1. Introduction
Virtual reality (VR) involves creating and implementing a simulated, realistic, three-dimensional environment [
1]. In other words, diverse virtual environments can be constructed in limited spaces by generating realistic images, sounds, and other sensations. Since environments generated by VR devices are similar to the real world, they have been used in various fields, especially as treatment options in hospitals. For example, VR devices have been used for social-adaptation training for social phobias, as well as for treating post-traumatic stress disorder (PTSD) [
2]. In addition, many researchers have utilized VR devices during their experiments to create environments and observe the corresponding responses [
3,
4,
5]. For instance, electroencephalogram (EEG) signals were measured in a VR environment, which are composed of three different traffic light situations (red, green, and yellow), and EEG signals were well discriminated according to the traffic light color [
6]. In another study, the anxiety was triggered by VR environment using stressful job interview situation and analyzed the changes in the cardiovascular activity [
7]. As a result, the cardiovascular change of people who had been trained in VR environment was less than that of untrained people. Taken together, the physiological signals seem to be well changed and modified with artificial VR environment.
Generally, stress is induced by physical, mental, or emotional tensions and causes changes in the body’s response. In addition, since VR environment can trigger unfamiliar stimuli for subjects, several phenomena such as the anxiety, mental concentration and nausea are regarded as stress in a broad sense. Especially, physiological changes can be caused by unfamiliar external environments and psychological changes [
8]. Among the factors that can induce physiological changes, in particular, the autonomic nervous system (ANS) is important in regulating the functions of the internal organs and maintaining homeostasis, as well as human physiological activities [
9]. Also, external factors including stress can affect ANS function, so that physiological phenomena and changes could diversely appear according to the levels of perceived stresses [
10].
The ANS, when affected by stress stimuli, secretes stress hormones such as cortisol and adrenaline within the blood vessels. This causes the activation of sympathetic nerves and the inactivation of parasympathetic nerves [
11]. As the result, physiological responses can appear in the body [
12]. For example, there is an evidence that cognitive loads can affect the cardiac function [
13]. Given that heartbeats depend on ANS activity, cardiac activity after tasks with cognitive loads can also be related to ANS changes.
Heart activity, skin sweating and skin temperature are known to be regulated by the ANS, so biosignals related to such activities (photoplethysmograms (PPG), electrodermal activity (EDA) and skin temperature (SKT)) could provide insights on ANS activity.
The PPG signal is measured from the finger, and reflects the change in blood volume in the peripheral blood vessels [
14]. Since the changes in blood volume are associated with cardiac activity, the peak positions of the PPG signal are similar to the R-peak positions of QRS complex in an electrocardiogram (ECG) [
15].
The heart rate variability (HRV) can be calculated from the intervals between the peaks of the PPG signal [
16]. The HRV has been used to investigate changes in the ANS, as well as diagnosing heart disease. In particular, it is noteworthy that several HRV parameters (e.g., the ratio of high-frequency and low-frequency (HF/LF) powers) are associated with the activities of sympathetic/parasympathetic nerves [
17].
The EDA is an electrical signal measuring continuous skin-conductance (SC) changes [
18]. In general, changes in SC are associated with sweat gland activity. Whenever sweat glands secrete sweat through the pores, SC peaks are generated. In particular, the amplitudes and frequency of SC peaks are related to the activation of the sudomotor nerve, which is part of the ANS [
19].
Lastly, changes in SKT are also associated with changes in the ANS. For instance, it is known that the combined inhibition of dopamine (DA) and norepinephrine (NE) reuptake by the activity of sympathetic nerves improves exercise performance and increases body temperature [
20]. Taken together, the three physiological signals (PPG, EDA, and SKT) are associated with the activities of the ANS. Considering that ANS changes can be induced by stressful tasks with cognitive loads, we can postulate that the features obtained from measured signals can reflect not only changes in the ANS, but also the stress levels [
17,
18].
Recently, mobile healthcare system has been studied for detecting stress levels using various wearable sensors measuring the physiological signals [
21,
22]. The newest developed wearable sensors can continuously measure the physiological signals including PPG, EDA, and SKT, so that the users can check their health state without any inconvenience [
23]. Mobile devices such as Galaxy gear and Apple iWatch are appropriate to be utilized in this field. Nowadays, most mobile device has high-performance microprocessor. With advances in engineering technology, the performance of microprocessors embedded in mobile devices has been improved and complex mathematical problem can be calculated as well. Eventually, classification methods that require the complex computation can work with the microprocessor embedded in mobile device.
Automatic classification methods using features extracted from biosignals have been developed [
24,
25,
26]. In general, conventional neural network algorithms are used to estimate optimal boundaries for separating distinct classes and also to perform iterative processes to obtain the optimized weights and biases of each layer. However, a long and complex iterative process is required to utilize large datasets, which include extensive biological information [
27]. Moreover, it is difficult to be operated on microprocessor having limited resources. To solve this problem, the extreme learning machine (ELM) was developed.
Since the ELM is based on single-hidden–layer feedforward neural networks (SLFNs), and randomly selects the input weights and biases for the hidden layer, it has low computational complexity as well as high classification accuracy [
28]. In addition, the kernel-based extreme learning machine (K-ELM), an extended version of the ELM, is more robust than the ELM, with smaller computation time [
27,
29]. In other words, the K-ELM uses fewer resources while showing high performance compared to other neural networks. Therefore, K-ELM algorithm can work in a microcontroller chip with a small memory.
In this study, a stressful task with cognitive loads was designed to measure stress-related biosignals in a VR environment. The task was composed of five sequential sessions with varying stress levels: baseline, mild stress, moderate stress, severe stress, and recovery. During this task, physiological signals (PPG, EDA, and SKT) were measured simultaneously. Then, we classified these five different states by combining K-ELM and the features obtained from the three physiological signals. Additionally, we evaluated whether the calculated features reflect intended stress levels, and compared the performances of the conventional machine learning algorithms with those of the proposed algorithm.
4. Discussion
As indicated in
Figure 1, we conducted a stress-inducing experiment while simultaneously measuring several physiological signals (photoplethysmogram (PPG), electrodermal activity (EDA), and skin temperature (SKT)). Physiological signals are regulated by the autonomic nervous system (ANS), and physical or mental stress affects the activity of the ANS [
3,
37]. In addition, there is evidence that the characteristics of the physiological signal measured before the stress task differ from that measured after the stress task [
13]. Therefore, we can expect that the stress states including two resting conditions could be well-discriminated by using the physiological signals. Accordingly, we implemented an automatic classification method to classify the stress levels. The overall feature preparation and classification processes are described in
Figure 7.
To evaluate the proposed method combining kernel-based extreme learning machine (K-ELM) with various features, we followed these three steps.
Firstly, we collected data related to the subjective stress levels from the subjects (STAI Y-1 and VAS). From the results, we confirmed that the subjective stress levels were correlated with the intended stress levels in our task. For instance, most of the subjects produced low scores on MIS-S and high scores on SES-S. In other words, our VR experiment could trigger the intended stress levels, and the physiological signals were measured under the intended stress conditions. Furthermore, we performed one-way ANOVA followed by the Tukey’s HSD and post-hoc comparisons to evaluate the difference of STAI-Y1 and VAS scores from three stress stimulation. As a result, the effects of three stress stimulation are significantly different in the STAI-Y1 (F-value: 172.29, p < 0.01) and VAS (F-value: 274.06, p < 0.01), respectively.
Next, we evaluated the classification accuracies of K-ELM while changing the input features—HRV, SC, SKT, and integrated feature (IT). In general, the performance of the conventional machine-learning algorithm was governed by the corresponding parameters [
27]. Since the K-ELM performance also depends on the parameters (kernel size γ and regularization coefficient C), we applied various values to K-ELM to find optimal parameters.
Figure 4 shows that the optimal parameters differ depending on the type of input feature. In other words, different parameters must be used to accurately evaluate the performances in each classification.
Table 3,
Table 4,
Table 5 and
Table 6 describe the averaged classification accuracies in HRV + K-ELM, SC + K-ELM, SKT + K-ELM, and IT + K-ELM, respectively. Except for SKT + K-ELM, the averaged classification rates of each classifier were more than 90%. In particular, the averaged classification rates of IT + K-ELM were the best (more than 95%). However, the averaged classification rates of SKT + K-ELM were approximately 77%. This implies that HRV or SC could reflect the stress-related ANS changes while SKT was not sufficiently able to show that. In fact, this result is consistent with previous studies [
18,
37].
It should be noted that the IT features showed the best results, which implies that the relationships among the features are also important for classifying the stress levels. The low classification results of the SKT feature might be because it is difficult to measure the core body temperature, which depends on the ANS [
20].
It is also noteworthy that we successfully discriminated two resting conditions, i.e., BA-S and RE-S, even though we provided the same conditions except for the task sequences. There is evidence that it takes some time for a physiological signal to return to its original state after a stressful condition [
13]. Our results could be interpreted in line with this point of view. However, high classification rates may be due to the sequence of experiments in this study. After each stress-inducing experiment, all subject had approximately 10 min of rest, but the subjects may not have been completely free from the effects of stress stimuli from previous session. This could be a limitation of our study. Therefore, in order to identify only the effect of the intended stress stimuli, it is necessary to design the experiment more elaborately, and we should use counterbalanced experiment order in future study. Furthermore, the effect of the previous stimuli on the next session might be more noticeable when a participant used a VR device for the first time. In general, unfamiliar VR environment may trigger nausea to beginners as well. Although, all subjects are approximately 25 years of age and very healthy, it might be difficult to exclude the unwanted additional stress factors. Therefore, it is possible that we have induced broader concept of stress besides our intended stress.
Figure 5 shows that the proposed method (IT + K-ELM) has the most excellent classification performance while using the minimum CPU resource allocation and memory usage during computation. In particular, computation time in
Figure 5a depends on the CPU resource allocation and mathematical complexity. Among them, multi-class support vector machine (mSVM) and kernel-based multi-class support vector machine (K-mSVM) had taken quite longer computation time compared to other classifiers such as linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), ELM and K-ELM. Basically, since the support vector machine (SVM) and kernel-based support vector machine (K-SVM) are binary classifier, the optimized multi-class solution can be obtained by repeatedly calculating the binary classifier problems. For instance, to obtain the k multi-class solution, binary classifiers should be calculated
k(
k − 1)/2 times [
43]. Therefore, the mSVM and K-mSVM can use many CPU resources due to iterative process. On the other hand, although computation times of LDA and QDA are faster than mSVM and K-mSVM, the ratios of memory usage are higher than mSVM, K-mSVM, ELM and K-ELM as in
Figure 5b. Basically, to obtain the discriminant hyperplane, the solutions of LDA and QDA are derived using the inverse matrix operations as well as eigenvalue and eigenvector. In particular, this process requires many memory usages [
44].
In addition, computation time of ELM is higher than that of K-ELM since the solution of ELM requires the calculation of the hidden node output matrix H in (8). Therefore, whenever the number of hidden node increases, the computation time also rises. Instead, K-ELM converts from
to kernel matrix
in (10) and the kernel matrix
is determined by users. Thus, computation time of K-ELM can be reduced compared to that of ELM.
Figure 5c describes that the proposed method has very small error rate despite low memory usage and short computation time. Taken together, our proposed algorithm (IT + K-ELM) is suitable for microprocessors embedded in mobile system and have excellent performance as well.
Finally, the organic relationships among the features were evaluated using self-organizing maps (SOMs). The shapes and distributions of the clusters in
Figure 6(a-ii,d-ii) look very similar. This implies that the HRV was a dominant feature in the IT features, and other signals might have played some supportive roles in our classifications. In particular, the clusters in
Figure 6(d-ii) were relatively well-organized into five clusters around the center of the map, in comparison with the clusters in
Figure 6(a-ii). On the other hand, the shapes of the clusters in
Figure 6(b-ii,c-ii) were quite disorganized and their positions were biased to one side. It implies that the SC and SKT features had an ambiguous characteristic with respect to stress-level discrimination.
It is also noteworthy that the SOM results were calculated from all subject data without group distinction. It implies that the clusters trained by the SOM algorithm exhibited characteristics reflecting the common stress levels of all subjects, and uniformed clusters indicate that the input features well represented the stress levels. According to our results, the SC and SKT features had high person-to-person variability, so the classification accuracies might be lower when we used them separately. When we integrated every feature, the classifier showed better performance, suggesting that feature integration partially improved the accuracy.
Lastly, we performed the statistical tests such as one-way ANOVA and post hoc test for investigating effects of each extracted feature in three stress sections (MIS-S, MOS-S and SES-S). As a result, a few features extracted from each physiological signal were significantly different (
p-value < 0.05). There are features of PPG (
,
and pNN50), EDA (
and
) and SKT (
). Especially, it is known that
,
and pNN50 can be changed whenever physiological or psychological changes of human occur [
37]. Besides,
and
are related to the ANS activities and the changes in ANS are induced by psychological and physical stress [
19]. In this experiment, since we restricted movements of the subjects, the changes in features were only triggered by stress stimulation.
In statistical analysis, the averages of
F-values, except for statistically significant features, were 10.55 (PPG), 0.77 (EDA) and 2.85 (SKT). Especially, the average of
F-value in PPG was higher than results of EDA and SKT. The higher
F-value is, the more significant difference between the groups is. Therefore, the features calculated by PPG have a close relationship with the stress state. As a result,
Figure 6a drawn by features of PPG was relatively well- organized into clusters compared to
Figure 6b,c which were described by EDA and SKT, respectively. Besides, classification rate in
Table 3 was higher than those in
Table 4 and
Table 5 since distinction of stress state was well expressed in features of PPG.