4.1. The Evaluation Process of the MR Training System
In this paper, the authors applied the MR to seismic retrofitting training. Every trainee wore Hololens2 to accept retrofitting training. The performances of different trainees in retrofitting operations were recorded. Finally, the recorded data and the trainee self-evaluation would be as parts of the original information for comprehensive evaluation. The evaluation data obtained from the MR seismic retrofitting training system are shown in
Figure 2.
The training scheme should be first determined before the comprehensive evaluation of the MR seismic retrofitting training. In order to improve the generalizability of this evaluating method, it is necessary to select a group with the same practical experience and education experience in retrofitting. Therefore, seven students were recruited for this study. Each participant’s current research was not related to seismic retrofitting, and none of them had work experience related to retrofitting. Before the training, the task description and self-evaluation questionnaire were prepared for participants. The purpose of issuing the task description was to explain the requirements of the experiment and briefly introduce the retrofitting tasks to be undertaken. The purpose of issuing the questionnaire was to support the participant in completing the trainee self-evaluation. Parts of retrofitting training process are shown in
Figure 3.
Marilyn Salzman et al. [
44] believe that the indicators influencing the training effect mainly include the features of the trainees, the features of the training system, the interactive experience and the learning experience. Since the participants recruited have the same education and retrofitting experience, this paper evaluated the MR seismic retrofitting training system from three aspects: the features of the trainees, the features of the training system, and the interactive experience. According to the research of Lee et al. [
45,
46,
47], the authors sorted out the indicators that affect the comprehensive evaluation results (as shown in
Figure 4).
The research of Pooya Adami [
48] shows that both qualitative analysis and quantitative analysis would be involved in the comprehensive evaluation of MR training. Therefore, this paper chose the quantitative-dominated hybrid method to analyze the evaluation indicators of the MR seismic retrofitting training.
For qualitative analysis, three aspects (the features of the trainees, the features of the training system, and the interactive experience) were considered in the comprehensive evaluation. For quantitative analysis, the weight of every evaluation indicator and the evaluation result were all expressed as a set of matrixes. The AHP [
49] is an analysis method that considers qualitative and quantitative data, so AHP was chosen as the analysis method in this paper.
The fuzzy comprehensive evaluation was used to evaluate the MR training system. According to the membership degree theory of fuzzy mathematics, the evaluation method can transport qualitative evaluation into quantitative evaluation. The quantitative evaluation results are expressed in the form of a matrix, which solves various non-deterministic problems [
50]. The specific comprehensive evaluation process is shown in
Figure 5.
4.4. Establish the Comprehensive Evaluation Model
The weight of each evaluation indicator can be determined after obtaining the judgment matrixes. In order to make the weight of each indicator more reliable and scientific, it is necessary to perform a consistency check.
The eigenvalue vector corresponding to the maximum eigenvalue of the judgment matrix is expressed as after normalization (the sum of all the is equal to 1), which means the influence of the comparison indicators on a certain indicator of the upper level. In other words, the is the weight of the indicator at the same level. The next step is to perform a consistency check. The consistency check refers to the allowable range of inconsistency for the matrix. Wherein the unique non-zero eigenvalue of the n-order uniform matrix is ; the maximum eigenvalue of the n-order reciprocal matrix is , if and only if , is a uniform matrix. The more is larger than , the more serious the inconsistency of the matrix.
CI is used to measure the consistency of the matrix. The smaller the
CI, the better the inconsistency. The greater the inconsistency of the matrix, the larger the judgment error caused. Therefore, the magnitude of the
value can be used to represent the degree of inconsistency of the matrix. The
CI is defined as:
where
is the eigenvalue, and
is the unique non-zero eigenvalue of the
-order uniform matrix.
When , the matrix is complete consistency; the closer to 0 the CI, the better the consistency; the closer to 1 the CI, the worse the consistency.
To measure the magnitude of the
CI, a random consistency index
RI is introduced:
The random consistency index
RI is related to the order of the judgment matrix. Generally, the larger the matrix order, the greater the possibility of a uniform random deviation. Considering that the deviation of consistency may be caused by random reasons, it is necessary to compare the
CI with the random consistency index
RI to judge whether the matrix has satisfactory consistency, and the comparison result can be defined as the coefficient
CR. The formula of
CR is as follows:
In general, if
CR < 0.1, the judgment matrix is considered to pass the consistency test. Otherwise, there is no satisfactory consistency, and the judgment matrix needs to be reconstructed. The excel template developed by Goepel [
53] was used in this paper to show the weight value and
CR value of each evaluation indicator. The weight values of all influencing indicators and
CR values of each judgment matrix can be obtained by calculating the numbers in the excel template. The calculation results show that the
CR value of each judgment matrix is less than 0.1, which indicates that the judgment matrix and analysis process are scientific and reliable. The detailed data on the weight values and
CR values of the evaluation factors involved in this study are shown in
Table 5.
As shown in
Table 5, the weights of the indicators in the decision level are as follows: the weight of the features of the trainees is 8.33%, the weight of the features of the training system is 19.32% and the weight of interactive experience is 72.35%. The weighted distribution indicates that the MR training system improvement should focus on the interactive experience, followed by the features of the training system. Through the discussion and analysis, the reasons for the weighted distribution of the decision layer indicators may be:
Interactive experience: the interactive experience occupies the most weighted distribution among the three indicators of the decision layer and is much more important than other evaluation indicators. MR includes VR and AR and realizes collaborative work based on immersive experience and virtual–real interaction. Collaborative work relies on the interaction between workers, environment and machines, which is reflected in the interactive experience of trainees.
As a unique feature of MR, collaborative work is also a very important function of seismic retrofitting. The seismic retrofitting task requires the cooperation of multiple workers, even the cooperation of workers and machines. When encountering difficult problems during retrofitting, it is also necessary to analyze and solve the problems with remote experts through the voice or video of the MR system. Collaborative work can maximize the advantages of each retrofitting worker through multi-person collaboration and can discover and even solve retrofitting mistakes in time through remote collaboration check of experts. Multiple people’s collaborative work not only ensures retrofitting safety of workers but also ensures retrofitting effect. Therefore, the weight of interactive experience is the largest.
The features of the training system: Among the three indicators of the decision layer, the weight of the features of the training system ranks second with 19.32%. The training features of the MR system are, of course, important, and a training system with strong learning ability and a better immersive naturally has great superiority in the training effect. However, from the perspective of the development of extended reality, VR has realized a great immersive experience with the virtual environment, and AR has realized the interaction between the virtual world and the real world. The training equipment for this research is MR equipment (Hololens2), which integrates the advantages of VR and AR. The related technologies about the subjective experience (such as immersive) have matured. Therefore, the features of the training system are not the most important and unique point.
In order to distinguish the differences between VR training and MR training and to better highlight the advantages of MR, the influence of the common features of VR, AR and MR on the comprehensive evaluation are deliberately weakened. Therefore, the weight of the features of the training system in the comprehensive evaluation ranks second with 19.32%.
The features of the trainees: Among the three indicators of the decision layer, the weight of the features of the trainees is the least, only 8.33%. The goal of this study was to evaluate the MR seismic retrofitting training system, which focuses on the comprehensive evaluation of the training experience and the effectiveness of the training system. Therefore, the features of the training system and interactive experience are more important than the features of the trainees. For the effectiveness of the training system, the trainees in this paper have the same ages, theoretical basis and retrofitting experience; there is little difference in trainees’ features. Thus, the weight of the features of trainees in the comprehensive evaluation is the least.