4.1. Summary of the TROFLEI Results Evaluation
The overall summary of the TROFLEI results data obtained from the experiments across all the underlying scales in each section is shown in
Table 7. Note that a total of 227 responses from participants are recorded for this dataset. The focused summary of this dataset on the Cronbach’s alpha value α for the reliability and mean values is shown in
Table 8.
As shown in
Table 8, the Cronbach’s alpha values as the results derived from the experiments are displayed for all the underlying scales in both the ‘Actual’ and ‘Ideal’ sections. For ‘(A) Student Cohesiveness’, the Cronbach’s alpha values for the ‘Actual’ and ‘Ideal’ are 0.88 and 0.87, respectively. Both values fall under the ‘Good’ category, according to the rating classifications in
Table 6. For the remaining seven underlying scales, all the sections have a Cronbach’s alpha value above 0.9, therefore categorizing them under the ‘Excellent’ category, according to the rating classifications in
Table 6. As all the eight underlying scales are well above the threshold for the ‘Good’ category, the questionnaire is deemed very reliable.
The mean values for each of the underlying scales, split into the ‘Actual’ and ‘Ideal’ sections, are also shown in
Table 8. As expected, for all the underlying scales, the mean values in the ‘Ideal’ section are higher than the mean values in the ‘Actual’ section. This is mainly due to the ‘Ideal’ section being referenced to the perfect VARTeL environment for each underlying scale, adding up to a perfect VARTeL environment that is able to effectively allow students to learn focused concepts flawlessly. It is aligned with the same trends as those of the observations in [
34,
35], with small differences of the Cronbach’s alpha values for the underlying scales.
As observed in the ‘Ideal’ section, the mean values range from 3.88 to 4.30, showing that students believe that various environment scales are effectively applied in an ideal VARTeL environment. For example, the highest mean value in the ‘Ideal’ section is 4.30, attached to ‘(F) Equity’. This shows the description of ‘The extent to which students are treated equally by the teacher in the VARTeL environment’ is extremely important in an ideal VARTeL environment.
The other underlying scales with mean values above 4.00 in the ‘Ideal’ section are ‘(D) Student Cooperation’, ‘(E) Differentiation’, ‘(G) Creativity’, and ‘(H) VARTeL’. Two of the more significant factors would be ‘(D) Student Cooperation’ and ‘(H) VARTeL’, as both can be further looked at as collaborative learning, which is beneficial to students’ learning efficacy. Through this, more activities focusing on collaborative learning should be introduced and implemented in the future VARTeL environment.
‘(E) Differentiation’ and ‘(G) Creativity’ can be linked to individual ability and imagination, respectively, where both can be explored in future VARTeL environment activities to expand on innovation. These activities can be tied with skills to train up future entrepreneurs, which are not only important in the engineering climate but also the entire society.
The other three underlying scales with values below 4.00 are ‘(A) Student Cohesiveness’, ‘(B) Student Involvement’ and ‘(C) Student Investigation’, with the lowest being ‘(B) Student Involvement’. But the lowest value, 3.88, is still relatively high on the scale, which is considered effective in the learning environment. ‘(B) Student Involvement’ and ‘(C) Student Investigation’ are tied to participation and problem solving, respectively, which are both important in the workplace and as an individual. They might be lower as the VR activities are currently more of a simulated environment rather than solving problems.
‘(A) Student Cohesiveness’ is mainly tied to students being knowledgeable and supportive of their classmates in the learning environment, which can be tied to team cohesiveness in a workplace after graduation. This can be lower as currently the VR activities are all individual activities, which results in no interaction among the students using the VR headsets. However, with the three underlying scales being lower than 4.00 and showing effectiveness over the others, this could either translate to a lesser importance or a need for improvement.
The mean values from the ‘Actual’ section ranging from 3.78 to 4.27 are used to compare the mean values in the ‘Ideal’ section. Similarly, five of the underlying scales from the ‘Actual’ section have a value above 4.00, while the other three have a value below 4.00. As explained above, the noticeable factor is that the mean values in the ‘Actual’ section are lower than in the ‘Ideal’ section. The smallest difference between the two sections is 0.03 in the ‘(F) Equity’, while the biggest difference is 0.11 in ‘(H) VARTeL’ and ‘(B) Student Involvement’, with the minor differences showing that the current VARTeL environment is comparable to what students have in mind. However, as the mean values in the ‘Actual’ section are comparable to the mean values in the ‘Ideal’ section, this shows that the students feel that the current VARTeL environment is efficient in delivering concepts in the classroom. Even so, there are still negative differences when comparing the mean values to the ‘Ideal’ section, which means that the VARTeL environment is still below what students expect and there is room for improvement. An argument could also be made that the VARTeL environment should be enhanced to make students believe the ideal environment can do much better than what they are currently expecting.
As there are 10 questions on the VARTeL environment and VARTeL academic in the pre-test and post-test questionnaires, a maximum score of 10 can be achieved in the Total Sum. A maximum score of 4 and 6 can be scored for the VARTeL environment sum and VARTeL academic sum, respectively, for both the pre- and post-tests.
4.3. Limitations and Future Works Recommendations
Although a pre-test and post-test were used to gauge the students’ academic knowledge for this experiment, the VARTeL questionnaire only consisted of a total of 10 questions and was not able to effectively gauge students’ academic improvement due to the VR games. Four out of ten questions were also specific to the VARTeL environment rather than the concepts of the VR games. The remaining six questions were split into three parts for each VR game, with only two questions pertaining to the concepts taught, which was insufficient when used to gauge the academic achievement of the students.
A focus group can be introduced outside of the class, where engineering students can be asked to take part in a paid or volunteer study. This focus group would go through the same concepts taught in the VARTeL environment but rather than going through the VR games, the students in the focus group would go through a lesson using a pen and paper. Other environments could also be tested to determine how effective a VARTeL environment is when compared to others. Examples such as ‘Actual Hands-On’, ‘Laboratory Pilot Plants/Simulations’ and ‘Computer Simulated Exercise’ are all environments that can be compared to determine the effectiveness of the VARTeL environment. Rather than just using the resultant TROFLEI dataset and VARTeL dataset, more specific questionnaires targeting academic achievement can be used in the future for comparison of how effective the VARTeL environment is. One example would be implementing the study of self-efficacy alongside the TROFLEI and determining any correlation between the two questionnaires. Another example would be having a grading factor outside of having questionnaires. This grading could be performed in the VARTeL environment through activities where students have to fulfil certain conditions, showing that the students have effectively picked up the concepts that are taught during the VR games. By having a grading factor for each individual student, a more effective correlation could be plotted to determine the effectiveness of the VARTeL environment.
The VR games can be changed to more engineering-focused concepts rather than STEM to determine the effectiveness of the VARTeL environment in imparting engineering concepts. Also, as the students in this study were all from an engineering background, the majority of the feedback received was that the students would like to see activities more focused on the engineering concepts, such as engines, manufacturing plants and CAD, rather than STEM concepts. By changing the VR activities to be mainly focused on engineering concepts, a clearer picture can be painted of the effectiveness of a VARTeL environment in an engineering classroom.