**5. Results**

#### *5.1. Impact on the Perceived Usability of the ERP System*

Perceived usability is important for user satisfaction, and has a significant impact on the frequency of use. Many definitions of usability can be found, among others, the definition by Nielsen [42] and the definition provided in ISO 9241-11:2018 [43]. The ISO 9241-11:2018 [43] defines usability as "the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use". On the other hand, Nielsen [42] defined usability with the use of five attributes: learnability, efficiency, memorability, errors, and satisfaction.

Usability can be evaluated using a variety of approaches. An example is the System Usability Scale (SUS) [41], a widely-used questionnaire that is simple to implement and analyse. It was designed to gather the users' subjective usability evaluation [41]. The SUS questionnaire consists of ten statements, evaluated with a Likert scale, which have to be evaluated immediately after the use of the system, even before the debriefings or discussion [41]. As shown in the surveys [44,45], SUS is the most commonly used standard questionnaire for measuring the usability of the serious game approaches.

Since SAP ERP is a complex solution combining multiple modules and having the user interface loaded with data and input fields, its use for newcomers can be a demanding task. When playing the ERPsim business simulation game, students worked with a real-life traditional SAP GUI interface. The aim of using the simulation game approach for the introduction of ERP concepts was to raise the perceived usability of SAP ERP by changing the students' focus from the user interface to the game. Therefore, by using the SUS questionnaire, we answered the research question RQ1: "What is the perceived usability of SAP ERP, when introduced via a simulation game, from the perspective of an ERP newcomer IT student?"

The three-year SUS score is presented in Figure 5. The average SUS value in 2017 was 59.17; in 2018, it was 61.61; and, in 2019, it was 61.43. All three cohorts of students, altogether 53 participants, assessed the usability of SAP ERP with SUS scores around 60 out of 100. As presented in the box-plot in Figure 5, the minimal SUS value was 40.00 in 2017, and 52.50 in both 2018 and 2019. On the other hand, the maximum SUS value was 75.00 in 2017 and 2018, and 77.50 in 2019.

Although the SUS score ranging from 0 to 100 offers transparent information, the understanding of the results can be increased if an adjective is added to the score. An example of an adjective rating was proposed by Bangor et al. [46]. Based on the empirical research, they proposed the use of the following adjectives: worst imaginable (mean SUS score, 12.5; standard deviation, 13.1), awful (mean SUS score, 20.3; standard deviation, 11.3), poor (mean SUS score, 35.7; standard deviation, 12.6), OK (mean SUS score, 50.9; standard deviation, 13.8), good (mean SUS score, 71.4; standard deviation, 11.6), excellent (mean SUS score, 85.5; standard deviation, 10.4), and best imaginable (mean SUS score, 90.9; standard deviation, 13.4). Based on the gathered SUS scores, as presented in Figure 5, the students evaluated the usability of the SAP ERP as OK, with scores of 59.17, 61.61 and 61.43, respectively.

**Figure 5.** System Usability Scale (SUS) score of SAP ERP for three generations of students.

Since related works allowing a comparison were not detected, a comparison of results was made with a SAP usability study done in a different environment and domain. Danda [47] carried out usability research on 103 existing SAP ERP users, using the SAP Fiori user interface. As the results show, the average SUS score was 34.47 [47]. The results gathered within our research deviate significantly in a positive way, despite the fact that the interface used was a traditional SAP GUI. Participating students assessed the usability of SAP ERP as OK, while participants in the study by Danda [47] evaluated the usability of SAP ERP as poor. Due to the perceived differences, we can conclude that the introduction of ERP concepts using a simulation game results in a positive impact on the perceived usability of the SAP ERP system.

#### *5.2. An Empirical Survey of Students' Feedback on Using Simulation Game Approaches*

In addition to perceived usability, we also gathered students' opinions about the selected experiential learning approach, focusing on acquisitions. Table 2 summarizes the results of a three-year survey, providing the answer to research questions RQ2 ( "Do students perceive that the use of a simulation game to introduce ERP concepts results in new knowledge and skills?") and RQ3 ("Do students believe that an introductory simulation game affect their intent in future course engagement?").

The questionnaire gathered students' opinions and experiences after playing the business simulation game. Students evaluated the given statements using a five-point Likert scale. The answer 5 stands for strongly agree, 4 for agree, 3 for neither agree nor disagree, 2 for disagree and 1 for strongly disagree. The empirical analysis of the results is presented in Table 2. The results are displayed by year, showing an average value for each year, followed by a detailed analysis presenting the frequency of each response in percentages.

As the results indicate, students have a positive opinion about the use of a simulation game approach for introducing the basic functionalities of SAP ERP, and also for learning the basic navigation and use of controls in SAP ERP. All three cohorts evaluated the mentioned aspect with average values from 4.4 to 4.6 on a scale of 1 to 5. The majority of students confirmed that the use of a simulation game contributed to the development of the technical skills necessary for using SAP ERP. In 2018 and 2019, all of the students agreed or strongly agreed with the statement, while, in 2017, the percentage of agreemen<sup>t</sup> was 83.34%.


**Table 2.** Empirical results of three year students' survey.

When asked about the newly obtained knowledge and skills about SAP ERP use and collaboration and coordination between different roles, students in 2017 evaluated the statements with 4.5 and 4.3 out of 5, while in 2018 they evaluated both statements with an average of 4.7, and in 2019 with an average of 4.3 and 4.4. The statements looked into students' perceptions of obtained knowledge, which contributed to answering the research question RQ2. The results are summarized and presented visually in Figure 6. All three groups, without disagreeing, confirmed that the use of a business simulation game resulted in anticipated knowledge and skills connected to collaboration and coordination between roles and for the use of SAP ERP. With regard to the findings provided by Cronan et al. [36] that there is a significant correlation between self-assessed knowledge and objective measures, the gathered results, due to the self-assessment type of questionnaires, seem to provide credible and valid results.

**Figure 6.** Evaluation of obtained knowledge and skills and students' engagement.

Two of the statements in the survey evaluated the students' competitiveness and motivation to win. As presented in Section 2, digital natives respond well to game-oriented learning [18], wherein competition in a goal-oriented environment suits their characteristics [21]. The results are presented in Table 2. The students from 2017, with an average value of 4.1, evaluated that their goal was to obtain new knowledge and not to win. In 2017 and 2019, the average values were lower, i.e., 3.4 and 3.7, reflecting the competitiveness of participating students. This was confirmed additionally within the statement measuring the motivation of getting the best possible results in comparison to other teams. The statement was assessed with an average value of 4.6, 4.4 and 4.1.

The debriefing phase in each round, aimed at reflecting the gathered experiences and, based on this, reforming the strategies used in a business simulation game. As the results demonstrated, in 2017, only 38.89% of participants followed a certain strategy during the game. In contrast, in 2018 and 2019, the percentage increased to 78.57% and 61.91%. In 2017, 55.56% of students thought that the teams were willing to share gathered experiences with others, while the number in 2018 is significantly lower, namely 28.58%. However, the average result in 2019 was even higher than in 2017, 76.19%. The mentioned results coincide with the results connected to the usefulness of the exchanged information. The result was the lowest in 2018, with an average value of 3.1, and the highest in 2019, with an average value of 4.0.

Regarding communication within the teams, the results sugges<sup>t</sup> that the communication was good, with average values of 4.3, 4.9 and 4.7. This could be expanded upon with the findings presented by Hwang [48] that teamwork, where one of the dimensions is also interaction with teammates, has an impact on team satisfaction as well as on team effectiveness. The findings are based on the survey results, in cases where students were using the ERPsim business simulation game. Hwang [48] concluded that the used business simulation game is also suitable for teaching and learning teamwork.

The motivation for the use of new learning approaches could also be gathered from the statement where students evaluated their future intent for attending lectures due to the use of the simulation game. The statement contributed to answering the research question RQ3. The results are visualized in Figure 6. The average number increased through the years—3.2 in 2017, 3.9 in 2018, and 4.0 in 2019—showing the positive affect of using the simulation game in the ERP systems course. Since it is possible to detect a significant drop of attendance in lectures in recent years, the results show that the experiential learning approaches incorporated into traditional learning strategies could also increase the students' level of commitment and engagement.
