A Practical Model of the Application of Information Technology in Various Fields of Online Education during the COVID-19 Pandemic: Mechanical Engineering, Traffic, Informatics and Statistics, Accounting and Auditing

Abstract

This paper deals with the issue of education during the COVID-19 pandemic in Serbia and Bosnia and Herzegovina and its effects on education development gained during that process. This topic is trending around sustainable education in urban and rural areas also including widespread areas and locations of students and faculties. In this paper, we present a model that was created in the early days of the COVID-19 pandemic and is functional even today. The model is supported on a website and it became a foundation for adding a great deal of material and solutions for more prosperous results in education on several faculties. The key findings imply that in the initial stages of the COVID-19 pandemic with the system establishment of lectures and examinations, the level of adopted skills improved significantly and exceeded those before the COVID-19 pandemic regarding the passing of exams and average grades on subjects. The ANN model was developed, which provides results in terms of successfully passing knowledge tests and average grades by subjects. The innovativeness of the model is a combination of input and output parameters supporting the possibility of its wide application in various branches of science, which resulted in intense application of this model in numerous courses.

1. Introduction

The COVID-19 pandemic has placed a lot of challenges in front of the entire world education system. Sudden and unexpected changes have affected nearly 1.6 billion students in more than 200 countries [1].

The example we present to you is regional and includes five universities in two countries of the Western Balkan region, the Republic of Serbia and Bosnia and Herzegovina. A total of 48 subjects are included in this course, which are lectured at nine faculties (five universities) and at all three levels of education (bachelor, master’s, and PhD studies). The number of teaching staff that participated in this newfound solution is 10 and the number of students who used the web platform reached 2900 in the two-and-a-half-year period. The domain visits in the highest frequencies resulted in more than 10,000 sessions per month, but it is still very high today, with a tendency to grow due to the advantages it provides, such as the possibility for students to become informed, listen to lectures, and to conduct trials and other knowledge tests at any given moment, which was not the case prior to the COVID-19 pandemic.

On the part of the teaching staff, multiple advantages have been created to automatically evaluate the tests, to facilitate data acquisition, and to improve the selection of questions on the tests in order to improve the quality of the adopted material.

Motivated by the initial successes in transferring knowledge through an Internet model based on a website, teachers developed the idea of improving the quality of the adopted knowledge led by the experience of earlier research related to the creating of various solutions through ANN (Artificial Neural Network) [2,3,4,5,6,7,8,9,10]. This experience was avant-garde, but it provided great opportunities, and, due to the positive feedback in the educational community, an increasing number of professors from various universities and faculties commenced to apply this novel solution to their subjects. Personal contacts of educators and acquaintances enabled the system to be more and more widespread and even the internationalization of the model transpired that educators and students were delighted. This originated solid ground to conduct research on a given topic and present the results of such work.

This can be explained by the fact that one smaller group of students was able to adapt to the new circumstances more rapidly, while the other larger group did so with a certain delay necessary to understand the new circumstances [11].

This paper was organized in the following manner. In Section 2, the authors have summarized the existing literature regarding the education during a pandemic that is related to the conducted research. In this section, a list of universities, faculties, and subjects were presented, where the platform with the ANN model for the selection of questions examinations was applied. Section 3 presents the online model itself, the web platform and its organization, ways of presenting knowledge, information, testing, automatic evaluation, etc. Additionally, the ANN model for the selection of questions for tests was presented. Section 4 displays the results of this way of working in terms of successfully passing knowledge tests and average grades by subjects before and during the pandemic in the period before and after the application of the ANN model.

2. Literature Review

Although there have been overwhelming challenges for educators, schools, institutes, and the government regarding online education from a different angle, there are several opportunities created by the COVID-19 pandemic for the unprepared and the distant plans of implementing an e-learning system [1,12,13,14].

Wargadinata et al. [15] conducted a study on the online learning process in the initial stage of the COVID-19 pandemic. This qualitative descriptive study included data obtained from observations, questionnaires, online interviews, and documentation in order to develop an online learning model. According to the obtained results, the WhatsApp Group emerged as the most effective among other online learning models. It is a familiar social media to all lecturers and students and through WhatsApp accounts students could communicate and share important files and learning resource links. The authors encourage dealing with obstacles such as limited internet access and level of digital literacy of students, as well as the development of other media for online learning as directions for future research.

Chakraborty et al. [11] studied the opinion of undergraduate students from one Indian university on different aspects of online education during the actual pandemic. Based on a questionnaire with 20 statements, the students provided their responses and descriptive results show that students indicated there are more advantages to offline learning in a physical classroom compared to online education. Additionally, a minority of students felt that online education is a better option compared to massive open online courses. However, they felt that professors improved their online teaching skills since the pandemic emerged.

Pokhrel and Chhetri [1] provided a comprehensive literature review on the impact of the COVID-19 pandemic on teaching and learning across the world. They highlighted certain weaknesses of online education such as online teaching infrastructure, exposure of teachers to online teaching, internet bandwidth, suitable pedagogy for online teaching and learning, platforms for higher education, and others. They encourage teachers and learners to continue using online tools in order to improve their teaching and learning in the future.

A survey by Adedoyin and Soykan [16] presented the challenges and opportunities of online learning. They pointed out that some universities experienced a crisis-response migration process toward digital transformation as a result of the pandemic. This caused the emergence of several key challenges emphasized in the study, such as dependence on technological infrastructure, digital competences, socio-economic factors (educational inequality), human and pets’ intrusions during online lessons, assessments and supervision, heavy workload, and compatibility.

Asgari et al. [17] conducted an observational study to identify issues and challenges that emerged after the transition to an online engineering education system during the COVID-19 pandemic. The extensive study was conducted at California State University and included 110 faculty members and 627 students from six engineering departments. Various logistical and technical issues regarding students and faculty were identified. More than half of the students indicated difficulties such as un-engagement, fatigue, and lack of focus during online classes and a lack of clear guidance or communication from instructors. Around half of the students experienced disconnection from their peers, technical difficulties, or a lack of technological equipment. Interventions regarding strategies for institution administration, engineering faculty, and engineering students were proposed.

Pirrone et al. [18] studied the differences between online distance learning and traditional face-to-face learning by focusing on learning strategy, metacognition, and attitudes toward school for Italian high school students. According to the questionnaire completed by 324 students, the results indicate that metacognition and study condition, as well as the attitude toward school, favors the face-to-face setting, while the learning strategy condition emphasized the use of subsidies and study flexibility as the benefits of online learning.

Sutarto et al. [19] performed qualitative research to understand and explore the teaching strategies at the Integrated Islamic Elementary School in Indonesia in order to increase students’ learning interests and responses in online learning during the pandemic. In that sense, the authors discussed several strategies such as providing understanding and attention to students by communication between teachers and students, as well as teachers and parents, then preparation of brief, clear, easy to understand, and interesting learning materials, the selection of simple and attractive social media platform such as WhatsApp, and conducting regular and continuous evaluations. The students provided different responses regarding the learning process during the pandemic; some saw fun and creativity, while others saw lack of connection with school friends during learning and playing.

Harvey et al. [20] proposed two goals in their study that are driven toward behavior change on individual and population levels to cope with the COVID-19 outbreak. Firstly, eight elements of habit formation that are highly relevant to prevention behaviors are considered. Those elements are identifying and addressing incorrect beliefs, setting goals, devising an action plan, establishing contextual cues, adding reinforcement, engaging in repetition, aiming for automaticity, and recognizing that change is difficult. The second goal was to propose strategies as part of behavioral intervention in order to promote habit formation for COVID-19 prevention behaviors and to break the bad habits that might spread the virus.

Lee et al. [21] examined the relationships among self-efficacy, task value, and the use of self-regulated learning strategies by massive open online course learners from a social cognitive perspective. The study showed that self-efficacy, as well as task value, are positively correlated to self-regulated learning strategies, and represent strong predictors of the use of self-regulated learning strategies. The authors suggested that learners can be provided with support to improve their self-efficacy and task value regarding massive open online courses in order to perform self-regulated learning strategies more frequently and efficiently.

Xie et al. [22] performed a review on the importance of online education due to the COVID-19 pandemic, which became a new normal, and the important role online education would have in the future. The authors exposed several advantages that increase the popularity of online education, such as flexibility, information accessibility, global reach, equity, and innovation. On the other side, major drawbacks are also highlighted, such as network instability and technological constraints, lack of a sense of belonging and connectedness, presence of distractions, and lack of engagement. As the future of online education, authors claim that online education will become an integral component of education after the pandemic meaning that the close connection between e-learning and traditional offline learning will be maintained.

In their survey, Lemay et al. [23] reported student perceptions of online learning before, during, and after the transition at a single college in Northeastern North America. Various contextual and institutional factors affected student perceptions and, even though a successful transition was achieved in terms of academic outcomes, the students reported high levels of stress and anxiety, as well as many social and instructional challenges during online lessons.

Adnan and Anwar [24] examined the attitudes of Pakistani undergraduate and postgraduate students toward compulsory distance learning courses during the COVID-19 pandemic. A total of 126 higher education students participated in the online survey and according to data analysis, the authors concluded that the majority of Pakistani higher education students do not consider online learning as effective as learning in conventional settings. The limitations in terms of internet access, interaction with students and instructors, and ineffective technologies as well as monetary issues seem to be the major challenges faced by students.

Pirrone et al. [25] studied the differences in anxiety, mental states, and metacognitive awareness of students toward mathematical subjects for both distance learning and in-person learning during the second wave of COVID-19. A total of 405 students from 25 middle schools in the Catania province in Italy participated in an online survey. More than 80% of the students prefer in-person math classes rather than distance math classes. However, less mathematical anxiety was experienced by the students during distance learning. On the other side, less anxiety but better mental states and metacognitive awareness resulted from in-person learning conditions. The authors conclude that the way of teaching mathematics is not as important as the motivation and dysfunctional beliefs, which are key issues that should be faced to develop students’ motivation, self-perception, and self-esteem.

Muthuprasad et al. [26] examined the perception and the preference of agricultural graduate students from different universities in India toward the online education caused by COVID-19. According to the results gathered after surveying 307 students, the majority of the respondents showed positive attitudes and expressed that they felt the online education was advantageous. Their preferences were driven toward recorded classes and quizzes after each class that would optimize learning efficiency, with well-structured content and flexibility as the main benefits of online learning. Similar to other studies, connectivity issues and technological constraints emerged as one of the main drawbacks of online classes compared to traditional classrooms.

Chen et al. [27] conducted a study to analyze the user satisfaction of six online education platforms used in China. The authors collected data from online and offline users using a questionnaire survey and web crawler technology. A back propagation neural network was employed to predict user satisfaction with online platforms and high prediction accuracy was achieved. According to the data analysis, user satisfaction is not influenced by personal factors as it is influenced by online education platform availability. The authors claim there are several issues referring primarily to technological problems, such as the design environment, platform customer service, network problems, etc., and that two-way interaction issues should be considered to improve the quality of online education.

Junus et al. [28] performed survey research to evaluate lecturer readiness for online classes during the COVID-19 pandemic. After analyzing both quantitative and qualitative data from lecturers coming from universities in Indonesia, it can be noted that lecturer readiness for online teaching was on a relatively high level, although they showed a medium level of e-learning before the pandemic, which alleviated these challenges. The main barriers the lecturers faced regarding online education were unstable internet connections and self-management issues.

Aydoğdu [3] used artificial neural networks to predict the performance of university students participating in an online learning environment considering several input variables. Different parameters, such as a number of neurons in the hidden layer, optimization algorithm, batch size, and epoch, were used to achieve a high prediction accuracy score. The attendance number of live sessions, the number of attendance to archived courses, and the time spent on the content left a larger impact on the output compared to other variables.

Bernard et al. [6] introduced LSID-ANN, a novel approach using artificial neural networks to identify students’ learning styles based on the Felder-Silverman learning style model (FSLSM).

Rivas et al. [10] applied several automatic learning methods to publicly available data sets, including tree-like models and different types of artificial neural networks. The study was aimed to predict master students’ results and grades and determine the factors that affect their performance.

Lau et al. [8] combined conventional statistical evaluations with artificial neural networks to model and predict students’ academic performance. The Levenberg–Marquardt algorithm was used as the back propagation training rule for neural networks. According to the overall results, ANN showed a good prediction accuracy of 84.8%.

Hamadneh et al. [7] proposed the combination of statistical analysis to identify factors that affect students’ performance and ANNs to predict their academic performance in the blended learning environment. The four factors: mid-exams, assignments, attendances, and virtual/face are considered to test the effects on students’ performance. A novel multilayer perceptron neural networks (MLPNN) was introduced to predict students’ academic performance. The appropriate training was achieved with an FFA swam intelligence algorithm.

Baashar et al. [4] emphasized the application of ANNs to predict students’ academic performances. They also pointed out the utilization of ANNs in combination with data analysis and data mining algorithms, which will allow researchers to evaluate their findings in assessing academic achievements.

Alsariera et al. [2] reviewed the accuracy of six machine learning algorithms: decision tree, artificial neural networks, support vector machine, K-nearest neighbor, linear regression, and Naive Bayes when predicting students’ academic performance in the period 2015–2021. Considering key features for predicting students’ academic performance, the ANNs demonstrated an excellent accuracy, followed by a decision tree, K-nearest neighbor, and Naive Bayes.

Baylari and Montazer [5] developed a personalized multi-agent e-learning system to estimate the ability of learners using item response theory to present personalized adaptive tests and an artificial neural network to diagnose a learner’s learning problem and recommend the appropriate learning material. The personalized and appropriate material recommendations can be provided with high precision using the proposed system.

Özbey and Kayri [9] proposed multilayer perceptron and radial-based function ANN methods to model the factors that affect transactional distance levels of university students who continued distance learning due to the COVID-19 pandemic. The authors collected research data from more than thousand students from various universities in Turkey and the results showed that multilayer perceptron ANN method provides better results compared to other ANN methods. The speed of the instructor to provide feedback on messages was shown as the most effective variable on transactional distance.

The ANN model proposed in this paper was developed by the first author and implemented in other fields [29,30], which was the inspiration to perform research on the possibility of the ANN being a helpful tool or selecting questions for tests on its own.

Considering the aforementioned, the authors, in order to facilitate the education of students, introduced a web solution for online learning as an option. With the outbreak of the pandemic, this proved to be a requisite solution, which through personal contacts, became increasingly popular in the local university community; therefore, the number of users expanded significantly. A particularly interesting solution is the specific solution of question selection on examinations based on ANN.

After developing the web platform-based model and the COVID-19 outbreak, there was an evident decline in points at final examinations and in average grades also. The hypothesis of this research was that the examinations based on ANN will incentivize the students to adopt knowledge on a higher level, which would result in higher examination scores and a higher average of grades in contrast to the period before this model was introduced. The periods before the ANN model are divided into pre-COVID-19 and post-COVID-19 periods.

The main contributions and innovations of this study are as follows:

• The development of a practical web platform for online education based on Moodle, JavaScript language, Excel models, Google tests, and ANN.
• To stimulate student’s motivation to work harder, the ANN model was designed to provide each student with individual questions that can be completed with the least probability to raise students’ awareness about the importance of adopting the entire learning material in a balanced way.

Table 1. The list of universities, faculties, and subjects involved in the model.

University	Faculty	Subject
University Business Academy in Novi Sad	Faculty of Economics and Engineering Management in Novi Sad	Fundamentals of mechanical engineering
		Technical drawing with descriptive geometry
		Engineer computer drawing
		Mechanics
		Driving dynamics
		Motor vehicles
		Internal combustion engines
		Means of transport and maintenance
		Maintenance of traffic and transport means
		Exploiting, diagnostics, and maintenance of road vehicles
		Exploiting and maintenance of motor vehicles
		The impact of traffic on the environment
		Contemporary aspects of transport vehicles and materials
		Quality management
		Business statistics
		Mathematics
		Basis of informational systems
		Quantitative methods
		Business informational systems
		Theories of traffic flow
		Freight transport technology
		Transport organization
		Capacity of roads
		Traffic research and measurement methods
University MB in Belgrade	Business and Law Faculty in Belgrade	Business mathematics
		Accounting
		Revision
		Statistics
Independent University of Banja Luka	Faculty of Ecology Faculty of Business Economy Faculty of Information Technologies Faculty of Pedagogy	Business statistics
		Advanced statistics
		Statistics and data processing
		Decision theories
		Econometrics
		Probability and statistics
		Operational research
		Mathematics
		Basics of informational systems
University “Union—Nikola Tesla”	Faculty of Business and Law	Probability and statistics
	Faculty of Informational Technologies and Engineering	Operational research
University of Novi Sad	Technical Faculty “Mihajlo Pupin” Zrenjanin	Materials 1
		Basics of mechanical design
		Machine design
		Intelligent machining processes
		Engineering materials
		Assembly technology

displays the list of universities, faculties, and courses that were included in the model and also used the ANN model.

3. Methodology

Framework

The research methodology was based on several successive phases, which are displayed in Figure 1. Phase 1 consisted of building the web platform as a whole so it could provide automatic tracking and data collecting, which refers to points collected pre-examination and during final examinations, so the final grades in phase 2 include codes created in the JS program language. With these newfound possibilities in phase 2, phase 3 provided us with statistic data analysis of points collected during pre-exams, final exams, and of the average grade during each course individually. These data could be further aggregated by types of pre-exam obligations, related subjects, or any other way in a short notice because the JS program language facilitated managing these data. With discovering a potentially interesting field for improving the adopted knowledge of students by statistical analysis in phase 3, we were brought to another hypothesis that the creating of the ANN model of selecting the examination questions could improve the level of adopted knowledge. The evident drawback was that the students who collected relatively high scores during the pre-exam obligations were far less motivated to study for the final examination. Phase 4 consists of identifying the parameters that are designated as inputs and outputs for the ANN model, designing instances, random selection of instances for determining the topology of ANN model, ANN training, ANN testing, and, finally, conducting these procedures. Furthermore, the ANN model was applied as a method of selecting questions on the final examination with a remark that procedures of determining the topology, training, and testing were repeated for each subject and exam individually in order to keep this model updated. After a longer period of applying the ANN model, we were enabled to compare pre-ANN model results in phase 6.

When the COVID-19 pandemic was announced, there was a website on the domain https://fimekdst.000webhostapp.com (accessed on 26 November 2022) that migrated several times to other domains. However, it maintained the fundamental information and functions for the students. It was launched by the professor of the University of Economic Academy, Milosevic, along with his associates. This was determined as the first phase commencement. This website still has activities in certain aforementioned subjects, which are used by the authors of this scientific paper. The subjects were designed with a multitude of links and materials, with added options for online teaching and a list of recordings that the students could access to during the pandemics. This completed the archive and learning material became gradually richer. In the period of education, mandatory platforms for online education (Moodle for example) provided students with easy access to sites and pages (landing pages) that helped integrating pre-COVID-19 and during COVID-19 solutions. Figure 2 displays the initial appearance of the homepage.

The next step was creating an Excel model of tasks for all subjects (Figure 3). Virtual workbooks presented a new possibility for subjects, with the same model, which meant students could change inputs by themselves in those tasks and obtain the final results automatically. The students could use this during and after tests. Page 2 is a clear example of an interactive workbook for practice.

The next logical step was to integrate the evaluation, which enabled the dislocated teaching staff to work, evaluate, and inform the students about their grades and points acquired during exams and pre-exams. JavaScript was utilized to perform these actions. This was the commencement of phase 2, which produced the possibility for data analysis on points and grades. Figure 4 displays a standard form of acquired points statistics organized by the educator, often used by more than one educator.

The inserted points are divided according to the teaching centers, cities, and faculties and subsequently they automatically generate a list for each faculty individually. All of this was divided into separate pre-exam and exam obligations (activity during class, colloquiums, written papers, oral exams, written exams). Figure 5 displays an example of a software-generating list that provides information instantly and is available to students at any moment with acquired points information, etc.

Many pages that were not possible to display due to the limited form of this scientific paper, referred to informing and setting up the learning materials for students and also useful video contents and web links. The most beneficial pages contained recorded lectures and thrived in excellence, not just on the level of adopting lectures, in time when students find it most convenient, but they also resolved the problem of varying the pace of knowledge adoption from the students. The ones in need could review those lectures at any given moment or remind themselves of a particular content. Figure 6 displays a page with links for lectures and information.

The total number of these pages, owned by the website, exceeded 300 webpages (an average of 10 pages per subject) and Google sheet documents; several hundred linked video lectures were also available on the Cloud platform. Figure 6 displays a page with subject information, test scheduling, and links for video lectures

The next step, occurring due to unpredictable circumstances, was to generate online knowledge evaluations. Through Google tests and similar methods (displayed in Figure 7), they provided a possibility for students to test themselves through randomly generated tests (with stratified questions) at any moment. Another advantage for examiners was that those tests could automatically evaluate themselves and consequently the number of complaints regarding the evaluated test was brought down to a neglectable number for most subjects. Much of the precious time was economized on the part of the teaching staff and that method had a tendency of becoming a regularity in the coming period. The automatic scoring of tests via the platform itself was indeed a particular advantage.

Ultimately, all the aforementioned parts were connected to official remote learning platforms such as the Moodle platform. This created an integral model (displayed in Figure 8). The new detail in Figure 8 is the ANN model that was a Decision Support Systems for deciding on selecting questions for trials and examinations. The model was created in phase 4, which was preceded by phase 3 after a period of obtaining data about acquired points and grades. Throughout this phase, we were able to analyze the data of all subjects both individually and collectively. The possibilities of data comparison are various, which led to certain conclusions.

With the stratified selection of questions, using the possibility that contained Google testing, particular questions were an option as first, second, third, or some other question with equal probability for selection. However, a student who focused only one group of questions was assured that there is a certain probability that they would pass at least one question and with this they would reach a sufficient number of points, when they add up the points from pre-exam obligations. With this practice, many students who worked diligently began to hold back and lacked motivation, so the average grade was decreasing in a rapid fashion due to low scores on the final testing in comparison to average scores.

The display of rates of correct answers for 2 questions on one of the tests is displayed in Figure 9, where it is obvious to notice the decreased percentage of points acquired on questions Q3 and Q7 after reaching a sufficient number of points acquired. Therefore, students massively opted for acquiring as many points as possible on their pre-exam obligations, so they could be excluded from studying for a final exam and vice versa. It is also possible that they were focused on other exams at the price of lower scores on the researched subject. The sample of researched students was 345 I 248 for questions Q3 and Q7 and these examples are the most evident.

After this, an idea emerged that the good students can be stimulated and motivated to work harder and achieve higher scores on their final exams by implementing uncertainty about which question will appear on the test and that they could even risk not completing one single question if they do not study thoroughly. The experience with working on ANN turned out to be crucial in the field of the reliability of technical systems. This approach was possible only where several tests were followed throughout the years by the previously explained online system of learning. This has established appropriate conditions for commencing the fourth phase—creating and applying the ANN model.

The model was created as a consequence of connecting this scientific work in the domain of predicting the failures of technical systems, which were the subject of authors scientific work in that period of time. What followed was the idea to develop a form of AI based on ANN in order to facilitate the selection of questions. So far, that selection has been performed randomly on stratified groups of questions or by the selection of educators themselves, which required an extra amount of work. The ANN model was designed to provide each student with individual questions that can be completed with the least probability that can raise students’ awareness about the importance of a balanced adopting of the entire learning material. Logically, the authors presumption was that this kind of AI assistance should put an end to patterned learning and lower scores on tests and provide a permanent quality as a final result.

The ANN inputs were:

• Freshman year on the faculty;
• Year of attending subject;
• Number of points acquired on a particular lecture (pre-exam obligations);
• Number of an exam question by order.

The outputs of the ANN model were:

• Percentage of points acquired for a particular exam question.

It was not possible to execute this model from the start.

The development of the neural network consisted of multiple phases that had to be performed for each subject individually, which was not simple in the beginning. However, where the collective database for learning ANN was sufficient, it was easier to overcome the complexity of it. The steps that were performed are following:

• Obtaining data instances for inputs and outputs;
• ANN architecture model selection based on a 20% sample;
• Learning ANN with 60% of randomly selected data;
• ANN testing based on remaining 20% of randomly selected data;
• Data preparation for next exam (each student has individual data that display as an input in ANN);
• Predictions of probability of correct answers for individual questions (each student individually);
• Selecting questions with the least probability of correct answer per student.

Input, even though intuitively determined, provided a required quality for the case of finding the answers for a designated question. The intention is to motivate students to learn all of the important questions by finding out which question has the least probability of correct answer for each student.

In the following, the ANN example is shown, which solved the probability of the correct answers for Q3 for each individual student.

Table 2. Data set used for selection, learning, and testing of ANN.

	Minimum	Maximum	Mean	Deviation
Start year	2014	2020	2019.5	0.67
Started course	2016	2022	2021.53	0.66
Points	30	60	44.81	8.62
Question	3	3	3	0
Probability	0.03	0.99	0.63	0.2

shows a preview of the data set. Here, the number of variables is 5 and the number of instances is 345.

Here, the number of variables is 5 and the number of instances is 345. The number of training instances is 207 (60%), the number of selection instances is 69 (20%), and the number of testing instances is 69 (20%).

The order selection algorithm selected for this application is in an incremental order. This method starts with the minimum order and adds a given number of perceptron’s in each iteration. The parameters of the order selection algorithm are displayed in

Table 3. The order selection algorithm.

	Description	Value
Minimum order	Number of minimum hidden perceptrons to be evaluated.	1
Maximum order	Number of maximum hidden perceptrons to be evaluated.	10
Step	Number of hidden perceptrons added in each iteration.	1
Trials number	Number of trials for each neural network.	3
Tolerance	Tolerance for the selection error in the trainings of the algorithm.	0.01
Selection loss goal	Goal value for the selection error.	0
Maximum selection failures	Maximum number of iterations at which the selection error increases.	5
Maximum iterations number	Maximum number of iterations to perform the algorithm.	1000
Maximum time	Maximum time for the order selection algorithm.	3600
Plot training error history	Plot a graph with the training error of each iteration.	TRUE
Plot selection error history	Plot a graph with the selection error of each iteration.	TRUE

.

Figure 10 displays the error history for the different subsets during the incremental order selection process. The blue line represents the training error and the orange line symbolizes the selection error.

Table 4. The order selection results by the incremental order algorithm.

	Value
Optimal order	6
Optimum training error	0.354425
Optimum selection error	1.09299
Iterations number	11
Elapsed time	00:17

displays the order selection results using the incremental order algorithm. They include some final states from the neural network, the error functional, and the order selection algorithm.

A graphical representation of the network architecture is displayed in Figure 11. It contains a scaling layer, a neural network and an unscaling layer. The yellow circles represent scaling neurons, the blue circles perceptron neurons and the red circles unscaling neurons. The number of inputs is 3 and the number of outputs is 1. The complexity represented by the numbers of hidden neuron is 6. The reason why this example has 3 inputs is because the fourth input is always 3. Therefore, a specific example of ANN provides us the probability of correct answers for question Q3 for each individual student.

The quasi-Newton method is used here as optimization algorithm. It is based on Newton’s method but does not require the calculation of second derivatives. Instead, the quasi-Newton method computes an approximation of the inverse Hessian at each iteration of the algorithm by only using gradient information. The quasi-Newton method parameters are displayed in

Table 5. The quasi-Newton method parameters.

	Description	Value
Inverse Hessian approximation method	Method used to obtain a suitable training rate.	BFGS
Training rate method	Method used to calculate the step for the quasi-Newton training direction.	BrentMethod
Loss tolerance	Maximum interval length for the training rate.	0.001
Minimum parameters increment norm	Norm of the parameters increment vector at which training stops.	1·10−9
Minimum loss decrease	Minimum loss improvement between two successive epochs.	1·10−12
Loss goal	Goal value for the loss.	1·10−12
Gradient norm goal	Goal value for the norm of the objective function gradient.	0.001
Maximum selection error increases	Maximum number of epochs at which the selection error increases.	100
Maximum iterations number	Maximum number of epochs to perform the training.	1000
Maximum time	Maximum training time.	3600
Reserve parameters norm history	Plot a graph with the parameter norm of each iteration.	FALSE
Reserve error history	Plot a graph with the loss of each iteration.	TRUE
Reserve selection error history	Plot a graph with the selection error of each iteration.	TRUE
Reserve gradient norm history	Plot a graph with the gradient norm of each iteration.	FALSE

.

Figure 12 displays the training and selection errors in each iteration. The blue line represents the training error and the orange line represents the selection error. The initial value of the training error is 18.4735 and the final value after 389 epochs is 0.350383. The initial value of the selection error is 25.5181 and the final value after 389 epochs is 1.41532.

Table 6. The training results by the quasi-Newton method.

	Value
Final parameters norm	23.3
Final training error	0.35
Final selection error	1.42
Final gradient norm	0.000982
Epochs number	389
Elapsed time	00:01
Stopping criterion	Gradient norm goal

displays the training results by the quasi-Newton method. They include some final states from the neural network, the loss functional, and the optimization algorithm.

4. Results

The outcome of the finalized ANN support system for decision making was to enable the determination of the probability of a correct answer for a given question according to calculated inputs. In the case of parameter Start year 2019 and Start course 2021, it was possible to display a probability graphic based on acquired points during pre-exam obligations (Figure 13).

Therefore, the question Q3 has a very small probability of a correct answer for around 54 points acquired—just 10%—while the sum of 60 points acquired has 70% probability and the sum of 48 points has the most probability and reaches around 85%. With this prediction, it was easy to access possibilities to accentuate specific teaching units and to improve knowledge adoption during the final course but to also raise awareness among students to focus on the whole learning material and adopt knowledge systematically.

Example Q3 is individual but it demonstrates possibilities of similar calculations for each question of each subject. The wide application of this concept resulted in very interesting outcomes.

If we use the percentage of passed examinations for every year and an average grade as a parameter of course’s success, then we can compare final results by dividing work periods into three intervals.

The intervals are:

• Period before the COVID-19 pandemic;
• Period after the COVID-19 outbreak, before introducing the ANN support system for selecting exam questions;
• Period after the COVID-19 outbreak, after introducing the ANN support system for selecting exam questions.

Phase 7 included comparing these three time intervals and resulted in following graphics of success rate criterion for different areas of subjects that applied the ANN model: Mechanical Engineering, Traffic Engineering, Informatics and Statistics, Accounting, and Auditing.

The average grade and completed tests percentage on these courses, for all three time intervals are measured in

Table 7. Average grades for groups of subjects for all three observed intervals.

Average Grades
	Period 1	Period 2	Period 3
Mechanical Engineering	7.9	7.6	8.4
Traffic Engineering	8.1	7.7	8.5
Informatics and Statistics	8.4	7.6	8.7
Accounting and Auditing	8.1	7.9	8.1

and

Table 8. Percentage of successful course taking for groups of subjects for all three observed intervals.

% of Success
	Period 1	Period 2	Period 3
Mechanical Engineering	83	73	89
Traffic Engineering	85	77	92
Informatics and Statistics	85	86	85
Accounting and Auditing	82	80	81

.

By observing these values of average grades and the percentage of passed courses, we can clearly notice a decrease in values almost becoming a regularity since the COVID-19 outbreak, which is a logical consequence of modified working methods and new online methods of learning. A majority number of educators and students were not prepared for these extraordinary events. However, since the stabilization and application of new models and ANN technology for exam question selection, these values have recovered and even improved significantly. We could say that the COVID-19 pandemic with all its challenges also brought in new aspects of work that turned out to be beneficial.

5. Discussion

At the time of the COVID-19 pandemic, it was clear that subjects with already implemented online working platforms were able to react faster to the new COVID-19 pandemic situation. The COVID-19 pandemic required people to adjust to online methods of work as an alternative (permanent or temporary) and also develop new services for the students. Online platforms mutually integrated and enriched each other with content. By connecting with the official services, educators’ personal services have become a part of the official parts of education. Educators’ personal services began to expand with more quality and become more adaptable, therefore they proved that they could function properly on an international level, as we have shown in this paper.

The general principles established in this paper show that the digitization of work provides online testing and better success metrics and answers on certain questions. Successful answer metrics on certain questions and connecting with the number of points during pre-exam obligations brought us to discover that better students do not perform with their best efforts during the final examinations compared to their effort during the course. Inspired by scientific work on ANN networks, educators have developed a form of AI capable to predict the probability of correct answers for each individual question based on a few key inputs.

Taking into consideration the review of recent studies cited in the literature, one of the main conclusions that is supported by the research through the paper is that the ANN model provided us the possibility to ask students about the least probabilities of correct answers and raise students’ awareness of learning the entire material altogether. This led to much better results in the third period of observation, better than ever before the pandemic. The rate of passed examinations increased by about 3% on average and the average grade increased for 0.3 on all subjects where the ANN model for selecting exam questions was applied.

Limitations and Future Research Directions

The drawbacks of online educations for this particular model, besides the learning to adapt on new terms that the ANN model requires, are that each time there is a new reaction of a student on new data selections, the training must be conducted for every new exam so the data can be updated and in accordance with contemporary practice.

The process of applying the ANN model is strenuous when you consider that numerous questions could be modified every following year or even between examinations. These are all aggravating factors, for they diminish the collection of data that are used for the ANN model.

Creating a universal model for all questions in all examinations is impossible because the ANN itself cannot differentiate between the new divisions of materials into exam questions but can only differentiate each question individually. Major modifications of the study programs would produce a new beginning of obtaining data and a lower accuracy of calculation by the ANN model.

The suggestion for further research is generating the assembly of ANN or a unique software that would facilitate the acquisition of data necessary for ANN training and new automatized training for saving time. Of course, time consumption for this model is not higher than the previous models but it is certain that we can save plenty of time with this integrated solution for all subjects.

A universal solution, as one of the requisite phases, would consider forming the elementary questions and answers that are integral parts of examination questions, so that every aspect of knowledge would apply the ANN model and establish resistance to changes of study programs.

The highest level of applying this model could suggest research that could provide that every type of learning material could be automatically decomposed to elementary questions and answers. This form of research is surely the most demanding and it is certain that it could be realized in the domain of individual courses and science disciplines before becoming a universal solution.

6. Conclusions

In this paper, we presented a web platform that was original and hybrid with a lot of subsystems that passed knowledge and information to students and teaching staff. With the COVID-19 outbreak, the platform expanded and enabled the quantification of numerous actives and examination scores of students. After acknowledging the drawbacks in knowledge adoption, an innovative ANN model was created that enabled the forming of individual questions that stimulated systematic learning and the better adoption of knowledge. The innovativeness of the model is a combination of input and output parameters, which enables the possibility of its wide application in various branches of science and resulted in the intense application of this model in numerous courses across universities in Serbia and Bosnia and Herzegovina.

The advantage of this model is its simple utilization, when the courses program is stable and unvarying for a longer period of time. The pace of calculating is almost instant and the obtained input data are easily updated.

The drawback of this model is that the probabilities of correct answers are examined individually instead of all the questions being examined simultaneously. However, this model would vary in a number of input parameters and it would be applicable to a number of courses. In the case of a large number of output neurons, the accuracy of the ANN model would be significantly reduced or the process would be burdened and, in numerous cases, probably impossible.