Knowledge Assessment Patterns for Distant Education: The Perceived Impact on Grading, Motivation, and Satisfaction

Abstract

With the increase in remote learning, the efficient implementation of distant knowledge assessments has become an essential topic. New challenges have arisen that have had to be adequately addressed and successfully solved. As a response, we introduced an assessment patterns catalogue for distant education, handling various challenges by proposing possible solutions. The catalogue presents a collection of proven practises targeting knowledge assessment in digital and distant environments. The paper presents the survey results from the final step of catalogue creation. The objective was to verify its suitability and expedite its use, focusing on several aspects of knowledge assessment. We focused on the perceived assessment patterns’ impact on grading objectivity and consistency, as well as students’ motivation and satisfaction with an implemented assessment, explored from the students’ and teachers’ perspectives. We gathered data using a uniform questionnaire distributed between students and teachers, both actively involved in distant knowledge assessment. Detailed data analysis highlighted the patterns with the highest perceived impact on the previously mentioned assessment aspects. We also analysed the top-rated patterns within the pattern categories. The results depict a high overlap between students’ and teachers’ perspectives, wherein patterns like Pentathlon, Statistical Validator, Game Rules, and Bonus Points were perceived as the patterns with the highest impact on grading objectivity and consistency, as well as the patterns with the most significant impact on students’ motivation and satisfaction.

1. Introduction

The main goal of education is to provide knowledge and skills within the teaching domain. To achieve this goal, a variety of approaches and methods are being used and constantly adapted in order to cope with emerging challenges within education. One of the goals of the UN for sustainable development is also quality education [1], whereby adaptation to current challenges is crucial for its achievement. An important role in providing sustainable development in society lies in higher education institutions [2,3]. Undoubtedly, the most significant changes in recent years were influenced by the COVID-19 pandemic. Remote learning also arose in environments where, until then, the activities were primarily practised in a person-to-person setting [4]. Higher education institutions had to move their learning activities to digital environments in a very short time, where reduced quality was not an option [5].

Together with teaching, knowledge assessments, a fundamental part of the educational process [6], were also moved to the remote environment. The latter added additional challenges to the already demanding task of implementing an efficient and fair assessment. The proven and widely used knowledge assessment practises (e.g., written exams) became practically unusable; therefore, an adaptation of what was used and known was necessary. Different studies addressed the educational process during the pandemic, some focusing specifically on knowledge assessment practises [4,6,7,8,9,10].

Many definitions of assessment exist. However, as pointed out by Ghaicha [11], Bachman [12] defined the assessment as an “act of interpreting information regarding student performance collected through any of a multitude of means or practices”. Various new practises and methods arose to facilitate the assessment process in distant environments. The limitations of having the process in-person are mostly gone. However, some parts of the educational process are still and will remain online. Although the shift in the teaching approaches was not promoted on the right footing, conditions, which put us out of our comfort zones, helped to push progress in the educational domain even further. This is especially true for the use and involvement of information technology in teaching activities, including knowledge assessment. Progressively, a new term emerged that describes the effective use of information and communication technology to achieve learning outcomes using a suitable pedagogical approach—smart education [13].

Already-proven best practises for distant education were collected in a catalogue of assessment patterns [14]. The catalogue was created in response to the situation in 2020, wherein we collected best practises and experiences that proved to be adequate in distance environments in higher education. It includes 47 educational patterns, collected in four categories, covering topics from designing knowledge assessments to patterns dedicated to communication before, within, and after knowledge assessment. Since the introduced catalogue [14] includes recurring practises derived from practical experiences, their general recognition, which will also result in broader acceptance, can only be achieved if the involved stakeholders grasp the effective and practical approach to addressing emerging challenges. Therefore, validating the created catalogue represents an essential step in catalogue introduction.

An initial validation was already performed, focusing on patterns, that are crucial for successful and efficient remote knowledge assessment from teachers’ [14], and students’ [15] perspectives. Therefore, this paper presents an in-depth analysis of teachers’ and students’ opinions regarding the proposed catalogue of assessment patterns for distant education. Our research was focused on two aspects. The first covers grading objectivity and consistency, and the second focuses on students’ motivation and satisfaction. When grading is objective and consistent, the difference that may appear when assessing students’ work is minimised [16].

Motivation represents an important element of teaching and learning [17,18]. According to Alsadoon et al. [18], motivation elevates the students’ interest in the learning domain and encourages their preparedness for learning. On the other hand, satisfaction is an emotion of joy and happiness from the individuals’ point of view, achieved when needs and desires are pleased [19]. It is a short-term attitude connected to subjective fulfilment of educational experience expectations [20].

However, research [21] suggest that students’ motivation decreases during online learning. By providing a positive and supportive environment, the students’ motivation could be increased [22]. Low engagement can result in poorly acquired knowledge; therefore, an important consideration when designing the catalogue was the impact of the patterns on students’ motivation and satisfaction.

Our research was performed from the teachers’ and students’ perspectives, using a uniform data collection approach, allowing a valid comparison within the groups. The presented research followed two research questions:

• RQ1—Which assessment patterns have the highest impact on grading objectivity and consistency?
• RQ2—Which assessment patterns have the highest impact on students’ motivation and satisfaction?

The rest of the paper is organised as follows. Section 2 summarises the existing related work and presents the structure and the creation of the catalogue of assessment patterns for distant education. The section depicts the catalog’s structure and briefly describes the defined patterns. Section 3 presents the applied research methodology, including the defined data collection method and research instrument. The results are presented in Section 4, starting with an overall analysis, which is deepened in Section 4.1 and Section 4.2. The research findings are discussed in Section 5, and the paper is concluded in Section 6.

2. Distant Assessment Pattern Catalog

Although distant education brings many benefits, various challenges arise that must be addressed efficiently [23]. Among the most important are the challenges connected to the lack of teachers’ knowledge and experience. Therefore, systematically collected and aggregated best practises can significantly contribute to the efficiency of remote education. Although the pandemic has ended, distant education in higher education has not disappeared. It proved to be a suitable solution in many contexts, and in addition, many educational institutions have significantly developed remote education infrastructure [6].

In distant education, a major challenge is the management and implementation of knowledge assessments [10]. The assessments have to be reliable and valid, even if implemented in remote settings, bringing additional challenges [10]. To successfully cope with those challenges, a well-defined and validated assessment pattern catalogue can be a great help. Its main goal is to group various ideas and proven practises, helping the involved parties when implementing an effective, fair, and motivational knowledge assessment process. Köppe et al. [24] defined educational patterns as “hypothesis solutions to recurring problems in an educational context”. According to a literature review [14], several available pedagogical patterns catalogues exist, e.g., [24,25,26,27]. The identified shortcoming lies in the lack of categorisation or cataloguing and the lack of consistency within authors [14]. The catalogues cover a wide educational area and connected aspects, wherein a few authors also address the knowledge assessment practises [28,29,30,31,32].

In response to the existing shortfall, we introduced a novel catalogue of assessment patterns for distant education [14]. The catalogue addresses two of the important challenges: efficient knowledge assessment and education in remote digital environments. The creation of the assessment patterns catalogue was conducted in a few sequential steps: (1) preliminary cataloguing; (2) organising workshops with practitioners and evaluating novelty with focus group members; (3) cataloguing; and (4) catalogue validation within pattern users [14].

According to first-hand experiences gathered during the COVID-19 pandemic, the preliminary cataloguing of assessment patterns began. A focus group of teachers and teaching assistants identified frequently used practises for distant knowledge assessments, resulting in the first set of patterns combined into the categorised catalogue. The initial version of the catalogue was later presented to the practitioners in the organised workshops. Workshop participants provided valuable feedback based on the patterns presentation and focused discussion. On the basis of the first two activities, the focus group iteratively finalised the cataloguing. The catalogue of assessment patterns for distant education consists of 47 patterns, divided into four semantic and two container categories [14]:

• D—Patterns for the assessment conceptual design;
• Q—Patterns for defining questions, answers, and schedules;
• E—Execution and grading patterns;
• C—Communication patterns;
• A—Anti-patterns;
• O—Other patterns.

In addition to pattern categorisation, we uniformly described each of the patterns, including attributes, namely, the pattern category, the challenge solved by the pattern, the main idea of the pattern, the context, participants, and environment, the implementation steps, variations, advantages and disadvantages, known uses, related patterns, and do not use with patterns. A short description of each pattern in the catalogue is presented in

Table 1. Short descriptions of the defined patterns [14].

Category	Pattern	Description
D	Open Book	All assets are allowed during an online assessment.
D	Stagewise Approach	Stagewise learning and remote assessment.
D	Pentathlon	The final grade comprises multiple assignments, e.g., hands-on, quizzes.
D	Continuous Testing	Integrate remote assessment into lectures as a tool for teaching.
D	Expert-Level	Remote assessments follow the complexity required for an expert.
D	Innovator	Be innovative in all aspects—use out-of-the-box online tools, and approaches.
D	Student Achievement Portfolio	Use assignments evolution/progress as an auditing trail for a later retrospective.
D	Colleague Bloom	Use Bloom’s taxonomy as guidance for remote assessment preparation.
Q	Expert Validator	Use colleague teacher or domain expert to validate questions and answers.
Q	Statistical Validator	Statistically validate online assignment responses.
Q	Hidden Validator	Continuously collect and validate students’ feedback in order to validate questions.
Q	Colleague Veto	Eliminate or correct questions in case of doubts or concerns of a colleague.
Q	Mathematical Validator	Follow method (e.g., 1 min/question) to set time-limits for remote assessment.
Q	Professional Multiplier	Multiply time required for completion by expert by three (or two).
Q	Dress Rehearsal	Rehearse online assignment or exam submission.
Q	Example Questions	Provide online question examples.
Q	Questions Donor	Motivate students to provide potential exam questions.
Q	Theme Variants	Prepare different online assignment variants for the same assessment type.
E	Timebox	Introduce time-limit for online assignments.
E	Randomised Order	Introduce random ordering of online questions and their answers.
E	Results With Delay	Display results with delay, fine-tune results—related to Objective Assessor.
E	Identity Guarantee	Use techniques that guarantee the authenticity of students (e.g., handwritten solutions).
E	Accessibility Adjustments	Allow online accessibility adjustments for students with special needs.
E	Objective Assessor	Eliminate questions that turn out to be problematic.
E	Criteria List	Define the list of criteria for a more objective e-solutions assessment.
E	Eurosong	Introduce peer evaluation with the Eurosong points system.
E	Bonus Points	Introduce bonus points to better engage students in an online setting.
E	Third Shift	Be available for remotely assisting students outside course time slots.
E	Number Draw	Introduce random order of online assignment presentations.
E	Impro League	Agile response to unexpected problems (e.g., internet failure).
E	Self-Assessment	Making a list of questions with answers for students’ preparation/self-assessment.
E	Personal Defence	Introduce remote oral defence of students’ e-solutions.
C	Appetizer	Provide example online exam questions.
C	Game rules	Explain remote assessment rules at first lecture and before assessment.
C	Time Reminder	During an online assessment, remind students about the remaining time.
C	To-Do List	Provide a checklist of the key parts of the remote assignments.
C	Emergency Call	Be available in the separated channel to address urgent student calls.
C	Member Channel	Encourage separated channel where students can communicate in smaller groups.
C	Academic Integrity Appeal	Emphasise the importance of academic integrity during online assessments.
C	Proactive Teacher	A teacher periodically contacts students to see if there are any problems.
C	Firewall	Explain/present assessment results remotely to minimise exam reviews.
C	Student Proxy	Urgent information can be distributed by student proxy. A voice of a class.
A	24/7	Unrealistic student expectations regarding teacher’s online responsiveness.
A	Full-Time Job	Addressing student problems remotely can become a full-time job.
A	Certification Centre	Avoid acting like a certification centre, where you focus only on assignment criteria.
O	Safe Exam Browser	Introduce the Safe exam browser to minimise cheating during the online assessment.
O	Big Brother	Introduce one or more cameras to supervise students remotely.

. In addition, an example of a complete description of chosen patterns can be found in the initial paper by Pavlič et al. [14].

The group of patterns for the knowledge assessment conceptual design (D) combines proven practises that are useful in the design and organisation of students’ knowledge assessments. Within this, teachers decide about allowed assets (e.g., Open Book) and the assessment structure (e.g., Pentathlon, Student Achievement Portfolio). The second group consist of patterns for defining questions, answers, and schedules (Q), wherein more detailed planning of the knowledge assessment instances is conducted. This includes question preparation (e.g., Question Donor, Colleague Veto), duration and content validation patterns (e.g., Professional Multiplier, Hidden Validator), and patterns used for the assessment preparation (e.g., Dress Rehearsal). Group execution and grading patterns (E) and communication patterns (C) consist of practises that could be applied within the knowledge assessment session implementation after the assessment design is finalised and questions are determined. Those are the patterns connected to grading (e.g., Criteria List, Results With Delay), implementation method (e.g., Eurosong, Personal Defence), and adjustments patterns (e.g., Accessibility Adjustment, Impro League). An important part of an efficient knowledge assessment is also appropriate communication. Therefore, practises that can be used when explaining the assessment process (e.g., Game Rules and To-Do List) are important, as well as solutions that could be used when challenges arise (e.g., firewalls and emergency calls).

The last sets of patterns are anti-patterns (A), which are poor practises that should be avoided (e.g., Certification Centre) and other patterns (O): patterns whose categorisation is not precise, since it depends on different factors and environments (e.g., Big Brother). As mentioned, a short description of each pattern is presented in Table 1, wherein the entire catalogue and its graphical representation can be found in our previous work [14].

The final step in catalogue design was the validation process. Validation is crucial for catalogue development and the expansion of its use. Although the catalogue was designed based on experiences gained in remote learning, most patterns could also be used in classroom settings. In related work, documented educational patterns were validated through peer reviews and discussions on domain events [33,34]. Further, a proven and established validation approach was not detected. As stated by Fioravanti and Barbosa [35], there is a gap in the validation of proposed pedagogical patterns. Consequently, we developed a questionnaire in order to review different quality aspects of the defined patterns. Teachers performed the first validation phase, while the second phase was conducted independently by students who were actively involved in the distant knowledge assessment process, implementing a majority of the presented patterns. The initial findings of the validation, addressing top-rated patterns crucial for successful and effective remote knowledge assessment, were already presented in our previous papers [14,15]. Hereinafter, we present an in-depth validation, analysing the aspects addressed by the formed research questions.

3. Research Methodology

A novel catalogue of assessment patterns for distant education was designed [14] in order to gather, define, and share proven practises for remote knowledge assessment. The final step of catalogue creation and validation aimed to research the perceived impact of the included patterns on grading objectivity and consistency and the impact of their use on students’ motivation and satisfaction. Since we are aware that opinions between teachers and students can vary significantly, we conducted the validation from both perspectives. The first validation phase was performed among teachers, namely, higher education teachers and their teaching assistants, and the second phase was conducted among students.

To answer the research questions presented in Section 1, we prepared a uniform questionnaire for teachers and students. The questionnaire was developed in sequential steps, considering the concepts of validity and reliability. The context of the research questions guided the definition of the variables, wherein each variable is represented in one statement. With this, we addressed the construct and content validity, assuring our tool measured the intended view of interest. The development of the questionnaire was conducted within a focus group, which was already a part of the catalogue creation presented in [14]. The focus group consisted of field experts, which also affected the tool’s validity. The instrument was then tested using a pilot group of teachers and students, where some minor changes were proposed. We implemented the test–retest approach to evaluate the reliability of the questionnaire. After introducing the changes to the questionnaire, the preparation of the research tool was concluded with a pilot data analysis to ensure suitable survey results.

In the questionnaire, participants evaluated four statements using a 5-point Likert scale for each of the 47 defined patterns. On the scale, 1 stands for completely disagree, while 5 stands for completely agree. The questionnaire is depicted in

Table 2. The questionnaire.

Statement	Scale
The pattern has a positive impact on grading objectivity.	5-point Likert scale
The pattern has a positive impact on grading consistency.	5-point Likert scale
The pattern has a positive impact on students’ motivation.	5-point Likert scale
The pattern has a positive impact on students’ satisfaction.	5-point Likert scale

.

The demographics of our research participants is summarised in

Table 3. Demographics of the research participants.

Teachers	Students
33 teachers	51 students
and teaching assistants	actively involved in remote assessment
11.1 years of teaching experiences	1st year Master’s Degree
5.4 different courses	focused on informatics

. Opinions were gathered from 33 teachers, including teachers and teaching assistants. On average, they had 11.1 years of teaching experience and were involved in 5.4 different courses. Teachers answered the survey in the context of assessment pattern catalogue creation.

Also, a sample of 51 students was involved in the research. All participating students were actively involved in the remote study process during the COVID-19 pandemic for almost two years, which also included the remote knowledge assessment. Since each study year comprises 10 study courses, each participant was included in knowledge assessment sessions in 20 courses. Therefore, they participated first-hand in the assessments containing many of the catalogue assessment patterns. At the time of the research, they were enrolled in the 1st year of the Master’s Degree study programme, focused on informatics.

Participation in the study was optional and anonymous, which was clearly stated to the potential participants. In order to be a part of the study, the participants had to give informed consent at the beginning of the questionnaire. With this, they were able to start the answering process. After the participants were selected, we organised a two-part session. In the first part, students were familiarised with the assessment patterns, and in the second part, all of them answered a questionnaire that was prepared for all 47 patterns.

After the survey, the results of both teachers and students were statistically analysed. According to the structure of a uniform questionnaire, we focused on two main viewpoints: (1) grading objectivity and consistency; and (2) students’ motivation and satisfaction. Each variable was represented by one statement, assessed by a 5-point Likert scale. The study’s main objective was to determine which of the defined assessment patterns was the most important for each concept. Therefore, a sorted list for each pattern category was created, based on the mean values that were gathered separately from students and teachers.

4. Results: The Impact of Defined Knowledge Assessment Patterns

Teachers and students answered four questions for each of the 47 identified patterns using a 5-point Likert scale. Tables 9 and 10 present average values for each pattern for students and teachers separately. The patterns are organised according to their categorisation presented in Section 2. Since the agreement with the statement was opined using a 5-point scale, numbers closer to 5 express strong agreement with the statement, while average numbers closer to 1 express strong disagreement.

The detailed analysis of the results is presented in Section 4.1 and Section 4.2, while

Table 4. Assessment conceptual design patterns’ impact on objectivity, consistency, motivation, and satisfaction.

	Patterns for the Assessment Conceptual Design (D)
		Students	Teachers
Objectivity	1.	Pentathlon	Pentathlon
	2.	Student Achievement Portfolio	Stagewise Approach
	3.	Innovator, Colleague Bloom	Student Achievement Portfolio
Consistency	1.	Pentathlon	Pentathlon
	2.	Student Achievement Portfolio	Continuous Testing
	3.	Colleague Bloom	Stagewise Approach
Motivation	1.	Pentathlon	Pentathlon
	2.	Open Book	Innovator
	3.	Innovator	Continuous Testing, Stagewise Approach
Satisfaction	1.	Open Book	Innovator
	2.	Innovator	Open Book, Pentathlon
	3.	Pentathlon	Stagewise Approach

,

Table 5. Patterns for defining the impact of questions, answers, and schedules on objectivity, consistency, motivation, and satisfaction.

	Patterns for Defining Questions, Answers, and Schedules (Q)
		Students	Teachers
Objectivity	1.	Expert Validator	Expert Validator
	2.	Statistical Validator	Statistical Validator
	3.	Colleague Veto	Colleague Veto
Consistency	1.	Expert Validator	Statistical Validator
	2.	Statistical Validator	Colleague Veto
	3.	Colleague Veto	Expert Validator
Motivation	1.	Questions Donor	Questions Donor
	2.	Example Questions	Example Questions, Hidden Validator
	3.	Statistical Validator	Dress Rehearsal
Satisfaction	1.	Example Questions	Example Questions
	2.	Questions Donor	Questions Donor
	3.	Expert Validator	Hidden Validator

,

Table 6. Execution and grading patterns’ impact on objectivity, consistency, motivation, and satisfaction.

	Execution and Grading Patterns (E)
		Students	Teachers
Objectivity	1.	Objective Assessor	Criteria List
	2.	Results With Delay	Results With Delay
	3.	Impro League	Objective Assessor
Consistency	1.	Objective Assessor	Criteria List
	2.	Impro League	Results With Delay
	3.	Results With Delay	Objective Assessor
Motivation	1.	Bonus Points	Bonus Points
	2.	Objective Assessor	Eurosong
	3.	Impro League	Self-Assessment
Satisfaction	1.	Bonus Points	Third Shift
	2.	Objective Assessor, Impro League	Bonus Points
	3.	Third Shift	Impro League

and

Table 7. Communication patterns’ impact on objectivity, consistency, motivation, and satisfaction.

	Communication Patterns (C)
		Students	Teachers
Objectivity	1.	Game Rules	Game Rules
	2.	To-Do List	Academic Integrity Appeal
	3.	Appetizer	To-Do List
Consistency	1.	Game Rules	Game Rules
	2.	To-Do List	Academic Integrity Appeal
	3.	Appetizer	Appetizer
Motivation	1.	Appetizer	Proactive Teacher
	2.	Game Rules	Appetizer
	3.	Emergency Call	Student Proxy
Satisfaction	1.	Emergency Call	Emergency Call, Game Rules
	2.	Appetizer	Appetizer
	3.	Game Rules	Proactive Teacher

present a bird’s-eye view of the analysed data. Table 4 summarises the results for the assessment conceptual design patterns. The table shows the top-rated patterns according to the students and teachers, regarding the perceived pattern’s impact on the aspects researched in the presented study, namely, grading objectivity and consistency and students’ motivation and satisfaction. An overlap could be seen between students and teachers, where some of the patterns take precedence in related categories. The pattern Pentathlon was perceived as the pattern with the higher impact on grading objectivity and consistency by both students and teachers. The pattern addresses the challenge connected to representatives of a single assessment; therefore, we propose that the final grade of the student is composed of multiple assignments, e.g., lab work, quizzes, oral and written exams. This also positively impacts students’ motivation since teachers can engage students with partial assessments throughout the semester. This was also later confirmed with empirical results, as seen in Table 4. Also, other patterns that anticipate a gradual path to the final grade were perceived as very important for grading objectivity and consistency. Those are the patterns Student Achievement Portfolio and Stagewise Approach.

On the other hand, the Open Book pattern was perceived as important for students’ motivation and satisfaction, especially from the students’ perspective. The Open Book pattern allows the use of all resources during the knowledge assessments. An appealing technique for the students needs special attention from the teachers, especially regarding the types of questions to be prepared. A pattern that highly impacts students’ motivation and satisfaction, perceived by students and teachers, is also the Innovator pattern. The pattern encourages the use of innovative, out-of-the-box tools and approaches, such as, for example, business simulation games in applicable study courses. As is known, in addition to challenges connected to remote education, challenges associated with maintaining the motivation and engagement of the digital generation are also present. The Innovator pattern allows one to address both challenges at the same time.

Table 5 depicts the top three patterns for the researched aspects within the group of patterns for defining questions, answers, and schedules. The highest impact on grading objectivity and consistency was perceived for the content validation patterns, namely, Expert Validator, Statistical Validator, and Colleague Veto. When applying the patterns Expert Validator and Colleague Veto, a domain expert or colleague reviews the questions to validate questions and answers and eliminate the problematic ones. While this is performed before the knowledge assessment process implementation, the pattern Statistical Validator is applied after the assessment is completed, looking for a potential negative deviation from the expected average result.

In the area of students’ motivation and satisfaction, the patterns Question Donor and Example Questions were perceived as very important among students and teachers. Both patterns give students the opportunity to practise and get to know the examples of possible questions. Even more, within the pattern Question Donor, students provide their own questions and answers that later become the candidates for the knowledge assessment. The pattern offers an alternative way for motivating students to examine the study materials by proactively participating in defining the knowledge assessment.

The third group within the catalogue consists of execution and grading patterns. The patterns with the most significant perceived impact on studied domains are presented in Table 6. The students perceived the pattern Objective Assessor as the pattern with the highest impact on grading objectivity and consistency. The pattern is closely related to the previously mentioned pattern Statistical Validator, wherein the teachers eliminate the unclear questions from the grading. Also, a high impact on objectivity and consistency was perceived for the pattern Results With Delay, where assessment results are published after the review and, according to the teachers’ perspective, for the pattern Criteria List. The latter requests that the grading is followed by a previously defined scoring rubric.

The teachers and students agreed that the pattern with the highest impact on student motivation is the Bonus Points pattern. The pattern aims to provide bonus points as a reward in order to encourage the students’ engagement. This is especially important in remote environments since the teachers do not have direct access to their listeners. High on the list was also the pattern Impro League, meaning that teachers react agilely and quickly to possible problems. Again, this frequently emerges when knowledge assessments are implemented in distant digital environments. Regarding students’ satisfaction, the pattern Third Shift was placed among the top three according to the teachers and students. The pattern anticipates that the teachers will also be available to students outside the dedicated course time slots. However, particular caution is required so that good practise does not turn into poor practise, i.e., the anti-pattern Full-Time Job.

The last content group in the published catalogue are the communication patterns. The patterns with the highest impact on objectivity, consistency, motivation, and satisfaction are captured in Table 7. Appropriately planned and implemented communication can significantly improve grading efficiency. Knowing what to do and when to do it, whether we are explaining the course assessment structure at the beginning or the knowledge assessment structure before the assessment, greatly impacts grading objectivity and consistency. This was confirmed by students and teachers, wherein the pattern Game Rules was perceived as the pattern with the highest impact on both of the mentioned categories. In addition, students believe that the pattern To-Do List also impacts grading objectivity and consistency. The pattern assumes that the teachers provide a checklist of the key parts and tasks of the knowledge assessment.

On the other hand, a significant impact on students’ motivation and satisfaction was perceived for the pattern Appetizer. The pattern assumes that the preparation could be more efficient if students are familiar with the examples of possible assessment questions. Therefore, teachers present example questions to students during the prior exam lectures or even during the lectures throughout the semester. The teachers also emphasise the pattern Proactive Teacher as the pattern with a high impact on students’ satisfaction and the pattern Student Proxy as the pattern with an impact on students’ motivation. Both patterns are intended to avoid unattended situations, which could be especially common when the knowledge assessment is implemented at a distance.

As depicted within the presented analysis, the patterns in the catalogue were categorised into different groups, wherein

Table 8. Impact of pattern group on grading objectivity and consistency, and students’ motivation and satisfaction.

	Objectivity		Consistency		Motivation		Satisfaction
	Students	Teachers	Students	Teachers	Students	Teachers	Students	Teachers
D	4.11	3.58	3.92	3.59	3.85	3.86	3.77	3.68
Q	3.98	3.53	4.07	3.56	4.00	3.45	4.03	3.72
E	4.06	3.50	4.04	3.54	3.91	3.64	3.85	3.75
C	4.23	3.18	4.24	3.25	4.26	3.85	4.27	4.20
A	2.39	2.64	2.36	2.69	1.58	2.18	1.45	2.17
O	2.50	2.28	2.51	2.36	2.71	3.17	2.88	3.54

represents the synthesised average values of the perceived impact on researched views. The table presents four groups of patterns, namely, patterns for the assessment conceptual design (D), patterns for defining questions, answers, and schedules (Q), execution and grading patterns (E), communication patterns (C), anti-patterns (A), and other patterns (O). For each group, the perceived impact on grading objectivity and consistency and students’ motivation and satisfaction is presented, as assessed by students and teachers.

According to the students, the communication patterns most significantly impact the grading objectivity, followed by the patterns for the assessment conceptual design for objectivity, and patterns for defining questions, answers, and schedules for consistency. The group of patterns with the highest impact on students’ motivation and satisfaction were again communication patterns, followed by patterns for defining questions, answers, and schedules, and execution and grading patterns. The analysis of teachers’ perspective, on the other hand, shows that the teachers perceived the group of patterns for the assessment conceptual design as the patterns with the strongest influence on grading objectivity and consistency, which was followed by patterns for defining questions, answers, and schedules. The ranking is understandable since the teachers are much more aware of the importance of knowledge assessment design. At the same time, this aspect was perceived as insignificant for students since they are not actively involved in preparing knowledge assessments. Regarding the perceived impact on students’ motivation, the teachers again ranked the patterns for the assessment conceptual design the highest, while in the area of satisfaction, they assessed communication patterns as the most important.

4.1. Perceived Impact on Grading Objectivity and Consistency

Table 4 and Table 6 provide a summary and highlight the top three patterns for which students and teachers perceived the highest impact on grading objectivity and consistency. The detailed analysis of gathered data, namely, average values for each pattern in the introduced assessment pattern catalogue, is presented in

Table 9. Impact on grading objectivity and consistency perceived by students and teachers.

	Pattern	Objectivity		Consistency
		Students	Teachers	Students	Teachers
D	Open Book	3.94	3.03	3.71	2.82
D	Stagewise Approach	3.68	3.82	3.85	3.88
D	Pentathlon	4.51	4.18	4.50	4.27
D	Continuous Testing	3.84	3.64	3.85	3.91
D	Expert-Level	3.52	3.27	3.37	3.18
D	Innovator	4.02	3.18	3.92	3.09
D	Student Achievement Portfolio	4.29	3.67	4.21	3.79
D	Colleague Bloom	4.02	3.42	3.96	3.76
Q	Expert Validator	4.47	4.33	4.42	4.03
Q	Statistical Validator	4.46	4.24	4.37	4.18
Q	Hidden Validator	4.04	3.27	4.02	3.42
Q	Colleague Veto	4.27	4.18	4.19	4.06
Q	Mathematical Validator	3.57	3.58	3.58	3.55
Q	Professional Multiplier	4.12	3.79	4.04	3.94
Q	Dress Rehearsal	3.96	3.00	3.98	2.94
Q	Example Questions	4.16	3.00	4.18	3.21
Q	Questions Donor	3.96	2.94	3.84	2.91
Q	Theme Variants	4.12	3.48	4.10	3.39
E	Timebox	3.69	3.52	3.65	3.64
E	Randomised Order	4.10	3.67	4.14	3.67
E	Results With Delay	4.45	4.15	4.41	4.15
E	Identity Guarantee	4.10	3.73	4.12	3.55
E	Accessibility Adjustments	4.35	3.55	4.37	3.55
E	Objective Assessor	4.75	4.09	4.58	4.03
E	Criteria List	4.37	4.70	4.35	4.79
E	Eurosong	3.04	3.03	2.96	3.09
E	Bonus Points	4.22	3.12	4.21	3.21
E	Third Shift	4.08	2.82	3.98	2.94
E	Number Draw	3.63	2.55	3.60	2.61
E	Impro League	4.40	3.18	4.46	3.24
E	Self-Assessment	4.04	3.03	4.12	3.12
E	Personal Defence	3.62	3.91	3.54	4.00
C	Appetizer	4.46	3.27	4.52	3.45
C	Game rules	4.65	4.18	4.67	4.24
C	Time Reminder	4.40	2.88	4.42	2.94
C	To-Do List	4.53	3.45	4.61	3.42
C	Emergency Call	4.45	3.06	4.40	3.06
C	Member Channel	4.17	2.48	4.15	2.52
C	Academic Integrity Appeal	4.08	3.55	4.02	3.55
C	Proactive Teacher	3.98	3.18	3.98	3.30
C	Firewall	3.74	3.00	3.74	3.18
C	Student Proxy	3.86	2.73	3.86	2.85
A	24/7	2.48	2.15	2.50	2.18
A	Full-Time Job	2.49	2.24	2.51	2.30
A	Certification Centre	2.53	2.45	2.53	2.61
O	Safe Exam Browser	2.55	2.82	2.53	2.76
O	Big Brother	2.23	3.09	2.19	2.85

. The table is colour-coded:

• Blue standing for students;
• Orange standing for teachers.

Additionally, the top-rated patterns are highlighted with the increasing colour intensity.

While the overlap between students’ and teachers’ perspectives was already seen in Section 4, the colour coding again shows a similar mindset but with some outliers. If we focus solely on values, ignoring the pattern rankings, hereinafter, we will point out the patterns with the largest difference in the average value assessed by the students and teachers. This indicates a difference in perceived objectivity and consistency among students and teachers.

The biggest difference in the assessment regarding the perceived impact on objectivity was for the patterns Member Channel in favour of students and Big Brother in favour of teachers. The difference for the pattern Member Channel was 1.69, wherein students perceived the pattern as more important for influencing objectivity than teachers. The pattern Member Channel is a communication pattern that encourages the use of separate channels where students can communicate in smaller groups. On the other end, the difference in average values in favour of teachers was for the pattern Big Brother. Therefore, with a difference of 0.86 in average values, the teachers perceived the pattern to have more influence on grading objectivity than students. In the published catalogue, we categorised the Big Brother pattern within other patterns. Although we are aware that it is sometimes inevitable to use the mentioned pattern, we believe that the objectivity and efficiency of the knowledge assessment could be ensured by using the combination of other proposed patterns. The mentioned patterns also received the largest difference in average values according to perceived impact on grading consistency. The differences were 1.63 for Member Chanel and 0.66 for Big Brother.

If we deeply analyse the difference between students and teachers for the top-rated patterns, differences can be observed. For example, the difference for the Pentathlon pattern was 0.33 regarding grading objectivity and 0.23 regarding grading consistency on the students’ side. On the other hand, the difference for the Objective Assessor pattern was 0.66 regarding objectivity and 0.55 regarding consistency, wherein again students perceived more impact.

An interesting analysis is also the average values of the patterns regardless of the defined categories. Students perceived the pattern Criteria List as the pattern with the highest impact on grading objectivity. The average value was 4.70. The pattern was followed by the patterns Expert Validator with an average value of 4.33 and Statistical Validator with a value of 4.24. According to the teachers, the patterns that impact the grading objectivity the most were the pattern Objective Assessor, with an average value of 4.75; the pattern Game Rules, graded 4.65 on average; and the pattern To-Do List, with a value of 4.53. According to the students, the patterns with the least influence on grading objectivity were the 24/7 pattern, valued at 2.15; Full-Time Job, with an average value of 2.24; and the pattern Certification Centre, with a value of 2.45. Similarly, teachers assessed the patterns Full-Time Job and 24/7 as the third and second least important pattern, while the pattern with the most minor influence on grading objectivity was, according to the teachers, Big Brother.

4.2. Perceived Impact on Students’ Motivation and Satisfaction

Table 6 and Table 7 provide a summary and highlights the top three patterns for which students and teachers perceived the highest impact on student motivation and satisfaction. The detailed analysis of the gathered data, namely, average values for each of the patterns, is presented hereinafter in

Table 10. Impact on students’ motivation and satisfaction perceived by students and teachers.

	Pattern	Motivation		Safisfaction
		Students	Teachers	Students	Teachers
D	Open Book	4.25	3.73	4.49	4.09
D	Stagewise Approach	3.66	4.06	3.62	3.82
D	Pentathlon	4.34	4.36	4.11	4.09
D	Continuous Testing	3.51	4.06	3.25	3.45
D	Expert-Level	3.17	3.24	2.92	3.03
D	Innovator	4.11	4.18	4.13	4.12
D	Student Achievement Portfolio	3.84	4.00	3.82	3.61
D	Colleague Bloom	3.90	3.24	3.80	3.24
Q	Expert Validator	4.08	2.94	4.21	3.42
Q	Statistical Validator	4.11	3.09	4.17	3.70
Q	Hidden Validator	4.09	4.21	4.08	4.09
Q	Colleague Veto	3.98	3.18	4.12	3.58
Q	Mathematical Validator	3.24	2.91	3.10	3.33
Q	Professional Multiplier	3.84	3.18	3.88	3.73
Q	Dress Rehearsal	4.04	3.39	4.02	4.00
Q	Example Questions	4.31	4.21	4.48	4.30
Q	Questions Donor	4.52	3.36	4.44	4.18
Q	Theme Variants	3.82	3.03	3.82	2.91
E	Timebox	2.94	2.61	2.73	2.42
E	Randomised Order	3.16	2.73	2.90	2.48
E	Results With Delay	3.80	2.82	3.56	3.39
E	Identity Guarantee	3.19	2.88	3.10	2.76
E	Accessibility Adjustments	4.47	4.06	4.48	4.39
E	Objective Assessor	4.69	3.76	4.73	4.39
E	Criteria List	4.26	3.55	4.22	4.15
E	Eurosong	3.48	4.27	3.63	3.91
E	Bonus Points	4.81	4.55	4.75	4.58
E	Third Shift	4.49	4.00	4.61	4.64
E	Number Draw	3.51	3.94	3.18	3.06
E	Impro League	4.66	3.64	4.73	4.42
E	Self-Assessment	4.36	4.15	4.39	4.36
E	Personal Defence	2.90	4.06	2.88	3.48
C	Appetizer	4.73	4.24	4.77	4.48
C	Game rules	4.67	4.00	4.63	4.52
C	Time Reminder	4.40	3.58	4.40	4.27
C	To-Do List	4.62	3.79	4.56	4.27
C	Emergency Call	4.63	3.97	4.75	4.52
C	Member Channel	4.43	4.00	4.40	4.06
C	Academic Integrity Appeal	3.65	3.42	3.73	3.58
C	Proactive Teacher	3.98	4.30	3.92	4.30
C	Firewall	3.45	3.21	3.43	3.85
C	Student Proxy	4.06	4.03	4.06	4.18
A	24/7	2.62	3.79	2.75	4.33
A	Full-Time Job	2.98	3.45	3.16	3.82
A	Certification Centre	2.52	2.27	2.73	2.48
O	Safe Exam Browser	1.58	2.09	1.48	1.85
O	Big Brother	1.57	2.06	1.42	1.76

. The table is colour-coded:

• Blue standing for students;
• Orange standing for teachers.

Additionally, the top-rated patterns are highlighted with the increasing colour intensity.

The biggest difference in the average values when assessing the perceived impact on students’ motivation was detected for the pattern Expert Validator and pattern 24/7. With the difference of 1.14, students perceived the greater influence of the Expert Validator pattern on their motivation. Therefore, students believe that using a colleague teacher or domain expert within the questions and answers validation significantly impacts motivation. On the other hand, the biggest difference was detected within the assessment of the 24/7 pattern. The teachers had a stronger perception of the influence of the anti-pattern on students’ motivation, with a difference of 1.17 within average values. The same pattern also occupies the same position in the assessment of the impact on students’ satisfaction, wherein the difference was even bigger, namely, 1.58. The 24/7 pattern is an anti-pattern covering unrealistic student expectations regarding teachers’ online responsiveness. However, since students assessed the pattern impact with only 2.62, we could assume that students themselves perceive the pattern as an anti-pattern, whereby they understand the reasonable concept of teacher availability.

Looking into the top-rated patterns regarding their impact on student’ motivation and satisfaction, we can detect only minor differences. Again, a very highly rated pattern is the Pentathlon pattern, wherein the difference between students’ and teachers’ assessments was only 0.02 for both aspects: motivation and satisfaction. A slightly greater difference was seen for the pattern Bonus Points: the pattern within the top three patterns from the execution and grading patterns. The difference when assessing the impact on motivation was 0.26, and when assessing the impact on satisfaction, it was 0.17.

The pattern Bonus Points was assessed with the highest average value when assessing the patterns’ impact on students’ motivation. Students assessed the pattern with an average value of 4.81 and teachers with 4.55. The pattern was followed by the patterns Pentathlon and Question Donor with 4.36, and the pattern Proactive Teacher with 4.30 from the teachers’ point. The students’ perspective presented with a slightly different sequence: the Bonus Points pattern was followed by Appetizer with 4.73 and Objective Assessor with an average value of 4.69. Regardless of the categories, the highly rated patterns in students’ satisfaction slightly overlapped between students and teachers. The highest teachers’ average values were given to the patterns Third Shift, Bonus Points and Emergency Call, namely, 4.64, 4.58, and 4.52. On the other hand, with the highest average values, students’ assessed Appetizer with 4.77, Bonus Point and Emergency Call with 4.75, and Impro League and Objective Assessor with 4.73, as the patterns with the greatest perceived impact on students’ satisfaction.

5. Discussion

Our catalogue of assessment patterns for distant education provides an extended group of proven practises, complementing the existing pedagogical patterns research domain (e.g., [24,25,26,27]), especially providing an essential addition to the knowledge assessment domain. The presented study explored the perceived impact of the defined patterns on grading objectivity and consistency and students’ motivation and satisfaction.

We defined a research instrument to implement the assessment. The questionnaire uses a Likert scale, wherein the results are presented as average values from students’ and teachers’ perspectives. The research was formed following two research questions, wherein the results are presented in Section 4 and the associated tables. The existing literature does not present any proven or established validation approach; moreover, Fioravanti and Barbosa [35] stated that there is a gap in the validation of proposed pedagogical patterns. For example, Köppe et al. [28] researched the proposed patterns, looking into students’ and teachers’ value and the effect on the final assessment. However, a research instrument was not defined, significantly affecting the possibility of expansion and reproducibility. Therefore, the defined questionnaire, presented in Table 2, significantly contributes to the existing body of knowledge.

Table 4, Table 5, Table 6 and Table 7 and Table 9 present and depict the results of the analysis addressing RQ1. We researched which patterns from the created catalogue significantly impact the grading objectivity and consistency. Table 9 presents a detailed analysis, assessing top patterns regarding the categories. It can be observed that the patterns with the highest perceived impact on grading objectivity are, according to the students, Objective Assessor, Game Rules, and To-Do List, and according to the teachers Criteria List, Expert Validator, and Statistical Validator. The patterns with the highest perceived impact on grading consistency are, according to the students, Game Rules, To-Do List, and Objective Assessor, and according to the teachers, Criteria List, Pentathlon, and Game Rules. We detected a great overlap, with some differences connected to patterns’ ratings.

The results of the analysis of the perceived impact on students’ motivation and satisfaction, as researched with RQ2, are depicted in Table 4, Table 5, Table 6 and Table 7 and Table 10. As can be observed from Table 10, the top-rated patterns regarding their impact on students’ motivation are, according to the students, Bonus Points, Appetizer, and Objective Assessor, and according to the teachers, Bonus Points, Questions Donor, and Pentathlon. Researching the area of students’ satisfaction, the top-rated patterns, according to the students, are Appetizer, Bonus Points, and Emergency Call, and according to the teachers, Third Shift, Bonus Points, and Emergency Call.

The implemented analysis also offers some additional insights into the considered domain, presented in Table 4, Table 5, Table 6 and Table 7. First, we will focus on communication patterns, since the patterns from this group are highly important for every researched area. As depicted in Table 7, we can see that the top three patterns, according to the students, with the highest perceived impact on grading objectivity are Game Rules, To-Do List, and Appetizer. On the other hand, the top three communication patterns with the highest perceived impact on students’ motivation, according to the teachers, are Proactive Teacher, Appetizer, and Student Proxy. Further, let us also look more deeply into the patterns for defining questions, answers, and schedules. We can see from Table 5 that according to the students, the top three patterns according to perceived impact on grading consistency are Expert Validator, Statistical Validator, and Colleague Veto. The top three patterns, according to the teachers, with the greatest perceived impact on students’ satisfaction are Example Questions, Question Donor, and Hidden Validator. The overview of Table 4, Table 5, Table 6 and Table 7 reveals the great overlap between students’ and teachers’ perspectives, wherein the top three patterns, if the rating is ignored, are often similar.

Our research was performed in the higher education environment, including students and teachers from IT-oriented study programmes. Therefore, our participants possessed advanced digital skills, which can foster a more positive attitude towards remote education and knowledge assessments as a whole, as they do not need to deal with the technological component. In addition, a positive attitude towards technology could also slightly affect the assessment score of the patterns with a stronger connection to IT, for example, communication patterns, wherein the communication path is realised remotely using technology. However, the defined patterns do not require advanced IT knowledge for teachers or students. Therefore, the results of the study are not contradictory if the backgrounds of the participants were different.

During the presentation of the results and detailed analysis, it was noticed that the applicability of the majority of presented patterns was not limited to remote knowledge assessments. Undoubtedly, a variety of them could also be used in the classroom setting. Therefore, one of our previous papers [36] researched the classroom applicability of the introduced catalogue. According to the results, the subset of the assessment patterns most suitable for the classroom consists of 12 patterns, among which are Pentathlon, Expert Validator, Colleague Veto, Bonus Points, Game Rules, Time reminder, and other [36].

Limitations

As in every research, the results can be affected by various factors. The limitations and potential threats to validity are presented hereinafter. The research participants were from the IT domain, namely, students in IT-oriented study programs and teachers involved in IT-oriented study programmes. Therefore, all of the participants had advanced IT knowledge. The research was performed in the higher education domain, involving students currently enrolled in the Master’s Degree study programme.

6. Conclusions

Different situations can confront us with previously unknown challenges, forcing us to find, define, and use proven possible solutions. One of the recent challenges we have had to cope with was distant education, which, practically overnight, became a part of our everyday life. The settings previously reserved for in-person sessions were moved to digital environments, including the knowledge assessment process. Some of the previous practises were, therefore, not efficient anymore, and a need for new and proven practises arose.

As an answer, we defined a novel catalogue of assessment patterns for distant education [14]. The final step of catalogue creation was the validation of the defined knowledge assessment patterns, wherein this paper presents the survey results implemented among teachers and students. The objective of the survey was to verify its suitability and expedite its use, focusing on several aspects of knowledge assessment. We addressed the validation from different angles, addressing grading objectivity and consistency, and students’ motivation and satisfaction. With the use of a uniform questionnaire, we gathered perspectives about the perceived impact on researched domains. According to the results, patterns such as Pentathlon, Expert and Statistical Validator, Game Rules, Results with Delay, and Objective Assessor had a high perceived impact on grading objectivity and consistency according to students and teachers. On the other hand, patterns such as Innovator, Example Questions, Question Donor, Bonus Points, and Appetizer had a high perceived impact on students’ motivation and satisfaction. The paper presents a detailed analysis, depicting top-rated patterns within defined pattern categories with corresponding average values, allowing an in-depth comparison among different aspects and views.

In future work, we plan to develop our catalogue of knowledge assessment patterns even further. Since one of our previous studies clearly confirmed the suitability of defined patterns for classroom settings, the catalogue’s further development is meaningful, wherein new and arising challenges must be addressed. One of them is, of course, practises related to the increased involvement of AI in the teaching process. We also plan to push catalogue dissemination further, by providing some additional analyses regarding the attitude towards the catalogue. Currently, the catalogue’s online collaborative platform is under construction in a limited number of languages. More translations will allow further propagation and development and the ability to propose new patterns.