On the Design of Constructively Aligned Educational Unit

Abstract

Modern pedagogy is moving away from traditional transmissive approaches, and it is extensively embracing constructive theory of learning. A prominent practical embodiment of this paradigm shift is a method called Constructive Alignment (CA). This approach focuses on learners’ actions and starts from a clear communication of the Intended Learning Outcomes (ILOs) of the focal unit. ILOs are made of content, a context, and an Educational Goal Verb (EGV). According to the Bloom Taxonomy, the EGV is the core of an ILO and refers to the action the learners are expected to be able to master after completing the educational unit. The ILO is then aligned to the course activity using the EGV (i.e., EGVs are enacted through Teaching and Learning Activities (TLAs) and verified through Assessment Tasks (ATs)). Despite the ILO definition being extensively investigated and described, the extant literature has poorly explored how to devise suitable TLAs and ATs, lacking comprehensive contributions that identify and describe the different kinds of TLAs and ATs available to course designers. In view of the above gap, the authors searched and reviewed the literature (scientific papers (i.e., top-down, deductive approach)) and practices in higher education (university websites and blogs (i.e., bottom-up, inductive approach)) to identify all the possible sources of TLA and AT descriptions available. The results propose standardized templates that support the course design process, providing extensive descriptions of TLA and AT based on the best practices identified. The proposed templates include the core dimensions that proved to be suitable for designing traditional and remote-learning activities. Finally, the examples provided in the paper show how to use these templates on a few kinds of selected on-campus and digital TLAs and ATs from the educational units identified in the Erasmus+ MAESTRO project, which is based on Industry 4.0 technological enablers and their application in support of manufacturing sustainability.

1. Introduction

An increasing number of higher education professionals are embracing a modern theory of learning based on a constructivist approach. Such an approach contrasts with the traditional passive transmission of knowledge between teachers and learners, in which the teacher is the focal figure of the learning process. Constructivism emphasizes that the learners build their knowledge through their own personal experiences and previous knowledge [1]. This theory focuses on a learner-centered model that puts emphasis on the learners rather than the teachers.

The constructivist theory inspired Biggs, who ultimately proposed Constructive Alignment (CA) in [2]. CA builds on the constructivist understanding of the learning process and is based on the principle of an aligned and outcome-based curricula design [2]. Implementing CA means devising Intended Learning Outcomes (ILOs) that are based on an Educational Goal Verb (EGV). The EGV refers to the action the learners are expected to learn after completion of the educational unit. The ILO, and related EGV, are then enacted through Teaching and Learning Activities (TLAs) and verified through Assessment Tasks (ATs). The alignment between ILOs, TLAs, and ATs is achieved by using the same action verb (i.e., EGV).

The salient aspects of the CA have been recently extracted and classified in a comprehensive taxonomy, called the ConstructiveAlignment (CONALI) ontology [3]. This ontology fulfills the current need to have a structured framework that describes and manages educational knowledge [4]. The main use of ontology is thus to collect and maintain educational knowledge, nowadays in the form of 3V (i.e., volume, variety, velocity) big data, as well as allow advanced analysis of the represented domain [5]. Working proficiently with big educational data requires a model that adds semantic meaning to the otherwise simple numerical data collected. As stated by the authors of the CONALI ontology [3,5], this effort is thus a fundamental support for the use of educational data and the emerging field of learning analytics.

The process of populating the ontology with instances can yield a knowledge base (i.e., well-structured data are paramount to enhance and improve the learning process). However, the lack of standardized CA data with respect to ILOs, TLAs, and ATs to populate such ontology is the main obstacle to its broader application and adoption, despite the significant potential already shown. TLAs and ATs are still sources of vague and shaded descriptions, while ILO description has lately converged to a standard template as numerous references have proposed a quite homogeneous and advanced view of the ILO’s attributes or dimensions.

Moreover, the education domain has been challenged by the ongoing digitization trend that is promoting the introduction of distance learning practice. Education designers seek support during the entire educational unit design process to adjust the courses to accommodate online approaches to their teaching. Such support is lacking as a comprehensive and homogeneous description of TLAs and ATs is lacking.

Given the above, this paper focuses on the definition of structured and standard dimensions of two of the three pillars of CA (i.e., TLA and AT). In particular, this work contributes to (i) identifying all the possible sources of TLA and AT descriptions available in the literature and on university websites, (ii) providing a standardized approach to describing TLAs and ATs based on best practices identified, which will ultimately populate the CONALI ontology, (iii) listing all the possible kinds of TLAs and ATs identified from the different sources analyzed, (iv) classifying them according to which EGV they are suitable for, and (v) providing examples of how to use this standard description on a few selected TLAs and ATs. The standardized template we propose will support educators in their endeavors to design constructively aligned courses.

The remainder of the paper is structured as follows. Section 2 provides a detailed overview of the CA and a discussion of the abovementioned lack of TLA and AT standardization. Section 3 describes the adopted methodology for the identification of the best practices investigated in the literature and on university websites for TLA and AT definitions. Section 4 provides an extended description of the information extracted and the final dimensions identified to standardize TLAs and ATs. It also reports two case studies on the design of courses adopting the proposed templates for validation. Section 5 and Section 6 discuss and conclude the work.

Table 1. List of acronyms in alphabetical order and their extended definitions.

Acronym	Definition
AM	Additive Manufacturing
AT	Assessment Task
CA	Constructive Alignment
CAA	Computer-Assisted Assessment
CAD	Computer-Aided Design
CoI	Community of Inquiry
CONALI	ConstructiveAlignment
EU	European Union
EGV	Educational Goal Verb
ILO	Intended Learning Outcome
KTH	Kungliga Tekniska Högskolan (Royal Institute of Technology)
LMS	Learning Management System
MAESTRO	Manufacturing Education for a Sustainable Fourth Industrial Revolution (an Erasmus plus project funded by the EU)
SOLO	Structure of the Observed Learning Outcome
TLA	Teaching and Learning Activity
3D	Three-Dimensional
UNIPI	University of Pisa

reports the acronyms used throughout the paper.

2. Literature Review

In this section, we report the main aspects of the CA approach (Section 2.1), focusing on detailed descriptions of the three pillars: ILO, TLA, and AT in Section 2.1.1, Section 2.1.2 and Section 2.1.3, respectively. Section 2.2 defines the research gap that set the stage for the current paper.

2.1. Constructive Alignment

The teacher’s role in the conventional learning model is to provide mainly frontal instructions, while the learners are relatively passive recipients of information and primarily perform individual work. This learning model is usually known as teacher-structured (i.e., a teacher-centered approach focused on what the teacher does). Modern pedagogy is moving away from such a model and adopting a constructivist approach to learning based on constructivist theory. This theory is built on a learner-centered model that puts emphasis on the learners rather than the teachers. The learners construct their knowledge [1] by bringing in a combination of previous knowledge, assumptions, motives, and intentions [2]. The learners are thus incentivized to actively participate in their learning process, mainly by being involved in collaborative group works [6].

With the constructivist theory in mind, Biggs proposed a systematic approach to course design, called CA [2]. The CA approach is based on the following activities:

-

Describe the ILO (i.e., what should the learners be able to do after completing the course?);

-

Create the TLA (i.e., what work must the learners do to reach the desired outcomes?);

-

Use AT (i.e., what would the learners do to prove they have reached the learning outcomes?).

The baseline of CA is that the TLA and AT must be aligned with the identified ILO. The alignment ILO-TLA-AT is realized with “action” verbs (i.e., EGVs) first specified in the ILO. The EGVs create a common link between the three elements [7]. Figure 1 presents how the three pillars of CA are aligned according to Biggs.

These EGVs are selected by looking at the modified Bloom taxonomy (Figure 2, right) [8]. The taxonomy classifies the learning objectives into six levels of understanding, from mere declarative knowledge represented in the semantic domain of remembering and understanding, to functioning knowledge that underpins actions by learners such as applying, analyzing, evaluating, and creating. Each level of understanding corresponds to a list of verbs that activate the corresponding abilities for a specific learning goal, as listed in Figure 2, along with the corresponding level of the Structure of the Observed Learning Outcome (SOLO) taxonomy (i.e., unistructural, multicultural, relational, extended abstract (Figure 2, left)).

The CA approach falls under the umbrella of outcome-based teaching [11]. Designing a course following the CA approach will create coherency that may improve the learners’ effectiveness in terms of the learning experience and academic grades [12].

2.1.1. ILO

The learners are provided with a set of ILOs that clearly specify the learning goals and competences that the learners are expected to achieve after attending a course. Biggs [13] proposed the SOLO taxonomy to clarify the learning outcomes for learners’ understanding levels (i.e., prestructural, unistructural, multistructural, relational, and extended abstract (from low to high, as shown in Figure 2, left)). Corresponding verbs dictated in the Bloom taxonomy (Figure 2, right) create nuances in the learning outcomes from a lower to a higher level of understanding that, in turn, reflect a challenge in the process of thinking. Selecting suitable verbs is thus important to define learners’ future competencies in a course, related to developing the intended learning outcomes [14].

Accordingly, the ILOs are expressed from the learner’s perspective. Their basic elements are (1) a verb (i.e., reflecting the educational goal and related level of understanding required), (2) content (i.e., the focal concept within the discipline), and (3) context (i.e., the domain in which the content is studied).

2.1.2. TLAs

The TLAs need to be designed after the learning outcomes are established to ensure that the chosen TLAs will fully realize the learning objectives [15]. The verb included in the TLA must reflect the one stated in the ILO to ensure their alignment.

There is a two-way interaction between the degree of learning-related activity and the teaching-related activity that stimulates the learning process as they jointly affect the level of engagement of the learners. On the one hand, teaching activities such as standard lectures or seminars require a passive learner activity that reflects a low level of learners’ engagement (e.g., explaining or describing). On the other hand, teaching activities such as case studies or debates require an active learner activity, and consequently, a higher engagement level (e.g., applying or theorizing) [15]. The most important task of a teacher is to set up suitable TLAs to engage learners to a maximum degree to achieve the level of understanding specified in the related ILO [15,16].

As shown in Figure 3, while the learners may play a passive role in adopting a surface approach, it is the course that provides the fertile ground on which this crops up.

The choice of active learning approach is very important to enact the EGV, meaning it is one of the seven principles for good practice in undergraduate education [17] that are the basis of effective TLA design in modern pedagogy. The rise of digital tools in the late 90s expanded the opportunities for learning by leveraging computer-mediated communication. In this context, the Community of Inquiry (CoI) framework emerged as a process of creating deep and meaningful (collaborative-constructivist) learning experiences through the development of three interdependent elements—social, cognitive, and the teaching presence [18]. The process relies on the seven principles that support CoI formation [19].

2.1.3. ATs

Following the well-known Deming cycle proposed for the quality-management domain and applicable to virtually control all kinds of activity [20], the AT can be considered the check (e.g., monitoring) stage, as shown in Figure 4.

Assessment design can have a big influence on what learners get out of the TLA [21,22,23]. Learners must not only study what is necessary to pass their assignments and exams. The assessment should serve as a means to measure the learners’ progress and get good feedback on their capabilities [24].

2.2. Gap in the Literature

The literature has extensively investigated the importance of designing an aligned curriculum embracing the CA approach. Scholars and practitioners have clearly and extensively agreed on the ILO definition, its dimensions and there is wide consent on the ILO being the basis for designing learning activities and assessment. However, the extant literature lacks comprehensive contributions that describe TLAs’ and ATs’ characteristic variables.

This paper reviews the literature (scientific papers) and practices in higher education (university websites and blogs) to exhaustively identify the possible sources of TLA and AT descriptions available. The result of this review is a set of standard templates that include the identified core dimensions of TLAs and ATs. These templates will support educators in the design process of aligned courses.

3. Methods

Concerning the identified research gap, the literature and best practices in higher education were investigated. The available sources presented several TLA and AT descriptions, as retrieved from scientific papers and websites, or blogs of worldwide universities engaged in advanced higher education. The websites were selected through the Google browser using the keywords “university”, “teaching and learning activities”, “assessment task”, “examples”, and synonyms (i.e., higher education and template). The relevant literature in the field was retrieved from scientific databases (i.e., Scopus and WebofScience) as they cover the most relevant available literature regarding the unit of analysis of this study. The articles were extracted using the following search string (“teaching and learning activities” OR (“assessment task”) AND (“course design” OR (“constructive alignment”)). The results of this search were filtered to exclude all publications not related to higher education. The expression AND NOT (“elementary school” OR “undergraduate” OR “high school”) was then added to the search string. An initial dataset of 100 papers, 50 universities’ websites, and 10 higher educational blogs was scanned to create a final set. Screening the abstracts allowed us to identify the most relevant publications for the scope of the study. The websites were manually analyzed and selected based on the university ranking and high relevance of their work in education design. The blogs were selected based on the reliability of the source and relevance of their content. The final dataset comprised 30 papers published in indexed and high-impact journals, 16 official university websites, and 2 blogs, as well as cross-referencing the citations reported on the university websites. This final set contained the most relevant sources and provides sufficient data. As a result of a thorough analysis, the resulting dataset allowed us to reach the saturation level for the purpose of this study (i.e., any other source analyzed would not bring any additional information to the study).

A cross-analysis approach was performed on the collected sources to extract the main dimensions that best characterized TLAs and ATs. The identified dimensions were thus abstracted, synthesized, and finally proposed in comprehensive and unified templates. The proposed templates aim at eliminating and merging all the redundancies in the literature, utilizing the best practices, and providing a standardized approach to describe TLAs and ATs.

Figure 5 shows the main steps of the methodology adopted in this paper.

To validate the proposed templates, a specific set of on-campus and digital TLAs and ATs was selected from the educational units identified in the Erasmus+ Manufacturing Education for a Sustainable Fourth Industrial Revolution (MAESTRO) project [25], to provide examples on how to use the proposed standard templates and validate the proposed standard.

4. Results

This section reports and summarizes the information retrieved from the scientific literature and the university websites and blogs, divided into TLA and AT dimensions. The section concludes with the validation of the proposed templates.

4.1. TLAs

Table 2. Dimensions and subdimensions identified via the information retrieved from the listed paper and the websites analyzed.

TLA Dimension	TLA Subdimension	Scientific Papers	Universities Websites and Blogs
Teaching and learning tasks (Section 4.1.1)	Teaching and learning task description and purpose	[7]	[26,27,28,29,30,31]
	Teaching and learning task type	[32,33,34]
Interaction (Section 4.1.2)	-	[33,35]	[26,27,28,30,31,36]
Alignment ILO-TLA (Section 4.1.3)	Soft skills activated	[37]	[26,27,28,38]
	Action verb (EGV)	[7,16,39,40]	[26,28,30,31]
Time requirements (Section 4.1.4)	-	-	[26,30,36]
Technological support (Section 4.1.5)	-	[34,41]	[26,27]
Mode and location of delivery (Section 4.1.6)	-	-	[26]

shows the identified dimensions for TLAs. Following that, such variables are described in detail.

4.1.1. Teaching and Learning Tasks

In this section, the retrieved information about teaching and learning tasks is summarized. The sources analyzed identified the following subdimensions:

• Teaching and learning task description and purpose

All the university websites and blogs analyzed identified this dimension that describes the intent of the activity and defines what the teacher does (e.g., presents a scenario) and what the learner does (e.g., analyze the case study and propose solutions) [7].

• Teaching and learning task type

This variable specifies what kind of activity aims to activate the EGV used in the ILO. Several activity types were identified from the literature and university websites, largely from [26,27,28,29,30,31,32,33,34]. Based on the level of engagement required, we can classify the learning activity as passive or active (Figure 3).

In passive learning, the level of engagement of the learners is medium-low. These activities correspond to basic educational learning objectives specified in the Bloom taxonomy such as remembering or understanding (Figure 2). A typical example of passive learning is the standard lecture. The standard lecture is meant to be information-giving for a large group of learners, supported usually by slides. Teachers can decide to introduce short interactions to involve the learners more. For instance, asking a question that requires them to summarize, explain, or identify an important aspect of the information just presented. Bain [32] specifies the typical key elements to make a lecture more interactive: they start with a question, then try to help learners understand the question’s significance (e.g., its implication, its connection to larger questions), they then encourage learners to engage critically with the question, to draw on evidence and reasoning to make arguments about how it should be answered, and they finish with a conclusion (which often does not include their own ‘answer’) [32].

According to [33], active learning is any learning activity that engages learners more than merely listening passively to an instructor’s lecture (Figure 2). These activities thereby correspond to advanced educational learning objectives specified in the Bloom taxonomy (Figure 2) such as apply, analyze, or create. The literature has identified a large spectrum of active learning activities. The list of activities presented below was compiled from the following sources [26,27,33,34].

-

Case studies: the teacher presents to the learners with a ‘story’ or scenario, e.g., a case of how a scientist identified new physics regularity [34]. The learners, divided into groups, discuss the scenario and work on answering related questions;

-

Group discussion: small groups are given a topic to discuss. The discussion is structured by presenting a set of questions the groups are supposed to focus on when giving their arguments. The active participation of each group member is maximized when the group is kept small (up to six people);

-

Fishbowl: this activity involves two groups: one discusses a topic and the second one observes the discussion. The observing group records their peers’ contributions and provides individual feedback afterward;

-

Peer evaluation: this activity is meant to be in pairs. The goal is for the involved learners to assess each other’s written work or oral presentation and give each other feedback focused on delivery and/or content;

-

Laboratory: the instructors give a set of guidelines to help the learners to navigate the concepts behind the experiments. The learners’ task is to observe a phenomenon, ask questions, and suggest a hypothesis or model. This activity creates the link between the theory presented in class and practice within a certain domain;

-

Presentations: learners, divided into groups or working as individuals, conduct research on a given topic, then prepare and deliver a short informative session to the wider group;

-

Panel: several ‘experts’ from industry or academia are invited to a session where each of them may briefly speak before the question session is opened;

-

Brainstorming: learners think and share as many different ideas as possible within the group. All ideas are recorded. The ideas are evaluated after a set time or when inspiration ends;

-

Mind maps: groups of learners discuss a given topic, suggesting and organizing ideas and information into clusters and determining how those clusters interact;

-

One-minute paper: the teacher poses a specific question that reflects a certain aspect of today’s lecture. The learners are given a few minutes to respond. The teacher collects the responses and assesses them;

-

Daily (or Weekly) journal: this activity allows a more in-depth discussion of or reaction to course material. Learners apply course concepts to their own life experiences and explore course content in broader contexts, including public debate;

-

Jigsaw method: as a collaborative learning method, learners work in groups. Each group has a specific piece of information in which, after some research, they become experts. The groups are shuffled: each member of the new groups has expertise in each area and they start discussing those areas;

-

Roleplay: groups/pairs/individuals ‘act out’ information on a specific topic, often in front of the class or group. A time limit is set for each group. It is important to allow time for participants to de-role/debrief. For instance, defending science experiments with small animals [34];

-

Debates: learners, divided into small groups, are assigned debate themes. Each group is split into teams, to which a role of evaluator and a position to defend (i.e., affirmative (for the theme) or negative (against the theme)) is given. During the debate, one team presents its arguments, and the opposite team responds with rebuttals. This format is repeated in cycles until the time is over [34].

4.1.2. Interaction

The university websites and blog [26,27,28,30,31,36] identified the interaction dimension. It emerges that different kinds of interaction may occur during the TLA (i.e., whether the learners need to carry out the activity individually, in pairs, or in groups). As for the group activities, the learner-learner interaction will be a collaborative exchange of ideas aimed at reaching the goal of the activity. The interaction serves as support for the learner to develop and enhance their learning skills. Such support may allow learners to overcome difficulties in their learning process. Studies have shown that small-group activities increase retention of the concepts and promote learners; better performance [35]. Moreover, the participation and engagement of each learner is paramount to reaching the objectives of the activity, promoting active and cooperative learning [33].

Distance learning has challenged the interaction among learners, and it is taking time for both learners and teachers to adjust to the innovative interaction platforms provided by digital technologies. Such a platform creates a new means of interactions, e.g., through a camera or chat [42].

Remote learning has also challenged how collaboration happens among learners. In a distance education context, the learners cooperate for their assignments solely online. The interaction among learners happens with extensive use of digital platforms, e.g., video conferencing tools, collaborative tools such as cloud and chat services, and blogs, which guarantee and increase the interaction among learners [43].

4.1.3. Alignment ILO-TLA

The most important task of a teacher is to design teaching and learning activities that may ultimately engage and guide learners to achieve the desired ILOs [16]. The ILO-TLA alignment makes sure that the selected learning activities enable learners to develop the skills, knowledge, and understandings required to achieve the unit ILOs.

The following subdimensions are identified:

• Soft skills activated

The following universities and blog [26,27,28,38] have specified that a TLA enhances specific soft skills, e.g., writing, presentation, deep thinking, critical thinking, questioning, and discussion facilitation skills, and active listening. Different activity types activate different soft skills. For instance, a debate activity requires learners to practice their information research skills when they seek evidence for their arguments, their presentation skills when they present their arguments to the opposition, and their critical thinking and questioning skills when they evaluate the arguments; a brainstorming activity requires learners to engage in the generation of creative thoughts, active listening, and discussion of others ideas, through which learners learn how to collaborate in groups and learn from each other; a laboratory requires learners to investigate, make a hypothesis, solve problems, use their critical thinking while assessing the phenomena, communicate, and collaborate [37].

• Action verb (EGV)

The alignment between TLA-ILO is created through the EGV (i.e., the action verb). The EGV emerges as an important dimension to be specified in the TLA description and can be observed in the workings of the following universities [26,28,30,31].

Each EGV corresponds to a certain level of knowledge and understanding expressed in the Bloom taxonomy.

Table 3. Connection between the level in the Bloom taxonomy to the specific educational goal verb and related learning activity examples.

Bloom Taxonomy	EGV	Learning Activities (Examples)
Remember	arrange, define, list, identify	Lecture, Reading materials
Comprehension	classify, discuss, present, rewrite	Mind map, Think-pair-share, Discussion, Reflection, Fishbowl
Apply	solve, calculate, demonstrate, organize, use	Case study in real-life situation, Problem-solving tasks, Roleplay, Group work, Laboratory
Analyze	categorize, contrast, compare, debate, inspect	Debates, Class discussion, Jigsaw method, Think-pair-share, Fishbowl, Laboratory
Evaluate	assess, conclude, justify, measure	Journal, Debates, Mind map, Peer evaluation
Create	design, develop, revise, formulate	Brainstorm, Design a presentation, Create a new report, Construct a roleplay

was compiled from the following references [28,39,40]. These references specify the learning activity type and the corresponding level of understanding according to the Bloom taxonomy.

4.1.4. Time Requirements

This dimension identifies the amount of time the tasks will take to be completed. Such time depends on the type of task. For instance, a “one-minute paper” activity will require a few minutes, while a debate activity will take a few hours. It can also be defined as an interval of time [26,30,36].

4.1.5. Technological Support

The technology support dimension identifies what support is needed to deliver the TLA, e.g., slides, video conferencing tools, digital whiteboard, clickers to poll the class, a recorded lecture with quizzes [26,27]. [41] addresses the importance of the use of technology as a means to increase the quality of learning. Integrating education technologies in teaching and learning practice may stimulate learners’ engagement with learning activities, making them, in turn, more active during their learning process and promoting a deeper understanding of concepts [34].

Technological support has been identified as an essential pillar of digital learning. Digital technologies are fundamental to create interaction and collaboration among learners and teachers. Such technologies become the means for the learning process in remote learning [42].

4.1.6. Mode and Location of Delivery

The University of Tasmania [26] clearly specifies the mode of delivery. Teachers will define the most appropriate mode to support the learners’ development of skills, knowledge, and understanding. The specified modes are on-campus (learning activities that occur when all learners are in the same physical space at the same time can support achievement of a wide range of learning outcomes), online (learning activities that are managed using the online environment can support achievement of a wide range of learning outcomes), and situated (fieldwork, practicums, and other situated learning can support achievement of a wide range of learning outcomes).

4.2. ATs

Table 4. Dimensions and subdimension identified via the information retrieved from scientific papers and the universities’ websites analyzed.

AT Dimension	AT Subdimension	Scientific Papers	Universities Websites and Blogs
Assessment tasks (Section 4.2.1)	Goal	[44,45]	[26,30,46,47,48,49,50,51,52,53,54,55]
	Activity type	[56,57,58,59,60,61]
	Timing	[23]
	Description/purpose/formatting requirements	[62]
Feedback (Section 4.2.2)	-	[63,64,65,66,67,68,69,70,71,72,73,74]	[26,30,46,47,48,49,50,51,52,53,54,55]
Alignment ILO-AT (Section 4.2.3)	Soft skills activated	[23]	[26,30,46,47,48,49,50,51,53,54,55]
	Action verb (EGV)	[75,76]
Marking criteria (Section 4.2.4)	-	[77,78]	[26,46,47,48,49,50,51,52,53,54]
Technological support (Section 4.2.5)	-	[72,79,80]	[26,46,47,49,50,51,52,53,54]
Plagiarism and cheating (Section 4.2.6)	-	[81,82,83,84,85,86]	[46,50,51,53,54]

shows the identified dimensions for TLAs. Following this, such variables are described in detail.

4.2.1. Assessment Tasks

In this section, we summarize the information retrieved about assessment tasks. The scientific literature is highlighted for each bullet point. As for university websites, the following sources have been considered for all the bullet points in this subsection [26,30,46,47,48,49,50,51,52,53,54,55].

• Goal

Formative assessment happens throughout the course. The primary goal of this kind of assessment is to provide immediate and meaningful feedback to the learners on their progress, enabling them to reflect on where they may be going wrong and allowing them to improve [44];

Summative assessment is generally at the end of a course or unit, such as a final exam or major essay. This kind of assessment is used to make judgments and formally measure learner achievement against learning outcomes [45].

• Activity type

The assessment methods referenced below have been compiled from the following sources [56,57,58,59] and from all the universities’ websites [26,30,46,47,48,49,50,51,52,53,54,55]. Assessment tasks can take many forms. Some of the more common assessment tasks are exams, tests, and essays. Examples of assessment types are listed below in alphabetical order. Each university may have specific types of assessment, and specific handbooks should be checked for details.

Among written assessments tasks, we report the following:

-

Abstract: learners are required to write an abstract of a research paper/article within a specified word limit, e.g., 200–300 words;

-

Annotated bibliography: learners produce a list of texts, primary sources, and internet sites on specified or agreed topics to a particular referencing convention. They annotate these with a commentary, which could include an evaluation of what they have read;

-

Article: a written piece on a specific topic in a specific style for a specific audience;

-

Case study: applying theoretical ideas to practical contexts. By using case studies and scenarios, teachers aim to develop learners’ reasoning, problem-solving, and decision-making skills [60];

-

Concept/mind maps: a concept map or conceptual diagram is a diagram that depicts suggested relationships between concepts, used by learners to map out their understanding of a particular concept by visual representation;

-

Essay: learners will synthesize and evaluate theoretical ideas and concepts and develop research skills. Additionally, they will learn academic and discipline-specific writing conventions such as formulating an argument, presenting evidence, integrating material from sources, and referencing appropriately, while building on their drafting, editing, and revising skills;

-

Forum posts: short written pieces of text about a particular topic or issue. These pieces are hosted on a discussion-based online forum, and readers (e.g., the instructor, other learners) can usually respond to each post;

-

Literature review: a written paper based on systematic and explicit identification, evaluation, and interpretation of existing bodies of work. Literature reviews require learners to conduct an extensive and critical review of literature related to a chosen topic;

-

Workbook: an objective record of observations, completed tasks, or goals that have been met;

-

Minutes: an official record of discussion and post-meeting action/s;

-

Open-book: learners have the opportunity to use any or specified resources to help them answer set questions under time constraints. This method removes the over-reliance on memory and recall and models the way that professionals manage information;

-

Portfolio: the learner selects the items of work to include to represent the learning that has occurred. Portfolios promote reflection, self-evaluation, and ongoing personal engagement, as well as demonstration of practical and creative abilities;

-

Problem sheet: learners complete problem sheets, e.g., on a weekly basis. This can be a useful way of providing learners with regular formative feedback on their work and/or involving elements of self- and peer-assessment;

-

Project: an application of theory in practice;

-

Quiz/Test: a non-invigilated response to a finite number of questions, which may be short-answer or multiple-choice, or a combination of both. Useful to assess a wide range of knowledge/skills across a module;

-

Question banks: learners are assessed on their ability to produce a certain number of questions on a topic. This helps learners to recognize what they do and do not understand about a topic and is a useful way for staff to collate a question bank that could be used for quick formative quizzes throughout the module;

-

Report: a structured piece of written work, usually divided into sections with subheadings. Reports will often be based on research that the learners have undertaken themselves. A report is a formal document that uses research to support a conclusion;

-

Reflection: the process of engaging the self in attentive, critical, exploratory, and iterative interactions with one’s thoughts and actions, and their underlying conceptual frame [61]. Reflective writing, where learners reflect on placements in light of theoretical learning, is often a component of professional placement subjects;

-

Thesis: an extended piece of research designed to set up and defend an intellectual position taken by its author;

-

Take-home examinations: non-invigilated, scheduled, timed assessments undertaken over a short period by learners at home.

Among oral and other performance-based assessments, we report the following.

-

Debate: a structured way of exploring the range of views on an issue. It consists of a structured contest of argumentation, in which two opposing individuals or teams defend and attack a given proposition;

-

Interview: an interaction between an instructor and learner, or between learners, that may replicate specific scenarios such as workplace interviews, research interviews, or other one-on-one oral interactions;

-

Individual presentation: helps learners to master oral communication and persuasive skills. The learners learn how to organize arguments using supporting evidence, select relevant material, and engage critically with ideas, while developing their understanding and confidence;

-

Group presentation: potential for sampling a wide range of practical, analytical, and interpretative skills;

-

Viva-voce: often used for assessing ‘borderline’ degree classifications but also useful to explore learners’ understanding of a wide range of topics;

-

Creative work: an original creative work, i.e., a tangible product of creative effort;

-

Demonstration: learners demonstrate their skills live, often in a simulated environment or a laboratory;

-

Performance: a live performance of a creative work, usually in front of an audience, though it could also be pre-recorded;

-

Recorded/rendered creative work: the work is presented through a recording or rendering medium;

-

Roleplay and simulations: forms of experiential learning. Learners take on different roles, assuming a profile of a character or personality, and interact and participate in diverse and complex learning settings;

-

Poster presentation: learners are asked to produce a poster on a particular topic. It can be used individually or in groups to assess a range of activities;

-

Teamwork: learners work together in teams to complete a task, assessment, or project. This emphasizes collaborative learning, problem-solving, and critical evaluation, and it is valuable preparation for the workplace.

• Timing

As for the assessment task timing, teachers must give feedback to learners as quickly as possible. The main purpose of assessment is to give learners advice on how to improve. Learners must have enough time to reflect on and incorporate the suggestion for their next assignment [66]. When thinking about the timing of the assessments, there are a few points that may be useful to consider during the planning stages [23]:

-

Different assessment methods are best suited to different times: conducting a formative assessment too late for the feedback to be enacted by a learner defeats the object of the assessment in the first place, as would summatively assessing learners too early, therefore, not allowing learners to learn and develop all the requisite skills;

-

Remember that learners are being assessed on other modules: ideally, the timing of assessments should not be considered in isolation, but rather in consultation with other teacher colleagues, to ensure that bunching and the associated ‘assessment overload’ is minimized, both for the learners and teachers;

-

Assessment timing needs to suit teachers as well as learners: the timing of assessments needs to be manageable, allowing teachers to have sufficient time to prepare and deliver the assessment, as well as mark the learners’ work and return grades/feedback in a timely fashion. The time required for this will depend very much on the chosen methods of assessment;

-

Formative assessment should start early: where assessment is to be used formatively, its use should begin as early as possible in the module to stand the best chance of motivating and positively influencing learners, and thus helping to direct their learning;

-

Assessment should be frequent, but not too frequent: it is important to find a balance between frequent assessment and over-assessing, providing learners with enough support and guidance as well as the time required to reflect and act on the feedback.

• Description/purpose/formatting requirements

When an assessment task needs to be completed by a learner, it is required that the teachers provide information on what the learners are expected to produce and how they must produce it. The task description needs to be developed to give clear, unambiguous direction as to what is required. An extensive description of the task itself may highlight the purpose, formatting requirements, deadlines, file type, submission method, marking grid, how the learners will receive feedback on their work, when they should receive their feedback, and any supporting materials learners may need to complete the assessment [62]. It is insufficient to just hand assessment documentation to learners (or markers) and expect them to know exactly what it means, especially in the early years of study. Time must be allocated to allow explanation and discussion of the expectations for the task, focusing on communication.

4.2.2. Feedback

Feedback can boost a learner’s confidence, self-awareness, and enthusiasm for learning. Such feedback can be considered effective. If given during the very first courses at university, the learners may be more incentivized to continue their education, which may support learner retention. The shift toward higher education may then be facilitated [63,64,65].

The main objectives of feedback are to justify to learners how their mark or grade was derived; identify and reward specific qualities in learners’ work; guide learners on what steps to take to improve; motivate them to act on their assessment and develop their capability to monitor, evaluate, and regulate their own learning [66]. The literature suggests the following seven principles of good feedback [67]:

• Helps clarify what good performance is (goals, criteria, expected standards);
• Facilitates the development of self-assessment (reflection) in learning;
• Delivers high-quality information to learners about their learning;
• Encourages teacher and peer dialogue around learning;
• Encourages positive motivational beliefs and self-esteem;
• Provides opportunities to close the gap between current and desired performance;
• Provides information to teachers that can be used to help shape the teaching.

The feedback modalities are identified, taking into account the above literature and the following university websites [26,30,46,47,48,49,50,51,52,53,54,55].

-

Qualitative/quantitative feedback: can be provided in the form of annotated comments on learners’ assessment tasks, assessment task advice and commentary, and audio or video files. Quantitative feedback is often provided in the form of numeric marks, rating scales, grades, or percentages [68];

-

Formal/informal feedback: formal feedback includes the likes of marking criteria, competencies, or achievement of standards, and is recorded for both the learner and organization as evidence. Informal feedback can be offered at any time as it is something that emerges spontaneously in the moment or during action [69];

-

Formative/summative feedback: as previously highlighted, formative feedback is thought to monitor the learning progress/success, to provide ongoing feedback that can be used by instructors to improve their teaching and by learners to improve their learning. Therefore, formative feedback is best given early in the course and before summative assessments. The goal of a summative assessment is to evaluate the learning at the end of an instructional unit by comparing it against some standard or benchmark. Therefore, summative feedback consists of detailed comments that are related to specific aspects of their work, clearly explain how the mark was derived from the criteria provided, and additional offer constructive comments on how the work could be improved [70,71];

-

Oral/written feedback: providing written feedback on learners’ work is the most used practice in many courses/programs. It is important to note that written feedback is a one-way message sent by the teacher/marker to a learner. It is essential for learners to be able to interpret and understand the message; otherwise, that feedback is redundant and ineffective. Hence, the feedback given needs to be specific and point directly to the part it applies to, to avoid ambiguity [72]. Providing verbal feedback in class can be a quick and valuable exercise. Face-to-face feedback to learners could be offered in the form of group feedback to cover common issues in the assignments, or an individual meeting, during which a detailed explanation and guidance on how to improve are required;

-

Learner/expert lead: among several possibilities, self-assessment provides quick feedback to the learner and reduces teachers’ workloads, while it can also help learners develop autonomy and improve their learning [87]. Peer-assessment is not unlike self-assessment and provides similar benefits to those outlined above, but peer-assessment offers the added advantages of learners working collaboratively and developing a better sense of their own learning by assessing other learners’ output and receiving feedback from a single source [73,74]. Yet, if learners are the sole arbiter of the marks awarded through self-/peer-assessment, it is likely that both learners and academics may be skeptical regarding the accuracy of the marks. A deliberate component of moderation of self-/peer-assessment is to accumulate evidence that can show how well learners’ judgments compare to those of academics. From the other side, teachers and invited guests from a professional, community, or industry body can be considered the most used and classical actors in expert-lead feedback [66].

4.2.3. Alignment ILO-AT

In this section, the alignment between the AT and the ILO is considered, both from a soft skills perspective and looking at the action verbs (e.g., EGV) activated with the assessment. The university websites use as sources were the following [26,30,46,47,49,50,51,52,53,54,55].

• Soft skills activated

Well-designed assessment can encourage active learning, especially when the assessment delivery is innovative and engaging. Peer- and self-assessment, for instance, can foster several skills, such as reflection, critical thinking, and self-awareness, as well as giving learners insight into the assessment process. Discussing the ways in which learners are assessed can also help to ensure that the aims and goals of the assessments are clear. Utilizing assessments that make use of technology, such as the use of online discussion forums or electronic submission of work, can teach learners new skills [23].

• Assessment tasks and EGV

Once the learning outcomes to be assessed are identified, it is important to check if the assessment task is aligned with the course’s, program’s, and institution’s ILOs. Aligning assessment tasks with ILOs at different levels enables assessment to be viewed as part of an integrated, collaborative learning experience. The learning is then deeper and more lasting as the learners see that the assessment tasks are not isolated experiences but are a coherent, connected experience.

According to [75,76], the connection between the EGV (i.e., the related level of understanding according to the Bloom taxonomy) and assessment tasks can come from:

-

Descriptive tasks: use EGVs such as describe, record, summarize, and define to ask learners to provide information or facts related to learning content (e.g., descriptive essay, article, abstract);

-

Analytical tasks: use EGVs such as analyze, report, relate, compare, and synthesize to ask learners to unpack and organize information (e.g., critical review, report, viva voce, case study);

-

Reflective tasks: use EGVs such as reflect, respond, and react to ask learners to explore personal experiences, opinions, events, and learning (e.g., interview, journal, presentation);

-

Persuasive tasks: use EGVs such as argue, persuade, defend, and discuss to ask learners to present perspectives to persuade (e.g., persuasive essay, presentation, article, report);

-

Creative tasks: use EGVs such as narrate, recount, and imagine asking learners to create original, imaginary responses (e.g., performance, portfolio, artefact, project);

-

Responsive tasks: use EGVs such as respond, apply, and review to ask learners to respond to hypothetical scenarios (e.g., report, journal, case study);

-

Design-based tasks: use EGVs such as design, plan, create, and build to ask learners to design or plan a new text, resource, or solution (e.g., portfolio, performance, demonstration);

-

Engagement-based tasks: use EGVs such as prepare, share, practice, and learn to ask learners to engage in opportunities and experiences (e.g., debate, role-play, placement, teamwork).

Table 5. Connection between the level in the Bloom taxonomy and the specific educational goal verb with related assessment task examples.

Bloom Taxonomy	EGV	Assessment Task (Examples)
Remember	arrange, define, list, identify	Multiple choice, quiz/test, question banks, take-home examinations
Comprehension	classify, discuss, present, rewrite	Concept/mind maps, interview, debate, problem sheet, minutes, forum posts, open-book, individual presentation, group presentation, viva-voce
Apply	solve, calculate, demonstrate, organize, use	Abstract, case study, problem-solving tasks, roleplay, group work, portfolio, workbook, project
Analyze	categorize, contrast, compare, debate, inspect	Thesis, annotated bibliography, literature review, debates, class discussion, jigsaw method, think-pair-share, fishbowl, laboratory
Evaluate	assess, conclude, justify, measure	Report, reflection, journal, debates, mind map, peer evaluation, group work, teamwork
Create	design, develop, revise, formulate	Project, thesis, article, essay, creative work, demonstration, performance, roleplay, recorded/rendered creative work,

summarizes the main assessment tasks retrieved from the literature, aligned with specific EGVs.

4.2.4. Marking Criteria

The quality of learners’ work or performance must be judged with reference to explicit or predetermined criteria and standards, and not by reference to the achievement of other learners [77]. Deciding on the model for setting marking criteria can depend on the ILOs of the course and the type of assessment task. The following universities have been accounted for in defining this section [26,46,47,48,49,50,51,52,53,54].

There are two main ways to provide marking criteria: (1) marking guides and (2) rubrics. The choice of using a marking guide or rubric to present the marking criteria will depend on the type of assessment task designed, the ILOs being demonstrated, and the learning technologies used [78].

4.2.5. Technological Support

The shift toward online testing is well-documented and different forms of Computer-Assisted Assessment (CAA) now exist. CAA can be defined as a broad term for the use of computers in the assessment of learner learning [80]. In the online and blended environment, there are many different learning technology tools that can be used to enhance the feedback and marking process for learners and teachers. Maximizing the functions of the Learning Management System (i.e., LMS, Canvas, Moodle, and Blackboard), online feedback processes can include creation, submission, grading, and feedback mechanisms all in the one place, which will return to the learners instantaneously, at any time and place. It is important to understand that using the wrong online tool for the feedback process can impact the efficiency and quality of the processes, just as choosing the right online tool can enhance the learning and teaching processes [72]. Digital assessment tools often have feedback and marking systems, including digital rubrics, comment banks, audio recording, and marking up materials. Among most digital features, video/audio are powerful tools in providing personalized and focused feedback [79]. Additional information can be retrieved at the university websites of [26,46,47,49,50,51,52,53,54].

4.2.6. Plagiarism and Cheating

Academic integrity is characterized by the values and behavior learners need to ethically undertake their studies, according to [46,50,51,53,54]. Assessment security emphasizes strengthening assessment tasks against cheating attempts and detecting cheating behavior. Plagiarism is a form of academic dishonesty. The design of the assessment plays an important role in discouraging learners from plagiarizing [81]. Knowing more about learners’ reasons for plagiarizing can help teachers to develop and use assessment and teaching strategies to discourage learners from plagiarizing in the first place [82,83,84,85,86].

4.3. Validation of the Proposed Template

The case study selected to validate the results of this work is a partial outcome of the European project MAESTRO [25]. In this project, each partner developed both theoretical and methodological learning material to define teaching and assessing strategies for the learning outcomes identified in the previous intellectual outputs. The developed learning material is individually presented as an independent educational unit featuring a stated Intended Learning Outcome (ILO) and related Teaching and Learning Activities (TLA), as well as an Assessment task (AT). The series of education units are developed to introduce specific applications of the technological enabler of the fourth industrial revolution that address improvement on the Sustainable Development Goals identified by the United Nations [88,89].

The courses selected from the project for this case study are “Biomedical Technologies” offered at the University of Pisa (UNIPI) and “Planning and Control” offered at Stockholm’s Royal Institute of Technology (KTH). The courses are presented in the following sub-paragraphs.

4.3.1. Biomedical Technologies Course

The biomedical technologies course focuses on conventional and unconventional manufacturing processes. Particular attention is given to the broad perspective of Additive Manufacturing (AM) technologies (i.e., fusion, binding, or solidifying liquid resin and powders) in the module of a manufacturing technology laboratory. At the end of the course, the learners can utilize AM machines to build parts through the selected AM process, starting from Computer-Aided Design (CAD) models; define machine process parameters; select material; optimize the energy consumption and costs. Moreover, the learners can design complex shapes and lightweight customized structures for biomedical supports, sometimes impossible to produce with conventional processes, and they may consider planning process chains or hybrid or additive, and conventional as well as unconventional, processes [90].

Following the framework of the MAESTRO project, the course was redesigned to meet sustainability goals according to the selected Industry 4.0 technology enabler (AM). This new perspective was developed by following the proposed template for the definition of TLAs and Ats, starting from the ILO that states “select and design a specific AM process for biomedical prosthesis by minimizing the environmental cost/impact”.

This ILO contains the EGVs select and design. Figure 6 and Figure 7 show TLA and AT standard templates, respectively, filled in as proposed in the Results sections (Section 4.1 and Section 4.2) to activate the select and design EGVs.

4.3.2. Planning and Control Course

The planning and control course focuses on forecasts and aggregate plans for production, and explains the lean philosophy and related tools. At the end of the course, the learners will be able to explain fundamental principles used in traditional production planning and control systems; develop aggregate plans for manufacturing a multi-component product; propose and motivate a Master Production Schedule and a Material Requirements Plan for a given aggregate plan; apply proper inventory control methods for a product with known demand; choose the best operations scheduling approach to optimize certain shop floor performance indicators; describe the principles of push and pull control policies; explain and utilize appropriate lean tools to continuously improve shop floor performance; apply value stream mapping for current and future states to a given case study.

Following the framework of the MAESTRO project, the course was redesigned to meet sustainability goals according to the selected Industry 4.0 technology enabler (augmented reality (AR) and virtual reality (VR)). This new perspective was developed by following the proposed template for the definition of TLAs and ATs starting from the ILO that states “explain and use suitable AR and VR implementations for assembly on a lean shop floor”.

This ILO contains the EGVs explain and use. Figure 8 and Figure 9 show TLA and AT standard templates, respectively, filled in as proposed in the Results sections (Section 4.1 and Section 4.2) to activate the explain and use EGVs.

5. Discussion

The main results of this paper are the proposed templates that collect and standardize the available literature and the best practice in the field of higher education course design. Special focus has been dedicated to the standardization of the main dimension of the TLAs and ATs since few contributions can be retrieved on this aspect in contrast to the vast literature on ILO design and description. Six dimensions were identified for TLAs and ATs. It could be observed that there are similarities among them. Both TLAs and ATs must include a description/purpose, activity type, align with the corresponding ILO through the action verb (EGV), and have technological support. Some of the dimensions needed to be described in further detail as they embed further aspects, for instance, the dimension “alignment” contains the “soft skills activated” and “action verb” subdimensions. These further details allow for a more complete description of the dimensions, facilitating the users’ understanding of them. Moreover, such level of detail may provide a more realistic picture of the learning approach chosen for the course designed.

The education domain has been experiencing a shift toward introducing distance learning to programs, boosted by the ongoing digitalization trend. Online and blended learning are thus becoming popular. This is challenging the entire course design process as educators are forced to reflect on whether the course’s activities and assessments may be carried out completely on-campus or online, or partially on-campus and online. The TLA and AT templates proposed in this work have proven to overcome this challenge as they are suitable for designing both on-campus and online TLAs and ATs. In particular, the identified dimensions indicate the “technological support” for TLAs and ATs and the “mode of delivery” for TLAs. As such, the course designers using these templates can specify the technology required to perform the activity or assessment and the preferred mode of delivery of teaching activities. This was verified in the case study used as a validation of the proposed templates. For instance, it could be observed that video conferencing tools (i.e., “technological support”) can be used to deliver a lecture. With this specified, it gives an indication for the learners that the lecture can be delivered online (i.e., “mode of delivery”). As for the assessment of a lecture, the technological support dimension in the AT template indicates the available online assessment tools (i.e., LMS) dedicated to the feedback and marking process.

The results of this work provide a valuable input to the second iteration of the CONALI ontology. The TLA and AT standard templates present ad-hoc guidelines to fully characterize TLAs and ATs available for education designers. Such guidelines contribute to the population of the ontology itself; the ontology’s classes are now presented with static attributes (i.e., dimensions and subdimensions) that allow us, in turn, to instantiate the whole ontology. Finally, the proposed improvements make the CONALI ontology suitable to create a solid education knowledge base that can be exploited to suggest the best way of teaching content based on the content available in the knowledge base. This brings a significant improvement to the state-of-the-art design tools in the education field, which is now dealing with increasing big education data.

6. Conclusions and Future Work

This work proposes a novel and standardized approach to describing TLAs and ATs that fulfills all the requirements found in literature, university websites, and blogs. The large amount of TLAs and ATs found are based on the EGVs included in the Bloom taxonomy. This study provides examples of how to apply the standardized approach to a few selected TLAs and ATs.

Although this is a preliminary approach, and further parameters need to be included, our ultimate goal is to provide an exhaustive classification. In this regard, future works are needed to expand the current study into an open handbook of CA educational unit design to support and inspire education designers.