*Article* **Does Postsecondary Education Attainment Matter in Community Service Engagement? Evidence from Across 18 OECD Countries**

**Hee Jung Gong <sup>1</sup> and Jung Eun Hong 2,\***


**Abstract:** This study is concerned with the central issues of community service engagement (CSE) in 21st century democratic societies around the world. To examine the factors influencing postsecondary education attainment's relationship to CSE, this study utilized data from the Organization for Economic Co-operation and Development (OECD) countries using ordinary least square (OLS) and two-level hierarchical linear modeling (HLM) methods, including various factors for each country's individual and country levels. The results show that attainment in postsecondary education at the individual level and investment and enrollments in tertiary education both have an influence on increasing CSE in 18 OECD countries. The present study is expected to contribute to an understanding of the relationship between postsecondary education and CSE across the world.

**Keywords:** postsecondary education; higher education; community service; civic engagement; educational attainment; OECD country; hierarchical linear model (HLM); PIAAC

#### **1. Introduction**

Encouraging responsible, active, participatory citizenship is a goal in the field of education and has globally been an important agenda item among researchers, educators, and policy makers [1]. Increasing civic engagement is a growing concern, as it has been proven to be an essential aspect of high-quality governance and a well-functioning democracy, producing better quality schools, faster economic development, and more effective governments [2]. In Organization for Economic Co-operation and Development (OECD) countries, especially, matters of economic disparity and environmental sustainability have increased calls for a more civil society [3]. In terms of the relationship between a nation and its civic affairs, according to a Washington DC Gallup Poll [4], adults in developed countries are more likely to be civically engaged than those in the developing world. Furthermore, the related literature has revealed that civic engagement can be cultivated by a nation through its economic status, culture, or social norms [5,6]. Despite variances in civic engagement behaviors around the world, cultivating community service engagement and thereby increasing civic engagement is an important task for all global citizens [1].

In the United States, individuals express their beliefs in the importance of individual effort and concern for others through volunteerism and the ethic of service [7]. Americans believe that one of the ways of passing this value on to younger generations is to participate in community service [7]. According to U.S. Bureau of Labor Statistics [8], 24.9% of the total U.S. population volunteered at least once and spent average 52 hours on volunteer activities during the period from September 2014 to September 2015. The three main organizations people volunteered for were religious (33.1%), educational or youth service (25.2%), and social or community service-related organizations (14.6%) [8]. Thus, K-12 schools strive to play a crucial role in addressing community needs and often utilize service-learning as a

**Citation:** Gong, H.J.; Hong, J.E. Does Postsecondary Education Attainment Matter in Community Service Engagement? Evidence from Across 18 OECD Countries. *Educ. Sci.* **2021**, *11*, 96. https://doi.org/10.3390/ educsci11030096

Academic Editor: Maria José Sousa

Received: 15 January 2021 Accepted: 23 February 2021 Published: 2 March 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

pedagogical method [9]. At the collegiate level, membership in Campus Compact, the U.S. coalition of colleges and universities committed to the public service purposes of higher education, grew from its founding three colleges in 1985 to over 1100 campuses in 2008 [10]. The increased pressure for educational accountability and commitment to public good has led universities to provide more opportunities for students to engage in community service and to learn through service-learning [11]. In addition to Campus Compact, a variety of professional and higher education organizations, such as the American Association of Colleges and Universities and the Engagement Scholarship Consortium, have been actively involved in community engagement in the United States.

Similarly, the United Kingdom places significance on building active citizens and believes that citizenship education is essential to achieving this goal [12]. Therefore, the national curriculum has included citizenship programs formally since 2002 with a goal to prepare students to become active, responsible members of society [13]. In the same manner, the majority of higher education institutions in the United Kingdom have been interested in promoting active citizenship, with an emphasis on volunteerism, so many institutions have provided various community service-learning programs for their students [14]. In addition, national organizations such as Volunteering Matters offer volunteer opportunities for the young generation.

In South Korea, the foundation of citizenship education has changed in accordance with Korean modern history from anticommunism in the 1950s (post-Korean war) to patriotism from the 1960s to the 1980s, to globalism and neoliberalism since the 1990s [15]. Recently, it has focused on preparing competitive workers rather than democratic citizens [15]. In K-12 education, community service activities are included in the formal school curriculum, so students tend to complete them mandatorily [16]. Korean higher education institutions do not seem to emphasize community service, so college students often participate in volunteer programs individually through non-profit organizations such as the Korean University Council for Social Service or religious organizations [17].

Last, surprisingly, citizens in South America show a relatively high level of civic engagement despite their countries' unfavorable political conditions and economic performances [18]. Citizens tend to actively participate in community activities to solve political and economic issues by raising their voices together [18,19]. The results of a survey of young teenagers from Chile and Columbia also revealed their civic engagement to be higher than the international mean, in spite of lower civic knowledge and attitudes [20]. In particular, various social and political issues in Chile have made universities' social responsibilities significant [21]. In 2001, many universities participated in the University Builds Country project to fulfill their responsibilities through service-learning [21].

As these countries have tried to expand their efforts for improving civic engagement, scholars and researchers also have revealed the relationship between education and civic engagement. In terms of the relationship between higher education and civic engagement (including community service engagement), past studies have focused on the connection between them and pointed out the importance of education in improving attitudes toward citizenship, civic education, political behavior, and social engagement [22–24]. Braskamp [25] also suggested that higher education should reinvestigate its role in preparing citizens for participation in a democratic society and the larger community. Since the 1970s, the field of postsecondary education has considered community service to be one of the most indispensable components of civic engagement and recognized its importance in student community service activities [26].

Although the importance of civic knowledge and engagement in the context of academia and society is convincing, thus far few empirical studies and trials have attempted to navigate the factors or determinants that influence community service engagement (CSE). There was particularly a dearth of recent empirical evidence from cross-national samples on the relationship between postsecondary education and CSE. This study, therefore, investigates how individual and national factors influence CSE. It focuses on the educational context, especially the influence of postsecondary educational (throughout this paper, the

terms higher education, postsecondary education, and tertiary education are regarded as concepts that can be interchangeable; however, we keep the original term from the given secondary datasets from each source) attainment at the individual and country levels within the OECD countries, using cross-national representative data and ordinary least square (OLS) regression and hierarchical linear modeling (HLM) methods. The specific research questions are as follows:


#### *Literature Review*

There is no single definition of civic engagement. It is sometimes defined narrowly, to focus on one specific perspective and activity, or broadly, to cover "all activity related to personal and societal enhancement which results in improved human connection and human condition" [27] (p. 22). However, generally it refers to "the ways in which citizens participate in the life of a community in order to improve conditions for others or to help shape the community's future" [5] (p. 236). Civic engagement has four key aspects: community service, collective action, political involvement, and social change [5,27]. The first aspect, community service, focuses on individual or group participation and engagement in voluntary service activities in the local community [28]. Taking collaborative and collective action with other community members to advance their common interest is a significant feature of civic engagement as well [29]. Civic engagement also includes active participation and involvement in the political process or democracy [22]. Lastly, civic engagement should strive towards positive social change, which benefits the entire community [30]. Therefore, various activities are often considered forms of civic engagement (e.g., community problem solving, volunteering, fund-raising, voting, protesting, submitting petitions, and canvassing) [22].

To promote students' meaningful, experiential, and active learning and to prepare them to become the leaders of our future society, higher education institutions have been offering service-learning opportunities for students participating in service activities as components of course work or extracurricular activities [11]. Service-learning should closely relate to students' academic curriculum; experiences that a student obtains from servicelearning need to meet the learning objectives of a certain course the student takes [11]. Therefore, unlike volunteerism, which more focuses on the receipient than the provider of volunteering, service-learning focuses on the development of students' learning through community service activities [11].

Emphasis on these activities in higher education institutions has naturally led to individuals in their 20s reporting higher participation in such activities than did previous generations [31]. Moreover, because a majority of colleges have begun to include community service experiences among their admission criteria, middle and high school students are participating in such activities at rates higher than those found among any other age groups, with numbers increasing steadily since the 1990s [5,32]. College students' participation in community service activities is especially important because people with prior volunteer experiences tend to remain involved in other forms of volunteer work continuously as they get older [26]. It seems that higher education institutions are the dominant gateway to promoting civic engagement among the youth in the long term.

Several studies have reported various benefits arising from students' service-learning experiences [11]. Through these experiences, students can improve their academic learning and develop practical skills [33]. They also provide opportunities for both personal growth (e.g., development of interpersonal, communication, problem-solving, and leadership skills) [34,35] and increased connections with their local community [36]. They are especially useful and critical in learning about diversity, because students' prejudices can be challenged by direct encounters with people with different identities (e.g., generations, socioeconomic status, ethnicity, and race) [37,38]. In addition, students can develop the capacity to be responsible members of society [35,39]. These activities also benefit students' future career by providing time to identify their values and consider career paths beneficial to their community [11].

Prior studies also identified an individual's motivations toward community service engagement, including altruism, patriotism, values, career, and enhancement [40–42]. Generation Z, who were born in the mid-1990s to 2002, especially seem to engage in community service activities to receive learning opportunities and to be beneficial for their career preparation [40]. Moreover, some scholars reported a close relationship between an individual's motivation to work in public sectors, such as government and non-governmental organizations (i.e., public service motivation) and engagement of community service activities, including volunteering [43,44]. In particular, people with a high level of commitment to the public interest tend to volunteer more often than others, and those people usually volunteer for political, religious, and charitable organizations [44].

In previous studies, background characteristics, such as gender, age, and religion, have been associated with engagement in community service [12,20,45]. These associated variables can be divided into three groups. First, as a proxy of economic capital, family background factors such as income have been reported as important attributes [12,45]. Second, as a proxy of social capital, parental education, occupation, and volunteer experience [45,46], along with the degree of social trust [47] and the service orientation of an individual's acquaintances [46,48], has correlated to CSE. Third, as a proxy of cultural capital, activities such as reading a book, going to a museum, or watching television news [20] and community organization experiences [31,32] have been associated with CSE. Other factors associated with the national level, such as characteristics of the individual's community [45] and governmental factors, such as media and ICT influence [49] or internationalization [50] also have an influence on CSE. Some studies also addressed education as one of the factors related to CSE; for example, the level of education, the desire for higher education attainment, and the availability of literacy resources at home are positively related to one's attitude towards civic engagement and CSE [20,51,52]. However, these studies were published almost 20 years ago, and there is still a dearth of recent literature examining the relationship between education, specifically focusing on postsecondary education, and CSE across OECD countries.

Comprehensively, even though postsecondary education has been gradually more interested in civic engagement and has put efforts into developing the curriculum and activities for community service, most previous studies focused on the individual factors influencing CSE and did not sufficiently consider educational factors using recently published large-scale data. Further, few trials have been conducted with a global perspective, and several studies were limited to a sample of one or a small number of countries. Thus, the conceptual model for our study, with literature support, is described in Figure 1.

**Figure 1.** Conceptual model of predictors influencing community service engagement

#### **2. Materials and Methods**

#### *2.1. Data and Sample*

The data for this study came from the Survey of Adult Skills, a product of the OECD Programme for the International Assessment of Adult Competencies (PIAAC), which assesses and compares the basic skills and competencies of adults in the 21st-century around the world. The survey was a large-scale study developed by the OECD, which surveyed 24 participating countries in 2012, nine additional countries in 2014, and another five in 2017. For this study, out of the 24 countries that were surveyed in 2012, 18 made their data available for public use. The surveys were conducted in multiple languages, all transcribed and subsequently translated so that the data could be available to the OECD in English. The raw data are organized by nation; thus, they had to be collapsed into one dataset. After the systematic missing values for a variable utilized for this study were deleted, the final dataset included a total of 110,288 individuals in 18 OECD countries. For the analysis, among 110,288 individuals, if there were missing data in a specific variable included in the analytic models, that observation was excluded from the analysis; thus, the number of individuals in OLS and the HLM model was 64,910.

#### *2.2. Variables*

The dependent variable of this study is CSE, measured by individuals' self-reported levels of participation in voluntary work. This includes unpaid work for a charity, political party, trade union, or other non-profit organization in the last 12 months. A Likert-scale was used for measuring the variable from (1) "never" to (5) "every day." It was redeemed as a continuing variable as often as possible so it could be used without any harm to the analysis [53].

Independent variables at the individual level (level-1) included personal factors, family background factors, economic factors, and educational factors. Personal factors included gender and age (16–55+). Family background included parental education (a proxy for social capital), the number of books at home (a proxy for cultural capital), and immigrant status. Economic factors included income level (a proxy for economic capital) and individuals' occupational status (e.g., student, employee, or retired). The educational factors that were used in the study included degree attainment (i.e., less than a high school diploma, high school diploma/some college but no degree, and college degree or higher).

In order to examine the effects of postsecondary educational attainment at the national level (level-2), other variables related to sectors of the economy, society, and technology were used as control variables, which were compiled from previous related studies [49,54,55]. The proportion of gross enrollment ratios in tertiary education for both sexes were used to assess postsecondary education attainment. In order to minimize the differences between the postsecondary education systems of each country, spending on tertiary education (1/1000 dollar) and the proportion of spending on tertiary education in the public sector were included as well. Gross domestic product (GDP) per capita (1/10,000 dollar) and the proportion of internet users were considered as control variables as well. All variables were drawn from indicators from World Bank and OECD data, aligning the same matched years (2012 and 2013) with the individual data from the PIAAC.

In the sample, in brief, 49% were female, 40% obtained greater than secondary education completion (postsecondary), 92% were employed or self-employed, and 89% were non-1st or 2nd generation immigrants. More specific descriptive statistics of the variables included in the analysis are presented in Table 1. It was confirmed that none of the missing data were biased towards a particular country or other variables.


**Table 1.** Descriptive statistics (n = 64,910).

Note: Descriptive statistics indicate the analytic sample number is limited to 64,910, and the observation has no missing data across all the variables. In the table, \* indicates the reference categories.

#### *2.3. Analytic Methods*

For the analysis of these data, the following three research hypotheses (alternative hypotheses) based on the aforementioned three research questions were established:

**Hypothesis 1:** *The level of CSE varies by educational level in OECD countries.*

**Hypothesis 2:** *Individual postsecondary educational attainment is positively associated with CSE.*

**Hypothesis 3:** *Postsecondary educational attainment at the country-level is positively associated with CSE.*

Specifically, for the first hypothesis test, we applied an ANOVA test with a Bonferroni post hoc test for multiple comparisons. Next, the second hypothesis was tested using OLS regression. Lastly, hierarchical linear modeling (HLM) was utilized for the testing of the third hypothesis.

The rationale behind the use of the HLM method, in particular, is the fact that data can be commonly grouped by hierarchical level [56]. For example, in this study, the behaviors of people are influenced simultaneously by their personal backgrounds but also by their country of residence. This means that variances in outcome variables are shared by the hierarchically structured data, on both the individual and national level. HLM allows for the estimation of OLS regression, taking into account the nested structure of the data, a unit of measurement. Thus, HLM can use data clusters to avoid grouping errors [57], which is helpful for drawing out accurate estimate slopes for each level.

In terms of HLM modeling, first, specifically, we fitted an unconditional model (basic model) to examine whether the average CSE differs between countries. After that, we added individual-level variables, including individual educational attainment and other covariates (Model 1). Finally, country-level variables, including the portion of postsecondary enrollment and other covariates at the country level, were included (Model 2). In Model 2, all the regression slopes of individual predictors were fixed at the country-level (level-2), since the outcomes among the given OECD countries showed only relatively small variations. This was also done to methodologically secure statistical stability [57]. The final model (Model 2) can be briefly represented as follows:

$$\begin{aligned} \text{(1)} \quad \text{Level-1:} \; \text{CSE}\_{ij} &= \beta\_{0j} + \sum\_{p=1}^{5} \beta\_{pj} Z\_{\pi ij} + \sum\_{p=6}^{13} \beta\_{pj} Y\_{\pi ij} + \sum\_{p=14}^{23} \beta\_{pj} X\_{\pi ij} + \sum\_{p=24}^{25} \beta\_{pj} \mathbb{W}\_{\pi ij} + \ [ \}, \\\ [ \} & \quad \ \gamma \sim N(0, \sigma^2) \end{aligned}$$

$$\text{(2)}\quad \text{Level -2: } \beta\_{0j} = \gamma\_{00} + \sum\_{c=1}^{5} \gamma\_{0c} T\_{cj} + \sqcap\_{0j} \sqcap\_{0j} \sim N(0, \tau\_{00})$$

 $\mathcal{S}\_{p\text{j}} = \gamma\_{p0\text{\textquotedblleft}p} \text{ ( $p = 1, 2, 3, ..., 25$ )}$ 

where *CSEij* is the CSE participation score for individual *i* in country *j*, *β*0*<sup>j</sup>* is an intercept for j country, *βpj* is the coefficients predicting CSE (*W*: educational attainment factors; *X*: economic factors; *Y*: family background factors; and *Z*: personal background factors), and *ij* is an error term to describe the unique effect of each individual. In addition, at level-2, *γ*<sup>00</sup> is an average *CSE*, *γ*0*<sup>c</sup>* indicates the coefficients predicting *CSE*, *Tcj* a>is country-level factors in the model, *γp*<sup>0</sup> is the average slope of the entire countries regarding individuallevel variables, and u0*<sup>j</sup>* is the level-2 random effect which is the variation of differences between countries.

#### **3. Results**

Prior to conducting OLS regression and hierarchical linear modeling techniques, we examined the mean CSEs using a one-way ANOVA test with the Bonferroni post hoc test for multiple comparisons, which addressed the first research question. Means and standard deviations, as well as the results of the ANOVA comparison of means analysis, are set forth in Table 2. On average, CSE was 1.58 during the 12 months prior to completion of the survey. This value is between the level of "never (1)" and "less than once a month (2)". However, worldwide, CSE varies from 1.31 (Poland) to 2.04 (USA), which yields a standard deviation of 0.99 among the individuals sampled within the 18 OECD countries. An alternative hypothesis (H1) predicts that for the individual participants in the CSE sample, CSE would be greater for individuals with higher rather than secondary education levels. As shown in Table 2, we rejected the null hypothesis, and the mean of CSE was not statistically equal for an individual with lower than high school completion, high school completion, and higher than high school completion. The detailed results of the Bonferroni post hoc test is also presented in Table 2. For those individuals with more than high school education (or postsecondary education level), the CSE in all 18 countries was higher than those with only high school completion or individuals with less than a high school completion.


**Table 2.** ANOVA results for differences in community service engagement (CSE) by educational level.

\*\*\* *p* < 0.001 \*\* *p* < 0.01. For these statistics, the sample is inclusive of observations suitable for analysis when the outcome variable has no missing values. The F-Statistic is calculated from variation between sample means divided by variation within the samples.

> With regard to research question 2, whether educational degree level has a statistical influence on CSE when considering other factors associated with it, the results of the OLS regression are shown in Table 3a,b. According to the results, which included all individuals in 18 countries in the sample, individuals with post-high school degrees were more likely to participate in CSE than individuals with high school completion only (β = 0.121, *p* < 0.001), when considering other factors as constant. In terms of the results of a statistical model applied in each of 18 countries individually, the coefficients in 12 countries out of the 18 were statistically significant. However, all individuals with a postsecondary education level participated more in CSE than did individuals who had equal to secondary education completion in all 18 countries, holding other personal background characteristics, such as gender, age, immigrant and working status, and social, economic, and cultural capital constant.

> In terms of the results from the HLM method for the last research question (see Table 4), the basic model was first run before the analysis. The indicator of the intra-class correlation coefficient (ICC) in CSE, calculated by dividing the between-country variance for the outcome variable by the total variance, turned out to be 0.055. This indicates that about 5.5% of the variability was caused by countries' individual factors or characteristics (in other words, accounted for by the between-country effect).


**Table 3.** Ordinary least square (OLS) estimates for educational attainment predicting community service engagement.

Reference group: Age1 (16–24), Ed2 (secondary education completion), Imgrt1 (1st generation immigrants), Income1 (no income), Culture1

(**a**) (cultural capital\_lowest quantile), ParEd1 (neither parent has attained upper secondary), Status3 (not working and looking for work). + *p* < 0.10, \* *p* < 0.05, \*\* *p* < 0.01, \*\*\* *p* < 0.001.


**Table 3.** *Cont.*

(**b**) Reference group: Age1 (16–24), Ed2 (secondary education completion), Imgrt1 (1st generation immigrants), Income1 (no income), Culture1 (cultural capital\_lowest quantile), ParEd1 (neither parent has attained upper secondary), Status3 (not working and looking for work).

+ *p* < 0.10, \* *p* < 0.05, \*\* *p* < 0.01, \*\*\* *p* < 0.001.


**Table 4.** Two-level hierarchical linear modeling (HLM) result of postsecondary educational attainment predicting CSE (n = 64,910 at Level 1; n = 18 at Level 2).

**Table 4.** *Cont.*


Reference group: Age1 (16–24), Ed2 (secondary education completion), Imgrt1 (1st generation immigrants), Income1 (no income), Culture1 (cultural capital\_lowest quantile), ParEd1 (neither parent has attained upper secondary), Status3 (not working and looking for work. \* *p* < 0.05, \*\* *p* < 0.01, \*\*\* *p* < 0.001

Next, according to the results of Model 1, including level-1 predictors and covariates, individual postsecondary educational attainment was significantly positively associated with CSE compared to the secondary educational attainment (*β* = 0.117, *p* < 0.001). At the same time, less than high school completion attainment negatively influenced CSE, compared to secondary educational attainment (*β* = −130, *p* < 0.001).

Additionally, the result of Model 2 shows that, globally, individuals with more than a high school education were more likely to participate in CSE compared to those who had only completed high school (*β* = 0.118, *p* < 0.001), when other factors at the individual and country levels remain constant in the model. On the other hand, individuals who completed lower than secondary education were associated with lower participation in CSE than those who completed secondary education. Meanwhile, the proportion of the postsecondary education enrollment at the country level (level-2) was positively associated with increased participation in CSE (*β* = 0.006, *p* < 0.05) across the OECD countries as well, when holding spending on tertiary education, the proportion of public-sector to privatesector spending on tertiary education, GDP per capita, and the proportion of internet users constant in the model.

#### **4. Discussion and Conclusions**

Recent decades have witnessed a rising emphasis on civic engagement and citizenship, and it is recognized as the leading civic education movement. The issues of civic engagement in postsecondary education are continuously discussed among policymakers, scholars, and practitioners in tertiary education settings [23,58,59]. Almost all higher education institutions state that their mission is educating students to become good, responsible citizens in our society [22,26,60–62]. That is, their ultimate goal is to prepare students for an active civic life. Thus, this study mainly examined the CSE globally, as a part of civic engagement activities, to explore how postsecondary education attainment affects participation in CSE. From the findings of the current study, educational degree attainment significantly influences the individual to engage in community service in most OECD countries, considering other factors associated with CSE. The study also revealed that postsecondary educational attainment plays an especially important role in increasing

CSE among individuals. In addition, attaining postsecondary education is associated with an increase in CSE as a part of civic engagement and citizenship, which goes beyond the individual level and impacts the whole country.

It is important to acknowledge the limitations of our work. The outcome variable was measured using a Likert scale, but the intervals on the scale would not carry the assumption that the differences between points on the scale are all the same. Likert and Likert-type responses are popular psychometric items, and debates on whether the analysis using the Likert scale should be estimated using parametric statistics or nonparametric statistics are ongoing. However, according to Carifio and Perla [63,64], Likert responses are approximate ratio data. Additionally, individual items on Likert scales are not independent and autonomous but instead are connected to each other to yield a single unified result, which provide more reliable and fundamental construct than any individual item. Moreover, we performed the ordered logistic regression, non-parametric function as well; however, the results show considerably similar findings, which included statistically significant factors on CSE, compared to the findings of this study. Therefore, for an intuitive interpretation, we assumed CSE Likert-scale is a continuous variable. In addition to that, there are other possible factors associated with CSE, and those factors, such as information on community environment, and where they live and belong, would be included in our analytical model. However, due to the characteristics of the secondary data, we could not use these variables in our analysis and made our best attempt to analyze the given data. We hope that other researchers will further expand the analysis using more inclusive variables, and test our model using different sources, and ultimately improve upon our study.

Despite the limitations, this empirical study's global perspective could have important implications for policy and research. First, this study comprehensively analyzed how a variety of influential factors, including personal and family characteristics (e.g., economic, social, and cultural capital), educational factors, and national factors influence CSE. Furthermore, it reveals whether each of these variables has a positive or negative influence on CSE. Accordingly, the results of this study can contribute to an appreciation of the real challenges surrounding CSE and civic engagement issues, as well as the role of higher education. Other factors (e.g., parental education and working status) that influence CSE vary from country to country. Thus, policymakers and educators in each country must compare and evaluate the results and utilize them to improve their country's CSE. In addition, the methodology of this study was especially comprehensive and robust since it used the representative OECD national large-data samples and utilized a variety of exploratory variables and advanced analytic method, including diverse contexts at the individual and country levels. Additionally, it employed HLM, which had been rarely used to analyze this topic. Therefore, this study will assist researchers and national leaders in ensuring that CSE can be affected not only by individual-level components but also by those on a country level.

The empirical evidence of the positive association between postsecondary educational attainment and CSE, closing the previous research gap, and simultaneously adding to the previous studies (e.g., [51]). However, there is a dearth of past empirical literature related to those issues in each country, and thus limited data are available on this subject. Further studies are needed to investigate relevant factors, such as the relationship between the postsecondary education system and government (governance) in each country at the national level. This in turn influences improvements in community service and, more broadly, civic engagement. Moreover, based on the findings of this study, average national higher education attainment in a country is revealed as a significant factor influencing the average CSE of the country, which also verified as the purpose of higher education such factors as public service to society and commitment to civic affairs. Educational attainment has been one of the consistent predictors from previous studies. However, our study newly confirmed the importance of higher education attainment specifically as a motivator toward CSE at the national level. Future studies could also attempt to consider historical and political context in the country to examine how the relationship

between postsecondary education and CSE would be changed depending on those national contexts. Furthermore, studies could potentially trace how civic engagement and CSE are influenced by the deliberate development level of postsecondary education. Those trials could contribute to an appreciation of the real challenges surrounding CSE and civic engagement and the role of general education and tertiary education systems around the world.

In order to be effective in the college and university setting, service-learning opportunities for students should not only advance their academic learning, but also promote their civic engagement and responsibility. Scholars have suggested several important components of this process: mutuality, personalization, inquiry, reflection, community, citizenship, and democracy [65,66]. The activities should be mutually beneficial for both actors and recipients, and students need to personalize the service by cultivating meaningful interactions with others. Pedagogically, students must have enough time to reflect on and interrogate their experiences, think through any issues, and find ways to solve them. The activities should be also directly or indirectly valuable to their community. Through these experiences, students will be able to develop their lifelong citizenship in a democratic society.

Higher education institutions should also play an important role in reducing disparity in civic engagement accessibility among students with different socioeconomic backgrounds by providing more opportunities and resources for students who do have less accessibility [22]. For example, student affairs departments and professionals at colleges and universities provide a variety of co-curricular or extra-curricular engagement bonding opportunities (e.g., experiential learning) both on and off campus together with community engagement activities. This approach ultimately should be "a transformation toward justice and the greater public good" [67] (p. 86). In addition, in the midst of the COVID-19 pandemic, due to the development of technologies, online community service engagement has been introduced [11]; thus, higher education practitioners try to spread knowledge about out this new medium to students or adults who have lower accessibility and knowledge of community and civic engagement activities.

Consequently, this study hopes to offer new insights and ideas as to how CSE can be examined in postsecondary education settings across the globe and lead to a fuller understanding of the factors that contribute to CSE from diverse contexts, including both individual and country levels in 18 OECD countries. Moreover, policy makers and practitioners can obtain additional insights into what factors function as a commonality of improving CSE and what factors do not. Simultaneously, investment in postsecondary education can play a pivotal role at both the individual and country level in bolstering CSE. This study, therefore, will contribute to improving civic engagement in educational contexts, especially in that of higher education, thereby promoting the value of democracy and social mobility to produce a better world.

**Author Contributions:** Conceptualization, H.J.G. and J.E.H.; Methodology, H.J.G.; Writing—original draft, H.J.G. and J.E.H.; Writing—review & editing, H.J.G. and J.E.H.. All authors have read and agreed to the published version of the manuscript.

**Funding:** There was no funding for this study.

**Institutional Review Board Statement:** Not applicable. In the paper preparation, authorization for the public-use of data utilization for this study was exempted by the PIAAC coordinator in OECD.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Data may openly be available in a public repository: https://www. oecd.org/skills/piaac/data/ (accessed on 15 February 2021).

**Acknowledgments:** The earlier version of this paper was presented at the annual meeting of the American Educational Research Association (AERA) at New York City in 2018 ("Civic engagement and political efficacy across OECD countries: Focusing on the impact of postsecondary education" by Hee Jung Gong). We thank scholars who provided valuable feedback and comments and also anonymous reviewers to improve this study.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


### *Article* **Cyber-Archaeometry: Novel Research and Learning Subject Overview**

**Ioannis Liritzis 1,2,3,4,5,\* and Pantelis Volonakis 1,5**


**Abstract:** The cyber archaeometry concerns a new virtual ontology in the environment of cultural heritage and archaeology. The present study concerns a first pivot endeavor of a virtual polarized light microscopy (VPLM) for archaeometric learning, made from digital tools, tackling the theory of mineral identification in archaeological materials, an important aspect in characterization, provenance, and ancient technology. This endeavor introduces the range of IT computational methods and instrumentation techniques available to the study of cultural heritage and archaeology of apprentices, educators, and specialists. Use is made of virtual and immersive reality, 3D, virtual environment, massively multiplayer online processes, and gamification. The VPLM simulation is made with the use of Avatar in the time-space frame of the laboratory with navigation, exploration, control the learning outcomes in connection to the archaeometric multisystem work. The students evidently learned to operate the VPLM following operations made via visual and home-made scripting, gaining experience in synergy, teamwork, and understanding. The resulting meaningful effects of the cyber-archaeometry with virtual operations and virtual hands, texts, and video equip students especially for e-learning with the required basic knowledge of mineralogical examination, which help to understand and evaluate mineral identification from material culture and provides readiness and capacity, which may be refined in a real polarized light microscopy (PLM) environment.

**Keywords:** educational; virtual environment; virtual reality; gamification; 3D modeling; cultural heritage; cyber-archaeology; microscope

#### **1. Introduction**

The higher education institutions are progressively looking for new ways to upgrade and update the quality of education, initiate student commitment, and manage knowledge resources. The high-tech development has a major impact on education, and technologymediated learning is constantly advancing with the introduction of blended learning in educational institutions. Along this rationale, the introduction of novel educational science learning approaches is most welcome. The current state of the art in cultural heritage and new technologies learning in university syllabuses is undermined and new learning management approaches that combine blended programs (in a classroom and from distance) benefits the self-efficacy of students [1]. The various tools used to study material culture and interpret the data and the results derive from the disciplines of information technology, geodesy, GIS, 3D, and virtual reality [2–5].

We are aware that the ability to transmit knowledge and interpretation depends on the complexity of various factors: technology, format, precision, deduction–induction,

**Citation:** Liritzis, I.; Volonakis, P. Cyber-Archaeometry: Novel Research and Learning Subject Overview. *Educ. Sci.* **2021**, *11*, 86. https://doi.org/10.3390/ educsci11020086

Academic Editor: Maria José Sousa

Received: 3 February 2021 Accepted: 17 February 2021 Published: 23 February 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

communication, context, ontology, etc. The archaeological information virtual ontology, or archaeological cybernetics, a further step forward shown here, refers to all the interconnective relationships generated by the datum, the transmission code, and its transmutability. The data are never neutral and, as a result, we must enhance the properties of the affordances [2,6–8].

A brief state of the art of the scientific literature of the novel subject and a comparison with works of a similar nature is unavoidably restricted by the basic foundations laid down by the tools used to record and handle big data taken from archaeological sites and grand excavation and digital imaging and documentation of unearthed finds with ultimate research, education, and pedagogical impact [6,7,9,10]. Earlier work on the present concept is only related to the initial presentation some five years ago [2,8]. Introducing a novel field, it requires an antecedent evolvement of the development of the digital and cyber archaeology task (scientific goal), the formulation, description, and justification of pursuing this new educational tool.

The information technology (IT), artificial intelligence, and high-tech in image processing have developed rapidly in the last decades in many aspects of our life, needs, and pursuits, and the archaeology, cultural heritage, of past cultures and environments have gained a lot. New terminology entered the scientific vocabulary and new disciplines emerged and gradually became establishing assets, such as: virtual environment, VE; virtual reality, VR; massively multiplayer online world, MMOW; virtual worlds; augmented reality, AR; gamification; serious games; and cyber–archaeology [2,11–13]. Thus, from simple digital recordings to multiple storage, interaction, and management of huge data, a cyberspace of the past is emerging.

It is this holistic approach we favor that integrates the insights of traditional archaeology, archaeological science, and digital archaeology, also offering a critical appraisal of the interface between digital methods and archaeological theory. It is the IT revolution that influences archaeological interpretations of techno-social change.

The first and pioneering monograph edited by Forte [14] on virtual archaeology and computer graphic representation of the past introduced and popularized the term virtual archaeology for the first time. The virtual archaeology is mainly visual, static, with graphics and orientated to photorealism [15]. Recently, new approaches have been added using various interactive practices. The 3D modeling is a very useful practice for the identification, monitoring, conservation, restoration, and enhancement of archaeological objects. In this context, the 3D computer graphics can support archaeology and heritage policy, offering scholars a "sixth sense" for the understanding of the past, as it allows them almost to live it [16].

In the late 2000s, cyber-archaeology (CA) transitioned to archaeology as a discipline. In 1997, it was first applied to anthropology and communication studies, where the connection between computer-mediated communications and online behavior as cultural artifacts was explained; see [6,17–20]. Cyber-archaeology was recontextualized when its meaning was expanded to include cybernetics after a workshop at a Theoretical Archaeology Group (TAG) meeting at Stanford University in 2009. CA is the digital management of much partial information in the field [6,7,21]. It is not necessarily visual, but dynamic, interactive, complex, autopoietic (self-organized) [22], and not necessarily oriented to photorealism.

It is Lake's article [18] that epitomized the history of archaeological computer simulation, starting with early 1970s simulation models, and focusing on those developed over the past twenty to twenty-five years, with a prelude to execution of laboratory exercises via browser.

The past cannot be remade but could be simulated, and CA is the process of simulation and reconstruction of archaeological finds or cultural materials. The archeology of the third millennium is able to process, interpret, and transmit much more data and information relative to the last two centuries. Cyber-archaeology provides new energy and excitement into grand narratives of technological revolution and culture change, yet it does further challenge the high-level theoretical explanations. The digital recording methods have

the potential to create large, regional-scale databases to ease investigation of high-level theoretical issues. In short, this field, emerging in the 2000s, has shown the potential of the IT revolution, which cuts beyond triteness and instead critically engages both its possibilities and constraints.

Most virtual archaeology research projects were visual-oriented in the 1990s; we now believe they will be cyber-oriented in the third millennium. The discussion of the phenomenology of cyber-archaeologies from virtual archaeology and related applications at epistemological, technological, and methodological levels through some significant theoretical approaches and case studies has been introduced by Forte [14] and later reviewed as cyber-archaeology [6]. Recently, Champion [23] argues that gaming in archaeological excavations are systems, experiences, or arguments.

Though this attempt is incomplete and very preliminary, the defiance is to draw up a cultural proclamation for the foundation of this work from different perspectives and with a variety of multidisciplinary contributions and theoretical discussions.

However, the development and applications to design cyber-archaeology's field of research coupled with archaeometry is being reconfirmed and really is proved valuable (in scholarly resources), and indeed, this post-modern revolution is more cyber than virtual, more sustainable than serving only academic interest [5,7,24,25]. Thus, in the field of archaeology and laboratories, the swiftly and progressive use of 3D digital technologies can design diverse and unexplored workflows in the spawning, portrayal, and communication of data [9,26]. This is making virtual labs practical and convenient and intelligent for e-learning purposes.

In such diverse discipline domains in cultural heritage, this cybernated migration of data and models creates unexpected results and more advanced knowledge, and Bateson thoughtfully (1972) forwards this process as the map-code of the cybernetic cycle. The learning of the code contains big data obtained in the field or processed in the lab, which are mandatory to handle by algorithms, interactors, and large storage machines, which after all generate a triggered feedback in lieu of predetermined inter-connections [6].

The simulation, that is, the enactive-dynamic behavior of the virtual actor and the digital ecosystem, is the focus of the cyber-archaeological process. As a result, different affordances and cybernetic models can be generated by the workflow capable of moving and migrating data from the fieldwork to a simulation environment: each can generate feedback and this is a new map code for the interpretation. The core of the process is not the model, data, or environment, but the interaction, the embodiment and the enacting that is the mental action or process of acquiring knowledge and understanding through thought, experience, and the senses. This is achieved and generated by mutual relations, i.e., it is addressed by cognition [6,22].

Along this amazing new information that offers novel investigatory tools to study the past, we have initiated the cyber-archaeometry project.

The IT and cybernated cultural heritage may be expanded to the archaeometry, also called archaeological science, an interdisciplinary field that emerged at Oxford in the 1960s [27]. Essentially, we make use of the CA methodology to a new concept in teaching higher education apprentices via a virtual environment for the investigation of cultural heritage and archaeological materials with natural sciences [28]. Archaeometry involves applications with the use of available instrumentation and methods to unearthed material culture of archaeological excavations, or basic research implying novel mechanisms for getting, e.g., the age or construction of equipment for solving a particular archaeological question.

It can be divided into seven categories with subdisciplines: (i) dating methods, i.e., physical and chemical dating methods, which provide archaeologists with absolute and relative chronologies, (ii) characterization and provenance methods, i.e., artifact analysis, mathematical methods for data treatment (including computer-based methods), (iii) prospection techniques, i.e., archaeo-geophysical, aerial, and remote sensing methods for the location of buried antiquities, (iv) bio-archaeological techniques for the study of ancient DNA and diet, nutrition, health, and pathology of people, (v) environmental approaches, which provide information on past landscapes, climates, flora, and fauna, (vi) conservation sciences, involving the study of decay processes and the development of new methods of conservation and restoration of ancient remains, and (vii) archaeoastronomy, which is the study of astronomical knowledge of ancient and prehistoric societies from orientated structures, devices, and literature sources [28].

Whichever these thematic divisions are, we stress and promote the concept of a perpetually accredited scientific holistic approach (PASHA), which provides current answers to questions arising from contemporary or future problematic issues and/or reassesses past results in the spirit of updating and reassessment. It is a kind of meta-archaeology, which involves philosophy, archaeology, and natural sciences.

Apart from applications, archaeometry also develops research into new methods and materials to improve errors, increase accuracy, and, thus, reliability. The important contribution of archaeometry to cultural heritage and archaeology remained, for most of the years of its development, known either to a few open-minded archaeologists or to a narrow group of academia [28].

Therefore, a modern approach to cultural education and archaeological sciences is affordable with the use of new technologies: from virtual archaeology in cyber-archaeology and to cyber-archaeometry.

The technological tools derived from the field of natural sciences used to investigate past cultures and archaeometry and the reconstruction of past cultures, achieved from analysis of material culture as well as ideas projected onto material culture, could be approached via theoretical modeling and virtual labs.

The aim of present work concerns a first contact with cyber-archaeometry, which has started with the simulation of a petrographic (optical) microscope with the use of avatar in the time-space frame of the laboratory. The use of virtual reality-gaming software was to enhance the effective education of students in the problem-solving exercises of the archaeometry laboratory, without necessarily their presence in the laboratory. The used digital tools follow the theory of mineral identification in archaeological materials (here granite from Osirion in Egypt). The 3D virtual lab, educational aims, anticipated results, benefits in training on a virtual environment, and learning outcomes will be discussed. The work concept was initiated having in mind the rapid development of e-learning and distance learning processes in higher education establishments, and the need for securing (virtual) hands-on experience of university students to expensive devices for archaeometric work in the field and the laboratory. The benefit of learning is enhanced from prohibited access to student classrooms due to disasters such as the current pandemic and includes accessibility to courses taught from distance [29]. The stakeholders are the apprentices, and the present novelty offers potential to support the managerial and policy decisions. Through properly developed tools to navigation, exploration, and control, one achieves the goal. This goal is made via virtual and immersive reality, 3D, virtual environment, virtual processes, gamification with serious games, and the use of avatars in the timespace frame of the laboratory. The integrated plan triggers show up and develop certain theoretical, cognitive, methodological, empirical, or practical (implemental) goals, which are critically discussed.

#### **2. Overview**

A brief review of the scientific literature and a comparison with works of a similar nature, as up to date as possible, is necessary, so that the theoretical new contributions of this research are detailed. However, as can be seen during the development of the present field from the digital and virtual applications, the sources used are pumping ideas from IT, engineering, and mathematics. Reportedly, the novel field of cyber-archaeometry is the simulation design principles that help students' learning and interactions with digital applications in archaeology and cultural heritage, with the experimentation tools in various learning environments. In this aspect, it recalls the integration of science, technology, engineering, mathematics, STEM, with arts and culture (STEMAC); involving computational thinking, engineering education epistemology, computational science education in education, and more generally in learning and teaching approaches and learning objectives [30–34]. The new transdisciplinary and interdisciplinary field that emerges of cultural heritage and archaeology in pedagogics is much valued, in particular in the contemporary emergence from lockdown due to the pandemic, but reappraisal of working conditions too.

The use of the prefix "cyber-" derives from cybernetics, a term that has been given several definitions from the early days of the 20th century [35,36], as a scientific field in retrospect (science of cybernetics and the cybernetics of science) [37] and in other disciplines [38,39]. The word "cybernetics" comes from the Greek word κυβερνητικη´ (kyvernitikí, "government"), i.e., all that is pertinent to κυβερνω´ (kyvernó), the latter meaning to "steer," "navigate," or "govern". Cybernetics has evolved in ways that distinguish first-order cybernetics (about observed systems) from second-order cybernetics (about observing systems), such as in the diffusion of water to obsidian hydration dating [40].

The use of cybernetics in the present work:


Use of cybernetics tools in online learning courses is developing to a sophisticated simulation process [41].

Accepted educational organization models are rapidly challenged by learning technologies. Developments since the 1970s have been reviewed, identifying how the three strands of (a) learning content development, (b) computer-mediated communication, and (c) learning management have been integrated into learning management systems (LMS) made possible by the World Wide Web.

It has been argued that mainstream LMS offer restricted pedagogic opportunities if they are adapted to existing organizational forms, instead of using alternative, easier, and more experienced organizational minimizing constraints. However, prophetically, Beer's work provides us with tools for the redesign of educational systems to make the most benefit from new technologies, guided by Illich's [42] critique of formal education.

Online learning includes offerings that run the gamut from conventional didactic lectures or textbook-like information delivered over the Web to Internet-based collaborative role-playing in social simulations and highly interactive multiplayer strategy games [43].

Furthermore, massive open online courses (MOOCs) provide new opportunities to a massive number of learners to attend free online courses from anywhere all over the world. MOOCs have features that make it an effective technology-enhanced learning model in higher education and beyond [44]. There are a lot of online learning platforms, such us Codecademy, Coursera, Edx, Udemy, etc. E-learning platforms are a fast-growing industry, especially after the advent of Covid-19.

Creating virtual environments offers new ways of educating students. Students could interact extensively with educational and laboratory material, even from their own space. The interaction is very beneficial compared to instructions based on texts or even videos that are not interactive. Virtual environments usually use 3D characters in a game environment, making the training exciting. Moreover, they support multimedia services, hence, students can watch videos, 3D animations, read text instructions, listen to audio instructions, and interact with 3D objects on stage. Those environments can be completely immersive, with 3D interactive functions that simulate, as accurately as possible, a real environment. Metadata is essential for virtual heritage to establish itself as a long-term research area, but metadata has to help the objectives of virtual heritage, which are arguably as much, or more, about education as they are about preservation [16,45].

Virtual training can enable many students to acquire knowledge and skills. On the other hand, virtual labs have been applied mainly in sciences such as physics, biology, chemistry, and in technological sciences.

In an earlier study about the reasons for creating virtual labs, the authors worked on the assumption that a significant portion of students go through labs with little thought about what they need to learn, and just follow closely the written instructions for the experiment to get the expected results. The authors also showed that a primary factor behind this trend is the rigidity imposed by training labs with strict time constraints, large numbers of students, the cost of materials, and security issues. Most of the evidence supporting the value of virtual workshops comes from student feedback. Moreover, the authors of that study found that 75% of students said the software gave them the freedom to explore, focus on the basics of science, repeat procedures, and was easy to use [46].

Several virtual microscope creation efforts have been made but for other purposes. From a search regarding virtual microscopes, we found several applications and approaches that do not meet the needs of an archaeometry laboratory but mainly applications of biology. Virtual microscopes were introduced into the teaching of histology and pathology at the University of Iowa (USA) in 2000 [47] and at the University of Leeds (UK) in 2005. However, the virtual microscope should not be compared with an electronic simulation of a microscope, which is obviously a complete operating model of a microscope.

At the University of Illinois at Urbana-Champaign in America—this work was funded by NASA in 2003 to provide simulated instrumentation for students and researchers from around the world, as part of a virtual laboratory. The simulation was made for an optical microscope that cost \$ 500000, with the aim that every scientist and student can use such a microscope for free. However, NASA has stopped financing it several years later and automatically stopped the development of software [48].

At the University of Delaware, flash technology video and simple operation optical microscope display of the instrument was performed, with no interaction [49].

At South Dakota State University (SDSU) and at New Mexico State University (NMSU), something similar was produced with a little effort to show the use of the microscope as a game [50].

At the Australian Centre for Microscopy and Microanalysis by University of Sydney, a Virtual Transmission Electron Microscopy (TEM) was done. It includes basic imaging with flash technology 2D [51].

The Open University (UK) also developed a virtual microscope (www.virtualmicroscope. org accessed on 3 February 2021). Students are not learning how to use the instrument, but they can enlarge and rotate photos of thin sections from several rocks [52].

Finally, at the Open University of Greece with OnLabs a software application was created, which implements a virtual world simulating the biology lab, not with an avatar; a similar concept to ours, along with the instruments and the rest of objects in it [53].

Regarding the national, regional, inter-regional, and international level, there have been some major efforts that have coined the later evolution of virtual 3D gaming pedagogical dimension. In Europe, the Digital Research Infrastructure for the Arts in Humanities (DARIAH)-funded project is a large-scale, long-term, pan-European endeavor aiming to enhance and support digitally enabled research across the arts and humanities. It does not include cyber-archaeology or cyber-archaeometry works, but remains in our opinion an inadequate level of simply a slow process digitalization of arts and humanities in general, archaeology and cultural heritage being a small part without expected establishment of "digital infrastructures" of the type and level presented in our present work. International appeal for synergy with DARIAH-EU has been endeavored by Schoch et al. [54], based upon work built on earlier interview-based and questionnaire survey research in the "Preparing DARIAH" and EHRI projects, and on synergies with projects such as eCloud, ARIADNE, and NeDiMAH.

In the US, California, San Diego, a web portal is the primary Internet vehicle for communicating with the public and researchers worldwide about At-Risk World Heritage and the Digital Humanities, a cyber-archaeology project awarded a \$1.06 million, two-year UC President's Research Catalyst Award from the University of California (UC) Office of the President to a consortium of archaeologists and information technologists on four UC campuses: UC San Diego, UCLA, UC Berkeley, and UC Merced. Cyber-archaeology integrated projects have been made on a regional and local scale [5,55]. The next US mission rests on the Qualcomm Institute (QI) at UC San Diego, which develops technological and institutional innovations including ancient cultures and cyber infrastructure applications from Mayas to Near and Middle East [56].

Visual Studies are established also at Duke University and the DIG@Lab, with the main research topics being digital archaeology, cyber-archaeology, classical archaeology (Etruscan and Roman Archaeology), and neuro-archaeology, and case studies in Europe, Asia, South America, Middle East, and the US [21].

Despite the large number of publications describing projects based on game engines, there are relatively few describing how game engines can be used as interactive frameworks for collaboration, teaching, and videoconferencing. Thus, a cylindrical stereo screen of the HIVE, Curtin University, Perth, Australia, has been developed for such a purpose [57]. It addresses issues that contribute to a serious challenge for virtual heritage: that there are few successful, accessible, and durable examples of computer game technology and genres applied to heritage. Moreover, it argues that the true potential of computers for heritage has not been fully leveraged and they provide a case study of a game engine technology not used explicitly as a game but as a serious pedagogical tool for 3D digital heritage environments. They combine immersive 3D models and video conferencing, particularly for large scaled cylindrical displays, such as the curved stereo display (e.g., the avatar mirrors that track gestures of the speaker and triggers slides by pointing at the relevant objects; another option is to simply have a hand that points to objects in the scene—the virtual hand moves and points according to the tracked hand of the speaker).

All these major enterprises provide an accurate, precise, workable, simulated, and learning environment of ancient sites, archaeological environments, and 3D artifacts, which contribute decisively to the integrated and holistic study of past cultures, making the field work and museum objects accessible to society and hitherto offer a superior pedagogical potential.

#### **3. Instrumentation**

Figure 1 shows the simulation of a petrographic (optical) microscope with the use of an avatar in the time-space frame of the laboratory, that navigates, explores, and controls the learning outcomes in connection to the archaeometric multisystem work.

**Figure 1.** Polarized light microscopy (PLM) with various components (© lab of archaeometry, University of the Aegean, Rhodes).

Use is made of virtual and immersive reality, 3D avatar, virtual environment, massively multiplayer online processes (MMOP) (virtual processes), and gamification with serious games. A demo is presented online [58]. The benefits include advantages concerning repeats as trial and error at any time, overcoming costly demands of purchasing electronic equipment.

#### *Virtual Development and Materials*

Here, we create a different software, which is based on 3D serious games, using immersive technology with a high degree of presence for the students. The use of avatars, 3D graphics, and gamification aims to ensure success not only in learning the microscope but also in detecting the minerals through information, images, and short videos including evaluation exercises for both knowledge and skills.

This virtual microscope has been designed to train students in learning and using the polarizing microscope. They use virtual hands to operate the instrument guided by speech and texts by human avatars, a laboratory assistant, and a geoarchaeologist (Figure 2A) and snap shots of the material culture from the Osirion Temple at Abydos, Egypt, along with the preparation of the thin section on a glass, the setting of the thin section onto a physical PLM table and images before focus, adjustment, and the clear image with the minerals (Figure 2B).

**Figure 2.** (**A**) Virtual microscope laboratory; the instructor, the apprentice, the VPLA and the operating hands. (© Authors 2021); (**B**) (i) A piece of granite from the Osirion Temple at Abydos, Egypt, (ii) the derived thin section on a glass, (iii) the setting of the thin section onto a PLM table, (iv) before focus, (v) adjusting the blurriness with rotating lenses, and (vi) the clear image with the ingredient minerals (© Lab of Archaeometry, Rhodes, Sample No OS-7/RHO-139).

It is very useful for students to learn the material culture composition and technology of stone implements from the archaeological excavations. It is imperative for the knowledge of knowing the recipes for making ceramics, the composition of rocks, and using databases to identify the types of manufactured artifacts and quarries used to produce implements or monuments. They become also acquainted with the ingredients of ceramics, become familiar with the minerals (main component of rocks), and acquire information on archaeo-materials.

The use of a polarizing microscope in the archaeometry laboratory concerns the enlargement and analysis of small samples, showing the structure of small fossils and the texture of rocks. The observation and analysis of the samples is done by examining a thin section, a few micrometers thick. Initially, a piece of material of about 1 mm and an area of 2 square centimeters is detached. This slice is then smoothed to the point where a flat surface is created like a mirror. The sanded surface is pasted on a glass surface and the sanding continues until a thickness of about 30 micrometers. A light beam of polarized light passes through the petrographic microscope to this thin section.

An example of analysis comes from the Abydos, the greatest of all cemeteries and the home of god Osiris. The adjoining building is the Osirion, which features a central "Island of Osiris" made of granitic stone and surrounded by an artificial canal and sandstone wall, all of which were deep underground in Pharaonic antiquity, invisible to the eye and unknown to all but the priests. A sample of the granitic assembly pillar was examined and was dated by Optical Stimulated luminescence (OSL) to 1980 ± 160 [59] in accord with the archaeological age Middle Kingdom, 11th to 14th dynasties, 2134–1690 BC. Mineralogical qualitative examination revealed: Quartz: moderate, Albite: moderate, Orthoclase/Microcline: low, Biotite: low, and Actinolite: sparse quantities [60].

Figure 3 shows the thin section microphotograph of the studied granite, and Figure 4 is the place of origin, the Osirion Island in Egypt, surrounded by sandstone walls and the granitic pillars.

**Figure 3.** Thin section microphotograph of granite OS7 (crossed polars, magnification x60). Qz, quartz, Plg (Ab), plagioclase feldspars (Albite); Mc, microcline; Bt, biotite.(© I. Liritzis).

**Figure 4.** (**Left**) Top view of Osirion at Abydos with the granitic pillars and the inner part flooded with water. (**Right**) I. Liritzis (left) and Prof. El Gohary (right). Sampling comes from the pillars (© I. Liritzis).

For the creation of the virtual class, the priority was the 3D model of the microscope (Figure 5a), which was given for free by the creator Olek Pieta. Two movements were added to the model of the microscope with the 3D Unity game machine (raising and lowering the bank for focusing and rotating the objective lenses for magnification). From the free libraries of 3D models on the Internet, furniture, and objects (Figure 5b) as well as 3D humans (Figure 5c) were added to the space of the laboratory. Basic 3D movements (walking, sitting, hand movements, speech movements) were added to the 3D humans to make their presence in the laboratory as real as possible. A 3D hand model (Figure 5d) was also placed so that the student could operate the microscope.

**Figure 5.** 3D models of microscope (**a**), furniture (**b**), humans (**c**), and hands (**d**) (© Authors).

The development of the application made by the 3D Models and Game Engine Unity3D. Visual scripting PlayMaker was used for most of the triggering scenarios between user and visual microscope. PlayMaker is a plugin for 3D Unity offering an intuitive structure with States, Actions, and Events to quickly build behaviors (Figure 6).

The students learn to operate the VPLM following the next operations, which have been made via visual scripting and other home-made scripts: (1) turn on the instrument by pressing the ON switch, (2) place the thin section of the sample they want to observe, on the microscope bank, (3) the thin section is placed on the bank, they must rotate the focus screws, (4) cautious rotation of the macro screw until they see the sample clearly and then they focus with the micrometer screw to see it as clearly as possible.

Texts and recorded texts from digital speakers were used to communicate with the students. For the educational material of mineral identification, a same virtual class was used with two boards in which the necessary texts, photos, and videos were presented. The navigation in the educational material can be done with the 3D hands of the virtual microscope but also with keyboard keys.

**Figure 6.** An example of programming inside 3D Unity with PlayMaker, which for every step the student controls his hands in space (passing through invisible trigger points) and activates respective algorithm and directs the apprentice to get the right answer (© Authors).

A logic diagram used to identify the minerals contained in granite rock (Figure 7). Students observing the properties of minerals through photos and video clips and by following the logical diagram are trained to recognize granite minerals.

The identification of minerals follows a gradual process and is examined visually by distinguishing the minerals according to properties that can be distinguished from the passage of polarized light such as color, relief, cleavage, and pleochroism by rotating the microscope bank.

The logic diagram for the identification of minerals in a granite thin section was coded with Playmaker and integrated into the software using the necessary images, adding text with explanations, and allowing the student to explore the next characteristic of the diagram with the (left) or (right) selection button of the teaching material board, pressing with keyboard keys or student 3D hands.

The first characteristic that is observed is the *color*. The colorless minerals (left) are usually quartz and feldspar and the colored minerals (right) biotite or hornblende.

The second characteristic that is observed is the *relief*. The relief of a mineral is how it appears to stand out in relation to the medium that surrounds it. The difference in refractive index between tangential crystals gives the impression that some of them are elevated relative to others. This also makes the boundaries of some minerals appear sharper.

For the colored minerals on the thin section of granite rock, the ones that have a low relief are usually biotite, chlorite (left) and those with moderate relief hornblende (right) (Figure 8).

**Figure 7.** Steps to identify minerals in thin sections of rocks (pers. Comm. Prof. I. Iliopoulos, University of Patras).

**Figure 8.** A photo capture of the virtual class (© Authors).

The third feature is *cleavage*. It is the property of a mineral to break or tear at certain levels, at which the atomic structure of the mineral is weak. For the thin section of granite in colorless minerals, if cleavage is not observed, no mineral is recognized. If cleavage is noted, then it is probably muscovite. No presence of cleavage also means no identification for the colored granite minerals. With cleavage we can identify biotite and chlorite (left) and hornblende (right) in conjunction with the next observation feature, pleochroism.

The colored minerals change the intensity of their color by rotating the bank of the microscope. This property (*pleochroism*) is important as it can be used to distinguish between minerals, which are difficult to distinguish macroscopically. Pleochroism occurs only in colored minerals and is observed only with a polarizer. For chlorite and biotite, it is observed that they change color from light brown to dark brown. To make this change visible, two video clips are included in the virtual microscope. Two video clips also presenting the color changing of the hornblende, where color ranges from yellow-green to brown-green.

During the mineralogical examination, the main diagnostic features for the identification of minerals are the crystalline form (color and relief), cleavage, and other optical properties as the polarized light pass through them [47].

These virtual operations with the virtual hands, texts, and video equip students with the required basic knowledge of mineralogical examination, which has a twofold value: (a) to understand and evaluate mineral identification from material culture, and (b) readiness and capacity that may be refined in a real PLM environment. Moreover, apprentices acquire a large experience on the content of archaeo-materials, learn to function the device and associated physical-chemical mechanism, become familiar and gain experience in an abundance of free chosen time.

#### **4. Discussion**

The construction of the virtual PLM (VPLM) forms the foundation for further development of virtual archaeometric equipment for learning the methods and extract the relevant information. It establishes a new convenient and low-cost application bound to make future archaeological analysis faster and more precise. Moreover, through remote connection, scholars from all over the world can participate in the identification and identification of antiquities. Because the operation can be repeated, this application gives all students the opportunity to operate. Analyzing various results from different angles and viewpoints is no longer just a standardized operation and answer. The physical device cannot return to the previous action for observation at any time during operation. This application can repeat steps so that students can analyze the results in more detail in their studies. Last, if the physical device is used for a long time, there will be problems with inaccurate identification. This application can indeed reduce the cost of equipment replacement and measurement errors.

With this rationale, the cyber-archaeometry (CAm) is the digital IT process of simulation, restructuring, and management of archaeometric processes from the field of natural sciences in relation to material culture, investigated variously (dating, prospection, analysis, technology, provenance, archaeoastronomy, etc.), either as optimum recruited image or as targeted research quest [2]. If this cyber era is seen as a retrospective concept, one has to compare the two approaches in the development of digital archaeometry from archaeological procedural (processualism) in post-procedural thinking, in order to achieve the analysis of hybrid forms of both approaches, achieved by procedural tools (statistical analysis and quantitative methods in different fields, mathematics, geography, archaeometry, anthropology, archeology, and related disciplines). The above is an example of the emergence of cyber-archaeometry.

It is most needed in the present era with the pandemic where online lectures and especially learning involve (virtual) hands-on instrumentation for measuring material culture.

One of the first findings (of archaeological processualism) from the digital point of view was the use of statistical processing and quantitative methods in various fields, including mathematics, geography, archaeometry, anthropology, archaeology, and related disciplines. The critique of subjective methodologies illustrated the need for hyper-taxonomies to understand the past, and this archaeology of computing seemed a tangible and sustainable way for the dream of the process: an objective "scientific" expounding.

In the field of data interpretation, processing, and exchange the digital representations provide new perspectives and a modern approach to training and education, but also to science. It is important to those who understand virtual cultural dynamics through virtual labs [2,6,8]. The motto, "The past cannot be remade but could be simulated" may now be rephrased to "The archaeometric instrumentation and methodologies cannot be available but could be simulated".

There is no doubt that the novel e-learning technologies for cyber-archaeometry have great potential for learning and the organization of education. However, from our project and student interactive process, it is concluded that it is the design and application conceptualization that determines the impact of the present and more archaeometric methodologies, and it is evident that people have widely differing views about their proper use. To understand these, it is prudent to recall and reflect on an earlier investigation about the short history of learning technologies [41]. The introduction of the Internet, and the World Wide Web, appears to have made possible holistic learning for all three of these aspects—content delivery, communications, and learner management—to be integrated into a single system. Hence, the learning management systems (LMS) or the virtual learning environments (VLEs) began to emerge, contributing to online access to computerbased materials culture, providing communication tools, and allowing teachers to provide assessments, track students, build course materials, and manage the whole process.

Cyber-archaeometry started with the VPLM but will have a rapid development, in the near future, for other techniques and methods of the archaeological sciences enriching the curricula with similar virtual environments. Such a case is the obsidian hydration dating, which is one of the dating methods to determine the age of an obsidian (natural volcanic glass), which was a sharp blade for prehistoric people's daily needs. A hydration layer is formed inside the rock, with a width that varies depending on the time of water penetration, the temperature and humidity of the environment, and its special physicochemical structure. The longer the diffusion lasts, the older the obsidian object. The 3D representation of the process of hydration of obsidian is primarily educational, so that students understand the mechanism of hydration in different sources of obsidian and from different environments through a visual language. At the same time, however, the 3D presentation of dating with obsidian hydration will prove the network of codes of interpretation (mathematical algorithms, equations) and diffusion time. Through the simulation, the disordered but with predominant orientation of moving water molecules into the obsidian tool surface, the apprentice gets a sense of the diffused water rim, which is a function of the age of the tool since its last use [40] (Figure 9).

**Figure 9.** Screenshot from the obsidian diffusion software; the obsidian blade, the simulated diffusion rim in dark blue, and the X-Y plot of concentration of water molecules as a function of depth (work in progress, see our video simulation in [58]).

Undoubtedly, technology is providing tools that provide radical new opportunities for education, but simply adding technology to the existing mix is not enough. We need to use technology to develop better pedagogies, and most importantly, to redesign educational organization at all levels, from the course to the national system, to allow potential benefits to be realized [61,62]. The rapid changes in technology affects educational roles and learning outcomes, and cyber-archaeometry falls within this new era. Surely the completion of a reasonable spectrum of available archaeometric methods to a virtual merging environment needs the financial support of the Institution, but from a complexity management perspective, it is important to note that new education subjects, such as the cyber-archaeometry, require a new role in educational policy. Our thesis towards the importance of the individual, along with the pre-existing and continuously evolved theoretical frameworks, concerns the general (world, language group, etc.) and local policy and organization, without discrimination. At any rate, it is not necessary for the board to know or understand all issues, but to ensure that intelligence and operation function well and are a properly balanced [63] viable system model restated by Liber [41].

The latter is reinforced by the results that showed that students as adult learners should be involved in the design and improvement of software. This is in line with the theory of Mindtools, according to which educational software should function as a tool in the service of students, to develop critical thinking and to acquire a high level of knowledge and skills [64].

#### **5. Pedagogical Assessment**

The experiencing in the acquisition of knowledge is efficient with an enactive cognitivism, taking cognition as an action of inculcation in the teaching. In terms of the novelty of the virtual lab in the cyber-archaeometry discipline, the cognition is enhanced by the comprehensive triggering experience and these capacities belong and refer to cultural contexts.

Hence, the attainment of data from material culture concerning constituents, physicalchemical mechanisms for dating, characterization, provenance, locating buried antiquities, and more, could be identified in the mutual interaction between virtual action and experience and consequently between action and knowledge.

The present project searched the ability of a virtual archaeometry lab with digital characters and gameplay elements, to increase student participation and to see how the learning performance was affected using virtual lab exercises. Equally important were the research questions posed as part of our goal: Could students express their educational needs? Could they be involved in the design of educational applications? What are the students' opinions about the effectivity of such educational software in archeology?

The construction of the VPLM, the students' reactions during the working, and the interactive process and operational tasks have produced interesting results.

Briefly, the meaningful effects of the cyber-archaeometry with a 3D VPLM (in a similar manner for other methods and electronic devices from archaeometry) are summarized as follows:

The educational aims and anticipated results that were satisfied:


The benefits in training and advantages of learning from the VPLM included:


• Partial steps can be repeated, giving students the opportunity to analyze the process from different perspectives and opinions.

Concerning the learning outcomes, students:


The results have shown that the 3D laboratory space, the game elements, and the automated exercises, excited the students and increased their desire to participate in the educational activities. A positive result was the satisfactory acquisition of knowledge based on their evaluation results, in relation to the level of information they met. The level of difficulty is a matter for further research, as it can also be a deterrent to using the software.

The existing learning processes in classrooms and laboratories (where available, literally in exceptionally few cases) follow the traditional PowerPoint or oral teaching followed (in some cases) by homework and essays. However, in the archaeology and cultural heritage investigation one needs measuring equipment, basics in natural sciences. In the development of cybernated methodologies, the analysis results are a message for the higher education service policy and practitioners in the institutions of different types and levels, which along with potential users' characterization open a new era in educational sciences.

With these results, the research problem and research goals and pedagogical aims are fully identified and satisfied in this work [57]. The scientific level of the present pedagogical concept has certainly the potential for generalization in any geographical, cultural, and organizational area.

Overall, handling scientific instruments, data collection, data processing and analysis, observation of results, interpretation-explanation of observations, and presentation of results are skills that are very important for students to acquire during their laboratory practice [65].

However, the instruments used in the archaeometry laboratory have a high cost of purchase and maintenance when used by students in the context of laboratory exercises. In addition, the time that students can use the instruments is determined by the opening hours of the laboratory and is relatively limited, while the number of students who practice on the same instrument makes it difficult to use.

Lack of resources in universities is a constant problem that in many countries creates an inability to perform many experiments in archaeometry laboratories. With the use of virtual labs—as, here, the cyber-archaeometry project—the above restrictions may no longer prevent students and researchers from enhancing their skills and knowledge in the most effective learning outcome, the experiential participation.

#### **6. Conclusions**

The VPLM for cultural heritage in the digital era is made via virtual tools and the development of the application is made with 3D modeling and the Game Engine Unity3D as well as for the purpose designed algorithms. The 3D virtual polarized light microscope has been constructed in a virtual laboratory environment and enables the student to comprehend complex physical/archaeometrical terminology and instrumentation. Making use of free codes and written scripts, apprentices followed a methodological way to identify minerals in archaeological materials, learning their characteristic properties in relation to operational modes of the PLM. Apprentices are becoming familiar and tuned with the modern trend of learning (distant learning), and institutions and tutors complement costly equipment for archaeometric results with a built-up cyber-archaeometry project. Here, we initiated a first step for the cyber-archaeometry, while next constructions could include spectroscopic and dating methods, etc. The benefits emerging from the concept of

developing virtual archaeometric environments include recording and managing many diversified big data, applicable to a wide spectrum of archaeometric devices and methods, and provide strength, modernization, and alternative experiences in education making use of the high-tech and IT tools.

Amongst the meaningful effects and merits of the cyber-archaeometry with a 3D VPLM are: execution of laboratory work from the Internet via a browser, making a virtual archaeometric lab for education of university students (e-learning or distance), and learning the operation and methodology of archaeometric devices for measurement processes related to chronology, provenance, and technology strengthen effectively students' knowledge and triggers interest and curiosity for interpretation of laboratory results. We have documented the benefits in training and advantages of learning from the VPLM and satisfactory learning outcomes, with major issues being the gaining of experience through synergy, teamwork, and understanding.

The research implies a new educational tool, which can be expanded to other devices and methods without limitations in the way of acquisition of knowledge in the particular course of archaeological sciences. It provides clues for an integrated policy to the current and future learning processes taken by higher education institutions and policy makers.

**Author Contributions:** Conceptualization, I.L.; Data curation, P.V., I.L.; Formal analysis, P.V.; Investigation, I.L., P.V.; Methodology, I.L., P.V.; Project administration, I.L.; Supervision, I.L.; Validation, I.L., P.V.; Visualization, P.V., I.L.; Writing—Original draft, I.L., P.V.; Writing—Review & editing, I.L., P.V. Both authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author.

**Acknowledgments:** We are grateful to E. Kiratzi, director of Fitch Lab at the British School of Athens and assistants for accepting us in the lab and offering valuable experience in mineral identification, and Assoc. I. Iliopoulos of Patras University, for invaluable help with microscopic mineral identification and the logic flow. I.L. is thankful to Henan University, Kaifeng, Key Research Institute of Yellow River Civilization and Sustainable Development & Collaborative Innovation Center on Yellow River Civilization jointly built by Henan Province and Ministry of Education for support of the Sino-Hellenic Academic Project.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


### *Article* **Using Peer Review for Student Performance Enhancement: Experiences in a Multidisciplinary Higher Education Setting**

**Juan Jose Serrano-Aguilera <sup>1</sup> , Alicia Tocino <sup>2</sup> , Sergio Fortes 3,† , Cristian Martín 4,† , Pere Mercadé-Melé 5,\* ,† , Rafael Moreno-Sáez 6,† , Antonio Muñoz 4,† , Sara Palomo-Hierro 5,† and Antoni Torres 7,†**


**Abstract:** Nowadays one of the main focuses of the Spanish University system is achieving the active learning paradigm in the context of its integration into the European Higher Education Area. This goal is being addressed by means of the application of novel teaching mechanisms. Among a wide variety of learning approaches, the present work focuses on peer review, understood as a collaborative learning technique where students assess other student's work and provide their own feedback. In this way, peer review has the overarching goal of improving the student learning during this process. Peer review has been successfully applied and analyzed in the literature. Indeed, many authors also recommend improving the design and implementation of self and peer review, which has been our main goal. This paper presents an empirical study based on the application of peer review assessment in different higher education BSc and MSc courses. In this way, six courses from different studies at the University of Malaga in Spain are subject to the application of peer review strategies to promote student learning and develop cross-wise skills such as critical thinking, autonomy and responsibility. Based on these experiences, a deep analysis of the results is performed, showing that a proper application of the peer review methodology provides reliable reviews (with close scores to the ones from the teacher) as well as an improvement in the students' performance.

**Keywords:** peer assessment; peer review; collaborative evaluation; higher education; rubric

### **1. Introduction**

The integration of the Spanish university education system into the European Higher Education Area (EHEA) has entailed a paradigm shift in the teaching-learning process, leading towards a student-centred learning (SCL) approach where the learner and their needs are the primary focus. Under this perspective, active collaboration replaces passive knowledge transmission, with both teachers and students becoming mutual active contributors to the education process. Thus, successful mutual responsibility becomes critical in enabling the development of the learners' autonomy [1]. In Spain, this change in

**Citation:** Serrano-Aguilera, J.J.; Tocino, A.; Fortes, S.; Martín, C.; Mercadé-Melé, P.; Moreno-Sáez, R.; Muñoz, A.; Palomo-Hierro, S.; Torres, A. Using Peer Review for Student Performance Enhancement: Experiences in a Multidisciplinary Higher Education Setting. *Educ. Sci.* **2021**, *11*, 71. https://doi.org/ doi:10.3390/educsci11020071

Academic Editor: Ismo Koponen Received: 31 December 2020 Accepted: 9 February 2021 Published: 13 February 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/ licenses/by/4.0/).

the teaching-learning paradigm (or from the system of credits based on teaching hours to workload) has entailed a conceptual change in the higher education system [2,3].

The transition towards SCL has required the adoption of different or tailored forms of learning, resources and evaluation [4,5], so the transmission of technical knowledge and the methods of assessment and evaluation are aligned with competence development. Evaluation systems are one of the issues which have been most affected by the convergence towards SCL since they are one of the essential elements in the planning and execution of the teaching-learning process [6]. As indicated by Rodriguez-Esteban et al. [7], today evaluation is not understood only as a final act, being rather a process that is part of the same learning system. Not only must they serve to accredit learning, but they must also help students to learn and teachers to improve their teaching [8].

In this context and among a wide variety of learning approaches, *peer review* stands out as a key tool for SCL. Peer review, applied to the education field [9], is a collaborative learning technique where students assess other student's work and provide their own feedback with the overarching goals of improving the student learning process during this process, enhancing the understanding of the peer's work (maybe with a different approach) and improving the quality of the final product [10]. It is also referred to in the literature as peer evaluation, peer response, formative peer assessment or peer editing [11]. From now on we consider the term *peer review* in this article. This technique aims to involve students in the evaluation process, allowing a review of the material from a critical perspective, so that, when they analyze different approaches and points of view that may not have been considered when performing their task, students will have a more global vision of their learning in the course. During this collaborative process, the transfer of knowledge is mutual: in those who give the feedback and those who receive it, since this activity triggers critical reasoning and self-assessment in both sides [12]. Rubrics, as identified by Hafner et al. [13] are normally used. These provide a common scheme (usually prepared by the teacher in collaboration with the students) for assigning marks in each step of the assigned task, guiding students during this process.

#### *1.1. Related Work*

Peer review has proved to be a powerful evaluation strategy that brings a number of advantages over classical methodologies, such as facilitating the students' acquisition and development of skills and promoting their capacity of self-direction of their own learning [14]. Previous studies as Nicol et al. [15] have shown that, during the review process and the generated feedback, the evaluative judgment of the students is improved, both about their peers work and their own work. The analysis of other perspectives also greatly contributed to provide them with a deeper understanding of the course material [16]. A growing ability to give constructive feedback during peer review has also been shown in first-year students [17], who may be unaware of professional standards and expectations and reluctant for critiquing work and writing thoughtful feedback. Promoting independent learning, increasing student motivation, building problem-solving skills are also notably advantages to remark in this learning process [18].

Peer review has been widely applied and analyzed in the literature. In Saiz et al. [14], peer review at university is analyzed, highlighting the characteristics of this strategy and the conceptual, institutional and relational difficulties of its implementation. Challenges and benefits of the implementation of self and peer review and identifying potential inhibitors in practice are explored in Adachi et al. [19]. Moreover, authors also make recommendations about improving the design and implementation of self and peer review. Amendola and Miceli [12] propose a peer review methodology completely conducted through online technologies (in particular using the Moodle e-learning platform) showing the benefits such as lack of space and time in the standard lesson environment applying this methodology. In Indriasari et al. [20], a survey of peer review of source code in higher education showing how such activities have been implemented in practice, examining instructor motivations and the primary benefits and difficulties of this practice have been

reported. Authors also identify a wide variety of tools to facilitate the peer code review process. Peer review has been been widely adopted by major massive open online course (MOOC) platforms, but there is little evidence about if it is appropriate or under what conditions. In Meek et al. [21], student performance, participation and opinions of a peer review task of a science course in a MOOC are examined. According to Reddy et al. [22], the positive impact of training in peer review learning experiences on science students over three years in higher education is demonstrated. In Gaynor [23] the quality of peer feedback, the importance of assessments and student perceptions are investigated.

Apart from the aforementioned topics, there are other issues which have also been previously addressed by the peer review literature, such as the influence of cultural perspectives [24,25] or the analysis of students perceptions [26,27]. Moreover, Panadero et al. [28] investigated the impact of friendship on students scoring, finding a positive impact of using rubrics on scoring objectivity. However, the overscoring effect generally observed among students was amplified when the use of a rubric was accompanied by a high level of friendship between assessor and assesses. The influence of peer-related factors such as gender on peer-awarded marks is studied by Lagan et al. [29]. They observed the presence of a slight and positive gender effect between participants of the same gender over participants of different gender, with female evaluators being more consistent at awarding marks. Another branch of the peer review literature focuses on the assessment of the effectiveness and reliability of the use of this technique for educational purposes [30,31].

Within the Spanish context, the literature also provides some experiences in applying peer review to the higher education context. For example, one of the first documented experiences was conducted by Sánchez Rodríguez et al. [32] in an Education Science course at the University of Malaga. They found that although students' scores were slightly lower and more concentrated than those awarded by teachers, both were strongly correlated. Regarding students' perceptions about the vast majority of the respondents showed a positive attitude about the possibility of valuing and being valued by peers. Conde et al. [18] applied peer review to technological courses at the University of Leon to help students in technical studies to develop specific abilities such as critical thinking and get more involved. They evaluated student's opinions and performance and found that the application of peer review increased students participation and led to higher scores. Moreover, also by comparing students and teacher scores, they observed a significant and strongly correlated relationship. The quantitative data analysis about students' perception showed that, in general, the level of satisfaction regarding the methodology's appropriateness and its beneficial role in acquiring critical thinking is elevated. Only first-year students showed some concern regarding its use. Finally, it is also worth mentioning the study of Dopico [33] at the University of Oviedo, who found that emotional aspects related to personal criteria other than prearranged ones seemed to intervene in peer review processes. Besides, he also observed that the use of digital tools for the peer review exercise enhanced the motivation of the students.

#### *1.2. Contributions of This Paper*

Despite these clear advantages, there are very few courses at the University of Malaga (UMA) in Spain that apply strategies of self-assessment, coevaluation or peer review methodologies. The fact of identifying errors of peers or own mistakes during evaluation, provides a stable critical base to continue the construction of knowledge, while giving confidence to students on the skills acquired with a strengthening of them. In the literature [34,35] it is also mentioned some factors that limits the adoption of this technique such as: (1) not having the necessary maturity to evaluate, (2) not taking the evaluation seriously, (3) having negative attitudes towards an evaluation of these characteristics and (4) considering the evaluation as an additional load.

Although all these studies give a valuable insight into the use of peer review for evaluation and assessment in higher education settings, each of them performed the peer review for the evaluation of different types of assignments and in different scientific fields. Starting from these efforts, and by adding to the literature, the study described in this

article aims to gain more insight into the possibilities of formative peer review and its application in the classroom, setting the first steps towards continuing the much-needed research on the use and performance of peer-assessment methodology in the future. Thus, in order to assess the effectiveness and reliability of peer review, this study aimed to answer two key questions:


By performing this peer review methodology we have also aimed at the development of certain transverse competences in students, such as responsibility and objectivity when making decisions. The study has been developed in six STEM (stands for science, technology, engineering and mathematics) courses at the University of Malaga, Spain. Students perform peer review of one or more tasks in these courses, such as the resolution of an exercise, the development of a report or the presentation of a project, which are carried out on the basis of similar statements and/or specifications with well-established rubrics. This methodology allows us to strengthen those concepts that teachers consider key in each of the courses using peer review learning processes based on error detection and observation of the different approaches that students expose when solving problems.

#### *1.3. Structure of the Paper*

The rest of the article is organized as follows. Methodology and developed experiments are detailed in Section 2. In Section 3 results are presented and discussed. Finally, our conclusions and future work are presented in Section 4.

#### **2. Methodology**

The methodology consists mainly of peer review evaluation of projects, class tests, works, activities, in addition to oral presentations. We have considered the case in which students know who they are evaluating [28]. One of the key objectives in the development of this activity is to ensure that students are able to make value judgments on the work done by their peers according to established criteria, in order to improve their degree perception of achievement of the subject. In order to make this possible, students need to have assimilated knowledge and be aware of what they are trying to evaluate.

The fact of knowing the students on whom evaluations are going to be made can lead to biased scores, depending on the affinity or not with the person being evaluated. It may even be that the rating is influenced by the possible consequences that may occur in personal relationships [28]. Therefore, in this case where the evaluated person is known, it becomes very necessary to have clearly defined and delimited the evaluation criteria, with specific headings for the qualification. In addition, it is convenient that there is an established weighting, where the qualification of the students influences the final grade of the task, but it is not determinant. In this line, we have not considered students weighting as we are on pilot experiences, but it is considered as ongoing work. However, the fact that student assessments have a direct impact on the score is a change in their usual role. It is important that students are aware that this activity prepares them, among other benefits, for the relevant critical thinking development in their upcoming professional life, so that without their involvement, the activity is meaningless.

#### *2.1. Research Context*

This study has been developed in the framework of a Teaching Innovation Project (PIE19-209) (https://www.uma.es/formacion/noticias/proyectos-de-innovacion-educativa-2019-2021/, accessed on 30 December 2020) grant funded by the University of Malaga. It has been carried out by a team of assistant professors from different STEM departments at the University of Malaga that have met with a common objective, as different weaknesses were recognized in particular learning points that we consider to be essential in our teaching activities. After researching and documenting it, we identified the peer review approach as an invaluable tool that could be developed in the context of the teaching

innovation call for projects that our University opens every two years. Thus, we decided to elaborate a proposal and this was submitted for evaluation. Then this project was granted one and a half years ago, although our activity on it goes back to January 2018. In order to assist the reader, details of our experiences are shown in Table 1.


**Table 1.** Summary of experiences.

In total there have been 409 students involved in this activity, distributed among the different courses offered by the University of Malaga. This provides a key multidisciplinary character to this work, starting from a relevant sample of students and applying peer review to a varied range of topics.

A total of six STEM courses have been chosen for this study. The peer review methodology had to be adapted to the nature of each of the subjects as well as to the preset teaching plans without disrupting the natural flow of the course. A different instructor is in charge of every experience since they belong to different STEM departments in the same institution (University of Malaga).

This work was motivated by the recurrent difficulty of a considerable proportion of students to acquire the necessary skills and thus solve problems that involve minimum requirements to pass the subjects. In this context, peer review is considered as a reinforcement activity that improves the way in which objectives are achieved. Additionally, given the wide range of possible solutions to science and engineering problems, students can propose different approaches that do not have to be strictly provided by the teacher. We believe that the peer review methodology can help the students to reach different solutions and understand multiple valid perspectives to address the presented challenges.

Finally, it has been observed that on many occasions it is easier for students to understand what their peers want to communicate to us than what the teacher is telling them. That is why a source to reinforce knowledge and contribute to their learning process has to be provided by their peers and we have tried to design a methodology to take advantage of this mechanism.

We propose students that voluntarily participate in this project to correct a series of activities of other classmates. In this way, they will review exercises using a detailed rubric provided at the time of carrying out their own exercises, so that they know the evaluation criteria. The evaluation of the students to their peers will be assessed according to the approximation of their scores to their peers with respect to that given by the teacher. Thus, we motivate the dedication on the part of the students. Moreover, it has to be noticed that this methodology provides a very interesting source of evaluation in the post COVID-19 environment as it provides data for a more enriched and constructive evaluation.

To sum up, we have conducted a series of experiments that contrast how the use of this methodology affects different scenarios with different conditions. In some cases, all students have taken part in the peer review voluntarily and in other cases there are groups in which it has been applied and others in which it has not.

#### *2.2. Research Method*

The method applied in this study has been divided into three well-differentiated steps in two types of experiences consisting of:


#### 2.2.1. Experiences Type I

With these experiences we aim to observe the impact that peer review has on the final score of the course. Students were put into teams to perform different activities in the classroom in order to assimilate basic concepts. The tasks that are being evaluated by peer review consist of solving exercises in which students apply the concepts learned in class following a theoretical approach. For example, the instructor explains the concept of the derivative and its applications and students have to solve a concrete problem that involves the derivative and that has not been previously solved in class. Therefore, the proposed tasks are very similar to the exercises to be solved in the exam. Afterwards, solutions were exchanged so that students could peer review each other. Finally, the teacher made a final correction of the results and the students' own corrections of their classmates. Accordingly, experiences 1, 2 and 3 have been conducted following a similar procedure so that the impact of peer review has been analyzed.

In these experiences, on-site activities have been carried out. Peer review was not implemented in all exercises as we considered that its application only on selected parts of the syllabus is a more effective way to preserve the appealing character of such a new methodology.

However, we have relied on the background and expertise of the instructor to choose those exercises that deal with fundamental concepts of the course and which, after previous years' experience, students find difficult to acquire. Between 2 and 3 activities have been carried out throughout the course, covering around 30–35% of the syllabus so as not to overload the students.

In addition, in these experiences, each student involved in the peer review has a period of time to carry out the exercises under the guidance of the instructor. These guidelines serve as a reference to accomplish the review of the exercises (between 2 and 3 peers) that takes place during the second stage. The number of exercises in each activity will be between 3 and 5. During this time, the student reinforces the knowledge that he/she has acquired after facing the proposed exercises they knew how to solve. In addition, although the student has the proposed solutions to those exercises, in some cases new possible solutions can be found. Finally, after the correction of the exercises, the instructor supervises all the scores in order to avoid deviations by students who mistakenly give wrong solutions.

#### 2.2.2. Experiences Type II

Experiences belonging to this type are intended to test students' critical thinking and the capacity for objectivity with respect to the teacher. For them, different class tasks are presented and must be evaluated by their own peers following detailed rubrics shown in Tables 2–4. In order to assist the reader, the difference between the score issued by the teacher and the score issued by the peers (students) will be represented graphically so that final conclusions can be drawn. Experiences 4, 5 and 6 have been carried out for this purpose.

In these experiences, we selected activities in the form of projects to be presented by the students, at all times under the instructor guidance. These projects focused on the fundamental topic of the subject while reinforcing other transverse skills such as public speaking, synthesis, slide preparation, etc.

In addition, in these experiences, the students had a detailed rubric prepared by the instructor before defending each of their projects, so that they were aware of the items to be assessed, both by their classmates and by the instructor himself (see Tables 2–4). Before starting, it was proposed that the students review the rubric in depth in order to know how it should be applied and resolve any relevant doubt of the process. From this point on, students began to prepare the project under the supervision of the instructor. When the time came for the presentations, the students were in charge of assessing following both the rubrics and the exhibitions of their classmates' projects. It was the instructor's task to make their own evaluations of the presentations, as well as to review each student's.

#### **Table 2.** Experience 4-Rubric.


#### **Table 3.** Experience 5-Rubric.



**Table 4.** Experience 6-Rubric. Mark: 1, 2 (Basic); 3, 4 (In Progress); 5, 6 (Achieved); 7, 8 (Highlighted).

#### *2.3. Procedure Description and Applications*

Peer review to strengthen concepts in students through an active methodology from the teaching and learning process to evaluation. The aim of the project is to involve students in the evaluation process, allowing a review of the material from a critical perspective, so that, seeing the need to analyze different approaches and points of view that may not have considered when performing their task, students will have a more global vision of their learning in the subject. This vision will allow us to identify possible errors, limitations and/or highlights, see different methods of resolution and also detect when there are successes or improvements to the techniques applied.

In short, the aim of the approach is that students have access to various forms of performance of the same task which will allow them to acquire a deeper knowledge of the subject. When students review projects or tasks of other classmates, they acquire a critical vision of the work, preparing them for their professional future, where any task is subject to public assessment. This is a motivating aspect for the students who see in the realization of this evaluation its application to the real world. This last concession does not intend at any time to discharge the responsibility of the teacher in terms of evaluation and qualification of students, since the teacher must monitor that both the qualification and the evaluation have been made based on objective criteria and, of course, the teacher has the ultimate responsibility.

The procedure is organized as follows. In a first face-to-face session, the knowledge test will be carried out. Students must identify themselves by name in the test that must be given to the teacher. However, in order to guarantee the anonymity of the answers, the name must be able to be replaced by an identification code that the teacher will establish.

The teacher must establish this identification code and associate it with the name of each student. This relationship will only be known by the teacher. The teacher will remove the student's name with the pertinent anonymous identification code. The teacher shall make the appropriate copies, so that each student can evaluate at least two tests from other students. In a second face-to-face session, and after the publication of the corresponding rubric, each student must evaluate the answers of at least two other peers. The possibility of having the rubric in the session is up to the teacher. In this second face-to-face session, the student will record the assessment he or she considers from the other classmates. The teacher will collect the evaluations on a corresponding card, which must contain the name of the student who has carried out the evaluation of the exercise and the code of the student evaluated. The teacher will have to make a definitive correction of the contents and determine how far they deviate from the score issued by the peers. In this sense, a smaller discrepancy between the teacher's and the student's score will guarantee that the student has assimilated the content and that the correction by pairs has been based on objective criteria.

During the academic year 2019–2020, the planned objective of the fieldwork by the teachers involved in this project and the students involved has been fulfilled for the realization of collaborative peer evaluation activities. A total of six experiences have been carried out in multidisciplinary subjects of different scores involving two of the three designed evaluation modalities: anonymous evaluation (the student does not know who he is evaluating) and public evaluation (the student knows who he is evaluating).

Under the mode of anonymous peer review, it is intended to limit the influence of bias due to any type of social relationship between students, deploying a series of protocols and procedures to ensure the anonymity of the contents of the peer evaluation. On the other hand, in the modality of public evaluation students are exposed to real evaluation situations that require the promotion of their critical sense under the application of evaluation validation mechanisms that ensure that the scores really reflect the acquisition of the objectives of the proposed task.

Next, we include a brief description of different experiences developed throughout the academic year 2019–2020. Derived from the multidisciplinary and enriching experiences developed during the last academic year, during the current academic year an exhaustive analysis of the results and information obtained will be carried out in order to obtain clear and concise information on the strengths and weaknesses of this peer evaluation system as well as to identify possible lines of improvement. This will contribute to a restructuring in the final phase of the project, where those improvements identified based on the results obtained in the first phase will be applied, if necessary, and the results obtained will be analyzed again to determine the effectiveness of the innovative proposal. Furthermore, during the present course, new experiences will be developed that will contribute to improve the quality of the results obtained from the present PIE (Teaching Innovation Project) with respect to the collaborative peer evaluation.

#### *2.4. Experiences*

This section presents the six experiences grouped in two different types as we have previously described.

#### 2.4.1. Experience 1

Participants: Students of the "Linear Algebra and Geometry" course in the "Mathematics" BSc and the "Mathematics and Computer Science" dual BSc.

Procedure: Contents included in this experience covers two of the total of four lessons of the course, where the other two followed a classic evaluation approach

The peer review took place in the classroom. The session took one hour and a half. Students were first grouped into 3–5 member teams to solve several activities in one hour. Once they solved the assigned problems, the solutions of each team were exchanged. At that time, the copies of the activities solved correctly were also handed out so that students had a correction guide. The instructor explained one standard way to solve each proposed activity and they appear in the correction guide. However, we are aware that in many cases there are different ways to solve a task, and this is highly valued.

Each group had to correct the activities solved by other groups in the last 30 min of the lecture. At the end of the session, the teacher collected all the activities, both the activities that were done and the corrections made by the students. It was possible to observe a great interest by the students to know if the corrections that they had proposed to their peer's exercises were correct or not. In addition, the attendance level for this sort of activities was very high. The fact of having their own peers at the time of being scored leveraged their motivation. From the 77 students in the group, 32 of them voluntarily performed the peer review task while another 45 students did not.

#### 2.4.2. Experience 2

Participants: Students of the "Algebraic Structures" course in the BSc degree of "Mathematics" and the "Mathematics and Computer Science" dual BSc.

Procedure: As in the previous case, the experience covers two lessons of the total of four. The peer review took place during an hour and a half session. These lessons were held virtually due to the COVID-19 using the Zoom application. This tool allowed the creation of small rooms so that students could be divided into 3–5 member groups. Students spent an hour solving the activities. Afterwards, they exchanged solutions, also virtually, and spent the last half hour of class correcting. Finally, the teacher collected all the activities and corrected them. Since they are mathematical exercises, in many cases there are different ways to solve them. Nevertheless, the correction by the students of exercises solved in a different but correct way from the one provided by the teacher is highly valued. From the 67 students, 29 of them voluntarily participated in peer review activities while another 38 did not.

#### 2.4.3. Experience 3

Participants: Student of "Statistics II" course in the "Marketing and Market Research" BSc degree.

Procedure: At first, the students had to group together and deliver the activities as a group. However, as a consequence of COVID-19, teaching was delivered online and the exercises were carried out individually. In this sense, the teacher created an appropriate task so that students would be able to manage all relevant issues under lesson 1. Specifically, a list of activities was given to the students before solving them in a virtual session. Students had to value both the resolution and the result of the activities of their classmates. From the 81 students in the group, 41 of them performed the peer review task voluntarily while

another 40 students did not. Since class attendance was not mandatory and these activities took place during on-site sessions, some students did not get involved in the experience. Others simply did not want to take part in them because of the additional workload. In addition, 133 students from two other academic groups of the same course did not do the peer review task and serves as a control group.

#### 2.4.4. Experience 4

Participants: Students of the "Wireless Networks" course in the "Telematics and Telecommunication Networks" MSc degree.

Procedure: This experience has adopted a model based on the development of projects whose objective is the definition of the wireless elements, technologies and architectures to be applied in order to solve a real-world use case. The general lines of work are proposed (smart cities, smart buildings, security. . . ) where the specific use is defined by the students (e.g., farm security sensors, healthcare monitoring in a hospital. . . ). The project teams were formed by 2–4 students each.

The evaluation of the projects was based on oral presentations of 10 min by each team. These were performed in front of all the students and a jury formed by the teacher and three other invited members with professional experience in wireless networks.

Hence, from the presentation, the projects get two evaluations: one from the jury and one from the rest of the class. Both evaluations were based on the same rubric (see Table 2). The rubric was known in advance for the students, and the evaluation was gathered online after each exposition through a Google Forms based poll.

#### 2.4.5. Experience 5

Participants: Students in the "Renewable Energies" course of the BSc degree "Energy Engineering".

Procedure: Students taking part in this experience were about to address the final year dissertation where they will be evaluated by means of a project report thesis and its defense.

On the first hand, students were grouped in teams of 2–3 members. Each team had to address a project of an installation based on renewable energies. This project had to be real and accurate, and it had to be properly developed and explained. For this purpose, students received previous instructions on how the report should be written. Technical aspects were explained during lessons in class.

Afterwards each team had to expose their project in a 10 min pitch, but because of the pandemic situation, on campus lessons were reduced to the minimum and virtual teaching was the recommendation. Because of that, an alternative activity was designed. This experience consisted in a peer review process where each student had to revise one project among those developed by the others teams. In that manner, every project would be evaluated by two or three students and the teacher. For this purpose a rubric (see Table 3) was designed and provided to each student so they would evaluate the project.

#### 2.4.6. Experience 6

Participants: Students of "Stochastic Models" course in the "Mathematics" BSc degree. Procedure: Following the aforementioned methodology an activity was developed consisting of the realization and presentation, by the students, of a project of analysis and forecast of a time series based on real data. For this purpose, groups of up to three participants were formed. Finally, each student evaluated both the work and the oral presentation of the other projects.

For the implementation of the proposed methodology, an evaluation rubric (see Table 4) was created with ten criteria and several aspects to be taken into account. In the evaluation form, open sections were also considered so that the students could include assessments and proposals for improvement, both of the rubric and of the development of the activity itself.

In order to have a comparison tool to assess whether the activity could be considered valid, the teacher also evaluated the projects and their exhibitions in the same way and under the same rubric used by the students.

#### **3. Results**

As previously detailed, a total of six experiences have been conducted, which are grouped in two categories. On the one hand, Type I experiences (1–3) aim to give new insights into the effectiveness of peer review, that is, how the application of peer review methodology can affect student performance. To that end we have analyzed the the scores of the final exam of the course as previously done by other authors, such as Amendola et al. [12], Conde et al. [18] and Li et al. [31]. In this way, groups with and without peer evaluation during the learning process have been compared.

On the other hand, Type II experiences (4–6) focus on evaluating the reliability of the scores provided by peers by analyzing them in terms of their statistical distribution and their relation to the marks provided by the instructor.

#### *3.1. Experiences Type I*

Results obtained from Type I experiences are displayed in Figures 1 and 2 according to the description given in experiences 1, 2, and 3 in Section 2.4. These diagrams display the number/fraction of students whose score is withing specific ranges. NP (standing for Non-Participant) refers to the fraction of students who did not attend to the final test. The rest of categories are defined following a 10-point scale. Since all the experiments were conducted in the context of the Spanish higher education system, it is considered that 5 is the minimum score to pass the exam. Accordingly, the following intervals [0–5), [5–7), [7–9) and [9–10] correspond to no pass, approved, outstanding and pass with distinction respectively.

Regarding experiences 1 and 2 (Figure 1), students were split into two subgroups: peer review (PR) and the control group (CG). As previously mentioned, the selection bias is based on a self assignment by the students themselves. Members of PR subgroup are those taking part in peer review activities during the learning process. On the other hand, members of CG are not taking any sort of peer review activity. In other words, they have followed a traditional learning process. In addition, the final exam of the course is formed by two tests: test 1 is related to the part of the syllabus where traditional learning process is used, whereas test 2 evaluates competences acquired in the second part of the subject (where only member of PR puts into practice peer review). This means that peer review activities have only been deployed by the members of subgroup PR in test 2 (bold in diagrams).

Based on this grouping, a comparison of the different subgroups based on mean scores and their standard deviation is of interest to check if the students' performance improves as a result of peer review activities. Left-hand side diagrams in Figure 1 describe the distribution of scores based on the number of students within each range, whereas right-hand side diagrams do the same but based on the percentage/fraction of students.

 **test 2 PR**

 test 2 CG

 test 1 CG

 test 1 PR

0

**Figure 1.** *Cont.*

(**d**) Experience 2 (fraction of students).

**Figure 1.** Score distribution obtained in type I experiences 1 and 2.

Relevant conclusions can be drawn after from the results compiled after conducting experience 1 (see Figure 1a,b). Based on the mean scores, the control group (CG) obtained similar grades in both tests (*µ* = 6.01 ± 2.48 for test 1 and *µ* = 6.12 ± 2.46 for test 2). It can be explained since CG members have followed the same traditional learning process for both tests. In contrast, a significant difference has been reported when comparing results between test 1 and test 2 by PR members. Peer review activities allowed an increase in the mean score of test 2 (*µ* = 8.26 ± 1.81) vs. test 1 (*µ* = 6.86 ± 2.37). This shows that peer review has positively contributed to the learning process in the second part of the course being the process of learning from peer's flaws a key asset.

Experience 2 provides analogous results (see Figure 1c,d). One can observe that peer review activities deployed in the second part of the subject (test 2) contributed to an increase in the average score of PR members by 2.41 points (from test 1 *µ* = 6.36 ± 2.37 to test 2 *µ* = 8.77 ± 1.51). It is nevertheless noteworthy that results compiled by CG members have also shown an increase between test 1 and test 2 (*µ* = 6.16 ± 2.24 vs. *µ* = 7.30 ± 2.72) but less significant. This shows that, despite the fact that other factors may affect the average score (some parts of the syllabus may result more complex), the effectiveness of peer review has been proven, as it has shown a consistent improvement in the students performance expressed in terms of their final score.

Regarding experience 3 two figures have been shown (Figure 2a,b). Unlike experiences 1 and 2, only three subgroups have been formed, and a single test has been done covering the whole subject. The first subgroup (PR) is formed by those students who voluntarily decided to take part in the peer review experience (it is displayed in bold in figures). Alternatively, subgroup CG1 is associated to students who did not take part in such activities. It is important to mention that both subgroups (PR and CG1) belong to the same academic group/shift. In addition, subgroup CG2 represents students of the same course but from a different academic group/shift who did not take part in peer review activities (note that due to classroom limitations a course can be divided into different academic groups/shifts). A similar comparison can be set based on the academic results of the three different subgroups. Scores by subgroup PR was 8.52 ± 1.57, which is clearly higher than those obtained by members of CG1 (*µ* = 6.52 ± 2.21) and CG2 (*µ* = 6.80 ± 3.30). In line with experiences 1 and 2, experience 3 also indicates that students taking part in peer review activities benefit noticeably from the experience. All students have gone through the same final evaluation tests and course syllabus.

**Figure 2.** Score distribution obtained in type I experience 3.

#### *3.2. Experiences Type II*

Results concerning type II experiences were assessed by calculating the existing distribution of the score difference [12,30,31] obtained by subtracting the reference-score from the score provided by peers. Here, the reference-score is the one provided by the instructor (experiences 5 and 6) or the average between the score provided by a panel of experts and the instructor (experience 4). A positive difference indicates that the peer's score is higher than the reference-score. In order to unify the criteria, scores will be expressed on a 0–10 scale before calculating the difference.

The items detailed in the rubrics aim to cover most of the competencies playing a relevant role, but they have been grouped into two main categories. The first one refers to items related to the contents (C) of the task carried out by the student group. It basically refers to how students have applied the specific concepts related to the main scope of the course. In contrast, formal aspects (F) refer to whether students have properly applied the available resources and competences to communicate their conclusions and results.

Figure 3 displays the distribution of the score difference grouped in the two aforementioned categories. Left-hand side figures refer to scores related to the contents of the tasks (Figure 3a,c,e). In general terms, scores provided by peers are distributed around the reference-score (i.e., score difference is zero). However, it can be observed that mean

values are slightly positive (from *µ* = 0.71 to *µ* = 0.08 in experiences 4 and 5 respectively) meaning that scores provided by peers are slightly higher than the reference-score (set by the instructor). The adjusted underlying Gaussian probability distribution has also been included just for comparison, where the standard deviation shows that the dispersion of the score difference is around two points (0-10 scale) in the worst scenario.

Right-hand side Figure 3b,d,f show the distribution of the score difference concerning the formal aspects (F category). It shows how dispersed the distribution is of the score difference but considering only interdisciplinary competences (not exclusively related to the topic of the course). Analogous distribution is observed in this case, with standard deviation lower than 2 points in all cases and mean values slightly positive. Only experience 5 (Figure 3) has a relatively higher mean value (*µ* = 1.68 points).

**Figure 3.** Results of Type II experiences according to rubric items category.

In order to answer the question about if peer review is an effective and reliable method for evaluation, the average of all the items of the rubric scores (for both categories) is represented in a single histogram for each experience. As displayed in Figure 4, a global perspective of the distribution is provided, where the mean value of the peer scores is significantly closer to the instructor's reference-grade. Note that in experiences 4 and 6, mean values are relatively low (*µ* = 0.72 and *µ* = 0.11 respectively). Only experience 4 shows a mean value slightly higher (*µ* = 0.96). However, regardless of the experience or classification assumed for the evaluated items, dispersion (standard deviation) is enclosed by 2 points out of 10. These results can be considered as reference values for the level of discrepancy and variability between the teacher and student qualifications in the peer review process.

(**c**) Experience 6 (all items).

**Figure 4.** Results of Type II experiences including all the items in the rubric.

#### **4. Conclusions and Future Work**

When used for educational purposes, peer review methodology is considered a learning technique where students proactively evaluate the work of other students. In this article, an empirical evaluation of peer review has been carried out in a multidisciplinary set of six science and engineering courses in the University of Malaga in Spain, where more than 400 students were involved. In order to answer the two key questions of this study, experiences have been grouped in two different types: (i) those analyzing the impact that peer review methodology has on the students learning process (effectiveness), and (ii) those assessing students' critical sense and the capacity for objectivity in their provided reviews (reliability).

Regarding the effectiveness of the peer review methodology (first key question), it has been possible to validate how the experience has improved the results of those students who have participated. This has been confirmed for different degrees. Here, it can be highlighted that all experiences show a clear improvement in student performance. This is reflected in the general course scores of the participants, being the increase of more than two points (0–10 scale) in the best of cases and at least of one point but always achieving positive results with respect to the control group. Regarding the second key question, we found that the scores provided by the students are very similar to the score assigned by the teacher. Therefore, students are able to evaluate consistently enough according to our results, confirming the feasibility of peer review as a reliable evaluation methodology. Regardless of the type of experience, this methodology has shown to enhance

the learning process prior to evaluation. Not only do students assimilate contents for their first test, but they also develop a well based criteria to identify the mistakes made by other classmates, encouraging their critical sense, and identifying the more complex concepts. This way, students acquire the ability to identify the most common errors allowing them to complement their knowledge of the course more effectively.

After analyzing the application of the peer review methodology, it can be concluded that all the experiences have successfully performed from both academical and motivational perspectives. In this sense, the motivational aspect is key, especially considering remote education conditions. From our point of view, this fact is crucial to encourage students to participate in these initiatives as they clearly improve their academic performance. Hence, peer review can be a key approach for its implementation in the current pandemic scenario where the use of new technologies has become essential in the educational model. Moreover, peer review seems to be an advisable complement to the traditional evaluation process which is frequently considered one of the weak spots of online teaching. In addition, peer review provides the instructor with an invaluable source of information to support a more accurate evaluation.

As future work, a study on how anonymous evaluation affects the results is proposed. It is expected that friendship bias could be mitigated by deploying a series of protocols and procedures to ensure the anonymous nature of the evaluation process. For example, this can be guaranteed by the random allocation of the exercises to be scored following a double-blind review process. This reduces the likelihood that the answers will contain any sort of information about the author of the question. These can be done during the development of the course in question, either in written form or through any computer tool within a classroom setting. In this context, tests might be individual to help the anonymity of the review process.

In addition, based on our experience, we propose a continuous improvement in the rubrics in order to increase the quality and objectivity of the review process. This will be achieved by the redefinition of their format and instructions, the elimination of the points that can be considered weak and the inclusion of new items that can be interesting for the evaluation. These proposals for improvement of rubrics will be gathered from both the analysis of current rubrics as well as the obtained results accompanied with a survey fulfilled by those students that voluntarily participated in the peer review. Moreover, it has been identified that in some cases the correction of exercises requires an excessive amount of time considering the planned students workload. For these, an improvement is deemed necessary to adapt the amount of exercises to be performed during class in order to dedicate more time to the correction to be performed by the students. Another option being envisaged is to dedicate one session to the development of the activities and another to their correction.

Finally, it is planned to study the impact of the peer review methodology in the same six courses in the next years in order to assess its continued use. Moreover, its application to other courses and fields is also planned in order to widen the multidisciplinary nature of the analysis.

**Author Contributions:** This research work has been a collaborative work, but task leaders were assigned in our project plan to organize the workflow. Literature review, S.P.-H., C.M. Conceptualization J.J.S.-A., and A.M.; methodology, P.M.-M., and A.T. (Alicia Tocino); software, J.J.S.-A., and S.F.; validation, S.F., A.T. (Antoni Torres), and R.M.-S.; formal analysis, C.M., and S.P.-H., investigation, C.M., and S.P.-H.; resources, A.T. (Antoni Torres), and P.M.-M.; data curation, A.T. (Antoni Torres), J.J.S.-A.; writing—original draft preparation, A.T. (Antoni Torres), and R.M.-S.; writing—review and editing, S.F.; visualization, R.M.-S., and A.T. (Alicia Tocino); supervision, A.M.; project administration, P.M.-M.; funding acquisition, A.T. (Alicia Tocino). All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was funded by the Spanish Teaching Innovation Project PIE19-209 ("Collaborative evaluation to strengthen student competencies through an active methodology from the teaching and learning process to evaluation") at the University of Malaga.

**Acknowledgments:** The authors are grateful for the helpful and constructive comments of the three reviewers in improving this paper. We would also like to express our gratitude to Pedro Rodríguez Cielos, who guided and helped us as a mentor to conduct this Teaching Innovation Project.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Abbreviations**

The following abbreviations are used in this manuscript:


#### **References**

