**Cooperative/Collaborative Learning**

Editor

**Robyn M. Gillies**

MDPI • Basel • Beijing • Wuhan • Barcelona • Belgrade • Manchester • Tokyo • Cluj • Tianjin

*Editor* Robyn M. Gillies School of Education The University of Queensland Brisbane Australia

*Editorial Office* MDPI St. Alban-Anlage 66 4052 Basel, Switzerland

This is a reprint of articles from the Special Issue published online in the open access journal *Education Sciences* (ISSN 2227-7102) (available at: www.mdpi.com/journal/education/special issues/cooperative learning).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. *Journal Name* **Year**, *Volume Number*, Page Range.

**ISBN 978-3-0365-2971-4 (Hbk) ISBN 978-3-0365-2970-7 (PDF)**

© 2022 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.

## **Contents**


#### **Kate Ferguson-Patrick**


Bringing Out-of-School Learning into the Classroom: Self- versus Peer-Monitoring of Learning Behaviour Reprinted from: *Educ. Sci.* **2020**, *10*, 284, doi:10.3390/educsci10100284 . . . . . . . . . . . . . . . . **189**

## **About the Editor**

#### **Robyn M. Gillies**

Robyn M. Gillies Ph.D. is a Professor of Education at The University of Queensland. Her research focuses on the social and cognitive aspects of learning through social interaction. She has spent over twenty years researching how students can be encouraged to engage in class and learn. Her research spans both primary and secondary schools and has focused on inquiry learning in science and mathematics, teacher and peer-mediated learning, student-centred learning, including cooperative learning, and classroom discourses and processes related to learning outcomes. Her recommendations on how teachers can translate research into practice have been widely profiled in the international literature and on the website of the Smithsonian Science Education Center in Washington, DC.

## **Preface to "Cooperative/Collaborative Learning"**

Collaborative learning is well recognised as a pedagogical practice that promotes socialization and learning for students from kindergarten to college level and beyond. The term collaborative learning is used very broadly in this Special Issue and includes cooperative learning, peer learning, and peer collaboration. These pedagogical approaches have been used successfully in both formal and informal settings in schools and the wider community to not only promote learning but also a greater understanding of others with diverse social, personal, and academic competencies.

The purpose of this Special Issue is to bring together a diverse range of international scholars to highlight recent developments in research on collaborative learning across different contexts, including subject areas and student populations to illustrate how this research has been translated into practice. The emphasis in this Special Issue is on research that has a strong evidence base for the work that is presented and includes empirical studies, best evidence synthesis of the relevant research, case studies, and theoretical papers. It also highlights how different technologies have been used to facilitate group interaction, dialogue, and learning. I believe there is much to be gained by sharing and learning about what happens in different disciplines and contexts and how different collaborative pedagogies can be implemented when needed to promote understanding and learning.

> **Robyn M. Gillies** *Editor*

## *Article* **Student Perceptions of a Synchronous Online Cooperative Learning Course in a Japanese Women's University during the COVID-19 Pandemic**

**Chie Sugino**

**Citation:** Sugino, C. Student Perceptions of a Synchronous Online Cooperative Learning Course in a Japanese Women's University during the COVID-19 Pandemic. *Educ. Sci.* **2021**, *11*, 231. https://doi.org/ 10.3390/educsci11050231

Academic Editor: Robyn M. Gillies

Received: 30 March 2021 Accepted: 7 May 2021 Published: 13 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Department of Culture and Tourism, Komazawa Women's University, Tokyo 206-8511, Japan; c-sugino@komajo.ac.jp

**Abstract:** As the coronavirus disease 2019 (COVID-19) pandemic expanded worldwide, most Japanese universities launched online learning as an emergent measure; hence, securing the quality of online learning remains a challenge. This study aimed to understand reasons behind students' preferred mode of online learning during the pandemic and to explore the impact of online cooperative learning on students' class participation by analyzing their voluntary comments. A qualitative content analysis identified three factors that are related to students' decisions and motivation about participating in synchronous online classes: mutuality resulting from interaction, the impact of COVID-19 on their life and learning, and individual circumstances. This small-scale study was conducted under the unusual circumstance of the pandemic, and the quality of student interaction was excluded from the analysis. However, their enjoyment arising from interaction encouraged their participation in a synchronous class and discussion. They expressed themselves and listened to others attentively, creating a favorable climate for learning. Students' positive interdependence observed in this study suggests that cooperative learning cultivates a classroom culture where students are willing to contribute without the fear of losing face. This study indicated that participation, cooperation, and active engagement create a positive feedback loop, promoting each aspect even in an online setting.

**Keywords:** informal cooperative learning; online learning; synchronous participation; small group discussions; case study; higher education; Japan; COVID-19

#### **1. Introduction**

The coronavirus disease 2019 (COVID-19) pandemic has forced universities worldwide to introduce online courses [1–12]. This new learning style will continue to be in demand, considering the risks of new infectious diseases and disasters [12] and the expansion of distance learning supported by the development of technology [7,13,14]. Although a major advantage of online courses in times of emergency has been their fast-growing implementation, ensuring the quality of online learning remains a challenge [2,4]; for instance, college students in Japan have expressed frustration about instructors' lack of information and communication technology (ICT) skills, preparation for online classes, and consideration for students [15]. Traditionally, among OECD countries, Japanese schools have the lowest utilization of digital devices during lessons [16]. This trend also applies to universities; 70.3% of departments had not introduced real-time online education by 2017 [17]. Accordingly, online classes in most Japanese universities began in response to the pandemic [18], and research on online learning has been a less represented area in Japan. Although many articles began to appear on university bulletins after the pandemic, these articles reported how online learning was introduced and managed in each institution [19–21], leaving online learning as a research area requiring urgent attention and analysis with regard to its impact on student learning in Japan in comparison to international situations [1–11]. This study explores student perceptions of a synchronous

online course using content analysis to understand the implications of their participation in online cooperative learning.

#### *1.1. Informal Cooperative Learning to Transform Classroom Culture*

In Japan, active learning has been rigorously introduced in college classrooms since the 2010s, with the expectation that it will promote students' active involvement in learning and improve their attributes [22]. However, it is difficult to bring active learning that emphasizes interaction into college classrooms with students who have been exposed to the traditional one-way instruction and have not been encouraged to express their opinions at school [23]. Activities to promote student engagement may ironically lead students to develop passive attitudes; since these activities place pressure on students to participate, they are not required to decide whether to be involved, and the nature of the group activities may blur the extent of each student's responsibility [24]. Thus, active learning requires a paradigm shift in students' concepts of college classes and participation and makes it essential to cultivate a classroom culture that welcomes students expressing opinions and asking questions. Discussions in small groups encourage interaction [25–27] and arouse students' interests, enhancing student learning experiences [25]. Small groups are a useful platform for students to pay attention to other members' remarks [26]. For that purpose, I used group discussions in my classes as a method of "informal cooperative learning" [28,29] to facilitate classroom interaction and learning from other students.

Johnson et al. define that "Cooperative learning is the instructional use of small groups so that students work together to maximize their own and each other's learning" ([28] p. 14). In "informal cooperative learning, students work together to achieve a joint learning goal in temporary, ad-hoc groups that last from a few minutes to one class period," so it is useful "to focus student attention on the material to be learned, set a mood conducive to learning, help organize in advance the material to be covered in a class session, ensure that students cognitively process the material being taught, and provide closure to an instructional session" in this kind of learning ([28] p. 16,17). Although cooperative learning in higher education has several theoretical bases, many research studies are based on social interdependence theory, which views positive interdependence as ideal to promote interaction when students realize that they need each other's effort to achieve their goals [29]. In line with this theory, Gillies indicates the following five points important to encourage cooperation among students: "establishing positive interdependence among group members; facilitating promotive interaction; encouraging individual accountability; explicitly teaching the appropriate social skills; and, encouraging groups to reflect on both the processes involved in managing the tasks and interacting with their peers" ([30] p. 51).

In Japan, Okada observes that some university students are uncomfortable with classroom interactions because of the fear of "losing face among peers" ([23] p. 95). The lack of opportunities to raise questions and express opinions in class makes it difficult for them to overcome their feelings of awkwardness. Similarly, when class discussions were introduced initially, the students in my classes were worried about joining discussions, sharing their thoughts, and communicating with others due to their lack of such experience. However, they eventually started enjoying exchanging opinions and creating a welcoming atmosphere for more reserved members [31]. This shows that students' negative preconceptions and passive attitudes about group discussions may change when they experience "positive interdependence" [29,32] through interactive communication.

#### *1.2. Spread of COVID-19 and Students' Experience of Online Learning*

The spread of COVID-19 infections in 2020 forced all classes to go online at the university in the present study, similar to the situation at other institutions [18–21]. Universities in metropolitan areas were particularly reluctant to resume face-to-face classes [33]; many cases of infection were reported among college-aged youth [34,35], which made the universities sensitive to the reporting of a cluster of infections or even a single infected student that could provoke criticism against the students and negatively affect the reputation of

these universities [36–39]. Thus, many universities developed and launched online classes to provide opportunities for learning in an emergent response to the pandemic [40]. The online classes continued throughout 2020 and are expected to continue, at least partially, throughout 2021 [41].

Although synchronous online learning was an emerging research area a few years ago [42], the pandemic has led to its sudden expansion and global attention. Over the past two years, thorough research on students' perceptions of synchronous online learning that has been introduced as emergency measures has been produced, reflecting experiences in higher education worldwide [1–11].

The sudden transition to online learning due to the pandemic provoked students' anxiety and nervousness [5,9]. They expressed their preferences for face-to-face classes [2,3,6,8,10]. High levels of interaction in face-to-face classes supported their preferences [2,8]; thus, those who preferred face-to-face learning experienced difficulty in adapting to online learning [2]. The lack of interaction with instructors and classmates caused isolation and loneliness [3,4]. While some students expressed a decreased quality of learning [2,4] and loss of motivation for learning during the pandemic [2,3,5,11], others welcomed the recording of live classes as flexible learning opportunities [9,10] and regarded online classes as improving their technology skills [10].

Students' perceptions of online learning depend on their "sense of community" ([7] p. 173) [10], which is enhanced by their "feelings of belonging" ([3] p. 13), "mattering as a student at the college" ([3] p. 14), and connectedness with faculty members and other students [2,10]. Although individual differences exist in their adaptability to the online environment [1,3,9,11], class interaction has a motivational impact on student learning [2,5,10]. Convenience and safety are associated with positive perceptions of online learning; the former saves the students time and money from not needing to commute [5,6,8]; and the latter maintains the necessary physical distance among students and faculty members [6].

The often-raised difficulties in online learning during the pandemic are distraction [2,4,6,8,10,11] and ICT issues [2,4–7,10,11]. Online activities may be useful for keeping students focused during the class [10]. With regard to students' ICT environment, Internet connectivity is particularly critical for students in rural areas where ICT infrastructure is developing [10] and for those who share communication devices and an Internet connection with their family members who work and study from home [2,6]. Students become fatigued from looking at screens [2] and are overwhelmed by heavy workloads [4,6] in the sudden shift to online learning. Students' negative perceptions toward online learning may influence their academic performance and motivation for learning, which may further deteriorate their attitudes [2].

#### *1.3. Implementing Synchronous Online Group Discussions*

Research on online learning prior to the pandemic indicates that interactive components are important for maintaining student concentration and preventing isolation during online classes [43]. Interaction positively influences student perceptions and participation in the class [44–47], creating a "sense of community" among the students [48,49]. Online interaction, such as small group work in synchronous settings, is useful because students feel others' presence [7,8,50,51]. However, implementing group work online requires careful planning, particularly with students who do not yet have sufficient experience in this type of learning.

In contrast to face-to-face sessions, nonverbal communication is restricted in online classes [44,52,53]. It is difficult for online participants to make eye contact and capture others' facial expressions [44]. An unstable Internet connection may also break off communication [7,10,44]. In this kind of environment, students may feel nervous about speaking and communicating online [4,53]. Those without personal computers find it frustrating to read materials and instructions shown on smartphone screens [54]. Thus, instructors in

online classes need to pay more attention "to creat[ing] a supportive climate" to encourage the students' participation ([52] p. 206) than they need to in offline settings.

The limited nonverbal communication results in a feeling of alienation among students [48,55,56] and may decrease their interest and motivation [55,56]. Audio and video use in online classes increase communication among students [49]. In particular, visual support helps students get acquainted with each other online [49,57].

While recognizing the benefits of seeing classmates in virtual classes, students participating with their videos turned off have often been observed [58,59]. Castelli and Sarvary [58] reported that 90% of the students in synchronous online classes did not use their cameras because they were concerned about their appearance and the background. These researchers also found that not turning on the camera, despite being encouraged to do so during the class, was regarded by these students as "unspoken social norms" ([58] p. 7). Participation with the video turned off may distract students' attention from the class because of their invisibility to their instructor [45,55]. However, Tonsmann concluded that the lack of video feed does not hamper communication and instruction when the students are communicating with each other verbally or through other interactive methods [58].

In addition, factors promoting extrinsic motivation, such as participation in an extra credit or an evaluative task, may be necessary to improve students' online participation [52,60].

It should be noted that synchronous online discussions may "stay at a fairly superficial level," as students regard it as an informal exchange ([61] p. 34). While the informal and impromptu nature of synchronous discussions may provide the students "security in being able to make comments with a level of confidence ([57] p. 197)," it could also "work against [student] engagement in purposeful dialogue" due to the lack of "sufficient reflective time to generate comments or input" ([57] p. 205). Difficult and complex topics require more preparation in advance and a longer time to discuss than less complex ones. For students without sufficient experience, an informal discussion may be a good option.

#### *1.4. The Objective of the Study*

This study aimed to examine the impact of online cooperative learning on students' class participation by analyzing the students' voluntary comments because "learner perceptions are of key importance in understanding online participation" ([62] p. 80). Since students who provided comments in addition to the reflections assigned to them tended to choose live participation and have a positive perception of the class, analyzing their responses helped highlight factors promoting their participation. The first objective of this study is to understand students' preferred mode of online learning during the pandemic and the reasons behind their preferences and the second is to explore the effects of online cooperative learning on their participation.

Accordingly, the following research questions were addressed in this study:


#### **2. Materials and Methods**

This study adopted qualitative research methodologies that are suitable to deal with descriptive data, such as interview texts, fieldnotes, and other documents, and to analyze them contextually [63]. The qualitative approach was appropriate for exploring student perceptions about participation in online cooperative learning classes using their writing as the data source.

#### *2.1. Context of the Study*

This study was conducted with students from a political science course at a women's university in suburban Tokyo. The course of choice was an introductory course about multicultural societies, immigration, and intercultural communication, intended for, but not limited to, first-year students.

Although classroom interaction was sacrificed mainly due to technical reasons during the spring semester at this university, the development of a virtual classroom system (Google Workspace for Education) and accompanying video conference tool (Google Meet) enabled us to use various tools for interaction, such as conducting polls and discussions in small groups and asking questions in the chat. I introduced interactive activities in the fall semester, starting with whole-class communication using chats and polls and then incorporating breakout discussion sessions. Table 1 shows the main topics of the course on which this research is based, and the interactive components employed in each class. The basic class delivery was synchronous and online (Table 2) and, considering student convenience and the communication environment, students could choose from either joining the synchronous class or watching the video of the synchronous class within three days after the class (asynchronous participation). Further, when moving from a lecture to a synchronous discussion session, students who did not wish to join the discussion could leave the class and watch the recorded wrap-up session later (hybrid participation); participation in the discussion was optional and did not count for extra credit so that students facing difficulty in live access were not disadvantaged. All the students, regardless of their preferred mode of participation, were required to submit reflections on the discussion topic of each class within three days after the class so that the next class could start with a feedback session on their reflections. I uploaded all the students' writings on the class post after deleting their personal information and anonymizing their names. I also commented and answered questions on some of these student inputs in the feedback session. When personal communication was necessary, I sent an email prior to the next class. As I tried to put the students at ease using small talk to create an accessible atmosphere, the feedback took 20–30 min, which was longer than the time taken in the previous year's face-to-face classes. I also encouraged the students to read their classmates' reflections, as one of the classes required students to respond to a reflection of their choice as an assignment. Each in-class activity took 20–30 min to fit in a 90 min class period.

A typical group-discussion-based class is shown in Table 2. After an introductory lecture on the discussion topic based on immigrant children and education, nine policy options on education in a multicultural society were shown to the students [64]. Those who did not prefer group discussion (hybrid participants) exited the class to write about their reflections on the topic. The students who joined the group session (synchronous participants) discussed prioritizing these policy options while representing the various stakeholders, such as an immigrant parent, a leader in the host community, school staff, and a member of the board of education as a coordinator of the discussion. After the discussion ended, the instructor wrapped-up the group session with all the synchronous participants by sharing each group's progress. Finally, the participants left the session to work on the same assignment given to the participants who did not join the discussion. The asynchronous participants watched the lecture videos and submitted their reflections. A student survey was conducted following this class.

In this class, students' participation style was roughly divided into three groups, 24 hybrid participants, 21 synchronous participants and 16 asynchronous participants. In a post-class questionnaire conducted after this class, among the synchronous participants, 15 students who joined the synchronous discussion stated that the synchronous class made them want to participate in the class; three students referred to the interactive aspects of the class; another three students felt that "watching the video later was bothersome."

For those who chose asynchronous/hybrid participation (37 students), poor quality Internet connection was a major reason for their choice (17 students). In addition, 10 students preferred asynchronous classes for convenience. While five students did not have specific reasons, four students expressed their negative feelings about (online) discussions. Two students mentioned their health conditions as the reason for not choos-

ing synchronous participation. Information on student participation gained from the questionnaire above was used as signposts for the content analysis.


**Table 1.** Course schedule and interaction.

**Table 2.** Basic structure of a discussion class and choices of participation.


#### *2.2. Participants*

The participants included 61 female undergraduate students registered for this course during the 2020 fall semester, who agreed to participate in this study when the semester ended. They were full-time students aged between 18 and 21 years. Among them, four were juniors, two were sophomores, and the rest were first-year students. The majors of the first-year students were not decided at the time of the semester, and, since there were just a few second- and third-year students in this study, it should suffice to describe their majors broadly as the humanities. Most of the participants had not experienced online learning before the pandemic and some of them did not have personal computers. The research ethics committee of the university approved this study (approval number: 2020–006).

#### *2.3. Data Collection*

The data for qualitative analysis were collected from students' comments submitted via Google Forms. In addition to the assignments consisting of reflections on the topics covered in the class, I provided a voluntary comment section after each class where the students could add messages to the instructor, along with any complaints and advice for class improvement. I made it clear that their comments in this section were not for evaluation and shared all the comments with the class in the feedback session. In addition, multiple-choice class surveys were occasionally conducted to determine the students' tendencies regarding class participation.

All data were obtained and analyzed in Japanese, following which they were translated into English for this study.

#### *2.4. Analysis*

This research employed a qualitative approach to explore student perceptions about participation in synchronous classes. The data were analyzed using inductive content analysis [65,66], which is suitable to "identify meaningful subjects answering the research question" [65] (p. 10). Further, a manifest analysis approach was chosen to understand what the students "actually say," staying "very close to the text" and focusing on "surface structure" [65] (p. 10). This approach was because the collected data comprised students' open-ended comments, rather than information gained through interviews where the researcher can clarify the intention and ask for elaboration.

The analysis procedure was based on that advanced by Bengtsson [65] and Elo and Kyngäs [66]. After 628 voluntary comments from 59 students were collected and reviewed, 158 comments were extracted for further analysis in light of the research questions. These 158 comments were broken down into 248 segments by theme. These segments were numbered, given headings, and open coded. During this process, a tentative coding list was created, and the codes were labeled for each heading. The list was updated several times to sort the headings into well-represented codes. The finalized codes were grouped into sub-categories by common concepts, and generic and main categories were generated accordingly. Finally, the results of the analysis were described.

#### **3. Results**

The coding process produced 31 codes, 13 sub-categories, and six generic categories under the three main categories of "mutuality, "impact of COVID-19," and "individual circumstances" (Table 3). Frequencies were calculated to avoid the duplication of entries by the same students. The main category, "mutuality," was related to the interactive aspect of the class; "impact of COVID-19" integrated students' responses to the pandemic and the new learning environment; and "individual circumstances" included the students' personal issues and external conditions beyond their control.

#### *3.1. Mutuality*

This category contained codes with greater frequencies than the other two main categories. Student comments on the interactive aspects of the class accompanied their expression of enjoyment and appreciation for opportunities to learn others' views. Such sentiments positively affected the class atmosphere so that they felt free to respond.

#### 3.1.1. Reflection of the Interaction

Students often referred to their enjoyment of the class, as evidenced by the following comments: "The class was enjoyable" and "I enjoyed learning in this class that was atypical in a good sense." Although it was difficult to identify which aspect of the class they enjoyed from these comments, the frequent reference to class interaction suggested that was the source of their enjoyment. Many students found it interesting to exchange opinions through the chat and became aware of others' reflections during the feedback; for instance, students stated that "opportunities to know others' opinions make online classes interesting" and "I enjoyed writing my comments in chat." A student wrote that she found it easier to respond online than in face-to-face classes, and this attitude was generally confirmed by the students' quick responses during the class. Their enjoyment also related to their expectations for the next class, expressed as their willingness to participate and interests

in upcoming topics: "I am excited about what topic we will discuss the next class" and "I want to join the next class discussion."


**Table 3.** Categories generated from the abstraction process.

Students' positive interdependence manifested in their affirmation of learning from others: "I was stimulated by different ideas at the wrap-up session" and "Group discussion gave me new perspectives"; assumption of responsibility for contribution: "I spoke and so did other members" and "All the members contributed actively"; awareness of support from the group members: "The members helped me in expediting the discussion [when I was allocated the leading role] "; and an indication of respect for other students: "The members of my group listened with an attitude that welcomes different views."

#### 3.1.2. Class Atmosphere

The students also mentioned their observation of peers' attitude in classes by other instructors: "Contrary to certain other classes where the students were forced into groups, the discussions in this class went smoothly, and it was easier to express opinions" and "In other classes, one or two students spoke in a group and the discussion did not continue, but students offered so many responses for one question in this class that we could spend dozens of minutes for discussion."

Although participation in discussions with the video turned on was encouraged at first to facilitate communication, no one took the lead, and the audio-based discussions continued. Despite this invisibility of the students, no serious problems during the interaction were reported. This was confirmed when the instructor visited each breakout room, where the students were having lively discussions. A student who experienced difficulties sensing who speaks next noticed that her group members "compensated for the disadvantages of online communication by voicing responses." Another student who was too shy to express her opinions wrote that she "became confident in a discussion because the members gave her positive responses without denying her opinion." When a student noticed her peer's silence, she wrote: "I will call her by name next time [to invite her opinion]." The students mutually made efforts to create a welcoming atmosphere where they felt comfortable expressing themselves.

The instructor's feedback and informal chats also helped the students form an attachment to the class and encouraged their participation: "I was glad when the instructor commented on my reflection, which does not happen in regular classes"; "I laughed out loud in front of my PC when I heard her feedback"; and "I like responding in chat because the instructor reads and comments on each entry."

#### *3.2. Impact of COVID-19*

This category reflected how the pandemic influenced their learning and daily lives. The restricted physical contact caused a shift to virtual communication, which made the students yearn for real exchange. Interactive components of the online class were regarded as second best to fulfilling their desire.

#### 3.2.1. Evaluation of Online Learning

Students' evaluation of the online tools used in class was positive overall, which promoted their participation and formed a positive perception of the class; for instance, they stated: "This class fully utilizes benefits of the online system" and "It is new and enjoyable to answer questions using Google Forms." They "got accustomed to responding in different tools" as the class introduced new functions.

Responses to the questionnaire at the end of the course revealed that although more than half of the students chose to participate mainly in the synchronous class, students were appreciative of the participation being optional, regardless of their actual choice. Having alternatives for participation reduced their stress about a steady Internet connection and accommodated their preferred learning style: "This was the only course where the students can choose from live or video class, and I felt safe when I had a problem in connection"; and "I appreciate that the instructor provided different styles for participation."

Some other students confessed their negative impressions about online learning: "It was difficult to communicate because I felt distance online but I had to accept it" and "I am not positive about online discussion looking at a screen." Those who participated in video classes commented that they "got frustrated with the one-way style" and "it was boring just to listen."

#### 3.2.2. Reaction to Social Distancing

The new social distancing measures isolated the students from society, as one student expressed: "I spent most of the time at home without opportunities in touch with the same aged youth." As a reaction to such a confined situation, the students sought actual college experiences in class; for instance, they stated: "I realized that I was a university student in this class when I communicated with other 19-year-olds with different backgrounds" and "I felt as if I was actually in university when I had chances to express my view and talk with others in a group."

The online interaction with the instructor and other students cultivated their sense of community, compensating to some extent for the lack of physical access to the campus: "The class interaction made me feel I was taking a class together with other students" and "Although it was online, I felt other students closer when the class shared their feelings and thoughts."

The real-time participation further enhanced their togetherness and helped them overcome the sense of distance originating from the remote nature of the online classes; as one student stated: "This is the only class in which I did not feel I was studying online!" The live class gave them the sense of participation: "I have a real feeling of participation when responding to questions."

#### *3.3. Individual Circumstances*

This category consolidated students' individual circumstances, including external issues beyond their control, representing their home environment and personal issues related to students' characteristics and physical conditions.

#### 3.3.1. Home Environment

Most of the technical problems the students reported concentrated on online connectivity. It was either an unstable Internet connection or application malfunctioning that restricted their access to the class. As elaborated by the students: "The communication environment made me choose asynchronous participation for the latter half of the course"; "I was shut out from the Google Meet"; "The battery of my smartphone was dead"; and "The quiz using Google Forms does not load well." At an early stage of the course, a student sent a request "to shorten the live session, as it costs more communication fee." This concern was responded to after the second class by offering three types of participation (Table 2) to the students, and the students appreciated having choices regarding how to participate.

Although expressed by a small number of the students, family affairs had a significant impact on their decision for class participation: "I was not able to join the discussion because my family came back." This family interference in class participation was a particularly critical concern for students sharing a room with their families who were working or studying from home. Another student commented that she was not able to find the time when she was visiting her parents' home in a rural area due to family matters.

#### 3.3.2. Motivation

This generic category had both motivators and demotivators. The former can be broken down into student sense of growth and their interests.

The classroom interaction gave them chances to explore certain issues themselves, which allowed them to broaden their perspectives: "I was able to think about issues that I had not usually paid attention to" and "This class often required me to express my thoughts, and it helped me consider different cultures."

Other students expressed their changes in attitudes: "I needed to have a broader point of view to live in this globalized world, and this class gave me a good opportunity to improve the way I think and to mature as a person" and "I was able to revisit my personal values at each class."

The students also commented on their expectation of personal growth: "I was not able to summarize my opinion today, so I would like to challenge it again in the next discussion." Some students who "do not usually enjoy group work" expressed their "determination to participate [in the next session] to get better at discussions."

In addition to students' experiences of personal growth, some of them were genuinely interested in listening to others' views; for instance, they stated: "It was interesting to find different sensibilities in other comments that mentioned things I had never questioned" and "I like a class where I can hear various opinions."

The topics covered in this course also motivated them: "I enjoyed this class because it delved into topics that other international studies courses did not deal with, such as international and cross-cultural marriages." When the students were interested in the topics addressed, they felt that the time in class passed quickly: "The content of the lecture is so interesting that ninety minutes are not enough" and "I enjoyed the quizzes today and the class ended too soon."

The final sub-category was demotivators. The students who preferred face-to-face classes and real interaction either expressed their resignation or desire: "I'd like a face-toface class but I accept the current style for now," "I wish this class could be held face-to-face because I like the class interaction," and "I really wish for face-to-face classes as soon as possible."

The students' health condition influenced their participation style: "I was about to join a live class, but my stomachache continued, and it was difficult for me to discuss." The students who suffered from occasional bad health appreciated the video class.

Students' invisibility to the instructor and their classmates affected their concentration, as stated by a student, "An online class environment where I am not being watched makes me feel sleepy." In a class survey at the end of the course, one of the respondents who said they were sometimes disturbed during the class explained that "when I attended a class from home, my attention went to other things, and I kept rewinding the video."

#### **4. Discussion**

The previous section presented findings of the content analysis. Students' voluntary comments were sorted into three main categories: mutuality, the impact of COVID-19, and individual circumstances. The mutual nature of interaction resulted in student enjoyment and promoted positive interdependence among participants. It also affected class atmosphere because the interactive components made the instructor seem approachable to the students and encouraged them to express themselves, creating a friendly environment that welcomed student voices. The social restrictions during the pandemic had impacts on the students' daily lives and learning. Social distancing forced them to take online classes, kindling their need for bonding and appreciation of the campus life. While it was fortunate that many of them saw positive aspects in online learning, there remained challenges that remote access needed to overcome, such as students feeling alienated and experiencing technical problems. Sensing possibilities for personal growth motivated them to participate in live online classes and discussions.

Based on the social interdependence theory of cooperative learning and in light of the insights gained from the literature, this section elaborates on the reasons behind students' decisions to participate in either synchronous classes or asynchronous classes and their motivation to join live group sessions according to the two research questions and the obtained findings.

#### *4.1. Student Choice Between Live and Video Classes*

Among the three main categories, the impact of COVID-19 should be recognized as the first element that influenced student choice between live and video classes; they would not have to attend online courses nor decide which class style to join were it not for the pandemic [3–6]. In relation to COVID-19, the following factors can be considered to have had an impact on the students' choice of online class participation: the reaction to COVID-19 and the adaptation to online learning.

#### 4.1.1. Reaction to COVID-19

The COVID-19 pandemic brought drastic changes and unpredictability to students' social life and daily routines, which intensified their anxiety [3–6,9]. The Japanese government declared the first state of emergency due to the pandemic in April 2020, when a new school year had just begun. Although elementary to high schools resumed face-to-face classes after a few months, most of the universities, particularly those in urban areas,

continued online learning throughout the school year. According to a nationwide student survey by the National Federation of University Co-operative Associations [67], 46.2% of students in Tokyo did not attend school during a given week between October and November 2020 when the survey was conducted; the percentage of the first-year students who enjoyed college life was 56.5%, which was the worst since 1983; the percentage of the first-year students who were worried about making friends increased by 20 points from the previous year.

This survey indicates that the first-year students suffered the most from COVID-19, as it is difficult for them to cultivate a sense of belonging when they are not allowed physical access to the university [3,5]. Moreover, as the pandemic expanded, the asymptomatically infected college-aged active youth were often blamed for passing on the virus unknowingly. Hence, many of the students refrained from going out and working part-time and stayed home instead, which made them long for real university experiences and connections with people. Such sentiments were expressed in their preferences for face-to-face classes, which is consistent with existing research [2,3,6,8,10], and their rating of synchronous online classes as the second-best choice. In Howland and Moore's research [43], students wanted to have a "face-to-face component" in virtual classes and introducing such a component helped ease their "feelings of isolation." The class interaction in this study also partially satisfied the students' desire for bonding. It created a sense of solidarity among them [7], and live access enhanced their feeling of taking classes together with other students, even if the students had connected from remote locations. In addition, the students felt an attachment to the class when their comments were responded to by the instructor, which reassured them that they mattered in this class as Besser et al. [3] point out. Their sense of mattering influenced the students' positive reaction. Combined with the occasional informal chat of the instructor, these factors created a friendly atmosphere when non-verbal communication was limited online [52].

#### 4.1.2. Adaptation to Online Learning

Student perceptions of online learning depended on the extent of instructor and students' adaptation to the new learning style [3–5,11]. Prior experience of online learning made it easier for them to adapt during the pandemic [2]. While instructors who had difficulties in using online technologies conducted lectures intended for knowledge transfer, those with online communication skills tended to introduce interactive components in classes. The former style was seen as one-directional and monotonous by students.

Most of the participants in this study were first-year students who took, on an average, 15–20 classes each semester. The classes of their very first semester in the university were conducted online as synchronous lectures or asynchronous video classes. At first, they resigned themselves to this unexpected learning style out of fear of the pandemic [5]. However, as the same class method continued in the second semester, their discontent with this one-directional instruction grew greatly. Their frustration was expressed as a lack of motivation [2,3,5,11] during online lectures and a request for opportunities to chat.

From the students' perspectives, their adaptation to online learning when they lack previous experiences of online learning is attributed to their communication environment and personal preferences. No matter how much they wish to join live classes, unstable connections, lack of appropriate devices, and device malfunctions prevent them from accessing the classes. Students who have negative perceptions of online learning are demotivated even in an ideal surrounding for online classes. Their negative attitude may exert adverse effects on their online experiences, which in turn may lead to poor academic achievement [2].

#### *4.2. Student Participation in Live Discussions*

A multidisciplinary literature review conducted by Rocca [52] concluded that it is essential for participation to be evaluated for grades or extra credit to increase the students' motivation to participate. Although Lee and Martin's research [58] was conducted in

asynchronous discussion classes with graduate students, their research results supported that of Rocca's [52], indicating that the participants' main motivation was extrinsic. In this study, student participation in real-time group discussions was voluntary and was not counted as an extra point, in contrast to the previous year's face-to-face setting, where group discussions were part of the class and evaluated for grades. Nevertheless, a quarter to a third of the students joined each discussion session. Those who remained for the discussion were not necessarily outspoken but placed a high value on exchanging opinions and wanted to receive direct feedback from their peers. As a result, cooperation among the students was promoted, creating "positive interdependence" [29,32]; this was because understanding others' views was considered necessary in the discussion, particularly when each member represented a different stakeholder. The need for positive interdependence prompted the students to listen attentively, respect others' views, and offer their thoughts in the group, and emerged as an important participation motivator [25–27]. In addition, as seen in 4.1.1., group discussions enhanced students' feeling of belonging to a learning community and promoted their participation [7,62].

The next section discusses what lies behind their motivation. The findings of the content analysis indicate their eagerness for direct interaction and confidence in personal growth.

#### 4.2.1. Eagerness for Direct Interaction

The content analysis confirmed that the students who willingly joined the discussions looked forward to opportunities for real communication. Interaction motivates student learning [2,7,10,47]. Responding to questions, reading others' comments, and listening to the instructor's feedback were the interactive elements of the class; however, all these elements were mediated by the instructor. As a student wrote, it was "a novel experience that live exchange develops discussion as members' add new perspectives to the comment expressed." Immediate communication let them feel the presence of others, which motivated them and encouraged their participation [50]. Students' welcoming reaction to interaction as observed in this study contrasts with the results by Abou-Khalil et al. [1], where students regarded online class interactions during the pandemic as "least effective strategies" (p. 14). The participants in both studies had experienced traditional one-way style instruction; however, the instructor in this study attempted to change the classroom culture by encouraging students' remarks and cultivating a friendly atmosphere. Such intentions might have positively influenced these students' perceptions about the interactions.

The students' eagerness compensated for the disadvantages of online communication, such as technical problems [7,10,44,47] and lack of non-verbal communication [8,48,52,53]. For instance, the students continued the discussion even when they did not see each other, as also observed in Tonsmann's research [59]. Considering that a video uses more data and may cause sound delay, participation with the video turned on may not be necessary for those who are motivated to participate. However, the number of participants is particularly crucial for an audio-based discussion because it is harder to identify the invisible speaker among a greater number of participants, and more participants place a burden on transmission speed; therefore, the optimum number of participants for audio discussions is three or four, slightly fewer than those of the face-to-face small group discussion in Fay et al.'s study [26].

#### 4.2.2. Confidence in Personal Growth

Many students wrote that they learned from others in interactive classes. After joining the live discussions, they felt that exchanging opinions would expose them to new ideas that they probably could not develop on their own, thereby broadening their viewpoints. The students' realization of changes in their ways of thinking and acting was linked with personal growth. Students' confidence increased their self-efficacy and helped them to have a positive view of online learning [2].Those who joined the live discussions also

tended to express their determination to overcome their weaknesses, which emerged as an expectation for personal growth.

Although this motivational factor is not peculiar to online discussions, student beliefs that discussions develop communication skills and make them more mature by broadening their perspectives promoted cooperation during discussions and created a welcoming atmosphere among the students. Voiced agreement, which encouraged students' speech and boosted their confidence, was consciously used in these discussions, in place of the nodding that typically accompanies face-to-face discussions. It may be necessary that instructors advise students about the appropriate behavior in advance [30], however, the students reacted sympathetically to facilitate interaction by themselves. The students' successful experiences further motivated them to participate, similar to what has been observed in face-to-face classes [31].

#### *4.3. Students' Lack of Video Participation*

Many students refer to the enjoyment of interacting with others, where interactive components impact their positive perceptions of, and participation in live classes [43–47]; likewise the present findings also emphasized the importance of interaction for online cooperative learning. All the students in this study joined with their video turned off, which was more than the number of students turning off their video in Castelli and Sarvary's observational study [58]. Although this study was not able to explore in detail the reasons for this, some students expressed self-consciousness about their appearance, such as "I hate to see my face on screen," and the participants seemed to be sharing a "social norm" [58] of keeping their videos turned off and audios muted in a plenary session. Considering that Google Meet, the university's online interaction tool, introduced a function to blur users' backgrounds in September 2020, it seemed that the students' concern about their appearance and shared social norms more strongly influenced their decision to turn off the video than did the presence of other family members in the background as indicated in previous research [58].

Since the students participated in this class with their videos turned off, the lack of non-verbal communication was equally shared by all. Thus, the students' feelings of isolation were mainly triggered by technical problems, an unstable Internet connection, and their devices malfunctioning; those who were able to join the interactions deepened their cohesion with the class [7,51]. Distraction due to invisibility during the class as shown in previous research [44,45,55] appeared as a demotivator in this study. In contrast, when the students enjoyed their interaction by answering quizzes and writing in the chat, they were able to maintain their focus in the class despite their invisibility, as reported in Muthuprasad et al. [10].

#### *4.4. Pedagogical Implications*

The mutuality that comes from the interactive nature of the class influenced student decisions to participate in live classes during the COVID-19 pandemic. Classroom interaction has a positive impact on reducing alienation arising from distant participation and invisibility. While interaction creates closeness similar to that established in face-to-face classes and enhances the advantages of cooperative learning, real-time participation requires a stable communication environment. For this reason, technical disturbances may increase student isolation in online learning. Thus, providing alternatives for participation and securing other interactive windows between the instructor and students are necessary [9,10,42] to avoid compromising their sense of participation. It may help to have the students facing difficulties in live access take the same quiz as those taken by synchronous participants, have access to the other students' comments using online class bulletin boards, and respond to others' comments and questions by email, in an asynchronous manner.

Instructors' efforts to "create a supportive climate" [52] are necessary in an online environment [53]; students will respond to them, affecting the class mood, whether it is conducted synchronously or asynchronously. In addition, reading others' comments on enjoyable and successful experiences during a discussion may encourage students to join the discussion on the next occasion.

In an asynchronous class, interaction and presence are different; students can communicate using the class chat and online posts even when they are not in the class [51]. Synchronous interaction accompanies presence, which enhances the students' "sense of community" [7]. When the students miss opportunities to connect with society, synchronous interaction makes them feel the presence of others and confirms that they are a part of a learning community [3]. However, their solidarity remained ad hoc at the class level. Online interactions, even in the case of a small group discussion among digital native students, did not necessarily help them develop mutual relationships beyond the class, as also observed by other researchers studying synchronous class activities [49,57]. The lack of non-verbal cues [44,48,52,55,56] might have constrained them from seeking friendship even in informal groups. Some of the online courses should be conducted in a "cooperative base group" with the same members; this can "provide students with long-term, committed relationships" ([28] p. 17) if online learning continues, since many first-year students express concerns about making friends at university [67]. In addition, blended learning that combine online and offline learning should be flexibly introduced to secure opportunities for real exchange among students [2,8].

Providing extra participation points may also motivate some students [52,60]. However, considering students' different communication environments [54] and instructors' difficulties in monitoring all the online breakout sessions, it seems fair to keep participation optional as long as there is another opportunity for those who do not participate to express and share their thoughts. In particular, when students engage in an audio-based discussion, those who remain quiet or do not respond, either intentionally or due to technical disturbances, may discourage their more motivated peers from participating.

#### *4.5. Limitations and Future Research*

This study was conducted in the unusual circumstance of most university classes in Japan being conducted online for the first time. Neither participants nor the instructor had ever experienced an online classroom in this university before the pandemic. Since the first-year students had not experienced any face-to-face university classes, they had no typical college classes to compare the online classroom to. Their anxieties and concerns over university education and the society in general due to the influence of COVID-19 were unprecedented. However, such sentiments were less directly expressed in this study than in the previous research [2,3,5,9] because it was beyond the scope of this study to explore students' feelings during the pandemic. The results indicate that acute-stage snapshots under stressful conditions had emerged from the lack of online class interaction in the previous semester. As online learning becomes the new normal in higher education due to the continuance of the COVID-19 pandemic, student responses and preferences for live classes may change; this should be the focus of future research.

Although this study addressed student interactions, the quality of the interactions was excluded from the analysis. Even when students participate enthusiastically and cooperatively and obtain good grades in online learning classes, their exchanges may remain superficial [61]; for instance, one of the students in this study confessed that she was not able to deepen her thoughts beyond the material provided for discussion. Synchronous discussions may fail to provide enough time for consideration and be challenging to some students [50]. Such frustration is observed in face-to-face discussions as well [31]; however, in comparison to regular classes, breakout room activities make it more difficult for instructors to render advice in a timely manner. An evaluation of the students' performance by their preferred mode of participation on a writing task related to the course topics may inform us of the qualitative impact on their performance; however, this goal was not pursued in this study in consideration of objectivity as the instructor and the researcher were the same.

It should be noted this research did not focus on the differences in student perceptions by their preferred mode of participation. There were five regular asynchronous participants, comprising of just about 10% of the sample, and, since they provided fewer voluntary comments than the synchronous participants, there were insufficient data for analysis. Large-scale research and different research designs are necessary to explore student perceptions by their preferred mode of participation further. Considering that this research was conducted in a women's university, balanced gender perspectives are necessary for generalization of the results. In addition, other methods of analysis should be employed to discover relationships among the codes found in this study, for the content analysis aimed at gaining "a condensed and broad description of the phenomenon" ([66] p. 108), and this study attempted to explore student perceptions of online cooperative learning.

#### **5. Conclusions**

A group discussion provides an effective cooperative learning opportunity for students to learn while helping each other. However, the COVID-19 pandemic changed classroom interaction in universities that shifted to full-time online learning to avoid physical contact between the students and faculty members in close and crowded spaces.

The pandemic also brought about a sudden change in students' relations with society, restricting their chances to connect with others. In particular, first-year students face difficulties in developing personal relationships on an online campus. Under such circumstances, it is critical to have interactive components in class to address student isolation. Thus, the course employed in this study encouraged classroom interaction using online tools and group discussions. Over half of the students mainly chose synchronous participation, and a quarter to third of the students joined the live online discussions. Through content analysis of the voluntary comments provided by the students, this study explored what influenced these students' decisions on online class participation and what motivated them to join group discussions. The abstraction and interpretation processes identified three factors related to students' decisions and motivations: mutuality resulting from interaction, the impact of COVID-19 on their lives and learning, and individual circumstances, including issues that were both personal and beyond their control.

This study indicates that synchronous interaction encouraged students to participate in online classes and confirms the results from the literature showing that interaction enhances motivation for learning in the situation emerging during the pandemic [7,8,10]. It is particularly important that students feel that they are essential parts of the learning community [3,7]. Thus, instructors need to be flexible and pay extra attention to students' learning environments and be flexible; this will help alleviate students' anxieties about online learning and isolation from society. Such strategies will expand instructors' options for classroom interactions, which can be updated with the latest online technologies.

Although this study was conducted on a small scale and in the broader context of the COVID-19 pandemic, the results revealed that the students who are motivated to learn from others can overcome the limitations of online communication through cooperation. The students, while acknowledging the disadvantages of online interaction, cooperated to compensate for the lack of non-verbal communication by intentionally voicing their agreement and adding constructive remarks in discussions. Thus, involving students in class interaction is vital to securing and improving the quality of education in a critical situation and moving toward a new normal of online learning in higher education.

Considering the background where active learning has been introduced broadly in universities in Japan, it is essential to change the students' view of university classes for the successful implementation of active learning. As Matsushita [24] warns, forcing students to join activities may deprive them of opportunities to make decisions regarding participation. The students in this study had to decide which style of class to join and then whether to join a discussion or not; they were responsible for their choices and learning that accompanies these decisions. Their enjoyment of the interaction influenced their decision about class participation. They were encouraged to express themselves and

to listen to others attentively in class, which created a favorable climate for cooperative learning. Students' positive interdependence, seen in their reflections, suggests that cooperative learning eventually cultivates a classroom culture where students are willing to contribute without the fear of losing face. This study indicated that participation, cooperation, and active engagement create a positive feedback loop, promoting each aspect even in an online setting.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted according to the Komawaza Women's University code of ethics for research involving human subject based on the guidelines of the Declaration of Helsinki, and the protocol was approved by the University's Ethics Committee (Approval number 2020-006 on 9 October 2020).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy concerns.

**Conflicts of Interest:** The author declares no conflict of interest.

#### **References**


## *Article* **Teacher Implementation of Cooperative Learning in Indonesia: A Multiple Case Study**

**Sari Karmina 1,2,\* , Ben Dyson <sup>3</sup> , Penelope Winifred St John Watson <sup>2</sup> and Rod Philpot <sup>2</sup>**

	- <sup>2</sup> Faculty of Education and Social Work, The University of Auckland, Auckland 1023, New Zealand;
		- p.watson@auckland.ac.nz (P.W.S.J.W.); r.philpot@auckland.ac.nz (R.P.)
	- <sup>3</sup> Department of Kinesiology, The University of North Carolina at Greensboro, Greensboro, NC 27412, USA; bpdyson@uncg.edu
	- **\*** Correspondence: sari.karmina.fs@um.ac.id

**Abstract:** The current study, a part of a bigger study, explored how teachers in Indonesia implemented cooperative learning (CL) in their distinct contexts. This multiple case study design used a qualitative interpretative approach. Following semi-structured interviews with 18 teachers who had attended CL professional development and used CL in their classrooms, four were purposefully selected as cases of teachers. Classroom observations, post classroom observation interviews, and field notes were employed to investigate the implementation of CL. The data were analysed using Miles, Huberman and Saldaña's framework. Five themes were generated from cross-case analysis: *implementation of CL principles, a lack of CL structures, a greater need of group orientation, group composition, and informal CL*. The findings indicate that when an innovative pedagogical practice such as CL is implemented in a culture different from the one in which it was constructed, unique contexts both enable and disrupt the successful implementation of CL. This study proposes that further research attention must be given to understanding the challenges faced by teachers shifting away from direct teaching to student-centred pedagogies such as CL.

**Keywords:** cooperative learning; pedagogy in Indonesia; teacher voice; qualitative research

#### **1. Introduction**

*1.1. Cooperative Learning and Culture Alignment*

Among constructivist educational premises and practices, CL, a learner-centred approach, which was developed in the United States of America in the 1970s, has been espoused as one of the most widespread and successful pedagogies [1,2]. Hundreds of evidence-based research studies have demonstrated that CL can provide students with opportunities to learn, work independently and in groups, and take greater responsibility for their learning and improve student outcomes [1,3–5]. In addition to improved learning, previous studies of CL have also shown that CL provides many social advantages for students, such as positive relationships between students and social support [6,7].

CL has enthused Indonesian teachers and researchers since it was introduced in the early 2000s [8]. The notion of learning together, teaching, and sharing with one another is attractive and promising, not only because of the potential for higher achievement proven in prior research [1,3–7], but also because of its potential for cultural aptness in the Indonesian contexts [9]. Indonesia's core values gotong-royong (mutual assistance) and musyawarah (consensus decision-making) are aligned with CL principles [9,10] – positive interdependence, individual accountability, promotive interaction, appropriate use of social skills, and group processing [1]. The concept of gotong royong guarantees that each individual of the community shares equal loads and responsibilities in order to achieve common social goals and the concept of musyawarah involves the process of

**Citation:** Karmina, S.; Dyson, B.; Watson, P.W.S.J.; Philpot, R. Teacher Implementation of Cooperative Learning in Indonesia: A Multiple Case Study. *Educ. Sci.* **2021**, *11*, 218. https://doi.org/10.3390/ educsci11050218

Academic Editor: Robyn M. Gillies

Received: 31 March 2021 Accepted: 29 April 2021 Published: 5 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

doing everything together in order to reach general agreement or common consent of all community members.

#### *1.2. Indonesian Teachers' Perception of Student-Centred Learning*

The implementation of CL in the Indonesian context, however, appears to clash with the prevalent teachers' beliefs of direct learning and teachers' perceptions of studentcentred learning. Seputro [11] described how a male Indonesian teacher in his/her study believed that students ought to be active and independent in their learning and that the teacher participant's role in teaching was as facilitator. The teacher's beliefs, however, were not aligned with his practice; that is, his actions were more consistent with direct instruction [11]. Seputro [11] reported that the teacher's beliefs were influenced by conservative Indonesian values where teachers are perceived as people who are to be listened to, modelled, and followed. Fourteen years after Seputro's study, Bjork [12] reported similar results. Bjork found that the 100 junior secondary school teachers participating in his study answered "often" and "always" when asked whether they used student-centred teaching methods; however, his observation revealed that 53 percent of the respondents used lecturing, 20 percent involved hands-on activities and only 5 percent included a class discussion. Bjork [12] reported that classes remained traditional dominated by rote learning.

These results show that CL fundamentally challenges Indonesian teachers' beliefs about teacher' role and position, and teachers' comments and views about student-centred teaching. The idea of learners learning together, teaching and sharing, is appealing, as CL practises the values of Indonesian cultures. The prevalent teaching approach, however, has been direct instruction which might impede the implementation of CL. It is, thus, important to explore how Indonesian teachers implement CL in their contexts. Moreover, little research has been reported on the implementation of CL in the Indonesian contexts. Most CL research in Indonesia has focused on the effectiveness of CL structures [13–18], rather than the effectiveness of teachers' incorporation of CL into their pedagogy. There is a paucity of qualitative research to more deeply interrogate the unique Indonesian teaching context. To build on this research gap, the purpose of this study was to investigate teacher implementation of CL in Indonesia using a multiple-case study.

#### **2. Literature Review**

#### *2.1. Teachers' Implementation of Cooperative Learning*

CL research has identified several factors that influence teachers use of CL such as students [19], and CL professional development [19,20]. Siegel [21], employing ethnography inquiry, conducted a qualitative study to explore variations in five teacher participants' implementations of CL. Siegel [21] reported that the teachers' implementation of CL was influenced by CL professional development and classroom experiences; lesson planning; and teaching contexts, which included lesson objectives, perceptions about student ability, task difficulty, curricular constraints, and collegial support. Likewise, Antil et al. [22] found that teachers' implementation of CL was influenced by training experiences. Dyson [23] and Dyson et al. [19] revealed that the use of CL involved changes in teaching role, lesson planning, and use of instructional time.

In a study that collected data through interviews and observations in Australia, 10 teachers who were trained in a 2-day CL workshop were asked to implement CL with some specific objectives such as the establishment of task interdependence and individual accountability, small-group orientation, and complex task construction [20]. The results show that the teachers had positive experiences using CL, but factors that impeded their implementation of CL were identified [20]. These factors were students' off-task behaviour during group work; time management and required preparation for CL; group formation; task construction; social-skill orientation for the students, especially to manage conflict; and assessing students in small groups [20].

In a recent study conducted in Aotearoa, New Zealand, Dyson et al. [19] aimed to investigate the implementation of CL in physical education classes by 12 generalist primary school teachers-who were responsible for teaching eight learning areas such as literacy, mathematics, physical education and science. Drawing from multiple data sources, Dyson et al. [19] highlighted several findings. The teachers in Dyson et al.'s study indicated that social skills were important features for students to possess in group work. Furthermore, Dyson et al. reported that due to the complexity of CL structures and their procedures, the teachers had difficulty choosing structures that were suitable for their physical education teaching. Finally, the teachers indicated that using CL gave more opportunities for their students to take responsibility for their own learning [19].

#### *2.2. Teachers' Implementation of Cooperative Learning in Asia*

CL studies conducted in East and Southeast Asia reported that there were potential cultural mismatches of Eastern cultures with CL principles. Thanh [24] conducted a research review of 17 studies, from 1990 to 2007, by investigating the effect of CL on academic performance in Confucian-heritage culture (CHC) countries such as China, Japan, Singapore, Vietnam, and Korea. Thanh revealed that CL did not improve students' achievement in approximately 50 percent of the studies. Thanh concluded that one of the main failures of the implementation of CL in CHC classrooms was the disjunction between CL and CHC cultural values [24–26]. In CHC countries, teachers are perceived as the source of knowledge, which seems to contradict the student-centred constructivist perspective underlying CL.

In a review of studies of the application of CL in Malaysia from 1996 to 2003, Zakaria and Ikhsan [27] reported additional challenges. Teachers, for example, had reservations as to whether their students could acquire knowledge by only learning from their peers and suggested that students might lack necessary skills to work in groups. A later study conducted by Arumugam et al. [28] reported that cultural norms such as budi bahasa (language of character) and gotong royong held by Malay students positively influenced their cooperative behaviours in CL groups when compared to Chinese-descendent students' cooperative behaviours. Arumugam et al. concluded that Chinese students, whose cultural roots were in Confucianism, which believes that teachers are authority figures who should be obeyed and respected, preferred to work individually on their projects rather than work in groups [28]. Arumugam et al.'s [28] study suggests that different cultures might support or impede the implementation of CL.

#### **3. Materials and Methods**

The current study employed a multiple case study design using interpretative qualitative approach to investigate the implementation of CL. We regard interpretivism as the most appropriate approach for the current study because we attempted to understand and interpret the teachers' conceptions of CL. These conceptions support the construction of a more complete picture of the phenomena as a whole [29]. In order to develop a deeper understanding of the implementation of CL, we employed a case study design, using the qualitative tradition of inquiry. A case study design has been described as offering more profound information and different insights into a phenomenon [30].

When a case study includes more than one single case, a multiple case study is required. A multiple case study design was chosen because understanding individual cases with their similarities and differences would "lead to better understanding, and perhaps better theorizing" [31] (p. 446). Moreover, Merriam [32] stated that "the more cases included in a study, and the greater the variations across the cases, the more compelling an interpretation is likely to be" (p. 40). The selection of multiple cases allowed the researchers to show the differences between commonalities among the cases' implementation of CL.

This paper reports a part of a bigger study that involved 18 teachers who had attended a CL professional development and had implemented CL for at least a year. Four teacher cases of 18 teachers were selected. The four teacher cases were selected as they were enthusiastic about practising CL, they used CL in almost every session, and were willing to undertake professional development throughout the research project. The participants

taught different subjects in different secondary schools as discussed more fully, below. The pseudonyms Jati, Budi, Nawang, and Krisentia were chosen by the teachers.

#### *3.1. Participants*

3.1.1. Jati

Jati was a teacher with 20 years' experience in secondary schools. Jati had attended several workshops in CL. One of the workshops provided him with some field practice and supervision. Jati also learned CL from MGMP (subject-teacher discussion forum) for Indonesian language teachers, in which he had held the position of the head of regional MGMP. The forum discussed the development of Indonesian language, current curriculum, lesson plans, and issues regarding the teaching and learning of Indonesian language. As well, the forum shared teaching approaches such as CL and teaching techniques.

#### 3.1.2. Budi

A mathematics teacher since 1983, Budi had learned about CL through professional development training running by the MoEC (Ministry of Education and Culture) in 2007. He had learned several CL structures such as Jigsaw [33] and STAD [34]. In 2011, he attended an inclusive 3-year programme run by an independent training institution in which CL was workshopped for 3 days. In the workshop, he received mentoring from a university lecturer that focused on the implementation of CL in the classroom.

#### 3.1.3. Nawang

Nawang had been an Indonesian language teacher for 23 years and completed her master's degree in education in 2012. Nawang learned CL through professional reading and through research. In her master's degree thesis, she compared the effectiveness of Think-Pair-Share [35] with the Group Investigation [36], in writing news. Nawang also studied CL through workshops conducted by an independent training institution and MGMP. With the independent training institution, she completed a 3-day workshop. In the first day of the workshop, she learned some CL structures. On the second day, she and her group made a lesson plan using CL structures, mentored by a university lecturer. On the third day, the mentor supervised them as they implemented the lesson plan in a real class.

#### 3.1.4. Krisentia

Krisentia, a social science teacher for 36 years, had been familiar with CL since 2006 when she joined professional development run by the MoEC. She has subsequently learned more about CL through books and workshops conducted by an independent training institution. She used several CL structures in her teaching such as Jigsaw and Inner and Outer Circle [35].

#### *3.2. Data Collection*

The current study employed five different data collection methods to investigate the implementation of CL in Indonesia. Semi-structured interviews, classroom observations, post-observation interviews and field notes were used as primary data and documents such as teachers' lesson plan and students' assessment as secondary data. The semi-structured interviews were conducted prior to the classroom observations in order to explore the teachers' knowledge of CL, experiences of learning CL, and the teachers' implementation of CL in their classrooms. The classroom observations that explored how the teachers implement CL and applied CL principles, CL structures, group composition, group interaction, and CL task were conducted over 12 weeks. Each case was observed four times during the study. All observations were video recorded. The video recording provided details of the practice of CL structures, the use of principles of CL, group composition, and the CL tasks.

The field notes, taken in a systemic manner with a focus on the researcher questions [37], were used as primary data to triangulate findings from classroom observations

and post-observation interviews. Moreover, the field notes were used to reflect the field researcher's observation and interpretations during classroom observations, post-observation interviews, and field observations such as school environment and school activities. The documents such as teachers' lesson plans, lesson materials, teachers' assessment sheets, students' peer-assessment sheets were analysed as secondary data. The teachers graciously provided these documents.

Ethics approval for the current study was received from the University of Auckland Human Participants Ethics Committee (Ref. 017950) on 4 October 2016. Signed consent was given by the schools and the study participants.

#### *3.3. Data Analysis*

The current study employed inductive analysis, using Miles, Huberman and Saldaña's [38] two-cycle framework to construct themes from the data. First-cycle coding includes techniques to identify, evaluate, and re-evaluate codes that emerge in the initial coding process [39]. Second-cycle coding provides the researchers with the opportunity to begin grouping first-cycle codes into categories and themes [38].

In the first cycle coding, initial coding and in vivo coding were used to code data from semi-structured interviews and post-observation interviews; process coding was used to code classroom observations; and descriptive coding was employed for field notes (see [39]). In the second cycle coding, pattern coding was used to condense the data into clusters that reveal categories. Categories from each case semi-structured interview data were analysed to look for larger categories. Larger categories were then scrutinised to find themes that reflected the teacher's perspectives, conceptions, and experiences of CL. Similar categories from classroom observations, post-observation interviews, and field notes of each case were analysed and clustered to create a larger category. Then, the larger categories from semi-structured interview data were compared and contrasted with larger categories from the other three data to generate themes to answer the research question; that is, how CL is implemented in Indonesian contexts. Five overarching themes were generated: implementation of CL principles, a lack of CL structures, a greater need for group orientation, group composition and informal cooperative learning.

#### **4. Findings and Discussion**

#### *4.1. Implementation of Cooperative Learning Principles*

Johnson and Johnson [40] emphasised the importance of understanding the five principles of CL for teachers to implement CL effectively in the classrooms, however, studies have revealed that often only some of the five principles are implemented in schoolbased settings [22]. The evidence in this current study showed that all four of the teachers had difficulties defining the principles underlying CL as described in the current literature.

"In groups, members should interact well so that they can produce something good". (Jati, Semi-structured Interview)

"In principle, a cooperative learning group should work together". (Budi, Semistructured Interview)

"In principle, cooperative learning is about cooperation and knowledge construction". (Nawang, Semi-structured Interview)

"It is a cooperation. I think they [the students] should not work individually but teamwork. That's all I know". (Krisentia, Semi-structured Interview)

However, through classroom observations, it was evident that each of the teacherparticipants established at least four principles of CL, positive interdependence, individual accountability, face-to-face promotive interaction, and interpersonal and small-group skills, through group roles, peer assessment, seating arrangement and group supervision.

Positive interdependence was practised when the teachers assigned the group members roles. The roles varied from readers, writers, and presenters in one lesson to characters of a story in which the groups performed a drama with different characters. Thus, each

of the group members was responsible for undertaking one aspect of the assignment. Johnson and Johnson [1] recommended that to maintain the positive interdependence in group is through task interdependence. The tasks require each member to have a role so that the group could finish the task in an allocated time. In one of Krisentia's classroom observations, for example, task interdependence was clearly demonstrated.

In a group of six, two members searched for answers to the assigned questions, two members wrote the answers on colourful sticky notes while the other two members were responsible for writing the results of the discussion on the project sheet. At the end of the task, two members, as representative of the group, were to present the completed project. (Krisentia, Classroom Observation 3)

Without each member's effort and responsibility to complete the task, the group would not be able to present the group project at the end of the session.

When every member of the group is responsible for his or her own task in order to contribute to the group's goal and ensure that every member completes their task, individual accountability increases [1,41]. In the current study, individual accountability was observed through the division of roles which resulted in an increase in responsibility for each group member. Gillies [41] stated that the shared responsibility that a group creates elevates the feeling of being accountable and the motivation to perform well. Individual accountability can be established through peer, group, and teacher assessment in regard to the student contribution to the group [41]. The evidence from the current study showed that peer assessment was utilised to help the students on the task, to help the group understand the task, and/or to help students improve the task. Nawang, for example, demonstrated a peer-assessment strategy that determined how the group performed certain aspects of the task and how the group performed the task overall.

Nawang showed a slide that contained information about the task. Students were to make a group of six, practise the story, and perform the story. She also gave a scoring sheet. The scoring sheet was used to evaluate other groups' performances. Nawang showed the scoring criteria in a slide. She explained the scoring criteria one by one. She also gave examples of each point such as in using intonation in telling the story. (Nawang, Classroom Observation 3)

This kind of accountability motivated each member to perform well in the task. Group assessment could be conducted through the agreement on certain tasks or roles with the group providing feedback on the task performance [1,41]. Group assessment in this current study was observed through the division of roles. Teacher assessment was conducted in every session by all teacher cases. The students were observed giving feedback regarding group processing during the group work and the group presentation.

The findings of the current study suggest that promotive interaction, the third element of CL, was demonstrated through the seating arrangement and the four teacher cases monitoring the groups' involvement. The four cases facilitated the students' interaction in groups by ensuring that they sat face to face while participating in discussion with their group mates. They also supervised and encouraged the groups to promote positive interaction among group members.

After setting up the group, Budi gave a mathematics problem to solve in the group. For the few minutes, some group members were doing the problem by themselves as Group 1, 2, and 3. Then when Budi was approaching the groups, they turned their chairs and started to discuss. After 2 min, Budi moved around checking the groups' answers or discussion. (Budi, Classroom Observation 1)

Oppositional interaction, however, as opposed to promotive interaction, occurred in some instances because of the group composition. In Jati's class, for example, when a group consisted of five boys and one girl, the girl worked independently without any interaction with other members. She worked through the task by herself, even though Jati had encouraged her to engage in the group discussion. The boys seemed to not include the girl in the group discussion.

There was one girl in Group B who hardly interacted with her group mates who were all boys. The boys did not ask her to join either. She may not have been comfortable being the only girl in the group. She did the task by writing down her opinions or arguments of the case in her book. There was an opportunity that at one point she came forward to represent the group. She wrote her opinions of the case on the board which likely did not represent her group's opinion but hers. (Jati, Classroom Observation 3)

However, in Budi's class, when a group consisted of three girls and one boy, the girls involved him in the discussion. This finding is consistent with previous research [42]. Webb [42] reported that in the majority-boy group, the boys focused their attention on the other boys and tended to ignore the girls, while in the majority-girl group, the girls gave more explanations and information to the boy than expected. Girls gave more explanations and responded more actively to requests to a boy member in the group. On the other hand, girls in the majority of male groups received less help and less responses [42].

When promotive interaction occurs, the other two principles of CL, appropriate social skills and group processing, are established [43]. The findings showed that the four cases attempted to establish social skills and group processing among the members of the groups through the use of movie clips related to cooperative values, peer mentoring, direct supervision, and group orientation. Budi introduced the cooperative norms and the importance of cooperation through videos. As observed in one of his classroom observations, he showed a short movie that taught the students the importance of working together to achieve a common goal. Krisentia encouraged the cooperative norms through peer mentoring in the classrooms, ensuring that the low achievers were on task, and, for the non-academic activities, assigning the senior students to mentor their juniors.

In the first day of school, I inform my class that I will treat everyone the same . . . I ask my students to coach their friends who need help [in learning]. The smart ones should help the low achieving in any situation. In the class [the observed class], there is Bella whom Ari helps in learning . . . Ray is coached by Eddy, and Tania is coached by Temy [all names are pseudonyms]. (Krientia, Post Observation Interviews)

Jati conducted a direct approach, working on the problematic individual or group, encouraging his students to improve social skills and checking if his students gave feedback and support to their group mates. Nawang established the two principles of CL by giving her students group orientation. She trained her students in how to cooperate in the groups, and thus her students did not encounter any conflicts or difficulties with cooperating as they worked in small groups.

Promotive interaction was present from the beginning of the lesson until the end of the lesson. All students in their groups communicated and they were on task, even the lowest achieving group. Nawang encouraged her students to work on the task. She often checked the groups and asked questions. She spent some time with the lowest achieving group. (Field researcher, Field Notes)

#### *4.2. A Lack of Cooperative Learning Strutures*

During the initial interviews, the four teachers indicated that they had used CL structures such as jigsaw, think-pair-share, and group investigation. However, only Nawang was observed using CL structure, think-pair-share, in one of her classroom observations.

Nawang asked the students to work in pairs. The students worked with their pair who was sitting in the same table. Nawang gave them a text and some comprehension questions to be discussed and analysed. She gave the pair 15 min to work. The pair started to work . . . Nawang asked the pairs to share with the whole class. (Nawang, Classroom Observation 2)

In general, the teachers asked the students to work in a small group to discuss answers to some questions from the textbooks or prepared by the teachers. At the end of the

discussions, each group or most groups, in turn, came to the front of the class to tell the class their answers to the questions. After the group presentation, other groups would ask questions, give feedback, or offer suggestions. The four cases mentioned that they did not know the names of the structures they used, or they just simply identified the structures like a group discussion. They reported that they could not remember the names of the structures because they learned many structures during CL professional development.

I did not know the name of the CL structures [used during classroom observation] because there were a lot of [CL] structures I learned. I am sure it was not Jigsaw [laughing] because there must be experts [in a group]. I didn't know exactly which type [CL] I used. From the characteristics [of the group activities] I know it [as] CL as there were cooperation and social interaction. The students learned from their friends. They gave feedback to others. Everyone had different opinions so that he or she would develop their understanding. You can see that they also gave feedback on the presentation such as the voice [of the reader] was not loud enough. (Jati, Post Observation Interview)

These findings suggest that the varieties of CL structures led to teachers' confusion in using the structures [2,19]. Dyson et al. [19] reported that the teachers in their study were confused by the variations of CL structures and in choosing the suitable structures for the physical education setting. Dyson and Rubin [44] and Goodyear and Casey [45] recommended that teachers focus on one or two CL structures at a time as this allows the teachers to move towards being more student-centred in their lessons.

The findings reveal that the teachers in the current study were uncertain as to whether they were using CL or other student-centred approaches. During the post-observation interviews, Nawang and Budi confirmed that they used problem-based learning in one of their classroom observations.

Now I'm not sure it was a Group Investigation. However, the most important was that the students were given a task in which they had to solve the problems. Please find the model for me [laughing]. I think it was problem solving [laughing] yes, yes [conforming her answer]. I think it was problem-based learning. (Nawang, Post Observation Interview)

In regard to the teachers' uncertainty about using CL or problem-based learning, Davidson and Major [46] pointed out that both of the approaches have many similarities so that teachers might find it challenging to distinguish between the two approaches. Davidson and Major [46] claimed, however, that when the two approaches were used together in sequence this would offer a powerful approach to develop students' intellectual and social skills.

The language factor might influence the teachers' forgetting the names of the CL structures. CL structures have English language names, providing a constraint for the teacher-participants as English was not their first language. An important example of teacher adaptation is when Krisentia changed two CL structure names into her first language (Javanese) and the Indonesian language because she had difficulties in pronouncing the English names of the structures. Krisentia said, "Mix Match is hard for me to pronounce so I changed it into *Gatuk Entuk*. The students then will mix, and they will find their match." Krisentia also used an adapted version of Inside-Outside Circle and named the structure *Sapi Kandang* which literally means "cows in a pen."

I divide the class into two groups, sapi [cows] and kandang [a pen]. The cows get questions . . . When the cows get a question, they say emoh [meaning no in Javanese]. It is so funny [laughing] . . . . Prior to this [the activity], I have asked my students to study the lesson thus the questions are related to the lesson. (Krisentia, Semi structured Interview)

Inside-outside circle is used to build students' social, knowledge and thinking skills [47]. In circles, students rotate and face their new partners. The students are given time to ask

and answer questions and discuss. Each time they switch partner, they can discuss something new or the same question [35].

#### *4.3. A Greater Need for Group Orientation*

Cohen and Lotan [6] recommended that teachers conduct an orientation session in which students are introduced to or reminded of the cooperative norms and roles and the central concepts of the task. These orientations are to prepare the students for the tasks and the challenges of working together. Teacher participants in studies by Dyson et al. [19] and Gillies and Boyle's [20] reported that initially students need to learn appropriate social skills to work in small groups. The findings of the current study reveal that only Nawang conducted task and group orientation. Nawang gave step-by-step procedures of the tasks and informed her students how to cooperate with their group mates.

Nawang informed the students the objectives of the lesson on the slides. The students read the objectives in unison . . . . Nawang showed a slide that contained information about the task. Students were to make a group of six, practice, and perform the story. She also gave a scoring sheet. The scoring sheet was used to evaluate other groups' performances. Nawang showed the scoring criteria on the slide . . . . Nawang showed the lesson procedures . . . (Nawang, Classroom Observation 3)

Nawang paid particular attention to the careful planning of the CL lesson in order to gain a very high level of students' involvement in groups. She said, "If we [teachers] want to implement cooperative learning, we have to be ready with the material, the instruction, the media, so that cooperative learning lessons could run smoothly. . . . students can do cooperative learning activities effectively." (Nawang, Semi-structured Interview)

In addition to the tasks and cooperative norms, Nawang gave an orientation to peer assessment, informing the groups about the criteria and the procedures of the assessment. Consequently, compared to the other three teacher cases, she had fewer problems with her students' behaviour and engagement in the group, and higher task completion. These findings support Golub and Buchs' [48] study. Golub and Buchs [48] reported that giving the students a cooperative norm orientation improved positive interactions. The students pay more attention, give more support, and ask more questions of their group mates.

#### *4.4. Group Composition*

The evidence in the current study showed that the four teacher cases acknowledged the benefits of heterogeneous groups. Previous research revealed that establishing heterogeneous groups in CL may cater to the diverse needs of the students [49] and impact students' performances [50]. However, the teacher cases in the current study had difficulties in forming heterogeneous groups. Forming heterogeneous groups required time in deciding the level of students' proficiency and students' needs because the groups consisted of highachieving and low-achieving students, different genders, and different needs depending on the students and the context. Jati reported, "most of the students ask for the same-gender groups. The boys group themselves with boys, so do the girls. It is hard to mix them." (Jati, Post Observation Interview). Having heterogeneous groups also meant that the teachers had to spend extra time on the seating arrangements. While managing time was important for the teachers because they struggled with finishing the lesson materials and preparing their students for tests, the teachers allowed friendship choice to dominate, which meant that teachers let the students choose their own group. Krisentia reported that forming a group based on a seating arrangement was fast and effective.

It [seating arrangement group formation] is fast. The students don't have to move, they just turn their chair. Communication is easier if they are close friends [close friends tended to sit in the same table]. Thus, it is faster [than heterogenous group formation]. (Krisentia, Post Observation Interview)

The teachers believed that friendship groups yielded more positive social interaction among group members which could increase active involvement in the tasks. Budi said," . . . I would just free the students to choose their groups. But the good side is they [the students] will feel happy when they work with their close friends and they can work better." This finding is supported by Gillies and Boyle's [20] study which revealed that friendship-based groups provide students with higher motivation to achieve group goals. In addition, the current study revealed that friendship groups did not take much time to arrange because the students tend to sit with their close friends, thus saving teaching time. The teachers' preference for friendship group composition was consistent with previous research findings. Kutnick, Blatchford, and Baines [51] found that the teachers in their study legitimised friendship groups chosen by their students, and friendship groups impacted on students' learning outcomes. Echoing Kutnick et al.'s [51] findings, Phuong-Mai, Elliott, Terlouw, and Pilot [25] and Thanh and Gillies [26] reported that friendship groups were more culturally responsive for Vietnamese students who had a strong sense of intimacy, group solidarity, and mutual support. South-East Asian countries such as Vietnam, Indonesia, and Malaysia are identified as having collectivist cultures in which cooperation is based on trust and identity [52]. Trust and identity are closely related because the condition for trust is identification with the group [53]. Brodt and Korsgaard [53] argued that trust is determined by the extent to which individuals define themselves in terms of particular group membership. Thus, in a collectivist society such as Indonesia, the new group should be based on trust, supporting the identity of each member. In the context of education in Indonesia, a group that supports personal relationships, such as friendship groups has a more powerful impact in determining the nature of group cooperation, which has the potential to engender more effective group processing. It is also worth noting that it is possible that friendship groups consist of close peers with mixed abilities, mixed interests and mixed genders.

#### *4.5. Informal Cooperative Learning*

In regard to Johnson and Johnson's [40] CL types, the teacher cases practised "informal cooperative learning" (p. 29), that is, informal CL groups as temporary groups that lasted from a short period to one class session. Informal CL aims to ensure active cognitive processing during a lesson. In the current study, the classroom observations revealed that the teacher cases would normally start with a review of the previous lesson, an introduction to the new lesson, and a short lecture. The teachers then grouped their students into anything from dyads, the smallest group, to octets, the biggest number in a group, and gave the groups a task. The groups lasted for a single activity or for the whole-class session. The tasks ranged from (a) a discussion on certain topics or questions prepared by the teachers, students, or the textbooks, to (b) a presentation on certain topics or the comprehension of texts or lessons, or to (c) a performance. Prior to doing the tasks, the teachers gave instructions about the tasks. At the end of the tasks, the teachers asked the groups to present the results of the discussion either on a piece of paper or orally. After the presentation of the task, the teachers encouraged the other groups to give reactions to the information presented. The teacher instructional procedures in using small-group learning were consistent with Johnson and Johnson's [40] informal CL.

The teachers in the current study practised informal CL because, firstly, the nature of group composition that the participants subscribed to was mostly friendship groups. This condition made formal cooperative learning and cooperative base groups difficult to implement because both forms require heterogeneous groups to work effectively (see [22,40]). Moreover, the current study revealed that the groups always changed in every group activity or every session which indicated that the teachers implemented informal cooperative learning. Secondly, the teachers did not use formal CL or cooperative-based groups because both forms required the teachers to provide more time to group the students in heterogeneous groups, design a task that promotes the principles of CL, evaluate each principles of CL, and assess the group learning. The teachers in the current study were reluctant to

take the time to establish formal CL group as they felt that it reduced the time available for material coverage and test preparation. Thirdly, the teachers received insufficient information about formal CL and cooperative base groups from their professional learning and development. The data from the professional development documents revealed that formal CL and cooperative base groups were not introduced. The CL professional development that the teachers attended focused on CL structures instead of CL principles.

#### **5. Limitations**

It is clear from the findings of the current study and previous research that the success of CL implementation requires the establishment of CL principles. Measuring the degree to which CL principles were incorporated was beyond the scope of this study. Knowing the nature of the interactions among group members would further help researchers analyse the degree of positive interdependence, individual accountability, promotive interactions, the interpersonal and small group skills, and group processing. Further, researchers could use CL validation tool developed by Casey, Goodyear, and Dyson [54] to examine the degree of CL principles.

#### **6. Conclusions**

The current study aimed to investigate the implementation of CL in the context of Indonesia. The evidence shows that the teachers in the current study demonstrated a desire to use a more student-centred teaching approach. The teachers subscribed to CL because the model offered strategies to engage the students to take ownership in their own learning and gave their students social and emotional benefits.

The current study reveals a considerable need for an integrated system of CL training programmes at both in-service and preservice levels in Indonesia. The findings indicate that the teachers in this study lacked conceptual understanding and knowledge of CL, which contributed to their difficulty in the implementation of CL. Their misconception of CL was caused by several factors. First, CL in-service training was typically delivered through short professional development programmes and there was no follow up or continual development. Therefore, the teachers understood CL as a theoretical model only. Second, the teachers integrated their knowledge of CL with other similar student-centred ideas such as problem-based learning and conceptual teaching and learning during their professional development. While CL requires teachers to incorporate the five principles to work effectively, other student-centred approaches do not have the same emphasis. Third, partial knowledge of CL in conjunction with a prior understanding of student-centred teaching has led to CL that is more consistent with group work and social construction of knowledge rather than CL. Thus, providing teachers with continuous professional development on CL involving teachers employing CL with close mentoring is suggested as likely to yield a more high-quality implementation of CL.

While this research was carried out in the specific context of Indonesian schools, the wider issues that this research has uncovered could be transferable to other contexts where teachers are shifting away from direct learning and toward constructivist student-centred pedagogical practices. As such, this study offers insight into the challenges involved in changing pedagogical practices and highlights to other researchers the fine-tuning that is needed to understand pedagogical training in and implementation of CL.

**Author Contributions:** Conceptualization, S.K.; Investigation, S.K.; Writing—original draft, S.K.; Writing—review & editing, B.D., P.W.S.J.W. and R.P. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Ethical review and approval were received from the University of Auckland Human Participants Ethics Committee (Ref. 017950) on 4 October 2016.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy.

**Acknowledgments:** This current study was supported by the Directorate General of Higher Education, the Indonesian Ministry of Research and Technology and Higher Education.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Is There a Right Way? Productive Patterns of Interaction during Collaborative Problem Solving**

**Noreen M. Webb 1,\* , Marsha Ing <sup>2</sup> , Eric Burnheimer <sup>1</sup> , Nicholas C. Johnson <sup>3</sup> , Megan L. Franke <sup>1</sup> and Joy Zimmerman <sup>1</sup>**


**Abstract:** Compelling research evidence shows benefits for student learning from explaining one's ideas and engaging with the ideas of others. However, whether certain patterns of group interaction may engender this productive student participation is unknown. Using data from two third grade mathematics classrooms, and over the course of six days during a five-month span, we investigated how students interacted with each other to solve problems when the teacher was not driving the interaction. We identified multiple profiles of group interaction that yielded highly-detailed participation for some or all students in the group. These profiles varied in terms of whether students interacted in an ongoing, sustained manner or interacted periodically but not continually, whether one or multiple students initiated problem-solving strategies, and whether group members worked jointly or largely separately on their strategies. No single profile of group interaction was either necessary or sufficient to lead to highly-detailed participation for all students in the group.

**Keywords:** cooperative/collaborative learning; small group learning; peer learning; peer collaboration; classroom-based talk; student interaction; mathematics; elementary schools; dialogue; problem-solving strategies

#### **1. Introduction**

Cooperative learning researchers have made great strides in identifying student participation in peer-led groups that is linked to student learning outcomes. At the heart of many researchers' perspectives about interaction among students that is productive for learning are explaining one's own ideas and engaging in the ideas of others. Some perspectives include exploratory talk [1], accountable talk [2,3], transactive discussions or transactive dialogues [4,5]; see also [6], argumentation [7–10], collaborative reasoning [11], coordinated talk [12], co-construction [13–17], co-regulation [18], and shared regulation [19–21].

In exploratory talk, for example, students offer the relevant information they have and engage critically but constructively with each other's ideas by jointly considering, evaluating, challenging, and building upon each other's hypotheses [22]. In contrast, disputational talk, characterized by disagreements but little constructive criticism of suggestions, and cumulative talk, characterized by positive but uncritical building upon each other's suggestions, are seen as less constructive and involve less student engagement with each other's ideas.

Explaining one's own ideas and engaging with others' ideas can promote learning in multiple ways. Developing and offering ideas to others, being challenged or questioned by others, and attending to others' thinking all encourage students to rehearse information in their own minds, monitor their own thinking, reorganize and clarify material for themselves, recognize and rectify misconceptions and gaps in their understanding,

**Citation:** Webb, N.M.; Ing, M.; Burnheimer, E.; Johnson, N.C.; Franke, M.L.; Zimmerman, J. Is There a Right Way? Productive Patterns of Interaction during Collaborative Problem Solving. *Educ. Sci.* **2021**, *11*, 214. https://doi.org/10.3390/ educsci11050214

Academic Editor: Robyn M. Gillies

Received: 29 March 2021 Accepted: 30 April 2021 Published: 3 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

make connections between new information and previously learned information, reconcile conflicting viewpoints, and acquire new strategies and knowledge and develop new perspectives [23–29].

Accumulating empirical evidence supports the hypothesized benefits of such participation for learning [30–33]. Studies comparing different approaches to training students on the use of productive dialogue, have shown positive effects on explaining their ideas and critically engaging with and evaluating each other's ideas on student mathematics and science achievement and reasoning ability [7,34–38]. Similarly, studies comparing different approaches to training teachers to implement instructional moves designed to promote student explaining and engagement with each other's thinking (e.g., asking students questions to clarify their thinking and give supporting reasons, and to discuss each other's ideas and predictions) have also shown positive effects on student achievement [39–50].

Correlational research also shows positive links between explaining and engaging with others' ideas and learning outcomes. Positive relationships with mathematics and science learning outcomes have been shown for providing explanations as part of arguments or justifications, and supporting, rebutting, and building on others' suggestions [37,51–58]. Furthermore, explaining and engaging with others' ideas (e.g., restating or paraphrasing another student's strategy and applying it) have been found to correlate with mathematics achievement among students who explicitly need help [59]. Recent studies show, moreover, that the relationships between participation and learning outcomes depend on the level of detail in explaining and engagement with others' ideas. For example, giving highlydetailed, fully complete, and correct explanations was more strongly related to achievement test scores than was giving incomplete or ambiguous explanations [60,61]. Similarly, students who engaged with others' ideas at a highly-detailed level (by adding details to other students' suggestions, or proposing alternatives), showed higher achievement than students who only repeated other students' suggestions, or who agreed or disagreed with others without providing reasons [62].

Close case-study analyses of student interaction reveal more details about the benefits for the participants in these conversations. Roscoe and Chi [54], for example, describe how students trying to explain to other students the structure and functioning of the human eye signaled that they did not fully understand (e.g., "This is something that I didn't really get before", p. 336), and revisited and reviewed material and rethought their ideas in the context of trying to provide more complete explanations, and arrived at a fuller understanding as a result. Brown, Campione, Webber, and McGilly [24] (pp. 177–178) illustrate how the group's challenge of an explainer's incomplete or incorrect ideas caused the explainer to re-examine her prior knowledge, to formulate and test predictions based on her incorrect mental model, and to use information provided by her peers in response to her predictions to revise her ideas (in this case, her ideas about animals' use of camouflage as a defense mechanism). Webb et al. [57] showed how, through the process of explaining their own thinking and engaging with others' ideas, students forged new connections between mathematical ideas and representations, and extended their problem-solving strategies.

While there is general agreement about the importance of explaining one's ideas and engaging with others' ideas, much less is known about whether there are certain patterns of group interaction that are more likely than others to produce this productive student participation for all students in the group. This paper explores how interaction in small groups solving mathematical problems unfolded, and examines how multiple features of the patterns of interaction corresponded to the emergence of highly-detailed participation in the group. In particular, we focus on two features of group interaction: (1) whether all students in the group have, and use, opportunities to lead the mathematics by contributing mathematical ideas; and (2) whether there is sustained interaction among the students. This paper then delves more deeply into the patterns of interaction in groups that are different or similar in respect to those features.

The importance of all students having opportunities to contribute ideas during group work finds support in previous research. Previous studies suggest that equitable opportunities to pose and develop ideas during group collaboration may prevent propagation of misleading or incomplete ideas, may prevent the development of negative self-perceptions that may arise when some students are marginalized or hindered from participating in group conversations, and may contribute to learning among all students in the group [63,64]. Inequitable participation opportunities within groups may arise when some members are positioned (by the teacher or by each other) as experts (or more capable) and others as novices (or less capable, the former having greater opportunities than the latter to contribute ideas, explain their thinking, and obtain feedback from other students about their ideas). In-depth studies of specific students within small groups have highlighted the dynamic, negotiated character of opportunities to contribute [65], and the ways that, for example, race, gender, and social status intersect with individuals' attempts to wield mathematical influence [66–69]. This work also documents how individuals embrace or resist particular positionings, and leverage different kinds of resources to reorganize access to the group's work. For example, Esmonde and Langer-Osuna [70] showed how taking on the role of critic enabled one student to position herself powerfully in relation to another member's attempts to lead the group's work.

The second feature of group interaction we examine is the degree of continuity of the interaction among group members. While previous research has not investigated whether interaction among group members must be sustained and ongoing for highly-detailed participation to take place, some hints have emerged. For example, Barron [12] described how group interaction that led to successful problem-solving work involved the rapid exchange of tightly coordinated conversational turns, frequent eye contact, and constant monitoring of each other's contributions. Group interaction that led to less successful outcomes had less eye contact, more self-directed talk, more interruptions, and more instances of ignoring students' suggestions, as well as episodes of silence [71].

Previous research suggests that these two features of group interaction may operate independently. Shah, Lewis, and Caires [72], for example, showed how ongoing interaction could be either equitable or inequitable. Dyads all exhibiting ongoing interaction varied in terms of the distribution of talk and the issuing of commands and directives, with consequences for individuals' opportunities to learn.

In this paper, we first examine the extent to which these two features of group interaction—all students taking the lead in contributing mathematical ideas in the context of solving problems, and sustained, continuous interaction among group members—are necessary for highly-detailed participation (explaining one's own ideas and engaging with others' ideas at a highly-detailed level) to emerge. We then analyze the patterns in how group interaction unfolded over time to better understand how highly-detailed participation emerged in groups that were similar or different in terms of whether all members of the group played an active role in contributing mathematical ideas, and the degree of continuity of interaction among group members.

#### **2. Materials and Methods**

#### *2.1. Participants*

The school district selected in this study (a large urban area in Southern California) expressed ongoing interest in improving mathematics instruction in alignment with the Common Core State Standards for Mathematical Practice [73]. The majority of students in this district are Hispanic or Latino students (greater than 75%) and a large proportion of students are classified as English Learners (over 40%) [74]. The demographics of students at the particular school used in this study were similar to the district. At this school, just over 80% of students identified as Hispanic or Latino with approximately 8%, 6%, and 2% identifying as White, African American, and Asian, respectively. Further, at the third grade level at the school (the grade level studied here), more than 84% of the students identified as Hispanic or Latino and approximately 7%, 6%, and 1% identified as White, African American, and Asian, respectively. Close to half of the students identified as female (in the district, school, and third grade at the school). In this school during that

particular academic year, 22% of the third graders met or exceeded the standard for math achievement. This was comparable to the math achievement for the district, where 29% of the third graders met or exceeded the standard for math achievement.

Two third grade teachers within the same elementary school in this particular district were recruited to participate in this study due to their successful efforts to create learning environments where students generated their own strategies for solving problems and explained and responded to each other's mathematical ideas. Both teachers had extensive experience with professional learning opportunities related to Cognitively Guided Instruction [75] and were interested in learning how to better support their students to engage with each other's thinking. Within these two classrooms, 98% of the enrolled students consented to participate. The student who did not consent to participate was not videotaped during any of the lessons. The final sample analyzed here included 45 third graders.

#### *2.2. Mathematics Lessons*

The two teachers coordinated their lessons throughout the academic year. They selected similar tasks and generally organized and sequenced their instructional time similarly. Each lesson typically started with a 10–20 min whole-class warm-up where students solved and discussed activities or problems (e.g., 2 h and 45 mi = \_\_\_ min).

Next, the lesson included 25–35 min of small-group collaborative problem solving where students worked together (mostly in pairs but occasionally in groups of three students) to make sense of and solve one or more story problems involving multiplication or division. An example of a problem that both teachers used was: "(Name of student) is allowed to watch T.V. for \_\_\_ min each day. How many minutes of T.V. can they watch in \_\_\_ days?" Students selected from three different number sets: (15, 8), (16, 6), (26, 4). The teacher encouraged students to generate, and carry out, multiple strategies for solving each number set. During small-group work, students were not provided with specific instructions about how to explain or engage with each other, nor were they given specific roles to play, or particular protocols to follow. On only a few occasions did the teacher provide any direction to the class during the transition to small-group work (e.g., "I want you to work on this with your partner. Make sure you try and understand each other's strategies. Stop once in a while to explain what you're doing. And that's it. You can go get started with your partner.").

The lessons concluded with a 15–20 min whole-class wrap-up during which students (or pairs) shared the strategies they had developed for the problems during their smallgroup collaborative problem-solving time.

#### *2.3. Data Collection*

On six days (two in January, two in March, and two in May), we video recorded student and teacher interactions in each classroom. On each day in each classroom, we used six video-recording devices to capture the interaction of six pairs (or triads) of students during small-group work. The use of multiple recording devices made it possible to record the interaction for half of the students in the class (approximately 12 students) on each day. We rotated which students were recorded on each day so that we recorded all students in the class during an adjacent pair of lessons. This procedure resulted in 68 video recordings of small-groups: 63 pairs and 5 triads.

We collected all written student work on every day of observation. The written work showed students' problem-solving strategies for the problems assigned in the small-group collaborative problem-solving portion of the lessons.

#### *2.4. Selection of Pairs for Analysis*

For the analyses presented here, we examined pairs whose dialogue developed naturally and was not driven by the intervention of the teacher in any significant way. More specifically, we included pairs in which the teacher did not visit the pair or checked-in only briefly. If the teacher's intervention lasted longer than a brief check-in, we included

the pair for analysis if the teacher did not steer or otherwise shape the trajectory of the conversation. We excluded pairs where the teacher played a large role in driving the pair's interaction, such as asking detailed questions that invited explanations, or inviting students to engage with each other's ideas. This resulted in the 55 videotapes of smallgroup work analyzed in this paper: 51 pairs and 4 triads. Because the overwhelming number of small-groups were pairs, we will use the term "pairs" to refer to both group sizes for convenience.

#### *2.5. Coding of Student Participation*

For each lesson in which a pair was videotaped, we coded each student according to whether they exhibited highly-detailed participation. Students were coded as exhibiting highly-detailed participation if they carried out highly-detailed explaining and/or highlydetailed engaging with others' ideas. Highly-detailed explaining consisted of explaining one's thinking with enough detail that most or all of their mathematical strategy was clear to the coders (e.g., For the problem, "\_\_\_students in Room 18 raised \_\_\_ each for the jog-athon. How many dollars did those students raise in all? (6, 15)": "I did six boxes then I put 15 [in each box] and split it into 10 and 5. And then I did 10, 10, 10, 10, 10, 10, I put equal 60. Then 5 times, 5, 5, 5, 5, 5, 5. And then I put equals 30. And then I put 60 plus 30 equals 90. And then my answer was 90.") Ambiguous or incomplete explanations were not coded as highly-detailed explaining (e.g., Now I know what I did. So we have to draw, like, 15, 15, 15. We can count by fifteens. 15, 20, 25, 30. 5, 10, 15, 20, 25, 30, 35 . . . This is 140.").

Highly-detailed engagement with others' ideas consisted of explaining another student's strategy in detail, extending the details of their partner's idea(s), or suggesting detailed alternatives. In the following example, Melanie suggested an alternative way to combine numbers that she thought was more efficient than what Noemi suggested for the problem that required totaling seven fifteens:


Simply repeating details of their partner's suggestions, asking general questions, asking questions about elements in a student's work, or voicing agreement or disagreement without adding any details or reasons were not coded as highly-detailed engagement with others' ideas. In the following example, Mira repeated a step her partner articulated, and asked a question about another number her partner had written, but did not extend the mathematical work:


Donatello: I'm counting by fifteens, seven times.

Mira: So you are counting by fifteens?

Donatello: 45 . . .

Mira: How do you know the next number is 30? (points to Donatello's notebook)

We then classified each pair according to the number of students in the pair who exhibited highly-detailed participation (for pairs: two, one, none; for triads: three, fewer than three, none).

To check rater consistency, the research team selected a random sample of 10 students for a randomly selected lesson. Two raters independently coded student participation for this sample of students. Exact rater agreement was high (above 97%) for both highlydetailed explaining and highly-detailed engagement with others' ideas.

#### *2.6. Coding of Group Interaction*

#### 2.6.1. Number of Students Leading the Math

The first group interaction feature coded was the number of students in the group who led the math in terms of contributing mathematical ideas. By leading the math, we mean that the students' efforts to contribute shaped the direction or nature of the mathematical work. This included making suggestions for alternative or more efficient problem-solving strategies, suggestions for how to begin or carry out an approach to solving the problem, making suggestions for alternative problem-solving strategies, suggesting new steps in a strategy, connecting a step with its relation to the story context, proposing a different way to carry out a particular step in a strategy, challenging the validity of others' problem-solving approaches or strategies, or exploring discrepancies that arose in problemsolving approaches, strategies, or results. Carrying out the steps of a strategy that another student proposed, repeating work that another student had described, asking questions of clarification, and confirming or disconfirming others' work (without providing justification or rationale), while important ways of contributing, were not, by themselves, bases for coding that a student led the mathematics in terms of contributing mathematical ideas.

As a preliminary step, we separated each video into segments corresponding to a solution strategy for a number set. We then assigned codes to each segment (as described below) and decided on the most representative code for the pair across all of the segments. Multiple members of the research team coded each video. Discrepancies were resolved through an iterative process of discussion in the whole research team and recoding, as described in a later section.

For each segment of a video, we assigned one of the following codes: (a) both students led the mathematics, (b) one student led the mathematics, or (c) no student led the mathematics (students were working on the problems but did not interact about the problems or their work). We then assigned one code that best represented the interaction in the pair across all segments. We selected a code as most representative by taking into account the number of segments with that code, the amount of interaction time with that code, and the centrality or importance of particular segments in terms of the pair's development of mathematical strategies and mathematical ideas (e.g., in some pairs, earlier segments showed the richest interaction around mathematical ideas; in other pairs, later segments involved in-depth interaction around discrepant strategies or answers). In addition, a pair was coded as both students leading the mathematics if both students led the math in most or all segments (or if both students participated but there was no clear leader), or if each student in the pair led the mathematics for different number sets and strategies (e.g., in some pairs, members of the pair alternated who led the mathematics).

#### 2.6.2. Continuity of the Group's Interaction

The second group interaction feature coded was the degree to which pairs showed sustained interaction. For the continuity of the interaction during collaborative work, we assigned each pair one of the codes in Table 1.

**Table 1.** Continuity of interaction among students in the pair.


#### 2.6.3. Patterns of How Group Interaction Unfolded over Time

We next created descriptions of how the interaction unfolded within each pair, starting from the suggestions of strategies to pursue, and continuing through how the pair interacted as they carried them out. These descriptions provided additional details about the evolution of the interaction across the entire small-group collaborative problem-solving time. Three of the research team members first generated and discussed the descriptions. The other three members of the team then independently reviewed the descriptions. Finally, the whole research team discussed changes that needed to be made. The final ten interaction patterns are presented and discussed in the next section on results.

#### 2.6.4. Coding Process for Group Interaction Variables

For the group interaction variables (number of students leading the math, continuity of interaction in the pair, and pattern of how group interaction unfolded over time), we conducted a team-based approach to make coding decisions [76]. The six members of the project's research team participated in the discussions around the development of the coding procedures and application of the codes. Multiple project members were responsible for coding different combinations of pairs. Refinement of codes and coding decisions occurred in an iterative fashion during weekly meetings over 19 months. During each meeting, project members raised questions and refined the coding scheme, compared coding results, and reviewed the coding of others. At every step, project members who had not initially performed the coding for a pair reviewed prior coding decisions, raised questions, and suggested alternatives. The purpose of this step was to check the dependability of applying the codes consistently across the entire data set and between the different coders [77]. Discrepancies between coders were discussed with the whole team and resolved. All project team members went back to the pairs they initially coded to ensure that the decisions about the discrepancies were consistently applied [78]. Discussions continued until consensus was reached for all codes for all pairs.

#### **3. Results**

In the following sections, we begin with a summary of the highly-detailed participation we observed from the pairs in our sample. Then, we describe the profiles of interaction in pairs and their relationship to highly-detailed participation. Next, we examine the patterns of interactions that underlie these profiles, and, lastly, we explore how highly-detailed participation emerged within these patterns.

#### *3.1. Summary of Highly-Detailed Participation in the Pairs*

Table 2 shows the breakdown of the pairs according to the number of students in each pair who participated at a highly-detailed level. In almost half of the pairs (24, 44%), both students exhibited highly-detailed participation. The majority of these pairs (19 of 24, 79%), showed both highly-detailed explaining and highly-detailed engagement with others' ideas. In the minority of pairs (5 of 24, 21%), students provided highly-detailed explanations of their own thinking without engaging at a highly-detailed level with their partner's ideas.

**Table 2.** Incidence of highly-detailed participation in the pair.


#### *3.2. Relationship between Leading the Math and Highly-Detailed Participation*

Table 3 presents the distribution of pairs according to how many students in the pair led the mathematics and the number of students in the pair who participated at a

highly-detailed level. As can be seen in Table 3, in the majority of pairs (36 of 55, 65%), both students in the pair were involved in leading the mathematics. In a minority of pairs (16 of 55, 29%), one student led the mathematics. In only a few pairs (3 of 55, or 5%) did no student lead the mathematics. When both students in the pair were involved in leading the math, it was very common for both students to participate at a highly-detailed level (22 of 36, 61%). When one student in the pair led the math, it was more common that one student participated at a highly-detailed level (10 of 16 pairs, 63%). Of these 10 pairs, most often (7 of 10, 70%), the student leading the math was also the student who participated at a highly-detailed level.


**Table 3.** Relationship between the number of students leading the math and the incidence of highlydetailed participation.

The relationship between the number of students in the pair who led the mathematics and the number of students in the pair who participated at a highly-detailed level was statistically significant (Fisher–Freeman–Halton Exact Test = 18.40, *p* < 0.001), with a large effect size (Cohen's *W* = 0.63) [79]. Even omitting the three pairs in which there was no interaction around the math, the results are statistically significant (Fisher–Freeman–Halton Exact Test = 12.31, *p* = 0.002), with a large effect size (Cohen's *W* = 0.48). This result shows that whether both students or one student led the math significantly related to the number of students in the pair who participated at a highly-detailed level.

#### *3.3. Relationship between Continuity of the Group's Interaction and Highly-Detailed Participation*

Table 4 presents the distribution of pairs according to the degree of continuity of the pair's interaction and the number of students in the pair who participated at a highlydetailed level. As can be seen in Table 4, the majority of pairs had an interaction that was sustained and ongoing (34 of 55 pairs, 62%). A minority of pairs had an interaction that was periodic (18 of 55, 33%). In only a few pairs (3 of 55, or 5%) was there minimal interaction (1-shot or none).


**Table 4.** Relationship between the degree of continuity of a pair's interaction and the incidence of highly-detailed participation.

While the degree of continuity in the pair's interaction was not significantly related to the number of students in the pair who participated at a highly-detailed level (Fisher– Freeman–Halton Exact Test = 6.24, *p* = 0.134), the effect size of the relationship was moderately large (Cohen's *W* = 0.41). This effect was due to the difference between the pairs with 1-shot or no interaction around the math and the pairs with ongoing or periodic

interaction. Omitting the three pairs in which there was minimal interaction around the math produced both a statistically insignificant effect (Fisher–Freeman–Halton Exact Test = 0.15, *p* = 1.00) and a very small effect size (Cohen's *W* = 0.03). This result shows that whether students interacted in an ongoing, sustained interaction or interacted periodically (but not continuously) was not associated with the number of students in the pair who exhibited highly-detailed participation.

#### *3.4. How Interaction Unfolded in the Pairs*

As shown above, when both students participated at a high level, it could involve either one or both students leading the mathematics, or either ongoing or periodic interaction among members of the pair. The question then emerges about the particular ways in which the interaction in these pairs unfolded over the course of their problem solving that were most likely to give rise to highly-detailed participation by both members of the pair. To investigate this question, we examined more closely the pairs in the first column of Table 3 (both students exhibited highly-detailed participation). We also examined the other pairs in the first row of Table 3—where both students led the mathematics and highly-detailed participation was exhibited by one or no member of the pair—to see whether interaction in these pairs unfolded in ways that may have been different from the pairs in the first column of Table 3. The results are summarized in Table 5, which lists the multiple patterns of how interaction unfolded among these 38 pairs over the course of their work together.

Several findings are apparent in Table 5. First, the interaction in the pairs began and evolved in multiple, different ways. In some pairs, students jointly produced and jointly carried out their strategies. In other pairs, students alternated initiating strategies for how to solve the problem and then worked on carrying out those strategies. In still other pairs, students generated and carried out their own strategies. In some pairs, one member of the pair generated the strategies and the other member of the pair worked on those strategies. A few pairs did not interact at all until they had completed some strategies. There was also variation in how continuous the pair's interaction was as they carried out their strategies. In some cases, students conferred in an ongoing, sustained way as they worked, while in other cases, students only occasionally conferred or conferred only at certain well-defined points during the group's work (such as when they completed a strategy).

Second, Table 5 shows that all of the patterns of interaction led to instances in which both students in the pair exhibited highly-detailed participation. That is, there were many ways in which group interaction could begin and unfold that led to both students exhibiting highly-detailed participation. It was not necessary, for example, for pairs to be jointly involved in generating and carrying out strategies. It was not even necessary for both members of the pair to be involved in initiating or generating the strategies. It was not necessary for students to carry out the same strategies. Finally, it was not necessary for students to confer regularly as they worked.

Third, while all of the patterns of interaction led to instances in which both students in the pair exhibited highly-detailed participation, it was not the case that they guaranteed this outcome. For the pairs we analyzed here, many patterns of interaction had instances in which only one or no student exhibited highly-detailed participation. Importantly, then, patterns of how interaction unfolded in pairs in which both students exhibited highlydetailed participation were very similar to those for pairs in which only one or no student exhibited highly-detailed participation. Students found different ways of working together and sharing their mathematical work. For some pairs, these patterns of working together led to both students exhibiting highly-detailed participation; in other pairs, these same patterns did not.


**Table 5.** Patterns of how the interaction in pairs unfolded over time.

<sup>1</sup> On this particular day, unlike the other days, the teacher gave the class 15 min of independent time to work on the problems before having them gather in groups to discuss their strategies and work on further number sets.

#### *3.5. Emergence of Highly-Detailed Participation during Pair Interaction*

This section presents more details for each of the patterns of interaction identified in Table 5 to show how highly-detailed participation emerged in the pair. For each of the ten patterns identified in Table 5, Table 6 presents an example in which both students in the pair exhibited highly-detailed participation.

As can be seen in Table 6, highly-detailed participation emerged in pairs in multiple ways. One common feature of pair interaction across the group interaction profiles was for students to give unsolicited explanations of how they had solved the problem (sometimes in conjunction with explicitly stating that they should explain their strategies). Other common features of interaction that led to highly-detailed participation (fully explaining one's own approach and/or engaging with others' ideas at a highly-detailed level) included a student suggesting a strategy to pursue, students disagreeing about an answer, one student questioning another student's work, one student appropriating another student's strategy, and students seeking confirmation of their strategy. These features (and the highly-detailed participation that ensued) occurred regardless of which student or students generated the strategies, which student(s) led the execution of the strategies, and how continually the students conferred about the work.


**Table 6.**How highly-detailed participation emerged in pairs' interaction1.


1 In these examples, both students in the pair exhibited highly-detailed participation. 2 On this particular day, unlike the other days, the teacher gave the class 15 min of independent time to work on the problems before having them gather in groups to discuss their strategies and work on further number sets.

#### **4. Discussion**

This paper investigated productive features of group interaction in pairs of students working to solve mathematical problems in which the students, and not the teacher, drove the interaction among students. Specifically, we examined patterns of group interaction that led to students giving highly-detailed explanations of their own ideas and engaging with others' ideas in highly-detailed ways, both of which have been found to be related to students' learning outcomes. Three major findings emerged. First, the number of group members taking the lead in contributing mathematical ideas about how to solve problems was significantly related to the number of students who exhibited highly-detailed participation. Contributing mathematical ideas did not only consist of making suggestions for how to begin or carry out problem-solving approaches. It also included reacting to, or building upon, others' ideas in ways that shaped the direction or nature of the work, such as suggesting extensions or alternatives. It did not include asking questions or disagreeing without offering new or alternative ideas.

The notion of taking the lead in contributing ideas is embedded in other researchers' perspectives about interaction among students that is productive for learning. For example, in exploratory talk [34–36], everyone "offers the relevant information they have" and "engages critically but constructively with each other's ideas" ([1], p. 187). In accountable talk ([2], p. 286), participants "build on each other's ideas . . . provide reasons when they disagree or agree with others" and "may extend or elaborate someone else's argument." In transactive discussions, students analyze and respond to their partners' ideas and assessments, rejecting ideas and proposing alternatives where relevant [5,6]. In argumentation, students present their ideas and listen to and criticize each other's explanations [7–10,30]. In collaborative reasoning, [40], after indicating their positions on an issue and offering reasons for them, students "either support and add to the reasons expressed or challenge and offer alternative reasons" ([11], p. 583). Co-construction [27] includes "additions (linking a new idea to someone else's idea or partial idea), corrections . . . or dialectical exchanges (disagreeing with the prior statement and offering a counterargument)" ([16], p. 394). The foregoing work shows that contributing ideas take many forms, and do not necessarily correspond with surface features of interaction such as which student talks the most or which student controls most of the conversation. What our findings make explicit is the importance of *every* student in a collaborative group having, and taking, opportunities to take the lead in contributing ideas that shape the group's work and move it forward.

Second, whether students interacted in an ongoing, sustained manner or interacted only periodically was not associated with the number of students in the pair who exhibited highly-detailed participation. While conferring in a sustained and ongoing fashion was one way in which students could converse productively, it was not the only way. Students could also interact with each other occasionally but not continuously, and still be engaged in ways that supported participation that led to highly-detailed participation.

Other researchers have also found that interaction around the task need not be continuous, and in particular that off-task interludes may be productive. For example, participation often characterized as off-task talk may serve to help groups ease into the task, may serve to help certain students gain access or re-access to the conversation, and can be used to resist the tendency of particular students to dominate groups' discussions [70,80].

Third, group interaction unfolded in many different ways over time, and multiple profiles of group interaction led to the emergence of highly-detailed participation for both students in the pair. In particular, highly-detailed participation among both members of the pair occurred whether both students or only one student initiated the problemsolving strategies, whether students worked on the same or different strategies, whether they worked through them jointly or separately, or whether they worked in an ongoing, sustained fashion or conferred only occasionally after periods of independent work. The multiplicity of productive group profiles is consistent with work by Shah and Lewis showing that collaboration that is equitable overall ("fair distribution of both participation opportunities and participation itself" ([65], p. 423) can have levels of equity and

inequity that fluctuate over the course of collaboration, such as students alternating taking leadership roles [65,72].

The assortment of productive group interaction profiles we observed suggests that there is not a single "right way" for students to interact. An implication of these findings is that, when seeking to promote collaborative problem solving, teachers should not assume that particular patterns of interaction are "productive" or "unproductive." Rather, it is important to look beyond surface features to attend to the substance of students' participation: do students find particular moments to ask each other questions or make suggestions, do their contributions shape and add details to emergent strategies, and do they create openings for others to participate in collaborative work? These results are consistent with other research that highlights the importance of collaborations that involve students explaining and engaging with the substance of one another's ideas, while also showing that avenues to meaningful engagement can take varied forms [70,71,81]. Who initiates problem-solving strategies, for example, may be less important than whether and how students' questions and suggestions are considered by others in the group and incorporated into the group's work.

These findings highlight the importance of teachers monitoring the participation occurring in collaborative groups and then finding ways to interact with groups in the moment to encourage students to contribute new ideas. A number of researchers have identified ways in which teachers can do this. For example, Gillies [45,46] describes teachers' mediating behaviors that can encourage students to suggest new ideas, including providing hints to consider new information or perspectives, questioning how multiple ideas are the same and different, and asking groups to evaluate whether all necessary ideas have been covered. Many of the talk moves described by O'Connor and Michaels [47] explicitly call for students to contribute new thinking, such as asking students to add on to others' ideas and to explain why they agree or disagree. Multiple instructional moves used by the teacher in collaborative reasoning discussions [11,50] function similarly, including asking for alternative perspectives, and challenging students to consider counter-arguments.

While the group interaction profiles that we observed to be associated with both students in the pair exhibiting highly-detailed participation are similar to profiles observed in prior research [82], we do not claim that this is an exhaustive set of profiles nor that this set of profiles will necessarily generalize to all classrooms. The teachers included in this study had extensive experience using students' ideas as a basis for instruction, and were recruited based on their interest in furthering their expertise in supporting student participation. The profiles of group interaction that may emerge in classrooms of teachers without this level of experience will be a fruitful area for further research.

Similarly, as previous studies have suggested, the variety of group interaction profiles observed must be considered in relation to the nature of the group's task. The tasks teachers created in this study in some ways varied from those recommended by other scholars [83–85]; they were neither ill-structured nor procedural in nature. Tasks were open in the sense that they did not require students to use particular strategies, and instead asked for students to solve problems in ways that made sense to them. Yet in other ways tasks were somewhat closed in that the story problems themselves were relatively straightforward and had a single answer. The influence of the design and enactment of tasks on the variety of group interaction profiles that may emerge remains to be investigated. In addition, teachers in this study did not install specific structures for who participates, when, and how [43]. How such structures may impact the variety of productive group interaction patterns that emerge is another topic for further study.

Still unknown is whether the group interaction profiles we observed may vary according to the membership of the group, or over time even for groups with the same composition [55,56]. For example, students may make different choices on when and how they participate based on the synergy with their partners on that day. Finally, most of the groups we observed were pairs. Whether and how the size of the group influences the profiles of group interaction that emerge remains to be investigated [58].

In conclusion, we observed a wide variety of interaction patterns in collaborative groups that led to productive participation by all members of the group. These results suggest that teachers and researchers should not be concerned with legislating ways in which groups should interact. Instead, it may be more fruitful to work toward creating opportunities for students to engage with each other during collaborative work in ways that make sense to them at that moment.

**Author Contributions:** Conceptualization, N.M.W., M.I., E.B., N.C.J., M.L.F. and J.Z.; Data curation, N.M.W.; Formal analysis, N.M.W.; Funding acquisition, N.M.W. and M.L.F.; Investigation, N.M.W., M.I., E.B., N.C.J., M.L.F. and J.Z.; Methodology, N.M.W.; Project administration, N.M.W. and M.L.F.; Resources, N.M.W.; Software, N.M.W.; Supervision, N.M.W. and M.L.F.; Validation, N.M.W.; Visualization, N.M.W.; Writing—original draft, N.M.W.; Writing—review & editing, N.M.W., M.I., E.B., N.C.J., M.L.F. and J.Z. All authors have read and agreed to the published version of the manuscript.

**Funding:** The research was funded by Institute of Education Sciences, grant number [R305A100181]; National Science Foundation, grant number [1919490]; Spencer Foundation, grant number [201500046]; UCLA Academic Senate Council on Research, grant number [2018 Faculty Research Grant].

**Institutional Review Board Statement:** Ethical clearance was obtained from the overseeing university in 2013 (IRB# 13-001403).

**Data Availability Statement:** The data are not publicly available due to privacy issues.

**Acknowledgments:** We wish to thank Angela Chan Turrou for her assistance on the project and the teachers who contributed their expertise to our research.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


## *Article* **Dual Effects of Partner's Competence: Resource Interdependence in Cooperative Learning at Elementary School**

**Céline Buchs 1,\* , Anissa Dumesnil <sup>2</sup> , Julien Chanal 3,4 and Fabrizio Butera <sup>5</sup>**


**Abstract:** A partner's competence should logically favor cooperative learning. However, research in cooperative learning has shown that a partner's competence may or may not activate a threatening social comparison and yields dual effects: It is beneficial when students work on complementary information while it is detrimental when students work on identical information. Two studies conducted at elementary school (study 1 with 24 fourth graders working on encyclopedic texts, and study 2 with 28 fifth graders working on argumentative texts) replicated that interaction: Information distribution (complementary vs. identical information) moderated the relationship between partner's competence and pupils' learning outcomes. The relation between partner's competence and students' performances was positive when working on complementary information, but negative when working on identical information. A third study confirmed that working on identical information led to a competitive social comparison whereas complementary information reinforced the pupils' cooperation perception. Contributions to cooperative learning research are discussed in terms of the competitive comparisons that may arise during cooperative learning at elementary school.

**Keywords:** social comparison; cooperative learning; resource interdependence; competence; elementary school

#### **1. Introduction**

Peer learning is encouraged by many instructional models in a variety of educational settings [1] from elementary school [2–6], through to secondary school [7–9], post-secondary school, and university [10–13]. Peer learning favors both work and play relations through a positive cognitive and social impact [14]. Among the peer learning methods, cooperative learning has been widely investigated due to its strong interconnections between theory, research, and practice [15–21]. Research that has compared cooperative learning with individual or competitive learning has consistently shown positive effects for a wide range of outcomes including learning (see [22] for a review of meta analyses).

Nevertheless, cooperative learning does not automatically lead to improved learning. The educational system, notwithstanding its formative mission, is fundamentally competitive [23,24], especially in the neo-liberal ideology in which most Western societies and educational institutions are embedded [25,26], and it is possible that this competitive atmosphere interferes with the effects of cooperative learning [27,28]. Indeed, a line of research carried out with university students has shown that focus on social comparison with a partner during cooperative learning may lead to a dual effect of partner's competence [29]. In this context, partner's competence can reduce as well as increase learning outcomes [30]. As competitive goals permeate the whole educational system, the aim of

**Citation:** Buchs, C.; Dumesnil, A.; Chanal, J.; Butera, F. Dual Effects of Partner's Competence: Resource Interdependence in Cooperative Learning at Elementary School. *Educ. Sci.* **2021**, *11*, 210. https://doi.org/ 10.3390/educsci11050210

Academic Editor: Robyn M. Gillies

Received: 29 March 2021 Accepted: 23 April 2021 Published: 29 April 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

the present research is to test that such a dual effect of partner's competence in cooperative learning also emerges at the level of elementary school.

#### **2. Theoretical Framework**

#### *2.1. Cooperative Learning in a Competitive Society*

Cooperative learning proposes that students work together to accomplish shared learning goals, and has been derived into a variety of methods [31]. One key factor consensually recognized as beneficial for cooperative learning is positive goal interdependence [18,32,33], in other words, the process whereby students perceive that they can reach their goal only if the other partners reach their own [34,35]. Perceived positive interdependence motivates students to learn, encourages partners to interact with each other in a constructive manner, and effectively helps them learn [19,35]. This is why in mastery-based contexts in general, and in cooperative learning settings in particular, partners are generally viewed as sources of informational support likely to contribute to one's own learning [36,37].

Despite an overall positive effect, however, cooperative learning efficacy varies across studies. The effect size revealed in meta-analyses varies from moderate to large (and Hattie's mega-analysis [22] reports effect sizes between 0.41 and 0.54), indicating that cooperative learning is a potentially powerful tool for learning in comparison with individual or competitive learning. Nevertheless, reviews only report between 53% [38] to 63% [39] of comparisons, indicating positive gains in favor of cooperative learning, which points to the question of what happens in the remaining cases.

In this article, we conceptualize cooperative learning as a powerful tool based on positive goal interdependence, but a tool that has to operate in a society increasingly based on the values of achievement, power, and competition. Kasser et al. [26] has suggested that the values of what they call corporate, neoliberal capitalism, in which are embedded the majority of citizens of Western countries, may interfere with goals directed toward cooperation. This analysis echoes the work of several authors that have noted that selection, grading, and ranking typical of most educational institutions are likely to promote competitive goals [23,40–42] and to enhance performance-avoidance goals [43]. One example drawn from higher education is the selection whereby a large number of students (from 9 to 55%, *M* = 31%) enrolled at universities in OECD countries (Organization for Economic Cooperation and Development) end up without a degree [44]. The competition involved at university can also be illustrated by the European official recommendations regarding credits: For students who pass, teachers have to use normative assessment with an a priori percentage of students that should obtain different grades, namely 10% of As, 25% of Bs, 30% of Cs, 25% of Ds, and 10% of Es [45]. This grading practice hinders the willingness to cooperate with a peer [46].

#### *2.2. Cooperative Learning and the Dual Effect of Partner's Competence at University*

One can expect that when cooperative learning is introduced, this competitive orientation would be reduced. Nevertheless, research has pointed out that competitive social comparisons are at work, even during cooperative learning at university [29,47–49].

More specifically, in order to investigate the conditions under which competitive social comparisons are likely to interfere with cooperative learning, a line of research has studied the effects of information distribution (that is resource interdependence: resource independence versus positive resource interdependence) during cooperative learning on texts (e.g., [50]). Positive resource interdependence refers to tasks and situations in which group members possess different complementary pieces of information, each member accesses only one part of the needed information, and relies upon communication with the partners to access the rest of the necessary information. Resource independence, on the other hand, involves access to the whole information by each member, which implies that students work on identical information and do not depend on one another for information [38].

The study of information distribution is particularly important for the present research because working on identical information has the property of allowing members to compare and judge each other's competence. Marshall and Weinstein [51] suggested that working on simultaneous similar tasks for all students with a unique theme induces a more social comparison of abilities than working on different themes at different times. Lambiotte et al. [48] suggested that even in a cooperative context, working on identical information stresses evaluative pressure between peers compared to working with complementary information. Moreover, working with identical information altered the representation of the interaction that should be typical of cooperative learning: The mere fact of reading the same text induced students to compare themselves to the partners, which led them to question their own competence and their partner's competence [29,50]. Despite the potential positive effects of social comparison (e.g., [52,53]), when working cooperatively with identical information, this competitive social comparison was responsible for the negative effect on learning outcomes of working on identical information [29,54].

Direct comparison of competences during interactions may elicit a motivation to protect self-esteem [55], transforming the partner into a potential competitor instead of a valuable resource. This implies that the others' competence may be threatening. This supposition has been put to the test by Buchs et al. [29]. Cooperative learning produces an environment in which the competences of group members should be viewed as supportive, and logically a more competent partner should lead to better learning than a less competent partner. However, when contrasting peer-learning methods on information distribution, an interaction between perceived partner's competence and resource interdependence condition appeared, indicating that for students working on complementary information, the more they perceived their partner as competent, the better they performed, whereas the relation was negative when they worked on identical information [29]. Results have been replicated when the competence of the partner was manipulated in order to test the causal link [30]: A confederate was trained to propose either a brilliant or an average oral performance. Results indicated that the partner's competence was positive for students' learning only in the positive resource interdependence condition, whereas under resource independence, the partner's competence was paradoxically detrimental to learning. In sum, even when cooperative learning is implemented, there are conditions—such as working on identical information—under which students interpret the competence of partners as a threat to their own competence, which reduces learning. Thus, information distribution may lead to two opposite effects of partner's competence.

#### *2.3. A Dual Effect of Partner's Competence at Elementary School?*

As competition permeates the educational system at all levels [23], the present article thus considers the possibility of competitive social comparisons during cooperative learning, even at elementary school that could lead to the dual effect of partner's competence presented above.

#### 2.3.1. Social Comparison and Competition in Children

At first glance, several elements suggest that social comparison feedback would not be used by young children to assess their intellectual ability because of the difficulty to coordinate their own and others' perspectives [56] and to make inferences regarding ability from normative information before the age of 6–8 [57]. Moreover, Stipek and Mac Iver [58] as well as Dijkstra, Kuyper, van der Werf, Buunk, and van der Zee [59] argue that pupils' attention to social comparison information as well as the need to outperform their classmates increased from the beginning to the end of elementary school and kept growing during the junior and senior high-school years. On this basis, elementary-school children could be expected to be far less prone to being threatened by social comparison than university students.

Looking more closely at the literature, however, it appears that preschoolers in natural settings display spontaneous active social comparisons [56], with a predominance of

differentiation/similarity and competition modes of social comparison. Butler [60] noted that children may display some negative outcomes of upward comparison already at the age of three; her results and those of Rhodes and Brickman [61] indicated that children at ages three to five use others' outcomes for self-appraisal even in a non-competitive context when interpersonal comparisons are salient. Moreover, Ruble, Eisenberg, and Higgins [62] concluded that most 5-year-old children are able to infer realistic evaluations of performance and display affective reactions using concrete interpersonal comparisons. Finally, young pupils start using social comparison information to evaluate themselves from the age of seven [59]. Children are able to use differences in social outcomes to make evaluative inferences in simple contexts when they can judge the outcome difference directly [60] and to explain their self-perceptions of smartness in social comparative terms [58]. Moreover, they actively search for social comparison information [63]. For instance, pupils from grade 3 inspect and evaluate peers' work and display "besting behaviors", revealing a need to excel over others, along with comparative evaluative statements, even when the context is not a competitive one [64]. In sum, from the age of seven or eight, children become increasingly sensitive to social comparisons [65–67].

#### 2.3.2. Social Comparison and Competition in School Environments

Again, at first glance one could expect that, compared with university, elementary schools are likely to favor an environment in which academic social comparison with peers is not overwhelming and threatening, as suggested by Mosatche and Bragonier [56]. Stipek and Mac Iver [58], in particular, proposed that intellectual tasks are less standardized and relative performance information less available in early years of elementary school, where normative information is not usually made salient. According to these authors, the nature of the tasks and the visibility of peer skills based on social comparison increase from early elementary (before grade 3) to upper elementary school (grades 3–6), and still increase at the middle or junior high level. This evolution is confirmed in studies that compare grades 4–5 to grades 6–7 [68], and in studies carried out at junior high school from grade 6 to 7 [69].

Relevant to this contention, achievement goal theorists have identified two separate classroom goals structures [70]; the distinction relies on the extent to which schools or classroom-level practices promote mastery goals (strong focus on understanding the classwork, efforts needed to improve learning) or performance goals (strong focus on normative evaluation and public social comparison of abilities). Elementary school appears to be more oriented toward mastery, autonomy, social and academic support, and cooperation [68,69], while focus on grades and normative comparisons increase from elementary school to middle school (see [70]). In elementary school, students are not usually assigned to classes according to their ability, special privileges offered for relative ability are less frequent, and normative evaluations are less likely.

However, data also indicate that in elementary school, social comparison is already likely to occur. Pepitone [64] underlined that being in a classroom implies a number of ingredients that reinforce the likelihood of social comparison: Novel tasks and new learning imply some degree of cognitive uncertainty, in a context where relevant others (people belonging to a reference group confronted to the same instruction) are available, and where evaluation is important. Moreover, the reward system, perceived focus on achievement and social pressure to perform well contribute to create an evaluative atmosphere in the classroom, both for children [59,65] and teachers [71,72]. While arguing that normative evaluation is less important in elementary school than in junior high school, Stipek and Mac Iver [58] underlined that focus on relative performance, the use of competitive activities in the classroom, and assignment to classes according to ability may appear as soon as grade 3 in elementary school. The way the student perceives the teachers' goals predicts the students' performance goal pursuit [73]. As early as grade 5, some elementary teachers are perceived by their pupils as high in performance, which impacts the goal structure of their classrooms [74]. Therefore, even if social comparison is less likely at elementary school than at university, pupils may be sensitive to competitive social comparisons at upper elementary school (from grade 3, i.e., eight years old).

#### *2.4. Hypotheses and Overview*

In sum, research on the effects of information distribution at university has shown that even minor contextual cues—such as for instance working on identical information—may elicit competitive social comparison concerns during cooperative learning. In particular, as noted above, results indicate a dual effect of partner's competence: Although for students working on complementary information, the higher their partner's competence, the better they perform, when they work on identical information this relation is negative [29,30].

Research also showed that social comparison may be related to competitive concerns and therefore be threatening—as early as elementary school, and the same concerns as those haunting university students may play a role at this level. Thus, drawing on research that has demonstrated that resource independence (working on identical information) triggers competitive social comparison whereas positive resource interdependence (working on complementary information) attenuates it and favors the recognition of one's partner's competence helpfulness (e.g., [50]), we predict that even with elementary school children, we should observe the same interaction between resource distribution and partner's competence on learning outcomes that was observed with university students. In a cooperative learning setting, when working on complementary information, the partner's competence should be positively linked with the pupils' learning, whereas when working on identical information, the partner's competence should be negatively linked with the pupils' learning.

Two experimental studies carried out in natural settings tested this hypothesis. To provide external validity, pupils worked on encyclopedic texts in study 1 and on argumentative texts in study 2. Moreover, study 1 involved fourth graders, and study 2 involved fifth graders. The third study aimed to test the effect of information interdependence on social comparison during cooperative learning at elementary school.

#### **3. Study 1**

#### *3.1. Materials and Methods*

#### 3.1.1. Participants

Twenty-four pupils, 11 boys and 13 girls, regularly attending grade 4 in a Swiss Frenchspeaking school and aged from 9.08 to 10.75 years (*M* = 9.87, *SD* = 0.45) participated in the study. Pupils worked in cooperative dyads under the supervision of the experimenter for one session. Ten pupils worked with complementary information and 14 worked with identical information. Their regular teacher composed the dyads so that the level in French was relatively equivalent inside the dyad and that pupils could work well together (no "enemies" together).

#### 3.1.2. Procedure

The experimental dyads did not work in their classroom (this applies to all three studies). Two or three dyads were brought out of the classroom at the same time; they worked in a room where they could use space in a comfortable way, without dyads interfering with each other and with a quiet environment. The experimenter requested the dyads work on two texts. Positive goal interdependence was kept constant in all conditions: Pupils were asked to work cooperatively and to try their best to promote both their own learning and their partner's learning. Their common goal was therefore to foster learning for all partners so that each learner could master the whole content [75].

Role interdependence was also constant, but roles were reversed for each text. In line with scripted cooperation procedures [76,77], two roles were introduced in order to facilitate the discussion. The summarizer role consisted of explaining as clearly as possible and in a detailed way the information contained in the text. Listeners had to ask questions, request clarifications, add comments, and identify errors or inconsistent information while

listening to the summarizer. Roles were presented to students as flexible roles introduced in order to facilitate discussion. The terms "Summarizers" and "Listeners" thus designated the students' primary role, but did not preclude interactive exchanges. For each text, one student played the summarizer role while the partner played the listener role. Roles and text order were counterbalanced. Dyads had 12 min to read and 5 min to discuss each text. Discussions of the dyads were recorded for the purpose of analyses. After working on the texts, they filled in a learning questionnaire on the two texts.

#### 3.1.3. Materials

Dyads worked on two encyclopedic texts reporting factual information: one text on ostriches and the other on chameleons. The length was quite similar (584 versus 605 words) and texts were structured in a similar way: one text (one page with nine paragraphs and eight pictures) and one figure with the explanation of some parts of the animal body (6–7 body parts). We checked that the specific content of these texts had not been addressed in the pupils' curriculum to ensure that the content was unfamiliar to them. A lexicon was used to answer vocabulary questions.

#### 3.1.4. Independent Variables


#### 3.1.5. Dependent Variable: Learning Outcomes as Listeners

Individual learning was assessed with a learning questionnaire regarding each text at the end of the session composed of nine multiple-choice questions per text (one question per paragraph). The multiple-choice test (MCT) was developed by pre-service teachers and perfectly matched the normal test format used by regular teachers. Each question had

three answers with only one being correct. One point was allocated for a correct answer, 0 for no answer and for mistakes. These criteria were explained to the pupils before the MCT. Thus, learning scores for each text ranged from 0 to 9, and we identified the role pupils endorsed for each text to compute their performance as summarizer and as listener. This score allowed us to test the relationship between the summarizers' competence and listeners' performance, as proposed by Buchs and Butera [30]. Additional variables were measured in the three studies. As they are not relevant to the present research question, they are not reported in the text, but can be obtained by writing to the corresponding author. had three answers with only one being correct. One point was allocated for a correct answer, 0 for no answer and for mistakes. These criteria were explained to the pupils before the MCT. Thus, learning scores for each text ranged from 0 to 9, and we identified the role pupils endorsed for each text to compute their performance as summarizer and as listener. This score allowed us to test the relationship between the summarizers' competence and listeners' performance, as proposed by Buchs and Butera [30]. Additional variables were measured in the three studies. As they are not relevant to the present research question, they are not reported in the text, but can be obtained by writing to the corresponding author.

blind regarding the experimental conditions. The inter-rater agreement was good enough (i.e., 88%) to keep only the pre-service teacher's coding as a measure of the

Individual learning was assessed with a learning questionnaire regarding each text at the end of the session composed of nine multiple-choice questions per text (one question per paragraph). The multiple-choice test (MCT) was developed by pre-service teachers and perfectly matched the normal test format used by regular teachers. Each question

*Educ. Sci.* **2021**, *11*, x FOR PEER REVIEW 7 of 17

3.1.5. Dependent Variable: Learning Outcomes as Listeners

summarizer's competence (from 7 to 26, *M* = 16.83, *SD* = 6.06).

#### *3.2. Results and Discussion 3.2. Results and Discussion*

As pupils learned in dyads, data are not independent, and therefore we conducted multilevel analyses. We built a two-level model with pupils nested in dyads. The tested model comprised our two predictors, namely resource interdependence (coded-1 for identical information and +1 for complementary information) and summarizer's competence (assessed by the total number of correct informational inputs, centered to its mean) as well as a term for the response considered (as a summarizer or as a listener) and the interactions between these three variables. As pupils learned in dyads, data are not independent, and therefore we conducted multilevel analyses. We built a two-level model with pupils nested in dyads. The tested model comprised our two predictors, namely resource interdependence (coded-1 for identical information and +1 for complementary information) and summarizer's competence (assessed by the total number of correct informational inputs, centered to its mean) as well as a term for the response considered (as a summarizer or as a listener) and the interactions between these three variables.

Results revealed that the predicted interaction between resource interdependence and summarizer's competence was significant, *b* = −0.20, *df* = 35.47, *t* = −2.07, *p* = 0.05, *R* <sup>2</sup> = 0.13. As illustrated in Figure 1, the relationship between the summarizer's competence and listener's learning was positive when pupils worked on complementary information (*b* = 0.10) and negative when they worked on identical information (*b* = −0.09). As few dyads were available to test our hypothesis, we conducted the same analysis using robust modeling [78]. Results showed a unilateral *p*-value for the main interaction equal to 0.06. Results revealed that the predicted interaction between resource interdependence and summarizer's competence was significant, *b* = −0.20, *df* = 35.47, *t* = −2.07, *p* = 0.05, *R<sup>2</sup>* = 0.13. As illustrated in Figure 1, the relationship between the summarizer's competence and listener's learning was positive when pupils worked on complementary information (*b* = 0.10) and negative when they worked on identical information (*b* = −0.09). As few dyads were available to test our hypothesis, we conducted the same analysis using robust modeling [78]. Results showed a unilateral *p*-value for the main interaction equal to 0.06.

**Figure 1.** Relationship between the partner's competence and listeners' learning in the two interdependence conditions (study 1).

The present study, carried out with elementary school children, built upon the known properties of information distribution and showed the same pattern between information distribution and the summarizer's competence on listeners' learning as Buchs et al. [29] and Buchs and Butera [30] with university students: It indicated a positive link between the summarizer's competence and listener's learning when working on complementary information and a negative link when working on identical information.

Nevertheless, it should be noted that working on encyclopedic texts led to very little interaction between pupils. Indeed, the coding of discussions revealed that listener's

involvement in informational exchange was rare. Of the 23 discussions, only few listeners provided an input or asked questions (including asking for some information or explanation, asking to repeat, and asked questions in order to check their understanding). More precisely, nineteen listeners brought no input at all, two proposed one informational input, one proposed three inputs and the last one proposed four inputs. Fifteen listeners asked no questions, five listeners asked only one question, one listener asked two questions, and two listeners asked three questions. As very few interactive patterns were found, we can underline that discussion on encyclopedic texts was essentially a summarizer's monologue.

In order to prompt more discussions, we proposed that students work on argumentative texts in study 2, and we carefully chose themes that potentially interested children with the classroom teachers. Furthermore, study 2 involved older children (grade 5) in order to try to promote student interactions. Finally, carrying out a second study with fifth graders working on argumentative texts enabled external validity.

#### **4. Study 2**

#### *4.1. Materials and Methods*

#### 4.1.1. Participants

As in study 1, pupils worked on dyads for one session. Twenty-eight pupils regularly attending grade 5 in two Swiss French-speaking classes, 15 boys and 13 girls, aged from 9.67 to 12.58 years (*M* = 10.89, *SD* = 0.58) participated in the study. Teachers composed the dyads in order to obtain a relatively homogenous French level and avoid "enemies" working together. Fourteen students worked on complementary information and 14 on identical information.

#### 4.1.2. Procedure

Dyads were randomly assigned to one of the two experimental conditions regarding information distribution (resource interdependence: independence vs. positive interdependence). The procedure was identical to study 1.

#### 4.1.3. Materials

In order to promote social interactions and discussion, we proposed that pupils work on argumentative texts. We looked for interesting themes for both girls and boys, and after discussion with the teachers, we chose one text on video games and another on dogs as pets. The length was quite similar: 650 words for the text on dogs versus 631 words for the text on video games. Texts were structured in the same way with five positive and five negative arguments in each text. We checked that the specific content of these texts had not been discussed previously in class. We built a lexicon in collaboration with teachers in case pupils asked vocabulary questions.

#### 4.1.4. Independent Variables

As above-mentioned, pupils worked either on identical, or on complementary information with each pupil playing the summarizer's role for one text and the listener's role for the other text. Summarizer's competence was assessed by counting the number of correct informational inputs (main correct information + detailed correct explanations) proposed. A pre-service teacher who followed a course on social interaction in peer learning coded all available discussions. Moreover, a social psychologist acquainted with coding students' interactions during cooperative learning independently coded the same discussions. Both coders were blind to the experimental conditions. The correlation between the two judges was 0.94 for both the number of main correct information and number of detailed correct explanations. We computed the mean of the two evaluations to measure the summarizers' competence (from 5 to 36, *M* = 17.52, *SD* = 8.40).

#### 4.1.5. Dependent Variable: Learning Outcomes as Listeners

Individual learning was assessed at the end of the session by a learning questionnaire for each text consisting of six multiple choice questions (one point for a correct answer and 0 for no answer and for mistakes), two fill-in-blanks questions (coded from 0 to 2 depending on the quality of the answer), and one open-ended question (coded from 0 to 3 points depending of the quality of the answer). Learning scores ranged from 0 to 13 for each text. The questionnaire was developed by pre-service teachers and approved by the two regular teachers. Three pre-service teachers coded each questionnaire, and reached an agreement for each learning outcome. Like in study 1, we identified the role pupils endorsed for each text to compute their performance as a summarizer and as a listener in order to test the relationship between the summarizers' competence and the listeners' performance.

#### *4.2. Results and Discussion*

First, it should be noted that, in this study, fifth graders working on argumentative texts displayed a higher level of interaction compared to fourth graders working on encyclopedic texts (study 1). Indeed, the number of listener's questions (whenever students asked for some information or explanation, asked for repeating, asked questions in order to check their understanding) indicated that in the 27 available discussions, twenty-four listeners brought at least one informational input (*M* = 4.00, max = 13, *SD* = 3.64) and twenty asked for at least one information or explanation (*M* = 2.15, max = 7, *SD* = 2.01). Thus, argumentative texts promote interactions between fifth graders.

With regard to the relation between the summarizer's competence and listener's learning, we conducted multilevel analyses, and built the same two-level model as for study 1. Again, the predicted interaction between resource interdependence and summarizer's competence was significant. As in study 1, we conducted the same analysis using robust modeling [78]. Results showed a unilateral p-value for the main interaction lower than 0.05 for pupils' response as a listener, *b* = −0.24, *df* = 54, *t* = −2.33, *p* = 0.02, and *R* <sup>2</sup> = 0.08, indicating that information distribution moderated the effect of the summarizer's competence. As illustrated in Figure 2, the relationship between the summarizer's competence and learning was positive when pupils worked on complementary information (*b* = 0.11) and negative when they worked on identical information (*b* = −0.13).

Study 2 replicated the predicted dual effect of partners' competence. The expected constructive relation between partners in cooperative learning was only found when pupils worked with complementary information. When pupils worked on identical information, the more their partner was competent, the less they performed.

It is important to note that the theoretical rationale underlining the hypotheses relied on previous research showing that compared with working on complementary information, working on identical information elicited more evaluative pressure [48], more negative reactions ([29], study 1), more perceived competence threat ([29], study 2), and more perceived competition ([50], pilot experiment), whereas working on complementary information reinforced students' cooperation ([29], study 1). Nevertheless, the quality of the relationship between partners was not investigated in study 2. Given the specific age group studied in the present research (elementary school pupils), it is important to assess whether information distribution yielded similar relational effects at elementary school. Thus, we conducted an additional study in order to test the hypothesis that working on identical information elicits perceptions of lower cooperation (here, mutual learning) and higher competitive social comparison than working on complementary information.

information reinforced students' cooperation ([29], study 1). Nevertheless, the quality of the relationship between partners was not investigated in study 2. Given the specific age group studied in the present research (elementary school pupils), it is important to assess whether information distribution yielded similar relational effects at elementary school. Thus, we conducted an additional study in order to test the hypothesis that working on identical information elicits perceptions of lower cooperation (here, mutual learning) and higher competitive social comparison than working on complementary information.

*Educ. Sci.* **2021**, *11*, x FOR PEER REVIEW 10 of 17

**Figure 2.** Relationship between partner's competence and listeners' learning in the two interdependence conditions (study 2). **Figure 2.** Relationship between partner's competence and listeners' learning in the two interdependence conditions (study 2).

#### **5. Study 3 5. Study 3**

*5.1. Materials and Methods 5.1. Materials and Methods*

5.1.1. Participants

5.1.1. Participants Forty-four pupils from two classes, regularly attending grade 5 in a Swiss Frenchspeaking school and aged from 9.92 to 11.68 years (*M* = 10.74, *SD* = 0.35), 21 boys and 23 girls, participated in the study. As in previous studies, pupils worked on dyads for one session. Twenty-two pupils worked with complementary information and 22 worked with identical information. Their regular teacher composed the dyads so that the level in French was relatively equivalent inside the dyad and that pupils could work well together Forty-four pupils from two classes, regularly attending grade 5 in a Swiss Frenchspeaking school and aged from 9.92 to 11.68 years (*M* = 10.74, *SD* = 0.35), 21 boys and 23 girls, participated in the study. As in previous studies, pupils worked on dyads for one session. Twenty-two pupils worked with complementary information and 22 worked with identical information. Their regular teacher composed the dyads so that the level in French was relatively equivalent inside the dyad and that pupils could work well together (no "enemies" together).

#### 5.1.2. Procedure

(no "enemies" together).

5.1.2. Procedure For each class, half of the dyads were randomly assigned to one of the two experimental conditions manipulating information distribution (resource interdependence: independence vs. positive resource interdependence). This study followed the same proce-For each class, half of the dyads were randomly assigned to one of the two experimental conditions manipulating information distribution (resource interdependence: independence vs. positive resource interdependence). This study followed the same procedure as far as working on texts is concerned.

#### dure as far as working on texts is concerned. 5.1.3. Materials

5.1.3. Materials We used the same material (two argumentative texts with a lexicon) as for study 2.

#### We used the same material (two argumentative texts with a lexicon) as for study 2. 5.1.4. Independent Variables

5.1.4. Independent Variables As in previous studies, students worked either on identical or on complementary information and played a summarizer role for one text and a listener role for the other text.

#### 5.1.5. Dependent Variable


experimenter asked to work cooperatively. To address this problem, we introduced a questionnaire saying that we had proposed the same activity to other fifth grade classrooms and recorded the answer of those pupils. We asked the pupils to indicate to which extent they felt the same. Sentences were proposed in the affirmative form and pupils had to indicate whether the same applied to them on a four-point scale ranging from 1 (very rarely) to 4 (very often). We proposed six items, derived from performance-approach and performance-avoidance goals, as both have been shown to be related to competition [79]: "Some pupils wondered how to appear good"; "Some pupils tried to be better than their partner"; "Some pupils wanted to compare themselves with their partner"; "Some pupils were afraid to be less strong then their partner"; "Some pupils were afraid not to explain well information"; "some pupils wondered whether they managed well". When answering the questionnaire, we separated the two pupils composing the same dyad, and read aloud the questions in order to be sure that pupils understood them. We computed the mean score of the six items for each pupil (*α* = 0.54).

#### *5.2. Results and Discussion*

The perception of cooperation and competitive social comparison are independent, *r* = 0.16, *p* = 0.31. As pupils learned in dyads, data are not independent, and therefore we built a two-level model with pupils nested in dyads.

Students reported that they perceived more cooperation when they worked on complementary information (*M* = 3.02, *SD* = 0.87) than on identical information (*M* = 2.40, *SD* = 0.95), *b* = 0.31, *df* = 40, *t* = 2.27, *p* = 0.03, *R* <sup>2</sup> = 0.11. In contrast, participants working on identical information reported more competitive social comparison (*M* = 2.08, *SD* = 0.40) compared to those working on complementary information (*M* =1.78, *SD* = 0.55), *b* = −0.15, *df* = 40, *t* = −2.07, *p* = 0.04, *R* <sup>2</sup> = 0.09.

The third experiment supported the hypothesis that overall working on identical information elicits perceptions of lower cooperation and higher competitive social comparison than working on complementary information. Thus, it seems that working on identical information may reduce perceived cooperation and stress competitive social comparison more than working on complementary information, even for elementary school children.

#### **6. General Discussion**

The very principle of cooperative learning consists in leading students and pupils to help each other with a view to learning together, and in creating a context in which mutual support is favored through positive interdependence. Therefore, in cooperative learning, the partners' competence should be a source of informational support and result in promoting learning outcomes. Nevertheless, research on the effect of information distribution on students' learning in cooperative dyads at university revealed that, even in a cooperative context, working on identical information may elicit competitive social comparison between peers. In particular, and this was the starting point of the present research, the logical positive relationship between partner's competence and students' learning appeared when they worked on complementary information; when they worked on identical information, however, the partner's competence yielded the paradoxical effect of negatively predicting the students' learning outcomes. These results were found both with perceived partner's competence [29] and with the actual (manipulated via the quality of informational input) partner's competence [30]. Research on the pervasive presence of neo-liberal, competitive values in Western societies [26,80] has led to the interpretation of this effect by considering that students socialized in a competitive society and that the educational system may switch very easily to a competitive mode as soon as focus on social comparison is salient in the task—as is the case for working on identical information despite cooperative instruction [81]. The present research studied the generality of this interpretation by testing that, if it is true that competitive societal values have permeated

the educational system [23], the paradoxical results described above should be likely to appear from the level of elementary school.

We tested the relationship between partners' competence (assessed by the quality of informational input on the text they were in charge to summarize) and the partners' learning outcomes for this text in two studies conducted with fourth graders (study 1) and fifth graders (study 2). In both studies, we found an interaction pattern indicating that information distribution moderated the relationship between the summarizer's competence and the listeners' learning outcomes. This interaction pattern has the same shape as the effect obtained by [29] with university students (see Figures 1 and 2), and in the present experiments, it was observed when pupils worked both on encyclopedic texts (study 1) and on argumentative texts (study 2). In line with our hypothesis, partner's competence was positively related to pupils' learning outcomes when they worked on complementary information, whereas this relationship was negative when they worked on identical information. In sum, the results of both studies underlined that even when cooperative learning is accurately implemented, even in elementary school, an element that favors focus on social comparison—like in this case working on identical information may render the partner's competence paradoxically detrimental for learning outcomes. Results regarding students' perception in study 3 supported the hypothesized information distribution properties: Students reported more mutual learning when they worked on complementary information whereas they reported a more competitive social comparison when they worked on identical information. Overall, these results underline that the dual effect of partners' competence may take place from elementary school.

In sum, we propose that the logical positive relationship between the partner's competence and students' learning appeared when they worked on complementary information for two intertwined reasons. On one hand, because each partner accesses different parts, the possibility of social comparison is weak. On the other hand, working cooperatively on complementary information creates an informational dependence because students must rely on what their partner gives them; thus, the reciprocal informational dependence renders the partner's quality of informational input crucial.

In contrast, threatening social comparison may explain the negative impact of partner's competence when working on identical information. When the situation activates social comparison, the partner's competence may elicit a potential threat to self-evaluation. Thus, when working on the same information, partner's performance may become a standard against which students can evaluate their own performance. Such a concern about relative performance has been shown to consume attentional resources because of ruminative thoughts about self-evaluation [82]. The more competent the partner, the more students may be concerned about their own performance. This distraction may interfere with complex information processing, and reduce learning from text. When the partner's competence is low, pressure to attain the same standard is weaker, and students may feel safer, with no intrusive thoughts distracting them from studying and discussing the text.

In addition, students may be guided by strategic information sharing and use in a situation perceived as competitive [83]. When a threatening social comparison is present, students may refrain from asking for information they need for fear of appearing incompetent or clarifying the information the partner needs in order to keep a competitive advantage. Moreover, the threatening social comparison that emerges when working on identical information may orient students toward competitive performance goals. Performance goal orientation may render the interaction with others less constructive [84] and enhance the willingness to exploit the partner's knowledge [85]. Of course, these are speculations and future research should design experiments that may directly test these ideas.

The small number of participants and dyads is obviously a limitation of present studies, which requires caution in the estimation of variance. The three studies were done with convenience samples and at that time, it was not possible to run more classes. Nevertheless, it should be noted that in both studies, the predicted interaction pattern was

observed, and that this was the same interaction observed by [29] with a sample of adults. It should also be reminded that the effects have been tested with different types of texts (encyclopedic and argumentative texts) and two different grades (grades 4 and 5), which speaks to the external validity of the present results. In other words, the limitation due to the small size of the samples is mitigated by the replication of the predicted effect.

A second limitation is that the present experiments were conducted outside the regular classroom setting, which might question the study's external validity. However, even if variations could be expected between experimental and in-classroom settings, with specific dynamics that depend on the specific relation with peers and teacher in regular classes, Marshall and Weinstein [51] pointed out that working on the same task in regular classrooms elicits the same social comparison of abilities that we observed in the present study. Moreover, studies conducted at university revealed the same pattern of results for a single non-class session [29,36,50] and regular workshop sessions [30].

A third limitation is that we did not test mediators. A reliable measure of such concerns with young pupils represents a challenge. In the present research, we relied upon the already known result that resource independence triggers competitive social comparison and positive resource interdependence attenuates it (e.g., [50]) to study the effect of information distribution on the relationship between partners' competence and pupil's learning. Furthermore, our third study confirms that, among our population of young children, information distribution has the same effect on social comparison as that described in the previous research with adults (resource independence elicits a competitive social comparison whereas resource interdependence reinforces pupils' cooperation). As elementary students could feel uncomfortable reporting competition while being asked to cooperate, we decided to conduct the third study as a separate study to investigate the information distribution effect on social comparison. However, a mediational analysis with a reliable measure of competitive social comparison concerns would represent an interesting addition from future studies.

#### **7. Conclusions**

These studies contribute to cooperative learning in two ways. First, they indicate that it is possible that competitive goals permeate the whole educational system, and that learners may feel threatened by their partner's competence, with detrimental effects on learning outcomes, already at the level of elementary school, even when cooperative learning instruction is implemented. Our results suggest that as soon as students can compare each other (when they work on identical information), partner's competence may become detrimental for cooperative learning. Future research may address more directly how threatening social comparisons intervene, and may investigate more generally how the promotion of self-enhancement values such as merit, social recognition, and power may hinder the benefits of cooperative learning; and how, on the contrary, promoting self-transcendence values of universalism and benevolence [80] may counteract these detrimental effects.

Second, the present results contribute to stress how important it is to reflect on the way cooperative learning is structured. The two methods manipulated in these studies varied only on one dimension, namely resource interdependence (working either on identical or on complementary information). However, results indicate that the mere fact of reading the same text, with its potential to generate a focus on social comparison [50], led the partner's competence to yield a detrimental effect on learning outcomes. This implies that it is important to reduce a competence threat when pupils work on identical information so that they can benefit from their partner's competence [54]. Preparing students to cooperate could be a useful preliminary step, so that they learn how to cooperate for learning [47,86].

Our results point out that subtle elements of the situation are sufficient to turn constructive interactions into a competence struggle. We hope that this work may orient future research to increase attention to the way cooperative learning is structured, which may influence its effectiveness.

**Author Contributions:** Conceptualization, C.B. and F.B.; Methodology, C.B. and F.B.; Analysis, J.C. and C.B.; Writing—Original, C.B. and A.D.; Writing—Review & Editing, C.B., A.D., J.C. and F.B. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** At the time of the studies, no approval was needed in Switzerland to conduct non-medical research on human subjects. As stated by the Federal Administration of the Swiss Confederation (https://www.bag.admin.ch/bag/en/home/medizin-undforschung/forschung-am-menschen/entstehung-humanforschungsgesetz.html), the law relating to research on human subjects (i.e., constitutional article n◦118b) came into effect in 1 January 2014. Given this legislation, the present research project was not submitted to any research ethics board. Each study was approved by the research commission in schools in partnership with the university and Genevan department of public instruction

**Informed Consent Statement:** As the participants were children, we asked for permission to (1) the head of school, (2) the teachers.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to the confidential information.

**Conflicts of Interest:** The authors declare no conflict of interest.

**Ethics Statement:** According to the regulations of the country where the study was carried out, approval from an ethical board was not required when data were collected.

#### **References**


## *Article* **The Transformative Role of Peer Learning Projects in 21st Century Schools—Achievements from Five Portuguese Educational Institutions**

**Ana Raquel Carvalho \* and Carlos Santos**

Digimedia, Department of Communication and Art, University of Aveiro, 3810-193 Aveiro, Portugal; carlossantos@ua.pt

**\*** Correspondence: raquelcabralc@ua.pt

**Abstract:** Rethinking the role of education in the 21st century implies acknowledging the power of learning and the urgency of making learning provision more meaningful, inclusive, and studentcentred, which assumes particular importance when learner disengagement is still a global issue in elementary and secondary education. Rooted in social constructivism principles, peer learning is a learner-centred approach that facilitates the development of soft and technical skills, with evidencebased contributions to learners' academic performance under the cognitive, affective, and social dimensions. This study aims to find evidence of the transformative role of peer learning projects in four Portuguese secondary schools and a higher education institution through teachers and peer teacher students' (PTS) perceptions of these projects' purpose, implementation, and impact on the educational community, particularly on PTS. Data were collected by means of a semi-structured in-depth interview and a survey by questionnaire, and content analysis and descriptive statistics were the techniques used. Results show cooperation and interpersonal skills' improvement as major strengths of these projects, whereas the challenges are mostly organisational, e.g., reduced teacher service time and coordination of learners' schedules. Conclusions highlight the potential of peer learning projects to promote pedagogical transformation and innovation in 21st century schools.

**Keywords:** peer learning; innovative practices; pedagogical transformation; 21st century skills; mixed methods

#### **1. Introduction**

Dealing with 21st century challenges from an educational viewpoint implies considering the power of learning and the key role of learning opportunities in individuals' life journeys [1]. As mentioned by Fullan, Quinn, and McEachen [2] (p. 5): "The new set of crises is forcing humankind to reconsider its relationship to each other, and to the planet and universe: it is essential that we proactively change the world through learning". According to the authors, in order to become deep, learning should be meaningful, happening in relation with others and giving learners the chance to find their purpose, develop abilities, and transform their own realities.

Student dissatisfaction toward school is a global issue yet to be solved [3–8], which might be a symptom of the still existing gap between current educational practice and the needs of 21st century learners and, as stated by González-Rodriguez, Vieira, and Vidal [4] (p. 214), when talking about early school leaving, may be "perceived to be an inefficiency of educational systems". According to Eurostat data on "early school leavers from education and training" referring to 2019 [9], the early school leaving average in the European Union (EU) is 10.2%, close to the goal of 10% or below, settled for EU countries by 2020 [10]. Based on the same data, Portugal is one of the member states registering "the largest reductions ( . . . ) between 2014 and 2019 in the proportion of early leavers" [9] (p. 2), having moved from 17.4% in 2014 to 10.6% in 2019, despite still being slightly above the EU target of 10%.

**Citation:** Carvalho, A.R.; Santos, C. The Transformative Role of Peer Learning Projects in 21st Century Schools—Achievements from Five Portuguese Educational Institutions. *Educ. Sci.* **2021**, *11*, 196. https:// doi.org/10.3390/educsci11050196

Academic Editor: Robyn M. Gillies

Received: 30 March 2021 Accepted: 19 April 2021 Published: 22 April 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

According to an OECD report on PISA results for Portugal [11] (p. 5), "many students, especially disadvantaged students, hold lower ambitions than would be expected given their academic achievement", which, in the case of Portugal, is still evident, with "one in four high-achieving disadvantaged students", as opposed to "one in thirty high-achieving advantaged students", not expecting to finish tertiary education [11] (p. 5). "Higher risks of social exclusion and lower civic engagement" [12] (p. 26) and "considerable difficulties in the labour market" [9] (p. 1) are reported to be linked to early leaving from education and training [4,12,13], and according to the European Commission [10] (p. 11), there is a correlation between "better educational achievement" and "more active civic participation", which, in the case of countries like Portugal, is reported to be "even more pronounced".

In a world where "scoring high on foundational subjects will not be enough to be competitive" [14] (p. 407), with "entrepreneurial, social and civic competences" [15] (p. 1) being as important as technical skills [16], and with the introduction of collaborative problem solving and learning with digital tools as basic competences for the demands of the upcoming decades [17], transforming education requires deep reflection, flexibility, and improvement of the mechanisms used to identify and assess new ways of learning [18] (p. 146). Voogt, Erstad, Dede, and Mishra [14] (p. 403) referred to the "shortage of creative and innovative workers", and the Council of the European Union [15] (p. 2) reports "a constant high share of teenagers and adults with insufficient basic skills", which, together with the acknowledgement of the growing but not always appropriate integration of digital technologies in contemporary education and learners' lives [7,18,19], might substantiate Erstad's position [3] (p. 76) that "new models of learning and knowledge creation are needed to prepare young people for their future work and citizenship".

The standardisation of learning derived from mass education principles still present in formal education settings, based on which academic achievement strongly relies on testing and reinforces competition [3,8,20]; the prevalence of a "top-down" organisation of "contemporary Western standards" of learning [21] (p. 32); as well as the "pedagogicisation of young people's everyday life" [3] (p. 76), based on excessive focus on academic achievement rather than on learners' interests, backgrounds, and identities, all substantiate Erstad's position [3] (p. 61) that there has been "a lack of understanding about the dynamic processes of learning as part of people's lives". According to the same author, it is essential to "look beyond school" [3] (p. 65) and find in engaging and motivating examples of informal learning environments, such as in community-based initiatives or, as noted by Pereira, Fillol, and Moura [7] (p. 47), simply through young people's informal ways of accessing knowledge "in their leisure time, in digital platforms, in peer communication", the strategies that might contribute to making learning meaningful. Although school is not the only place where learners actually learn, Miño-Puigcercós [6] stresses the central role that school has in learners' lives, which should validate the promotion of alternative ways of engaging learners and favour the strengthening of the bonds between their knowledge, their interests, and the new experiences that school should provide for them.

Rooted in social constructivism principles [22,23], peer learning is a student-centred approach that gives teachers and learners the chance to experience new roles, privileging the human essence of education based on personal interactions [18,24–27]. With evidence-based benefits under the cognitive, but especially affective and social, dimensions [23,28–31], peer learning is in line with Dewey's [32] (p. 46) vision of learning "as an active and constructive process", and Vygotsky's principle of autonomous but scaffolded access to knowledge through the "zone of proximal development" [33] (p. 1), based on which both peer teacher students (PTS) and peer learners (PL) are given the chance to be co-constructors of their learning process [3,25,28,34]. Over the past few decades, improvements in peer learning delivery [30] have contributed to its popularity, particularly in cross-level programs in higher education [22,23,28,29], and justify the presence/emergence of different peer learning varieties, such as peer tutoring, and cooperative learning [24], "the longest established and most intensively researched" [30] (p. 632)—peer teaching [23], peer mentoring [30,35], peer-assisted learning [36], or peer instruction [37]. As stated by

Topping [30] (p. 633), at least 13 organisational dimensions can influence peer learning delivery (e.g., the context; participants' features, such as age, year of study, ability, and role; curriculum content; objectives; assessment of students; and characteristics of the program, such as being voluntary or compulsory). According to a meta-analysis by Balta, Michninov, Balyimez, and Ayaz [28] (p. 67), "learning with and from peers is not always effective", and in order to be successful, peer learning requires planning and structure. Topping [30] (pp. 631–632) distinguishes between the most "archaic perceptions of peer learning", where only the best students were given the chance to transmit their classmates the knowledge instructed by teachers, and more recent trends, according to which "peer helping interaction is qualitatively different" [30] (p. 632), with PTS being cognitively closer to PL and where both "feel equally valuable and worthwhile", being "active participants in the learning process" [30] (p. 643). In order to become deep, learning occurring within the scope of peer helping interaction should result from "reflexive knowledge building" situations, promoted by peer discussion, reflection, and PLs' scaffolded access to knowledge, and not from "knowledge telling" scenarios [22] (p. 3). Simultaneously, by revising information, monitoring, and correcting, PTSs are given the chance to practice cognitive tasks such as summarising, questioning, classifying, and inferencing, considered vital in the development of PTSs' critical thinking and autonomy [29,30]. According to Topping [30] (p. 638), "the greater the differential in ability" between PTS and PL, "the less cognitive conflict and the more scaffolding might be expected".

Within the context of elementary and secondary education, despite the lower number of existing studies, peer learning is considered for its potential to combat early school leaving rates, promoting low-risk environments, and helping learners with behavioural problems to change their attitudes towards school through assuming new roles and restoring their sense of belonging to their educational community [38–40]. However, based on a literature review, more evidence is requested on the effectiveness of peer learning programs [40] as well as on their impact on learners' academic performance [29,35,41], particularly in the case of PTS, since most studies focus on the contributions of peer learning delivery to PL [35,42]. The complexity of isolating variables in the teaching and learning context and identifying cause–effect relationships, as well as the limited design of most studies, very frequently associated with quasi-experimental methods, are pointed out as possible reasons for this [29,41]. Besides this, Marshall, Dobbs-Oates, Kunberger and Greene [35] also report the need for more studies involving peer mentoring programs of different institutions and disciplines, at least in higher education, as well as how important it is for educational institutions to understand the challenges and benefits experienced by PTS when the goal is to run effective peer mentoring programs. Grounded in the circumstances imposed in school year 2019/2020 due to the COVID-19 pandemic, the Portuguese Ministry of Education established a set of guidelines for elementary and secondary schools for school year 2020/2021 [43], with the purpose of supporting learners to restore their learning processes and promote pedagogical innovation. Among the strategies determined in the guidelines is the implementation of cross- and/or same-level peer learning programs by each school, according to which volunteering students become PTSs of their schoolmates, "helping them develop skills, clear doubts, integrate at school, and prepare for assessment tasks" [43] (p. 46). According to the same document [36], the personal, interpersonal, and academic skills development promoted by peer learning, as well as its principles, are in line with what is expected from school and also mirrored in the document released by the Portuguese Ministry of Education with the skills expected from students when they finish compulsory education [44].

Based on learner engagement issues and inequalities, especially in elementary and secondary education in Portugal, on the need for more meaningful and innovative pedagogical approaches as key elements to promote learner engagement, on the potential of peer learning to promote learners' ability to be co-creators of their learning process, but also on the need for more studies focusing on peer learning contributions to PTSs' academic performance as well as involving programs from different institutions and disciplines,

this study aims to find evidence of the strengths and challenges of peer learning delivery in four Portuguese basic and secondary schools and a higher education institution and, through the conclusions drawn, assess its transformative role in 21st century schools. Eight teachers in charge of five peer learning projects and 63 peer teacher students (PTSs) from those projects were surveyed on motivational aspects, human and organisational variables, and participation impact. The sample included teachers and PTSs due to their leading and active roles in the projects. Through a mixed-methods approach, qualitative and quantitative data, collected between December 2018 and January 2019 with the support of a semi-structured in-depth interview and a survey by questionnaire, were analysed by means of content analysis and descriptive statistics. The ultimate goal of the study is to shed light on the transformative role of peer learning programs in schools as well as encourage and support the implementation of similar bottom-up initiatives, improving teachers' experience at school and learners' engagement, inclusiveness, and empowerment.

#### **2. Materials and Methods**

#### *2.1. Purpose of the Study*

This multiple case study integrates one of the stages of a research project designed within the scope of educational design research, with the purpose of finding evidence of the strengths and challenges of peer learning delivery in Portuguese schools and, based on the conclusions drawn, informing the development of a prototype of a peer learning program, expected to be an innovative solution to support teaching and learning practice in elementary and secondary education.

Within the scope of this paper, analysis of peer learning delivery in the case of the five projects is expected to answer the following research questions: What was the purpose of the projects?; How were they implemented?; What were the strengths and challenges of project implementation, according to the perceptions of the teachers and peer teacher students who took part in the inquiry?

#### *2.2. Sample*

Purposeful sampling was the technique used to select the five peer learning projects that constitute this study, based on the following criteria: (i) current/recent implementation in Portuguese educational institutions; (ii) inclusion of cross-level peer learning programs; (iii) voluntary participation of PTS; (iv) evidence of the project outcomes for the learners involved and the educational community. Three of the projects were selected based on news articles reporting their positive impact on the schools in which they were held, one was previously known by the researchers, and another was chosen for convenience due to being implemented by one of the researchers in the corresponding institution. Four of the projects were held in basic and secondary schools and one in a higher education institution. The five institutions were located in four Portuguese districts, namely Vila Nova de Gaia (1 project), Aveiro (2 projects), Santarém (1 project), and Leiria (1 project).

A total of eight teachers and 63 PTS were surveyed, as listed in Table 1. Peer learners (PLs) were not included as part of the study sample, based on the fact that they were not involved in project organisation. As for the teachers, seven worked in basic and secondary schools, among which two assumed non-teaching roles (i.e., a librarian teacher and an educational psychologist), and one lectured in a higher education institution. The commonality between the eight teachers was essentially the fact that all were in charge of the peer learning projects in the corresponding educational institutions and, based on their experience, would be expected to provide valuable insights into the human, pedagogical, and organisational matters of the projects. Regarding PTS, 68.2% were attending upper-secondary education when they participated in the projects: in projects D and E, both scientific–humanistic and scientific–technological courses (42.9%); in project A, only scientific–humanistic courses (9.5%); and in project B, only scientific–technological courses (15.9%). Project C involved former students, in the case of PTS (31.7%), and PLs attending the curricular unit of "Multimedia Laboratory 4" (LabMM4), which integrates the curricu-

lum of a bachelor's degree course on new communication technologies at a Portuguese higher education institution. PLs were attending elementary and/or lower secondary education in four of the projects: elementary education (projects B and E); and lower secondary education (projects A and E). The age of 68.2% of the PTSs ranged from fourteen to eighteen years old (projects A, B, D, and E), and of 31.7% from nineteen to twenty-six years old (project C). The sample included thirty-six female (57.1%) and twenty-seven male (42.9%) PTSs. Among PTSs, the major differences stand between those belonging to 4 of the projects (A, B, D, and E), attending basic and secondary education, and those assuming their role within the context of higher education (project C). This fact is also echoed in the two different age groups represented in the sample, according to which 68.2% of the PTSs are teenagers, as opposed to 31.7% who are young adults. Based on a preliminary data analysis, no significant discrepancy was found between the perceptions of older and younger PTSs regarding similar matters, which substantiates the fact that the impact of PTSs' age group differences was not further assessed within the scope of the study. Teachers and PTSs signed a declaration of informed consent, which, in the case of students under eighteen years old, was done by their legal representatives, to enforce the applicable law regarding participants' confidentiality and anonymity.



Own source.

#### *2.3. Data Collection Tools and Data Analysis*

A semi-structured in-depth interview, to be applied to the teachers, and a survey by questionnaire, to be answered by PTSs, were created and validated for the purpose of this study. Data collection took place from December 2018 to January 2019.

The interview was segmented into three parts, namely the purpose of the projects, human and organisational variables, and results, and was intended to promote teachers' reflection on project implementation aspects as well as on the strengths and challenges resulting from it. Teachers were interviewed individually in the schools in which the projects were developed. The content was recorded, transcribed, and treated under content analysis with the support of qualitative data analysis software WebQDA.

The survey by questionnaire was created on Google Forms and completed by PTSs online. It was segmented into three parts, namely socio-demographic data, initial motivation, and participation in the project, including fifteen closed-ended questions and eight open-ended questions. The survey was intended to identify the profile of the PTSs involved, their motivations to voluntarily integrate the projects, and finally their perceptions of participation in the whole experience. Content from the open-ended questions was analysed under content analysis, also with the support of software WebQDA, and quantitative data were analysed with the support of SPSS by means of descriptive statistics.

Within the scope of this study, teachers' views covered a wider number of variants, whilst PTSs' perspectives were highlighted regarding their reflections about participation in the projects. Although predominantly qualitative, the mixed-methods approach of the study was intended to facilitate triangulation of teachers and PTSs' perceptions of similar variables and, as mentioned by Fraenkel, Wallen, and Hyun [45], regarding the advantages of qualitative and quantitative data combination, contribute to the validity of the study.

Table 2 provides an overview of the interview content in focus, based on the established dimensions, categories, and subcategories resulting from content analysis.


**Table 2.** Overview of interview corpus selected categories and subcategories of analysis.

Own source.

Table 3 lists selected open-ended questions from the survey by questionnaire to be paired with categories "peer teacher students" and "project implementation" of the interview, and also establishes a parallel with a closed-ended question of the same survey, selected to promote data triangulation.

**Table 3.** Open-ended and closed-ended question pairing (survey by questionnaire).


### Own source.

#### **3. Results**

#### *3.1. Purpose of the Projects*

Most projects (A, D, and E) were created with the purpose of "tutoring disadvantaged students and/or providing study support to learners in need". "Preventing early school leaving" was the second most mentioned purpose (B and E). Interestingly, in the case of project B, "early school leaving" was considered regarding PTS attending scientific– technological courses, corresponding to vocational training study programs, and, in the case of project E, regarding PL, as noted in the following comments:

"We always tried to motivate learners and decrease early school leaving rates, making them feel valued and belonging . . . something students from Scientific-Technological courses report is that they feel a little lessened compared to students from Scientific-Humanistic courses. So, we tried to make them realise things could be different and, by creating bonds with us, teachers, give them the selfconfidence they needed for peer learning sessions"

Teacher 7 reported the following:

"In the beginning, the project was considered for supporting learners at risk of early school leaving; the school Director integrated a group that visited several schools and realised many students did not have any support at home to study or do their homework. And this was something that disturbed us and has increased over time"

(T7).

Teacher 5 mentioned two complementary yet opposite purposes, considering both students with learning issues and more proficient learners, namely "increasing the attractiveness level of the subject" and "challenging learners to learn more":

"In a certain way, it was the high retention rates of LabMM4, probably the second curricular unit with more students having more difficulties . . . lacking foundational knowledge of programming and many of them having failed it more than once before. Something had to be done, but obviously without sacrificing the high standards of the subject . . . at the same time, another interesting thing to explore was 'what can challenge proficient learners to learn beyond what is addressed in classes and not leave anyone behind' . . . "

(T5).

In the case of project B, "accepting an external proposal" was another of the purposes mentioned. In this case, it was a regional Centre of Competence, based on a partnership established with public and private schools, with the intent of subsidising equipment acquisition.

Regarding their nature, four out of the five projects were defined for their humanist and humanitarian nature and two of them also for their motivating and innovative character. Teacher 3 highlighted: "Simply motivating! We were able to create a network within our educational community, involving different stakeholders, and it was inspiring to everyone, thus being so enriching". Teacher 5 associated pedagogical innovation with the new roles assumed by the teachers and PTSs: "The challenge had very much to do with, not losing sight of project design and scientific validation, but giving PTS the chance to take responsibility for as many decisions as possible and implement challenges based on their own ideas".

Overall, the peer learning programs implemented in schools A, B, D, and E were in line with peer tutoring principles, where PTSs helped PLs to restore and improve their skills based on the scaffolded progress provided in all subjects of PLs' curricula, except for project B, according to which the support was centred on a specific subject of PLs' curricula, namely "Robotics". In the case of project C, the peer learning program took place not only within the scope of a specific curricular unit of PLs' syllabus, namely LabMM4, but also included challenge-based and project-based features, according to which each peer learning session (PLS) was implemented based on storytelling and role-playing techniques, and its organisation involved previous collaborative work between PTSs regarding logistical, communication, and pedagogical matters, under the teacher's supervision.

#### *3.2. Project Organisation and Implementation*

In general, these projects were created based on partnerships, in the case of projects D, B, and E, correspondingly, between the school board, one/two teachers and an existing Nucleus of Citizenship & Personal Development with years of practice and expertise gathered at school or external entities such as a Competence Centre or a Creative Learning Foundation. In the case of project A, the school psychologist, and of project E, the librarian teacher, were, and continued to be, in partnership with other teachers, key elements in creating and implementing the projects. Most projects (B, C, and D) have been implemented since 2016/17 or later and only two (A and E) since 2010/11 or later.

As for institutional support, the encouragement and availability of school boards was unanimously highlighted, and in the case of projects B, C, and D, teachers also underlined

that supporting volunteering initiatives and/or technologically enhanced pedagogical solutions was part of the institutional policy. However, teachers from the five projects reported not having specific service time on their schedules allocated for these peer learning projects, except for two hours of the non-lecturing component (A, D, and E), which was also mentioned as not being sufficient for the management and supervision of the projects. In the case of project E, the librarian teacher transferred six hours allocated for library projects to this peer learning project.

Regarding the way that the remaining teaching staff welcomed the projects, besides curiosity, support, and recognition, there was also some doubt and fear, especially deriving from giving PTSs so much responsibility: "Some teachers immediately supported the idea, but others said I would probably regret giving PTS so much responsibility . . . it was too risky" (T5); "In the beginning there was some fear of whether it was going to work or not, but as soon as results emerged, which was also obvious in learners' attitudes, the community surrendered..." (T7).

As regards recognition of PTSs' volunteering work, projects D and E were the ones where participation in the projects was awarded with a school honor roll diploma at the end of the school year.

Table 4 provides an overall picture of peer learning session (PLS) delivery per project, mostly based on operational variables.


**Table 4.** Peer learning session implementation.

Own source.

Regarding location, the option of having no fixed room for PLS was ultimately connected to preserving the relaxing atmosphere for the sessions or to the sense of openness and proximity between the institution and the students involved: " . . . we intend to promote a relaxing atmosphere, where learners feel at ease, so they might search for familiar locations: the library or the recreation room, where they also find desks and in quieter moments can talk . . . " and proximity between the institution and the students involved: "… we intend to pro-

[T1].

As for frequency and duration, most PLS happened on a weekly basis, in PTSs' free time, usually in the afternoon, and lasted between 45 and 90 min. PLS of project C could take up to 2 h 30 min, depending on group performance, and happened at night. In terms of work dynamics, most projects (B, D, and E) relied on small group interactions, with one PTS for two or three PLs. As for frequency and duration, most PLS happened on a weekly basis, in PTSs' free

Regarding teacher supervision, in PLS, teachers were usually not present, and those on-site took a non-participatory though supportive role. According to the teachers, the most significant challenges that PTSs had to deal with were personal insecurities, impact on PLs' attitudes, and availability over time. on PLs' attitud

#### *3.3. Outcomes of Project Implementation*

#### 3.3.1. Strengths and Challenges

moments can talk…" [T1].

Based on teachers' and PTSs' answers to the open-ended questions on the strengths and challenges deriving from participation in and implementation of the projects, Figure 1 provides an overview of the variables that more teachers and PTS referred to regarding the projects' strengths and the major challenges dealt with. Based on teachers' and PTSs' answers to the open the projects' strengths and


Teachers and peer teacher students' perceptions of project implementation o **Figure 1.** Teachers and peer teacher students' perceptions of project implementation outcomes (own source).

The following extracts from teachers' comments exemplify some of the indicators in-The following extracts from teachers' comments exemplify some of the indicators included in Figure 1:

"When they work and ask each other for help, when they assume others know it better, they are collaborating with each other."

[T7]

"The whole experience shows that also PTS who did not score so high in tests, here have the opportunity to shine, and they do because they are available, and because of their attitude and commitment. All these components are assessed, so they see their academic results improve."

[T1]

"One of the PTS once told me, almost disappointed, he was going to work with a Romani PL . . . after some time, the same PTS reported the close and affectionate bond both were creating . . . I know he will never look at the Romani community the same way . . . "

[T6]

In the case of PTSs' comments, the following extracts stood out:

"It was so enriching . . . helping others and finding new ways of explaining things helped me strengthen my own weaknesses."

```
[PTS, project C]
```
"The PL I worked with improved his academic results, started to have study habits and it was good to me as it has helped me improve my self-esteem."

[PTS, project A]

"I got a clearer perception of what teamwork involves."

[PTS, project B]

"I had the chance to create bonds with children and I loved it. I became more organised and it was really nice to feel that an action can change someone's life, including mine."

[PTS, project D]

#### 3.3.2. Participation Impact on Peer Teacher Students' Academic Performance

As regards PTSs' perceptions of participation impact on their academic performance, data collected from the selected closed-ended question: Q.4, "Assess the impact of your participation in the project within the scope of the items listed below. Choose from 'really improved' to 'regressed'—focused on variables "motivation for learning"; "self-confidence"; "sense of belonging to the group"; "sense of belonging to school"; "collaborative skills"; "communication skills"; "knowledge mastery"; and "leadership skills".

Results related to the eight variables assessed are shown in Figure 2. According to this figure, "collaborative skills" is the most impacted variable, with the highest score under both options "really improved" (29%) and "improved" (59%). "Sense of belonging to the group" (24%), "to school" (19%), and "motivation for learning" (19%) were the second and third variables most marked as having "really improved". Furthermore, PTSs also considered that their "self-esteem" (59%), "communication skills" (56%), and "motivation for learning (54%) were the variables with the highest scores for having "improved". Inversely, variable "leadership skills" was considered by more PTSs (41%) as not having changed, although 48% still marked it as having "improved" and 8% as having "really improved". Overall, the impact of participation in the projects was significantly positive, with all variables being scored higher for having "improved" over time.

Peer teacher students' perception **Figure 2.** Peer teacher students' perceptions of participation impact on their academic performance (own source).

Furthermore, in regard to the above-mentioned data, a binomial test was performed with the support of SPSS based on the following hypothesis:

$$H\_0: p = 50\% \tag{1}$$

$$H\_1: p > 50\% \tag{2}$$

The cut-off point selected was level 2, splitting the sample into two groups:

$$\text{Group } 1 \, \text{:} \, \text{:} \, \text{:} \, \text{:} \, \text{:} \, \tag{3}$$

("did not change"/"regressed") ("did not change"/"regressed")

$$Group\,\,2:\,\,\stackrel{\}}{\colon}\,2\,\tag{4}$$

variables "leadership skills" and "knowledge mastery" ("really improved"/"improved")

("really improved"/"improved")

according to PTSs' perceptions, participation in the projects had t "leadership skills" and "knowledge mastery" as having "really improved" or "improved" over time. Figure 3 provides the binomial test results and, according to them, only in the case of variables "leadership skills" and "knowledge mastery" were the *p*-value results higher than = 0.05, which means that for the remaining variables, the *p*-value results show statistical significance. Based on this, it can be inferred that there is statistical evidence that, according to PTSs' perceptions, participation in the projects had a significant impact on their academic performance, since all variables assessed, except "leadership skills" and "knowledge mastery", were rated by more than half of the PTSs as having "really improved" or "improved" over time.

2: ≥ 2

3.3.3. Recommendations to Prospective Participants of Peer Learning Projects

Finally, recommendations highlighted by the teachers as to requisites for peer learning project implementation focused on the professional skills that teachers require to manage and supervise peer learning projects, namely "commitment, availability, and tolerance";

human and interpersonal skills like "working in multidisciplinary teams", "creating networks", "giving PTS as much responsibility as possible"; and also organisational variables like "planning short PLS", "promoting interdisciplinarity", "efficiently managing learners' schedules"; and "having institutional support".


**Figure 3.** Binomial test results. (own source).

tion" to be consistent over time.

age and supervise peer learning projects, namely "commitment, availability, and to ance"; human and interpersonal skills like "working in multidisciplinary teams", "creat-In the case of recommendations given by PTS to prospective PTS, the majority referred to the importance of personal and behavioural requirements like "being calm and patient", "relaxed", "being committed and available"; as well as humanist and humanitarian aspects like "enjoying helping others" and "focusing on the benefits of participation" to be consistent over time.

#### ing networks", "giving PTS as much responsibility as possible"; and also organisational **4. Discussion**

variables like "planning short PLS", "promoting interdisciplinarity", "efficie ing learners' schedules"; and "having institutional support". ferred to the importance of personal and behavioural requirements like "being calm and Based on the study results, it may be inferred that to be successful, peer learning project-based initiatives depend on integrated coordination between the school board, the teachers involved, and the remaining teaching staff. Other elements of the pedagogical

staff holding a more flexible schedule, such as educational psychologists and/or librarian teachers, are welcomed for the support provided as well as the possible allocation of extra service time for such initiatives, as happened in projects A and E. According to the European Commission's report on "recommended instruction time in compulsory education" [46] (p. 134), "schools may manage up to 25% of the curriculum in a flexible way" and, in this way, develop curricular enrichment activities. As a matter of fact, the major challenges reported by teachers in this study focused on organisational aspects regarding the need for more teacher service time provision for the projects and for better planning to facilitate the management of PTSs' and PLs' schedules. Based on the reported guidelines [46], schools currently have more tools to provide these projects with better conditions, so that peer learning implementation does not solely depend on the volunteering commitment of hardworking teachers.

As stated by some of the teachers, another challenge that they faced was the reluctance of other teachers to believe in learners' capacity to assume a new role that required so much responsibility and autonomy. Bearing in mind the significance of learner-centred approaches in 21st century schools [24,25,27,28,30,31], and the key role of teachers in reducing the gap between learners' needs and current educational practice [3,6,47], it is mandatory that teacher training meets the standards of 21st century educational scenarios [6,7] and supports teachers to (re)signify the meaning of teaching and learning in the academic community [1,18].

According to the study results, most peer learning sessions happened on a regular basis, which, on the one hand, reinforces affective bonds and promotes learners' scaffolded progress, in part also due to the distinguishing role of ICT tools based on their ubiquity, immediacy, and effective promotion of collaboration and communication between PTS and PL [48], but on the other hand requires consistent effort, commitment, and availability over time, which, for some PTSs, may be a challenge, as stated by the teachers. However, based on PTSs' answers, the major challenges reported were related to effectively performing their new role, mostly regarding communication and choice of appropriate pedagogical strategies, and this is in line with the teachers' opinion that dealing with "personal insecurities" was probably PTSs' major challenge. Curiously, by comparing the results, the same PTSs reported "the improvement of teaching and learning skills", when answering the open-ended question on the main benefits of participation, and registered major development of their "collaborative" and "communication skills" (as stated in Figure 2), when answering the selected closed-ended question on participation impact on their academic performance. Only a very small percentage of PTSs reported a regression in most of the skills in focus, having associated PLs' lack of commitment as the main challenge they faced, and the personal development and satisfaction deriving from helping others as the main benefits of the experience. All this may indicate that independently of age group differences, PTSs' perceptions tended to converge and that, despite the demanding tasks of PTSs' new roles, most students realised what effective peer helping interaction requires [29,30] and were able to overcome insecurities, adapt, and assume an active and constructive attitude, as recommended by Binkley, Erstad, Herman, Raizen, Ripley, Miller-Ricci, and Rumbler [17] (p. 32), when authors associate "sophisticated thinking, flexible problem solving, and collaboration and communication skills" as the "new standards for what students should be able to do", "to be successful in work and life" and based on which schools must promote transformation. Simultaneously, and based on Gillie's [24] remarks on how to effectively promote successful cooperative group work, it is essential to provide PTSs with the skills and support that may help them to internalise key elements described by the author, for promoting "successful cooperation" (p. 3), namely ensuring "positive interdependence", based on which learners develop their sense of belonging to the group and influence within the group, "individual accountability" [24] (p. 3), improving PTSs' ability to "actively listen to others, ( . . . ), constructively critiquing the ideas of others, sharing resources, taking turns" (p. 4), effectively promoting "interactions", and finally "group processing" or reflecting on achievements and challenges to be further improved

(p. 4). When researching into the effectiveness of peer learning programs and their layout, it is therefore relevant to assess how PTSs are prepared for the role and how this impacts the way that they perform this role.

Besides this, the reported development of "personal and interpersonal skills", both by teachers and PTSs, as major strengths deriving from participation in the projects, also substantiates the position of authors who state the emotional and social nature of human beings [6,26,49,50] and the critical impact of emotions on cognition [49]. Together with skills like creativity and critical thinking, all these assumptions contribute to a comprehensive perception of what learning means, and, as highlighted by Erstad [3] (pp. 63–64): "By combining 'knowing' and 'becoming' ( . . . ) we open up a more dynamic understanding of learning". Based on this, it may be inferred that, despite including voluntary PTSs who, for that reason, were willing to participate in the projects and were less likely to have engagement issues at school, the five projects gave learners from different course backgrounds the chance to strengthen their affective and social bonds, clearly expressed in PTSs' positive perceptions of participation impact on their sense of belonging to school and to the group, but also on their self-esteem and motivation for learning. These facts show these projects' potential to captivate different learner profiles, including those of students with higher risk of social exclusion and learning engagement issues, giving them the opportunity to feel valuable and to restore their relationship with school [6,38–40], their appreciation for learning, and after all to be used by 21st century schools as a bottom-up pedagogical solution to promote learner engagement and combat early school leaving.

#### **5. Conclusions**

According to the study results, most of the peer learning projects analysed show similarities regarding their purpose, organisation and implementation features, and impact on the corresponding participants and educational communities, especially when sharing the same educational context, as happened with four of the projects, held in basic/elementary and secondary schools. Coincidently, major differences were reported between these and the project implemented in higher education, specifically regarding the peer learning principles of the program and the role given to PTSs, whose tasks/responsibilities are, within this scope, more student-centred but also require more autonomy and availability over time, which seems to be one of the challenges that PTSs have to cope with in secondary education, especially when such project-based initiatives are extra-curricular. As for the achievements, no significant differences were found between the perceptions of teachers and PTSs' in the five projects, which highlights the unanimity of the benefits reported within this scope.

Based on the study results, its contributions are expected to be of particular interest to researchers, educational leaders, teachers, and the educational community in general. By analysing human, pedagogical, and organisational variables of peer learning projects implemented by different educational institutions, involving learners from distinctive courses and disciplines, the study provides comparative results that are expected not only to consolidate findings but also to add insights into the establishment of guidelines for peer learning delivery [40], particularly in basic and secondary education. Besides this, study results on PTSs' participation impact on their academic performance, based on these students' perceptions of the strengths and challenges experienced and complemented by the teachers' views on the same matters, is expected to expand knowledge of a peer learning area reported in the literature for being "less studied" [35] (p. 1), and contribute to providing clarity on how to effectively organise and implement peer learning programs, bearing in mind not only PLs but also PTSs' gains resulting from it [29,41]. The choice of a mixed-methods approach that combines the richness of qualitative data with the validity provided by quantitative data [45] is also expected to be an alternative to methods reported in the literature for not being the most effective when the purpose is to analyse educational variables and identify cause–effect relationships [29,41]. Contributions of the study are expected to be particularly useful for teachers and educational institutions that might

be interested in using peer learning project-based initiatives as a bottom-up solution to implement more meaningful and innovative pedagogical approaches and promote learner engagement, inclusiveness, and empowerment. It may hence be inferred that, according to the study findings, the peer learning projects in focus have promoted the development of what Voogt, Erstad, Dede, and Mishra [14] (p. 407) call "key areas" of 21st century curriculum, namely "foundational", "meta", and "humanistic knowledge", providing all learners with the opportunity to find their place at school [6], engage "in what matters for them and their communities" [3] (p. 67), and play a transformational role in the schools to which they belonged.

As for the limitations of the study, despite including five educational institutions that, especially in the case of basic and secondary schools, may be considered representative of the corresponding educational context in Portugal, due to the small size of the sample, the findings may not be generalisable. In addition, further investigation would be needed to complement the findings for the evidence of peer learning contributions on learners' academic performance, especially regarding the development of cognitive and metacognitive skills. In terms of methods, including PLs in the study sample and assessing their own perceptions of participation in the projects, particularly regarding the main challenges and benefits resulting from it, would promote triangulation of the data gathered and more robust confirmation of findings. Although it would be impossible to conceive it with all learners included in the sample, using the focus group technique with a reduced but representative number of PTS selected from the five projects would be a valuable way to add in-depth understanding of these learners' perceptions of the matters in focus. Future research on peer learning project-based delivery in similar contexts in other countries, including longitudinal assessment of PTSs' and PLs' satisfaction toward learning, may be a valuable contribution to complementing findings on the challenges and strengths of peer learning delivery and support its widespread use in more educational scenarios. Besides this, based on the pedagogical innovativeness of peer learning, research on the strengths and challenges of peer learning deployment during the COVID-19 pandemic, either as a blended learning or a distance learning solution, would add valuable insights into the perceptions of its effectiveness when being mediated by digital technologies and into its adaptability potential, not only to diverse contexts and audiences but also to new and challenging educational scenarios. Simultaneously, implementing studies involving different educational institutions and learner profiles, based not only on participants' perceptions but also on complementary data (e.g., input from learners' interactions within the scope of the peer learning tasks) would promote the conditions to assess gains more effectively to the learners involved in peer learning programs under the cognitive, affective, and social dimensions.

**Author Contributions:** Conceptualization, A.R.C. and C.S.; methodology, A.R.C. and C.S.; software, A.R.C. and C.S.; validation, C.S. and a group of experts.; formal analysis, A.R.C. and C.S.; investigation, A.R.C. and C.S.; resources, A.R.C. and C.S.; data curation, A.R.C. and C.S.; writing—original draft preparation, A.R.C. and C.S.; writing—review and editing, A.R.C. and C.S.; visualization, A.R.C. and C.S.; supervision, C.S.; project administration, C.S. funding acquisition, A.R.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by The Portuguese National Funding Agency for Science Research and Technology ("Fundação para a Ciência e a Tecnologia – FCT") under Grant reference SFRH/BD/146606/2019. The APC was funded by Digimedia – Digital Media and Interaction Research Centre, University of Aveiro.

**Institutional Review Board Statement:** Ethical review and approval were waived for this study, after assessment of the Ethics Committee of the University of Aveiro, due to the fact that the nature, scope, context, and purpose of the data collected and processed were decided not to be likely to result in a high risk to the rights and freedoms of the persons involved. The study was implemented in compliance with the applicable law regarding data protection as well as participants' confidentiality and anonymity.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The authors ensure that the data shared are in accordance with consent provided by participants on the use of confidential data. However, the data supporting reported results cannot be made available at the moment of this article publication, based on data availability restrictions regarding the research project within the scope of which the present study was implemented, imposed until the project has finished. Notwithstanding, access to the data supporting study results is expected to be possible from the first quarter of 2022 onwards.

**Acknowledgments:** The support of the Portuguese National Funding Agency for Science Research and Technology ("Fundação para a Ciência e a Tecnologia – FCT") as well as of Digimedia – Digital Media and Interaction Research Centre is gratefully acknowledged. We would also like to thank the schools that integrated this study for their receptiveness. The presented data were originally collected in Portuguese and this is a free proposed translation by the authors.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


## *Article* **Assessing the Differential Effects of Peer Tutoring for Tutors and Tutee**

**Allen Thurston 1,\*, Maria Cockerill <sup>1</sup> and Tien-Hui Chiang <sup>2</sup>**


**Abstract:** There is strong evidence that peer tutoring, as a form of cooperative learning, has a positive impact on tutor and tutee outcomes. However, little previous research has been reported as to the differential effects of engaging in cooperative learning in dyads for peer tutors and peer tutees, respectively. A randomised controlled experimental study was undertaken involving 295, 11- to 13-year-old students, drawn from 12 classrooms, across three secondary/high schools situated in areas of low-socio-economic status, in the north east of England. In total, 146 students engaged in cooperative learning for a period of 12 weeks, and 149 students served as a comparison group. Gains were significantly greater on independent standardised reading comprehension tests for those engaged in cooperative learning than those in comparison classes, and greater for tutors than tutees. The results are explored by critically reflecting on the underlying theories of education that may be at play in classrooms using this form of cooperative learning.

**Citation:** Thurston, A.; Cockerill, M.; Chiang, T.-H. Assessing the Differential Effects of Peer Tutoring for Tutors and Tutee. *Educ. Sci.* **2021**, *11*, 97. https://doi.org/10.3390/ educsci11030097

Academic Editor: Robyn M. Gillies

Received: 13 January 2021 Accepted: 24 February 2021 Published: 2 March 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

**Keywords:** cooperative learning; peer tutoring; reading comprehension; paired reading; high school; secondary school

#### **1. Introduction**

Peer tutoring (often referred to as 'tutoring') is a structured form of peer learning. It involves two students working together in a structured manner, with one taking the role as tutor, and the other taking the role as tutee. Paired reading is a form of peer tutoring that has been the subject of historical and recent research endeavor. There are a number of distinct forms of paired reading. Therefore, the literature presented in this manuscript will confine itself to reported research where the form of paired reading was of the same design and nature as the technique used by the research team in this study. Paired reading is generally implemented as a cross age/cross ability intervention, where the teacher manages the overall classroom, but has little individual interaction with pairs during the peer tutoring process. There is strong evidence that peer tutoring, as a form of cooperative learning, has a positive impact on both tutor and tutee, with an indicative average effect size (ES) of +0.48 [1]. It has been found to be particularly beneficial for children in high areas of social disadvantage and those with special educational needs [2]. However, previous studies have not explored the differential outcomes of peer tutoring/paired reading for those acting as peer tutor and peer tutee, respectively. The research used a randomised controlled trial experimental design to study whether tutors or tutees gain greatest benefit when undertaking cooperative peer learning (in the form of paired reading) in dyads, when reading in secondary/high school. The peer tutoring intervention reported in this manuscript was similar to a paired reading used in previous interventions that had been reported to result in positive ES of +0.2 in a randomised controlled trial in 129 elementary schools [3]. The peer tutoring intervention included training for teachers that focused on the theory and pedagogy of cooperative learning. Teachers implemented the intervention

over a 12-week time period, during which the research team made observations on the efficacy of the implementation in schools, and measured pre/post intervention changes in reading comprehension using an independent, age-standardised reading test.

Three interlinked theoretical perspectives are appropriate to consider when thinking about peer tutoring. Piaget [4] proposed that understanding developed in children, through the processes of assimilation and accommodation. If Piagetian based peer tutoring can provide the right balance between the disequilibrium, caused through cognitive challenge, and social exchanges between peers, effective learning takes place [5]. In the described process of paired reading that follows, this process may be thought to be in action when students acting as peer tutors correct reading errors of peers to improve the lexicon of their tutees. The second theoretical frame at play in paired reading is Vygotsky's theories around supported performance and the Zone of Proximal Development [6–8]. This process will be evident in paired reading as the peer tutee is asked to pick a challenging book that is just beyond their independent readability level. The third theoretical perspective that is relevant here is Social Interdependence Theory [9,10]. This combines elements of individual and group performance. Peer tutoring, in the form of paired reading, involves cognitive challenge from peer tutors and post-interactive reflection and restructuring by tutees. Both tutor and tutee have to fulfil their roles effectively. This creates a social interdependence between tutor and tutee that underpins cognitive developments. The individual successes of tutors and tutees are linked through common goals, and mutual interdependence on each other, for gains to accrue. Without both tutor and tutee performing their roles in accordance with prescribed patterns for interaction, neither can gain benefit from the interaction. For co-operative learning to be most effective during peer tutoring/paired reading, social interdependence must be present in the form of:


Paired reading is a structured form of cooperative learning, with high focus error correction, questioning for understanding, and formative feedback on performance [11]. During paired reading, two students read a book together. The peer tutor is generally an older student with more advanced reading comprehension ability that the tutee [11]. The text read should be above the independent reading age of the tutee, but below the independent reading ability of the tutor. This allows the tutor to correct any errors in reading. In addition, the tutor needs to understand what is read, and think about what questions need to be asked of the tutee during the interaction. The tutee needs to answer questions posed by the tutor. Without the appropriate gap, both tutor and tutee can be under stimulated [12]. Both of these processes require social interdependence [9,10] and processing of prior knowledge using a metacognitive strategy to link previous learning to the text being read. This also facilitates self-regulation and should concomitantly result in enhanced metacognition [13]. This may facilitate assimilation of the learning and accommodation of new ideas, eventually leading to the development of new cognitive understandings and equilibration as a result of post-interactive reflection.

When paired reading is implemented with reasonably high integrity, improvements in reading comprehension are typically good [3,14,15]. Paired reading has been reported to be an effective way of raising reading attainment in both primary/elementary aged students [16] and secondary/high school aged students [11]. In an experimental study, positive effect sizes were reported for experimental classes in respect of the number of words read correctly in reading aloud (ES = +0.22), and the number of questions answered correctly about passages read (ES = +0.55) in a study of paired reading in 20 experimental and

20 control schools. The study involved paired reading amongst nine-and-a-half-year-old pupils for 35 min per day, 3 days a week over a 15-week period [17]. Process observations have been reported to be an effective way of assessing implementation integrity in paired reading initiatives. In a 4-year study involving 33 control and 56 experimental students aged 7–10 years-old, undertaking paired reading for 30 min per week process observations showed significant advantages for experimental pupils in reading aloud (F(2,46) = 21.26, *p* < 0.001), academic talk (F(2,46) = 10.34, *p* < 0.01) and question asking F(2,46) = 4.73, *p* < 0.05) when compared to control pupils. In this study, significant advantages in reading attainment were also reported for experimental pupils (F(2,175) = 16.43, *p* < 0.0001) [12]. In a small-scale study, paired reading was shown to enhance positive learning interactions amongst socially rejected and isolated boys during a five-week trial [18]. Cross-age co-operative learning in reading with second and fifth graders was reported to enhance reading attainment in a quasi-experimental study involving 454 students from 19 schools in Belgium. Multi-level modeling revealed gains for cross-age pairings, but no significant gains for same-age pairings compared to control groups [14,15]. Oral reading with scaffolding from teachers and parents was demonstrated to be effective at raising oral and silent reading ability in a randomised study of 400 students in Grades 3–5 [19]. Therefore, there is significant evidence that paired reading has beneficial effect on student reading comprehension attainment [3]. However, whether it is tutors or tutees who actually get the most gain is generally not reported. This is because results are reported as an overall effect for both tutors and tutees. This gap in the reported literature led to the development of the following research question:

What are the differential effects on reading comprehension, for tutors and tutees, when engaged in peer learning?

In order to answer this research question, a research programme was developed that had the following aims and objectives:


#### **2. Materials and Methods**

#### *2.1. Reading Attainment Measure*

The main outcome measure was the Granada Learning New Group Reading Test (NGRT). The NGRT was an independent, standardised measure of attainment in reading comprehension that involved students completing stand-alone sentences by filling in the missing word, choosing the correct word to complete sentences within passages of text and selecting the correct answer to questions about passages of text that were presented with increasing levels of difficulty. The versions used were electronic tests A (pre-test) and B (post-test). The computer test self-adapts according to the response of the test taker. This process continues until a convergence is reached between the difficulty level of the questions and the ability of the pupils to answer them correctly. Passages are also of different difficulty levels, and students are presented with texts of greater or lesser difficulty according to their previous answers. The tests are internationally available, independently designed instruments with good reliability. Cronbach alpha was reported to be 0.846 for a sample of 2574 students drawn from the schools selected in this study [20]. The NGRT measured two dimensions of reading comprehension: sentence completion and passage comprehension. These are combined to give an overall standardised reading age score, where 100 is the age-standardised score for the age of the child and 10 is one standard deviation from the norm. The choice of this reading comprehension test was made by the

funder who expressed a desire to be able to track reading comprehension development across a number of funded studies.

#### *2.2. Observations*

Researchers visited all six intervention classes to undertake classroom observations. These were conducted using standardised, previously validated, observation templates to determine efficacy of paired reading, developed in previous research [3]. All observations were conducted by one member of the research team. All data were recorded in written format, in real time. The observation process was as follows. Firstly, the observer made general written notes regarding the classroom atmosphere and structure for cooperative learning. Then the observer moved to undertake observations on the efficacy of the paired reading process. Whilst the class undertook paired reading, a random sub-sample of five pairs were selected for observation from each class. Behaviours were observed and recorded in real time during a series of three, one-minute observation windows for each of the five pairs (15 min total observation time). During observations, the researcher observed and recorded behaviours during reading and looked to record the frequency and efficacy of the error correction process (a process that should have contained a mistake in reading, a mistake corrected by the peer tutor, evidence that the tutee repeated the corrected word, and that the tutor praised the tutee for reading the mistaken word correctly), and also recorded the number of times the peer tutor asked questions about the book.

The format of each observation was as follows: Pair 1 was observed for a one-minute window and behaviours recorded. Next, pair 2 was observed, then pairs 3, 4 and 5. After the cycle was completed, observation again cycled to pair 1, pair 2, pair 3, pair 4 and finally pair 5 for the second window. This sequence was repeated for the third observation window (resulting in a 15-min period of observation). This meant that there were 90 one-minute observation windows spread evenly as 15-min observation periods undertaken in each of the six classrooms implementing paired reading. Behaviours were recorded as they were observed. So, for instance, if a pair was reading alone and made a mistake, the mistake was corrected by the tutor, praise was given, the pairs started to read together, and the tutee signalled to read alone; each of these behaviours would be recorded each time it occurred. Total behaviours in each category of observed behaviour were tallied for each pair. The same person conducted all observations. Reliability trials were conducted at the start and the end of the observations using pre-existing videos of the paired reading process to measure intra-rater reliability during the observation timeframe. The reliability trial involved undertaking observations on a 15-min prerecorded video. The mean alpha for observation reliability was 0.93.

#### *2.3. Sample*

The study was conducted in a large metropolitan borough in the north east of England, United Kingdom. It occupied around 80 km<sup>2</sup> and had a population of 191,659. The borough had areas of extreme deprivation, with at least one ward in the most deprived 1% of all electoral wards in the United Kingdom [21]. The project comprised 12 classes, including six Year 7 classes and six Year 9 classes, drawn from three of the eight secondary/high schools within this geographic area comprising 295 (149 control/146 intervention) students. The mean age of students at pre-test was 155.32 months (*SD 12.48*); Special Educational Needs provision was 101 students with Action/Action Plus and 10 Statement of Special Educational Needs. Six students were reported as having English as Additional Language. Ethnicity of the sample, as reported on the school information management system, reported by parents at school enrollment was: 283 Caucasian, 3 Chinese, 1 Pakistani, 1 White-Asian, 1 White-Other, 1 Asian-Other, 1 Black-Other, and 4 Other-mixed). Free School Meals rates were School A = 36.5%, School B = 30.1%, School C = 34.1%, compared to a national average = 28.8% [22]. Free School Meals are a measure of social deprivation often used in the United Kingdom. Children are entitled to Free School Meals if they received certain government benefits paid to families who have low incomes. The chronological age

of the sample at pre-test was 155.32 months (*SD 12.48*), and the reading comprehension age determined by NGRT pre-test was 149.06 months (*SD 37.81*). However, the range of chronological age was 137–174 months, whereas the range in NGRT reading comprehension age was 60–291 months.

#### *2.4. Randomisation*

Classes were randomised to condition pairwise for each teacher who volunteered to implement the technique. Classes were grouped in pairs for each of the six teachers (12 classes in total). Then each pair of classes was randomised to condition using a random number generator programme for iPhone: Version 5.5 123 The Random Number Generator by Nicolas Dean. This was set to generate a number of 0 = control (*n* = 6) and 1 = paired reading (*n* = 6) classes. This generated a sample of equal number of classes, where control classes and intervention classes were taught by the same teacher. Teachers signed Memorandum of Understanding stating that they would only use the paired reading technique with target classes, and would use business as usual for control classes.

#### *2.5. Paired Reading*

The paired reading technique involved supported reading, error correction, and switching between the tutor and tutee reading together, and the tutee reading alone. The book chosen by pairs had to be above the independent readability level of the tutee, but below that of the tutor and appropriate to their interest. This facilitated the tutor helping the tutee through the error correction process. Readability level was decided by using a simple test. Tutees randomly selected 20 words from the book from four different pages. If the tutee could read between 15 and 19 words, the book was deemed to be at the right level of readability. Tutees could select any genre of book or reading material (whether fiction, non-fiction, comic, magazine, newspaper). Teachers also occasionally checked the appropriateness of readability of books during observations. The tutor and tutee started by reading together. The tutee signalled to read alone. Upon making an error, the tutor waited 4–5 s and if the tutee did not self-correct, the error was corrected by the tutor. The tutee repeated the error word correctly and the pair read together again until the tutee signalled to read alone. The tutee read alone until the next error at which point the error correction process would be repeated. The tutor was also asked to formulate questions to ask the tutee as reading progressed. In the form of paired reading employed, the asking of questions was reserved for the role of tutor (as it had been in earlier iterations of the technique). The other important role for the tutor was to praise the tutee's reading. There was a set number of times to use praise which included after mistakes were corrected, when switching to reading alone, when the tutee was reading difficult sections of text independently and during reading alone by the tutee (e.g., for good use of expression). Both the tutor and tutee recorded sessions in a logbook. This noted what went well and what the pair were working to improve.

#### *2.6. Matching of Pairs*

Pairs were matched on the basis of previous reading comprehension attainment. Students within classes were ordered from highest to lowest in reading comprehension attainment using the pre-test NGRT result. The top-attaining tutor in the Year 9 class tutored the top-attaining tutee in the Year 7 class; the second top tutor tutored the second top tutee in the younger class, and so on. Once matched, the advice given to teachers was that pairs stayed together for the duration of the intervention period. At the beginning of the intervention period, teachers were allowed some latitude to switch pairs who were clearly not able to form a working partnership. These processes were adopted on the basis that previous research indicated that an attainment gap was preferable to optimise the interactions and benefit within pairs [16]. The matching technique was originally reported and described in some detail by Fuchs et al. [17,23]. It had also been used for paired

reading in the Fife Peer Learning study which used a similar technique with primary school students [3]. A manual describing the technique used is available online [24].

#### *2.7. Continuing Professional Development*

The first continuing professional development (CPD) event for teachers took place at the end of February (one day), prior to the implementation of the 12-week peer tutoring programme in schools. In mid-April, following the implementation of four weeks of the programme in schools, a second CPD event was held (half day). Following implementation of the programme, a final CPD event was held.

#### *2.8. Training of Pupils*

Training videos were provided for the intervention classes. Videos were produced during the pilot study and contained local students demonstrating the technique with local accents. Video footage was captured and edited to ensure all the components of paired reading were represented in the training video.

#### *2.9. Length, Duration and Implementation of Intervention*

The intervention took place for 30 min, once per week, over a period of 12 school weeks. This spanned a period from February to July in one school year (given that both half-term and Easter holidays fell within this period). This gave a total minimum and maximum duration of between 4 and 6 h. When the technique was implemented, the teachers swapped half their students (i.e., half of the Year 9 students went to the Year 7 teacher's classroom, whilst half of the Year 7 students went to the Year 9 teacher's classroom).

#### *2.10. Planned Analysis*

Planned analysis was to look at post-test outcomes in reading comprehension for tutors and tutees engaged in paired reading, compared to a suitable control group, on the NGRT comprehension reading test using pre-test scores as a covariant in an ANCOVA statistical test. It was calculated that if previous effects from randomised trials were replicated, that to detect an effect size of +0.23, 295 students would be required to detect significant differences between groups at *p* > 0.05 and 80% power, assuming that correlation between pre- to posttest was 0.7. The ES used in the power calculation was a best estimate based on previous reported ES from differing reading assessments (+0.2 [3] and +0.4 [24]) from studies that researched the same paired reading techniques used in primary/elementary school.

#### *2.11. Ethics*

The study was approached with equipoise underpinning expected outcomes. Although a randomised trial in younger children had previously been undertaken, no previous randomised controlled trial with children of the age in this study had been undertaken. Students in the control group were in a wait-treatment group. This meant that if positive outcomes accrued for those undertaking paired reading, the resources and training would be available to those students and staff who wanted to avail of them and had been in the control group. The research was approved by the School of Education Ethics Committee at Queen's University Belfast, but also had to receive individual ethical approval from every individual headteacher (this was because in the jurisdiction that this work was undertaken, any research undertaken in the schools must be ethically approved by the school headteacher). Opt-out consent was used at the individual student level, where students, or their parents, could opt-out of having their data used for research purposes.

#### **3. Results**

#### *3.1. Effects of Paired Reading on Reading Comprehension Performance*

Data from the NGRT are presented as an age-standardised overall reading comprehension score in Table 1. ANCOVA analysis indicated that those students acting as peer

tutors in the intervention made significantly higher gains on the overall age-standardised reading comprehension scale than those acting as peer tutees (F(1, 140) = 8.37, *p* < 0.05). Statistical analyses using ANCOVA indicated that gains were significant on the overall age-standardised reading comprehension scale for Year 9 students acting as tutors (F(1, 145) = 4.29, *p <* 0.05) than students in the control groups. Year 7 students in the intervention actually performed more poorly than controls, with a small negative Effect Size of −0.07, but analysis indicated that this difference was not significant (F(1, 139) = 1.63, *p* = not significant). These results indicated that peer tutoring, in the form of paired reading, had significant benefit in reading comprehension to those students who acted as peer tutors.


**Table 1.** Granada Learning New Group Reading Test overall age-standardised reading comprehension scores for intervention and control students.

#### *3.2. Observations of Implementation Fidelity*

Results of observations in classrooms are reported in Table 2. Adherence to the technique was good in most classrooms. The process of paired reading was implemented with high integrity in respect of seating arrangements and book choice. All pairs observed were appropriately seated and had selected a suitable book for the observation lesson. Observations indicated good implementation of the peer feedback and error correction. In total, 28 out of 29 errors were correctly spotted and corrected, during 90 one-minute observation windows, with the error rate being slightly lower than that reported in the Fife Peer Learning Project [25]. However, opportunities for praise were lower than optimal, being observed only 12 times (whereas the number of times praise was observed, should have been higher than the number of errors made, as the rubric was that the tutor praised the tutee after they corrected a word). Questioning was less frequent than had been observed in the Fife Peer Learning Project [25], with only 16 questions asked by peer tutors during 90 min of observation. Overall, observations indicated that teachers established paired reading in their classrooms and that tutors were monitoring reading, and correcting mistakes effectively. There was an indication that tutors could have helped tutees make sense of what was being read by asking more questions. No questions were asked by tutees during observations, but this is not a surprise as they were not required to ask any.


**Table 2.** Observations of reading in classrooms.

#### **4. Discussion**

Overall ES for tutors were similar to those previously reported (ES +0.24 for tutors), for instance an overall ES of +0.24 was reported in the Fife Peer Learning study [3]. However, the overall ES of +0.09 for secondary school students (for all tutors and tutees) was lower than the overall ES reported for primary/elementary school students in the Fife Peer Learning study (note that the Fife study did not separate gains for tutors and tutees) [3]. The main difference between the current study and previous reported studies was the fact that analysis indicated that gains mainly accrued for tutors only, rather than for both tutors and tutees. It is often reported that peer tutors should gain more than peer tutees when undertaking peer learning. However, this is the first definitive evidence that this does in fact occur, as previous studies have only reported composite effects of peer tutoring for both tutors and tutees [25]. The asking of questions by peer tutors, whilst monitoring and correcting errors (which was done with a high degree of fidelity), is a demanding role. The hope was that the work would take place in the Zone of Proximal Development of the tutee, where they developed their lexicon and their understanding of what they were reading. On reflection, had the tutees been challenged to ask questions of their tutors during reading, this may have helped them develop a more complex understanding of the text. This would be so because to ask a question, the reader must have a metacognitive understanding of what is being read. The asking of questions by tutors only may account for why gains tend to accrue for tutors, rather than tutees. Educators might wish to examine how to increase cognitive demand for tutees during the paired reading process (e.g., is there a role for tutees to ask questions also).

Previous studies had predicted greatest gains using this cooperative learning technique for tutors in high poverty areas [26,27]. However, these reports were anecdotal. This study establishes, in a scientific study, that these benefits do accrue in that way for the sample of secondary/high school students from this background (the mean Free School Meal rate for this sample was 33.56%, compared to the national average of 28.8%), who also had lower than mean age-standardised reading comprehension attainment scores than the normalised sample with which psychometric properties of NGRT were established. In this study, it was tutors who showed the only significant gains. Inherent within the classroom organisation of peer tutoring dyads, there is often an embedded message about the status of students. Tutors are perceived as higher status than tutees [28]. Enhanced satisfaction with learning and achievement were reported in a sample of 104 twelve-year-old students in a reciprocal peer tutoring study. However, these gains were only evident when students were acting in the role of tutor [29]. This is why reciprocal tutoring has often been reported to be beneficial to use in schools [30]. It may also be a more plausible reason as to why gains mainly accrued for the tutor in the peer learning process.

It was noted that overall effect sizes on NGRT scores were modest (overall ES +0.13). This ES is lower than when the same technique was implemented in elementary/primary schools where reported ES was +0.2. However, the intervention reported here ran for only 12 weeks and this may not have been long enough for gains to maximise. In contrast, the Fife Peer Learning study ran over a period of 104 weeks [3]. In addition, there was some concern over the low number of errors being made during the cooperative interactions between students. In comparison, when implemented with Year 6 and Year 4 students in elementary/primary schools in the Fife Peer Learning study, mistake rates that were linked to optimal reading attainment gains were one mistake about every two minutes (a mistake rate of 0.5 mistakes per minute) [25]. However, the mistake rate in this study was approximately one every three minutes (a mistake rate of 0.32 mistakes per minute). The explanation for this may be that reading development for students in the sample had reached a stage where the range of new word attack skills, e.g., use of phonics to approach new or unknown words, was quite comprehensive. They were therefore able to pronounce words that were used in their selected reading efficiently. The widespread use of phonics teaching in England has resulted in a school-aged population who can read sentences they may not necessarily have full understanding of. This has been termed 'barking at print' [31]. Therefore, even when students did not know what a word meant, they would be able to say it without error. Alternatively, the interest level of students may not be matched by the reading level of books, as publishers and authors make books with readability age targets that are wide enough to appeal to a mass audience and are generally reported to be at a level akin to that of Year 4 and Year 6 students [32]. This level of readability would indicate a potential use of reading ages in books of about 144 months, where in fact the mean comprehension reading age determined by the NGRT test of the sample in this study was 149 months. This means that the book readability age ranges were likely to be below the reading comprehension age of the majority of the students. With this being the case, the error correction component of paired reading is only likely to be beneficial in similar secondary/high school settings for those of low literacy attainment. A larger randomised controlled trial reported that it was only low reading comprehension-attaining tutors in secondary/high school settings in the north east of England that benefited from use of paired reading techniques similar in nature to the use of paired reading reported in this study [20]. This may add weight to this argument. Other ways to optimise the use of paired reading may include ensuring that both tutees and tutors ask questions. This should enhance cognitive load on the tutees, bringing it closer to that demanded of the tutors. It may also create more opportunities for promotive positive interaction. This has been reported to be a fundamental requirement of successful cooperative learning techniques according to Social Interdependence Theory [9,10]. Finally, there may be a need to pay attention to the use of praise. Given the way that self-concept in an academic subject area has been repeatedly shown to predict better future school attainment [33], it would make sense for tutees to be the recipients of praise that may raise their self-concept levels of themselves as a reader.

What were the sociological factors at play that may explain these results? According to Bourdieu [34–36], education achievements are not mainly determined by mental abilities, but habitus that develops within a specific social space through gradually assimilating the structural features of the context in which the actor is situated. Because the volume of cultural capital directly affects such structural features, when its strength increases, the actor is likely to develop an academic habitus and vice versa. Because habitus functions as a certain form of disposition, directing the actors how to perceive, judge and react towards the outside information, it becomes the core framework regulating people's minds and behaviours.

It has been well documented that linguistic abilities come to regulate cognitive development [8,37]. Code theory proposed by Basil Bernstein [38] looked at the range and type of language structures employed by families and within schools. Bernstein classified codes as being elaborated when language and discourse were rich, and children grew up being

able to articulate themselves effectively. This theory provides a convincing insight into the linkage between cognitive development and linguistic abilities. This language-rich culture thus facilitates the children's development of an elaborated code that helps them undertake logical reasoning and decode the theoretical, and often abstract, concepts embedded within texts that they are exposed to in school. This is particularly important when undertaking reading comprehension work, when students must move from the text to abstract conceptualisation of meaning. Those students with poor reading comprehension may well lack the development of this required 'scholastic' code. In this manner, texts can be conceptualised as vertical discourse, and characterised as theoretical, systematic and logical [39]. Students growing up in linguistic or literacy poverty, where their language may not be stimulated to the same extent as children from more literate backgrounds and homes, may be afforded fewer opportunities to expand their ability to express abstract terms in verbal discourse, developing the required lexicon and comprehension to fully articulate with school-based texts [39]. As a result, they tend to develop a restricted code (the features of which have been reported to be short, unorganised and unsystematic discourse). Because a restricted code differs greatly from the written language often found in books, it can then be difficult for students with less well-developed language skills to understand the abstract meanings of texts and make the jump from text to abstract imagination of meaning [40]. In order to improve their outcomes when learning, weak social relation such as that which occurs during groupwork/cooperative learning (weak framing) can help to transform vertical discourse (strong classification) into understandable information [41].

Findings from this study suggest that peer tutoring/cooperative learning, along with extended periods of academic learning comprising language rich experiences, are two core elements in developing the linguistic structures and competency to become a successful comprehender of written text. The language immersion of the paired reading processes, by which high quality discourse about text in a suitably structured environment is shared, provides a good context and medium in which to develop linguistic and literacy skills for students who have been unable to develop these skills previously.

#### *Limitations*

This study had a number of limitations. The sample size was low and drawn from a limited number of school settings. There is a need to undertake a study with a wider number of schools, classes, teachers and students. Classes were assigned to condition at the class level, so interpretation of individual analysis is problematic. There is now a need to undertake a cluster randomised controlled trial of paired reading that takes account of potential clustering effects in design and analysis. There is also a need to explore for which students gains accrue. This study appears to indicate that gains are most likely to accrue for those acting in the role of tutors. There is also a need to explore where gains accrue in terms of reading comprehension ability, to determine whether paired reading in secondary/high school should be a general or targeted intervention.

#### **5. Conclusions**

In conclusion, the observed effects of paired reading in primary school settings do not transfer for all pupils in a secondary school implementation. Gains are greatest for those acting in the role of tutor. This may be due to reading ages of texts not being high enough to provide challenge and extend the lexicon of tutees. Reading comprehension ages are likely to be lower than the chronological age of secondary school students, but higher than the readability of available texts at an appropriate interest level for students. There was some evidence for this in the fact that mistakes made were at a lower rate than those reported for a sample using the same technique in primary school [25] However, further work is required to explore these patterns in more detail, and determine whether they can be generalised to a larger population of students.

**Author Contributions:** Conceptualization, A.T. and M.C.; methodology, A.T. and M.C.; formal analysis, A.T., M.C. and T.-H.C.; investigation, A.T., M.C. and T.-H.C.; resources, A.T. and M.C.; data curation, A.T. and M.C.; writing—original draft preparation, A.T., M.C. and T.-H.C.; writing—review and editing, A.T., M.C. and T.-H.C.; funding acquisition, A.T. and M.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by a grant under the 'Literacy Catch Up Round' by Educational Endowment Foundation R2999EDU.

**Institutional Review Board Statement:** This research was approved by the Ethics Committee of the School of Education, Queen's University Belfast.

**Informed Consent Statement:** All data was collected and held in accordance with the informed consent procedures of the Ethics Committee of the School of Education, Queen's University Belfast and data was processed and held within the legal framework of Data Protection Framework of the United Kingdom.

**Data Availability Statement:** The anonymised data set is publicly available from the 'Activities' section of Allen Thurston's homepage at Queen's University Belfast at https://pure.qub.ac.uk/en/ persons/allen-thurston/activities/ or by emailing Allen (details above).

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study (other than stipulating the main outcome measure); in the collection, analyses, or interpretation of data; in the writing of the manuscript; in line with the terms of the grant application the funders gave permission for this work to be published.

#### **References**


## *Article* **Peer Assessment: Channels of Operation**

**Keith Topping**

School of Education, University of Dundee, Dundee DD1 4HN, UK; k.j.topping@dundee.ac.uk

**Abstract:** The present paper offers a definition of peer assessment and then reviews the major syntheses on its effectiveness. However, the main part of this paper is preoccupied with how to do PA successfully. A typology of 44 elements explains the differences between the many types of peer assessment. Then a theoretical model outlines some of the processes which may occur during PA. Initially, only a few of these will be used, but as those engaged in PA become more experienced, an increasing number of elements will feature. However, these may not appear in the linear order set out here, and indeed may be recursive. The implications for the design and organisation of PA are outlined, as well as the implications for future research.

**Keywords:** peer assessment; schools; universities; effects; how to; typology; theory; model

#### **1. Peer Assessment: Channels of Operation**

Feedback is widely considered important in education [1] and peer assessment (PA) is one method of enhancing the speed and quantity of feedback, if not the quality. Many professions may expect to engage in PA as part of their vocations, so its value goes beyond school and university.

#### **2. What Is Peer Assessment?**

A widely quoted definition of PA is: "an arrangement for learners to consider and specify the level, value or quality of a product or performance of other equal-status learners" [2] (p. 256). However, other similar terms (synonyms) are in the literature (e.g., peer grading/marking—giving a score to a peer product or performance; peer feedback peers giving elaborated feedback; peer evaluation—more usually in workplaces regarding skill and knowledge; or peer review—more usually in academia regarding assessment of written papers).

#### **3. Does Peer Assessment Work?**

PA is not just for managing assessment burdens for teachers, but more importantly a mechanism for more effective learning, particularly with elaborated feedback. For the assessor, the intellectual demands of reflecting, making a balanced assessment, formulating and delivering feedback can all lead to learning gains [3]. For the assessee, the intellectual demands of receiving and evaluating the feedback, deciding what aspects to implement and what not, and reflecting on other issues prompted by the feedback (but not contained within it) can all lead to learning gains [4].

The evidence on PA with all kinds of learners is generally positive, from the earliest reviews (e.g., [5] on peer grades and feedback; [6] on peer grades) to the latest metaanalyses (e.g., [7,8]). An early systematic literature review on the effects of PA appeared in 2009 [9]. Fifteen studies from 1990 to 2009 dealt with effects on achievement. However, only one of these studies included students from a school, the remainder consisting of university students. PA had positive effects. The authors offered four underlying constructs: psychological safety, value diversity, interdependence and trust. Psychological safety was defined as a belief that it was safe to take interpersonal risks in a group of people. Value diversity referred to differences in opinion about what a team's task, goal or mission should

**Citation:** Topping, K. Peer Assessment: Channels of Operation. *Educ. Sci.* **2021**, *11*, 91. https:// doi.org/10.3390/educsci11030091

Academic Editors: Robyn M. Gillies and James Albright

Received: 11 January 2021 Accepted: 20 February 2021 Published: 25 February 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

be—it should be low for PA to be effective. Interdependence has been long studied, but needs to be perceived by the participants rather than assumed by teaching staff. It requires that multiple perspectives are made explicit and students are individually responsible for an active contribution to group discussions. In respect of trust, several studies noted that students felt uncomfortable criticising one another's work, or at least initially found it difficult to rate their peers.

Another study [10] considered what quality criteria were specifically relevant to PA. One hundred and thirty-two studies of PA were selected, together with 42 studies for a qualitative analysis. Nowhere was any distinction made between studies based in school, higher education or other settings. Studies were evaluated with regard to two quality criteria: (1) the recognition of educational measurement criteria, and (2) the consideration of student involvement in the assessment of learning. Where emphasis was placed on authenticity and future learning needs across the lifespan, PA had much to recommend it in terms of generalisability, particularly utility in contexts beyond the present institution.

Only one review was solely concerned with PA in schools [11], analysing 26 studies of peer response on writing proficiency. The author noted that several studies had indicated that peer response was effective, but had not explored why. Many studies appeared to combine instruction in strategies, rules for interaction, and/or genre knowledge—and this seemed to be effective compared to individual writing.

The first meta-analysis of PA [12] studied PA in digital platforms since 1999, again mainly in universities, finding a moderately strong average correlation between peer and teacher ratings of 0.63. This correlation was higher when: (a) the PA was paper based rather than computer assisted; (b) the subject area was not medical/clinical; (c) the course was graduate level rather than undergraduate or in school; (d) individual work instead of group work was assessed; (e) the assessors and assessees were matched at random; (f) the PA was voluntary instead of compulsory; (g) the PA was not anonymous; (h) peer raters provided both scores and qualitative comments instead of only scores; and (i) peer raters were involved in developing the rating criteria.

Turning to the latest meta-analyses, one [7] found an overall effect size (ES) of 0.29 in 58 studies (an effect size is a number measuring the strength of the relationship between two variables, which can apply across all studies). Significant moderator variables were found of training and online/digital (moderator variables are third order variables that affect the size or nature of the relationship between an independent and dependent variable). Another meta-analysis [8] found an overall ES of 0.31 in 54 studies, but no significant moderator variables. In both cases the ESs were lower than previous studies.

In PA studies, it is often assumed that teacher "expert" assessment should be the criterion for validity, but both these studies showed PA was more reliable and had higher ESs than teacher assessment, although in fact teacher assessment is not very reliable [13].

#### **4. Digital Peer Assessment**

It is unsurprising that digital PA has been separately reviewed, given the widespread recent move towards online methods and the fact that PA in large university courses can only be managed by such means (e.g., [14,15]). The first meta-analysis [16] found 37 controlled studies from 1999 to 2018. Eight studies were in school and the rest in higher education, and again this mixing of contexts without discrimination is a weakness. Of the 37 studies, 19 examined outcomes (overall ES 0.58) and 17 the effects of extra supporting strategies (ES 0.54).

These ESs would be considered "moderate" by most researchers, but are larger than those reported most recently for PA in general (above), suggesting that (despite some disadvantages), digital PA has countervailing advantages that make it more effective than face-to-face PA. Training and anonymity improved outcomes, and duration of PA was also important (6–10 weeks being the optimum). However, direct comparison of online and offline learning was rare—most studies compared online PA to no PA.

However, here we are less concerned with whether PA works and more concerned with the how of PA, and we will consider a typology of PA and then a theoretical model of PA. Together, these should give practitioners a clearer idea of the how to successfully design and implement a PA project, and researchers a clear idea of the broad context of PA.

#### **5. Typology of Peer Assessment**

Several studies compare two or three types of PA, but the variety in types of PA goes far beyond that. Teachers need to be able to clearly categorise what they want to do—in a way which will also remind them of variables which they might have forgotten. It is important to be aware of what you are not doing as well as what you are. Different kinds of PA are more or less suitable for particular classroom contexts, different levels of maturity in the students, different subjects and assessed activities, and these are judgements the teacher must make.

A typology of relevant variables was first described in 1988 [2], Subsequently, a more developed inventory was offered [17]. Further developments [18] (pp. 12–13) in 2018 outlined 44 variables (see Table 1).


**Table 1.** Variations in Peer Assessment.


**Table 1.** *Cont.*

Proceeding through the list, firstly the objectives for the exercise may vary—the teacher may target cognitive and/or metacognitive gains, teacher time saving, or other goals. There may be other gains, such as social gains or attitudinal gains (e.g., better relationships, improved self-confidence, improved motivation). Do you see peer assessors and assessees talking more out of class? Do you feel that some students are more engaged in what they are doing as a result of PA?

A key difference is whether the PA is formative or summative or both. Will it serve to give students indications of how to improve their work (formative), so the final version can be better? Or will it just indicate to the students how good or bad their work was (summative), with no opportunity for improvement?

Similarly, the PA can be quantitative (assigning a number with respect to a grade) or qualitative (giving rich verbal feedback on positive and negative aspects and possibilities for improvement), or both. If students are merely to give a grade, they will need considerable experience in grading before their grades can be considered reliable. Further, even if they are reliable, they do not give the assessee any clues on how to improve their work the next time. By contrast, qualitative feedback gives rich ideas on how to improve the current piece of work, let alone future pieces of work. The assessee may not agree with all of these, but some negotiation of the nature of improvement can follow.

Will PA be voluntary or compulsory? When it is used in a class, it would be a normal expectation that all students would participate, but if it is compulsory from the beginning, some students might be very resistant to participation. It might be better to say that it will be voluntary at the beginning. So few students are likely to opt out, that after a short while those who have opted out will realise that their opposition is unusual if not a little bizarre, and agree to join in.

Will you use some form of digital technology? This could be all online or it could be blended, with some face-to-face contact. This could help even if the PA was mostly occurring in class, e.g., having students rehearse their oral presentations on video on their mobile phones until they are satisfied with the performance, then have them upload the final version to a common location (e.g., GoogleDocs) for everyone to see, then meet face to face to discuss and conduct PAs. For more remote students and during pandemics with lockdowns, all PA will have to be online. If neither of these is relevant, it all could be face to face unless the number of students is too large to allow this.

Other differences between types of PA are more subtle. For example, are the PAs on single pieces of work, or are they of several pieces of work? A piece of writing is relatively easy to assess, as it has a beginning and an end. But even here you should not assume that peer assessors are only relevant after the writing has been completed. They could for instance be involved again as the writer tries to improve the piece of writing. Other products of work may be more complicated. For example, in PA of a group presentation, should the quality of discussion prior to the presentation itself be peer assessed?

Are PAs on the same kind of product? The product or output assessed can vary writing, portfolios, presentations, oral statements, and so on. Assessment of writing is very different to assessment of an oral statement, which is in turn very different to PA in music or physical education. Students will need some experience of each kind of PA before they have confidence that they can manage the necessary tasks.

PA can operate in different curriculum areas or subjects, which may impose different demands. For example, in physical education classes, can peers be trained to investigate differences in the way the other student runs, or catches a ball, or throws a javelin, and so on? In foreign language learning, how quickly might students be able to accurately respond to the comments or questions of a peer in the foreign language?

The participant constellation can vary, with consequent variation in joint responsibility for the assessed product. Assessors and assessed may be individuals, pairs or groups. Will you have one assessor and one assessee in a pair? Or a small group where everyone assesses all the productions of the other members of the group? Will their PA be reciprocal? Or will you have one cooperative group assessing another cooperative group—again, reciprocal or not? Be careful in supposedly cooperative groups that all members of the group have contributed. You could invite the group to assess each of its members on the size of their contribution to the group proceedings. Then the responsibility for the finished product is not unfairly apportioned to the lazy members of the group.

Will it be anonymous or not? Of course, if you have reciprocal face-to-face PA in one classroom, it is impossible to make it anonymous. But if you have one class assessing the work of another class, and giving feedback in writing or over the internet, it might be much more possible. But will you actually want the feedback to be anonymous? Peer feedback from somebody you know might be more powerful than that from somebody who is anonymous. But if you do not know your assessor, you might feel safer initially if they were anonymous.

Clarification of the assessment criteria is essential, and peers may or may not be involved in establishing these criteria. In general, however, peers should always be involved in the development of the assessment criteria, even if the teacher has their own ideas or there is some external assessment system that needs to be acknowledged. The fact that the peer group will eventually come up with very similar criteria to those the teacher would have given does not take away from the value to the peers of feeling engaged in the process. As a result, they know the criteria better from the outset.

Rubrics or structured formats listing assessment criteria for feedback may or may not be provided. However, assessment rubrics almost always help the assessors and the assesses. As above, they should be developed by the peer group. But having these criteria written down will help add consistency to the PA.

Training in PA may be given to assessors and/or assesses to a greater or lesser extent. It is surprising how many projects in the literature appeared to give no training to the peer assessors. Some training will be needed—the only question is: how extensive will it be? It cannot go on too long or the peer group will become restless to get some "real" activity. However, it should not merely involve the teacher talking. Some encounters with real life examples and some practice in actually applying PA should certainly feature as part of the training.

Is any feedback provided expected to be balanced between positive and negative, or only one of these? When you are starting with PA, you might be inclined to ask the peer assessors to provide only positive feedback. Then you get them used to the idea of being positive. Later, you can also ask them to give "suggestions for improvement", which of course are open to discussion. Once students are competent with both aspects of feedback, you can give them free rein, except that every piece of assessed work should have some positives and some negatives.

Is feedback expected to lead to opportunities to rework the product in the light of feedback, or is there no opportunity for this? Of course, we all hope that the current version of our work is the final one, so there might be some resistance to (apparently endlessly) reconsidering—although this is almost always going to result in a better piece of work. Negative feedback indicates where the work needs improving, and hopefully there will be time available to achieve this. A related question here is that of audience—why should the peer assessee try to improve the work? Who will tell the difference? Students need to see what the point is of improving.

Is feedback expected to include hints or suggestions for improvement? Negative feedback will be much more acceptable if it is accompanied with some suggestions for improvement, even if those suggestions are not accepted. They give the assessee something to think about, and maybe they will then come up with a completely different way of doing things.

The nature of subsequent PA activity may be very precisely specified or it may be left loose and open to student creativity. Again, this may be a developmental issue, in that at the beginning, peer assessors and assessees may need a fairly strict procedure. Later, however, this may become looser, so that assessors may begin to give more feedback in their own time, as they develop a sense of responsibility towards their assessee.

Does the interaction involve guiding prompts, sentence openers, cue cards or other scaffolding devices? At the beginning of PA, one, some or all of these are a good idea, as some students will have little idea how to begin a PA conversation. Giving them some questions to use to get them started is an excellent idea—they do not necessarily need to use them.

PA can be one-way, reciprocal, or mutual within a group. If you have an older class assessing a younger class, directionality is likely to be one-way. If you are working with same-ability pairs in one class, directionality is likely to be reciprocal. If you are working with groups, does the group decide on a mutually agreed assessment for another group, or are the separate PAs of the other group to be taken into account? (requiring an agreed group assessment gives the group another valuable learning experience).

Matching of students may be deliberate and selective or it may be random or accidental. If the teacher is new to the class, it may need to be random. If the teacher knows something about the class members, one can be more careful. Matching may take account only of academic factors, or also involve social differences. The most able assessing the least able is not recommended. You may decide that you want the top half of the class assessing the bottom half of the class. Or you may decide that you want students to be matched based on having similar abilities, especially if you are doing reciprocal PA. Or you may decide that while ability is relevant, personality and social issues are also relevant.

Assessors and assessed may come from the same year of study or from different years. If you have a colleague from a class of a similar age who is also interested, you could certainly see if the two classes could be matched up for the purposes of PA. If the classes are more or less of the same size, you have an ideal opportunity. But many teachers will want to experiment first within their own class.

The assessors and assessees may be of the same ability, or deliberately of different ability. If they are of the same ability, you can expect a rich dialogue between them. If they are of different ability, the flow may be more one way, with the more able child dominating the proceedings.

The amount of background experience the students have in PA can be very variable. PA may represent a considerable challenge to, and generate considerable resistance in, new initiates. If they have previous experience, it may have been positive, negative or both. So, bear in mind the previous experience that these students might have had in previous classes. You might want to ask them about that right at the beginning.

Students from different cultural backgrounds may be very different in acceptance of PA. In particular, students from a Middle Eastern or Asian background may have great difficulty accepting PA. In the case of Middle Eastern students, resistance might have a lot to do with gender, as boys might be very reluctant to accept advice from a girl. In the case of Asian students, the idea that there is not one right answer which the teacher already knows can be rather startling, and also lead to resistances.

Gender may thus make a difference, and thought should be given to the implications of same-sex or cross-sex matching. With Middle Eastern students, same-sex matching might be easier to start with. We have some evidence from peer tutoring that same-sex matching is generally more effective for boys, but of course that leaves you with the question of what to do with the girls. So, there is no easy answer here. Of course, if there is no face-to-face contact (as in an online environment), gender may not be apparent.

Place can vary: most PA is structured and can occur in class, but it can also be informal and occur outside of class. Once students become really involved in it, you may find they are having PA conversations in break time. Indeed, in some cases, taking PA into their homes and using it with older and younger siblings.

Similar variation occurs with respect to the time when the PA takes place: How long are the sessions, how many sessions? Generally, the morning is best for thinking activities, but maybe PA could also fit into the afternoon when the timetable perhaps feels a little looser. If a big and complicated piece of work is being peer assessed, a good deal of time might be needed, but this should be broken into smaller sections of no longer than one period, and some structure provided so that students do not go off track. Make sure you give enough time so that the PA is actually finished in the tine specified.

What degree of justification for opinions is expected of the assessor? In the beginning it will be hard enough to get peer assessors to give suggestions for improvement, without expecting them to say why they think what they think. But with experience, peer assessors may become more adept at this—and also be more careful about not giving an opinion until they are sure they can justify it.

Will all PAs be confidential to the assessing pair and the teacher, or will they be made publicly available? At the start you will want to keep the PAs confidential to each assessing group. Once you have checked some of them for reliability, and you are satisfied about reliability, you may wish to operate a more open system. This could of course become competitive, and you would not wish what you had hoped would be a positive social experience degenerate into a competition.

Another issue is the extent to which the process of PA is monitored by supervisory staff. With PA in one class, it is relatively easy for the teacher to keep an eye on the situation. But PA between classes can become tricky in terms of keeping an eye on the situation. Obviously, you will want to be alert to any problems and able to nip them in the bud.

The extent to which the reliability and validity of the PA is moderated by supervising teachers is also an issue. While this generally comes up mainly with summative quantitative PA, it can also be relevant where students are giving elaborated verbal feedback. Sometimes this feedback may seem so strange that you are tempted to intervene—but remember, it is for the assessee to comment first, so give them the chance to say that the PA is nonsense.

Inspecting a sample of the assessments is particularly important where the assessment is summative. Is the task a simple surface task requiring limited cognitive engagement, or a highly complex task requiring considerable inference of the part of assesses, or does a simple initial task develop into increasingly complex tasks? If it is complex, you might be particularly inclined to pay some attention to the process.

In relation to this, what quantity and quality of feedback is expected, and is this elaborated and specific, or more concise and general? Time will be a major factor here. Initially, you might want to ask your assessors to give two positive points of feedback and two points where improvement might help. Should this latter be about a minute point (such as a spelling) or much broader (such as the structure of a piece of writing), or do you want to say that one should be broad but the other can be small?

To what extent is the feedback tending toward the objective and definitive, as it might be in response to a simple task, or to what extent more subjective, as it might be with a more complex task? What effect might this have on the amount of disputation that ensues? Is there time for the assessees to actually make all the suggested improvements?

How are assessees expected to respond to feedback? Are their revisions to be none, few or many, simple or complex? Again, given the time constraints, you may wish to put some sort of quota on this—perhaps a maximum of three revisions to be done in 20 min, or some such.

What extrinsic or intrinsic rewards are made available for participants? The USA has been much criticised for its use of extrinsic rewards. First, it is worth thinking about what the students might get out of PA in intrinsic terms. Once over their first shock, do the assessors get more pride in what they are doing, more involvement as they engage their assessee(s) in conversation, and so on? Do assessees seem to respond at all to the deeper and quicker suggestions for improvement they get from a peer assessor (as compared to a teacher)? Might this activity become self-sustaining without it having to be inflicted on the students? Can you reflect this back to the students?

Another issue is whether the PA is aligned with the traditional forms of assessment. Will the PA be taken into account when grading students at year end, for example, or does all of the assessment information for this have to be generated by the teacher? Do all students have to sit formal examinations irrespective? If so, is there any way you can use PA to help them prepare for these examinations?

What transferable skills relevant to other activities might be measured as by-products of the process? Are you seeing improved social or communicative skills which might generalise beyond the PA situation? Or writing skills or presentation skills? Or music skills or physical education skills? Might any of these endure beyond school or university? These are important by-products which should be taken into account when you are considering the success or otherwise of your PA project.

Finally, is the PA being evaluated, as one would hope with any new venture, or is its success or failure just assumed? Time spent evaluating is costly, and could be spent doing something else, but if you are to persuade the powers that be (within your school/university or wider than that) that PA is worthwhile, you are going to need some evidence that looks at least a little bit objective.

Thus, it is clear that PA is not just one method, but many. Labels can be given to some of these variations, distinguishing formative from summative PA, qualitative from quantitative, structured from unstructured, unidirectional from reciprocal or mutual, sameyear from cross-year, and same-ability from cross-ability PA, for instance.

Using Table 1, teachers will be able to decide and see what kind of PA they intend to implement. Importantly, because all the variations are listed, teachers will not overlook any issue they should have considered. There are rather a large number of variables in the table, and some researchers have proposed clustering these. The difficulty is that different researchers propose different clusters, so I have left the list un-clustered. Now let us consider theory in PA, and see how we can relate that to the typology just explicated.

#### **6. Theoretical Issues in PA**

PA theory is rather scarce. An early contribution in 1998 [19] regarding peer learning in general introduced the idea of distributed cognition leading to distributed metacognition. Subsequently a conceptual framework for PA in teacher education was articulated [20]. Later, researchers used expectancy theory regarding students' motivation for PA, emphasising the belief that performance would lead to valued outcomes [21]. The cognitive underpinnings of PA were explored in 2010 [22]. In 2016, a model was advanced [23] describing how PA operated in marking/grading, analysis, feedback, conferencing and revision, noting that investigating learning opportunities was more useful than investigating student/instructor grade relationships. More recently, researchers [24] have explored the theoretical underpinnings of PA in a digital world.

However, a more comprehensive and integrated theoretical model of the cognitive processes involved in PA has been proposed [18] (chapter 4, pp. 103–109), encompassing: organisation; cognitive conflict; individualisation and engagement; scaffolding and error management; communication; affect; intersubjectivity; practice and generalisation; reinforcement; metacognition, self-regulation and self-efficacy; and levels of learning. This illuminates many of the processes which may occur during PA, either deliberately or accidentally. This model has since been simplified and further developed, as in Figure 1.

**Figure 1.** Theoretical Model of Peer Assessment.

— Of course, not all partnerships will show all these features when they are first developing. Some may not show many features even when somewhat developed. The purpose of the theoretical model is to enable partners (perhaps with professional help) to see what new functional areas their relationship might develop into next. A more elaborate relationship is likely to be more satisfying for both assessor and assessee, and lead to enhanced educational outcomes. The model also gives professionals a framework within which they can counsel partners towards more effective experiences. Thus, it has strong practical implications for improvement of PA quality. A most important point is that both partners can be expected to benefit in all these ways—both as assessor and as assessee.

The model also has implications for research. The design of new PA interventions to be evaluated could be tested against this model, to ensure all aspects had been considered. The other question is which of these elements might be the most effective in any particular context. Research could possibly investigate the relative efficacy of each part of the model, while holding the other parts constant, but this would only be relevant to the context in which the PA was occurring. It may be that the whole proves greater than the sum of its parts.

#### *6.1. Organisation*

Many of the issues regarding organisation will have become clear from reading of the foregoing typology section—what kind of PA is proposed and what planning decisions do you need to make [25]? What organising time for PA will do is enable participants to get together and focus on the task(s) in hand. This has an effect on attention and concentration. Issues include the need and pressure inherent in PA toward increased time looking at the task and maybe thinking about it (time on task) and time observably involved in doing something active leading to task completion (time engaged with task)—the two being different concepts. The need for both helper and helped to elaborate goals and plans, the

immediacy of feedback possible within the small group or one-on-one situation, and the variety of a novel kind of learning interaction are also included in this category.

The issues of gender and race can also present problems. First, gender—should you pair with the same gender or mix genders, or does it not matter? There is no one right answer here. The main issue may be how to engage males, since often more females volunteer for such activities. Nonetheless, the presence of a male figure is particularly important for boys, but may also be important for girls.

Second, race—should you try to pair participants of the same race, or does it not matter? One problem here is determining exactly how the participants are located in terms of race, which might depend on how recently they have arrived in the host country (this might not be an issue for participants who have generations of experience of the host culture and speak the host language as a native would). However, many recent immigrants might call themselves citizens of their new country, but their culture and beliefs might still owe much to the country of origin of themselves or their parents. This may lead to issues of acceptance. Further, even within one country there are often a great many cultural and religious differences. So, you cannot assume that because both participants come from the same country, they will be well matched—indeed, sometimes quite the opposite.

#### *6.2. Cognitive Conflict and Co-Construction*

From Organisation, we proceed to more abstract and psychological variables. Conflict and Co-Construction are very much part of informal learning. Conflict is a clash of opposite opinions, which need to be worked through and resolution found. Co-construction is collaborating with others in building knowledge together—jointly investigating, analysing, interpreting and reorganising. Both are needed to liquify primitive cognitions and beliefs.

When the pair first meet, they will need to talk to decide their first area of inquiry. Then they will need to find out where each other is in their area of inquiry. What they will discover is not only that the knowledge of both is somewhat patchy, but that the assessee (and maybe the assessor!) holds some ideas very dear which are not helpful—in fact, they are wrong, or at best unduly simplistic. What will follow is a somewhat heated conversation where the pair try to determine a consensus on what they both already know about the subject which is actually correct. This is known as a period of "cognitive conflict"—disagreement about thinking.

Once the pair have established this baseline, they are in much better shape to proceed to build correct knowledge which is new for the assessee (and maybe for the assessor). However, this will be done gradually, and result in the assessee (and maybe the assessor) re-tuning their existing knowledge into something more complex and refined, adding new elements to it in a way that coheres rationally with what is agreed to be already known, or perhaps even restructuring existing knowledge to accommodate the new knowledge. This kind of "cognitive co-construction" by mutual agreement leads to a state known as "intersubjectivity" or shared understanding (for this area of inquiry) between the pair. Intersubjectivity is the sharing of subjective states by two or more individuals—they agree on a given set of meanings or a definition of the situation [26].

The notion of cognitive conflict reflects Piagetian schools of thought [27]. It concerns the need to loosen cognitive blockages formed from old myths and false beliefs by presenting conflict and challenge via one or more peers. Teachers focus on learning as if the pupil was a blank slate. But, in fact, the pupil's head is full of all kinds of stuff, much of it factually or conceptually erroneous. So, unlearning wrong stuff is as important as learning new stuff. Peers can be good at rooting out misconceptions in their partner—they certainly have more time for it than the teacher does.

The Russian psychologist Vygotsky [28] was famous for investigating cognitive coconstruction between more able and a less able participants. He found that it was important that the level of challenge was appropriate for the assessee—within their "zone of proximal development" (the level where the assessee could not perform unaided but could perform successfully with some help from a more knowledgeable other).

From Conflict and Co-Construction, there are five different options, all of which interact with each other and have an influence on the linear steps which follow them (see Figure 1) [29]. We will take these five variables in the order in which they appear.

#### **7. Engagement**

Engagement describes intensity of arousal and involvement with the task. It encompasses curiosity, interest, attention, responsiveness, investigation, discovery, anticipation, persistence and initiation [30]. Any activity which is of interest to the pair will result in a focus of attention on the joint interactive task (and pairs should not try to engage with activities which are only of interest to one member of the pair). There will be concentration and arousal gains. Of course, if one member of the pair becomes too much like a teacher (didactic—maybe even bossy), the concentration and response of the assessee may suffer. So, some form of equal sharing of the interactivity is needed, which can be helped if the assessor is not an "expert" in the field (or is pretending not to be).

A great advantage is the immediacy of response from one to another [31]—especially high in face-to-face contact, albeit rather slower in messaging at different times (asynchronously) via the internet. This keeps the interaction speeding along at a good pace, even if there are diversions where the members of the pair do not agree and a compromise has to be negotiated. As the relationship develops, pairs are able to make goals and plans for the future about issues they will explore in future meetings [32]. Of course, there will be lots of talking, so any hope that PA will be quiet is unrealistic—there will be noise—but of course it will be productive noise and it is unlikely to disturb the pairs.

#### **8. Individualisation**

An immediate benefit of PA to both members of the pair is that both are receiving more than usual individual attention, intended to be specifically relevant to their immediate concerns. This might mean that the assessee gets more attention than in a regular class, while the assessor also gets more attention than in the course of usual everyday events, and in both cases this attention is closely focused on the mental activity of the other person, i.e., it avoids other distractions, is highly engaging, and requires new thinking. Nonetheless, more individual attention would rapidly lose its appeal if it had no active content.

An associated advantage is Individualisation—the content, pedagogy and pace of learning are based upon the unique abilities and interest of each learner—and perhaps their culture, socioeconomic status, language, gender, motivation, ability/disability, personal interests and so on (this is also known as Differentiation). Each member of the pair will increasingly respond to their partner in a way which is tailored to the needs of that partner. As time goes by, the assessor will modify the difficulty and other characteristics of the material under discussion so that the individual assessee can readily understand it—although this will take some time to develop [33,34]. Of course, the partner should not be "dumbing down" the issue too much so it is too easy—a certain amount of challenge is always needed.

Various forms of interactivity will take place. There will be many opportunities to question—from both members of the pair. A question is any sentence which has an interrogative form or function. Assessor questions act as instructional stimuli suggesting elements to be learned. Young children are often very good at asking questions, especially if they are encouraged—although sometimes their questions are too big to find an answer [35]. Equally, the assessor can question strategically—not offering just a closed question or one where the answer is self-evident, but asking a question which leads the student on from where their thinking has got to. Learning skilful questioning is highly desirable in assessors—and of course assessees will learn it and use it with their eventual assessees in later years. A good question promotes a high quality of answer—not just "yes" or "no", but an elaborated statement which indicates the reasoning behind the student's opinion. Of course, the opinion may be quite wrong, and the partner then has to skilfully question to get the student to see alternative perspectives.

#### *8.1. Communication*

Much of PA is about communication—the act or process of using words, sounds, signs, or behaviours to express or exchange information or to express your ideas, thoughts, feelings, etc., to someone else. Listening, explaining, questioning, summarising, speculating and hypothesising are all valuable skills of effective PA which should be transferable to other contexts. PA pairs will communicate in the common vocabulary of everyday people, not in the rather technical and complex language teachers sometimes use. This enables children to be much more talkative than they might otherwise be. Vygotsky [27] said that you only really know something when you have the language to express it to another person, and PA gives students the chance to develop the language to express their thoughts—including their deepest thoughts, which might quite surprise their partner. Both parties also need to carefully listen to the other as they attempt to explain their point of view, then ask questions which lead to further elaborations—or maybe a realisation that the first view was wrong or incomplete.

Of course, there needs to be care that a given explanation is not too abstract for the assessee to grasp. Exemplification can be very helpful here—a concrete example often works wonders. Students often make their initial stumbling explanations too longwinded and partners can help them by encouraging them to clarify, simplify or summarise. Summarising teaches assessees how to discern the most important ideas in a text, how to ignore irrelevant information, how to integrate the central ideas in a meaningful way and improves their memory.

Some students will be reluctant to offer half-formed thoughts, and the partner will encourage them to say something, because everything can be revised and improved later once you have something to start with. Similarly, the idea of rehearsing an idea should be shared (not just repeating it but adapting it at each stage), so that with continuous improvement it will eventually be worth sharing with other pairs or the whole group [36]. As an idea develops pairs can speculate freely or hypothesise, allowing their imagination to run riot, then later bring their ideas back and rationalise or summarise them for wider consumption. Needless to say, this process presents many learning opportunities for the assessor as well.

#### *8.2. Social*

Every learning interaction requires the use of social skills by both members of the pair [9]. Social skills are the skills we use to communicate our messages, thoughts and feelings and interact with each other, both verbally and non-verbally, through gestures, body language and our personal appearance. At a more advanced level, such skills include empathy and self-control. If they do not already know each other, at first meeting both assessor and assessee might need some way of introducing themselves and beginning to talk about what might be learned first. If need be, they can be given some training and a list of tips about this. The notional assessor will need to learn not to be bossy and not to talk too much of the time—in other words, not be too much like a professional teacher. The assessee will need not to be over-powered by their partner and be prepared to expose their initially rather faulty thinking, as well as accepting both criticism and praise without becoming upset or over-excited. Both members of the pair will need to show some social tolerance of the peculiarities of their partner. Of course, social skills developed with one partner will only partly transfer to interaction with a new partner. Apart from these functional issues, the participants should develop a sense of social connectedness and trust in each other.

#### **9. Emotion (Affect)**

Emotion has a particularly strong influence on selectivity of attention, as well as motivating action and behaviour. A trusting relationship with a peer who holds no position of authority might facilitate self-disclosure of ignorance and misconception, enabling subsequent diagnosis and correction that could not occur otherwise. Modelling of enthusiasm and competence and belief in the possibility of success by the helper can influence the

self-confidence of the helped, while a sense of loyalty and accountability to each other can help to keep the pair motivated and on-task.

Negative emotions such as anxiety, depression, anger and frustration can be the cause or effect of problems with learning and lead to a maladaptive and self-defeating pattern of behaviour which prevents learning [37]. At first meeting, a degree of anxiety is normal. Both partners are entering a new situation, which is unknown. As the pair get to know each other better and learn to trust each other (bearing in mind the assessor is not the same kind of authority figure as a teacher), their anxiety about each other should reduce and their self-esteem (or self-confidence) should grow. Of course, for some pairs, there might be a longer period of social as well as cognitive conflict before things settle down.

In the longer run, other emotional factors come into play. The assessee might be anxious about the material to be assessed. Here it will be important that the assessor is positive and encouraging and reassures the assessee that they felt the same way before they learned it, but now they are quite happy and confident with it. In other words, the assessor should be encouraging and demonstrate a model of coping and confidence. As the assessee becomes more confident, they will feel more able to disclose their thinking, which may well be faulty, and this will enable diagnosis and correction by their partner.

As time goes on, both members of the pair should develop more certainty about what is being assessed, and with that will come higher desire and confidence (motivation) to proceed to the next thing [38]. Added to this is the fact that the partners come to be accountable to each other—because they have a better and better relationship, they do not want to let their partner down. This gives them a stronger sense of responsibility for their learning. This responsibility leads to a stronger sense of ownership of their own learning—it is truly theirs rather than being inflicted upon them by an outside organisation.

#### **10. Prompting (Scaffolding) and Error Management**

Once assessees have the confidence to express their thinking out loud, it will become evident that they are making errors, or perhaps leaving gaps in their line of reasoning. How should assessors intervene? Particularly when one partner is more able in the area of interest than the other, they are likely to be involved in "prompting"—saying something to encourage or remind someone to do or say something, without telling them what they have to say [39]. Prompting is definitely not just telling them the "right answer"—if assessors do this, they are paying too much attention to correctness and not enough to the development of the thought processes required for the assessee to arrive at the right answer by themselves. Of course, the latter takes longer, but the assessee learns the thinking involved and can then use these skills to solve other similar problems.

"Scaffolding" is another word sometimes used in this context [40]. When grasping the concept is just too difficult for the assessee, the assessor provides some steps which lead the assessee in the right direction—without giving the answer. Like prompting, this is a skill that assessors have to develop over time—another of the benefits for them.

Error management is directed at dealing effectively with errors after they have occurred, with the goal of minimising negative and maximising positive error consequences. One of the major issues is the question of how errors should be corrected [41]. Even when the assessee's error seems glaring to the assessor, the assessee may be very emotionally attached to it, so it is no use just saying that is "wrong". The first issue is identifying the error—sometimes the assessor will miss errors without noticing or at first may choose to concentrate on major errors and overlook minor errors. When the assessor spots an error, they should not immediately go into a mini-lecture about it. Instead, they should wait till the end of the sentence then simply point to or say what the error was, and see how the assessee responds—they may be able to self-correct, which is a much more productive way of progressing [42].

The other issue is diagnosing the kind of error—what does it tell us about the assessee's faulty thinking and what might we need to address to resolve that faulty thinking? It follows from what has been said above that errors need to be discussed between the partners, so they can arrive at a newly constructed form of truth before going on. If the assessee still cannot grasp the concept, the assessor may have to resort to giving a more concrete example or modelling or demonstrating how that bit of the problem can be solved. Again, skill development for the assessor. One of the great advantages is that errors can be corrected almost immediately. In a classroom, students might have to wait much longer, unless they were using some kind of computer application which offered corrective feedback.

Generally, errors should be corrected in a positive way through discussion, prompting, scaffolding and if necessary, modelling—demonstrating the relevant behaviour [43]. Assessors should also remember that once the error has been identified, they should pause or allow some "wait time" to allow the assessee to try to self-correct [44]. With a bit more thinking they might manage it on their own, and that would make for better learning than too much interference by the assessor. However, particularly with very difficult concepts, the assessor will need to monitor and control the flow of information so that the assessee is never presented with too large a chunk of material which they cannot assimilate. The concept of zone of proximal development is again highly relevant. The cognitive demands upon the assessor in terms of monitoring learner performance and detecting, diagnosing, correcting and otherwise managing misconceptions and errors are great. Herein lies much of the cognitive exercise and benefit for the helper.

The greater the differential in ability or experience between the assessor and the assessee, the less cognitive conflict and the more scaffolding might be expected. Too great a differential might result in minimal cognitive engagement for the assessor and unthinking but encapsulated acceptance with no co-construction. Of course, if the assessor is older, more experienced, and therefore more credible but actually has no greater correct knowledge or ability than the helped, then a mismatch and faulty learning might occur in a different way.

#### **11. Practice and Fluency**

PA enables and facilitates a greater volume of engaged and successful practice, leading to consolidation, fluency and automaticity of thinking, and social, communicative and other core skills. Much of this might occur implicitly, i.e., without the assessee or assessor being fully aware of what is happening.

PA might occur more frequently than interaction between the teacher and each student in the classroom. So, there are more opportunities to repeat similar tasks until the principles are really well understood. This also enables and facilitates a greater volume of engaged and successful practice—the actual application or use of an idea, belief, or method, as opposed to theories relating to it. Of course, the practice needs to be correct practice, or the assessee will overlearn mistakes! [45].

This more frequent practice leads to greater consolidation and fluency in understanding and performance. Someone is said to be fluent if their use of the language appears fluid, smooth, natural, coherent, and easy. Fluency is characterised by the language user's automaticity, their speed and coherency of language use, and the length and rate of their speech output. The flow is smoother because some of the learning has become automatic—it does not have to be consciously remembered but is put into operation without really thinking about it. The more learning is at the automatic level, the greater will be the retention of that learning—it is truly embedded in the assessee's consciousness. Much of this automaticity is implicit, i.e., the assessee is not really consciously aware of it.

#### **12. Feedback and Reinforcement**

Another great benefit of PA is feedback—information about their performance given to learners to praise positive aspects and point out areas needing improvement—which of course then needs to be acted upon. PA increases the quantity and immediacy of feedback to the learner very substantially. Feedback can also help develop the leaner's capacity to monitor, evaluate and regulate their own learning [46]. Feedback from assessors is more

frequent than with classroom learning [47]. Assessees can be frequently encouraged as they struggle with difficult concepts. As they say things that are partially right, the assessor can start by pointing out what they have said which is good and useful, then move on to point out where their reasoning is less good (always positive before negative) [48].

Positive reinforcement is the action or process of encouraging or strengthening a pattern of behaviour by associating some positive event with the behaviour, so it is more likely to occur again in the future. Usually positive reinforcement will be praise, but this should clearly specify exactly what is being praised. The role of praise is an interesting issue. One might say that all students should be praised as much as possible. However, even some young children are not happy with an excess of praise, perhaps because they feel they have to learn trust in the person who is praising before they can accept it [49]. Praise may not be appropriate in that context at that time. However, it can be helpful if assessees can be encouraged to give praise to assessors also, since the feedback process should be two-way.

Where praise is given a variety of forms of verbal praise are need, not just a routine and repetitive "good". In addition, the praise needs to be accompanied by non-verbal signals, so that the assessee is convinced that the assessor actually means it. Giving a variety of both verbal praise and non-verbal praise is a skill that has to be developed, and this extends the assessor's repertoire. Beyond the partnership, there may be explicit reinforcement for the pair in the form of social acknowledgement and status, official accreditation, or even more tangible reward. However, tangible reward which is not necessary is not likely to act as a reinforcer.

Some of this feedback and reinforcement will be implicit (the partners not consciously aware of it), but some will be explicit (the partners are consciously aware of it). However, indiscriminate reinforcement which is not linked directly with good performance or is predominantly for effort rather than performance will not be nearly as effective in promoting good learning. Explicit reinforcement might stem from within the partnership or beyond it, by way of verbal and/or non-verbal praise, social acknowledgement and status, official accreditation, or even more tangible reward. However, reinforcement should not be indiscriminate or predominantly focused on effort.

#### **13. Generalisation**

Generalisation accepts that humans recognise the similarities in knowledge acquired in one circumstance and that this enables transfer of that knowledge into new and somewhat different situations. Once the assessee has really learned a concept, they can begin to apply it to other similar problems. An obvious example would be in mathematics, where once a principle is grasped, it can be applied to many similar problems. PA can lead to generalisation from the specific example in which a concept is learned, extending the ability to apply that concept and its developmental variants to an ever-widening range of alternative and varied contexts [50]. In the first instance, much of this would be supported by the assessor, but as time goes on, it should become increasingly independent—the assessee managing this without much scaffolding. Likewise, in the first instance, it would be implicit, but as time goes on and the assessee is made aware of what is happening, it should become increasingly explicit.

#### **14. Metacognition**

Metacognition is awareness and understanding of one's own thought processes thinking about thinking—which leads to the ability to control and direct those thought processes (see self-regulation in the next section). In a learning situation, it means becoming sharply aware of how you are thinking to learn, and consequently how that thinking can be made more efficient [51]. Assessors will usually become more metacognitively aware first, then the assessee may then follow. Metacognition is always explicit—it is always fully in consciousness and is intentional. It can be summarised in the catch phrase: I know I know; I know I know how; I know I know when and if.

As the learning relationship develops, both assessor and assessee should become more consciously aware of what is happening in their learning interaction, and more able to monitor and regulate the effectiveness of their own learning strategies in different contexts. Development into fully conscious explicit and strategic metacognition not only promotes more effective onward learning, it should make the assessor and assessee more confident that they can achieve even more, and that their success is the result of their own efforts. In other words, they attribute success to themselves, not external factors, and their self-efficacy is heightened.

#### **15. Self-Monitoring and Self-Regulation**

As learners become more sophisticated, they become more metacognitively aware and, through this, more able to self-monitor their own thinking. Self-monitoring can be defined as the process of attending to one's own actions and noting or recording the presence or absence of a specified relevant behaviour [52]. This of course requires multitasking—not only thinking, but also thinking about thinking. So, it is not easy. Beyond this, the learner should become more able to self-regulate or control their thinking about similar and then new topics in different contexts, so that many false paths are avoided and the logical consistency of their reasoning improves [53]. Self-regulated learning refers to one's ability to understand and control one's learning environment and includes goal setting, self-instruction, and self-reinforcement. This self-regulation can be both implicit and explicit.

#### **16. Confidence (Self-Efficacy) and Self-Attribution**

As the learner develops metacognition and self-regulation, an emotional change is likely to occur. Because the assessee is so much more aware of their thinking and in control of it, they feel increasingly confident about their mastery of this area of inquiry. Confidence means believing in your own ability, skills and experience and your ability to succeed. Of course, some students are over-confident, but many are under-confident. As assessors become increasingly competent, they also become increasingly confident. Confidence is also known as self-efficacy [54,55].

Furthermore, the students attribute this improvement to their own ability, rather than to the support of the assessor or any even more distant external factors. Self-attribution bias refers to an individual's tendency to attribute successes to their own personal skills and any failures to factors beyond their control [56]. Assessors attribute success to themselves as well as the efforts of the assessee. This self-attribution can be summarised in the catch phrase: I want to know; I want to know how, when, if; I believe I can know how, when, if.

#### **17. Level of Learning**

Surface learners have an unreflective approach—there is a focus on memorising and reproducing the learning material, knowledge is fragmented, facts are not elaborated upon and there is no real interaction with or connection between with ideas. The underlying argument is not comprehended and the learning task is treated as a monotonous chore. The learning is driven by external incentives or punishments, such as an impending test, i.e., is extrinsic. The aim is to recite and regurgitate the material inactively, forgetting it as soon as the external accountability requirement has passed [57].

By contrast, deep learners relate the topic and its ideas to past knowledge and experiences. They think critically about newly learned material and tie it in with information from other sources. They recognise a structure in the content. Their motivation comes from within and is intrinsic—they want to learn. They aim to understand the meaning behind the material and can create new arguments based on the new information. They retain much of what they learn.

Obviously, the aim is to enhance deep learning and reduce surface learning. However, all learners may need to engage at the level of surface learning before they can develop into deep learners, in relation to any particular topic of inquiry. The role of the assessor in

encouraging deep learning is cognitively challenging for them, and enhances their own level of thinking. As the PA relationship develops, the model continues to apply as the learning moves from the shallow, instrumental, surface level to the strategic level and on to the deep level as the students pursue their own goals rather than merely those set for them.

#### **18. Type of Learning**

Learners need to possess and be aware of three kinds of knowledge: declarative, procedural, and conditional. Declarative knowledge is factual information that one knows; it can be declared—spoken or written. Procedural knowledge is knowledge of how to do something, formed by doing, of how to perform the steps in a process; for example, knowing how to pronounce a multi-syllabic word. Conditional knowledge is about when to use a procedure, skill, or strategy and when not to use it—why a procedure works and under what conditions; and why one procedure is better than another. For example, learners need to know under what conditions to draw a diagram to more effectively illustrate points that they are making. In PA, all these kinds of learning are needed [58]. However, the usual tendency is to over-emphasise declarative knowledge at the expense of the two other kinds, so this needs to be struggled against.

These affective and cognitive outcomes feed back into the originating subprocesses—a continuous, iterative process and a virtuous circle. Of course, it is unlikely that PA in practice will neatly follow these linear stages. Some may be missing (and the teacher can prompt for their insertion). Sometimes one will occur before another which appears to follow it in the model. Most likely a number of events will occur which seem to be combinations of items. Even where students work through the whole model on one task, they may begin again at the outset on a new task.

#### **19. Conclusions**

Different individuals within the same learning partnership, and with different partner relationships, are likely to follow somewhat different pathways to the same learning goals. If one characteristic of the assessors and assessees is that they are developmentally young or slow learners themselves, then few of the channels in the model will develop automatically, intersubjectivity is likely to be primitive, and more training and closer monitoring, coaching and management will be necessary. Although all channels in the model might be eventually utilised to some extent by both members of a pair, their different learning styles might lead them to use some channels more than others in ways unique to themselves. This highlights the individualisation which is inherent in PA, but takes the notion much further than the mere individualisation of learning tasks or surface learning behaviours.

The point of the model is to enable learners (whether assessees or assessors) to see what channels they are currently not using enough or not using at all and encourage them to use additional channels as suits their personal learning styles to maximise the effectiveness of their learning. For professionals, this theoretical model is something of a mixed blessing. Just when they thought they knew how PA should work, along comes a model that makes everything seem rather more complicated. Of course, professionals should be encouraged to think of the model in terms of a step-wise progression for each pair. Having identified which elements any PA pair are not doing, the professional selects the one most obviously missing and desirable element and advises the pair to engage in it. Later, she/he selects the next most obviously missing element, and so on . . . So, professionals are never actually faced with trying at one moment to get the PA pair to engage in all the elements, which is likely to be too complex and counter-productive.

In that respect, a useful task for professionals is to explain the model to users in simple terms, discuss how it applies to present learning and how future learning might take advantage of additional opportunities. In this respect, it may become a feature in initial training in PA—or perhaps in a second phase of training after some initial experience. It can provide a framework for helping the learning partners themselves to reflect upon their own process—a tool for self-assessment or PA of the process which might further

enhance metacognition [18]. The model can also be used profitably as a template (or observational checklist) for monitoring PA as it is happening—a tool to structure monitoring and diagnostic fault finding.

For future research, the template provided by the model should prove useful for the design of new PA methods. Further research might seek to explore the validity of the model empirically, or of the relative effectiveness of different elements of the model with different learners. Research into the impact of the use of the model in monitoring implementation integrity (quality of delivery of an intervention) also would be worthwhile.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** No new data were gathered during the research reported in this article.

**Informed Consent Statement:** Not Applicable.

**Data Availability Statement:** Not Applicable.

**Conflicts of Interest:** The author declares no conflict of interest.

#### **References**


## *Article* **Integrated STEM for Teacher Professional Learning and Development: "I Need Time for Practice"**

**Andrea C. Burrows 1,\* , Mike Borowczak <sup>2</sup> , Adam Myers <sup>3</sup> , Andria C. Schwortz 1,4 and Courtney McKim <sup>5</sup>**


**Abstract:** This study compares three pre-collegiate teacher professional learning and development (PLD) integrated science, technology, engineering, and mathematics (STEM) experiences framed in astronomy. The study is set in the western United States (USA) and involves 60 pre-collegiate teachers (in the USA these are K-12 teachers) over the course of three years (June 2014–May 2017). During the PLDs, astronomy acted as a vehicle for pre-collegiate STEM teachers to increase their STEM content knowledge as well as create and implement integrated STEM classroom lessons. The authors collected quantitative and qualitative data to address five research questions and embraced social constructionism as the theoretical framework. Findings show that STEM pre-collegiate teachers are largely engaged with integrated STEM PLD content and embrace astronomy content and authentic science. Importantly, they need time to practice, interpret, translate, and use the integrated STEM content in classroom lessons. Recommendations for PLD STEM teacher support are provided. Implications of this study are vast, as gaps in authentic science, utilizing astronomy, PLD structure, and STEM integration are ripe for exploration.

**Keywords:** professional learning and development; teacher; authentic science; integrated STEM; astronomy; partnership and collaboration; interdisciplinary; differentiating instruction; pre-collegiate classrooms; real-world connections

#### **1. Setting the Stage**

A study in 2003 asked, 'What about the teachers?' and probed into studies on teacher understanding of astronomy [1]. At the time, there was a dearth of specific studies relating to pre-collegiate teachers (or K-12 teachers in the USA) and astronomy education, although some researchers had already begun investigating the need for pre-collegiate, astronomy education [2]. In 2008, another study highlighted the increase in research related to astronomy projects and future astronomy education research efforts, including investigating technological solutions, utilizing real scientific data, and exploring ongoing professional learning and development (PLD) [3]. Today, almost 20 years after questioning where teachers fit into astronomy education, researchers are exploring several teaching realms and extending studies to include expanded research questions and considerations [4–7]. This study adds to the existing literature and expands the literature by examining teacher professional learning and development with integrated STEM over multiple years. The purpose of the study was to focus on STEM integration as a central PLD tenet and demonstrating connections between STEM disciplines in solving real-world problems [8].

**Citation:** Burrows, A.C.; Borowczak, M.; Myers, A.; Schwortz, A.C.; McKim, C. Integrated STEM for Teacher Professional Learning and Development: "I Need Time for Practice". *Educ. Sci.* **2021**, *11*, 21. https://doi.org/10.3390/educsci 11010021

Received: 26 October 2020 Accepted: 8 December 2020 Published: 5 January 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Pre-collegiate teachers instruct students from about age five through age 18, and the teachers explore how to engage these students during their teacher education program prior to certification and in PLD sessions after their certification. Most of the 'before' work occurs in an academic setting, while most of the 'after' work occurs in teacher PLD. In this article, 'teacher(s)' refers to pre-collegiate or K-12 teachers.

Astronomy PLDs, which lend themselves to physical manipulation and modeling, can utilize the USA's Next Generation Science Standards (NGSS) [9]. NGSS include components of science and engineering practices (SEP) and crosscutting concepts (CCC) that enhance authentic science experiences to spark the imagination of teachers' lessons [10,11]. Thus, there is a connection between astronomy, STEM disciplines, and the NGSS, even if a teacher is not teaching astronomy. The authors caution readers that astronomy and physics disciplines require content knowledge in all of the STEM disciplines (including computer science), and all too often authentic-science PLDs fail to embrace STEM in a cross-disciplinary manner. Instead, PLDs often present concepts in an isolated manner without showing the complexities that could create rich experiences for the teachers.

To add to this pre-collegiate STEM picture, current research encourages teachers and communities to incorporate extracurricular opportunities, classroom experiences, and hands-on projects to increase pre-collegiate students' STEM interest [12,13]. Hence, authentic science, and what that term actually means, is currently in the spotlight [11]. In this article, 'authentic science' refers to a real-world problem viewed through the teachers' questions, data collection, analysis, and community dissemination. Therefore, the problem facing researchers now is in the understanding of how to effectively use astronomy in authentic science settings in teacher PLDs to provide them with the support needed to translate the PLD content and pedagogy into useful STEM classroom experiences.

Because astronomy intrinsically encompasses cross-curricular concepts, such as light and energy, it is a natural approach to teach integrated STEM [14]. After an extensive literature review, researchers found STEM integration "emphasized the need for complex, authentic, or real-world problems [with] shared practices, skills, and concepts across disciplines as a motivation for integrating those disciplines [and] frequently encourage or even require student-centered teaching strategies and classroom structures, and collaboration and teamwork . . . [as] essential ingredients . . . " [14] (p. 11).

As such, the PLD teams selected astronomy as a vehicle to teach integrated STEM because students interested in STEM are often attracted to STEM through authentic science experiences [13]. Additionally, researchers have shown that astronomy is a vehicle to teach integrated science and math components [15]. As the authors structured the PLD they asked, 'If astronomy is the "gateway science" to physics and coding, then wouldn't the scaffolding, intersection, and support of the disciplines be vital for teachers to teach STEM disciplines well?' Hence, the authors argue that by offering PLD focused on astronomy, teachers can strengthen their STEM content knowledge and their understanding of how the STEM disciplines integrate with one another.

Further, the authors argue carefully structuring astronomy-rich teacher PLD can offer the support needed for teachers to use astronomy as a focus for STEM lessons, including authentic science practices. Carefully structured PLDs would provide a translation of the pure STEM astronomy content into STEM discipline topics as presented in the NGSS [9]. For example, skills such as computer coding, telescope control, and database manipulation should be included in astronomy PLDs to position the PLD facilitators towards understanding what the teachers understand and what is actually working for them. One participant aptly stated, 'I need time for practice,' and this short sentence altered the PLD teams' perceptions. Hence, what does the literature show researchers about PLD, authentic science, integrated STEM, and where to go next with teachers?

#### **2. Literature Review**

For the PLD leaders, including those in astronomy, researchers have identified general effective PLD practices—which should translate to the STEM classroom—including

items such as core disciplinary content, materials, hands-on activities, real-world issues, reflection and collaboration [16,17]. Specifically, researchers include: (a) Improvement of content knowledge, pedagogy, and dispositions [18–21]; (b) Creation of quality instructional materials [19,22–24]; (c) Use of authentic science and inquiry practices [11,19,25–27]; (d) Consideration of socioscientific issues and links to other educational areas [19,27]; (e) Iterative cycles of use and reflection [19,20,28]; and (f) Partnership development [29]. Although not the focus of this article, the three PLDs described here fit these aforementioned criteria. These effective PLD practices are in addition to or expansion of the integrated STEM practices described in the previous section (including real-world problems, shared practices/skills/concepts, and student-centered teaching strategies).

In any PLDs conducted, along with the general attributes mentioned earlier, authentic science should be utilized if possible. As stated earlier, authentic science in this article is defined as a real-world problem viewed through the teachers' focus on and questions about research, data collection, analysis, and community dissemination. Authentic science, according to Spuck [11] and other researchers, incorporates: (a) A real-world problem, (b) Exploring and summarizing information, (c) Using scientific instruments and technology, (d) Using mathematics in data analysis [30], (e) Analyzing evidence and using findings to form conclusions, (f) Developing and refining questions and presenting new questions as a result of the work, (g) Developing and refining procedures and methods, (h) Communicating the methods used and results of the work to others for critique, (i) Collaborating with others in meaningful ways throughout the process, and (j) Recording the results of participant work making it accessible to the broader scientific community [11,19]. Additionally, PLDs should last 50+ hours, which is the minimum suggested amount to enact teacher change [31].

Although the educational research community realizes that content knowledge alone does not directly equate with effective teaching [32], the community does know that teachers with less content knowledge are less confident and effective in some skills such as discussion and questioning [33]. Hwang, Hong, & Hao [34] found teachers value PLD when they acquire pedagogical content knowledge (PCK). They find PCK more valuable than content knowledge (CK) because of how closely related PCK is to the act of teaching. With the need for PLD to focus on PCK the researchers first asked themselves, what do teachers need to know about science content? "To teach students the knowledge and skills required for science proficiency, teachers need knowledge and skills that are congruent with them" [33]. Thus, teachers need content knowledge (CK) and the expertise to use them in engaging ways (PCK). With the ultimate goal being to increase teachers' PCK the focus was on authentic science.

Subsequently, which STEM teachers engage students? It is likely teachers took science coursework implementing traditional teaching methods such as lecture and 'cookbookstyle' laboratory experiences without authentic classroom connections [35,36]. Because teachers typically teach according to how they were originally taught, implementing authentic science inquiry practices could be difficult for a teacher who was taught using traditional methods such as lecture [19]. However, teachers can engage students in meaningful STEM experiences with learning from carefully structured PLDs [13]. To encourage active science teaching, some researchers have created authentic science, astronomy-based teacher PLDs to support STEM teaching. Additionally, teachers use the NGSS, or similar documents, to guide their STEM disciplines' content curriculum. The evaluations from the three years of PLD experiences show that there is increased participant content knowledge, greater pre-collegiate student impact through teacher implementation, and an appreciation for authentic experiences for integrated STEM [25,35].

The gap in the literature that the authors' address revolves around one central concept, making STEM integration a central PLD tenet and demonstrating connections between STEM disciplines in solving real-world problems [8]. Although researchers can identify characteristics that compose a quality STEM teacher, unknowns still include content knowledge gains on specific STEM items, perceptions of different PLD iterations, the comparisons

of those iterations, and the use of authentic science experiences in STEM teacher planning and implementing which can only happen with an increase in teachers' PCK. Although the PLDs in this study were structured around astronomy, there were daily attempts to expand on areas of STEM integration and show connections to other disciplines. As astronomy can be a vehicle to engage students, the authors wanted to know if teachers could transform PLD astronomy-based content, with explicit connections to STEM integration, to a variety of STEM classroom and activity uses.

#### **3. Materials and Methods**

This study utilized both quantitative and qualitative data collection over a three-year period in three separate PLDs interacting with 60 K-12 teachers in the western USA to address three questions. According to Decker and McGill [37] "reporting can be improved across all disciplines to improve the quantity of data needed to replicate studies and to provide complete data sets that provide for the comparison of collected data". As such, the PLD authors of this work have included as much detail as possible in this section.

To fully investigate both the PLD structure and its use for integrated STEM, the authors of this study investigated several research questions. The study's research questions include: (1) How effective were the PLDs at developing teacher knowledge of astronomy content? (2) How do K-12 teachers perceive astronomy PLDs? (3) How do K-12 teachers use astronomy in planning and implementing STEM classroom activities? (4) Does using an astronomy framed PLD promote using integrated STEM content in lesson planning? and (5) How do astronomy PLDs translate into teacher classroom activities in a variety of STEM disciplines to showcase integrated STEM?

#### *3.1. Social Constructionist Theoretical Framework*

With the need for an interactive nature in PLD settings [38], the authors embraced a social constructionism theoretical framework. As outlined by Koro-Ljungberg. Yendol-Hoppey, Smith, and Hayes [39], social constructionism is interpretivist in nature and describes the socialization, roles, dialogue, and transformation of the participants. Through this lens, data collection is through interactive means such as discourse analysis (not used in this study), group assignments (e.g., activities and project feedback), and archival materials (e.g., lesson plans). The validity of the material is through member checking and triangulation [40,41] and the participants negotiate to transform future practices. The social constructionist lens fits the three PLDs described in this research study as the authors encouraged collaboration and interaction, and they interpreted what the teachers experienced through quantitative and qualitative measures in social settings through several methods, which are described in the following section. The authors used mixed methods design and the data collected in one project (or phase) contributed to data collected in projects two and three (other phases) as an iterative process [42].

#### *3.2. Participants*

Participants included 60 STEM teachers in three PLDs conducted over three years. The first and second PLDs were named Launching Astronomy: Standards and STEM Integration (or LASSI), and LASSI was conducted for two years (2014 and 2015). The third PLD was named Robotics, Applied Mathematics, Physics, and Engineering Design (or RAMPED), and the first year of RAMPED was conducted in 2016. Teachers were invited to participate in LASSI and RAMPED by responding to an email invitation sent through the school districts' listserv as well as direct email invitations from the PI, and they self-selected for the PLDs. A breakdown of demographics per experience is shown in Table 1.


**Table 1.** Demographic breakdown: Teaching Level: (El)ementary, (M)iddle, (H)igh, (O)ther; Primary Subject Taught: (S)cience, (T)echnology, (E)ngineering, (M)ath, (O)ther/(A)ll; Gender: (F)emale and (M)ale.

#### *3.3. Instruments*

During the three separate PLDs there were seven types of data collected. Three of the seven types were quantitative instruments and four data points were qualitative components. The three quantitative components included: (a) pre/post-tests on astronomy content; (b) the Misconceptions-Oriented Standards-Based Assessment Resources for Teachers tests (MOSART: https://www.cfa.harvard.edu/smgphp/mosart/); and (c) the Science Teaching Efficacy Belief Instrument (STEBI: http://stelar.edc.org/sites/stelar.edc.org/ files/Science-TE-2fbsc7e.pdf). The four qualitative components included: (d) open-ended questions; (e) anonymous notebook feedback; (f) archival lesson plans; and (g) classroom observations.

#### 3.3.1. Mixed Method Instrument—(a) Pre/Post Astronomy Content Tests & (d) Open-Ended Questions

The pre/post-test on astronomy content tests were developed and examined by the LASSI and RAMPED project teams. Daily content pre/post questions, designed by session and team leaders, were administered at the start and end of the project. The pre/post-tests were created by the PLD teams. The PLD teams were comprised of the faculty leading the specific professional development session. The pre/post-tests were given via the computer by the faculty conducting the PLD (although the faculty could not see the answers or the raw data), and the external evaluator accessed and analyzed the data. The surveys and the open-ended questions were collected directly by the external evaluator. Participants used a code number (instead of a name) when submitting those scores and answers to the external evaluator. In this case, the faculty had no access to the raw data or when/how the participants answered the questions.

#### 3.3.2. Quantitative Instrument—(b) MOSART Test

The MOSART test is an assessment developed by Harvard University, and the Astronomy MOSART was developed with support from the U.S. National Aeronautics and Space Administration (NASA) grant, and it is conceptual in nature. The MOSART project provides tests educators can use to assess their students' understanding of science concepts. The tests can be used to assess one's subject matter knowledge and what impact a PLD has had on enhance one's subject matter knowledge. The MOSART test involved 20 items measuring concepts such as astronomy, chemistry, earth science, and physics. The psychometric properties are detailed on the MOSART website (https: //www.cfa.harvard.edu/smgphp/mosart/testinventory\_2.html).

#### 3.3.3. Quantitative Instrument—(c) STEBI Survey

The STEBI is an assessment developed by Riggs and Enochs in 1989 [43]. The STEBI is used to assess one's science teaching self-efficacy and outcome expectancy. It is comprised of 25 items measured on a 5-point Likert scale. There are two versions, one for in-service teachers and one for pre-service teachers. The version used in this study was geared toward in-service teachers. The LASSI and RAMPED PLD teams used the in-service STEBI teacher edition (http://stelar.edc.org/sites/stelar.edc.org/files/Science-TE-2fbsc7e.pdf).

#### 3.3.4. Qualitative—(e) Notebook Feedback

Additional written data collected from participants included anonymous notebook feedback (in a hardcopy bound notebook). In the notebook, a free response space and a question were posed each day of the PLD so that the participants, at any time during the PLD, could write their thoughts about the day's events (e.g., What do you want us to know about today?). Typically, the team read the notebooks at the end of each week, reflected, and made alterations to the following week's schedule.

#### 3.3.5. Qualitative—(f) Lesson Plans

Teachers were expected to create lesson plans based on the content they were exposed to in the PLD. This lesson plan was required to be submitted as part of their work in the PLD. The lesson plan was also intended to be implemented in the following academic year. The lesson plans were submitted by the teachers to an online site (http://uwyo.edu/wycs/ lessons), where the lesson plans were evaluated. The lesson plans did differ from teacher to teacher as they were created to be implemented in the teacher's classroom. However, each lesson plan contained the same basic components in various orders (i.e., pre-test, objectives, catch/hook, activity, review, post-test, standards connections).

#### 3.3.6. Qualitative—(g) Classroom Observations

The classroom observations were conducted by teaching peers in the same PLD, and the written feedback, pictures, and notes from the observations were collected for analysis through Google forms. Each classroom observation was a minimum of 30 min long and the notes taken were guided by the following questions: What did you observe happening in the classroom?; What did you observe happening with the students?; How did the teacher conduct the activity/lesson?; Describe any lesson successes.; and Describe any lesson challenges.

#### **4. Data Analysis**

For research question number one ("What astronomy content do K-12 teachers learn?") the team used the pre/post content questions as well as the MOSART survey data. For research question number two ("How do K-12 teachers perceive astronomy PLDs?") the team used the STEBI survey, open-ended questions, and anonymous feedback notebook. For research question number three ("How do K-12 teachers use astronomy in planning and implementing STEM classroom activities?") the team used the PLD lesson plans and classroom observations. For research questions four and five ("Does using an astronomy framed PLD promote using integrated STEM content in lesson planning?" and "How do astronomy PLDs translate into teacher classroom activities in a variety of STEM disciplines to showcase integrated STEM?") the PLD team used the qualitative methods of open-ended questions, anonymous feedback notebook, lesson plans, and classroom observations.

The team performed a summary analysis of quantitative data collected from each of the surveys and pre/post-tests during all three PLDs. As a first pass, the authors summarized the prior analysis, which consisted of the use of a *t*-test to determine individual questions with significant changes between pre-to-post assessment [25,29]. Additionally, a matched paired analysis was performed to assess the effect size on the MOSART misconceptions test. Secondly, the authors used a *t*-test to determine if there were significant changes between groups of pre-to- post assessments during each PLD. Assessments were grouped by overarching subject matter, and *p*-values greater than 0.001 were deemed insignificant (and less than 0.001 were deemed highly significant). Additionally, the authors identified the percentage of technical and conceptual questions of each pre-to-post assessment subject matter group. Four team members coded the three years of qualitative data (e.g., lesson plans, open-ended questions, anonymous notebook feedback) creating major themes by

frequency. The authors agreed on the trends that written feedback showcased. Three major themes, and their subthemes, emerged including: (A) STEM discipline content learning (S, T, E, M, and astronomy); (B) Perceptions—positive or negative—regarding the PLD; and (C) Astronomy use in STEM classrooms—in regard to planning, and implementing. The authors validated the data collected through member checking and triangulation of all the data sets. The PLD team talked with the teachers and reported external evaluator weekly summary results to them at the end of weeks one and two. The team asked for feedback on the legitimacy of the information that they were given. Lastly, the team asked if the themes and subthemes were correct. No participant identified any comments that he/she wrote, but a few comments were reclassified after discussion with the groups. For example, the comment that included, 'I need time for practice' was originally put into the STEM astronomy category, but after group discussion, it was moved to the 'negative perceptions' category. Hence, there is inter-rater reliability (>80%) as well as the validity of the data in regard to quantitative and qualitative measures as established through member checking and triangulation. The authors used the qualitative findings to understand teacher appreciation of the PLDs more effectively as well as their needs for creating and implementing astronomy, STEM, and integrated STEM lessons. Overall, the data sets were utilized to triangulate the findings.

#### **5. The Professional Learning and Development Structure**

Each of the three PLD sessions met during the summer over the course of three years (2014–2016). The sessions were held on a university campus and included teachers from around the region. Several of the teachers participated in more than one summer session. However, although a few teachers repeated from year to year, each set of teachers was unique from one year to the next. General format for the sessions included various topics covered over two-day periods. The two-day format would include content coverage, application, and lesson development time. Specifically, each session consisted of 1.5 days of engagement with the material and a domain faculty expert. The remaining half of a day was devoted to lesson planning time. The summer portion of the PLDs were two-week intensive experiences, but the PLDs persisted into the following academic years. After PLD participants (i.e., teachers in this study) completed the summer PLD sessions they were supported throughout the following academic year with lesson plan implementation. This support included close contact with the primary instructor during the PLD sessions and then virtual meeting sessions with the teachers as needed.

The first and second PLD experiences, both named LASSI (Launching Astronomy Standards and STEM Integration), immersed 30 teachers in 25 days of an astronomy research project over two summers (2014 and 2015). The reader can see some pictures and examples of LASSI activities on the following sites: http://uwyo.edu/wycs/lessons. The third PLD experience, named RAMPED (Robotics, Applied Mathematics, Physics, and Engineering Design), exposed another 30 teachers to 16 days of six sessions. Of the six RAMPED sessions, only one—named 'space'—specifically addressed astronomy content. The reader can see some pictures and examples of RAMPED activities on the following site: http://uwyo.edu/wycs/lessons.

As an overview to the PLD designs, see Table 2. In the 2014 LASSI PLD, teachers explored different activities and authentic problem bases. In the 2015 LASSI PLD, teachers created a research question to pursue with faculty support [25]. Observing with the Wyoming Infrared Observatory (WIRO) telescope (http://physics.uwyo.edu/~WIRO/) and viewing at the STAR observatory (https://www.uwyo.edu/physics/observatories/) were participant experiences.

#### **Table 2.** Overarching PLD design for three years (2014–2016).

#### **PLD Design: 8-Hour Days For 2 Weeks**

**2014 LASSI**

**Goal:** Expose K12 teachers to astronomy lessons and show connection with all STEM content.

**Emphasis**: Astronomy exposure with STEM connections

**Daily structure:** Scheduled activities (~6 h/day) collaborative work time (~2 h/day).

**Data collection & analysis:** During on-campus activities with peers and faculty support

**Data presentation:** At the end of the two-week session to their peers, highlighting main "take-aways" and key content knowledge gains. No outsiders attended the presentations.

**Examples:** How can I use a solar system scaling activity to teach metrics, scale, and distance? How far away is Mars, and can we reach it in our lifetime? Would it be possible, with current technology, to colonize Mars?

**Remaining participant need:** Project work with space to fail

**2015 LASSI**

**Goal:** Encourage K12 teachers to create, explore, collect data, analyze data, & present their own astronomy research question or project; focus on data from an observatory or general topic. Use integrated STEM concepts.

**Emphasis:** Authentic science astronomy experience/project

**Biggest change from 2014 LASSI:** Focus shifted to independent authentic research projects

**Daily structure:** Collaborative work time with faculty support (~6 h/day) and one planned activity a day (~2 h/day).

**Data collection & analysis:** During peer collaborations on & off campus, some faculty support, minimal during activities.

**Data presentation:** At the end of the two-week session to peers and others, using electronic poster format, highlighting the main research question, data, analysis, and lessons learned. Several (~5) outsiders attended the presentations.

**Examples:** (Observatory) Analysis of spectra; (General) Does Venus have phases like the moon?; How can Jupiter's moons be distinguished from each other?; Can the size of a celestial body be calculated using angular size and portions?

**Remaining participant need:** Project work with more structure and faculty supported space

#### **2016 RAMPED**

**Goal:** Expose K12 teachers to all aspects of STEM using different STEM disciplines (including astronomy), and assist them in creating classroom lessons showcasing their learning. Showcase integrated STEM activities.

**Emphasis:** Different and engaging STEM exposures

**Biggest change from 2015 LASSI:** Focus shifted to different content & technology exposure; less independent work time.

**Daily structure:** Planned activities (~6 h/day) and collaborative work (~2 h/day).

**Data collection & analysis:** During on-campus planned sessions with faculty support.

**Data presentation:** At the end of the two-weeks to their peers and others, using a printed poster, conference-like session, highlighting main the K12 classroom connections and lessons learned. Many (>15) outsiders attended the presentations.

**Examples:** Using SDSS, how can you create a list of potential changing-look quasars?; How can Raspberry Pis be used to teach K12 weather concepts?; How does NetLogo show and allow modeling manipulations of natural phenomenon? **Remaining participant need:** Less areas of focus (max. 4)

> While in the 2016 RAMPED PLD, each teacher picked four of six sessions to attend during the 10 summer days (note that six additional PLD days were included in the academic year). Only of the sessions was strictly astronomy focused ('space'). Importantly, for this article, the 'space' session is singled out of the six sessions for data analysis. For context, the other five RAMPED PLD sessions focused on computer science/modeling, virtual reality, Arduinos, Raspberry Pis, and robotics.

> Although the PLD teams created these three PLDs at different times with different goals, all three—based on the literature—encompassed high-level content, engaging activities, and varying levels of authentic science. The same primary investigator (PI—first author) constructed all three PLDs in collaboration with higher education faculty and various school districts over the three years to use astronomy and STEM authentic science

approaches for translation to STEM classroom use. Innovative technologies, such as webbased communities and Jupyter notebooks, which are web-based programming notebooks, were emphasized during the 2016 RAMPED PLD.

As the literature highlighted, all three PLDs notably and specifically addressed core disciplinary content (e.g., distances, energy, light, geology, stars, quasars, Sloan Digital Sky Survey—known as SDSS), creation of materials (lesson plans on scale), hands-on activities (e.g., chocolate chip pancakes & the universe; excel pattern recognition), realworld issues (e.g., how do we know it's a quasar?), reflection (e.g., what isn't working in the astronomy project, why, and what can I do about it?), and collaboration (e.g., peers, faculty on and off-campus, students, authors). The authentic science included in the PLDs mirrored the literature by using real-world problems (identified by both participants and faculty), while the K-12 teachers focused on research questions, data collection, analysis, and community dissemination.

#### **6. Results**

Results are shown here aggregated by research question and supporting data throughout all three years of the PLDs. In each of the following three sections, the first table highlights the various data sources on the left and contains three columns—one for each PLD (2014 LASSI, 2015 LASSI, and 2016 RAMPED). The individual cells contain either a (1) summary of that data along with any additional references to more detailed descriptions, or (2) note if that data was collected or omitted from this work.

#### *6.1. Astronomy Content Knowledge Gains*

The authors collected data on teacher content knowledge gains via pre-to-post tests and the MOSART survey (see Section 3.3.2) over the three PLD years (see Table 3). Overall, all three PLDs showed gains in teachers' content knowledge after the two-week PLD sessions. In 2014 LASSI PLD the MOSART test was used to assess misconceptions and astronomy knowledge of the STEM teachers. The teachers had few astronomy misconceptions after the summer PLD. The most prevalent teacher misconception was the primary source of energy for stars. The total possible score was 12, and one teacher received a three (low score) while six teachers received high scores (two with a 10, one with an 11, and three with a 12). Five additional questions from the MOSART were administered involving drawings, and the high score was five. Three teachers received a low score (one with a one, and two with a two), while five teachers received a high score (four with a four, and one with a five). Five of eight STEM teachers had a misconception that pertains to eccentricity of the ellipse representing the Earth's path around the sun.


**Table 3.** Summary of quantitative data collected on teacher content knowledge gains, from pre-to-post and MOSART survey, with a reference to detailed pre-to-post data.

> During the 2015 LASSI two-week PLD, the faculty presented a wide spectrum of astronomy and STEM topics, including: how to construct a telescope, identifying parts of the telescope, ways to use angular size and distance in STEM lessons, explaining Kepler's Laws,

the Doppler Effect and Newton's laws, and how to use Arduinos. Other specific topics teachers explored included black holes, wormholes, circuits, Galilean moons, orbital resonance, applying spectral graphs, constellations, movement of light, wavelengths, relation to elements, information about astronomers (e.g., Tycho Brahe), electromagnetism, prisms, stars, galaxies, gravity, CCDs, wavelengths, Kirchoff's circuit laws, and spectroscopy. The 2015 LASSI pre-to-post questions were skewed towards technical understanding. The test was neither differentiated for elementary and secondary teachers, nor did it take into account the different backgrounds of the teachers. The teachers' content scores increased pre-to-post-tests on all but two topics (see Table 4).

**Table 4.** Summary of significant (*p* < 0.001) teacher content gain in pre-to-post surveys and technical versus conceptual question ratio (grouped by content focus area).


During 2016 RAMPED, five items, selected from each of four MOSART tests (Physics, Earth Science, Chemistry and Astronomy/Space), were pooled into a pre/post outcome assessment. Matched pair analysis indicated an increase in scores from pre-to-post that was statistically significant (*t* = 2.38, *p* = 0.0244) with a high correlation (r = 0.94) between the pre- to- post-test scores, therefore it was a very small effect size (0.15), and the difference is trivial. Additionally, half of the participants scored above 60% on the MOSART assessment.

During the 2016 RAMPED 'space' session, teachers were asked about repositories of astronomy data for future classroom use, and the pre-space session test showed only 15% could identify them, whereas the post-space session test showed that 90% could identify the correct databanks (*p* < 0.00001). The RAMPED session questions were skewed towards conceptual understanding.

#### *6.2. Teacher Perceptions of Astronomy*

The teams collected data on teacher astronomy perceptions from various sources over the three PLD years (see Tables 5–7). Overall, all three PLDs had high teacher perception value, and a ratio of three positive comments for every negative comment. In the 2014 LASSI PLD, each day the STEM teachers rated the usefulness of the day's events that were listed on the daily agenda. Overall, LASSI met the PLD needs and interests of participating teachers at a high level based on the mean quality rating of each session at 4.0 or higher on a scale from 1 to 5 (5 high). Ninety percent of the teachers reported being highly engaged in the sessions at least 75% of the time.

The most highly rated events were those that involved teachers in hands-on activities. These activities included: (A) The sidewalk solar system, (B) Solar system size and scale, (C) The distance lab, (D) The crawl of the crab, (E) Math connections, (F) Writing lesson plans and (G) Lesson planning and collaboration. Collaboration is an important aspect of the social constructionism framework that was employed in these PLDs, and the teachers appreciated interacting and learning with their colleagues. All of the teachers

were interested in leaving the PLD with lessons they could use in their classrooms. Time spent collaborating on lesson planning was always rated 'very useful.' The STEM teachers wanted to develop lessons in astronomy that they could take back to their classrooms and immediately use. Session activities that included food were popular because teachers reported that, in general, their students like these kinds of activities. One teacher wrote about a mathematics session activity, "I can see some possibilities for integrating it . . . preferably with food (fractions). What fraction is a Cheez-It of a graham cracker?" Science and mathematics topics were the focus of the summer (without a specific computer science presence).

**Table 5.** Overview of teacher astronomy perceptions findings based on various sources, with references to detailed data.


**Table 6.** Excerpts of qualitative responses concerning teacher perceptions of astronomy PLDs, as reported in open-ended questions and anonymous notebook feedback.

#### **Teacher Perceptions of Astronomy (Open Ended Questions + Notebook Excerpts)**

#### **2014 LASSI**

"I enjoyed the hands-on part of the labs where we actually did some of the work we expect the students to do. I also liked when we were asked how we would adapt each part of the PD for use in our own class."

"Variety of instructors and ideas presented, some was challenging, some more basic—good mixture."

"This activity has students partake in research . . . similar to what astronomers do for research . . . "

#### **2015 LASSI**

"I have a pretty solid physics background (introductory only) and felt the concepts were interesting and helped elevate my knowledge."

"I appreciated having the opportunity to visit with experts. I learned a lot during my discussions."

"The topics were presented in a way that I could grasp the information even with my limited background experience in astronomy. I found every lesson very interesting and appreciated that I could relate most to my professional growth as a classroom teacher." "The first week the material was really too complex for me, but during the 2nd week things came together and I felt really proud of the learning that had happened in the two-week PD."

#### **2016 RAMPED**

"This was my first experience with the space concepts. I was overwhelmed; my table partner again, was invaluable to my comfort with the material. I need time for practice."

"The tutorials were good and helped to fill in some of the holes that were present for those that were not in the space session earlier. I do feel as though there was an expectation that we would quickly pick up Python and be able to extend on our own."

"I have never participated in a better, more relevant professional development."

**Table 7.** Aggregation of qualitative teacher responses of astronomy content knowledge gains, as reported in open-ended questions and anonymous notebook feedback of qualitative responses concerning teacher perceptions of astronomy PLDs.


"This week I finally realized how computer science fits into STEM, computer science was foundational to the experiments we were doing with the Sloan data."

> Additionally, nearly two-thirds (64%) of the teachers reported that they anticipated that what they learned in the PLD sessions would impact their teaching in their own classrooms in 2014-15. After each PLD session, participating teachers responded to the question, 'How much do you anticipate what you learned during today's PLD session will impact your teaching in your own classroom in 2014-15?' All the responses were aggregated by theme. Based on this pooled data 64% of the teachers anticipated that their own classrooms would be impacted 'quite a bit' (45%) and 'very much' (19%) by what they learned during the PLD session. Although 2% said that they weren't sure or 'not at all,' the other 33% said their classrooms would be 'somewhat' (9%) and 'moderately' (24%) impacted.

> In the 2015 LASSI PLD, the overall quality of the second week of the PLD was rated higher than the first PLD week. On a scale of 0 to 4 (0—poor, 1—fair, 2—good, 3—very good and 4—excellent), the mean overall quality ratings were 2.32 for the first week and 3.67 for the second week. Based on the data, the lower mean rating of the first week's session was likely due, at least in part, to teachers' opinions of the level of complexity of the topics included each week: the topics were rated as 'just right' by 41% of the teachers in week 1 and 76% of the teachers in week 2.

> Premium experiences were applied. The teachers praised the interactions with the astronomers at the WIRO (http://physics.uwyo.edu/~WIRO/), and the hands-on expert explanations. Also, on the top of the list, the teachers valued the opportunities to make things, use new resources (e.g., Makey Makey; https://makeymakey.com/), and develop lesson plans for STEM classes from the content learned during the PLD. Remote viewing, engaging presentations, stellar lectures, expert guest speakers, energetic astronomers, the planetarium, the Jelm Mountain Observatory (http://physics.uwyo.edu/~WIRO/), break-out sessions, hands-on activities, developing lesson plans, and the 3-color images lesson were all highlights of the second week. Computer science was specifically a focus of the PLD, as teachers needed to write code in order to obtain data on stars and quasars. The STEM teachers used code to transform the raw data retrieved from the telescope.

> Thirteen teachers (59%) were enthusiastic about the lessons and topics they were taking back to their own classrooms, including: making pin-hole telescopes to view the total

eclipse, using the Bayer filter in photography, using remote telescopes, using spectrums to reinforce the importance of graphing, sharing information about women in astronomy, general information about the solar system, introducing astronomy vocabulary, using linear regression to find Hubble's constant, using sinusoidal functions in trigonometry when studying wavelength, frequency, and energy, and how to determine size and mass of a planet. One teacher was thinking about starting an after-school astronomy club. Eight teachers (36%) were either not sure how they would use what they learned during the PLD or could only think of a few ways that they could use the information in their own classrooms.

Eleven 2014 LASSI teachers (50%) planned to use some combination of the following from the PLD: remote telescope operations such as Sloan Digital Sky Survey (SDSS), Stellarium, and Micro-Observatory Robotic Telescopes, plus online databases. A few teachers intended to have students build and/or use Galilean telescopes, and one emphatically stated that phases of the moon could be observed without a telescope. One did not know what he/she would have students use in lieu of the WIRO, STAR, and rooftop telescopes, while four replied that they could not use their projects with the grade levels that they teach.

One teacher pulled out the popular culture aspect of one presentation and activity and explained, "I liked the discussion on UFOs. It's helpful to discuss the difference between science and pseudoscience with [students] and what better time to discuss this difference then when asked about UFOs. I will use [the expert's] statement [with my classes]. It is impossible to disprove that something does not exist, and therefore, the question of whether something does not exist is pseudoscience not science."

In 2015 LASSI, the STEM teachers were asked, 'In what ways are you a more knowledgeable educator as a result of this summer's focus on guided inquiry and authentic research experiences?' Although a couple of the teachers commented on their better content knowledge in astronomy, and one noted a better understanding of the importance of independent research and its processes, far more teachers' remarks pointed to other areas including (1) better understanding of and appreciation for the integration of mathematics and sciences areas and the integration of STEM into their classrooms, (2) a better grasp of what constitutes successful guided inquiry, and (3) how to improve and use classroom discussions more effectively. The teachers noted that effective STEM integration and appropriate guided inquiry could impact student learning and career decisions and encourage students' ownership of their learning. One respondent said that a better understanding of the inquiry-based approach derived from having become 'a responsible participant in their own learning' during the project experiences.

In 2016 RAMPED, the percentages of teachers who were either 'somewhat' or 'very satisfied' with the PLD at the end of week one and end of week two were 93% and 97%, respectively. The likelihood that teachers would recommend more PLD like RAMPED increased from a mean of 8.86 (end of week one) to 9.07 (end of week two) on a scale of 0 to 10 (0 not at all likely and 10 extremely likely). The STEM teachers made connections with other teachers who they did not know earlier, enthusiasm for working collaboratively with one another increased, and by the end of the second PLD week the teachers were excited about being able to continue to network with collaborators they met during the PLD in order to share ideas and learn from one another on how to implement project-based activities so that students could be engaged and learning high-level science. A teacher expressed an example of the support teachers can offer each other and the increased comfort with one another that they had at the end of the PLD's second week, "These connections allow me expanded collaboration. I am able to bounce ideas off of teachers not just housed in my district but others as well. They give me the insight I haven't before seen."

Some teachers struggled with the amount of and complexity of content, including the computer science components. Teachers rated the extent (0 to 3, not at all = 0 and large extent = 3) to which the space session was useful to them. The mean rating for space was 1.95. Within the space session, the STEM teachers utilized computer science by obtaining and manipulating SDSS data through development of Python code to automate the process and visualize the results.

Specifically, regarding the 'space' session, 20 teachers chose to attend the summer session, and of those teachers, 14 felt that the space session was useful to a 'moderate' (45%) or 'large' (25%) extent. Five of the teachers felt the 'space' PLD was 'not at all useful' (5%) or to a 'small extent useful' (20%). One teacher stated it would have been helpful to have the first session be about programming and then one or two sessions about how to use that programming with the technology from the sessions. The teacher stated, "Programming was very overwhelming and got in the way of learning the technology." However, another teacher commented that "the Space session showed [me] how coding relates to [astronomy]."

Based on data from the teachers, the strengths of the project were the depth of content, exposure to cutting edge technology that can be used in the classroom, involvement in programming and coding, knowledgeable presenters and team, impressive organization, applicability of the content to the classroom, collaborations, enhanced problem-solving skills, acquisition of technology skills, availability of (patient) individual help, the fun of living in a dorm, resources available for loan to the classroom, intelligent discussions, and exposure to many STEM areas. One teacher summed up the experience and stated, "I have never participated in a better, more relevant PLD. The applications for my classroom will really support me in helping my students to understand and apply 21st-century skills." After the 16 days of the 2016 RAMPED PLD, 97% of the teachers reported that they were 'moderately' or 'extremely' satisfied with the experience. One teacher noted, "Great insight into where we need to support our students and prepare them for these amazing experiences. The instructors were beyond amazing and supportive, the content was interesting, and the use of hands-on materials and the problems [presented] were very beneficial . . . ." The external evaluator exclaimed, 'I have never seen a project with such high participant satisfaction. And I've evaluated hundreds of projects." The authors include these evaluation scores and quotes to situate the conclusions.

The STEBI was used as a second teacher perception assessment. Mean scores decreased from pre-to-post on the STEBI for personal science teaching efficacy belief. None of the pre/post changes on the STEBI were statistically significant.

#### *6.3. Evidence of Astronomy Planning and Implementation*

The authors collected data on teacher planning and implementation via participant PLD developed lesson plans and classroom observations over the three PLD years (see Table 8). Overall, there was a shift in PLD lesson planning from science and mathematics, to science and astronomy, and finally technology and astronomy. Moreover, teachers vested in astronomy (such as the 2014 LASSI participants) were more likely to create astronomy laden lesson plans, while those only interested in astronomy were less likely to include specific astronomy topics in their STEM lesson plans (such as the 2016 RAMPED participants). One may expect all teachers to create lessons focused on astronomy since that was the focus of the first two PLDs. However, this was only found in lesson plans for astronomy teachers. Other STEM teachers were able to create lesson plans not directly related to astronomy based on the PLD content. This illustrates the value in offering PLD in a field not directly related to one's content area and the value of STEM integration.

During 2014 LASSI, the teachers focused on utilizing science and mathematics in their lessons (see Figure 1 and Table 9). One teacher focused on student understanding of scale by 'walking the solar system.' Another teacher emphasized light, specifically having students investigate visible, infrared, and ultraviolet light. All teachers used standards, objectives, pre/post essential questions, a catch, activity, review, and assessments in their lesson planning.

The 2015 LASSI teachers used science and astronomy on equal footing (see Tables 9 and 10). This group of teachers created lesson plans where students could use constellation-viewers, engage in spectroscopy labs, and explore the inverse square law with lasers. Again,

all teachers used standards, objectives, pre/post essential questions, a catch, activity, review, and assessments in their lesson planning.

During 2016 RAMPED there was another shift in lesson plans, but this time the teachers focused heavily on technology (see Figure 1 and Table 10). One would expect to see a focus on technology integration with that being the focus of the PLD. However, some PLD never make an impact on the actual classroom [37]. This PLD did show potential for an impact on the teacher's classroom by evidence of the lesson plans. Again, all teachers used standards, objectives, pre/post essential questions, a catch, activity, review, and assessments in their lesson planning. A teacher stated, "I've learned so much, have used the instruction this year and have plans to use it next year. Next year two [sic] other teachers will also be using the instruction."

**Table 8.** Summary of collected data for the planning and use of astronomy in the classroom, with references to detailed data.


**Table 9.** Aggregation of qualitative teacher responses on planned STEM lessons with bolded integrated STEM aspects, as reported in lesson planning documents.


"I like how the idea of algorithms was brought down to an elementary school level [in the] discussion of sequencing." "I came away with an incredible amount of curriculum, **ideas and connections** that I can use in my classroom!"

"The **applications for my classroom** will really support me in helping my students to understand and apply 21st-century skills."


**Table 10.** Breakdown of seven observed lessons with astronomy connections, where the frequency counts are shown in parentheses.

**Figure 1.** Contents of lessons broken down by STEM core areas as well as any specific Astronomy lesson topics. Note that all astronomy lessons are science lessons as well.

The teachers recorded ways they planned to use what they learned in the sessions in their own classrooms. Four teachers (13%) were not sure about how they would use what they learned in their own classrooms and eight teachers (27%) made general statements of classroom inclusion. For example, one teacher described, "I plan to use SDSS as an introduction hook and as enrichment."

Classroom visits and lesson plans showed the team that teachers can and do use astronomy to teach STEM lessons (see Figure 1). Classroom observers, peer leaders, were invited to 21 of the 30 STEM teachers' classrooms where the STEM teacher picked the lesson and content to showcase. There was no requirement to show classroom astronomy use, but instead the teachers were asked to show the incorporation of something new that they learned from the PLD. Of the 21 classrooms visited, seven of the 2016 RAMPED teachers (33%), used astronomy in the lesson presented. The observers noted six specific instances where the STEM teachers applied astronomy as a vehicle for STEM.

Teachers that used astronomy in classes (see Table 10) included: (a) A high school chemistry teacher that used SDSS, and the Jupyter Notebook technology, to teach chemical gas laws; (b) Two elementary teachers that used mathematics and the order of magnitude, introduced in the space session, to teach about the numbers of red blood cells and other parts of the human body; (c) A high school physics teacher that used space materials as a catch for plasma content and other physics topics; (d) A high school calculus teacher

that used space orbits to teach about the ellipse; (e) A middle school teacher that used material distribution in space to teach about diffusion and movement of gas particles, observing spatial patterns, and analyzing data and graphs; and (f) An elementary teacher that read from a children's space book to introduce a writing lesson's components. Overall, observational findings indicate that astronomy, which is outside of teachers' primary area of expertise, is utilized about a third of the time after exposure to astronomy PLD experiences.

#### **7. Discussion and Conclusions**

Overall, the PLD teachers learned new STEM content, perceived that the astronomy PLDs were beneficial, and used astronomy in classroom lessons approximately a third of the time, although not in the integrated STEM fashion that the authors would have liked to observe. As found in previous research, the STEM teachers in these three PLDs were caught in an ebb and flow of engagement [37] and seemed to cling to some pieces of the PLD and disregard others. The PLD team (the faculty and team members who designed the PLD) allowed the teachers to grapple with concepts, exactly as their students would in the classroom, by using patience, positive reinforcement, and challenging questions instead of answers, as authentic science experiences. Similar to the Socratic method, this is the same approach that STEM teachers are encouraged to use in their own classrooms. This approach showed mixed results, as some of the teachers initially disengaged with the process because of the unease they felt with creating research questions, collecting data, analyzing the data, and disseminating to the community. Especially early in the PLD, some of the teachers wanted short, clearly directed activities without having to create questions or consider alternatives. The PLD team faced these challenges and made adjustments for each summer, achieving various levels of success.

The PLD teams have started to understand the PLD dynamics and how astronomy fits into integrated STEM education. The teams saw that content learning was directly tied to the activities and experiences created for and/or experienced by the teachers. This corroborates the literature on learning [37]. The PLD value teachers perceive might best be shown in the application to their classroom. Teachers find PLDs useful when the content is relevant for their classroom context [44–46]. However, when asked what they learned about astronomy, most of the participants' statements were general in nature, and they cited general background astronomy knowledge, optics, angular size and distance, telescope construction and use, object identification (e.g., star, quasar, galaxy), waves, gravity, Kepler's Laws, SDSS data repository, and spectra. The general nature of the statements speaks to the value of the PLD for STEM teachers, but this potential was left unexplored in many instances. Teachers were able to find value from the PLD, no matter the specific courses they taught, but not through specific content or to the extent that the PLD team wanted to observe. True authentic science experiences were included and contributed to knowledge gains, but the authors question the longevity and depth of the astronomy or integrated STEM learning due to the common content utilized. These areas could be ripe for future research directions.

Teacher perceptions of the PLDs were consistently high. The teachers noted that the PLD ideas and concepts presented were useful and relevant, the topics were complex, the variety of facilitators and experts was helpful, the challenge of learning difficult material was stimulating, and the hands-on experiences that mimicked student work was insightful. The teachers agreed that disseminating the results allowed them to synthesize the PLD concepts. Thus, the STEM teachers wanted and valued the astronomy PLDs, although not one of the teachers specifically taught astronomy classes (which made this field rich in integrated STEM possibilities). Again, this finding speaks to the value of PLD geared towards content and activities that can be integrated into teachers' classrooms. STEM teachers wanted involvement with astronomy activities and projects that provided information that could be used in the classroom. The authors value the high ratings of the PLDs, and anticipate more changes in the PLDs, but also know that there is room for improvement. For example, after this study, the most beneficial STEM PLD organization

seems to include: (1) use of (astronomy) authentic science projects, (2) structured support for content and use of technology, (3) providing extension activities utilizing content and technology for practice, and (4) providing space for participant practice, collaboration, and sharing ideas for skillset development. This structure is conceptual and not linear and allows the PLD providers a frame to build activities, practice time, and other elements onto the complete PLD that participants experience. Thinking of integrated STEM frameworks [14,47] when creating the PLD can also provide a guide for aspects to consider and emphasize. Planning with these concepts in mind could alleviate some challenges. For example, although the use of science projects and activities speaks to an increase in teachers' pedagogical content knowledge (PCK), which was the ultimate goal of the PLD, more subject integration was expected. Acquiring a deeper level of PCK translates to teachers having more ideas to represent complex ideas [45], but more work on how this happens is required.

In general, STEM teachers' use of astronomy in the classroom varied from 'infrequently' to 'often', considering their feedback and observations in the classroom. The authors anticipated that the majority of lesson plans would show new STEM integration, specifically with an astronomy focus. However, the team noticed that the teachers' lesson plans focused on connecting astronomy only to the content that they had previously taught, using astronomy in advanced questioning or enhanced activities/projects, connecting astronomy content to their standards (e.g., NGSS), using the new technologies presented, and/or engaging students in new skills correlating with their existing content. With a third of the teachers showcasing an astronomy connected lesson during observation, the authors are both encouraged about the astronomy topic use and concerned that more teachers did not identify a connection to showcase the astronomy content or use it to teach their STEM subject. Unfortunately, not one teacher implemented an observed lesson with an authentic, real-world science focus. The authors are left wondering if the PLDs emphasized authentic science enough (as defined earlier in this work), or if perhaps the task of using authentic science in pre-collegiate classrooms seemed overwhelming to place into a classroom context? With Kanli [46] stating that the "importance given to astronomy teaching in science and physics education has been gradually increasing," STEM teachers should engage students inside and outside of the classroom in STEM opportunities that include astronomy, but the best path to this outcome is still under review.

What are the lessons learned from the comparisons in these three PLDs? With all of the data presented, the authors argue that careful structuring of PLD leads to stronger content knowledge and use of content (astronomy in this instance) to teach various disciplines (STEM disciplines in this instance). How can PLD facilitators check on the authentic science aspect of an astronomy PLD? One way might be to pay special attention to the dissemination, or presentations, given by the teachers. When the PLD teams listened closely for STEM teacher content gains, perceptions, and future planning in their presentations, opportunities to improve the PLDs emerged. One such instance was when the PLD team used only oral presentations in the 2014 LASSI PLD, then oral, electronic, and group posters in the 2015 LASSI PLD, and finally oral and printed individual posters in the 2016 RAMPED PLD. In the final iteration, the STEM teachers stated that they participated in a meaningful, classroom changing PLD, and the authors believe that this was due in part to the changing nature of the dissemination skillset where the teachers had to 'make sense' of what they learned in order to present to others. Future PLD iterations will include teacher products showcasing all STEM subjects.

Another important discussion waits at the intersection of using an astronomy PLD to assist teachers with integrated STEM classroom applications. Did the astronomy structured PLD work to deliver integrated STEM content? The answer is partly (see Table 10), because the teachers did use all of the STEM disciplines to create their lessons, but they were not utilized in a consistent or evenly distributed manner. As one content area was incorporated more, another content area was not used as much. In all three years, the pattern of using some disciplines was accompanied by using others less. It is possible that the teachers

focused on what was emphasized with the PLD group on a particular day or over a particular week, but refined research questions and data gathering methods would be required to address this speculation. After the teachers planned a lesson, were they able to translate the astronomy content into observable integrated STEM discipline lessons? Here the outlook was less optimistic as only 7/21 (33%) of teachers implemented an observable integrated STEM lesson, and only 6/21 (29%) used astronomy as the vehicle to showcase the STEM discipline. The STEM discipline was also secondary to the astronomy content addressed, which was not the intended delivery mechanism for the teachers. The number of observable integrated STEM lessons using astronomy could have been impacted by school curricula demands, pressure to address standards without a real-world application, or a multitude of other options, but as a group, the majority of teachers could not find a means of using astronomy in authentic, integrated ways to make STEM more engaging in their classrooms. This reality is disappointing to the authors, but it encouraged the PLD teams to realize more explicit connections to STEM disciplines (and showcasing integrated STEM) were needed to enable teachers to translate the STEM spaces into activities for their classrooms. PLD barriers to teaching lessons that lead to improved student success encompass teacher lack of knowledge base, formative feedback, and district factors, and although these challenges were addressed in the PLDs there is still room to understand why content and skillset transfer is slow (such as improvement science education—or ISE—which focuses strongly on context through six principles) [48]. Other studies show that teacher awareness of discipline connections and finding integration valuable [49] as well as bridging a discipline-based to problem-based approach and teachers' professional mindset [50] are also factors that should be considered. The authors of this article are experimenting with using explicit non-examples and examples of both integrated STEM lesson plans and teaching recordings to showcase successful STEM teacher authentic, integrated STEM lessons through purposeful iterations. These example lessons show realworld applications, integrated skillsets, and apply student-centered pedagogies as the STEM disciplines are teased apart for critique. Educational researchers should consider creating studies with a focus on the transfer of integrated STEM to teacher lesson plans, implementation, and student learning.

After reviewing the results of this study, the PLD team offers that researchers must focus on assisting teachers in translating the concepts of astronomy (or any discipline) seamlessly into the content and skills that their students should understand, and PLD leaders must give teachers time to practice, fail, and repeat. The practice should occur outside the classroom and then inside the classroom. Showing the opportunities and providing extended time to try them are basic ideas that could be overlooked in PLDs. Experiences could include skillsets such as coding, collecting telescope data, analyzing data sets, and the like. The content STEM teachers learn, the astronomy opportunities they experience, their perceptions while learning, and then how PLD facilitators support those STEM teachers through 'content to classroom' translations, is an important aspect to consider when creating an astronomy PLD for integrated STEM use. The transfer of content knowledge from facilitators to teachers is vitally important if STEM teachers are to gain the knowledge needed to use astronomy to inspire their students with authentic projects and experiences in STEM subjects. This transfer occurs with hands-on experiences and time to fail and try again. At the end of the three summers, there were clearly teacher content gains in knowledge, positive astronomy PLD perceptions, and integrated STEM classroom uses, but the authors believe that more could be achieved especially using PLDs to kickstart integrated STEM spaces in classroom activities. While the outlook for using astronomy themed PLDs has promise, there are limitations that must be considered when recreating or utilizing this work.

#### **8. Implications, Limitations, and Future Directions**

Teachers, PLD facilitators, and higher education faculty and their partners need to examine astronomy for classroom STEM enhancements, as it can provide a means for teachers to spark student interest and engage in hands-on projects and experiences. Implications of this research include gaps in authentic science, online learning platforms/environments, astronomy PLD structure, and STEM integration. Future researchers should consider focusing on the exploration and solutions in light of NGSS and teacher interest and involvement in astronomy PLDs.

Limitations of this work fall into five main categories. First, participants come mostly from one state and they self-selected to learn about astronomy as a means to teach STEM. Thus, this might be a participant pool that is not representative of the general STEM population. Second, satisfying the content needs of teachers over a wide range of student ages (5 to 18 years) was a challenge in all three years. Therefore, explicitly anticipating the different content and technology needs of elementary, middle, and high school teachers would benefit a PLD team. Third, teacher observations were conducted by different peer project leaders, and no official protocol was used to evaluate the observed sessions, although an outline of what to observe was included in the online form. Thus, comparing and contrasting the observed lessons was difficult and does not allow the team to analyze the lessons as completely as desired. Fourth, a limitation in interpreting the content gains is the short time between the administration of the pre-test and post-test. Lastly, the authors worked with the participants as education and science faculty, and they could have swayed the results due to interactions, lack of interactions, and/or misconceptions and miscommunication.

Future teacher astronomy PLD questions include: (1) Why do teachers utilize lowerlevel astronomy content in lessons? Is it an issue in understanding the astronomy content?; (2) Do the STEM teachers require more exposure to authentic science astronomy PLDs in order to create more robust STEM lessons that use authentic science?; (3) Why are a majority of STEM teachers seemingly reluctant to engage in astronomy laced authentic science projects by creating their own research questions, investigating and analyzing the data, and disseminating their findings?; (4) Since they are the experts of their content and classrooms, why do STEM teachers desire others (e.g., PLD leaders) to map out the astronomy connections with STEM content and standards that are taught in their classrooms?; (5) What are the impacts of using PLDs focused on science versus mathematics with STEM teachers?; (6) What type of authentic astronomy PLD experiences do teachers connect to the classroom most readily?; (7) How do integrated STEM experiences impact teachers' planning of classroom integrated STEM lessons?; and (8) What impacts teachers' use of classroom integrated STEM lessons?

In the end, the authors reflect on one quotation from a participant; "I need time for practice". It is important for both the authors and all educators to remember that 'I need time for practice' is a small statement with a big message. For PLD teams, they must form purposeful, structured time for teacher experiences and feedback while focusing on both explicit STEM content and integration, but also showcasing and supporting how the PLD translates into STEM classroom contexts.

**Author Contributions:** Conceptualization, A.C.B., M.B., A.M.; methodology, A.C.B., A.C.S.; validation, C.M.; formal analysis, A.C.B., A.C.S., C.M.; investigation, A.C.B., M.B., A.M.; resources, A.C.B., M.B., A.M., A.C.S., C.M.; data curation, A.C.B., A.C.S., C.M.; writing—original draft preparation, A.C.B.; writing—review and editing, A.C.B., M.B., A.M., A.C.S., and C.M.; visualization, A.C.B., M.B.; project administration, A.C.B.; funding acquisition, A.C.B., M.B., A.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the USA Wyoming Department of Education, Math and Science Partnership grant numbers WY140202 and WY1601506MSPA2, and USA National Science Foundation #1339853. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Institutional Review Board of the University of Wyoming (protocol #20150921AB00901, 21 September 2015).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Data for this study is held by the first author. There are no publicly available datasets.

**Acknowledgments:** The authors would like to thank all of the teachers and PLD team members for their efforts and dedication during the PLDs. Additionally, they would like to thank the University of Wyoming's College of Education, College of Engineering and Applied Science, and the College of Arts and Science for supporting this interdisciplinary work focusing on authentic science and integrated STEM.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


## *Article* **Dialogic Teaching during Cooperative Inquiry-Based Science: A Case Study of a Year 6 Classroom**

#### **Robyn M. Gillies**

School of Education, The University of Queensland, Brisbane 4072, Australia; r.gillies@uq.edu.au

Received: 28 September 2020; Accepted: 9 November 2020; Published: 12 November 2020 -

**Abstract:** Teachers play a critical role in promoting dialogic interaction in their students. The purpose of this case study was to investigate how one very effective teacher taught two, cooperative, inquiry-based science units to her Year 6 class. In particular, the case study focused on how she used different discourses to capture students' curiosity in the inquiry-based tasks, provided hands-on activities to enable them to test out their hypotheses and develop explanations for what they found in order to help them become more scientifically literate and have a broader understanding of the role of science in the world in which they live. The results showed that the students engaged constructively with their peers on the inquiry group tasks; they used the correct scientific language to discuss phenomena, make claims, and compared findings. Furthermore, they became more adept at expressing their opinions and providing explanations and justifications for the 'scientific' positions they had adopted across the six inquiry-based science lessons; core cognitive practices that support learning. This case study highlights the importance of utilizing both authoritative and dialogic discourse to challenge and scaffold students' thinking to support enhanced understandings and reasoned argumentation during inquiry-based science. This case study fills a gap in the literature on how teachers can utilize different communicative approaches during inquiry-based science units to promote student engagement and learning.

**Keywords:** cooperative learning; collaboration; inquiry-based science; discourse; dialogue

#### **1. Introduction**

Teachers play a pivotal role in inducting students into ways of thinking and reasoning by making explicit how to express ideas, seek assistance, contest different propositions, and reason cogently [1]. It is well known that students learn when they have opportunities to interact with others where they actively listen to what others have to say, reflect on different propositions, present ideas and information, and, in turn, learn to incorporate different concepts and perspectives into their own understandings. However, such interactions do not happen without modelling and guidance from the class teacher who must construct learning situations where students have opportunities to learn the language of science.

Through modelling and guidance, students learn to better understand how different scientific practices of investigation can be undertaken. Such practices involve thinking like scientists, so students learn to be more meta-cognitive and self-reflective in the feedback they provide to their peers. When this occurs, Herrenkohl, Tasker, and White [2] found students were better able to link their hypotheses, scientific investigations, and data analyses to more coherent accounts of their investigations. Furthermore, they demonstrated positive gains on an individually administered follow-up inquiry test and a group project score that was administered pre- and post-intervention, adding credence to the importance of teaching students how to dialogue together. In fact, Chinn et al. [3] argue that there is a large volume of research that demonstrates that when students participate in inquiry-based

science, such experiences lead to positive gains in students' explanatory conceptions and reasoning competencies; core cognitive practices that support student learning.

#### *1.1. Dialogic Teaching and Learning*

Harnessing the power of talk to stimulate and promote students' thinking and learning requires a special type of teaching commonly referred to as dialogic teaching. Dialogic teaching involves five key principles: It is collective and involves the teacher and students discussing learning tasks together in order to clarify concerns and promote understandings; it is reciprocal where the teacher and students present ideas, consider alternative perspectives, and work together to clarify issues; it is supportive enabling students to share their ideas in the context of an environment that respects the ideas of others; it is cumulative in that both the teacher and students build on each other's ideas to develop cogent lines of inquiry; and it is purposeful with the teacher structuring the discussion with a specific educational purpose [4].

When teachers engage in dialogic teaching with their students, Alexander [5] notes, there is more talk about how the discussion will proceed, including the ground rules to be followed. Teachers' questions are often more open and purposeful, enabling students to contribute to the discussion in a more meaningful way. Additionally, students are encouraged to speculate, think aloud, and help each other with both students and teacher building on each other's ideas as they develop "coherent lines of thinking and enquiry" ([6], p. 8). Boyd and Markarian [7] noted that dialogic teaching is evident when teachers use a tone of voice that is conversational, they listen to what others have to say, and encourage students to elaborate on their ideas or questions.

Teaching and learning in the dialogic classroom, Reznitskaya and Gregory [8] argue is characterized by authority over the content and form of discussion shared among group members; questions are open and more divergent and designed to promote meaningful inquiries; teachers provide students with constructive feedback; and students take on key responsibilities for the flow of the discussion. This includes managing turns so all students have opportunities to participate, ask questions, critique others' answers, introduce new topics, and suggest changes to processes of the discussion [8]. Furthermore, students are encouraged to explain their thinking as they work collaboratively with others to co-construct new knowledge and mutual understandings. Gillies [9] reported that when these conditions exist, students are more likely to engage in dialogic exchanges that support enhanced disciplinary knowledge and reasoned argumentation.

Alexander [10] reported on a randomized control trial of a dialogic teaching intervention involving 5000 Year 5 students and 208 teachers across four large cities in the United Kingdom. The intervention was implemented for 20 weeks and involved a cyclic program of planning, goal setting, and review to encourage teachers to extend their own and their students' repertoires of classroom talk to promote dialogue and argumentation. An independent follow-up evaluation of the program indicated that students in the intervention groups were two months ahead of their peers in the control groups on standardized tests of English, mathematics, and science. Similarly, Howe, Hennessy, Mercer, Vrikki, and Wheatley [11], in a large study of 72 demographically diverse classrooms of students, aged 10–11 years, studying mathematics, literacy, and science, found that as long as students engaged extensively in seeking or providing elaborations and challenging others' previous contributions, these categories of language were found to be positively associated with curriculum mastery. Furthermore, elaborations were positively associated with positive attitudes towards learning.

Garcia-Carrion, deAguileta, Padros, and Ramis-Salas [12], in a review of the social impact of dialogic teaching and learning noted that there is accumulated evidence from small and large-scale studies that dialogic teaching contributes to academic achievement and social cohesion, providing children, regardless of their origin, culture, or background with the same opportunities to participate in dialogic spaces that promote their learning and development.

Others who have investigated the role of different types of talk in classrooms are Scott and Mortimer [13] who developed a tool for analyzing the different forms and functions of discursive interactions in high school science classrooms. One form of interaction that is particularly prominent is the interactive and dialogic approach where the teacher actively listens to students' ideas and seeks to explore and elicit their ideas further by asking questions that demonstrate a genuine interest in the issue under discussion. A second form of interaction is the authoritative and interactive approach where the teacher is primarily focused on presenting a scientific point of view and leads the students through a series of questions and answers designed to clarify that point of view. Scott, Mortimer, and Aguiar [14] argued that shifts between these styles of interacting are an inevitable part of teaching science as the authoritative approach is often used to introduce new ideas while the dialogic provides opportunities to explore these ideas further. In a study of two secondary science classrooms, Bosser and Lindahl [15] found that in managing classroom discussions, the teachers needed to alternate between both authoritative and dialogic talk, making specific use of each, as well as interactive and non-interactive communicative approaches. This was particularly important when the teacher needed to ensure that students were provided with opportunities to investigate the complexity of an issue by using an interactive/dialogic-fixed end approach (a guided authoritative approach) as well as being responsive to different perspective presented during a discussion (dialogic/interactive approach). In short, Bosser and Lindahl posited that no specific communicative approach is more appropriate than another and teachers need to consider their choices in relation to the specific goals of their teaching.

In summary, studies by Alexander [4,5,10], Boyd and Markarian [7], Garcia-Carrion [12], Gillies [1], Reznitskaya and Gregory [8], and Wolfe and Alexander [9] highlight the importance of dialogic teaching where students are actively engaged in meaningful discussions with each other as they interrogate a topic, explain their thinking, and work constructively together to construct new understandings and conceptions. Scott and Mortimer [13] and Scott, Mortimer, and Aguiar [14] argued that in such classrooms, teachers need to be very adept at using language that not only builds on students' ideas (interactive and dialogic approach) but also enables the teacher to present and model the canonical ways of reasoning in science (authoritative and interactive approach) relevant to the topic under discussion. Bosser and Lindahl [15] noted that the type of communicative approach that the teacher chose to use was very dependent on the purpose of the lesson.

#### *1.2. Inquiry-Based Science*

One of the main goals of science education is to nurture students' scientific dispositions, develop their capability to engage in scientific inquiry, and teach them how to reason scientifically (National Research Council [16]. The four essential elements in any science education program, Osborne [17] argues are: (a) The conceptual, which builds students understanding of the knowledge and ideas of science; (b) the cognitive, which promotes students' ability to reason critically and scientifically; (c) the nature of science, which focuses on how scientific knowledge is produced and what makes it reliable; and (d) the social and affective, which are designed to promote intellectually engaging experiences. Moreover, Osborne argues that "dialogic enquiry is central to learning as it demands the use of epistemic processes–describing, explaining, predicting, arguing, critiquing, explicating and defining" (p. 180). When students have opportunities to engage in cooperative scientific inquiries, they learn to listen to what others have to say, ask and refine questions, debate ideas, make predictions, plan and undertake investigations, use a variety of tools and artefacts to collect and analyze data, and communicate their ideas and findings to others [18–20].

Kang and Keinonen [21] investigated the effect of four student-centered approaches on students' interest and achievements in science. The authors used a large-scale dataset derived from the Program for International Students Assessment (PISA) 2006. The four approaches to teaching that were assessed were relevant topic-based, open- and guided-inquiry, and discussion-based approaches. The results demonstrated that relevant topic-based approaches and guided inquiry-based learning were strong positive predictors for students' achievement in science and they were also positively associated with students' interest. In contrast, open inquiry-based learning and discussion-based learning were strong negative predictors of students' achievement and interest in science. Kang and Keinonen

concluded that students become more interested in science when they have opportunities to participate in more guided-inquiry learning where connections are made between what they are learning in school science and their real-life experiences and it is this increased interest in science that accounts for better achievement outcomes.

One approach to teaching inquiry that adopts a guided-inquiry approach is the 5Es model of inquiry instruction [22]. In this instructional model, students are provided with opportunities to participate in learning tasks that arouse their curiosity, challenge their conceptions or misconceptions, explore potential solutions, and re-organize and re-construct their current knowledge and understandings to develop explanations, based on the evidence they derive from their own investigations. The advantages of this approach to inquiry learning is that it taps students natural desire to learn how the world works and provides a structure for understanding the nature and development of scientific knowledge and how it is practiced through learning experiences that are meaningful to the learner [18,23]. In this way, students learn how to represent ideas, use scientific tools, and interact appropriately with peers in their cooperative, inquiry science groups in the classroom [24,25].

#### *1.3. Purpose of the Case Study*

Given that many teachers face challenges in teaching students how to engage in discourses about scientific issues, identify and question scientific claims, and draw evidence-based conclusions about problem-based issues, the purpose of this case-study was to investigate how one exemplary Year 6 teacher taught two inquiry-based science units from two units in the Australian Curriculum—Science (ACARA, 2011). A case study design was chosen as it enabled the researcher to provide an in-depth exploration of an individual teacher's approach to teaching inquiry-based science to her students [26]. In particular, the case study focused on how she used different discourses to capture students' curiosity in the inquiry-based tasks and provided opportunities for them to investigate different topics together to develop explanations for what they observed. The study also investigated the language that the students used as they worked on the different inquiry-based tasks across Lessons 1–6.

#### **2. Method**

#### *2.1. Context for the Study*

The case study reported here is part of a larger study involving eight Year 6 teachers from three schools in Brisbane, Australia. All teachers had agreed to teach two units from the Australian Curriculum—Science (ACARA, 2011) for once a term for two consecutive school terms (see [24]). The case study teacher was one of four teachers who were identified as very effective teachers, based on indicators of teacher effectiveness adapted from the research on teacher effectiveness in teaching science, the Australian Professional Standards for Teachers developed by the Australian Institute for Teaching and Learning (AITSL) and previous research by Gillies [27–29] on the effects of cooperative learning on students' task behavior, language, and learning. All teachers were videotaped for approximately one hour for each of the six, inquiry-based science lessons across the two inquiry-based science units. Two groups of students in each teacher's classroom were also videotaped as they worked on their small group activities.

All teachers in the larger study [24] had participated in three days of professional learning workshops that provided them with the background information on the two inquiry-based science units from the Biological Science and Earth and Space Science strands of the Australian Curriculum-Science. As this curriculum has a focus on teaching through inquiry, emphasis was given to ensuring that teachers were introduced to the 5Es model of inquiry [30,31], the importance of students working cooperatively [32], and the importance of dialogic approaches (i.e., Exploratory Talk) for promoting interactions and learning in science [28,33].

#### *2.2. Measures*

#### 2.2.1. Teacher Language

An observational schedule that coded the teacher's use of scientific language was used in this study. The six categories of language that were identified and coded according to frequency included: (a) Making basic statements where the teacher presents a specific point of view or presents facts; (b) asks an open question, which is designed to elicit information; (c) asks a closed question such as an initiation-response-evaluation question or a short answer response; (d) mediates students learning through challenging and scaffolding their thinking; (e) encourages students efforts or ongoing engagement with the task; and (f) uses maintenance language designed to ensure the resources needed to complete the task are available. Teacher language was coded from the observational schedule that was developed for the larger study by Gillies and Baffour [24]. The teacher was videotaped for approximately one hour for the six inquiry-based science lessons across two school terms. All video data were coded by three research assistants who were trained teachers, with two being science teachers, and all had had extensive experience coding video data (inter-observer reliability for the frequency of the six language categories ranged 94–96%). In addition, all video data were transcribed enabling the identification of language that was authoritative and interactive or dialogic and interactive [13]. Authoritative and interactive language is characterized by the teacher leading the students through a series of questions and answers with the aim of reaching a clear understanding of the topic under investigation (discussion occurred between the two research assistants who were science teachers on the authoritative/interactive and the dialogic/interactive language categories until 100% agreement was achieved). In the authoritative and interactive approach, the teacher and students take turns discussing an issue with various voices not taken into account as attention is focused on one perspective [14]. Dialogic and interactive language is characterized by the teacher seeking to elicit and explore the students' ideas about a particular topic by asking questions that probe students' points of view [12]. Dialogic and interactive language involves the teacher and students taking turns to express ideas or comment on an issue with different voices represented and taken into account [14].

#### 2.2.2. Student Language

Five categories of student language were coded during the group activities, based on research by Gillies et al. [24] that identified the types of language students use during inquiry-based science activities and included: (a) Social language (students provide directions, affirm support, suggest solutions, and negotiate actions to take); (b) basic statement (share information, brief statement on everyday knowledge); (c) basic scientific language (correct use of scientific terms and contributes an idea with reason); (d) moderate scientific language (evidence based on conceptual knowledge, strategy to verify information); and (e) advanced scientific language (cites evidence to speculate, reason, challenge others' propositions, negotiate meaning). Video data of the two groups of students were also coded for frequency by the three research assistants (as mentioned above). Additionally, each lesson was fully transcribed, enabling the identification of language that involved the students providing relevant information to the group, utilizing the correct scientific language, and contributing ideas and information that were based on their conceptual knowledge and understandings.

#### **3. Results and Discussion**

Table 1 presents the total percentage of teacher language categories used during the six inquiry-based science lessons. While the teacher used all six categories of language across the six lessons, it was apparent that the language used in each lesson was very dependent on the type of inquiry-based task the students were expected to complete. For example, in Lessons 1 and 2, the teacher asked more open questions and mediated language as she challenged the students to link their understandings of earthquakes to the task they were undertaking. In contrast, in Lesson 3, the students were working more independently in their small groups to construct an earthquake

proof building so the language the teacher used involved more mediation and maintenance where she challenged and scaffolded their thinking and assisted the students with any technical problems or resources they needed to complete this task. In Lesson 4, the students were working together to construct sentences using as many words as they could from their 'word wall' on Infectious Diseases. In this lesson, the language involved the teacher making more basic statements and using language that had a maintenance focus.


**Table 1.** Percentages of teacher language used during the six inquiry-based science lessons.

Lesson 5 began with a video on bacteria, followed by a discussion about how bacteria are important. In this lesson, the teacher focused on providing basic information, mediating students' thinking and learning, and ensuring that students had the assistance and resources needed for the task. Finally, Lesson 6 was a role play by the teacher where she pretended that she had contracted a disease and the students who were "doctors" had to deduce what that disease was on the basis of the research they had done previously on diseases and the clues she provided in the role play. The language that the teacher used involved predominantly making basic statements and providing factual information and asking open questions as the students worked on resolving this dilemma.

Table 2 presents the total percentage of student language categories used during the six inquiry-based science lessons. As with the teacher language, it was apparent that the language used in each lesson was very dependent on the type of inquiry-based task the students were expected to complete and the language the teacher had modelled prior to the small group tasks. For example, in Lesson 1, the students were engaged in determining the magnitude of an earthquake so 90% of the total language involved the students providing assistance by making basic statements (facts relevant to the topic) and using basic and moderate scientific language. Certainly, over 60% of the teacher's language in this lesson had involved asking open questions and mediating the students' thinking and learning so they were 'attuned' to the importance of synchronizing with each other and providing appropriate information and assistance as needed to complete the task.

In Lesson 2, the students were focused on constructing a gum drop and testing its strength and stability so they were predominantly using social language where they offered support and suggestions to each other and used basic scientific language, which involved contributing ideas and requesting explanations. The teacher's language during this lesson mainly involved asking open questions and mediating the students' thinking and learning; language that promoted student engagement with the task.

The students were certainly engaged with constructing an earthquake proof building in Lesson 3, so a considerable percentage of their language interactions were social as this was a hands-on activity with the students making suggestions on how to stabilize and brace the building. The teacher's language in this lesson involved challenging and scaffolding the students thinking (mediating) and making comments designed to ensure the students had the technical assistance and resources needed to complete the task.


**Table 2.** Percentages of students' language used in their science groups during Lessons 1–6.

In Lesson 4, the students were engaged in playing a card game that required them to recognize different diseases, their mode of transmission, and how transmission can be prevented. As this was a hands-on activity, the language the students used was mostly social and involved making basic statements about the task while the teacher's language predominantly involved making basic statements and ensuring that the students had the resources needed to complete this activity (maintenance).

Lesson 5 began with the teacher showing a short video on different types of micro-organisms and discussing how they affect people. In this instance, the language used by the groups involved making basic statements to facilitate completion of the task with the teacher using a combination of making basic statements, asking open question, mediating the students' thinking and learning, and maintenance language.

Finally, Lesson 6 involved a role play with the students tasked with identifying the 'disease' the teacher had caught on her holiday. One hundred percent of the student language in both groups involved the use of scientific language as the students worked to resolve the problem scenario.

#### *3.1. Teacher–Student Language Interactions during Lesson 1–6*

**Lesson 1:** The teacher is recapping on the information from the students' previous lesson on earthquakes and the investigation they are going to do in their groups on identifying the magnitude and location of earthquakes using a Richter or modified Mercalli scale. The vignette below represents a few minutes of the teacher and students discussing the investigation that the students will be undertaking in their small groups.


any of those facts. Last 4 h would be red, last 24 h that orange colour, and we can see one here, and then other. So, those others have to be within those last 7 days though (Mediates)


In the interaction above, it is interesting to note how the teacher is guiding the discussion using a combination of authoritative and interactive discourse [13] where she is focusing on consolidating the students' understanding of earthquakes and some of the terms, such as tectonic plates, magnitude, and epicenter, that they are likely to encounter in their small group discussions. It is interesting to note that of the eight turns initiated by the teacher, two involved open questions and five were designed to encourage thinking. All were followed by the students elaborating on the information (providing an explanation) or generating a closed question or short answer response, indicating that the students were attending to the discussion topic. An examination of Table 1 shows that approximately 66% of the language that the teacher used during this lesson involved asking open questions and mediating the students' thinking, language designed to promote discussion and learning.

The following is an example of the discussion that one group had on earthquakes, seismographs, and S and P waves (forces) that follow an earthquake.

#### **Students interactions in a cooperative group during Lesson 1**


Although this interaction represents only a few minutes of the group's discussion, over 50% of their total interactions involved the students providing more details or elaborating on the information under discussion. In short, they were actively attending to the discussion and this is reflected in the scientific language the students used. An examination of Table 2 indicates that 90% of the language the students used in the lesson involved productive scientific talk where the group members provided relevant information, utilized the correct scientific language, and contributed evidence, based on conceptual knowledge and understandings.

**Lesson 2:** The purpose of this lesson to plan and test out how braces affect the strength of different shapes. The lesson began with the students watching a short video on how to construct a building and the different types of support that were needed to stabilize it.


The teacher, again, is guiding the discussion but, in the vignette above, the language is more dialogic and interactive where the teacher and students together consider different ideas. Of the seven turns initiated by the teacher, four were open and two were closed questions and all were followed by either short answer responses or explanations, indicating that the students were actively engaged in the discussion. In fact, an examination of Table 1 shows that 50% of the total language that the teacher used consisted of open questions and mediation, language designed to challenge and scaffold the students' thinking.

**Students interactions in a cooperative group during Lesson 2** (The students were constructing a gum drop dome and evaluating its strength and stability)


Constructing a gum drop dome (see Figure 1) to learn how the different geometric shapes contributed to its stability and strength was certainly an activity that engaged the students' attention and challenged their thinking. Of the 14 turns in the above vignette, 6 involved explanations or reasons why the dome was able to withstand the different forces. This pattern of interaction was repeated across the two student groups that were videotaped with over 66% of the language used identified as productive scientific talk.

**Figure 1.** Gum drop dome.

**Lesson 3:** The purpose of this lesson is to have the students plan and construct an earthquake proof building using the different bracing techniques that they have been learning are used. The lesson began with the teacher recapping on the importance of designing a building that would withstand the different forces unleashed in an earthquake, the strength of the materials needed, and the types of bracing that they needed to consider. In the vignette below, the teacher is moving around the groups to check on their understandings of the task and scaffold and challenge the students' thinking on how they can manage this task.


The teacher's language in the above vignette is more authoritative and interactive because of the nature of the task that the students are completing in their small groups. In fact, 4 of the 15 turns involve the teacher providing basic information or re-affirming the groups' decisions. An examination of Table 1 indicates that this is consistent with the language the teacher used in Lesson 3 with the teacher engaging in more maintenance language designed to help the students continue working on the activity in conjunction with language designed to scaffold and challenge their thinking.

#### **Students interactions in a cooperative group during Lesson 3**

In the vignette below, the students are discussing how they would brace a building (Figure 2) to ensure it would withstand an earthquake. This activity followed immediately on the class discussion that the teacher had with the students in Lesson 3 about planning and designing a model of an earthquake-proof building. The students are using their previous understandings of different shapes, structures, and the strength of different bracing techniques in the construction of this building. The students will be able to test their construction during a simulated earthquake.


The students certainly enjoyed this hands-on activity of designing, constructing, and testing out the viability of their building. All three members of the group contributed to the activity by either asking questions to clarify their understandings or providing explanations to help each other understand the basic science of bracing. An examination of Table 2 shows that this pattern of interaction was consistent across the inquiry-based science lesson for this group. However, the second group (vignette not included) did not demonstrate this type of language interaction. In fact, 77% of this group's language was socially orientated with members directing the actions of the group or making suggestions about actions to take; ways of interacting that demonstrated that they were actively involved in the task.

**Figure 2.** Bracing on building in Christchurch, NZ after an earthquake.

**Lesson 4:** The teacher is encouraging the students to work together to construct sentences using as many words as they can from their "word wall" on Infectious Diseases (new science unit of work). Terms such as bacteria, organisms, pathogen, disease, infection, and vaccination are some of the words the students will be using in the sentences they construct. In this lesson, the teacher moves around the groups, helping students, clarifying technical terms, and reminding groups to share their knowledge.

T: Everyone's going to have that knowledge, that's right. OK? So I want you to have "knowledge epidemic", and then I want you to, on the next line, write the short version of the date for me. Because we're going to add to our knowledge epidemic page throughout this unit. Now cast your eyes over to our word wall . . . I've put up some of the words that we've been looking at this week. Infectious, pathogen, bacteria, antibodies, vaccination, disease, microbe . . . I want you to see if you can put those words into a sentence that makes sense. You could use all of them; you could make a couple of sentences . . . Just show me that you're kind of having—showing me what knowledge you've gained so far this week. It could be a sentence about a patient. It could be a sentence about a—a factual sentence. It could be a made-up—we've been reading stories about people who have infectious diseases. So you could ha- give a person a name. And have a character in your sentence. It's up to you how you write it. OK, it might be t- um, telling somebody about your illness, or something like that. How you choose to write it is up to you. You may not get all the words into your sentence. But give it a go. See how many words you can make in your sentence that would make sense . . . Go. (Encourages thinking). The teacher then moves around the groups to assist students with their inquiries.

The language the teacher uses in this lesson is more authoritative and non-interactive as over 60% of her language involved making basic statements and language that included assisting students to resolve problems or continue to work on their current activity (maintenance language) (See Table 1). With this type of interaction, the teacher is presenting a specific point of view; in this case, she is recapping on the information collected on the word wall and directing the students to use that information to construct their sentences.

**Students interactions in a cooperative group during Lesson 4** (the students are playing a card game on the different types of infectious diseases)

In the vignette below, the students are constructing sentences about different infectious diseases that they have been studying. They have been learning that infectious diseases are disorders caused by organisms such as bacteria, viruses, fungi, or parasites. While many of these organisms live in and on the human body, they are usually harmless or even helpful. However, under certain conditions, some organisms may cause disease. Some infectious diseases can be transmitted from person to person while others are passed on by insects or other animals. Consuming contaminated food or water is another way in which infectious diseases can be transmitted. The students have been learning that many of these infectious diseases have signs and symptoms, depending on the organism responsible for the infection. In the activity below, the students are working together to construct sentences

that provide information about the different terms (i.e., infectious, pathogen, bacteria, antibodies, vaccination, disease, microbe) that they have been learning about in their science unit on Diseases.


There is no doubt that the students in the vignette above are focused on the task as they pose questions and provide directions to help complete the task. An examination of Table 2 indicates that this pattern of interaction is consistent with the language the students in both groups used in Lesson 4 where over 50% of the language involved students responding with basic information to others' questions.

**Lesson 5:** The students have just finished watching a short video clip on different types of good and bad bacteria. Terms such as bacteria and viruses are microorganisms, symptoms of the Ebola virus, immunization, transmission, toxin, and yeast infection are discussed as the teacher helps the students to recap on information they have learned in previous lessons.


16. T: Yoghurt! OK, so yoghurt can be . . . where will it fit? (Closed question)

It is interesting to note that over 30% of the language used in this vignette involves the teacher challenging and scaffolding the students' learning. This pattern is consistent with the language the teacher used across the full lesson when she provided basic information, mediated students' learning, and encouraged ongoing involvement in the task (see Table 1). In this sense, the language used was dialogic and interactive as she encourages the students to investigate different ideas while simultaneously challenging and scaffolding their learning.

#### **Students interactions in a cooperative group during Lesson 5**

In the vignette below, the students are writing sentences about the different types of good and bad bacteria and where they are found (Figure 3).


**Figure 3.** Single-cell bacterium.

Here is no doubt that the students in the above vignette are synchronized with each other as they ask and respond to questions. In fact, two of the responses provided reasons or explanations as to why some bacteria are important. An examination of Table 2 indicates that during this lesson, the students predominately responded with brief statements, based on what they had been learning.

**Lesson 6:** The teacher has just finished a role play where she told the class she went to Nepal and while she was there, she contracted a disease. The students have been studying different diseases and the effects they have on people so they are about to begin working in their small groups to deduce the name of the disease, based on the clues the teacher has provided.


The language the teacher used in the above vignette was certainly shaped by the role play she had presented as the students' task was to identify the disease she had contracted while holidaying in Nepal. She keeps reminding the students that they had to be able to indicate 'why' they have decided it is a certain disease and be prepared to 'justify' their choice. The students are certainly 'tuned-in' to the teacher as every question or statement involves a student providing a reason, explanation, or short answer response. An examination of Table 1 indicates that over 60% of the teacher's language during this lesson involved her in making basic statements (about the task) and asking open questions. The language used was authoritative and interactive as she encouraged the students to think carefully about the disease that they chose, provide reasons for their choice, and be prepared to discuss why they eliminated other diseases.

#### **Students' interactions in a cooperative group during Lesson 6**

In the vignette below, the students are discussing the possibility that the disease the teacher contracted was malaria.


While the vignette above represents only a few minutes of the students' discussion, it is interesting to note that they very readily identify that malaria is transferred by being bitten by a mosquito and that it is controlled by a DDT program (an insecticide to control malaria). All three students are involved in the discussion, either through asking questions, responding to them, or providing information or help. An examination of Table 2 indicates that 100% of the language involved the students in both groups using basic, moderate, and advance scientific language to gain deeper insights of the issue they were investigating. This was the only lesson where the students' utilized all forms of scientific language, clearly indicating that they had developed a clear understanding of the importance of using the correct nomenclature and the role it played in helping them to engage in scientific discussions.

#### *3.2. Summary of Teacher–Student Language during Lessons 1–6*

In the six lessons outlined above, it is interesting to note how the teacher's language was very dependent on students' previous understandings of the topic and the type of inquiry-based task they were expected to complete. For example, in Lessons 1, 2, and 3 the teacher used more authoritative and interactive discourse where she recapped on the students' technical knowledge of earthquakes while simultaneously posing a series of open questions that challenged and scaffolded their thinking. The students, in turn, responded with more details about the topic, asked closed questions to elicit information, or provided short answer responses; all indicative of students who are motivated and engaged with the task.

It was interesting to note that while the teacher's interactions with the students were essentially authoritative as she began each lesson by introducing the problem and then guiding students towards developing an understanding of earthquakes and the issues of how to brace buildings to manage the various forces they released, there were occasions when her language became dialogical as she encouraged the students to express their opinions and provide reasons and justifications for the positions they had adopted [13]. Similarly, in Lessons 4, 5, and 6 where the teacher again used more authoritative language designed to ensure students understood the specific terms associated with their unit of work on Infectious Diseases. Again, there were occasions when the teacher's language was dialogical as she challenged and scaffolded the students' thinking to engage in more meaningful inquiries about the topics they were discussing. This type of discourse was particularly evident in Lesson 6 where the students had to critique the 'scientific' cues that emerged during the role play and try to reconcile them with their own investigations on infectious diseases to identify the particular 'disease' the teacher had acquired on her holiday in Nepal.

Scott et al. [14] and Aguiar, Mortimer, and Scott [20] maintain that both authoritative and dialogic discourses are necessary with the former enabling teachers to develop the canonical scientific view or ways of reasoning in science while the latter encourages investigation of the ideas presented. When teachers interact with students to share their knowledge and understandings about science and scaffold their thinking, they also interact socially with students, which it is argued helps to keep them engaged in the discussions and motivated to learn science concepts. In short, it is through talking and thinking together that students learn to make connections between everyday ideas and scientific ideas to construct new understandings [34].

If students are to engage successfully in science then they need to have opportunities to participate in dialogic exchanges with each other where they learn to express their views about the problem at hand, probe and explore the views of others, and reconcile these views with the scientific perspective [13,35]. Inquiry-based science instruction, Harris and Rooks [25] propose, places higher demands on students in terms of participation, personal responsibility, and cognitive effort than would normally occur in classrooms where teachers use traditional transmission modes of teaching. In inquiry-based learning classrooms, students cooperate to plan and conduct investigations and engage in discussions and debate with each other and the teacher [30–34]. In so doing, students have opportunities to ask and answer questions to help make their thoughts, reasons, and knowledge explicit, learn new ways of using language that they can appropriate for themselves, and make sustained contributions to class discussions; ways of interacting that promote understanding and learning in science [33,34].

#### *3.3. Limitations*

There are two limitations to this study. First, this is a case study of one Year 6 teacher who taught two units of inquiry-based science across two school terms. While the data presented enabled the author to undertake an in-depth exploration of this teacher's approach to teaching inquiry-based science to her students, the generalizability of the insights gained are limited. Second, because of the constraints that operated around the teacher's class time, there were no pre-intervention measures collected on the students' discourse or reasoning and problem-solving skills during their small group discussions, limiting the interpretations that can be placed on the insights gained. These are issues that need to be addressed in future studies.

#### **4. Conclusions**

The purpose of this case-study was to investigate how one very effective teacher taught two, cooperative inquiry-based science units to her Year 6 class. In particular, the case-study focused on how she used different discourses to capture students' curiosity in the inquiry-based tasks, provided hands-on activities to enable them to test-out their hypotheses and develop explanations for what they found in order to help them become more scientifically literate and have a broader understanding of the role of science in the world in which they live. The results showed that the students engaged constructively with their peers on the inquiry group tasks; they used the correct scientific language to discuss different phenomena, make claims, and compare findings. Furthermore, they became more adept at expressing their opinions and providing explanations and justifications for the 'scientific' positions they had adopted across the six inquiry-based science lessons; core cognitive practices that support learning. This case study highlights the importance of utilizing both authoritative and dialogic discourse to challenge and scaffold students' thinking to support enhanced understandings and reasoned argumentation during inquiry-based science. This case study fills a gap in the literature on how teachers can utilize different communicative approaches during inquiry-based science units to promote student engagement and learning.

**Funding:** This research was funded an Australian Research Council Grant: ARC-SRI: Science of Learning Research Centre (project number SR120300015).

**Conflicts of Interest:** The author declares no conflict of interest.

#### **References**


**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Cooperative Learning in Swedish Classrooms: Engagement and Relationships as a Focus for Culturally Diverse Students**

#### **Kate Ferguson-Patrick**

School of Education, University of Newcastle, Newcastle, NSW 2289, Australia; kate.fergusonpatrick@newcastle.edu.au

Received: 21 September 2020; Accepted: 28 October 2020; Published: 31 October 2020

**Abstract:** The ongoing refugee crisis makes intercultural competence and culturally responsive education crucial issues in schools. At the same time, increased migration poses new challenges for social cohesion in countries around the world. How schools and classrooms can be fair and inclusive in terms of experiences and outcomes for migrant and refugee students is therefore a key question. This paper will explore the increase in migration of newly arrived students in Sweden, and how teachers in this country are catering for diverse students through cooperative learning. I explore cooperative learning as an inclusive and culturally responsive pedagogy that can be effectively used in schools to support all students and especially 'refugee' or newly arrived students. Using theory from cooperative learning and Stembridge (2020) as a theoretical framework, I particularly focus on analysis using two of Stembridge's themes of Culturally Responsive Education: *Engagement* and *Relationships* to analyze Swedish primary school classroom observations and teacher interviews and find commonalities between these two themes and the key ideas in cooperative learning. This research is built on the premise that there is more need in education research for up to date observations into the classroom factors that support or hinder learning and the way that within-class groupings can support diversity and inclusivity. Cooperative learning allows participants to develop a commitment to fairness, social responsibility and a concern for others and this particularly caters for our diverse student populations.

**Keywords:** culturally responsive education; cooperative learning; relationships; engagement; inclusivity; diversity; newly arrived; refugee

#### **1. Introduction**

The UN Refugee Agency's figures show that there were 79.5 million displaced people worldwide at the end of 2019. Refugees who have fled their countries to escape conflict and persecution accounted for 26 million, with more than half being under the age of 18, reaching a record high, with the highest increase being between 2012 and 2015 due to the Syrian conflict (UNHCR, 2019). The provision of educational opportunities is seen as one of the highest priorities for refugee communities [1] yet research into refugee education has been described as a wasteland [2] little focus on how to best educate such a diverse and marginalized group. The 2030 Agenda for Sustainable Development adopted by the United Nations in 2015 includes 17 sustainable development goals (SDGs), with a comprehensive global goal on education (SDG4). SDG4 on education is to "ensure inclusive and equitable quality education and promote lifelong learning opportunities for all" [3]. This is a key focus of this paper which argues cooperative learning is an inclusive pedagogy and provides students with lifelong skills including social and emotional skills. When students are engaged in cooperative work with shared engagement, this can contribute to effective social inclusion due to intergroup contact [4].

It is important that we not only ensure that Interculturalism allows refugee, migrants, and newly arrived peoples to live side by side with each other harmoniously in society, but that we also provide such an environment in classrooms with our students able to live together in harmony with improved dialogue and fruitful relationships [5,6]. Now more than ever we need to ensure our multicultural societies construct the kinds of 'spaces that need to reflect and incorporate the diversity of their citizens, including those who cross their borders as refugees and immigrants' [7]. He has long argued for the need for equity pedagogy, which he says exists when teachers use techniques and teaching methods that facilitate the academic achievement of students from diverse racial and ethnic groups and from all social classes. Using teaching techniques that cater to the learning and cultural styles of diverse groups and using the techniques of cooperative learning are some of the methods that teachers have found effective with students from diverse racial, ethnic, and language groups [8].

Schools, in particular, have the responsibility to help refugees as they can be seen as 'the major avenue by which students with refugee experiences can be acculturated [and can] provide environments which are inclusive, tolerant and accepting of difference' [9]. It is equally important that we recognize that all students have unique experiences including histories, life experience and different beliefs which enrich our classrooms. Culturally responsive pedagogies which 'respond to students' identities by respecting, valuing and drawing on these attributes and rich histories and lived experiences' [10] can help to bridge the gap between these home experiences with school [11].

#### *Some Definitions of the Terms*

The country of study in this research project is Sweden. There are different terms used throughout the world for students who are sometimes known as newly arrived or refugee students and also as asylum seekers. I chose to use the term newly arrived students as the study is situated in Sweden and this is the term used in schools. Nilsson and Bunar [12] provide definitions of the terms according to Swedish schooling systems:

*In Sweden, the educational authorities (Skolinspektionen, 2009, 2014; Skolverket, 2008; Utbildningsdepartementet, Ds:2013:6) define a newly arrived student as a student who has migrated for any reason (for example, as a refugee, for family reunion, or labour migration), who does not possess basic knowledge of the Swedish language, and who starts school just prior to or during the regular academic year.* (see Bunar, 2010)

Nilsson and Bunar [12] further state the need for teachers to understand that not all newly arrived or refugee students are the same,

*Migrant children are heterogeneous in regard to pre-migration factors, such as social and educational background, family relations, and upbringing in a rural or urban area, as well as trans-migration factors, including the reason for migration (voluntarily or forced), experiences of war, persecution and trauma, background from a country with a long-lasting war or warlike situation (for example, Afghanistan and Somalia), or a country where sudden crises caused a large-scale flight.* (such as Iraq after 2003 or Syria after 2011)

The influx of refugees during 2015 increased the number of foreign-born individuals in Sweden by 163,000 [13]. This has been reported as the greatest proportion of refugee migration in Sweden to date [14]. McIntyre et al. [15] provide a good policy context of the Swedish system for newly arrived students demonstrating how Swedish schooling systems and policy favors inclusion:

*A cross-party refugee agreement in 2015 outlined policies for ensuring that all schools be prepared to receive newly arrived students (Proposition 2015*/*16:184, 12), that municipalities should place students in a range of schools to avoid segregation and that support for students in their mother tongue should be provided.* [15]

Every fourth (24.9%) resident in Sweden has an immigrant background and every third (32.3%) has at least one parent born abroad. Migration can have a longer lasting effect on people than just the initial effects of arriving in a new country with different cultural and language expectations. Research has shown that, 'childhood immigration and the timing of this event after an early period of childhood socialization and optimal learning may trigger adverse life-course trajectories that leave a lasting impact on adult socioeconomic attainments' [16]. It has also been shown that region of origin also has an effect, with immigrants arriving from low income region origins resulting in longer lasting impacts. This demonstrates that teachers should be aware of this sensitive period in newly arrived students' lives and support these students in an inclusive cooperative learning environment at school.

As newly arrived students are challenged due to speaking different languages at home and school, as well as requiring more support in schools to help with their learning [17] they also need to help overcome the longer lasting effects migration can have by being further supported in schools with teachers that that understand and use Translanguaging (developed from the Welsh term Trawsieithu). This is the 'strategy of planned and systematic use of two languages for teaching and learning inside the same lesson in bilingual education' [17]. This allows students to see the respect and appreciation the school has of their multilingual capabilities.

An equity pedagogy, such as cooperative learning, is needed: this creates citizens committed to social justice with an understanding of intercultural competence and on the need to build attitudes and dispositions which link, rather than compete with, others. At the same time we need to understand also as educators the complexities of diversities and lenses of identity that illustrate the need to understand diversity in a nuanced way, and that this notion of diversity is a strength in our classroom [10]. The results of a recent survey, Diversity Barometer (2016), have shown that in Sweden, 'In general, attitudes toward cultural diversity and migration became more negative compared to the results of similar studies carried out during 2005–2014 . . . those who have had less contact and experience of interacting with foreigners have a more negative attitude toward migration' [13]. There is a general pattern found from this survey whereby young people have more positive attitudes towards diversity. If cultural and ethnic diversity are perceived negatively in schools by students and their parents this can 'lead to both an increase in and normalization/legitimization of discrimination and social exclusion of certain groups, which in turn may aggravate tensions, conflicts and even violence in different segments of society' [13]. This is an important point. I argue all students are heterogeneous in regard to their social, educational and family backgrounds and we need to particularly ensure we do not create educational spaces which exclusively privilege the majority culture.

Risannen [18] has noted the change in Sweden to a Multiculturalist orientation in educational discourses from one of assimilationism with cultural needs of different groups now more likely to be addressed. An increased use of cooperative learning that teachers in Sweden are embracing may well be changing young people's attitudes to diversity. A cooperative learning Facebook page for teachers in Sweden https://www.facebook.com/groups/kooperativt now has 25,000 members. The group's purpose is to provide support for teachers about how collaboration between students can be used to strengthen learning. The number of PhD theses written on cooperative learning in the past few years has also increased dramatically, with 66 produced in the past three years (http://www.diva-portal.org/) so it is both a popular and well researched pedagogy in Sweden, as well as being widely used in Swedish classrooms. Having positive attitudes to diversity is important if we want our students to develop intercultural competence and thrive in a diverse world. Intercultural competence is seen as "a complex of abilities needed to perform effectively and appropriately when interacting with others who are linguistically and culturally different from oneself" [19] I argue that cooperative learning helps to develop these abilities.

#### **2. Twenty-First Century Pedagogies to Support a Globalized World**

An equity pedagogy has long been argued for but more recent arguments for twenty-first century pedagogy imply a new way of learning and teaching, a way that assists and enhances a different

set of beliefs and theories more appropriate for a new century—a globalized century. There is some consensus that students of current educational institutions 'will need to function well in the globalized world in terms of competition for opportunities and collaboration with individuals from different cultures' [20]. As a result of globalization, twenty-first century students need to be interconnected and interdependent to address global issues such as human rights abuses [21,22] and understand the key challenges facing our globalized world.

Education for the 21st century needs to develop students' skills that allow them to move between workplaces and develop the metacognitive skills necessary to do this well. These skills are not simply memorization and repetition but include social skills and attitude sets. Twenty-first century skills have been described often as being the 4Cs, comprised of Communication, Collaboration, Critical thinking and Problem-solving, and Creativity and Innovation. These skills help our students engage with the global economy [23,24] and global problem-solving [25]. Some authors have argued that these four skills can actually be developed into ten skills, especially when students are involved in teaching and learning activities driven by problem solving and in a world with ICT [26]. Two of these skills particularly relevant to cooperative learning are cross/intercultural competence and co-responsibility. 'Cross/inter-cultural competence addresses learners' capacity to communicate, collaborate and work in multicultural and global environments. Co-responsibility refers to a culture of sharing that necessitates shifting to less ego-centric principles and practices' [26], pp. 5–6. These skills are now even more needed with the added social challenge our word is facing, 'As the global population continues to grow migration, urbanisation and increasing social and cultural diversity are reshaping countries and communities' [27]. This report on the future of education by the OECD states that education needs to be about equipping, 'students with the skills they need to become active, responsible and engaged citizens'. They list social and emotional skills (e.g., empathy, self-efficacy and collaboration) as some of these important skills. Skills such as collaboration, as well as flexibility, adaptability and initiative, global awareness and citizenship are important skills according to the OECD [28]. We need our students to develop these essential skills in diverse classrooms, to be globally competent, and it is important that we consider the kinds of classrooms in which this can occur.

Culturally responsive pedagogies need to be taught by teachers who understand how to develop students' 21st century skills which also include the understanding of the sixth pillars for 21st century learning. These include Delors [29] four pillars learning to know, learning to be, learning to live together and learning to do, and UNESCO's fifth pillar learning to transform oneself and society [30] and a more recently added one learning to give and share [26]. This paper focuses on one such pedagogy for equity that incorporates a number of these six pillars, cooperative learning. Three particularly relevant ones in are 'learning to be', 'learning to live together' and 'learning to transform oneself and society'. Learning to be involves developing 'one's personality and be able to act with greater autonomy, judgment, and personal responsibility. This relates to personal skills and dignity' [31] and learning to live together involves developing 'an understanding of other people and their history, traditions, and spirituality, in order to participate and cooperate with others in all human activities. This has to do with social skills and social capital' [31]. Learning to transform oneself and society has links to cooperative learning as it is about developing 'respect for the environment, for social solidarity, and for a nondiscriminatory, gender-sensitive world' [31].

#### *2.1. Culturally Responsive Education for Equity: Developing 21st Century Skills Like Collaboration through Cooperative Learning*

This paper explores how cooperative learning is used in schools in Sweden, a country which has had a significant number of refugee students arriving in the past few years. The use of cooperative learning is a way of creating a classroom ambience, or democracy stance [32,33] in that classroom in which there is genuinely opportunity for all. Hek [34] argues from the UK experience that, "It is not an exaggeration to say that refugee children's well-being depends to a major degree on their school experiences, successes and failures . . . . [and] Educational progress and emotional well-being are mutually dependent" (p. 29).

Cooperative learning is about learning together in small groups (learning to live together) and is about the way teachers develop the relationships in that classroom and create a particular environment, or 'that certain climate' [35] using methods and strategies to maximize participation and engagement from all students. The UK SPRinG program also found that group work can particularly help relationships and social inclusion [36]. This is further supported by research that confirms small group learning 'is very useful help for the marginalized children to be incorporated into their team' [37].

Cooperative learning is important in diverse classrooms. Cooperative learning is a particular type of group work and the benefits of using this have been researched over decades. Diversity can bring social problems and conflict if teachers do not consider this when new students arrive in their classrooms. Positive social and emotional development lays the foundations for wellbeing and good mental health and so this is also an important focus for schools [38,39] especially with newly arrived or refugee students who may have experienced trauma and change. Cooperative learning experiences can be used to increase students' cooperative predispositions leading to pro-social behaviors and reducing bullying and harm intended aggression in students [40]. Interactions among students are crucial to cooperative learning and the interactions that occur in the groups help to facilitate learning [41] with positive relationships occurring as students not only help each other but also enhance thinking.

One of the major spin-offs from such improved relationships in the classroom is ensuring that all students are included in learning. When teachers actively include all pupils through identifying individual strengths and giving them roles in groups, then it recognizes diversity as a learning resource and changes the conditions that influence students' participation in the classroom [42,43]. This equity is fundamental to an intercultural classroom. A cooperative classroom can also enhance such a classroom of tolerance. As students learn to appreciate the skills of others and as they are positively connected in their learning tasks, they become more tolerant of students they may have otherwise not appreciated. Students are able to develop 'learning to be' as they develop personal skills when working in cooperative groups. Empathy and the ability to trust others in their group is developed and enhanced in such classrooms.

An important aspect of developing a collaborative learning space is also the work of teachers who develop caring spaces for their students. Care can be developed through supportive and cooperative pedagogies used by teachers. As argued by Sellars [44],

*Caring is not just to be read about, discussed or subtly mandated as part of a hidden curriculum (Giroux and Penna, 1979), it is practised as collaborative, not competitive learning. It is a daily, ongoing commitment to explicit, strategic pedagogical approaches for engaging students with each other in positive and mutually supportive interactions.*

Caring goes hand in hand with teachers who want to develop good relationships with their students. As Noddings [45] argues, "If we cared deeply about fostering growth and shaping both acceptable and caring people, we could surely find ways to extend contact between teachers and students. There is no good reason why teachers should not stay with one group of students for three years rather than one in the elementary years." (p. 224). Noddings goes on to argue "an ethic of caring, involves modeling, dialogue, practice, and confirmation" (p. 222). Teacher modelling of caring occurs when they encourage their students to recognize and value themselves; when they have open dialogue; when they encourage students to practice care towards others; and demonstrate high expectations of their students by affirming what they know about them and by allowing them to grow accordingly. This comes from teachers having a genuine and thorough knowledge of each of their students, developed over time. Relationships take time to build and this notion of keeping students with their teachers for three years is apparent in many of the Swedish classrooms I visited.

#### *2.2. Culturally Responsive Education for Equity: Engagement and Relationships through Cooperative Learning*

Culturally responsive teaching can be seen as 'using the cultural knowledge, prior experiences, frames of reference, and performance styles of ethnically diverse students to make learning encounters more relevant to and effective for them' [46]. Culturally responsive education (CRE) has recently been defined as 'a framework that is useful for closing Equity gaps in school achievement outcomes' [47]. Stembridge goes on to argue that as teachers we have the choice of what to teach, the pedagogies we use and that the 'strategies we choose in teaching should reflect the needs of our learners (Equity) . . . and Equity isn't possible without the element of connection' (p. 27, 28). It is important then that the school spaces, the classrooms, that these students are in, have teachers who are culturally responsive [48] as well as necessary to explore the kinds of practices used by teachers who demonstrate connection. 'Our students and the communities we serve must feel connected to us, and we are better teachers when we are connected to them' [47].

#### *2.3. Stembridge's Themes of Culturally Responsive Education*

Examining the six themes of culturally responsive education [47] allows us to see these 'as essential for the bridging of Equity and pedagogy' (p. 67). This article will examine two of the six themes being *Engagement* and *Relationships* (you can read more about the other four themes in Stembridge [49] as these two themes were the ones that were seen as predominant in the data during analysis as well as mostly being closely matched to other aspects of the literature about cooperative learning. Stembridge argues that although they can be seen as separate components they should also be seen as 'an interrelated ecosystems of intellections' (p. 67). I will therefore briefly describe the other four themes.


frame and deliver their opportunities to learn-what we offer in the content of the experience-must be substantive and meaningful" (p. 103). He further argues that cognitive engagement and rigor help to reinforce each other.

#### *2.4. Engagement and Relationships*

Now I will explore more fully the two themes that will be used for analysis in this article: *Engagement and Relationships* alongside the literature that links these themes to other research and in particular research on cooperative learning. Engagement is defined as being the most valuable commodity in the interest of learning, as being engaged is needed by our students for us to have a chance to teach them. Stembridge goes on to explain the importance of context with engagement as 'human beings are highly social species and interpersonal and cultural contexts matter' (p. 71). There are three types of engagement according to Stembridge [47]: (behavioral, affective and cognitive) and they are not mutually exclusive (See Table 1).

**Table 1.** Types of engagement as a theme and goal for culturally responsive education (CRE).

Behavioral Engagement can be thought of as the physical investments that students make in their learning. In the simplest terms, behavioral engagement entails the students' willingness to follow the rules and accept the behavioural guidelines in instructional spaces.

In classrooms, affective engagement looks like interest (versus boredom), active (versus passive) learning, and the students feeling of belonging to the school community (versus a sense of isolation).

Cognitive engagement can be thought of as the intellectual investments that students make in their learning. It speaks to the extent and intensity of self-regulation in attending to one's own learning, and also the willingness and ability to be strategic in the building of understandings and completion of tasks.

When students are affectively engaged Stembridge argues 'they are more likely to be cognitively and behaviourally engaged as well' (p. 74). Stembridge further argues that our students can best manage their own engagement when they are taught and supported and given the tools and techniques to engage.

Stembridge's second theme of culturally responsive education is *Relationships*. Stembridge argues that, 'relationships are particularly vital for students with more exposure to the elements of social, economic and educational disadvantage—relationships for them are the channel through which their investment with school is personalized' (p. 88). There is a wealth of research into the impact of relationships on students' achievements and performance at school and I argue too that using cooperative learning with students is a way of building relationships.

#### *2.5. Cooperative Learning and Engagement and Relationships*

Research into the benefits of cooperative learning has identified a number of outcomes that can be related to engagement and relationships. Language minority children can especially benefit from cooperative learning as a culturally responsive pedagogy [50]. Students with diverse needs and backgrounds, such as migrant and newly arrived students, need a safe and secure space to learn where they feel valued [51].

In a cooperative learning environment, students are more likely to be engaged [52–55]. Engagement occurs in these classrooms when teachers teach the tools and techniques to engage; for example, teachers who support their students by highlighting and teaching small group and interpersonal skills, are also able to develop engagement by using cooperative learning structures that support the specific skills.

Trusting relationship building is crucial, especially when teachers want to develop students who learn to respect other students whose values and ideas might differ to their own [56]. Cooperative learning requires this careful teaching of social skills in order to improve social skill development [57–62] and this in turn helps to develop relationships as students work together better when their social skills are well developed. Cooperative learning has a lasting impact on student learning as children who were trained initially in the processes and skills required for successful cooperative learning demonstrated ongoing positive relationships with other students with the ability to help each other and enhance thinking in future cooperative learning experiences [41]. The trusting relationships that are built through cooperative strategies in classrooms will develop collaborative skills that are crucial for the development of both the children's emotional, as well as academic development' [63]. Cooperative learning also helps to promote socialization and learning among students [42]. Additionally, cooperative learning can improve social problems [64], alleviate bullying [65] and help students manage conflict [66]). These all relate to better relationships between students in the classroom. Research [49,59,67] has found that when using cooperative learning students demonstrated the ability to provide explanations, instructions and develop implicit understanding of the needs of other group members which have developed positive relationships. Furthermore, research by Gillies and Ashman [68–70] found that students in structured groups exhibited more cooperative behaviors than other students, giving more unsolicited explanations, tuning in to their group members and offering help without it being requested. This also positively affects achievement [71]. These all relate to the better engagement of students, both cognitively and behaviorally.

#### **3. Methodology**

#### (1) Subjects of the Study and Setting

This paper focusses on classroom observations made in four Swedish classrooms in three different schools in late 2019 in Uppsala, Dallarna and Orebro. A Swedish teacher-researcher assisted me in locating schools which were interested in using cooperative learning and the schools were chosen as they were situated in areas which had increased populations of diverse students, including newly arrived or refugee students. Five teachers in the study consented and were selected as all regularly utilized cooperative learning strategies in their classrooms. Five teachers in the study consented and were selected as all regularly utilized cooperative learning strategies in their classrooms.

#### (2) Data Collection and Research Instruments

As argued by Quennerstedt [72], "Observational research is time consuming and therefore restricts the number of observation sites, but fruitful in allowing close-up examinations of how teachers and pupils talk and act in practice" (p. 612). This is further argued by [73], who also advocates for more research on 'teachers' integration of CL into day-to-day school practice' (p. 3). Baines et al. [74] concur that there is also a need to research how teachers implement cooperative learning in their own teaching practice.

I spent a day at each of the three schools. At least one classroom observation (between 30 min and an hour for each) was made with some teachers consenting to two observations. Only students and their parents who had given consent were observed (teachers organized the classroom so that I could focus on certain groupings and students working or organized the class so that only the consenting students were present). Follow up semi-structured interviews with the teachers were also conducted and recorded that day or the following day about their use of cooperative learning pedagogies. I used an observation proforma to look for aspects of democracy classrooms, but due to the language issues of me not understanding the Swedish teachers' and students' conversations, I wrote field notes and watched closely for signs that the spaces were welcoming and caring, the students felt secure and able to learn, and noted which pedagogies the teacher used to determine how they were inclusive, particularly focusing on cooperative learning. I examined the teachers' use of cooperative learning and asked them to define cooperative learning. During the semi-structured interviews that followed the classroom observations, I was able to follow up on what was observed and question further about their use of cooperative learning to determine the choices the teachers made and find out more about

what was occurring. This was necessary due to the language barrier of the classes being taught in Swedish and myself being an English-speaking researcher.

#### (3) Data Analysis

Interviews were recorded and transcribed and fieldnotes were written soon after the observation. These were read and analysed in order to classify the written and oral statements into categories of similar meaning [75]. After a number of iterations, I went back to the literature and examined the data using this lens, particularly focussing on Stembridge's themes of culturally responsive education for equity [47] situated in the data. This provided a theoretical framework and I embarked upon thematic analysis following the principles of coding, identifying open coding of the comments made by teachers during the interviews and fieldnotes recorded during classroom observations. Using Nvivo, I identified the links between the comments raised by the teachers, who had discussed their use of cooperative learning, or aspects of cooperative learning sighted in lessons (i.e., topic or axial coding), to propose higher order conceptualisation of the main themes related to the sub-themes of Stembridge's themes of *Engagement and Relationships* as concepts that are used to bridge equity and pedagogy alongside the literature on cooperative learning. I particularly noticed during the analysis process that many comments focused on the *Engagement* of students and the *Relationships* developed between teacher and students and among students, and therefore used ideas from Stembridge to analyze these encounters. Cooperative learning use occurs in classrooms with teachers who care about relationships [32]. During cooperative learning, when students learn together in small groups, teachers develop the relationships in that classroom and create a particular environment, or climate [35]. Engagement is important when working in cooperative learning groups for the learning to be successful, and social skills are necessary for students to be able to work successfully together [57–62] and remain engaged. Teachers who support their students by highlighting and teaching small group and interpersonal skills are assisting their students to remain engaged in learning activities.

#### (4) Ethical Considerations

Ethical considerations were made by applying and gaining consent through my University Ethics committee; seeking advice from Swedish researchers as well as reading about Swedish ethics requirements [76] to ensure the guidelines matched good research practice in Australia. After consenting to my visit, Principals sought voluntary permission from teachers and teachers sought consent from parents and students. Schools translated information statements and consent forms for parents and students so that informed consent could be given. Good research practice was followed by seeking written informed consent from all participants, providing information about the study, and ensuring results were reported confidentially.

#### *3.1. The Schools and Teacher Participants*

#### 3.1.1. School 1

The first school is a large public school in the northern part of a University city in the south of Sweden, west of Stockholm. Among the residents of the city are immigrants from around 165 different countries. The school focus is on life skills, which for them is about how to create a positive group climate through regular social and emotional training. In the school itself there are 597 pupils, which the Principal tells me has doubled in the past 5 years due to the number of new arrivals in the area. There are 47 pupils from ethnic minorities in the schools and 12% of students with another language as their first language, and a total of 27 different languages other than Swedish spoken at the school.

In Sofia's Year 4 class of 25 students (24 present that day) there are only three students with diverse backgrounds, from Syria, Iran and England. All the other students are born in Sweden, although many have parents who were born in other countries and moved to Sweden.

#### 3.1.2. School 2

The second school is in a small multicultural town of about 50,000 in the province of Dalarna, approximately 200 km south of Sweden and is a F-6 school with approximately 200 students. Currently, the major language groups in the town include Somali, Thai, Arabic and Kurmanji, Persian, Turkish, Albanian and Tigrinya. It is a rural school set close to meadows and forests and has an Equal Treatment Plan demonstrating its commitment to social justice and it also describes itself as a school characterized by care, with a strong emphasis on developing active and responsible citizens.

The two teachers (Elsa and Klara) I observed team-teach a Year 3 class of 34 students who are aged 9 and 10. They work between two classrooms, sometimes working as a whole class and other times splitting into two smaller units. They are both experienced teachers but are new to the pedagogy of cooperative learning. They are starting out and are hoping their Principal will let them attend a course after my visit.

#### 3.1.3. School 3

The third school is located in one of the four largest cities in Sweden and is located about 70 km north of Stockholm. The school has approximately 450 pupils. The city this school resides in has a 21 per cent population of the population born in foreign countries. Parents of the students in the school are not all first-generation Swedish residents. Some come from Spanish, German, Arabic, English, and Dutch backgrounds, so some students have mother tongue language studies that are arranged from the local authority. This school is in a village of about 2500 people.

In Jocelyn's year 1 class (ages 6 and 7) of 20 students, there are no students born outside of Sweden, and this is typical of other classes in the school despite the rise in immigration and hence now having a more ethnically, religious and linguistically diverse population in the area.

In Peta's year 4 class (ages 9 and 10) of 22 students, there is one student whose parents came from Iran and all other students are born in Sweden from Swedish-born parents.

#### **4. Results and Discussion**

#### *4.1. Narratives from the Classrooms*

#### 4.1.1. School 1: Sofia

I observed Sofia teach a year 4 Geography (social studies lesson) one Friday morning at 10 a.m.

Sofia told me these are new working groups from Monday. They have had the opportunity this week to make a group name and next week they will add a group slogan to their group poster. These are displayed in the room to give the students a sense of group identity.

(Karusellen) Carousel strategy (or Floorstorming) is the cooperative learning activity Sofia is using in today's lesson. Sofia told me that they have been learning about particular terms and their definitions in Geography like 'Map', 'Geography', 'Sweden', 'Natural landscape', Cultural landscape'. Each group member was given a role with a role card (these were Organiser, Secretary, Reporter, Encourager). The role cards had descriptions of the role and how to fulfil it on the back as a reminder. Sofia told me this was the first time she had given them the card but they were familiar with using roles. (She hoped it would help me if I could see who had each role).

They were given a piece of paper per group and one pen per group and were asked to work together to come up with ideas as to what the word meant to them. I watched the secretary write but others made suggestions as to what they could write. After a given amount of time they rotated clockwise to the next table and piece of paper. After 30 min all students were still engaged and were huddled together. After 40 min this was still the case and the class were asked for some reflection.

Calmly Sofia asked them to discuss together the following questions:

How did I cooperate in the group? Did I do my thing?

Did everyone get a turn to talk and did my friends listen to me?

Can the group now better talk and describe the definitions they have been working with about Geography?

She asked for a brief report back about the content of the task and these questions from the Reporter role.

Each Reporter gave some answers to the questions but often they answered to 'how did you work cooperatively?' simply with 'good'. In one group however they reflected that as the role cards were a novelty (the first use of them) they often just thought about this and forgot the other important aspect of talking and listening to others (in other words the task itself!).

#### 4.1.2. School 2: Elsa and Klara (Class Share)

I observed Elsa teach a Maths class to 14 students at 10 a.m. on a Thursday morning. Half of the class are in a music lesson and will do this same lesson later.

An overview of the lesson has been written on the board and their names are called out to put them into groups of 4 and 5. Elsa had decided to ensure there was a strong maths student and a weaker maths student in each pairing to ensure heterogeneous groupings.

A question (a maths word story problem) is written on the smartboard and they are asked to think about this by themselves. Each child is given a piece of paper and asked to think and write by themselves, then work in their pairs and talk to decide on an answer. They work quietly sharing ideas about how to solve the problem in pairs and then in fours.

Later on in the morning I see the other half of this class having a quiet reading lesson taught by Klara. They are asked to read a chosen book alone and then move into reading with a partner when they are ready (and they can see another child is ready to find a partner). This works seamlessly with different students accepting the offer to read and all being accepted in an inclusive way. The class reads and shares quietly in a sustained way for another 30 min. At times Klara suggests a text to a student (if she feels they may not be challenging themselves in this independent reading lesson) but other times she lets them have the freedom to choose and relax and read a book they just really want to read.

At the end of the school day the class comes together as a group of 34 students. They do a fun game thinking of how many words they can think of with 'ta' in them. After working for a while, they are asked to find a shoulder partner (this is a partner closest to one of your shoulders—either on the left or right hand side) and share a word in turn. The students are settled, engaged and obviously respectful of their partners.

#### 4.1.3. School 3: Jocelyn

I observed Jocelyn teach Swedish language lesson one Tuesday morning. The students arrive in the classroom just after 8 a.m. and quiet music is playing with the teacher greeting each student as they come in with either a high five or a hug (the students choose this). They seem very settled as they arrive with one more reluctant student (who I was told had just started school). The other students knew each other from previous classes (in preschool).

Jocelyn takes the roll by pointing to pictures of the children, pictures they have drawn of themselves. The music is on to settle and calm the students, the teacher makes time for a personal communication and greeting with each student and the roll taking shows that each student is valued by having their artwork displayed and this is being highlighted as important to the students at the start of each day.

Jocelyn goes over the day, date, and month with the students. They then count how many school days they have been at school and do this will bundles of sticks (developing place value information–groups of ten). The reticent child who did not want to come in class is sitting close to Jocelyn. She doesn't participate and has her head lowered. They then go over the 'y-chart in the classroom. There is one on taking turns and one on classroom voices. Gentle reminders from Jocelyn tell the students about today's focussed social skill on taking turns. What does taking turns 'look like', 'sound like', 'feel like'?

She sets up an activity. They are to go into pairs. They make the groups by finding someone with the same length arm. The activity is a lesson where one child is a detective and the other a secretary. The word detective has to spy on the words selected at the front. They are words they are focussing on at the time—common frequency words. The detective comes back and tells the secretary the word and how to spell it, then they swap roles. I notice different strategies being used to remember the words and how to write them. The pairs then go into groups of 4 and are asked to share their words. They are asked to do their secret handshakes. I notice by this time—the reticent girl has found a partner, she is participating in the group activity and she is smiling when they do the secret handshake.

Later, I observe the same class from 12.55–13.25. They are working on a maths activity with their shoulder partners. Before working they are reminded of the Y-charts in the room and particularly the ones about taking turns and using classroom voices. They are asked to find a partner and then one of them is to go and get the required materials—the one with the eldest father. They collect a small whiteboard, a small eraser and a blank sheet of paper. One member of the group using the counters on the whiteboard to make all the combinations that add up to 4. As they are made the other partner writes it as an algorithm on their blank sheet. Then they swap roles and one student makes a different group on the whiteboard with the counters showing it also with whiteboard marker and the other student writes the algorithm on their paper. They share the whiteboard and the counters but they have their own individual recording sheet for the algorithms. I notice the girl who has been shy earlier in the day working alongside a boy. There are several missing values on the sheet and they need to make them all have something in them before they are finished. They encourage each other to stay on task and to keep going. The lesson finishes with a pack up—each child is responsible to place their work in their own work folder and then find a matching number in the room and say the number in English.

#### 4.1.4. School 3: Peta

Peta's year 4 class is organised into tables of 2 to enable paired work and working in fours. I observe her class in a social science lesson one Tuesday morning at 10 a.m.

They move from quiet reading time just before and one child tries to disrupt the class–he is loud and is finding it hard to participate—he is asked to go to the time out area which is a small area in an enclosed space at the back of a room—like a small room. I ask Peta about him later and she tells me he finds visitors in the classroom difficult and prefers to have time away from it when this happens—she said he was quite stressed and the other students actually cope with him quite well although she does sometimes ensure he works in a three so if he is not coping and leaves the group for time out there are still two students to work together.

All students are working in pairs for this activity. They are given a bundle of definitions of words they are using in social science (some are terms they have used for the past four weeks and some are new terms not introduced before) and sort them into two piles so they half the cards each. On one side the cards have a definition and the other side the word. They look at the word decide if they know it for sure, or not sure, or kind of know what it means.

The student with the card reads the word . . . and talks about what they think it means if they know. They then look at the definition and sort into green (yes, they were correct), amber (kind of correct but not completely) or red (were not correct). Then the next student has a turn and they continue until all cards are sorted. Afterwards they look at the words again that are in amber and red and have another try and re-sort.

They work steadily on this lesson for 30 min and appear engaged throughout—improving their piles in the green section as they go.

The narratives from the classrooms give us a sense of these types of cooperative classrooms which have a certain classroom ambience, or democracy stance [32]. The kinds of practices we can see being used by these teachers demonstrate connection between the teachers and their students as well as among students. As Stembridge [47] argues, 'we are better teachers when we are connected to them' (p. 28). I will explore, through these narratives as well as from comments from the teachers during

their interviews, the themes of Engagement and Relationships in relation to literature on cooperative learning, as these are essential themes for teachers seeking to bridge equity through pedagogy.

Engagement is evident in these classrooms when the teachers employ tools and techniques to engage; for example, when Sofia mentions the importance of developing a good classroom climate and supports her students by highlighting and teaching small group and interpersonal skills which are important in a cooperative learning environment [62,77]

*"whole class level climate, climate* . . . *that's very, very important".* (Sofia, Interview)

*"That's The first thing* . . . *when someone speaks, you listen, and you look at that person, look at them, and respect".* (Sofia, Interview)

Climate is particularly important for newly arrived students [51].

*Behavioral engagement* is very apparent in all the classrooms observed and helps to develop that particular safe and secure class climate. There is an obvious intent from the students to follow the class rules. Sofia mentions the importance of helping the students create the rules at the beginning of the year to establish this climate and develop that respect, and asks questions like . . .

*"what kind of class do we want? So we can learn best? What should we do because we always have a relationship? Would you say* . . . *everyone here has a responsibility?"* (Sofia, Interview)

Jocelyn mentions too that she is able to develop engagement by using cooperative learning structures that support the specific skills.

*"the big thing is to talk about how to work in groups that you need to have. And also, it's important to thank each other, I think, this positivity in the group that feel that you need to belong with someone, and that you're not alone. The social skills [are important]."* (Jocelyn, Interview)

During the observation, I notice that this behavioral engagement is apparent in Jocelyn's classroom: Before working they are reminded of the Y-charts in the room, in particular the ones about taking turns and using classroom voices. Y-charts are charts that are split into three sections like a Y—each section asks students to consider what a particular social or small group skill 'looks like' 'sounds like' or 'feels like'.

This shows the class are used to thinking about and reflecting on the social skills necessary to work cooperatively.

In Elsa and Klara's classroom behavioral engagement is demonstrated by the students as they seamlessly move into reading with a partner when they are ready (and they can see another child is ready to find a partner). Different students accept the offer to read and all are accepted in an inclusive way. The class reads and shares quietly in a sustained way for 30 min. The teachers also talk about this behavioral engagement in their joint interview and how there is a need for modelling by the teacher,

*"We model our behaviour. We reflect on what's happened and ask how do we carry on? And we say 'stop' if it's just bad behaviour* . . . *if they have had bad behaviour to each other, we have done a lot of talking with them, they talk to each other and tell how they feel and [how they should] act towards each other. And now we try to let them solve problems. 'Okay, go in there and talk about it".* (Elsa and Klara, Interview)

Not only is teacher modelling of caring occurring in these classrooms [44,45] they are also learning how to help their students successfully manage conflict [66]. Students also learn to appreciate the skills of others and become positively connected in their learning tasks, in turn becoming more tolerant of students that they work with [42,43]). This type of acceptance and equity is fundamental to an intercultural classroom.

*A*ff*ective engagement* was demonstrated in Sofia's classroom when I observed that after 30 min into the lesson all students were still engaged and were huddled together; also, no one was excluded. This affective engagement was also demonstrated by the shy girl in Jocelyn's class who, although new to the school and reticent that morning to engage, demonstrated that feeling of belonging to her class community in a very short time. She found a partner, she participated in the group activity and she was smiling when they did the secret handshake. This is another way Jocelyn developed caring relationships in the class: this type of bonding brings them closer together in order to develop that democratic culture [78] where they are able to make choices about the secret handshake together and where they become like a community of many voices and ears together [33]. This community has allowed her to be included—there were not opportunities for her to be excluded as there would if she was asked to find a friend to work with. In this environment, she found someone like everyone else did.

Active learning was apparent in Elsa and Klara's classroom during the maths lesson involving small group learning. A question (a maths word story problem) was written on the smartboard and they were asked to think about this by themselves. Each child was given a piece of paper and asked to think and write by themselves, then work in their pairs and talk to decide on an answer. They worked quietly, sharing ideas about how to solve the problem in pairs and then in fours. There was no disengagement in the task observed and all students are involved in the learning [37]. Affective engagement goes hand in hand with cognitive engagement [47].

*Cognitive engagement* was also demonstrated by Jocelyn using cooperative learning structures. She had modelled social skills and their importance in how to tackle a task with others in the classroom too when she had her students examine the 'y-chart' in the classroom. Gentle reminders told the students about these social skills, focusing on aspects from the y-chart about what taking turns 'looks like', 'sounds like', or 'feels like'. The students in her class also demonstrated cognitive engagement when they shared the resources (whiteboard and the counters) demonstrating positive interdependence and then completing their own individual recording sheet for the algorithms demonstrating individual accountability.

*"And then it's important also to have a structure like 'Do we belong together', that we have a structure that you know how to use. And then that they always need something like, if they work together, they always need to, to, they always need each other. No-one can just say, Oh, you can do it all* . . . *I also have to participate. Also, that something that makes them meet each other [work together rather than alone]".* (Jocelyn, Interview)

She was a teacher who had learnt to use cooperative learning structures to support them in learning to work cooperatively. In turn the students are able to tune in to group members as this socialization is improved which also improves learning [42]. Good social and group skills then allow students to have the ability to provide explanations, instructions and develop an implicit understanding of the needs of other group members [49,79].

Cognitive engagement was again observed in Jocelyn's classroom when I observed the word activity lesson where one child was the detective and the other the secretary. The word detective had to spy on the words selected at the front. They were words being focused on at the time—common frequency words. The detective came back to tell the secretary the word and how to spell it, then they swapped roles. I noticed different strategies being used to remember the words and how to write them demonstrating students modelling different strategies to others and heightened engagement through the use of cooperative learning [52–55]. The pairs then moved into groups of four and were asked to share their words. They worked in a sustained way doing this, swapping roles and keeping each other on task. The use of different roles for individual accountability supported this cognitive engagement.

Cognitive engagement was evident in Jocelyn's classroom, with different students reminding others to stay on task, encouraging each other to keep going (both affective and behavioral engagement), but ultimately showing strategic understanding in how to complete a cooperative learning task (cognitive engagement).

*I notice the girl who has been shy earlier in the day working alongside a boy. At one point she says, "This isn't good," about what she has done—and the boy she is working with replies—"Yes it is–let's keep going!".* (Jocelyn, observation field notes)

*On another table a boy is resting his head on the table looking sleepy—his partner looks at the timer Jocelyn has put on the board and says "quick—you need to fill in your sheet". There are a number of missing values on the sheet and they need to make them all have something in them before they are finished.* (Jocelyn, observation field notes)

This gentle encouragement to stay on task and also to keep going was a sign that the pairs were supportive of each other and also enabled them to do a great job of the activity. These practices helped to develop social and emotional learning in the classroom [57,59–61] with positive and respectful relationships [41,71], a sense of belonging and 'relational trust' [80] being an obvious value that was promoted.

*Relationships* have a huge impact on the way a teacher can manage and develop the students in their classrooms. Research into the benefits of cooperative learning has also identified a number of outcomes including improved social skills and relationships between students. Ongoing positive relationships are developed when teachers take time to develop these skills [41]. Sofia stressed the importance of developing good relationships in her classroom.

*"I would say it's about cooperation in our relationships, and about how we learn together. That's what I think, what if you don't learn alone* . . . *This is so much better. And I think that especially* . . . *, you learn in a group where you learn from each other".* (Sofia, Interview)

#### She went on:

*"The first thing I would say, you need to do when working with the group is to be tight and be I mean, to be safe and* . . . *to be, you know, accepted to [be able to ] say [the] wrong things, instead of being* . . . *laughing at each other, you need to feel very safe in the group".* (Sofia, Interview)

Being safe and accepted in a group is particularly important for diverse students and newly arrived pupils who may be struggling with limited language skills and who are 'learning to be' as these students are often learning to act with greater autonomy in unfamiliar situations [29]. Being safe and included is also important as they learn to work alongside others and to participate and cooperate with others—'learning to live together' [29]. Again, Sofia talked about the safety the students felt in the small group situations when working in CL groups, but she also commented on how she can develop deeper relationships with her students as a result of the Swedish system of having more than one year with the students as their teacher. In this school, the teachers kept the students for three years.

*"So you can also see so many di*ff*erent sides of one student, you know, you can see [child's name], she's very shy in that way [in the whole class situation] but when she comes into this group, she's just so safe. I can take a step back. And I can see new sides of my students. And I think that's very, very important. because I also think you have a* . . . *when you're so many years with your students, you have you have, you know, sometimes you can have a specific idea about the students, you can, you know, is always something new to, to look for".* (Sofia, Interview)

She developed these relationships in the classroom through practical activities (poster and slogan making for group identity).

*"Student (from Syria) has been here for four weeks, you know. Her mum texted me two days ago, she said, this is the dialogue she told me. 'It's gonna be so fun'. She replied, 'Now, why is the school fun?' She replied 'Because Sofia is giving me so much encouragement, feedback, and tells us that we're going* . . . *we're doing a good job'. And she [the parent] tells me, so I* . . . *I didn't know that!"* (Sofia, Interview)

Similarly, Jocelyn developed relationships with her students by greeting her students outside the classroom. Immediately I saw this was a classroom with a democracy feel where relationships are prioritized [32].

*"I think that you want to see a child and have a relationship with them, know a little bit behind their name know a little bit about the parents, I'm very involved with the parents. They know me very much. Yes, I call them we have SMS text messages. I really work with my parents, and the kids know that, Okay. And I also work of course, with the kids. But this is what you call it. It's, it's a back and forth".* (Sofia, Interview)

Elsa and Klara developed the relationships in the classroom over time. They noted that the students "didn't always work like this" and that at times it was difficult to get them working in groups, but that now they are seeing developments in the relationships after the class has worked closely with cooperative learning for the past three years. I noticed in their classroom that they were willing to work with anyone. It didn't matter who worked with who. Elsa and Klara claimed, "it wasn't like that in the beginning" and now they are able to say to their students who may be experiencing issues with each other:

*"Now we try to let them solve problems. Okay, go in there and talk about it. Because it can [be necessary] you [the teacher] can't solve this* . . . *".* (Elsa and Klara, Interview)

As argued by Stembridge [47], ' . . . relationships for them [students who may experience social, economic or educational disadvantage] are the channel through which their investment with school is personalized' (p. 88). It is therefore important to develop acceptance and inclusion in the classroom and through this tolerance, active participation will be further developed [81]. Sofia mentioned she does this:

*"If I show them that they are accepted. You know, I'm interested, okay. 'Can you tell me about your culture?' 'What did you do when you were somewhere else?' You know, [give them the chance to talk]* . . . *And, you know, [some people] don't worry about what we have opinions about, you know, eg [they say] "I don't believe in other people coming to Sweden", because we have many, many, many immigrants coming. So right now it's, it's a very rough world [for some students]".* (Sofia, Interview)

Sofia mentioned how she was now a much more student-centered teacher since realizing the importance of developing strong relationships, and she is able to listen to her students so much better now as a result of this.

*"That's what I think is the big change for me that I don't need to be the focus. Yeah* . . . *I can still please just go ahead and I can watch them. Yes, like, so that's a new thing for me. I never used this before. Before this thing [using CL] and I was always listening [now]. I was always standing in the front and talking to them, talking and never giving them the opportunity. I never tried that [before]".* (Sofia, Interview)

Positive and respectful relationships and interactions with teachers have been shown to improve both engagement at school and academic motivation [54], with students performing better academically when they experience this sense of belonging at school [82]. As positive and respectful relationships are built, students in these classrooms have a sense of belonging, with many opportunities to talk about values [83] as they build a collaborative community.

Similarly, Gay [46] cites the importance of culturally responsive caring, as caring for students instead of about them. She encourages teachers to go beyond simply feeling concern for students and calls on us to become actively engaged in creating positive outcomes for culturally and linguistically diverse students. Culturally responsive caring begins with building relationships as an avenue to better understand students [84].

It has also been shown by a number of researchers that when students' teachers are caring and understanding, they become more willing to participate in class due to feelings of safety. The ability to be risk takers when participating in discussions also shows they are more likely to become more involved in society [85–87].

#### **5. Conclusions**

These classroom snapshots of learning, and discussions with teachers in a time of increased migration have demonstrated that the themes of Relationships and Engagement [47] are themes that bridge equity and pedagogy. Cooperative learning is one such pedagogy that develops these relationships and encourages an engaging and caring environment. The teachers have managed to develop strong relationships and have succeeded in strong engagement in their rooms: care has been evident in these classrooms. Their 'ethic of care' [45] is evident as they demonstrate how they have encouraged their students to recognise and value themselves. According to Kostoulas-Makrakis and Makrakis [88] 'The most valuable thing we can offer our learners is genuine care, hope, happiness and love' (p.178).The open dialogue between teachers and students and students and students is also evident, as is the practice of care towards each other in the rooms. Classroom climates which demonstrate the teachers know their students and allow them to grow accordingly is also apparent. It is increasingly important that in teacher education we provide 'training to gain awareness of the refugee experience as well as of the cultural backgrounds of refugee learners so that they can be responsive to refugee leaners' needs and be sensitive to trauma reactions' [88].The use of cooperative learning has shown teachers with a commitment to fairness, social responsibility and a concern for others. Now more than ever, our students need education that gives them the skills to understand and respect others as well as to be comfortable with difference [89]. It is important for us to revolutionize classrooms to enable our increasingly diverse student population to thrive. This is particularly vital for students with greater exposure to the elements of social, economic and educational disadvantage. Future research should focus on the importance of culturally responsive pedagogies and their ability to cater for diversity in a way that supports newly arrived students. This small study has its limitations in that it only focusses on one country and a small number of classrooms and teachers. However, the positive results from the study suggests more observational studies of teachers using culturally responsive pedagogies, such as cooperative learning are needed, to ensure our diverse population are catered for and nurtured in classrooms all around the world.

**Funding:** This research was funded by a "University of Newcastle FEDUA SNAPP grant". **Conflicts of Interest:** The author declares no conflict of interest.

#### **References**


**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Bringing Out-of-School Learning into the Classroom: Self- versus Peer-Monitoring of Learning Behaviour**

#### **Yelva C. Larsen 1,\*, Jorge Groß <sup>2</sup> and Franz X. Bogner <sup>1</sup>**


Received: 7 September 2020; Accepted: 10 October 2020; Published: 16 October 2020

**Abstract:** Based on classroom management fostering autonomy support and intrinsic motivation, this study examines effects of reciprocal peer-monitoring of learning behaviours on cognitive and affective outcomes. Within our study, 470 German secondary school students between 13 and 16 years of age participated in a multimodal hands- and minds-on exhibition focusing on renewable resources. Three groups were separated and monitored via a pre-post-follow up questionnaire: the first conducted peer-monitoring with the performance of specific roles to manage students' learning behaviours, the second accomplished a self-monitoring strategy, while the third group did not visit the exhibition. In contrast to the latter control group, both treatment groups produced a high increase in short- and long-term knowledge achievement. The peer-monitored group scored higher in cognitive learning outcomes than the self-monitored group did. Interestingly, the perceived level of choice did not differ between both treatment groups, whereas peer-monitoring increased students' perceived competence and simultaneously reduced the perceived level of anxiety and boredom. Peer-monitoring realised with the performance of specific roles seems to keep students "on task" without lowering indicators for students' intrinsic motivation. Herewith, we are amongst the first to suggest peer-monitoring as a semi-formal learning approach to balance between teacher-controlled instruction and free-choice exploration.

**Keywords:** peer-regulation; self-regulation; peer monitoring; workbook guidance; mobile science centre; semi-formal learning; hands-on learning; intrinsic motivation; autonomy in group learning; classroom management

#### **1. Introduction**

School research, teachers and institutions must have an interest in improving students' engagement and academic success. Teachers' supporting autonomy style and the basic psychological needs for autonomy, competence, and relatedness are major predictors for school engagement and academic success [1–3]. As a basis for this, firstly, stimulating learning environments are needed to foster students' active involvement and capacity to take responsibility for their learning. Secondly, strategies of classroom management are required that reduce teachers' role of being authoritarians but are still tailored to the demands of teaching.

#### *1.1. Bringing the Science Centre into Schools*

In contrast to everyday school life, "Scientific field trips to science centres can generate a sense of wonder, interest, enthusiasm, motivation, and eagerness to learn, which are much neglected in traditional formal school science" [4] (p. 125). Science centers often present fluid transitions between education and entertainment [5], providing opportunities for cognitive [6] as well as social and affective learning [7,8]. They offer authentic experiences with natural and technological phenomena combined with an autonomous, active and student-centred learning. This focus on a person's active involvement during a task should address students' engagement in a particular way [9]. Especially parameters such as intrinsic motivation, science learning achievement, interest in science and self-confidence can benefit from these learning experiences [10,11].

Although advantages of out-of-school learning seem obvious, mainly monetary or logistic constraints (e.g., due to location, organization and implementation of scientific fieldtrips) often prevent field trips in daily school routine [11,12]. Efficient practical approaches for the implementation of scientific activities inside schools are still rare.

Bringing portable exhibits into schools could valuably complement out-of-school learning offers [4]. This "semi-formal" learning approach should be adapted simultaneously to teachers' and students' needs: Curriculum connection is one of the most often mentioned motivations for teachers to leave the classroom [13–15]. Used as a supplement to "normal" classroom instruction, portable exhibits provide opportunities to reinforce or expand the classroom curriculum [16]. Exhibits can easily be presented within a conceptual framework; they introduce a learning topic, summarize it, or deepen and extend it. Portable exhibits, put up in schools could link student-centred attributes with pre-determined learning objectives.

For the developed of exhibits and related tasks in alignment with school curricula and practical implementation schools, science centres and museums should cooperate to create a shared vision [4,17]. The integration of portable interactive augmented reality approaches into teaching have already shown positive effects on learning [18]. However, little literature considers the structure needed to support semi-formal learning environments [19] and the question remains which strategies could balance between teachers' control and students' autonomy.

#### *1.2. Structure and Autonomy Support*

#### 1.2.1. What Is the Right Degree of Structure?

Degrees of structure usually are discussed controversially. In particular, the relationship between autonomy support and structure has been portrayed in different ways: from being antagonistic to curvilinear, and independent [20], whereas Cheon and colleagues found a positive correlation [21]. Teachers often prefer "structured student engagement" (e.g., docent-led guidance, worksheets or other writing activities) to free-choice learning [22,23]. Besides curricular specifications, the demand to keep students "on task" seems to be triggered by teachers' concerns about their ability to manage or control student behaviour, in particular within learning environments outside the classroom [11]. A feeling of not being able to maintain control is frequently cited as heightening teachers' stress [24,25].

In contrast, informal learning settings often provide unstructured and non-sequential learning environments [26,27]. Less structure, missing accountabilities and an oversupply of stimuli as well as information may hamper a student's focus on predetermined learning objectives [9] and intended connections, underlying the different types of exhibits [9,28]. Minimal guidance has been criticized regarding the achieved knowledge gain, see, e.g., [29–31]. A meta-analysis of Alfieri and colleagues clearly showed that directly guided instruction significantly improved knowledge gains in comparison to non-guided discovery learning settings [32]. Unlike rigid or unstructured formats, the more structured ones are often associated with an increase of feelings such as autonomy and achievement compared to either rigid or completely unstructured formats [33–36]. Different approaches seemingly minimize the amount of choice in favour of a combination of guidance and exploration [37–40]. Still, teacher-controlled guidance often reduces the opportunities to explore and is not always adjusted to the students' needs, in particular when taking intrinsic motivation and learning emotions into account [41,42].

#### 1.2.2. Intrinsic Motivation as a Predictor for Students' Engagement and Academic Success

According to Self-Determination Theory (SDT), intrinsic motivation results from the perception that one's basic needs for autonomy, competence and relatedness are satisfied [43]. Needs for competence imply to interact effectively with one's environment, to have opportunities to express one's abilities and to experience skilfulness [43]. The need for relatedness is associated with a feeling of connectedness to others within a social context rather than being isolated from others [44]. Autonomy is defined as the need to be the origin of one's own behaviour and the experience of volition, choice, and self-regulation [43].

Students' basic needs satisfaction predicts students' school engagement for learning activities, see, e.g., [45–47]. As a multidimensional construct, school engagement represents the way in which students act, feel and think [48] and in which they participate. It involves a behavioural, cognitive and emotional dimension: Behavioural engagement includes the compliance with school rules, students' involvement and persistence in school-related activities [49]. Cognitive engagement includes the understanding and use of self-regulated learning strategies such as planning, monitoring and evaluating one's own progress [50] and the motivation to learn [33]. Emotional engagement includes the emotional quality of a person's active involvement; it refers to students' attitudes towards learning, classmates and teachers and their personal identification with school and schoolwork [51].

Melchor and colleagues [2] assumed a mediation model (Figure 1) that is based on the positive correlation between autonomy support, psychological needs satisfaction, school engagement and academic success. They confirmed this model in a survey with 2034 Angolan students and 2302 Dominican Republic students.

**Figure 1.** Predictors for cognitive outcomes and academic self-concept hypothesized by the mediation model [2].

In line with SDT, the mediation model stresses the importance of autonomy-supportive strategies to support students' engagement and academic success. Stimulating learning environments should clearly foster students' perception of having control over their learning process, but they should also meet school demands [20]. Peer-monitoring could be a promising approach to combine both, autonomy support and structure within group learning activities.

#### *1.3. Peer- versus Self-Monitoring to Regulate Students' Learning Behaviour*

As a student-centred learning approach, group learning is frequently used within formal and informal learning settings. Cooperative learning environments and learning settings in which one person (the tutor) imparts knowledge to another person (the tutee) combine cognitive activity with social interactions [52–55]. In comparison, instructional monitoring is typically based on behavioural activity, making it particularly suitable for classroom management. Hereby, self-monitoring is a self-management strategy that involves observing, assessing and regulating one's own behaviour [56]. It has been found to be a key process in self-regulated learning [57]. In contrast, reciprocal peer-monitoring implies monitoring classmates as well as being monitored. Brown and colleagues [58] defined peer-monitoring as peer-observing and checking the behaviour of others within the group regarding appropriateness and effectiveness. For both strategies of monitoring teachers as well as students could define target behaviours. Self-monitoring usually requires minimal training and is easy

to implement [59], in contrast previous training for peer-monitoring depends on monitor variables (e.g., single vs. multiple behaviours) and the specific behaviour that is required. Interventions of self-monitoring, e.g., in [60–64] and peer-monitoring, e.g., in [58,63] have been applied to promote academic performance and "on-task" learning behaviours among students of different ages and across a variety of learning disabilities and emotional-behavioural difficulties. Thereby, self-monitoring was often used in learning settings with no possibilities for teacher control and in multiple contexts like math [65] and writing or reading performance [60].

Previous research can only be compared to a certain extent: Self-monitoring is mostly used in learning settings where students work on their own (e.g., while doing homework); whereas, peer-monitoring always implies social interactions. The scope of monitoring within and between both methods could differ from very little to intensive monitoring. In addition, monitoring could be used within different learning contexts and in combination with other instructional approaches, such as the use of a graphic organizer [60]. Thus, outcomes could differ fundamentally.

In the majority, both self- and peer-monitoring supported desired behaviours within the specific contexts and revealed to be equally effective, or superior to, teacher-directed procedures when knowledge gain was taken under consideration (see [58,66] for peer-monitoring and [67,68] for self-monitoring).

In contrast to positive effects of self-regulated learning on intrinsic motivation, self-efficacy and self-awareness [57], results of peer-monitoring are ambiguous. Henington and Skinner [67] stated that "peer monitoring can bring about both positive and negative social, emotional, and cognitive changes in students who are being monitored, as well as in those students doing the monitoring." (p. 251). Within the mediation model [2] autonomy support is an important predictor of students' intrinsic motivation. Peer-monitoring as "learners keeping an eye on whether their partners are going through appropriate and effective processes and procedures of learning" [58] (p. 174), could minimize students' perception of self-control in comparison to self-monitoring. Still, there is some evidence that regulation is not correlated with a feeling of restriction if guided by a person of the same age [46] and reciprocal peer-monitoring could establish a positive interdependence among students. Additionally, taking cooperative learning into account, peer-monitoring can strengthen prosocial behaviour [69]. In conclusion, so far, students' perception of peer-monitoring in comparison to self-monitoring is difficult to predict.

#### *1.4. Study Question*

To answer the research question if reciprocal and similar-age peer monitoring of students' activities should be recommended as guidance within semi-formal learning environments, we compared it with self-monitoring. Thus, we are among the first to test self- versus peer-monitoring with regard to effects on students' (1) cognitive achievement, (2) intrinsic motivation (perceived competence and perceived choice) and short-term learning emotions (anxiety, boredom, interest and well-being) as predictors of students' emotional engagement. Differences of previous literature in research designs, learning settings and the scope of monitoring prevents us from drawing concrete hypotheses, e.g., [63,70]. The comparison with self-monitoring could reveal important tendencies if peer-monitoring might be a key to guide group learning activities within semi-formal learning.

#### **2. Materials and Methods**

#### *2.1. Instructional Concept*

We designed a mobile exhibition to test for differences between peer- and self-monitoring. In line with an "Education for Sustainable Development" (ESD), our exhibits focused on the potential of renewable resources to replace fossil fuels, especially by focusing on bioenergy. As an interdisciplinary educational goal, ESD became part of the curriculum across all schools in most federal states within Germany. Content-wise, it addresses the Sustainable Development Goals (SDGs), defined as

17 interlinked goals "to achieve a better and more sustainable future for all" [71]. As an essential part of the Agenda 2030 for Sustainable Development, the SDGs were set by the United Nations General Assembly in 2015 and are intended to be achieved by the year 2030. As part of young students' scientific literacy, it is not only imperative to sensitise their awareness of the finiteness of fossil fuels, but also to inform them about possible alternatives and related technologies. In contrast to wind, water and solar energy, educational approaches concerned with the topic of bioenergy are still limited and many students show only little knowledge of the use of biomass as an energy source [72,73]. This lack of knowledge seems to be independent of age and origin [74,75]. Within the exhibition, references can be made in particular to SDG 7, which proposes to "Ensure access to affordable, reliable, sustainable and modern energy for all" and to SDG 12, which calls to "Ensure sustainable consumption and production patterns".

Our exhibition offers eight mobile exhibits that focus on the heterogeneous nature of products based on renewable resources and used for bioenergy. They could be presented in out-of-school settings as well as inside schools. Exhibit modules can be used independently as solitary elements. We placed a special focus on the application of "hands-on" exhibits in combination with multi-sensory exploration as they are considered more attractive and leading to longer-lasting attention compared to non-participatory exhibits [76,77]. Exhibits 1–4 covered a comparison of products based on renewable resources and mineral-oil-based products by including four aspects:


Exhibits 5–7 showed application forms of bioenergy based on renewable gaseous, solid and liquid biomass-based fuels; they included:


Exhibits 8 presented an interactive computer game as a voluntary learning activity that was not included in the knowledge questionnaire. The "energy race" required physical effort (displayed as kW) by riding a fixed bicycle on a roll.

#### *2.2. Workbook Guidance*

In favour of one independent variable, we combined both peer- and self-monitoring with an instructional workbook, providing learners with guidance about how to interact with the exhibits—cf. [78] (p. 162 ff.). Following Krombaβ and Harms [79], the workbook consisted of a closed task setting and was subdivided into sections; each section corresponded to a "hands-on" exhibit. The informational sheets provided clearly defined solutions and were, together with the workbook contents, subsequently monitored in the knowledge questionnaire. We focused on factual information in favour of a between-groups comparison with only one independent variable (peer- versus self-monitoring). Thus, we deliberately avoided open-ended tasks—cf. [4], in favour of group comparisons.

#### *2.3. Implementation*

Within the framework of "Bringing the science centre to school" the mobile exhibition was offered to secondary schools (Level 2 of the International Standard Classification of Education) in Bayreuth (Bavaria) and the surrounding area. The data were collected by carrying out a questionnaire survey with a pre-, post- and retention-test design, including a control.

In total, 470 students (8th/9th grade) participated, with an average age of 14 years (8th) and 15 years (9th grade). Classes from different schools were randomly assigned to the experimental groups, but students from the 8th and 9th grades were not mixed.

The visit of the exhibition was part of an educational programme: Two groups, a self-monitored group (*n* = 218, female: 47.7%) and a peer-monitored group (*n* = 230, female: 49.4%) participated in our preliminary session (20 min), the workbook-guided visit (90 min) of the exhibition and the debriefing session (15 min) (see Figure 2). An additional control group (*n* = 22, female: 34.8%) did not participate in the educational programme. The preliminary session of the programme was divided in two parts: The first part comprised a topical introduction that provided background knowledge for a more profitable learning experience. The second included an introduction to the process of monitoring (see Section 2.4). The debriefing session (15 min) summarized the learning outcomes. The same person conducted the preliminary and the debriefing sessions to avoid any "teacher effects" within test results; during the workbook-guided visit of the exhibition, there was no teacher guidance. Per day, two classes visited the exhibition with a time frame of 135 min per class. The number of students in the exhibition did not exceed 34 at any given moment. Classes were randomly divided into self- and peer-monitored groups and subdivided, also randomly, into groups with up to four students each. Both treatment groups completed identical exhibit-related and workbook-guided activities. We specified different rotation sequences for each group in the workbook to provide a smooth cycle between exhibits. Each group had a time frame of up to 15 min per exhibit.

**Figure 2.** Overview of the study design including the two interventions and the control group (white boxes), applied scales (grey boxes) and time of testing indicated by an arrow.

#### *2.4. Self- versus Peer-Monitoring*

The experimental groups differed in internal regulation (self-regulated behaviour) and externally regulated behaviour (peer-regulated behaviour). We defined particular target behaviours that should help students to keep the focus on specific learning objectives—see [66], which were communicated to both groups. The self-monitored group assessed individual learning behaviour by checkmarks that were placed on supplementary sheets. In addition, behavioural rules were fixed on a poster that was put up in front of the class (self-monitored group) or connected with specific roles and written down

on role cards (peer-monitored group). Each peer-monitored group received a little bag with the same amount of role cards as group members. Before starting with an exhibit, each person had to draw a card out of the bag and put it back when finished. Meanwhile, role card holders were responsible for the adherence of role instructions (Table 1) by the rest of the group. Requisites visualized specific roles; e.g., glasses for the role as a reader.


**Table 1.** Description of role-specific instructions as target behaviours for peer-monitoring.

#### 2.4.1. Measurement of Science Learning Outcomes

We applied a pre-, post- and retention test to monitor pre-knowledge as well as the short- and long-term learning effects of the self- and peer-monitored groups. For the assessment of knowledge achievement (as the dependent variable), we developed a multiple-choice questionnaire with 21 items and four distractors per item. A pilot study validated the multiple-choice questionnaire on the basis of 25 students (female: 45.8%) without visiting the exhibition. Questions covered factual knowledge with regard to workbook tasks and information sheets (see Table 2).


**Table 2.** Listing of four item examples of the science achievement questionnaire <sup>1</sup> .

<sup>1</sup> Note that more than one correct answer could be given.

We used the same questions and students' incorrect answers as distractors in the multiple-choice questionnaire and conducted a distractor analysis (*n* = 51, female: 36.2%; note that gender distribution was not balanced). Based on this analysis, we selected distractors with a solution probability that varied from 20% to 80%. We applied an item difficulty index of about 25% easy items, 50% medium items and 25% difficult items [80]. Multiple-choice items had a sufficient level of internal consistency, determined by Cronbach's alpha coefficients (*n* = 470) of 0.651 (pre-knowledge), 0.751 (post-knowledge) and 0.688 (retention-knowledge) [81]. In the main study, students completed the pre-test one week before participation, the post-test immediately after, and the retention test six to eight weeks later (Figure 1). Students were not informed about the test schedules or repeated applications; see, e.g., [82]. Additionally, students were assured that their responses would have no influence on their school marks. We advised teachers not to make any thematic connection to the exhibition until finishing the study with the retention questionnaire. In order to exclude external learning variables not due to the programme—e.g., [83], a control group (*n* = 22, female: 34.8%) completed the questionnaire three times without participating in the programme. In the statistical analysis, we considered gender differences because the distribution was not balanced.

#### 2.4.2. Measurement of Predictors for Short-Term Learning Emotions and Intrinsic Motivation

Within the post-questionnaire, we assessed students' short-term learning emotions and indicators for students' intrinsic motivation as dependent variables. The German translation [84] of the subscales—perceived competence (6 items, e.g., "I was pretty skilled at this activity.") and perceived choice (7 items, e.g., "I did this activity because I wanted to.")—of the intrinsic motivation inventory [85] was applied as well as subscales for the short-term learning emotions (4 items each, which also included reversed items) of anxiety (e.g., "I felt under pressure during the activity."), boredom (e.g., "I felt bored."), interest (e.g., "I want to learn more about that topic.") and well-being (e.g., "I enjoyed the activity"); based on Gläser-Zikuda and colleagues [86] and adapted by Meissner and Bogner [87]. Both instruments used a 4-point Likert scale (1 = very high, 4 = very low). We assessed sufficient Cronbach's alpha coefficients between 0.730 and 0.849 for all scales [81].

#### *2.5. Statistics*

SPSS Statistics was used for data analysis. The effect size (r = z √ N) was calculated according to Field [88] with 0.10 regarded as a small, 0.30 as a medium, and 0.50 as a large effect size [89]. We conducted a non-parametric Mann–Whitney test for inter-group differences and a Wilcoxon signed-rank test for within group differences, due to the non-normal distribution of our scores (Kolmogorov–Smirnov test, *p* < 0.05). Additionally, a binary logistic regression measured the relationship between the post- or retention knowledge (categorical dependent variable) and scenario, pre-knowledge and gender (independent predictor variables). Results display the odds ratio as a probability score where the independent variable is a predictor of the dependent variable. We included the effect of pre-existing knowledge, gender and the interaction as predictor variables. The odds ratio shows the effect of the variables in the equation (reference category: peer-monitored group, right answer, male); odds ratio = 1 reflects no difference in knowledge gain, odds ratio >1 indicates a higher, and odds ratio <1—a lower probability to give correct answers.

#### **3. Results**

For none of the group comparisons significant differences were calculated for different grades or gender.

#### *3.1. Within-Group Comparison of the Control Group*

A control group (*n* = 22) completed our questionnaires at the three test intervals. A comparison between pre-test (25th/75th, P = 6.0/11.25, median = 10), post-test (25th/75th, P = 8.75/12, median = 10) and retention test (25th/75th, P = 6/13, median = 10) revealed neither external short- and long-term nor persistent learning effects: Pre- versus post-test (z <sup>=</sup> <sup>−</sup>1.28, *<sup>p</sup>* <sup>=</sup> 0.2, r <sup>=</sup> 0.27); pre- versus retention test (z <sup>=</sup> <sup>−</sup>0.68, *<sup>p</sup>* <sup>=</sup> 0.50, r <sup>=</sup> 0.15); post-test versus retention test (z <sup>=</sup> <sup>−</sup>0.49, *<sup>p</sup>* <sup>=</sup> 0.63, r <sup>=</sup> 0.10).

#### *3.2. Science Learning Outcome*

Sum scores of correct answers of both peer- and self-monitoring showed a highly significant knowledge increase in short- and long-term learning outcomes in comparison to pre-knowledge. Long-term achievement decreased in both groups in comparison to short-term learning outcomes (Figure 3).

**Figure 3.** Box-and-whisker plot showing sum scores of knowledge achievement of the self- and peer-monitored group at the three times of testing. Note: The box-and-whisker plot indicates the median, lower and upper quartile and the 9th and 91st percentile; \*\*\* significant at <0.001, \* significant at <0.05.

#### *3.3. Peer- versus Self-Monitoring*

− − − At all testing times, the peer-monitored group showed higher achievement outcomes (pre-, postand retention test) than the self-monitored group (Table 3). Differences remained if effects of the predictor variables of pre-knowledge and gender were excluded. In detail: Our inter-group comparison of science knowledge achievement scores (21 items) showed that, immediately after the programme, the peer-monitored cohort (25th/75th, P = 13.0/18.0, median = 16) performed significantly better than the self-monitored cohort (25th/75th, P <sup>=</sup> 10.0/16.25, median <sup>=</sup> 13), (U <sup>=</sup> 17837.50, z <sup>=</sup> <sup>−</sup>5.30, *<sup>p</sup>* <sup>&</sup>lt; 0.001 \*\*\*, r = 0.25). The better performance of the peer-monitored group (25th/75th, P = 10.0/16.0, median = 14) compared to the self-monitored group (25th/75th, P = 9.0/14.0, median = 12) still persisted six weeks after the programme (U<sup>=</sup> 18872, z <sup>=</sup> <sup>−</sup>4.54, *<sup>p</sup>* <sup>&</sup>lt; 0.001 \*\*\*, r <sup>=</sup> 0.21). A higher score of the peer-monitored cohort (25th/75th, P = 7.0/13.0, median = 10) was already visible in the pre-knowledge test (U= 21835, <sup>z</sup> <sup>=</sup> <sup>−</sup>2.37, *<sup>p</sup>* <sup>&</sup>lt; 0.018\*, r <sup>=</sup> 0.11) in comparison to the self-monitored cohort (25th/75th, P <sup>=</sup> 7.0/11.0, median = 9).



Note: Wilcoxon signed-rank test z; \*\*\* significant at *p* < 0.001, exact one-tailed significance; effect size r.

We applied a logistic regression to test for the influence of pre-existing knowledge and gender in post- (Table 4) and retention knowledge (Table 5). Asterisks indicate that all predictor variables (scenario, pre-knowledge and gender) for post- as well as for retention knowledge were statistically significant (with the exception of gender in step 5 for the retention knowledge): Depending on the predictor included in the model, the odds ratio reveals a probability, which is 33.1% (odds ratio = 0.669) to 39.8% (odds ratio = 0.602) higher, that the self-monitored group responds incorrectly in the post-knowledge test (Table 4). The retention knowledge test shows similar results, with a probability that is 22.3% (odds ratio = 0.777) to 40.8% (odds ratio = 0.592) higher (Table 4).

**Table 4.** Logistic regression predicting "post-knowledge" from scenario, pre-knowledge and gender by odds ratio.


*n* = 9534 (\* significant at *p* < 0.05, \*\* significant at *p* < 0.01).

**Table 5.** Logistic regression predicting "retention-knowledge" from scenario, pre-knowledge and gender by odds ratio.


*n* = 9534 (\* *p* < 0.05, \*\* *p* < 0.01).

#### *3.4. Short-Term Learning Emotions and Intrinsic Motivation*

Students of the peer-monitored group had a significantly lower perception of their level of boredom and anxiety and a significantly higher perception of their level of competence in comparison to the self-monitored group (Table 6). We did not record any differences with regard to the perception of choice, well-being and interest. Within all between-group comparisons, the effect size r was small.

**Table 6.** Between-group comparison of predictors for the perception of short-term learning emotions (left side) and intrinsic motivation (right side).


Note: Mann–Whitney U test and z; \*\* significant at *p* < 0.01, \* significant at *p* < 0.05, exact one-tailed significance; effect size r.

#### **4. Discussion**

Based on the mediation model (see Figure 1), we assumed that peer-monitoring should support students' capacity to take control of their learning, if monitoring is not connected with a feeling of restriction. We proposed peer-monitoring of students' activities as an instructional guidance within semi-formal learning environments. In comparison to previous studies that had a strong focus on academic achievement, the study at hand examined not only students' learning outcomes, but also affective predictors for students' classroom engagement. The between-group comparison underlined that the application of peer-monitoring resulted in an increased level of knowledge combined with a better perception of competence, while anxiety and boredom levels diminished.

In favour of comparability, cognitive and affective effects are considered separately, even though engagement is a multifaceted construct and cognitive learning outcomes, intrinsic motivation and short-term learning emotions should not be regarded as isolated factors but as influencing each other; see [90,91].

#### *4.1. Within-Group Comparison*

Both the self- and the peer-monitored group demonstrated a significant increase in short- and long-term cognitive learning. This is in line with previous research in peer-monitoring [58,66] and self-monitoring [67,68] that focused on students' knowledge gain.

We did not compare workbook with non-workbook guidance, but we still can conclude that the applied workbooks provided guidance about how to interact with the exhibit and seemed to be effective with regard to factual knowledge. Our results support the combination of instructions for behavioural activity with an additional assistance to structure cognitive activity, such as workbooks. They are in accordance with Krombaβ and Harms [79], who showed positive effects of worksheets for acquiring basic knowledge in a natural history museum. Researchers provide counterarguments like the restriction of exploration [92,93] and pro-arguments, e.g., facilitating group dialogues and curriculum connection [94] with regard to worksheets in an out-of-school learning setting. In accordance with Kisiel [14,23] we emphasize that not simply the presence or absence, but the realisation of well thought-out workbooks or -sheets matters. Especially workbooks that are in line with inquiry- and problem-based learning could be very engaging.

To conclude this point, well-designed workbooks that fit curriculum standards and the learning objectives intended by the design of exhibits can help to bring more scientific activities into the classroom and create a shared vision between schools, science centres and museums; see [4,94] (p. 1389).

#### *4.2. Between-Group Comparison (Peer- versus Self-Monitoring)*

#### 4.2.1. Science Learning Outcomes

The peer-monitored instruction revealed better short- and long-term learning outcomes compared to the self-monitored group. Efficiency of peer-monitoring has been shown especially with regard to its function of keeping learners on track [58,66]. McCurdy and Shapiro revealed teacher- and peer-monitoring of process learning behaviours to be equally effective [95]. It seems likely that a higher percentage of "on-task" behaviour is also connected with better learning outcomes. Likewise, self-monitoring could increase "on-task" behaviours, particularly when incorporated with other instructional approaches [60]. The assessment of one's own learning behaviour requires self-regulation. However, this should be expected from the studies' participants who attended secondary school with an age range of 13 to 16 years. Particularly young children or student populations with high incidence of disabilities or learning difficulties face challenges in self-regulation during independent learning [95,96].

As an additional explanation, the distribution of tasks between learners within peer-monitored groups could have a positive effect on learning outcomes. King [97] introduced the idea of distributed metacognition by the distribution of components of control among collaborating learners (p. 61, 70 f.). Although we did not perform an observational study, we detected interesting group dynamics within the self-monitored group: Some students took over a "leader function", while others became passive followers. Scharfenberg and colleagues showed in a video analysis that even in hands-on lessons, passive followers remain passive [98]. In the peer-monitored group, activity cards challenged all participants to accept responsibility for a specific task, engaging thus both active and passive students, as well as low and high achievers.

#### 4.2.2. Predictors for Students' Emotional Engagement

Neither effect sizes of short-term learning emotions nor predictors for intrinsic motivation revealed high values (Table 3). Thus, significant differences between results should not be overestimated.

#### Students' Intrinsic Motivation

In comparison to self-monitoring, peer-monitoring resulted in a higher perception of competence and choice, both subscales are indicators for intrinsic motivation; e.g., [45,99].

Peer-assisted learning promotes competence and autonomy among students, which has a positive effect on students' self-conceptions; for a review, see [100,101]. However, peer-monitoring considered as peer-controlled learning behaviour should not be equated with peer-assistance. A higher perception of competence and choice in peer-monitoring, compared to self-monitoring, is an interesting result.

The assignation and performance of specific roles might be connected with the higher perception of competence. A feeling of competence implies to interact effectively with one's environment and to have opportunities to express one's abilities and to experience skilfulness. Reciprocal and same-age peer-monitoring could establish a positive interdependence among students that might trigger these characteristics. Within Self-Determination Theory, the three "psychological needs" of competence, relatedness and autonomy interact with each other; see, e.g., [45,47]. In favour of having a reasonable number of questionnaire items, we did not measure social relatedness, although collaboration and cooperation are seen as critical in learning and engagement [102]. Social relatedness is an important factor, in particular with regard to informal learning settings and might play an important role especially for young people [103,104]. Working in small groups in combination with the application of roles could have influenced group dynamics and social relatedness positively —cf. [105].

When coming back to our initial question, the lack of significant differences in the perception of control is a promising result. It indicates that in comparison to self-monitoring, peer-monitoring might not be perceived as an external control of behaviour. Finally, intrinsic motivation is a multifaceted construct and we cannot explain all interactions between the psychological needs, but we can still conclude that peer-monitoring seems to shape a social environment, which fosters single predictors for intrinsic motivation.

#### Short-Term Learning Emotions

Self- and peer-monitored instruction did not provoke notable effects on well-being and interest, but peer-monitoring lowered the perceived level of anxiety and boredom. Short-term learning emotions cannot always be expected to be long-lasting, but they can very much influence learning processes [106–108]. Anxiety and boredom are defined as specific "achievement emotions" which correspond to students' learning achievements [91].

The application of roles that specified particular target behaviours may also explain why students felt less bored: Role keepers had to make sure that the rest of the group followed instructions connected with their specific role; additionally, their behaviour was monitored by group members. It is likely that both actions kept students more active than those in the self-monitored group.

Interestingly, Pekrun and colleagues [109] stated: "Self-regulating one's own learning may induce positive feelings, whereas external control may induce anger, anxiety, or boredom" (p. 99). In 2010, they confirmed the inverse relation of boredom with self-regulation [110].

In view of this, peer-monitoring in combination with the performance of specific roles did not seem to be perceived as an external control and a loss of self-regulation. This may be due to the fact that peers are not perceived as an authority; each person in the group performs a specific task that could be experienced as part of a game, as "without rules there is no game" [111]. We tried to avoid the risks of comparisons within the groups, the "competence threat" and social divisiveness [112] by applying a structured switching of roles. Interestingly, Ryan, Stiller and Lynch [113] described pupils as internalizing extrinsic school regulations when they felt secure and cared for. The reduced perception of anxiety is an intriguing phenomenon; Gläser-Zikuda and colleagues [86] described self-regulated learning as requiring a variety of strategies that need to be performed before they may be experienced as emotionally positive and valuable. Although both groups were expected to follow similar behavioural rules, the higher perception of anxiety within the self-monitored group could be based on insufficient experience in self-regulated learning, creating insecurity. In different contexts, the new and unknown may have overwhelmed students and influenced the cognitive [114] as well as the affective outcomes [115] negatively. With the application of role cards, students knew and told each other how to behave, apparently being emotionally reassuring, and thereby reducing the level of perceived anxiety [116].

#### **5. Conclusions**

In accordance to Malcolm, Hodkinson and Colley [117] we conceive "formality" and "informality" as attributes present in all circumstances of learning that should not be considered separately. We are amongst the first to suggest peer-monitoring conducted with the performance of specific roles in combination with workbook guidance to better link these attributes. The application of role cards provides an easy and cost-efficient implementation to define and target specific learning behaviours. Hence, peer-monitoring is not an informal support but refers to a learning situation planned by a professional teacher and the amount of structure ought to be adapted to requirements, purpose and context individually. In conclusion, peer-monitoring in combination with well-designed workbooks seems a fruitful approach that implies reduced teacher control but is still tailored to teachers' needs. This could provide an enriching repertoire of student-centred learning opportunities, suitable not only in out-of-school contexts, but also during classroom teaching. Our research contributes to the fields of "semi-formal learning" as well as motivation and autonomy in group learning.

#### *Limitations and Recommendations*

Behind the background that autonomy support is a determining factor to improve students' engagement and academic success, a limitation of this study was that the educational environment provided limited possibilities for students to be creative. In favour of a between-group comparison, we deliberately avoided open-ended tasks and had a focus on factual information. Learning tasks could have been presented in a much more stimulating way, especially when taking inquiry-based learning and problem solving into account. This also points out a research challenge: self-control individualises learning processes and makes it difficult to compare the efficiency of different methodologies. In a broader context politicians, economists and scientists should discuss approaches that bridge the gap between curricular demands, social structures characterised by comparability and students' individual needs for autonomy.

**Author Contributions:** Funding acquisition, F.X.B.; Supervision, F.X.B.; Writing—original draft, Y.C.L.; Writing—review & editing, J.G., F.X.B. and Y.C.L. All authors have read and agreed to the published version of the manuscript.

**Funding:** Funding was provided by the University of Bayreuth and the Bioenergy-Region Bayreuth. Additionally, the German Research Foundation (DFG) in the funding program Open Access Publishing under grant agreement no. LA 2159/8-6. funded this publication. The funders had no role in the design of the study, in the collection, analyses, or interpretation of data, in the writing of the manuscript, or in the decision to publish the results. This article reflects only the authors' views. The funders are not liable for any use that might be made of the information contained herein.

**Acknowledgments:** We are very thankful to all students for participating and we greatly appreciate the support of the Bioenergy-Region Bayreuth represented by B. Rothammel.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Education Sciences* Editorial Office E-mail: education@mdpi.com www.mdpi.com/journal/education

MDPI St. Alban-Anlage 66 4052 Basel Switzerland

Tel: +41 61 683 77 34 Fax: +41 61 302 89 18

www.mdpi.com ISBN 978-3-0365-2970-7