Next Article in Journal
STEM Teachers’ Digital Competence: Different Subjects, Different Proficiencies
Previous Article in Journal
Characteristics of Filipino Online Learners: A Survey of Science Education Students’ Engagement, Self-Regulation, and Self-Efficacy
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Investigating Metacognitive Strategies and Exam Performance: A Cross-Sectional Survey Research Study

by
Jolie V. Kennedy
1 and
David R. Arendale
2,*
1
Educational Technology and Learning Design, Empire State University, Saratoga Springs, NY 12866, USA
2
Curriculum and Instruction, University of Minnesota-Twin Cities, Minneapolis, MN 55455, USA
*
Author to whom correspondence should be addressed.
Educ. Sci. 2023, 13(11), 1132; https://doi.org/10.3390/educsci13111132
Submission received: 7 August 2023 / Revised: 29 October 2023 / Accepted: 2 November 2023 / Published: 13 November 2023

Abstract

:
This investigation used cross-sectional survey research methods in a high-enrollment undergraduate history course, setting out to examine test performance and metacognitive strategies that subjects self-selected prior to class, during class, and during the exam. This study examined the differences in exam scores between one group of students who self-reported completing specific metacognitive strategies and one group of students who self-reported not completing them. An online survey instrument was used to collect data from 121 students about the frequency of occurrence of specific behaviors. Frequencies and an Independent Samples T-Test were used to analyze metacognitive strategies and exam performance. The results showed the following strategies were statistically significant at the 0.05 alpha level: (1) read or listened to assigned readings and audio files before they were discussed during class; (2) frequently took part in small group discussion at the table during the class session; (3) created outlines for each of the potential essay questions to prepare for the examination; and (4) made an outline of the essay question before beginning to write while taking the exam. Limitations of the study, implications of the results, and recommendations for future research are provided. With the challenges of supporting students to earn higher grades and persist toward graduation, faculty members need to join the rest of the campus to be active agents in supporting students through simple learning strategies and effective student behaviors embedded into their courses. This may require extra time and effort to engage in professional development to learn how to embed practice with metacognitive strategies during class sessions.

1. Introduction and Background

It is essential that undergraduate instructors embed academic support activities within their courses to enable students to learn effective study behaviors, practice using metacognitive strategies to monitor and modify learning activities, and practice effective examination-taking strategies before the students compete in high-stakes examinations in their classes. Referring students to tutoring programs, writing centers, and learning centers located somewhere on a college campus after the student has received a failing first major examination grade may be too late. Student retention research has identified that the most prevalent decision time for students to drop out of college is within the first 6 weeks of their first academic term [1,2,3]. The 6-week mark of the academic term correlates in many first-year classes with the first major examination [1].
Instructors can form a vital partnership with the campus learning assistance programs and other services to support student persistence toward graduation. With little hope of increased funding for existing academic support programs, the campus must add new partners to support student success. Student achievement and persistence to graduation are everyone’s responsibility. This includes faculty members, student services, academic advisement, learning assistance centers, and others at the institution [1,2,3,4,5,6,7,8,9]. The classroom instructor can help accomplish this through simple and no-cost/low-cost enhancements to the course curriculum.
This research study focused on actions that a classroom history instructor took to assist students in earning higher grades on their first major examination of the academic term. It was hoped that the students would also employ the newly acquired strategies in other courses to help increase their ability to learn more, be effective and efficient, and earn higher grades. One of the authors of this study had been involved with promoting peer assistance programs for adoption at other institutions. He had often wondered how he would conduct a first-year college course and seek to integrate the learning and metacognitive strategies into it rather than simply referring students to walk across the college campus to seek help. He received the opportunity to test his hypothesis when he was recruited for a tenure-track position at a university where he would have responsibility for teaching two introductory history courses each academic term. The total annual enrollment of students in the history course ranged from 140 to 300. The history course fulfilled several graduation requirements and attracted students from a wide range of academic majors from across the institution. More information about the history course is provided below and later in this article.
The setting for this study was a first-year global history course at a large research-intensive university with selective admissions. Even with a highly selective admissions process, university leaders and faculty members were concerned with unacceptable rates of low grades and course withdrawals. An entering student who earned a cumulative grade point mean below 2.0 on a 4.0 scale would be placed on academic probation at this institution. A consecutive academic term of a similar below 2.0 grade point mean would result in academic dismissal from the institution. Admission to academically rigorous programs such as Science, Technology, Engineering, and Mathematics (STEM) requires a cumulative student grade point mean above 3.5. Earning low grades in just a few courses could result in loss of the opportunity to pursue such academic programs.
The institution provided some tutoring and writing center services for students if they were able to navigate an unfamiliar environment to locate them. In addition to the challenge of locating the tutoring services was the students’ lack of time. Because of the high tuition at the institution, many students worked two and sometimes three part-time jobs to pay for college. Many enrolled in the history classes were first-generation college students from historically underrepresented and under-resourced backgrounds. Self-destructive and self-sabotaging behaviors resulted from the students feeling like imposters who did not deserve to be enrolled at the institution [10]. The institution prohibited offering developmental-level courses or study strategy courses, which could have better prepared students for the academic rigor of the course.
A major pedagogical approach selected by the instructor of the history course for this study was Universal Design for Learning (UDL) [11,12,13,14,15]. UDL posited that small changes could be made in the curriculum to make it more accessible for most or all students enrolled in the course. Rather than assuming new students could navigate a huge and unfamiliar college campus to locate supportive resources, the decision was made by the course instructor to make changes in the approach and the curriculum. One of the coauthors of this research study has written previously on the comprehensive changes made in the curriculum of his history course following UDL principles [16,17,18].

1.1. Literature Review

A review of the professional literature provides a context for this history course and discussion of the results from this research study. The following topics were examined for relevant research studies: student retention and persistence to graduation, Universal Design for Learning, cognition and metacognition, constructivism, and a sustainable ecosystem for student retention and graduation. Each of these topics guided curricular choices by the instructor for embedding academic support into the course.

1.1.1. Student Retention and Persistence to Graduation

The landmark researcher regarding college student departure before completing a degree was Vincent Tinto [1,2]. It was his research that identified that the most likely time for a student to withdraw from college was at the end of 6 weeks of their first term. Tinto’s research identified six themes leading to early withdrawal from college: (a) difficulty with the adjustment from secondary school to college; (b) difficulty with their new day-to-day schedule that was less structured than secondary school; (c) difficulty making connections between what they understand and the new academic content presented by the instructor, which resulted in cognitive incongruence; (d) heightened sense of social isolation in comparison with their previous lives in secondary school; (e) high financial need due to living expenses and college tuition, books, and fees; and (f) social groups in their previous communities that encouraged dropping out of college and returning back [1]. Failure to deeply integrate into the new learning environment made it easier for the student to drop out.
Four of these themes help explain academic behaviors and outcomes for new college students. The first theme of having difficulty adjusting to the academic rigor of college can result from a lack of successful learning strategies needed for more challenging course material. The second theme of having daily difficulty can manifest with the quick pace of many college courses with reading assignments, daily quizzes, and voluntary class attendance. Experiencing incongruence can become commonplace if the student is unable to independently make connections between what they already know with the new course content of class lectures, reading assignments, which may not be discussed by the course instructor, and other class experiences. The final theme of social isolation can be experienced if the student fails to seek out other students in the course to study the course material and prepare for the examination. As described earlier, all of these themes can lead to developing a sense of imposter syndrome, which can lead students to falsely assume that they do not belong in college, even for those who were successful in secondary school [10].
In addition to the decision to drop out of the institution at this critical period in the academic term is dropping the course and switching academic majors to a less academically demanding one [19]. This has been especially troubling with students initially committed to a Science, Technology, Engineering, and Mathematics (STEM) major persisting at the institution but transferring to another subject area. The diminishing number of STEM students in the pipeline for graduation is troubling. Particularly at risk are women and students of color.
Tinto’s focus on frontloading academic support for new students into the first 6 weeks of the course led the instructor to encourage student metacognitive behaviors before the first exam. Decades of experience of the course instructor had confirmed this need since attempting to remediate students who failed the first exam often results in course withdrawals. Understanding the need for changes early in the academic term led the history instructor to pursue new pedagogies. As indicated earlier, UDL was an important model to emulate in the course [11,12,13,14,15].

1.1.2. Universal Design for Learning

Universal Design for Learning (UDL) is an educational framework based on research in the learning sciences, including cognitive neuroscience that guides the development of flexible learning environments to accommodate individual learning differences [11,12,13,14,15]. Recognizing that the way individuals learn can be unique, the UDL framework creates a curriculum that provides multiple means of representation to give learners various ways of acquiring information and knowledge, multiple means of expression to provide learners alternatives for demonstrating what they know, and multiple means of engagement to tap into learners’ interests, challenge them appropriately, and motivate them to learn [11,13,14]. Several other researchers have updated UDL to make the learning process more inclusive for students of color [12,15].
UDL was a helpful guide for the inclusion of new cognitive learning activities into the course. The next step was the inclusion of metacognitive learning activities where students would monitor their academic behaviors and select appropriate learning activities depending upon the nature of the academic task. That is the next topic in this review of the professional literature.

1.1.3. Cognitive Strategies and Metacognitive Strategies

Cognitive strategies are actions and mental activities to gain new information [20]. Examples include memorization of facts, reflecting on previous class lectures, repeating the problem-solving process, self-monitoring their learning strategies, and summarizing an assigned reading selection [21]. Studies indicate that students who applied strategies for learning had a better performance than students who did not apply them to the same degree [22,23]. Metacognitive strategies reflect upon various cognitive strategies and strategically select the ones based on the learning situation most likely to yield the highest mastery [24,25]. A common term to concisely describe this is ‘thinking about thinking’ [26]. Employing metacognitive learning practices has been associated with higher academic performance from elementary through postsecondary education [9]. Examples of these metacognitive actions include (a) self-awareness of strengths and weaknesses as a learner; (b) planning how to approach a learning task; (c) selecting a particular learning activity or strategy for a specific learning task; and (d) self-correcting in response to a self-assessment of progress towards completing a learning task successfully. Another term sometimes used for metacognition is mindfulness [27] when the student reflects on the learning task and carefully selects the most appropriate learning strategy. Careful attention is continuously exercised with the learning task to make necessary changes [28]. An additional term that is often synonymous with metacognition is self-regulated learning [29].
Several research studies reported using “exam wrappers” to increase students’ metacognitive awareness of their examination skills [30,31,32,33,34]. A highly cited reference to this approach described its use in multiple academic disciplines [33]. The authors identified several keys for effective use of this exam reflection approach: embedding into the course material and the associated exam, being explicit about the usefulness of the activity, and encouraging the students to take time for this small amount of time, which can yield high results with increased exam scores. This approach applied to a nursing course by providing activities that assisted students in reviewing their performance on a nursing exam with feedback concerning their choices and gave them time to reflect on how to make more effective exam preparations for the next time [34]. Other researchers extended the use of these exam reflections in computer science [30], psychology [31], and Food Science and Human Nutrition [32].
Some institutions have approached the issue of developing metacognitive learning strategies through activities that occurred outside the classroom. Many campus learning centers offer study skill workshops or academic courses on applied metacognition. A frequently cited researcher in this area is McGuire [35]. Her books have been used to inspire such activities. Another approach to an outside-of-course approach is Supplemental Instruction (SI) [36]. The early report of the SI model shares how a study group is attached to a course with a historically high rate of D, F, W, F, and I grades. Most often, this is a voluntary participation study group for students in a course. These outside-of-classroom activities do not require time or planning by the course instructor.
Thus far in this literature review, the focus has been on acquisition and practice with learning strategies and metacognitive strategies. The next section is focused on the use of active learning activities within the classroom. A classical learning theory that guides some of these activities is constructivism.

1.1.4. Constructivism

The Constructivist Theory was groundbreaking in understanding how children and young adults learn. Jean Piaget formulated it as a leader of developmental psychology [37,38,39]. His theory emphasized the learner’s active role in constructing knowledge through interactions with their environment. According to Piaget, children and young people are not passive recipients of information but active agents who construct their understanding of the world through a series of states. Several of Piaget’s theory concepts are highly relevant to this research study. First, schemas are the basic building blocks of knowledge. They are mental structures used by young people to organize and interpret their experiences. Schemas evolve and become more complex as individuals interact with their environment. This interaction with the environment includes two complementary processes: assimilation and accommodation. Assimilation is when new information is incorporated into existing schemas, while accommodation modifies existing schemas to fit newly gained information [37,38,39].
A critique of Piaget’s theory has been its application to college-aged students as it has primarily been applied to younger children. Another criticism was his research was based upon Western, middle-class children, limiting its generalizability to other cultures and socioeconomic backgrounds. Other researchers have since validated the theory with field tests with other young people in other cultures [38].
This theory inspired a number of classroom activities by the students. These included a student-constructed examination preparation website, a course content review podcast, history event videos, guided classroom discussions, history simulations, and more. The theory-guided activities meant that students must engage independently and in small groups to create knowledge about new history topics and display skills in demonstration of mastery.
The final topic to explore in this brief review of the professional literature is an emerging one of colleges developing a multi-faceted support system for all students to achieve high mastery of challenging content material as evidenced with their grades and to complete their academic program of study.

1.1.5. Improved Student Retention Involves the Entire Campus

It is essential for the entire college to be part of the support system for students to achieve higher academic performance and persistence to graduation. The literature supports that a sustainable learning environment requires an ecosystem that supports students via instructors; student services such as advising, counseling, new student orientation, and health services; and other parts of the campus. For example, not surprisingly, Tinto’s research has led the way with this comprehensive approach [1,2]. Astin’s landmark study on factors that influenced student outcomes identified more than 100 variables that can influence the students [40]. A short list of researchers who have identified evidence-based strategies to support student development include Burke [41], Habley et al. [42], and Braxton et al. [43]. Faculty members are not alone as being essential for the ecosystem supporting student success. Unsurprisingly, a complex and sustainable learning environment is required to support student development through their stages of life. One foundational approach to supporting student success is through educators’ practice of scaffolding learners’ use of metacognitive strategies in their courses.
This study investigated learners’ use of metacognitive strategies and their exam performance. The primary aim of this study was to investigate the difference, if any, in mean scores on the first major exam between students who reported completing specific metacognitive strategies and those who reported not completing them. The following research question guides this study: what is the difference, if any, in mean scores on the first major exam between students who applied metacognitive strategies and those who did not?

2. Materials and Methods

A cross-sectional survey research design was selected to investigate the differences, if any, in exam performance between two groups of students and their use of metacognitive strategies [32,33,34]. Data were gathered to systematically compare the exam performance of two groups of students and to gain insight into the metacognitive strategies of these undergraduate students. We gathered the data during the fall semester of 2012. The participants were asked to complete an online questionnaire after completing the first major exam of the academic term. The survey instrument was used previously in the class for the preceding two academic years to find out its usefulness for students to reflect upon their academic behaviors. Students provided feedback to suggest refinements to the survey instrument.

2.1. Setting

The Global History and Culture course enrolls undergraduate students at a four-year university in the midwestern United States. The four-credit course provided an overview of significant historical events around the globe since the Industrial Revolution in western Europe and the United States. The course fulfilled graduation requirements for the areas of historical perspectives and cultural diversity. The instructor intentionally structured the course for students to engage in various active learning strategies, employ learning technologies, engage in online course enrichment activities, and participate in exam preparation activities before taking the first major exam of the academic term. These course enrichment activities were inspired by the best practices of Universal Design for Learning [11,12,13,14,15] and other educational practices.
These diverse learning activities are divided into the following categories that occurred during or outside of class:
  • Active learning strategies during the class included small group discussion sessions that lasted between 2 and 5 min.
  • Used study guides provided by the instructor for assigned readings or audio narrations of text material before they were discussed during a class session. A print translation of the audio was provided.
  • The instructor spent a few minutes during class helping students to practice with several examination preparation activities: (a) predicting potential multiple-choice questions based on the assigned readings, lectures, and other class activities; (b) creating detailed outlines of the potential exam essay questions; (c) writing responses to the potential exam essay questions; (d) marking up the potential multiple-choice exam questions by circling key terms to increase understanding; (e) constructing a brief outline before writing for a potential essay question; and (f) looking back to the potential exam vocabulary terms for inclusion of them as appropriate into the essay response.
  • PowerPoint slides were provided before class sessions to students in a variety of formats so that additional notes could be added to them.
  • Audio podcasts constructed and voiced by students provided a course unit summary, suggestions for exam study strategies, and an explanation by the course instructor of nuances of the potential essay exam questions.
  • An exam study guide constructed by students provided several summaries of the course topic, definitions for key vocabulary terms previously identified by the course instructor, and several outlines of potential essay questions. The course instructor monitored the contributions for accuracy. The instructor provided a dozen potential essay questions covering all major topics for the exam. During the exam, the instructor selected two of the dozen essay questions for the students to complete.
  • The course instructor hosted an online exam preparation session. He discussed key vocabulary terms and nuances of understanding what was required of the essay questions without answering them, and answered questions from students. The online session was recorded and made available for students to listen if they were unable to participate during the live recording. Some students reported they met in small groups to view the sessions.
  • Students could elect to review recorded class lectures online to more deeply understand challenging course material and for a missed class session.
  • Students could form study groups that met outside of class sessions.
  • Students could seek assistance from the free campus tutoring or writing center.

2.2. Participants

This study surveyed students in an undergraduate history course offered at a four-year, public, research university in the midwestern United States during the fall academic term of 2012. Of the 148 total students in the course, 121 students completed the online survey for this research study with an 82% response rate. Table 1 presents a profile of the sample based on participant demographics. Based on the population demographics of undergraduate students system-wide at the university during the fall of 2012, the sample of students taken from the history course appears to represent the general population.

2.3. Survey Instrument

The online survey instrument consisted of 7 close-ended questions and 1 open-ended question (see Appendix A). The survey took approximately 10 min to complete. In the first section of the questionnaire, items focused on activities that students may or may not have completed before the exam such as exam preparation resources, assigned readings, and audio files; class session activities; and examination preparation activities, and the approximate number of hours spent preparing for the exam and the number of class periods missed before the exam. The second section of the questionnaire focused on activities that students may or may not have completed during the exam. The third section of the survey questionnaire focused on plans for potential changes with the next exam. Data collected from this third section will be the subject for a separate research publication. This survey instrument is similar to “exam wrappers”, which were used for research studies at other postsecondary institutions [30,31,32,33,34].

2.4. Data Analysis

To answer the research question concerning the differences in mean major exam scores for metacognitive strategies, the researchers coded the survey responses to determine the frequencies of strategies and conducted the independent samples t-test. Demographic data were obtained from the institution’s student information database. The data were entered into SPSS along with student test scores on the first major exam.

3. Results

The results of data analyses on the primary research question are reported using frequencies and two-sample T-tests. To examine the metacognitive strategies and the first major exam performance, researchers asked survey questions about students’ self-reported activities prior to and during the exam. Frequencies and the results of the two-sample t-test are reported.

3.1. Descriptive Statistics

The descriptive statistics provide a view of the data that informs the characteristics of the participants through the frequencies of which they reported preparing for the first major examination of the academic term. The results of the questions about behaviors related to examination preparation include the use of resources, completion of assigned readings and audio files, student activities during the class sessions, examination preparation activities, hours spent on preparing for the exam, and the number of class periods missed prior to the exam. The frequencies of examination preparation resources are reported in Table 2. Nearly all students (f = 114, 94%) who completed the survey used the online exam study guide website constructed by students in the course. Approximately half of the students listened to one or more of the previous class lectures (f = 65, 54%), listened to the online exam review session by the instructor (f = 63, 52%), and listened to the unit course podcast one or more times (f = 61, 50%). Fewer participants reported to have studied with other students in the class just before the exam (f = 14, 12%), studied with other students in the class weekly (f = 4, 3%), and going to the campus tutoring or writing center (f = 3, 2%).
Based on the frequency chart (Table 2), students overwhelmingly favored using the online exam study guide website as a resource for preparing for the examination. Students preferred working individually to prepare for the examination given that listening again to previous class lectures, listening to the online exam review session by the instructor, and listening to the podcast were more frequently reported than studying with other students or seeking help through tutoring or the writing center.
Table 3 presents the frequencies for the completion of assigned readings and audio files, which were part of their ungraded homework. Only one-third of the students surveyed (f = 40, 33%) reread assigned material or listened to the audio files again before the exam. However, over half of the students read or listened to them before class (f = 80, 66%), read or listened to them at least once before the exam (f = 67, 55%), and took written notes about the assigned readings and audio files (f = 62, 51%).
When thinking about student behaviors related to the assigned readings and audio files, the frequency chart (Table 3 shows that most students read or listened to the material before class. The next most frequent behavior reported by students was reading or listening to the material before the exam. Nearly the same number of students also took written notes. A much smaller number of students reread or listened to the material again before the exam.
Table 4 presents the frequencies of class session activities. Most students participated in all five of the in-class activities. Most frequently reported, during class, students wrote additional comments about the material presented in the lecture (f = 103, 85%). Many students printed or downloaded the lecture slides (f = 95, 79%), participated in small group discussions (f = 91, 75%), wrote supplemental lecture notes (f = 87, 72%), and reread the lecture notes before the exam (f = 83, 69%).
Table 5 presents the frequencies for the examination preparation activities. Just over half of the students surveyed created outlines for each of the potential essay questions (f = 63, 52%). Fewer students practiced writing answers to the essay questions (f = 52, 43%) or tried to predict multiple-choice questions based on the lecture notes and assigned readings (f = 49, 40%).
The frequencies of hours spent preparing for the exam are shown in Table 6. Most students spent 4 to 6 h preparing for the exam (f = 41, 34%). An almost equal number of students spent 1 to 3 h (f = 23, 19%), 7 to 9 h (f = 27, 22%), or over 10 h (f = 28, 23%). Only one student reported to have spent less than 1 h preparing for the exam (f = 1, 1%).
The frequencies of class periods missed for any reason before the exam are displayed in Table 7. Overwhelmingly, students reported not missing any class periods before the exam (f = 71, 59%). If students missed a class period before the exam, then most frequently it was only one class (f = 24, 20%) or two classes (f = 14, 12%). Fewer students reported missing three or more classes (f = 12, 10%).
Table 8 shows the frequency of student activities during the exam. Most frequently, students reported that they looked back at the vocabulary and multiple-choice questions when answering the essay questions (f = 82, 68%). Many students made a brief outline before writing (f = 79, 65%). Some students also marked up the exam questions by underlining or circling keywords to make it easier to understand (f = 68, 56%).

Data Conditions

Prior to conducting independent samples T-tests on the data, four data conditions were examined. While participants in the sample were not randomly selected, the demographics in Table 1 show that the sample is reasonably representative of the population demographics. Given that the sample size is sufficiently large (N > 30), the Central Limit Theorem applies to the sampling distribution and researchers can assume a normal distribution. The samples are independent of one another given that students were categorized as either having completed or not completed an activity. When comparing the two groups, the standard deviations were examined to determine the similarity of variances. From this information, researchers were able to determine that data conditions were met prior to running independent samples t-tests in SPSS.

3.2. Independent Samples t-Test

A two-sample t-test was conducted to examine the differences in mean scores on the first major exam between one group of students who completed metacognitive strategies and one group who did not. Our hypothesis was that the population mean score of students who completed the activities (μ1) would be greater than the population mean score of students who did not complete the strategy (μ2). Thus, the alternative hypothesis was Ha: μ1 − μ2 > 0, (α = 0.05). Of the 22 behaviors measured on the survey, five were found to be statistically and practically significant. See Table 9 for a summary of the results of the independent samples t-test.
Results indicated that there was a statistically significant difference in mean scores on the first exam between the group of students who read or listened to assigned readings and audio files before they were discussed during class and the group of students who did not (Item 4a), t(118) = 1.856, p = 0.033, d = 0.35. The results also indicated that there was a statistically significant difference in the mean exam scores between the group of students who frequently participated in small group discussions at the table during class session activities and the group who did not (Item 5d), t(118) = 2.287, p = 0.012, d = 0.47. Item 6a was found to have the highest statistical and practical significance. The mean exam scores for the group of students who created outlines for each of the potential essay questions while preparing for the exam were higher than the group who did not, t(118) = 3.940, p < 0.001, d = 0.73. There was a statistically significant difference in the mean exam scores between the group of students who made a brief outline of the essay question before beginning to write while taking the exam and the group who did not, t(118) = 1.685, p = 0.0475, d = 0.33. Finally, Item 4b. Read or listened to assigned readings and audio files at least once before the exam narrowly missed statistical significance at the 0.05 level, t(118) = 1.635, p = 0.0525, d = 0.30.

4. Discussion

This study investigated metacognitive strategies and exam performance. Four behaviors emerged as statistically significant, or in the case of one behavior, nearly statistically significant: (a) read or listened to the assigned readings and audio files before they were discussed during the class session; (b) frequently were actively engaged in brief small group discussions at designated times during class sessions; (c) created outlines for each potential essay question before the examination; and (d) made an outline of their written response to the examination essay question before they began to write©.
These behaviors by the group of students who earned higher grades on the first major examination in the course have been previously identified as best practices for examination preparation in the professional literature [33,34]. Multiple exposures to academic content are linked with higher retention and understanding rates [8,33]. Active learning in the classroom is frequently cited by UDL as a key component of increased engagement and learning [11,12,13,14,15]. Constructing outlines of potential essay questions requires students to construct their potential responses before the examination and enhance their deeper understanding of the subject material [28,29,30,31,32]. Creating outlines of potential responses to an essay question during the exam requires the deployment of metacognitive thinking before beginning the writing activity [33].
It was surprising that some of the frequently chosen behaviors by students were not statistically significant: (a) listened to the course exam review podcast, (b) listened to a recording of a class session an additional time or after they had missed class, and (c) participated in an online exam review session with the course instructor or listened to a recording of that event before the exam. Anecdotal evidence from some students in the class reported they were satisfied that the class sessions were recorded for them if they missed class. This was especially noted by student–athletes who missed class sessions for scheduled athletic events.
Few students reported they studied with other classmates outside of class sessions or used the free campus tutor program or the writing center. It is speculated the low rate of participation in these activities is due to the complicated and intense work schedules of students who were enrolled full-time, worked one or more part-time jobs, and often commuted to the college campus. Also, it may be due to some students preferring to study independently rather than working in collaborative study groups outside of class [14]. An additional research study will report data gathered from the same students through the instrument (Exhibit A) to study the potential impact on the final course grade. Perhaps additional exposure to the exam preparation activities will have a higher influence on the final course grade.
Speculation about the lack of statistical significance of listening to the podcast, viewing recorded class sessions, and participating in the online instructor-led exam review session could be the lack of significant student interaction while listening or viewing the online event. The course instructor needs to reflect on how to increase the relevance of the podcast and increase student interaction during the online review session. As stated above, another research study in this history course will repeat this research design by examining the final course grade. Perhaps the results will differ.
While students cannot change their past academic performance, they have free will to choose academic behaviors during college. In this research study, five behaviors were identified that were statistically significant. It required action by the course instructor and the students for those behaviors to occur or be enhanced.

4.1. Implications

The results from this study confirm previous research on the influence of high-yield, low-effort strategies to improve student examination scores [33,34]. Students can work individually and in groups to prepare for major course examinations. With guidance from the instructor regarding constructing their podcasts and the exam review website, they can use their own words to make sense of challenging course material. Student engagement is increased and is more easily integrated into their thinking through the voices of other students.
The post-examination survey (Exhibit A) was reported by students in the history course as highly satisfactory for providing recommendations for study strategies by the students who earned the highest marks. The course instructor categorized the responses from the students between those who earned an A or B grade and those who earned a C or below. The instructor provided a summary to the students within a week of the exam through a class handout and posting it to the course website. Rather than the instructor exhorting students to adopt particular behaviors, students compared their behavior to what the students who earned A or B grades did. Students informed themselves with better study behaviors.
Instructors can make examination preparation easier by being more transparent with students regarding expectations and nuances of language. Both identifying key vocabulary terms and providing concise definitions of them can help students to comprehend and remember them more efficiently. Rather than making a mystery of potential major exam essay questions, they could be identified in advance. Without providing the answer, the instructor could identify all the key parts of the question and clarify the expectations for a comprehensive response. Sometimes the error by students on examinations is their imprecise reading of the question rather than incomplete prior knowledge needed for higher marks regarding the multiple-choice, vocabulary, and essay questions. The instructor could also use a detailed rubric for grading them, which is provided to the students afterwards. For the instructor of this study, complaints from students regarding the grading of essay questions dropped to nearly zero after implementing the detailed grading rubric.
Recent learning technology advancements make it easier to implement many of the activities used by the instructor in this research study. Course Management Systems (CMSs) often provide many tools for students to work collaboratively in producing exam preparation resources. Google Workspace provides similar tools for creating YouTube videos, collaboratively constructed webpages, and more. One-button creation of podcasts is free and easy through Spotify. Finally, providing some more time during class for more small group discussions can help students to put into their own words what is being learned and increase their comprehension of challenging academic content.

4.2. Recommendations for Further Research

Replication of research studies is a recommended action with any previous research study. The current approach by the course instructor with embedded metacognitive learning activities, an active learning classroom, recommended examination strategies, and student-created exam preparation resources such as the exam website and the podcast could be repeated. The new studies could occur in other academic disciplines at the lower-division or upper-division undergraduate or graduate level. The studies could occur at various institutional sizes, types, and student demographics.
Different approaches could be taken to prepare students for success on course assessments. Alternative means for assessing student decisions could be through individual or small group interviews to understand student decision-making processes for examination preparation more deeply. Effective faculty development programs could be created and assessed regarding the effectiveness of curricular transformation and improved achievement and persistence by students, especially those from disadvantaged and historically underrepresented populations.

4.3. Limitations

There are several limitations for this research study. The following are some of them. This study was conducted at one institution in one history course taught by one instructor. It is possible a wider range of academic subjects and a wider range of instructors could have different results. Second, all students were in the lower division of the undergraduate curriculum. It is possible that a different experience would result with upper-division undergraduate or graduate students. The third is that the study depended upon perceptual recall, which reflected their interpretations, judgments, and potential bias in students self-reporting their study behaviors. By its nature, this research is subject to the limitations of the self-reported data of the survey. We did not conduct individual or small group interviews with the students. These limitations can guide replication research studies to confirm these results and probe more deeply into student intentions and actions regarding metacognitive strategies.

5. Conclusions

Students can engage in metacognitive learning strategies to earn higher grades, which can contribute to higher course completion and persistence toward graduation. Rather than expecting students to navigate an unfamiliar campus environment to identify resources to help them, instructors can scaffold their metacognitive skill development within their classrooms with minimal effort. This study explored some pedagogical choices based on current theoretical models and previously published research studies. With limited institutional funding for more learning centers and tutors and a widespread decision to prohibit developmental-level courses, the classroom presents an untapped learning environment for students to obtain the help they need to meet rigorous expectations for academic performance. However, this choice requires a willingness by instructors to invest a minimal amount of time in learning about these activities and then providing space within the curriculum for their cultivation. This will require time and effort by instructors to engage in professional development regarding how to scaffold metacognitive strategies within their courses. This small investment can bloom higher academic achievement, which is a goal of students, instructors, and the institution.

Author Contributions

Conceptualization, J.V.K. and D.R.A.; methodology, J.V.K.; software, J.V.K.; validation, J.V.K. and D.R.A.; formal analysis, J.V.K.; investigation, J.V.K. and D.R.A.; resources, D.R.A.; data curation, D.R.A.; writing—original draft preparation, J.V.K. and D.R.A.; writing—review and editing, J.V.K. and D.R.A.; visualization, J.V.K.; supervision, J.V.K.; and project administration, D.R.A. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

This research study was exempt from the review process. The study was conducted in accordance with the Declaration of Helsinki and approved by the Institutional Review Board of the University of Minnesota-Twin Cities (protocol code 1508E77086, 28 January 2019).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Data are unavailable due to privacy.

Acknowledgments

Technical help was provided by Jennifer Franko with accessing student data from the university student database.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Survey Instrument
  • Please type your name (Last Name, First Name)
  • Please type your UofM Student Number
Section One: The following focus on activities you did or did not do BEFORE the exam.
3.
Exam Preparation Resources (Checkmark each activity that you did from the following)
  • Listened to the unit course podcast one or more times.
  • Studied with other students in the class weekly.
  • Studied with other students in the class just before the exam.
  • Studied the online exam study guide website constructed by students.
  • Listened to one or more of the previous class lectures.
  • Listened to the online exam review session by the instructor.
  • Went to the campus tutoring or writing center.
  • Other resources I used (Please type response inside the box).
4.
Assigned readings and audio files (Checkmark each activity you did from the following)
  • Read or listened to them before they were discussed during class.
  • Read or listened to them at least once before the exam.
  • Reread or listened to them again before the exam.
  • Took written notes about the assigned readings and audio files.
5.
Class session activities (Checkmark each activity that you did from the following)
  • Printed or downloaded to your computer the lecture PowerPoint slides.
  • During class, wrote additional comments about material presented in the lecture.
  • During discussion time, wrote notes about the video clips and assigned readings.
  • Frequently participated in the small group discussion at the table.
  • Reread the lecture notes before the exam.
6.
Examination preparation activities (Checkmark each activity that you did from the following)
  • Created outlines for each of the potential essay questions.
  • Tried to predict potential exam multiple-choice questions based on lecture notes and the assigned readings.
  • Practiced writing answers to several of the potential essay questions.
  • Any other things you did to prepare for the exam? (Please type your response inside the box.)
7.
Approximate number of hours spent preparing for the exam.
(Select the one choice that represents how many total hours you studied for the exam.)
  • Less than 1
  • 1 to 3
  • 4 to 6
  • 7 to 9
  • 10 to 14
  • More than 15
8.
Number of class periods missed for any reason before the exam.
  • Zero
  • One
  • Two
  • Three
  • Four
  • Five or more
Section Two: These are activities that you did or did not do DURING the exam.
9.
Activities while taking the exam (Checkmark each activity you did from the following)
  • Made a brief outline of the essay question before beginning to write.
  • Marked up the exam questions by underlining or circling keywords to make them easier to understand.
  • While answering the essay questions, you looked back at the vocabulary and multi-choice questions to locate information to incorporate into the essay.
Section Three: Plans for potential changes with the NEXT exam.
10.
If you plan to make changes preparing for the next exam, please list one or more of them. (Please type your response inside the box.)

References

  1. Tinto, V. Leaving College: Rethinking the Causes and Cures of Student Attrition, 2nd ed.; The University of Chicago Press: Chicago, IL, USA, 1993. [Google Scholar]
  2. Tinto, V. Completing College: Rethinking Institutional Action; The University of Chicago Press: Chicago, IL, USA, 2012. [Google Scholar]
  3. McKinney, L.; Novak, H.; Hagedorn, L.S. Giving up on a course: An analysis of course dropping behaviors among community college students. Res. High. Educ. 2019, 60, 184–202. [Google Scholar] [CrossRef]
  4. Banks, T.; Doby, J. Mitigating barriers to persistence: A review of efforts to improve retention and graduation rates for students of color in higher education. High. Educ. Stud. 2019, 9, 118–131. [Google Scholar] [CrossRef]
  5. Winkelmes, M.-A.; Boye, A.; Tapp, S. (Eds.) Transparent Design in Higher Education Teaching and Leadership; Routledge: London, UK, 2019. [Google Scholar]
  6. Zamora-Polo, F.; Sanchez-Martin, J. Teaching for a better world: Sustainability and sustainable developmental goals in the construction of a change-maker university. Sustainability 2019, 11, 4224. [Google Scholar] [CrossRef]
  7. Ellis, R.A.; Goodyear, P. The Education Ecology of Universities: Integrating Learning, Strategy, and the Academy; Routledge: London, UK, 2019. [Google Scholar]
  8. Conefrey, T. Supporting first-generation students’ adjustment to college with high-impact practices. J. Coll. Stud. Retent. Res. Theory Pract. 2021, 23, 139–160. [Google Scholar] [CrossRef]
  9. Tight, M. Student retention and engagement in higher education. J. Furth. High. Educ. 2019, 44, 689–704. [Google Scholar] [CrossRef]
  10. Canning, E.A.; LaCosse, J.; Kroeper, K.M.; Murphy, M.C. Feeling like an imposter: The effect of perceived classroom competition on the daily psychological experiences of first-generation college students. Soc. Psychol. Personal. Sci. 2020, 11, 647–657. [Google Scholar] [CrossRef]
  11. Burgstabler, S.E. (Ed.) Universal Design in Higher Education: From Principles to Practice, 2nd ed.; Harvard Education Press: Cambridge, MA, USA, 2015; Available online: https://www.washington.edu/doit/universal-design-higher-education-principles-practice-1 (accessed on 1 July 2023).
  12. Fritzgerald, A. Antiracism and Universal Design for Learning: Building Expressways to Success; CAST Publishing: Lynnfield, MA, USA, 2020. [Google Scholar]
  13. Gordon, D.; Meyer, A.; Rose, D.H. Universal Design for Learning: Theory and Practice; CAST Professional Publishing: Lynnfield, MA, USA, 2014. [Google Scholar]
  14. Ross, D.H.; Meyer, A. Teaching Every Student in the Digital Age: Universal Design for Learning; Association for Supervision and Curriculum Development: Alexandria, VA, USA, 2002. [Google Scholar]
  15. Tobin, T.J.; Behling, K.T. Reach Everyone, Teach Everyone: Universal Design for Learning in Higher Education; West Virginia University Press: Morgantown, WV, USA, 2018. [Google Scholar]
  16. Arendale, D.R. Preliminary results of pilot study concerning use of emerging technologies by students within an introductory history course. Centerpoints Newsl. 2007, 4. Available online: https://hdl.handle.net/11299/200384 (accessed on 1 July 2023).
  17. Arendale, D.R. Podcasting: Learning on-the-go. In Cultivating Change in the Academy: 50+ Stories from the Digital Frontiers at the University of Minnesota in 2012; Duin, A.H., Nater, E.A., Anchesaria, F.X., Eds.; University of Minnesota: Minneapolis, MN, USA, 2012. [Google Scholar]
  18. Arendale, D.R.; Ghere, D. Teaching history using Universal Instructional Design. In Pedagogy and Student Services for Institutional Transformation: Implementing Universal Design in Higher Education; Higbee, J.L., Goff, E., Eds.; University of Minnesota, College of Education and Human Development: Minneapolis, MN, USA, 2008; pp. 113–130. Available online: https://hdl.handle.net/11299/200460 (accessed on 6 August 2023).
  19. Seymour, E.; Hewitt, N.M. Talking about Leaving: Why Undergraduates Leave the Sciences; Westview Press: Boulder, CO, USA, 1997. [Google Scholar]
  20. Dunlosky, J.; Rawson, K.A.; Marsh, E.J.; Nathan, M.J.; Williangham, D.T. Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychol. Sci. Public Interest 2013, 14, 4–58. [Google Scholar] [CrossRef]
  21. Pintrich, P.R.; Brown, D.R.; Weinstein, C.E. (Eds.) Student Motivation, Cognition, and Learning: Essays in Honor of Wilbert J. McKeachie; Lawrence Erlbaum Associates: Mahwah. NJ, USA, 1994. [Google Scholar]
  22. Nussbaumer, A.; Dahn, I.; Kroop, S.; Mikroyannidis, A.; Albert, D. Supporting Self-Regulated Learning. Responsive Open Learning Environments: Outcomes of Research from the Role Project; Springer: Berlin/Heidelberg, Germany, 2015; pp. 17–48. [Google Scholar] [CrossRef]
  23. Pintrich, P.R.; De Groot, E.V. Motivational and self-regulated learning components of classroom academic performance. J. Educ. Psychol. 1990, 82, 33–40. [Google Scholar] [CrossRef]
  24. Dennis, J.L.; Somerville, M.P. Supporting thinking about thinking: Examining the metacognition theory-practice gap in higher education. High. Educ. 2023, 86, 99–117. [Google Scholar] [CrossRef]
  25. Schraw, G.; Moshman, D. Metacognitive theories. Educ. Psychol. Rev. 1995, 7, 351–371. [Google Scholar] [CrossRef]
  26. Dinsmore, D.L.; Alexander, P.A.; Loughlin, S.M. Focusing the conceptual lens on metacognition, self-regulation, and self-regulated learning. Educ. Psychol. Rev. 2008, 20, 391–409. [Google Scholar] [CrossRef]
  27. Kaplan, M.; Silver, N.; Lavaque-Manty, D.; Meizlish, D. (Eds.) Using Reflection and Metacognition to Improve Student Learning: Across the Disciplines, Across the Academy; Taylor & Francis: Abingdon, UK, 2013. [Google Scholar]
  28. Shapiro, S.L.; Carlson, L.E.; Astin, J.A.; Freedman, B. Mechanisms of mindfulness. J. Clin. Psychol. 2005, 62, 373–385. [Google Scholar] [CrossRef] [PubMed]
  29. Boekaerts, M.; Pintrich, P.R.; Zeidner, M. (Eds.) Handbook of Self-Regulation; Academic Press: Cambridge, MA, USA, 2000. [Google Scholar]
  30. Craig, M.; Heap, D.; Horton, D.; Zingaro, D. Introducing and evaluating exam wrappers in CS2. In Proceedings of the SIGCSE Conference, Memphis, TN, USA, 2–5 March 2016; Association for Computing Machinery: New York, NY, USA, 2016; pp. 285–290. Available online: https://dl.acm.org/doi/pdf/10.1145/2839509.2844561 (accessed on 6 August 2023).
  31. Edlund, J.E. Exam wrappers in psychology. Teach. Psychol. 2020, 47, 156–161. [Google Scholar] [CrossRef]
  32. Gezer-Templeton, G.G.; Mayhews, E.J.; Korte, D.S.; Schmidt, S. Use of exam wrappers to enhance students’ metacognitive skills in a large introductory food science and human nutrition course. J. Food Sci. Educ. 2017, 16, 28–36. Available online: https://ift.onlinelibrary.wiley.com/doi/pdfdirect/10.1111/1541-4329.12103 (accessed on 6 August 2023). [CrossRef]
  33. Lovett, M.C. Make exams worth more than the grade. In Using Reflection and Metacognition to Improve Student Learning: Across the Disciplines, Across the Academy; Kaplan, M., Silver, N., Lavaque-Manty, D., Melzlish, D., Eds.; Taylor & Francis: Abingdon, UK, 2013; pp. 18–32. [Google Scholar]
  34. Sethares, K.A.; Asselin, M.E. Use of exam wrapper metacognitive strategy to promote student self-assessment of learning: An integrative review. Nurse Educ. 2022, 47, 37–41. [Google Scholar] [CrossRef] [PubMed]
  35. McGuire, S.Y. Teach Students How to Learn: Strategies You Can Incorporate into Any Course to Improve Student Metacognition, Study Skills, and Motivation; Taylor & Francis: Abingdon, UK, 2015. [Google Scholar]
  36. Friedlandeer, J. Delivering academic assistance: Exemplary approaches. J. Dev. Remedial Educ. 1984, 7, 13–16. [Google Scholar]
  37. Flavell, J.H. The Developmental Psychology of Jean Piaget; Van Nostrand Reinhold: Washington, DC, USA, 1963. [Google Scholar]
  38. Lourenço, O.; Machado, A. In defense of Piaget’s theory: A reply to 10 common criticisms. Psychol. Rev. 1996, 103, 143–164. [Google Scholar] [CrossRef]
  39. Piaget, J. Genetic Epistemology; Columbia University Press: New York, NY, USA, 1970. [Google Scholar]
  40. Astin, A. What Matters in College? Four Critical Years Reconsidered; Jossey-Bass: Hoboken, NJ, USA, 1997. [Google Scholar]
  41. Burke, A. Student retention models in higher education: A literature review. Coll. Univ. 2019, 94, 12–21. [Google Scholar]
  42. Habley, W.R.; Bloom, J.L.; Robbins, S. (Eds.) Increasing Persistence: Research-Based Strategies for College Student Success; Jossey-Bass: Hoboken, NJ, USA, 2012. [Google Scholar]
  43. Braxton, J.; Doyle, W.R.; Hartley, H.V., III; Hirschy, A.S.; Jones, W.A.; McLendon, M.K. Rethinking College Student Retention; Jossey-Bass: Hoboken, NJ, USA, 2014. [Google Scholar]
Table 1. Sample and population demographics of undergraduates in Fall of 2012.
Table 1. Sample and population demographics of undergraduates in Fall of 2012.
SamplePopulation
Age
<192621%722416.5%
19–205445%16,53037.7%
21–243327%15,73035.9%
25–3465%31947.3%
35+22%11172.6%
Gender
Female6049.6%21,79849.8%
Male6150.4%21,89850%
Ethnicity
American Indian22%7991.8%
Asian and Int’l3731%641314.6%
Black1512%17754.1%
Hawaiian 00%1180.3%
Hispanic65%10352.4%
White6150%33,23875.9%
Table 2. Frequency of use of the examination preparation resources.
Table 2. Frequency of use of the examination preparation resources.
Frequency
a. Listened to the unit course podcast one or more times6150%
b. Studied with other students in the class weekly43%
c. Studied with other students in the class just before the exam1412%
d. Studied online exam study guide website constructed by students11494%
e. Listened to one or more of the previous class lectures6554%
f. Listened to the online exam review session by the instructor6352%
g. Went to the campus tutoring or writing center32%
Table 3. Frequency of completion of assigned readings and audio files.
Table 3. Frequency of completion of assigned readings and audio files.
Frequency
a. Read or listened to them before they were discussed during class8066%
b. Read or listened to them at least once before the exam6755%
c. Reread or listened to them again before the exam4033%
d. Took written notes about the assigned readings and audio files6251%
Table 4. Frequency of class session activities.
Table 4. Frequency of class session activities.
Frequency
a. Printed or downloaded to your computer the lecture PowerPoint slides9579%
b. During class, wrote additional comments about the material presented in the lecture10385%
c. During discussion time, wrote notes about video clips and assigned readings8772%
d. Frequently participated in the small group discussion at the table9175%
e. Reread the lecture notes before the exam8369%
Table 5. Frequency of examination preparation activities.
Table 5. Frequency of examination preparation activities.
Frequency
a. Created outlines for each of the potential essay questions6352%
b. Tried to predict potential exam multiple-choice questions based on lecture notes and the assigned readings4940%
c. Practiced writing answers to several of the potential essay questions5243%
Table 6. Frequency of number of hours spent preparing for the exam.
Table 6. Frequency of number of hours spent preparing for the exam.
Frequency
a. less than 1 h11%
b. 1 to 3 h2319%
c. 4 to 6 h4134%
d. 7 to 9 h2722%
e. 10 to 14 h1815%
f. more than 15 h108%
Table 7. Frequency of the number of class periods missed for any reason before the exam.
Table 7. Frequency of the number of class periods missed for any reason before the exam.
Frequency
a. 07159%
b. 1 2420%
c. 21412%
d. 387%
e. 432%
f. 5 or more11%
Table 8. Frequency of activities while taking the exam.
Table 8. Frequency of activities while taking the exam.
Frequency
a. Made a brief outline of the essay question before beginning to write7965%
b. Marked up the exam questions by underlining or circling keywords and phrases to make it easier to understand6856%
c. While answering the essay questions, you looked back at the vocabulary and multi-choice questions to locate information to incorporate into the essay8268%
Table 9. Results of independent samples t-test.
Table 9. Results of independent samples t-test.
Metacognitive Task StrategyCompletersNon-Completerst(118)pd
MSDMSD
4a. Read or listened to them before they were discussed during class. 168.8921.20161.0223.511.8560.0330.35
4b. Read or listened to assigned readings and audio files at least once before the exam.169.1822.12162.5622.021.6350.05250.30
5d. Frequently participated in the small group discussion at the table. 168.8321.25158.3023.592.2870.0120.47
6a. Created outlines for each of the potential essay questions. 173.3819.51158.2622.523.9400.0000.73
9a. Made a brief outline of the essay question before beginning to write. 168.6523.08161.4919.941.6850.04750.33
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Kennedy, J.V.; Arendale, D.R. Investigating Metacognitive Strategies and Exam Performance: A Cross-Sectional Survey Research Study. Educ. Sci. 2023, 13, 1132. https://doi.org/10.3390/educsci13111132

AMA Style

Kennedy JV, Arendale DR. Investigating Metacognitive Strategies and Exam Performance: A Cross-Sectional Survey Research Study. Education Sciences. 2023; 13(11):1132. https://doi.org/10.3390/educsci13111132

Chicago/Turabian Style

Kennedy, Jolie V., and David R. Arendale. 2023. "Investigating Metacognitive Strategies and Exam Performance: A Cross-Sectional Survey Research Study" Education Sciences 13, no. 11: 1132. https://doi.org/10.3390/educsci13111132

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop