*Case Report* **The Psychological Impact of the COVID-19 Pandemic in Remote Learning in Higher Education**

**Lívea Dornela Godoy 1,†, Raquel Falcoski 1,2,†, Roberta Monteiro Incrocci 1, Fabiana Maris Versuti <sup>2</sup> and Fernando E. Padovan-Neto 1,\***


**Abstract:** Stressful events can cause a significant impact on education; however, it is not yet clear how the interplay between anxiety, work, and social dysfunction relates to learning impairments. In this study, we investigated the impact of the COVID-19 pandemic on students' learning and mental health. This study was conducted during four modules of a remote Psychopharmacology course between 5 October and 20 December 2020. We collected data from 28 Psychology undergraduate students at the University of São Paulo, Brazil. We used pre- and post-test multiple-choice questions to obtain a quantitative measure of learning. Students completed an online survey to report demographic information, functional impairment (Work and Social Adjustment scale; WSAS), generalized anxiety (Generalized Anxiety Disorder scale; GAD-7), coronavirus anxiety (Brazilian adapted version of the Coronavirus Anxiety Scale; CAS-BR), and self-perception with the remote lectures' methodology. In our sample, 42.9% of respondents experienced symptoms of generalized anxiety disorders (GAD-7 ≥ 15), and 53.6% had moderate to severe functional impairment (WSAS > 20). We also observed an overlapping profile of highly anxious and dysfunctional students. A chi-square test of independence revealed a relation between pairs of multiple-choice questions answers and GAD-7 scores, indicating that less anxious students were more likely to perform better in pairs of pre- and post-tests. Intriguingly, the correlational analysis suggested that students with moderate to severe functional impairment (WSAS scores > 20) were less likely to change from an incorrect to a correct answer to pairs of pre- and post-tests. This data suggests that psychological distress and anxiety states might influence students' ability to coordinate social and work activities and performance during remote learning. Although this study evaluated a small sample of students, our data highlights the importance of investigating anxiety and functional impairment measures as part of the remote-learning curriculum.

**Keywords:** Coronavirus Anxiety Scale; psychology; working and social adjustment scale; GAD-7; anxiety; functional impairment

### **1. Introduction**

Since May 2020, the world has been facing the COVID-19 pandemic, and students have faced severe alterations in routine worldwide. Brazil is one of the leading countries in the number of infections and deaths and, as of 20 December 2020, Brazil held the third most significant number of confirmed cases of COVID-19 in the world at 7,162,978 and the second most considerable rate of confirmed deaths from this disease at 185,650 deaths [1]. One of the few collective prophylactic measures nationally adopted in Brazil was the suspension of in-person learning activities. Specifically, most college courses were transferred to a

Incrocci, R.M.; Versuti, F.M.; Padovan-Neto, F.E. The Psychological Impact of the COVID-19 Pandemic in Remote Learning in Higher Education. *Educ. Sci.* **2021**, *11*, 473. https://doi.org/10.3390/educsci 11090473

**Citation:** Godoy, L.D.; Falcoski, R.;

Academic Editor: Kelum A. A. Gamage

Received: 19 July 2021 Accepted: 23 August 2021 Published: 27 August 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

remote environment to keep the undergraduate students actively enrolled in classes during social distancing [2].

In the face of the many constraints, uncertainties, and adversities related to the COVID-19 pandemic, mental health issues have gained public and research attention, with recent alarming rates of indicators of depression and anxiety [3–5]. Recent studies pointed out that the most affected groups were young adults, women, individuals with a previous mental disorder diagnosis, and individuals at higher risk from coronavirus [6–8]. Adaptability to remote learning during the COVID-19 pandemic can be challenging in many South American countries. Poor delivery of educational content to undergraduate students can make students anxious and trigger mental health disorders.

The main objective of this study was to determine whether high levels of anxiety and functional impairment impact students' learning performance in remote education during the COVID-19 pandemic. Our first aim was to assess the impact of psychological distress and anxiety on students' performance in pairs of pre- and post-classes multiple-choice questions. To test the working hypothesis that students with high levels of psychological distress and anxiety would have a negative performance during remote learning, we correlated changes from an incorrect to a correct answer to pairs of pre- and post-tests with psychometric instruments that measure functional impairment (Work and Social Adjustment scale; WSAS), generalized anxiety (Generalized Anxiety Disorder scale; GAD-7), and coronavirus anxiety (Brazilian adapted version of the Coronavirus Anxiety Scale; CAS-BR). Our second aim was to perform a qualitative analysis on protective factors that were likely to contribute to effective learning in the context of remote education during the COVID-19 pandemic. We analyzed students' perceptions of learning strategies to test the hypothesis that active methodologies (such as Information and Communication Technologies; ICT) can overcome dysfunctional and anxiogenic states to favor the learning process.

#### **2. Materials and Methods**

#### *2.1. Ethics*

The ethics committee of the Faculty of Philosophy, Sciences, and Letters of Ribeirão Preto at the University of São Paulo (FFCLRP/USP; CAAE 32077620.1.0000.5407) approved this study.

#### *2.2. Experimental Design*

Measures of learning were obtained quantitatively using pre- and post-test multiplechoice questions within a virtual learning environment. We developed 4–6 multiple choice questions for each of the 4 study modules allocated in a one-semester psychopharmacology course. All questions encouraged critical thinking about psychopharmacology and clinical knowledge relevant to clinical psychology practice. Sets of study material were selected for each module and made available to students at the University of São Paulo online platform one week before the online lectures. Students answered the same multiple-choice questions at an online platform (Socrative) before (pre-test) and after (post-test) a two-hour talk. At the end of the course, students were asked to complete an online survey at Google Forms platform to report demographic information, functional impairment, generalized anxiety, and self-perception with the methodology of the online lectures.

#### *2.3. Participants*

A total of 28 Psychology undergraduate students of the Faculty of Philosophy, Sciences, and Letters of Ribeirão Preto, University of São Paulo, participated in this study. The remote psychopharmacology course occurred between 5 October and 20 December 2020. Before providing informed consent, participants were informed about the nature, objectives, risks/benefits, and anonymity of responses for this study. Inclusion criteria were age 18 and voluntary participation in the survey without compensation.

#### *2.4. Measures*

#### 2.4.1. Basic Information

Participants reported their age and history of anxiety disorder. Students were also asked to rate, using a 5-point anchored Likert scale (1 = Strongly disagree to 5 = Strongly agree), if they had studied before synchronous classes as instructed (e.g., "I always studied the topics suggested by the professor before the synchronous online lecture").

#### 2.4.2. Psychological Effects of the Coronavirus Outbreak

Students were asked to rate, using a 4-point time anchored scale, how often they experienced the psychological effects of preoccupation (1 = 1 to 3 h, to 4 = 7 h or more) and fear (1 = less than 1 or 2 days, to 4 = almost every day) of the coronavirus outbreak (Lee, 2020). Coronavirus preoccupation was measured by the item, "Over the last two weeks, how much time did you spend thinking about or watching media about coronavirus?" Coronavirus fear was measured by the item, "Over the last two weeks, how often have you experienced significant anxiety, fear, or worry about coronavirus?"

#### 2.4.3. Coronavirus Anxiety

The Brazilian adapted version of the Coronavirus Anxiety Scale is a psychometric instrument used to measure coronavirus-related fear and anxiety [3,18]. The CAS-BR items measure physiologically-based symptoms experienced when triggered by coronavirusrelated information and thoughts. Using a 5-point time anchored scale (1 = not at all to 5 = nearly every day over the last two weeks), students rated how frequently they experienced each anxiety symptom. The original CAS study reported that CAS exhibited good diagnostic properties with an optimized cut score of ≥9 (90% sensitivity and 85% specificity) to classify anxious adults with dysfunctional anxiety [9]. Lee's replication study proposed to lower the CAS cut-off score to ≥5 when assessing the general population [10]. In the Brazilian study, Youden indices were all very low (<0.50) and authors were unable to identify the optimal cut-score for psychiatric screening purposes [3]. Therefore, we did not use a cut-score for measuring coronavirus anxiety in our study sample. The CAS-BR exhibited good internal consistency reliability (α = 0.83) in this study.

#### 2.4.4. Generalized Anxiety

The adapted version of the Generalized Anxiety Disorder scale is often used to indicate clinical symptoms of generalized anxiety [11]. Students were asked to rate seven items of the GAD-7, using a 4-point time anchored scale (0 = not at all to 3 = nearly every day), regarding how frequently they experienced symptoms of generalized anxiety over the past two weeks. GAD-7 cut points of 10 and 15 can be interpreted as representing moderate and severe anxiety levels, respectively. The Brazilian adaptation of the GAD-7 presented an excellent indicator regarding validity and reliability in previous studies [12]. GAD-7 exhibited good internal consistency reliability in this study (α = 0.88), according to previous studies [3,9,13].

#### 2.4.5. Functional Impairment

The adapted version Work and Social Adjustment Scale (WSAS) is a psychometric instrument used to measure functional impairment [14]. Students were asked to rate five items of the WSAS, using a 9-point severity scale (0 = not at all to 8 = very severely), regarding how much impairment they experienced due to fear and anxiety over the coronavirus (e.g., "Because of my fear and anxiety over the coronavirus, my ability to work or study is impaired."). The WSAS is a five-item scale that measures (1) the ability to work or study; (2) home management; (3) social leisure activities; (4) private leisure activities; and (5) the ability to form and maintain close relationships with others. Students were stratified by severity based on WSAS ratings [14]: a WSAS score above 20 suggests moderate to severe psychopathology; scores between 10 and 20 are related to significant functional impairment, and scores below 10 appear to be related to subclinical symptomatology. According to previous studies, this adapted scale exhibited good internal consistency reliability (α = 0.85) [3,4,8].

#### 2.4.6. Students' Perception of Learning Strategies

Students rated, using a 5-point anchored Likert scale (1 = Strongly disagree to 5 = Strongly agree), 9 questions relating to their perceptions of answering pairs of multiplechoice questions in online lectures in structuring or advancing their learning (e.g., "I felt that I learned more during classes when I answered the questionnaires").

#### *2.5. Statistical Analysis*

The demographic and psychometric characteristics of the sample were summarized as means (M) and standard deviations (SD). Continuous data normality was checked out with the Shapiro–Wilk test and analyzed using independent samples t-test and Pearson's correlation as indicated. Pairs of answers to multiple-choice questions were treated as categorical data (expressed as proportions) and analyzed using Pearson's chi-square test of independence. Pairs of multiple-choice questions were normalized to the maximum number of questions that each student answered in all four study modules for correlations with CAS-BR, GAD-7, and WSAS scores. Spearman's rank correlation coefficient assessed the correlation between continuous variables, normalized (proportions), or ordinal data (coronavirus fear and preoccupation). Venn diagrams were constructed online [15]. Data were analyzed using JASP (Version 0.14.1, USA), and significant results were considered when *p* < 0.05.

#### **3. Results**

#### *3.1. Participants*

The study's sample consisted of 20 women (71.4%) and 8 men (28.6%). Students had a combined mean age of 21.2 (19–26). There was no age difference between women and men (t(26) = 1.11, *p* > 0.05). The independent samples t-tests revealed that the total CAS-BR, GAD-7, and WSAS scores were not different between gender (t(26) = 1.48, *p* = 0.15; t(26) = 0.95, *p* = 0.36; t(26) = 0.59, *p* = 0.55, respectively). Therefore, data from men and women were pooled for the subsequent analysis. Most of the participants reported never being diagnosed with an anxiety disorder (85.7%).

#### *3.2. Psychological Effects of the Coronavirus Outbreak, Generalized Anxiety, and Functional Impairment*

Our data demonstrated good convergent validity between all psychometric instruments used in this study (Table 1). In our sample, 42.9% (*n* = 12) of the students had severe anxiety indicators according to GAD-7 scores ≥ 15. Furthermore, 53.6% (*n* = 15) of the students had moderate to severe functional impairment according to WSAS scores > 20. Although we did not use a cut-score for measuring coronavirus anxiety in this study, the averaged CAS-BR scores in our sample (M = 2.75, SD = 3.17) were very similar to our previous study [3].

Most of the students spent 1 to 3 h (*n* = 15; 53.6%), followed by 3 to 5 h (*n* = 7; 25.0%), 5 to 7 h (*n* = 4; 14.3%), and 7 h or more (*n* = 2; 7.1%), thinking about and/or watching media about the coronavirus. Coronavirus preoccupation correlated positively with GAD-7 but not with CAS-BR and WSAS scores (Table 1). When asked about anxiety, fear, or worry about the coronavirus during the past two weeks, most of the participants spent less than a day or two (*n* = 10; 35.7%) or 3 to 7 days (*n* = 10; 35.7%), followed by nearly every day feeling elevated anxiety about the coronavirus (*n* = 8; 28.6%). Coronavirus fear correlated positively with CAS-BR, WSAS, and GAD-7 scores (Table 1). There was a strong correlation between coronavirus preoccupation and fear (Table 1).


**Table 1.** Matrix correlation between psychological scales (items 1 to 3), psychological effects of coronavirus pandemic (items 4 and 5), multiple-choice questions (items 6 and 8), and respective means and standard deviations. Pearson's chi-square values are shown across the table (\* *p* < 0.05, \*\* *p* < 0.01, \*\*\* *p* < 0.001, # *p* = 0.057).

Psychological constructs are represented by the Brazilian Coronavirus Anxiety Scale (CAS-BR), Generalized Anxiety (GAD-7), and Working and Social Adjustment Scale (WSAS). Psychological effects are represented by coronavirus preoccupation and fear. Multiple-choice questions are represented by correct to correct, incorrect to correct, and other (incorrect to incorrect or correct to incorrect) answers in the pre-test and the post-test.

#### *3.3. Analysis of Student's Performance on Multiple-Choice Questions*

Since not all students who completed the survey answered all the pre- and postmodule multiple-choice questions of all four evaluated study modules, only students who provided answers to all the pre- and post-tests in a particular study module were included in our analysis. Therefore, we used 422 pairs of pre- and post-tests, and 138 pairs of questions were excluded from our analysis. First, we checked if frequencies of correct and incorrect answers collected in the pre-test and the post-test were independent of the number of multiple-choice questions collected in one module (*n* = 16), two modules (*n* = 52), three modules (*n* = 134), and four modules (*n* = 220) (Figure 1). The Chi-square test revealed that frequencies of correct or incorrect answers were independent of the number of questions collected in both the pre-test (χ<sup>2</sup> (3) = 3.72, *p* = 0.29; Figure 1A) and the post-test (χ<sup>2</sup> (3) = 3.46, *p* = 0.32; Figure 1B). Therefore, although not all students answered the same number of pairs of questions during this study (i.e., some students did not participate in all study modules), the proportion of correct and incorrect answers were consistent in all four study modules. We then pooled the frequencies of correct and incorrect answers and found a strong association between educator's synchronous intervention and correct/incorrect answers (χ<sup>2</sup> (1) = 29.74, *p* < 0.001; Figure 1C). The percent of correct answers increased from 62.3% (*n* = 263) in the pre-test to 79.4% (*n* = 335) in the post-test. Likewise, the percent of incorrect answers decreased from 37.7% (*n* = 159) in the pre-test to 20.6% (*n* = 87) in the post-test, confirming the potential utility of pre- and post-test multiple-choice questions as a measure of learning.

A more detailed analysis (Figure 1D) demonstrated that 56.4% (*n* = 238) of responses to pairs of multiple-choice questions were correct, suggesting that students consulted the available online material one week before the online lectures. In addition, a total of 23.5% (*n* = 99) of responses to pairs of pre- and post-tests were changed from incorrect to correct, suggesting that students could learn new content. Other combinations (20.1%) of multiple-choice questions (i.e., correct to incorrect or incorrect to incorrect; *n* = 25 and 60, respectively) also occurred, suggesting that students were either uncertain about the correct answer or were not engaged in the learning tasks. In addition, we cannot exclude potential issues of failure in delivering learning content.

**Figure 1.** Correct and incorrect answers in study modules. As evidenced by the module-to-module comparison (**A**,**B**) and overall frequency of correct answers (**C**), there is an increase of correct answers from the pre-test to the post-test. Frequencies of correct (black) and incorrect (white) answers shown in the pre-test (**A**) and the post-test (**B**) also demonstrate data were independent of the number of multiple-choice questions collected in modules. The percent of changed answers (**D**) from correct to correct (black) were 56.4% (*n* = 238), incorrect to correct (gray) were 23.5% (*n* = 99), and other combinations of multiple-choice questions (stripes) were 20.1% (encompassing both correct to incorrect and incorrect to incorrect; *n* = 25 and 60, respectively).

#### *3.4. Impact of Generalized Anxiety and Functional Impairment on Student's Performance on Multiple-Choice Questions*

We used the GAD-7 and WSAS scales to check how students who experienced clinically significant impairment because of generalized anxiety (GAD-7 scores ≥ 15) and functional impairment (WSAS scores > 20) performed across all four study modules (i.e., correct/correct, incorrect/correct, and other pairs of multiple-choice questions). Because some students who completed the survey did not participate in all four study modules, we compared the number of questions each student answered according to GAD-7 and WSAS cut-scores. Our results indicated that all students answered to a similar number of pairs of multiple-choice questions in all four study modules according to GAD-7 (t(26) = 0.38, *p* < 0.05; Figure 2A) and WSAS cut-scores (t(26) = 0.89, *p* < 0.05; Figure 2C).

**Figure 2.** Profile of students' performance in pairs of multiple-choice questions according to GAD-7 and WSAS scores. Students with low anxiety scores (GAD-7 < 15; white) were more likely to answer correctly in tests, whereas students with moderate to severe functional impairment (WSAS scores > 20; black) were less likely to change from an incorrect to a correct answer in the multiple-choice test (**A**–**D**). Box plot distribution of pairs (pre- and post-test) of answers per student in all four modules (**A**,**C**) and proportion of pairs of answers of multiple-choice answers (**B**,**D**) were shown according to GAD-7 and WSAS severity cut-scores. Correct to correct (C to C), incorrect to correct (I to C), and other combinations (both correct to incorrect and incorrect to incorrect) were represented in the histogram chart. GAD-7 scores ≥ 15 and WSAS > 20 (black) cut-scores show moderate to severe anxiety and functional impairment indicators. Venn Diagrams (**E**,**F**) show the proportion of students categorized in the cut-score psychometric instruments and the overlap of GAD-7 ≥ 15 and WSAS > 20 scores and GAD-7 < 15 and WSAS < 20 scores.

A chi-square test of independence was performed to examine the relationship between pairs of answers and GAD-7 or WSAS. The relation between pairs of answers and GAD-7 was significant (χ<sup>2</sup> (2) = 6.77, *p* < 0.05), indicating that students with GAD-7 scores < 15 were more likely to be able to identify correct answers, to change from an incorrect to a correct answer, and to make more minor mistakes (i.e., change from correct-to-incorrect or incorrect-to-incorrect answers) in pairs of multiple-choice questions (Figure 2B). The relation between pairs of answers and WSAS was not significant (χ<sup>2</sup> (2) = 1.25, *p* < 0.05; Figure 2D). Venn diagrams indicate participants' relationship with the study sample with scores above (Figure 2E) and below (Figure 2F) GAD-7 and WSAS cut-scores.

There was no correlation between GAD-7 scores and answers to pairs of multiplechoice questions (Table 1). Interestingly, there was a negative correlation between WSAS scores and the percent of answers to pairs of pre- and post-tests that were changed from incorrect to correct (Table 1), suggesting that higher WSAS scores impact students' ability to learn new content.

#### *3.5. Students' Perception of Learning Strategies*

Students were asked to rate five questions relating their perceptions of the benefit and usefulness of assigning pre- and post-test multiple-choice questions in online lectures in structuring or advancing their learning. Most students considered their experience with this methodology as positive (Figure 3). Overall, students either agreed or strongly agreed that the teaching method helped develop study skills. They reported that multiple-choice questions helped guide the essential concepts in the lecture, and they felt they learned more when answering the questionnaires.

**Figure 3.** Most students considered the ICT methodology a positive experience in structuring or advancing their learning. Students were asked to rate their perceptions through a 5-point agreement scale, ranging from strongly disagree to strongly agree, in questions regarding the methodology, including the teaching method, the use of multiple-choice questions, and the synchronous online lectures.

#### **4. Discussion**

This study examined the impact of ICT on students' learning during a remote psychopharmacology college course during the COVID-19 pandemic and its relationship with students' mental health. A high proportion of students were under elevated anxiety levels and functionally impaired at the end of the academic year during the COVID-19 pandemic. Multiple-choice questionnaires showed that students could still learn new content during remote learning. Remarkably, psychometric measures of generalized anxiety (GAD-7) and functional impairment (WSAS) were associated with lower performance in measures of learning. Our study highlights the importance of investigating anxiety and functional

impairment indicators during COVID-19 pandemic remote learning. According to students' perception of learning strategies, it is likely that the use of active learning methodologies (i.e., the teaching method, the use of multiple-choice questions, and the synchronous online lectures) was able to encourage students' engagement. Different from the passive learning approach, active learning strategies contribute positively to remote teaching. Brazil has the world's highest prevalence of anxiety (WHO, 2017). The COVID-19 pandemic aggravated the symptoms of anxiety and distress in the Brazilian population [3,16,17]. Specifically, women and young adults were amongst the most affected groups [3,6–8]. Studies conducted with college students in Brazil and other countries demonstrated higher anxiety, depression, and stress levels. Therefore, the COVID-19 pandemic has had a negative psychological impact on students.

#### *4.1. The Impact of Psychological Distress and Anxiety on Students' Performance in Pairs of Preand Post-Classes Multiple-Choice Questions*

The CAS-BR is an important psychometric instrument to measure coronaphobia in Brazilian adults. It can be used as a parameter to estimate the psychological and psychiatric impact of the COVID-19 pandemic on the Brazilian population. The CAS-BR items (dizziness, sleep disturbances, tonic immobility, appetite loss, and nausea/abdominal distress) represent physiological symptoms related to clinically elevated fear and anxiety [9]. CAS-BR scores in this study sample were very similar to what we observed in our previous study [3]. Our data demonstrated that CAS-BR scores correlated positively with generalized anxiety (GAD-7), functional impairment (WSAS), and fear about the coronavirus. Unexpectedly, there was a lack of correlation between the CAS-BR scores and coronavirus preoccupation. This finding was also reported in another study and might suggest that our sample shows a more complex behavioral manifestation [4]. Therefore, the results of our study demonstrated that CAS-BR scores were related to adverse psychological effects of the COVID-19 pandemic and support the construct validity of the CAS-BR and replicate previous findings.

According to the original study [11], GAD-7 cut points ≥ 15 can be interpreted as representing severe anxiety levels. Our study demonstrated that 42.9% of the students were likely to exhibit generalized anxiety symptoms according to this cut-score. A high proportion of students (53.6%) also presented moderate to severe functional impairment indicators according to WSAS scores ≥ 21 [14]. This result indicates that the significant distress due to the pandemic could contribute to functional impairment. Students showed high anxiogenic and dysfunctional scores, but an overlap of these psychological states is further represented by a significant correlation between the GAD-7 and WSAS scores. Functional impairment is often considered in tandem with anxiety during clinical assessment. Despite the multidimensional nature of the first, it is argued that GAD-7 seems to be more deeply related to the social dysfunction dimension [18]. However, extremely high scores on the GAD-7 scale may be related to a decline in overall functional status [11]. Surprisingly, in our study, most of the participants reported they have never been diagnosed with anxiety disorder, and healthy controls usually present low scores on these scales [19]. WSAS and symptoms-based instruments are often combined in primary care mental health [20], but WSAS seems to address a more general aspect that is responsive to diagnostic scales [20,21].

A recent study in the US also showed that anxiety, depression, and anger symptoms predicted significantly more work and social adjustment problems in WSAS [22]. It has been shown that the GAD-7 and the WSAS scale items reveal a clear bi-factor pattern (general trait measured by the test overall, as well as specific or secondary traits measured by the subscales), with medium to high loadings on both general and individual factors. Those psychometric instruments enable assessing negative experiences that can include both anxiety and depressive states [20]. Therefore, our data add to the discussion that WSAS is a screening tool sensitive not only to clinical change but to different clinical subgroups, representing different levels of severity of mental distress.

We verified if exacerbated signs of generalized anxiety and functional impairment would impact students' performance at the end of the academic year. Overall, students

showed a good performance in multiple-choice questions on all four study modules. The high percent of pairs of correct answers detected in both pre-test and post-test (56.4%) suggested that students engaged in active learning and consulted the available online material one week before the online lectures. This observation is also supported by the fact that many students agreed that "I always studied the suggested material before classes." We also found that 23.5% of the responses to pairs of pre- and post-tests were changed from incorrect to correct, indicating that students could learn new content during remote teaching. Learning can be verified when the learner can present behavior that he did not present before the teacher's intervention [23]. Therefore, the difference between the learner's performance before and after the teacher's intervention is a valuable measure of learning and suggests that students learn new content [24].

It is broadly accepted that emotional context modulates action–outcome contingencies in an uncertain environment. Previous studies demonstrated that disrupted learning ability is present in high trait social anxious individuals [25,26]. In addition, chronic stress is known to impair retrieval and working memory and contribute to the development of anxiety [27]. Considering the great adversities imposed by the pandemic and the highly dysfunctional scores, many students still cope with learning. The methodology we used likely contributed to overcoming dysfunctional and anxiogenic states and contributed to the learning process.

Even though most students learned despite the high dysfunctionality, lower GAD-7 scores were associated with better learning performance in pairs of multiple-choice questions. We also found a negative correlation between WSAS scores and the percentage of pairs of pre- and post-tests that were changed from incorrect to correct. Taken together, our data suggest a negative association between anxiety-related psychological distress and students' ability to learn new content. Therefore, this data highlights the need to investigate the impact of anxiety and dysfunctionality on students' learning during remote teaching.

#### *4.2. Protective Factors That Can Contribute to Effective Learning in the Context of Remote Education*

Anxiety disorders are related to disrupted learning in uncertain environments [28,29]. A large-scale longitudinal epidemiologic assessment of college students in China demonstrated that mental health concerns significantly increased throughout the pandemic [30]. Anxiety levels were mapped in medical and non-medical students during remote and hospital classes in the United Arab Emirates during the pandemic. Interestingly, non-medical students reported higher levels of anxiety before and after online learning than medical students. While medical student anxiety levels decreased following online learning, nonmedical students' anxiety levels increased [31]. These results could foster discussions on protective and teaching strategies that support education in contexts of adversity.

As a strategy to keep the fundamental interactions and the exchange of experience in the virtual environment, the importance of tutorial mediation and the strengthening of the self-learning process associated with the mediation should be highlighted. Following up student activities, motivating learning, guiding, and providing conditions for autonomous learning could significantly contribute to student engagement and functionality in this new college format [32].

Educators, more than ever, are struggling to adapt to remote classes and keep a supportive and engaging learning environment [33]. Online platforms and learning tools became essential in higher education to establish active communication between students and educators [2,33]. Therefore, a dynamic learning environment is necessary for proper content delivery and strengthening social ties during social distancing. Even though face-toface teaching is irreplaceable, it is needed to explore ICT in the learner–content interaction, learner–instructor interaction, and learner–learner interaction [34]. Formative assessment in higher education is essential to enhance teaching and learning practice, especially in a remote environment. Educators need to adopt personalized, self-tailored, active methodology strategies to build contextualized active and meaningful learning. Unlike the traditional education model based on passive content exposure, active learning can

increase cognitive flexibility and overcome rigid mental models associated with inefficient automatic responses [35]. Therefore, when students are encouraged to learn autonomously and actively, they are more likely to connect with new knowledge to previous knowledge and acquire new problem-solving skills.

Students should be the protagonists of their learning to develop socioemotional skills such as critical thinking, problem-solving, cooperation, mutual respect, perseverance, and flexibility [36,37]. Online interaction is an essential resource for communication between professors and students, and, if used correctly, it can maintain or even strengthen social bonds during the physical distancing period. Thus, the combination of active teaching methodologies (including those mediated by ICT), immediate feedback during remote teaching, and test-taking strategies can be influential protective factors for the psychological instability generated by the pandemic in the educational field.

It is possible to assess the effectiveness of the teaching method by applying the same multiple-choice questions before and after each class using software that provides real-time feedback. The first round of questions is applied before the lecture and can access students' previous knowledge. In a well-designed methodology, correct and incorrect answers guide the professor on concepts to emphasize during teaching. On the other hand, the second round of the same multiple-choice questions provides the students' performance and the ability of the instructor to teach the concepts he considers relevant to a particular topic [38]. Therefore, learning can be verified when the learner can present behavior that he did not present before the teacher's intervention [23]. Furthermore, the difference between the learner's performance before and after the teacher's intervention is a valuable measure of learning [24].

Although this study was conducted in a small sample, educational technologies based on active methodologies favored the educational process more effectively and autonomously. This observation is supported by the fact that many students had a very positive impression of strategies used in the classes. Because students have different learning rates, it is essential to allow sufficient time to access the learning materials. Flexible schedules, follow-up activities, and student development should be considered to minimize the psychological impact caused by the COVID-19 pandemic. In other words, the educational process should focus on human development. Strategies to increase interactions within the virtual learning environment, tutorial mediation, and strengthening the self-learning process should also be used. Follow-up activities, tutorial guidance, and autonomous learning could significantly contribute to students' engagement and functionality during remote learning. In addition, it is suggested that incorporating students' perspectives (such as interests, tastes, and expectations) contribute to the learning process [39]. Collectively, these results presented here show the importance of adaptations of teaching-learning methodologies during the pandemic period and demonstrate that online classes cannot be a repository of digital content.

Our study has several limitations. First, we evaluated a small sample in a particular context of remote learning. Therefore, generalization of the results requires future studies. A control experiment conducted in a non-pandemic period would help to understand better the impact of psychological distress and teaching methodology in remote learning. Second, we were unable to map differences across gender and ages. Third, despite the general satisfaction with the methodology, we cannot exclude potential failure problems in delivering learning content. Nevertheless, this study has several strengths. Although we collected data in a small sample, we were able to run within the experimental design and correlate students' performance in multiple-choice questions with psychometric tools. It is worth noting that when we look closely to the student sample in our previous study [3]; *n* = 104; age: M = 24.50, SD = 5.63), CAS-BR (M = 2.53, SD = 2.88), GAD-7 (M = 11.42, SD = 5.29), and WSAS scores (M = 17.50, SD = 9.81) were very similar to the sample used in the current study (*n* = 28, see Table 1).

#### **5. Conclusions**

In conclusion, psychometric measures of generalized anxiety and functional impairment were associated with lower performance in measures of learning. Therefore, the data presented here highlight the need for college institutions to investigate how psychometric instruments relate to learning in disruptive situations such as the COVID-19 pandemic as part of the remote-learning curriculum. Furthermore, according to students' perception of learning strategies, we conclude that active learning strategies could support students and promote psychological protection in adverse contexts. Even teachers who already used online environments in their practices had to abruptly adapt to the new teaching context with the advent of emergency remote education [40]. Thus, there is a need for a more in-depth discussion on teacher training and the role of pedagogical practices for remote teaching. On the other hand, new contexts and necessities in education generate learning communities that facilitate and perpetuate learning. Interaction and collaboration between educators and learners must occur in a dynamic, flexible, and adaptable fashion.

**Author Contributions:** Conceptualization and methodology, F.E.P.-N.; formal analysis, F.E.P.-N. and L.D.G.; methodology, R.M.I. and F.E.P.-N.; data curation, R.M.I.; All author contributed to writing the paper, from the original draft preparation to its final version; supervision, F.E.P.-N., F.M.V., R.F. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by CAPES-PROEX and São Paulo Research Foundation (FAPESP, grant 2017/00003-0).

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board (or Ethics Committee) of Conselho de Ética e Pesquisa (protocol code 32077620.1.0000.5407 approved in August 2020).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Data of this article will be made available by the authors without undue reservation.

**Acknowledgments:** We kindly acknowledge the Psychology Department at the Faculty of Philosophy Sciences and Letters of Ribeirão Preto at the University of São Paulo.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

