Next Article in Journal
Dust Retention Effect of Greenery in Typical Urban Traffic Landscapes of Nanjing—In the Case of Xuanwu Avenue in Nanjing City
Next Article in Special Issue
The Use of Gamification and Web-Based Apps for Sustainability Education
Previous Article in Journal
Bamboo Scaffolding as a European Promising Opportunity: A Structural Feasibility Study
Previous Article in Special Issue
Enhancing Empathy for Justice: A Methodology for Expansive Teacher Professional Development through Creative Body-Based Learning
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sustainability
Volume 16
Issue 2
10.3390/su16020916
sustainability-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Classroom Emotion Monitoring Based on Image Processing

by
Cèlia Llurba
*,
Gabriela Fretes
and
Ramon Palau
Department of Pedagogy, Universitat Rovira i Virgili, Crta Valls s/n, 43007 Tarragona, Spain
*
Author to whom correspondence should be addressed.
Sustainability 2024, 16(2), 916; https://doi.org/10.3390/su16020916
Submission received: 29 December 2023 / Revised: 12 January 2024 / Accepted: 17 January 2024 / Published: 22 January 2024
(This article belongs to the Special Issue Sustainable Education: Innovations and Developments of Educational Technology)
Browse Figures
Versions Notes

Abstract

:
One challenge of teaching and learning the lack of information during these processes, including information about students’ emotions. Emotions play a role in learning and processing information, impacting accurate comprehension. Furthermore, emotions affect students’ academic engagement and performance. Consideration of students’ emotions, and therefore their well-being, contributes to building a more sustainable society. A new way of obtaining such information is by monitoring students’ facial emotions. Accordingly, the purpose of this study was to explore whether the use of such advanced technologies can assist the teaching–learning process while ensuring the emotional well-being of secondary school students. A model of Emotional Recognition (ER) was designed for use in a classroom. The model employs a custom code, recorded videos, and images to identify faces, follow action units (AUs), and classify the students’ emotions displayed on screen. We then analysed the classified emotions according to the academic year, subject, and moment in the lesson. The results revealed a range of emotions in the classroom, both pleasant and unpleasant. We observed significant variations in the presence of certain emotions based on the beginning or end of the class, subject, and academic year, although no clear patterns emerged. Our discussion focuses on the relationship between emotions, academic performance, and sustainability. We recommend that future research prioritise the study of how teachers can use ER-based tools to improve both the well-being and performance of students.

1. Introduction

1.1. Conceptualisations of Emotions

The concept of emotions has been addressed by many authors in the fields of psychology and philosophy. First, Darwin proposed that emotions have a biological basis and that the facial expressions of emotions have an evolutionary origin [1]. According to James, emotions are the perceptions of specific bodily changes in response to emotional stimuli [2]. Based on that, the authors of [3] worked in collaboration with William James on the development of the James–Lange theory, which proposes that emotions result from the perception of bodily responses that occur in emotional situations. For Freud, emotions are internal impulses that may be influenced by unconscious conflicts [4]. Meanwhile, Lazarus proposed that emotions result from the cognitive evaluations we make of events or situations in relation to our personal goals and values [5].
From the phenomenological perspective of Husserl [6], emotions can be regarded as the most intricate stratum of the affective experience. They exhibit a dual nature, encompassing a physical facet involving bodily changes in response to emotional states, and a cognitive facet characterised by evaluating or appreciating the possibilities of a given situation [6]. These two dimensions are intricately interlaced, giving rise to the following four foundational attributes: affective intentionality, bodily resonance, inclination towards action, and function significance.
Ekman and Friesen focused on universal emotions and facial expressions [7]. Ekman’s theory of emotions has been widely studied and is one of the most influential. According to Ekman, there are six basic universal emotions that are recognised in different cultures—happiness, sadness, surprise, fear, anger, and disgust. These basic emotions are discrete, meaning they comprise distinct patterns of psychological, physiological, and neurobiological features that distinguish them from one another. They serve as the foundation for more complex emotional experiences, developed to satisfy our needs in specific environments or circumstances.
Ekman’s classification has been used in several practical applications, such as in face design and the human emotion identification process known as emotional recognition. This technology analyses facial expressions from still images and videos to reveal information about a person’s emotional state. In the process, a standardised classification of emotions facilitates communication and information sharing among researchers. To further the research with that goal, here, we detected students’ facial emotions using an ER technology developed from Ekman’s theory, with the addition of a neutral emotion.

1.2. Emotions and Learning

Several authors have investigated how emotions influence the learning process. The Russian term perezhivanie, from Vygotsky’s theory, refers to a deeply lived experience that is charged with emotional meaning. This idea suggests that emotional experiences play a key role in the internalisation of knowledge and skills. Rich emotional experiences may, in particular, lead to deeper and more lasting internalisation of information, which may influence long-term academic performance [8]. Accordingly, one theory of emotional intelligence [9] highlights the importance of emotional skills in academic performance and learning. According to [9], emotions have a significant impact on motivation, the ability to regulate stress, and the ability to relate to others. Furthermore, Damasio uses the somatic theory of emotions to highlight the fundamental role they play in cognitive processes [10]. For instance, emotional experiences are closely linked to decision-making and memory formation and can facilitate the consolidation and retrieval of information, influencing the learning process. The emotional dimension of the learning experience, specifically the feelings associated with perceiving and processing new information, may thus form an essential component of the acquired knowledge and abilities [11,12]. If this is the case, then promoting students’ social–emotional well-being by monitoring their emotions may contribute to fostering a positive classroom climate and creating healthier and more equitable educational communities, which is an important component of sustainable development [12,13].
Pekrun’s theory of emotions [14] provides a conceptual framework by which to understand how emotions affect learning and academic performance. This framework distinguishes two types of emotions in the academic context: activation emotions and value emotions. Activation emotions are related to perceptions of control and may include anxiety and boredom. Value emotions, meanwhile, are linked to the importance and value of the task and include emotions such as joy and pride. Under this paradigm, students experience a wide range of emotions in the academic environment, except for disgust [15]. In previous research, anxiety constituted 15–27% of all emotional episodes in three academic situations (study, class, test/exam). While that was considerable, the finding indicated that most emotions in academic situations were not anxiety-related. Pleasant emotions, such as enjoyment, satisfaction, hope, pride, and relief, contrasted with negative emotions like anger, anxiety, embarrassment, and boredom. Students also mentioned less frequently experienced emotions, such as hopelessness, and social emotions such as gratitude, admiration, contempt, and envy.
The studies linking pleasant emotions to achievement show that joy, hope, and pride positively correlate with students’ academic self-efficacy, academic interest and effort, and overall achievement [16]. Positive emotions are hypothesised to facilitate approach-related activities, and these activities are likely to provide academic benefits, particularly as the student moves toward a desired goal [17,18]. Furthermore, Fredrickson [19,20] suggested that positive emotions enhance academic competence because they encourage exploring, integrating diverse materials, and broadening potential methods of solving problems.
On the topic of unpleasant emotions, Fredrickson [21] found that young children with negative emotionality struggle with higher-order cognitive processes because of their lack of reflective planning and problem-solving skills. When a student’s experience of negative emotion leads to fixation on the object of the emotion (such as when a child dwells on the morning’s event that caused their anger), cognitive resources are redirected from educational materials to distractions that hinder learning. In this way, negative emotions interfere with scholar activities by limiting cognitive resources necessary for integrating and recalling important details.
In relation to this, it has been found that the anxiety and frustration from struggling with maths exercises can cause a negative perception and dislike of the topic [22,23]. Additionally, research has found that individuals experiencing unpleasant emotions (such as sadness, frustration, or boredom) tend to process new information in a rigid and shallow manner [24,25].
Furthermore, strong and persistent anxiety impairs learning, though occasional and mild anxiety has its benefits. The same goes for other emotions and moods, which may have contrasting learning effects. Additionally, according to Efklides [26], a certain pessimism or seriousness can be advantageous for analytical and quantitative tasks, while a positive mood may be more beneficial for creative and heuristic thinking.
From another perspective, some empirical findings support the notion that the effect of emotionality on achievement might be indirect, through cognitive processes, interpersonal relationships with the teacher and peers, and motivational mechanisms, such as engagement, enjoying school, and staying on task [27]. In this way, a link is proposed between motivation and academic emotions [28], as motivational behaviours involve positive and negative academic emotions, which students experience in academic settings and which explain all kinds of psychological processes during learning, as Pekrun called them [15]. Specifically, positive academic emotions are usually beneficial, but negative academic emotions, such as dissatisfaction and uneasiness, can have contradictory effects. Regardless, it should be observed that students’ academic emotions often impact their performance.
The effects of emotions on learning in facilitating or impeding the acquisition of new knowledge by influencing its value and desirability have been described as hedonic affects [29]. For example, the satisfaction of solving maths problems enhances the acquired knowledge and skills, serving as motivation to continue the activity. Likewise, students who are in a positive frame of mind are more likely to think creatively and learn a topic meaningfully [30].
Teachers also influence students’ emotions. The relationship between teachers’ social and affective strategies and students’ academic performance in an English language class was explored by Saeidi and Jabbarpour [31]. The researchers propose that language teachers should utilise affective strategies, such as humour, positivity, fairness, encouragement, and politeness, to effectively teach and enhance student achievement.

1.3. Emotions’ Assessment

Until now, it has been difficult to recognise students’ emotions [32]. Self-assessment is one of the most widely used ways to measure emotions [33,34,35]. Individuals report their emotional experience using rating scales or questionnaires that request ratings of intensity and valence (positive or negative) and descriptions of the emotions experienced. Alternatively, in clinical or research settings, researchers can use direct observation of emotions through facial expressions and nonverbal language [36].
Taking another approach, physiological responses, such as heart rate, skin conductance, and brain activation patterns, can infer the presence and intensity of emotions [37,38]. These measurements may be obtained using technologies such as electrocardiograms (ECGs), electrodermal activity monitors, or neuroimaging techniques [39].
Adding to those options, in recent years, advancements in technology have allowed for the development of AI-based cameras that can detect and recognise facial expressions, enabling the assessment and monitoring of basic emotions in various settings, including the classroom [40,41]. Hence, the use of facial expression recognition technology has become a promising tool for assessing and monitoring emotions in real-time, providing valuable insights for both researchers and practitioners in the fields of psychology, education, and healthcare [42].
Facial expressions serve as a conduit for understanding individuals’ emotional states [43]. In the educational field, employing AI-based methodologies enables continuous monitoring of students’ emotions. This approach facilitates monitoring how diverse methodologies, educational scenarios, evaluative contexts, etc., influence emotions. In this way, it allows for pre-emptive and targeted interventions to guide or enhance emotional experiences. Against that background, the purpose of this study was to explore whether the use of these advanced technologies can assist the teaching–learning process, despite the complexity of the teaching task, while ensuring the emotional well-being of students. To that end, continuous monitoring with a laptop camera and software was used in this study to explore the basic emotions of secondary school students in the classroom. The specific objectives were as follows:
I.
Analyse the students’ manifestations of emotions in the classroom.
II.
Compare the emotions at the beginning and at the end of the class.
III.
Relate the different emotions to the subject and academic year.

2. Materials and Methods

This study adopted an exploratory observational design to monitor the emotions of secondary school students in six different class groups over 4 weeks.
The secondary school is located in Cambrils, a coastal town in northeastern Spain, which has three cohorts of four secondary education courses, and two cohorts of two bachelor’s degree courses. The average number of students per class is 32 and the number of students that participated in the experiment is shown in Table 1. Classes are equally mixed by gender.
The experiment occurred during the first term of the school year. Most students had already studied the subject technology in previous years, but it was a new subject for the first-grade students in secondary education. The same went for the optional subject, the Green project. Six class groups were recorded during the experiment. The first two groups consisted of 16 students from the first year of secondary school, aged 12–13, who were enrolled to study technology and the Green project. The second group consisted of 24 students from the fourth year of secondary school, aged 15–16, who were enrolled in robotics. The third and fourth groups consisted of 24 students from the first year of the bachelor’s degree, aged 16–17; those in the third group were studying technology, while those in the fourth group were studying robotics. The last group comprised 12 students from the second year of the bachelor’s degree, aged 17–18, who were enrolled in technology. Every student group had a female teacher assigned to them. Figure 1 illustrates the arrangement of the students in the classroom, and Figure 2 shows some students attending class.

2.1. Materials

We used a laptop, an Intel Core i5, PC Notebook HP ProBook 640 G2, with Intel Core i-5 6200U @ 2,3 GHz, RAM 8 GB, with a webcam.
The laptop was placed in a high position in front of the class, in order to focus on the students’ faces, with the webcam directed towards them.
The experiment led us to take 47 videos. The dataset was annotated in a semi-automated procedure, with both manual and automatic annotations.

2.2. Experiment’s Procedure

Previously, we investigated the effect of the emotions of two students attending classes in two different subjects for several hours. To end that work, in this experiment, we used an improved configuration and a scenario of more students. The aim was to develop and apply a code capable of detecting faces and ER, then transfer the data collected into a database for further analysis. Based on that, we would explore the initial links between students’ emotions, subjects, time of day, and academic performance [40].
Python was used as the programming environment in which we developed the code for acquiring and processing images (face detection, identification, and ER). Besides the language itself, Python has many instructions in libraries that can simplify complex tasks by introducing just a few lines of code. In addition, to facilitate programming, a code editor (IDE) was needed to create and execute the code; in this case, we used Visual Studio Code. Furthermore, Py-Feat (Python Facial Expression Analysis Toolbox) from a GitHub portal was also used to promptly process, analyse, and visualise the facial expression data.
In each class, the students were recorded using the camera on the laptop to obtain data; 50 min to 1 h of video was recorded and saved in an .mp4 file for each class. The video included as many students as there were in the field of view of the webcam. This camera could clearly focus on the front of the room as well as the back, which allowed us to obtain the full effect of using such an imaging tool in a classroom. Then, videos were uploaded and stored in a Google Drive. After that, a code was applied to split the videos into consecutive frames, one every 10 s, which were saved as images in .png files for further data analysis.
A code capable of detecting and identifying faces and analysing facial expressions was developed, with Py-Feat the chosen tool for obtaining the emotions of the students attending class. The results were entered into a .csv file containing all emotions.
Prior to discerning the emotions, the code gives different action units (AUs) for every detected face, which is a quantitative method for describing facial movements. We extracted the AUs of each facial part from the videos, and we added intensity to the code to obtain the value of each emotion (continuous values from 0 to 1). This data processing of AUs was performed independently for each face.
It should be noted that in order to be included, it was necessary that a student was within the focus of the webcam and was looking straight ahead, or at least the camera detected enough of their face to extract data that could be analysed.
In the AU detection task, we utilised the unit defined by the Facial Action Coding System [44] to capture and interpret facial muscle movements associated with different expressions [45]. In this research, we particularly focused on emotion detection.

2.3. Computer Code

The code used for the experiment was the Windows computer interpreter Command Prompt (Figure 3), which allowed us to convert the recorded videos of 50 to 60 min duration into images, keeping an image from every 10 s, thus achieving an average of 300–360 images per recorded class.
Then, these images, with the help of a code editor for a desktop computer, Visual Studio Code (Figure 4), were processed in the same command to be converted into AUs and emotions. The images saved for the ER through the code were deleted once all the data were available since these images were no longer useful.
The code protected the privacy of the students, making it impossible to track a particular student.
As shown in Table 2, a .csv file was obtained with all the emotions obtained from the students detected in the images.
Ethical approval was given by the Ethics Committee of the Rovira i Virgili University before data collection because this experiment involved contact with humans, and more precisely, with minors (under the age of 18). They reviewed and approved this experiment with the reference number: CEIPSA-2021-TD-0019. Written informed consent was obtained from all students and their parents. In addition, at the beginning of the academic year, the parents of all students in the school signed a consent form in favour of or against their children being photographed or recorded. Students without parental consent sat out of the camera’s reach.

3. Results

We chose to apply nonparametric data analysis techniques because they are useful when data cannot be assumed to follow a specific parametric distribution, such as the normal distribution.

3.1. Students’ Emotions That Manifested in the Classroom

The heatmap in Figure 5 shows the correlations observed between the variables of interest. Pearson correlation coefficients were used to evaluate the strengths and directions of the linear relationships between the variables. We found that all emotions correlated with each other, positively or negatively.
We highlight the positive correlations, which were significant between the emotions disgust and anger (0.447, p < 0.001) and between sadness and fear (0.417, p < 0.001). The significant negative correlations, meanwhile, were between anger and fear (−0.321, p < 0.001) and between anger and sadness (−0.291, p < 0.001).

3.2. Comparison of Emotions at the Beginning and at the End of the Class

When comparing the emotions during the first five minutes of class and the last five minutes (Table 3), using a nonparametric Mann–Whitney U test, we found significant differences (Table 4).
Specifically, fear was present with greater intensity at the beginning of the class (1.634 × 106, p > 0.001) (Figure 6). Happiness followed the same trend, with greater intensity at the beginning than at the end of the class (1.658 × 106, p < 0.001) and, to a lesser level, the same was true of anger (1.607 × 106, p = 0.002).
On the contrary, sadness was somewhat less intense at the beginning and increased at the end of the class (1.425 × 106, p = 0.003). Finally, there were some emotions, including a neutral emotion, for which no significant differences were observed; or in the case of disgust, the observed difference had no practical significance when we analysed the descriptive information of the variable.

3.3. Different Emotions According to the Subject and Academic Year

Our Kruskal–Wallis analysis (Table 5) indicated statistically significant differences in all emotional responses among academic years. Specifically, anger varied significantly between academic years (1, 5, 6) (H = 215.711, p < 0.001), as did disgust (H = 222.708, p < 0.001) (5, 6). Fear and surprise showed significant differences (H = 13.627, H = 127.004, p < 0.001) between academic years (1, 5), while happiness and sadness (H = 556.320, H = 190.068, p < 0.001) also varied (1, 5, 6).
Having identified differences among groups through the Kruskal–Wallis analysis, subsequent post hoc comparisons were executed by employing Dunn’s test with Bonferroni adjustment to evaluate specific variations between pairs of groups (Table 6). The outcomes were as follows:
-
Anger: Differences were observed in all pairs of subjects, with the most pronounced difference (p < 0.001) between the first year of the bachelor’s degree (5), with the least anger, and the second year of the bachelor’s degree (6), with the greatest anger.
-
Disgust: There were significant differences between the first year of secondary school (1) and the first (5) and second years of the bachelor’s degree (6), both comparisons with a significance of p < 0.001. Students in the first year of secondary school had the lowest disgust and those in the second year of the bachelor’s degree had the highest.
-
Fear: No significant differences were observed between the groups.
-
Happiness: Significant differences were found between the first year of secondary school (1) and the first (5) (p < 0.001) and second year (6) (p < 0.001) years of the bachelor’s degree. Students in the first year of the bachelor’s degree displayed lower happiness compared to those in the other two academic years.
-
Sadness: There were significant differences between all pairs of academic years (p < 0.001). Students in the first year of secondary school (1), who exhibited the greatest level of sadness, differed significantly from those in the first (2) and second (6) years of the bachelor’s degree; and to a lesser extent, there was a significant difference between those in the first and second years of the bachelor’s degree, with greater sadness among those in the second year.
-
Surprise: Significant differences were observed between the first year of secondary school (1) and the first year of the bachelor’s degree, with greater surprise among students in the latter. Similarly, differences existed between the first and second years of the bachelor’s degree, with greater surprise among students in the first year.
Table 6. Kruskal–Wallis results for emotions by subjects.
Table 6. Kruskal–Wallis results for emotions by subjects.
EmotionFactorStatisticdfp
AngerSubject1915.6812<0.001
DisgustSubject592.1342<0.001
FearSubject648.5362<0.001
HappinessSubject449.2892<0.001
SadnessSubject1813.7242<0.001
SurpriseSubject18.8572<0.001
After detecting significant differences between groups using Kruskal–Wallis analysis, post hoc comparisons were carried out using Dunn’s test with Bonferroni adjustment to assess specific differences between pairs of groups. The results highlighted the following:
-
Disgust: Differences between all subject pairs (p < 0.001). The greatest disgust was in Tech, followed by Glob and, finally, Green.
-
Fear: Differences between subject pairs Glob and Green (p < 0.001) and Green and Tech (p < 0.001). The greatest fear was found in Green.
-
Happiness: Differences between all subject pairs (p < 0.001). The greatest happiness score appeard in Tech.
-
Sadness: Differences between all subject pairs (p < 0.001). The highest sadness score was in Green.
-
Surprise: Differences between all subject pairs (p < 0.001). The highest surprise score was in Tech.

4. Discussion

In this research, it was observed that students experience many emotions throughout a class [46]. Emotions significantly influence our cognitive functions [47], linked to cognitive skills such as attention, working memory, planning, decision-making, critical thinking, problem-solving, and reasoning [48]. However, we did not find a clear pattern for associating emotions with a subject, generating results that may initially seem contradictory. For instance, a student may be frustrated by their lack of understanding of a subject, and stress can either enhance or hinder learning and memory, depending on its intensity and duration. This study did not consider non-academic factors that could affect the emotional experience in the classroom. These factors include physical or mental conditions, events before class, influence from close peers, and expectations of meeting someone.
In terms of learning performance, pleasant emotions, such as enjoyment of learning, have been correlated with better performance on placement tests [49]. In addition, research has shown that emotional–psychological satisfaction is a determinant variable in students’ academic performance. Regarding the presence of emotions in different school subjects, in technology, we observed greater emotions, as well as positive emotions that would promote learning such as surprise and happiness. Although these results are not conclusive, it is important to note that emotions affect learning in science, depending on the subject matter. In terms of negative emotions, it has been found that in physics and chemistry subjects, similar to technology, students show low interest, likely as they consider those to be difficult, boring, or useless subjects [50]. In secondary education, emotions were more positive towards natural sciences and more negative towards physics and chemistry [51]. Similarly, Dávila [52] pointed out that students in compulsory secondary education often experience negative emotions such as boredom, nervousness, and worry when learning physics and chemistry. However, it is important to consider that embracing pessimism or seriousness can be beneficial for analytical and quantitative tasks [27]. In the Green project, which required a more practical and holistic learning approach than technology, there was greater fear and sadness, which theoretically would not encourage learning. However, in order to fully interpret this outcome, we need to consider the context, external factors, and personal variables, rather than just the subject. Nonetheless, it seems that the ability to detect and understand emotions in the classroom context offers the potential to improve pedagogical practices, especially in subjects like the Green project, which may mean education on sustainability can become more effective.
In regard to the moment an emotion arose during the lesson, we found more fear at the beginning of the class than at the end. We propose that this was due to students adapting to their school environment and their initial lack of knowledge about what to expect from the class and teacher. Likewise, we found greater happiness at the beginning of class compared to the end, which may be due to the excitement of joining peers and the positive expectation of learning new things, which may diminish as the demands of academic tasks progress.
In terms of the relationship between emotions and academic years, no definite pattern was evident, with pleasant and unpleasant emotions existing simultaneously in different academic years. In the first year of secondary education, we observed increased anger and sadness, but also more joy; in the bachelor’s degree, we observed greater disgust and surprise. Based on these findings, we propose that emotions may be more connected to the subject than the academic year.
Nevertheless, the school and classroom environment are important factors influencing achievement emotions [53]; furthermore, it can be assumed that classmates play an important role in affecting students’ achievement emotions. Similarly, it can be expected that students’ valuation of subjects is influenced by parents who value a subject highly or like a particular subject, and by teachers who teach a subject with enthusiasm. In this regard, our results show individual differences in emotional experiences in teaching and subjects, but these results are sample-specific; therefore, more research is needed.
This study is connected to goals 3 and 4 of the Sustainable Development Goals (SDGs) due to its interdisciplinary nature. Health and Well-being Goal 3 comprises ensuring healthy lives and promoting well-being for everyone, at all ages. In this sense, our developed tools have the potential to be used to prevent mental health problems, as they allow for the early identification of negative and positive emotions. Our research also contributes to promoting a healthy school environment by considering students’ emotions. Meanwhile, Goal 4, Quality Education, focuses on achieving inclusive, equitable, and quality education for all individuals and encouraging lifelong learning opportunities. In this regard, understanding emotions can make it easier to support the participation of all students, which is essential to ensuring that everyone has equal opportunities to engage in and contribute to learning.
There are several implications of this study, which could affect the entire educational community: students, teachers, administrators, families, and mental health professionals. For a start, our findings extend the latest understanding of how emotions affect the learning experience of each student in different disciplines. Accordingly, they may help educators adapt their teaching methods to address students’ emotional needs. The findings also identify factors in each subject or moment during the lesson that relate to certain emotions, which may facilitate the implementation of strategies to mitigate emotions considered unpleasant and encourage pleasant ones. An additional implication of this study is that it may affect how the classroom’s emotional climate is assessed, potentially in terms of developing future monitoring tools. Alternatively, the findings presented herein demonstrate that the tools we utilized could be applied with precision in other secondary schools, as the system is based on a simple and successfully tested code. We wish to highlight though that the assessment of their advantages in terms of ethical considerations rests with the appropriate experts, despite our evidence of privacy preservation.
This study’s limitations include how the precision of emotion monitoring via camera can be influenced by environmental conditions, camera quality, and student movement. Furthermore, emotions are complex phenomena that can be produced not only by the classroom context but also by subject content or by prior, external, or personal factors. Related to this, emotional responses to an academic situation differ for each individual. However, here, we have tried to find general tendencies. A final limitation to note is that the algorithm categorises emotions but does not assign them to a specific individual. Although this aspect fully protects the students’ privacy, it does not allow for monitoring emotions throughout the sessions or relating them to other variables, such as academic performance.

5. Conclusions

This study’s purpose has been achieved as we have provided evidence that technology can serve as a valuable tool to support the teaching–learning process, prioritising the emotional well-being of students. Furthermore, the concrete aims of this study—(1) to analyse the students’ manifestations of emotions in the classroom, (2) compare the emotions at the beginning and at the end of the class, and (3) relate the different emotions to the subject and academic year—have also been fulfilled, at least partially. However, we have not obtained conclusive results for the second and third objectives because some results seem inconclusive or contradictory. Perhaps this is because we have evidenced the complexity of the emotional phenomena and/or perhaps it is because more data are needed to analyse and find patterns that more clearly relate emotions to the moment in the lesson, academic year, and subject.
From a technological point of view, this study makes a useful contribution in that we developed and applied an innovative code system to detect students’ emotions during class. The system uses a vision-based model, with a webcam that records the class and then a developed code that detects and analyses the students’ facial expressions, categorising them into one of six basic emotions or a neutral emotion. The developed emotional expression recognition software is sufficiently accurate to identify the emotions of the learners, though it is only possible to obtain suitable images of the participants in close-up and when they are looking fully or partially at the camera.
Future research should focus on specific ways to improve the integration and effectiveness of emotion monitoring in the classroom, considering ethics. It would be useful to develop an interactive tool for teachers, such as by investigating how to design interfaces and tools that enable teachers to interpret and use emotional information effectively in the classroom, with the aim of improving students’ well-being and performance, or developing systems that provide specific suggestions (change in methodology, personalised attention) for how to address identified emotional needs.
Emotion monitoring is set to continue to grow as part of a holistic approach to education that considers academic development but also the well-being of students and their readiness to contribute to a more sustainable society, while empowering the students themselves.

Author Contributions

Conceptualisation, R.P.; methodology, G.F.; software, C.L.; validation, C.L. and G.F.; formal analysis, G.F.; investigation, C.L. and G.F.; resources, C.L.; data curation, G.F.; writing—original draft preparation, C.L. and G.F.; writing—review and editing, R.P.; visualisation, C.L. and G.F.; supervision, R.P.; project administration, R.P. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

This study was conducted in accordance with the Declaration of Helsinki and approved by the Ethics Committee of Universitat Rovira i Virgili (CEIPSA-2021-TD-0019). Date of approval: 2022/02/08.

Informed Consent Statement

Informed consent was obtained from all subjects involved in this study.

Data Availability Statement

Data are unavailable due to privacy or ethical restrictions.

Acknowledgments

We wish to acknowledge NAIST (Nara Institute of Science and Technology), specifically Hiroki Tanaka, Kana Myamoto, Kazuhiro Shidara, Takeshi Saga, and Satoshi Nakamura.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Darwin, C.R. The Expression of the Emotions in Man and Animals, 1st ed.; John Murray: London, UK, 1872. [Google Scholar]
  2. James, W. The Principles of Psychology; Henry Holt and Co.: New York, NY, USA, 1890; Volume II. [Google Scholar]
  3. Lange, C. The Emotions; William and Wilkins: Baltimore, MD, USA, 1885. [Google Scholar]
  4. Freud, S. A Project for a Scientific Psychology, Standard Edition; Hogarth: London, UK, 1895; Volume 1, pp. 283–397. [Google Scholar]
  5. Lazarus, R.S. Thoughts on the Relations between Emotion and Cognition. Am. Psychol. 1982, 37, 1019–1924. [Google Scholar] [CrossRef]
  6. Fuchs, T. Depression, intercorporeality, and interaffectivity. J. Conscious. Stud. 2013, 20, 219–238. [Google Scholar]
  7. Ekman, P.; Friesen, W.V. Measuring facial movement. Environ. Psychol. Nonverbal. Behav. 1976, 1, 56–75. [Google Scholar] [CrossRef]
  8. Taype-Huarca, L.A.; Zavalaga-del Carpio, A.L.; Fernández-González, S.V. Usefulness of the Perezhivanie Construct in Affectivity and Learning: A Systematic Review. In Affectivity and Learning: Bridging the Gap Between Neurosciences, Cultural and Cognitive Psychology; Springer: Berlin/Heidelberg, Germany, 2023. [Google Scholar]
  9. Goleman, D. Emotional Intelligence; Bantam Books, Inc.: New York, NY, USA, 1995. [Google Scholar]
  10. Damasio, A. Descartes Error: Emotion, Reason, and the Human Brain; Avon Books: New York, NY, USA, 1994. [Google Scholar]
  11. UNESCO. Educación para el Desarrollo Sostenible: Hojas de ruta; Organización de las Naciones Unidas para la Educación, la Ciencia y la Cultura, Francia: Paris, France, 2020. [Google Scholar]
  12. Bower, G.H.; Forgas, J.P. Mood and social memory. In Handbook of Affect and Social Cognition; Forgas, J.P., Ed.; Lawrence Erlbaum Associates Publishers: Mahwah, NJ, USA, 2001; pp. 95–120. [Google Scholar]
  13. Cristóvão, A.M.; Valente, S.; Rebelo, H.; Ruivo, A.F. Emotional education for sustainable development: A curriculum analysis of teacher training in Portugal and Spain. Front. Educ. 2023, 8, 1165319. [Google Scholar] [CrossRef]
  14. Minsky, M. The Emotion Machine: Commonsense Thinking, Artificial Intelligence, and the Future of the Human Mind; Simon & Schuster: New York, NY, USA, 2007. [Google Scholar]
  15. Pekrun, R. The Control-Value Theory of Achievement Emotions: Assumptions, Corollaries, and Implications for Educational Research and Practice. Educ. Psychol. Rev. 2006, 18, 315–341. [Google Scholar] [CrossRef]
  16. Linnenbrink-Garcia, L.; Patall, E.A.; Pekrun, R. Adaptive Motivation and Emotion in Education: Research and Principles for Instructional Design. Policy Insights Behav. Brain Sci. 2016, 3, 228–236. [Google Scholar] [CrossRef]
  17. Stoeber, J.; Pekrun, R. Advances in test anxiety research. Anxiety Stress Coping 2004, 17, 205–211. [Google Scholar] [CrossRef]
  18. Davidson, R.J.; Jackson, D.C.; Kalin, N.H. Emotion, plasticity, context, and regulation: Perspectives from affective neuroscience. Psychol. Bull. 2000, 126, 890–909. [Google Scholar] [CrossRef]
  19. Rothbart, M.K.; Bates, J.E. Temperament. In Handbook of Child Psychology, 6th ed.; Eisenberg, N., Damon, W., Lerner, R.M., Eds.; Social, Emotional, and Personality Development; Wiley: New York, NY, USA, 2006; pp. 99–166. [Google Scholar] [CrossRef]
  20. Fredrickson, B.L. What good are positive emotions? Rev. Gen. Psychol. 1998, 2, 300–319. [Google Scholar] [CrossRef]
  21. Fredrickson, B.L. The role of positive emotions in positive psychology. The broaden-and-build theory of positive emotions. Am. Psychol. 2001, 56, 218–226. [Google Scholar] [CrossRef]
  22. Blair, C. School readiness: Integrating cognition and emotion in a neurobiological conceptualization of children’s functioning at school entry. Am. Psychol. 2002, 57, 111–127. [Google Scholar] [CrossRef] [PubMed]
  23. Carver, C.S.; Scheier, M.F. Origins and functions of positive and negative affect: A control-process view. Psychol. Rev. 1990, 97, 19–35. [Google Scholar] [CrossRef]
  24. Goetz, T.; Pekrun, R.; Hall, N.; Haag, L. Academic emotions from a social-cognitive perspective: Antecedents and domain specificity of students’ affect in the context of Latin instruction. Br. J. Educ. Psychol. 2006, 76, 289–308. [Google Scholar] [CrossRef] [PubMed]
  25. Ahmed, W.; van der Werf, G.; Minnaert, A. Emotions, Self-Regulated Learning, and Achievement in Mathematics: A Growth Curve Analysis. J. Educ. Psychol. 2013, 105, 150–161. [Google Scholar] [CrossRef]
  26. Efklides, A. Interactions of metacognition with motivation and affect in self-regulated learning: The MASRL model. Educ. Psychol. 2011, 46, 6–25. [Google Scholar] [CrossRef]
  27. Clore, G.L.; Schiller, A.J.; Shaked, A. Affect and cognition: Three principles. Curr. Opin. Behav. Sci. 2018, 19, 78–82. [Google Scholar] [CrossRef] [PubMed]
  28. Zhou, T.; Lu, Y.; Wang, B. Integrating TTF and UTAUT to explain mobile banking user adoption. Comput. Hum. Behav. 2010, 26, 760–767. [Google Scholar] [CrossRef]
  29. Teimouri, Y. L2 selves, emotions, and motivated behaviors. Stud. Second. Lang. Acquis. 2017, 39, 681–709. [Google Scholar] [CrossRef]
  30. Ormrod, J.; Anderman, E.; Anderman, L. Educational Psychology: Developing Learners, 10th ed.; Pearson Education: London, UK, 2019. [Google Scholar]
  31. Saeidi, M.; Jabbarpour, N. EFL Teachers Socio-Affective Strategy use in relation to students’ academic achievement. Int. J. Acad. Res. 2011, 3, 476–750. [Google Scholar]
  32. Fried, L. Teaching Teachers about Emotion Regulation in the Classroom. Aust. J. Teach. Educ. 2011, 36, 117–127. [Google Scholar] [CrossRef]
  33. Bradley, M.; Lang, P. Measuring emotion: The self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 1994, 25, 49–59. [Google Scholar] [CrossRef] [PubMed]
  34. Harmon-Jones, C.; Bastian, B.; Harmon-Jones, E. The Discrete Emotions Questionnaire: A New Tool for Measuring State Self-Reported Emotions. PLoS ONE 2016, 11, e0159915. [Google Scholar] [CrossRef] [PubMed]
  35. Wang, J.; Liu, Y.; Wang, Y.; Mao, J.; Yue, T.; You, F. SAET: The Non-Verbal Measurement Tool in User Emotional Experience. Appl. Sci. 2021, 11, 7532. [Google Scholar] [CrossRef]
  36. Kim, C.-M.; Hong, E.J.; Chung, K.; Park, R.C. Driver Facial Expression Analysis Using LFA-CRNN-Based Feature Extraction for Health-Risk Decisions. Appl. Sci. 2020, 10, 2956. [Google Scholar] [CrossRef]
  37. Landowska, A. Emotion Monitoring—Verification of Physiological Characteristics Measurement Procedures. Metrol. Meas. Syst. 2014, 21, 719–732. [Google Scholar] [CrossRef]
  38. Brouwer, A.M.; Zander, T.O.; Van Erp, J.B.; Korteling, J.E.; Bronkhorst, A.W. Using neurophysiological signals that reflect cognitive or affective state: Six recommendations to avoid common pitfalls. Front. Neurosci. 2015, 30, 136. [Google Scholar] [CrossRef] [PubMed]
  39. Chen, J.; Li, H.; Ma, L.; Soong, F. DEEMD-SPP: A Novel Framework for Emotion Recognition Based on EEG Signals. Front. Psychiatry 2022, 13, 885120. [Google Scholar] [CrossRef]
  40. Llurba, C.; Fretes, G.; Palau, R. Pilot study of real-time Emotional Recognition technology for Secondary school students. Interact. Des. Archit. 2022, 52, 61–80. [Google Scholar] [CrossRef]
  41. Zhu, Y.; Li, Y.; Yang, Y.; Zhai, X. Intelligent technology drives higher education reform: Key points and reflections on the 2023 Horizon report: Teaching and learning edition. Open Educ. Res. 2023, 29, 19–30. [Google Scholar]
  42. Rößler, J.; Sun, J.; Gloor, P. Reducing Videoconferencing Fatigue through Facial Emotion Recognition. Future Internet 2021, 13, 126. [Google Scholar] [CrossRef]
  43. Krumhuber, E.; Skora, L.; Hill, H.; Lander, K. The role of facial movements in emotion recognition. Nat. Rev. Psychol. 2023, 2, 283–296. [Google Scholar] [CrossRef]
  44. Ekman, P.; Friesen, W.V. Facial Action Coding System (FACS) [Database record]. APA PsycTests. 1978. [Google Scholar] [CrossRef]
  45. Vu, T.; Huỳnh, V.T.; Kim, S. Vision Transformer for Action Units Detection. arXiv 2023, arXiv:2303.09917. [Google Scholar] [CrossRef]
  46. Pekrun, R.; Stephens, E.J. Academic emotions. In APA Educational Psychology Handbook. Individual Differences and Cultural and Contextual Factors; Harris, K.R., Graham, S., Urdan, T., Graham, S., Royer, J.M., Zeidner, M., Eds.; American Psychological Association: Washington, DC, USA, 2012; Volume 2, pp. 3–31. [Google Scholar] [CrossRef]
  47. Tyng, C.M.; Amin, H.U.; Saad, M.N.M.; Malik, A.S. The influences of emotion on learning and memory. Front. Psychol. 2017, 8, 1454. [Google Scholar] [CrossRef]
  48. Cristofori, I.; Cohen-Zimerman, S.; Grafman, J. Executive Functions. In The Frontal Lobes. Handbook of Clinical Neurology; D’Esposito, M., Grafman, J., Eds.; Elsevier: Amsterdam, The Netherlands, 2019; Volume 163, pp. 197–219. [Google Scholar] [CrossRef]
  49. De la Fuente, J.; Sander, P.; Cardelle-Elawar, M.; Pignata, S. Effects of level of regulatory teaching on achievement emotion in the learning process: Anxiety and achievement emotions on higher education. In Teaching and Learning; Vargas, M., Ed.; Nova Science Publishers, Inc.: New York, NY, USA, 2016; pp. 131–151. [Google Scholar]
  50. Solbes, J. Revisiting the teachers’ lounge: Reflections on emotional experience and teacher identify. Teach. Teach. Educ. 2011, 26, 616–621. [Google Scholar] [CrossRef]
  51. Brígido, M.; Couso, D.; Gutiérrez, C.; Mellado, V. The Emotions about Teaching and Learning Science: A Study of Prospective Primary Teachers in Three Spanish Universities. J. Balt. Sci. Educ. 2013, 12, 299–311. [Google Scholar] [CrossRef]
  52. Dávila, M.A. Las emociones y sus causas en el aprendizaje de Física y Química, en el alumnado de Educación Secundaria. Rev. Eureka Sobre Enseñanza Y Divulg. De Las Cienc. 2017, 14, 570–586. [Google Scholar] [CrossRef]
  53. Gläser-Zikuda, M.; Stuchlíková, I.; Janík, T. Theoretical and research papers. Emotional Aspects of Learning and Teaching: Reviewing the Field—Discussing the Issues. Orb. Sch. 2013, 7, 7–22. [Google Scholar] [CrossRef]
Sustainability 16 00916 g001
Figure 1. Classroom arrangement.
Figure 1. Classroom arrangement.
Sustainability 16 00916 g001
Sustainability 16 00916 g002
Figure 2. Students attending class.
Figure 2. Students attending class.
Sustainability 16 00916 g002
Sustainability 16 00916 g003
Figure 3. Sample of the code in Command Prompt.
Figure 3. Sample of the code in Command Prompt.
Sustainability 16 00916 g003
Sustainability 16 00916 g004
Figure 4. Sample of the code in Visual Studio Code.
Figure 4. Sample of the code in Visual Studio Code.
Sustainability 16 00916 g004
Sustainability 16 00916 g005
Figure 5. Spearman’s rho heatmap. Note: Pink tones represent negative correlations between variables, and blue tones positive ones. The intensity of the colour shows the strength of the correlation: the paler shades represent the weakest correlations and the darkest represent the strongest. Asterisks represent different significance levels associated with p-values. Two asterisks (**) p < 0.01, and three asterisks (***) p < 0.001.
Figure 5. Spearman’s rho heatmap. Note: Pink tones represent negative correlations between variables, and blue tones positive ones. The intensity of the colour shows the strength of the correlation: the paler shades represent the weakest correlations and the darkest represent the strongest. Asterisks represent different significance levels associated with p-values. Two asterisks (**) p < 0.01, and three asterisks (***) p < 0.001.
Sustainability 16 00916 g005
Sustainability 16 00916 g006
Figure 6. Emotional changes at the start and end of class.
Figure 6. Emotional changes at the start and end of class.
Sustainability 16 00916 g006
Table 1. Students involved in the experiment.
Table 1. Students involved in the experiment.
StudentsLevelClass
16First-year secondary school2 groups
24Fourth-year secondary school1 group
24First-year bachelor’s degree2 groups
12Second-year bachelor’s degree1 group
Table 2. Example of some of the emotions detected among a class.
Table 2. Example of some of the emotions detected among a class.
AngerDisgustFearHappinessSadnessSurpriseNeutral
0.031770.00240.138080.122030.185580.0822040.43848
0.002330.00020.218800.395560.063040.0283360.29173
0.000450.00050.097310.021840.238040.1913830.45042
0.004940.00260.230400.154940.249080.1320830.22589
Table 3. Statistics of emotions at the beginning and end of the class.
Table 3. Statistics of emotions at the beginning and end of the class.
Descriptive Statistics
AngerDisgustFearHappinessSadnessSurprise
Mode0.004 a9.359 × 10−4 a0.036 a0.023 a0.046 a0.039
Median0.0110.0020.1200.0560.1220.160
Mean0.0810.0120.1460.1120.1630.223
Std. D.0.1720.0470.1220.1510.1400.211
Min.2.410 × 10−51.920 × 10−51.800 × 10−41.680 × 10−42.674 × 10−46.730 × 10−5
Max.0.9950.8760.8680.9420.9330.950
25th percentile0.0039.746 × 10−40.0540.0250.0530.051
50th percentile0.0110.0020.1200.0560.1220.160
75th percentile0.0580.0060.2030.1260.2370.332
a More than one mode exists. For nominal and ordinal data, the first mode is reported. For continuous data, the mode with the highest density estimated is reported, but multiple modes may exist. We recommend visualising the data to check for multimodality.
Table 4. Results of the Mann–Whitney U test.
Table 4. Results of the Mann–Whitney U test.
Independent-Samples t-Test
Wdfp
Anger1.607 × 106 0.002
Disgust1.599 × 106 0.004
Fear1.634 × 106 <0.001
Happiness1.658 × 106 <0.001
Sadness1.425 × 106 0.003
Surprise1.488 × 106 0.413
Neutral1.448 × 106 0.028
Note. Mann–Whitney U test.
Table 5. Kruskal–Wallis results for emotions by academic year.
Table 5. Kruskal–Wallis results for emotions by academic year.
EmotionFactorStatisticdfp
AngerAcademic year215.7112<0.001
DisgustAcademic year222.7082<0.001
FearAcademic year13.6272<0.001
HappinessAcademic year556.3202<0.001
SadnessAcademic year190.0682<0.001
SurpriseAcademic year127.0042<0.001
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Llurba, C.; Fretes, G.; Palau, R. Classroom Emotion Monitoring Based on Image Processing. Sustainability 2024, 16, 916. https://doi.org/10.3390/su16020916

AMA Style

Llurba C, Fretes G, Palau R. Classroom Emotion Monitoring Based on Image Processing. Sustainability. 2024; 16(2):916. https://doi.org/10.3390/su16020916

Chicago/Turabian Style

Llurba, Cèlia, Gabriela Fretes, and Ramon Palau. 2024. "Classroom Emotion Monitoring Based on Image Processing" Sustainability 16, no. 2: 916. https://doi.org/10.3390/su16020916

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop