*Article* **Effect of Adding Emotion Recognition to Film Teaching—Impact of Emotion Feedback on Learning through Puzzle Films**

**Shang-Chin Tsai \* and Hao-Chiang Koong Lin**

Department of Information and Learning Technology, National University of Tainan, Tainan City 70005, Taiwan; koong@gm2.nutn.edu.tw

**\*** Correspondence: maneytsai@gmail.com

**Abstract:** In this study, the scientific puzzle film, "Story of the Comet", is taken as a case to implement scientific teaching to guide students to find correct answers, through which it can train their learning and judging abilities. The students in the experimental group received the scientific teaching guiding system of the puzzle film "Story of the Comet" with a facial emotion recognition system to recognize the emotional reaction of the subjects at the moment. According to their facial expressions of "disgust", "sadness", or "joy" appearing in the moment, the system presented differently captioned positive encouragement cards particularly designed for four different levels, for when the subjects answered the questions incorrectly at different levels and their emotions were detected at the same time. Furthermore, the positive encouragement cards encouraged the subjects to complete the puzzle film learning process. The subjects were students in the higher grades of Grade 5 and Grade 6 in elementary school. A total of 130 students participated in this experiment and were randomly divided into two groups. Both the control group (i.e., the group without emotion recognition) and the experiment group (i.e., the group with emotion recognition) received a before-watching test of learning effectiveness. After implementing the scientific teaching of the puzzle film "Story of the Comet", both the control group and the experimental group also received an after-watching test of learning effectiveness. Finally, the subjects filled out a "learning satisfaction" questionnaire, "system availability" questionnaire, and "system satisfaction" questionnaire. The analysis of the results of the two groups' tests and questionnaires: a comparative analysis of learning effectiveness indicates that there is a statistically significant difference between the choice answers of the two groups after the interactive teaching; for the experimental group, the average correct answers in the after-watching test was 5.86, which is 2.48 more than the before-watching test; that of the control group was 4.74, which is 1.47 more than the before -watching test. For comparative analysis of questionnaires for "learning satisfaction" and "system satisfaction", the statistical data analysis indicates that the experimental group was more satisfied than the control group.

**Keywords:** emotion recognition; puzzle film; learning satisfaction; emotion feedback; learning effectiveness

### **1. Introduction**

In this era, in public and private universities in Taiwan, despite teachers' great flow of speech on the podium, students in their seats usually mind their own business, chatting, sleeping, eating bento, playing mobile phones and video games, and watching movies; few students listen to teachers attentively [1]. The inattention of students at different levels, including elementary, junior high, and senior high schools, should not vary much. According to the concentration survey of Parenting magazine, more than 90% of teachers from elementary and middle schools believe that students do not concentrate enough, while "cannot understand contents in class", "teaching methods cannot attract students", and "teaching contents are too simple, which they have learned in continuation class and

**Citation:** Tsai, S.-C.; Lin, H.-C.K. Effect of Adding Emotion Recognition to Film Teaching—Impact of Emotion Feedback on Learning through Puzzle Films. *Sustainability* **2021**, *13*, 11107. https://doi.org/10.3390/su131911107

Academic Editor: Carlos Salavera

Received: 20 August 2021 Accepted: 4 October 2021 Published: 8 October 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

daycare centers" are the three major reasons for their inattention [2]. Therefore, learning effectiveness is very poor. How to raise students' interest in learning and guide them to study attentively is an important topic of the education circle. As positive emotion improves students' learning interest and motivation [3], by using the function of emotion detection and adding emotional feedback through encouragement cards, this study explored whether the addition of timely encouragement cards can improve the learning effectiveness of students.

### *Research Question*


### **2. Literature Review**

With the rapid development of science and technology and the innovation of teaching aids, the narrative teaching method in the classroom is no longer favored by students, and the visualization of an image is better than purely text description in cognition [4,5]; in the following four teaching methods—(1) digital learning by an interactive film; (2) digital learning by a non-interactive film; (3) digital learning without using film; (4) traditional teaching in class—comparing learning achievements based on the four different learning environments and learning equipment, results confirmed that students viewing interactive films in a digital learning environment achieved significantly better learning results and higher satisfaction [6]. With both sound and light effects, films are an excellent instrument to attract students' attention. The audio–visual experience of film can improve students' awareness and encourage critical thinking skills [7]. On the other hand, in addition to the function of an oral account, films allow the teacher to replay and pause, which makes them a good medium to promote learning [8–10]. Learning effectiveness is an indicator for judging students' learning outcomes. The purpose of evaluating effectiveness is to let students become aware of their own learning situation and to serve as a basis for teachers to improve teaching and students to improve learning. [11] Learning effectiveness is a measure of the performance indicators or a behavior change after a student participates in a learning activity [11,12]. Generally, we can use two data to measure the effect of learning. The first is academic achievement, such as semester grade or test score [13,14], and the second is learning satisfaction [15–17]. Correspondingly, Huang et al. [18], they also use academic achievement and learning satisfaction as the two criteria to measure the learning effect of students. Learning satisfaction can be regarded as the learner's feelings about learning activities [19,20], or the learner's attitude [19] or sense of pleasure [21]. Piccoli et al. [17] and Maki et al. [16] believe that learning satisfaction expresses learners' satisfaction with the learning process and learning results. Therefore, learning satisfaction is a very suitable criterion for evaluating learners' satisfaction with classroom learning. As a result, we can further understand the learning effect of students based on academic achievement and learning satisfaction [22].

With the rapid changes of the times, it is more important to use effective ways to enable learners to quickly absorb the information they have learned in this fast-developing environment. Because of this, digital learning has stood out from the digitization of traditional methods. So far, it has evolved from inefficient computer-aided teaching to intelligent guidance learning through intelligent classrooms and mobile devices (digital learning by mobile equipment) [23]. In addition, studies found that observation and recognition of the learners' emotional status enable teachers to adopt actions of significant

impact on teaching quality and execution to a large extent and interact with students in more humanized ways. Therefore, an intelligent teaching system should consider relations among emotions, perceptions, and behaviors in the learning environment [24]. It is very important to explore students' emotions, as there are inextricably connections between emotion and perception, which facilitates effective learning. The study also found that perception plays a role in emotion generation. Positive emotions can actually improve students' learning interest and motivation [3]. Therefore, an emotion recognition system can help teachers interact fully and accurately so as to encourage students to learn actively. In this way, teachers will have the opportunity to monitor and propose appropriate expression models, especially when dealing with students' negative emotions. Therefore, "effective emotion detection" allows the design of modular and reusable activities suitable for students' learning styles, which can provide stronger and powerful educational activity planning [25]. According to the studies of Elias and Weissberg [26]; and Payton, Wardlaw, Graczyk, Bloody, Trompsett, and Weissberg [27], if children can use emotional intelligence in class, they can achieve better results in tests or other forms of academic performance. In addition, the study finds the relationship between emotional factors and students' academic performance, which believes that students with healthy emotions have the opportunity to succeed in college, and one of the basic mechanisms of learning is students' active participation. To be more exact, when a student begins to actually engage in the learning process, the student will usually be more active rather than assume passivity or total acceptance. In view of this, how to help students speed up their learning and obtain better learning results have always been the research topic of many researchers who are concerned about this issue.

Film teaching materials integrate various media materials such as images, pictures, music, sounds, and words to present information in an appropriate way and appearance. Therefore, it is normal to use films as a teaching aid in school teaching or in the daily work of industry, commerce, and society. As a cognitive theory, dual coding theory was proposed by Allan Paivio of the University of Western Ontario in 1971. Paivio believed that the "formation of imagery is beneficial to learning". According to Paivio, people can use "verbal association" and "visual imagery" to further remember what they have learned [28]. Dual coding theory proposes that both visual information and verbal information are the way to express information, which is different in their processing methods. They are processed in different blocks of the brain, and information processed by each channel creates its own unique representation. When learning information, if people use visual code and verbal code at the same time, that is, generate a relevant image in visual code and link it with a verbal code, such an ability to use the two different codes improves the chances of remembering the item [29]. In addition, some people like to receive information through visual modes such as charts and illustrations, while others like to obtain information by reading or listening to verbal information; Richardson [30] called the former a "visualizer", while the latter is a "verbalizer". Through the study, Richardson found that there are obvious differences between visualizers and verbalizers in many characteristics; visualizers tend to think specifically, while verbalizers tend to think with more abstract symbols; visualizers like to think with physical images or mental images. Their thinking mode is subjective and self-oriented, while the thinking mode of verbalizers is objective and task-oriented.

According to the Cognitive Theory of Multimedia Learning of Mayer [31–33], there are two separate channels (auditory and visual) for processing information; there is limited channel capacity; and learning is an active process of filtering, selecting, organizing, and integrating information. Mayer points out that "people learn more from words and pictures" "rather than just words" [34,35]. However, only adding words to pictures is not an effective way to achieve this goal. The purpose of multimedia learning is to make teaching media according to people's way of thinking. This is the basis of Mayer's cognitive theory of multimedia learning. When it comes to multimedia learning, the theory puts forward three main assumptions:


Image, sound, action, and plots are the four characteristics of films. Combining these characteristics, it becomes a powerful force, which is far more than the media effect of pure text or ordinary text plus pictures. The advantage of film is that it has sound and light effects at the same time, giving viewers extreme sensory stimulation. Therefore, it is undeniable that film itself has a unique charm in attracting learners' attention. Furthermore, due to the development of digital audio–visual technology and the population of networks, digital audio–visual information is everywhere in our daily life. Many teachers have used film as an auxiliary tool for teaching in school classes, but the format of films is mostly in a one-way communication mode, that is, relying on the authors' intention to produce and play the content of the film, and the audience is expected to watch and accept the content. However, in the digital era, the one-way communication mode cannot meet the information audio–visual needs of the audiences. Responding to this trend, producers integrate new media materials and new technology and produce two-way films with interactive characteristics, through which users can participate and even dominate the content. In recent years, interactive films have been widely used in advertising, marketing, education, publicity, navigation, and other aspects of daily life. Interactive films have become an efficient instrument in education and communication.
