MultiFusedNet: A Multi-Feature Fused Network of Pretrained Vision Models via Keyframes for Student Behavior Classification

Abstract

This research proposes a deep learning method for classifying student behavior in classrooms that follow the professional learning community teaching approach. We collected data on five student activities: hand-raising, interacting, sitting, turning around, and writing. We used the sum of absolute differences (SAD) in the LUV color space to detect scene changes. The K-means algorithm was then applied to select keyframes using the computed SAD. Next, we extracted features using multiple pretrained deep learning models from the convolutional neural network family. The pretrained models considered were InceptionV3, ResNet50V2, VGG16, and EfficientNetB7. We leveraged feature fusion, incorporating optical flow features and data augmentation techniques, to increase the necessary spatial features of selected keyframes. Finally, we classified the students’ behavior using a deep sequence model based on the bidirectional long short-term memory network with an attention mechanism (BiLSTM-AT). The proposed method with the BiLSTM-AT model can recognize behaviors from our dataset with high accuracy, precision, recall, and F1-scores of 0.97, 0.97, and 0.97, respectively. The overall accuracy was 96.67%. This high efficiency demonstrates the potential of the proposed method for classifying student behavior in classrooms.

1. Introduction

An essential part of a teacher’s role is the effective communication of the teaching material to students. The conventional approaches to learning, such as rote learning, are ingrained in our educational culture [1] and routinely employed in general classrooms. The classroom activities of the traditional educational methods involve a variety of exercises, including memorizing, creating rules and formulas, and ensuring that pupils complete a large amount of homework or take exams. However, despite the students’ listening to and memorizing an excessive amount of information, the traditional educational method does not result in an increased interest level among students in their studies. Furthermore, these traditional learning techniques could result in additional adverse effects on students, such as a lack of critical thinking abilities, inadequate problem-solving strategies, and strained relationships in small groups or the entire classroom. So, we use the professional learning community (PLC) technique [2] to improve student teaching in a classroom environment and overcome the constraints of the traditional approach [3]. However, when teachers use the PLC approach for schooling, they will observe different student behaviors during the learning process. In this stage, it is challenging to simultaneously distinguish an individual student’s behavior from that of the whole class.

In recent years, artificial intelligence (AI) has become more powerful than ever [4], and many industries, including education, are using it to become leaders in their fields. Many student behavior and classroom assessment researchers have recently created and deployed machine learning and deep learning models [5] to help resolve difficulties such as the difficulties in understanding student behavior, student engagement in the classroom, and students’ emotional facial expressions [6,7,8,9,10,11]. AI technology in the computer vision domain can help teachers analyze student behavior better. Furthermore, this technology can be used to develop an improved teaching approach and learning model that will help students enhance their enthusiasm for studying. In addition, this approach and model improve the capacity of students to solve issues and form stronger relationships among themselves.

While AI can potentially analyze student behavior, limitations persist in this analysis due to data complexity, dynamic classroom environments, and behavior variability. The multifaceted nature of student behavior poses challenges for models to capture behaviors precisely. AI model accuracy is also influenced by classroom size, as larger classes can overwhelm the model’s ability to track individual behaviors effectively. Additionally, varying classroom contexts, such as in terms of teaching approaches, classroom layouts, lighting conditions, and classroom learning culture, can significantly impact student behavior.

To overcome these limitations, we propose AI technology [12], based on a novel hybrid approach involving a combination of convolutional neural networks (CNN) and recurrent neural networks (RNN) [13,14], for the classification of student behavior. Our framework utilizes keyframe selection in video preprocessing and multiple-feature fusion from various pretrained CNN families to extract features from each keyframe and fused features. We then leverage a deep sequence neural network with an attention mechanism to classify student behavior in the classroom. We collected the dataset from mathematics classrooms in a primary school at Sisaket Rajabhat University Demonstration School, Thailand. The student’s behavior dataset consists of five categories: hand raising, interacting, sitting, turning around, and writing. Our work uses a student behavior classification approach that is different from that of other researchers, who typically use behavior detection to alert teachers about inappropriate learning behaviors. Student behavior classification, a critical component of student analysis, is essential for designing learning units in each subject. The use of the PLC approach in the classroom makes it even more essential to analyze student behavior in relation to learning outcomes.

The major contributions of this paper are as follows:

• We have designed an efficient keyframe selection method based on the K-means and the sum of absolute differences (SAD) of the L channel in the LUV color space to extract keyframes from a random length of video clips. The proposed method guarantees the extraction of a unique sequence for any given video clip.
• Four features are fused to characterize the behavior type efficiently. High-level features are extracted from the natural RGB images, while motion features are extracted from the optical flow. To overcome problems regarding various luminance conditions and data deficiency, features from Lab color components and data augmented (DA) are also extracted and fused with the other high-level features. We propose this multi-feature fusion, ensuring that the feature extraction process captures the most relevant information from the video keyframes covering spatial and temporal features.
• To achieve efficient classification, we introduce an innovative hybrid network that combines a bi-directional long short-term memory (BiLSTM) model with an attention mechanism (AT) to classify student behavior in a classroom setting effectively. The BiLSTM component excels at capturing long-range temporal dependencies within the student behavior data. Simultaneously, the integrated AT empowers the model to identify and prioritize the most relevant data segments for precise classification. Our architectural enhancements include the use of 512 LSTM cells, equally distributed into 256 cells each for both forward and backward directions. This architecture is robust and can support 7168 feature dimensions in each of the ten keyframe feature fusions.

The remainder of this paper is organized as follows: Section 2 reviews the related literature regarding student behavior in the classroom. Section 3 describes the materials and methods, including the student behavior data collection, keyframe extraction, and selection methods, the multiple pretrained models for feature extraction and the fusion approach, and the proposed overall framework using a deep sequence model based on the BiLSTM-AT network to classify students’ behaviors. Section 4 describes the experimental setup, dataset setup, and evaluation metrics. Section 5 presents the experimental results and discusses the findings. Section 6 concludes the paper and presents directions for future work.

2. Literature Review

Analyzing student behavior is important for enhancing teaching approaches and improving student learning. In this context, Zheng et al. [15] proposed a system to analyze student behavior by automatically detecting hand-raising, standing, and sleeping behaviors based on object detection. The researchers adopt ResNet-101 into a Faster R-CNN network with different sizes of regions of interest, including low-level and high-resolution feature maps and high-level semantic information. The results showed an mAP of 57.6%, a 3.4% increase in accuracy compared to the original model mAP of 54.2%, and a fast speed. Jisi et al. [16] presented a new feature fusion network that outperforms recognition algorithms for student behavior recognition in education. The network combines spatial affine transformation and CNNs to extract detailed features from video data and uses a weighted sum method to fuse spatial-temporal features for classification. They evaluate the model with the human behavior dataset on HMDB51 and UCF101, with better results than other state-of-the-art recognition algorithms. Hu et al. [17] proposed a bimodal learning engagement recognition method for large-scale classrooms. They used deep learning techniques to recognize three levels of engagement (high, medium, and low) based on emotional and behavioral features. In addition, they designed a bimodal network using ResNet50 and CoAtNet with a KNN classifier and obtained an accuracy of 93.94%. Their method outperformed state-of-the-art techniques.

Zhou et al. [18] proposed a deep learning network for student behavior recognition in the classroom using a 10-layer deep CNN (CNN-10) to extract vital information from the human skeleton. The experimental results showed that their method could effectively exclude irrelevant information such as students’ physique, dress, and classroom background and focus on key effective information such as hands up, head down, listening, and class standing, resulting in higher recognition accuracy. Lin et al. [19] presented a student behavior recognition system based on skeleton pose estimation and person detection using the OpenPose framework. To reduce errors, they applied pose estimation and person detection techniques, followed by skeleton data preprocessing to eliminate several joints with weak effects on behavior. Feature vectors of human postures were extracted from normalized joint locations, joint distances, and bone angles. This work applied the deep neural network constructed to recognize student behaviors, such as looking, asking, bowing, and bored behavior. The scheme outperformed the skeleton-based one in terms of complexity, with an average precision of 15.15% and an average recall of 12.50%. Fu et al. [20] also conducted learning behavior analysis in classroom teaching, which included different behaviors such as listening, fatigue, hand-up, turning sideways, and read-write. They applied OpenPose to extract key points from human-skeleton faces and fingers for detecting the human body. Afterwards, they classify each behavior using the CNN network. The model produces a result with a 92.86% accuracy in a real-life classroom teaching environment.

Huang et al. [21] presented a novel student action recognition method for classrooms using YOLOv3. Their method could accurately and quickly identify student behaviors such as fatigue, hand-ups, bowing, turning sideways, body position, and facial expressions. Zhang et al. [22] proposed an improved YOLOv3 to detect student behavior in the classroom, such as sleeping, using mobile phones, taking notes, and others. The model added an attention mechanism CBAM module to the original network. The experiment presented was quick and effective, and YOLO-CBAM can also detect small targets. The method achieves high accuracy, even though detecting student behavior is challenging. Ren et al. [23] improved YOLOv4 to detect students who sleep or use a smartphone by enhancing a feature extraction network that combined top-down and bottom-up paths. The original YOLOv4 was compared with an improved network (PANet and Faster R-CNN), and the results showed that the enhanced network can achieve better results and is suitable for student detection and recognition tasks. Tang et al. [24] presented a classroom behavior detection algorithm based on an improved YOLOv5 object detection model. They combined a weighted bidirectional feature pyramid network (BiFPN) with the feature pyramid network and the path aggregation network (FPN+PAN) structure of YOLOv5. This allowed them to extract the fine-grained features of various behaviors from different scales of the object. They also added a CBAM to the neck network to improve the detection accuracy. Finally, they improved the original non-maximum suppression algorithm using a distance-based intersection ratio to enhance the discrimination of occluded objects. Yang et al. [25] presented a student classroom behavior detection method based on an improved YOLOv7 to address the low accuracy of the existing methods. To improve the detection accuracy in crowded scenes, they integrated the biformer attention module and Wise-IoU into the YOLOv7 network. Their method achieved a [email protected] of 79% using the SCB dataset, which includes 18.4 k of labels and 4.2 k of images covering three behaviors: hand-raising, reading, and writing. This represents a 1.8% improvement over previous results.

Wang et al. [26] proposed a novel student classroom behavior detection system that combines deformable DETR with a Swin Transformer and a lightweight Feature Pyramid Network (FPN). To address the limitations of CNN-based target detection methods, their system employs a feature pyramid structure and the CARAFE lightweight operator to process multi-scale feature maps effectively and enhance the network’s detection accuracy. This system achieved a significant 6.1% improvement in detection accuracy compared with the original deformable DETR model and outperforms Faster R-CNN, SSD, and YOLOv3 [21], v5 [24], and v7 [25].

According to recent papers, the majority of research has focused on improving accuracy through feature extraction, model combining, or both. For instance, ResNet-101 has been integrated with a Faster R-CNN network to detect student behavior [15]. In addition, the spatial affine transformation network has been combined with CNNs for feature extraction and the detection of student behavior [16]. ResNet50 and CoAtNet have been employed to create a bimodal network for feature extraction [17]. CNN-10, a 10-layer deep CNN, has been used for feature extraction to recognize student behaviors [18], while the OpenPose framework has been utilized to focus on key information [19,20]. YOLOv3 uses multiple convolutional layers to isolate important features [21], and CBAM has been incorporated into YOLOv3 for feature extraction [22]. For enhanced accuracy, a weighted BiFPN combined with FPN+PAN has been used for feature extraction [23], and a CBAM has been added to it [24]. In YOLOv7, the biformer attention module and Wise-IoU have been applied for feature extraction [25]. To enhance feature extraction, the deformable DETR, Swin Transformer, FPN, and the CARAFE operator have been combined [26]. The use of integrated models and combination methods for feature extraction is crucial to improving the accuracy of student behavior detection and classification tasks.

Previous diverse research has demonstrated that a combination of sophisticated feature extraction techniques and model integration strategies can enhance the accuracy of student behavior detection and classification. Our comprehensive analysis of existing approaches found that no single algorithm or model can consistently achieve the highest levels of exactness across all aspects of student behavior analysis. Therefore, it is essential to employ diverse techniques to capture the multifaceted nature of student behavior and achieve the most accurate and comprehensive results.

In addressing the aforementioned limitations and leveraging the strengths of diverse approaches, we apply a critical feature extraction and fusing methodology using multiple pretrained models [27]. To optimize feature extraction, we leverage the unique strengths of each model [28]. We then use deep sequence neural networks to classify and recognize student behavior. While existing research has introduced various methods for feature extraction, a robust feature extraction technique is still required that can be used with our dataset and in a different context. Our work takes a distinctive and innovative approach to enhance the accuracy and efficiency of the feature extraction and classification processes.

3. Data Collection

The dataset was collected in an actual classroom environment [29] using a purposive sampling technique focusing on a first-grade primary school at Sisaket Rajabhat University Demonstration School, Thailand. Specifically, data was obtained for students studying mathematics in the second semester of 2020. The number of students was 24 per classroom, and a PLC teaching approach was used. For the data collection, the educational researcher assigns the teams, designs the layout for the student seating, and sets up a video camera to record all students. The video cameras were placed in front of the whole class on the left and right sides. The top view of the seating layout is depicted in Figure 1a. When a class starts, the photographic specialist records the video until the course ends. Figure 1b shows an actual classroom scene. A SONY Handy Cam 60X (Tokyo, Japan) was used to record the images with a frame rate of 30 frames per second.

The expert analyzed and collected behavioral clips by cropping the original video file and uploading it to the cloud server storage. This research focused on the student’s behavior during lesson activities, and we labeled each clip with one of five categories: hand-raising, interacting, sitting, turning around, and writing. The dataset collection sample for each behavior is shown in Figure 2.

The PLC teaching approach aims to encourage collaboration among educators to enhance teaching, improve student outcomes, and foster continuous improvement within the school. PLCs strive to create an environment that supports educators in optimizing teaching for diverse student needs, promoting academic success through data-driven decisions, ongoing development, and a student-centered focus [2]. By observing how a particular behavior influences and positively impacts students’ learning [1,2,3], teachers gain firsthand insights into the effectiveness of student learning. This observation-based approach encourages teachers to analyze the correlation between student behaviors and the resulting outcomes in students’ understanding. Our collected dataset encompasses five distinct classes of student behaviors observed within a classroom environment employing the PLC teaching method. The classification of student behaviors in our dataset aligns with the core objectives of the PLC [2].

4. Methods and Experiment

4.1. Overview of Framework

Our overall framework can be divided into three parts: pre-processing, pretrained models, and the deep sequence model. In the pre-processing phase, we choose ten keyframes by applying the SADLK algorithm. Next, we use multiple pretrained models for feature extraction based on the feature fusion approach. During this process, we apply data augmentation (DA) to RGB keyframes. The original RGB keyframe extracts the feature from Inception-V3. For keyframe augmentation techniques such as rotation, width shift, height shift, and horizontal flip, we use the ResNet50-V2 for feature extraction. Color injection applies the color transform technique from RGB to L*a*b* color space and extracts the feature by using VGG-16 in the optical flow keyframe using EfficientNet-B7 to extract features. Our overall process pipeline is depicted in Figure 3.

The MultiFusedNet has been designed based on a deep sequence model with the BiLSTM-AT network to classify the students’ behaviors.

Our model consists of 158 million parameters, 14 layers with different output sizes, and trainable parameters. The models use frame features as input and a mask input to handle keyframe sequence features. The layer configuration has two BiLSTMs with attention machine unit layers with 512 and 256 hidden units, each followed by a dropout layer with a 0.3 rate to prevent overfitting. The dropout layers ensure the model learns to rely on the different hidden units, preventing it from becoming too specialized for the training data. In addition, a dense layer with 64 hidden units, relu activation, and L2 regularization are added, followed by a third dropout layer with a 0.3 rate. These additions further improve the model’s generalization capability. The output layer uses softmax activation for multi-class classification. We then feed the features into the BiLSTM-AT network with a 64-batch size. The keyframe selection and multi-feature fusion with the BiLSTM-AT networks are called “KMFF-BiLSTM-AT”, and the same without the keyframe are called “MFF-BiLSTM-AT”. Figure 4 illustrates the overall framework of MultiFusedNet.

4.2. Keyframe Extraction

Analyzing videos often involves processing a large number of frames, many of which may contain redundant information. This leads to a time-intensive undertaking, resulting in low accuracy. Nevertheless, using keyframes [30,31] can substantially enhance the processing speed of video analysis. We adopt the SAD [32] in the LUV color space [33], focusing on the L channel and the K-mean algorithm (SADLK) for the frame extraction pre-processing and select a keyframe before transferring it to the multiple pretraining model for feature extraction. The keyframe selection is performed according to the Algorithm 1 given below [34].

Algorithm 1: SADLK Keyframe Selection
Input:  Video frames ($\mathrm{R}\mathrm{G}\mathrm{B}$)
Keyframe count ($\mathrm{K}$)
Output:  Selected keyframes
Procedure:
1.	Step 1: Convert RGB to LUV in the L channel
2.	For each frame$i$in range($1,\mathrm{N}+1$):
3.	//$\mathrm{Given} \alpha =0.2126729, \beta =0.7151522, \gamma =0.072175$
4.	${\mathrm{Y}}_{\mathrm{i}}=\alpha \cdot {\mathrm{R}}_{\mathrm{i}}+\beta \cdot {\mathrm{G}}_{\mathrm{i}}+\gamma \cdot {\mathrm{B}}_{\mathrm{i}}$
5.	${\mathrm{L}}_{\mathrm{i}}^{\mathrm{*}}=116\cdot \mathrm{f}({\mathrm{Y}}_{\mathrm{i}}/{\mathrm{Y}}_{\mathrm{n}})-16$
6.	// Store ${\mathrm{L}}_{\mathrm{i}}^{\mathrm{*}}$ values in $\mathrm{L}\mathrm{U}{\mathrm{V}}_{\mathrm{L}}$ array
7.	End for
8.	Step 2: SAD Calculation
9.	For each frame$i$in range($1,\mathrm{N}+1$):
10.	$\mathrm{S}\mathrm{A}{\mathrm{D}}_{\mathrm{i}}={\sum }_{(\mathrm{x},\mathrm{y})}\mid \mathrm{L}\mathrm{U}{\mathrm{V}}_{\mathrm{L}}\left[\mathrm{i}\right](\mathrm{x},\mathrm{y})-\mathrm{L}\mathrm{U}{\mathrm{V}}_{\mathrm{L}}[\mathrm{i}+1](\mathrm{x},\mathrm{y})\mid$
11.	End for
12.	Step 3: Keyframe Selection using K-means
13.	// Initialize cluster centroids C with K random SAD values
14.	While centroids are not converged
15.	For each$\mathrm{S}\mathrm{A}{\mathrm{D}}_{\mathrm{j}}$in$\mathrm{S}\mathrm{A}{\mathrm{D}}_{\mathrm{L}}$
16.	// Assign$\mathrm{S}\mathrm{A}{\mathrm{D}}_{\mathrm{j}}$to the closest cluster${\mathrm{C}}_{\mathrm{i}}$
17.	${\mathrm{C}}_{\mathrm{i}}=\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}\left|\right|\mathrm{S}\mathrm{A}{\mathrm{D}}_{\mathrm{j}}-{\mathrm{C}}_{\mathrm{i}}\left|\right|$
18.	End for
19.	End while
20.	Step 4:
21.	Sort selected keyframes by frame sequence number (ID value)
22.	// Output the selected keyframes
23.	Return Selected keyframes in chronological order
End Procedure

4.3. The Pretrained Models

The application of multiple features is beneficial for recognizing correct behaviors. In addition, color information is important, and is necessary to provide consistent scene luminosity, regardless of the condition of the captured image. Features of the optical flow also provide critical information regarding moving trajectories to distinguish behaviors. For each video, we extracted ten original RGB keyframes. We applied one technique of rotation, flipping, and shifting to each keyframe for geometric transformation and a color transformation for data augmentation purposes. Furthermore, we included an optical flow image for each keyframe. Thus, we obtained 40 keyframes per video clip in total. To obtain the necessary features for the classification of student behavior in the classroom, we use four pretrained models: Inception-V3, ResNet50-V, VGG-16, and EfficientNet-B7 for features including the RGB color, augmented data, luminosity in the L*a*b* color space, and optical flow, respectively. These multiple features are concatenated and fed into the classification network. Visualizations for RGB, DA, L*a*b*, and the optical flow keyframe are shown in Figure 5.

• Inception-V3: The Inception-V3 is a highly regarded CNN architecture developed by Google AI that has made significant strides in the field of computer vision. It is known for its ability to extract high-level features from images, which has consistently led to superior performance compared to other CNN architectures. The establishment of Inception-V3 was a turning point in network scaling strategies. According to Szegedy et al. [35], the architects behind Inception-V3 aimed to leverage additional computational resources efficiently, resulting in innovative techniques such as factorized convolutions and advanced regularization. The architecture went through rigorous evaluation, particularly in the ILSVRC ImageNet classification challenge of 2012.

We utilize the key success of Inception-V3 for high-level feature extraction from an image frame. The design of Inception design incorporates inception modules, which are clusters of convolutional layers with different filter sizes. This innovation allows Inception-V3 to analyze image frames across multiple scales, capturing features of varying granularities and complexities in the student behavior dataset. This aspect makes Inception-V3 well-suited for extracting features from RGB keyframes, which contain information about the student’s appearance. The architecture for feature extraction from the video keyframe is depicted in Figure 6.

• ResNet50-V2: The ResNet50-V2 is a highly effective CNN model developed to address the problem of vanishing gradients in deep networks. It is an improved version of ResNet50 that leverages innovative techniques to improve accuracy and performance. One of the key features of ResNet50-V2 is its residual connections, which enable direct information flow between layers, thus mitigating the vanishing gradient challenge. ResNet50-V2 uses bottleneck layers that reduce network parameters while maintaining accuracy via 1 × 1 convolutions followed by 3 × 3 convolutions. These advancements are supplemented by improved batch normalization and initialization techniques [36,37].

In the classroom, students’ behavior classification needs to detect the head and face of students, such as when they are sitting or turning around. Leveraging the capabilities of ResNet50-V2, we implemented this network to extract keyframe features. The main difference between Inception-V3 and ResNet50-V2 is that the latter focuses on keyframe augmentation techniques [38] to enhance various keyframe features, whereas Inception-V3 focuses on the original RGB keyframe. The model architecture designed for the feature extraction of DA is illustrated in Figure 7.

• VGG-16: The VGG-16 excels at feature extraction from images [39], resulting in its popularity. With 16 layers, including 13 convolutional and 3 fully connected layers, VGG-16 processes input images, gradually reducing spatial resolution while deepening channel features. Trained on ImageNet, housing over a million images and 1000 classes [40], VGG-16 employs supervised learning to adapt its weights. This learning equips VGG-16 to recognize common patterns and features in images, making it invaluable for diverse image analysis tasks.

To extract features from each video keyframe, we first apply an image transform from RGB to CIELAB or L*a*b* color space [41]. We then feed the transformed image through a VGG-16 model, which utilizes a global average pooling layer for its final pooling operation. This layer significantly reduces the spatial dimensions of the feature maps to 1 × 1 by computing the average values across the remaining pixels in each map. As a result, the output of the VGG-16 model for each video frame is a single vector containing 512 features that encapsulate the global characteristics of the input image. The model takes as input a video frame with dimensions of 224 × 224 × 3 and outputs a vector of 512 features. This pretrained model extracts the features from the keyframe based on the CIELAB color space. Figure 8 shows the architecture of the feature extraction model for Lab color space.

• EfficientNet-B7: EfficientNet is a family of CNNs that scale their depth, width, and resolution using a compound coefficient to ensure uniform scaling. EfficientNet models have been pretrained on ImageNet, a large-scale dataset with 1.3 million images from 1000 object classes [40]. According to Tan and Le (2019) [42], EfficientNet models outperform other CNN models in terms of accuracy while being faster and smaller. The EfficientNet-B7 module performed feature extraction with the transfer learning technique. Each input keyframe with dimensions of 224 × 224 × 3 is processed through a stack of 810 layers, without fully connected layers, responsible for making the final classification predictions for feature extraction. EfficientNet generates an output of feature descriptors with 2560 dimensions for each keyframe. These feature descriptors are input to the BiLSTM model for video representation and classification. EfficientNet-B7 is a pretrained model that is good at extracting features from optical flow keyframes, which contain information about the student’s movements. It is a relatively efficient model, which means that it can extract features from optical flow keyframes quickly and without using a lot of resources. This pretrained model is also the strongest for extracting the features, even with a few color differences [43]. We apply this pretrained model to extract the features from the optical flow keyframe [44]. The architecture of the feature extraction model for optical flow is shown in Figure 9.

4.4. BiLSTM-AT Deep Neural Network

RNNs are effective at modeling sequential patterns in time-series data such as video clips. However, the vanishing gradient problem can hinder RNN training. To overcome this issue, two RNN variations have been introduced: long short-term memory (LSTM) and gated recurrent units. The LSTM network has the same structure as RNNs but with an additional “memory cell” unit that maintains information for longer periods, helping to overcome the vanishing gradient problem. LSTM networks are used in a variety of applications, including video classification. However, LSTMs have a limitation in that they only capture past context. To fully understand any video, it is essential to consider both past and future contexts. Thus, bidirectional LSTMs (BiLSTMs) are a better option for video classification. BiLSTMs preserve information in both directions, allowing them to learn long-term dependencies in the data more effectively [45,46]. The BiLSTM model is highly efficient and more powerful when integrated with an attention mechanism for video classification tasks [47]. The BiLSTM model consists of two hidden layers: the forward hidden layer $\left(h_t^f\right)$ and the backward hidden layer $\left(h_t^b\right)$. The forward hidden layer processes the input vector $x_t$ in ascending order, with $t$ values ranging from $1$ to $T$. Conversely, the backward hidden layer processes the input vector in descending order, with $t$ values ranging from $T$ to $1$. The output $y_t$ is generated by combining the results of $h_t^f$ and $h_t^b$. Our work employs the BiLSTM-AT network, which comprises two modules: the BiLSTM and the attention mechanism. This network combines the current target forward $h_t^f$ and backward $h_t^b$ hidden layer with the context vector $C_t$ to generate an attention vector output for classifying student behavior. The structure of BiLSTM-AT is illustrated in Figure 10.

We implement the BiLSTM-AT model to classify student behavior [47]. After preprocessing the video to extract keyframes, we employ feature fusion to concatenate the keyframe extraction features from Inception-V3, ResNet50-V2, VGG-16, and EfficientNet-B7 to create a fused feature vector, denoted by $X_F=[F_{inception}, F_{resnet},F_{vgg}, F_{efficientnet}]$ for frame $F$. The sequence of feature vectors ${X=[X_1,X_2,X}_3,\dots ,X_{max\_sequence}]$, represented by the vector $X_{max\_sequence}$, denote the length of a keyframe in each video clip. To capture temporal dependencies, we compute forward and backward hidden states $h_t^f$ and $h_t^b$, and forward and backward cell states $c_t^f$ and $c_t^b$ for each keyframe feature vector $X_t$ by LSTM processing. We implement all the step processes of BiLSTM-AT using the following equations:

Forward LSTM equations:

$$h_t^f={LSTM}_f\left(X_t,h_{t-1}^f,c_{t-1}^f\right)$$

(1)

$$c_t^f={cell}_f\left(X_t,h_{t-1}^f,c_{t-1}^f\right)$$

(2)

At time step $t$, $h_t^f$ is the forward hidden state generated by ${LSTM}_f$, by processing $X_t$, an input feature vector typically from a video keyframe. $h_{t-1}^f$ and $c_{t-1}^f$ represent the prior forward hidden and cell states at $t$−1, capturing essential temporal information. $c_t^f$, the forward cell state at time step t, is the output of ${\mathrm{c}\mathrm{e}\mathrm{l}\mathrm{l}}_{\mathrm{f}}$ processing $X_t$, influenced by $h_{t-1}^f$ and $c_{t-1}^f$, collectively preserving critical information in the video sequence.

Backward LSTM equations:

$$h_t^b={LSTM}_b\left(X_t,h_{t+1}^b,c_{t+1}^b\right)$$

(3)

$$c_t^b={cell}_b\left(X_t,h_{t+1}^b,c_{t+1}^b\right)$$

(4)

In time step t, $h_t^b$ represents the backward hidden state, which ${LSTM}_b$ generates by processing $X_t$ in a reverse sequence, often starting from the end of the input sequence. $h_{t+1}^b$ and $c_{t+1}^b$ denote the subsequent backward hidden and cell states at $t$ + 1, capturing information concerning the sequences’ upcoming context. $c_t^b$, the backward cell state at time step t, results from ${cell}_b$ processing $X_t$ in reverse order, under the influence of $h_{t+1}^b$ and $c_{t+1}^b$. This arrangement collectively preserves vital information about the sequence’s future context.

Concatenation of Hidden States: For each time step t, we combine the forward and backward hidden states to create the final representation, $h_t$:

$$h_t=\left[h_t^f,h_t^b\right].$$

(5)

This concatenation ensures that we capture the context in both the forward $h_t^f$ and backward $h_t^b$ directions, enabling a more comprehensive understanding of the keyframe sequence.

Attention Mechanism: In the optional inclusion of an attention mechanism, we compute attention scores $A_t$ for each keyframe $X_t$ based on a query ($Q$), key ($K$), and value ($\mathit{V}$) vectors. This mechanism enhances the model’s ability to focus on essential frames within the sequence:

$$A_t=softmax(Q_t·K_t^T)$$

(6)

where $A_t$ is the attention score for the keyframe $X_t,$ reflecting the model’s emphasis on the keyframe at time step t. Softmax is the mathematical function that converts the dot product of the query $Q_t$ and transposed key $K_t$ vectors into a probability distribution for easier interpretation.

Context Vector Calculation: The context vector $C$ is generated as a weighted sum of the value vectors $V_t$ using the attention scores $A_t$:

$$C={\sum }_{t=1}^{\mathrm{m}\mathrm{a}\mathrm{x}\_\mathrm{s}\mathrm{e}\mathrm{q}\mathrm{u}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}}{A}_t·V_t$$

(7)

where $C$ summarizes information from the selected keyframes by assigning more importance to those frames identified as crucial by the attention mechanism.

As part of the attention mechanism, each keyframe feature vector $X_t$ is element-wise multiplied by the context vector $C$ to produce an attended representation $Y_t$. This process allows the model to prioritize and enhance the significance of features within each keyframe, thus improving the accuracy of student behavior classification.

4.5. Experiments

Our methodology employs multiple stages: First, pre-processing by frame extraction and selection are performed using the SADLK algorithm. Second, features are extracted from various pretrained models to obtain feature fusion. Lastly, behavior classification based on a hybrid model using a deep sequence neural network is utilized to classify student behavior based on the extracted features.

4.5.1. Student Behavior Dataset

As detailed in Section 3, regarding student behavior data collection, we have compiled a dataset of student behavior and divided it into five distinct classes: hand-raising, interaction, sitting, turning around, and writing. Hand-raising represents students raising their hands to express opinions or ask questions. Interactions denote students engaged in group discussions and problem-solving. Sitting indicates students are listening to the teacher’s explanations during lectures. Turning around represents students observing their friends’ activities, and writing captures students solving math problems and recording their answers on paper.

The student behavior dataset comprises 578 video clips with a 30 frames per second (fps) frame rate. The videos encompass various durations, with a minimum of 0.7 s (21 frames) and a maximum of 83.87 s (2516 frames). Each clip contains a total of 40 keyframes, which amounts to a total of 23,120 keyframes. The dataset is divided into training, validation, and testing subsets with 70, 15, and 15% of the data, respectively. Each video clip has an original keyframe of ten RGB, ten geometric transformations, ten color transformations, and ten optical flow keyframes. To ensure that frame overlap is considered, we use the SADLK algorithm to select ten keyframes from each video clip at a consistent 30 fps frame rate, regardless of the frame length.

4.5.2. Experimental Environment

To implement our proposed framework, the software and hardware specifications are as follows: Python version 3.11, TensorFlow and Keras version 2.12, OpenCV 4.7 on an Ubuntu operating system version 22.04 LTS, server processor Intel Xeon Gold 6230R of 4.0 GHz with 128 GB RAM, and GPU Nvidia Quadro RTX 8000. We perform experiments on multiple pretrained models with the SADLK algorithm keyframe selection methods and BiLSTM-AT to classify student behavior in the classroom.

4.5.3. Model Hyperparameters Tuning

The model is compiled with sparse categorical cross-entropy loss, the Adam optimizer with a learning rate of 0.001, and accuracy as the evaluation metric. The model is trained for 50 epochs, and the layer configuration of the proposed multiple pretrained and BiLSTM-AT is used for classifying student behavior. To load the data into the model, we choose ten keyframes and then extract features using the multiple pretrained models. The image size of the keyframes is 224 × 224 with three channels of RGB. All of the features are fused, and the number of features is 7168. The complete layer configuration is listed in

Table 1. The layer configuration of the proposed BiLSTM-AT model for student behavior classification.

Layers	Type	Output Shape	Parameters
Layer 1	Multiple Pretrained Models	(10 × 224 × 224 × 3)	124,179,959
Layer 2	Input 1	(None, 10, 7168)	0
Layer 3	Input 2	(None, 10)	0
Layer 4	BiLSTM (unit = 512)	(None, 10, 512)	15,206,400
Layer 5	Attention	(None, 10, 512)	1
Layer 6	Multiply	(None, 10, 512)	0
Layer 7	GlobAveragePooling1D	(None, 512)	0
Layer 8	Dropout 1 (value = 0.3)	(None, 512)	0
Layer 9	Reshape	(None, 1, 512)	0
Layer 10	BiLSTM (unit = 256)	(None, 256)	656,384
Layer 11	Dropout 2 (value = 0.3)	(None, 256)	0
Layer 12	Dense 1	(None, 64)	16,448
Layer 13	Dropout 3 (value = 0.3)	(None, 64)	0
Layer 14	Dense 2	(None, 5)	325
Total parameters: (15,879,558)
Trainable parameters: (15,879,558)
Non-trainable parameters: (0)
Activation function (ReLU, softmax)
Loss function (sparse categorical cross-entropy)
Optimizer (Adam)
Epochs (50)
Batch-Size (64)
Learning rate (0.001)

. Our model uses all the parameters of the multiple pretrained models during training, and none are non-trainable.

4.5.4. Evaluation Metrics

When assessing the efficacy of multiclass student behavior classification models, performance is evaluated by computing accuracy, precision, recall, and the f1 score based on confusion matrices. In precision metric measures, TP and FP represent the number of true-positive and false-positive samples in the positive class, respectively, while recall measures the proportion of correctly identified real positive samples. FN indicates false-negative counts. Accuracy is determined by comparing the number of correct predictions for each class with the total number of predictions for all classes and is expressed as a ratio. The formulae for calculation are [48]:

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}$$

(8)

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}$$

(9)

$$\mathrm{F}1\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2\frac{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\times \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}$$

(10)

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{\mathrm{N}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r} \mathrm{o}\mathrm{f} \mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t} \mathrm{c}\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s}}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l} \mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r} \mathrm{o}\mathrm{f} \mathrm{c}\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s} \mathrm{a}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{m}\mathrm{p}\mathrm{t}\mathrm{e}\mathrm{d}}$$

(11)

5. Results and Discussion

In this section, we analyze the accuracy of the single-pretrained and multiple-pretrained models with keyframe selection methods and use the BiLSTM-AT to classify the students’ behavior. Furthermore, the performance of our overall framework has been evaluated with a large dataset consisting of the HMDB51 [49] and UCF101 [50] public datasets. The results of the single-pretrained model and the multiple-pretrained models are described in Section 5.1 and Section 5.2. Finally, the performance evaluation with a large dataset is explained in Section 5.3. A summary of the discussion results is presented in Section 5.4.

5.1. Single Pretrained Model Results with BiLSTM-AT

The BiLSTM-AT network, without the keyframe selection and using single pretrained models, was used for feature extraction; we call this model “S-BiLSTM-AT.” The models were compared with Inception-V3, ResNet50-V2, VGG-16, and EfficientNet-B7. The VGG-16 pretrained model has a higher accuracy than the other single pretrained models with different parameters, with an accuracy of 88.89%. However, the single pretrained model cannot obtain the necessary features to identify a similar activity and classify the students’ behavior, which is a complex task with our dataset. Nevertheless, the results show that all four models achieve relatively high accuracy, precision, recall, and F1 scores. This suggests that all four models are effective at student behavior recognition, but VGG-16 with BiLSTM-AT is the most effective. The results are displayed in

Table 2. The results of various single pretrained models compared with the BiLSTM-AT network. The bold represents the best value.

Classifier	Total Parameters	Classification Overall Accuracy (%)
		Precision	Recall	F1	Accuracy
Inception-V3 + BiLSTM-AT	5,393,798	0.85	0.85	0.85	84.44
ResNet50-V2 + BiLSTM-AT	5,393,798	0.90	0.88	0.88	87.78
EfficientNet-B7 + BiLSTM-AT	6,442,374	0.88	0.88	0.88	87.78
VGG-16 + BiLSTM-AT	2,248,070	0.90	0.89	0.89	88.89

.

We conducted experiments using BiLSTM-AT with different pretrained models. Specifically, we used two, three, and four pretrained models. The two pretrained models were Inception-V3 and ResNet50-V2. The three pretrained models were Inception-V3, ResNet50-V2, and VGG-16. The four pretrained models were Inception-V3, ResNet50-V2, VGG-16, and EfficientNet-B7. We referred to these models as “IR-BiLSTM-AT”, “IRV-BiLSTM-AT”, and “IRVE-BiLSTM-AT” or “MFF-BiLSTM-AT”, respectively. The results of these experiments are displayed in

Table 3. The results of IR-BiLSTM-AT, IRV-BiLSTM-AT, and MFF-BiLSTM-AT models.

Classifier	Total Parameters	Classification Overall Accuracy (%)
		Precision	Recall	F1	Accuracy
IR-BiLSTM-AT	9,588,102	0.91	0.90	0.90	90.00
IRV-BiLSTM-AT	10,636,678	0.91	0.92	0.91	91.11
MFF-BiLSTM-AT	15,879,558	0.93	0.93	0.92	92.22

.

5.2. Multiple Pretrained Model Results with BiLSTM-AT

The BiLSTM-AT network uses the keyframe selection with multiple pretrained models for feature extraction. The model has the same hyperparameter tuning and training with the same parameters between MFF-BiLSTM-AT and KMFF-BiLSTM-AT in our overall frameworks. The results are shown in

Table 4. Comparison of the overall accuracy results of our dataset with the five models of the proposed framework. The bold represents the best value.

Classifier	Total Parameters	Inference Time (min)	Classification Overall Accuracy (%)
			Precision	Recall	F1	Accuracy
S-BiLSTM-AT	2,248,070	04.74	0.90	0.89	0.89	88.89
IR-BiLSTM-AT	9,588,102	10.61	0.91	0.90	0.90	90.00
IRV-BiLSTM-AT	10,636,678	13.94	0.91	0.92	0.91	91.11
MFF-BiLSTM-AT	15,879,558	20.07	0.93	0.93	0.92	92.22
KMFF-BiLSTM-AT	15,879,558	21.80	0.97	0.97	0.97	96.67

, and the training and validation accuracy is shown in Figure 11. The sample model prediction results are shown in Figure 12.

We compare the performance of the five models of our proposed framework: S-BiLSTM-AT, IR-BiLSTM-AT, IRV-BiLSTM-AT, MFF-BiLSTM-AT, and KMFF-BiLSTM-AT. The accuracy of the KMFF-BiLSTM-AT model was found to be higher than the accuracy of the other models. However, even though our framework exhibits a high level of performance, there are still some limitations. For example, there is some confusion when making predictions about actual classes; some students turned around, but the model predicted hand-raising behavior because the student also moved one hand. In addition, when students interacted with their friends, the model predicted that they would turn around as they moved their heads to respond to another student.

The results show that after fusing the features, the single pretrained model is less efficient than the multiple pretrained models, with an accuracy of 88.89 and 92.22%, respectively. However, when compared with applying the keyframe selection methods and using multi-feature fusion, the KMFF-BiLSTM-AT model was the most efficient, with an accuracy of 96.67%. The results can be found in Table 4.

5.3. Comparison with a Large Dataset

The overall performance of our proposed framework was evaluated using the large public UCF101 and HMDB51 datasets. We selected ten keyframes from each video clip using the SDALK algorithm and then used our proposed novel multi-feature extraction and fusion methodology. Finally, the BiLSTM-AT network is used to classify human action recognition. The dataset is split into three groups consisting of “Training”, “Validation”, and “Testing”, which have percentage data values of 70:15:15, respectively. The number of video clips is shown in

Table 5. Dataset settings for training, validation, and testing.

Dataset	Classes	Training (70%)	Validation (15%)	Testing (15%)	Total Clips (100%)
UCF101	101	9664	1705	1951	13,320
HMDB51	51	4907	865	994	6766

.

Our proposed KMFF-BiLSTM-AT model obtained high accuracy compared with the state-of-the-art models. The model achieved the highest accuracy of 97.54% with the UCF101 dataset and 72.43% with the HMDB51 dataset.

Table 6. Comparison of our methods with the state-of-the-art models with the UCF101 and HMDB51 public datasets.

Methods	UCF101 (%)	HMDB51 (%)
iDT+HSV [51]	87.90	61.10
C3D [52]	85.20	-
Deeper temporal net [53]	84.90	-
Two-stream [54]	88.00	54.90
FstCN [55]	88.10	59.10
TDD+FV [56]	90.30	63.20
scLSTM [57]	84.00	55.10
VideoLSTM [58]	89.20	56.40
L2STM [59]	93.60	66.20
STS [60]	90.10	62.40
STS-ALSTM [60]	92.70	64.40
S3D+RANs [61]	93.30	71.20
TSN [62]	93.90	69.90
TSN+AS [62]	94.50	71.60
3DCNN+Two-level+AT [63]	95.68	72.60
KMFF-BiLSTM-AT (10 Keyframes) (Ours)	97.54	72.43

summarizes the comparative results.

5.4. Discussion

When comparing the results of the student’s behavior dataset with only a single pretrained model, we found that VGG-16 with the BiLSTM-AT model had a higher accuracy than the others. However, even though the VGG-16 pretrained model is excellent for extracting features, it is still insufficient for identifying and classifying students’ behaviors in the classroom using the BiLSTM-AT model. When we apply the multiple pretraining models based on multi-feature fusion and keyframe selection methods to address this limitation, the MFF-BiLSTM-AT and KMFF-BiLSTM-AT models achieve a higher accuracy than the S-BiLSTM-AT model. Nevertheless, of the multiple pretrained models, the KMFF-BiLSTM-AT obtained an accuracy higher than the MFF-BiLSTM-AT. The model prediction results for KMFF-BiLSTM-AT are higher than all of our proposed models, indicating that this is an excellent, accurate, and robust model for classifying the complexity of our student behavior datasets.

Moreover, our proposed framework adopted keyframe extraction and selection methods using the SADLK algorithm with multiple feature fusion and applied the data augmentation technique combination with optical flow motion identification on BiLSTM-AT networks—the overall accuracy evaluation with the two largest datasets, HMDB51 and UCF101, of our proposed framework is excellent compared with state-of-the-art models.

6. Conclusions and Future Work

There have been several attempts to analyze students’ behavior in a classroom. However, it has been difficult to find an effective method for this analysis due to the limited classroom environment and the interactions between many students and a teacher. We propose a novel MultiFusedNet network model that can optimally perform individual behavior analysis in situations where lighting changes are diverse and groups of students are intermingled. To select keyframes that have representativeness from a video input of arbitrary length, the SADLK algorithm was designed as an efficient tool to select a specified number of keyframes by identifying the spatial and color differences between frames and applying the k-mean method. The MultiFusedNet network analyzed the form of a given action based on multi-features, which are extracted and mixed from the selected keyframe. The spatial and temporal features are extracted from RGB images, including L*a*b* color images that are robust to changes in lighting, geometric transformed images that ensure data diversity, and optical flow images that contain motion vectors.

We used the various spatial and temporal features in the depth network BiLSTM with an attention mechanism model. With highly complex student behavior datasets, the model can effectively analyze complex student behavior datasets, even when the data is similar across classes. By leveraging the capabilities of our framework, teachers or educators can develop novel techniques to help students learn more effectively. The model’s ability to analyze student behavior can identify relationships between learning outcomes, activities, and behaviors, enabling instructors to design more personalized and targeted learning plans based on the PLC approach. In future work, we aim to expand our collection of student behavior datasets with diverse labeling schemes while adding new categories to our existing datasets. To enhance the effectiveness and accuracy of our overall framework, we plan to integrate pose estimation of the human skeleton into our existing multiple features and use a BiLSTM-AT deep sequence neural network to classify students’ behaviors. The goal of this work is to comprehensively study student behavior in classrooms across diverse schools in Thailand. By partnering with researchers and using AI technology, the aim is to support teachers in analyzing and improving students’ behaviors. This approach will enhance the classroom environment, address learning needs, and improve outcomes in all subjects.