A Novel Method for Cross-Modal Collaborative Analysis and Evaluation in the Intelligence Era

Abstract

The development of intelligent information technology provides an effective way for cross-modal learning analytics and promotes the realization of procedural and scientific educational evaluation. To accurately capture the emotional changes of learners and make an accurate evaluation of the learning process, this paper takes the evaluation of learners’ emotional status in the behavior process as an example to construct an intelligent analysis model of learners’ emotional status in the behavior process, which provides effective technical solutions for the construction of cross-modal learning analytics. The effectiveness and superiority of the proposed method are verified by experiments. Through the analysis of learners’ cross-mode learning behavior, it innovates the evaluation method of classroom teaching in the intelligent age and improves the quality of modern teaching.

1. Introduction

1.1. Educational Evaluation in the Age of Intelligence

With the development of science and information technology, contemporary education has undergone a series of reforms and innovations with a focus on improving the quality of education rather than increasing the level of its scale and speed. Establishing a high-quality educational system has become the aim for the country and society as a whole.

In recent years, technologies such as big data, the Internet of Things, artificial intelligence, short video, virtual reality (VR), augmented reality (AR), and the metaverse have been gradually applied to the field of education. The upgraded level of information technology accelerates reform and innovation in education and profoundly affects the connotation, method, and path of educational evaluation, making educational evaluation transfer from a result-oriented, subjective direction to a process-oriented, concomitant, and scientific direction. There is an emphasis on the innovation of evaluation tools and the application of modern information technologies such as artificial intelligence and big data to explore and carry out the longitudinal evaluation of students’ learning at all grades, such as the horizontal evaluation of moral, intellectual, physical, aesthetic, and labor factors.

In accordance with the national aim and requirements of education, which lie in the scientific, procedural, and intelligent standards of educational evaluation, and with the use of advanced technologies in information and artificial intelligence, this paper adopts an innovative teaching and learning evaluation model that focuses on intelligence education evaluation methods. It is deemed essential for the development of education reform in terms of the integration of information technology, intelligent technology, and educational philosophy.

1.2. Proposition of Research Questions

Education in the age of intelligence emphasizes the cultivation of students’ abilities in terms of innovation, interdisciplinary problem-solving, teamwork, and other comprehensive qualities through the process of hands-on practice. The biggest difference compared with traditional education is that it focuses more on the process of learning than the ultimate results. The emotional status generated by students during the learning process can, to some extent, reflect their learning interests and the learning effects, and therefore form the mechanism of learning eventually. The status can be recorded and analyzed through cross-modal data such as facial expressions, actions, language, and heartbeat.

Cross-modal collaborative analysis and evaluation can provide an effective technical solution for the construction of a teaching evaluation system in the Age of Intelligence. Thus, the data analysis and conclusions from the evaluation system can shed some light on research questions such as how to improve the quality of intelligent education.

In the smart education environment, based on technologies such as the Internet of Things and artificial intelligence, as well as intelligent sensing devices such as eye trackers and electroencephalographs, it is possible to monitor and record complex teaching processes in real-time and form cross-modal data such as audios, texts, images, videos, etc. Based on cross-modal learning analysis technology, collaborative analysis of student learning behavior data is carried out to mine and restore the students’ learning emotional status information implied by the cross-modal and accompanying behavioral information in the classroom. A scientific model further reveals the implicit relationship between teaching and learning process information and classroom education quality, and a new evaluation model of classroom teaching and learning has been constructed. This evaluation method is more comprehensive and scientific by eliminating the weaknesses of the traditional educational evaluation model, which emphasized tests and grades, and by optimizing teaching methods based on evaluation and improving the quality of classroom teaching. The purposed method focuses on evaluating and developing students’ innovative thinking ability, comprehensive quality, personality, and mental health.

Taking the cross-modal learning behavior data as the carrier, with technologies such as artificial intelligence, big data, etc., this research collects students’ emotional status through the whole process of learning, conducts a collaborative analysis of learning behavior, and thus constructs a new evaluation method of teaching that links learning behavior with learners’ emotional status. This research indicates significance in research and real-world practice in terms of achieving a comprehensive and accurate assessment of a student’s ability, cognitive level, personality traits, mental health, etc., explaining and exploring cross-modal data-driven educational phenomena and educational laws, offering guidance on the optimization of the teaching process, and improving the goal of classroom teaching quality.

2. Literature Review

Research on cross-modal learning analysis could be dated back a decade, and the academic community continues to focus on its development. In 2012, scholars raised the idea of applying cross-modal data such as text, video, and audio in the field of cross-modal interaction research, a new direction for the research of cross-modal learning interaction, which put cross-modal learning as the core status of educational evaluation. In 2012, the International Conference on Cross-Modal Interaction (ICMI) established the frontier status of cross-modal learning analysis by organizing relevant scholars to participate in the seminar on “Cross-Modal Learning Analysis” for the first time in the form of workshops [1], declaring that the aim of this research field is to combine cross-modal analysis technology with learning science research to help researchers better understand the mechanism of learning. Since 2014, with the gradual development of blended learning, learning analysis, and big data technologies, a large amount of online and offline learning data have become available for identifying patterns such as learning emotions by extracting and analyzing cross-modal feature data. In 2016, the International Conference on Learning Analysis and Knowledge (LAK) set up a data challenge workshop for cross-modal learning and analysis, organized relevant scholars to participate in the practical work of cross-modal data analysis, and explored the development direction of learning analysis research supported by cross-modal data [2]. The application of artificial intelligence technology has been widely spread in cross-modal learning analysis since 2018. By applying artificial intelligence to education, Li Songqing [3] proposed to use cross-modal machine learning to create and replicate human cognition, analyze and design artificial artifacts based on artificial intelligence technology through cross-modal learning, and ultimately support, help, and expand human cognitive abilities. The American Educational Research Association (AERA) annual meeting in 2019 proposed to use cross-modal data such as text, charts, audio, and video to construct a research paradigm based on cross-modal narratives to seek evidence of fairness in educational research [4].

In terms of specific research content, Hu Qintai et al. [5] analyzed the interpretability problem of cross-modal learning behavior and used the HDRBM (Hybrid Deep Restricted Boltzmann Machine), a deep neural network model, to obtain the implicit physiological and emotional characteristics of students. This research used the Bayesian network and the junction tree algorithm to analyze the interpretability of the results, which has a certain effect on improving the interpretability of learning behavior analysis. From a cross-modal measurement perspective, Kyllonen et al. [6] designed and developed a computational model framework for assessing learning status. This framework is capable of capturing, analyzing, and measuring complex human behavior as well as analyzing noisy, unstructured, and cross-modal data. The analysis process utilizes cross-modal data, including audio, video, and activity log files, to construct an Analytic Hierarchy Process for modeling the temporal dynamics of human behavior. Riquelme et al. [7] used transponders to collect students’ voice data to analyze the continuity and motivation of group collaboration. Poria et al. [8] used an ensemble feature extraction method to develop a new cross-modal information extraction agent. In particular, the developed method exploited joint tri-modal features (text, audio, and video) to infer and aggregate semantic and emotional information on user-generated cross-modal data. In terms of analysis methods, Scherer [9] used a cross-modal sequence classifier to analyze the expressions of laughter in the process of multi-party dialogue in the natural environment. At the same time, multi-dimensional data channels are used to extract the frequency and spectral features from audio streams and motion-related behavioral features from video streams. On this basis, the Hidden Markov Model and the Echo State Network are used to identify the paralinguistic behavior in the communication process, and a classification model with a certain level of accuracy is obtained. Researchers from the Norwegian University of Science and Technology collected cross-modal data of learners in adaptive learning activities and used the fuzzy set qualitative comparative analysis (fsQCA) method to describe the relationship between learner participation patterns and learning performance [10]. Vicente et al. proposed a Wearable Internet of Things in Education (WIoTED) system based on IoT technology and real-time monitoring data from wearable devices and used machine learning techniques and cross-modal learning analysis methods to build a model that could “explain” student engagement. This study selects a decision tree and rule system based on a set of correlated variables, and the obtained rules could be easily interpreted by non-professionals [11].

Overall, the research on cross-modal learning analysis has a rapid speed of development in spite of its comparatively short history. In recent years, it has received extensive attention from academic groups across different disciplines. The evolutionary process covers knowledge model and framework design, cross-modal feature extraction and sentiment analysis, cross-modal representation learning, and in-depth learning. Its research ideas and research results have further enriched the fields of learning analysis and learning science. Major research teams in China are from normal universities such as Beijing Normal University, East China Normal University, and South China Normal University. By analyzing cross-modal learning from the perspective of learning science, the research mainly focuses on theoretical discussion (Such as Mu Su [12], Wang Weifu [13]) and framework construction (Such as Zhou Jin [14], Zhang Qi [15], Mu Zhijia [16], Li Qing [17], etc.).

Previous studies indicate that cross-modal learning analysis is an effective method for analyzing learning behavior and obtaining learners’ status. However, current methods rarely apply theories to practice from the perspectives of computer science and data science. Meanwhile, the accuracy, comprehensiveness, and interpretability of analytical models need to be improved. Based on deep learning and cross-modal learning, we established a multi-modal sentiment analysis model for learning behavior, and the robustness of the analytical model was improved using deep learning and pre-training, implying an important research trend.

3. Research Method

3.1. Problem Definition

This paper processes and analyzes multi-modal data (visual, text, audio, etc.) through in-depth learning algorithms such as neural networks and attention mechanisms. It also constructs an intelligent analysis model of the emotional status of the learner’s learning process, which can accurately capture the emotional changes produced by learners during the learning process for collaborative analysis and evaluation.

3.2. Model Architecture

By studying cross-modal data processing, cross-modal data fusion, pre-training model construction, and collaborative analysis of learner learning status, we propose an architecture to identify the mechanism of deep inquiry learning occurrence and improve the accuracy of students’ emotional status analysis during the learning process. We term our architecture ‘BLBA-MODEL’ (Bi-LSTM and Bi-Attention Mechanism Model). Its main framework is shown in Figure 1, which can be divided into three modules: data feature extraction, data fusion and parsing, and emotional status assessment of the learning process. These three modules are illustrated as follows.

3.2.1. Representation, Normalization, and Alignment of Cross-Modal Learning Behavioral Data

The data in the learning process includes multiple modalities such as video, text, and audio. In multi-modal data, each modality provides specific information for other modalities, and there is a certain correlation between the modalities. In this paper, the text, audio, and visual in the video of the learner’s learning process are used as data for processing, and the two techniques of batch normalization and layer normalization are used to remove the dimension and realize the normalization of the training data. Formula (1) is a specific representation of normalization. For input $z_i^l$, after calculating its mean $\mu$ and variance ${\sigma }^2$, normalization is performed, where $\gamma$ and $\beta$ represent the learnable parameter variables of scaling and translation, respectively. For each neuron, before the data enters the activation function, the mean and variance of each batch are calculated along the channel so that the data maintains a normal distribution with a mean of 0 and a variance of 1 to avoid the disappearance of the gradient.

$${\tilde{z}}_i^l=\frac{z_i^l-\mu }{\sqrt{{\sigma }^2+\stackrel{\prime }{o}}}\times \gamma +\beta$$

(1)

The aligned hidden vectors are spliced as the input of the subsequent module; it is the LSTM-Attention (Long Short-Term Memory and Attention Mechanism) combination module. The technical route of characterization, normalization, and alignment of cross-modal streaming media data is shown in Figure 2.

3.2.2. Fusion and Analysis of Cross-Modal Learning Behavior Data

After normalizing and aligning the cross-modal data, the information interaction within and between modalities is identified by fusing the feature representations of various modal information. This paper proposes the LSTM-Attention combination module to realize this kind of information interaction and takes the interaction of three modalities (audio, text, and visual) in the video as an example to illustrate in detail. Since we only used text, audio, and visual as our inputs for subsequent experiments, we will no longer regard video as a separate input.

3.2.2.1. Unimodal Feature Information Interaction

Considering the video information as a set of several utterances, we use multiple independent Bi-LSTMs (Bi-Directional Long Short-Term Memory) to capture the context-related semantic information of each modality, and the output $h_{ij}$ after the Bi-LSTM layer can be obtained.

$$h_{ij}=\overrightarrow{LSTM}\left(x_{ij},h_{j-1}\right)\oplus \overleftarrow{LSTM}\left(x_{ij},h_{j-1}\right)$$

(2)

Among them, $x_{ij}$ represents the input feature of utterance $j$ in video $i$, $h_{j-1}$ represents the hidden layer state of utterance $j-1$, and $h_{ij}$ represents the output of Bi-LSTM layer.

Then the matrix of video $i$ after the Bi-LSTM layer is expressed as:

$$H_i^{s_m}=\left[h_{ij}:j\le L_i\right], H_i^{s_m}\in {ℝ}^{q\times d}$$

(3)

where $d$ represents the feature dimension.

Through this layer processing, three modal feature representations with context-related information in video $i$ can be obtained: textual feature $H_i^{s_t}$, audio feature $H_i^{s_a}$, and visual feature $H_i^{s_v}$.

3.2.2.2. Bimodal Feature Information Interaction

After obtaining the above unimodal feature representation, three modalities are combined in pairs (text + audio, audio + visual, and text + visual) to obtain three sets of modal pairs. Next, we proposed a BAM-model (Bimodal Attention Mechanism model) to capture the interaction information between modalities and contexts. The model structure is shown in Figure 3. The specific process and formula are as follows:

The first step is to multiply the feature representation matrices obtained in Section 3.2.2.1 to obtain the cross-modal information matrix, where $u,w=\left\{t,a,y\right\}$:

$$C_i^{uw}=H_i^{s_u}\cdot H_i^{s_w}{}^T$$

(4)

$$C_i^{wu}=H_i^{s_w}\cdot H_i^{s_u}{}^T,$$

(5)

The SoftMax function (SoftMax logical regression) is used to model multiclass classification problems where we want the output to be a probability distribution of the different possible classes. The second step is to demonstrate a SoftMax calculation on the cross-modal information matrix to obtain the attention score:

$${\alpha }_i^{uw}=softmax\left(C_i^{uw}\right), {\alpha }_i^{uw}\in {ℝ}^{q\times q}$$

(6)

$${\alpha }_i^{wu}=softmax\left(C_i^{wu}\right), {\alpha }_i^{wu}\in {ℝ}^{q\times q},$$

(7)

In the third step, the attention score is multiplied by the feature matrix, and the modal feature representations with information interaction are obtained, respectively:

$$H_i^{d{1}_u}=({\alpha }_{uv}{H}_i^u)·H_i^u, H_i^{d{1}_u}\in {ℝ}^{q\times d}$$

(8)

$$H_i^{d{1}_w}=({\alpha }_{wu}{H}_i^w)·H_i^w, H_i^{d{1}_w}\in {ℝ}^{q\times d},$$

(9)

After the above operations, text feature representations $H_i^{d{1}_t}$, $H_i^{d{3}_t}$, audio feature representations $H_i^{d{1}_a}$, $H_i^{d{2}_a}$, and visual feature representations $H_i^{d{2}_v}$, $H_i^{d{3}_v}$ can be obtained, respectively.

Figure 3. BAM-model.

Figure 3. BAM-model.

3.2.3. Assessment of Learning Emotional Status

The feature representation of each modality with contextual interaction information can be obtained through the module processing in the first two sections. In this section, the attention mechanism is used to weigh the importance of each modality of information to filter out redundant information, classify emotions according to the final fusion information and obtain students’ emotional status throughout the whole process of learning. The specific process is as follows:

First, we use the fully connected layer to splice the above modal features and obtain the feature representation to enter the attention mechanism to filter redundant information:

$$R_i^t=\tanh \left(W_i^t\left[H_i^{s_t}\oplus H_i^{d{1}_t}\oplus H_i^{d{3}_t}\right]+b_i^t\right)$$

(10)

$$R_i^a=\tan \mathrm{h}\left(W_i^a\left[H_i^{s_a}\oplus H_i^{d{1}_a}\oplus H_i^{d{2}_a}\right]+b_i^a\right),$$

(11)

$$R_i^v=\tan \mathrm{h}\left(W_i^v\left[H_i^{s_v}\oplus H_i^{d{2}_v}\oplus H_i^{d{3}_v}\right]+b_i^v\right),$$

(12)

where $W_i^m$ is the weight and $b_i^m$ is the bias.

Then, the feature representation $R_i^m$ of each modality is sent to the attention mechanism model, and the multi-modal fusion feature representation is obtained by weighted summation:

$${⋎}_m=softmax\left(\tan \mathrm{h}\left(W_{att}^m{R}_i^m+b_{att}^m\right)\right)$$

(13)

$$R_i^m{}^{*}={\displaystyle \sum }_m{R}_i^m{⋎}_m{}^T,$$

(14)

Among them, $W_{att}^m$ represents the weight, $b_{att}^m$ represents the bias, and ${⋎}_m$ represents the normalized weight.

Finally, through the fully connected layer and using the SoftMax function for emotion prediction and classification, the final original learned emotion and cognitive state of the model are obtained:

$$y_i=softmax\left(W_q·\left(\tanh \left(W_p·R_i^m{}^{*}+b_p\right)\right)+b_q\right)$$

(15)

Among them, $W_p$ and $b_p$ are the weights and biases of the SoftMax layer, and $W_q$ and $b_q$ are the weights and biases of the fully connected layer.

4. Results and Discussion

This section systematically analyzes the effect of the proposed model on the multi-modal sentiment analysis task.

4.1. Experimental Datasets

Since the data set for students’ emotional behavior in the learning process involves issues such as students’ privacy, no data set has been found as an experimental data set for targeted model training and analysis. Therefore, this paper uses the universal public data set CMU-MOSEI to conduct experiments. On the one hand, it can verify the sentiment classification performance of the model. On the other hand, it can expand the scope of application of the model and facilitate subsequent targeted work.

The dataset was collected from 1000 speakers and contains a total of 3228 videos with 23,453 annotated sentences and 250 different topics. In this dataset, males and females make up about 57% and 43%. We divide the dataset into a training set and a test set; the details are shown in

Table 1. Training set and test set of the CMU-MOSEI dataset.

CMU-MOSEI Dataset	Videos	Number of Utterances	Positive Sentiment	Negative Sentiment
Training set	2250	16146	7869	8277
test set	679	4634	2541	2093

. The dataset has six different annotations for sentiment, namely Happiness, Sadness, Anger, Disgust, Surprise, and Fear. For classification prediction, videos with sentiment values greater than or equal to 0 are marked as positive sentiment classifications, and videos with sentiment values less than 0 are marked as negative sentiment classifications.

4.2. Ablation Experiment

This paper divides the feature input into different combinations to verify the importance of multi-modal information and performs sentiment classification prediction, respectively. This paper uses ACC (accuracy) and $F1$ indexes to evaluate algorithm performance:

$$Accuracy=\left(TP+TN\right)/\left(TP+FP+TN+FN\right)$$

(16)

$$F1=2TP/\left(2TP+FP+FN\right),$$

(17)

where TP is true positive, TN is true negative, FP is false positive, and FN is false negative. The experimental results are shown in Figure 4.

4.3. Comparative Experiment

This paper compares the proposed model with existing multi-modal sentiment analysis models, and we use the same dataset to experiment with these methods. The benchmark methods are as follows:

(1)

SVM-MD: An SVM model using multi-modal features trained by early fusion.

(2)

LF-LSTM: LSTM network with late fusion method.

(3)

MFN: A neural network structure for multi-view sequential learning based on attention networks and gated memory, which well models the interaction between modalities.

(4)

MARN: The method of combining a multi-attention module and a recurrent neural network is adopted to obtain the interaction information between three modalities through a multi-attention unit, and the recurrent network is used as a memory unit for storage.

(5)

Graph-MFN: Based on the MFN method, a method of combining the dynamic fusion graph with it is introduced; that is, the multi-modal dynamic fusion graph is combined with the context memory fusion network.

(6)

MMMU-BA: A method that utilizes the correlation of contextual information of target video segments between different modalities to assist multi-modal information fusion.

4.4. Experimental Results and Analysis

It can be seen from the ablation experiment results that the performance of text-based sentiment analysis in unimodality is the best. The sentiment analysis performance of all modal combinations in bimodality is generally better than that of the best unimodality analysis type. The trimodal sentiment analysis model performed the best among all models. Its accuracy was increased by 1.36% and 1.61% compared with the best models in the unimodal and bimodal, respectively. Its $F1$ index is improved by 2.33% and 1.98% compared to the best model in unimodal and bimodal, respectively. Therefore, the effective combination of multiple modalities can well obtain sentiment classification prediction and improve the model’s performance. Moreover, during the process of students’ learning, it can accurately grasp the emotional status of students by capturing the information from multiple modalities, adopting multi-modal emotional analysis, and presenting evaluation results that could provide emotional guidance to students promptly or adjust the difficulty level and teaching mode of the course to help students acquire knowledge much better and achieve the learning aims much easier.

From the experimental data in

Table 2. Comparative experimental results of different models.

CMU-MOSEI	Method Model
	SVM-MD	LF-LSTM	MFN	MARN	Graph-MFN	MMMU-BA	BLBA-MODEL
ACC (%)	68.34	80.61	77.13	76.8	76.9	80.27	80.78
$F1$ (%)	67.92	79.46	76.35	77.01	77.14	79.23	80.41

, the proposed method has achieved excellent results accuracy and $F1$ index. Compared with early fusion, the performance improvement is the most obvious. Compared with recent more advanced deep learning and other methods, the accuracy and $F1$ index are also 0.17% and 0.95% higher. Therefore, the model can extract meaningful information from multi-modalities for fusion and accurately classify and predict, which is helpful to apply to the scenarios where students learn emotional status analysis in the process of learning.

5. Conclusions

With the development of intelligence, the single source and simple structure of analytics data in the past have gradually become unsuitable for the era of rapid development. Based on deep learning, this paper proposes a multi-modal emotional analysis method to obtain people’s sentiment status and make accurate evaluations of people’s behavior processes. All the above experiments demonstrate this study method’s effectiveness and excellent performance. In future work, we will use emotional analysis in intelligent education to build a learning analysis model between different subjects and find out the correlation between them while studying the learning behavior of each subject. We will use different dataset related to the student’s learning behavior process to build a complete cross-modal collaborative analysis framework.