1. Introduction
Human memorability can be regarded as the criterion for judging whether the content of multi-media can be remembered or not [
1]. In current social media, humans are exposed to continual and vast multi-media information, such as images, videos, audio and text. Human memorability has different responses to different multi-media contents, some of which are stuck in our memory while others are forgotten easily. For example, if the content appeals to the observers, such as their favorite celebrities or shocking natural scenes, the media information can be remembered easily. Therefore, research on media information memorability can contribute to advertising, intelligent recommendation, and so many other applications.
Media memorability aims to score the probability of being remembered according to the media content. Intuitively, media memorability is affected by subjective factors, but some studies have proved that memorability is the inherent attribute of media information [
2]. For example, Isola et al. [
2] demonstrate that image memorability is a stable intrinsic property. The image memorability (IM) score is defined as the percentage of correct detection by different observers. Based on observation, many works focus on exploiting the factors affecting image memorability and designing models for predicting image memorability. More recently, the amount of videos on social media, such as Tik-Tok, Youtube, etc., has been growing exponentially. Therefore, video memorability is nascent as a new research field. Compared with images, videos are dynamic and contain temporal information. Therefore, the study of video memory is different from that of images. Kar et al. [
3] began to wonder what makes a video memorable and propose that the video frames can be used to extract features for predicting video memorability. Similarly, the video memorability (VM) score is defined as the percentage of correct detection by different observers.
However, the drawback of current research on video memorability is that the main attributes affecting memorability prediction have not been determined, making the prediction model’s design more challenging. This paper analyzes and experimentally verifies the critical impact factors in video memorability and proposes a framework for efficiently predicting video memorability. Firstly, visual and semantic factors play a decisive role in video memorability prediction. The combination of vision and semantics has produced a powerful effect on human memory. Secondly, multi-source feature fusion usually yields better results than isolated modeling generally. However, merging multiple features of the same information source adds complexity and causes heterogeneous errors [
4]. Thirdly, it has been studied that the memory interval affects the video memorability [
5]. Thus, we consider the memory interval by taking the memory after memorizing a few minutes as the short-term memory and using the long-term memory to define the memory performance after 24–72 h. Choosing different media information sources can predict the change in video memorability from different angles. Finally, we choose temporal 3D information, spatial information, and semantics derived from the three media information, including videos, images and captions, respectively.
Considering the redundancy effect of feature fusion, we use different base individual learners for each media information source to predict memorability scores. Concretely, we train the Resnet3D network [
6] model to predict video memorability scores. For the selection of the spatial information, based on the previous experimental results [
4], Local Binary Patterns (LBP) [
7] features are more favorable for predicting video memorability scores than RGB [
8] and High Osmolarity Glycerol (HOG) [
9]. As for the text features, the semantics are phrases or sentences sufficient to describe the video scene, and we extract features from semantics as input to the semantic modal model. Then, we design a new framework, the Adaptive Multi-modal Ensemble Network (AMEN), to predict video memorability efficiently. Specifically, we integrate the memorability scores of different models using the weighted method and then use the stochastic gradient descent algorithm to obtain the best prediction results. In addition, the framework updates the weights of each learner adaptively instead of assigning weights manually. In general, the ensemble learning [
10] appliance has a more vital generalization ability than the base learning appliance. Compared with feature fusion, the weighted ensemble learning method can fully guarantee the independent information of each mode and reduce the error and redundancy caused by feature fusion. As mentioned above, people’s memory for a certain video decreases with time, so we predict the video memorability scores for both short-term and long-term memory, that is, how well humans remember a video a few hours after watching it and how well humans remember the same video a few days later. Our experiments are conducted in both short-term and long-term memory.
The main contributions of our method are summarized as follows:
We analyze and experimentally verify how to select impact factors to predict video memorability and conduct three main factors that affect video memorability, i.e., temporal 3D information, spatial information and semantics derived from video, image and caption, respectively.
We propose a new adaptive multi-modal ensemble network (AMEN) for video memorability according to the selected impact factors. It eliminates the error caused by the heterogeneous gap via integrating the optimal base learners corresponding to each media source instead of fusing the heterogeneous modalities directly.
We design an adaptive learning strategy to update the weights based on the importance of memorability which is predicted by the base learners and contributes to obtaining the best performance without any manual tuning.
The rest of our paper is organized as follows. Related works about media memorability are summarized in
Section 2. Our method is described in
Section 3. Experimental results are presented in
Section 4. The conclusion is made in
Section 5.
2. Related Works
In this section, we summarize the previous works on media memorability and related concepts of media memorability. The work of Isola et al. [
2] led to a pioneering study of media memorability. To measure image memorability, Isola et al. first proposed the memorability score of each image by the memory game. Based on the visual memory game, Isola et al. [
2] proved that image memorability is the intrinsic property of each image and showed that the image memorability score has sufficient consistency of each image across various viewers by the consistency analysis. With the development of research on image memorability, more researchers have devoted themselves to studying what makes the image memorable and how to utilize machine learning algorithms to predict image memorability scores. Isola et al. [
11] further explored the fact that image features, attributes and labels have a positive effect on the memorability of an image and use the support vector regression algorithm to predict image memorability scores. Intrinsic and extrinsic properties [
12] that make media information memorable have been studied in recent years. Moreover, objects, emotions, saliency, and aesthetics contribute to making an image memorable [
13,
14,
15,
16]. It has to be mentioned that Dubey et al. [
17] specifically discussed what makes an object memorable. To better understand what makes an image forgettable or memorable, Basavaraju et al. [
14] were committed to studying the role of depth and motion and showed that depth and motion are helpful. Although many properties have been mined related to the memorability of the image, researchers only use traditional and simple image features, such as Pixels, GIST [
2,
11,
18], to predict image memorability. In other words, much of the image information has not been mined and utilized.
Contrast to that, scholars attempted to extract deep features from deep learning algorithms to predict image memorability with the deep learning algorithm becoming extremely popular in the 2010s [
19,
20]. Khosla et al. [
21] first used fine-tuned Hybrid Convolutional Neural Networks (CNNs) [
22] to extract deep features, and the performance outperformed all other features at that time. Then, Zarezadeh et al. [
23] used three common convolutional networks types to derive deep features from predicting image memorability and they drew a conclusion that deep features outperformed traditional features which are universally used, such as the SIFT, SSIM, and HOG
. Due to the application of deep learning, Squalli-Houssaini et al. [
24] presented their computation model, which is based on a deep learning frame to predict memorability scores while support vector regression is used in previous studies. When predicting image memorability, several different datasets and ground truth have been constructed by various scholars. Isola et al. [
2] established the Scene UNderstanding (SUN) memorability dataset and the Large-scale Memorability (LaMem) dataset was built by Khosla et al. [
21]. SUN memorability and the LaMem dataset are also the most used in further research. The point of adaptive semi-supervised feature selection [
25] is to attempt to apply memorability. Akagunduz et al. [
18] proposed the concept of Visual Memory Schema (VMS) and built the VISCHEMA image set. Based on these datasets, more research on image memorability will be carried out in the future.
As we have mentioned before, different media leave different impressions on people, some of which can be remembered, while others are ignored. Getting inspired by the memorability of the image, video memorability is studied gradually, and video memorability scores are defined as the percentage of correct detection of each video by different participants [
3]. The research on image memorability can be said to be the cornerstone of other media memorability research. In the paper by Goswami et al. [
26], they contributed to face memorability. WuLin Wang et al. [
27] raised a video hashing method based on memorability features. A system for memorability estimation was proposed by Han et al. [
28], which predicts the memorability scores of a video clip by learning from brain functional magnetic resonance imaging (fMRI). Influenced by these studies, Kar et al. [
3] began to wonder what makes a video memorable. Kar et al. [
3] proposed that the video frames can be used to extract features for predicting video memorability. Their studies have shown that saliency, color, scene complexity, background simplicity, object occurrence, and object attributes are related to the memorability scores. In the same year, Shekhar et al. [
29] focused on analyzing which fusing features could more accurately predict memorability scores. However, in previous experiments measuring video memorability scores, the data volume of the annotated dataset is insufficient. Cohendet et al. [
5] constructed a large-scale dataset named VideoMem, and the composition of VideoMem is 10,000 videos with corresponding video memorability scores. It is worth noting that VideoMem includes short-term human annotations and long-term annotations since the memory of people changed over time. Cohendet et al. proved that video semantics information is beneficial for predicting video memorability scores.
At the same time, Awad et al. [
30] attempted to enrich the memorability annotations of the dataset TRECVID 2019. They provided partial data with memorability annotations to carry out the memorability prediction task in MediaEval 2020 [
31]. A great many novel ideas have been conceived at previous MediaEval conferences [
32,
33,
34,
35,
36]. Such as MediaEval 2018, many researchers have put forward their ideas and carried out experiments to verify them. Lryva et al. [
37] used CNNs to extract video, text, and image features. Then, features were fused to obtain a vector as global features to predict video memorability. Lryva et al. [
38] proposed a video memorability prediction framework based on late fusion of text, visual and motion features. Kleinlein et al. [
39] proved that text features are effective in the representation of visual semantics required for the video memorability prediction model. Ali et al. [
40] propose a novel framework to fuse the text, visual and motion features to predict video memorability. In recent years, with the development of ensemble learning and cross-modal [
41,
42,
43,
44,
45], studies on ensemble learning and multi-modal are applied in various fields. Chen et al. [
46] presented a Group Ensemble Network (GENet) and a survey on ensemble learning [
47] was expanded. Zhou et al. [
10] proposed the method of domain adaptive ensemble learning. These research studies also encourage scholars to use ensemble learning methods to predict video memorability. Zhao et al. [
48] proposed the ensemble methods with text, image, audio, and video features that are extracted. However, the method of the multi-modal approach to predict memorability also has some problems. Simply speaking, the ensemble weight is not updated, and the size of the dataset they used is small. In the paper by Azcona et al. [
49], in MediaEval 2019, they used ensemble transfer learning methods with semantics and their extract features to predict media memorability scores. Specifically, each feature gets a relatively good result through the base learner, then each base learner is integrated into a strong learner by weight or others. Zhou et al. [
50] introduce the knowledge about ensemble learning in detail and provide the latest inspiration for the current research. In the latest study, Newman et al. [
7] analyze the influence of semantics and time decay for video memorability and construct a multi-modal memorability dataset named Memento10k. Unlike VideoMem, Memento10k includes the memorability annotations that occur delays ranging from several seconds to ten minutes. They propose a SemanticMemNet that can predict video memorability at an arbitrary delay.
With the enlightenment of the above academic research, we propose a new adaptive multi-modal ensemble network (AMEN) for video memorability according to the selected impact factors. It eliminates the error caused by the heterogeneous gap via integrating the optimal base learners corresponding to each media source instead of fusing the heterogeneous modalities directly. In addition, we design an adaptive learning strategy to update the weights based on the importance of memorability which is predicted by the base learners and contributes to obtaining the best performance without any manual tuning.
3. Proposed Methods
Many studies [
5,
7,
31,
46,
48] have proved that both vision and semantics play a key role in video memorability prediction. However, there is no definite content about the selection of impact factors for video memorability. We take the video itself, the video key frame and the corresponding captions of the video as video, image and text, respectively, and mine the feature information contained in these three media information first. Next, the features of each media information source and the individual learner required are analyzed and selected to be put into the ensemble network. In
Section 3.1,
Section 3.2 and
Section 3.3, we specifically elaborate on the reason for selecting features and individual learners needed in the proposed Adaptive Multi-model Ensemble Network (AMEN). Now, we briefly explain the symbols needed for feature selection and video memorability scores calculation. Given a video instance
= (
), where
is video feature,
is image feature and
is text feature. The memorability score obtained from the video information source of video
i is called video-output, termed as
, and the score obtained from the image source is addressed as image-output and defined as
. Similarly, the score obtained from the text source is called text-output, which is denoted as
. The final video memorability score for video
i obtained by our model is named video-memorability-output, which is referred to
and
n represents the number of samples.
3.1. Video Representation
ResNet [
51] made a name for itself and influenced the direction of deep learning in academia and industry. Research shows that the depth of the network is an essential factor in achieving good results. However, the gradient dispersion/explosion becomes an obstacle to the training of the deep network, leading to the failure of convergence. The ResNet model provides better performance of the network to make up for this disadvantage, but also becomes a relatively advanced network model. Considering the feature redundancy and other problems brought by the direct use of the network to extract video features, such as C3D features, we adopt the end-to-end Resnet3D model by feedback process to directly predict the video memory score, which can achieve better prediction effect and make full use of video information. The experimental results also prove our guess. Based on this, we fine-tune the ResNet3D model [
6] to use an end-to-end approach to predict video memorability scores rather than extracting features directly for video features.
Based on previous studies, we chose the ResNet3D model with 34 residual blocks and replaced the classification layer of the last layer with the full connection layers to obtain the memorability scores. Compared with the ResNet3D model fine-tuned by Cohendet et al. [
5], we added two hidden layers and one output layer in the full connection layer, with 100 neurons in the first hidden layer, ten neurons in the second hidden layer and output in the output layer. In the training stage, the loss function was adjusted from the
L1 loss function to the MSE loss function, and the optimization algorithm was adapted from the Adam algorithm to the SGD algorithm. Given a video instance
= (
), output values of the trained model were selected to obtain with the fine-tuned ResNet3D model instead of extracting the feature
directly. For example, let us define
as the parameter of the video branch, the learned video-output
of video
i can be represented as follows:
where
represents the function of video features
, which is used to obtain the memorability score of the video, and
=
.
3.2. Text Representation
Firstly, for text data, stopwords were used to process some unnecessary words. Then, we used the count-vectorizer method to extract text content as semantic features. For each training text, it only considers the frequency of each word. Countvectorizer converts the words in the text into a word frequency matrix, which uses the fit-transform function to calculate the number of times each word appears. After converting words into vectors, we used principal component analysis (PCA) to reduce dimension. PCA is a standard data analysis method that is often used for dimensionality reduction of high-dimensional data and can extract the main feature components of data. After PCA dimensionality reduction, the dimension of text features corresponding to each video is 500-dimensional. Considering that the Random Forest (RF) algorithm has strong model generalization ability and fast training speed and the multi-layer perceptron (MLP) can quickly regress high-dimensional features into a memorability score, we compared and analyzed these two methods through experiments to more accurately predict the memorability score of the base learner, and then the final video memorability score is predicted more accurately. Finally, we used Spearman’s correlation coefficient to judge the correlation. The experimental results show that the multi-layer perceptron (MLP) can predict the memorability scores more efficiently as an individual learner. The following results are described in
Section 4.2.1.
Given a video instance
= (
), transforms semantic information of video
i into 500-dimensional vector features
. Let us define
as the parameter of the text branch, the learned image-output
of video
i can be represented as follows:
where
defines the function of text features
, which is used to obtain the memorability score of the semantic derived from the video
i, and
=
.
3.3. Image Representation
For the selection of image features, we referred to some valuable image memorability literature, such as HOG, LBP, and RGB, which can be considered. Considering that the image information source may cause a lot of feature redundancy after fusing multiple features, or the size of the constituent feature vector is too large compared with the size of the dataset [
1]. After our previous experimental comparison [
4], we finally chose the LBP feature as the image feature of modeling. LBP is the abbreviation of Local Binary Pattern (Local Binary Pattern) and an effective texture description operator, which measures and extracts the local texture information of the image, which has obvious advantages such as gray invariance and rotation invariance. To a certain extent, the problem of illumination change is eliminated. In addition, it has the advantages of rotation invariance, low texture feature dimension, and fast calculation speed.
After determining the image features, based on our previous method of predicting video memorability [
4], Random Forest (RF), Support Vector Regression (SVR), and fully connected layer (MLP) were chosen as regression models to predict memorability scores. Random Forest (RF) shows higher efficiency in predicting memorability scores using LBP features than the other two methods. Given a video instance
= (
), we used the obtained LBP features
. Let us define
as the parameter of the text branch, the learned image—output
of video
i can be represented as follows:
where
denotes the function of text features
, which is used to obtain the memorability score of the image derived from the video
i, and
=
.
3.4. Weighted Ensemble
Ensemble learning is an algorithm that builds and combines multiple primary learners to achieve a more robust learning capability. Most integration methods use the same basic learning algorithm to produce homogeneous basic learners, that is, the same kind of learners produce homogeneous integrations, but in this paper, we tried to use a variety of learning algorithms to train different kinds of learners to produce heterogeneous integrations, and these learners were called individual learners. The ensemble learning method improves the generalization ability by combining a group of individual learners rather than choosing the best one among them, so the combination method used is very important.
After consideration, we used the weighted method to combine the individual learners of video, image, and text. The method is the most popular and primary combination method, assuming a set of T individual learner
, where the output of the learner
on example
x is
. The task is to combine the output result of
to obtain the final prediction result on the actual value variable. The weighted method brings the combined results by assigning different importance weights to the output results of each learner. The weighted method obtains mixed results by giving different importance weights for the output results of each learner. Specifically, it obtains the combined output
in such a way that
where
represents the weight of
, usually with the constraint
and
.
After we obtained
and
, as mentioned in
Section 3.1,
Section 3.2 and
Section 3.3, we utilized the weighted method to obtain the
. We assigned different initial weights to the outputs of respective models such as
,
,
for video
i. The initial learned
of video
i can be expressed as follows:
As described in
Section 1, video memorability is the probability of being remembered by people. That is to say, the final predicted video-memorability-output (
) is in the range of
. Therefore, the sigmoid function was used to control the prediction result within range
. The final video-memorability-output (
) is
Based on the video-memorability-output (
) for each video
i, we used short-term memory and long-term memory as groundtruth. We optimized our model by minimizing the mean square error (MSE) between the predicted value and the true value through the stochastic gradient descent (SGD) algorithm, in which the loss function
L can be calculated as follows:
where
represents the groundtruth of short-term or long-term memorability scores, the number of the sample is
n, and
denotes the predicted video memorability scores, which are video-memorability-outputs.
To prevent overfitting,
L1 regularization and
L2 regularization were added, and the final loss function
L was
where
is the regularization coefficient,
is
L1-norm and
is
L2-norm.
For
n samples, the video-memorability-outputs (
) is calculated represented as follows:
where
,
,
,
,
and
.
3.5. Evaluation Indicator: Spearman’s Rank Correlation Coefficient
In this paper, Spearman’s rank correlation coefficient is used to calculate the correlation between the video memorability scores predicted by our model and the real video memorability scores. The closer the Spearman coefficient is to 1, the higher the correlation between the predicted value and groundtruth. On the contrary, the closer the coefficient is to −1, the lower the correlation. Indicators commonly used in statistics to measure the correlation between two variables include Person correlation coefficient [
52], Spearman’s rank correlation coefficient [
53] and Kendall correlation coefficient [
54]. Pearson’s correlation coefficient applies to continuous data where there is a linear relationship between two variables and the population of the two variables is normally distributed or near-normal unimodal distribution. However, as can be seen from
Figure 1, our data are not normally distributed, whether short-term or long-term memory scores. Although Spearman’s rank correlation coefficient and Kendall’s can be used for data with non-uniform distribution, Kendall’s coefficient is more suitable for the multi-column rank correlation degree method. Furthermore, Spearman’s rank correlation coefficient has less strict requirements on data conditions than Pearson’s correlation coefficient. As long as the observed values of the two variables are paired with rank assessment data, Spearman’s rank correlation coefficient can be used to analyze the overall distribution patterns and sample sizes of the two variables.
Spearman’s correlation coefficient [
53] is defined as the Pearson correlation coefficient between rank variables. For sample size
n,
n original data are converted into rank data, and the correlation coefficient
is
where
and
are the predicted value and the groundtruth of video
i,
and
are the mean of
and
. Original data is assigned a rank based on its average descending position in the overall data. In practice, the links between variables are irrelevant, so the
can be calculated in a simple step. The difference between the ranks of the two variables observed,
is
where
and
are the order in the list after rearranging the original data
and
in ascending order.
3.6. Framework Overview
As is shown in
Figure 2, the AMEN framework adaptively integrates three individual learners to predict the final video memorability score based on the selection of the key features of the three media information sources and the corresponding individual learners. Specifically, the fine-tuned ResNet3D model, RF, and MLP that have been fine-tuned for the three information sources of video, image, and text are trained to obtain their respective output values and then use the weighted method to update the weight of each output value to obtain the final video memorability score.
Figure 2.
The overall framework of the proposed Adaptive Multi-model Ensemble Network (AMEN), see Algorithm 1.
Figure 2.
The overall framework of the proposed Adaptive Multi-model Ensemble Network (AMEN), see Algorithm 1.
Algorithm 1: AMEN training process. |
![Applsci 12 08599 i001]() |