A few recent works on audio and video matching methods are reviewed below. Further, the work of affective music content analyses and feature extraction, affective video content analyses and feature extraction, as well as cross-modal retrieval and matching are also discussed in the sections below.
2.1. Audiovisual Affective Feature Extraction
In music emotion recognition (MER), the extraction of musical features is essential for both classification and regression tasks. Music emotion recognition (MER) is an interdisciplinary computational task [
17], spanning fields such as psychology, neuroscience, and deep learning [
7]. Currently, research on music affective recognition is primarily focused on two aspects: improving the accuracy of affective feature extraction for improved affective content expression and enhancing the accuracy and effectiveness of emotion recognition to better identify and understand affective cues in music.
Traditional features such as MFCC and average power were used as input features for music in [
18,
19,
20,
21,
22]. In [
23], the author selected force, range, time domain, and other variables as variables to extract music features, and trained a forward neural network (FNN) to improve the average recognition accuracy of the model to 86%. In addition, in order to better identify music affective information, it is necessary to construct a suitable network. In [
15], an improved SVM algorithm was adopted for high-precision music classification. Hizlisoy et al. [
24] constructed a hybrid structure based on LSTM and DNN, providing the energy and MFCC of logarithmic Mel filter banks to a convolutional neural network layer to obtain audio features. Zhong et al. [
25] proposed a new network model, CBSA (CNN BiLSTM self-attention), in which a CNN is used to extract audio features, BiLSTM is used to capture temporal relationships between features, and a self-attention model is used to enhance the model’s representation ability.
Videos contain rich content and affective information, making research on video affective semantics significant. Currently, research on video affective analysis primarily revolves around refining methods for extracting affective feature. Traditional visual features primarily consist of color, texture, and region characteristics. With the advancement of deep learning technologies, these techniques are increasingly being utilized for the recognition and extraction of image emotional features [
26,
27,
28,
29,
30,
31,
32]. Liao et al. [
33] extracted two features, saliency target and facial target area, from an image and constructed corresponding deep learning frameworks to obtain their respective affective polarities. Lee et al. [
34] first extracted relevant information about the facial region in an image, and then used attention mechanism based methods to extract relevant scene information from the remaining regions. Finally, they combined both types of information to recognize emotions. Zhan et al. [
16] extracted three types of contextual information from images, including multimodal information, situational environment information, and depth map information, to complete affective recognition of image characters. This method can reduce dependence on facial expression information.
2.2. Audiovisual Association Matching
In the field of audiovisual cross-modal matching and retrieval, existing research has proposed methods such as label-based matching, self-supervised matching, and similarity-based matching. Label-based matching matches music and videos based on their label categories, considering a match when the categories align [
14]. Self-supervised learning is a variant of unsupervised representation learning that exploits the intrinsic characteristics of unlabeled data as supervisory signals [
35,
36]. Surís et al. [
37] proposed a self-supervised method that learns the direct correspondence between music and video from raw data, with artistic alignment of music and video serving as the supervisory signal for matching video segments with their corresponding music segments. Audiovisual matching methods based on similarity can be further divided into cognitive similarity and feature similarity. Cognitive similarity relies on human perception of emotions, such as using the valence–arousal (V-A) emotion model for emotional labels, and calculating the cross-modal similarity between music and video through cosine distance and Euclidean distance [
17]. In contrast, feature similarity-based matching analyzes the associations between data at the feature level, with a typical approach being to learn shared representations of feature data from different modalities.
Currently, based on the cross-modal information used when learning shared representations, cross-modal retrieval can generally be categorized into retrieval and matching methods based on inter-data similarity and those based on inherent data correlations.
Based on inter-data similarity: Masci et al. [
9] utilized a twin neural network to encode data from different modalities into binary codes, with performance superior to state-of-the-art hashing approaches. Cao et al. [
38] proposed a deep hybrid structure called the correlation hashing network (CHN), which addresses the similarity issue of data in the shared representation space. Shen et al. [
39] proposed a deep binary encoding method termed textual–visual deep binaries (TVDB) that is able to encode information-rich images and descriptive sentences. Liang et al. [
40] expanded the traditional linear projection by utilizing affine transformation techniques, thereby constructing a universal similarity model. Nie et al. [
41] proposed a novel deep hashing method (Dmfh) that can conduct feature learning and hash code learning simultaneously.
Based on inherent data correlations: Rasiwasia et al. [
42] proposed canonical correlation analysis (CCA) to enable joint modeling of text and images. Andrew [
43] introduced deep neural networks into the aforementioned concepts of CCA and defined deep CCA. Through the integration of DNNs, the linear mapping of data was substituted to achieve the maximization of data correlation. Shao et al. [
44] employed a sequence of progressively enhanced DCCA models to incrementally improve feature representation and matching performance, thereby modeling the correlation between images and text. Zeng et al. [
45] proposed a novel deep architecture composed of cluster-CCA and a deep triplet neural network model. This architecture fully considers the appropriate placement of each data point in the shared subspace based on pairwise correlations and semantic label assignments. Vukotic et al. [
46] detailed a method for weight distribution and structural symmetry between autoencoders of different modalities to enhance the correspondence between data. Wei et al. [
47] proposed an alternating optimization framework. This framework optimizes one modality’s autoencoder while keeping the weights of the other modalities’ autoencoders fixed, thereby enhancing the efficiency of cross-modal retrieval. He et al. [
48] generated pairs of similar and dissimilar data in the latent space of the joint representation using maximum likelihood estimation and conducted comparative observations between the two. Yan [
49] proposed a text-to-image retrieval system based on deep convolutional neural networks and utilized a margin-based objective function to constrain the similarity between data points. Zhang et al. [
50] employed a combination of generative adversarial networks (GANs) and hashing functions to achieve unsupervised cross-modal hashing. Nawaz et al. [
51] proposed a cross-modal retrieval method based on local features, which utilizes CNNs and RNNs to learn global and local features for each modality and employs a metric learning-based approach to achieve cross-modal matching. Gu et al. [
52] employed multiple generators and discriminators to accomplish the tasks of image generation and discrimination. Su et al. [
53] employed a bidirectional recurrent neural network to construct semantic relationships between images and text. Li et al. [
54] introduced the cycle GAN framework to learn the joint representation of data in both real and binary values. Wang et al. [
55] proposed a feature pseudo-labeling method named FPL. Integrating model predictions with feature similarity information enhances the robustness and accuracy of deep neural networks in the presence of label noise. Wang et al. [
56] introduced a multimodal sentiment analysis method named CCDA, which integrates self-attention and cross-attention mechanisms. By incorporating a cross-correlation loss function, it effectively captures and fuses information across different modalities, significantly improving the model’s performance in multimodal sentiment analysis tasks.
The key to cross-modal prediction tasks is to find a suitable feature fusion method. Traditional early fusion methods amalgamate features from diverse modalities at the extraction stage, typically through concatenation or weighted summation [
57]. Subspace-based techniques map disparate modal features into a common subspace for fusion at a higher-level representation [
58]. Attention-based methods learn importance weights for different modalities [
25]. Representation learning approaches aim to learn a shared subspace by optimizing their mutual correlation, yielding equivalent dimensional representations across all modalities [
43].
Canonical correlation analysis (CCA) is considered a conventional representation learning method that takes the linear transformation of two modal datasets as input. However, this approach fails to guarantee the efficacy of feature extraction when variables exhibit nonlinear relationships. To address this issue, kernel canonical correlation analysis (KCCA) and deep canonical correlation analysis (DCCA) are employed to learn complex transformations of distinct datasets. In addition, cluster-CCA establishes all possible correspondences and utilizes CCA for projection learning, segregating different categories within the shared subspace to enhance intra-class similarity among data from various modalities [
59].
KCCA utilizes kernel functions to map the original variables into a high-dimensional feature space, where potentially nonlinear relationships can be transformed into linear relationships [
60]. The objective of KCCA is to find the optimal projection directions for two sets of variables in this high-dimensional space such that the correlation between the projected sets of variables is maximized. The core idea of this method is that even if the relationships between variables in the original data space are very complex, these relationships can be simplified and revealed in the new feature space through appropriate nonlinear transformations. Unlike CCA, KCCA is suitable for high-dimensional data and complex data relationships, as kernel methods can map data into a high-dimensional space without increasing computational overhead.
The core idea of DCCA is to project data from multiple modalities into a low-dimensional space, maximizing the correlation of the projected data in this lower dimension [
41]. This method involves two deep neural networks, each dedicated to representation learning for data from the two modalities. In nonlinear feature mapping, DCCA exhibits superior performance, capable of maximizing the associativity of multimodal data and thereby addressing the issue of nonlinear transformation of input data. In contrast to CCA and KCCA, DCCA is tailored for temporal contexts, as it is capable of capturing the dynamic relationships between data points at various time points.
Cluster-CCA is a method employed when data can be categorized into distinct clusters or groups, facilitating the analysis of variable relationships between these clusters. By utilizing standard CCA for projection learning, it aims to maximize the correlation between two different modalities. Within the shared subspace, cluster-CCA seeks to isolate different categories, thereby enhancing the similarity of data from various modalities within each category. However, when it is applied to large datasets, the computational complexity is high and the processing efficiency is low. As with the above methods, this method is also suitable for small and medium-sized datasets. In the field of audiovisual retrieval, Zeng et al. [
45] have achieved remarkable results by employing cluster-CCA for feature fusion, reaching an accuracy of 74.66% on the VEGAS dataset.
According to the above research, most existing algorithms measure cross-modal data in a common space for matching and retrieval. However, these methods may not finely capture the correspondence between different modalities, posing a challenge to effectively discover and capture valuable information.
To address the aforementioned issues, this paper integrates the cognitive and perceptual characteristics of emotions, constructing a matching model based on label similarity and feature fusion. In terms of feature fusion, this study used statistical methods to represent audio and video data as two multivariate datasets and established connections between them. Subsequently, the model utilized CCA and its variants to find the maximum correlation between these datasets, aiming to identify the greatest match between the two. On one hand, feature fusion can overcome the gap in cross-modal heterogeneity. On the other hand, constructing a factor space based on affective labels can fully explore the affective correlations across different modalities.