Most Significant Impact on Consumer Engagement: An Analytical Framework for the Multimodal Content of Short Video Advertisements

Abstract

The increasing popularity of short videos has presented sellers with fresh opportunities to craft video advertisements that incorporate diverse modal information, with each modality potentially having a different influence on consumer engagement. Understanding which information is most important in attracting consumers can provide theoretical support to researchers. However, the dimensionality of the multimodal features of short video advertisements is often higher than the available data, posing specific difficulties in data analysis. Therefore, designing a multimodal analysis framework is needed to comprehensively extract and reduce the dimensionality of the different modal features of short video advertisements, thus analyzing which modal features are more important for consumer engagement. In this study, we chose TikTok as the research subject, and employed deep learning and machine learning techniques to extract features from short video advertisements, encompassing visual, acoustic, title, and speech text features. Subsequently, we introduced a method based on mixed-regularization sparse representation to select variables. Ultimately, we utilized multiblock partial least squares regression to regress the selected variables alongside additional scalar variables to calculate the block importance. The empirical analysis results indicate that visual and speech text features are the key factors influencing consumer engagement, providing theoretical support for subsequent research and offering practical insights for marketers.

1. Introduction

The rise of Web 2.0 ushered in the golden age of short videos. Short video platforms incorporate advertising elements into entertainment-oriented short videos and recommend them to potentially interested consumers based on targeted recommendation algorithms, giving them a natural advantage in traffic aggregation [1]. This “advertising + short video platform” business model has also been widely adopted by platforms such as TikTok, Kwai, and YouTube. According to a report by QuestMobile [2], as of December 2024, TikTok’s deduplicated active user base reached 978 million, far exceeding that of other short video platforms. This vast user base provides TikTok with immense market potential. Advertisers, brands, and influencers alike can find a large number of potential consumers or viewers on this platform. This market potential helps attract more business collaborations, driving TikTok’s commercialization and profitable growth.

On short video platforms, users can customize their short videos with color filters, video editing tools, background music, and subtitles, easily enabling self-expression [3]. Influencers, based on their personal brands, gather followers with similar interests. When the number of followers accumulates to a certain level, they begin to monetize through short video advertisements or live streaming sales [4]. Short videos are essentially social media that combine audiovisual elements. For influencers, one of the greatest challenges lies in attracting as many consumers as possible by posting engaging content [5]. This aspect has also garnered attention from the academic community.

Therefore, researchers have begun to extract features from various modalities of short videos to explore the antecedents that influence consumer engagement, interaction, and purchasing behaviors. For example, Wei et al. [6] extracted consumers’ perceived warmth and competence traits from the comment texts and title texts of short videos and found a significant impact on consumer engagement. Xiao et al. [7] extracted multimodal features from the speech text, audio signals, and visual signals of short videos and similarly discovered the influence of these features on consumer engagement behavior. However, it seems that there is no framework available to quantitatively inform researchers which modal features are the key factors influencing consumer engagement in short video advertisements.

Essentially, short videos are social media platforms that integrate audiovisual elements and contain various modal data [8]. It is necessary to identify which modal features are the key factors influencing consumer engagement, providing theoretical support for researchers studying short videos and optimization suggestions for short video marketing. Lu et al. [9] made a preliminary exploration of this problem. They extracted visual, acoustic, and title features from short videos through quantitative methods and found that title and visual features are important factors influencing user engagement in short videos. However, they overlooked the importance of speech text features. Furthermore, their research only focused on short entertainment videos, and whether the multimodality of short video advertisements exhibited consistency remains to be further explored.

With the aim of resolving this issue, this study designs a multimodal analysis framework that extracts multiple modal features from short video advertisements, optimizes variable selection methods based on regularization representations, and performs multiblock regression to analyze which features are more significant for consumer engagement. As shown in Figure 1, we designed a multimodal data analysis framework to investigate the impact of short video advertising content attributes on consumer engagement. Firstly, VGG16+LSTM [10,11], Fourier transform, and Word2Vec [12] were used to extract the visual, acoustic, and title text features of short video advertisements, respectively. Using the Baidu API, the audio was converted to speech text, and the speech text features were retrieved using the BERT model [13]. The different modalities of features were grouped together as different blocks. Subsequently, given that some blocks had extremely large dimensions, a mixed-regularized sparse representation-based method was proposed to select features for these blocks independently. Finally, the chosen multimodal features and consumer engagement indicator variables were regressed using the multiblock partial least squares (MBPLS) method [14], and the block importance (BIP) is utilized to compare various data patterns and calculate the actual contribution of each block to the prediction of consumer engagement.

2. Related Work

2.1. Short Video Advertising

The content of short video advertising is typically more condensed than that of regular adverts, catering to users’ fragmented and fast-paced viewing patterns. They are frequently presented in imaginative and engaging ways, which better grabs viewers’ attention and piques their interest. As a result, short videos are more popular and possess a larger user base. The vast user base of short video platforms provides ample opportunities for sellers to promote their products [15]. This marketing strategy is known as short video business, and the short videos containing product promotions are considered short video advertisements [16]. Several platforms can be used to edit short videos. For example, TikTok has powerful editing and customization functions that sellers can use to market their products with an exciting style to attract consumers. Consumers can interact with sellers by liking, commenting, collecting, sharing videos, and purchasing their preferred products through the “little yellow car” link. Unlike traditional video advertising, short videos effectively reach consumers during various fragmented moments, shaping the product brand image and inciting consumer purchasing intent through repeated playbacks [17].

Current research primarily focuses on influencers and the content attributes of short videos. For influencers, Zhang and Zhang [18] discovered that both the presentation style and sales approach of influencers in short video advertisements could significantly influence consumers’ neural engagement. Haq and Chiu [19] found that the positive image of influencers could be transferred to the platform and product image, thereby influencing consumers’ intention of online engagement as well as their emotional, cognitive, and social behaviors. Tan et al. [20] extracted images of doctors from short videos and found that the image features were significantly correlated with the behavior of the viewers’ engagement, and the doctor with formal uniforms or videos taken in the study room or at home on the white wall would receive more engagement.

In terms of short video content attributes, Chen et al. [21] reported that the duration, title, dialogue cycle, and content type of short videos released by the health sector in China would affect the level of user engagement. Using the text mining method, Xiao et al. [16] found that the performance expectation, entertainment, contact strength, and sales approaches of short video advertisements significantly correlated with consumer engagement, and product type had moderate effects. Zhang et al. [22] reported that the combination of audio features in short videos could enhance user engagement. Through the study of content matching, information association, story, and emotion of short videos, Dong et al. [23] found that the content features of short videos markedly affected consumer engagement, and the release time of videos had moderation effects between short video emotion and consumer engagement.

Essentially, short video advertisements contain multiple modes of content, including visual, acoustic, and speech text. Visual content refers to all components that make up a short video and convey information and feelings through visual elements, such as images, colors, lighting effects, etc. Acoustic content mainly refers to sound elements in short videos that affect viewers’ auditory perception, including human voices, music, and sound effects. Speech text content mainly refers to the written content presented through speech in short videos, which can directly convey information and express emotions, working together with the video images to create the complete content and atmosphere of the short video. Title content is a brief and concise text description set for the video to summarize or attract viewers to click and watch, which can affect the video’s exposure, click-through rate, and viewers’ first impression. In order to provide researchers with a theoretical basis for selecting the content modalities and to offer influencers advice on creating engaging video content, it is necessary to further explore which factors are the key influencers in short video marketing.

2.2. Consumer Engagement Behaviors

Consumer engagement behaviors refer to the various actions and decision-making processes exhibited by consumers during the purchase, use, and evaluation of products or services [6,7,8]. It serves as a pivotal metric for assessing the effectiveness of corporate activities, particularly drawing significant attention in the field of relationship marketing [16]. This behavior stems from consumers’ intrinsic motivations, encompassing word-of-mouth promotion, recommendations, assistance to others, posting reviews, and even legal actions [24]. It embodies the depth of consumers’ connections with services, products, brands, and events, serving as the cornerstone of brand relationship building and reflecting consumers’ cognitive, emotional, and behavioral responses towards brands [6,25].

In the realm of short videos, consumer engagement centers around three key dimensions: emotional connection, cognitive processing, and specific behaviors [6,16,23], with behavioral engagement (likes, comments, collects, and shares) being particularly crucial [7]. This is attributed to three reasons: firstly, the behavioral engagement indicators on short video platforms are intuitive and susceptible to the bandwagon effect, becoming vital benchmarks for evaluating video quality [26]; secondly, videos with high engagement can garner more exposure through the platform’s traffic allocation mechanism, directly impacting sales [6]; and thirdly, behavioral engagement is an external manifestation of consumers’ emotional and cognitive investments, with high-quality video content prompting greater behavioral engagement [7,27]. Furthermore, behavioral engagement also mirrors consumers’ evaluations, feedback, and suggestions regarding brand services [7]. Specifically, likes signify emotional identification and social identity, comments reflect opinions, emotional feedback, and social interaction [28], collects cater to content preferences and review needs [6], while shares underscore recognition, recommendation, and social interaction demands, embodying trust in the platform [7]. Understanding these behavioral patterns holds practical significance for short video marketing strategies.

2.3. Multimodal Content Analysis

The multimodal content analysis model involves various aspects, including the understanding and generation of multiple modalities such as images, videos, audio, and text. Researchers typically use machine learning and deep learning algorithms to predict patterns based on specific modal features or explore the impact of multimodal content on consumer behavior using regression models. In terms of pattern prediction, Fu et al. [29] extracted multimodal features from short video advertisements and designed a time series prediction model using a hierarchical attention model and customized LSTM, predicting advertisement sales effectively. Guo et al. [30] extracted shared, unique, and conflicting information from multimodal features, using a hierarchical multitask classification module to capture dependencies, accurately predicting consumer responses. Yang et al. [31] calculated engagement and product heat maps based on video dimensions, length, and product image matching, deriving a score that captures product engagement within the video.

In terms of consumer behavior, Xiao et al. [7] extracted emotional features from speech text, energy features from audio signals, and color perception features from video signals in short video advertisements, discovering through regression models the significant impact of these multimodal features on consumer engagement behavior. Wang et al. [32] found that human voiceovers in short video advertisements can reduce consumer cognitive load and enhance purchase intention compared to AI voiceovers. Additionally, they discovered that human voiceovers are preferred without subtitles, while the difference diminishes with subtitles. Lu et al. [9] extracted deep features of acoustics, vision, and titles from various entertainment-type short videos using deep learning methods, finding the importance of titles and visual features in consumer engagement behavior.

These studies typically select several different modal features to explore specific issues. However, the basis for selecting different modal features lacks objective evidence. Although Lu et al. [9] conducted a preliminary exploration of the importance of multimodal features in entertainment-type short videos, they overlooked the crucial speech text feature in short videos. Furthermore, the consistency of the importance of these multimodal features in the context of short video advertisements remains to be further explored. Therefore, this study aims to extract visual, speech text, title, acoustic, and scalar features from short video advertisements, design multimodal analysis algorithms, and to explore the key features that influence consumer engagement behavior.

2.4. Multimodal Features and Variable Screening Methods

Different modes affect the user’s choice in a multimodal system, and multimodal analysis often involves high-dimensional features of these modes. Feature extraction becomes crucial when dealing with numerous variables [33]. In the current research, the three multimodal features in short videos that researchers are typically most concerned with are visual, textual, and acoustic features.

Regarding visual features, researchers typically rely on machine learning and deep learning techniques to capture images from video frames and compute the visual features of the video based on these images. For example, Xiao et al. [7] used OpenCV to extract color features from each frame of the entire short video, calculating the overall color features of the video by averaging color values. However, this approach ignores the issue of varying lengths in short videos. Lu et al. [9] further confirmed the importance of the first 10 s of short video advertisements and proposed using the ResNeXt-50 model to extract video frame features on a second-by-second basis, which exhibits high efficiency in batch and systematic processing of video data. Additionally, there are more complex image feature extraction methods. Yu et al. [34] used the quantum evolution method to extract the inner image features based on the quantum evolution method. Tian and Wang [35] utilized a quantum kernel clustering method to test the image marginal features, which showed evident feature distribution and sound consistency.

Regarding textual features, natural language processing has made great strides. For example, Lu et al. [9] successfully extracted deep features from short video titles using the Word2Vec method. In fact, if the training sample is large enough, the traditional Word2Vec method can effectively extract word vectors for vocabulary modeling [12]. However, Word2Vec is a method based on word vectors and has limited understanding of sentence semantics. Therefore, researchers have proposed large language models such as ERNIE [36] and BERT [13], which can learn a significant amount of prior language knowledge through pre-training. This significantly enhances the effectiveness of semantic extraction and, therefore, they are also widely applied in the extraction of deep features for empirical research [9,37].

Regarding acoustic features, researchers typically rely on the Fourier transform to extract features such as the spectral centroid, spectral variance, spectrum bandwidth, zero-crossing rate, and RMS (Root Mean Square) energy [7,9]. These features often carry different implications and can typically provide a good explanation for a particular aspect of the audio. For instance, spectral variance can indicate the degree of audio diversity, spectral centroid can represent the characteristics of pitch, and zero-crossing rate can reflect the level of audio fluctuation frequency [7]. Furthermore, there are also hybrid feature extraction methods. Hu and Flaxman [38] used deep neural networks LSTM [10] and an initial model to extract text features and image features, respectively. Gallo et al. [39] encoded the text through the convolution, maximum pooling, and complete connection layers. Then, they drew the extracted text features on the image to enhance the image information.

Short videos have multimodal attributes, which result in higher dimensions of extracted features compared to the sample number. Selecting the variables of the extracted short video features is necessary. The feature selection process aims to reduce the dimensions of the feature space by screening a subset of the original feature space, which can effectively reduce the dimensions of the feature space [40]. Many studies have emphasized the importance of feature selection and its optimization process [41,42,43,44]. As a result, Zhao and Liu [41] proposed a spectral feature selection that measured feature correlation and developed feature selection based on the correlation by constructing spectral maps. Li et al. [42] used non-negative spectrum analysis in non-negative discriminative feature selection to obtain more accurate cluster labels and guide the feature selection process. Shi et al. [43] proposed a robust spectral feature selection method that utilized a stable partial learning method and spectral regression to build graphs and process the learned errors. Zhang et al. [44] suggested an optimization method incorporating space reduction and local filter search techniques based on average mutual information and feature redundancy to tackle the feature selection problem through global exploration capabilities.

3. Materials and Methods

3.1. Data Collection and Pre-Processing

Nelson [45] classified products into search products and experience products based on the ability of consumers to assess the products before making a purchase. Search products can be quickly evaluated by comparing product qualities, while experience products can only be assessed by physically seeing, smelling, touching, or using them [46].

In this study, food (such as snacks, beverages, light refreshments, bread, noodles, and so on) is used to represent experience products, and digital products (such as portable chargers, mobile phone accessories, speakers, headphones, and so on) are used to represent search products. According to Xiao et al. [16], consumer engagement metrics stabilize when short videos are posted for over 30 days. Therefore, we selected short video advertisements posted over 30 days to maintain the variables’ validity. From January to May 2023, we gathered 906 short video advertisements related to food and digital products from data analysis websites for short videos and live broadcasts (https://dy.huitun.com (accessed on 1 July 2024)). After excluding invalid and missing data, we collected 501 in the food category and 405 in the digital product category. For each video, the collected information includes the video title, video publishing time, video duration, likes, comments, collects, shares, author fan numbers, and product price.

Then we used the web crawler tool Python (version 3.7) to crawl these 906 short video advertisements through video links and extracted the features through machine learning and deep learning methods. As shown in

Table 1. Extracted features of short video advertisements.

Modality Blocks	Features	Item	Description	Dimension
Block 1	Visual	Beginning image	Features of the first 10 s of a short video frame	1280
Block 2	Acoustic	Zero-crossing rate	The rate of symbol change in acoustic signals	1
		Spectral centroid	The position of the spectral centroid	1
		Spectrum roll-off	Measurement of the shape of acoustic signals	1
		Chromaticity frequency	The 12 different semitones or chromatics in music	1
		Spectral bandwidth	The effective range of the frequency spectrum	1
		RMS energy	The energy or intensity level of the sound signal within the selected time period	1
		MFCC	Reflecting the nonlinear perception features of the human ear to sound frequencies	20
Block 3	Speech text	Author’s description	Speech text of short video authors	768
Block 4	Title	Video title	The average value of word vectors of titles	100
Block 5	Scalar	Duration	Number of seconds of a short video advertisement	1
		Speed	The speed of voice playback	1
		Title length	Word count of title	1
		Time gap	The released days of the short videos	1
		Product price	The price of the products	1

, the extracted features include the title, speech text, acoustic, visual, and scalar. Scalar features include the duration, speech speed, title length, published time, and product price of short video advertisements. The likes, comments, collects, and shares of short video advertisements are combined as the indicator variables of consumer engagement. In order to ensure that our method is properly tested and to avoid errors caused by the sample distribution problem, we evaluated the statistical distribution of consumer engagement indicator variables. As shown in

Table 2. Statistical distribution of consumer engagement.

Statistic	Likes	Comments	Collects	Shares
Skewness	6.12	22.24	5.74	6.87
Kurtosis	46.14	542.57	40.19	67.81
Max	1,069,000	690,000	266,000	302,000
Min	175	15	8	47
Mean	44,581.63	3229.13	11,023.74	8164.47
Std	96,193.25	26,367.08	26,193.67	20,462.20

, the skewness and kurtosis of the indicator variables exceed the acceptable range of normal distribution (skewness > 3; kurtosis > 7), indicating excessive dispersion of the dataset. Therefore, in this study, we use the logarithm of indicator variables for analysis. Furthermore, in our later writing, likes, comments, collects, and shares stand for In (likes), In (comments), In (collects), and In (shares), respectively.

3.2. Feature Extraction

Short video advertisements can be analyzed by extracting features, which encompasses identifying visual, speech text, acoustic, title, and scalar elements, and examining these components to understand their contributions to the advertisements’ overall message and impact on consumers.

Visual feature extraction: Ouzir et al. [47] conducted an analysis of consumers’ electroencephalogram (EEG) while they watched short video advertisements and found that consumers’ perception of changes within each second of the short video frames was not significant, allowing for the analysis of consumers’ cognitive processes on a second-by-second basis. Furthermore, they discovered that consumers’ brains only need a few seconds to process the visual features on social media. Xiao et al. [7] indicated that in short videos, consumers typically use only a very short amount of time to judge their interest in the current content and to decide whether to continue watching. This point was also supported by Lu et al. [8], who demonstrated that the visual content within the first 10 s of a short video is a crucial factor influencing consumer engagement.

Based on their research, our study captured the first ten seconds of the short video advertisement by second, using a pre-trained network, visual geometry group 16 (VGG16) [11], to extract the features of each image into a 4096-dimensional vector. Considering that after extracting visual features using VGG16, we needed to further extract temporal sequence features using LSTM, we excluded the three hidden layers of VGG16 in order to avoid the influence of task-specific information that they might introduce on the feature extraction process. Given the continuous nature and strong temporal continuity of short videos, we employed the LSTM model to extract the time-series features from these 10 images, which generated a 1280-dimensional vector to represent the visual features of the short video. The basic structure of LSTM consists of four main components: the input gate, the forget gate, the output gate, and the memory cell. LSTM is a special type of recurrent neural network that excels in processing time-series data and can handle long-term dependency issues with high performance. The playback of short videos was a continuous process with temporal solid continuity. The LSTM model was used to extract the time-series features of these 10 images. Finally, a 1280-dimensional vector was generated to represent the visual features of the short video.

Speech text feature extraction: The Baidu API was used to convert the speech to text for each short video advertisement. Since speech text is usually concise and contains strong logic, most short videos have speech text of less than 512 characters. The content at the beginning of the short video plays a crucial role in determining whether the viewer continues to watch it. The Bert model [13] was used to extract features of the first 512 characters of each speech text. Specifically, the official pre-trained model of Google, Bertbase Chinese, was selected to train the speech text. The features of speech text were characterized through the first unique marker “[CLS]” vector of the pre-training model, which encoded the semantics of the entire sentence.

Title feature extraction: The title of each article in a corpus of 162,000 Chinese articles was downloaded as training data using the Python package Jieba. The titles were then segmented, stop words were removed, and lexical representations of each title were obtained for word vector training. The corpus’s XML files were converted to text files, and the text files underwent pre-processing that simplified stopword removal and traditional Chinese word segmentation to prepare the data for training. Then, the Chinese word vectors were trained using the Word2Vec algorithm in the Python package Gensim. Finally, each word of the titles was transformed into a 100-dimensional vector, and the average of all word vectors for each title was used to represent the title feature.

Acoustic feature extraction: The Python Librosa analyzed acoustic signals in frames using the Fourier transform. A set of acoustic features was extracted using the zero-crossing rate, spectral centroid, spectrum roll-off, RMS energy, spectrum bandwidth, chromaticity frequency, and Mayer frequency cepstrum coefficient. Extracting various features from each frame of an acoustic signal is possible. These features include the zero-crossing rate, spectral center, spectrum attenuation, RMS energy, spectral bandwidth, and chromaticity frequency. Moreover, the Mel frequency cepstrum coefficient, a small set of 20 coefficients, can also be extracted. As a result, 26-dimensional features can be extracted from each frame of the acoustic signal.

Scalar feature extraction: The duration, speech speed, title length, gap time, and product price of each short video advertisement were used as scalar features.

This study used five features from short video advertisements, including scalar, speech text, title, acoustic, and visual features, for multimodal analysis. Each short video divided the scalar, speech text, title, acoustic, and visual features into five blocks, resulting in a 2179-dimensional vector comprising all blocks.

3.3. Method

3.3.1. Mixed-Regularization Sparse Representation-Based Method

The $l_2$ norm is unsuitable for variable screening, and the $l_1$ norm can eliminate numerous highly correlated variables. Therefore, we have devised a mixed-regularization sparse representation (MSR) method to refine the $l_1$-regularization-based variable screening approach. This innovative method integrates the $l_1$ and $l_2$ norms, thereby enabling us to preserve crucial, highly correlated variables and bolster the overall stability of the screening process. Before discussing the MSR method, we first specify some symbols: for vectors $a={(a_1,a_2,\dots {,a}_p)}^T$, the $l_1$ and $l_2$ norms of $a$ are ${‖a‖}_1=\left|a_1\right|+\left|a_2\right|+\dots +\left|a_p\right|$ and ${‖a‖}_2={(a_1^2+a_2^2+\dots +a_p^2)}^{1/2}$, respectively, where $\left|a_i\right|$ represents the absolute value of $a_i,$ and the superscript $T$ represents the transposition of a matrix or vector.

For the data $X=(X_1,X_2,\dots {,X}_p)\in R^{n\times p}$ and $y={(y_1,y_2,\dots {,y}_n)}^T\in R^{n\times 1}$, where $n$ is the data number, and $p$ is the feature number. The core of the MSR method is to find a weight vector $w={{(w}_1,w_2,\dots ,w_p)}^T\in R^{p\times 1}$ to combine the independent variable $X$ to allow the distance between $X$ and $y$ be closer. This process indicates that MSR must find the weight vector $\tilde{w}$ that minimizes the right side of (1).

$$\tilde{w}={argmin}_w\{{‖y-Xw‖}_2^2+\lambda (\rho {‖w‖}_1+(1-\rho ){‖w‖}_2^2\left)\right\}$$

(1)

where $\lambda$ is the penalty coefficient; $\rho$ is the mixed parameter with the value in [0, 1].

Due to the fact that in solving the minimum value of Equation (1), the absolute value is not differentiable, we use the coordinate descent method [48] to solve this optimization problem. For Equation (1), we rewrite each element of $w$ as an independent variable to Equation (2), then the optimization problem is transformed to solve for the minimum value of $f\left(w\right)$.

$$f\left(w\right)=f\left(w_1,w_2,\dots ,w_p\right)={‖y-Xw‖}_2^2+\lambda (\rho {‖w‖}_1+(1-\rho ){‖w‖}_2^2$$

(2)

Considering that $f\left(w\right)$ is a differentiable convex function, the local minimum value of $f\left(w\right)$ is the global minimum value. Therefore, we sequentially find the partial derivative of $w_i$ ($i=\mathrm{1,2},\dots ,p$), calculate the minimum value of the univariate linear equation, and conduct a descent calculation to obtain the global minimum value of $f\left(w\right)$. The $t$-th $(t=\mathrm{1,2},..,N)$ iteration process is shown in (3):

$$\begin{gathered} w_1^{\left(t\right)}={argmin}_{w_1}f(w_1,w_2^{\left(t-1\right)},w_3^{\left(t-1\right)},\dots ,w_p^{(t-1)}) \\ {w}_2^{\left(t\right)}={argmin}_{w_2}f(w_1^{\left(t\right)},w_2,w_3^{\left(t-1\right)},\dots ,w_p^{(t-1)}) \\ \dots \\ {w}_p^{\left(t\right)}={argmin}_{w_p}f(w_1^{\left(t\right)},w_2^{\left(t\right)},\dots ,{w_{p-1}^{\left(t\right)},w}_p) \end{gathered}$$

(3)

The values of $w_{i+1}$ to $w_p$ in $f\left(w\right)$ are all taken from the ($t-1)$-th iteration, and the values of $w_1$ to $w_{i-1}$ are all taken from the $t$-th iteration. The goal is to determine the current minimum value of the parameter $w_i$ of the $t$-th iteration. All values of $w$ are updated in every iteration. When the value of the iteration error ${‖w^{\left(t\right)}-w^{\left(t-1\right)}‖}_1$ is less than the given threshold, we consider the descent to be convergent, where $w^{\left(t\right)}$ and $w^{\left(t-1\right)}$ are computed in the $t$-th and the ($t-1)$-th iteration, respectively. When the descent is convergent, $\tilde{w}$ is estimated by $w^{\left(t\right)}$. Then, $f\left(\tilde{w}\right)$ is the minimum value of (2). The independent variables corresponding to elements of $\tilde{w}$ with a value of 0 will be removed, thus achieving the effect of variable selection. The iterative process is shown in Algorithm 1.

Algorithm 1. Coordinate descent method

Input: Given a starting point $w_0\in R^{p\times 1}$

Repeat       for $t$ = 1, 2,…, $p$ do            for $i$ = 1, 2,…, N do                             $w_i^{\left(t\right)}={argmin}_{w_i}f(w_1^{\left(t\right)},\dots ,w_{i-1}^{\left(t\right)},w_i,w_{i+1}^{\left(t-1\right)},\dots ,w_p^{(t-1)})$            end        end Until convergence

Output: $w_i^{\left(t\right)}$ when convergence

3.3.2. Multiblock Partial Least Squares

The multiblock partial least squares (MBPLS) method can be used in data with a large number of variables and blocks [49]. In MBPLS, different blocks are processed in parallel to fit the regression model with the dependent variable [50]. The results of the calculation can be used for interpretation and prediction. These include the block weight, load, and score, as well as the super weight and super score. Therefore, MBPLS’s suitable block scaling can offers better explanations than the standard partial least squares method, as it can store all variables in a single block [51]. Super-weights indicate the relative importance of blocks, facilitating an explanation of noteworthy blocks and dominating variables influencing them. The proportional relevance of the variables in the block depends on their numerical values because super-weights in MBPLS are generated using block scores.

In the partial least squares (PLS) method, for each potential variable (LV) number, the predicted block $X_i$ and the response variable $y$ are decomposed into the following equations:

$$X_i=t_i{p}_i^T+E_i$$

(4)

$$y=u{q}^T+F$$

(5)

where $i=\mathrm{1,2},\dots M$; $M$ is the number of blocks; $t_i$ and $u$ is the score vector; $p_i$ and $q$ is the load vector, $E_i,$ and $F$ is the residual error. The initial score vector $u$ extracted from the response variable $y$ is in all blocks $X_i$ for each LV number. This process is used to regress the assigned variable weights to $w_i$. Then, $w_i$ and $X_i$ are multiplied to obtain a block score $t_i$. All $t_i$ are combined to obtain the superblock $B=\left[t_1,t_2,\dots ,t_M\right]$. According to (5), the super weight $w_t$ and super Score $t_t$ are obtained as follows:

$$w_t={\displaystyle \frac{Bu}{u^{T}u}}=[w_{t_1},w_{t_n},\dots w_{t_M}]$$

(6)

$$t_t={\displaystyle \frac{B{w}_t}{{w_t}^T{w}_t}}$$

(7)

In (5), $w_{t_i}$ corresponds to the weight of variables in the block $i$. When the LV number of each block is the same, $t_t$ will be equal to the score in regular PLS, making MBPLS perform quantitative predictions like PLS. The significant benefit of MBPLS is that it improves the model’s interpretability since the relative importance of blocks is determined by $w_t$, which creates a better explanation of the blocks of interest and their dominant factors. However, $w_t$ depends on the numerical value of the variable in the block. Moreover, blocks with larger values may not contribute more to the MBPLS model, as certain variables may not provide predictive information. In addition to $w_t$, a more plausible explanation and prediction result might arise by assigning each block a weight that represents its contribution to the model.

4. Results

4.1. The Result of MSR

In order to investigate whether the proposed variable screening method can effectively achieve the goal of variable screening, we tested the variable screening results of the MSR method under different values of the penalty coefficient and the mixed parameters. We evaluated the magnitude of the independent variable’s representative error (RE) value in relation to the consumer engagement indicator variables. The penalty coefficient $\lambda$ can increase the sparsity of the MSR method. Therefore, we suggest $\lambda$ take values at [0.1, 2]. If $\lambda$ is too small, MSR cannot achieve the variable screening. If $\lambda$ is too large, MSR will eliminate too many variables. Therefore, the parameter $\rho$ for mixing is meant to combine the regularized representations of $l_1$ and $l_2$. When testing the MSR method, we only considered the values of $\rho$ at 0.25, 0.5, and 0.75. Considering that the dimensions of scalar features are too small (only 5 variables), we only performed variable selection on visual, speech text, title, and acoustic features.

Specifically, we considered the variables of these four modalities as four blocks and applied the MSR method to perform variable selection for each block individually. The results of visual feature variable selection for food and digital products short video advertisements under different parameters are shown in Figure 2 and Figure 3. As $\lambda$ increases, more variables are removed. For the same $\lambda$, the larger $\rho$ is, the more variables are removed, and this trend remains consistent for each consumer engagement indicator variable for food and digital products. Therefore, we calculated the mean RE ($M_{RE}$) value of these four consumer engagement indicator variables under each set of parameters $\lambda$ and $\rho$. To avoid excessively eliminating variables, we selected the value of ($\lambda$, $\rho$) as the final parameter setting when the ER values no longer decrease significantly as ($\lambda$, $\rho$) increases. For food short video advertisements, when ($\lambda$, $\rho$) = (0.5, 0.5), $M_{RE}$ value significantly decreases to 0.39, and no longer changes significantly with the increases of ($\lambda$, $\rho$). For digital product short video advertisements, when ($\lambda$, $\rho$) = (0.5, 0.75), $M_{RE}$ value significantly decreases to 1.50, and no longer changes significantly with the increases of ($\lambda$, $\rho$). At this point, the variable dimensions for both types of products were reduced by at least 60%, so we selected these two sets of ($\lambda$, $\rho$) as the visual features.

The process of selecting parameters for variable selection of speech text, title, and acoustic features is similar. We have included the results of these features under different parameter selections in the Figure A1, Figure A2, Figure A3, Figure A4, Figure A5 and Figure A6. Finally, the selected parameters for different consumer engagement indicator variables of food and digital products are shown in

Table 3. Parameter values of each feature.

Feature Type	Parameter (Food)		Parameter (Digital Products)
	$\mathit{\lambda }$	$\mathit{\rho }$	$\mathit{\lambda }$	$\mathit{\rho }$
visual	0.5	0.5	0.5	0.75
speech text	0.5	0.25	0.5	0.5
title	0.5	0.75	0.5	0.5
acoustic	1	0.75	0.5	0.75

, and the reserved variables are listed in

Table 4. Number of variables retained by each block under different parameters.

Feature Type	Food				Digital Products
	Likes	Comments	Collects	Shares	Likes	Comments	Collects	Shares
visual	500	479	482	477	301	270	262	310
speech text	436	436	395	410	240	228	208	233
title	48	50	53	58	58	57	48	59
acoustic	14	14	12	13	14	18	13	18

. The variable selection process of MSR involved selecting parameters ($\lambda$, $\rho$) to optimize the representative error of the features corresponding to different consumer engagement indicators across various blocks. Consequently, the choice of parameters ($\lambda$, $\rho$) varies under different product types and consumer engagement indicators, leading to differences in the number of variables retained. However, regardless of the number of variables retained, their representative error for the dependent variable is below 5 (representing a significant reduction in relative error compared to when no variable selection is performed). This indicates that after MSR screening, the retained variables from different blocks maintain a consistent level of representativeness for consumer engagement indicators. In subsequent BIM analysis, this ensures a high degree of robustness.

To verify the results of the MSR method, we compared the variable screening results of the MSR method with random forest regression (RFR), t-distributed stochastic neighbor embedding (t-SNE), principal component analysis (PCA), and correlation analysis (CA) methods.

Table 5. Analysis of MBPLS model prediction results.

Method	Index	Food				Digital Products
		Likes	Comments	Collects	Shares	Likes	Comments	Collects	Shares
MSR	R	0.685	0.616	0.740	0.712	0.718	0.615	0.654	0.684
	MAE	0.112	0.087	0.065	0.104	0.094	0.105	0.101	0.100
	MSE	0.019	0.012	0.007	0.017	0.014	0.017	0.017	0.017
RFR	R	0.593	0.504	0.622	0.657	0.459	0.383	0.540	0.532
	MAE	0.087	0.093	0.073	0.089	0.098	0.082	0.076	0.085
	MSE	0.012	0.014	0.009	0.013	0.015	0.011	0.009	0.012
t-SNE	R	0.652	0.595	0.611	0.628	0.454	0.361	0.513	0.531
	MAE	0.085	0.089	0.073	0.092	0.103	0.089	0.088	0.089
	MSE	0.011	0.013	0.010	0.012	0.015	0.013	0.012	0.013
PCA	R	0.542	0.460	0.559	0.554	0.286	0.227	0.374	0.351
	MAE	0.090	0.094	0.078	0.102	0.105	0.079	0.087	0.097
	MSE	0.013	0.015	0.010	0.015	0.017	0.011	0.011	0.015
CA	R	0.560	0.507	0.592	0.626	0.049	0.464	0.554	0.518
	MAE	0.092	0.090	0.075	0.093	0.094	0.075	0.074	0.087
	MSE	0.013	0.014	0.009	0.014	0.014	0.009	0.009	0.013
All	R	0.416	0.337	0.529	0.589	0.362	0.297	0.272	0.340
	MAE	0.148	0.113	0.082	0.118	0.131	0.127	0.137	0.131
	MSE	0.033	0.020	0.011	0.022	0.027	0.028	0.030	0.028

presents the regression results using the MSR method, comparison method, and unscreened variables. After applying the variable screening of MSR, the two short videos showed significant improvements in the evaluation indicators R, MSE (Mean Absolute Error), and MAE (Mean Squared Error) of consumer engagement regression compared to unfiltered variables. The R value is the correlation coefficient between predicted values and actual values. The closer the R value is to 1, the better the predictive capability of the model. MAE measures the average difference between predicted values and actual values, with a smaller value indicating more accurate predictions by the model, and it can be directly compared with the actual dimension of the data. MSE measures the average of squared prediction errors, with a smaller value indicating more accurate predictions by the model, and it is suitable for situations where prediction errors are relatively large.

Results showed that the MSR method had significantly higher R values for the regression evaluation indicators of consumer engagement variables than for the other variable screening methods, with little difference in MSE and MAE. These results suggest that the MSR method is better than the comparison method.

4.2. The Result of MBPLS

Figure 4 shows the explained variance of consumer engagement indicators in each latent variable (LV). The statistical principle of MBPLS in capturing the variability information of data is similar to that of principal component analysis (PCA). The higher the ranking of LVs, the better they can explain the variance generated by consumer engagement indicators. Based on the attenuation situation shown in Figure 4, when LV exceeds 3, the value of the interpretive variance is less than 0.1. Thus, we reserve the first three LVs for further analysis.

In the analysis of short video advertising variables, we focused on the importance of individual consumer engagement indicators. Figure 5 and Figure 6 show the block importance (BIP) values of the three LVs for food and digital products. As shown in Figure 5 and Figure 6, the speech text features and visual features of the two types of videos are the most important. However, the importance of speech text features and visual features to food is slightly higher than that to digital products. This may be because, in short video advertisements, visual appeal and descriptive language can directly convey the characteristics of food, whereas the functions and features of digital products require more detailed demonstrations and explanations, and consumers often rely on more information to evaluate digital products [16]. Indeed, the cognitive processes consumers undergo when perceiving these two product types diverge, and numerous studies have confirmed the moderating effect of product type in short video advertisements [16,18]. Nevertheless, this particular aspect does not constitute the primary focus of our study.

Due to the fact that we control the gap time of videos to be greater than 50 days, the proportion of scalar features in BIP is very small, which indirectly verifies the rationality of our handling of control variables. Compared to food, the importance of digital product titles for consumer engagement is higher, and the importance of acoustic features for comments is also higher. Furthermore, the BIP of speech text features is greater than that of visual features in each label of consumer engagement indicator variables, indicating that in short video advertisements, hearing is more important than vision.

5. Discussion

The primary goal of this study was to identify the key factors influencing consumer engagement in short video advertisements for experience and search products. To this end, we designed a multimodal analysis framework that extracts visual, speech text, acoustic, title, and scalar features from short video advertisements through machine learning and deep learning methods. Considering that the extracted feature dimensions were too high, we proposed the MSR method to optimize the problem of $l_1$-regularization in variable selection. The MSR method reduced the computational complexity and improved the accuracy of the model through high-performance dimensionality reduction. To intuitively display the importance of different features, we visualized the results of the block importance analysis using the MBPLS regression model.

In order to better fit the MBPLS regression model, when using the MSR method, we designed a parameter selection rule that aims to retain more important variables while keeping the representation error as low as possible, and we kept around 40% of the variables in each block. In addition, when using $l_1$-regularization to screen variables, the features with high dimensionality in visual and speech text variables, $l_1$-regularization almost removed all variables under each consumer engagement indicator. Therefore, the result of $l_1$-regularization was not listed in the comparison method, displayed in Table 4. Our MSR method solved the problem of $l_1$-regularization in ultra-high-dimensional feature screening. Our method yields better results than random forest regression, decision tree regression, principal component analysis, and correlation analysis, demonstrating its robustness further.

Our study found that speech text and visual features were the key factors influencing consumer engagement behaviors. In short video advertisements, the speech text conveys core product information, such as features, advantages, and promotional content, clearly and accurately, helping consumers quickly understand and reduces misunderstandings. At the same time, by telling stories, citing data, and asking questions, the speech text enhances the persuasiveness of the advertisement, making consumers more likely to generate purchase intentions [52]. On the other hand, the visual features of short video advertisements can capture consumers’ attention immediately. This visual impact helps the advertisement stand out among numerous pieces of information, increasing its exposure rate and memorability. Furthermore, visual features can also touch consumers’ emotions, enhancing the persuasiveness and appeal of the advertisement [7,20]. Content-related research in the field of short video marketing should prioritize extracting information from these two features, which can increase the persuasiveness of the research results.

We also found that, although not as important as the speech text and visual features, title features are more significant than acoustic features. In short video advertisements, the title is often a brief summary of the video content, helping users quickly understand the theme and main points of the video [6]. This aids consumers in judging whether the video aligns with their interests and needs. However, our study merely ranks the importance of multimodal features in short video advertisements and provides a theoretical basis for related research. Features that rank lower are merely relatively less important. In fact, according to the findings of Xiao et al. [7], acoustic features such as zero-crossing rate and spectral centroid have a significant impact on consumer engagement behavior. Our results suggest that relevant researchers should prioritize visual and speech text features in short video advertisements.

Lu et al. [8] analyzed consumer engagement in short videos, emphasizing the significance of visual and title features. However, they ignored the speech text features. Because multimedia relies heavily on audiovisual elements, the significance of speech text features in short videos cannot be ignored. In fact, our results confirm that speech text features are significantly more important for consumer engagement than title features. On the other hand, for acoustic features, we added spectral bandwidth and RMS energy based on the research by Lu et al. [8]. These two indicators can effectively measure the effective frequency range and acoustic energy of the spectrum. Although some important acoustic features have been added, they are generally far less significant than speech text and visual features as a whole.

Furthermore, in contrast to the conclusion of Lu et al. [8] on entertaining short videos, our results showed that visual and speech text are significantly more important for consumer engagement in short video advertisements for search and experience products. These findings provide new insights into the multimodal analysis of short videos. Additionally, title features were more important than acoustic and scalar features, which might be attributed to viewers’ propensity to peruse short videos with a more hedonistic and leisurely mindset, paying closer attention to the audio and visual content of the advertisements.

5.1. Theoretical Implications

This study has several theoretical implications.

Firstly, this study offers a fresh perspective on short video advertisements by analyzing them through a multimodal lens, thereby enriching the existing literature on social media advertising. While prior research on short videos has primarily centered around social and commercial attributes, leveraging scalar features, unstructured text data, or questionnaires, multimodality research has largely been confined to sentiment analysis and classification tasks in specific scenarios and datasets [53]. Notably, there is a dearth of studies exploring the multimodal features of short video advertisements. Our work stands as the first to quantitatively identify the crucial factors that drive consumer engagement in short video advertisements from a multimodal perspective, providing valuable theoretical insights to researchers and contributing to the broader understanding of social media advertising.

Secondly, based on the content attributes of short video advertisements, this study considers product types, analyzes experience and search products, respectively, and explores the impact of visual features, speech text features, acoustic features, and title features on consumer engagement. Results show that visual features and speech text features are the most important, and the conclusions for the two product types are consistent. Subsequent research can focus on these two points. From this perspective, this work has made a certain contribution to consumer engagement behavior.

Finally, this study can address the challenges of processing multi-dimensional patterns in social media. From the perspective of short video advertising content, the short videos posted by influencers encompass both textual data (speech transcripts and headline text data) aimed at attracting consumers and non-textual data (visual and auditory data) [54]. Our data analysis method can reduce the dimensionality of various modal features and explore the relationship between multimodal features and consumer engagement. Our multimodal analysis framework is also applicable to the study of user behavior in other social media contexts. Specifically, our framework comprehensively extracts multiple modal features of short video advertisements, and the proposed dimensionality reduction method requires less computation and exhibits higher computational efficiency. In the study of patterns in social media, it is suitable for classification and regression tasks.

5.2. Practical Implications

From a marketing perspective, this study has practical significance for both short video sellers and platform operations. Short video advertisements are information streams based on an audiovisual stimulus, recommended by consumer preferences. When releasing short video advertisements, sellers should consider video display, text introduction, tone/music, title, length, speaking speed, etc. Our results show that whether it is search products or experience products, the speech text and visual features of advertisements are the most important. Since short video users typically make judgments about their preference for the current content in extremely short periods of time [7], sellers need to capture the consumer’s attention through visual and speech text elements at the beginning of short video advertisements. When the consumers have aroused interest at the beginning of the short video, they are more likely to engage with likes, comments, collects, and shares. Although title features are not as important as visual and speech text features, they are also more important than acoustic and scalar features. On the TikTok platform, titles are partially obscured due to length and so, sellers should keep video titles brief.

For platform operations, our multimodal analysis framework extracts four different types of short video features. For the extracted high-dimension features, we propose the MSR method to screen variables and use the regression model MBPLS to achieve BIP analysis of blocks. Short video platforms analyze users’ habits of watching short videos and label them according to their interests. At the same time, the labels of each uploaded video are also calculated, and then the tags of the video are mapped to users with the same tags [1]. Technically speaking, the map methods are based on the deep features of short videos. Our work can provide a certain reference for the platform. Our feature extraction is quite extensive and can supply positioning suggestions based on big data algorithms, which can promote short video marketing.

6. Conclusions and Future Work

The aim of this study was to delve into the intricate relationship between the multimodal features of short video advertisements and their impact on consumer engagement. The multimodal analysis results indicate that visual and speech text features are the key factors influencing consumer engagement, providing theoretical support for subsequent research. To the best of our knowledge, the features extracted in our analysis represent the most comprehensive set currently available for short video advertisements. To gain insights into the significance of various features, we leverage MBPLS as a regression model. This approach enables us to assess the influence of different types of features on consumer engagement through the calculation of BIP values. By employing variable screening and multiblock regression analysis, our framework uncovers the criticality of different blocks within the multimodal feature set.

This study has several limitations, including potential future research directions. Firstly, our text features only consider speech text and titles, whereas short videos actually contain other forms of text (e.g., subtitles, on-screen text). However, these forms of text appear less frequently in short video advertisements. Future research could collect more data on short video advertisements to explore the importance of these forms of text. Secondly, according to our results, the speech text and visual features of short video advertisements have the most important impact on consumer engagement. Our work has only considered the deep features, which are essentially a black box. Further refining of interpretable speech text and visual features is worth exploring, such as calculating the creativity, narrative, color complexity, and psychological factors of short video advertisements [55]. Thirdly, although the MSP and MBPLS methods in our framework perform well in data analysis, this process can be further refined, such as considering the continuity of short videos and studying the influencing factors of consumer engagement in short video advertisements through time series analysis methods, which may lead to new discoveries.