Reducing Video Coding Complexity Based on CNN-CBAM in HEVC

Abstract

High-efficiency video coding (HEVC) outperforms H.264 in coding efficiency. However, the rate–distortion optimization (RDO) process in coding tree unit (CTU) partitioning requires an exhaustive exploration of all possible quad-tree partitions, resulting in high encoding complexity. To simplify this process, this paper proposed a convolution neural network (CNN) based optimization algorithm combined with a hybrid attention mechanism module. Firstly, we designed a CNN compatible with the current coding unit (CU) size to accurately predict the CU partitions. In addition, we also designed a convolution block to enhance the information interaction between CU blocks. Then, we introduced the convolution block attention module (CBAM) into CNN, called CNN-CBAM. This module concentrates on important regions in the image and attends to the target object correctly. Finally, we integrated the CNN-CBAM into the HEVC coding framework for CU partition prediction in advance. The proposed network was trained, validated, and tested using a large scale dataset covering various scenes and objects, which provides extensive samples for intra-frame CU partition prediction in HEVC. The experimental findings demonstrate that our scheme can reduce the coding time by up to 64.05% on average compared to a traditional HM16.5 encoder, with only 0.09 dB degradation in BD-PSNR and a 1.94% increase in BD-BR.

1. Introduction

To transport huge volumes of video data effectively, some new versions of flexible video standards like H.264 [1], H.265 [2], and H.266 [3] are continually being developed. Each of these standards provides many efficient coding methods, greatly increasing the code complexity. For example, to obtain the best CU, versatile video coding (VVC) uses highly adaptable block coding (i.e., quad tree-based merge and transform (QTMT) structure), increasing the intra-frame coding complexity by an average of 18 times compared to HEVC [4]. It will become considerably more difficult if the rate distortion optimization (RDO) is also used. The VVC standard’s practical application is constrained by its high level of complexity, particularly in real-time settings. Given that both VVC and HECV use block-based encoding constructions with some variations in specific technologies as well as the fact that deep learning-based accelerating methods typically require the power of a graphics processing unit (GPU), HEVC is more developed and widely used than VVC in this regard. Thus, to estimate the performances of the various methods and validate the efficacy of the suggested algorithm, this work utilized HEVC without losing its universality.

Plenty of approaches to simplify HEVC coding have been developed during the past 10 years. It has been demonstrated that the standard reference software HM’s recursive CU partition procedure consumes up to 80% of the intra-frame encoding time. Therefore, reducing the complexity of quad-tree CU partitioning without compromising coding efficiency is a much-anticipated problem in the field of video compression. Currently, complexity reduction methods are mainly categorized into heuristics and learning-based methods, and there are also individual papers that have proposed other ways to reduce the coding time, like in [5,6].

The RDO search for CU partitions can be streamlined using several middle characteristics in heuristic strategies [7,8,9,10,11,12,13,14,15]. The approach by Shen et al. [7] circumvented the large CU size of the close-by coding tree unit (CTU) and offered an early CU size determination method based on the texture homogeneity measured by the mean absolute distortion. Quantization parameters (QPs), RD cost distributions, and video content were used to update an adaptive early termination threshold in Zhang et al.’s [8] statistical modeling-based CU partition technique. Lim et al. [9] developed a quick joint approach to address the issue of high complexity when the PU crosses the RDO. In particular, it performs jump and split termination judgments on prediction units (PUs) using Bayes’ rule and freshly set decision parameters, avoiding RD processing. Shen et al. [10] originally used a class conditional probability density function using one-two essential features and prior experience to identify whether or not the CU is divided. They then developed a Bayesian threshold method to ensure that the loss of the large coding unit (LCU) partition was kept to a minimum. The technique is limited in its ability to encode various scenarios because it lacks statistical features that are simple to compute. Zhang et al. [11] overcame the constraints of [10] by presenting a statistical-based model solution to the high-complexity of the CU partition and PU mode decision-making process. Using statistical qualities, the method extracted as many key features of the image as possible, minimizing the coding time to a greater extent.

Learning-based approaches have gradually replaced traditional heuristics to extract more features and lower the complexity of the CU partition in today’s environment, where artificial intelligence technologies are constantly encouraged [16,17,18,19,20]. To achieve more efficient results, we used machine learning (ML) methods such as support vector machines (SVMs) [16,17,18], random forests (RFs) [19], or Bayesian decision rule [20] classifiers (not the block similarity metrics), which are used in statistical-based methods to simplify the complexity of the CU decision module. Unlike the ML approaches, the CNN methods have been proven to be effective in automatic feature extraction [21,22,23,24,25,26,27,28,29,30,31,32]. CNN-based approaches, in comparison to heuristic-based methods, exhibit superior generalization ability and automatic feature learning. They can be effectively employed in various video coding and decoding scenarios, thereby enhancing the universality of the proposed algorithm. Consequently, CNN-based methods are increasingly gaining attention and finding diverse applications in the field of video coding and decoding.

Although the complexity can be greatly reduced using both heuristic and learning-based approaches, heuristic methods often struggle to extract sufficient features from video content, and the learning-based methods lead to a significant increase in complexity at deeper CTU depths. Therefore, when designing the network structure, not only must sufficient features be extracted, but the encoding time must be considerably reduced within the acceptable range of model calculation. In this paper, we propose a fast partition algorithm in CU based on machine learning and heuristic methods. Firstly, a CNN that can quickly predict all CU partition results was proposed to adapt to three CU sizes of 16 × 16, 32 × 32, and 64 × 64. In addition, the convolution blocks were designed to enhance the connection between the blocks in the intra-frame space. Secondly, a CBAM was introduced into the proposed CNN to construct CNN-CBAM with better performance, consequently refining the feature map while improving the network performance. Finally, the experimental data revealed that the CNN-CBAM significantly reduced the intra-frame coding time while somewhat degrading the RD performance.

To conclude, our primary contribution is in the three categories listed below:

• We propose a CNN for variable CU sizes to properly fetch the global information of the current CTU. The developed CNN can accurately anticipate the size of the HEVC coding units, resulting in a reduced intra-frame coding time.
• Embedding a CBAM into CNN leads to the development of the first CNN-CBAM, which is capable of predicting CU partitions more accurately in HEVC, so that it can obtain the key information of the different sequences.
• The CNN-CBAM is embedded in the HEVC coding framework to predict the CU partition in advance. This method can avoid the repeated iterations of traditional RDO to extract the optimal CU depth, therefore, it can remarkably decrease the coding complexity while sustaining good coding performance.

The rest of this paper is organized as follows. Section 2 describes the proposed CNN-CBAM algorithm in detail. Section 3 reports the experimental results. Section 4 summarizes the paper and discusses some future work.

2. The Proposed Methods

2.1. Traditional CU Partition

The high complexity of HEVC is mainly attributed to the CU partition in which RDO is the core process, as previously mentioned. According to the image content of each LCU, each is divided into four CU sizes with varying depths, and the specific CU sizes of the four are shown in Figure 1. As already noted, the RDO technique is utilized to traverse in order to find the ideal CU sizes, and the procedure consumes the majority of the intra-frame coding time, realizing a full LCU delineation. However, only one CU partition from 1 (maximum size 64 × 64 CU are not partitioned) to 64 (CTU are fully partitioned into 8 × 8 CU with depth = 3) is chosen in the final CU partition, which significantly increases the amount of redundant coding time.

2.2. Proposed Algorithm

2.2.1. Proposed Structure of CNN

Deep learning has been brought to video coding to reduce the time overhead of intra-frame coding by training a large number of parameters to learn the encoder CU partition rules. These methods can be broadly classified into two kinds: single CNN [21,22,23,24,25,26,27] and single CNN with some auxiliary data [28,29,30,31,32], For example, CNNs were initially used in video codecs by Liu et al. [19], who presented a deep CNN to minimize the amount of CU/PU candidate patterns. Furthermore, size-specific down-sampling for different CU/PU sizes resulted in texture loss variances for different CU/PU layers, which had an effect on the anticipated results. To better understand the CU partition rules in the encoder, Li et al. [22] suggested a deep CNN method trained on numerous parameters to address the problem of redundant computation during intra-frame CU partition. Fan et al. [28] introduced an efficient block partitioning CNN method that not only improved on the CNN, but also incorporated an adaptive threshold technique for the accurate management of CNN prediction errors. Zhang et al. [32] investigated the correlation between texture complexity, quantization parameters, and CU depth and proposed a CNN scheme based on texture classification aiming at accelerating CU partition. However, the huge reduction in the complexity of these methods based on CNN comes at the expense of a minor drop in RD performance. The latest study [28] showed a more significant improvement in both the complexity reduction and compression efficiency. However, there are still some shortcomings. For example, in [28], although the different CU sizes could share the same convolution structure, the extracted global information was limited for larger CU sizes. In addition, this method used RDO in uncertain regions with a threshold of approximately 0.5, further increasing the computational complexity. Inspired by [21,28], we propose a novel CNN-based approach to further reduce the coding complexity while sustaining the compression performances without a loss of generality.

Unlike in [21,28], the proposed CNN considers the global image information and designs a convolution operation that is compatible with the current CU size, thus the feature extraction is realized between adjacent scaling CU and the information interaction can be carried out in the same path. In addition, a convolution block with superposition performance was designed to extract the detailed features of the image blocks. Thus, the effective learning of the global information is achieved. The proposed CNN not only accurately predicts the CU partition to replace unnecessary encoding processes, but also strengthens the connection between CNN and different CU sizes.

The proposed CNN is shown in Figure 2. Note that we selected original image luminance information as input to the network since it contained more visual information. There are three different branches of the proposed CNN, namely B1, B2, and B3, corresponding to the different CU sizes. The different branches are first pre-processed separately, and then the pre-processed CU information is passed through the convolution, concatenation, and fully connection layers in sequence until the CU partition decision results are output. The specific structures are described as follows:

Convolution Layer: The initial 64 × 64 luminance pixel matrix serving as the input to this network is the corresponding preprocessing process before entering each of the three branches, with the preprocessed CU block serving as the input to the convolutional layer of that branch alone. Noted that in this paper, we fixed the size of the convolution kernel to be same as the step of this layer.

For a further understanding of deeper features, the convolution block was designed and set in the convolution path $C_{i-j} \left(i=1, 2, 3;j=3, 4\right)$ and $C_{3-2}$. Figure 3 depicts the structure of the convolution block. Firstly, the input blocks are convolved along principal and lateral branch paths to obtain two feature maps, Con-1 × 1 and Con-2 × 2, respectively, then to obtain the convolution blocks, the feature maps of the two different paths are stacked in the channel dimension, which can help the network learn about the many combinations of CU sizes. The convolution kernels of the main and side branch are 1 × 1 and 2 × 2, respectively. Meanwhile, the step size of both convolutions is set to 1, which aims to ensure that the feature maps of Con-1 × 1 and Con-2 × 2 have the same size to facilitate feature fusion. Those features contain rich texture information for learning CU partition. Secondly, the convolution kernels of the $C_{i-1 }\left(i=1, 2, 3\right)$ are set to half of the size of the current CU to obtain the global information of the image as much as possible. These features are closely related to the current CU partition. Finally, to fuse the feature maps of the two paths, the $C_{i-4} \left(i=1, 2, 3\right)$ and $C_{i-1} \left(i=1, 2, 3\right)$ channels are set to 64 and 128, respectively. In addition, a non-overlapping 2 × 2 max pooling, computation with a step size 2 is used to filter the features in the main channel $C_{i-j} \left(i=1, 2, 3;j=2, 3, 4\right)$. This drastically decreases the number of parameters and the risk of over-fitting while preserving the main features.

Concatenation Layer: The fused feature maps obtained by the three branches are turned into three one-dimensional vectors respectively in the concatenation layer. The one-dimensional vector contains all the key information features of the current CU.

Fully connected layer: In each of the three branches, the connection layer’s three one-dimensional vectors are used as inputs to the fully connected layer. The input contains local features collected from various layers for each category. Furthermore, because QP is critical in the CU partition process, it is spliced into the fully connected layer of each branch as supplementary features. These above features will be integrated and categorized in this layer until the learning of all of the CTU partition rules is complete, and then the output results will be used to judge the current CU.

Re LU and Sigmoid are used as the convolutional and output layer activation functions, respectively. Furthermore, the method proposed employs an early termination strategy to eliminate superfluous CU partition operations during coding.

Table 1. The parameters of the proposed CNN.

Branch	Layer	Output Size	Proposed CNN Configuration
B1	C1-1	2 × 2 × 128	S-32, Con-32 × 32
	C1-2	16 × 16 × 64	S-4, Con-4 × 4
	C1-3	16 × 16 × 64	S-1, Con-[2 × 2, 1 × 1]
	pooling	8 × 8 × 64	S-2, MP-2 × 2
	C1-3	8 × 8 × 128	S-1, Con-[2 × 2, 1 × 1]
	pooling	4 × 4 × 128	S-2, MP-2 × 2
	Concatenation	/	2560
	FCon-1	/	512
	FCon-2	/	256
B2	C2-1	8 × 8 × 64	S-16, Con-16 × 16
	C2-2	8 × 8 × 64	S-4, Con-4 × 4
	pooling	4 × 4 × 64	S-2, MP-2 × 2
	C2-3	4 × 4 × 128	S-1, Con-[2 × 2, 1 × 1]
	pooling	2 × 2 × 128	S-2, MP-2 × 2
	C2-4	/	S-1, Con-[2 × 2, 1 × 1]
	pooling	/	S-2, MP-2 × 2
	Concatenation	/	1024
	FCon-1	8 × 8 × 64	256
	FCon-2	8 × 8 × 64	16
B3	C2-1	4 × 4 × 64	S-8, Con-8 × 8
	C2-2	4 × 4 × 128	S-1, Con-[2 × 2, 1 × 1]
	pooling	8 × 8 × 64	S-2, MP-2 × 2
	C2-3	8 × 8 × 64	S-1, Con-[2 × 2, 1 × 1]
	pooling	4 × 4 × 64	S-2, MP-2 × 2
	C2-4	4 × 4 × 128	S-1, Con-[2 × 2, 1 × 1]
	pooling	2 × 2 × 128	S-2, MP-2 × 2
	Concatenation	/	1024
	FCon-1	/	256
	FCon-2	/	16

displays the parameters of each layer of the proposed CNN, where full convolution, convolution, maximum pooling, and step size are abbreviated as FCon, Con, MP, and S, respectively.

2.2.2. Structure of CBAM

The proposed convolution block attention module (CBAM) by [33] can improve the network model performance by learning to focus or suppress key feature information, hence facilitating network information transmission. Figure 4 depicts the CBAM’s detailed construction.

For the given input feature map $\mathit{F}\in R^{C\times H\times W}$, where C, H, and W denote the number of channels, height, and width of the feature map, respectively. The CBAM sequentially contains ${\mathit{M}}_{\mathit{C}}\in R^{C\times 1\times 1}$ (channel attention map) and ${\mathit{M}}_{\mathit{S}}\in R^{1\times H\times W}$ (spatial attention map). The overall CBAM can be characterized as:

$${\mathit{F}}^{\prime }={\mathit{M}}_{\mathit{C}}\left(\mathit{F}\right) \otimes \mathit{F},$$

(1)

$${\mathit{F}}^{″}={\mathit{M}}_{\mathit{S}}\left({\mathit{F}}^{\prime }\right) \otimes {\mathit{F}}^{\prime },$$

(2)

where $\otimes$ means the elements of the feature map and the two attention modules to produce the corresponding weight product operation. The two attention modules are detailed below.

Channel attention module: First, in $\mathit{F}\in R^{C\times H\times W}$, maximum pooling and average pooling are carried out to summarize the spatial information of the feature map and produce two distinct spatial context descriptions, i.e., ${\mathit{F}}_{max}^c$ (maximum pooling) and ${\mathit{F}}_{avg}^c$ (average pooling). Second, the ${\mathit{F}}_{avg}^c$ and ${\mathit{F}}_{max}^c$ are passed to a hidden layer and multi-layer perceptron (MLP) to produce a channel attention map ${\mathit{M}}_{\mathit{c}}\in R^{C\times 1\times 1}$. In addition, we set the drop channel size of the hidden layer to $R^{\raisebox{1ex}{$c$}\!\left/ \!\raisebox{-1ex}{$r$}\right.\times 1\times 1}$ to reduce the model parameter overhead, where $\raisebox{1ex}{$c$}\!\left/ \!\raisebox{-1ex}{$r$}\right.$ is an integer. Then, after implementing a shared network for each descriptor, two new descriptors are generated (i.e., ${\mathit{F}}_{avg\prime }^c$ and ${\mathit{F}}_{max\prime }^c)$. Finally, we utilize an element-by-element summing approach to obtain F’ as the output. Therefore, the channel attention module is computed as:

$$\begin{gathered} {\mathit{M}}_{\mathit{C}}\left(F\right)=\alpha \left(MLP\left(Avgpool\left(\mathit{F}\right)\right)+MLP\left(Maxpool\left(\mathit{F}\right)\right)\right), \\ =\alpha \left({\mathit{W}}_1\left({\mathit{W}}_0\left({\mathit{F}}_{avg}^c\right)\right)+{\mathit{W}}_1\left({\mathit{W}}_0\left({\mathit{F}}_{max}^c\right)\right)\right), \end{gathered}$$

(3)

where α represents the sigmoid function and ${\mathit{W}}_0\in R^{C/r\times C}$ and ${\mathit{W}}_1\in R^{C\times C/r}$ are the MLP weights.

Spatial attention module: Firstly, the first attention module’s output result F’ is used as the second attention module’s input. Secondly, the maximum pooling and average pooling are performed along the channel, and the pooling results are concatenated to yield 2D valid feature descriptors (i.e., ${\mathit{F}}_{avg}^s\in R^{1\times H\times W}$(average pooling) and ${\mathit{F}}_{max}^s\in R^{1\times H\times W}$(maximum pooling)). Finally, for cascade-based feature descriptors, a convolution layer is applied to produce ${\mathit{M}}_{\mathit{s}}\left(\mathit{F}\right)\in R^{H\times W}$ (spatial attention map). Therefore, the spatial attention module is computed as:

$$\begin{gathered} {\mathit{M}}_{\mathit{S}}\left(\mathit{F}\right)=\alpha \left(f^{7\times 7}\left(\left[Avgpool\left(\mathit{F}\right);Maxpool\left(\mathit{F}\right)\right]\right)\right), \\ =\alpha \left(f^{7\times 7}\left(\left[{\mathit{F}}_{avg}^s;{\mathit{F}}_{max}^s\right]\right)\right), \end{gathered}$$

(4)

where $f^{7\times 7 }$ represents a 7 × 7 convolution operation and α is the sigmoid function. The results by [33] showed that the filter size of 7 × 7 had good performance, so this paper selected this filter without loss of generality.

In summary, the CBAM introduces the spatial and channel attention from the spatial and channel dimensions to notice important information about the object’s content and thus better describe the key features of the image. Secondly, adding CBAM to the network induces the original network to spotlight the target objects, thus improving the generalization and characterization ability of the network. More importantly, CBAM is a lightweight network that improves network effectiveness without adding additional computational complexity. Based on the above advantages, we introduced the CBAM into the proposed CNN and then achieved a better texture representation performance.

2.2.3. Proposed CNN-CBAM Algorithm

As above-mentioned, to increase the link between CUs of various sizes and the convolution structure, the size of the convolution kernel in $C_{i-1} \left(i=1, 2, 3\right)$ is half of the current CU size. The feature map obtained from the convolution is a global description of the current CU information, which is critical in the CU partition of the output layer. However, when the CNN performs feature mapping, the weights assigned to each channel are the same that result in the poor description of the key information. In light of this, as shown in Figure 5, we introduced CBAM in the CNN structure to better characterize key features of the image by adaptively assigning the weights within the channels according to the content of the image and focusing on important regions of the image content.

2.2.4. Loss Function of Proposed CNN-CBAM

Cross entropy is derived from the concept of information entropy in information theory and is commonly used to measure the disparity between different probability distributions. Specifically, it treats the ground-truth labels as one probability distribution (referring to the true CU partition labels of the video sequence by HM16.5), while the model’s predicted output can be treated as another probability distribution (CNN-CBAM model). The cross-entropy loss function is calculated as follows:

$$H\left(p,q\right)=-{\sum }_{i=1}^N{p}_i\log \left(q_i\right),$$

(5)

where $p_i$ denotes the probability of category i in the ground-truth label and $q_i$ denotes the probability of category i in the model’s predicted output.

This loss function is not only the most often used function for classification tasks, but it also sets the neural network’s direction of convergence, allowing the neural network to perform well during the initial training. On this premise, the CNN-CBAM network model was trained in this research using the cross-entropy loss function. Meanwhile, applying the Sigmoid function to the gradient descent algorithm can effectively solve the problem of a reduced learning rate.

Supposing that there are M training samples, the prediction values in the B1, B2, and B3 branches are shown as ${\left\{y_{B1}^{\ast }\left(k\right)\right\}}_{k=1}^4$, ${\left\{y_{B2}^{\ast }\left(k\right)\right\}}_{k=1}^4$, and ${\left\{y_{B3}^{\ast }\left(k\right)\right\}}_{k=1}^4$, respectively. The corresponding true values are denoted as ${\left\{y_{B1}\left(k\right)\right\}}_{k=1}^4$, ${\left\{y_{B2}\left(k\right)\right\}}_{k=1}^4$, and ${\left\{y_{B3}\left(k\right)\right\}}_{k=1}^4$. The samples’ loss function $Lr$ is expressed as follows:

$$\begin{gathered} Lr={{\displaystyle \sum }}_{k=1}^{1}H\left(y_{B1}^{\ast }\left(k\right), y_{B1}\left(k\right)\right)+{{\displaystyle \sum }}_{k=1}^{4}H\left(y_{B2}^{\ast }\left(k\right), y_{B2}\left(k\right)\right) \\ +{{\displaystyle \sum }}_{k=1}^{16}H\left(y_{B3}^{\ast }\left(k\right),y_{B3}\left(k\right)\right), \end{gathered}$$

(6)

by minimizing the loss function, the CNN-CBAM can be trained in this manner.

$$L=\frac{1}{M}{{\displaystyle \sum }}_{m=1}^{M}Lr.$$

(7)

2.3. Proposed CNN-CBAM Algorithm and Flowcharts

The overall CNN-CBAM algorithm is shown as Algorithm 1. The algorithm contains the two most important parts. (1) Training the CNN-CBAM model to obtain the partition labels of CU in the image. (2) We embed the trained model into HEVC to predict the CU partition in advance.

Algorithm 1 (CNN-CBAM)

1: Input: current image information   2: Output: return a set of split CU with RDO costs   3: Processed processing: I-pro   4: {// train CNN-CBAM model to obtain current CU split_flag   5: If W = 64 and H = 64 then   6:      If split_flag = 0 then   7:     skip CU partition   8:   Else W = 32 and H = 32 then   9:      If split_flag = 0 then  10:      skip CU partition  11:     Else W = 16 and H = 16 then  12:      If split_flag = 0 then  13:       skip CU partition  14:      Else W = 8 and H = 8 then  15:       split_flag ← 0  16:      End if  17:     End if  18:     End if  19: End if  20: }  21: frontier ← stack (LCU)  22: decision results ← ∅  23: While is empty (frontier) is false do  24:  CU block ← POP (frontier)  25:  If execute Model (CNN-CBAM, CU block) is not split do  26:     decision results ← decision results ∪ CU block  27:  Else  28:     Sub CU ← GET (Sub CU, CU block)  29:     Sub CU RDO ← ∅  30:     For sub-CU in Sub CU do  31:     If is leaf (sub-CU) is true then  32:      Sub CU RDO ← Sub CU RDO ∪ calculate (RDO, sub-CU)  33:     Else  34:      PUSH (frontier, sub-CU)  35:     End if  36:     End for  37:     Parent RDO ← calculate (RDO, CU block)  38:     If parent RDO < sum of 4 sub-CU RDO then  39:     decision results ← decision results ∪ CU block  40:     Else  41:     decision results ← decision results ∪ Sub CU  42:     End if  43:  End if  44: End while  45: Return decision results

The framework of the CNN-CBAM is displayed in Figure 6. The proposed method firstly avoids unnecessary redundant computation of violent RDO compared with the traditional CU classification method; secondly, the proposed CNN, which is compatible with different levels of CU, can predict all the CU partition results at one time, significantly reducing the coding complexity. In addition, the convolution block is designed to enhance the information exchange of the same path of the CU to obtain richer texture features. Finally, CBAM is introduced into CNN to build CNN-CBAM. The CNN-CBAM is embedded in the HEVC coding framework to predict the partition of CU in advance. The CBAM can effectively improve the intermediate features by dynamically suppressing or emphasizing the feature map through the attention mechanism, thus improving the representational ability of the CNN-CBAM network without increasing the computational complexity.

3. Experimental Results

3.1. Experimental Conditions

Experimental data: The database for the CU partition of HEVC(CPH) [34] (2000 ultra-high definition (UHD) images were selected from the original image datasets) was used to test the network presented. First, these UHD images were randomly divided into validation, testing, and training, which was 5%, 10%, and 85% of the entire dataset, respectively. Generally speaking, the more the training data, the more accurate the predicted CTU partition. Second, each group contains four subsets equally (i.e., 768 × 512, 1536 × 1024, 4928 × 3264, and 2880 × 1920). Finally, the CPH video sequences are encoded with HM16.5 to different QPs to obtain the containing CU partition after training, testing, and validation of the data labels.

In addition, all trainable parameters are randomly initialized when training the model. The specific hyper parameter settings are shown in

Table 2. Specific hyper parameter settings of CNN-CBAM.

Hyper Parameters	Value
Model initialization	Random initialization
Distribution type	Truncated normal distribution
Mean	0
Standard deviation	0.1
Batch size (R)	64
Momentum	0.9
Initial learning rate	0.01
Learning rate decay	Decrease by 1% every 2000 iterations
Total iterations	1,000,000

.

Standard experimental configuration: In this study, to encode the standard video test sequence, four QP values {22, 27, 32, 37} and encoder_intra_main.cfg [35] were chosen using the Benchmark software HM16.5 as the experimental platform.

Table 3. Experimental conditions.

Name	Model
Operating system	Window 11
python	3.6
TensorFlow	1.8
Processor	AMD Ryzen 7 5800 H with Radeon Graphics @3.20 GHz
Running memory	8.00 GB RAM
Graphics Card Adapter	NVIDIA GeForce GTX1650

shows the specific experimental conditions.

Objective evaluation indicators: Validation studies were carried out on 18 reference sequences with varied resolutions provided by JCT-VC in this publication for fair comparison. Simultaneously, to evaluate the overall effectiveness of the suggested strategy, we settled on four criteria (i.e., $∆T$, $∆T_{avg}$, BD-PSNR [36], and BD-BR). The specific calculation formulas are as follows:

$$∆T=\frac{T_{proposed}-T_{HM16.5}}{T_{HM16.5}}\times 100\%,$$

(8)

$$∆T_{avg}=\frac{1}{4}{\sum }_{i=1}^4\frac{T_{proposed}\left(Q{P}_i\right)-T_{HM16.5}\left(Q{P}_i\right)}{T_{HM16.5}\left(Q{P}_i\right)}\times 100\%,$$

(9)

$$BD-PSNR=\frac{1}{r_h-r_l}{{\displaystyle \int }}_{r_l}^{r_h}\left(D_{proposed}\left(r\right)-D_{HM16.5}\left(r\right)\right)dr,$$

(10)

$$BD-BR=\frac{1}{D_h-D_l}{\int }_{D_l}^{D_h}\left(r_{proposed}-r_{HM16.5}\right)dD,$$

(11)

where $T_{HM16.5}$ and $T_{proposed}$ denote the coding time by HM16.5 and the proposed method, respectively; $D_{proposed}$ and $D_{HM16.5}$ are the two corresponding RD curves; $r_l$ and $r_h$ are the low and high ends of the output bit rate range, respectively; $r_{HM16.5}$ and $r_{proposed}$ are the corresponding bit rates of the original one in HM16.5 and the proposed method, respectively; $D_l$ and $D_h$ are the low and high ends of the output RD curve range, respectively.

3.2. Ablation Experiments

To evaluate the method of the CNN-CBAM and CNN, three ablation experiments were performed by using the same configurations for the first 20 frames of each sequence under the four QP values. The results are displayed in Figure 7, where the point values indicate the average of the five types (A–E) of sequences at the four QPs.

Figure 7a indicates that the coding time saved by CNN-CBAM and CNN was very similar and that there was almost no difference. This is because the introduced CBAM is a lightweight network and does not significantly increase the complexity of the network. From Figure 7b,c, we can see that the CNN-CBAM had a different degree of improvement in RD performance compared to the CNN, which is because the CNN-CBAM network can refine the feature map after adding CBAM, thus improving the representational capability of the CNN. Therefore, the RD performance of CNN-CBAM is more advantageous compared with the CNN.

3.3. Analyzing Experimental Outcomes

3.3.1. Performance Comparison with HM16.5 Encoder

To evaluate the proposed CNN-CBAM algorithm, we performed it and the original one in HM16.5. It has already been mentioned how the four indicators were obtained (i.e., BD-BR, BD-PSNR, $∆T$, and $∆T_{avg})$.

Table 4. Results for the A–E sequences.

Resolution	BD-BR	BD-PSNR	$\mathit{∆}\mathit{T}\left(\mathit{\%}\right)$				$\mathit{∆}{\mathit{T}}_{\mathit{a}\mathit{v}\mathit{g}}$
	(%)	(dB)	QP = 22	QP = 27	QP = 32	QP = 37	(%)
People on Street	2.11	−0.12	−62.59	−68.02	−65.43	−64.56	−65.15
Traffic	2.47	−0.13	−67.50	−66.77	−69.64	−73.14	−69.26
Average Class A	2.29	−0.13	−65.05	−67.40	−67.54	−68.85	−67.21
Basketball Drive	3.86	−0.09	−71.37	−77.46	−80.00	−82.38	−77.80
BQ Terrace	1.37	−0.09	−65.25	−64.76	−74.41	−67.51	−67.98
Cactus	2.02	−0.05	−60.66	−64.91	−68.34	−71.48	−66.35
Kimono	1.47	−0.05	−83.50	−85.35	−89.99	−90.42	−87.32
Park Scene	2.15	−0.09	−74.99	−67.27	−70.63	−74.17	−71.77
Average Class B	2.17	−0.07	−71.15	−71.95	−76.67	−77.19	−74.24
Basketball Drill	2.94	−0.14	−46.77	−51.38	−68.23	−74.97	−60.34
Party Scene	0.86	−0.07	−42.50	−40.93	−42.54	−45.13	−42.78
Race Horses C	1.64	−0.10	−54.81	−54.63	−57.61	−62.58	−56.51
BQ Mall	1.18	−0.07	−46.86	−50.37	−53.94	−58.06	−52.33
Average Class C	1.65	−0.10	−47.53	−49.10	−55.35	−59.97	−52.99
Basketball Pass	1.96	−0.11	−62.21	−72.57	−75.99	−78.17	−72.24
Blowing Bubbles	0.74	−0.04	−37.07	−40.11	−45.55	−51.86	−43.76
BQ Square	0.24	−0.02	−64.49	−44.78	−46.21	−47.60	−50.77
Race Horses	1.04	−0.07	−45.57	−47.00	−49.34	−53.55	−48.87
Average Class D	1.00	−0.06	−52.44	−51.18	−54.20	−57.81	−53.91
Four People	2.61	−0.15	−68.61	−75.95	−66.71	−76.08	−71.84
Johnny	3.39	−0.14	−71.93	−74.21	−76.60	−78.31	−75.26
Kristen & Sara	2.82	−0.14	−70.09	−71.45	−73.58	−75.15	−72.57
Average Class E	2.94	−0.14	−70.21	−73.87	−72.30	−76.51	−73.22
Average	1.94	−0.09	−60.91	−62.07	−65.20	−68.02	−64.05

indicates that the computational complexity of the suggested method was significantly lower than that of the method in HM16.5. On the one hand, the proposed approach saved an average 64.05% of time. The complexity reduction was especially evident for class B with gentle motion, reaching 74.24%, and even for class C sequences with vigorous motion, reaching 53.21%. For sequences such as Kimono, which has a background with high texture complexity and slow-moving foreground targets, the average time saving even reached 87.32%. The proposed CNN-CBAM method for the CU partition’s accurate prediction accounted for the proposed algorithm’s low complexity, thereby avoiding unnecessary redundant iterations. On the other hand, although there was a decrease in RD performance per sequence, it was not significant. The average loss of video quality was only 0.09. There was only a 1% average bit rate increase. Undeniably, the proposed algorithm, in general, maintained a good rate–distortion performance.

To display the proposed algorithm’s RD performance, we took the two sequences with the greatest difference in performance loss among the test sequences as an example. Figure 8 depicts the RD curves of the Basketball Drive and BQ Square sequences. The figure indicates that the suggested algorithm’s RD performance lines up with the original approach in HM16.5. The experimental results showed that various sequences with varied video properties had comparable characteristics. In addition, the average loss of the video quality was only 0.09 dB, with a maximum drop of 0.15 dB (for the Four People sequence) and a minimum drop of 0.02 dB (for the BQ Square sequence), which was almost negligible. Undeniably, the network structure we devised efficiently decreased the coding complexity while maintaining the coding quality.

3.3.2. Comparison of Subjective Quality

To assess the subjective quality, we compared the experimental results of the proposed algorithm and the one in HM16.5 on sequences with different characteristics. For example, the Class A Traffic sequence with a flat mean, the Class B Cactus and Class E Kristen & Sara sequence with a complex background, and the Class C BQ Mall sequence with complex heterogeneous texture. To make it easier to see the image loss details, we provide a local magnification area for subjective quality comparison, as shown in Figure 9. The subjective effects of the suggested method and the one in HM16.5 were almost identical. The PSNR loss of Traffic, Cactus, BQ Mall, and Kristen & Sara in the suggested method was 0.0181, 0.0239, 0.0068, and 0.0317, respectively, when compared to the methodology in HM16.5. Thus, the method proposed has an excellent performance in both the image quality and coding efficiency.

Moreover, to further demonstrate why the suggested method is of low complexity, we compared the CU partition results of the proposed algorithm to that of HM16.5. Figure 10a,b show a zoomed-in demonstration of the two algorithms in specific regions (r1–r6). On the one hand, as illustrated in Figure 10a, although HM16.5 exhibited strong results as a high-efficiency algorithm for CU partition, this efficiency may come at the cost of increased computational complexity. However, compared with Figure 10a, the CU partitioning of our scheme was very consistent with that of HM16.5, which is one of the important reasons for the high coding performance obtained by our method. On the other hand, from a subjective perspective, the HM16.5 method divides some regions that do not need CU partition, while the proposed method does not divide these areas, which better reflects the subjective feelings. Specifically, Figure 10a,b highlights yellow regions that are not divided in our method but are divided in HM16.5. This is one of the reasons for the low complexity of our scheme. It is noteworthy that similar results were obtained for other test sequences of JCT-VT, which proves the robustness and effectiveness of the proposed algorithm.

3.3.3. Comparison of the Proposed Algorithm with Three Existing Algorithms

We compared the proposed method to the competing algorithms including the heuristic-based scheme by Zhang et al. [11], the CNN-based algorithm by Liu et al. [21], and the CNN-based scheme by Fan et al. [28] to further illustrate its superior performance, as shown in

Table 5. The BD-BR, BD-PSNR, and complexity comparison between the proposed algorithm and three algorithms of Refs. [11,21,28].

Sequence	Algorithm [11]			Algorithm [21]			Algorithm [28]			Proposed Algorithm
	BD-BR	BD-PSNR	$\mathit{∆}{\mathit{T}}_{\mathit{a}\mathit{v}\mathit{g}}$	BD-BR	BD-PSNR	$\mathit{∆}{\mathit{T}}_{\mathit{a}\mathit{v}\mathit{g}}$	BD-BR	BD-PSNR	$\mathit{∆}{\mathit{T}}_{\mathit{a}\mathit{v}\mathit{g}}$	BD-BR	BD-PSNR	$\mathit{∆}{\mathit{T}}_{\mathit{a}\mathit{v}\mathit{g}}$
	(%)	dB	(%)	(%)	dB	(%)	(%)	dB	(%)	(%)	dB	(%)
People on Street	1.77	−0.10	−53.34	2.31	−0.12	−62.30	2.84	−0.15	−57.56	2.11	−0.12	−65.15
Traffic	2.38	−0.12	−58.51	2.35	−0.11	−60.80	2.64	−0.13	−65.53	2.47	−0.13	−69.26
Average Class A	2.08	−0.11	−55.93	2.33	−0.12	−61.55	2.74	−0.14	−61.55	2.29	−0.13	−67.21
Basketball Drive	2.18	−0.06	−57.53	3.67	−0.09	−71.00	3.91	−0.11	−72.26	3.86	−0.09	−77.80
BQ Terrace	1.47	−0.07	−56.21	2.32	−0.12	−62.40	1.99	−0.09	−58.21	1.37	−0.09	−67.98
Cactus	2.20	−0.07	−58.25	2.57	−0.09	−59.90	2.18	−0.07	−65.95	2.02	−0.07	−66.35
Kimono	1.82	−0.06	−67.30	3.07	−0.10	−59.80	2.84	−0.10	−80.90	1.47	−0.05	−87.31
Park Scene	2.35	−0.10	−61.97	1.82	−0.07	−59.30	2.57	−0.11	−65.09	2.15	−0.09	−71.77
Average Class B	2.00	−0.07	−60.25	2.69	−0.09	−62.48	2.70	−0.10	−68.32	2.17	−0.08	−74.24
Basketball Drill	2.06	−0.10	−48.93	3.98	−0.18	−56.60	2.27	−0.11	−53.75	2.94	−0.14	−60.34
Party Scene	1.37	−0.11	−48.44	1.95	−0.13	−50.60	0.66	−0.05	−42.83	0.86	−0.07	−42.78
Race Horses C	2.50	−0.14	−47.77	2.03	−0.12	−54.20	1.87	−0.10	−60.06	1.64	−0.10	−56.51
BQ Mall	3.15	−0.17	−56.31	2.77	−0.15	−57.30	2.31	−0.12	−55.79	1.18	−0.07	−52.33
Average Class C	2.27	−0.13	−50.36	2.68	−0.15	−54.68	1.78	−0.10	−53.11	1.66	−0.10	−52.99
Basketball Pass	2.72	−0.15	−54.25	2.92	−0.16	−59.60	1.52	−0.09	−57.30	1.96	−0.11	−71.84
Blowing Bubbles	1.43	−0.09	−46.92	2.44	−0.13	−53.00	0.64	−0.04	−40.32	0.74	−0.04	−43.76
BQ Square	1.20	−0.10	−45.47	1.39	−0.11	−53.40	0.52	−0.04	−46.04	0.24	−0.02	−50.77
Race Horses	/	/	/	2.41	−0.14	−52.70	1.45	−0.08	−52.01	1.04	−0.07	−48.87
Average Class D	1.78	−0.11	−48.88	2.29	−0.14	−54.68	1.03	−0.06	−48.92	1.00	−0.06	−53.91
Four People	2.71	−0.15	−59.89	3.06	−0.16	−59.60	1.76	−0.10	−64.78	2.61	−0.15	−71.84
Johnny	2.48	−0.10	−64.67	4.58	−0.18	−72.70	2.31	−0.09	−75.44	3.39	−0.14	−75.26
Kristen & Sara	3.25	−0.16	−62.36	3.10	−0.15	−69.30	2.21	−0.11	−73.34	2.82	−0.14	−72.57
Average Class E	2.81	−0.14	−62.31	3.58	−0.16	−67.20	2.09	−0.10	−71.19	2.94	−0.14	−73.22
Average	2.18	−0.11	−55.77	2.71	−0.13	−59.69	2.03	−0.09	−60.35	1.94	−0.09	−64.05

. The meanings of BD-BR (%), BD-PSNR (dB), and $∆T_{avg}$ (%) were the same as noted above. On the one hand, for the coding complexity, we noted that the proposed scheme was reduced on average by 64.05%, better than both the 59.69% in [21] and 60.35% in [28], and even about 10% higher compared to [11]. The method has such a low complexity because it can extract all features of the current CTU pixels from both the global information and refined texture features, while the encoding process does not require any intermediate features, and the CNN-CBAM can predict all of the CU partition results at one time, skipping more encoding unit depth computation processes, and thus effectively removing redundant CU partition computations. In addition, as above-mentioned, for some regions, especially for the regions with gentle movement and simple textures, the CU partition of the CNN-CBAM better reflected the subjective feelings than the method in HM 16.5. For these regions, the CNN-CBAM terminated the CU partition early, which further reduced the intra encoding time.

On the other hand, for the RD performance, all four methods in Table 5 showed a slight loss compared with the method in HM16.5; the proposed method was superior to the methods in [11,21,28] both in terms of the code rate and video quality with 2.18% and −0.11 dB, 2.71% and −0.13 dB, and 2.03% and −0.09 dB, respectively. Our method had a better generalization ability compared with competing algorithms. Meanwhile, the CBAM algorithm could better identify the features of the current block and induce the CNN to focus on the target object correctly, thus improving the network prediction performance.

In summary, compared to heuristic-based methods [11], the CNN-CBAM based scheme excels in feature extraction and adaptively adjusts to scene characteristics, optimizing the network structure. In contrast to CNN-based schemes [21,28], the distinctive convolutional block design in CNN-CBAM strengthens the interaction of information between blocks, leading to the acquisition of more abundant texture features. Moreover, the incorporation of CBAM enhances image quality without incurring additional computational complexity. It is undeniable that our method has an advantage over competing methods.

In Figure 11, the results of the comparison between the other three algorithms and the proposed algorithm are presented in the form of three-dimensional visualization graphs, which vividly show the differences between the four algorithms on the three metrics. Note that the values of BD-PSNR (dB), $∆T{ }_{all}$ (%), and BD-BR (%) were the average of all 18 test sequences. Undeniably, the proposed algorithm is more advantageous in terms of the overall criteria for assessment compared with the three methods in [11,21,28]. On the one hand, the proposed method has significant advantages in reducing the encoding complexity since the proposed CNN-CBAM can classify blocks more comprehensively and faster, and consequently reduce many RDO candidates. On the other hand, the feature refinement process of the CBAM enables the CNN-CBAM to effectively utilize the given features, which enhances the neural network’s capacity for generalization and maintains good coding performance.

We further used a metric (i.e., figures of merit (FoM) mentioned in [37]) to more objectively show the complexity–RD performance of the proposed method. The FoM is calculated as follows:

$$FoM=\left|\frac{BD-BR}{\Delta T_{avg}}\right|\times 100$$

(12)

the FoM can be used to compare the strengths and weaknesses of different algorithms and to find the optimal balance between low RD performance loss and high complexity reduction (i.e., the better the performance of the method, the smaller the FoM value, and vice versa).

Table 6. Comparison of the FoM of the four methods.

Parameter	Algorithm [11]	Algorithm [21]	Algorithm [28]	Proposed
FoM	3.9089	4.5401	3.3637	3.0312

presents the FoMs of the different approaches including the proposed one and the existing methods in [11,21,28]. A lower value for FoM is desirable as it indicates a better compromise between RD efficiency and reduced computational complexity. Table 6 shows that the proposed method’s FoM was 3.0312, which was significantly lower than the other methods in [11,21,28]. The result demonstrates that the proposed CNN-CBAM attained a favorable balance between the low encoding complexity and high RD performance.

4. Conclusions and Feature Works

In this paper, we proposed an optimization algorithm for CNN based on the hybrid attention mechanism (i.e., CNN-CBAM), which aimed to accelerate CU partitioning in the HEVC. Firstly, we extracted the global information and local texture features of the video sequence using CNN. Then, we introduced CBAM to CNN to construct CNN-CBAM, which was then embedded into the coding framework of HECV to predict the CU partition in advance. CBAM mainly extracted the nonuniform fusion of the fetched multi-channel features, which in turn improved the information representation of the fused features. Finally, the experimental data showed that the CNN-CBAM could reduce the intra-frame coding time to 64.05%. Despite a slight performance loss in BD-PSNR and BD-BR (with a BD-PSNR and BD-BR of −0.09 dB and 1.94%, respectively), relatively good video quality was maintained.

Although some findings have been made in this study, there is room for further refinement and expansion in the future. We can further optimize and refine the CNN-CBAM model by combining the rules of inter-frame coding. In addition, we will further explore how to apply the optimized CNN-CBAM to VVC. With such improvements, we expect to achieve better results and advance the technology in the field of video compression.

In conclusion, the algorithm presented in this paper offers a promising solution for accelerating CU partition in the HEVC. However, future research endeavors should focus on refining and extending the algorithm to achieve even better results.