Content-Symmetrical Multidimensional Transpose of Image Sequences for the High Efficiency Video Coding (HEVC) All-Intra Configuration

Abstract

Enhancing the quality of video coding whilst maintaining compliance with the syntax of video coding standards is challenging. In the literature, many solutions have been proposed that apply mainly to two-pass encoding, bitrate control algorithms, and enhancements of locally decoded images in the motion-compensation loop. This work proposes a pre- and post-coding solution using the content-symmetrical multidimensional transpose of raw video sequences. The content-symmetrical multidimensional transpose results in images composed of slices of the temporal domain whilst preserving the video content. Such slices have higher spatial homogeneity at the expense of reducing the temporal resemblance. As such, an all-intra configuration is an excellent choice for compressing such images. Prior to displaying the decoded images, a content-symmetrical multidimensional transpose is applied again to restore the original form of the input images. Moreover, we propose a lightweight two-pass encoding solution in which we apply systematic temporal subsampling on the multidimensional transposed image sequences prior to the first-pass encoding. This noticeably reduces the complexity of the encoding process of the first pass and gives an indication as to whether or not the proposed solution is suitable for the video sequence at hand. Using the HEVC video codec, the experimental results revealed that the proposed solution results in a lower percentage of coding unit splits in comparison to regular HEVC coding without the multidimensional transpose of image sequences. This finding supports the claim of there being increasing spatial coherence as a result of the proposed solution. Additionally, using four quantization parameters, and in comparison to regular HEVC encoding, the resulting BD rate is −15.12%, which indicates a noticeable bitrate reduction. The BD-PSNR, on the other hand, was 1.62 dB, indicating an enhancement in the quality of the decoded images. Despite all of these benefits, the proposed solution has limitations, which are also discussed in the paper.

1. Introduction

Digital video coding has important applications in video storage and communications. This is because the raw videos are composed of a sequence of non-compressed images that require massive storage space and cannot be used for video communications. Video coding standards compress these raw videos with a compression ratio of up to 1000:1, making storage and the communication of the videos feasible. Such compression ratios are possible because in video coding algorithms, lossy compression is used mainly in the many-to-one quantization of DCT coefficients. This can also be combined with a reduction in the spatial and temporal resolutions of the input videos, all of which are performed with the constraint of maintaining the highest subjective quality of the reconstructed video as possible.

Briefly, the first usable video coding standard was H.261, which was created in 1988 [1]. This was followed by MPEG-1 in 1992 [2], which introduced bidirectional frame coding. Soon after, MPEG-2 was finalized in 1995 [3], which introduced half-pixel motion estimation. MPEG-2 became very popular as it was the digital format for TV signals broadcast over the air, as well as cable and satellite TV systems. MPEG-2 also introduced scalable or multilayer video coding, which never became popular as it generates inferior quality to the simpler alternative based on bit stream switching. In 1996, H.263 [4] was published for use in low-bitrate communications. H.263 introduced quarter-pixel motion estimation. Additionally, MPEG-4 was created in 2003 [5], and it included object-based compression, which did not become popular simply because image segmentation was inaccurate at that time. However, AVC, which is also known as H.264 [6], has become very popular, being used in streaming platforms. A decade later, HEVC was introduced [7]; it went back to the basic block-based DCT compression and discarded object-based coding. It introduced the concept of large coding units with adaptive recursive splits, making it suitable for ultra HD stream. Lastly, VVC was introduced in 2021 [8], where coding units became double their counterpart in HEVC, allowing for greater flexibility in adaptive recursive splitting according to the video content.

Many authors have proposed solutions for enhancing the quality of coded video. The challenge here is to enhance the quality whilst remaining compliant with the syntax of the underlying video codec. There are two approaches for doing so: in the first approach, the enhancement algorithms are performed as a pre- and/or post-processes with respect to video coding; on the other hand, the second approach enhances coding algorithms that are already compliant with the video codec syntax. These algorithms include bitrate control, two-pass encoding, in-loop filtering, motion estimation, and the selection of sizes and coding modes for coding units.

For video coding enhancements based on two-pass encoding, the work in [9] proposed an efficient low-latency two-pass encoding solution for live video streaming applications. In the first pass, feature variables are extracted to predict the bitrate optimal constant rate factor to be used in the second-pass constrained variable bitrate coding. Compared to single-pass encoding, the solution resulted in bitrate savings of up to 11%. Additionally, the work in [10] proposed a two-pass encoding rate–distortion optimization method to improve the coding efficiency of HEVC. In the first pass, a video frame is HEVC-encoded to obtain the rate–distortion model of coding units and the number of bits allocated for the frame. In the second pass, an optimal equation combines the rate–distortion model and the rate–distortion, which is used to determine the Lagrange multiplier and quantization parameter for each coding unit. This results in rate–distortion improvement performance of up to 5.6% in comparison to HEVC coding. More recently, the study in [11] proposed a constrained two-pass per-title VBR encoding scheme, which is an optimized bitrate ladder approach for live video streaming that reduces storage and delivery costs while improving quality of experience. Evaluated with the HEVC encoder, the solution achieved an average bitrate reduction of 18.80% compared to standard HTTP live-streaming CBR encoding. Additionally, it results in a 68.96% reduction in storage space and an 18.58% reduction in encoding time, demonstrating its efficiency for adaptive live streaming. Moreover, the work in [12] addressed complexity reduction in two-pass rate control for the versatile video encoder. The authors propose spatial and temporal sub-sampling during the first encoding pass to speed up the overall process. The encoding process achieves an 18% speedup while incurring only a 0.48% loss in coding efficiency.

For video coding enhancements using bitrate control algorithms, the work in [13] proposed the utilization of entropy-based visual saliency models within the framework of HEVC. Consequently, the quantization parameters are adjusted according to visual saliency relevance at the coding tree unit level. Efficient rate control is achieved by allocating bits to salient and non-salient coding tree units by manipulating the quantization parameters according to their perceptual weighted map. Bitrate reductions up to 6.6% in comparison to HEVC are reported. The work in [14] proposed SSIM–MSE distortion models at the coding tree unit level to enable the performance of SSIM-based rate–distortion optimization with a simpler R-DMSE cost scaled by the SSIM-based Lagrangian parameter. Compared to HEVC encoding, the proposed solution results in bitrate savings of 5%, 11%, and 17% using the same SSIM in all-intra, hierarchical, and non-hierarchical low-delay-B configurations. The work in [15] propose a novel bitrate control method that takes into account the distortion characteristics of inter-frame coding for updating the parameters of bitrate control. The paper also proposed a low-complexity I-frame quantization parameter decision strategy for low delay scenarios in which estimated distortion characteristics and the previous quantization parameters are made use of. The proposed method resulted in 2.6% bitrate saving in comparison to HEVC coding. More recently, the work in [16] proposed a CTU-level bit allocation improvement scheme for intra-mode rate control. A dataset is created using natural images, and various metrics are applied to determine the significance and complexity of each coding unit. The most important coding units are weighted differently, and their optimal adjustment values are incorporated into the dataset. A PLS regression model is then used to refine the bit allocation weights. The proposed method improves rate control accuracy by 0.453%, Y-PSNR by 0.05 dB, BD rate savings by 0.33%, and BD-PSNR by 0.03 dB compared to the standard video coding rate control algorithm. Moreover, the work in [17] presented a neural network-based rate control method for intra-frame coding. A neural network-based model is developed to predict bit allocation by mapping video content features to estimated bit usage at both the frame and CTU levels. Additionally, an improved parameter updating algorithm is introduced at the frame level. Experimental results show that ENNRC achieves 7.23% BD rate savings while providing more accurate bit allocation compared to VVC’s default rate control algorithm.

For video coding enhancements using the pre/post-processing of input and decoded images, the work in [18] proposed a frame-level filtering solution based on CNNs for enhancing the decoded video quality of HEVC. This is achieved using a deep neural network architecture for post-filtering the decoded all-intra videos. The proposed solution serves as an alternative to the HEVC in-loop filtering for intra-coded frames. The solution resulted in a BD rate saving of 11.1% compared with the HEVC reference model. The work in [19] introduced CNN-based up-sampling for intra-frame coding in HEVC, which down-sampled blocks before coding and then used a custom CNN to up-sample them. Implemented into HEVC, a BD rate saving of up to 9.0% is reported. Additionally, the work in [20] presented a quality enhancement CNN (QE-CNN) for HEVC that improved the quality of locally decoded images without altering the encoder. The reported BD rate enhancements averaged 8.31%. In [21], a post-processing solution was proposed which enhances the quality of the locally decoded images of a VVC codec by training a deep CNN network that receives the locally decode images, their prediction errors, and quantization maps. When deployed, the network enhances the quality of locally decoded images, which are consequently used for the prediction of future images, thus reducing the BD rate by 1.52% without increasing the computational complexity. Other similar solutions that work on enhancing locally decoded images include the work reported in [22] using a squeeze-and-excitation CNN, resulting in 10.05% BD rate savings. Lastly, in [23], the authors proposed a patch-wise spatial–temporal quality enhancement network which extracts and fuses spatial–temporal features. Using the HEVC baseline under LDP configuration, the work resulted in a BD rate savings of 17.24%.

On the other hand, the objective of this work is to enhance video coding through a pre- and post-processing approach, where input images undergo a content-symmetrical multidimensional transpose (CSMT) before compression. The transpose is content-symmetrical, meaning the video content remains unchanged but its spatial axes are permuted. As explained in later sections, the proposed method is particularly effective for the all-intra configuration in video coding. After decoding, the images are restored to their original form using the same CSMT. We analyze the impact of this transformation on the homogeneity of raw images and its influence on the coding process by examining the percentage of coding unit (CU) splits. Additionally, we propose a lightweight two-pass encoding approach, where the suitability of the video sequence for CSMT is assessed in the first pass before proceeding with full encoding in the second pass.

The rest of the paper is organized as follows: Section 2 provides on overview of the proposed CSMT of input videos. In Section 3, we introduce the overall system architecture. Section 4 discusses the limitations of the proposed system, Section 5 presents the experimental results, and Section 6 concludes the paper.

2. Proposed Solution

The proposed solution relies on the content-symmetrical multidimensional transpose (CSMT) of YUV video images as a pre-process to video coding and as a post-process to video decoding. The CSMT results in video images of temporal slices composed of temporal information from many images over a small spatial area, which is one line from each image. This transpose is illustrated in Figure 1, where the upper part represents input images and the lower part represents output images, where each is a temporal slice composed of individual spatial lines of input images.

Mathematically, a sequence of images can be represented as a three dimensional matrix as follows:

$$I={\left[i_{xyz}\right]}_{w.h.f}$$

(1)

where I is a sequence of images; $i_{xyz}$ are pixel values at coordinates x; y in an image at index z; and w, h, and f represent the number of rows, number of columns, and total number of frames, respectively. The following constraints apply: $1\le x\le w,1\le y\le h, \mathrm{a}\mathrm{n}\mathrm{d} 1\le z\le f$.

The CSMT of I can be represented through swaps in indices and swaps in the dimensions as follows [24].

$$I^T={\left[i_{xzy}\right]}_{w.f.h}$$

(2)

Applying multidimensional transpose twice on the set of images returns them to their original form. This is mathematically expressed as follows:

$${\left(I^T\right)}^{T_{\left(\mathrm{2,3}\right)}}=I$$

(3)

where T^(2,3) represents swapping the second and third axes (z and y) to restore the original order of the images.

Note that this approach can be used on a group of images, with a total number of images in each group equal to the number of rows in each image. This guarantees that the transposed images have the same spatial dimensions as the input images. Figure 2 shows example images transposed using CSMT, belonging to different video sequences.

The figure lists both positive examples, where the proposed solution was suitable, and negative examples, where it was unsuitable. Suitability here refers to whether or not the proposed transpose resulted in video compression enhancements in terms of BD rate and BD-PSNR. Such suitability of the proposed solution to various sequences can be detected prior to full encoding, as shall be elaborated upon in this section.

In general, it was experimentally observed that on average, the images transposed using CSMT have lower spatial variance and entropy than the original images. More specifically, using the video sequences listed in the experimental results section, it was observed that the average spatial variance per image dropped from 1624 to 1466, and the average entropy per image dropped from 7.03 bits to 6.86 bits, which represent 9.73% and 2.42% drops in variance and entropy, respectively.

These statistics indicate that the images transposed using CSMT are good candidates for the all-intra coding configuration of the HEVC video codec. Clearly, such images are not good candidates for inter-frame coding as the transpose increases the spatial homogeneity at the expense of the temporal resemblance.

Therefore, one research question raised in this research work is related to whether or not the content-symmetrical multidimensional transpose of images enhances the efficiency of video coding using the all-intra configuration. Another research question is related to whether or not individual video sequences can be examined to attain the suitability of the proposed solution prior to full encoding. These questions are addressed in the sections to follow.

3. Overall System Architecture

To address the two research questions posed in the previous section, we propose the system architecture illustrated in Figure 3.

The first block in the figure pertains to the decision logic in which a decision is made on whether or not the proposed solution is suitable for a particular video sequence. The details of this decision logic are presented later in this section. If a video sequence is deemed suitable for the proposed solution, then the input images are subjected to CSMT prior to video encoding. After that, the compressed video is either transmitted and/or stored. This is followed by video decoding, where CSMT takes place again prior to video display. As noted in the previous section and illustrated in Equation (3), transposing the transposed sequence puts it back to its original non-transposed form.

On the other hand, if a video sequence is deemed unsuitable for the proposed solution, then it goes through the typical video coding pipeline illustrated in the figure. Note that the figure is illustrated in such a way as to emphasize that the proposed solution is a pre- and a post-process to video coding, without the need for video syntax amendments or modifications to the compression algorithm itself.

The decision logic block in Figure 3 is elaborated upon in Figure 4. Basically, this can be considered a lightweight first encoding pass in which the input images are temporally subsampled prior to encoding. One valid approach is to use systematic sampling in which every kth image from the input sequence is retained. Formally, with N images, to retain n images only, the value of k is calculated as follows.

$$k=⌊{\displaystyle \raisebox{1ex}{$N$}\!\left/ \!\raisebox{-1ex}{$n$}\right.}⌋$$

(4)

Consequently, the images to retain will be at the following indices:

Retained images indices = p, p + k, p + 2 k,…, p + (n − 1) k

(5)

where p is a number between 1 and k.

In Figure 4, the systematic temporal sub-sampling step makes sense as in the all-intra coding configuration, temporal dependencies between video frames are not taken into account; rather, each video frame is compressed as an intra-frame in isolation from the surrounding video frames. As such, temporal sub-sampling does not affect the coding efficiency of the underlying video.

After the systematic temporal sub-sampling, the images are subjected to CSMT and encoded. Since the encoder results in locally decoded images, there is no need to run the decoder separately. The PSNR of the locally decoded images and the resultant bitrate of the encoder are compared against the case of encoding without CSMT. A decision on whether or not to use the proposed solution is then made. These statements are revisited and further elaborated upon in the experimental results section.

4. Limitation of the Proposed System

In the experimental results, the proposed solution shows a clear enhancement over ordinary video coding in terms of bitrate and PSNR. However, for completeness, this section lists the limitations of the proposed solution.

First, and as mentioned previously, the proposed solution works for the all-intra coding configuration only. This is because the CSMT of input images typically results in more spatially homogeneous images at the expense of reducing the temporal resemblance. The statement can be verified experimentally by comparing the entropy of predicted images with and without the use of the proposed solution. A predicted image in this context is the difference between an image and its motion compensated for by its predecessor, which is represented as follows:

$$\begin{array}{c}\hfill {\mathit{P}\mathit{r}\mathit{e}\mathit{d}\mathit{e}\mathit{c}\mathit{t}\mathit{e}\mathit{d}\_\mathit{i}\mathit{m}\mathit{g}}_{\mathit{x},\mathit{y}}^{\mathit{f}}={\mathit{i}\mathit{m}\mathit{g}}^{\mathit{f}+1}\left(\mathit{x},\mathit{y}\right)-{\mathit{i}\mathit{m}\mathit{g}}^{\mathit{f}}\left(\mathit{x}-{\mathit{V}\mathit{x}}_{\mathit{x},\mathit{y}}^{\mathit{f}},\mathit{y}-{\mathit{V}\mathit{y}}_{\mathit{x},\mathit{y}}^{\mathit{f}}\right)\hfill \\ \hfill \mathit{f}=1..\mathit{N}-1,\hfill \\ \hfill \mathit{x}=1..\mathit{w},\hfill \\ \hfill \mathit{y}=1..\mathit{h}\hfill \end{array}$$

(6)

where ${Vx}^f \mathrm{a}\mathrm{n}\mathrm{d} {Vy}^f$ are the x and y motion vector components of the fth image, and w and h are the width and height of the image, respectively.

Having calculated the entropy of the predicted images with and without CSMT, averaged over all images, the results are 4.1 bits and 3.99 bits, respectively. Since the entropy is increased, this indicates that the proposed solution reduces the temporal resemblance; hence, it is not suitable for inter-frame coding.

The second limitation of the proposed solution is that it is more suitable for off-line coding as opposed to compressing a live stream of images. This is because, as in the proposed solution, to perform CSMT and create images with the same dimensions as the input (rows × columns), we need to have access to a number of images, which is equal to the number of rows. Again, each transposed image has one row of pixels from each image up to the number of rows in the input images. For example, if the rows and columns of images are 480 and 832, then we need access to 480 images to perform the CSMT, resulting in images with the dimensions as the input, which are 480 rows and 832 columns. This creates an initial delay equal to 480 images, which might not be suitable for encoding a live stream of images.

One can alleviate such an initial delay by creating transposed images of size (64 × 832) instead of (480 × 832), and 64 is chosen as it is the size of the largest coding unit in HEVC. While such a solution reduces the initial delay, it results in splitting images into many smaller images. When compressed using video coding, each smaller image will have its own frame header, which contains many syntax elements according to the underlying video coding standard used. These extra headers increase the coding bits and thus reduce the efficiency of the proposed solution. Reducing the initial delay is therefore not feasible; thus, the proposed solution is more suitable for off-line coding, which is the second limitation of this work.

5. Experimental Results

In this section, the video sequences reported in

Table 1. Video sequences used and their resolutions.

Seq. ID	Name	W × H	Num. of Frames
1	BasketballDrill	832 × 480	500
2	BasketballPass	416 × 240	500
3	BlowingBubbles	416 × 240	500
4	BQMall	832 × 480	600
5	BQSquare	416 × 240	600
6	Cactus	1280 × 768	500
7	CITY	704 × 576	300
8	ParkScene	1280 × 768	240
9	PartyScene	832 × 480	500

are used to generate the experimental results. The diverse characteristics of the sequences guarantee that the proposed solution is not suitable for all of them; however, the proposed decision logic introduced in Section 2 will be applied to identify such sequences.

In digital video compression, a typical experimental setup involves reporting the bitrate and PSNR using four quantization parameters (QPs). In HEVC video coding, these QPs are 22, 27, 32, and 37 [25]. As mentioned previously, the proposed solution is for the all-intra configuration of HEVC.

In Section 2, it was mentioned that the proposed multidimensional transpose of input videos reduces variance and entropy on average, making them more suitable for the all-intra coding profile. These statistics are calculated on the raw YUV images. However, additional statistics can be derived from the actual all-intra coding process to further analyze the impact of the proposed transpose on coding unit (CU) splitting. It is well known that CU splits are applied automatically by the encoder to achieve smaller block sizes that are more homogeneous in content. Therefore, a higher percentage of CU splits indicates that the input video frames are less homogeneous. The experiments are repeated four times, once for each of the quantization parameters (QP) used (22, 27, 32, and 37). The results are presented in

Table 2. Percentage of coding unit splits across different QPs.

Seq. ID	QP22	QP27	QP32	QP37	Avg.
1	0.83	0.74	0.56	0.39	0.63
2	0.66	0.56	0.48	0.39	0.52
3	0.89	0.83	0.75	0.60	0.77
4	0.69	0.63	0.56	0.46	0.59
5	0.91	0.84	0.79	0.75	0.82
6	0.71	0.56	0.47	0.36	0.52
7	0.59	0.59	0.58	0.49	0.56
8	0.63	0.53	0.43	0.30	0.47
9	0.90	0.87	0.82	0.75	0.83
Avg.	0.76	0.68	0.60	0.50	0.64
(a) Original video sequences without multidimensional transpose.
Seq. ID	QP22	QP27	QP32	QP37	Avg.
1	0.54	0.49	0.43	0.36	0.45
2	0.79	0.72	0.62	0.51	0.66
3	0.79	0.75	0.66	0.53	0.68
4	0.65	0.55	0.48	0.40	0.52
5	0.82	0.76	0.65	0.59	0.70
6	0.53	0.44	0.39	0.32	0.42
7	0.72	0.72	0.67	0.53	0.66
8	0.75	0.65	0.53	0.35	0.57
9	0.66	0.65	0.61	0.53	0.61
Avg.	0.70	0.64	0.56	0.46	0.59
(b) Video sequences with the proposed multidimensional transpose.

.

In the tables, the last rows are the average per QP, and the last columns are the averages per video sequence. As can be seen in both parts of Table 2, on average, the percentage of coding unit splits for all QPs are reduced when applying the proposed multidimensional transpose of the input videos. This indicates that the proposed solution, on average, results in a more homogeneous video content suitable for the all-intra profile. Other conclusions can be drawn from the tables. This includes the fact that on average, the percentage of coding unit splits is reduced for most video sequences but not all, hence the need for the two-pass encoding, as illustrated in Figure 4 of Section 3.

In the same tables, the last row represents the average percentage of CU splits for each QP, while the last column represents the average per video sequence. As seen in both parts of Table 2, on average, the percentage of CU splits is lower when applying the proposed multidimensional transpose. This indicates that the proposed solution results in more homogeneous video content, making it better suited for the all-intra profile.

Additionally, while the CU split percentage decreases for most video sequences, it does not do so for all cases. This further justifies the need for two-pass encoding, as illustrated in Figure 4 of Section 3.

In the following experiments, the bitrate and PSNR are reported for both the proposed solution and the standard, unmodified HEVC encoding, which serves as a benchmark in this study. These results, obtained using four QP values, are presented in

Table 3. Bitrate in kilo/s and PSNR in dB for the proposed solution and the standard HEVC encoding at four different QPs for the all-intra coding configuration.

	Proposed		Ref. HEVC			Proposed		Ref. HEVC
Seq.	Bitrate	PSNR	Bitrate	PSNR	Seq.	Bitrate	PSNR	Bitrate	PSNR
1	9194.06	42.90	11,308.06	41.77	1	5726.02	39.80	6156.61	38.39
2	2214.90	43.75	2374.41	42.60	2	1449.26	39.97	1406.72	39.00
3	2773.23	41.60	4257.62	41.00	3	1644.99	38.20	2521.83	36.90
4	11,841.29	41.50	10,698.50	41.80	4	6802.44	38.33	6399.47	38.98
5	3150.51	42.04	5150.29	41.38	5	2047.15	38.31	3450.51	36.95
6	29,508.62	40.52	36,861.16	40.61	6	12,756.76	37.80	17,324.99	37.44
7	21,742.35	41.56	15,690.92	42.09	7	13,741.02	37.23	9608.79	38.23
8	24,842.78	41.70	30,625.08	41.66	9	12,931.56	38.75	16,597.48	38.52
9	15,579.65	41.69	24,631.49	41.07	10	9730.40	38.14	15,504.52	36.87
Avg.	13,427.49	41.92	15,733.06	41.55	Avg.	7425.51	38.50	8774.55	37.92
(a) QP 22					(b) QP 27
	Proposed		Ref. HEVC			Proposed		Ref. HEVC
Seq.	Bitrate	PSNR	Bitrate	PSNR	Seq.	Bitrate	PSNR	Bitrate	PSNR
1	3476.53	36.55	3315.22	35.78	1	2038.09	33.30	1825.51	32.80
2	895.36	36.25	788.12	35.60	2	522.79	32.70	426.73	32.46
3	944.59	34.70	1388.92	33.20	3	520.21	31.50	704.44	29.90
4	3822.91	35.10	3750.57	35.96	4	2059.26	31.85	2105.14	32.90
5	1288.78	34.62	2260.92	33.04	5	808.83	31.16	1416.00	29.15
6	7123.97	35.65	9242.55	34.99	6	3903.78	33.13	4908.85	32.49
7	7664.29	33.13	5461.26	34.62	7	3613.21	29.50	2828.63	31.28
8	6762.86	35.92	8531.70	35.53	8	3476.59	33.19	4086.85	32.72
9	5830.74	34.57	9356.95	33.05	9	3288.26	31.05	5090.69	29.31
Avg.	4201.11	35.17	4899.58	34.64	Avg.	2247.89	31.93	2599.21	31.45
(c) QP 32					(d) QP 37

.

As can be seen from the results, on average, the proposed solution reduced the bitrate and enhanced the PSNR of the coded videos at all four quantization parameters. However, looking at the results of individual videos, we notice some exceptions. To clearly point out the video sequences that were suitable or unsuitable for the proposed solution, the results in Table 3 are further summarized as percentage bitrate decreases and PSNR differences, as reported in

Table 4. Summary percentage bitrate deceases and PSNR differences between the proposed solution and HEVC encoding.

	QP22		QP27		QP32		QP37
Seq.	Bitrate Decrease	ΔPSNR	Bitrate Decrease	ΔPSNR	Bitrate Decrease	ΔPSNR	Bitrate Decrease	ΔPSNR
1	18.69%	1.13	6.99%	1.41	−4.87%	0.77	−11.65%	0.50
2	6.72%	1.15	−3.02%	0.97	−13.61%	0.65	−22.51%	0.24
3	34.86%	0.60	34.77%	1.30	31.99%	1.50	26.15%	1.60
4	−10.68%	−0.30	−6.30%	−0.65	−1.93%	−0.86	2.18%	−1.05
5	38.83%	0.67	40.67%	1.36	43.00%	1.58	42.88%	2.01
6	19.95%	−0.08	26.37%	0.37	22.92%	0.66	20.47%	0.64
7	−38.57%	−0.53	−43.00%	−1.00	−40.34%	−1.49	−27.74%	−1.78
8	18.88%	0.05	22.09%	0.23	20.73%	0.39	14.93%	0.47
9	36.75%	0.62	37.24%	1.27	37.69%	1.52	35.41%	1.74
Avg.	13.94%	+0.37	12.87%	+0.59	10.62%	+0.53	8.90%	+0.49

.

Here the percentage bitrate decrease in Kbit/s is calculated as follows.

$$Decrease precent={\displaystyle \frac{\left(bitrate\left(HEVC\right)-bitrate\left(Proposed\right)\right)}{bitrate\left(HEVC\right)}}\ast 100$$

(7)

And the PSNR differences are calculated as follows.

$$\Delta \mathrm{P}\mathrm{S}\mathrm{N}\mathrm{R}=\mathrm{P}\mathrm{S}\mathrm{N}\mathrm{R}\left(\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{e}\mathrm{d}\right)-\mathrm{P}\mathrm{S}\mathrm{N}\mathrm{R}\left(\mathrm{H}\mathrm{E}\mathrm{V}\mathrm{C}\right)$$

(8)

As can be seen in the table, on average, the proposed solution resulted in around 14%, 13%, 11%, and 9% bitrate decreases when the proposed solution is used. The best decrease in bitrate results are obtained when using the lowest QP, which is 22 in this work. The bitrate savings then slightly decreases as the QP increases, which is understood in video encoding. The reported percentage decreases in bitrate are considered remarkable given that the PSNR also increased on average. This indicates that the proposed solution of applying a multidimensional image sequence transpose as a pre- and post-process in all-intra video coding works remarkably well.

However, the results cannot be perfect, with a closer look at the results reported for individual video sequences in both Table 3 and Table 4, one can observe that when using the proposed solution, video sequences 4 and 7 constantly generate worse results than standard HEVC encoding. These sequences are BQMall and City, samples images of which are displayed in Figure 2 above. Hence, the proposed lightweight two-pass encoding solution is needed to identify if a sequence is suitable or not for the proposed CSMT, as shall be elaborated upon in this section. For further visualization of the reported results, we also display samples of the rate–distortion curves of both positive and negative results (sequences 4 and 7) in Figure 5.

Figure 5a presents sample sequences where the proposed solution worked well, and Figure 5b presents the curves for the two sequences (4 and 7) that generated negative results. Note that the rest of the sequences that generate positive results are not added to Figure 5a as their curves overlap with existing ones.

Moreover, the areas between the curves of the reference HEVC and the proposed solution can be quantified by means of BD-PSNR and BD rate [26]. A positive BD-PSNR indicates that on average, the PSNR of the proposed solution is higher that of the reference HEVC encoder, while a negative BD rate indicates that on average, the bitrate of the proposed solution is lower than that of the reference HEVC encoder. So, in summary, in this work, it is desired to report a positive BD-PSNR and a negative BD rate. These results are reported in

Table 5. BD-PSNR and BD rate results for the proposed solution in comparison to regular HEVC encoding.

Sequence ID	BD-PSNR (dB)	BD Rate (%)
1	1.09	−18.61
2	0.31	−3.43
3	3.80	−46.11
4	−0.97	18.94
5	6.24	−50.06
6	1.49	−31.76
7	−3.51	68.32
8	1.35	−25.64
9	4.77	−47.76
Avg.	1.62 dB	−15.12%
Avg exc. 4 and 7	2.72 dB	−31.91%

.

As can be seen in the table, all sequences generated the desired values for the BD-PSNR and BD rate, except for sequences 4 and 7. The average BD-PSNR and BD rate, excluding these sequences, is reported as 2.72 dB and −31.91%, respectively.

To further investigate the proposed solution, we present its effect on the encoding process in terms of the percentage of splits applied to coding units during the compression. It is known in video coding that video frames are divided into blocks of pixels with a typical size of 64 × 64 pixels, referred to as largest coding units (LCUs). These coding units are then recursively divided into smaller units according to the spatial characteristics of the pixels or their prediction residuals. The smallest size of a coding unit is 4 × 4 pixels.

Table 6. Percentage of coding unit splits per video sequence per QP value for both the proposed solution and regular HEVC coding.

	QP 22		QP 27		QP 32		QP 37
Seq.	Prop.	HEVC	Prop.	HEVC	Prop.	HEVC	Prop.	HEVC
1	0.54	0.83	0.52	0.74	0.47	0.56	0.40	0.39
2	0.79	0.66	0.72	0.56	0.62	0.48	0.51	0.39
3	0.79	0.89	0.75	0.83	0.66	0.75	0.53	0.60
4	0.65	0.69	0.66	0.60	0.59	0.53	0.51	0.43
5	0.82	0.91	0.76	0.84	0.65	0.79	0.59	0.75
6	0.53	0.71	0.41	0.56	0.35	0.47	0.27	0.36
7	0.72	0.59	0.73	0.61	0.71	0.59	0.59	0.50
8	0.75	0.63	0.64	0.53	0.52	0.43	0.37	0.30
9	0.66	0.90	0.67	0.85	0.63	0.80	0.55	0.71
Avg.	0.70	0.76	0.65	0.68	0.58	0.60	0.48	0.49

lists the average percentage of coding unit splits per video sequence per QP value for both the proposed solution and regular HEVC coding.

It is interesting to observe that on average, the percentage of coding unit splits is lower in the proposed solution for all of the four quantization parameter values. This indicates that the proposed CSMT of the input image sequences results in more spatially homogeneous regions and, consequently, the decreased need for splitting the coding units. In turn, this results in fewer coded elements and lower syntax overhead in the output video bit stream, which also justifies the decrease in bitrate reported in Table 3 and Table 4 above.

Another interesting finding from the results in Table 6 is related to the percentages of splits reported for sequences 4 and 7. In general, the encoding of these sequences using the proposed solution resulted in a higher percentage of splits for the coding units, which can also help in justifying why they resulted in higher bitrates compared to the regular HEVC encoding process.

Lastly, we present the results of the proposed lightweight two-pass solution, which is used to detect if a video sequence is suitable for the proposed CSMT prior to full encoding or not. As mentioned in Section 3, systematic sampling is used for temporal sub-sampling of the video frames prior to encoding. In this work, we retain one video frame out of each 25 frames. This is a suitable step size as it encodes one or two frames per second depending on the temporal resolution of the input video, thus reducing the number of frames to be encoded by a factor of 25. Again, as mentioned in Section 3, this arrangement works for the all-intra configuration as there are no temporal dependencies between frames; thus, systematic sampling applies.

In

Table 7. BD-PSNR and BD rate results for the proposed solution in comparison to regular HEVC encoding. The first set of results belongs to the lightweight first pass, and the second set of results belongs to the second pass or the full-encoding pass.

	Lightweight First Pass		Full-Encoding Second Pass
Seq.	BD-PSNR (dB)	BD RATE (%)	BD-PSNR (dB)	BD RATE (%)
1	1.42 dB	−23.22	1.09	−18.61
2	0.57	−7.65	0.31	−3.43
3	3.78	−45.90	3.80	−46.11
4	−0.80	15.59	−0.97	18.94
5	6.11	−49.35	6.24	−50.06
6	1.45	−30.77	1.49	−31.76
7	−3.53	69.04	−3.51	68.32
8	1.34	−25.48	1.35	−25.64
9	4.84	−48.13	4.77	−47.76
Avg.	1.69	−16.21	1.62	−15.12
Avg. exc. 4 & 7	2.79	−32.93	2.72	−31.91

, we report the BD-PSNR and BD rates resulting from this lightweight first-pass encoding. For completeness, we also replicate the BD-PSNR and BD rates reported in Table 5 for the second-pass encoding, which is the full-pass encoding.

It is observed that the BD-PSNR and BD rate results are very similar in both encoding passes. The averages of the first pass are 1.69 dB and −16.21%, while in the second pass, the averages are 1.62 dB and −15.12% for BD-PSNR and BD rates, respectively. This indicates that the first lightweight pass can be used to make a decision on the suitability of the proposed solution for the underlying sequence prior to full encoding. In the table, it is shown that the BD-PSNR and BD rate results are positive and negative, respectively, except for sequences 4 and 7. In summary, in the first lightweight encoding pass, if the BD-PSNR and BD rate values are positive and negative, respectively, then the proposed solution of CSMT in the all-intra encoding configuration is a viable solution.

For completeness, the computational time for the first- and second-pass encodings are reported in

Table 8. The computational time required for the first- and second-pass encodings averaged over all sequences using the four quantization parameters.

	First Pass (s)	Second Pass (s)	$\frac{\mathbf{P}\mathbf{a}\mathbf{s}\mathbf{s}\_2}{\mathbf{P}\mathbf{a}\mathbf{s}\mathbf{s}\_1}$
Proposed	61.42	1497.15	24.37
HEVC	62.32	1504.80	24.15

. The reported times are averaged over all sequences using the four quantization parameters. The results are generated using a laptop with Windows 10 OS with a 10th generation Intel Core i9 processor, 32 GB RAM, and a NVIDIA Quadro T2000 GPU.

The results in the table show that the first pass is indeed a lightweight encoding pass as it requires a fraction of the time needed for full encoding. More specifically, the required time is reduced by around 24~25 times. This is expected as the systematic temporal subsampling applied retained every other 25th image of the input sequence.

Table 9. The computational time required for the proposed pre- and post-multidimensional transpose.

Seq. ID	Resolution	Num. Frames	Seconds	Per Frame	Per 25 Frames
1	832 × 480	500	8.98	0.018	0.449
2	416 × 240	500	1.17	0.002	0.059
3	416 × 240	500	1.17	0.002	0.059
4	832 × 480	600	10.78	0.018	0.449
5	416 × 240	600	1.41	0.002	0.059
6	1280 × 768	500	23.59	0.047	1.180
7	704 × 568	300	6.42	0.021	0.535
8	1280 × 768	240	11.33	0.047	1.180
9	832 × 480	500	8.98	0.018	0.449
			Avg.	0.020	0.491

reports the additional time required to perform the proposed multidimensional transpose as a pre- and a post-process to video compression. The results are reported according to the spatial resolution used, as reported in Table 1.

As shown in the table, the required time for the multidimensional transpose as a pre- and post-process depends on the spatial resolution of the input video. On average, the required time per frame varied from 0.002 to 0.047 s, and the required time per 25 frames varied from 0.059 to 1.18 s. These results correspond to spatial resolutions varying from 416 × 240 to 1280 × 768, respectively.

There are no similar existing solutions to compare the results of this work against; nonetheless, we compare its bitrate savings against solutions that enhanced the quality of HEVC as a pre/post-process or using two-pass encoding. As reported in the introduction, ref. [18] used a CNN-based filtering solution for locally decoded images, and ref. [19] down-sampled blocks as a pre-process and used CNN for up-sampling them as a post-process. Additionally, ref. [20] used a CNN-based enhancement approach for locally decoded images. The work in [11] used two-pass encoding, and the work in [17] proposed a solution for enhancing the video quality of intra-frame coding.

Table 10. BD rate savings of existing HEVC quality enhancement solutions based on pre/post-processes.

Solution	Ref. [18]	Ref. [19]	Ref. [20]	Ref. [11]	Ref. [17]	Proposed One-Pass	Proposed Two-Pass
BD RATE	−11.1%	−9.0%	−8.31%	−28.25%	−7.23%	−15.12%	−31.91%

lists the BD rate savings of these solutions in comparison to the proposed one.

In the Table, the DB rate of the proposed solution is presented in two variants: with and without the two-pass solution. In the later, the bitrate reductions are higher as detailed in Table 7.

6. Discussion

As mentioned previously, in addition to enhancing the bitrate savings, the proposed solution has an additional advantage related to its simplicity as it is purely based on pre- and post-processes of CSMT of image sequences and does not require deep learning or other similar predictive solutions.

In general, real-world video coding scenarios can be divided into two categories: those that require live streaming, and those that rely on pre-encoded video. The former includes applications such as video conferencing, live broadcasting, and live surveillance, while the latter encompasses video streaming services, Blu-ray disc content, and video-on-demand platforms.

While the proposed solution is not suitable for live broadcasting, it is well-suited for scenarios that rely on pre-encoded video. This is because the additional time required for the first lightweight encoding pass and the pre- and post-processing steps (involving the multidimensional transpose) can be performed offline, without real-time constraints.

Future work directions include reducing the latency of the proposed solution by applying the multidimensional transpose at the CU level instead of the frame level. A future study will be required to examine the impact of this approach on syntactical video overhead. Additionally, latency can be further reduced by utilizing deep learning to predict whether a video sequence is suitable for the multidimensional transpose without requiring the first lightweight encoding pass. Reducing the latency can help in implementing this solution in live video streaming. Another research direction involves experimenting with a modified multidimensional transpose, where video frames are constructed using planes of pixels arranged from right to left instead of from top to bottom.

7. Conclusions

In this work, the enhancement of the quality of all-intra video coding using a content-symmetrical multidimensional transpose (CSMT) of image sequences as a pre- and post-process to video coding was proposed. Having applied the proposed CSMT to various video sequences, it was reported that the average spatial variance per image dropped by 9.73%, and the average entropy per image dropped by 2.42%, thus indicating the suitability of the proposed solution for the all-intra configuration of the HEVC codec. However, in the context of inter-frame coding, the entropy of the predicted images with and without the proposed solution increased from 3.99 bits to 4.1 bits. This indicates that the proposed solution reduces the temporal resemblance; hence, it is not suitable for inter-frame coding.

In the experimental results, it was observed that the proposed solution is not suitable for all video sequences; therefore, the proposed two-pass solution was used. The BD-PSNR and BD rate results of the first pass with temporal subsampling are very similar to those of the full encoding; thus, such lightweight first-pass encoding can be used to detect the suitability of the proposed solution prior to the full encoding of the underlying video sequence. This arrangement worked well as temporal subsampling can be applied to the all-intra configuration where temporal dependencies are irrelevant.

Furthermore, the BD rate and BD-PSNR results of the proposed solution using various video sequences and four different quantization parameters resulted in −15.12% and 1.62 dB, respectively. This indicates a clear reduction in bitrate and an increase in video quality. With the two-pass encoding, having eliminated the video sequences that are not suitable for the proposed solution, the two measures increased to −31.9% and 2.72 dB, respectively.

Future work will involve reducing latency by means of applying the multidimensional transpose at the CU level instead of the frame level. Additional reductions can be achieved by using deep leaning to predict the suitability of the solution on individual video sequences to replace the two-pass coding solution.