Adaptive Transmission Strategy for Non-Uniform Coding of 360^∘ Videos

Abstract

A 360° video offers a more immersive experience, gaining increasing popularity among users. However, enhancing the transmission efficiency of 360° videos under limited bandwidth conditions remains a significant challenge. This paper segments the video into three areas: the attention area, the edge area, and the viewpoint-switching transition area. Based on the segmentation of these three distinct video areas, a novel non-uniform coding transmission method for 360° videos is presented, along with mathematical modeling to define the optimization problem. A heuristic algorithm is subsequently introduced, which adaptively determines the optimal number of tiles and allocates the bitrate for each tile in real time to enhance the user’s quality of experience (QoE). Finally, a simulation platform has been developed to validate the efficacy of the proposed algorithm by conducting comparative analyses with existing algorithms.

1. Introduction

Due to their ability to provide an immersive experience, 360° videos are becoming increasingly popular among users. When watching 360° panoramic videos, users can switch viewpoints freely by wearing head-mounted display devices or by rotating handheld devices to access the surrounding 360° video content. However, 360° videos are characterized by their large encoding volume, necessitating substantial bandwidth for transmission. For instance, a 4K-resolution 360° video requires up to 4.2 Gbps of network bandwidth [1]. Consequently, the transmission of 360° videos within bandwidth limitations presents a substantial challenge. Focusing on transmitting only the video data relevant to the current viewpoint can significantly reduce the data load. Researchers frequently employ the tile-based encoding method for 360° videos, as it minimizes the chances of stuttering during viewpoint transitions [2].

The authors of Ref. [3] introduced a multimodal spatiotemporal attention transformer designed to generate multiple viewpoint trajectories and their associated probabilities based on a historical trajectory. The approach aims to better handle abrupt changes in the user’s perspective. By framing viewpoint prediction as a classification problem, the method leverages the attention mechanism to capture both spatial and temporal features from input video frames and viewpoint trajectories, thereby enabling accurate multi-viewpoint prediction [3].

Many scholars have conducted research on 360° transmission technology and have achieved significant results. Tile-based encoding is currently a common encoding scheme for 360° videos that can reduce the overall transmission load, thereby enhancing the user’s media experience. However, since each tile is encoded and decoded individually, a problem arises: when the tile size is small, the overall video encoding volume increases; when the tile size is too large, more stuttering and buffering may occur during viewpoint switching. Therefore, tile size directly affects the user’s quality of experience (QoE). Furthermore, when users watch videos, their range of attention is limited, and this range changes with the distance between their eyes and the viewing device. Determining the optimal number and size of tiles, along with the appropriate bitrate for each tile, based on the user’s eye-to-device distance, poses a significant challenge that requires careful consideration.

To address the aforementioned issues, this study first proposes a non-uniform coding method for 360° videos, as shown in Figure 1. The 360° video is divided into three different areas: the attention area (AA), the field-of-view (FoV) area, and the out-of-field-of-view (OFoV) area. The tile sizes in these three areas vary. The size of the attention area is calculated based on human physiological principles and the distance from the eyes of the user to the viewing device. The tile sizes for the field-of-view and out-of-field-of-view areas are calculated using the algorithm introduced in this paper. Subsequently, a 360° video adaptive transmission architecture with non-uniform coding, tailored to the user’s viewing distance, is presented. Finally, an optimization algorithm is proposed to choose the suitable number, size, and bitrate of tiles for the user.

The rest of this paper is organized as follows. Section 2 examines the current state of research in adaptive 360° video transmission. In Section 3, the architecture, model, and analysis of 360° non-uniform coding are presented. Section 4 proposes an intelligent algorithm. In Section 5, we establish a simulation environment and conduct a performance evaluation to validate our approach. Finally, in Section 6, we summarize our findings and highlight future research directions.

The primary contributions of this research are as follows:

• Introduction of an adaptive transmission framework for 360° videos based on non-uniform coding, which, for the first time, calculates the user’s attention range through the user’s video viewing environment (distance between the eyes and the screen) and dynamically segments video fragments into tiles of varying numbers and sizes, selecting appropriate bitrates for each tile in real time.
• Mathematical modeling of the non-uniform coding adaptive transmission method for 360° videos and identification of the optimization problems that need to be addressed. A complexity analysis of the problem to be solved is conducted, proving it to be NP-hard.
• Proposal of a non-uniform coding algorithm for 360° panoramic video tiling and bitrate selection (NETBS), which selects the appropriate number and size of tiles and suitable bitrates for the user in real time based on the network environment. Finally, the performance of the proposed algorithm is confirmed through simulation experiments and subjective user experiments.

2. Related Works

As a popular media format today, 360° videos have become a hot research topic. Numerous scholars have conducted extensive research on this subject.

In recent research, the authors of [4] investigated a field-of-view (FoV) prediction technique based on spherical convolution. The method combines salient features extracted from 360° videos with FoV feedback, forming a multi-source prediction framework. The authors of [5] proposed a novel transcoding-supported VR video caching and delivery framework designed to optimize computational and storage resource utilization in edge-enhanced next-generation wireless networks. Additionally, the authors of [6] introduced a single-frame and multi-frame joint network, which can upscale low-resolution spherical videos to high-resolution outputs. In [7], the collaborative transcoding and caching process of 360° videos was modeled as a Markov decision process (MDP), and the authors employed a model-free deep reinforcement learning method to derive caching replacement and computational power allocation strategies. They also proposed a novel FoV-guided scheme to accelerate the training of the DDPG agent. Moreover, in a multi-access edge computing (MEC) caching system, a tile-based caching strategy for 360° videos has been proposed. The authors applied the Combinatorial Multi-Armed Bandit (CMAB) theory, which can optimize users’ quality of experience and reduce MEC energy consumption without prior knowledge of video content popularity [8]. A viewpoint reconstruction-based 360° video caching solution has been proposed, wherein the authors designed a QoE-driven reconstruction triggering scheme to maximize viewers’ QoE. The scheme determines whether reconstruction is needed based on current caching information and network conditions. To efficiently use cache space for viewpoint reconstruction, the authors proposed a heuristic aggregation-based cache replacement algorithm that carefully selects tiles to store, thereby enhancing the probability of successful viewpoint reconstruction [9]. To optimize end-to-end video streaming, a viewport-aware real-time 360-degree video streaming framework has been proposed. The framework decomposes the transmission optimization problem into two sub-problems, which are asynchronously optimized for upstream and downstream processes. The method prioritizes the uploading of more popular content at higher bitrates based on users’ real-time viewing interests [10]. Lastly, a practical neural-enhanced 360-degree video streaming framework called Masked360 has been proposed. The framework significantly reduces bandwidth consumption by transmitting only the low-resolution masked version of each video frame instead of the full frames, thereby conserving bandwidth [11].

Caching can significantly reduce video transmission latency. A novel intelligent edge caching framework has been proposed, which maximizes users’ video experience by considering fairness and long-term system costs based on a new edge caching architecture [12]. A multi-category 360° video streaming strategy based on multi-agent deep reinforcement learning (MADRL) for joint edge caching and bitrate selection has also been introduced. The approach enables more refined performance control by adopting different caching strategies and bitrate selection methods for various video categories [13]. Furthermore, an edge-client collaborative caching and super-resolution strategy based on MADRL has been proposed. The authors first constructed a video experience evaluation function that includes video quality, temporal smoothness, rebuffering time, and device energy consumption. They then modeled the edge-client collaborative caching and super-resolution problem as a multi-agent cooperative Markov decision process, summarizing it as an optimization problem. To address decision coupling among agents, a multi-agent dual-agent regularized critic algorithm based on adaptive learning has been proposed. The algorithm improves users’ video experience through appropriate collaborative caching and super-resolution methods [14].

A multi-control adaptive 360° video streaming algorithm has been proposed, where the authors integrated multiple auxiliary controls with the main control to adapt to changes in the user viewport [15]. In [16], the authors explored Non-Orthogonal Multiple Access (NOMA) and Scalable Video Coding (SVC) technologies and subsequently proposed a new multicast transmission scheme for 360° videos, aiming to maximize users’ video experience through multicast video data transmission. In another study, a transformer-based saliency model was proposed, first trained on 2D images for viewpoint prediction. Based on these results, the authors introduced a new bitrate allocation algorithm that can enhance users’ video experience even in the absence of head-movement data [17]. Additionally, a new spherical super-resolution method was introduced to enhance users’ viewing experience [18]. Guo et al. proposed an optimal wireless streaming scheme for transmitting multi-quality tiled 360° VR video from a multi-antenna server to multiple single-antenna users based on Multi-Input Multi-Output (MIMO) and Orthogonal Frequency-Division Multiple-Access (OFDMA) technologies. The method minimizes total transmission power by jointly optimizing beamforming, subcarrier allocation, transmission power, and rate allocation [19]. An optimal wireless streaming technology for transmitting multi-quality tiled 360° VR video from a server to multiple users has been introduced. This technology maximizes transmission efficiency by leveraging the characteristics of multi-quality tiled 360° VR video and user-side computational resources to increase multicast opportunities [20]. Guo et al. studied the optimal streaming method for tiled 360° VR video from a server (base station or access point) to multiple users, considering random viewing directions and stochastic channel conditions. They adopted Time-Division Multiple Access (TDMA) to optimize transmission time and power allocation, thereby reducing the average transmission energy [21]. Furthermore, Guo et al. investigated the adaptive streaming of tiled 360° videos in a multi-carrier wireless system from a multi-antenna base station (BS) to single-antenna users. Their goal was to enhance video quality while minimizing rebuffering time by adjusting the encoding bitrate for each Group of Pictures (GOP) and tuning transmission for each slot [22].

A novel joint utility-based two-tier 360° video streaming system has been proposed. The method finely estimates the contribution of each tile, allowing for longer buffer durations and bandwidth-sharing strategies. The optimal bitrate allocation strategy is determined by dynamically selecting tiles based on Model Predictive Control (MPC) [23]. Additionally, an online adaptive bitrate algorithm for 360° videos has been proposed. The authors aimed to improve users’ video experience by coordinating the download of video segments from the server. By downloading data within the user’s field of view, network bandwidth is conserved, and the real-time adjustment of tile bitrates reduces buffering occurrences [24]. A new method called DCRL360 has been proposed, where the authors incorporated automatic curriculum learning and deep reinforcement learning into the bitrate allocation and tile scheduling algorithms for 360° videos, thereby enhancing the algorithm’s efficiency [25]. For optimal adaptive 360° video streaming, a synergistic spatiotemporal user-perceived viewport prediction scheme, SPA360, has been proposed. This scheme employs a user-perceived viewpoint prediction model, providing a transparent solution for field-of-view prediction [26]. Furthermore, a tile-weighted rate-distortion (TWRD) packet scheduling optimization system has been proposed to reduce the amount of data needed for transmission and enhance users’ video experience. The authors then introduced an attention transformer to calculate discardable data packets during transmission and finally proposed a dynamic programming algorithm to solve the problem [27].

The aforementioned studies have made significant contributions to the transmission of 360° video. In these studies, 360° videos are segmented into tiles for transmission, with all tiles being the same size. However, encoding each tile separately can lead to a large amount of video encoding data if the tiles are too small, and if the tiles are too large, it can affect the user’s viewpoint-switching experience. Finding the optimal tile size to balance encoding volume and viewpoint switching remains a challenge.

To address this, this paper proposes a non-uniform coding method for 360° panoramic video tiling and bitrate selection. This method segments the 360° video into tiles of different sizes according to the human-eye attention mechanism. The central area, which is the focus of attention, is divided into a single tile, while the other areas are segmented using the proposed tile-partitioning algorithm. This algorithm determines the final number of tiles, their sizes, and bitrates, thereby enhancing users’ media experience. In previous studies, we also conducted related research, which was published in various journals. Building upon this research, we conducted the present study. The differences between this study and our previous research are shown in Appendix A.

3. Architecture and Model

In this section, we outline the concept and non-uniform coding transmission system architecture of the proposed transmission method. Initially, we digitize the model of the transmission system and formulate the optimization problem. Subsequently, we introduce a heuristic algorithm designed to solve the optimization problem presented in this paper.

3.1. Non-Uniform Coding Transmission System

Figure 2 presents the 360° adaptive transmission architecture with non-uniform coding proposed in this paper. The transmission process is as follows: the user predicts the bandwidth of the network for the next moment using a bandwidth prediction algorithm. Based on the user’s predicted bandwidth and the eye-to-device distance, the algorithm proposed in this paper calculates the required number, size, and bitrate of video tiles for the next moment. The calculation results are then sent to the media server, which encodes and transmits the video tiles to the user based on the received information.

3.2. System Model

This paper divides a 360° video into four areas: an attention area (AA), an out-of-field-of-view (OFoV) area, a peripheral area (PA), and other areas (OAs). The AA represents the most important part of the video, the area where the user’s attention is concentrated, and it has the greatest impact on the user’s QoE, hence its highest weight. In our approach, this part is segmented into a single tile. The PA is slightly less important and has a lesser impact on the user’s QoE, but it is still within the user’s field of view and can be affected by the user’s viewpoint switching. Therefore, the number and size of tiles in this part need to be calculated by the algorithm. Although the OFoV area is outside the user’s field of view, it can become part of the FoV with a change in the user’s perspective. Thus, it has the least impact on the user’s QoE and the lowest weight.

Definition 1

(Attention Area (AA)). The central area of the field of view, representing the region where the user’s attention is concentrated.

Definition 2

(Out-of-Field-of-View (OFoV) Area). The area within the user’s current field of view, excluding the AA.

Definition 3

(Peripheral Area (PA)). The area outside the user’s field of view but surrounding it.

Definition 4

(Other Areas (OA)). Any regions of the 360° video that do not fall into the aforementioned categories.

The user’s video experience is quantified as the $QoE$, which is composed of three parts: the $QoE$ gain from the AA, the $QoE$ gain from the OFoV area, and the $QoE$ gain from the PA. Due to the uneven distribution of user attention across different areas while watching a 360° video, the $QoE$ gains from these areas vary. For instance, the AA, which captures the most user attention, contributes the most to the overall $QoE$.

The QoE (quality of experience) of the user, as defined in this text, is composed of three parts: one part is the objective quality of the video, another part is the stability of the playback, and the third part is the latency in video transmission. These three parts are each represented by different weighting parameters. The influence of each part on the user’s QoE is not uniform, and these three weighting parameters can be adjusted according to the user’s own preferences. Therefore, the user’s QoE can be calculated using Equation (1):

$$QoE=\alpha ·Qo{E}_A+\beta ·Qo{E}_O+\gamma ·Qo{E}_P,$$

(1)

where $\alpha$, $\beta$, and $\gamma$ represent different weighting parameters, indicating the attention distribution in different regions.

$$\begin{array}{cc}\hfill Qo{E}_A& =Qo{S}_A-\xi R{J}_{\mu }^i-TTP,\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill Qo{E}_O& =Qo{S}_O-\xi R{J}_{\mu }^i-TTP,\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill Qo{E}_P& =Qo{S}_P-\xi R{J}_{\mu }^i-TTP,\hfill \end{array}$$

(4)

$$R{J}_{\mu }^i=\left(|j_{\mu }^i-j_{\mu -1}^i|+1\right)N_{\mu }^i,1\le i\le M,$$

(5)

where M represents the number of video tiles that need to be transmitted. $R{J}_{\mu }^i$ represents a monotonically increasing function of $N_{\mu }^i$. Here, $N_{\mu }^i$ denotes the number of bitrate switches a user has encountered in the previous segments. The parameter $\xi$ controls the balance between the bitrate and stability. The current bitrate of the user is given by $j_{\mu }^i$, and $j_{\mu -1}^i$ is the corresponding video bitrate index. $TTP$ stands for the FoV switching delay in the video.

$$TTP=\psi (t_d+t_c),$$

(6)

where $t_d$ represents the sum of the time taken to segment a 360-degree video segment into tiles and the time for encoding and decoding these tiles; $t_c$ represents the time taken for the 360-degree video data to be transmitted from the server to the client; and $\psi$ represents the weight of these two times on the user’s QoE.

$$QoS=MSSIM(P_{\mu },Q_{\mu }),$$

(7)

where $P_{\mu }$ and $Q_{\mu }$ represent the original video on the server and the reconstructed video on the client after transmission, respectively. Their resolutions are $r_v^x\times r_v^y$ and $r_f^x\times r_f^y$, respectively.

The $MSSIM$ can be computed using the following equations. The entire screen of the media player, with a resolution of $DL$, is divided into s regions.

$$\left\{\begin{array}{c}MSSIM(P_{\mu },Q_{\mu })={\displaystyle \frac{1}{s}}\sum _{j=1}^{x}SSIM(x_{\mu ,j},y_{\mu ,j})\hfill \\ \\ s={\displaystyle \frac{DL}{min\left\{{(r_c^x\times r_c^y)}_1,{(r_c^x\times r_c^y)}_2,\dots ,{(r_c^x\times r_c^y)}_M\right\}}}\hfill \\ \\ SSIM(x_{\mu ,j},y_{\mu ,j})=g(x_{\mu ,j},y_{\mu ,j}),\hfill \end{array}\right.$$

(8)

where g is a function of $x_{\mu ,j}$ and $y_{\mu ,j}$. $x_{\mu ,j}$ is the original j-$th$ tile video. $y_{\mu ,j}$ is the aware j-$th$ tile video. Moreover, there is a one-to-one mapping relationship between $y_{\mu ,j}$ and $b_{\mu ,j}^j$. $b_{\mu ,j}^j$ is the bitrate of the j-$th$ tile.

$$MSSIM(P_{\mu },Q_{\mu })=f\left(b_{\mu }^i\right),$$

(9)

f is a mathematical function that represents the one-to-one mapping relationship between the $MSSIM$ value and the video bitrate. The bitrate of the $\mu$-$th$ segment of the i-$th$ tile is denoted by $b_{\mu }^i$.

The predicted network bandwidth, denoted as $\mathcal{B}$, is determined by the following equations:

$$\mathcal{B}=\left\{\begin{array}{cc}{\displaystyle \frac{{\sum }_{k=1}^N\left({\beta }_{\nu }^k\times {\tau }_s\right)}{{\tau }_{\nu }-{\tau }_{\nu -1}}},\hfill & \mathrm{if}\mathrm{it}\mathrm{is}\mathrm{the}\mathrm{initial}\mathrm{segment}\hfill \\ \\ \gamma \times {\displaystyle \frac{{\sum }_{k=1}^N\left({\beta }_{\nu -1}^k\times {\tau }_s\right)}{{\tau }_{\nu -1}-{\tau }_{\nu -2}}}+(1-\gamma )\times {\mathcal{B}}^{*},\hfill & \mathrm{for}\mathrm{subsequent}\mathrm{segments}\hfill \end{array}\right.$$

(10)

In this context, $\mathcal{B}$ signifies the forecasted bandwidth. The term ${\beta }_{\nu }^k$ represents the bitrate of the $\nu$-$th$ segment of the k-$th$ tile. The parameters ${\tau }_{\nu }$ and ${\tau }_{\nu -1}$ denote the reception and transmission times, respectively. The playback duration of the segment is denoted by ${\tau }_s$. The term ${\mathcal{B}}^{*}$ refers to the previously available bandwidth, and $\gamma$ is the weighting factor for the current bandwidth. This framework remains effective even when alternative methods are utilized for estimating the available bandwidth.

The aggregate bitrate of the segments is expressed as follows:

$$\sum _{k=1}^N{\beta }_{\nu }^k=max\left\{\sum _{k=1}^N{\beta }_{\nu ,m}^k|m=1\dots L,\sum _{k=1}^N{\beta }_{\nu }^k\le \mathcal{B}\right\},$$

(11)

where L represents the number of quality layers for the video.

As shown in Figure 3, the user’s area of attention while watching the video is related to the distance between the user’s eyes and the viewing device. Assuming the user’s attention angle is $\theta$, the value of $\theta$ is related to the user’s eyes. The relationship can be described by the equation $tan(\theta /2)=(p/2)/d$, where the variable d represents the distance between the user’s eyes and the viewing device. The side length P of the attention area can be calculated using Equation (12):

$$p=tan\left({\displaystyle \frac{\theta }{2}}\right)·d·2.$$

(12)

Consequently, the optimization objective of this paper is as follows:

$$\begin{array}{cc}\hfill & \mathrm{maximize}\left(QoE\right)\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\mathrm{Equations}\left(1\right)–\left(12\right)\hfill \end{array}$$

(13)

3.3. Complexity Analysis

Within the aforementioned problem, it is necessary to determine the transmission rate for each tile. Initially, we demonstrate that this problem is NP-hard.

Our approach involves proving the complexity of the optimization problem delineated by Equation (13) through the identification of a specific instance of this problem. Subsequently, we show that this particular instance is analogous to the classic NP-hard exact cover problem.

Proof. 

Consider a particular scenario of Equation (13), where the current bandwidth is sufficient to transmit the lowest bitrate video for all tiles. In this case, Equation (13) simplifies to

$$\begin{array}{c}\mathrm{maximize}\sum _{i=1}^M\left(Qo{E}_i\right)\hfill \\ \mathrm{subject}\mathrm{to}\sum _{i=1}^M{N}_i⩽B\hfill \end{array}.$$

(14)

The variable $N_i$ represents the total bitrate of all transmitted tiles.

Equation (13) can be converted into a set problem. Consider a family P comprising d sets, each containing bit-vectors. The aggregate length of P is $3d$. For example, the families P: 110001 and 001110 are associated with $\{x_1,x_2,x_6\}$ and $\{x_3,x_4,x_5\}$, respectively. A sequence can be represented as an integer in the base $(N+1)$ system. Consequently, the integer representing the sequence is determined by

$$U_i=\sum _{x_i\in Q_i}{(N+1)}^{i-1}.$$

(15)

This methodology enables the formation of various integers: $U_1,U_2,\dots ,U_d$, and B. A subfamily of P that precisely covers $T=\{x_1,\dots ,x_{3d}\}$ exists if and only if the integer representing the set $U_i$ matches B. Moreover, B aligns with the sequence $11\dots 1$, which has a length of $3d$.

$$\sum _{i=0}^{3d-1}{(N+1)}^i=B.$$

(16)

Now, suppose there exists a set $T\subseteq \{1,2,\dots ,N\}$ such that ${\sum }_{i\in T}{U}_i=B$. In base $(N+1)$ arithmetic, the elements in the resulting set of the sum of T are either 1 or 0. When the elements of set T are combined, the total number of summands is less than $N+1$. Consequently, there is no carry in the addition. If ${\sum }_{i\in T}\left(U_i\right)=B$, each position in all sets has exactly one number 1. Thus, the subfamily $U=\{Q_i:i\in T\}$ precisely covers $\{x_1,\dots ,x_{3d}\}$, thereby demonstrating sufficiency.

Now, if $U=\{Q_i:i\in T\}$ covers $\{x_1,\dots ,x_{3d}\}$ exactly, it is evident that ${\sum }_{i\in T}\left(U_i\right)=B$. Hence, the necessity is established.

This concludes the demonstration.    □

Thus, the issue outlined in Equation (13) is classified as NP-hard. NP-hard problems are notoriously challenging to solve optimally. The 360° video adaptive transmission algorithm with non-uniform coding is employed.

4. Algorithm

This section primarily describes the 360° video adaptive transmission algorithm with non-uniform coding used to solve the problem. The algorithm is divided into two parts: the non-uniform tiling algorithm and the bitrate allocation algorithm. It addresses the issues of how to segment video fragments and how to allocate bandwidth resources to each video tile. The pseudocode for the specific algorithm is described below.

The Algorithm 1 first initializes the user’s field-of-view (FoV) position and size, defines different regions (AA, OFoV, PA, and OA), and sets the number of tiles and video quality levels, as described in lines 1–5. Next, in line 6, it initializes the utility function matrix E. Lines 7–14 use nested loops to iterate through all possible combinations of tile numbers for the PA and OFoV regions ($x_1$ and $x_2$). For each combination, lines 10–12 further loop to calculate the quality of experience (QoE) for each tile at different quality levels, and line 13 stores the maximum QoE value in the utility function matrix E. Lines 15–16 sort the values in matrix E in descending order and select the tile and quality level corresponding to the maximum value. Lines 17–24 iterate through the tiles, checking whether the remaining bandwidth B is sufficient. Lines 19–22 ensure that the selected tile’s video bitrate does not exceed the bandwidth limit. If the condition is met, the highest quality video is selected, and the bandwidth is updated; otherwise, the loop is exited. Finally, line 25 calculates the overall QoE for all video tiles, and lines 26–28 determine the optimal tiling scheme and bitrate configuration that maximize the overall QoE.

Algorithm 1: Non-uniform coding for 360° panoramic video tiling and bitrate selection algorithm (NETBS)

Initialize the user’s field-of-view (FoV) position and size. Define the FoV attention area (AA), out-of-field-of-view (OFoV) area, peripheral area (PA), and other areas (OAs). The AA area consists of 1 tile. Initialize the number of tiles, where AA: 1 tile; PA: $x_1$ tiles; OFoV: $x_2$ tiles. The video can be divided into $x_3$ quality levels. Initialize the utility function matrix E, where $E=QoE/bitrate$. for $x_1=1$ to x do      for $x_2=2$ to y do           for $i=1$ to $1+x_1+x_2$ do                for $j=1$ to $x_3$ do                     Calculate QoE                end for                Store the value of $max({\mathrm{QoE}}_1,{\mathrm{QoE}}_2,\dots ,{\mathrm{QoE}}_{x_3})$ in the utility function $E(i,j)$           end for           Sort the values of $E(i,j)$ in descending order           The highest value of $E(i,j)$, where i represents the i-th tile and j represents the j-th quality level           for $i=1$ to $1+x_1+x_2$ do                if $B\ge E(i,j)$ then                     Select the j-th quality level video for the i-th tile                     $B=B-\mathrm{bitrate}(i,j)$                else                     break                end if           end for           Calculate the overall QoE for all video tiles      end for end for Calculate the maximum QoE to determine the tiling scheme values $x_1$ and $x_2$, as well as the bitrate for each video tile. The values of x and y, respectively, represent the maximum number of segments in the PA and OFoV area, which can be defined by the user according to their specific requirements.

The Complexity Analysis of NETBS

The time complexity of this algorithm is $O(n^2·mlogm)$, where n represents the maximum values of x and y, which are the maximum number of segments in the peripheral area (PA) and out-of-field-of-view (OFoV) area, respectively. Assuming x and y are of the same order of magnitude, the outer double loop iterates over all combinations of x and y, resulting in $O\left(n^2\right)$ iterations. The middle loop processes $(1+x+y)·x_3$ elements for each combination, which is of the order $O(n·m)$. The sorting operation has a time complexity of $O(mlogm)$, where m is the number of quality levels. Hence, the overall time complexity is $O(n^2·mlogm)$.

The total number of operations for this algorithm is calculated as follows: the outer double loop iterates $x\times y$ times, where $x=10$ and $y=10$, totaling $10\times 10=100$ iterations. The maximum number of iterations for the middle loop is $(1+x+y)\times x_3$, where $x_3=5$, resulting in $(1+10+10)\times 5=21\times 5=105$ iterations. The time complexity of the sorting operation is $O((1+x+y)\times x_{3}log((1+x+y)\times x_3))$, which equates to $105\times 6.72=705.6$. Therefore, the total number of operations is $100\times 105+100\times 705.6$ = 10,500 + 70,560 = 81,060 operations.

Assuming each basic operation takes an average of 10 clock cycles and the Intel Core i9-14900HX CPU has a clock speed of 3.9 GHz, the time for each operation is $10\times {\displaystyle \frac{1}{3.9\times {10}^9}}$ s, or ${\displaystyle \frac{10}{3.9\times {10}^9}}$ s. The total time T is $81,060\times {\displaystyle \frac{10}{3.9\times {10}^9}}$ s, approximately $2.079\times {10}^{-4}$ s, which converts to 207.9 ms. Therefore, the estimated time to run this algorithm is 207.9 ms. Based on the above analysis, it can be concluded that our proposed algorithm (NETBS) fully supports real-time operation.

5. Simulation

5.1. Experimental Method

In this section, we implement the proposed transmission approach using MATLAB and evaluate the performance of the allocation method in NS3. In our experiment, the position of the user was generated randomly. The distance between the two base stations was fixed at 2 km. The simulation parameters are summarized in

Table 1. Simulation parameters.

System Bandwidth	20 MHz
Number of RBs	0–200
BS Tx Power	30 dBm
Subcarriers per RB	12
Subcarrier Spacing	15 KHz
Bandwidth per RB	180 KHz
End-to-End RTT	100 ms
Pathloss Model	COST 231 HATA urban
Fading Model	Rayleigh Fading
Antenna Type	Omnidirection
Doppler Shift	30 Hz
Thermal Noise Density	−174 dBm/Hz
Modulation/Coding Rate Settings	M-QAM
TCP Layer	TCP SACK
TCP Receive Window	65,535 Bytes
Distance From Base Station	2 km

. Based on these parameters, we used the model proposed by Zhao to calculate the bandwidth of the 5G network in the experiment [28]. We simulated background traffic using the ON/OFF model. The 4K video sequences “Crosswalk” and “Boat” [29] were encoded using the scalable H.264 reference software JSVM (version 9.19) [30]. Both video sequences consist of 300 frames, with a frame rate of 60 frames per second and a GoP size of eight images. The videos were set to play repeatedly. We extracted six layers employing a layer extraction technique for DASH video sources. The resolutions of the layers are as follows: the first layer is 2160P, followed by layers with resolutions of 1440 P, 1080 P, 720 P, 480 P, and 360 P. The “Crosswalk” video contains more dynamic content compared to the “Boat” video. Finally, a subjective quality assessment experiment was conducted. In this study, video data was re-encoded based on the transmission strategies derived from the simulation environment. The video was segmented and encoded into tiles according to the calculated strategies. These videos were then played for the participants to watch. A total of 21 participants were recruited for the experiment. Under the same conditions, they watched videos processed by different algorithms and rated them based on their subjective experiences. The scores ranged from 1 to 6, with higher values indicating a better subjective user experience.

Some important parameters need to be explained.

The $PSNR$ (peak signal-to-noise ratio) is a metric utilized to evaluate the quality of reconstructed images or videos after compression by comparing the original and compressed versions. It is calculated using the mean squared error (MSE) between the original and compressed images, with higher $PSNR$ values indicating better quality and lower levels of distortion or noise.

The $SSIM$ (structural similarity index measure) is a perceptual metric that assesses the visual similarity between two images, focusing on their structural information, luminance, and contrast. It provides a value between 0 and 1, where 1 indicates identical images and 0 signifies no similarity. This makes the SSIM a measure of image quality that aligns more closely with human visual perception compared to traditional error-based metrics like the $PSNR$.

The stability of the playback represents the stability of the video quality during playback, indicating the variation in video quality during playback. In the transmission of 360° videos, the quality level within the field of view is represented by the average quality level of all tiles. It is related to the switching of video quality at different times and the total number of switches. Its value ranges from 0 to 1, with a value closer to 1 indicating more stable video quality. The specific calculation method is shown in Equation (17). In this paper, the complete method for calculating video quality levels is as follows: the quality level of the current video playback is determined by calculating the average quality level of all tiles within the user’s field of view.

The NETBS analysis includes the following performance metrics:

• Efficiency: This metric is evaluated using objective quality indices like the $PSNR$ and $SSIM$. It measures video quality and transmission effectiveness by comparing the sampled image with the original resolution.
• Cumulative Distribution Function (CDF): This metric assesses the distribution of video quality levels that are transmitted.
• Playback Stability: This metric evaluates the stability of video playback, using a specific calculation method:

  $$PS={\displaystyle \frac{{\sum }_{e=1}^n\frac{{\sum }_{d=0}^{I-1}|j_{i-d}^{\prime }-j_{i-d-1}^{\prime }|\ast \omega \left(d\right)}{{\sum }_{d=1}^I{j}_{i-d}^{\prime }\ast \omega \left(d\right)}}{n}}.$$

  (17)
• QoE: The QoE defined by Equation (1) is normalized to enhance its accuracy:

  $$Qo{E}_{nor}=(QoE-Qo{E}_{min})/(Qo{E}_{man}-Qo{E}_{min}).$$

  (18)

The instability index $PS$ is computed by taking the weighted sum of all bitrate changes in the previous I segments and dividing this by the weighted sum of the maximum received bitrate levels over the same duration. Segment i has its highest received bitrate level, denoted as $j_i^{\prime }$.

To evaluate the performance of the proposed NETBS algorithm, we conducted simulation experiments and compared it with other algorithms. The first algorithm, referred to as Base-M, sends only the minimum-quality level video to ensure smooth video playback. The second comparison algorithm is NETBS-S, a variant of NETBS. This algorithm allocates the same bitrate to all video tiles within the same region instead of using the utility function to allocate bitrates to each tile individually. The third comparison algorithm is TBRA [31], which dynamically divides panoramic videos into different numbers of tiles. The fourth comparison algorithm is Option-2 [32], which differentiates a transition region in the user’s FoV movement direction to enhance viewpoint-switching sensitivity based on uniform tiling. The fifth comparison algorithm is Live360 [10], which optimizes 360° video transmission using dynamic programming. The sixth comparison algorithm is the full-frame technique (FFT), where the 360° video is encoded as a single block without tiling, and the basic DASH protocol is used for transmission.

5.2. Experimental Results

The $PSNR$ and $MSSIM$ values in this study were determined by comparing the received video with the original highest-quality video. If the resolution of the received video differed from that of the original, the received video was upsampled to match the original video’s resolution prior to calculating the $PSNR$ and $MSSIM$ values. Figure 4 illustrates a comparison of the $PSNR$ values for different algorithms. Specifically, Figure 4a presents the $PSNR$ values when users viewed the video “Boat”, and Figure 4b shows the $PSNR$ values for the video “Crosswalk”. The results demonstrate that the proposed method, NETBS, achieved the highest $PSNR$ values, whereas Base-M, which only transmits the lowest-quality video, achieved the lowest $PSNR$ values.

Figure 5 shows a comparison of the $MSSIM$ values for different algorithms. Figure 5a shows the $MSSIM$ values when users watched the video “Boat”, and Figure 5b shows the $MSSIM$ values when users watched the video “Crosswalk”. The $MSSIM$ values ranged from 0 to 1, with higher values indicating that the received video was closer to the original video. The results show that the proposed NETBS algorithm achieved the best $MSSIM$ values, indicating the highest video quality obtained by the proposed algorithm.

Figure 6 shows the CDF values of video quality levels received by clients over the entire duration of watching different videos. Figure 6a shows the CDF of video quality levels when users watched the video “Boat”, and Figure 6b shows the CDF of video quality levels when users watched the video “Crosswalk”. The figures demonstrate that the proposed NETBS method resulted in the highest proportion of high-quality videos received by clients. The Base-M method only transmitted the lowest-quality-level video, resulting in a 100% proportion of the lowest-quality-level video.

Figure 7 illustrates the QoE when users watched different videos. Figure 7a represents the QoE when users watched the video “Boat”, and Figure 7b represents the QoE when users watched the video “Crosswalk”. As an important parameter of the QoE, latency occurred during tile transmission when users switched perspectives, and the size of the tile also affected the transmission latency. From the figures, it can be seen that the performance of the proposed NETBS algorithm was superior to that of the other four algorithms.

Figure 8 shows the subjective video experience scores of users for different algorithms. As depicted in the figure, the proposed algorithm NETBS achieved the highest scores, indicating that the proposed non-uniform tiling strategy and bitrate selection algorithm provide a better media experience for users.

Figure 9 shows the bitrates of the 360° video segments in the transmission strategies calculated by different algorithms. This study considers only the bitrate of the tiles within the field of view, and the bitrate for the FFT algorithm is calculated based on the proportion of the view area to the total video area. From the figure, it can be observed that the BASE-M and FFT algorithms achieved lower bitrates because BASE-M only transmitted video data at the lowest quality level, and the FFT algorithm allocated more network resources to out-of-view videos. The bitrates of the other algorithms were similar, while the proposed NETBS algorithm achieved a higher QoE because it allocated more network resources to the tiles that had the greatest impact on users’ QoE.

Table 2. Playback stability.

PS	Boat	Crosswalk
NETBS	0.8952	0.9475
BASE-M	1	1
NETBS-S	0.8752	0.9203
TBRA	0.8031	0.8262
Option-2	0.8149	0.8302
Live360	0.8912	0.9455
FFT	0.9856	0.9912

shows the stability of playback when users watched different videos, where higher values indicate more stable video quality during transmission. The table indicates that Base-M had a consistent value of 1, as it only sent the lowest-quality-level video, resulting in no quality jitter. The proposed NETBS method outperformed the other methods.

Table 3. Execution times.

TIME/Microsecond	Boat	Crosswalk
NETBS	402.254	424.671
BASE-M	0.672	1.239
NETBS-S	356.241	402.576
TBRA	504.235	428.245
Option-2	321.245	421.487
Live360	312.125	264.588
FFT	1.875	3.698

shows the average execution time for a single run of each algorithm. In this study, Matlab was used as the simulator for the algorithms, and the computer’s CPU was the latest i9-14900. Each algorithm was run 500 times, and the average execution time was calculated. The results indicate that all algorithms met the real-time computation requirements for video. Additionally, the execution time can be further reduced if the algorithms employ parallel programming techniques.

6. Conclusions

This paper presents an innovative adaptive transmission strategy designed to optimize the transmission efficiency of 360° videos under bandwidth-constrained conditions. As 360° videos gain increasing popularity among users, their high bandwidth demands pose significant challenges. To address this issue, this paper first segments the 360° video into three distinct areas: the attention area, the out-of-field-of-view area, and the viewpoint-switching transition area. Based on this segmentation, a novel non-uniform coding transmission method is proposed, along with detailed mathematical modeling to identify the optimization problems inherent in the transmission process. To achieve real-time optimized transmission, this paper introduces a heuristic algorithm, termed the non-uniform coding for 360° panoramic video tiling and bitrate selection algorithm. This algorithm dynamically determines the optimal number and size of video tiles based on the user’s viewing environment, such as the distance between the user’s eyes and the screen, and allocates the appropriate bitrate to each tile. In this way, the proposed strategy effectively reduces unnecessary data transmission without compromising QoE, thereby enhancing overall transmission efficiency. To validate the effectiveness of the proposed algorithm, this paper constructs a simulation platform and conducts extensive experimental comparisons with various existing transmission algorithms. The experimental results demonstrate that the NETBS algorithm performs excellently across multiple key metrics, such as $PSNR$, $SSIM$, and QoE. These findings indicate that, compared to traditional uniform coding transmission strategies, the proposed algorithm not only delivers high-quality video transmission but also significantly enhances the user’s viewing experience. Furthermore, subjective user evaluation experiments reveal that videos transmitted using the NETBS algorithm exhibit better playback stability and image quality, aligning more closely with user expectations. Overall, the main contribution of this paper lies in the proposal of a non-uniform coding-based adaptive transmission strategy. By dynamically adjusting the number, size, and bitrate of video tiles, this strategy effectively improves the transmission efficiency of 360° videos under various network conditions while significantly enhancing user experience quality.