Block Recovery Rate-Based Unequal Error Protection for Three-Screen TV

Abstract

This paper describes a three-screen television system using a block recovery rate (BRR)-based unequal error protection (UEP). The proposed in-home wireless network uses scalable video coding (SVC) and UEP with forward error correction (FEC) for maximizing the quality of service (QoS) over error-prone wireless networks. For efficient FEC packet assignment, this paper proposes a simple and efficient performance metric, a BRR which is defined as a recovery rate of temporal and quality layer from FEC assignment by analyzing the hierarchical prediction structure including the current packet loss. It also explains the SVC layer switching scheme according to network conditions such as packet loss rate (PLR) and available bandwidth (ABW). In the experiments conducted, gains in video quality with the proposed UEP scheme vary from 1 to 3 dB in Y-peak signal-to-noise ratio (PSNR) with corresponding subjective video quality improvements.

1. Introduction

Recently, various multimedia services over wireless and wired networks using video streaming technologies have been emerging [1]. In particular, both continuous packet losses and rapid bandwidth changes in a time-varying wireless environment lead to serious video quality degradation. This paper proposes an unequal error protection (UEP) algorithm of scalable video to minimize video quality degradation caused by the packet losses over time-varying wireless channels.

Scalable video is an efficient encoder which can provide spatial, temporal, and quality scalabilities without any additional computations [2,3]. The characteristic of scalable video allows one single scalable bitstream to provide different video quality and resolution to different video receivers. Depending on channel characteristics or decoding capability of the receiver, it is possible to transmit, as well as decode, the partial or whole bitstreams of scalable video from the server, the intermediate node between the server and the receiver. Although the scalable embedded bitstream can easily adapt to various network environments, the packet losses in unstable channel conditions can cause serious video quality degradation in the compressed bitstream due to the strong spatio-temporal dependencies between scalable video layers. Since the structure of scalable video compression has strong spatial and temporal dependencies, the packet losses in hierarchical coding structures with scalability can substantially deteriorate the received video quality. Thus, considering both the hierarchical coding structure and the dependencies between quality layers, the effect on overall video quality degradation from packet losses in each layer is utilized as weighting in various UEP algorithms.

Figure 1 depicts the conceptual diagram of a three-screen TV using scalable video coding (SVC) with high-definition TV (HDTV), standard-definition TV (SDTV), and low-definition TV (LDTV) (such as mobile device). A traditional solution for source coding is to prepare multiple pre-encoded video files (chunks) according to the performance capability of each targeted device: such as screen resolution, computational power, and available network bandwidth. Due to the multiple redundancies, the video streaming server requires relatively large storage as well as a significant bandwidth adaptation problem in wireless error-prone networks.

There is another solution using transcoding with down sampling. By transcoding one high resolution and high bit rate video sequence for targeted screen resolution and bandwidth of TV clients, a video service provider (VSP) can reduce storage redundancies as well as adapt to bandwidth fluctuations. However, this solution requires too much computing power to support multiple TVs that have variable screen resolutions at the same time. Thus, it is very hard to apply in a real-time video streaming system.

This paper proposes a new video streaming solution for a three-screen TV that has low computational complexity and error robustness. First, it proposes the combined source and channel coding technologies: (1) application-level unequal error protection (UEP) technology for recovering packet losses over error-prone networks; (2) SVC-based adaptive layer-switching method that enables the server to select best scalable layers of SVC network abstraction layer (NAL) bitstreams according to the available bandwidth.

This paper is organized as follows: Section 2 gives an overview of the H.264 SVC, H.265 SHVC, and channel coding with Raptor forward error correction (FEC) codes. Section 3 proposes a new UEP algorithm to minimize the video quality degradation from packet losses, and an adaptive SVC layer switching method with Raptor FEC overhead adaptation. Section 4 presents the system design and implementation while also verifying the performance of proposed system. Finally, Section 5 provides conclusions.

2. Background

Before explaining the proposed UEP algorithm and layer-switching method, this paper describes in detail the source coding technology with SVC and channel coding technologies including FEC.

2.1. Scalable Video Coding: H.264 SVC and H.265 Scalable HEVC (SHVC)

The H.264 SVC is an H.264 advanced video coding (AVC) scalable extension that combines spatial, temporal, and quality scalabilities simultaneously [4]. Because of the scalable feature, SVC supports multiple screen resolutions, frame rates, and video qualities within a single bitstream file that consists of multiple layers, as shown in Figure 2.

In the layered structure of the SVC encoder, the original high quality video input is spatially down-sampled for lower layers, and each layer encodes the input video using an inter-layer prediction. Due to the layering architecture, SVC has the following advantages: (i) it can reduce server-side disk storage and overall network bandwidth without the use of the kind of transcoding that has high computational complexity; (ii) because the layers of SVC have priorities (the lower layer is more important than the higher layer), the UEP method can be applied to each layer packet; (iii) SVC can support the diverse screen resolutions of user devices as well as network bandwidth (BWs.)

The scalable high efficiency video coding (SHVC) standard of joint collaborative team on video coding (JCT-VC) is as shown in Figure 3. It has a low computational complexity for enhancement layers (ELs) by adding the reconstructed and up-sampled base layer (BL) picture to the reference picture lists (RPLs) in EL. Additionally, the SHVC uses multiple-loop decoding to make a decoder chipset simple while the SVC uses single-loop decoding. SHVC also has a standard scalability with supporting AVC standard with BL and high efficiency video coding (HEVC) standard with EL.

2.2. Reed-Solomon (RS) and Raptor FEC Codes

Reed-Solomon (RS) based unequal loss protection algorithm is performed [5,6,7,8,9] in the FEC assigner. Here, we denote the layered unit data (block) in a temporal layer t of a quality layer q as B(t, q). When the number of FEC packets and video data are represented by F(t, q), the reconstruction mechanism of the RS code is composed of continuous B(t, q) video packets and F(t, q) parity packets. If the minimum number of successfully received packets is more than B(t, q), then the original source packets can be reconstructed. The FEC assignment scheme is proposed to maximize the block recovery rate (BRR). A similar local hill-climbing search method is utilized for assignment of FEC packets for each block with low complexity.

The Raptor FEC, developed by Amin Shokrollahi in 2001, is also called a rateless code because the amount of encoded data is not fixed in advance [10]. It is used in the multimedia broadcast and multicast services (MBMS), digital video broadcasting-handheld (DVB-H), and reliable multicast transport (RMT) working group [11]. The encoding processes of the Raptor codes are as shown in Figure 4. The processes consist of two steps: (i) pre-coding, which encodes the input symbols using a traditional erasure correcting code; and (ii) Luby transform (LT) coding [12], which creates the needed number of encoding symbols. The LT code makes the Raptor codes have low computational complexity.

In the (n, k) systematic Raptor codes, k source symbols are pre-coded to k intermediate symbols, and then pre-coded to k + s pre-coding symbols, where s is the number of redundant symbols of pre-code. Finally, pre-coding symbols are encoded into n encoding symbols using LT-code.

2.3. UEP Research

In Figure 5, once the encoder has decided the priority of a picture, the UEP and/or transmission scheduler can use the priority in both robust streaming and quality of service (QoS) handling. It applies different FEC overheads to pictures according to picture priority (PPn).

There have been several research studies on FEC algorithms regarding the error propagation effect on video quality degradation in both base layer and enhancement layers [5,6,7,8,9,13,14]. In a study by Nafaa et al. [5], a FEC assignment algorithm using the property of continuous packet losses is presented. Various UEP algorithms based on rate-distortion model have been presented for reducing video quality degradation from packet losses [9,13]. Adaptive FEC assignment algorithms have been developed for video streaming over best-effort networks [13,14,15]. To adapt time-varying channel environments, an optimal packet scheduling for rate-distortion was proposed [14,15,16,17]. Also, a rate-distortion optimized scheduling algorithm was presented to adapt the streaming video over time-varying channels for minimizing the total distortion under the transmission rate constraint [18,19]. There have been research studies on transmitting SVC video sequence safely over multicast channel or MIMO wireless systems using Raptor codes [20,21,22], research studies on the priority-based SVC or SHVC video streaming over multi-channel were introduced in [23,24], and research studies on the UEP method with Raptor codes with SVC or HEVC temporal layer priorities have been presented in [25,26,27,28,29].

3. BRR-Based UEP Algorithm and Scalable Video Layer-Switching

3.1. Performance Metric

This section introduces a new performance metric to predict the video quality from packet losses. In general, the packet loss recovery procedure is as follows. Assuming that video packets of B(t, q) are encoded as N(t, q) packets on the server side where N(t, q) = B(t, q) + F(t, q), where F(t, q) represents FEC packets assigned in B(t, q). If the minimum number of successfully received packets is more than B(t, q), the original source packets can be reconstructed. The block recovery rate (BRR) of B(t, q) is calculated as

$$R_b\left(t,q\right)={\displaystyle \sum }_{i=B\left(t,q\right)}^{N\left(t,q\right)}{C}_i^{N\left(t,q\right)}·{\left(1-p\right)}^i·p^{N\left(t,q\right)-i}$$

(1)

where p is the packet loss rate and $C_i^{N\left(t,q\right)}$ represents the combinations of the number of packets i received successfully within N(t, q). To reflect error propagation effects of scalable video in (1), the prediction directions for each block are considered. Figure 6 shows the hierarchical prediction structure of scalable video.

The block information of temporal layer t is referenced by that of the lower temporal layer (t − 1). Also, the block information of quality layer q in temporal layer t is affected by the information from the lower quality block (q − 1) in the same temporal layer (t). For example, R_b(1, 1) is affected by both R_b(0, 1) in the same quality layer of the lower temporal layer and R_b(1, 0) in the lower quality layer of the current temporal layer. Based on the hierarchical prediction structure between the temporal and quality layer, (1) can be updated as

$${\widehat{R}}_b(t,q)=\{\begin{array}{ll}{R}_b(t,q),\hfill & ift=0,q=0\hfill \\ R_b(t,q-1)\cdot R_b(t,q),\hfill & ift=0,q\ne 0\hfill \\ R_b(t-1,q)\cdot R_b(t,q),\hfill & ift\ne 0,q=0\\ R_b(t-1,q)\cdot R_b(t,q-1)\cdot R_b(t,q),\hfill & otherwise\hfill \end{array}$$

(2)

In (2), ${\widehat{R}}_b(t,q)$ is computed by the temporal and quality layer index in scalable prediction structure. If t = 0 and q = 0, since there are no reference blocks, ${\widehat{R}}_b(t,q)$ is the same as $R_b(t,q)$. If t or q is not 0, it is influenced by the lower scalable block with non-zero temporal or quality layer index. Finally, when both t and q are not zero, ${\widehat{R}}_b(t,q)$ is affected by the lower temporal and quality blocks. Based on (2), the average block recovery rate (R_avg) in a group of pictures (GOP) is calculated as

$$R_{avg}=\frac{{\displaystyle {\sum }_{t=0}^{T-1}{\displaystyle {\sum }_{q=0}^{Q-1}{\widehat{R}}_b(t,q)}}}{T\cdot Q}$$

(3)

where T and Q represent the maximum number of temporal and quality layers, respectively.

3.2. Proposed FEC Assignment Algorithm

Based on the performance metric model (R_avg) in (3), we propose a FEC assignment algorithm which allocates the amount of FEC packets to each block in order to maximize the average block recovery rate (R_avg). Similar to the simple local hill climbing method, we find the B(t, q) that can achieve the largest value of R_avg for a given FEC assignment.

Based on (3), we define R_avg(F(t, q)) as the average block recovery rate when FEC packets are assigned into B(t, q). If the amount of the increased average block recovery rate by the allocated number of FEC packet in B(t, q) is defined as $\delta (t,q)$, it is calculated as

$$\delta (t,q)=R_{avg}(F(t,q)+1)-R_{avg}(F(t,q))$$

(4)

Using (4), the B(t, q) which can achieve the largest increased value of average block recovery rate is selected as

$$B(\widehat{t},\widehat{q})=\underset{t,q}{\arg \max }\delta (t,q)$$

(5)

The number of FEC packets of the selected B($\widehat{t},\widehat{q}$) is updated by 1. The accumulated value of F(t, q) is represented as F_sum(t, q) and is updated by 1. If the overall number of assigned FEC packets (F_sum(t, q)) exceeds the limited channel resource (T_pkt), the proposed algorithm is terminated.

3.3. Proposed Adaptive Scalable Video Layer-Switching

The proposed SVC layer-switching method selects the best layer(s) for streaming automatically. The switching is the easiest natural bandwidth adaptation method in the middle of streaming and has very low computational complexity without any transcoding that requires very high computing power. Thus, these SVC features enable the following working scenarios; a VSP server sends whole SVC layers to an individual home gateway, and the switching is performed in the gateway according to the link-quality feedback from the client. For example, if a user stays in an area that has a good channel quality, the gateway switches its service to a higher level with multiple ELs. In this system, the channel condition is measured using signal strength indicator (RSSI)-based feedback, packet loss ratio (PLR), and available bandwidth.

In the implemented system, network abstraction layer (NAL) packets are analyzed and extracted the real-time layer-switching, as shown in Figure 7. In addition, the instant decoder refresh (IDR) information of SVC is the starting point of the proposed layer-switching method.

4. Experimental Results and Discussion

4.1. H.264 SVC Unequal Error Protection Using Reed-Solomon(RS) Codes without Layer Switching

This section reveals simulation results of the proposed FEC algorithm for the performance analysis by using version 9 of the Joint Scalable Video Model [30]. We used common intermediate format (CIF) video sequences ‘Foreman’ and ‘Mobile’ sequences with a frame rate of 30 fps. The number of frames was 81 and the size of the group of pictures (GOP) was 16 frames.

There were five temporal layers with one quality base layer and two quality enhancement layers. Quantization parameters (QPs) were set to be 40, 30, and 25. A two-state Markov channel model described in [31] was used for modeling the packet losses with the average burst length of Lb and the average packet loss rate of Pb. The value of Lb is set as 2. Using different packet loss rates (5%~15%) the proposed unequal error protection method using block recovery rate in (3) (BRR-UEP) was compared with other unequal error protection schemes.

• BRR-UEP: The proposed block recovery rate-based UEP (BRR-UEP) performs the unequal FEC assignment by calculating the block recovery rate for each layer through the hierarchical prediction structure analysis in quality and temporal layers including the packet loss rate.
• RD-UEP: For allocating unequal amount of FEC protection to different video packets, the rate-distortion based UEP (RD-UEP) method utilizes the simple rate-distortion model without considering the recovery rate for each block according to the current packet loss rate [15,17].
• Equal EP: The equal error protection (EP) allocates an equal amount of FEC packets for all packets in different layers without considering the performance metric.

Figure 8 shows the variations of R_avg in (3) according to the total number of FEC packets (T_pkt) in variable channel environments. As the packet loss rates are increased, the block recovery rate is gradually decreased. Furthermore, it is observed that as the number of FEC packets is increased, the value of R_avg is increased.

Figure 9 shows the distribution of F(t, q) for each temporal (t) and quality (q) layer at different packet loss rates Pb = 5% and Pb = 15%, respectively. Redundant FEC packets are intensively assigned to the lower block index. This is because based on a hierarchical prediction structure in the scalable video, packet losses in lower blocks have stronger error propagation effects than those in the higher blocks.

Figure 10 shows the peak signal-to-noise ratio (PSNR) comparisons of BRR-UEP, RD-UEP, and Equal EP. It is observed that Equal EP shows the lowest PSNR values. The proposed BRR-UEP scheme results in higher PSNR values compared to the RD-UEP schemes by 2 dB for the ‘Foreman’ case and 1 dB for the ‘Mobile’ case, respectively. Those average PSNR gains are from overall frames. Compared to the RD-UEP method, it is observed that the BRR-UEP scheme could provide better video quality by efficiently limiting the error propagation through the BRR-based FEC assignment. In future research, the proposed method will be compared to other latest research studies, including the work by Song and Chen [22].

The proposed UEP method was validated using NS-3 network simulator [32]. The network topology was implemented as shown in Figure 11. The video server produces layer based scalable video packets with temporal and quality scalabilities and transmits real-time transport protocol/user datagram protocol/internet protocol (RTP/UDP/IP) packets to the corresponding receivers with various decoding capabilities through a wireless access point (AP). Exponential traffic using UDP packets is additionally added to simulate a congested environment.

4.2. H.265 SHVC Unequal Error Protection Using Raptor Codes without Layer Switching

This subsection explains the experimental results of H.265 SHVC-based UEP without layer switching. For verifying the proposed methods, ultra-high definition (UHD) test sequence ‘PeopleOnStreet’ which is defined in the common test condition (CTC) of JCT-VC standard work was used. The resolution of base layer and enhancement layer were 1920 by 1080 and 3840 by 2160, respectively. QP values were also applied for those base layer and enhancement layers as 34 and 36, and three picture coding structures (random access (RA), all intra (AI), and low-delay B (LDB)) were tested. For the SHVC, SHM version 4.1 encoder was used, and the ratio of spatial scalability was 2:1 (2160 p for the enhancement layer and 1080 p for the base layer) [33,34,35]. The experiment was conducted without the simulation codes but the fully implemented UEP system. The experimental setup and results are as shown in

Table 1. Experimental environments. QPs, quantization parameters; RA, random access; AI, all intra; LDB, low-delay B.

Encoder and Decoder	SHM 4.1. (JCT-VC SHVC Reference SW), OpenHEVCDecoder (Opensource SW), and Implemented Raptor En/Decoder
Test sequence name	PeopleOnStreet
Resolutions for base layer and enhancement layer	1920 by 1080, 3840 by 2160
QPs for base layer and enhancement layer	34, 36 (base layer QP + 2)
Picture coding structures	RA, AI, LDB

and

Table 2. Y-PSNR results with Raptor overhead 5% and 10%.

Coding Structure	Methods	Overhead 5% with Base PLR 12%	Overhead 10% with Base PLR 17%
RA	Uniform	17.80	15.16
	Proposed	18.28	23.72
AI	Uniform	20.03	17.44
	Proposed	21.28	29.48
LDB	Uniform	15.44	14.64
	Proposed	16.20	22.60

. In the result, the base layer was protected more and the enhancement layer was sacrificed for the base layer in given Raptor overheads (e.g., 5% and 10%) with base PLRs (e.g., 12% and 17%). The Table 2 shows the proposed UEP method provided better objective video qualities with Y-PSNR values.

4.3. H.264 SVC Unequal Error Protection Using Raptor Codes with Layer Switching

This subsection explains the experimental results of H.264 SVC-based UEP with layer switching. The implemented codes were based on ‘OpenSVCDecoder’, and the Raptor FEC codes with UEP method were added on the decoder. The implementation included RSSI measurement module and layer switching module with the IDR picture in it. First, for verifying the benefit of the scalable video layer-switching method, Figure 12 shows the reactions of the proposed method according to the measured link quality (RSSI) in real moving experimentation.

Figure 12a–c shows the measured network conditions such as RSSI, ABW, and PLR during user movements. As the mobile TV client was getting farther from the wireless AP, the value of RSSI was decreasing and it increased again when the mobile TV was coming towards the AP. From 50 to 90 s, the ABW was decreased and PLR was increased. Thus, the implemented server decreased SVC layers that have to be transmitted from three layers (BL + two ELs) to BL only by the 110 s point and increased the layers again based on the network quality feedback from the mobile TV client. The Raptor overhead adaptation was applied from 20% to 52% during the period. With these adaptive features of the proposed system, most of the packet losses were recovered, as shown in Figure 12f. The layer switching occurred on IDR pictures only when there were severe network condition changes, and all decoded pictures were upscaled for full resolution. Thus, the switching did not make a noticeable inconvenience.

5. Conclusions

This paper proposed an unequal error protection (UEP) algorithm based on the FEC mechanism in order to improve scalable video streaming quality. It first proposed a low complexity performance metric, block recovery rate (BRR) using a hierarchical coding structure of scalable video and packet loss rate. Then, it developed the FEC assignment algorithm to maximize the value of BRR. The simulation results showed that the proposed algorithm outperforms previous UEP algorithms by 1 to 2 dB for variable channel condition. In addition, in the implemented three-screen TV system, the proposed scalable video layer-switching method selects appropriate layers to be sent according to ABW by RSSI-based feedback.