Reducing System Load of Effective Video Using a Network Model

Abstract

Recently, as non-face-to-face work has become more common, the development of streaming services has become a significant issue. As these services are applied in increasingly diverse fields, various problems are caused by the overloading of systems when users try to transmit high-quality images. In this paper, SRGAN (Super Resolution Generative Adversarial Network) and DAIN (Depth-Aware Video Frame Interpolation) deep learning were used to reduce the overload that occurs during real-time video transmission. Images were divided into a FoV (Field of view) region and a non-FoV (Non-Field of view) region, and SRGAN was applied to the former, DAIN to the latter. Through this process, image quality was improved and system load was reduced.

1. Introduction

Recently, various fields and services, including cloud computing, have been growing at a greater rate than expected, and related systems are being developed. The live streaming industry, especially, is growing faster than ever. High-performance portable devices are popular for watching video content anytime, anywhere through a network, such as a Wi-Fi or 5g network [1]. However, when using real-time streaming, the simultaneous use of platforms, networks, and streaming formats can cause various problems for users and viewers. For example, connection delays due to network congestion or limits to server capacity due to excessive user access may result in great inconvenience. This paper proposes a new method, focusing on reducing the amounts of data transmitted and improving video quality as a solution to various problems with real-time streaming services [2].

In this paper, we propose the following method, considering that the solution to various problems in real-time streaming services is to maintain and improve video quality and reduce the amount of data transmitted. First, the problem is solved by using the characteristics of the non-FoV region existing in the image apart from the viewing region. During streaming or learning through a network, the video transmitted to the user shows all areas of the image; therefore, more time is taken up and unnecessary data consumption occurs in transmission, which causes problems such as high traffic. To avoid this, we used screen division to maintain image quality without compromising QoE (Quality of Experience). Through this, we tried to reduce the overall traffic of the network by fixing the quality of FoV and lowering the quality of the non-FoV area. Second, system load was reduced by varying the learning method according to the viewing area. The reason for dividing into an FoV region and a non-FoV region is to apply deep learning to each image. Previously, a method was used to reduce system load by only guaranteeing the quality of the FoV region through segmentation and treating the non-FoV region as background. However, this only reduced the load on the system and could not maintain the quality for the entire area. When the change in viewpoint was applied, the viewer had no choice but to check that the quality of the image changed slowly. We therefore applied this technique to maintain the minimum frame and quality even outside the viewing area. The viewing area transmits high-quality images and the non-FoV area uses low-quality images to which DAIN (Depth-Aware Video Frame In-interpolation) is applied, thereby reducing network load and transmission. Third, the user does not continuously transmit the area that the viewer first saw, but selectively transmits the image as FoV information is received. This allows for real-time transmission while maintaining image quality, since the viewing area can change continuously. Finally, SRGAN (Super Resolution Generative Advisory Networks) is applied to the FoV area to maintain and improve image quality, and DAIN and SST are applied to the non-FoV area to adjust the image. The image that is the result this process is designed to be a selective output determined by the viewpoint of the viewer.

2. Related Work

2.1. SSAS (Sptial Segmented Adptive Streaming)

SSAS, which uses an adaptive streaming method to reduce network traffic, is set up as an FoV that transmits the user’s viewing area to the center. The FoV area viewed by the user transmits the original image quality, the original image quality is transmitted to the center of the FoV, and the lower the quality of the FoV video, the less network traffic is transmitted [3].

Figure 1 shows the user screen treated according to the SSAS method. The brightest part means highest quality; the darker the color, the lower the quality of the video.

2.2. 360° Video

In general, video is recorded in a single direction or in the direction of a viewpoint, so only the recorded video or the played view can be outputted. However, in the case of 360° or VR∙AR, all areas may be affected or only specific areas depending on the recording device used. The above method can produce contents using multiple cameras. It can also create and express camera objects by using a program. Typical programs are Unity, Unreal Engine, and 3D Capture. Otherwise, 360° videos can be created, as shown in Figure 2.

This method can be easily manufactured using a camera equipped with a 180° fisheye lens. This is a feature of the 360° camera [4]. Stitching that uses multiple cameras to shoot an object in multiple directions and then turns each image into a single image represents the final image in a spherical shape, so this image must be converted into a standard 2D image. In order to represent a spherical image as a 2D image, a process called warping is performed, and distortion occurs in the process. The 2D images derived from 360° images by means of this process are displayed as inverse warping [5,6].

Figure 3 shows a 360° image (a) and an inverse warping image (b). It is difficult to see the entire 360° image on one screen. Warping of the 3D image is performed in order to produce a 2D image which is easier to see. It is very difficult to remap 360° images to a 3D form in the way described above. However, this can be easily done with a program. Video games that use 360° video require a high FPS (Frames Per Second). Therefore, we propose a method that can adjust FPS using the minimum amount of video.

2.3. DAIN (Depth Aware Video Frame Interpolation)

The DAIN network is a method that can have many different applications, such as slow-motion creation, new view compositing, and frame recovery [7,8]. The high frame rate is a very attractive form of video streaming for viewers and users. However, problems such as network delay or motion blur which can be caused by high frame rates can affect system load [9]. Even if you try to use a network, such as a CNN, to solve the problem, the effect is not significant. Therefore, high-quality frame problems for video are still difficult to solve with basic networks. There are several approaches to handling large motions, such as fine-tuning strategies or adjusting architecture by means of flow estimation.

Among these approaches, PWC-Net can be used to estimate accurate flow [10]. Another approach is to estimate the mask through the pixel method. There is also a learning method that adapts the space changes and neighboring pixels that have been recently discussed. This method produces effective results by extracting and synthesizing specific frames through a pre-trained classification network. However, these methods may not be efficient as they are highly dependent on the model data that is required for training. Using the DAIN network, an estimated image is synthesized by calculating the intermediate frame between two existing images. This network is a highly effective means of achieving optical flow estimation, depth estimation, content extraction, amongst other things. In particular, it is excellent for estimating flow and depth in both directions for two input frames. It contributes to the intermediate flow by adopting a projection layer through the input frame. Since multiple flow vectors can meet at the same location, the contribution of each flow vector is calculated based on depth value. The proposed approach creates a flow with clearer motion boundaries due to the influence of depth, not just flow. Figure 4 shows a model of DAIN.

The details of the depth perception proposed are shown in the following figure.

Figure 5 is a representation of flow projection. Flow projection produces an average flow vector (green arrow). For a depth-aware flow projection layer, the projected flow will point to pixels with a smaller depth value (red arrow). On the other hand, the flow vector does not pass through these pixels, and so it can be that there are voids in the intermediate flow. The empty space created here can be compensated by calculating the average flow between neighbors. With the following formula, we can see the content of the depth.

The projected flow F_t→0 is defined by:

$${\mathrm{F}}_{\mathrm{t}\to 0}\left(\mathrm{x}\right)=-\mathrm{t}\ast \frac{{{\displaystyle \sum }}_{\mathrm{y}\in \mathrm{S}}{\mathsf{\omega }}_0\left(\mathrm{y}\right){\ast \mathrm{F}}_{0\to 1}\left(\mathrm{y}\right)}{2}$$

(1)

The flow F_0→1 (y) passes through x at time t. One can approximate F_t→0(x) by −t F_0→1(y). Similarly, we approximate F_t→1(x) by −(1−t) F_1→0(y).

Where the weight w₀ is the reciprocal of depth:

$${\mathsf{\omega }}_0\left(\mathrm{y}\right)=\frac{1}{{\mathrm{D}}_0\left(\mathrm{y}\right)}$$

(2)

D₀ is the depth map of the input frame. The flow in the empty space position is computed by averaging the available flows from its neighbors:

$${\mathrm{F}}_{\mathrm{t}\to 0}\left(\mathrm{x}\right)=\frac{1}{\left|\mathrm{N}\left(\mathrm{x}\right)\right|}{\displaystyle \sum }_{{\mathrm{x}}^{\prime }\in \mathrm{N}\left(\mathrm{x}\right)}{\mathrm{F}}_{\mathrm{t}\to 0}\left({\mathrm{x}}^{\prime }\right)$$

(3)

where N (x) = {x’: |S(x’)| > 0} is the 4-neighbors of x. From (1) and (3), we obtain the dense intermediate flow fields F_t→0 and F_t→1 for warping the input frames. If the above method is used, we can get many complements by learning the hierarchical features that extract information from most networks and contents [11].

2.4. SRGAN (Super Resolution Generative Advisory Networks)

When a low-resolution image is input, SRGAN regenerates it into a high-resolution image to further enhance the textural detail. Figure 6, below, shows the process of SRGAN [12].

The network process applies input data and analyzes inputted images pixel by pixel through convolution in order to extract the features of highly related pixels and apply them through the network. BN (Batch Normalization) enables learning in artificial neural networks by means of backpropagation. As the depth of the layer included in the artificial neural network increases, the gradient value transmitted by back propagation converges to 0 or diverges to achieve gradient vanishing or gradient exploding. Therefore, to improve overall learning speed, we used BN, a technique for reducing the frequency of occurrence of the problem by regularizing the values inputted to each layer, for SRGAN. Using the ReLU function, this was supplemented to prevent the loss of the gradient of the Deep Neural Network. By means of the method described, SRGAN learns one network with two networks connected. In the generator network, learning is carried out by comparing and analyzing the original image and the generated image. Additionally, this method removes the sigmoid from the network structure and calculates the loss function. After that, the final loss value is calculated using the calculated adversarial loss value and the content loss value. “Dense layer” refers to a layer that is combined with all neurons of the previous layer in the artificial neural network. Leaky ReLU is a modification of the ReLU function, and is a function that takes a negative number and reduces the value by 1/10. The following equation is the formula for the constructor (2) and the identifier (3) of SRGAN.

$${\hat{\mathsf{\theta }}}_{\mathrm{G}}=\underset{{\mathsf{\theta }}_{\mathrm{G}}}{\mathrm{argmin}}\frac{1}{\mathrm{N}}{\mathrm{l}}^{\mathrm{SR}}\left({\mathrm{G}}_{{\mathsf{\theta }}_{\mathrm{G}}}\left({\mathrm{I}}_{\mathrm{n}}^{\mathrm{LR}}\right),{\mathrm{I}}_{\mathrm{n}}^{\mathrm{HR}}\right)$$

(4)

$$\underset{{\mathsf{\theta }}_{\mathrm{G}}}{\min }\underset{{\mathsf{\theta }}_{\mathrm{D}}}{\max }{\mathrm{E}}_{{\mathrm{I}}^{\mathrm{HR}}~{\mathrm{P}}_{\mathrm{train}}\left({\mathrm{I}}^{\mathrm{HR}}\right)[\log {\mathrm{D}}_{{\mathsf{\theta }}_{\mathrm{D}}}\left({\mathrm{I}}^{\mathrm{HR}}\right)]}+{\mathrm{E}}_{{\mathrm{I}}^{\mathrm{LR}}~{\mathrm{P}}_{\mathrm{G}}\left({\mathrm{I}}^{\mathrm{LR}}\right)[\log (1-{\mathrm{D}}_{{\mathsf{\theta }}_{\mathrm{D}}}({\mathrm{G}}_{{\mathsf{\theta }}_{\mathrm{G}}}({\mathrm{I}}^{\mathrm{LR}})))]}$$

(5)

• G = Constructor that creates a super-resolution image result when loss resolution is inputted;
• D = A delimiter to distinguish whether the generated image is an original or a generated image;
• ${\mathsf{\theta }}_{\mathrm{G}}$ = G parameter;
• ${\mathsf{\theta }}_{\mathrm{D}}$ = D parameter;
• ${\mathrm{l}}^{\mathrm{S}\mathrm{R}}$= The loss function value determined by comparing it with the original image (HR).

2.5. SRCNN (Image Super Resolution Using Deep Convolutional Networks)

SRCNN is a technology for converting a low resolution (LR) image into a high resolution (HR) image or for improving the quality of an image. It is mainly applied in supervised learning and provides a low-resolution and ground-truth (GT) image as a model. It is a model that learns to reduce the difference between HR and GT results [13]. The overall structure is as follows.

Figure 7 show that SRCNN receives a low-resolution image as input, goes through a series of processes, and outputs a high-resolution image. It can be expressed by the following process and formula.

(1)

Bicubic interpolation

By using bicubic interpolation—a traditional upscaling method—the size of LR is the same as that of HR. Bicubic interpolation is a method of estimating the value of a new pixel as the pixel is increased by using the values of 16 adjacent pixels and weights according to distance.

(2)

Patch extraction and representation

The image is divided by the specified patch size and the feature map of the corresponding patch is extracted through a convolution operation. In this case, the patch size is arbitrarily designated. Afterwards, only meaningful values of the feature map are left through the activation function (ReLU).

Y stands for input low-resolution image, F(Y) stands for reconstructed output high-resolution image, W_i stands for filter, B_i stands for bias, and * stands for the convolution operation and ReLU(max(0, x)) function.

$${\mathrm{F}}_1\left(\mathrm{Y}\right)=\max \left(0,{\mathrm{W}}_1\ast \mathrm{Y}+{\mathrm{B}}_1\right)$$

(6)

(3)

Non-linear mapping

Create another feature map by applying the convolution operation to the feature map delivered in (1).

Through this process, a nonlinear relationship between interpolated LR (ILR) and HR is estimated.

That is, the feature map created through this process becomes a value representing the feature of the corresponding patch area of the HR image.

$${\mathrm{F}}_2\left(\mathrm{Y}\right)=\max (0,{\mathrm{W}}_2{\ast \mathrm{F}}_1\left(\mathrm{Y}\right)+{\mathrm{B}}_2)$$

(7)

(4)

Reconstruction

In order to combine the feature maps created in this way into one image, there will be overlapping areas. Finally, a feature value corresponding to one pixel is created by applying a convolution operation to them. This is applied to all HR pixels to create a new HR image.

$$\mathrm{F}\left(\mathrm{Y}\right)={\mathrm{W}}_3{\ast \mathrm{F}}_2\left(\mathrm{Y}\right)+{\mathrm{B}}_3$$

(8)

In summary, this is a process of dividing the LR in which bicubic interpolation is applied in patches to obtain an estimated value through a convolution operation for each of the divided features, and where the combined HR is reconstructed by applying the convolution operation again. This structure is represented in Figure 8.

Mean Squared Error (MSE) is used as the loss function for learning the mapping function F.

2.6. RTSP (Real Time Streaming Protocol)

RTSP is a network protocol designed for communication systems to control streaming media servers. Figure 9 shows a media server and client architecture using RTSP. Such a network protocol is used for a system that streams media in real time to a client with video or audio functions and is used to remotely control a media server. Commands are similar to VCR operation methods, such as PLAY and PAUSE, and access the server based on time information. It can operate on a variety of transport layer protocols, including TCP and UDP, and forwards packets after establishing an RTP/RTCP channel using a trusted connected TCP for sending control messages.

3. System Model or Proposal Method

3.1. Front on the Backward Frame

Based on the PIP (Picture in Picture) method, we propose the FBF (Front on the Backward Frame) method to reduce system load without deteriorating QoE. First, a low-resolution image is transmitted in the background. After that, the FoV area is transmitted in high resolution to maintain the quality. In general, 360° video streams video with different resolutions according to the influence of network traffic, while the proposed method streams video to different areas depending on the FoV. Figure 10 shows the FBF algorithm configuration diagram.

The server serves a low-quality background image to the client and the client receiving the FoV area outputs the high-definition area created by the transcoder using the conversion event. It is difficult to handle multiple clients at the same time, the reason being that a problem occurs in the conversion delay. Therefore, we solved the problem by pre-processing before streaming. Figure 11 shows the preprocessing of the FBF algorithm.

Streaming using the FBF algorithm requires two preprocessing steps. The first step is to create a video streaming background. The background is outputted as a low-quality video. The second step requires region-division encoding because we have to stream FoV regions individually. The original video is segmented according to the number of target segmentation segments. The server receives the system information and then selects the video, so it creates a file containing the time information. Videos of different qualities generated by preprocessing are stored on the server and they are streamed in ways that are chosen by the client.

Figure 12 shows the process of an FoV switching event as experienced by the client. The FoV transition event depends on the user’s point of view. When viewing a 360° video in a PC environment, an event occurs through the mouse sensor, but in the case of a mobile device, it is operated through a gyroscope sensor. The FoV area changed by the environment according to the user’s FoV is downloaded in the next buffering and is displayed as an image overlaid on the background image. In order to produce a video according to the proposed method, a transcoding process is needed.

The transcoder converts the original image into three types of quality: low, medium, and high. The next step is to divide the number of divisions for the FoV area and store them according to the key frame. Thus, the user receives the background video while being presented with an image of the right quality for their viewing area. This method allows the video to be streamed to the viewing area including the FBF algorithm with real-time characteristics.

3.2. SST (Spatial Segmented Transform)

The SST method involves a network that performs conversion after setting an area for a designated object or other object. Since it is a format that learns not only the salient objects in the image but also background features through deep learning, the information about the areas other than the designated area is converted. This can compensate for the problems of underfitting and overfitting [14]. Finally, the undesignated part is converted around the FoV area. The SST method was produced based on spatial transformer networks technology. The proposed method divides the image by region and the image quality is transformed through deep learning to compensate for loss of image quality. When the image is artificially divided into individual areas and passed through a network, quality conversion occurs, so SST is used to prevent data loss in whatever conditions obtain without changing the shape of the image. Figure 13 shows the basic structure of SST.

Figure 14 shows how the network handles the rest of the original image except for the RoI (Region of interest). In order to determine the RoI at an early stage, a method of using a designated area or target is employed, though information can also be extracted using a network such as CAM Shift. In the last step, non-RoI is extracted and converted into a new frame after the conversion process. This represents the process of outputting as video. The method is used to reduce system load.

3.3. Segmentation

The FBF algorithm is a method of converting a high-quality area according to a user’s area. At the initial stage, quality was lowered and the frame was set to 30 frames to reduce the transition delay of the high-quality area. The segmentation process was designed according to the generic HLS protocol. Also, unlike the basic structure, classification according to quality was added. A transcoder module and a segment module were implemented in the server to reduce system load on mobile phones and PCs. The information for each area of the content is stored separately, and a packet to be played back can be selected. We designed a protocol to classify chunk files in order to properly select them according to FoV. The m3u8 in each layer stores information about whether or not the program was viewed and the path to the file. The next layer stores the route of the TS segment divided by time and conforms to the HLS standard. The proposed server uses a split transmission method, so no RTSP or RTMP server is required. Preprocessing and TS-packetized modules were installed on the Apache web server. The preprocessing module re-encodes the content uploaded to the server by area and quality. The TS-packetized module divides each video into area and quality and reconstructs them into MPEG-2 TS packets. This will create an m3u8 file listing the HLS Segmenter in shown Figure 15.

3.4. Investigation of Loss Comparison

We decided to use mean opinion score (MOS) testing to determine the super-resolution network to be applied to FoV.

Figure 16 shows the ability of SRGAN according to MOS testing. The MOS test is the evaluation of 12 versions of each image on the BSD100. The networks used for evaluation were nearset, bicubic, SRCNN, SelfExSR [15], DRCN [16], ESPCN [17], SRResNet, and SRGAN. Based on this, SRGAN was selected, and SRCNN, which is used in many environments, was applied to the experiment to compare the proposed architecture.

First of all, SRCNN has the advantage of having a very simple structure and a high level of accuracy. Its speed is fast, even if only the CPU is used. In addition, it can make the model bigger and deeper, and better performance can be expected as the data set is diversified. It also has the advantage of processing three RGB channels and obtaining high-resolution images at the same time, so it can be used in many places. SRGAN’s strength, meanwhile, is that it handles texture well and has a fast speed. By actively utilizing ResNet and skip-connection in the generator network, it shows good performance in saving time and memory. The following

Table 1. Quantitative measures with Ch. 1–3.

	SRCNN		SRGAN
	PSNR	SSIM	PSNR	SSIM
Ch. 1	43.25	0.9869	85.16	0.99
Ch. 2	43.08	0.9829	84.83	0.98
Ch. 3	58.11	0.9767	114.43	0.97

shows the performance.

After learning using the DIV2K data set [18], we performed standard quantitative measurements using PSNR and SSIM. The focus of this work was the perceptual quality of super-resolution images.

4. Simulations

4.1. Simulations Environment Architecture

Figure 17 shows the overall system structure and how it is that the original video is outputted in the server. In order to change the resolution of the image, the image is divided and the resolution is then adjusted. These images are merged and transmitted by the server. Segmentation, recognition, transformation, and networks are used in this process.

4.2. System Configuration

In this section, we show the proposed model. The experimental method in Figure 18 shows an example of image segmentation. When dividing the image, the image is adjusted to the absolute size, so the area can be divided according to the location of the FoV. This method is a flexible and objective method that can change the viewpoint by extracting a box-shaped image in units of regions. Algorithms and computational methods of networks are configured according to each domain.

Figure 19 and Figure 20 show the results of applying the image segmentation method to 360° video and general video. Since the image is scaled and segmented based on its absolute size, the area can be segmented according to the position of the FoV. As shown in Figure 18, this method can be easily and quickly applied to visual viewpoint transformation because it is extracted in the shape of a box. Therefore, flexible calculations are possible when applying networks and algorithms.

Figure 21 shows how the client and server behave according to user requests.

The server prepares video for output through preprocessing.

When the user’s request is confirmed, the requested video is sent. Whenever the user’s field of view is switched, the client sends an event to the server and requests the video it needs. When the server receives the video, each network works to output the video.

Figure 22 shows how video is handled by the network depending on the user’s point of view. In this simulation, the image was segmented into nine parts and divided into FoV and non-FoV areas according to the user’s point of view. It shows how the image is outputted after SRGAN is applied to the FoV area and DAIN and SST algorithms are applied to the non-FoV area.

4.3. Evaluation

We performed several tests on the system using various images. The data for each video was measured via live streaming. Figure 23 shows the image used for the simulation.

The used video displays information about each video in the form of a Channel (Ch.). The 360° real-time game screen, the 2D game screen, and broadcast screen were used in Ch. 1–3. For comparison, the results of applying the entire area, including the original, and the network after segmentation, including SRGAN and SRCNN, are shown in each diagram.

Table 2. Compare each network of the 360° video.

	Original	SRGAN (Scaling)	SRGAN (by One Scaling)	SRCNN (Scaling)	SRCNN (by One Scaling)
Bitrate	719 Kbps	7365 Kbps	2520 Kbps	14,676 Kbps	5021 Kbps
Latency	--	16,974.42	1886.05	30,780	7695
Network Bandwidth	0.7112 Mbps	7.3264 Mbps	2.4624 Mbps	14.3336 Mbps	4.8184 Mbps

and

Table 3. Bitrate and size comparison for each of the 360° videos.

Video	Bitrate	Data Rate
No. 1(5120*2880), FoV	3651 Kbps	120 Mb
No. 2(1280*720)	312 Kbps	10.1 Mb
No. 3(1280*720)	276 Kbps	8.94 Mb
No. 4(1280*720)	349 Kbps	11.2 Mb
No. 5(1280*720)	445 Kbps	14.3 Mb
No. 6(1280*720)	317 Kbps	10.2 Mb
No. 7(1280*720)	260 Kbps	8.18 Mb
No. 8(1280*720)	404 Kbps	13.0 Mb
No. 9(1280*720)	296 Kbps	8.14 Mb
Total	6310 Kbps	204.06 Mb

correspond to Ch. 1 in Figure 23. As a result of the simulation, it can be seen that bitrate differs by 5.8 times compared to SRCNN (scaling) based on FoV (by one scaling) value, and SRGAN (scaling) differs by 2.9 times. Latency values showed a high difference of 16 times for SRCNN and 9 times for SRGAN. And it can be seen that the difference in Network Bandwidth is about 5.8 to 2.9 times. The overall bitrate and data rate by one scaling are shown in Table 3, and in the FoV area, 4K or higher quality is transmitted. The video bitrate range of 4K or higher requires about 13,000–51,000 kbps, but the bitrate in the FoV area was reduced to 3651 kbps.

Table 4. Compare each network of game streaming video.

	Original	SRGAN (Scaling)	SRGAN (by One Scaling)	SRCNN (Scaling)	SRCNN (by One Scaling)
Bitrate	301 Kbps	6937 Kbps	1054 Kbps	13,823 Kbps	4729 Kbps
Latency	--	755.24	83.90	373.92	92.98
Network Bandwidth	10.568 Kbps	229.048 Kbps	79.072 Kbps	413.16 Kbps	154.736 Kbps

and

Table 5. Bitrate and size comparison for each game streaming video.

Video	Bitrate	Data Rate
No. 1 (426*240)	67 Kbps	616 kb
No. 2 (426*240)	78 Kbps	710 kb
No. 3 (426*240)	65 Kbps	592 kb
No. 4 (426*240)	92 Kbps	833 kb
No. 5 (1704*960), FoV	854 Kbps	7622 kb
No. 6 (426*240)	93 Kbps	841 kb
No. 7 (426*240)	83 Kbps	752 kb
No. 8 (426*240)	101 Kbps	918 kb
No. 9 (426*240)	85 Kbps	776 kb
Total	1518 Kbps	13,660 kb

corresponds to Ch. 2. As a result of the simulation, it can be seen that the FoV (by one scaling) of the bitrate differs by 13 times in SRCNN (scaling) and 6.5 times in SRGAN (scaling). Latency showed a high difference of 4.4 times in SRCNN and 9 times in SRGAN. And it can be seen that network bandwidth differs by about 5 to 2.8 times. Table 5 shows the overall bitrate and data rate of one scaling. In the FoV area, HD level quality is transmitted. In HD quality, the video bitrate range needs about 1500–6000 kbps, but the FoV area was able to check it at 854 kbps.

Table 6. Compare each network of broadcast streaming video.

	Original	SRGAN (Scaling)	SRGAN (by One Scaling)	SRCNN (Scaling)	SRCNN (by One Scaling)
Bitrate	247 Kbps	6510 Kbps	989 Kbps	12,972 Kbps	4437 Kbps
Latency	--	457.72	50.85	225.4	56.35
Network Bandwidth	8.496 Kbps	87.52 Kbps	29.416 Kbps	171.224 Kbps	57.56 Kbps

and

Table 7. Bitrate and size comparison for each streaming video broadcast.

Video	Bitrate	Data Rate
No. 1(426*240)	95 Kbps	648 kb
No. 2(426*240)	85 Kbps	584 kb
No. 3(426*240)	75 Kbps	515 kb
No. 4(426*240)	110 Kbps	748 kb
No. 5(1704*960), FoV	159 Kbps	1069 kb
No. 6(426*240)	97 Kbps	661 kb
No. 7(426*240)	80 Kbps	549 kb
No. 8(426*240)	98 Kbps	667 kb
No. 9(426*240)	90 Kbps	613 kb
Total	891 Kbps	6054 kb

correspond to Ch. 3. As a result of the simulation, it can be seen that the FoV (by one scaling) of the bitrate differs by about 13 times in SRCNN (scaling) and 6.5 times in SRGAN (scaling). Latency values also showed a high difference of 4.4 times for SRCNN and 9 times for SRGAN. It can be seen that network bandwidth differs from about 5.8 to 2.9 times. For Ch. 3, it is the same size as Ch. 2. However, since Ch. 2 is in the form of a game, more load is inevitably added, so Ch. 3 shows a lower value than Ch. 2. In the case of Ch. 3, as with Ch. 2, HD quality is transmitted, and a bitrate of about 159 kbps was confirmed. When the video is transmitted by applying the proposed method, it shows a low bitrate value even though it is a high-resolution video. It can be seen that the overall value decreased by about 40–60%.

5. Conclusions

The ideal in video streaming today is high-quality, persistent video. However, image quality may deteriorate depending on the user’s system and the network environment, and a system with a higher specification may be required. In addition, it is thought that bandwidths currently used are insufficient to cover the streaming market in the future, as more broadcasts are showing real-time motion through VR or 360-degree video. In particular, videos with resolutions over 4K may have insufficient bandwidth depending on the streaming environment.

In this paper, we have proposed a method to reduce system and network load according to the video viewing area. We aimed to reduce the bitrate by at least 30–40% and derived a bitrate reduction of about 40–60% using the proposed method. According to this method, after segmenting the image, SRGAN is applied to the FoV region to transmit an image of higher quality than the original image, and the DAIN + SST algorithm is applied to the non-FoV region to adjust FPS. In this way, we were able to solve the phenomenon of quality degradation that can be observed depending on the streaming environment. In addition, in order to reduce the burden on viewers due to the switching delay that occurs during region conversion using this method, streaming is enabled using the video region of the preprocessing part of the server.

In conclusion, using the above method, the user can transmit a high-quality image using a small capacity, and the viewer can watch the high-quality image. However, the use of various networks and algorithms requires a high-spec server. Therefore, in future research, instead of reducing the load on the network by reducing preprocessing as much as possible, we plan to conduct research to increase the FoV area, improve QoE, and reduce the load on the server.