Intelligent Caching for Mobile Video Streaming in Vehicular Networks with Deep Reinforcement Learning

Abstract

Caching-enabled multi-access edge computing (MEC) has attracted wide attention to support future intelligent vehicular networks, especially for delivering high-definition videos in the internet of vehicles with limited backhaul capacity. However, factors such as the constrained storage capacity of MEC servers and the mobility of vehicles pose challenges to caching reliability, particularly for supporting multiple bitrate video streaming caching while achieving considerable quality of experience (QoE). Motivated by the above challenges, in this paper, we propose an intelligent caching strategy that takes into account vehicle mobility, time-varying content popularity, and backhaul capability to improve the QoE of vehicle users effectively. First, based on the mobile video mean opinion score (MV-MOS), we designed an average download percentage (ADP) weighted QoE evaluation model. Then, the video content caching problem is formulated as a Markov decision process (MDP) to maximize the ADP weighted MV-MOS. Owing to the prior knowledge of video content popularity and channel state information that may not be available at the road side unit in practical scenarios, we propose a deep reinforcement learning (DRL)-based caching strategy to solve the problem while achieving a maximum ADP weighted MV-MOS. To accelerate its convergence speed, we further integrate the prioritized experience replay, dueling, and double deep Q-network technologies, which improve the performance of DRL algorithm. Numerical results demonstrate that the proposed DRL-based caching strategy significantly improves QoE, and achieves better video delivery reliability compared to existing non-learning approaches.

1. Introduction

The advent of the 5G era and rapid development of cellular vehicle-to-everything (C-V2X) technologies have realized a massive number of vehicles equipped with advanced intelligent devices (e.g., high-definition (HD) players, 3D navigation equipment, vehicle mixed reality (MR) glasses), which bring significant mobile data traffic and service explosion [1]. Furthermore, the global mobile data traffic is expected to reach 77 exabytes per month by 2022, as reported by Cisco, where 79 percent of this data comes from mobile video [2]. Additionally, innovative mobile video technologies have inspired a wide range of video streaming applications in vehicular networks, becoming one of the most popular and indispensable services.

Generally, a massive amount of HD mobile video data is downloaded through the backhaul network of a macro base station (MBS), resulting in congested traffic loads. Consequently, the limited backhaul bandwidth capacity poses one of the most significant challenges to HD video data delivery services in the internet of vehicles (IoV) [3]. To effectively alleviate the backhaul traffic load, reduce the content delivery latency, and meet the diverse quality of service (QoS) data transmission requirements [4], multi-access edge computing (MEC) has been proposed as an efficient paradigm [5], which enables data caching at the edge of mobile networks, thus offers fast, and video delivery. The quality of experience (QoE) of mobile video streaming is a key criterion for measuring mobile video quality in mobile networks [6].

One promising method to improve the QoE of mobile users is to bring video content close to mobile users by deploying the MEC server (MECS) in mobile edge networks. Hence, mobile users can download the requested HD video from the small base stations (SBS) directly instead of acquiring the relevant contents from the MBS through the backhaul network, which reduces transmission delay for video distribution. However, the existing mobile network edge caching strategies can hardly be applied directly in vehicular edge caching networks for the following reasons [7,8]. First, high vehicle mobility and limited caching storage capability bring difficulties in achieving HD video delivery among vehicle users. Second, the high bitrate of the emerging Ultra HD videos (e.g., 1080p, 2K, and 4K video that request at least 12–25 Mbps user data rate [9]) may incur vehicle users to be unable to complete HD video download tasks within service hours of road side unit (RSU). Owing to the limited backhaul capacity, the MBS, which took over the service, can hardly provide bitrate HD video requirements, leading to failure to download tasks. Additionally, considering the high vehicle mobility, the video transmission time is limited by the time duration a vehicle is within the signal coverage of an RSU [10]. As a result, the traditional mobile network QoE evaluation method relying on available service time is infeasible to apply vehicular caching networks directly.

Adaptive bitrate (ABR) video streaming has recently become a technology to improve QoE [11]. However, many works have focused on the ABR video process, ignoring the heterogeneity of networks and user capabilities that impact ABR video caching. MECS can connect with vehicle users for high-speed and stable transmissions, but its storage capacity is limited [12]. Caching high-bitrate videos can achieve higher QoE than low-bit-rate videos, but requires higher caching resources. Hence, there is a trade-off between caching some high-bitrate videos and diverse videos [13,14]. Moreover, the duration for downloading HD video is limited by the constrained time that a vehicle passes through the RSU service area [15]. To achieve maximum QoE, the caching decision of MECS in vehicular networks becomes challenging to support mobile videos of multiple bitrate levels [16]. Motivated by the abovementioned challenges, in this paper, we propose a deep reinforcement learning (DRL)-based intelligent caching strategy for mobile video in vehicular networks. Major contributions are summarized as follows:

1

We propose a mobile video caching framework in vehicular networks, where vehicle users can access the high-resolution video from the MECS deployed on RSU directly without occupying the backhaul network to release the bandwidth resource and reduce the data transmission delay.

2

We designed an average download percentage (ADP) weighted mobile video mean opinion score (MV-MOS) model for vehicle users. Then, the video content caching problem is formulated as a Markov decision process (MDP) to maximize the ADP weighted MV-MOS.

3

Based on a deep Q-learning network (DQN), we propose an intelligent caching algorithm to solve the problem while achieving a maximum ADP weighted MV-MOS. Furthermore, a P3DVQC caching scheme in vehicular networks was proposed to improve the performance of algorithm by integrating the prioritized experience replay, dueling, and double DQN technologies.

The remainder of this paper is organized as follows. The related work is presented in Section 2. We describe the system model in Section 3. In Section 4, we formulate intelligent caching for mobile video problems. Section 5 gives the solution of the proposed method. Numerical results and discussion are presented in Section 6. Finally, a conclusion is drawn in Section 7.

2. Related Work

A considerable study has been conducted on the design of mobile edge caching algorithms for mobile networks. The authors in [17] proposed proactive caching of popular content during off-peak periods to reduce peak traffic demands. The authors in [18] conducted a comprehensive survey regarding different aspects of mobile edge caching. Furthermore, they have discussed caching schemes based on different caching locations and performance criteria. The work in [19,20] leveraged edge nodes (e.g., small cells) to store popular content, such as multimedia files, to reduce latency and improve the performance of 5G networks. The authors in [21] propose a novel deep learning-based proactive caching framework in cellular networks that obtain higher backhaul offloading and user satisfaction. Along this line of mobile edge caching, video placement has been studied over different heterogeneity of networks. Considering that user devices and preferences and needs for specific videos may vary, adaptive bitrate (ABR) streaming becomes a pivotal technique to improve the quality of delivered video on networks. The authors in [22] envision a collaborative joint caching and processing strategy for multiple bitrate video delivered to adapt to the heterogeneity of user capabilities and wireless communication conditions. According to the authors in [23] scalable video coding (SVC) based video services are considered to formulate the joint video quality selection and caching problem, to maximize vehicular user’s QoE. The solutions based on SVC on a system called DASH to ensure the quality of streaming media services have been proposed in recent studies [24].

Many works focus on the optimization efficiency of video caching and reducing video delivery delay in vehicular networks. The work in [25] proposed a cooperative transmission strategy for video transmission in small-cell networks with caching. Authors in [26] investigate a problem of cooperative mobile edge caching for scalable video streaming in HetNets. Adaptive video technology is applied to popular streaming services such as YouTube, Netflix, and Youku to provide smooth streaming and improve quality, such as Microsoft smooth streaming, Adobe’s HDS, and Apple’s HLS [27]. These streaming services encode videos into multiple versions with discrete bitrates. The authors in [28] proposed a mechanism based on MEC to cache only the highest available bit-rate video content while converting it to the requested lower bit-rate version using the available processing power of MEC. Many recent works focus on an adaptive bitrate streaming to cope with time-varying channels incurred by vehicular users’ high mobility in IoV. In [29], the authors use a technique to effectively use both ABR streaming and BS caching in vehicular networks with high channel variations. Quality of experience serves as a direct evaluation of vehicle users’ experiences in mobile video transmission, and thus the authors in [30] propose a deep learning-based QoE prediction approach with a large-scale QoE dataset for mobile video transmission. The work [31] proposed to simultaneously optimize energy consumption and QoE metrics in video streaming over software-defined mobile networks (SDMN) combined with MEC.

Zhao et al. [32] by considering the interaction between video encoding and edge caching, the authors proposed a QoE-driven cross-layer optimization scheme for secure video transmission over the backhaul links in cloud-edge networks. Liang et al. [33] proposed enhancing the quality of experience-aware wireless edge caching with bandwidth provisioning in software-defined wireless networks. Latency is decreased, and the utilization of caches is improved in the proposed scheme. Huang et al. [34], based on the video popularity and the wireless resource conditions of the network, proposed a joint cache allocation and video delivery scheme for the video streaming system. Alberto et al. [35] presented demonstrates the possibility of developing a DRL-based quality optimization framework which can guarantee an adequate QoE. Li et al. [36] studied a QoE-driven mobile edge caching placement optimization problem for dynamic adaptive video streaming that by the optimal caching placement of representations for multiple videos, they maximize the aggregate average video distortion reduction of all users while minimizing the additional cost of downloading. Qiao et al. [37] proposed a deep deterministic policy gradient (DDPG)-based cooperative caching scheme to jointly optimize the content delivery and content placement in vehicular networks.

3. System Model

As shown in Figure 1, we consider a highway vehicular networks scenario that includes an MBS, several RSUs, and vehicle users (VUs). The MBS connects to the core network (CN) through the backhaul link. Denote $\mathcal{M}=\left\{1,\dots ,M\right\}$ and $\mathcal{U}=\left\{1,\dots ,U\right\}$ be the RSUs set and vehicle users set, respectively. Each RSU is equipped with an MECS of size ${\phi }_{MEC}$ to store a number of popular video replicas, which helps reduce the delay of content delivery and improve QoE of VUs. Vehicular users can access nearby RSU or MBS and download videos from the MECS or CN. We assume that each vehicular user request one of the interesting videos once the vehicle enters the service coverage areas of RSU but experiences a higher download rate if the requested one is pre-cached in MECS. Set the vehicle user arrival probability to a Poisson distribution of B different parameters that obey the Markov process, then the parameter of vehicle user arrival probability at time slot $\tau$ is recorded as $\lambda \left(\tau \right)\in \left\{{\lambda }_1,{\lambda }_2,\dots ,{\lambda }_B\right\}$. Then, at time slot $\tau$, the probability that the number of vehicle users $U\left(\tau \right)$ in the service coverage is expressed as

$$P\left(U\left(\tau \right)\right)=\frac{{\left(\lambda \left(\tau \right)\right)}^{U\left(\tau \right)}}{U\left(\tau \right)!}{e}^{-\lambda \left(\tau \right)}.$$

(1)

We define the state transition probability of the parameter of arrival probability $\lambda \left(\tau \right)={\lambda }_i,i\in \left\{1,\dots ,B\right\}$ at time slot $\tau$ transfers to $\lambda \left(\tau +1\right)={\lambda }_j,j\in \left\{1,\dots ,B\right\}$ at time slot $\tau +1$ as ${\vartheta }_{i,j}$, the transition matrix $\mathsf{\Gamma }$ of parameter $\lambda$ is expressed as

$$\mathsf{\Gamma }=\left(\begin{array}{ccc}{\vartheta }_{1,1}& \dots & {\vartheta }_{1,B}\\ ⋮& \ddots & ⋮\\ {\vartheta }_{B,1}& \cdots & {\vartheta }_{B,B}\end{array}\right).$$

(2)

3.1. Caching Model

The video content popularity reflects the statistical results of the video requested by vehicle users over a period. As a result, the time slot varying scale of video content popularity is much larger than the cache content update of the MECS. Furthermore, the time slot varying scale of the video content delivery process is much smaller than the cache content update of the MECS. Hence, the mobile video caching in the vehicular network can be modeled as a multi-time scale model, as shown in Figure 2.

Popularity variation time scale: We assume that one video popularity variation time slot is included $K_{CN}$ caching placement time slot. The time scales of popularity variation can be defined as $t^x,x\in \left\{1,2,3,\dots ,K_{CG}\right\}$, which containing $K_{CG}$ time slots, and define the length of the one MECS caching update period is ${∆t}_y^x$, then the length of ${∆t}^x$ is expressed as

$${∆t}^x=K_{CN}{∆t}_y^x.$$

(3)

Caching placement time scale: The times scales of caching placement can be defined as $t_y^x,y\in \left\{1,2,3,\dots ,K_{CN}\right\}$, and caching video content of MECS is updated once every time slot of $t_y^x$. Suppose $\mathcal{D}$ is the length of service covered, the length of ${∆t}_y^x$ at vehicles speed $v\left(t_y^x\right)$ is calculated as

$${∆t}_y^x=\frac{\mathcal{D}}{v\left(t_y^x\right)}.$$

(4)

Video delivery time scale: The rapid movement of the vehicle causes the geographic location and channel status to change, the transmission rate of the vehicle user is time-vary. The ${∆t}_y^x$ is discretized into $K_{SN}$ segments to simplify the model. Then the times scales of video delivery can be defined as $t_{y,z}^x,z\in \left\{1,2,3,\dots ,K_{SN}\right\}$. It can be considered that the communication rate in each content distribution period ${∆t}_{y,z}^x$ remains unchanged. The length ${∆t}_{y,z}^x$ of the time slot $t_{y,z}^x$ is calculated as

$${∆t}_{y,z}^x=\frac{{∆t}_y^x}{K_{SN}}.$$

(5)

The video requested by vehicle users (such as 4K/8K high definition movies, etc.) comes from a video content library $\mathcal{F}=\left\{f_1,f_2,\dots ,f_F\right\}$, which containing F video files. Assumed that $\mathcal{F}$ obeys Zipf’s law [27], and the videos of $\mathcal{F}$ are arranged in descending order of their content popularity. Using constant bitrate (CBR) technology to encode the videos of $\mathcal{F}$ into a constant bitrate multiple bitrate video $\mathcal{L}=\left\{{\ell }_1,{\ell }_2,\dots ,{\ell }_L\right\}$, its bitrate level is still arranged in descending order ${\ell }_1>{\ell }_2>\dots >{\ell }_L$, that is, ${\ell }_1$ is the replicas of video with the highest bitrate. Without loss of generality, assume that the time lengths of the videos in $\mathcal{F}$ are equal, denoted as $\varpi$, it can be considered that after coding using the CBR technology, all video copies of the same bitrate level in $\mathcal{F}$ have the same size. The replicas size $g_j$ of video f at the ${\ell }_j$ bitrate level is expressed as

$$g_j=\varpi {\ell }_j.$$

(6)

Hence, the caching state matrix $X\left(t_y^x\right)$ of MECS at the time slot $t_y^x$, is defined as

$$X\left(t_y^x\right)=\mathcal{F}\times \mathcal{L}={\left\{x_{{}_{f_i,{\ell }_j}}\left(t_y^x\right),f_i\in \mathcal{F},{\ell }_j\in \mathcal{L}\right\}}_{F\times L}.$$

(7)

where F is the number of videos and L is bitrate level. The $X\left(t_y^x\right)$ can be further expressed as $X\left(t_y^x\right)\in {\left\{0,1\right\}}_{F\times L}$. Then caching state variable of video $f_i$ with bitrate level ${\ell }_j$ at the time slot $t_y^x$ is expressed as

$$x_{f_i,{\ell }_j}\left(t_y^x\right)=\left\{\begin{array}{c}1,\mathrm{if}\mathrm{MECS}\mathrm{cached}\mathrm{the}{\ell }_j\mathrm{level}\mathrm{of}\mathrm{video}{f}_i\hfill \\ 0,\mathrm{otherwise}\hfill \end{array}\right..$$

(8)

The video caching state vector is defined as ${\chi }_{\mathcal{F}}\left(t_y^x\right)=\left\{{\chi }_{f_1}\left(t_y^x\right),{\chi }_{f_2}\left(t_y^x\right),\dots ,{\chi }_{f_F}\left(t_y^x\right)\right\}$, ${\chi }_{f_1}\left(t_y^x\right)$ is the number of video copy of $f_1$ in MECS cached at the time slot $t_y^x$.

$${\chi }_{f_i}\left(t_y^x\right)=\sum _{j\in L}{x}_{f_i,{\ell }_j}\left(t_y^x\right)\le 1.$$

(9)

The Formula (9) is the constraint that the MECS can only cache one video copy of the bitrate for the same video simultaneously. Additionally, existing research shows that low bitrate video copies can be obtained from high bitrate video transcoding by MECS. Hence, multiple video copies which are cached simultaneously will cause heavy waste of caching resources [34]. Therefore, the constraint condition of the caching capacity of MECS is expressed as

$$\sum _{i\in F}\sum _{j\in L}{g}_j{x}_{f_i,{\ell }_j}\left(t_y^x\right)\le {\phi }_{MEC}.$$

(10)

Based on Zipf’s law, the video’s requested probability of $\mathcal{F}$ can be denoted as $P_{\mathcal{F}}^x\left(f\right)=\left\{P_{\mathcal{F}}^x\left(f_1\right),P_{\mathcal{F}}^x\left(f_2\right),\dots ,P_{\mathcal{F}}^x\left(f_F\right)\right\}$. Then video $f_i$ is the requested probability by the vehicle user at the time slot $t^x$ and can be expressed as

$$P_{\mathcal{F}}^x\left(f_i\right)={\left({f_i}^{\mu \left(t^x\right)}\sum _{k=1}^F{f_k}^{-\mu \left(t^x\right)}\right)}^{-1},$$

(11)

where $\mu \left(t^x\right)$ is popularity parameter at the time slot $t^x$, which reflects the shape of video popularity distribution. Without loss of generality, suppose $\mu \left(t^x\right)$ obeys a Markov process $\mu \left(t^x\right)\in \left\{{\mu }_1,{\mu }_2,\dots ,{\mu }_G\right\}$, which containing G parameter. Let ${\varphi }_{i,j}$ denote transition probability from popularity parameter ${\mu }_i$ at the time slot $t^x$ to ${\mu }_j$ at time $t^{x+1}$. Then the state transition probability matrix $\mathrm{\Phi }$ of the video popularity parameter $\mu$ is calculated as

$$\mathrm{\Phi }=\left(\begin{array}{ccc}{\varphi }_{1,1}& \dots & {\varphi }_{1,G}\\ ⋮& \ddots & ⋮\\ {\varphi }_{G,1}& \cdots & {\varphi }_{G,G}\end{array}\right).$$

(12)

Hence, the cumulative distribution function of the top S videos in $\mathcal{F}$ requested by the VUs is expressed as

$$P_{CDF}^x\left(S\right)=\underset{i=1}{\sum ^S}{P}_{\mathcal{F}}^x\left(f_i\right)=\underset{i=1}{\sum ^S}\frac{f_i^{-\mu \left(t^x\right)}}{{\sum }_{k=1}^F{f}_k^{-\mu \left(t^x\right)}}.$$

(13)

When the caching state matrix is $X\left(t_y^x\right)$, and caching vector is ${\chi }_{\mathcal{F}}\left(t_y^x\right)$, then the caching hit rate at the time slot $t_y^x$ is expressed as

$$\mathcal{H}\left(t_y^x\right)=\sum _{i=1}^F{P}_{\mathcal{F}}^x\left(f_i\right){\chi }_{f_i}\left(t_y^x\right).$$

(14)

Hence, the $\mathcal{H}\left(t_y^x\right)$ can be rewritten as

$$\mathcal{H}\left(t_y^x\right)=\sum _{i=1}^F{P}_{\mathcal{F}}^x\left(f_i\right){\chi }_{f_i}\left(t_y^x\right)=\frac{{\sum }_{i=1}^F{f}_i^{-\mu \left(t^x\right)}{\chi }_{f_i}\left(t_y^x\right)}{{\sum }_{k=1}^F{f}_k^{-\mu \left(t^x\right)}}.$$

(15)

Then the caching loss rate at the time slot $t_y^x$ is calculated as

$$\overline{\mathcal{H}}\left(t_y^x\right)=1-\mathcal{H}\left(t_y^x\right)=\frac{{\sum }_{i=1}^F{f}_i^{-\mu \left(t^x\right)}\left(1-{\chi }_{f_i}\left(t_y^x\right)\right)}{{\sum }_{k=1}^F{f}_k^{-\mu \left(t^x\right)}}.$$

(16)

3.2. Communication Model

We assume that the access system of the vehicular network is based on orthogonal frequency division multiple access (OFDMA). Therefore, every vehicle linked to RSU or MBS is assigned an orthogonal subchannel. Thus, we do not consider interference among different links. For simplicity, the channel gains are assumed to remain constant during one video delivery period. They have the same distribution, so it is sufficient to concentrate on one vehicle user to study the performance of interest. The transmission rate between vehicle and RSU at time slot $t_{y,z}^x$ can be calculated by

$${\mathcal{R}}_{MEC}\left(t_{y,z}^x\right)=W_r\left({log}_2\left(1+\frac{P_r{\left|h_r\left(t_{y,z}^x\right)\right|}^2}{{\delta }^2}\right)\right),$$

(17)

where $W_r$ is the channel bandwidth allocated by RSU to vehicle users, $P_r$ is the signal transmit power of RSU, ${\delta }^2$ is the noise power, $h_r\left(t_{y,z}^x\right)$ is the RSU channel gain at time slot $t_{y,z}^x$, it can be expressed as

$${\left|h_r\left(t_{y,z}^x\right)\right|}^2=G_r{\left|d_{1,u}\left(t_{y,z}^x\right)\right|}^{-ϵ}{\left|h_0\left(t_{y,z}^x\right)\right|}^2,$$

(18)

where $ϵ$ is the path loss coefficient, $h_0\left(t_{y,z}^x\right)$ is a complex Gaussian distributed random variable for Rayleigh channel fading, $|.|$ is absolute operation, ${\left|h_0\left(t_{y,z}^x\right)\right|}^2\sim CN\left(0,1\right)$, $G_r$ is the antenna gain coefficient of RSU, $d_{1,u}\left(t_{y,z}^x\right)$ represents the distance between the vehicle user and RSU. If the MECS does not cache any bitrate video copies of the video requested by the vehicle user, MBS will take over the service of the vehicle user through the backhaul network, and distribute video copies with the highest bitrate under the constrained backhaul bandwidth to the VUs. The transmission rate between the vehicle user and MBS is expressed as

$${\mathcal{R}}_{MBS}\left(t_{y,z}^x\right)=B_0\left({log}_2\left(1+\frac{P_{bs}{\left|h_{bs}\left(t_{y,z}^x\right)\right|}^2}{{\delta }^2}\right)\right),$$

(19)

where $B_0$ is MBS allocates bandwidth to each vehicle user, $P_{bs}$ is the MBS transmit power, MBS channel gain at the time slot $t_{y,z}^x$ is calculated as

$${\left|h_{bs}\left(t_{y,z}^x\right)\right|}^2=G_{bs}{\left|d_{0,u}\left(t_{y,z}^x\right)\right|}^{-ϵ}{\left|h_0\left(t_{y,z}^x\right)\right|}^2,$$

(20)

where $G_{bs}$ is the antenna gain coefficient of MBS, $d_{0,u}\left(t_{y,z}^x\right)$ represents the distance between the vehicle user and MBS. Due to ${\mathcal{R}}_{MBS}$ is both limited by the wireless access network and the backhaul bandwidth $C_{BM}$, when the number of vehicle users $N\left(t_{y,z}^x\right)$ connected to MBS at time $t_{y,z}^x$ does not reach the maximum number of bearer users $N_{Max}$ of MBS, the principle of fairness will be taken into consideration. The bandwidth resource required for caching update will allocate a size of $C_{MBS}$ backhaul bandwidth to each vehicle user. $N_{Max}$ is defined as

$$N_{Max}=⌊\frac{C_{BM}}{C_{MBS}}⌋,$$

(21)

where $⌊.⌋$ is the rounding down operation. When the number of vehicle users connected to MBS exceeds $N_{Max}$, the backhaul resources will be average allocated to users, and the transmission rate ${\mathcal{R}}_{Back}\left(t_{y,z}^x\right)$ can be expressed as

$${\mathcal{R}}_{Back}\left(t_{y,z}^x\right)=\left\{\begin{array}{c}{C}_{MBS},N\left(t_{y,z}^x\right)\le N_{Max}\\ \frac{C_{BM}}{N\left(t_{y,z}^x\right)},N\left(t_{y,z}^x\right)>N_{Max}\end{array}\right..$$

(22)

Hence, the service state probability can be expressed as

$$P_{Ar}\left(U\left(t_{y,z}^x\right),N\left(t_{y,z}^x\right)\right)=\left(\begin{array}{c}U\left(t_{y,z}^x\right)\hfill \\ N\left(t_{y,z}^x\right)\hfill \end{array}\right)\mathcal{H}{\left(t_y^x\right)}^{\left(U\left(t_{y,z}^x\right)-N\left(t_{y,z}^x\right)\right)}\overline{\mathcal{H}}{\left(t_y^x\right)}^{N\left(t_{y,z}^x\right)}.$$

(23)

where $U\left(t_{y,z}^x\right)$ is the number of vehicle users at the time slot $t_{y,z}^x$, $N\left(t_{y,z}^x\right)$ is the number of vehicle users who download the video by the backhaul link. The average transmission rate by backhaul link at the time $t_{y,z}^x$ is calculated as

$${\mathcal{R}}_{Bh}\left(t_{y,z}^x\right)=\sum _{N\left(t_{y,z}^x\right)=0}^{\mathbb{E}\left[U\left(t_{y,z}^x\right)\right]}{P}_{Ar}\left(\mathbb{E}\left[U\left(t_{y,z}^x\right)\right],N\left(t_{y,z}^x\right)\right){\mathcal{R}}_{Back}\left(t_{y,z}^x\right)$$

(24)

where $\mathbb{E}$ is the expect operation. Then, the transmission rate of MBS can be rewritten as

$${\mathcal{R}}_{BH}\left(t_{y,z}^x\right)=min\left\{{\mathcal{R}}_{MBS}\left(t_{y,z}^x\right),{\mathcal{R}}_{Back}\left(t_{y,z}^x\right)\right\}.$$

(25)

3.3. ADP Weighted MV-MOS Model

In summary, the multiple bitrate video distribution strategy is that if the video requested by the vehicle user has been pre-cached in the MECS, the video file with the highest bitrate within the MECS will be sent to the vehicle user directly. Otherwise, VUs request the lowest bitrate replicas of videos via the MBS. The intelligent caching in IoV problem aims to maximize vehicle users’ QoE. The MV-MOS is a mobile video experience metric at the device level, it has been widely used in the QoE evaluation of mobile video, e.g., [33,34,38]), so it is expressed as

$$f_{MOS}\left({\ell }_j,{\mathcal{R}}_{{\ell }_j}\right)=Mv\left({\ell }_j,{\mathcal{R}}_{{\ell }_j}\right).$$

(26)

where $Mv\left({\ell }_j,{\mathcal{R}}_{{\ell }_j}\right)$ is required data transmission rate, ${\ell }_j$ is the video definition level, and ${\mathcal{R}}_{{\ell }_j}$ is the minimum bitrate requirement corresponding to the video definition level ${\ell }_j$. Then mobile video mean opinion score for the vehicle user to obtain the video from the MECS at the time $t_{y,z}^x$ is calculated as

$$Q_{MEC}\left(t_{y,z}^x\right)=\sum _{i\in F}\sum _{j\in L}{x}_{f_i,{\ell }_j}\left(t_y^x\right)P_{\mathcal{F}}^x\left(f_i\right)f_{MOS}\left({\ell }_j,{\mathcal{R}}_{MEC}\left(t_{y,z}^x\right)\right).$$

(27)

Hence, mobile video mean opinion score for the vehicle user to obtain a video from MBS at time slot $t_{y,z}^x$ can be expressed as

$$Q_{BH}\left(t_{y,z}^x\right)=\sum _{i\in F}\left(1-{\chi }_{f_i}\left(t_y^x\right)\right)P_{\mathcal{F}}^x\left(f_i\right)\underset{j\in L}{max}{f}_{MOS}\left({\ell }_j,{\mathcal{R}}_{BH}\left(t_{y,z}^x\right)\right).$$

(28)

When the number of users in the service coverage area at time slot $t_{y,z}^x$ is $U\left(t_{y,z}^x\right)$, and the number of VUs downloading videos through MBS is $N\left(t_{y,z}^x\right)$, all are obtained from the MECS, mobile video mean opinion score for the vehicle user is expressed as

$$Q_M\left(U\left(t_{y,z}^x\right),N\left(t_{y,z}^x\right)\right)=\left(U\left(t_{y,z}^x\right)-N\left(t_{y,z}^x\right)\right)P_{Ar}\left(U\left(t_{y,z}^x\right),N\left(t_{y,z}^x\right)\right)Q_{MEC}\left(t_{y,z}^x\right).$$

(29)

The mobile video mean opinion score for the $N\left(t_{y,z}^x\right)$ VUs who obtain videos from MBS through the backhaul network at the time slot $t_{y,z}^x$ is expressed as

$$Q_B\left(U\left(t_{y,z}^x\right),N\left(t_{y,z}^x\right)\right)=P_{Ar}\left(U\left(t_{y,z}^x\right),N\left(t_{y,z}^x\right)\right)N\left(t_{y,z}^x\right)Q_{BH}\left(t_{y,z}^x\right).$$

(30)

When the number of vehicle users via the RSU service at the time slot $t_{y,z}^x$ is $U\left(t_{y,z}^x\right)$, and the number of vehicle users downloading videos through MBS is $N\left(t_{y,z}^x\right)$, then the average mobile video mean opinion score at the time slot $t_{y,z}^x$ is expressed as

$${\overline{Q}}_{AV}\left(U\left(t_{y,z}^x\right),N\left(t_{y,z}^x\right)\right)=\frac{Q_M\left(U\left(t_{y,z}^x\right),N\left(t_{y,z}^x\right)\right)+Q_B\left(U\left(t_{y,z}^x\right),N\left(t_{y,z}^x\right)\right)}{U\left(t_{y,z}^x\right)}.$$

(31)

Hence, the average mobile video mean opinion score at time slot $t_{y,z}^x$ is expressed as

$$Q_{Aver}\left(t_{y,z}^x\right)=\sum _{N\left(t_{y,z}^x\right)=0}^{\mathbb{E}\left[U\left(t_{y,z}^x\right)\right]}{\overline{Q}}_{AV}\left(\mathbb{E}\left[U\left(t_{y,z}^x\right)\right],N\left(t_{y,z}^x\right)\right).$$

(32)

Therefore, the average MV-MOS of vehicle users for one period $t_y^x$ is expressed as

$$Q_{MOS}\left(t_y^x\right)=\frac{{\displaystyle \sum _{z\in K_{SN}}}{Q}_{Aver}\left(t_{y,z}^x\right)}{K_{SN}}.$$

(33)

The multiple bitrate level of mobile video not only affects the resolution of the video, but is also related to the size of the video file. Owing to the service time of RSU for vehicle users is limited, it may cause the vehicle user to only complete a small part of the download task within the time of the RSU service range, which means bad QoE. When the vehicle user leaves the RSU service coverage area, MBS will continue to distribute the unfinished part of the replicas of video. This is very likely to be difficult to meet the QoE requirements of the corresponding bitrate replicas of video due to the low MBS transmission rate, resulting in serious video service fail. To deal with the above problems, it introduces the evaluation index of the ADP of vehicle users into the QoE evaluation model, then proposes an ADP weighted MV-MOS model that comprehensively considers the completion of the video download task. The calculation formula of ADP for mobile video is expressed as

$$P_{ADP}\left(f_i,{\ell }_j,{\mathcal{R}}_a\left(t_y^x\right)\right)=\left\{\begin{array}{c}\frac{{\displaystyle \sum _{z=1}^{t_{SN}}}{\mathcal{R}}_a\left(t_{y,z}^x\right)t_{y,z}^x}{g_j},{\displaystyle \sum _{z=1}^{t_{SN}}}{\mathcal{R}}_a\left(t_{y,z}^x\right){∆t}_{y,z}^x<g_j\\ 1,{\displaystyle \sum _{z=1}^{t_{SN}}}{\mathcal{R}}_a\left(t_{y,z}^x\right){∆t}_{y,z}^x\ge g_j\end{array}\right.,$$

(34)

where ${\mathcal{R}}_a\left(t_y^x\right)$ is transmission rate, $g_j$ is file size of the video bitrate level ${\ell }_j$. When vehicle users obtain video from the MECS, the $P_{MEC}\left(t_y^x\right)$ can be expressed as

$$P_{MEC}\left(t_y^x\right)=\sum _{i\in F}\sum _{j\in L}{x}_{f_i,{\ell }_j}\left(t_y^x\right)P_{\mathcal{F}}^x\left(f_i\right)P_{ADP}\left(f_i,{\ell }_j,{\mathcal{R}}_{MEC}\left(t_y^x\right)\right).$$

(35)

When a vehicle user obtains a video from MBS through the backhaul at the time slot $t_y^x$, the $P_{BH}\left(t_y^x\right)$ is calculated as

$$P_{BH}\left(t_y^x\right)=\sum _{i\in F}\sum _{j\in L}{x}_{f_i,{\ell }_j}\left(t_y^x\right)P_{\mathcal{F}}^x\left(f_i\right)P_{ADP}\left(f_i,{\ell }_j,{\mathcal{R}}_{Bh}\left(t_y^x\right)\right).$$

(36)

Hence, the average ADP at time slot $t_y^x$ is calculated as

$${\overline{P}}_{ADP}\left(t_y^x\right)=\sum _{K=0}^{\mathbb{E}\left[U\left(t_y^x\right)\right]}{P}_{Ar}(\mathbb{E}\left[U\left(t_y^x\right)\right],K)\left(\frac{K{P}_{BH}\left(t_y^x\right)}{\mathbb{E}\left[U\left(t_y^x\right)\right]}+\left(\frac{\left(\mathbb{E}\left[U\left(t_y^x\right)\right]-K\right)P_{MEC}\left(t_y^x\right)}{\mathbb{E}\left[U\left(t_y^x\right)\right]}\right)\right),$$

(37)

In summary, the ADP weighted MV-MOS at the time slot $t_y^x$ is expressed as

$$J\left(t_y^x\right)={\overline{P}}_{ADP}\left(t_y^x\right)Q_{MOS}\left(t_y^x\right).$$

(38)

4. Problem Formulation

4.1. ADP Weighted Mobile Video Mean Opinion Score

This section, we focus on the problem of intelligent caching for mobile video streaming in the IoV. In mobile networks, the mobile user’s location is considered static, and then the QoE evaluation method cannot be applied to the dynamic scene of the vehicular network. To adapt to the vehicle’s fast mobility scenario, we formulate the problem of mobile video caching in IoV as an optimized expression under multiple constraints to maximize the ADP weighted MV-MOS. The objective function can be formulated as

$$\begin{array}{c}\mathcal{P}:argmaxJ\left(t_y^x\right),\hfill \end{array}$$

(39a)

$$\begin{array}{c}\mathrm{s}.\mathrm{t}.\sum _{j\in L}{x}_{f_i,{\ell }_j}\left(t_y^x\right)\le 1,\hfill \end{array}$$

(39b)

$$\begin{array}{c}\sum _{i\in F}\sum _{j\in L}{g}_j{x}_{f_i,{\ell }_j}\left(t_y^x\right)\le {\phi }_{MEC},\hfill \end{array}$$

(39c)

$$\begin{array}{c}{\mathcal{R}}_{Back}\left(t_{y,z}^x\right)\le {\mathcal{R}}_{MEC}\left(t_{y,z}^x\right),\hfill \end{array}$$

(39d)

$$\begin{array}{cc}\hfill {\mathcal{R}}_{BH}\left(t_{y,z}^x\right)& =min\left\{{\mathcal{R}}_{Back}\left(t_{y,z}^x\right),{\mathcal{R}}_{MBS}\left(t_{y,z}^x\right)\right\},\hfill \end{array}$$

(39e)

where (39b) is a restriction of multiple bitrate copies. In the same time slot, the MECS can only caching one replica of video of the same video content. (39c) is a restriction of the storage capacity of the MECS, (39d) is transmission rate constraint, (39e) is bandwidth constraint, and optimization problem (39a) is a dynamic optimization problem under multi-dimensional constraints.

4.2. Average Mobile Video Mean Opinion Score

For better experimental comparison, this section formula a sub-problem $\mathcal{P}1$ for the optimization problem of Formula (39a), which is to relax (39a) and strip the ADP weighting term to replacement optimization objective is Formula (33), so the multi-dimensional constraint expression can be formulated as

$$\begin{array}{c}\mathcal{P}1:argmax{Q}_{MOS}\left(t_y^x\right),\hfill \end{array}$$

(40a)

$$\begin{array}{c}s.t.\sum _{j\in L}{x}_{f_i,{\ell }_j}\left(t_y^x\right)\le 1,\hfill \end{array}$$

(40b)

$$\begin{array}{c}\sum _{i\in F}\sum _{j\in L}{g}_j{x}_{f_i,{\ell }_j}\left(t_y^x\right)\le {\phi }_{MEC},\hfill \end{array}$$

(40c)

$$\begin{array}{c}{\mathcal{R}}_{Back}\left(t_{y,z}^x\right)\le {\mathcal{R}}_{MEC}\left(t_{y,z}^x\right),\hfill \end{array}$$

(40d)

$$\begin{array}{cc}\hfill {\mathcal{R}}_{BH}\left(t_{y,z}^x\right)& =min\left\{{\mathcal{R}}_{Back}\left(t_{y,z}^x\right),{\mathcal{R}}_{MBS}\left(t_{y,z}^x\right)\right\}.\hfill \end{array}$$

(40e)

In fact, $\mathcal{P}1$ is a simplification of $\mathcal{P}$, then constraint conditions of $\mathcal{P}1$ the same as $\mathcal{P}$, but only the optimization objective is different. So the parameters definition and constraint conditions of (40) are the same as (39).

5. Deep Reinforcement Learning-Based Caching Solution

The video content caching problem in this section is formulated as MDP. A novel caching scheme DQN-based is proposed to achieve maximum ADP weighted MV-MOS, to solve this complex problem. Furthermore, we also improve the convergence speed of DQN and enhance the performance of proposed algorithm. The MDP can be represented by the 4-tuple $<\mathcal{S},\mathcal{A},\mathcal{E},\mathcal{R}>$. $\mathcal{S}$ is the set of environment states, $\mathcal{A}$ denotes the set of agent actions, $\mathcal{E}$ represents the state transition probability, and $\mathcal{R}$ indicates the reward function.

5.1. Markov Decision Process

At the beginning, the agent will observe environment information. The environment state is represented as

$$s_{t_y^x}=\left\{\lambda \left(t_y^x\right),\mu \left(t_y^x\right),X\left(t_{y-1}^x\right)\right\},$$

(41)

The action $a_{t_y^x}$ is represented as

$$a_{t_y^x}=\left\{X\left(t_y^x\right)\right\},$$

(42)

After an action $a_{t_y^x}$ is taken, the reward $\mathcal{R}\left(s_{t_y^x},a_{t_y^x}\right)$ is represented as

$$\mathcal{R}\left(s_{t_y^x},a_{t_y^x}\right)=argmax{\mathbb{E}}^{\pi }\left\{\sum _{y=0}^{\infty }{\omega }^{t_y^x}\mathcal{R}\left\{s_{t_{y+1}^x}\left|s_{t_y^x},a_{t_y^x}\right.\right\}\right\},$$

(43)

where ${\omega }^{t_y^x}\in \left(0,1\right]$ is discounted factor. Hence, the optimal caching strategy ${\pi }^{*}$ in the IoV is expressed as

$${\pi }^{*}\left(s_{t_y^x}\right)=argmaxQ\left(s_{t_y^x},a_{t_y^x}\right).$$

(44)

In summary, the agent perceives the environment state $s_{t_y^x}$, then selects and executes an action $a_{t_y^x}$, after the system environment will feedback an immediate reward $\mathcal{R}\left(s_{t_y^x},a_{t_y^x}\right)$, it is represented as

$$\mathcal{R}\left(s_{t_y^x},a_{t_y^x}\right)=J\left(t_y^x\right).$$

(45)

Otherwise, the system environment will feedback on a cost (e.g., −1).

5.2. DRL-Based Caching Algorithm

The high mobility of vehicle users leads to environmental information dynamic changes in vehicular networks. To cope with the challenge, we consider a novel algorithm that effectively solves excessive state and action space in a dynamic environment by combining deep learning and reinforcement learning to solve the complex caching problem. Especially, as the update rule of the DQN does not require knowledge about the transition and reward functions. Therefore, we proposed the P3DVQC to solve the caching problem. The parameters of the P3DVQC main networks update formula is expressed as

$${\theta }_{t_{y+1}^x}=\alpha \left(\gamma Q\left(s_{t_{y+1}^x},\underset{a_{t_{y+1}^x}}{max}Q\left(s_{t_{y+1}^x},a_{t_{y+1}^x};{\theta }_{t_y^x}\right);{\theta }_{t_y^x}^{-}\right)+{\theta }_{t_y^x}-Q\left(s_{t_y^x},a_{t_y^x};{\theta }_{t_y^x}\right)\right)\nabla Q\left(s_{t_y^x},a_{t_y^x};{\theta }_{t_y^x}\right)+\mathcal{R}\left(t_y^x\right),$$

(46)

where $\alpha$ is learning rate, $\gamma$ is discount factor, ∇ is gradient operator. The P3DVQC is integrated with prioritized experience replay (PER) technology to achieve priority sampling by changing sampling distribution to improve the performance of the DQN. The P3DVQC sampling weight formula is expressed as

$$P_{PER}^i=\frac{{\rho }_{PER}^i}{{\displaystyle \sum _{\mathrm{i}\in \Omega }}{\rho }_{PER}^i},$$

(47)

where ${\rho }_{PER}^i$ is priority parameter of experience pool sampling i, it can be expressed as

$${\rho }_{PER}^i={\left({∆E}_{TD}\left(t_{y+1}^x\right)+\epsilon \right)}^{\varrho },$$

(48)

where $\epsilon$ is disturbance coefficient, ${∆E}_{TD}\left(t_{y+1}^x\right)$ is a distance of $<s_{t_y^x},a_{t_y^x},\mathcal{R}\left(t_y^x\right),s_{t_{y+1}^x}>$ between output and target value. It is calculated as

$${∆E}_{TD}\left(t_{y+1}^x\right)=\left|\gamma Q\left(s_{t_y^x},\underset{a_{t_{y+1}^x}}{max}Q\left(s_{t_{y+1}^x},a_{t_{y+1}^x};{\theta }_{t_y^x}\right);{\theta }_{t_y^x}^{-}\right)+\mathcal{R}\left(t_y^x\right)-Q\left(s_{t_y^x},a_{t_y^x};{\theta }_{t_y^x}\right)\right|,$$

(49)

where ${\theta }_{t_y^x}$ is the parameter of P3DVQC main network, ${\theta }_{t_y^x}^{-}$ is parameter of P3DVQC target network, $|.|$ is absolute operation. In addition, P3DVQC also integrated dueling technology to improve the deep Q-learning algorithm. The framework of the proposed P3DVQC is shown in Figure 3. The target network output function is expression as

$$Q\left(s_{t_y^x},a_{t_y^x};{\theta }_{t_y^x},{\alpha }_{t_y^x},{\beta }_{t_y^x}\right)=V\left(s_{t_y^x};{\theta }_{t_y^x},{\beta }_{t_y^x}\right)+A\left(s_{t_y^x},a_{t_y^x};{\theta }_{t_y^x},{\alpha }_{t_y^x}\right),$$

(50)

where ${\alpha }_{t_y^x}$ is parameter of advantage function $A\left(s_{t_y^x},a_{t_y^x};{\theta }_{t_y^x},{\alpha }_{t_y^x}\right)$, ${\beta }_{t_y^x}$ is parameter of value function $V\left(s_{t_y^x};{\theta }_{t_y^x},{\beta }_{t_y^x}\right)$. The pseudo code of the proposed DRL-based caching algorithm is provided in Algorithm 1.

Algorithm 1 Prioritized experience replay Dueling Double DQN IoV QoE Caching (P3DVQC)

1: 1. Initialize: 2: $s_{t_y^x}\in \mathcal{S}$, $a_{t_y^x}\in \mathcal{A}$, $\alpha$, $\gamma$, the size of minibatch $\mathcal{G}$ and the memory pool $\mathcal{N}$ 3: Parameters ${\theta }_{t_y^x}$ and ${\theta }_{t_y^x}^{-}$ 4: Parameters ${∆E}_{TD}\left(t_y^x\right)$ 5: Parameters $V\left(s_{t_y^x};{\theta }_{t_y^x},{\beta }_{t_y^x}\right)$, $A\left(s_{t_y^x},a_{t_y^x};{\theta }_{t_y^x},{\alpha }_{t_y^x}\right)$, 6: 2. Learning: 7: for$t_y^x\in K_c$do 8:    Choose the $\pi \left(t_y^x\right)$ by $\epsilon -greedy$ policy 9:    Observe environment, evaluate and estimate $s_{t_y^x}$ 10:    Perform $a_{t_y^x}$ according to $\pi$ and observe feedback 11:    while $t_{y,z}^x\in T_s$ do 12:       Observe: environment and communication state 13:       Calculate: $\mathcal{H}\left(t_y^x\right)$, ${\left|h_r\left(t_{y,z}^x\right)\right|}^2$, ${\left|h_{bs}\left(t_{y,z}^x\right)\right|}^2$, ${\mathcal{R}}_{MEC}\left(t_{y,z}^x\right)$, ${\mathcal{R}}_{MBS}\left(t_{y,z}^x\right)$ 14:       Calculate: $Q_{MEC}\left(t_{y,z}^x\right)$,$Q_B\left(t_{y,z}^x\right)$ 15:       Obtain: ${\overline{Q}}_{AV}\left(U\left(t_{y,z}^x\right),N\left(t_{y,z}^x\right)\right)$, $Q_{Aver}\left(t_{y,z}^x\right)$ 16:    end while 17:   Calculate: $\mathcal{R}\left(s_{t_y^x},a_{t_y^x}\right)$,$s_{t_{y+1}^x}$ 18:    Store: transition $<s_{t_y^x},a_{t_y^x},\mathcal{R}\left(t_y^x\right),s_{t_{y+1}^x}>$ 19:    if $ER{P}_t>N_{max}$ then 20:      for $j\in N_F$ do 21:         Calculate: ${\rho }_{PER}^i$, sort by SumTree, and sample $<s_{t_y^x},a_{t_y^x},\mathcal{R}\left(t_y^x\right),s_{t_{y+1}^x}>$ 22:         Update: ${∆E}_{TD}\left(t_{y+1}^x\right)$ and function parameters 23:         Update: ${\theta }_{t_{y+1}^x}$ with SGD algorithm 24:         Update: Q, V, A, ${\theta }_{t_y^x}^{-}$, and network parameters 25:      end for 26:    end if 27: end for

6. Numerical Results and Discussion

The simulations of the caching scheme are carried out in this section. The performance of P3DVQC was compared with the baseline schemes. The simulation environment for the mobile video caching system in IoV was programmed in Python. In addition, the TensorFlow platform was used to implement the P3DVQC caching scheme based on the open-source package convolutional neural networks. The main system parameters used in the simulations are summarized in

Table 1. System Parameters.

Notation	Definition	Parameter	Value
$P_{bs}$	40 dbm	${\phi }_{MEC}$	$[500,2000]$ Mb
$\gamma$	$\left[0.1,0.9\right]$	$\beta$	$[0.5,0.8]$
${\mu }_m$	$[0.4,1.5]$	$\alpha$	$\left(0,1\right]$
$ϵ$	4	${\delta }^2$	$-102$ dBm
$P_r$	33 dbm	L	$[5,10]$
$\theta$	$\pi /6$	$C_{MBS}$	$[3,6]$ Mbps
$\mathcal{F}$	$[10,50]$	$C_{BM}$	$[10,100]$ Mbps

. For performance comparison, five benchmark schemes were presented:

(1) Resolution Optimal Caching Scheme (ROCS): The scheme always priority caches the video copies with the highest bitrate level to realize the rapid delivery of high-quality video content and improve the quality of experience of vehicle users.

(2) Fluency Optimal Caching Scheme (FOCS): The scheme caches the video copies with the lowest bitrate to cache as many videos as possible and realizes the diversity of the cache to improve the cache hit rate.

(3) Random Caching Scheme (RCS): The scheme selects the video copies with the random algorithm, which means an equal probability choose each copy cached until the maximum cache capacity of MECS is reached.

(4) Cost Efficient Scheme (CES): The scheme only depends on the backhaul network delivery of video files, not to use of MECS caching equipment to minimize device cost and energy consumption.

(5) Brute Force Scheme (BFS): The scheme is optimal, but it is obtained based on ideal conditions where all system information is known. In actual scenarios, system information, such as wireless channel state information, cannot be obtained in advance. Therefore, BFS cannot carry out in real vehicular network scenarios.

6.1. Algorithm Analysis

Figure 4 shows the performance of the two DRL-based caching algorithms proposed. One is based on the traditional deep Q-network IoV caching algorithm (DVQC), and another is the P3DVQC algorithm which integrates the prioritized experience replay, the dueling, and the double deep Q-network technologies. The size of the experience replay pool is 3000 in the simulation. Figure 4 also shows that the P3DVQC completes the convergence at nearly 5000-time slots, while the DVQC at about 6000-time slots. That is due to the P3DVQC employing the PER technology improving the deep neural network training efficiency. In addition, the P3DVQC shows more stability than DVQC at the convergence state, this is beacuse the P3DVQC integrated DDQN and Dueling technology, over-estimation of DQN is effectively eliminated, avoiding unnecessary misselections to achieve rapid convergence to the optimal caching strategy.

Figure 5 shows the convergence performance comparison between the P3DVQC and the DVQC. Before 3000 time slots is the experience replay pool initialization stage. After 3000 time slots, the P3DVQC achieves a faster convergence rate than DVQC. That is due to the P3DVQC employing the PER to achieve TD weighting of the experience pool samples, which makes deep neural networks more effective. The P3DVQC algorithm determines the priority of the sample selected for training according to the size of TD errors to improve the training effect and speed up the convergence speed. However, the DVQC algorithm selects samples for training through a uniform sampling strategy, so the the convergence speed is slower than P3DVQC.

6.2. Average MV-MOS Performance

Figure 6 shows the average MV-MOS performance comparison between the P3DVQC scheme and the five baseline schemes. BFS is the ideal upper bound algorithm of the experiment, it is obtained by traversing all the solution spaces and requires knowing all the system information, which requires a lot of time overhead to search the solution space, hardly realized in the actual scene. The CES scheme does not use any caching technology, so its performance is the lowest. Secondly, the RCS chooses video content caching by the random algorithm, resulting in the performance being very unstable. The FOCS only caches the lowest bit rate video. Because the lowest bit rate video file is small, the FOCS can cache more different video content varieties, but it cannot effectively provide high-definition video. ROCS caches high-bitrate HD videos as much as possible, so vehicle users can achieve high average MV-MOS if the required HD video is pre-cached in MECS. However, the size of high bitrate video content is larger, so the higher storage overhead leads to a lower cache hit rate and performance. The P3DVQC proposed by this paper comprehensively considers the multiple bitrate video quality, backhaul bandwidth, and caching capacity to maximize objective. Therefore, the caching strategy obtained by the P3DVQC is that it caches a mixture of low and high bitrates. Figure 6 shows that the proposed can converges to the upper bound, and its performance is better than other benchmark algorithms.

6.3. ADP Weighted MV-MOS Performance

Figure 7 shows that ADP weighted MV-MOS performance indicators by different caching schemes. The ROCS scheme achieves a poor performance in ADP weighted MV-MOS, almost the same as the CES algorithm. This is because the video’s bitrate is directly proportional to the file size. A high bitrate means a large storage cost. Hence, when the service time is fixed, the ROCS can only complete a small proportion of tasks, which will cause vehicle users to only enjoy a short time of the high-definition video within the service area. It will immediately interrupt when it leaves the caching service area. Therefore, based on the average MV-MOS model cannot be directly applied to the vehicular networks. Figure 7 also shows that the ADP weighted MV-MOS performance of the P3DVQC can converges to the optimal BFS, which is better than other benchmark algorithms.

Figure 8 shows the cumulative reward of the five algorithms in Figure 7. The BFS is a straight line in Figure 7, so the cumulative gain is meaningless. To increase the caching hit rate, FOCS only caches the lowest bitrate videos so that its solution space is much smaller than the P3DVQC, so the cumulative return obtained in the early stage is more significant. However, with the P3DVQC converging to the optimal strategy, it achieves a greater cumulative return than the FOCS. Other benchmark algorithms can be seen from the analysis of Figure 7, and their performance is lower than the P3DVQC.

7. Conclusions

In this paper, a DRL-based P3DVQC algorithm is proposed to solve the mobile video caching problem while achieving the maximum ADP weighted MV-MOS. The numerical results show that compared with other benchmark schemes, the proposed has a faster convergence speed and significant performance.