LFDC: Low-Energy Federated Deep Reinforcement Learning for Caching Mechanism in Cloud–Edge Collaborative

Abstract

The optimization of caching mechanisms has long been a crucial research focus in cloud–edge collaborative environments. Effective caching strategies can substantially enhance user experience quality in these settings. Deep reinforcement learning (DRL), with its ability to perceive the environment and develop intelligent policies online, has been widely employed for designing caching strategies. Recently, federated learning, when combined with DRL, has been in gaining popularity for optimizing caching strategies and protecting data training privacy from eavesdropping attacks. However, online federated deep reinforcement learning algorithms face high environmental dynamics, and real-time training can result in increased training energy consumption despite improving caching efficiency. To address this issue, we propose a low-energy federated deep reinforcement learning strategy for caching mechanisms (LFDC) that balances caching efficiency and training energy consumption. The LFDC strategy encompasses a novel energy efficiency model, a deep reinforcement learning mechanism, and a dynamic energy-saving federated policy. Our experimental results demonstrate that the proposed LFDC strategy significantly outperforms existing benchmarks in terms of energy efficiency.

1. Introduction

In recent years, with the rapid development of cloud computing and the Internet of Things (IoT), the combination of cloud and edge has become one of the research hotspots. Traditional cloud computing has the advantage of strong resource service capability but the disadvantage of distant transmission [1], while edge computing has the advantage of low transmission latency but limited resource constraints [2]. Cloud–edge integration refers to a computing mode that combines cloud-computing resources with edge-computing resources. The goal is to make more efficient use of edge-computing resources, while also fully leveraging the advantages of cloud computing [3,4].

Caching plays a pivotal role in reducing bandwidth traffic and the number of remote server accesses, thereby enhancing the system response time and reliability in the collaborative environment of cloud–edge computing [5]. However, caching in cloud–edge environments poses several challenges. Firstly, limited hardware resources impose constraints on cache capacity [6,7]. Secondly, restricted network bandwidth may result in prolonged response times for the cache when cache hit rates are low [8]. Cloud–edge caching is typically divided into cloud cache and edge cache, and a caching policy is employed to ensure that popular content is cached as much as possible at the edge. However, existing cloud–edge caching policies face two major problems. Firstly, due to the heterogeneous resources and communication links of the edge base station (BS), such caching strategies can become exceedingly complex. Secondly, the dynamics of user demands lead to real-time changes in content popularity, necessitating time-varying cache content. This requires caching policies capable of adapting in real time to the dynamic caching environment. Therefore, effectively managing caching in real time to improve cache hit rates and reduce response time remains a persistent and crucial challenge.

DRL, an AI model that can dynamically generate policies based on the environment, has seen successful applications in various cloud–edge collaborative environments, including wireless access network mode selection and resource management [9], autonomous driving strategy [10], and computation offloading optimization [11]. In the context of cache optimization, DRL methods have been utilized to improve cache efficiency by predicting content popularity [12] and dynamically adjusting cache policies [13,14,15]. Despite these achievements, using DRL-based caching strategies in cloud–edge environments still poses two significant challenges: (1) limited hardware resources, which leads to non-negligible energy consumption during DRL training, and (2) the need to jointly schedule cloud and edge storage resources for caching, which raises security and privacy concerns. Federated learning can be conducted at local BSs for learning and inference, and models can be aggregated and distributed in the cloud. Therefore, under federated learning, each BS can take advantage of its local characteristics while also learning from the experiences of other BSs, allowing it to adjust caching policies in real-time while maintaining data privacy. Building on the work of [16] that proposed dynamically adjusting the number of local iterations depending on resource utilization in federated learning, we propose a new federated deep reinforcement learning framework that can effectively reduce energy consumption while ensuring cache performance through the sharing and collaboration of models among multiple nodes. The federated nature of the learning framework maximizes data security and prevents data leaks. This paper’s main contributions include the following:

(1)

Creating a system model for the cache environment that combines cloud and edge, including network, cache performance, DRL, and energy-consumption models, and proposing a new objective function of the cache energy efficiency ratio to balance performance and energy consumption.

(2)

Designing a new federated deep reinforcement learning method that dynamically adjusts cloud aggregation and edge training or decision making to reduce energy consumption while ensuring cache efficiency.

(3)

Presenting a series of simulation experiments that demonstrate the effectiveness of the proposed strategies in reducing the training energy consumption of the system while maintaining cache performance.

This article is structured as follows. In Section 2, we conduct a comprehensive review of related literature. Section 3 outlines the system’s model and problem definition. Section 4 presents our proposed federated mechanism for achieving energy-efficient caching. The effectiveness of our proposed strategies is validated through simulation experiments in Section 5. Finally, Section 6 concludes the paper by summarizing our contributions.

2. Related Works

Research on the cache-enabled cloud–edge collaborative architecture can be broadly categorized into two groups.

The first group employs traditional methods based on convex optimization or probabilistic modeling to solve the content cache placement problem in cloud–edge environments. This category includes classic algorithms, such as least recently used (LRU) [17], first in first out (FIFO) [18], and least frequently used (LFU) [19]. However, these methods disregard the popularity trend of the content, which can result in significant performance degradation [12]. K. Poularakis et al. [20] optimize the hit rate by jointly optimizing content placement and user association. Poularakis et al. [21] propose a greedy service placement algorithm that maximizes the number of service requests by considering collaborative service placement, but it is not able to account for random requests in a dynamic environment. Similarly, Yang et al. [22] address the joint service caching and load scheduling problem to minimize the average request delay. Zhang et al. [23] employ probabilistic caching in heterogeneous networks to enhance traffic offloading and optimize edge device cache resources. Since cloud–edge collaboration is constantly evolving, the content cache problem in cloud–edge environments requires continuous optimization. Nevertheless, these strategies are often challenging to adapt to the dynamic cloud–edge environment.

The second category of research pertains to machine learning algorithms, such as machine learning, deep learning, and federated learning [24]. This approach involves learning key attribute features in the cloud–edge environment, including user request behavior, content popularity, and user mobility distribution, in order to optimize content cache strategies. Reinforcement learning [25] and deep reinforcement learning [15] exhibit significant potential in designing content cache strategies. Specifically, Rathoret et al. [26] propose a proactive caching framework based on deep learning that achieves significant improvements in feedback link and QoE. Wan et al. [27] employ an improved DRL method to optimize caching for enhancing video streaming quality of service. To mitigate the excessive network resource consumption of deep reinforcement learning during the transmission of training and testing data, Wang et al. [13] propose a caching framework based on federated deep reinforcement learning. Li et al. [12] present the CVC video caching framework based on federated deep reinforcement learning to reduce communication costs.

While machine-learning-based methods can achieve higher caching efficiency by adapting to the caching environment, they themselves consume significant amounts of energy. Reducing training energy consumption offers three benefits: (1) enhancing system efficiency alleviates base station energy burden, and lowers operational costs; (2) improving device reliability and stability, and reducing maintenance and replacement costs; and (3) reducing the system’s carbon footprint, supporting environmentally responsible edge computing. To address this issue, we propose a novel federated deep reinforcement learning mechanism that incorporates energy-saving properties to reduce the energy consumption associated with machine learning.

3. System Model and Problem Formulation

This section models the network topology, cache hit rate, deep reinforcement learning and its energy consumption model, respectively, and presents a cache energy efficiency ratio problem. Some of the key parameters are shown in

Table 1. Key notations and descriptions.

Notation	Description
$\mathcal{N}=\left\{1,\dots ,n,\dots ,N\right\}$	Set of BSs
$\mathcal{U}=\left\{1,\dots ,u,\dots ,U\right\}$	Set of UEs
$\mathcal{F}=\left\{1,\dots ,f,\dots ,F\right\}$	Set of content
$\mathcal{T}=\left\{1,\dots ,t,\dots ,T\right\}$	Epochs of federated learning/caching decision.
$v_{u,n}$,$v_a$,$v_b$	Transmission rate between UEs and BSs, BSs and Cloud, BSs and BSs.
$D_n$, $D_f$	The size of the content in BSn, the size of the content f.
$P_{u,f}$	UE u preferences for content f.
${\Re }_u$, ${\Re }_n$	Number of requests from UE u, number of requests received by BSn.
$H_{t,n}$, $H_{t,system}$	The cache hit efficiency of BSn in epoch t, The cache hit efficiency of system in epoch t.
$E_{t,n}^{cmp}$, $E_{t,n}^{up}$	The energy consumption respectively, of local learning and upload in epoch t.
$E_{t,system}$	The system energy consumption for learning in epoch t.
$s_i=(s_{i,u}^r,s_{i,n}^c)$	The caching state.
$\varphi \left(s_i\right)=\{a_i^{local},a_i^{BS-BS},a_i^{cloud}\}$	The caching decision.
$R_{i,n}(s_i,\varphi \left(s_i\right))=H_{i,n}$	The Double Deep Q-Network (DDQN) reward.
$Loss\left(w_i\right)$	The loss function of local training.
${\nabla }_{w_i}loss\left(w_i\right)$	gradient update formula for $w_i$.
${\beta }_{t,n}$	The weight factor for adaptive iteration times.
$D_{t,n}$	Amount of data for a single training session in BSn in epoch t.

.

3.1. Networks Model

As shown in Figure 1, in our network architecture, all data are sourced from cloud servers. The core network communicates with the cloud through a backhaul network and with edge base stations (BSs) through frontier networks. Each BS is equipped with an access point that provides wireless connectivity to users. When the requested content is available in the cache of a BS within its service range, it is referred to as a cache hit. $\mathcal{N}=\left\{1,\dots ,n,\dots ,N\right\}$ represents the set of BSs, each with a cache capacity of $C_n$ for storing content. $\mathcal{U}=\left\{1,\dots ,u,\dots ,U\right\}$ represents the set of user equipment (UE), randomly distributed within the service range of the BSs. $\mathcal{F}=\left\{1,\dots ,f,\dots ,F\right\}$ represents the set of content items, each with its request frequency $r_{u,f}$. $\mathcal{T}=\left\{1,\dots ,t,\dots ,T\right\}$ represents the epochs of aggregation/distribution for federated learning.

For the sake of illustration, we assume that the size of all content is equal to M bits. This assumption has been widely considered in previous studies (e.g., [28,29,30]) because large contents are often divided into roughly equal-sized chunks. BSs can directly communicate with other BSs to retrieve the requested content from neighboring BSs. Since retrieving content from neighboring BSs is faster and less costly than retrieving it from cloud servers, priority is given to retrieving content from the cloud last. There are three states when a user requests content, namely local hit, retrieval from a neighboring BS, and retrieval from the cloud. That is, when a request arrives at a local BS, if the requested content is cached locally, the local BS immediately returns the cached content. If the local BS misses, it retrieves the content from a neighboring BS and returns it to the user. If the neighboring BS also misses, the content is retrieved from the cloud to serve the request.

3.2. Cache Performance Model

The caching performance is primarily determined by the quality of user service, namely, the latency in accessing the content. To this end, it is necessary to establish an overall delay model for cloud–edge caching and to ensure that the requested content by the user is represented as a hit in its local BS (resulting in minimum content transfer delay). Similar to [31], the transmission medium of content between the BS and UE is wireless and is determined by the Shannon formula:

$$v_{u,n}=Blog(1+\frac{q_u{g}_{u,n}}{\varpi +{\displaystyle \sum _{i\in U\backslash \left\{u\right\}:a_i=a_u}}{q}_i{g}_{i,n}})$$

(1)

where B represents the channel bandwidth, $\varpi$ represents the background noise, $q_u$ represents the transmission power between the BS_n and the UE u, and $g_{u,n}$ represents the channel gain. $a_i=a_u$ represents that UE u has the content it needs.

Caching content raises the issue of content popularity. High content popularity indicates that users request the content more frequently, while low content popularity indicates that users request the content less frequently. Therefore, content popularity can directly affect cache performance. As a result, edge BSs should cache more popular content. In this article, content popularity is defined as a measure of user preference for content f. Similar to [13], $d_{U-B}$ is defined as the transmission delay between the user and the BS.

$$d_{U-B}=\sum _{f\in \mathcal{U}}\sum _{f\in \mathcal{F}}{P}_{u,f}\frac{D_f}{v_{u,n}}$$

(2)

where $D_f$ represents the size of content f, and $P_{u,f}$ is the UEs’ preferences for content f:

$$P_{u,f}=r_{u,f}/{\Re }_u$$

(3)

$$\begin{array}{l}subjectto:\\ f\in \mathcal{F}\\ {\displaystyle \sum _{f\in \mathcal{U},f\in \mathcal{F}}}{P}_{u,f}=1\end{array}$$

${\Re }_u$ represents the total number of requests made by UE u, while $r_{u,f}$ represents the number of requests made by UE u for content f. Similarly, we can obtain the transmission delay between the cloud and the edge:

$$d_{C-B}=\sum _{f\in \mathcal{U}}\sum _{f\in \mathcal{F}}{P}_{u,f}\frac{D_f}{v_a}$$

(4)

$v_a$ denotes the transmission rate from the cloud to the edge BS, and similarly, by setting $v_b$ as the transmission rate between BSs, we can obtain the transmission delay between BSs:

$$d_{B-B}=\sum _{f\in \mathcal{U}}\sum _{f\in \mathcal{F}}{P}_{u,f}\frac{D_f}{v_b}$$

(5)

The cache hit rate can directly evaluate the effectiveness of caching policies and reflect the performance of caching systems. For cloud–edge caching systems, the cache hit efficiency is another way of expressing cache efficiency, which is more representative of the efficiency of caching in real scenarios because it is closely related to the transfer latency in various cases [15]. As the edge has a faster response time than the cloud, caching at the edge for a requested content is considered a cache hit. Taking into account the issue of transmission delay, in the t-th round of federation, the cache hit efficiency for all contents within a single BS can be represented as

$$H_{t,n}=\sum _{u\in \mathcal{U}}\sum _{f\in \mathcal{F}}\frac{d_{U-B}}{d_{U-B}+x_{f,B}{d}_{B-B}+x_{f,C}{d}_{B-C}}$$

(6)

where $x_{f,B}$ and $x_{f,C}$ are decision variables representing whether the requested content f comes from the neighboring BS ($x_{f,B}=1$ if yes, 0 otherwise), or the cloud ($x_{f,C}=1$ if yes, 0 otherwise). Therefore, the formula for the cache hit efficiency of the system in the t-th epoch of federation is as follows:

$$H_{t,system}=\sum _{n\in \mathcal{N}}{H}_{t,n}$$

(7)

$$\begin{array}{l}subjectto:\\ f\in \mathcal{F}\\ n\in \mathcal{N}\\ u\in \mathcal{U}\\ x_{B}and{x}_c\in \left\{0,1\right\}\\ {\sum }_{f\in \mathcal{F}}{P}_{u,f}=1\end{array}$$

Based on the paper [30], it is known that maximizing the cache hit efficiency, i.e., minimizing the average delay of the system, is an NP-complete problem. We prove this in the next section. Moreover, achieving satisfactory results through heuristic algorithms such as GREEDYAD is difficult [31]. The difficulty in obtaining effective strategies with heuristic algorithms for edge caching can be attributed to their inability to adequately address dynamic environments, heterogeneous resources, scalability, and collaboration requirements. Therefore, we introduce an approach to optimize the caching efficiency through DRL methods.

3.3. Np-Complete Proof

Theorem 1.

The cloud–edge caching problem is an NP-complete problem.

Proof of Theorem 1.

If the set cover (SC) problem for the NP-complete set can be reduced to a cache placement problem, then we can prove that the decision problem for cache placement is NP-complete. Firstly, we define two sets, the first being set $G=\left\{g_1,g_2,\dots ,g_I\right\}$, and the second being $H=\left\{h_1,h_2,\dots ,h_J\right\}$, representing the subsets of G, where each element $h_j$ has a weight. The objective of the SC problem is to select a subset from H such that the union of all elements in the subset equals the set G, and the sum of the weights is either minimized or maximized.

The steps to simplify the SC problem to the cache placement problem are as follows: Firstly, set the number of contents to 1. Let $G=\left\{g_1,g_2,\dots ,g_I\right\}$ represent all BSs, and let $H=\left\{h_1,h_2,\dots ,h_J\right\}$ represent the different ways to content caching on BSs. For instance, $h_1=\left\{1,3,5\right\}$ denotes caching content on $g_1$, $g_3$, and $g5$, while $h_2=\left\{2,4,6\right\}$ denotes caching content on $g_2$, $g_4$, and $g_6$. Then, let the cache value and link cost respectively denote the weight of each subset.

Therefore, our problem can be formulated as an SC problem as follows: we have a set $V=\left\{v_1,v_2,\dots ,v_N\right\}$ for all edge servers, and each element in the set $C=\left\{c_1,c_2,\dots ,c_K\right\}$ represents all possible caching decisions for a piece of content on all BSs. For each caching decision $c_k$, there is a cache hit efficiency. The cache hit efficiency of caching decision $c_k$ is the content caching revenue obtained with $c_k$. The objective of our problem is to find a subset $C^{\prime }=\left\{c_1,c_2,\dots ,c_K^{\prime }\right\}$, which is a subset of C, and the union of all elements in $C^{\prime }$ equals V such that the sum of all the content caching revenues is maximized.    □

The set cover problem can be simplified to the content caching placement problem through the transformation described above. By integrating practical variables into the framework of the set cover problem, we identified that these practical variables correspond to the general variables in the set cover problem. In essence, our problem takes into account the same factors as the set cover problem and can be considered a variation of it. Consequently, the data caching problem is classified as an NP-complete problem.

3.4. Drl Energy Consumption Model

In the context of cloud computing, the significant performance advantages of cloud-based operations outweigh the energy consumed in transmitting and aggregating model parameters. Accordingly, this study concentrates solely on the energy expenditure involved in training at the edge, encompassing the local training energy consumption and the energy consumption associated with the transmission of model parameters at the edge. As demonstrated in the research conducted by [32], the computational burden is approximately proportional to the data volume in a nearly linear fashion. Specifically, assuming that a base station (BS) trains a single data sample, it necessitates $c_n$ CPU cycles per bit. Taking into account the training data volume $D_t$, the total energy consumption of $B{S}_n$ during the t-th training epoch is expressed in accordance with [33,34]:

$$E_{t,n}^{cmp}=L_n{E}_{t,n}^{cmp}=L_n{\zeta }_n{c}_n{D}_{t,n}fr{e}_n^2$$

(8)

$$\begin{array}{l}subjectto:\\ L_n\ge 1\\ n\in {\mathcal{N}}_1\\ {\mathcal{N}}_1\subset \mathcal{N}\end{array}$$

The effective capacitance coefficient of the computing chip group of BS_n is denoted as ${\zeta }_n$, while $L_n$ represents the count of local training iterations conducted by BS_n. To compute the local model, BS_n is required to perform CPU cycles. Based on Equation (8), it is apparent that the energy expenditure associated with local training can be appropriately regulated by adjusting the number of local training iterations $L_n$. Assuming that $P_n$ signifies the transmission power of $B{S}_n$, the energy consumption involved in uploading the model can be derived as follows:

$$E_{t,n}^{up}=P_n\frac{{\wp }_{t,n}}{v_a}$$

(9)

$$\begin{array}{l}subjectto:\\ n\in {\mathcal{N}}_3\\ {\mathcal{N}}_3\subset \mathcal{N}\end{array}$$

Here, ${\wp }_n$ represents the model size in $B{S}_n$. The total energy consumption of the system in the t-th epoch is

$$E_{t,system}=\sum _{n\in {\mathcal{N}}_1}{E}_{t,n}^{cmp}+\sum _{n\in {\mathcal{N}}_2}{E}_{t,n}^{up}$$

(10)

$$\begin{array}{l}subjectto:\\ {\mathcal{N}}_1\subset \mathcal{N}\\ {\mathcal{N}}_3\subset \mathcal{N}\end{array}$$

In this context, ${\mathcal{N}}_1$ denotes the group of BSs that necessitate training during the t-th epoch, while ${\mathcal{N}}_3$ signifies the set of BSs that require model aggregation during the same epoch. The selection of ${\mathcal{N}}_1$ and ${\mathcal{N}}_3$ shall be performed utilizing a dynamic and energy-efficient federated approach, which will be elaborated comprehensively in Section 4.

3.5. Problem Formulation

This section is devoted to the investigation of optimizing cache hit efficiency and base station (BS) selection for online training with the aim of maximizing the ratio of cache efficiency to training energy consumption. The total number of federated aggregation epochs is represented by T. To improve the cache performance while minimizing training energy consumption, this study proposes a hit efficiency to energy (HE) gain metric, defined as the ratio of the hit efficiency to the energy consumption of the selected BSs. The objective is formulated as follows:

$$P1:max\sum _{t\in \mathcal{T}}\frac{H_{t,system}}{E_{t,system}}$$

(11)

$$\begin{array}{l}subjectto:\\ {\mathcal{N}}_1\subset \mathcal{N}\\ {\mathcal{N}}_3\subset \mathcal{N}\\ L_n\ge 1\\ f\in \mathcal{F}\\ n\in \mathcal{N}\\ u\in \mathcal{U}\\ x_{B}and{x}_c\in \left\{0,1\right\}\\ {\sum }_{f\in \mathcal{F}}{P}_{u,f}=1\end{array}$$

The application potential of energy efficiency ratios, such as the one proposed herein, is considerable. Specifically, the aim of attaining reliable training efficiency while simultaneously reducing energy consumption in the context of online training can be expanded to a broader range of scenarios.

4. Low-Energy Federated Deep Reinforcement Learning for Caching

This section first designs the local iterative DRLs, followed by a systematic planning of the approach for optimizing federated learning.

4.1. Local DRL Model Design

When a request for content is received by a local base station (BS) from user equipment (UE), the cloud–edge system conducts a search for the content and subsequently delivers it to the UE. The content may be stored in the local BS, neighboring BSs, or the cloud. The objective is to cache the content proactively prior to user requests in order to ensure high cache hit efficiency. To achieve this goal, the proactive caching policy is updated in each epoch of global federation, based on the local deep reinforcement learning (DRL) training performance of the BS. Therefore, it is essential to design the training process of the local DRL. Generally, the design of DRL involves three steps: (1) designing the system state, action, and reward; (2) defining the caching policy based on the system state, action, and reward; and (3) continuously optimizing the caching policy through online iteration.

DRL distinguishes itself from reinforcement learning (RL) by utilizing deep network stacking, which enhances the generalization capability of the trained neural network as the sample data grow larger. The first step in DRL is to define the state $s_i$ in the environment, followed by selecting an action a based on $s_i$ and the action probability. This is then followed by a new state $s_{i+1}$ and a reward R. In the context of the cloud–edge collaborative caching environment, the cache state is defined for content f at each decision i. We model the content replacement process in the cache of BS as an Markov decision process (MDP) [35]. The states, system actions, and rewards of the cache and request are shown below:

$$s_i=(s_{i,u}^r,s_{i,n}^c)$$

(12)

Let $s_{i,u}^r:=s_{i,u,f}^r,f\in \mathcal{U},f\in \mathcal{F}$ and $s_{i,n}^c:=s_{i,n,f}^c,n\in \mathcal{N},f\in F$ denote the content request state and content cache state, respectively. $s_{i,u,f}^r=1$ and $s_{i,n,f}^c=1$ indicate that the UE u requests content f and content f has been cached in BS_n, respectively. Each iteration i has F elements, and the set of epochs is the cache state over a period of time. This also indicates the cache state required for one iteration during online training.

Since the UE can obtain the requested content in the local BS, neighboring BSs or the cloud, there are three types of actions for the agent in a decision epoch: $a_i^{local}$ replacing the content to the local edge node, $a_i^{BS-BS}$ neighbouring edge nodes collaborating on the content request, and $a_i^{cloud}$ processing the content request in the cloud. Then the action policy under state $s_i$ is defined as

$$\varphi \left(s_i\right)=\{a_i^{local},a_i^{BS-BS},a_i^{cloud}\}$$

(13)

where $a_i^{local}=\{a_{i,1}^{local},a_{i,2}^{local},\dots ,a_{i,f}^{local}\}$ denotes the local processing action, $a_{i,f}^{local}\in \{0,1\},$$f\in \mathcal{F}$, and $a_{i,f}^{local}=1$ indicates that content f needs to be replaced by the requested content in iteration i; otherwise, $a_{i,f}^{local}=0$. If the requested content is not cached in the local BS, it needs to be processed in the neighboring BS. $a_i^{BS-BS}=\{a_{i,1}^{BS-BS},a_{i,2}^{BS-BS},\dots ,a_{i,N}^{BS-BS}\}$ denotes the processing action of the neighboring BSs, $a_{i,n}^{BS-BS}\in \{0,1\}$, and $a_{i,n}^{BS-BS}=1$ indicates that the requested content is provided in BS_n. If the UE is unable to get its requested content in either BS, the local BS has to decide whether to forward the request to the cloud for processing. Let $a_i^{cloud}\in \{0,1\}$, and $a_i^{cloud}=1$ means the request is handed over to the cloud for processing.

After an action is performed under state $s_i$, a reward is obtained. From formula (7), it can be seen that the reward of the cache should maximize the cache hit efficiency. Therefore, the cache reward $R_{i,n}(s_i,\varphi \left(s_i\right))$ is represented in iteration i as follows:

$$R_{i,n}(s_i,\varphi \left(s_i\right))=H_{i,n}$$

(14)

when making caching decisions, it is necessary to ensure that each BS does not cache more files than its capacity. The capacity limits are defined as follows:

$$\sum _{f\in \mathcal{F}}{s}_{i,n,f}^c{C}_f\le C_n$$

(15)

MDP is usually defined as a transition ${\delta }_i=\{s_i, \varphi \left(s_i\right),R_{i,n}(s_i,\varphi \left(s_i\right)),s_{i+1}\}$ under DRL. By learning in a series of transitions ${\delta }_i$, DRL can give the best strategy to maximize the return in the current state. The action–value function Q function is used to evaluate the expectation starting at state $s_i$. Then, the Q function is defined as

$$Q(s_i,\varphi \left(s_i\right))={\mathbb{E}}_{\varphi }\left[\sum _{i=1}^{\infty }{\gamma }^{i-1}R(s_i,\varphi \left(s_i\right))|s_i;\varphi \right]$$

(16)

where $\gamma \in (0,1]$ is the factor discounting future rewards. The optimal value $Q^{*}(s_i,\varphi \left(s_i\right))={max}_{\varphi }{Q}_{\varphi }(s_i,\varphi \left(s_i\right))$, and $Q^{*}(s_i,\varphi \left(s_i\right))$ can be written as follows:

$$Q^{*}(s_i,\varphi \left(s_i\right))={\mathbb{E}}_{s_{i+1},\varphi }\left[R_i+\gamma {max}_{\varphi }{Q}^{*}\left(s_{i+1},\varphi \right)|s_i,\varphi \right]$$

(17)

A local BS training model was specifically developed utilizing the double Q-learning (DDQN) algorithm as described in [36], with a focus on states, actions, and rewards as depicted in Figure 2. Two deep neural networks were employed to learn an equation $Q(s_i,\varphi \left(s_i\right);w)$ that approximates $Q^{*}(s_i,\varphi \left(s_i\right))$. The objective of $Q^{*}(s_i,\varphi \left(s_i\right);w)$ is $y_i=R_i+\gamma {max}_{\varphi }Q(s_{i+1},\varphi ;w)$. Consequently, the parameters w can be updated to optimize $Q^{*}(s_i,\varphi \left(s_i\right))$.

Figure 2 illustrates the training process of DDQN, where every individual local BS maintains a finite-size experience replay pool, which is refreshed with the latest transitions. The Q network (MainNet) of the deep reinforcement learning (DRL) framework is leveraged to determine system actions, while the $\widehat{Q}$ network (TargetNet) is utilized to evaluate the selected actions. The $\widehat{w}$ in the $\widehat{Q}$ network is periodically updated with $w_i$ in the Q network. During the training process, the agent randomly selects a minibatch ${\delta }_i$ from the experience replay pool and trains the Q network by minimizing the loss function for each iteration,

$$loss\left(w_i\right)={\mathbb{E}}_{{\delta }_i}\left[{\left(R(s_i,\varphi )+\gamma \widehat{Q}\left(s_i,\underset{{\varphi }_{i+1}}{\arg \max }Q(s_{i+1},{\varphi }_{i+1};w_i);\widehat{w_i}\right)-Q(s_i,{\varphi }_i;w_i)\right)}^2\right]$$

(18)

Thus, we can obtain the gradient update formula for $w_i$,

$$\begin{array}{c}\hfill {\nabla }_{w_i}loss\left(w_i\right)={\mathbb{E}}_{{\delta }_i}[(R(s_i,{\varphi }_i)+\gamma \widehat{Q}(s_i,\underset{{\varphi }_{i+1}}{\arg \max }Q(s_{i+1},{\varphi }_{i+1};w_i);w_{i+1})-\\ \hfill Q(s_i,{\varphi }_i;w_i))·{\nabla }_{w_i}Q(s_i,{\varphi }_i;w_i)]\end{array}$$

(19)

The online iterative process used by DDQN for caching is represented by Algorithm 1. First, each BS initializes MainNet and TargetNet based on its historical information and obtains the initial reply memory $\Omega$ (shown as lines 1–5). Next, DDQN is executed to train the caching process and update all parameters provided that there is no requested content f in BS_n and the BS_n storage is full (shown as lines 8–17).

Algorithm 1 Local DDQN process for caching.

1: for each BSn do   2:    Initialize replay memory $\Omega$.   3:    Initialize action-value function Q with random weights $w_0$.   4:    Pretraining the MainNet and TargetNet by $w_0$ and $\widehat{w_0}$ from local historic information.   5:    Save $\{s_0,{\varphi }_0,R_0,s_1\}$ in $\Omega$.   6: end for   7: for each BSn, each iteration $i\in L_n$ do   8:    $B{S}_n$ receive a request content f.   9:    if $s_{i,n,f}^c=1$ then 10:      Provide content directly f to UE u. 11:    else 12:      Receive the current state $s_i$, and execute ${\varphi }_i^{*}=argmaxQ(s_i,\varphi \left(s_i\right);w_i)$. 13:      Observe current reward $R_i$ by formula (14) and next state $s_{i+1}$. 14:      Store transition $\{s_i,{\varphi }_i,R_i,s_{i+1}\}$ to $\Omega$. 15:      Sample random mini-batch ${\delta }_i$ of $\Omega$. 16:      Update MainNet $w_i$ and TargetNet $\widehat{w_i}$ by gradient descent as formula (19). 17:      Update the caching state $s_i$. 18:    end if 19: end for

4.2. Federated Mechanism

Owing to its distributed and privacy-preserving attributes, federated learning has garnered widespread adoption in domains such as cloud computing, edge computing, and device-to-device (D2D) computing [37]. By leveraging the federated approach to optimize the training models of multiple agents, superior efficiency can be achieved as compared to centralized training, while ensuring complete privacy. Consequently, this manuscript puts forth a novel dynamic energy-conserving federated mechanism to tackle the challenge of the energy efficiency ratio in cache for cloud–edge systems.

Due to the regional characteristics of local BSs, their training strategies are more tailored to local demands. In order to ensure that BSs with local advantages do not lose their cache efficiency due to the federated mechanism, it is also necessary to provide sufficient training for BSs with lower cache efficiency. The basic idea of the proposed mechanism is to determine the aggregation and training of BSs based on their data request quantity and cache hit efficiency. Specifically, the cache hit efficiency of each BS is calculated at the end of each epoch t. If a BS has a lower data request quantity, it will not participate in the aggregation process (which will not affect the effectiveness of the global model) and save sent power, while BSs with higher cache hit efficiency will not participate in online training (as local BSs have their own cache advantages at this time).

This approach has two issues. On the one hand, some BSs may have a higher hit efficiency due to receiving fewer requests, but the model is not sufficiently trained. Aggregating the model at this point will reduce the performance of the global model. On the other hand, some BSs may have a lower hit efficiency due to receiving too many content requests, but this type of model has a strong generalization ability and can significantly improve the performance of the global model. To this end, we dynamically partition the local BSs and incorporate a mechanism for adjusting the local iteration count dynamically to ensure cache efficiency while reducing energy consumption.

Assuming that each BS possesses adequate computational capabilities to execute the training tasks, the dynamic federated mechanism proposed in this study is deployed across both cloud and edge BSs, as demonstrated in Figure 3 and Algorithm 2. The said mechanism comprises three distinct steps.

Algorithm 2 Dynamic low-power federated DRL for caching (LFDC).

Initialized process in BSs:   1: Initialize $w_{0,n}=w_0$.   2: Initialize number of local iterations $L_0$, global epoch T, and step size $\eta$.   3: Pretrained parameters $w_{0,n}$ from Algorithm 1 (lines 1–5).   4: for$t=1,2,3,\dots ,T$do Aggregate model and select BSs process in cloud:   5:    Receive $w_{t,n}$ from ${\mathcal{N}}_3$ BSs.   6:    Update global model $w_{t+1}$ according to (26).   7:    Computing weight factor $\beta$ by (22), and set $L_{t,n}=\left[{\beta }_{t,n}{L}_0\right]$.   8:    ${\mathcal{N}}_1=\left\{n\right|{\Re }_n>{\Re }_{threshold}$∧$H_n<H_{threshold}\}$, ${\mathcal{N}}_2=N\backslash {\mathcal{N}}_1$.   9:    Send training requests and global models to BSn that need to participate in training (${\mathcal{N}}_1$). Local online training process in BSs: 10:    for each BSn do 11:      if $n\in {\mathcal{N}}_1$ then 12:         Processing online training by Algorithm 1 (lines 7–19) 13:         Set local parameter $w_{t,n}$ by (23). 14:      else 15:         Processing strategy of $t-1$ epoch in BSn (${\mathcal{N}}_2$). 16:      end if 17:      Computing ${\Re }_n$ and $H_n$ of BSn. 18:      ${\mathcal{N}}_3=\left\{n\right|{\Re }_n>{\Re }_{threshold}\}$. 19:      for $n\in {\mathcal{N}}_3$ do 20:         Sending models to the cloud. 21:      end for 22:    end for 23: end for

(1) BSs are selected for participation in training by defining two thresholds: the request quantity threshold (${\Re }_{treshold}$) and the cache hit efficiency threshold ($H_{threshold}$) (lines 8, 12). Commencing from the $t_1$-th iteration, the cloud selects the BSs (${\mathcal{N}}_1$) that have received more requests than the request quantity threshold in the previous t − 1 iterations. Next, the BSs with cache hit efficiency lower than the efficiency threshold are chosen for online training (lines 11–13), while other BSs (${\mathcal{N}}_2$) adopt the cached strategy obtained after the previous $t_1$-th iteration of training (line 15). This strategy facilitates continued training for BSs with low caching efficiency and fully reflects the local content request characteristics for BSs with higher efficiency, thereby sustaining a high cache performance. Simultaneously, as the number of BSs participating in online training in each round diminishes, the overall energy consumption of the system reduces. $w_{t,n}$ represents the model parameters in epoch t. For the local dataset $D_{n,t}$ of each BS, its model loss function is given by

$$F_n\left(w_{t,n}\right)=\frac{1}{D_{t,n}}\sum _{d\in D_{t,n}}{f}_{d,n}\left(w_{t,n,d}\right)$$

(20)

$$\begin{array}{l}subjectto:\\ {\mathcal{N}}_1\leftarrow \left\{n\right|{\Re }_{t-1,n}>{\Re }_{threshold},H_{t-1,n}<H_{\mathrm{threshold}}\}\end{array}$$

where ${\mathcal{N}}_1$ represents the set of BSs that require local online training. Therefore, the local problem is

$$w_{t,n,d}=\arg \min F_n\left(w_{t,n}\right|w_{t-1,n})$$

(21)

that is, to minimize the local loss function relative to the previous epoch (lines 10–18).

(2) A novel strategy for the dynamic adjustment of local iteration times in BS models that participate in local online training. Our approach aims to obtain the neighboring BSs experience without compromising the unique local characteristics of each BS model. To achieve this, we propose dynamically adjusting the local iteration times based on the difference between the model parameters obtained from the previous global training (t − 1) and the current global model parameters at time t. Specifically, we define a weight factor $\beta$ as the basis for this dynamic adjustment strategy (line 7).

$${\beta }_{t,n}=\frac{\sqrt{{\displaystyle \sum _i^I}(w_{t,n}-w_t)}}{\frac{1}{N}\sum _{n\in \mathcal{N}}\sqrt{{\displaystyle \sum _i^I}(w_{t,n}-w_t)}}$$

(22)

The dimensionality of the model parameters is denoted by I. Thus, for BS_n, the number of local training iterations is determined by $L_{t,n}=\left[{\beta }_{t,n}{L}_0\right]$, where $\left[{\beta }_{t,n}{L}_0\right]\ge 1$. Models with large differences in parameter values have lower global performance and may not meet the dynamic requirements of requests, therefore requiring more local iterations ($L_n$) to ensure their dynamic performance. Additionally, reducing the number of local iterations can lead to lower energy consumption of the system, as shown in Equation (8). When a local model of a participant is close enough to the global model, the number of local training rounds for this participant can be appropriately reduced to reduce the training energy consumption. Local parameter updates can be expressed as follows:

$$w_{t+1,n}=w_{t,n}-\eta \nabla F_n\left(w_{t,n}\right)$$

(23)

(3) The conventional approach to model aggregation involves weighting the training data size of each model to create a global model. However, in cloud–edge caching environments, the number of content requests and the cache hit rate are more critical technical indicators. This is because, for any BS_n, as the number of content requests increases, the caching strategy must adapt more dynamically to maintain request efficiency. Additionally, a higher cache hit rate implies a higher cache efficiency for the current BS_n. In light of this, we propose using aggregation weights that take into account the importance of the cache hit efficiency because $H_t,n$ is the sum of the cache hit efficiency of all requests received by BS_n at the t-th epoch:

$${\varpi }_{t,n}=\frac{H_{t,n}}{{\displaystyle \sum _{n\in \mathcal{N}}}{H}_{t,n}}$$

(24)

The global objective of FL is to minimize the global loss function:

$$\underset{w}{min}{F}_t\left(w\right)=\frac{1}{D_t}{\displaystyle \sum _{n\in {\mathcal{N}}_2}}{\displaystyle \sum _{i\in D_{t,n}}}{f}_{i,n}\left(w\right)$$

(25)

$$\begin{array}{l}subjectto:\\ {\mathcal{N}}_3\leftarrow \left\{n\right|{\Re }_n>{\Re }_{threshold}\},D_t={\displaystyle \sum _{n\in {\mathcal{N}}_3}}{D}_{t,n}\end{array}$$

Let ${\mathcal{N}}_3$ be the set of BSs that require model aggregation, characterized by a content request number exceeding a predetermined threshold ${\Re }_{threshold}$. The models associated with these base stations must be transmitted to the cloud for aggregation. At time t, the total data quantity can be represented by $D_t={\displaystyle \sum _{n\in {\mathcal{N}}_3}}{D}_{t,n}$, where $D_{t,n}$ refers to the data amount associated with each base station. Therefore, the parameter update of the global model at this stage can be expressed as (lines 5–9)

$$w_{t+1}=\frac{{\displaystyle \sum _{n\in {\mathcal{N}}_3}}{\varpi }_{t,n}{w}_{t+1,n}}{{\displaystyle \sum _{n\in {\mathcal{N}}_3}}{\varpi }_{t,n}}$$

(26)

It should be noted that the edge BSs are divided into two categories: BSs for online training and BSs with fixed models. The BSs involved in online training need to receive global model parameters from the cloud and update their cache strategies in real time through DRL. The cache strategy for fixed-model BSs remains unchanged. The energy consumption of sending models from BSs is explained in Section 3.3. Both BSs for online training and BSs with fixed models need to calculate their cache hit efficiency after each round of local training. The online training BSs and model invariant BSs at the edge can be dynamically switched based on changes in the cache hit efficiency and content request number.

5. Simulation Experiments

In this section, we compare the performance of the proposed LFDC algorithm in terms of cache hit efficiency, energy consumption, and HE gain with centralized DRL ([38]), distributed DRL ([13]), the classical federated learning algorithm FedAvg ([39]), and traditional caching algorithms, including first in first out (FIFO, [18]), least frequently used (LFU, [19]), and least recently used (LRU, [17]). Furthermore, we evaluate the performance of the proposed algorithm under different hyperparameters. The following are three comparison algorithms also based on DRL:

(1)

Centralized DRL: A DDQN model is deployed in the cloud to train and make decisions on global dynamic caching.

(2)

Distributed DRL: A DDQN model is deployed in each BS, which is trained and makes decisions based on local data.

(3)

FedAvg: A DDQN model is deployed in each BS and aggregated and distributed through the federated averaging method.

5.1. Simulation Setting

We implemented the proposed LFDC in Python 3.9, with PyTorch set up on a 64-bit CentOS Linux workstation with an Intel Xeon W-2155 CPU, 128 GB RAM, and 2080Ti GPU. In the simulation, there are $\left|N\right|=10$ BSs with $\left|U\right|=500$ randomly distributed within the coverage area of the BSs. The cache capacity of each BS is 1 GB. The maximum coverage radius of each BS is 500 m, with a channel bandwidth of 20 MHz and a transmission power of 500 W. The path loss $g_{u,n}$ is modeled as $30.6+36.7lg\left(d{t}_{u,n}\right)dB$, where $d{t}_{u,n}$ denotes the distance between UE u and BS_n.

Moreover, to validate the effectiveness of the proposed algorithm in practical scenarios, we obtain content from actual datasets. We collect content request records from the publicly available Movielens dataset. MovieLens consists of 27,753,444 ratings from 283,228 users on 58,098 movies, as well as related information about the movies, such as their titles and genres. We assume that the number of ratings is equal to the number of requests, and the time when a review is posted is considered to be the time when a request is initiated. We record the tags of each video as video features and divide the videos into several sub-files $C_f$ of equal size, based on their duration. To simulate users’ mobility, we set the mobility coefficient to 0.1 so that after each epoch, a portion of the users served by the base station (BS) will be randomly replaced. For each BS selected for local training, the initial value of $L_0$ for the number of local training iterations is set to 10, with a minibatch size of 32. The optimizer used is Adam, with a step size of 0.01. We use a dataset consisting of 300 h of data for experimentation, with each epoch being one hour, and Pandas and Requests are used to retrieve and process the data. Other parameters for the experimental setup are shown in

Table 2. Parameter values.

Parameter	Value	Description
T	1000	Number of global epoch
$L_0$	10	Number of initial local iterations
$c_n$	20 cycles/bit	CPU cycles per bit for training one data sample
$fre$	4 GHz	Computation capacity of BSs
$P_n$	500 W	Transmit power of BSn
B	20 MHz	Channel bandwidth of BSs downlink
$v_a$	1000 Mbps	Transmission speed between cloud and BSs
$v_b$	1 Gbps	Transmission speed between BSs
$D_f$	10 Mbit	Content size
${\zeta }_n$	$1.2\times {10}^{-28}$	Effective capacitance coefficient of BSn
${\wp }_n$	$1\times {10}^4$ bit	Parameter size of model
$\gamma$	0.9	Discount factor
$\eta$	0.01	Step size
$ϵ$	0.1	State transition probability

.

5.2. Evaluation Results and Analysis

The experiment initially compared the cache hit efficiency of the LFDC algorithm with other algorithms. As shown in Figure 4a, with the increase in global epoch numbers (i.e., online training/online caching time), the LFDC exhibited improved caching performance. Learning-based algorithms initially underperformed compared to traditional cache optimization algorithms, but they gradually demonstrated better cache performance with ongoing online training. In contrast, the cache hit rates of FIFO, LFU, and LRU fluctuated over time and, due to their inability to adapt to the environment, had lower cache hit efficiency than learning-based algorithms. LFDC essentially converged at 300 global epochs, while other learning algorithms converged earlier, which is attributed to the fact that some BSs did not participate in training in each epoch. Due to the optimization of the federated process, the system cache hit efficiency of LFDC had a clear advantage over other learning algorithms. Starting from the 100th epoch, the average cache hit efficiency of LFDC was 23.87%, 12.51%, 13.66%, 55.27%, 64.36%, and 38.03% higher compared to Central DRL, distributed DRL, FedAvg, FIFO, LFU, and LRU, respectively. As shown in Figure 4b, as the number of UEs increases, the number of content requests also increases. The proposed LFDC algorithm likewise outperforms other algorithms. However, in Figure 4a,b, the curve trends are contrary. This is because, under the fixed cache capacity of the BS, as the number of less-popular new request content increases, a large portion of the new request content is not replaced by the local BS.

Figure 5 compares the training energy consumption of LFDC with distributed DRL and FedAvg. As the number of epochs increases, LFDC exhibits superior energy performance compared to the other two learning algorithms. This is because, firstly, FedAvg requires all BSs to continuously undergo online training and constantly upload their parameters. Although distributed DRL does not involve parameter uploading, its BSs are always engaged in online training, and the number of local training iterations remains constant. The LFDC method, by selecting suitable BSs for training, choosing appropriate parameters from these BSs for uploading, and dynamically adjusting the number of local training iterations, achieves better online training energy efficiency than the other learning algorithms.

Figure 6 investigates the HE gain of the proposed LFDC, where T increases from 20 to 1000. Overall, after 1000 epochs, the proposed LFDC achieves the highest HE gain of 1.988439791 compared to the existing distributed DRL (1.078518039) and FedAvg (0.947555). This is because, in the environment deploying the LFDC algorithm, BSs obtain the experiences of other BSs while catering to local request characteristics. Furthermore, the dynamic federated strategy reduces the training energy consumption. In other words, LFDC can achieve a higher training energy efficiency ratio compared to other classical algorithms.

Figure 7 presents the comparison results of the cache hit efficiency (a), training energy consumption (b), and HE gain (c) for the LFDC algorithm without an adaptive local iteration count. As can be seen in Figure 7a, when the adaptive local iteration count is removed, i.e., the local iteration count is fixed at 10, the cache hit efficiency of LFDC shows a slight improvement. This is because the increased local iteration count can better adapt to the local content request situation. Simultaneously, Figure 7b shows that the total training energy consumption of the LFDC algorithm without adaptive local iteration count increases by 27.19% compared to LFDC, resulting in a 25.03% decrease in the HE gain evaluation as seen in Figure 7c. In summary, the LFDC algorithm with adaptive local iteration count can significantly enhance the training energy efficiency ratio, which means it can provide better cache performance under the same training consumption rate.

Finally, we compare the cache performance and HE gain under different step sizes $\eta$ in Figure 8. As the learning rate increases, the performance of cache hit efficiency does not change significantly. However, when the step size factor $\eta$ is within the range of 0.001 to 0.01, the HE gain exhibits stable performance variations. Therefore, we use a step size of $\eta$ = 0.01 as an empirical value to maintain the stability and effectiveness of the algorithm.

6. Conclusions

In this paper, we studied the cloud–edge collaborative caching in cloud–edge integrated environments. To improve the cache hit efficiency and reduce training energy consumption, we proposed a low-training-energy federated deep reinforcement learning for caching (LFDC) algorithm. The algorithm optimizes cache decision making based on user preferences, and optimizes federated decision making based on cache hit efficiency and the number of UE requests after cache decision making. This algorithm can effectively reduce training energy consumption while ensuring cache hit efficiency and improving the efficiency ratio in cloud–edge collaboration.