Joint Latency-Oriented, Energy Consumption, and Carbon Emission for a Space–Air–Ground Integrated Network with Newly Designed Power Technology

Abstract

Ubiquitous connectivity is envisaged for the space–air–ground-integrated network (SAGIN) of future communication to meet the needs of quality of service (QoS), green communication, and “dual carbon” targeting. However, the offloading and computation of massive latency-sensitive tasks dramatically increase the energy consumption of the network. To address these issues, we first propose a SAGIN architecture with energy-harvesting devices, where the base station (BS) is powered by both renewable energy (RE) and the conventional grid. The BS explores wireless power transfer (WPT) technology to power an unmanned aerial vehicle (UAV) for stable network operation. RE sharing between neighboring BSs is designed to fully utilize RE to reduce carbon emissions. Secondly, on the basis of task offloading decisions, the UAV trajectory, and the RE sharing ratio, we construct cost functions with joint latency-oriented, energy consumption, and carbon emission. Then, we develop a twin delayed deep deterministic policy gradient (TD3PG) algorithm based on deep reinforcement learning to minimize the cost function. Finally, simulation results demonstrate that the proposed algorithm outperforms the benchmark algorithm in terms of reducing latency, energy saving, and lower carbon emissions.

1. Introduction

To accommodate the diverse quality-of-service (QoS) requirements of a wide range of applications in various scenarios, sixth-generation (6G) communication networks are envisioned to offer low latency and ubiquitous connectivity [1]. They are also expected to provide simultaneous wireless connectivity that is 1000 times higher than that provided by fifth generation (5G) networks [2]. In particular, a space–air–ground integrated network (SAGIN) is considered as a promising paradigm for future communication to provide large-scale coverage and network performance enhancement [3].

On the other hand, mobile edge computing (MEC) is considered a key technology for future communication network service for latency-sensitive tasks [4,5], addressing the requirement for massive low-latency control (less than 1 ms end-to-end latency) for 6G networks [6]. However, the execution of tasks would consume a considerable amount of energy. Green communication in 6G should be manifested by reducing the total energy consumption in order to achieve an energy efficiency goal [7]. In addition, the dual carbon goal of future communication networks is to reduce carbon emissions by 50% [8]. Therefore a joint latency-oriented to guarantee QoS, energy consumption for green communication, and carbon reduction to achieve a dual carbon goal for SAGIN meets the needs of future communication network development [9,10].

Several studies have investigated how to reduce network latency in order to guarantee QoS. Peng et al. [11] considered two typical MEC architectures in a vehicular network and formulated multidimensional resource optimization problems, exploiting reinforcement learning (RL) to obtain the appropriate resource allocation decisions to achieve high delay/QoS satisfaction ratios. By jointly optimizing computation offloading decisions and computation resource allocation, Zhao et al. [12] effectively improved the system utility and computation time. Abderrahim et al. [13] proposed an offloading scheme that improves network availability while reducing the transmission delay of ultra-reliable low-latency communication packets in the terrestrial backhaul, satisfying different QoS requirements.

A few other studies have considered the issue of energy consumption in networks. The authors of [14,15] considered a multiuser MEC system with the goal of minimizing the total system energy consumption over a finite time horizon. Chen et al. [16] investigated the task offloading problem in an ultra-dense network and formulated it as a mixed-integer nonlinear NP-hard program, which can reduce 20% of the task duration with 30% energy savings compared with random and uniform task-offloading schemes.

Some scholars have concentrated their attention on latency and energy consumption in the SAGIN architecture. Guo et al. [17] investigated service coordination to guarantee QoS in SAGIN service computing and proposed an orchestration approach to achieve a low-cost reduction in service delay. To achieve a better latency/QoS for different 5G slices with heterogeneous requirements, Zhang et al. [18] intelligently offloaded traffic to the appropriate segment of the SAGIN. In order to ensure QoS for airborne users, as well as to reduce energy consumption, Chen et al. [19] designed an energy-efficient data-saving scheme, with extensive simulations confirming the effectiveness of the proposed scheme in terms of both energy consumption and processing delay in a SAGIN.

Although the aforementioned studies represented a considerable amount of exploration of the issues of latency reduction and energy consumption reduction, the research reported in [20] highlights that the dense deployment of high-energy-demand communication devices corresponds to increased energy consumption of wireless access networks. Moreover, future networks must have reduced carbon emissions. Fortunately, energy harvesting technology converts ambient energy to electric energy, which can be used to help reduce the carbon footprint of wireless networks [21]. In particular, a green approach to cellular base station (BS) has been proposed, which involves the adoption of renewable energy (RE) resources [22].

Consequently, the subject of utilizing RE to power BSs in order to reduce the carbon footprint of networks has attracted academic attention. Yuan et al. [23] proposed an energy-storage-assisted RE supply solution to power a BS, in which a deep reinforcement learning (DRL)-based regulating policy is utilized to flexibly regulate the battery’s discharging/charging. On the basis of off-grid cellular BSs powered by integrated RE, Jahid et al. [24] formulated a hybrid energy cooperation framework that optimally determines the quantities of RE exchanged among BSs to minimize both related costs and greenhouse gas emissions. Despite their advantages, RE harvesting technologies are still variable and intermittent compared to traditional grid power. An aggregate technology that combines RE with traditional grid power is the most promising option to reliably power cellular infrastructure [25].

However, existing research efforts are limited to one or two aspects of the network in terms of reducing latency, energy saving, or access to new power supply technology. Relatively little attention has been paid to entire SAGIN with integrated energy access technology. Apart from the issues considered above, there are still a number of challenges associated with a SAGIN. First of all, the long-term performance of the network needs to be considered, as the arrival, transmission, and processing of tasks represent a stochastic and dynamic process over a period of time. The use of DRL in artificial intelligence techniques can address stochastic and complex problems, as well as dynamic problems. Secondly, due to the fact that future communication is likely to comprise a cellular network capable of providing energy [26], it is extremely critical to power an unmanned aerial vehicle (UAV) in order to maintain the sustainability of the network. Fortunately, radio-signal-based wireless power transfer (WPT) can provide battery-equipped devices with a sustainable energy supply. For instance, the authors of [27,28,29,30] considered a wirelessly powered multiaccess network wherein a hybrid access point powers wireless devices via WPT.

Motivated by the above limitations and challenges, we inquire into two issues in SAGINs. One is how to combine latency-oriented, energy consumption, and carbon emissions to model the objective function in a SAGIN with new power access technology. The other is how the relevant influence factors affect latency, energy consumption and carbon emissions.

In this article, we first develop a SAGIN with a newly designed power supply technology. In order to maintain the sustainability of the network, the BS explores the WPT to power the UAV. The UAV is able to dynamically adjust its trajectory to cope with task processing and charging. The combination of latency, energy consumption, and carbon emissions is formulated as the cost function. Obviously, our research target is to minimize the cost function.

The main contributions of this article can be summarized as follows:

• We propose an architecture for a SAGIN with a newly designed power supply technology. This network offers ubiquitous connectivity while adapting to the requirements of high reliability and green communication.
• We develop a cost function with joint latency-oriented, energy consumption, and carbon emissions, which facilitates decreases in the latency of task processing, energy consumption, and the carbon footprint of the network through optimization.
• We put forward a twin delayed deep deterministic policy gradient (TD3PG) method. This DRL-based algorithm is capable of sensing parameter changes in the network and dynamically updates the offloading decision to minimize the cost function.
• We conduct experimental evaluations and comparisons. The proposed algorithm is compared with three benchmark algorithms. The results show that the proposed algorithm is outstanding in terms of reducing task latency, network energy savings, and lower carbon emissions.

The rest of this article is organized as follows. In Section 2, the system model with SAGIN is presented. Section 3 provides the problem formulation. In Section 4, a TD3PG algorithm is designed to obtain the optimal values. Section 5 demonstrates the simulation results and performance analysis. Finally, Section 6 is devoted to the conclusions.

2. System Model

A SAGIN that consists of M ground user devices, N BSs, K UAVs, and a low-earth-orbit (LEO) satellite is considered.

As shown in Figure 1, each BS, each UAV, and the LEO satellite is separately equipped with a server to provide the devices with computation services. Devices are randomly distributed on the ground, satisfying $m\in \mathcal{M}=\{1,2,\cdots ,M\}$. BSs are deployed on buildings with a height of $H_{BS}$ and satisfy $n\in \mathcal{N}=\{1,2,\cdots ,N\}$. The flight altitude of the UAV k is fixed as H. The communication range covers the preset range around the UAV, which satisfies $k\in \mathcal{K}=\{1,2,\cdots ,K\}$. The set of time slots is indicated as $t\in \mathcal{T}=\{1,2,\cdots ,T\}$, and the size of each time slot is $\tau$. The UAV acts as a mobile BS and does not take into account the function of a relay. The communication coverage of the UAV is smaller than that of the BS, and the communication range of the BS is smaller than that of the LEO satellite. Since one LEO satellite can sufficiently cover the area of the scenario, it is assumed that a LEO satellite is available within the communication coverage at all times. The meteorological environment remains constant. For ease of understanding, the key symbols used in this article are listed in

Table 1. Summary of key notations.

Notation	Definition
$\mathcal{M}$, $\mathcal{N}$, $\mathcal{K}$, $\mathcal{T}$	The set of devices, BSs, UAVs, and time slots
M, N, K, T	Number of devices, BSs, UAVs, and time slots
${\lambda }_m\left(t\right)$, $s_m\left(t\right)$, $e_m\left(t\right)$, $t_m\left(t\right)$	The arrival rate, data size, required number of CPU cycles per unit of data, and the latency constraint for the computation subtask of device m in time slot t
$L_n\left(t\right)$, $T_n\left(t\right)$, $D_n^{res}\left(t\right)$	The electrical load, communication traffic, and fixed standby time for BS n in time slot t
$R_{m,n}\left(t\right)$, $R_{m,k}\left(t\right)$, $R_{m,sat}\left(t\right)$	Wireless communication data rate in time slot t between devices and BSs, UAVs, and the LEO satellite, respectively
$B_{m,n}\left(t\right)$, $B_{m,k}\left(t\right)$	Bandwidth allocated to device m by BS n and UAV k in time slot t
$t_m^{l,wait}\left(t\right)$, $t_k^{u,wait}\left(t\right)$, $t^{s,wait}\left(t\right)$	Queuing delay for subtasks to be computed locally, offloaded to the UAV, and offloaded to the satellite for execution in time slot t
$f_m\left(t\right)$, $f_{m,n}\left(t\right)$, $f_{m,k}\left(t\right)$, $f_{m,sat}\left(t\right)$	Computational resources allocated to device m, BS n, UAV k, and the LEO satellite, respectively, in time slot t
$f_m^{max}$, $f_n^{max}$, $f_k^{max}$, $f_{sat}^{max}$	The total computational resources of device m, BS n, UAV k, and the LEO satellite, respectively
$p_m$, $\kappa$	Device upload power and the CPU effective switched capacitance
$v_k\left(t\right)$, ${\theta }_k\left(t\right)$, ${\Psi }_k\left(t\right)$, $v_{max}$, H	The average flying velocity, the flying directional angle, the horizontal position coordinates, the maximum speed, and the height of UAV k, respectively
$D_{min}$, ${\alpha }_0$, ${\beta }_0$	Safety distance among UAVs, the path loss index, and the channel gain at unit distance
$a_{m,k}\left(t\right)$, $o_{k,n}\left(t\right)$	Whether device m is within the communication range for UAV k and whether UAV k is within the WPT range for BS n in time slot t
$t_m^l\left(t\right)$,$E_m^l\left(t\right)$	Subtask execution time and energy consumption for local computation in time slot t
$t_n^{bs}\left(t\right)$,$E_n^{bs}\left(t\right)$	Subtask execution time and energy consumption for offloading to BS n in time slot t
$t_k^{uav}\left(t\right)$, $E_k^{uav}\left(t\right)$	Subtask execution time and energy consumption for offloading to UAV k in time slot t
$t_m^{sat}\left(t\right)$,$E_m^{sat}\left(t\right)$	Subtask execution time and energy consumption for offloading to the LEO satellite in time slot t
$B_k^H\left(t\right)$, $E_k^F\left(t\right)$	The harvested energy and flight energy consumption for UAV k in time slot t
${\sigma }^2$, ${\zeta }_{m^{\prime },n^{\prime }}\left(t\right)$, ${\zeta }_{m^{\prime },k^{\prime }}\left(t\right)$	Noise power and the interference from all other BSs and UAVs
$B^{bot}$, $B^{capacity}$	Minimum power threshold and maximum battery capacity for UAV batteries, respectively
$B^{res}\left(t\right)$, $B^{battery}$	Reserved energy in time slot t and maximum battery capacity for BS backup battery, respectively
$\iota$, $\eta$, $\chi$	Power division factor, loss factor for energy transfer, and energy conservation efficiency, respectively
$G_n^t$, $G_k^r$, $P_n$	Antenna power gains of the BS transmitter and the UAV receiver, BS n transmitting power
$P_0$, $P_1$, $U_{tip}$	Blade profile power in hovering state, induced power in hovering state, and tip speed of the rotor blade, respectively
$v_0$, $\rho$, A, $d_0$, s	Mean rotor-induced velocity in hover state, air density, rotor disc area, fuselage drag ratio, and rotor solidity, respectively
${\delta }_0$, ${\delta }_1$	Coefficients related to the electrical load for BSs
${\vartheta }_0$, ${\vartheta }_1$, ${\vartheta }_2$	Preference for latency, energy consumption, and carbon emissions, respectively
$T\left(t\right)$, $E^{ec}\left(t\right)$, $C^{RE}\left(t\right)$	The total task execution latency, the energy consumption for task execution and UAV flight, and the reduction in carbon emissions by reducing traditional power gird energy consumption in time slot t, respectively
$e_{uav}$, $e_{sat}$	Energy consumption per unit of computational resources of the UAV and LEO satellite

.

Each BS is independently equipped with solar panels, a wind turbine, a backup battery, and an electricity converter to provide uninterrupted energy supply. Adjacent BSs are able to share energy through an energy-sharing line. In order to maintain the stable operation of the UAV, the BS is able to supply the UAV with energy via WPT.

Given that virtual reality, augmented reality, and mixed reality are typical applications based on the 6G [31], it is assumed that each device has one total task that can be divided into a number of subtasks. The subtasks of device m can be represented as $〈s_m\left(t\right),e_m\left(t\right),t_m\left(t\right)〉$, where $s_m\left(t\right)$ and $e_m\left(t\right)$ denote the subtask’s size (bit) and required number of central processing unit (CPU) cycles per unit of data (cycles/bit) in time slot t, respectively; and $t_m\left(t\right)$ indicates the latency constraint for execution of the subtask, which contains the latency for the transmission, computation, and waiting of the subtask. Subtasks follow a Poisson distribution with an arrival rate of ${\lambda }_m\left(t\right)$.

The relevant system models in SAGIN are enumerated below.

2.1. Backup Battery Model for BS

$L_n\left(t\right)$ denotes the electrical load for BS n in time slot t, which is related to the communication traffic ($T_n\left(t\right)$) [32]. Hence, $L_n\left(t\right)$ can be expressed as

$$L_n\left(t\right)={\delta }_0+{\delta }_1{T}_n\left(t\right),$$

(1)

where ${\delta }_0$ and ${\delta }_1$ are coefficients.

The BS backup battery is required to reserve capacity $b_n^{res}\left(t\right)$ to meet the energy consumption need for the electrical load of BS n during the fixed standby time ($D_n^{res}\left(t\right)$) in time slot t. $b_n^{res}\left(t\right)$ can be represented as

$$b_n^{res}\left(t\right)={\int }_0^{D_n^{res}\left(t\right)}{L}_n\left(\tau \right)d\tau .$$

(2)

During time slot t, the RE harvested by N BS backup batteries is $\mathbf{H}\left(t\right)=[h_1\left(t\right),h_2\left(t\right),\dots ,$$h_N\left(t\right)]$. Energy sharing with the adjacent BS is denoted as ${\mathbf{E}}^{\mathbf{share}}\left(t\right)=[e_1^{share}\left(t\right),e_2^{share}\left(t\right),\dots ,$$e_N^{share}\left(t\right)]$. Let ${\mathbf{B}}^{\mathbf{res}}\left(t\right)=[b_1^{res}\left(t\right),b_2^{res}\left(t\right),\dots ,b_N^{res}\left(t\right)]$ denote the energy reserved by the backup battery to support the current communication traffic of the BS meets at a fixed standby time. ${\mathbf{E}}^{\mathbf{bs}}\left(t\right)=[e_1^{bs}\left(t\right),e_2^{bs}\left(t\right),\dots ,e_N^{bs}\left(t\right)]$ is the energy consumption of each BS. Suppose that the energy remaining in the backup battery is ${\mathbf{B}}^{\mathbf{bs}}\left(t\right)=[b_1^{bs}\left(t\right),b_2^{bs}\left(t\right),\dots ,b_N^{bs}\left(t\right)]$. Then, the capacity of the backup battery during the next time slot can be derived as

$${\mathbf{B}}^{\mathbf{bs}}(t+1)=min\{{\mathbf{E}}^{\mathbf{gird}}\left(t\right)+\mathbf{H}\left(t\right)+{\mathbf{B}}^{\mathbf{bs}}\left(t\right)-{\mathbf{E}}^{\mathbf{share}}\left(t\right)-{\mathbf{E}}^{\mathbf{bs}}\left(t\right),B^{battery}\},$$

(3)

where $B^{battery}$ is the capacity of the backup battery. ${\mathbf{E}}^{\mathbf{gird}}\left(t\right)$ is the energy supplied by power grid when RE harvesting is unable to maintain the BS energy consumption, which can be constructed as

$${\mathbf{E}}^{\mathbf{gird}}\left(t\right)=[{\mathbf{B}}^{\mathbf{res}}\left(t\right)+{\mathbf{E}}^{\mathbf{bs}}\left(t\right)-\mathbf{H}\left(t\right)+{\mathbf{E}}^{\mathbf{share}}\left(t\right)-{\mathbf{B}}^{\mathbf{bs}}{\left(t\right)]}^{+},$$

(4)

where ${\left[x\right]}^{+}=max\{x,0\}$. Considering the cost and size of BS energy harvesting equipment, the optimal unit size is finitely large. Battery lifetime ($B^{life}$) is a factor to be considered, which is related to the number of charge/discharge cycles. In this case, the following constraints are required:

$$\begin{array}{cc}\hfill & B^{battery}⩾{\int }_0^{D_n^{res}\left(t\right)}{L}^{max}d\tau ,\hfill \end{array}$$

(5a)

$$\begin{array}{cc}\hfill & CPR=\frac{B^{life}{B}^{battery}}{C^{battery}},\hfill \end{array}$$

(5b)

$$\begin{array}{cc}\hfill & C^{battery}\le C^{expect},\hfill \end{array}$$

(5c)

where $L^{max}$ is the maximum electrical load for the BS. $CPR$ is defined as the cost–performance ratio of the backup battery, which is as large as possible. $C^{battery}$ and $C^{expect}$ denote the actual and maximum expectation cost of the backup battery, respectively.

Furthermore, adjacent BSs are able to share energy through physical resistive power line connections to maximize RE utilization. This approach may be reasonable for energy sharing among BSs within small zones [24]. The remaining energy of the RE may be shared with other BSs. Then, energy sharing (${\mathbf{E}}^{\mathbf{share}}\left(t\right)$) can be expressed as

$${\mathbf{E}}^{\mathbf{share}}\left(t\right)={\upsilon }_n·{[\mathbf{H}\left(t\right)+{\mathbf{B}}^{\mathbf{bs}}\left(t\right)-{\mathbf{B}}^{\mathbf{res}}\left(t\right)-{\mathbf{E}}^{\mathbf{bs}}\left(t\right)]}^{+},$$

(6)

where ${\upsilon }_n$ is the RE transfer ratio with ${\upsilon }_n\in [0,1]$. When the backup batteries are fully charged, the excess RE (${\mathbf{E}}^{\mathbf{exc}}\left(t\right)=[e_1^{exc}\left(t\right),e_2^{exc}\left(t\right),\dots ,e_N^{exc}\left(t\right)]$) is insufficient to be sold to the power grid. Adding additional energy storage devices increases the cost. Therefore, the excess RE is discarded. The loss factor for energy transfer between BSs is $\eta$, which satisfies $\eta \in [0,1]$. Thus, the energy loss (${\mathbf{E}}^{\mathbf{loss}}\left(t\right)=[e_1^{loss}\left(t\right),e_2^{loss}\left(t\right),\dots ,e_N^{loss}\left(t\right)]$) can be expressed as

$${\mathbf{E}}^{\mathbf{loss}}\left(t\right)=\eta {\mathbf{E}}^{\mathbf{share}}\left(t\right)+{\mathbf{E}}^{\mathbf{exc}}\left(t\right).$$

(7)

It is crucial to reduce energy loss (${\mathbf{E}}^{\mathbf{loss}}\left(t\right)$) when evaluating the reduction in carbon emissions.

2.2. Communication Model

Each subtask can be offloaded to a BS, UAV, or the LEO satellite for execution in one of these ways in time slot t. Thereafter, the task communication model is described as follows.

2.2.1. Offloading to the BSs

Let $R_{m,n}\left(t\right)$ denote the wireless transmission rate between device m and BS n in time slot t, which can be expressed as

$$R_{m,n}\left(t\right)=B_{m,n}\left(t\right){log}_2\left[1+\frac{p_m{G}_{m,n}\left(t\right)g_{m,n}\left(t\right)}{{\sigma }^2+{\zeta }_{m^{\prime },n^{\prime }}\left(t\right)}\right],$$

(8)

where $B_{m,n}\left(t\right)$ represents the channel bandwidth between device m and BS n. $p_m$ is the transmission power of device m. $G_{m,n}\left(t\right)$ is the path loss, which is related to the distance ($d_{m,n}\left(t\right)$) between device m and BS n in time slot t. $g_{m,n}\left(t\right)$ is the small-scale channel fading subject to Rayleigh fading. ${\sigma }^2$ denotes the noise power. ${\zeta }_{m^{\prime },n^{\prime }}\left(t\right)$ is the interference from the other BSs.

2.2.2. Offloading to the UAVs

If device m is in the communication coverage area of UAV k in time slot t, $a_{m,k}\left(t\right)=1$ is assumed. Otherwise, $a_{m,k}\left(t\right)=0$. As the altitude of the UAV is much higher than that of the devices, the line-of-sight channels of the UAV communication links are much more predominant [33]. The wireless channel of the UAV is considered a free-space path loss model. Therefore the channel gain ($g_{m,k}\left(t\right)$) is ${\beta }_0{\left(d_{m,k}\left(t\right)\right)}^{-{\alpha }_0}$, where ${\beta }_0$ denotes the channel gain at unit distance, ${\alpha }_0$ represents the path loss index, and $d_{m,k}\left(t\right)$ is the Euclidean distance between the m-$th$ device and the k-$th$ UAV. Hence, the transmission rate ($R_{m,k}\left(t\right)$) can be formulated as

$$R_{m,k}\left(t\right)=B_{m,k}\left(t\right){log}_2\left[1+\frac{a_{m,k}\left(t\right)p_m{g}_{m,k}\left(t\right)}{{\sigma }^2+{\zeta }_{m^{\prime },k^{\prime }}\left(t\right)}\right],$$

(9)

where $B_{m,k}\left(t\right)$ is the channel bandwidth, and ${\zeta }_{m^{\prime },k^{\prime }}\left(t\right)$ represents the co-channel interference from all the other UAVs.

2.2.3. Offloading to the LEO Satellite

The channel conditions between the LEO satellite and ground devices are mainly affected by the communication distance and rainfall attenuation. Since the distance between LEO satellite and devices remains almost unchanged, a fixed transmission rate ($R_{m,sat}\left(t\right)$) between devices and the LEO satellite is adopted. It is usually lower than the transmission rate between devices and UAV communication [34,35].

2.3. Computational Model

The task can be executed in four ways: executed locally, offloaded to BSs, offloaded to UAVs, or offloaded to the LEO satellite. Due to the small data size of the computational result, the delay and energy consumption required to transmit the computational result can be omitted [36].

2.3.1. Local Computation

The device is capable of varying CPU cycle frequency ($f_m\left(t\right)$) dynamically to obtain better system performance. Consequently, the latency of local task execution ($t_m^l\left(t\right)$) can be obtained as

$$t_m^l\left(t\right)=\frac{s_m\left(t\right)e_m\left(t\right)}{f_m\left(t\right)}+t_m^{l,wait}\left(t\right),$$

(10)

where $t_m^{l,wait}\left(t\right)$ indicates the queuing delay of the task for the device. Assuming the energy consumption per CPU cycle as $\kappa f_m^2\left(t\right)$, where $\kappa$ is the CPU chip-related switching capacitor [37], the energy consumption $E_m^l\left(t\right)$ of task execution on the device can be expressed as

$$E_m^l\left(t\right)=\kappa f_m^2\left(t\right)s_m\left(t\right)e_m\left(t\right).$$

(11)

2.3.2. Offloading to the BSs

It is assumed that tasks are executed instantly without queuing delay [30]. Because the BS typically has a relatively stable energy supply and a powerful multicore CPU when tasks are offloaded to the BS. $f_{m,n}\left(t\right)$ represents the allocated computational resources for the n-th BS. The task execution latency ($t_n^{bs}\left(t\right)$) for offloading to the BS can be expressed as

$$t_n^{bs}\left(t\right)=\frac{s_m\left(t\right)}{R_{m,n}\left(t\right)}+\frac{s_m\left(t\right)e_m\left(t\right)}{f_{m,n}\left(t\right)}.$$

(12)

The energy consumption ($E_n^{bs}\left(t\right)$) required for task execution in the BS can be obtained as

$$E_n^{bs}\left(t\right)={\delta }_0+{\delta }_1{s}_m\left(t\right).$$

(13)

2.3.3. Offloading to the UAVs

The task execution latency ($t_k^{uav}\left(t\right)$) for offloading to the UAVs can be expressed as

$$t_k^{uav}\left(t\right)=a_{m,k}\left(t\right)\left[\frac{s_m\left(t\right)}{R_{m,k}\left(t\right)}+\frac{s_m\left(t\right)e_m\left(t\right)}{f_{m,k}\left(t\right)}+t_k^{u,wait}\left(t\right)\right],$$

(14)

where $f_{m,k}\left(t\right)$ is the allocated computational resources for the k-$th$ UAV.

$e_{uav}$ is defined as the energy consumption per unit of computational resources of the UAV. The energy consumption ($E_k^{uav}\left(t\right)$) required for task execution by the UAV can be expressed as

$$E_k^{uav}\left(t\right)=a_{m,k}\left(t\right)\left[\frac{p_m{s}_m\left(t\right)}{R_{m,k}\left(t\right)}+e_{uav}{s}_m\left(t\right)e_m\left(t\right)\right].$$

(15)

2.3.4. Offloading to the LEO satellite

$f_{m,sat}\left(t\right)$ represents the allocated computational resource for the LEO satellite. The task execution latency ($t_m^{sat}\left(t\right)$) for offloading to the LEO satellite can be expressed as

$$t_m^{sat}\left(t\right)=\frac{s_m\left(t\right)}{R_{m,sat}\left(t\right)}+\frac{s_m\left(t\right)e_m\left(t\right)}{f_{m,sat}\left(t\right)}+t^{s,wait}\left(t\right).$$

(16)

The energy consumption ($E_m^{sat}\left(t\right)$) required for task execution by the LEO satellite can be obtained as

$$E_m^{sat}\left(t\right)=\frac{p_m{s}_m\left(t\right)}{R_{m,sat}\left(t\right)}+e_{sat}{s}_m\left(t\right)e_m\left(t\right),$$

(17)

where $e_{sat}$ is the energy consumption per unit of computational resources of the LEO satellite.

2.4. WPT Model

In order to maintain the sustainability of the network, the BS explores the WPT to power the UAV. Figure 2 shows the WPT principle adopted in the SAGIN. $\iota$ is the power division factor. The power divider divides the received radio signal into signal power with a ratio ($\iota$) for energy harvesting and signal power with a ratio $(1-\iota )$ for information decoding. In the energy harvester, the received signal is converted to the required energy and stored in the battery for the UAV.

Rather than considering the information decoder, the energy converted in the energy harvester is taken into account, and signal power is allocated to the energy harvester as much as possible. In addition, the impact on the health of the UAV battery of applying WPT technology is not considered. In the case in which UAV k is within the energy transmission coverage area of BS n in time slot t, $o_{k,n}\left(t\right)=1$. Otherwise, $o_{k,n}\left(t\right)=0$. A linear energy harvesting model is adopted. Then, the energy harvested by UAV k can be expressed as

$$B_k^H\left(t\right)=\sum _{k=1}^K{o}_{k,n}\left(t\right)\chi P_n{G}_n^t{G}_k^r{h}_{n,k}\left(t\right)·\tau ,$$

(18)

where $\chi$ denotes the energy conservation efficiency, which satisfies $\chi \in (0,1]$. $P_n$ is the BS transmitting power, and the BS adopts constant power transmission. $G_n^t$ and $G_k^r$ are the antenna power gains of the BS transmitter and the UAV receiver, respectively. $h_{n,k}\left(t\right)$ is the channel gain between the n-th BS and the k-th UAV in time slot t.

2.5. Flight Model

Continuous variables such as the average velocity ($v_k\left(t\right)$) and the directional angle (${\theta }_k\left(t\right)$) are considered to adjust the trajectory of UAV k. The horizontal position coordinates and the moving distance of UAV k in the current time slot are assumed to be ${\Psi }_k\left(t\right)={\left[x_k\left(t\right),y_k\left(t\right)\right]}^{\mathrm{T}}$ and $d_k\left(t\right)$, respectively. The position coordinates of UAV k in the next time slot can be expressed as

$$\left\{\begin{array}{c}{x}_k(t+1)=x_k\left(t\right)+d_k\left(t\right)cos{\theta }_k\left(t\right),\hfill \\ y_k(t+1)=y_k\left(t\right)+d_k\left(t\right)sin{\theta }_k\left(t\right).\hfill \end{array}\right.$$

(19)

Referring to [38], the propulsion power consumption of UAV k can be modeled as

$$P\left[v_k\left(t\right)\right]=P_0\left(1+3v_k^2\left(t\right)/\phantom{3v_k^2\left(t\right)U_{tip}^2}{U}_{tip}^2\right)+\frac{1}{2}{d}_0\rho sA{v}_k^3\left(t\right)+P_1{\left[\sqrt{1+v_k^4\left(t\right)/\phantom{v_k^4\left(t\right)4v_0^4}4v_0^4}-v_k^2\left(t\right)/\phantom{v^{2}2v_0^2}2v_0^2\right]}^{\frac{1}{2}},$$

(20)

where $P_0$ and $P_1$ are constants representing the blade profile power and induced power in hovering state, respectively. $U_{tip}$ denotes the tip speed of the rotor blade. $v_0$ is the mean rotor-induced velocity in the hover state. $d_0$ and s are the fuselage drag ratio and rotor solidity, respectively. $\rho$ and A denote the air density and rotor disc area, respectively. The flight energy consumption of UAV k in time slot t can be expressed as

$$E_k^F\left(t\right)={\int }_t^{t+\tau }P\left[v_k\left(t\right)\right]dt.$$

(21)

The system model and the SAGIN are enumerated in this section. In the following section, the cost function is constructed.

3. Problem Formulation

How to integrate latency-oriented, energy consumption, and carbon emissions to model the objective function in a SAGIN is the key to solve the first issue. According to the system model mentioned above, the cost function is first constructed on the basis of the system model. The formulation of the problem is demonstrated in this section.

The network cost function ($U\left(t\right)$) is constructed by joint task execution latency ($T\left(t\right)$), energy consumption ($E^{ec}\left(t\right)$), and carbon emissions ($C^{RE}\left(t\right)$), which can be represented as

$$U\left(t\right)={\vartheta }_{0}T\left(t\right)+{\vartheta }_1{E}^{ec}\left(t\right)+{\vartheta }_2{C}^{RE}\left(t\right),$$

(22)

where ${\vartheta }_0$, ${\vartheta }_1$, and ${\vartheta }_2$ represent the preference for latency, energy consumption, and carbon emissions, respectively, satisfying ${\vartheta }_0+{\vartheta }_1+{\vartheta }_2=1$. $T\left(t\right)$, $E^{ec}\left(t\right)$, and $C^{RE}\left(t\right)$ are expressed as

$$T\left(t\right)=\sum _{\genfrac{}{}{0pt}{}{m\in \mathcal{M},\hfill }{n\in \mathcal{N},k\in \mathcal{K}}}\left[t_m^l\left(t\right)+t_n^{bs}\left(t\right)+t_k^{uav}\left(t\right)+t_m^{sat}\left(t\right)\right],$$

(23)

$$E^{ec}\left(t\right)=\sum _{\genfrac{}{}{0pt}{}{m\in \mathcal{M},\hfill }{n\in \mathcal{N},k\in \mathcal{K}\hfill }}[E_m^l\left(t\right)+E_n^{bs}\left(t\right)+E_k^{uav}\left(t\right)+E_m^{sat}\left(t\right)+E_k^F\left(t\right)],$$

(24)

$$C^{RE}\left(t\right)=\sum _{n\in \mathcal{N}}\xi \left[h_n\left(t\right)-e_n^{loss}\left(t\right)\right],$$

(25)

where $\xi$ is the conversion factor of carbon emissions per unit of energy and is related to the type of energy source. $t_m^l\left(t\right)$, $t_n^{bs}\left(t\right)$, $t_k^{uav}\left(t\right)$, and $t_m^{sat}\left(t\right)$ represent the time when the tasks are executed locally or offloaded to the computation server in time slot t, respectively. In particular, each subtask can only be executed in one of these ways. $E^{ec}\left(t\right)$ includes energy consumption for tasks execution and UAV flight in time slot t. $C^{RE}\left(t\right)$ is the reduction in carbon emissions achieved by reducing traditional power gird energy consumption.

The network cost function is formulated as a minimization problem as follows.

$$\begin{array}{c}\hfill min{\sum }_{t\in \mathcal{T}}U\left(t\right)\end{array}$$

(26)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& o_m^L\left(t\right),o_{m,n}^N\left(t\right),o_{m,k}^K\left(t\right),o_m^{sat}\left(t\right)\in \{0,1\},\hfill \end{array}$$

(26a)

$$\begin{array}{cc}\hfill & {\mathrm{o}}_m^L\left(t\right)+\sum _{n=1}^N{\mathrm{o}}_{m,n}^N\left(t\right)+\sum _{k=1}^K{\mathrm{o}}_{m,k}^K\left(t\right)+{\mathrm{o}}_m^{sat}\left(t\right)=1,\hfill \end{array}$$

(26b)

$$\begin{array}{cc}\hfill & t_m^l\left(t\right),t_n^{bs}\left(t\right),t_k^{uav}\left(t\right),t_m^{sat}\left(t\right)\le t_m\left(t\right),\hfill \end{array}$$

(26c)

$$\begin{array}{cc}\hfill & 0\le {\theta }_k\left(t\right)\le 2\pi ,0\le v_k\left(t\right)\le v_{max},\hfill \end{array}$$

(26d)

$$∥{\Psi }_{k1}\left(t\right)-{\Psi }_{k2}\left(t\right)∥\ge D_{min},k1\ne k2,$$

(26e)

$$B^{bot}\le B_k^K\left(t\right)\le B^{capacity},$$

$$B^{res}\left(t\right)\le B_n^{bs}\left(t\right)\le B^{battery},$$

(26f)

$$f_m\left(t\right)\le f_m^{max},\sum _{m=1}^M{f}_{m,k}\left(t\right)\le f_k^{max},$$

$$\begin{array}{cc}& \sum _{m=1}^M{f}_{m,n}\left(t\right)\le f_n^{max},\sum _{m=1}^M{f}_{m,sat}\left(t\right)\le f_{sat}^{max},\hfill \end{array}$$

(26g)

$$\begin{array}{cc}\hfill & \forall m\in \mathcal{M},n\in \mathcal{N},k\in \mathcal{K},t\in \mathcal{T}.\hfill \end{array}$$

(26h)

Constraints (26a) and (26b) imply that the task is indivisible and can be executed locally or remotely by the computation server. (26c) constrains the latency limit of the task. Constraint (26d) defines the range of angles and velocities that control the UAV’s flight trajectory. (26e) ensures a safe distance between UAVs, where $∥·∥$ is the Euclidean norm. Constraint (26f) indicates the minimum and maximum thresholds for the UAV and BS backup batteries. (26g) guarantees that the computational resources allocated to the task by the device and other computational servers do not exceed the maximum computation capacity. (26h) specifies the range of variables.

It is a challenge to solve the above problem, owing to the dynamic and complex characteristics of the network environment. It is crucial to design a satisfactory solution that can operate in real time and is capable of handling high-dimensional dynamic parameters. Therefore, in the following section, DRL is used to learn the network and to deal with an optimization problem with various constraints.

4. TD3PG Algorithm Based on the Markov Chain Model in a SAGIN

DRL aims to learn an award-maximizing strategy through interactions of intelligent agents with the environment. In order to solve the second issue with DRL, the cost function is reformulated as a Markov decision process (MDP). TD3PG, a DRL-based algorithm, is proposed to optimize the objective problem by training the near-optimal model.

4.1. MDP Formulation

We assume a typical MDP with four tuple elements $〈\mathbf{S},\mathbf{A},\mathbf{M},\mathbf{R}〉$, where $\mathbf{S}$ is the state space. $\mathbf{A}$ is the action space. $\mathbf{M}$ denotes the state transition function, which indicates the probability of the transition to next the state ($s_{t+1}$) after performing $(s_t,a_t)$. $\mathbf{R}$ is the reward function. Detailed descriptions of the elements in the tuple are presented below.

• State space: State space is designed to store the DRL agent’s observations in the environment and to guide the generation of actions. The state space includes the remaining energy of the UAV ($B_k^K\left(t\right)$), the BS ($B_n^{bs}\left(t\right)$), the reserved energy for the BS backup battery ($B_n^{res}\left(t\right)$), the current horizontal location of the UAV (${\Psi }_k\left(t\right)$), and the amount of the current task remaining ($s_m\left(t\right)$). Then, the system state space can be defined as

  $$s_t=\{B_k^K\left(t\right),B_n^{bs}\left(t\right),B_n^{res}\left(t\right),{\Psi }_k\left(t\right),s_m\left(t\right)\}.$$

  (27)
• Action space: Action space is designed to store the actions performed by the DRL agent in the environment in order to obtain feedback from the environment and the next state. The action space contains the offloading decisions ($o_m^L\left(t\right),o_{m,n}^N\left(t\right),o_{m,k}^K\left(t\right),o_m^{sat}\left(t\right)$, i.e., offloading the task to a local device, BS, UAV, or the LEO satellite, respectively). The navigation speed ($v_k\left(t\right)$) and the rotation angle (${\theta }_k\left(t\right)$) used to control UAV trajectory are continuous values in the $\left[0,v_{max}\right]$ and $\left[0,2\pi \right]$ intervals, respectively. $f_m\left(t\right)\in \left[f_m^{max}/2,f_m^{max}\right]$ denotes the dynamic computational resources of the device. ${\upsilon }_n$ is the proportion of shared RE harvested by the BS. Therefore, the system action space can be expressed as

  $$a_t=\{o_m^L\left(t\right),o_{m,n}^N\left(t\right),o_{m,k}^K\left(t\right),o_m^{sat}\left(t\right),v_k\left(t\right),{\theta }_k\left(t\right),f_m\left(t\right),{\upsilon }_n\left(t\right)\}$$

  (28)
• Reward: Reward is the feedback from the environment to the DRL agent after performing the current action. The research goal is to minimize the cost function. The algorithm depends on the reward to evaluate its decisions by learning the SAGIN environment. Due to the fact that the goal of RL is a cumulative overall reward over time to maximize reward, the reward function is reformulated as the negative value of the optimization cost function. Then, the reward can be expressed as

  $$R_t=-U\left(t\right),$$

  (29)

  where the coefficients ${\vartheta }_0$, ${\vartheta }_1$, and ${\vartheta }_2$ in $U\left(t\right)$ are required to satisfy ${\vartheta }_0,{\vartheta }_1>0$ and ${\vartheta }_2<0$, respectively.

Figure 3 shows the strategic framework for the scenario proposed in the SAGIN. In time slot 1, tasks are set to be computed locally or be offloaded to the computation server without exceeding the latency constraints, incurring the latency of local computation or transmission, as well as the corresponding energy consumption. In time slot 2, tasks that are transferred to the server are computed. Because the device is within the communication range of the UAV, new tasks can be optionally offloaded to the computation servers, including the UAV waiting for execution. Since the UAV is also within the energy transmission coverage area of the BS, the BS can charge the UAV by WPT when the UAV battery power falls below a threshold. In time slot 3, the remaining tasks in the UAV’s computing queue that have not been computed (black) in the previous time slot generate a waiting delay. It is necessary to wait until execution is completed before calculating the tasks for the current time slot. At the end of each time slot, the remaining battery power is calculated. Excluding the energy that needs to be reserved, the BS backup battery shares the excess RE with other BSsin a certain percentage. When the remaining power of the BS backup battery is lower than the required reserved power (red), power from the grid is utilized to replenish it. In each time slot, the task execution latency, total network energy consumption, and reduction in carbon emissions are jointly formulated as a cost function. The reward state space and action space are fed into the TD3PG algorithm, which ultimately feeds back offload decisions, local computation resources, the UAV trajectory, and the RE sharing ratio for the scenario. Eventually, new actions are executed in the scenario.

4.2. TD3PG Algorithm

Owing to the fact that the action space contains continuous variables, TD3PG is adopted, which is an off-policy algorithm based on the actor–critic structure for dealing with optimization problems with a high-dimensional action space [39].

Compared to the conventional actor–critic-based deep deterministic policy gradient (DDPG), which would result in cumulative errors, TD3PG solves the former overestimation problem. It has two pairs of critic networks in which the smaller Q value of the two Q functions is chosen to form the loss function in the Bellman equation. Hence, there is less overestimation in the learning process. Moreover, the application of a delayed policy technique enables TD3PG to less frequently update the actor network and the target network than the critic networks. In other words, the model does not update policy unless the model’s value function is updated sufficiently. The less frequent policy updates result in a value evaluation with lower variance, producing a better policy [40].

TD3PG is also able to smooth the value estimate by applying a smoothing technique for a target policy that adds noise (clipped random noise) to the target actions and average in mini batches, as shown below:

$$ϵ\sim \mathrm{clip}\left(\mathrm{N}\left(0,\tilde{\omega }\right),-c,c\right).$$

(30)

According to the structure of the TD3PG algorithm depicted in Figure 4, the TD3PG architecture contains one actor network and two critic networks. The actor network is parameterized by ${\theta }_{\pi }$. The two critic networks are parameterized by ${\theta }_{Q1}$ and ${\theta }_{Q2}$, respectively.

The agent first observes the current state ($s_t$) and picks an action ($a_t$) via the actor network. After performing $a_t$ in the environment, the agent is able to observe the immediate reward ($R_t$) and the next state ($s_{t+1}$). Then, the agent selects the policy that maximizes the cumulative reward, which converts the state ($s_t$) to the action ($a_t$). At this point, the transition ($〈s_t,a_t,R_t,s_{t+1}〉$) is obtained and stored in the replay memory for further sampling.

The training process of the TD3PG algorithm is summarized as Algorithm 1.

The minimum Q-value estimate of these two critic networks is used to calculate the target value ($y_t$), which can be obtained as

$$y_t=R_t+\gamma ·\underset{i=1,2}{min}{Q}_{{\theta }_{Qi}}\left(s_{t+1},(\pi \left(s_{t+1}\right|{\theta }_{\pi }^T)+\epsilon )|{\theta }_{Qi}^T\right),$$

(31)

where $\gamma$ is a discount factor, ranging from 0 to 1.

The loss function of the two critic networks can be defined as

$$Ls\left({\theta }_{Qi}\right)=\mathrm{E}\left[{(y_t-Q_{{\theta }_{Qi}}(s_t,a_t|{\theta }_{Qi}))}^2\right],i=1,2.$$

(32)

Then, the gradient of the loss function is calculated as

$${\nabla }_{{\theta }_{Qi}}Ls\left({\theta }_{Qi}\right)=E\left[\right(y_t-Q(s_t,a_t|{\theta }_{Qi})){\nabla }_{{\theta }_{Qi}}Q(s_t,a_t|{\theta }_{Qi})],i=1,2.$$

(33)

The policy gradient of the actor network can be expressed as

$${\nabla }_{{\theta }_{\pi }}J\approx E\left[{\nabla }_{a}Q(s_t,a_t|{\theta }_{Q1}){|}_{a_t=\pi \left(s_t\right)}{\nabla }_{{\theta }_{\pi }}\pi \left(s_t\right|{\theta }_{\pi })\right].$$

(34)

The parameters of the target networks are updated by means of

$$\begin{array}{c}{\theta }_{\pi }^T\leftarrow \omega {\theta }_{\pi }+(1-\omega ){\theta }_{\pi }^T\hfill \\ {\theta }_{Qi}^T\leftarrow \omega {\theta }_{Qi}+(1-\omega ){\theta }_{Qi}^T,i=1,2,\hfill \end{array}$$

(35)

where $\omega$ is decaying rate of both the actor and critic networks.

Algorithm 1 TD3PG algorithm.

Require: Training episode length: $max$-$episode$. discount factor: $\gamma$. Ensure: Loss, gradients.   1: Initialize the actor network ${\pi }_{{\theta }_{\pi }}$ with parameter ${\theta }_{\pi }$. Initialize the critic network $Q_{{\theta }_{Q1}}$, $Q_{{\theta }_{Q2}}$ with parameters ${\theta }_{Q1}$ and ${\theta }_{Q2}$, respectively.   2: Initialize the target network by: ${\theta }_{\pi }^T\leftarrow {\theta }_{\pi }$, ${\theta }_{Q1}^T\leftarrow {\theta }_{Q1}$, ${\theta }_{Q2}^T\leftarrow {\theta }_{Q2}$.   3: Initialize replay buffer $\mathcal{R}$.   4: for episode = 1 to $max$-$episode$ do   5:     Reset simulation parameters and obtain initial observation state s.   6:     for t = 1 to T do   7:        Get action after adding exploration noise $a_t\sim \pi \left(s_t\right|{\theta }_{\pi })+\mathrm{N}\left(0,\delta \right)$.   8:        Obtain the reward $R_t$ and update $s_{t+1}$.   9:        Store the transition $〈s_t,a_t,R_t,s_{t+1}〉$ in replay buffer $\mathcal{R}$.  10:       Sample a random mini-batch of V tuples from replay buffer $\mathcal{R}$.  11:       Compute greedy actions for next states through $\tilde{a}\leftarrow \pi \left(s_{t+1}\right|{\theta }_{\pi }^T)+ϵ$, where $ϵ\sim \mathrm{clip}\left(\mathrm{N}\left(0,\tilde{\omega }\right),-c,c\right)$.  12:       Compute target value $y_t$ based on Equation (31).  13:       Update the critics by minimizing the loss function Equation (32).  14:       Update the actor network parameter ${\theta }_{\pi }$ by using the sampled policy gradient Equation (34).  15:       Update the target networks Equation (35).  16:     end for  17: end for

4.3. Complexity Analysis

The proposed TD3PG algorithm contains both the actor network and critic networks. Therefore, their complexities are evaluated separately. Assuming that the actor network has $L^a$ layers with $n_l$ neuron nodes in the l-th layer (l ≤ L_a), the complexity of the l-th layer is $\mathcal{O}\left(n_{l-1}{n}_l+n_l{n}_{l+1}\right)$ [41]. The complexity of the actor network is $\mathcal{O}\left({\sum }_{l=2}^{L^a-1}\left(n_{l-1}{n}_l+n_l{n}_{l+1}\right)\right)$. Assuming that the critic network has $L^c$ layers with $m_l$ neuron nodes in the j-th layer (j ≤ L_c), the complexity of the j-th layer is $\mathcal{O}\left(m_{j-1}{m}_j+m_j{m}_{j+1}\right)$ [41]. Similarly, the complexity of the critic network is $\mathcal{O}\left({\sum }_{j=2}^{L^c-1}\left(m_{j-1}{m}_j+m_j{m}_{j+1}\right)\right)$. Hence, the overall computational complexity of the TD3PG algorithm is $\mathcal{O}({\sum }_{l=2}^{L^a-1}(n_{l-1}{n}_l+n_l{n}_{l+1})+{\sum }_{j=2}^{L^c-1}(m_{j-1}{m}_j+m_j{m}_{j+1}))$, which is similar to that of the DDPG algorithm.

Overall, TD3PG is an off-policy algorithm based on the actor–critic structure for dealing with optimization problems with a high-dimensional action space. It solves the overestimation problem of DDPG. Moreover, the application of a delayed policy technique enables TD3PG to produce a better policy. It is an improvement over the DDPG algorithm, but it does not considerably increase the complexity.

5. Results and Discussion

In this section, numerical simulations are used to evaluate the performance of the proposed TD3PG algorithm. Specifically, the simulation settings and benchmark strategies are first elaborated. Afterwards, an analysis of the test results is conducted.

5.1. Simulation Settings

We consider an experimental area of 800 m × 800 m in which M = 50 devices are uniformly distributed. The data size and latency requirements of each computation subtask obey a uniform distribution. Each device has a subtask in each time slot with the data size randomly distributed in $U[3,8]$ Mbits. The latency requirements of the subtasks are randomly distributed in $U[0.8,1.2]$ s. There are N = 2 BSs, K = 3 UAVs, and one LEO satellite in the experimental area. The fixed altitude of the UAVs is H = 100 m, and their communication coverage is within a circular radius of 173.2 m. The communication coverage radius of the BSs is about 250 m. The LEO satellite covers the entire experimental area, with a communication coverage of about 200,000 km${}^2$. The bandwidths of the UAVs and BSs are 5 and 20 MHz, respectively. A DJI M300RTK quadcopter with the addition of a computation server, communication equipment, wireless charging equipment, and a wheelbase of 895 mm was selected. The total weight of the UAV is about 9 kg, with the communication equipment (6G BS) accounting for 1 kg. One charge can support UAV flight for 55 min. In order to simplify the battery parameters and to demonstrate the role of the WPT as quickly as possible, the battery capacity of the UAV is set to ${10}^4$ J. During a charge of a UAV, the less RE loss collected by the BS backup battery, the more RE gained by the UAV through WPT. Hence, the more reduced carbon emissions can be captured. Corresponding simulation parameters are illustrated in

Table 2. Simulation parameters.

Parameter	Value	Parameter	Value
M, N, K	50, 2, 3	$v_0$	4.03 m/s
$s_m$	$U[3,8]$ Mbits	$P_0$	79.86 W
$e_m$	800 cycles/bit	$P_1$	92.48 W
$t_m$	$U[0.8,1.2]$ s	$\rho$	1.225 kg/m${}^3$
$p_m$	0.1 W	A	0.503 m${}^2$
$f_m^{max}$, $f_n^{max}$	0.25, 20 GHz	$\chi$	0.9 [30]
$f_k^{max}$, $f_{sat}^{max}$	10, 50 GHz	s	0.05
$D_{min}$	20 m	$d_0$	0.6
$v_{max}$	23 m/s	$\eta$	0.1
H	100 m	$G_n^t$	41 dB
${\alpha }_0$	2 [42]	$G_k^r$	41 dB
${\sigma }^2$	${10}^{-14}$ W	$U_{tip}$	120 m/s
$\kappa$	${10}^{-27}$ J/Hz${}^3$/s	$B^{battery}$	${10}^6$ J
$e_{sat}$	1 W/GHz	$B^{capacity}$	${10}^4$ J
$e_{uav}$	1.5 W/GHz	${\vartheta }_0$, ${\vartheta }_1$, ${\vartheta }_2$	0.12, 0.98, −0.1

.

The simulations are run with Python 3.8.8 and Tensorflow 2.6.0 on a computer with 1 CPU: an Intel(R) i5-9400. The time for each training episode is about 22 s. To evaluate the performance of our proposed algorithm, we compare the proposed TD3PG algorithm with three conventional benchmarks.

• DDPG Algorithm: A DRL-based algorithm, DDPG also contains two types of neural networks—an actor network and a single critic network—whereas TD3PG has two critic networks. Other parameters are consistent with those of TD3PG.
• Full Offload: As local computation does not guarantee latency constraints, in addition to consuming a lot of energy, tasks are all offloaded to the computation server for computation. The computation server distributes computation resources evenly based on the number of connected devices. All excess RE is shared with neighboring BSs.
• Greedy on Edge Algorithm (Greedy-edge): Since edge computing has a low computational cost, the computational resources of the UAV can be fully utilized, resembling the “Greedy on edge” algorithm reported in [43]. In time slot t, 60% of the total tasks are offloaded to the UAV, which distributes computational resources evenly based on the number of connected devices. All excess RE is shared with neighboring BSs.

5.2. Results and Analysis

In order to evaluate the impact of different network parameters on the performance of the proposed TD3PG-based algorithm, a normalized cumulative reward is adopted. The normalized cumulative reward is defined as the ratio between the difference and the extreme difference in the cumulative reward, where the difference is the difference between the cumulative reward and the minimum cumulative reward. A larger normalized cumulative reward means better performance. In Figure 5, combinations of four learning rates are presented to evaluate the impact of different learning rates. With an increased number of episodes, the normalized cumulative reward value fluctuates with different learning rates. Until around 2000 episodes, the four cases of normalized cumulative reward have different convergence trends. The performance of the proposed algorithm is the best when ${\alpha }_{\pi }=0.0001$ and ${\alpha }_{\mu }=0.0001$, corresponding to the highest convergence reward curve among the four cases.

In Figure 6, the impacts of different discount factors are evaluated. According to the convergence results, the normalized cumulative reward in the four cases gradually become smaller as $\gamma$ keeps increasing. In particular, compared with $\gamma =0.69$, $0.79$, and $0.89$, higher performance is exhibited when $\gamma =0.99$. Hence, $\gamma =0.99$ is the best discount factor among the proposed discount factors.

Figure 7 shows the variation of the network cost function with the number of devices under different benchmark algorithms. Both the proposed algorithm and the DDPG algorithm demonstrate good performance in terms of joint latency-oriented energy consumption and carbon emissions. Specifically, for both algorithms, the value of the network cost function is always lower than other benchmarks for different numbers of devices and increases at a slow rate as the number of devices increases. This is due to the fact that the proposed algorithm and the DDPG algorithm dynamically adjust the offloading strategy to obtain a lower value of the cost function, optimizing the trajectory of the UAV and the sharing ratio of the RE in real time, ultimately obtaining a lower value of the cost function. In particular, the cost function under the TD3PG algorithm is lower and shows better performance compared to that under the DDPG algorithm.

In Figure 8, with the exception of full offload, the total energy consumption under other benchmark strategies increases with the maximum computation capacity of the device. Due to the ability of the proposed algorithm and the DDPG algorithm to dynamically adjust the local computational resources according to the reward function, the total energy consumption achieves optimal performance when $f_m^{max}$ is low. In particular, the proposed algorithm shows the best performance with $f_m^{max}$ between 2.0 GHz and 2.5 GHz. However, the full offload policy offloads all tasks to the computation server with no significant change in total energy consumption.

Figure 9 compares the variation of the total network energy consumption with different values of energy consumption per unit of computational resources for the LEO satellite $e_{sat}$ under different benchmark algorithms. With the increase in $e_{sat}$, the total energy consumption under the proposed algorithm is always the lowest. The total energy consumption of the network under the greedy-edge algorithm is the highest. On the one hand, it makes no better offloading decision. On the other hand, in terms of energy consumption, the device is unable to intelligently change the computational resources, resulting in increased energy consumption for local computation. Under the full offload benchmark algorithm, more tasks are offloaded to the LEO satellite for execution. The total energy consumption grows more significantly when $e_{sat}$ increases. The DDPG algorithm dynamically adjusts task offloading according to the reward value, with the rate of increase in total network energy consumption gradually decreasing as $e_{sat}$ grows. The high altitude of the LEO satellite leads to increased transmission latency. Therefore, few tasks are offloaded to it. The variation in total network energy consumption with the proposed algorithm is slight when $e_{sat}$ increases. Similarly, the proposed algorithm demonstrates excellent performance as the energy consumption per unit of computational resources for the UAV ($e_{uav}$) increases.

In order to illustrate the performance of the proposed algorithm in optimizing the UAV trajectory to reduce latency, three cases with different benchmark algorithms are analyzed. These three cases are trajectory optimization, the trajectory fixed in one direction, and the UAV flying in place. Simultaneously, 50 devices are compared in terms of total execution latency for all tasks throughout the entire period. As shown in Figure 10, the lowest overall latency of the three cases is trajectory optimization, which reduces the overall latency by 2.93% compared to the UAV flying in place and 5.78% compared to the overall time reduction for a trajectory fixed in one direction. The total task latency under both the proposed algorithm and the DDPG algorithm is lower in each flight trajectory mode, significantly improving the latency/QoS. With both algorithms, the UAV moves towards the location with the appropriate number of devices. As offloading too many tasks to the UAV leads to an increase in total latency, the greedy-edge and full offload algorithms show poor performance. The greedy-edge algorithm offloads a higher percentage of tasks to the UAV, resulting in a higher task latency. Meanwhile, the local computational resources do not correspond to the maximum computation capability, which leads to a longer computation time. This is the reason why the total task latency of the full offload algorithm is lower than the greedy-edge algorithm.

Since the UAVs have limited battery resources, the remaining power of the UAV needs to be maintained as much as possible to ensure the sustainable operation of the network. Figure 11 shows the average residual power of the UAV at each time slot for different benchmark algorithms. Among these benchmark algorithms, both the proposed algorithm and the DDPG algorithm employ WPT, which ensure that the UAV always has remaining power to maintain operation. However, the other two do not have WPT. So the power is quickly depleted and unable to maintain the sustainable operation of the network. It is worth noting that the energy consumed by UAV computation is not large compared to the energy consumed by UAV flight. Hence, the difference in the average remaining power of UAV under the remaining three benchmark algorithms is small. The greedy-edge algorithm and the full offload policy offload a large amount of tasks to the computation server. So the average remaining power of the UAV is lower. The proposed algorithm, on the other hand, integrates dynamic task offloading with UAV trajectory optimization, which not only maximizes the relative residual power of the UAV but also enables the UAV to replenish power through WPT when its power falls below a threshold value, maintaining sustainable network operation.

In order to evaluate the performance of the proposed algorithm in reducing the carbon emissions of the network, Figure 12 shows the average loss ratio of RE for different loss factors of energy transmission. The smaller the average loss rate of RE, the more carbon emissions are reduced. As the two algorithms can intelligently optimize the sharing proportion (${\upsilon }_n\left(t\right)$) of RE, the proposed algorithm shows better performance compared to the DDPG algorithm. It is worth noting that the average loss ratio of RE under the DDPG algorithm at $\eta =0.05$ is higher than that $\eta =0.1$. It may be that too much energy is carried over when too little energy is shared. When RE arrives in the next time slot, the total remaining energy exceeds the backup battery capacity, which leads to an increase in the amount of RE ($e_n^{exc}\left(t\right)$) that is discarded.

6. Conclusions

In this article, a SAGIN scenario with new designed power technology is proposed for the first time. A cost function is formulated with joint latency/QoS, green communication, and peak carbon dioxide. Then, a TD3PG method is proposed to optimize the cost function. Finally, the proposed algorithm is compared with three benchmark algorithms. Many valuable findings can be captured. Simulation results showcase that the cost function obviously decreases. The proposed approach improves the sustainability of the network using WPT technology, while offering advantages in terms of task latency and energy saving. In particular, by optimizing the UAV trajectory under this algorithm, the total task latency is reduced by 2.93% and 5.78% compared to a single trajectory and no trajectory, respectively. The proposed algorithm also shows better performance when it comes to reducing the carbon emissions of the network by optimizing the the sharing proportion of RE. The research reported in this article contributes to the development of future network in the direction of smart, green, and low-carbon applications, which is also important for improving the QoS in future communication network. In the future, a software-defined network will be utilized to better optimize the energy resources and wireless resources in a SAGIN.