Adaptive Control for Underwater Simultaneous Lightwave Information and Power Transfer: A Hierarchical Deep-Reinforcement Approach

Abstract

In this work, we consider a point-to-point underwater optical wireless communication scenario where an underwater sensor (US) transmits its sensing data to a remotely operated vehicle (ROV). Before the US transmits its data to the ROV, the ROV performs simultaneous lightwave information and power transfer (SLIPT), delivering both control data and lightwave power to the US. Under the considered scenario, our objective is to maximize energy harvesting at the US while supporting predetermined communication performance between the two nodes. To achieve this objective, we develop a hierarchical deep Q-network (DQN)–deep deterministic policy gradient (DDPG)-based online algorithm. This algorithm involves two reinforcement learning agents: the ROV and US. The role of the ROV agent is to determine an optimal beam-divergence angle that maximizes the received optical signal power at the US while ensuring a seamless optical link. Meanwhile, the US agent, which is influenced by the decision of the ROV agent, is responsible for determining the time-switching and power-splitting ratios to maximize energy harvesting without compromising the required communication performance. Unlike existing studies that do not account for adaptive parameter control in underwater SLIPT, the proposed algorithm’s adaptive nature allows for the dynamic fine-tuning of optimization parameters in response to varying underwater environmental conditions and diverse user requirements.

1. Introduction

Underwater optical wireless communication (UOWC) is a cutting-edge technology that uses light to transmit data through water, enabling high-speed and reliable communication in underwater environments. It has gained significant importance in various scientific and industrial applications, including underwater sensing, environmental monitoring, underwater robotics, and offshore exploration. Compared to traditional acoustic-based communication, UOWC offers several advantages, such as higher data rates, wider bandwidths, lower latencies, and immunity to electromagnetic interference. These advantages make it a promising solution for fulfilling the increasing demands of underwater communication systems [1,2,3].

However, despite the numerous advantages of UOWC, it faces several challenges that can significantly impact its performance. A primary issue is signal attenuation and fading, which arise due to the inherent properties of water, such as absorption, scattering, and turbulence. Another critical challenge is the misalignment between the transmitter and receiver, often caused by factors like water currents or the movement of underwater vehicles, leading to degraded link quality and reduced communication range. To address these challenges, various studies have been conducted, focusing not only on evaluating UOWC system performance considering optical signal attenuation and fading [4,5,6,7], but also on mitigating the effects of unpredictable misalignment between the transmitter and receiver [8,9,10,11].

Meanwhile, in recent times, the development of simultaneous lightwave information and power transfer (SLIPT) techniques [12] has garnered significant importance in the field of UOWC, leading to the emergence of underwater SLIPT. SLIPT is an extended concept of simultaneous wireless information and power transfer, called SWIPT [13,14], to the optical domain, utilizing light signals for both data transmission and power transfer. SLIPT offers the unique capability to not only transmit data but also provide power simultaneously, thus addressing the challenges of power supply in underwater environments. The ability to simultaneously transfer data and power represents a significant technological advancement in underwater communication and holds great promise for facilitating reliable and sustainable operations in challenging underwater environments. The work of [15] introduced an overview of various SLIPT techniques in the time, power, and spatial domains. Moreover, it presented two underwater proof-of-concept demonstrations of time-switching (TS) SLIPT. In [16], an underwater SLIPT system was designed that consisted of a laser diode (LD)-based transmitter and a multi-element receiver with a single-photon avalanche diode and a solar panel. In [17], the authors investigated closed-form expressions for energy harvesting (EH), bit error rate, and spectral efficiency (SE) over log-normal turbulence channels under different underwater SLIPT methods. Optimization problems were then formulated for each method, and the optimal TS and power-splitting (PS) ratios were determined. The work of [18] presented the constellation design for an optimized color-shift keying system to maximize the minimum distance between the constellation points while mitigating the total received current constraint to optimize communication performance. In [19], the evaluation of communication link performance and charging speed was conducted under an actual experimental setup of an underwater SLIPT system. For the expansion of the UOWC range, a dual-hop structure with an underwater SLIPT was introduced based on the TS method [20]. Subsequently, expressions for the average BER at the target node and the harvested energy by the relay node were derived over underwater attenuation channels. The work of [21] considered a cooperative non-orthogonal multiple-access-assisted uplink UOWC system based on SLIPT. In particular, in the process of performance evaluations, various practical assumptions, including misalignment at the relay node, were reflected.

However, despite these research achievements in the field of UOWC, it is worth noting that the existing works on underwater SLIPT, including [15,16,17,18,19,20,21], have not addressed the adaptive control of TS and PS ratios in combination with the beam-divergence angle, considering changes in the underwater environment. This aspect is crucial because the adaptive control of these parameters plays a vital role in providing seamless communication service while maximizing EH in dynamic and time-varying underwater environments. In real-sea conditions, unlike on land, the UOWC channel is subject to a range of external factors such as water currents, salinity, and temperature fluctuations, which can cause rapid and unpredictable changes in channel characteristics. These challenges make it particularly difficult to guarantee consistent and reliable communication performance in underwater environments, further underscoring the importance of adaptive control strategies.

1.1. Contributions

We highlight our contributions in this work as follows:

• In this study, our objective is to develop an online algorithm for UOWC that adaptively determines the TS and PS ratios of SLIPT as well as the beam-divergence angle to maximize EH while ensuring seamless communication performance between a remotely operated vehicle (ROV) and an underwater sensor (US) with SLIPT capabilities. To carry out the ROV missions set in this study, we consider a hybrid UOWC system that utilizes both LD and light-emitting diode (LED) technologies. LD-based UOWC is employed for control data and power transmission from the ROV to the US via SLIPT, whereas LED-based UOWC is used for sensing data transmission from the US to the ROV.
• To address the challenges of this communication scenario, we propose a hierarchical deep Q-network (DQN)–deep deterministic policy gradient (DDPG) algorithm. This algorithm involves two reinforcement learning (RL) agents: the ROV agent and the US agent. The role of the ROV agent is to determine the beam-divergence angle that maximizes the received optical power at the US node while ensuring a seamless optical link. On the other hand, the US agent, influenced by the decisions of the ROV agent, is responsible for determining the TS and PS ratios that maximize the EH without compromising the required communication performance.
• Through extensive simulations, we demonstrate that the proposed algorithm successfully maximizes the EH while maintaining the predetermined communication requirement at the US. The adaptive nature of the algorithm allows it to dynamically adjust the system parameters in response to changing underwater environmental conditions and sensor requirements, therefore enabling efficient and sustainable energy transfer and communication in underwater environments.

1.2. Organization

The rest of this paper is organized as follows. In Section 2, we formally present our underwater UOWC scenario between the ROV and US. Section 3, we introduce a hybrid TS and PS SLIPT technique and its corresponding performance metrics. In Section 4, we first formulate an optimization problem to achieve our objective and then propose an online learning algorithm (i.e., hierarchical DQN–DDPG algorithm) to solve the problem in real time. In Section 5, we provide an evaluation of the performance of our proposed algorithm based on extensive simulations. Finally, conclusions are drawn in Section 6.

2. System Model

2.1. Network Model

We consider a three-dimensional (3D) underwater communication network in which a US communicates with the ROV as shown in Figure 1. More specifically, the US is fixed onto the seafloor and measures, at regular intervals, a variety of underwater environmental data depending on its purpose. On the other hand, the ROV can conduct many shallow and deep underwater missions, such as marine science and oil and gas extraction missions, which would otherwise be very difficult or dangerous for humans to do, even if diving in a submersible or submarine. In these applications, the motions of the ROV are guided either by a human pilot on a surface support vessel through an umbilical cable that provides power and telemetry or by an automatic pilot system [22]. This study assumes that the ROV is controlled by a human pilot through an umbilical cable and that it has two missions: (1) collecting sensing data measured by the US and (2) wirelessly transferring power to the energy-deprived US for battery charging. To support these ROV missions, this study adopts hybrid LD–LED-based UOWC. The modems for this hybrid UOWC are installed at the bottom of the ROV and at the top of the US, respectively, to align their beams for optical links. More specifically, to simultaneously transmit both power and control data (e.g., wake-up and communication completion data) from the ROV to the US, we employ LD-based SLIPT. For such missions, adopting LD-based communication is reasonable because it is an effective method for transferring power with high efficiency compared to that of LED-based UOWC. On the other hand, to transmit underwater sensing data collected by the US to the ROV, we employ LED-based communication. This is because LED-based communication can support reliable data transmission over a relatively large FOV, even when the ROV and US are not perfectly aligned owing to various factors in the water. As illustrated in Figure 1, the specific ROV operation procedure for achieving these missions is as follows:

• First, the ROV is launched into the ocean from the support vessel using a launch and recovery system (LARS). Once in the water, the ROV moves to the location where the US is located. At this location, the ROV performs SLIPT to not only transmit control data (e.g., wake-up or communication completion data) but also transfer power.
• Perceiving the control data and power, the US proceeds to transmit its collected sensing data to the ROV via LED-based UOWC. Although LED-based UOWC may have a relatively lower data rate compared to that of LD-based UOWC, it still provides a sufficient data rate (e.g., more than Mbps [23]) to transmit the sensing data with high reliability.
• Once the data reception process is complete, the ROV is retrieved and brought back to the support vessel for recovery, which is facilitated by the LARS.

2.2. Signal Model

We consider UOWC that is based on intensity modulation and direct detection (IM/DD), in which the light intensity is modulated as an information-bearing signal, and information is recovered at the receiver side by measuring the intensity of the received light [24]. Under IM/DD, the information bits are modulated via M-ary pulse amplitude modulation (M-PAM), where M denotes the modulation level.

Let T be the time duration of a data frame consisting of M-PAM symbols, such that the symbol interval can be expressed as $T_s=T/M$. We denote the M-PAM symbol as x. Since M-PAM differentiates information solely based on signal amplitude without incorporating phase information, each M-PAM symbol can be geometrically represented as one of the one-dimensional signal points with possible values:

$${\displaystyle \frac{\left(m-1\right)A}{M-1}},m=1,2,\dots ,M,$$

(1)

where $A\in \left[0,\left(I_{max}-I_{min}\right)/2\right]$ is the peak amplitude, and $I_{max}$ and $I_{min}$ denote allowable the maximum and minimum input bias currents, respectively. The instantaneous emitted optical intensity signal can be expressed as follows:

$$P_{\mathrm{tx}}=\delta \left(x+B\right),$$

(2)

where $\delta$ denotes the slope efficiency of LD. Meanwhile, $B=I_{max}-A$ is the DC bias which plays a role in guaranteeing that the resulting signal power is non-negative [25].

When an optical signal is transmitted in an underwater environment, the optical signal undergoes both path loss and fading induced by turbulence. Thus, the instantaneous received optical power $P_{\mathrm{rx}}$ at the receive node can be expressed as

$$P_{\mathrm{rx}}=h{P}_{\mathrm{tx}}=h_{\mathrm{AL}}{h}_{\mathrm{GL}}{h}_{\mathrm{F}}{P}_{\mathrm{tx}},$$

(3)

where h is the underwater channel coefficient which is affected by the 3D position of ROV, and it includes the attenuation loss $h_{\mathrm{AL}}$, geometrical loss $h_{\mathrm{GL}}$, and fading $h_{\mathrm{F}}$. Specific definitions of these terms are presented in the next subsection. The solar panel converts the optical intensity into an electrical current. The received electrical signal can be given as follows [17]:

$$i_{\mathrm{RX}}=rh\delta B+rh\delta x+n=I_{\mathrm{DC}}^{\prime }+I_{\mathrm{AC}}+n,$$

(4)

where r is the solar panel responsivity, and n is the additive white Gaussian noise (AWGN) with zero mean and variance of ${\sigma }^2$. In (4), the first term (i.e., $I_{\mathrm{DC}}^{\prime }$) and second term (i.e., $I_{\mathrm{AC}}$) in the right equation are the AC and DC components of the received signal, respectively.

2.3. Underwater Optical Channel Model

In this subsection, we describe each channel component of the underwater channel coefficient defined in (3). In UOWC, the path loss experienced by a transmitted optical signal can be characterized by two components: attenuation loss and geometrical loss. The attenuation loss is caused primarily by the absorption and scattering of light in the water medium, whereas the geometrical loss arises as a result of the transmitted beam spreading and propagating between the ROV and US.

First, for the computation of the attenuation loss in water, many past studies have commonly adopted the Beer–Lambert (BL) formula [26], which assumes that the ROV and US are aligned perfectly. However, under the considered UOWC scenario, misalignment between the ROV and the US is unavoidable because unpredictable shaking and movement of the submerged ROV might occur due to various external factors, even when hovering. To include this issue, the inclination angle ${\theta }_0$, which refers to the angular difference between the center of the transmit node’s optical signal and the receiving node, is modeled as the Gaussian random variable with a mean of ${\overline{\theta }}_0$ and variance of ${\sigma }_{{\theta }_0}^2$ [27]. Therefore, the attenuation loss of an underwater optical signal, accounting for misalignment, can be expressed as

$$h_{\mathrm{AL}}=exp\left\{-c\left(\lambda \right){\displaystyle \frac{d}{cos\left({\theta }_0\right)}}\right\},$$

(5)

where d is the distance between the ROV and the US and $c\left(\lambda \right)$ is the extinction coefficient which is defined as the summation of the absorption coefficient $a\left(\lambda \right)$ and scattering coefficient $b\left(\lambda \right)$, i.e., $c\left(\lambda \right)=a\left(\lambda \right)+b\left(\lambda \right)$. The absorption and scattering coefficients are affected by the wavelength of light and the type of water. Specific values of the absorption and scattering coefficients for different water types at a specific wavelength are presented in [9].

On the other hand, the geometrical loss of an underwater optical signal, accounting for the misalignment, can be expressed as [28]:

$$h_{\mathrm{GL}}=\left\{\begin{array}{c}{\displaystyle \frac{A_{r}cos\left({\theta }_0\right)}{2\pi d^2\left[1-cos\left(\theta \right)\right]}},\theta \ge {\theta }_0\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.$$

(6)

where $A_r$ is the aperture area of the selected optical receiver. It should be noted that $h_{\mathrm{GL}}$ can have a positive value only when the beam-divergence angle $\theta$ at the transmitter is equal to or greater than the inclination angle ${\theta }_0$ (i.e., $\theta \ge {\theta }_0$).

Another important factor affecting the UOWC channel is underwater turbulence, which arises from refractive-index fluctuations caused by variations in the salinity and temperature of the water medium [6]. These turbulence-induced fluctuations result in fluctuations in the received signal intensity, commonly referred to as fading. In particular, in vertical underwater links, the salinity and temperature gradients change with depth. To model the channel such that this characteristic is considered, the underwater vertical link can be approximated as a series of non-mixing layers, each with different properties [7]. Nevertheless, since this work considers short-range communication (i.e., short link distance) between the two nodes for efficient SLIPT, a single underwater vertical link is considered. As such, under the assumption of weak turbulence, which is typically observed for short link distances [29], $h_{\mathrm{F}}$ can be modeled with a log-normal (LN) distribution, and its probability density function (PDF) is given by

$$f_{h_{\mathrm{F}}}\left(h_{\mathrm{F}}\right)={\displaystyle \frac{1}{h_{\mathrm{F}}\sqrt{2\pi \left(4{\sigma }_x^2\right)}}}exp\left(-{\displaystyle \frac{{\left(ln\left(h_{\mathrm{F}}\right)-2{\mu }_x\right)}^2}{2\left(4{\sigma }_x^2\right)}}\right),$$

(7)

where ${\mu }_x$ and ${\sigma }_x$ are the mean and variance, respectively, of the log-amplitude coefficient $X=0.5ln\left(h_{\mathrm{F}}\right)$. Because the fading coefficient does not change the average power, its amplitude should be normalized, i.e., $E\left[h_{\mathrm{F}}\right]=1$, such that ${\mu }_x=-{\sigma }_x^2$ [30]. The variance of ${\sigma }_x$ can be computed as

$${\sigma }_x^2=0.25ln\left(1+{\sigma }_h^2\right),$$

(8)

where ${\sigma }_h^2$ is the scintillation index achieved under spherical waves. Under the assumption of quasi-static fading, $h_{\mathrm{F}}$ remains constant during the symbol interval.

3. Underwater Hybrid Time Switching–Power Splitting (TS–PS) SLIPT

Various SLIPT methods, such as AC–DC separation, TS, PS, and hybrid TS–PS SLIPT methods, have been introduced in [17]. Among them, we adopt the hybrid TS–PS SLIPT method, which, as the name implies, is a combination of the TS and PS methods. More specifically, we denote $T_{\mathrm{EH}}\le T$ and $T_{\mathrm{ID}}\le T$ as the time durations allocated to EH and information decoding (ID), respectively, within the given time duration of a data frame. Because $T_{\mathrm{EH}}+T_{\mathrm{ID}}=T$, these durations are defined by $T_{\mathrm{EH}}=\left(1-\tau \right)T$ and $T_{\mathrm{ID}}=\tau T$, respectively, where $\tau \in \left[0,1\right]$ denotes the time-switching factor. The hybrid TS–PS method consists of two phases and its brief procedure is described as follows:

• In the first phase (referred to as EH mode), with a duration of $T_{\mathrm{EH}}$, only EH is conducted (similar to the TS method). In this phase, there is no need to transfer information, such that DC bias can be set to its maximum value (i.e., $B=I_{max}$) to maximize EH, which leads to $A=0$. The AC component in this phase is blocked by an inductor, such that only the DC component (i.e., $I_{\mathrm{DC}}^{\prime }$) passes through the EH block as illustrated in Figure 2.
• In the second phase (referred to as PS mode) with a duration of $T_{\mathrm{ID}}$, the receiver conducts the PS method to perform EH and ID at the same time. For this, the received signal power is split into two streams using the power-splitting factor $\rho \in \left[0,1\right]$. As a result, $\left(1-\rho \right)i_{\mathrm{RX}}$ and $\rho i_{\mathrm{RX}}$ are dedicated for EH and ID, respectively. Through the suppression of the AC or DC component, the inputs to the EH and ID blocks can be presented as $\left(1-\rho \right)\left({I^{\prime }}_{\mathrm{DC}}+\varsigma {\overline{I}}_{\mathrm{AC}}\right)$ and $\rho \left({\overline{I}}_{\mathrm{AC}}+n\right)$, respectively, where $\varsigma$ is the AC–DC conversion efficiency [31], and ${\overline{I}}_{\mathrm{AC}}=rh\delta \mathrm{E}\left[x\right]=rh\delta A/2$ is the average of the AC component.

3.1. Performance Metric 1: Energy Harvesting

The maximum output power of a solar panel in the EH mode can be achieved at its maximum power point (MPP) [32] and is given as follows [33]:

$$P_{\mathrm{MPP}}=F{I}_{\mathrm{DC}}{V}_{\mathrm{OC}}.$$

(9)

In (9), $V_{\mathrm{OC}}$ is the open circuit voltage which is calculated as $V_{\mathrm{OC}}=V_{t}ln\left(1+I_{\mathrm{DC}}/I_0\right)$, where $V_t$ is the thermal voltage and $I_0$ is the dark saturation current [34]. Furthermore, $I_{\mathrm{DC}}$ is defined as $I_{\mathrm{DC}}={I^{\prime }}_{\mathrm{DC}}+\iota \varsigma {\overline{I}}_{\mathrm{AC}}$, where $\iota$ (which takes a value of 0 or 1) is used to indicate whether or not AC component is used for EH, i.e., if only the DC component is used for EH, then $\iota$ is set to zero. Meanwhile, F is the fill factor defined as $F=I_{\mathrm{MPP}}{V}_{\mathrm{MPP}}/I_{\mathrm{DC}}{V}_{\mathrm{OC}}$, where $I_{\mathrm{MPP}}$ and $V_{\mathrm{MPP}}$ are the optimal values of MPP voltage and MPP current, respectively. The optimal values of $I_{\mathrm{MPP}}$ and $V_{\mathrm{MPP}}$ can be obtained automatically using dynamic tracking techniques for a given temperature and irradiance [35]. Based on (9), we can obtain the energy harvesting condition on $I_{\mathrm{DC}}$ by multiplying $T_{\mathrm{EH}}$ with $P_{\mathrm{MPP}}$ as follows

$$E=T_{\mathrm{EH}}{P}_{\mathrm{MPP}}=T_{\mathrm{EH}}F{I}_{\mathrm{DC}}{V}_{t}ln\left(1+{\displaystyle \frac{I_{\mathrm{DC}}}{I_0}}\right).$$

(10)

3.2. Performance Metric 2: Spectral Efficiency

Similar to [36], a low bound of SE (bps/Hz) for IM/DD systems conditioned on $I_{\mathrm{DC}}$ is expressed as follows:

$$\eta \ge {\displaystyle \frac{T_{\mathrm{ID}}}{T}}\left[{\displaystyle \frac{1}{2}}{log}_2\left(1+{\displaystyle \frac{e}{2\pi }}\beta \right)\right],$$

(11)

where $\beta$ denotes the average electrical signal-to-noise ratio (SNR). Since the AC component carries the information, $\beta$ can be expressed as follows:

$$\beta ={\displaystyle \frac{{\overline{I}}_{\mathrm{AC}}^2}{{\sigma }^2}}={\displaystyle \frac{{\left(rh\delta \mathrm{E}\left[x\right]\right)}^2}{{\sigma }^2}},$$

(12)

where ${\sigma }^2=N_0/2T_s$ is the noise variance and $N_0$ is the noise power spectral density.

4. Proposed Algorithm

In this study, the US conducts underwater SLIPT with the objective of maximizing EH while satisfying the SE requirement for control data reception by jointly controlling the TS and PS ratios. In this point, we emphasize that the EH, as well as SE at the US, is obviously affected by the received electrical signal power, which can vary with the beam-divergence angle chosen by the ROV.

Thus, to achieve this objective, we aim to jointly optimize not only the beam-divergence angle at the transmitting node (i.e., ROV) but also the TS and PS ratios at the receiving node (i.e., US) by solving the following optimization problem:

$$\begin{array}{c}\underset{\tau ,\rho ,\theta }{max}E\left(\tau ,\rho ,\theta \right)\hfill \\ \mathrm{s}.\mathrm{t}.\eta \left(\tau ,\rho ,\theta \right)\ge {\eta }_{th},\hfill \\ 0\le \tau ,\rho \le 1,\hfill \\ {\theta }_{min}\le \theta \le {\theta }_{max}.\hfill \end{array}$$

(13)

To solve problem (13), we can decompose the problem into two subproblems. This is because when determining the beam-divergence angle, the TS and PS ratios do not have an impact. Conversely, when determining the TS and PS ratios, the beam-divergence angle does have an effect. More specifically, the first subproblem (P1) is to determine the beam-divergence angle (i.e., $\theta$) of the ROV to maximize the received electrical signal power (3) at the US while maintaining a seamless connection between the two nodes:

$$\begin{array}{c}\left(\mathrm{P}1\right)\underset{\theta }{max}{i}_{\mathrm{RX}}\left(\theta \right)\hfill \\ \mathrm{s}.\mathrm{t}.{\theta }_{min}\le \theta \le {\theta }_{max}.\hfill \end{array}$$

(14)

On the other hand, the second subproblem (P2) is to simultaneously determine both the TS and PS ratios (i.e., $\tau$ and $\rho$) at a given $i_{\mathrm{RX}}\left(\theta \right)$ to maximize EH while supporting the SE requirement at the US:

$$\begin{array}{c}\left(\mathrm{P}2\right)\underset{\tau ,\rho }{max}E\left(\tau ,\rho |i_{\mathrm{RX}}\left(\theta \right)\right)\hfill \\ \mathrm{s}.\mathrm{t}.\eta \left(\tau ,\rho \right)\ge {\eta }_{th},\hfill \\ 0\le \tau ,\rho \le 1.\hfill \end{array}$$

(15)

Since each subproblem should be solved on different sides (i.e., (P1) and (P2) on the ROV and US sides, respectively), we refer to the ROV and US as agents, each of which solves the subproblem on its side.

4.1. ROV Agent: Beam-Divergence Angle Adaptation

The role of the ROV agent is to choose the beam-divergence angle $\theta$ at every time slot by learning the inclination angle ${\theta }_0$ between the ROV and US. We note that the inclination angle between the two nodes can vary at every time slot because of the unpredictable shaking and movement of the submerged ROV due to various external factors in the ocean. To solve problem (P1) for an underwater environment, we can define a Markov decision process (MDP) for the ROV agent, which can be represented as a tuple $\left({\mathbf{S}}^{\mathrm{ROV}},{\mathbf{A}}^{\mathrm{ROV}},r^{\mathrm{ROV}}\right)$ where ${\mathbf{S}}^{\mathrm{ROV}}$, ${\mathbf{A}}^{\mathrm{ROV}}$, and $r^{\mathrm{ROV}}$ refer to the state space, action space, and reward for the ROV agent.

We denote $a^{\mathrm{ROV}}\left(t\right)=\theta \left(t\right)\in {\mathbf{A}}^{\mathrm{ROV}}$ as an element in action space ${\mathbf{A}}^{\mathrm{ROV}}$ which represents a set of discrete beam-divergence angles in degrees that the ROV can choose at time:

$${\mathbf{A}}^{\mathrm{ROV}}\left(t\right)=\left\{{\theta }_{min},{\theta }_{min}+\alpha ,\dots ,{\theta }_{max}-\alpha ,{\theta }_{max}\right\},$$

(16)

where ${\theta }_{min}$ and ${\theta }_{max}$ are the minimum and maximum beam-divergence angles supported by the optical modem installed on the ROV, respectively, and $\alpha$ is the gap between two consecutive beam-divergence angles.

We also denote ${\mathbf{S}}^{\mathrm{ROV}}\left(t\right)$ as the state space at time slot t, which includes various information that affects the action selection of the ROV agent:

$${\mathbf{S}}^{\mathrm{ROV}}\left(t\right)=\left\{{\mathbf{s}}_{\mathrm{his}}^{\mathrm{ROV}}\left(t-1\right),{\theta }_0\left(t-1\right),{\theta }_{\mathrm{gap}}\left(t-1\right)\right\},$$

(17)

where ${\mathbf{s}}_{\mathrm{his}}^{\mathrm{ROV}}\left(t-1\right)$ contains historical information on the actions and rewards experienced by the ROV agent, ${\theta }_0\left(t-1\right)$ refers to the inclination angle at time slot $t-1$, and ${\theta }_{\mathrm{gap}}\left(t-1\right)$ refers to the difference between $\theta \left(t-1\right)$ and ${\theta }_0\left(t-1\right)$. When defining ${\mathbf{s}}_{\mathrm{his}}^{\mathrm{ROV}}\left(t-1\right)$, we adopt the concept of a sliding window of size l at each time slot to limit the size of the state space as the learning progresses. Thus, ${\mathbf{s}}_{\mathrm{his}}\left(t-1\right)$ can be expressed as follows:

$${\mathbf{s}}_{\mathrm{his}}^{\mathrm{ROV}}\left(t-1\right)=\left\{a^{\mathrm{ROV}}\left(t-l\right),r^{\mathrm{ROV}}\left(t-l\right),\dots ,a^{\mathrm{ROV}}\left(t-1\right),r^{\mathrm{ROV}}\left(t-1\right)\right\}.$$

(18)

For the reward function of the ROV agent, we adopt the received electrical signal power defined in (4) under the chosen action as follows:

$$r^{\mathrm{ROV}}\left(t\right)=i_{\mathrm{RX}}\left(\theta \left(t\right)\right).$$

(19)

The reward data obtained at the US are fed back to the ROV during the transmission of sensing data from the US to the ROV via LED-based UOWC.

4.2. US Agent: SLIPT Adaptation

Given the received electrical signal power $i_{\mathrm{RX}}\left(\theta \left(t\right)\right)$ affected by the action of the ROV agent, the US agent aims to maximize EH at the US while guaranteeing the SE requirement for control data transmission. For this purpose, we can also define an MDP for the US agent, consisting of a tuple $\left({\mathbf{S}}^{\mathrm{US}},{\mathbf{A}}^{\mathrm{US}},r^{\mathrm{US}}\right)$ where ${\mathbf{S}}^{\mathrm{US}}$, ${\mathbf{A}}^{\mathrm{US}}$, and $r^{\mathrm{US}}$ refer to the state space, action space, and reward for the US agent.

Unlike the ROV agent, which chooses only one discrete value (i.e., beam-divergence angle), the US agent needs to determine two different continuous values, i.e., the TS and PS ratios, such that the action space can be defined as follows:

$${\mathbf{A}}^{\mathrm{US}}\left(t\right)=\left\{\tau ,\rho \right\}.$$

(20)

The state space for the US agent includes historical information on the actions and rewards experienced by the US agent as well as current channel quality between the two nodes as follows:

$${\mathbf{S}}^{\mathrm{U}S}\left(t\right)=\left\{{\mathbf{s}}_{\mathrm{his}}^{\mathrm{U}S}\left(t-1\right),h\left(t\right)\right\}.$$

(21)

Similar to that done for the ROV agent, the concept of a sliding window of size l is adopted to define ${\mathbf{s}}_{\mathrm{his}}^{\mathrm{U}S}\left(t-1\right)$ as follows:

$${\mathbf{s}}_{\mathrm{his}}^{\mathrm{U}S}\left(t-1\right)=\left\{a^{\mathrm{U}S}\left(t-l\right),r^{\mathrm{U}S}\left(t-l\right),\dots ,a^{\mathrm{U}S}\left(t-1\right),r^{\mathrm{U}S}\left(t-1\right)\right\}.$$

(22)

To achieve the objective of the ROV agent, we define a reward function, which is influenced by the chosen action set (i.e., $\left\{\tau ,\rho \right\}$):

$$r^{\mathrm{US}}\left(t\right)=\left\{\begin{array}{c}E\left(\tau ,\rho |\theta \right)\\ \eta \left(\tau ,\rho |\theta \right)-{\eta }_{th}\end{array}\right.\begin{array}{c},\eta \ge {\eta }_{th}\\ ,\mathrm{otherwise}.\end{array}$$

(23)

The aforementioned reward indicates that if the SE requirement is satisfied, the reward is set to the EH at the US. Otherwise, the reward sets the difference between the achieved and required SE to prevent the achieved value from becoming smaller than the required value.

4.3. Proposed Algorithm

To obtain the solutions of such two MDPs, this study proposes a hierarchical DQN–DDPG-based online learning algorithm that determines not only the beam-divergence angle at the ROV but also the TS and PS ratios at the US in real time. The conceptual structure of the proposed hierarchical DQN–DDPG-based online learning algorithm is illustrated in Figure 3.

First, since the role of the ROV agent is to determine the discrete beam-divergence angle, it adopts the DQN algorithm [37]. At each time t, the ROV agent constructs state $s^{\mathrm{ROV}}\left(t\right)\in {\mathbf{S}}^{\mathrm{ROV}}$, for which it needs to gather historical information on its reward from the US through feedback. As mentioned previously, although such feedback data are transmitted together with the sensing data, LED-based UOWC from the US to the ROV can offer a sufficient data rate (e.g., more than Mbps) to transmit them. After constructing $s^{\mathrm{ROV}}\left(t\right)$, the ROV agent makes a decision to choose the beam-divergence angle based on the $\epsilon$-greedy algorithm. The ROV agent chooses its action by the following equation with probability $1-\epsilon$:

$$a^{\mathrm{ROV}}\left(t\right)=\underset{a\in {\mathbf{A}}^{\mathrm{ROV}}\left(t\right)}{argmax}{Q}^{\mathrm{ROV}}\left(s^{\mathrm{ROV}}\left(t\right),a|\Theta \right),$$

(24)

where $Q^{\mathrm{ROV}}\left(s^{\mathrm{ROV}}\left(t\right),a\right)$ is the Q-function achieved by action a in state $s^{\mathrm{ROV}}\left(t\right)$, and $\Theta$ is a set of weights for the deep neural network (DNN) of the ROV agent. With probability $\epsilon$, it randomly chooses its action in the action space ${\mathbf{A}}^{\mathrm{ROV}}\left(t\right)$. At each time slot, the weights of the DNN are updated using the mean-squared error (MSE) loss function as follows:

$${\mathcal{L}}^{\mathrm{ROV}}\left(\Theta \right)=\underset{e^{\mathrm{ROV}}\sim {\mathcal{D}}^{\mathrm{ROV}}}{\mathbb{E}}{\left[y^{\mathrm{ROV}}-Q^{\mathrm{ROV}}\left(s^{\mathrm{ROV}}\left(t\right),a^{\mathrm{ROV}}\left(t\right)|\Theta \right)\right]}^2,$$

(25)

where ${\mathcal{D}}^{\mathrm{ROV}}$ denotes the experience replay buffer for the ROV agent which contains tuples $e^{\mathrm{ROV}}=\left(s^{\mathrm{ROV}},a^{\mathrm{ROV}},r^{\mathrm{ROV}},{s^{\mathrm{ROV}}}^{\prime }\right)$. Meanwhile, $y^{\mathrm{ROV}}$ is the target value for updating $\Theta$, and is expressed as follows:

$$y^{\mathrm{ROV}}=r^{\mathrm{ROV}}+\gamma \underset{a}{max}{Q}^{\mathrm{ROV}}\left({s^{\mathrm{ROV}}}^{\prime },a|\tilde{\Theta }\right),$$

(26)

where $\gamma$ denotes the discount factor, and $\tilde{\Theta }$ denotes the set of weights for the target network. Algorithm 1 describes the DQN algorithm implemented by the ROV agent.

On the other hand, unlike the ROV agent, which chooses a discrete value as an action, the role of the US agent is to determine two contribution values (i.e., TS and PS ratio) with the same bound (i.e., $0\le \tau ,\rho \le 1$) as an action. For this, we adopt the DDPG algorithm [38], which is one of the representative deep RL (DRL) algorithms for finding a continuous action vector. Let $Q^{\mathrm{US}}\left(s,a|{\theta }_Q\right)$ be a critic network with weights ${\theta }_Q$ that estimate the Q-function. Additionally, let $\mu \left(s|{\theta }_{\mu }\right)$ be the actor network with weights ${\theta }_{\mu }$ which specifies the current policy by deterministically mapping states to a specific action. Then, the gradient of the accumulated discounted reward (denoted as J) can be expressed as follows:

$${\nabla }_{{\theta }_{\mu }}J=\underset{e^{\mathrm{US}}\sim {\mathcal{D}}^{\mathrm{US}}}{\mathbb{E}}\left[{\nabla }_{{\theta }_{\mu }}\mu \left(s|{\theta }_{\mu }\right){|}_{s=s^{\mathrm{US}}}{\nabla }_a{Q}^{\mathrm{US}}\left(s,a|{\theta }_Q\right){|}_{s=s^{\mathrm{US}},a=\mu \left(s^{\mathrm{US}}\right)}\right],$$

(27)

where ${\mathcal{D}}^{\mathrm{US}}$ denotes the experience replay buffer for the US agent, which contains tuples $e^{\mathrm{US}}=\left(s^{\mathrm{US}},a^{\mathrm{US}},r^{\mathrm{US}},{s^{\mathrm{US}}}^{\prime }\right)$. Similar to the ROV agent, the US agent updates its Q-function by minimizing the MSE loss function as follows:

$$\begin{array}{cc}\hfill {\mathcal{L}}^{\mathrm{US}}\left({\theta }_Q\right)& =\underset{e^{\mathrm{US}}\sim {\mathcal{D}}^{\mathrm{US}}}{\mathbb{E}}\left[{\left(y^{\mathrm{US}}-Q^{\mathrm{US}}\left(s^{\mathrm{US}},a^{\mathrm{US}}|{\theta }_Q\right)\right)}^2\right],\hfill \end{array}$$

(28)

where $y^{\mathrm{US}}$ is the target value for updating $Q^{\mathrm{US}}$, and is also expressed as follows:

$$y^{\mathrm{US}}=r^{\mathrm{US}}+\gamma Q^{\mathrm{US}}\left({s^{\mathrm{US}}}^{\prime },\mu \left({s^{\mathrm{US}}}^{\prime }|{\tilde{\theta }}_{\mu }\right)|{\tilde{\theta }}_Q\right),$$

(29)

where ${\tilde{\theta }}_Q$ and ${\tilde{\theta }}_{\mu }$ are the sets of the weights of the target network with respect to $Q^{\mathrm{US}}$ and $\mu$, respectively. Algorithm 2 explains the DDPG algorithm implemented by the US agent.

In Algorithm 2, $\mathcal{N}$ is the noise process for constructing the exploration policy, $\phi$ is a predetermined value for the repetitive initialization of $\mathcal{N}$, and $\omega$ is the weight for soft target updates, and X is the number of samples in the mini-batch.

To facilitate a clear understanding of Algorithms 1 and 2 described earlier, we have provided flow charts for each algorithm, as shown in Figure 4.

Algorithm 1: DQN algorithm for determining the beam-divergence angle at the ROV agent.

Algorithm 2: DDPG algorithm for determining TS and PS ratios at the US agent.

4.4. Verification for Online Operation via Computational Complexity Analysis

We analyze the time computational complexity of the proposed hierarchical DQN– DDPG algorithm using big O notation denoted by $\mathrm{O}\left[·\right]$.

In the training stage, the ROV agent first executes Algorithm 1 (i.e., beam-divergence angle decision algorithm), which is based on the DQN consisting of two DNNs having the same structure, e.g., a Q-network and a target network. Let $L^{\mathrm{DQN}}$ and $m_l^{\mathrm{DQN}}$ as the number of layers of the DNNs and the number of neurons in the l-th layer among $L^{\mathrm{DQN}}$ layers. According to [39], the computational complexity of each training step in the DNN using a fully connected network can be presented as $O\left[{{\rm Y}}^{\mathrm{DQN}}{\displaystyle \sum _{l=1}^{L^{\mathrm{DQN}}-1}}{m}_l^{\mathrm{DQN}}{m}_{l+1}^{\mathrm{DQN}}\right]$, where ${{\rm Y}}^{\mathrm{DQN}}$ is the mini-batch size of the DQN. Thus, total training computational complexity of Algorithm 1 is $O\left[2T_{\mathrm{C}\mathrm{V}}{{\rm Y}}^{\mathrm{DQN}}{\displaystyle \sum _{l=1}^{L^{\mathrm{DQN}}-1}}{m}_l^{\mathrm{DQN}}{m}_{l+1}^{\mathrm{DQN}}\right]$, where $T_{\mathrm{C}\mathrm{V}}$ is the number of time slots until performance of the hierarchical DQN–DDPG algorithm converges. Next, the US agent executes Algorithm 2 (i.e., TR and PS ratios decision algorithm), which is based on the DDPG network consisting of two DNNs with a different structure, e.g., an actor network and a critic network. Assuming that that the actor and critic networks contain $L^{\mathrm{ACT}}$ and $L^{\mathrm{CRIC}}$ fully connected layers, respectively, total training computational complexity of Algorithm 2 is $O\left[T_{\mathrm{C}\mathrm{V}}{{\rm Y}}^{\mathrm{DDPG}}\left({\displaystyle \sum _{l=1}^{L^{\mathrm{ACT}}-1}}{m}_l^{\mathrm{ACT}}{m}_{l+1}^{\mathrm{ACT}}+{\displaystyle \sum _{l=1}^{L^{\mathrm{CRIC}}-1}}{m}_l^{\mathrm{CRIC}}{m}_{l+1}^{\mathrm{CRIC}}\right)\right]$, where $m_l^{\mathrm{ACT}}$ and $m_l^{\mathrm{CRIC}}$ are the numbers of neurons in the l-th layer among $L^{\mathrm{ACT}}$ and $L^{\mathrm{CRIC}}$ layers, respectively: ${{\rm Y}}^{\mathrm{DDPG}}$ is the mini-batch size of the DDPG network. As a result, in the training stage, total computational complexity of the proposed hierarchical DQN–DDPG algorithm is $O\left[2T_{\mathrm{C}\mathrm{V}}{{\rm Y}}^{\mathrm{DQN}}{\displaystyle \sum _{l=1}^{L^{\mathrm{DQN}}-1}}{m}_l^{\mathrm{DQN}}{m}_{l+1}^{\mathrm{DQN}}\right]+O\left[T_{\mathrm{C}\mathrm{V}}{{\rm Y}}^{\mathrm{ACT}}\left({\displaystyle \sum _{l=1}^{L^{\mathrm{ACT}}-1}}{m}_l^{\mathrm{ACT}}{m}_{l+1}^{\mathrm{ACT}}+{\displaystyle \sum _{l=1}^{L^{\mathrm{CRIC}}-1}}{m}_l^{\mathrm{CRIC}}{m}_{l+1}^{\mathrm{CRIC}}\right)\right]$.

In the test stage, the computational complexity of the proposed hierarchical DQN– DDPG algorithm in each time slot can be dramatically reduced to $O\left[{\displaystyle \sum _{l=1}^{L^{\mathrm{DQN}}-1}}{m}_l^{\mathrm{DQN}}{m}_{l+1}^{\mathrm{DQN}}\right]+O\left[\left({\displaystyle \sum _{l=1}^{L^{\mathrm{ACT}}-1}}{m}_l^{\mathrm{ACT}}{m}_{l+1}^{\mathrm{ACT}}\right)\right]$. This is because once the performances of the networks finally converge, we do not need iterations for updating the target network for DQN and the critic network for the DDPG network with reply buffers, respectively. This indicates that the proposed algorithm is capable of being implemented in real-time operations.

5. Simulation Results

To assess the validity of the proposed algorithm, we conduct numerical simulations to evaluate the performance of the proposed hierarchical DQN–DDPG algorithm and then compare it with those of a number of already existing UOWC algorithms. For simulations, a UOWC scenario between the ROV and the US is considered in which the ROV is randomly shaking even when hovering. As aforementioned, the shaking of the ROV is affected by the inclination angle ${\theta }_0$ which is modeled as a Gaussian random variable with a mean of ${\overline{\theta }}_0$ and variance of ${\sigma }_{{\theta }_0}^2$.

5.1. Simulation Parameters

For numerical simulations, we set static parameters as follows: ${\theta }_{min}=3^{\circ }$, ${\theta }_{max}=5^{\circ }$, $\varsigma =1$, $T=1$ s, $\overline{\theta }=2.5^{\circ }$, ${\sigma }_{{\theta }_0}=0.5$, and $d=10$ m, respectively. Also, the size of sliding window l for ${\mathbf{s}}_{\mathrm{his}}^{\mathrm{ROV}}\left(t-1\right)$ and ${\mathbf{s}}_{\mathrm{his}}^{\mathrm{US}}\left(t-1\right)$ is set to 3. Other static system and channel parameters adopted for our simulations are from [17], which are summarized in

Table 1. List of static network and channel parameters [17].

Parameter	Value
Time duration of a data frame T	1 s
Distance between ROV and US	10 m
Receiver Aperture diameter $A_r$	0.2 m
Extinction coefficient $c\left(\lambda \right)$	0.15 (clean ocean)
Solar panel responsibility r	0.6 A/W
Slope efficiency of LD $\delta$	1.33 W/A
Maximum input bias current $I_{max}$	1.2 A
Minimum input bias current $I_{min}$	0.2 A
Fill factor F	0.75
Thermal voltage $V_t$	0.025 W
Dark saturation current $I_0$	${10}^{-9}$ A
Noise power spectral density $N_0$	${10}^{-19}$ W/Hz
Mean of inclination angle ${\overline{\theta }}_0$	0.0436 rad
Standard deviation of inclination angle ${\sigma }_{{\theta }_0}$	0.0087 rad

.

Regarding the learning environment, the DQN structure for the ROV agent is a fully connected neural network with two hidden layers containing 68 neurons in each layer. Other learning hyperparameters for the DQN are summarized in

Table 2. List of DQN hyperparameters.

Hyperparameter	Agent
$ϵ$ for $ϵ$-greedy	0.01
Mini-batch size	64
Optimizer	Adam
Activation function	Relu
Learning rate	${10}^{-4}$
Experience replay buffer size	2000
Discount factor	0.99
Considered time slots for ${\mathbf{s}}_{\mathrm{his}}\left(t\right)$	2

.

On the other hand, the DDPG structure for the US agent is a fully connected neural network with two hidden layers, where the first and second hidden layers contain 512 and 256 neurons, respectively. Other learning hyperparameters for DDPG are summarized in

Table 3. List of hyperparameters for DDPG networks.

Parameter	Value
Mini-batch size	64
Experience replay buffer size	${10}^6$
Discount factor	0.99
Learning rate of actor	${10}^{-4}$
Learning rate of critic	3 × ${10}^{-4}$
Soft update rate of target parameters	2 × ${10}^{-1}$

. These two DRL agents are implemented using Keras Python libraries with a TensorFlow backend.

5.2. Benchmark Algorithms

For the performance comparison, the proposed algorithm is compared with these already existing SLIPT algorithms: AC–DC separation (ADS) method [40], TS method [41], and PS method [42]. In the ADS algorithm, the AC (e.g., $I_{\mathrm{AC}}+n$) and DC (e.g., $I_{\mathrm{DC}}^{\prime }$) components of the received signal (4) are separated by the capacitor and inductor and are then used for ID and EH at the receiver, respectively. Since ID and EH are conducted simultaneously at the receiver, we set $T_{\mathrm{ID}}=T_{\mathrm{EH}}=T$. In the TS method, the receiver switches only in time between the ID and EH modes by a factor of $\tau$. In other words, the TS method is equivalent to executing only the first phase (i.e., EH method) of the hybrid TS–PS method described in Section 3. On the other hand, in the PS method, the received electrical power $i_{\mathrm{RX}}$ is split into two streams with a factor of $\rho$, i.e., $\left(1-\rho \right)i_{\mathrm{RX}}$ and $\rho i_{\mathrm{RX}}$, which are used for EH and ID, respectively, during the time duration of a data frame ($T_{\mathrm{ID}}=T_{\mathrm{EH}}=T$). In other words, the PS method is equivalent to executing only the second phase (i.e., PS method) of the hybrid TS–PS method.

5.3. Simulation Results

Figure 5 compares the performance of the proposed algorithm with those of existing algorithms at a given $\eta =5$ [bps/Hz]. In the figure, the deterministic TS and PS methods execute the TS and PS algorithms with deterministic TS and PS ratios (e.g., $\tau =\rho =0.5$ in this simulation). By contrast, the adaptive TS and PS methods refer to the DRL algorithm, which adaptively determines only the TS and PS ratios, respectively. For a fair comparison, we set the state space and reward function of the adaptive TS and PS methods to be the same as those of the US agent of the proposed algorithm. Regarding the performance metrics for comparison, we employ the moving averages of EH and SE with window sizes of 100. From Figure 5, it can be observed that the proposed algorithm achieves at least an 11% improvement in EH performance while meeting the communication requirements at the US, compared to benchmark SLIPT algorithms. This is because, by learning a time-varying underwater environment, the proposed algorithm can conduct online and adaptive control of the optimization parameters (e.g., TS and PS ratios and beam-divergence angle) in (13) to achieve our objective. Furthermore, although both the adaptive TS and PS methods can fulfill the SE requirement, the EH performance of the adaptive PS method is better than that of the adaptive TS method under the given SE requirement. The difference in EH performance between the two methods can vary according to changes in the SE requirement at the US. By contrast, in the case of the deterministic TS and PS methods, they cannot fulfill the SE requirement, which limits the utilization of these algorithms in the considered network scenario.

Figure 6 presents the performance of the proposed algorithm with respect to variations in the SE requirement at the US, demonstrating that the proposed algorithm can fulfill a variety of SE requirements. Moreover, it is observed that as the SE requirement increases, the EH performance at the US decreases. This is because, to achieve a higher SE requirement, more time or more power should be allocated, which leads to a decrease in the amount of EH.

Figure 7 presents a performance comparison between the proposed algorithm when both the ROV and US agents are considered (hereafter referred to as the proposed algorithm with two agents) and the proposed algorithm when only the US agent is considered (hereafter referred to as the proposed algorithm with only the US agent) under given $\theta ={\theta }_{min}$ and $\theta ={\theta }_{max}$, respectively. From Figure 7, it can be observed that the proposed algorithm with two agents exhibits the best EH performance while guaranteeing SE requirements. By contrast, the constrained case, i.e., the proposed algorithm with only the US agent under given $\theta ={\theta }_{min}$, exhibits severe performance degradation. This is because, at $\theta ={\theta }_{min}$, cases wherein the chosen beam-divergence angle is smaller than the inclination angle (i.e., $\theta <{\theta }_0$) frequently occur due to irregular shaking or movement, which results in $i_r=0$. Moreover, although the proposed algorithm with only the US agent under given $\theta ={\theta }_{max}$ supports a stable and better EH performance compared with that under given $\theta ={\theta }_{min}$, its performance is still worse than that of the proposed algorithm with two agents. This is because although the maximum beam-divergence angle may have an advantage for a seamless connection, it offers the worst SNR performance when the link is connected. This result demonstrates the validity of the adaptive control of the beam-divergence angle at the ROV in the proposed algorithm.

Furthermore, to check whether the beam-divergence angle chosen at the ROV implementing the proposed algorithm is adaptively controlled depending on the degree of misalignment, we conduct simulations that measure the average of the beam-divergence angles chosen at the ROV agent for 10,000 time slots with respect to changes in the mean of inclination angle, i.e., ${\overline{\theta }}_0$. In Figure 8, it can be observed that as the degree of misalignment between the ROV and US is growing larger (i.e., increase in ${\overline{\theta }}_0$), the ROV chooses a wider beam-divergence angle. For example, when there is a slight misalignment (i.e., ${\overline{\theta }}_0=2$), the ROV chooses the minimum beam-divergence angle, i.e., $\theta ={\theta }_{min}=3$. By contrast, as the change in misalignment becomes larger, a wider beam is selected, and eventually, the widest beam, i.e., $\theta ={\theta }_{max}=5$, is selected. This tendency is reasonable because the proposes algorithm sets the beam-divergence angle as narrow as possible to maximize the SNR under the assumption of no change in the misalignment; however, in the opposite case, it sets the beam as wide as possible to prevent disconnection between the two nodes.

6. Conclusions

This work studied an adaptive control mechanism for a UOWC between the ROV and a US endowed with SLIPT capabilities. The primary goal is to maximize EH at the US while sustaining a predefined SE performance level between the two nodes. To address this objective, we proposed a hierarchical DQN–DDPG-based online algorithm that involves two RL agents: the ROV agent, which optimizes the beam-divergence angle to enhance the received optical power at the US while maintaining an uninterrupted optical link, and the US agent, which determines the TS and PS ratios to maximize EH without compromising the SE requirements. Extensive simulation results demonstrated the effectiveness of the proposed algorithm, achieving at least an 11% improvement in EH performance while meeting the communication requirements at the US, compared to benchmark SLIPT algorithms. The adaptability of the algorithm to dynamically adjust optimization parameters in response to varying underwater environmental conditions and user requirements enhances the integration of energy transfer and communication in underwater contexts. Furthermore, the exploration of additional communication performance requirements within the proposed optimization framework will be addressed as part of future research.