Improvement of the Control of a Grid Connected Photovoltaic System Based on Synergetic and Sliding Mode Controllers Using a Reinforcement Learning Deep Deterministic Policy Gradient Agent

Abstract

This article presents the control of a grid connected PV (GC-PV) array system, starting from a benchmark. The control structure used in this article was a cascade-type structure, in which PI or synergetic (SYN) controllers were used for the inner control loop of i_d and i_q currents and PI or sliding mode control (SMC) controllers were used for the outer control loop of the u_dc voltage from the DC intermediate circuit. This paper presents the mathematical model of the PV array together with the main component blocks: simulated inputs for the PV array; the PV array itself; the MPPT algorithm; the DC-DC boost converter; the voltage and current measurements for the DC intermediate circuit; the load and connection to power grid; the DC-AC converter; and the power grid. It also presents the stages of building and training the reinforcement learning (RL) agent. To improve the performance of the control system for the GC-PV array system without using controllers with a more complicated mathematical description, the advantages provided by the RL agent on process controls could also be used. This technique does not require exact knowledge of the mathematical model of the controlled system or the type of uncertainties. The improvement in the control system performance for the GC-PV array system, both when using simple PI-type controllers or complex SMC- and SYN-type controllers, was achieved using an RL agent based on the Deep Deterministic Policy Gradient (DDPG). The variant of DDPG used in this study was the Twin-Delayed (TD3). The improvement in performance of the control system were obtained by using the correction command signals provided by the trained RL agent, which were added to the command signals u_d, u_q and i_dref. The parametric robustness of the proposed control system based on SMC and SYN controllers for the GC-PV array system was proven in the case of a variation of 30% caused by the three-phase load. Moreover, the results of the numerical simulations are shown comparatively and the validation of the synthesis of the proposed control system was obtained. This was achieved by comparing the proposed system with a software benchmark for the control of a GC-PV array system performed in MATLAB Simulink. The numerical simulations proved the superiority of the performance of control systems that use the RL-TD3 agent.

1. Introduction

The importance of studying renewable energies from the phenomena of their generation, from sources including solar, wind, water, geothermal, etc., to their integration into microgrids or main grids is undeniable [1].

In parallel with these studies, studies on the control systems used for the generation of energy from renewable sources have also been intensified. Thus, there have been studies on hybrid microgrids [2,3,4], the optimization of the process for battery charging in microgrids [5,6], the optimization of converters in microgrids [7,8], problems regarding the defects that can occur in microgrids [9], as well as elements regarding the dispatching of microgrids by economic criteria [10,11,12,13,14].

A specific problem that is addressed in this article is the control system for the connection of the PV array system to a main grid. This problem involves the study of a chain of primary elements, consisting of the following blocks: the inputs for the PV array; the PV array itself; the MPPT algorithm; the DC-DC boost converter; the voltage and current measurements for the DC parameters in intermediate circuit; the DC-AC converter; the PLL (phase locked loop); the load and connection to power grid; and the core element that controls these blocks, called the voltage-source converter (VSC). The main objective of the control system is to stabilize the u_dc voltage as precisely as possible, including under variation caused by the three-phase load [15].

To achieve this goal, a series of adaptive control-type [16], robust control-type [17,18] and predictive control-type [19] algorithms can be used. Furthermore, fuzzy logic and neuro-fuzzy systems [20,21], genetics [22], particle swarm optimization (PSO) [23], RL [24], and passivity theory control systems are a special category of these control systems [25].

Given that the description equations of GC-PV array systems are nonlinear, a control system that ensures parametric robustness is provided by the SMC [26]. The SYN control systems [27], which can be considered as an extension of the SMC, also receive special emphasis.

The control systems that are based on RL for process control are organized as a series of tasks, which run on a computer for the control of an industrial process but do not require an explicit mathematical description [28,29,30].

This article starts from a benchmark presented in MATLAB Simulink [15], which was resumed in order to compare the best results obtained in [26,27,31,32]. After presenting the main characteristics of the benchmark system, numerical simulations are reported based on the theoretical elements presented in the preceding sections. Thus, starting from the cascade control structure in which the PI-type controllers were used in the inner control loop of i_d and i_q currents, an SMC-type controller was used in the outer control loop of u_dc voltage and the elements regarding the RL-TD3 agent were used, the superior performance of the control system for the GC-PV array was obtained.

Moreover, in the second part of the numerical simulations, starting from the peak performances presented in [32,33] regarding the cascade control system in which an SYN-type controller was used in the inner control loop of i_d and i_q currents, an SMC-type controller was used in the outer control loop of u_dc voltage and the elements regarding the RL-TD3 agent were used, the superior performance of the control system for the GC-PV array was obtained both in terms of the direct comparison of these performances and the robustness provided by the control system under parametric variations, such as the variation caused by the three-phase load.

The main contributions of this article are as follows:

• The proposal of a cascade control system structure for the GC-PV array system, in which an SMC-type controller is used for the outer u_dc voltage in the DC circuit control loop and SYN-type controllers are used in the inner control loops in the i_d and i_q currents;
• Improvements in the performance of the control system for the GC-PV array system when using simple PI-type controllers or complex SMC-type or SYN-type controllers through the use of an RL agent that is based on TD3;
• Validations of the results performed through a MATLAB Simulink environment to show the improvements in the performance of the control system for the GC-PV array system by using the RL-TD3 agent, even under parametric uncertainties; for example, a variation of 30% from the nominal value caused by the three-phase load.

The rest of the paper is organized as follows. Section 2 presents the mathematical model of the GC-PV array system. Section 3 describes the RL agent used for process control. Section 4 presents a correction of the control signals and the MATLAB Simulink implementation of the control for the GC-PV array system based on PI-type controllers using the RL-TD3 agent. Section 5 presents a correction and MATLAB Simulink implementation of the command signals for the control system for the GC-PV array system based on SMC- and SYN-type controllers using the RL-TD3 agent. The results of the numerical simulations are presented in Section 6 and Section 7 presents our conclusions.

2. Grid Connected PV Array System: The Mathematical Model

The schematic block of the main circuit for the GC-PV system is presented in Figure 1 [15,27,31]. The input quantities for the PV array model were provided by radiation and temperature. A component of utmost importance was the power point tracking (MPPT) module, which acted on the DC boost converter to obtain the maximum efficiency of the energy received from the PV array. A detailed description of the MPPT is presented in [15,26,31]. A three-phase DC–AC converter, which powered a three-phase load, was added to the diagram in Figure 1. The controller proposed and described in the following sections acted on the three-phase DC–AC converter in order to stabilize the u_dc voltage as precisely as possible, even under a significant variation caused by the three-phase load. Using the notations presented in Figure 1, Equations (1)–(4) could be written as below:

$$C_1\frac{d{u}_{PV}}{dt}=i_{PV}-i_s$$

(1)

$$u_{PV}=R_1{i}_s+L_1\frac{d{i}_s}{dt}+u_s$$

(2)

$$C_2\frac{d{u}_{dc}}{dt}=i_{dc1}-i_{dc2}$$

(3)

$$u_{abc}-e_{abc}=R_3{i}_{abc}+L_3\frac{{di}_{abc}}{dt}$$

(4)

where the output voltages of the DC–AC is noted with u_abc (represented by the VSC with the form $u_{abc}={\left[\begin{array}{ccc}{u}_a& u_b& u_c\end{array}\right]}^T$), the grid voltages are denoted by e_abc with the form $e_{abc}={\left[\begin{array}{ccc}{e}_a& e_b& e_c\end{array}\right]}^T$ and the alternating currents are denoted by i_abc with the form $i_{abc}={\left[\begin{array}{ccc}{i}_a& i_b& i_c\end{array}\right]}^T$.

In Equation (5), the well-known Park’s transformation based on P matrix is presented:

$$P=\left[\begin{array}{ccc}\sin \left(\omega t\right)& \sin {\displaystyle \left(\omega t-\frac{2\pi }{3}\right)}& \sin {\displaystyle \left(\omega t+\frac{2\pi }{3}\right)}\\ \cos \left(\omega t\right)& \cos (\omega t-\frac{2\pi }{3})& \cos (\omega t+\frac{2\pi }{3})\\ \frac{1}{2}& \frac{1}{2}& \frac{1}{2}\end{array}\right]$$

(5)

The transformation from the coordinates abc reference frame to the d−q reference frame was performed using Equation (5): u_dq0 = Pu_abc, e_dq0 = Pe_abc, i_dq0 = = Pi_abc. Equation (4) transformed as follows:

$$u_{dq0}-e_{dq0}=R_3{i}_{dq0}+L_3\frac{d{i}_{dq0}}{dt}+L_3\left[\begin{array}{c}-\omega i_q\\ \omega i_d\\ 0\end{array}\right]$$

(6)

Equation (6) could be written by components as follows:

$$L_3\frac{d{i}_d}{dt}=-R_3{i}_d+\omega L_3{i}_q-e_d+u_d=u_{3d}+u_d$$

(7)

$$L_3\frac{d{i}_q}{dt}=-R_3{i}_q-\omega L_3{i}_d-e_d+u_q=u_{3q}+u_q$$

(8)

where u_id and u_iq are the control variables used for the command of the DC–AC. In the above equations, we noted that $u_{3d}=-R_3{i}_d+\omega L_3{i}_q-e_d$ and $u_{3q}=-R_3{i}_q-\omega L_3{i}_d-e_q$.

Following [15,26,31], the duty cycle D used for the control of the DC boost converter was described by means of Equations (9) and (10):

$$i_{dc1}=\left(1-D\right)i_s$$

(9)

$$u_s=\left(1-D\right)u_{dc}$$

(10)

A general block diagram of the entire application described in this article is shown in Figure 2. The chain of primary elements consisted of the following blocks: the simulated inputs for PV array; the PV array itself; the MPPT algorithm; the DC-DC boost converter; the voltage and current measurements for the DC intermediate circuit; the load and connection to power grid; the DC-AC converter; and the power grid. It can also be noted that the main element that we focus on in this article is the control block of the three-phase DC-AC converter. The main objective of the control system is to stabilize the u_dc voltage as precisely as possible, including under variation caused by the three-phase load. While in the classic case, the control system is built with PI-type controllers for the two voltage and current control loops, this article presents an improvement in the performance of the control system through the use of an RL-TD3 agent. Furthermore, in the complex case of a control system in which the control of the voltage loop is performed by an SMC-type controller and the control of the current loop is performed by an SYN-type controller, there was an improvement in the performance of the GC-PV control system through the use of the RL-TD3 agent that was created and trained accordingly.

3. Reinforcement Learning for Process Control

The RL for process control is organized as a series of tasks, which run on a computer to control an industrial process but do not require an explicit mathematical description. Thus, the RL process interacts with the controlled process in the sense of transmitting decisions (commands), which must reach the maximum of a set cumulative “Reward”. Figure 3 presents the schematic block diagram for an RL of the process control system. It can be noted that “Observation” and “Reward” are input signals for the RL. Observations are signals that characterize the process and are measurable along with their rate of change or error relative to a reference. Actions are the control quantities that act on the controlled process. Over time, Actions are selected so that the cumulative Reward increases in order to reach an optimal value. The Reward is expressed in terms of the square error of process signals and the square of the past Actions. The RL contains an optimal “Policy”, which is analogous to the operating mode of a process controller. The process contains the usual elements, namely a plant, reference signals, converters, filters and sensors.

The usual stages for the design of an RL process are the following [28,29,30]:

• The Problem statement represents the RL agent and its capability to interconnect with the components of the process;
• The Process creation represents the dynamic model type of the GC-PV’s controlled process and its interface;
• The Reward creation represents the mathematical relationship of the Reward in order to carry out the performance measurements for the execution of the proposed task;
• The Agent training represents an RL agent that is trained to realize the Policy based on the Reward, RL algorithm and controlled process.
• The Agent validation represents the stage where the performance is evaluated after training;
• The Deploy policy represents the step that performs the implementation of the trained RL agent within the GC-PV control system.

In this article, we used an RL-TD3 agent, which was an improved variant of the RL-DDPG-type agent. This type of agent is an actor-critic agent that calculates the long-term maximization of the Reward.

The steps performed by an RL agent during the training period are as follows [28,29]:

• For the Observation of the current state S, the action $A=\mu (S)+N$ is selected, where N is the stochastic noise obtained from the noise model;
• Action A is executed, then Reward R and the next Observation S’ are calculated;
• The experience $\left(S,A,R,S^{\prime }\right)$ is stored;
• M experiences $\left(S_i,A_i,R_i,S_{\prime }^i\right)$ are randomly generated;
• For $S_i^{\prime }$, which is a terminal state, we can obtain the value function target y_i that is set to R_i.

Alternatively, this is calculated by the Equation (11) [28,29]:

$$y_i=R_i+\gamma \cdot \min \left(Q_k^{\prime }\left(S_k^{\prime },clip\left({\mu }^{\prime }\left(S_k^{\prime }\left|{\theta }_{\mu }\right.\right)+\epsilon \right)\left|{\theta }_{Q_k^{\prime }}\right.\right)\right)$$

(11)

The value function target is equal to the sum of the experience Reward R_i and the minimum discounted value for the future Reward from the critics.

At every training step, the parameters of each critic are updated and minimized using the following expression:

$$L_k=\frac{1}{M}{\displaystyle \sum _{i=1}^M{{\displaystyle (}{y}_i-Q_k{\displaystyle (}{S}_i,A_i\left|{\theta }_{Qk}{\displaystyle )}{\displaystyle )}\right.}^2}$$

(12)

At every step, the actor’s parameter values are updated, thereby maximizing the Reward:

$${\nabla }_{{\theta }_{\mu }}J=\frac{1}{M}{\displaystyle \sum _{i=1}^M{G}_{ai}{G}_{\mu i}}$$

(13)

where G_ai, G_μi and A are represented by the following expressions, respectively:

$$G_{ai}={\nabla }_A\min \left(Q_k\left(S_i,A\left|{\theta }_Q\right.\right)\right)$$

(14)

$$G_{\mu i}={\nabla }_{{\theta }_{\mu }}\mu \left(S_i\left|{\theta }_{\mu }\right.\right)$$

(15)

$$A=\mu \left(S_i\left|{\theta }_{\mu }\right.\right)$$

(16)

Additionally, the parametric updates are realized for a selected smoothing coefficient τ, as in the following equations:

$${\theta }_{Q{k}^{\prime }}=\tau {\theta }_{Qk}+\left(1-\tau \right){\theta }_{Q{k}^{\prime }}$$

(17)

$${\theta }_{{\mu }^{\prime }}=\tau {\theta }_{\mu }+\left(1-\tau \right){\theta }_{{\mu }^{\prime }}$$

(18)

4. Correction of the Control Signals Used for the Control of a Grid Connected PV Array System Based on PI Controllers Using RL-TD3 Agent

The classic control system for a GC-PV system is presented in detail in [15] and can be considered as the benchmark for the performance of the control system. The control system is also presented in [27,31], both under low voltage and normal operation conditions. Figure 4 shows the schematic diagram of the classic control system for the GC-PV array, which consists of a cascade structure in which PI-type controllers and control loops are used for the control of currents i_d and i_q (inner control loop) and for the control of the u_dc voltage (outer control loop).

Figure 5 shows the model MATLAB Simulink implementation of the control system for the GC-PV system based on PI-type controllers using an RL-TD3 agent for the correction of control signals, i.e., a customization of Figure 2 for the control system presented in this section. Thus, the RL-TD3 agent that learned the behavior of the control system for the GC-PV array was used, which supplied the correction command signals for the three command inputs of the cascade-type control system (i_dref, u_dref and u_qref) after the training phase, so that the improved GC-PV control system would produce a superior performance.

The steps in Section 3 were followed to implement the RL-TD3 agent. In first step, the deep neural network (DNN) object was created, which was characterized through two inputs (Observation and Action) and one output. An example code sequence from the software program developed in the MATLAB environment for the design of the neural network is shown in Figure 6 and its graphic representation is presented in Figure 7.

To train the RL-TD3 agent to control the GC-PV system, 200 episodes were chosen, with the step number for each episode being around 100 and the time sampling of the agent being 10⁻⁴ s. The RL-TD3 agent training stage could be stopped when the cumulative average Reward was greater than −190 for a period of 100 consecutive episodes or after the 200 training episodes that were initially set had finished. To improve the RL-TD3 agent’s performance during training, Gaussian noise overlapped the signals that were received and transmitted by the proposed agent.

4.1. Implementation of the RL-TD3 Agent for the Correction of Commands for the Outer Voltage Control Loop

The model MATLAB Simulink implementation of the control system for the GC-PV array based on PI-type controllers using the RL-TD3 agent for the command correction of the i_dref current, which represents the outer loop for the control of the u_dc voltage, is shown in Figure 8. Figure 9 presents subsystem diagram of the MATLAB Simulink implementation of the RL-TD3 agent. In this case, the corrected command signals of the RL-TD3 agent were added to the command signal i_dref. The Observations were represented by the following signals: u_dc and u_dcerror.

The Reward at every step in this case was calculated using the following equation:

$$r_1=-\left(Q_1{u}_{dc\_error}^2+R{\displaystyle \sum _j{\left(u_{t-1}^j\right)}^2}\right)$$

(19)

where Q₁ is 0.5 and R is 0.1.

The training time in this case was 2 h, 37 min and 12 s. The graphical results for this training stage are presented in Figure 10.

4.2. Implementation of the RL-TD3 Agent for the Command Correction of the Inner Currents Control Loop

The model MATLAB Simulink implementation of the inner loop of the GC-PV array (which controls the i_d and i_q currents) based on the RL-TD3 agent is shown in Figure 11. After the learning stage, the RL-TD3 agent supplied correction signals for the command signals u_d and u_q. Figure 12 presents the block diagram of the MATLAB Simulink subsystem implementation of the RL-TD3 agent. The Observation consisted of the following signals: i_d, i_q, i_derror and i_qerror.

The Reward at every step in this case was calculated using the following equation:

$$r_1=-\left(Q_1{i}_{derror}^2+Q_2{i}_{qerror}^2+R{\displaystyle \sum _j{\left(u_{t-1}^j\right)}^2}\right)$$

(20)

where Q₁ = Q₂ = 0.5, R is 0.1 and $u_{t-1}^j$ is the Actions from the previous step.

The training time in this case was 3 h, 12 min and 42 s. The graphical results for this training stage are presented in Figure 13.

4.3. Implementation of the RL-TD3 Agent for the Command Correction of the Outer Voltage Control Loop and Inner Current Control Loops

The model MATLAB Simulink implementation of the command correction for the outer voltage control loop and the inner current control loops based on the RL-TD3 agent is shown in Figure 14. Figure 15 shows the block diagram of the MATLAB Simulink subsystem implementation of the RL-TD3 agent. In this case, the correction signals of RL-TD3 agent were supplied to the command signals u_d and u_q and also to the i_dref signal. The Observations were represented by the following signals: u_dc, u_{dc_error}, i_d, i_q, i_{d_error} and i_{q_error}.

The Reward at every step in this case was calculated using the following equation:

$$r_1=-\left(Q_1{u}_{dc\_error}^2+Q_2{i}_{derror}^2+Q_3{i}_{qerror}^2+R{\displaystyle \sum _j{\left(u_{t-1}^j\right)}^2}\right)$$

(21)

where Q₁ = Q₂ = Q₃ = 0.5 and R is 0.1.

The training time in this case was 1 h, 27 min and 33 s. The graphical results for this training stage are presented in Figure 16.

5. Correction of the Control Signals for the Control System of the Grid Connected PV Array Based on SMC and Synergetic Controllers Using the RL-TD3 Agent

In this section, we present the design and synthesis algorithms of the SMC and SYN controllers of the control system for the GC-PV array. Figure 17 presents the schematic diagram of the control system for the GC-PV array based on the SMC and SYN controllers. This control system consisted of a cascade in which the control loops were used for the control of the i_d and i_q signal currents (inner control loop with the SYN controller) and the control of the u_dc signal voltage (outer control loop with the SMC controller).

Moreover, the RL-TD3 agent that learned the behavior of the GC-PV control system was used, which supplied the correction signals for the three control inputs of the cascade-type control system (i_dref, u_dref, u_qref) after the training stage, so that the improved control system would produce a superior performance, even when the control system used SMC- and SYN-type controllers.

5.1. Sliding Mode Control

Based on the elements presented in Section 2 and denoting the switching functions of the DC-AC converter as S_a, S_b and S_c, the following equation could be written within the abc frame reference:

$$C_2\frac{d{u}_{dc}}{dt}=i_{dc1}-\left(i_a{S}_a+i_b{S}_b+i_c{S}_c\right)$$

(22)

By using the transformation in (5), the switching functions Sd and S_q could be obtained:

$${\left[\begin{array}{ccc}{S}_d& S_q& 0\end{array}\right]}^T=P{\left[\begin{array}{ccc}{S}_a& S_b& S_c\end{array}\right]}^T$$

(23)

With these, Equation (22) became:

$$C_2\frac{d{u}_{dc}}{dt}=i_{dc1}-\frac{3}{2}\left(i_d{S}_d+i_q{S}_q\right)$$

(24)

Similar to [15,31,33], the same MPPT algorithm was considered, so we then focused on obtaining the SMC and SYN command laws. Furthermore, by following [26], i_qref = 0 was selected and Equation (24) became:

$$C_2\frac{d{u}_{dc}}{dt}=i_{dc1}-\frac{3}{2}{i}_{dref}{S}_d$$

(25)

Moreover, to obtain the reference current i_dref using the SMC design procedure, the state variable x₁, as in Equation (26), and the switching surface S, as in Equation (27), were added:

$$x_1=u_{dc}-u_{dcref}$$

(26)

$$\left\{\begin{array}{c}S=c_1{x}_1+x_2\\ \dot{S}=c_1{x}_2+{\dot{x}}_2\end{array}\right.$$

(27)

In Equation (27), the state variable x₂ was defined as follows:

$$x_2={\dot{x}}_1=-{\dot{u}}_{dc}$$

(28)

Equation (29) was then necessary to achieve convergence:

$$\dot{S}=-\epsilon \mathrm{sgn}S-kS$$

(29)

where ε and k are positive constants.

Using calculus, the following could be obtained:

$${\ddot{x}}_1={\dot{x}}_2=-{\ddot{u}}_{dc}=\frac{3}{2}\frac{S_d}{C_2}{\dot{i}}_{dref}-\frac{{\dot{i}}_{dc1}}{C_2},$$

(30)

and so, the next equation could be written:

$$-\epsilon \mathrm{sgn}S-kS=c_1{x}_2+\frac{3}{2}\frac{1}{C_2}{S}_d{\dot{i}}_{dref}-\frac{{\dot{i}}_{dc1}}{C_2}$$

(31)

Following [32,33], to improve the convergence and smoothing of the high frequency oscillations, the sgn function was replaced with the following function defined by Equation (32):

$$h(x)=\frac{2}{1+e^{-a(x-b)}}-1$$

(32)

For a = 4 and b = 0, $h\in [-1\hspace{0.33em}1]$ and a smoothed transition were achieved for this interval. Thus, the output of the designed SMC-type controller was obtained by:

$$i_{dref}=\frac{2}{3}\frac{C_2}{S_d}{\displaystyle \underset{0}{\overset{t}{\int }}\left[-\left(c_1{x}_2+kS-\epsilon h(S)\right)+\frac{{\dot{i}}_{dc1}}{C_2}\right]}dt$$

(33)

Figure 18 presents the block diagram of the MATLAB Simulink subsystem implementation of the proposed SMC controller.

5.2. Synergetic Control

For a nonlinear system in the form of (34), an SYN control law could be synthesized that could be seen as a generalization of the SMC-type control law [27,32,33]:

$$\dot{x}=f\left(x,u,t\right)$$

(34)

where x is the state vector $x\in {\Re }^n$, $f\left(.\right)$ is the continuous nonlinear function and u is the input control vector $u\in {\Re }^m,\hspace{0.33em}\left(m<n\right)$.

The macro variable $\psi \left(x,t\right)$ was chosen and it was defined for each input control according to the states of the system. The forced evolution of the states according to the following equation was imposed for the synthesis of the SYN-type control law:

$$T\dot{\psi }+\psi =0$$

(35)

where T > 0 is selected to achieve the desired convergence rate.

By differentiating the chosen macro variable, ψ was obtained by the following expression:

$$\dot{\psi }=\frac{\partial \psi }{\partial x}\dot{x},$$

(36)

After inserting Equation (36) into Equation (35), the following could be obtained:

$$T\frac{\partial \psi }{\partial x}\dot{x}+\psi =0$$

(37)

By inserting the explicit forms of the $\dot{x}$ states into Equation (37), we could obtain the control law given by the next equation:

$$u=u\left(x,\psi (x,t),T,t\right)$$

(38)

The outputs of the SYN controller were given by u_d and u_q.

For the d axis and k_d > 0, we selected the chosen macro variable ψ_d in the following form:

$${\psi }_d=\left(u_{dcref}-u_{dc}\right)+k_d\left(i_{dref}-i_d\right)$$

(39)

We defined the state variable x₂ as in Equation (40):

$$\left\{\begin{array}{c}{x}_1=u_{dref}-u_{dc}\\ x_1=i_{dref}-i_d\end{array}\right.$$

(40)

From Equation (40) and for the slow mode variations of the reference quantities or for a quasi-stationary regime, the next expression could be obtained:

$$\left\{\begin{array}{c}{\dot{x}}_1=-{\dot{u}}_{dc}\\ {\dot{x}}_2=-{\dot{i}}_d\end{array}\right.$$

(41)

Based on these, Equation (39) became:

$${\dot{\psi }}_d={\dot{x}}_1+k_d{\dot{x}}_2=-{\dot{u}}_{dc}-k_d{\dot{i}}_d$$

(42)

For T = T₁, Equation (40) became:

$$T_1\left(-{\dot{u}}_{dc}-k_d{\dot{i}}_d\right)+\left(u_{dcref}-u_{dc}\right)+k_d\left(i_{dref}-i_d\right)=0$$

(43)

Using Equation (7), Equation (43) could be written in the following form:

$$-T_1{\dot{u}}_{dc}-T_1{k}_d\frac{1}{L_3}\left(u_{3d}-u_d\right)+\left(u_{dcref}-u_{dc}\right)+k_d\left(i_{dref}-i_d\right)=0$$

(44)

After rearranging the terms in Equation (44), we could obtain the following expression:

$$T_1{k}_d\frac{1}{L_3}{u}_d=-T_1{\dot{u}}_{dc}-T_1{k}_d\frac{1}{L_3}{u}_{3d}+\left(u_{dcref}-u_{dc}\right)+k_d\left(i_{dref}-i_d\right)$$

(45)

Thus, the control law u_d was obtained:

$$u_d=\frac{L_3}{T_1{k}_d}\left[-T_1{\dot{u}}_{dc}-T_1{k}_d\frac{1}{L_3}{u}_{3d}+\left(u_{dcref}-u_{dc}\right)+k_d\left(i_{dref}-i_d\right)\right]$$

(46)

For the q axis and k_q > 0, we selected the macro variable ψ_q in the following form:

$${\psi }_q=i_{qref}-i_q$$

(47)

We could define the state variable x₃ as:

$$\left\{\begin{array}{c}{x}_1=u_{dcref}-u_{dc}\\ x_2=i_{dref}-i_d\\ x_3=i_{qref}-i_q\end{array}\right.$$

(48)

For i_qref = 0, Equation(48) could be written in the following form:

$$\left\{\begin{array}{c}{\dot{x}}_1=-{\dot{u}}_{dc}\\ {\dot{x}}_2=-{\dot{i}}_d\\ {\dot{x}}_3=-{\dot{i}}_q\end{array}\right.$$

(49)

Thus, the macro variable derivative ψ_q, which was defined in Equation (47), was obtained:

$${\dot{\psi }}_q={\dot{x}}_3$$

(50)

For T = T₂, Equation (40) became:

$$-T_2{\dot{i}}_q+\left(i_{qref}-i_q\right)=0$$

(51)

Using Equation (8), Equation (51) could be written in the following form:

$$-T_2\frac{1}{L_3}\left(u_{3q}+u_q\right)+i_{qref}-i_q=0$$

(52)

After rearranging of the terms in Equation (52), we could obtain the following expression:

$$\left(u_{3q}+u_q\right)=\frac{L_3}{T_2}\left(i_{qref}-i_q\right)$$

(53)

Thus, the control law u_q was obtained:

$$u_q=\frac{L_3}{T_2}\left(i_{qref}-i_q\right)-u_{3q}$$

(54)

Figure 19 presents the MATLAB Simulink implementation subsystem of the designed SYN controller.

Figure 20 shows the model MATLAB Simulink implementation of the control system for the GC-PV array based on the SMC (MATLAB Simulink subsystem implementation shown in Figure 18) and SYN (MATLAB Simulink subsystem implementation shown in Figure 19) controllers using the RL-TD3 agent for the correction of the control signals.

Following on from the aspects presented in Section 4, this section continues to present the ways in which the performance of the GC-PV system could be improved by using the RL-TD3 agent, even when using complex SMC- and SYN-type controllers, i.e., a customization of Figure 2 for the control system presented in this section.

5.3. Implementation of the RL-TD3 Agent for the Correction of the Outer Voltage Control Loop Using SMC and Synergetic Control

The block diagram of the MATLAB Simulink subsystem implementation of the GC-PV control system using SMC and SYN controllers and the improved performance of the RL-TD3 agent being used for the outer control loop is shown in Figure 21. The correction signals of the RL-TD3 agent were added to the command signal i_dref, the RL-TD3 block structure was similar to that in Figure 9 and the Reward was given by Equation (19).

The training time in this case was 3 h, 12 min and 17 s. The graphical results for this training stage are presented in Figure 22.

5.4. Implementation of the RL-TD3 Agent for the Correction of the Inner Currents Control Loop Using SMC and Synergetic Control

The MATLAB Simulink subsystem block implementation of the GC-PV control system using SMC and SYN and the improved performance from using the RL-TD3 agent in the inner control loop is presented in Figure 23. The correction signals of the RL-TD3 agent were added to the command signals u_dref and u_qref, the RL-TD3 block structure was similar to that in Figure 12 and the Reward was obtained using Equation (20).

The training time in this case was 3 h, 3 min and 36 s. The graphical results for this training stage are presented in Figure 24.

5.5. Implementation of the RL-TD3 Agent for the Correction of the Outer Speed Control Loop and Inner Current Control Loops Using SMC and Synergetic Control

The MATLAB Simulink subsystem block implementation of the GC-PV control system using SMC and SYN and the improved performance from using the RL-TD3 agent in the outer and inner control loops is presented in Figure 25. The correction signals of the RL-TD3 agent were added to the command signals i_dref, u_dref and u_qref, the RL-TD3 block structure was similar to that in Figure 15 and the Reward was obtained using Equation (21).

The training time in this case was 2 h, 15 min and 24 s. The graphical results for this training stage are presented in Figure 26.

6. Numerical Simulations

This section starts with the benchmark that was presented in MATLAB Simulink [15] and resumed to compare the best results obtained in [26,27,31,33]. After presenting the main characteristics of the benchmark system, this section presents the numerical simulations that were based on the theoretical elements presented in the previous sections. Thus, starting with the cascade control structure in which PI-type controllers were used for the inner control loop of the i_d and i_q current signals and the outer control loop of the u_dc voltage signal and using the elements regarding the RL-TD3 agent, a superior performance of the control system for the GC-PV array was obtained.

Moreover, in the second part of the numerical simulations and starting from the peak performances presented in [32,33] regarding the cascade control system in which an SYN-type controller was used for the inner control loop of the i_d and i_q current signals, an SMC-type controller was used for the outer control loop of the u_dc voltage signal and the elements regarding the RL-TD3 agent were used, a superior performance of the control system for the GC-PV array was obtained, both in terms of the direct comparison of these performances and the robustness provided by the control system under parametric variations, such as the variation caused by the three-phase load.

Regarding the characteristics of the benchmark system presented in Section 2, we note that it was a 100 kW model in which the value of the u_dc voltage in the DC intermediate circuit was set to a value of 500 V; therefore, one of the objectives of the control system is to maintain this voltage and the voltage value supplied by the DC–AC converter was 260 V. The load was connected to the main grid via a 25 kV–260 V transformer. The nominal value of the three-phase load was 10 kvar. The MPPT algorithm used was that presented and implemented in the benchmark system [15,31] and was kept unchanged so as to be able to compare the performances of different the control systems for the GC-PV array. The PV array consisted of 330 modules that could supply 100.7 kW (305.2 W/modules) and in which the short circuit current of each module was I_sc = 5.96 A and the open circuit voltage was V_oc = 64.2 V. The sampling period for the PWM generator was 1 ms and the sampling period for the voltage and the current were 100 ms. Similar to the benchmark, the control system was bypassed for the first 50 ms.

The simulation of the PV array operation was dependent on the evolution of the irradiance and temperature input signals, which is shown in Figure 27. The time evolution of the irradiance and temperature in Figure 27 and the signal type 1 PV array are noted to be those used in the numerical simulations of the GC-PV array control system when using PI-type controllers.

Figure 28 shows the time evolution of the u_dc voltage for the irradiance and temperature of the control GC-PV array system using PI-type controllers with the signals of the type 1 PV array. The steady-state error of the control system based on PI controllers was 1 V, i.e., 0.2%, and the overshooting was neglectable.

Next, Figure 29, Figure 30, Figure 31 and Figure 32 show the evolution over time of the following quantities of interest of the control system: the i_d and i_q currents; the power P_mean and voltage U_mean of the PV; the duty cycle of the DC-DC converter; the modulation index of the DC-AC converter; the u_a voltage and the i_a current of the main grid; and the power flow P between the PV and the main grid. Thus, the evolution of the i_d and i_q currents, with the reference current i_qref = 0 and the i_d current following the i_dref reference current, are presented in Figure 29. The evolution of the power P_mean and voltage U_mean of the PV, the duty cycle of the DC-DC converter and the modulation index of the DC-AC converter are presented in Figure 30. The time evolution of the u_a voltage and the i_a current of the main grid are presented in Figure 31. The evolution of the power flow P between the PV and the main grid is presented in Figure 32.

Regarding the performance of the control system for the GC-PV array using PI-type controllers, a step variation was applied from 500 V to 550 V at 1 s. The result of the numerical simulation is presented in Figure 33.

As a result of the design of the RL-TD3 agents, their training and the numerical simulations related to the cases in Section 4.1, Section 4.2, Section 4.3 and Section 3 are presented Figure 34, Figure 35 and Figure 36.

Figure 37 presents the comparative responses of the control systems for the GC-PV array for a step signal between 500 V to 550 V that was based on PI-type controllers and three variants of this type of control system using the RL-TD3 agent for the correction of the command signals (outer and inner control loops).

Table 1. The performances of the GC-PV array control system based on PI-type controllers using the RL-TD3 agent.

Controllers for the GC-PV Array	Response Time (ms)	Voltage Ripple (V)	Overshooting (%)	Steady-State Error (%)
PI	40.4	57.87	<0.5	0.2
PI using the RL-TD3 agent for the correction of the idref command	37.1	57.23	<0.5	0.2
PI using the RL-TD3 agent for the correction of the udref and uqref commands	35.9	56.67	<0.5	0.2
PI using the RL-TD3 agent for the correction of the udref, uqref and idref commands	33.8	56.22	<0.5	0.2

presents the comparative performances of these systems in controlling the GC-PV array variants, i.e., the response times, and the ripple of the voltage error signal obtained using the Equation (55). In all of these cases in which the RL-TD3 agent was used, the overshooting was almost zero and the steady-state error was less than 0.2%. In the presented numerical simulations, it can be seen that the use of an RL-TD3 agent contributed to the improvement in the performance of the GC-PV array control system.

$$u_{dc\_rip}=\sqrt{\frac{1}{N}{\displaystyle \sum _{i=1}^N{\left(u_{dc}(i)-u_{dcref}(i)\right)}^2}}$$

(55)

where N is the sample number, u_dc is the voltage and u_dcref is the reference voltage.

The time evolution of the irradiance and temperature in Figure 38 and the signal type 2 PV array was noted as being used in the numerical simulations of the GC-PV array control systems using SMC- and SYN-type controllers.

Figure 39 presents the response of the SMC-type control, which was designed for the control of the u_dc voltage, combined with the SYN-type control, which was designed for the control of the i_d and i_q currents, for the DC voltage reference u_dcref = 500 V. In the detail in Figure 39, it can be observed that the steady-state error was 0.1 V, i.e., 0.02%.

To demonstrate the parametric robustness of the control system, Figure 40 and Figure 41 show the evolution of the u_dc voltage in the case of a variation of 30% from its nominal value that was caused by the load. Therefore, it can be noted that the control system for the GC-PV array using SMC- and SYN-type controllers maintained its performance in each of these cases.

Following the case in which the input for the GC-PV array was provided by the type 2 PV array signals, Figure 42, Figure 43, Figure 44 and Figure 45 present the evolution over time of the following quantities of interest of the control system: the i_d and i_q currents; the P_mean power and U_mean voltage of the PV; the duty cycle of the DC-DC converter; the modulation index of the DC-AC converter; the voltage u_a and current i_a of the main grid; and the power flow P between the PV and the main grid. Thus, the time evolution of the i_d and i_q currents, with the reference current i_qref = 0 and the i_d current following the i_dref reference current, are presented in Figure 42. The time evolution of the power P_mean and voltage U_mean of the PV, the duty cycle of the DC-DC converter and the modulation index of the DC-AC converter are presented in Figure 43. The evolution of the u_a voltage and i_a current of the main grid are presented in Figure 44. The time evolution of the power flow P between the PV and the main grid is presented in Figure 45.

Regarding the performance of the control system for the GC-PV array using SMC- and SYN-type controllers, a step variation was applied from 500 V to 550 V at 1 s. The result of the numerical simulation is presented in Figure 46.

As a result of the proposed RL-TD3 agents, their training and the numerical simulations related to the cases in Section 5.3, Section 5.4 and Section 5.5 are presented in Figure 47, Figure 48 and Figure 49.

Figure 50 presents the comparative response of the control for the GC-PV array system for a step variation from 500 V to 550 V based on SMC and SYN controllers and three variants of this type of control system using RL-TD3 agent for correction of the command signals (outer and inner control loop).

Table 2. The performances of the GC-PV control systems based on SMC and SYN controllers using the RL-TD3 agent.

Controllers for the GC-PV Array	Response Time (ms)	Voltage Ripple (V)	Overshooting (%)	Steady-State Error (%)
SMC and SYN	14.1	55.63	<0.2	0.02
SMC and SYN using the RL-TD3 agent for the correction of the idref command	13.7	55.12	<0.2	0.02
SMC and SYN using the RL-TD3 agent for the correction of the udref and uqref commands	13.5	54.58	<0.2	0.02
SMC and SYN using the RL-TD3 agent for the correction of the udref, uqref and idref commands	13.2	54.03	<0.2	0.02

presents the comparative performances of these systems in controlling the GC-PV array variants, i.e., the response times, and the ripple of the voltage error signal obtained using Equation (55). In all of these cases in which the RL-TD3 agent was used, the overshooting was almost zero and the steady-state error was less than 0.02%. In the presented numerical simulations, it can be seen that the use of an RL-TD3 agent contributed to the improvement in the performance of the GC-PV array control system.

It can be observed that, between PI-type control system and the PI–RL-TD3 agent control system, the response time was improved by approximately 7 ms and between the SMC and SYN control system, the SMC–RL-TD3 agent and SYN–RL-TD3 agent, the response time was improved by approximately 1 ms, i.e., a decrease in the response time by approximately 18% in the first case and approximately 7% in the second case. However, between the simplest and most complex cases, the decrease in the response time was about 27 ms, which indicates a decrease of about 70%. In the same way, the rest of the performances could be evaluated as relative or absolute units.

7. Conclusions

This paper described the control system for a GC-PV array, starting from a benchmark system. The control structure was a cascade-type structure in which PI or SYN controllers were used for the inner control loops of i_d and i_q signal currents and PI or SMC controllers were used for the outer control loop of the u_dc signal voltage in the DC intermediate circuit. The paper presented the model of the PV array together with the main component blocks: the simulated inputs for the PV array; the PV array itself; the MPPT algorithm; the DC-DC boost converter; the voltage and current measurements for the DC intermediate circuit; the DC-AC converter; the load and connection to the power grid; and the power grid. It also presented the stages of building and training the RL-TD3 agent. Additionally, the comparative results are shown for cases in which the RL-TD3 agent was properly trained and provided correction signals that were added to the command signals u_d, u_q and i_dref. The parametric robustness of the proposed control system for the GC-PV array based on SMC and SYN controllers was proven in the case of a variation of 30% that was caused by the three-phase load. Moreover, the results of the numerical simulations are presented comparatively and the validation of the synthesis of the proposed control system for the GC-PV array was obtained. This was performed by comparing the system to the software benchmark of a control system for a GC-PV array that was implemented in MATLAB Simulink. The numerical simulations proved the superiority of the control system that used the RL-TD3 agent.