Adaptive Neural Network Q-Learning-Based Full Recurrent Adaptive NeuroFuzzy Nonlinear Control Paradigms for Bidirectional-Interlinking Converter in a Grid-Connected Hybrid AC-DC Microgrid

Abstract

The stability of a hybrid AC-DC microgrid depends mainly upon the bidirectional interlinking converter (BIC), which is responsible for power transfer, power balance, voltage solidity, frequency and transients sanity. The varying generation from renewable resources, fluctuating loads, and bidirectional power flow from the utility grid, charging station, super-capacitor, and batteries produce various stability issues on hybrid microgrids, like net active-reactive power flow on the AC-bus, frequency oscillations, total harmonic distortion (THD), and voltage variations. Therefore, the control of BIC between AC and DC buses in grid-connected hybrid microgrid power systems is of great importance for the quality/smooth operation of power flow, power sharing and stability of the whole power system. In literature, various control schemes are suggested, like conventional droop control, communication-based control, model predictive control, etc., each addressing different stability issues of hybrid AC-DC microgrids. However, model dependence, single-point-failure (SPF), communication vulnerability, complex computations, and complicated multilayer structures motivated the authors to develop online adaptive neural network (NN) Q-learning-based full recurrent adaptive neurofuzzy nonlinear control paradigms for BIC in a grid-connected hybrid AC-DC microgrid. The proposed strategies successfully ensure the following: (i) frequency stabilization, (ii) THD reduction, (iii) voltage normalization and (iv) negligible net active-reactive power flow on the AC-bus. Three novel adaptive NN Q-learning-based full recurrent adaptive neurofuzzy nonlinear control paradigms are proposed for PQ-control of BIC in a grid-connected hybrid AC-DC microgrid. The control schemes are based on NN Q-learning and full recurrent adaptive neurofuzzy identifiers. Hybrid adaptive full recurrent Legendre wavelet-based Neural Network Q-learning-based full recurrent adaptive NeuroFuzzy control, Hybrid adaptive full recurrent Mexican hat wavelet-based Neural Network Q-learning-based full recurrent adaptive NeuroFuzzy control, and Hybrid adaptive full recurrent Morlet wavelet-based Neural Network Q-learning-based full recurrent adaptive NeuroFuzzy control are modeled and tested for the control of BIC. The controllers differ from each other, based on variants used in the antecedent part (Gaussian membership function and B-Spline membership function), and consequent part (Legendre wavelet, Mexican hat wavelet, and Morlet wavelet) of the full recurrent adaptive neurofuzzy identifiers. The performance of the proposed control schemes was validated for various quality and stability parameters, using a simulation testbench in MATLAB/Simulink. The simulation results were bench-marked against an aPID controller, and each proposed control scheme, for a simulation time of a complete solar day.

1. Introduction

Renewable resource-based distributed generators (DGs), like wind turbines (WTs), photovoltaic (PV) generators, micro-turbines (MTs) etc., are in fashion nowadays to reduce greenhouse emissions and to improve the efficiency of the electric power system. A large power system is divided into many small microgrids by the integration of DGs, energy storage system (ESS), and loads, and, thus, making it more controllable, flexible, and effective [1,2,3,4].

The DC microgrid has no frequency issues, produces low power loss, due to the absence of reactive power, and has simple dynamic variations and easy power conversions. These advantages of the DC microgrid make it superior to the AC microgrid. However, hybrid AC-DC microgrids are gaining huge attention, due to their capability of combining the benefits of both AC and DC networks [1,2,5,6,7,8]. The AC and DC microgrids are connected through a bidirectional AC-DC interlinking converter, thus, making a hybrid AC-DC microgrid that is capable of utilizing the advantages of both sides and which can entertain multiple types of loads. However, new technological challenges arise for the smooth and reliable operations of hybrid AC-DC microgrids [1,2,9,10,11,12,13].

AC and DC subgrids are interconnected via bidirectional interlinking converters (BIC). The BIC is responsible for smooth bidirectional power flow, power balance, microgrid stability, voltage solidity, frequency and transients firmness. Failure in the right performance of BIC leads to instability of the microgrid and causes power loss. The unpredictable nature of renewable DGs, nonlinear uncertain loads, switching harmonics, and bidirectional power flow between the utility grid and storage devices produces a variety of stability issues, like net active-reactive power flow on the AC-bus, increased THD level, voltage fluctuation, and frequency instability. Therefore, the key challenge in hybrid AC-DC microgrid is the control problem [2,14,15,16,17,18,19].

Generally, the whole network and, particularly, the BIC requires intelligent adaptive control to ensure the coordination of DC and AC subsystems and appropriate power flow across the hybrid microgrid [1,2,20,21]. The problem is non-trivial and becomes more difficult, due to the difference in response time of AC and DC networks. The faster response of the DC network over the AC network, and interactions between corresponding sides, may lead to destabilization of the power system [22,23]. Thus, the key issue in the hybrid AC-DC microgrid is the control of BIC.

Several control schemes are reported in the literature for the control of BIC in a hybrid AC-DC microgrid. Reference [24] presented coordination control and dual closed-loop control for the interlinking converter. However, the study lacked real time simulation results and presented two cases with specific variation in the generation and loads. Reference [2] designed a uniform control scheme for the bidirectional interlinking converter for economical and resilient objectives of hybrid AC-DC microgrids. However, the study lacked a real time simulation testbed and was studied for specific power flow cases. Reference [25] presented the voltage regulation of the DC-bus by designing a model-based controller for the bi-directional interlinking converter. However, the study required proper modeling of the system and the PID control used might not work for large complex power systems. Reference [26] presented the application of a passivity framework to AC-DC grids. However, a lossless line was assumed in the study and no real time simulation results were presented. Reference [27] presented virtual inertia support for BIC in the hybrid AC-DC microgrid. However, use of PI control and only the regulation of active power reduced its impact for large power systems with various operating points. Many other articles are found in the literature but the lack of real time systems and a small number of DGs reduce their productivity.

This article presents three novel control schemes for a bidirectional interlinking converter for a hybrid AC-DC microgrid. The control schemes are hybrid adaptive NN Q-learning-based full recurrent adaptive NeuroFuzzy (FRNF) control schemes. The difference in the three proposed control schemes is based on the different variants in the antecedent and consequent parts of adaptive NeuroFuzzy architecture. The variants in the antecedent part are Gaussian membership function and B-Spline membership function, while the variants used in the consequent part are Mexican hat wavelet, Legendre wavelet, and Morlet wavelet. The real time simulation testbed consisted of multiple DGs and an utility grid connected to DC- and AC-buses via converters and transformers. The real time solar and wind profiles, along with maximum power point tracking, used in this article were obtained from our previously published articles [28,29,30,31,32]. The use of multiple DGs, real time solar and wind profiles, and one day complete simulation are the key features of this study. The performance of the proposed control scheme was verified through various quality and stability parameters, using MATLAB/SIMULINK software, 2015a. The various results obtained from the simulation were compared with an aPID controller and against each other. Table 1 gives the features and contribution of the proposed solution, compared to previously reported work.

The article is divided into sections for understanding. Section 2 describes the testbed, Section 4, Section 5, Section 6, Section 7, Section 8, Section 9, Section 10, Section 11, Section 12 and Section 13 discusses the detailed modeling of the proposed control schemes, Section 14 discusses the various results of the simulation, Section 15 concludes the article, and Section 16 shows suggested future work.

Remark

This work is the extension of the authors’ previous work reported in [28,29,30,31,32], which focused on maximum power point tracking of the variable speed wind energy conversion system, PV-farm, swift response of solid oxide fuel cell, and charging/discharging scheduling of the plug in hybrid vehicles (PHEVs) in a grid-connected hybrid AC-DC microgrid. The voltage, frequency and THD regulations were considered in the above studies. However, the control of the interlinking converter in this study regulates the concerns (voltage, frequency, THD) at a higher efficiency level. Hence, the new control typologies were developed to increase the stability and reliability of the hybrid microgrid.

Table 1. Main contributions, features and limitations of the existing and proposed work.

Ref. No.	Year	Frequency Regulation	Voltage Regulation	THD Reduction	Limitations/Features
[33]	2016	✗	✓	✓	☞	Independent of communication and model parameters. However, general hybrid AC-DC model is assumed with negligence of frequency regulation.
[34]	2017	✗	✓	✗	☞	Applicable only on low voltage hybrid grid. Frequency regulation and THD reduction are neglected.
[35]	2018	✗	✓	✗	☞	Leader follower consensus and particle swarm optimization are prone to SPF ${}^1$. Only voltage regulation is considered.
[36]	2019	✗	✗	✗	☞	Synchronization control increases the risk of SPF. Only voltage regulation is considered.
[37]	2020	✓	✓	✗	☞	Hierarchical control strategy has drawback of SPF. THD reduction is neglected.
[18]	2021	✗	✓	✓	☞	Model predictive control algorithm requires model and parameters. It is complex and consumes more time. Moreover, frequency regulation is neglected.
[38]	2021	✗	✗	✓	☞	Plug-in-hybrid vehicles and other renewable resources are not taken into account. THD reduction is neglected.
[39]	2021	✗	✓	✗	☞	PI controller cannot operate under highly nonlinear conditions and prone to SPF. Only voltage regulation is considered.
[40]	2022	✓	✓	✗	☞	Coordinated control scheme requires communication network. THD regulation is neglected.
[41]	2022	✗	✓	✓	☞	Multiple control schemes are integrated. SPF of any control scheme results in power instability that may lead to blackout. Only wind power, PV and BSS are considered. Frequency regulation is neglected.
Proposed	2022	✓	✓	✓	☞	Model free control strategy is used. No communication required. Advantages of Q-learning and full recurrent neurfuzzy are combined to avoid SPF. Complete microgrid with multiple DGs is considered. Real time solar irradiance, temperature and wind speed is used. Optimal active-reactive power flow, voltage-frequency regulation, and THD reduction are considered.

¹ SPF = Single-Point-Failure.

Table 1. Main contributions, features and limitations of the existing and proposed work.

Ref. No.	Year	Frequency Regulation	Voltage Regulation	THD Reduction	Limitations/Features
[33]	2016	✗	✓	✓	☞	Independent of communication and model parameters. However, general hybrid AC-DC model is assumed with negligence of frequency regulation.
[34]	2017	✗	✓	✗	☞	Applicable only on low voltage hybrid grid. Frequency regulation and THD reduction are neglected.
[35]	2018	✗	✓	✗	☞	Leader follower consensus and particle swarm optimization are prone to SPF ${}^1$. Only voltage regulation is considered.
[36]	2019	✗	✗	✗	☞	Synchronization control increases the risk of SPF. Only voltage regulation is considered.
[37]	2020	✓	✓	✗	☞	Hierarchical control strategy has drawback of SPF. THD reduction is neglected.
[18]	2021	✗	✓	✓	☞	Model predictive control algorithm requires model and parameters. It is complex and consumes more time. Moreover, frequency regulation is neglected.
[38]	2021	✗	✗	✓	☞	Plug-in-hybrid vehicles and other renewable resources are not taken into account. THD reduction is neglected.
[39]	2021	✗	✓	✗	☞	PI controller cannot operate under highly nonlinear conditions and prone to SPF. Only voltage regulation is considered.
[40]	2022	✓	✓	✗	☞	Coordinated control scheme requires communication network. THD regulation is neglected.
[41]	2022	✗	✓	✓	☞	Multiple control schemes are integrated. SPF of any control scheme results in power instability that may lead to blackout. Only wind power, PV and BSS are considered. Frequency regulation is neglected.
Proposed	2022	✓	✓	✓	☞	Model free control strategy is used. No communication required. Advantages of Q-learning and full recurrent neurfuzzy are combined to avoid SPF. Complete microgrid with multiple DGs is considered. Real time solar irradiance, temperature and wind speed is used. Optimal active-reactive power flow, voltage-frequency regulation, and THD reduction are considered.

¹ SPF = Single-Point-Failure.

2. System Overview and Model Description

Figure 1 sketches the proposed microgrid. It consists of AC and DC-buses which are connected to each other via a main interlinking inverter. The DC-bus has multiple renewable energy resources connected to it through various AC-DC converters. The 165 F super-capacitor bank, 260 kW Photovoltaic (PV) array, 260 kW solid oxide fuel cell (SOFC), 150 kW electrolyzer, 200 Ah batteries, and 100 kW wind turbine (WT) are connected to the DC-bus through AC-DC and DC-DC converters. The AC-bus is connected with 200 kVA micro-turbine, 11 kV grid, bidirectional smart charging station (CS), and the AC-load through transformers, as well as AC-DC and AC-DC-AC converters. The DC-bus is interlinked with the AC-bus through the main inverter. The technical details of the proposed microgrid and its components are given in Appendix A [28,29,30,31,32].

Modeling and Description of Interlinking Inverter

AC and DC sub-grids are linked with each other via an interlinking power converter in the smart microgrid hybrid power system (SMG-HPS). The role of the interlinking converter is important in terms of bidirectional power transfer and stability of the whole SMG-HPS. The uncertain load and generation conditions enforces the interlinking converter to operate in various modes. If no power transfer takes place to either side, the operational mode is known as stop mode. If power transfers from the DC subgrid side to the AC subgrid side, the operational mode is known as inverter mode. If power transfers from the AC subgrid side to the DC subgrid side, the operational mode is known as rectifier mode.

Smooth power transfer between AC and DC subgrids is guaranteed by the interlinking converter. During grid-connected mode, the power balance is ensured by the utility grid, while the DC-bus voltage stability is ensured by an interlinking converter. The increasing use of electronic devices in the power sector, at the household and industrial levels, dramatically increases the complexity of the power system and reduces efficiency. The increase in THD level, and fluctuations in the frequency and voltage profiles, reduce the life of many electric components, as well as decrease the reliability and quality of power.

The specifications of interlining inverter used in this research work are given in

Table A9. Parameters of Interlinking Inverter.

Type	Zhejiang, China CHZIRI-2VF
Rated power	400 kW
Rated voltage	200/540 V
Inductance L-filter	2.1 $\mathsf{\mu }$H

, Appendix A.9.

The interlinking or main inverter acts as a bridge between AC and DC sub-grids. It controls the active and reactive power and, thus, ensures the voltage stability. The terminal voltages of this voltage source converter are as follows:

$$\begin{array}{c}\hfill v_{an}=L_{11}\frac{d{i}_a}{dt}+R_{11}{i}_a+v_{an-AC}\end{array}$$

(1)

$$\begin{array}{c}\hfill v_{bn}=L_{11}\frac{d{i}_b}{dt}+R_{11}{i}_b+v_{bn-AC}\end{array}$$

(2)

$$\begin{array}{c}\hfill v_{cn}=L_{11}\frac{d{i}_c}{dt}+R_{11}{i}_c+v_{cn-AC}\end{array}$$

(3)

where, inductance and resistance are shown by $L_{11}$ and $R_{11}$, respectively. The dq-reference frame is calculated by using Park’s transformation as follows:

$$\begin{array}{c}\hfill \frac{d{i}_d}{dt}=\omega i_q-\frac{R_{11}{i}_d}{L_{11}}+\frac{1}{L_{11}}(v_d-v_d^{*})\end{array}$$

(4)

$$\begin{array}{c}\hfill \frac{d{i}_q}{dt}=\omega i_d-\frac{R_{11}{i}_q}{L_{11}}+\frac{1}{L_{11}}(v_q-v_q^{*})\end{array}$$

(5)

Voltages in the dq-reference frame are calculated using the following equations:

$$\begin{array}{c}\hfill v_d=L_{11}\frac{d{i}_d}{dt}-\omega L_{11}{i}_q+v_d^{*}+R_{11}{i}_d\end{array}$$

(6)

$$\begin{array}{c}\hfill v_q=L_{11}\frac{d{i}_q}{dt}-\omega L_{11}{i}_d+v_q^{*}+R_{11}{i}_q\end{array}$$

(7)

In the dq-reference frame the power is calculated using the following relation:

$$\begin{array}{c}\hfill P_{dq}=\frac{3}{2}(v_d{i}_d+v_q{i}_q)\end{array}$$

(8)

3. Supervisory Control of Microgrid and Operation Strategy

The principal purpose of supervisory control is to provide continuous and reliable power to all the connected loads. The loads used in this research work are residential load ($P_L$) and CS load ($P_{CS}$). The significant power is transmitted to these main loads. However, the excess power during off-peak hours is transmitted to BSS and SC for storing purposes and later use. The electrolyzer also acts as a load and obtains power from the DC-bus of SMG-HPS. The supervisory control enforces the use of renewable energy to entertain loads and store electrical power in various storing devices. However, in the case of a deficit power, the extra power is obtained from BSS, SC, and CS (PHEVs).

The modes of operation of supervisory control depend on the balance of available and required power.

3.1. Modes of Operation of Supervisory Control System

The modes are based on the deficit and excess power. The description of the modes used in this research work is given under.

3.1.1. Mode of Power Deficit

The operational modes of supervisory control systems with deficit power are described below.

Mode 1: WT, PV and BSS Fulfill the Load Demand

In this mode, only RES WT, PV, and BSS provide power to fulfil the load demand. The WT and PV provide their maximum output powers while BSS power is restricted to up to 20% of the state-of-charge (SOC). The mathematical representation of this mode is:

$$\begin{array}{c}\hfill P_{Load}\ge P_{ren}+P_{BSS}\end{array}$$

(9)

where, $P_{Load}=P_{CS}+P_L=V_L\times I_L$ is the load demand, $P_{ren}=P_{WT}+P_{PV}$ is the cumulative renewable power, and $P_{BSS}$ is the BSS power for $SO{C}_{BSS}\ge 20\%$. The BSS discharges in this mode of operation.

Mode 2: WT, PV, BSS and SC Fulfill the Load Demand

If the load demand is higher than the power produced in mode 1, then deficit power is obtained from SC to fulfil the load demand. The mathematical representation of this mode is:

$$\begin{array}{c}\hfill P_{Load}\ge P_{ren}+P_{BSS}+P_{SC}\end{array}$$

(10)

where $P_{SC}$ is the power obtained by SC for $SO{C}_{SC}\ge 20\%$. All other sources are kept OFF and power is obtained only through RES, BSS, and SC during this mode of operation.

Mode 3: WT, PV, BSS, SC and SOFC Fulfill the Load Demand

If the power demand of load is not fulfilled from RES and the discharging of BSS and SC, then the deficit power is obtained from SOFC to satisfy the load demand. The SOFC tracks a power reference that depends on the deficit power. The mathematical representation of this mode is:

$$\begin{array}{c}\hfill P_{Load}\ge P_{ren}+P_{BSS}+P_{SC}+P_{SOFC}\end{array}$$

(11)

where $P_{SOFC}$ is the power obtained from SOFC.

Mode 4: WT, PV, BSS, SC, SOFC and Grid Fulfill the Load Demand

If the power required by the load increases from mode 3, then the extra power deficit is obtained from the utility grid to satisfy the load requirements. The mathematical expression for this mode is:

$$\begin{array}{c}\hfill P_{Load}\ge P_{ren}+P_{BSS}+P_{SC}+P_{SOFC}+P_{Grid}\end{array}$$

(12)

where $P_{Grid}$ is the power obtained from the utility grid.

Mode 5: WT, PV, BSS, SC, SOFC, Grid and MT Fulfill the Load Demand

If the power delivered from mode 4 cannot satisfy the load demands during peak hours, then the deficit power is obtained from the MT. The mathematical relation for this mode is:

$$\begin{array}{c}\hfill P_{Load}\ge P_{ren}+P_{BSS}+P_{SC}+P_{SOFC}+P_{Grid}+P_{MT}\end{array}$$

(13)

where $P_{MT}$ is the power obtained from MT to satisfy the load demand.

3.1.2. Modes of Excess Power

The modes of supervisory control systems with excess power are described below.

Mode 6: Excess Power Given to Electrolyzer

If the power obtained from RES and BSS is more than the power required by the load, then the extra energy is absorbed by the electrolyzer. This mode can be mathematically represented as:

$$\begin{array}{ccc}\hfill P_{Load}& \le & P_{ren}+P_{BSS}\hfill \end{array}$$

(14)

$$\begin{array}{ccc}\hfill P_{excess}& =& P_{Load}-P_{ren}-P_{BSS}\hfill \end{array}$$

(15)

$$\begin{array}{ccc}\hfill P_{elect}& =& P_{excess}\hfill \end{array}$$

(16)

where, $P_{excess}$ is the extra amount of power, $P_{elect}$ is the electrolyzer power. The BSS is in discharge mode.

Mode 7: Excess Power Given to SC and Electrolyzer

If the power generated from RES and BSS is greater than the load demand, while SOC of SC is less than 90%, then the excess energy is utilized to charge SC. In the case of more surplus power, the electrolyzer is entertained as well. The mathematical representation of this mode is:

$$\begin{array}{ccc}\hfill P_{Load}& \le & P_{ren}+P_{BSS}\hfill \end{array}$$

(17)

$$\begin{array}{ccc}\hfill P_{excess}& =& P_{Load}-P_{ren}-P_{BSS}\hfill \end{array}$$

(18)

$$\begin{array}{ccc}\hfill P_{SC}& =& P_{excess}\hfill \end{array}$$

(19)

$$\begin{array}{ccc}\hfill P_{elect}& =& P_{excess}-P_{SC}\hfill \end{array}$$

(20)

where $P_{SC}$ shows the power absorbed by SC in charging mode. In this mode, RES and BSS are providing power, while load, SC and electrolyzer are absorbing power.

Mode 8: Excess Power Given to SC, Grid, and Electrolyzer

In this mode of operation, the power generated from the RES is greater than the load demand. Therefore, the excess energy is given to SC for its charging. If SMG-HPS still has surplus power generated from RES then the excess energy is delivered to the utility grid during its peak hours and the remaining surplus power is given to the electrolyzer for the production of hydrogen gas. The mathematical expression for this mode is:

$$\begin{array}{ccc}\hfill P_{Load}& \le & P_{ren}+P_{BSS}\hfill \end{array}$$

(21)

$$\begin{array}{ccc}\hfill P_{excess}& =& P_{Load}-P_{ren}-P_{BSS}\hfill \end{array}$$

(22)

$$\begin{array}{ccc}\hfill P_{SC}& =& P_{excess}\hfill \end{array}$$

(23)

$$\begin{array}{ccc}\hfill P_{Grid}& =& P_{excess}-P_{SC}\hfill \end{array}$$

(24)

$$\begin{array}{ccc}\hfill P_{elect}& =& P_{excess}-P_{SC}-P_{Grid}\hfill \end{array}$$

(25)

where $P_{Grid}$ shows the power absorbed by the utility grid during peak hours.

Mode 9: Excess Power Given to SC, Grid, and Electrolyzer, while BSS Is Disconnected

This mode of operation is similar to mode 8. However, in this mode, the BSS is disconnected, i.e., $P_{BAT}=0$. The excess power generated from RES is delivered to SC for charging, for the utility grid during peak hours, and for the electrolyzer for production of hydrogen gas, respectively. The mathematical expression is:

$$\begin{array}{ccc}\hfill P_{Load}& \le & P_{ren}\hfill \end{array}$$

(26)

$$\begin{array}{ccc}\hfill P_{excess}& =& P_{Load}-P_{ren}\hfill \end{array}$$

(27)

$$\begin{array}{ccc}\hfill P_{SC}& =& P_{excess}\hfill \end{array}$$

(28)

$$\begin{array}{ccc}\hfill P_{Grid}& =& P_{excess}-P_{SC}\hfill \end{array}$$

(29)

$$\begin{array}{ccc}\hfill P_{elect}& =& P_{excess}-P_{SC}-P_{Grid}\hfill \end{array}$$

(30)

The supervisory control flow chart is given in Figure 2.

4. Description and Modeling of Proposed Control Schemes

This section discusses the mathematical modeling of, and describes, the three proposed hybrid adaptive NN Q-learning0based full recurrent adaptive NeuroFuzzy (NNQLNF) control schemes. The proposed control schemes consist of two parts: NeuroFuzzy parameter tuning (NFPT) and optimal action-value function $Q^{*}(x,u)$ estimator network (QEN). $Q^{*}(x,u)$ is estimated using a backpropagation (BP) neural network (NN). The NeuroFuzzy systems used are discussed in Section 6 and Section 7. The NeuroFuzzy parameters are updated using $Q^{*}(x,u)$. The action exploration modifier (AEM) guarantees the trail of all possible actions. The NNQLNF control technique does not depend on prior information for future driving conditions, due to online adaptation of the learning algorithm and fuzzy parameters, which make it a prominent and advantageous control paradigm.

5. Neural Network Q-Learning-Based Full Recurrent Adaptive NeuroFuzzy Control

In control problems, Q-learning and actor-critic learning are two major types of reinforcement learning. The actor-critic learning estimates the state value function and chooses an optimal action for every state. However, in Q-learning, the system approximates the action-value function for all action-state sets and chooses the optimal control technique based on this [42,43].

The schematic of NNQLNF is given in Figure 3. It consists of two parts: the QEN network, that estimates $Q^{*}(x,u)$, and the FRNF architecture, that tunes the parameters. The QEN comprises a BP NN for estimation purpose, while the FRNF is an intelligent identifier discussed in Section 6 and Section 7. Three different FRNF architectures are used as FPT, and, thus, three distinct hybrid adaptive FQL embedded FRNF control paradigms are derived, which are used in this research work.

5.1. Back Propagation NN for Estimating $Q^{*}(x,u)$

If the initial state is x and initial action is u, then the expected discounted sum of rewards gives the action-value function $Q(x,u)$, as given below [42]:

$$\begin{array}{c}\hfill Q(x,u)=E\left(\underset{k=1}{\sum ^{\infty }}{\gamma }^k{r}_{t+k+a}\right|x_t=x,u_t=u)\end{array}$$

(31)

where, E is the expected value function, and u is the action value. The optimal action-value $Q^{*}(x,u)$ is given as:

$$\begin{array}{c}\hfill Q^{*}(x,u)=E(r\left(x_{t+1}\right)+\gamma \underset{u^{\prime }}{\max }{Q}^{*}(x_{t+1},u^{\prime })|x_t=x,u_t=u)\end{array}$$

(32)

QEN approximates/predicts the ideal action-value function $Q^{*}(x,u)$ related to different inputs and control output states. The approximation property of BP NN is used for the estimation of $Q^{*}(x,u)$.

The internal architecture of QEN is given in Figure 4. The network consists of a three-layer structure: input node, hidden node, and output node. The inputs are state variable and control action. However, the output is the required QEN i.e., $Q(x,u)$.

The $Q(x,u)$ can be represented mathematically as:

$$\begin{array}{c}\hfill Q(x,u)=f\left(V\right)\end{array}$$

(33)

where, V gives the weighted sum input of the output node and is given as:

$$\begin{array}{c}\hfill V=\underset{k=1}{\sum ^{10}}{w}_{(40+k)}\times y_k\end{array}$$

(34)

where, $w_{(40+k)}$ represents the weight between hidden and output node and $y_k$ is the output of hidden layer and is given as:

$$\begin{array}{c}\hfill y_k=f\left(a_k\right)\end{array}$$

(35)

where, $a_k$ gives the added input of kth hidden node and is given as:

$$\begin{array}{c}\hfill a_k=\underset{i=1}{\sum ^4}{x}_i\times w_{(i-1,j)}\end{array}$$

(36)

where, $w_{(i-1,j)}$ shows the weight between input and hidden node, x = $x_i$ represents the QEN’s input, and f shows the activation function of the node. The sigmoid function is used as the activation function of the node and is given as:

$$\begin{array}{c}\hfill f\left(x\right)=\frac{1}{1+ex{p}^{-x}}\end{array}$$

(37)

Generalized policy iteration is used for tuning the parameters of QEN. The optimal action-value function is approximated by reducing temporal difference (TD) error, ${\delta }_t$ continuously with the help of NN. The TD error is given as follows:

$$\begin{array}{c}\hfill {\delta }_t=r_{t+1}+\gamma \underset{u^{\prime }}{\max }{Q}^{*}(x_{t+1},u^{\prime })-Q(x_t,u_t)\end{array}$$

(38)

The cost function is the mean square error and is given as:

$$\begin{array}{c}\hfill E=\frac{1}{2}{\delta }_t^2\end{array}$$

(39)

The gradient descent method is used for fast convergence. The rule for weight update is given as:

$$\begin{array}{ccc}\hfill w(t+1)& =& w\left(t\right)-\zeta \frac{\partial E}{\partial w}\hfill \end{array}$$

(40)

$$\begin{array}{ccc}\hfill \frac{\partial E}{\partial w}& =& {\sigma }_t\frac{\partial {\delta }_t}{\partial w}=-{\delta }_t\frac{\partial Q(x_t,u_t)}{\partial w}\hfill \end{array}$$

(41)

$\frac{\partial Q(x_t,u_t)}{\partial w}$ can be obtained by using chain rule for $w_{(40+k)}$ and $w_{(i-1,j)}$ as shown below:

$$\begin{array}{c}\hfill \frac{\partial Q(x_t,u_t)}{\partial w_{(40+k)}}=\frac{\partial Q(x_t,u_t)}{\partial V}\times \frac{\partial V}{\partial w_{(40+k)}}\\ \hfill =f^{\prime }\left(V\right)\times y\left(k\right)\end{array}$$

(42)

$$\begin{array}{c}\hfill =y_k\times Q(x_t,u_t)\times (1-Q(x_t,u_t))\\ \hfill k=1,\dots ,10\end{array}$$

(43)

$$\begin{array}{ccc}\hfill \frac{\partial Q(x_t,u_t)}{\partial w_{(i-1,j)}}& =& \frac{\partial Q(x_t,u_t)}{\partial V}\times \frac{\partial V}{\partial y_j}\times \frac{\partial y_j}{\partial a_j}\times \hfill \\ \frac{\partial a_j}{\partial w_{(i-1,j)}}\hfill & & \end{array}$$

(44)

$$\begin{array}{ccc}& =& f^{\prime }\left(V\right)\times y_j\times w_{(40+j)}\hfill \\ \hfill \times f^{\prime }\left(a_j\right)\times u_i& & \end{array}$$

(45)

$$\begin{array}{ccc}\hfill & =& w_{(40+j)}\times u_i\times Q(x_t,u_t)\times \hfill \\ & & [1-Q(x_t,u_t)]\times y_j\times [1-y_j]\hfill \\ & & i=1,\dots ,4j=1,\dots ,10\hfill \end{array}$$

(46)

$\frac{\partial Q(x_t,u_t)}{\partial u}$ can also be obtained using chain rule as given below:

$$\begin{array}{ccc}\hfill \frac{\partial Q(x_t,u_t)}{\partial u}& =& \frac{\partial Q(x_t,u_t)}{\partial V}\times \hfill \\ \hfill & & \underset{j=1}{\sum ^{10}}\left(\frac{\partial V}{\partial y_j}\times \frac{\partial y_j}{\partial w_j}\times \frac{\partial w_j}{\partial u}\right)\hfill \end{array}$$

(47)

$$\begin{array}{ccc}& =& f^{\prime }\left(V\right)\times \underset{j=1}{\sum ^{10}}\left(w_j^{\left(1\right)}{f}^{\prime }\left(w_i\right)\times w_{j,4}^{\left(2\right)}\right)\hfill \end{array}$$

(48)

$$\begin{array}{ccc}& =& Q(x_t,u_t)\times \left(1-Q(x_t,u_t)\right)\times \hfill \\ & & \underset{j=1}{\sum ^{10}}\left(w_i^{\left(1\right)}\times w_{i,4}^{\left(2\right)}\times y_j\times [1-y_j]\right)\hfill \end{array}$$

(49)

Also;

$$\begin{array}{ccc}\hfill \frac{\partial Q(x_t,u_t)}{\partial u}& =& \frac{\partial Q(x_t,u_t)}{\partial V}\times \hfill \\ \hfill & & \underset{j=1}{\sum ^{10}}\left(\frac{\partial V}{\partial y_j}\times \frac{\partial y_j}{\partial a_j}\times \frac{\partial a_j}{\partial u}\right)\hfill \end{array}$$

(50)

$$\begin{array}{ccc}& =& f^{\prime }\left(V\right)\times \underset{j=1}{\sum ^{10}}(w_{(40+j)}{f}^{\prime }\left(a_5\right)\hfill \\ \hfill & & \times w_{(30+j)}{\displaystyle )}\hfill \end{array}$$

(51)

$$\begin{array}{ccc}& =& Q(x_t,u_t)\times \left(1-Q(x_t,u_t)\right)\times \hfill \\ & & \underset{j=1}{\sum ^{10}}(w_{(40+j)}\times w_{(30+j)}\times \hfill \\ \hfill & & y_j\times [1-y_j]{\displaystyle )}\hfill \end{array}$$

(52)

It must be noted that the control output of FRNF is an input to the NN too.

6. Full Recurrent Adpative NeuroFuzzy Architectures

A variety of FRNF identifiers are used in order to identify the nonlinear $\widehat{f}\left(x\right)$ and $\widehat{g}\left(x\right)$ functions for different sub-systems in SMG-HPS. The seven-layered FRNF system uses NeuroFuzzy concept for estimation.

Fuzzy logic uses IF-THEN rules for approximation of unknown functions using standard fuzzy model. The unknown functions, $\widehat{f}\left(x\right)$ and $\widehat{g}\left(x\right)$, can be identified by the standard fuzzy model using a set of rules:

$R^m$: IF $x_1$ is $A_1^j\dots$ and $x_n$ is $A_n^j$ THEN y is ${\beta }_l^j$

Let fuzzy logic controller have q inputs, ${\rho }_1,{\rho }_2,\dots ,{\rho }_q$. The output of NeuroFuzzy system is given as:

$$\begin{array}{ccc}\hfill \mathrm{{\rm Y}}& =& \frac{{\sum }_{l=1}^m{\prod }_{j=1}^q{\mu }_{F_j^l}\left({\rho }_j\right){\beta }_l}{{\sum }_{l=1}^m{\prod }_{j=1}^q{\mu }_{F_j^l}\left({\rho }_j\right)}\hfill \end{array}$$

(53)

where ${\mu }_{F_j^l}$ is the membership function, ${\rho }_j$ and ${\beta }_l$ are adjustable parameters. It is the point in R (set of rules) at which ${\mu }_{{\beta }_j}$ achieves its maximum value. m is the number of fuzzy rules used to construct the identifier, $F_j^l$ is the jth fuzzy set corresponding to the lth fuzzy rule, and ${\beta }_l$ is centroid of the lth fuzzy set corresponding to identifier output, $\widehat{f}\left(x\right)$ and $\widehat{g}\left(x\right)$. Equation (53) can be written for $\widehat{f}\left(x\right)$ and $\widehat{g}\left(x\right)$ using fuzzy-basis function vector $\xi \left(x\right)$, as:

$$\begin{array}{c}\hfill \widehat{\mathbf{f}}\left(\mathbf{x}\right)={\xi }_{\mathbf{f}}^{\mathbf{T}}\xi \left(\mathbf{x}\right)\end{array}$$

(54)

and

$$\begin{array}{c}\hfill \widehat{\mathbf{g}}\left(\mathbf{x}\right)={\xi }_{\mathbf{g}}^{\mathbf{T}}\xi \left(\mathbf{x}\right)\end{array}$$

(55)

where

$$\begin{array}{c}\hfill {\beta }_f={[{\beta }_{f1}{\beta }_{f2}\dots {\beta }_{fm}]}^T\end{array}$$

(56)

and

$$\begin{array}{c}\hfill {\beta }_g={[{\beta }_{g1}{\beta }_{g2}\dots {\beta }_{gm}]}^T\end{array}$$

(57)

and $\xi \left(x\right)$ is given as

$$\begin{array}{c}\hfill \xi ={[{\xi }_1{\xi }_2\dots {\xi }_m]}^T=[\frac{{\prod }_{j=1}^q{\mu }_{F_j^1}\left({\rho }_j\right)}{{\sum }_{l=1}^m{\prod }_{j=1}^q{\mu }_{F_j^l}\left({\rho }_j\right)}\\ \hfill \dots \frac{{\prod }_{j=1}^q{\mu }_{F_j^l}\left({\rho }_j\right)}{{\sum }_{l=1}^m{\prod }_{j=1}^q{\mu }_{F_j^l}\left({\rho }_j\right)}]\end{array}$$

(58)

Several mathematical relations and functions are available for designing a fast and robust NeuroFuzzy identifier. The following variants were used to design antecedent and the consequent part of the fuzzy logic system for this research work.

6.1. Variants of Antecedent Part

The transformation of continuous input variables into linguistic variables is fuzzification. A membership function is always required for the transformation. The importance of the membership function is based on its shape, that translates complete information of the plant (uncertainties and nonlinearities) in a fuzzy inference system.

6.1.1. Gaussian Membership Function

The Gaussian membership function has the following properties:

• local and nonlinear nature
• smooth output

Gradient-based techniques are highly suitable for use, due to the continuous differentiable nature of the Gaussian membership function. It is expressed as:

$$\begin{array}{cc}\hfill {\mu }_j^r\left(x_i\right)& =exp\left[-{\left(\frac{x_i\left(k\right)+m_{ij}}{{\sigma }_{ij}}\right)}^2\right]\end{array}$$

(59)

where $m_{ij}$ and ${\sigma }_{ij}$ are the mean and variance of the ith input and jth membership function.

6.1.2. B-Spline Membership Function

The B-Spline membership function (locally controllable membership function formed by polynomial pieces) is used as a variant of the antecedent part. The B-Spline membership function is defined as [44]:

$${\varrho }_{ij}\left(x_i\right)=O_i^{\left(2\right)}=\begin{array}{c}{\displaystyle \sum _{k=0}^n}{\mathrm{{\rm Y}}}_k{\mathrm{\Delta }}_{k,p}\left(x_i\right)\hfill \end{array}\begin{array}{c}1\le p\le n.\hfill \end{array}$$

(60)

where, ${\varrho }_{ij}$ represents the degree of B-Spline membership function, ${\mathrm{{\rm Y}}}_k$ is the control point with $k=0,1,2,3,\dots ,n$ with $n+1$ total control points. p is the order of B-Spline basis function. ${\mathrm{\Delta }}_{k,p}\left(x_i\right)=\mathrm{\Delta }\left(x_i\backslash {\wp }_1,{\wp }_2,\dots ,{\wp }_{n+p}\right)$ is the kth B-Spline basis function and is given by the following Cox-de Boor recursion formula [44]:

$$\begin{array}{c}\hfill {\mathrm{\Delta }}_{k,p}\left(x_i\right)=\left\{\begin{array}{c}\begin{array}{c}1\mathrm{if}p=1\hfill \end{array}\begin{array}{c}x\in \left[{\wp }_i{\wp }_{i+1}\right[\hfill \end{array}\hfill \\ \begin{array}{c}\left(\frac{x-{\wp }_i}{{\wp }_{i+p-1}-{\wp }_i}\right){\mathrm{\Delta }}_{i,p-1}+\left(\frac{{\wp }_{i+p}-x}{{\wp }_{i+p}-{\wp }_{i+1}}\right)\hfill \\ {\mathrm{\Delta }}_{i+1,p-1}\mathrm{if}p>1,x\in \left[{\wp }_i{\wp }_{i+p}\right[\hfill \end{array}\hfill \\ \begin{array}{c}0\mathrm{if}p=1\hfill \end{array}\begin{array}{c}x\notin \left[{\wp }_i{\wp }_{i+1}\right[\hfill \end{array}\hfill \end{array}\right.\end{array}$$

(61)

where, $\wp =[{\wp }_1,{\wp }_2,\dots ,{\wp }_{n+p}]\in \mathbb{R}$ is the knot vector such that ${\wp }_{i+1}-\wp i\ge 0$.

This research work employed a second-order B-Spline membership function with nine control points and thirteen-knot vectors.

6.2. Variants of Consequent Part

The consequent part generates weights based on different mathematical functions, like the Fourier series function, wavelet networks, and polynomial NN. The operation of the consequent part takes place in parallel to the antecedent part and produces the final output of the identifier at the defuzzification layer. The variants of the consequent part used in this research work are given below.

6.2.1. Fuzzy Wavelet Neural Networks (NNs)

For a better estimation of nonlinear functions, wavelet NNs were proposed as a substitute to feedforward NNs. Due to numerous neurons, NNs may get struck in the local minima, that results in slower convergence of the network. To get rid of this, wavelet functions can be used in the structure. Wavelets are waves having a limited duration and zero mean value. The localization characteristics of wavelets, and the fast learning abilities of NNs, result in better outcomes for complex nonlinear system modeling. The schematic diagram of wavelet NN is given in Figure 5.

Following are the wavelet activation functions used in this research work.

• Mexican hat wavelet (MHW) is a negative normalized, non-orthogonal second derivative of Gaussian function. MHW function is expressed as;

  $$\begin{array}{ccc}\hfill {\mathrm{\Psi }}_i\left(x_i\right)& =& {\left|d_{ij}\right|}^{-\frac{1}{2}}\psi \left(z_i\right)\hfill \end{array}$$

  (62)

  where, ${\mathrm{\Psi }}_i\left(x_i\right)$ is the family of wavelets obtained by single $\psi \left(x_i\right)$ function, $z_{ij}=\frac{x_i\left(k\right)+H_{ij}{F}_{ij}-t_{ij}}{d_{ij}}$, $H_{ij}$ represents the output of the consequent part, $F_{ij}$ is adaptive recurrent feedback weight, $t_{ij}$ is the translation, and $d_{ij}\ne 0,$ for $i=1,2,\dots ,n$ is dilation respectively.
• Morlet wavelet (Mor-W) is given as [44]:

  $${\mathrm{\Psi }}_{ij}\left(z_{ij}\right)=exp\left[-0.5{\left(z_{ij}\right)}^{2}cos\left(5z_{ij}\right)\right]$$

  (63)

  where;

  $$z_{ij}=\frac{x_i\left(k\right)+H_{ij}{F}_{ij}-t_{ij}}{d_{ij}}$$

  (64)

  where $H_{ij}$ represents the output of the consequent part, $F_{ij}$ is adaptive recurrent feedback weight, $t_{ij}$ is the translation, and $d_{ij}$ is dilation of Mor-W.
• Legendre wavelets (Leg-W) are also known as spherical harmonic wavelets. They are based on Legendre polynomial, compactly supported, and orthonormal wavelets. They can be expressed as [45]:

  $$\begin{array}{c}\hfill {\mathrm{\Psi }}_{pq}^k\left(x\right)=\left\{\begin{array}{c}\begin{array}{c}{2}^{p+1}\sqrt{2k+1}\times \hfill \\ L_k\left(2^{p}x-q-0.5\right),\hfill \\ 0,\hfill \end{array}\begin{array}{c}\forall \frac{q}{2^p}\le x\le \frac{q+1}{2^p}\hfill \\ \\ otherwise\hfill \end{array}\hfill \end{array}\right.\end{array}$$

  (65)

  where, $p=1,2,\dots ,m$ and $q=0,1,\dots ,2^p-1$ shows decomposition level and integer translation, respectively. $L_k$ shows the Legendre polynomial as given in (66), with k being the degree of the polynomial [45].

  $$\begin{array}{c}\hfill L_p\left(x\right)=\frac{1}{2^{p}n!}\frac{d^n}{d{x}^n}{\left(x^2-1\right)}^p\end{array}$$

  (66)

  where, $p\ge 0,x\in \left[-11\right]$. The first four Legendre polynomials for $p=0,\dots ,3$ used in this work are given below [45]:

  $$\begin{array}{ccc}\hfill L_o\left(x\right)& =& 1,\hfill \\ \hfill L_1\left(x\right)& =& x,\hfill \\ \hfill L_2\left(x\right)& =& 0.5\left(3x^2-1\right),\hfill \\ \hfill L_3\left(x\right)& =& 0.5\left(5x^3-3x\right).\hfill \end{array}$$

  (67)
  Six Leg-W basis functions were used in this research work for $p=1$ and $q=0,\dots ,2$ defined on $\left[01\right]$.

7. Proposed Full Recurrent Adaptive NeuroFuzzy Identifier

The full recurrent adaptive NeuroFuzzy (FRNF) has seven layers as shown in Figure 6. The antecedent part consists of the first three layers, whereas the remaining four layers are consequent part layers. The n number of input signals in the first layer is equivalent to the m number of nodes and these nodes are used for input distribution.

Let $I_i^k$ and $O_i^k$ represents input and output of the ith node in the kth layer. The operation function of nodes and the signal propagation in each layer of FRNF is given below.

• Layer 1

The input is received at this layer and is passed to the next layer. The temporal relationship in the network is shown by the feedback connection of this layer.

The output is: $O_i^{\left(1\right)}=x_i$.

• Layer 2

One of the variants described in Section 6.1 is used as the membership function in this layer. This layer estimates the membership degree and fuzzy sets.

The output of this layer, in the case of the B-Spline membership function, is:

$$O_i^{\left(2\right)}={\varrho }_{ij}\left(k\right)+\upsilon {\varrho }_{ij}(k-1)$$

(68)

where, $\varrho$ represents one of the variants given in (60), $\upsilon$ is the closed-loop adjustable feedback gain of the antecedent part in case of B-Spline membership function.

The output of this layer in the case of Gaussian membership function is:

$$\begin{array}{ccc}\hfill O_i^{\left(2\right)}& =& exp\left[-{\left(\frac{x_i\left(k\right)+O_i^{\left(2\right)}(k-1){\theta }_{ij}+m_{ij}}{{\sigma }_{ij}}\right)}^2\right]\hfill \end{array}$$

(69)

where, $O_i$ represents the output of the ith node, superscript (2) indicates layer number, $ij$ subscript shows the jth term of the ith input, $x_i$ is the input, and ${\theta }_{ij}$ is the recurrent weight, ${\sigma }_{ij}$ and $m_{ij}$ are the variance and mean of the ith input and jth membership function. It must be noted that the recurrent weight of the Gaussian membership function in antecedent part is adaptive.

• Layer 3

In the rule layer, the product of the membership function is calculated. The number of rules in this layer determines the number of nodes.

The output, in the case of B-Spline membership function, is:

$$\begin{array}{ccc}\hfill O_i^{\left(3\right)}& =& \stackrel{n}{\prod _{i=1}}{O}_i^{\left(2\right)}\hfill \end{array}$$

(70)

For the Gaussian membership function, the output of this layer is:

$$\begin{array}{ccc}\hfill O_i^{\left(3\right)}& =& \stackrel{n}{\prod _{i=1}}exp\left[-{\left(\frac{x_i+O_i^{\left(2\right)}(k-1){\theta }_{ij}-m_{ij}}{{\sigma }_{ij}}\right)}^2\right]\hfill \end{array}$$

(71)

• Layer 4

This layer approximates the weighted firing strength and shows the THEN-part of fuzzy rules. The inputs to this layer are the error signal and own weighted feedback signal.

The output of this layer for Leg-W given in Section 6.2 is:

$$\begin{array}{c}\hfill O_i^{\left(4\right)}=H_i\left(k\right)+{\delta }_i{H}_i(k-1)\end{array}$$

(72)

where, for Leg-W using (65), $H_i={\displaystyle \sum _{p=1}^N}{\displaystyle \sum _{q=0}^{2^p-1}}{\displaystyle \sum _{k=0}^K}{w}_{nm}^k{\mathrm{\Psi }}_{pq}^k$, $w_{nm}$ represents Leg-W coefficients. The feedback weight is a closed-loop fixed gain for this layer.

In the case of MHW and Mor-W, the output of this layer is:

$$\begin{array}{c}\hfill O_i^{\left(4\right)}=w_i\underset{i=1}{\sum ^n}{\mathrm{\Psi }}_{ij}^k\left(Z_{ij}\right)\end{array}$$

(73)

where, $w_i$ is the weight and ${\mathrm{\Psi }}_{ij}^k\left(Z_{ij}\right)$ is given in Equations (62) and (63) for MHW and Mor-W, respectively.

It must be kept in mind that the feedback gain for MHW and Mor-W is adaptive, while the feedback gain for all other variants used is conventional feedback adjustable fixed gain.

• Layer 5

In the first defuzzification layer, the sum of products of antecedent and consequent parts of (70) and (72) is calculated.

The output is:

$$\begin{array}{ccc}\hfill O_i^{\left(5\right)}& =& \underset{i=1}{\sum ^n}{O}_i^{\left(4\right)}{O}_i^{\left(3\right)}\hfill \end{array}$$

(74)

• Layer 6

In the second defuzzification layer, the sum of all the rules from (70) is calculated.

The output is:

$$\begin{array}{ccc}\hfill O_i^{\left(6\right)}& =& \underset{i=1}{\sum ^n}{O}_i^{\left(3\right)}\hfill \end{array}$$

(75)

• Layer 7

The required nonlinear functions are approximated in the output layer.

The output is:

$$\begin{array}{c}\hfill \widehat{u_f}=O_f^{\left(7\right)}=\widehat{f}\left(x\right)=\frac{O_{fi}^{\left(5\right)}}{O_{fi}^{\left(6\right)}}\end{array}$$

(76)

$$\begin{array}{c}\hfill \widehat{u_g}=O_g^{\left(7\right)}=\widehat{g}\left(x\right)=\frac{O_{gi}^{\left(5\right)}}{O_{gi}^{\left(6\right)}}\end{array}$$

(77)

7.1. Optimization Algorithm

The training of FRNF is for the adjustment of input-output pairs or a given function by fine-tuning network parameters. Mean square error is used as the cost function for the training purpose, given as [46,47,48,49,50,51];

$$\begin{array}{ccc}\hfill E& =& \frac{1}{2}{(\widehat{y}-y)}^2,\hfill \end{array}$$

(78)

where E is the identification error, $\widehat{y}$ is the approximated output of the subsystem, and y is the actual output of the subsystem, respectively. The gradient descent method is used for fast cost function reduction and convergence [46,47,48,49,50,51]. The general equation is given as follows:

$$\begin{array}{ccc}\hfill \mathrm{\Omega }(k+1)& =& \mathrm{\Omega }\left(k\right)-\gamma g_k\hfill \end{array}$$

(79)

where, $g_k$ is the gradient of cost function at kth iteration, $\gamma >0$ is the learning rate and k is the iteration index.

8. Full Recurrent Adaptive NeuroFuzzy Identifiers

The parameters of FRNF were tuned, based on the estimation of $Q(x,u)$, as discussed in Section 5.1. The optimization of FRNF output was achieved by updating the parameters of FRNF and maximizing the action-value function $Q(x,u)$ w.r.t. control output u. The parameters of FRNF were tuned using the gradient descent method, described in Section 7.1. The following three different combinations of FRNF algorithms were used in this research work for tuning of parameters.

8.1. FRNF-HBs-LegW Identifier

In this FRNF the identifier was the B-Spline membership function, described in Section 6.1.2, while the consequent part was the Legendre wavelet given in Section 6.2.1. The proposed identifier was a seven-layered scheme, discussed in Section 7. The schematic diagram of the identifier is shown in Figure 7. Both the antecedent and consequent parts were conventional recurrent.

The update equations of all the parameters of the antecedent part and the consequent part, according to (79), used the following chain rules:

$$\begin{array}{c}\hfill \frac{\partial E}{\partial {\zeta }_{ij}}=\frac{\partial E}{\partial {\xi }_i}\frac{\partial {\xi }_i}{\partial {\varsigma }_i}\frac{\partial {\varsigma }_i}{\partial {\zeta }_{ij}}\end{array}$$

(80)

where, ${\zeta }_{\mathit{i}\mathit{j}}$ shows the variants of B-Spline membership function defined in (60). The update equations for B-Spline membership function in antecedent part is:

$$\begin{array}{c}\hfill {\mu }_{ij}(p+1)={\mu }_{ij}\left(p\right)+\gamma e_q\frac{{\xi }_j({\beta }_i-\widehat{u}u)}{{\mu }_{ij}{\displaystyle \sum _{j=1}^n}{\xi }_j}\end{array}$$

(81)

where, $\widehat{u}$ is the output of identifier given in Equation (76), $e_q=\left(e\frac{\partial y}{\partial \zeta }\right)$, ${\xi }_j$ is output of Equation (70), $\beta$ is output of Equation (72), update parameters are rules and weights $\left[\mu \right]$.

The chain rule for consequent part is given below:

$$\begin{array}{c}\hfill \frac{\partial E}{\partial {\vartheta }_{ij}^{\left(p\right)}}={\psi }_{ij}^{\left(p\right)}\frac{{\xi }_j\left(x_i\right)}{{\displaystyle \sum _{j=1}^n}{\xi }_j\left(x_i\right)}\end{array}$$

(82)

where ${\psi }_i$ represents Leg-W as given in (65) and $\mathbf{\vartheta }$ shows the Leg-W variants. The update equations are:

$$\begin{array}{ccc}\hfill {\vartheta }_{ij}^p(p+1)& =& {\vartheta }_{ij}^p\left(p\right)+\gamma e_q\frac{{\xi }_j}{{\displaystyle \sum _{j=1}^n}{\xi }_j}\times {\psi }_{ij}^{\left(p\right)}\hfill \end{array}$$

(83)

where, $e_q=\left(e\frac{\partial y}{\partial \vartheta }\right)$, ${\xi }_j$ is output of Equation (70), and update parameters are polynomials of Leg-P $\left[{\vartheta }_{ij}^p\right]$ given in Equation (67).

8.2. FRNF-MHW Identifier

In this FRNF, the identifier is given in Equation (76), the antecedent part in this control scheme is the Gaussian membership function, described in Section 6.1.1, while the consequent part is the MHW described in Section 6.2.1. The FRNF-MHW identifier has a seven-layered architecture, given in Section 7. The schematic diagram of identifier is shown in Figure 8. The recurrent part in this identifier is adaptive for both antecedent and consequent parts.

The update equations of all the parameters of the antecedent part, according to (79), uses the following chain rules:

$$\begin{array}{c}\hfill \frac{\partial E}{\partial {\zeta }_i}=\frac{\partial E}{\partial {\xi }_i}\frac{\partial {\xi }_i}{\partial {\mu }_i}\frac{\partial {\mu }_i}{\partial {\zeta }_i}\end{array}$$

(84)

where, ${\mathit{\zeta }}_{\mathit{i}}$ shows the variant like mean, variance and feedback weight of Gaussian membership function.

$$\begin{array}{c}\hfill m_i(k+1)=m_i\left(k\right)+\gamma e_q\left[\frac{({\beta }_i-u){\xi }_i}{{\displaystyle \sum _{j=1}^n}{\xi }_i}\right]\times \\ \hfill \left[\frac{x_i\left(k\right)+{\mu }_i(k-1){\theta }_i-m_i}{{\left({\sigma }_i\right)}^2}\right]\end{array}$$

(85)

$$\begin{array}{c}\hfill {\sigma }_i(k+1)={\sigma }_i\left(k\right)+\gamma e_q\left[\frac{({\beta }_i-u){\xi }_i}{{\displaystyle \sum _{j=1}^n}{\xi }_i}\right]\times \\ \hfill \left[\frac{(x_i\left(k\right)+{\mu }_i(k-1){\theta }_i-m_i)}{{\left({\sigma }_i\right)}^3}\right]\end{array}$$

(86)

$$\begin{array}{c}\hfill {\theta }_i(k+1)={\theta }_i\left(k\right)-\gamma e_q\left[\frac{({\beta }_i-u){\xi }_i}{{\displaystyle \sum _{j=1}^n}{\xi }_i}\right]\times \\ \hfill \left[\frac{x_i\left(k\right)+{\mu }_i(k-1){\theta }_i-m_i}{{\left({\sigma }_i\right)}^2}\right]\end{array}$$

(87)

where, $\widehat{u}$ is the output of identifier, $e_q=\left(e\frac{\partial y}{\partial \zeta }\right)$, ${\xi }_j$ is output of Equation (70), $\beta$ is output of Equation (72), update parameters are mean, variance, and feedback weight $\left[m\sigma \theta \right]$.

The chain rule for the consequent part is given below:

$$\begin{array}{c}\hfill \frac{\partial E}{\partial {\vartheta }_i}=\frac{\partial E}{\partial {\xi }_i}\frac{\partial {\xi }_i}{\partial {\beta }_i}\frac{\partial {\beta }_i}{\partial {\mathrm{\Psi }}_i}\frac{\partial {\mathrm{\Psi }}_i}{\partial z_i}\frac{\partial z_i}{\partial {\vartheta }_i}\end{array}$$

(88)

where here ${\mathrm{\Psi }}_i$ represents MHW given in Equation (62), $z_i$ defined below is an intermediate state variable and $\vartheta$ shows the variants of MHW like, translation, dilation, feedback weight etc.

$$\begin{array}{c}\hfill z_i=\left(\frac{x_i\left(k\right)+H_i{F}_i-t_i}{d_i}\right)\end{array}$$

(89)

$$\begin{array}{c}\hfill t_i(k+1)=t_i\left(k\right)-\gamma e_q\left(\frac{{\xi }_i{w}_i}{{\displaystyle \sum _{j=1}^n}{\xi }_i}\right)\times \\ \hfill \left[e^{-0.5{\left(z_i\right)}^2}\frac{\left[\frac{0.5}{z_i}-3.5z_i+{\left(z_i\right)}^3\right]}{|d_i{|}^{3/2}}\right]\end{array}$$

(90)

$$\begin{array}{c}\hfill d_i(k+1)=d_i\left(k\right)-\gamma e_q\left(\frac{{\xi }_i{w}_i}{{\displaystyle \sum _{j=1}^n}{\xi }_i}\right)\times \\ \hfill \left[e^{-0.5{\left(z_i\right)}^2}\frac{\left[0.5-3.5z_i^2+{\left(z_i\right)}^4\right]}{|d_i{|}^{3/2}}\right]\end{array}$$

(91)

$$\begin{array}{c}\hfill F_i(k+1)=F_i\left(k\right)+\gamma e_q\left(\frac{{\xi }_i{w}_i}{{\displaystyle \sum _{j=1}^n}{\xi }_i}\right)\times \\ \hfill \left[e^{-0.5{\left(z_i\right)}^2}\frac{\left[\frac{0.5}{z_i}-3.5z_i+{\left(z_i\right)}^3\right]}{|d_i{|}^{3/2}}\right]H_i\end{array}$$

(92)

where, $e_q=\left(e\frac{\partial y}{\partial \vartheta }\right)$, ${\xi }_j$ is output of Equation (70), and update parameters are translation, dilation, and feedback weight $\left[tdF\right]$.

The weight of the consequent part is updated according to the following chain rule:

$$\begin{array}{c}\hfill \frac{\partial E}{\partial w_i}=\frac{\partial E}{\partial {\xi }_i}\frac{\partial {\xi }_i}{\partial {\beta }_i}\frac{\partial {\beta }_i}{\partial w_i}\end{array}$$

(93)

where $w_i$ represents the weight of the consequent layer.

$$\begin{array}{c}\hfill w_i(k+1)=w_i\left(k\right)+\gamma e_q\left(\frac{{\xi }_i}{{\displaystyle \sum _{j=1}^n}{\xi }_i}\right)\left[{\psi }_{i+1}+{\psi }_i\right]\end{array}$$

(94)

where, $\psi$ shows the MHW given in Equation (62).

8.3. FRNF-Mor-W Identifier

In this FRNF, the identifier is given in Equation (76), the antecedent part in this control scheme is the Gaussian membership function, described in Section 6.1.1, while the consequent part is the Morlet wavelet given in Section 6.2.1. The FRNF-Mor-W identifier has a seven-layered architecture, given in Section 7. The schematic diagram of the identifier is shown in Figure 9. The recurrent part in this identifier is adaptive for both antecedent and consequent parts.

The update equations of all the parameters of the antecedent part, according to (79), use the following chain rules:

$$\begin{array}{c}\hfill \frac{\partial E}{\partial {\zeta }_i}=\frac{\partial E}{\partial {\xi }_i}\frac{\partial {\xi }_i}{\partial {\mu }_i}\frac{\partial {\mu }_i}{\partial {\zeta }_i}\end{array}$$

(95)

where, ${\mathit{\zeta }}_{\mathit{i}}$ shows the variant like mean, variance and feedback weight of Gaussian membership function.

$$\begin{array}{c}\hfill m_i(k+1)=m_i\left(k\right)+\gamma e_q\left[\frac{({\beta }_i-\widehat{u}){\xi }_i}{{\displaystyle \sum _{j=1}^n}{\xi }_i}\right]\times \\ \hfill \left[\frac{x_i\left(k\right)+{\mu }_i(k-1){\theta }_i-m_i}{{\left({\sigma }_i\right)}^2}\right]\end{array}$$

(96)

$$\begin{array}{c}\hfill {\sigma }_i(k+1)={\sigma }_i\left(k\right)+\gamma e_q\left[\frac{({\beta }_i-u){\xi }_i}{{\displaystyle \sum _{j=1}^n}{\xi }_i}\right]\times \\ \hfill \left[\frac{(x_i\left(k\right)+{\mu }_i(k-1){\theta }_i-m_i)}{{\left({\sigma }_i\right)}^3}\right]\end{array}$$

(97)

$$\begin{array}{c}\hfill {\theta }_i(k+1)={\theta }_i\left(k\right)-\gamma e_q\left[\frac{({\beta }_i-u){\xi }_i}{{\displaystyle \sum _{j=1}^n}{\xi }_i}\right]\times \\ \hfill \left[\frac{x_i\left(k\right)+{\mu }_i(k-1){\theta }_i-m_i}{{\left({\sigma }_i\right)}^2}\right]\end{array}$$

(98)

where, $\widehat{u}$ is the output of identifier given in Equation (76), $e_q=\left(e\frac{\partial y}{\partial \zeta }\right)$, ${\xi }_j$ is output of Equation (70), $\beta$ is output of Equation (72), update parameters are mean, variance, and feedback weight $\left[m\sigma \theta \right]$.

The chain rule for the consequent part is given below:

$$\begin{array}{c}\hfill \frac{\partial E}{\partial {\vartheta }_i}=\frac{\partial E}{\partial {\xi }_i}\frac{\partial {\xi }_i}{\partial {\varsigma }_i}\frac{\partial {\varsigma }_i}{\partial {\mathrm{\Psi }}_i}\frac{\partial {\mathrm{\Psi }}_i}{\partial {\vartheta }_i}\end{array}$$

(99)

where ${\mathrm{\Psi }}_i$ represents Mor-W as given in (63) and $\mathbf{\vartheta }$ shows the variants of Mor-W-like translation, and dilation as given in (63). The update equations are:

$$\begin{array}{ccc}\hfill t_{ij}(p+1)& =& t_{ij}\left(p\right)+\gamma e_q\mho \hfill \end{array}$$

(100)

$$\begin{array}{ccc}\hfill d_{ij}(p+1)& =& d_{ij}\left(p\right)+\gamma e_q\mho z_{ij}\hfill \end{array}$$

(101)

$$\begin{array}{ccc}\hfill w_{ij}(p+1)& =& w_{ij}\left(p\right)+\gamma e_q\frac{{\xi }_j}{{\displaystyle \sum _{j=1}^n}{\xi }_j}\hfill \end{array}$$

(102)

where, $e_q=\left(e\frac{\partial y}{\partial \vartheta }\right)$, ${\xi }_j$ is output of Equation (70), $\mho =\frac{{\xi }_j{w}_j^{\left(5\right)}}{{\displaystyle \sum _{j=1}^n}{\xi }_j}\times \frac{\left(cos\left(5z_{ij}\right)e^{-0.5z_{ij}^2}{z}_{ij}+5sin\left(5z_{ij}\right)e^{-0.5z_{ij}^2}\right)}{d_{ij}^{\left(5\right)}}$, and update parameters are rules, translation, dilation, and weight $\left[\mu tdw\right]$.

9. Exploration Policy and Action Modifier

If all the actions from all the states are tried, then $Q(x,u)$ converges to $Q^{*}(x,u)$, with a probability of 1 [52]. In this research work, an exploration policy was implemented that guaranteed the trail of all possible actions for the control output $\widehat{u}$ given by the FRNF. The AEM generated a control output $u_c$. The $u_c$ was the sum of a disturbed action $u_d$ and $\widehat{u}$. The standard deviation of $u_d$ was ${\sigma }_{Q\left(t\right)}$, which was recommended by FRNF and its normal distribution was zero. The problem of exploration was solved by this AEM in reinforcement learning [42].

$$\begin{array}{ccc}\hfill u_c& =& \widehat{u}+u_d{u}_d\sim N(0,{\sigma }_{Q\left(t\right)})\hfill \end{array}$$

(103)

where,

$$\begin{array}{ccc}\hfill {\sigma }_{Q\left(t\right)}& =& \frac{k}{[1+2exp(maxQ(x,a)\left)\right]}\hfill \end{array}$$

(104)

where, k is a variable that shrinks or expands $u_d$.

10. Proposed Hybrid Adaptive Neural Network Q-Learning-Based Full Recurrent Adaptive NeuroFuzzy Control Paradigms

Using different combinations of the FRNF identifier algorithms, discussed in Section 8, the following three hybrid adaptive Q-learning control paradigms were proposed:

• Hybrid adaptive full recurrent Legendre wavelet-based Neural Network Q-learning control
  In this control scheme, the NNQLNF was embedded with an FRNF-Leg wavelet identifier, discussed in Section 5.1 and Section 8.1.
• Hybrid adaptive full recurrent Mexican hat wavelet-based Neural Network Q-learning control
  In this control scheme, the NNQLNF was embedded with the FRNF-MHW identifier, discussed in Section 5.1 and Section 8.2.
• Hybrid adaptive full recurrent molet wavelet-based Neural Network Q-learning-based control
  In this control scheme, the NNQLNF was embedded with the FRNF-Mor-W identifier, discussed in Section 5.1 and Section 8.3.

11. Implementation Procedure of Hybrid Adaptive NNQLNF Control Paradigms

The following steps took place for the implementation of the proposed hybrid adaptive NNQLNF control paradigms [42].

• In the first step, the initialization of $Q(x_t,u_t)$, the parameters of FRNF and the weights $w_1-w_{40}$, $w_{41}-w_{50}$ of QEN took place.
• Control output $u_t$ was obtained from the FRNF identifier.
• $u_t$ was then processed by AEM, according to Equation (103).
• $u_c$ was the actual control output fed to the system.
• The estimated $Q(x_{t+1},u_{t+1})$ was obtained from QEN, depending on control action, previous and current states.
• The performance of controller r, $Q(x_t,u_t)$ and $Q(x_{t+1},u_{t+1})$, given in Equation (38), were used to estimate the TD error from Equation (38).
• QEN was updated, based on Equations (43) and (46).
• Parameters of FRNF were updated.
• $Q(x_t,u_t)$ was updated to $Q(x_{t+1},u_{t+1})$.
• If the parameters of QEN and FRNF were not updated for a specific interval of time, then the learning procedure terminated; otherwise, the algorithm was repeated from step 2 again.

12. PQ Control of Interlinking Inverter Using Hybrid Adaptive NN Based Q-Learning Full Recurrent Adaptive NeuroFuzzy Control Paradigms

This section focuses on the PQ control of the interlinking inverter of SMG-HPS using the adaptive hybrid NNQLNF control paradigms, discussed in Section 5. The main aim of this research was to minimize the THD, frequency, and voltage fluctuations, which arise due to the use of multiple renewable energy sources and non-renewable energy sources along with converters and nonlinear loads. The objective was achieved by PQ control using the three proposed adaptive hybrid NNQLNF-based control schemes on interlinking inverters. The results were compared with an aPID control scheme.

13. Formulation of Control Problem

The active and reactive powers were obtained from the AC-bus. The difference between actual and reference power generated the error signal, e, which was used for tuning the parameters of the proposed control schemes.

$$\begin{array}{c}\hfill e_p=P_{ref}-P_{AC}\end{array}$$

(105)

$$\begin{array}{c}\hfill e_q=Q_{ref}-Q_{AC}\end{array}$$

(106)

where, $e_p$ is the real power error, $P_{ref}$ is the real reference power, $P_{AC}$ is the real AC bus power, $e_q$ is the reactive power error, $Q_{ref}$ is the reactive reference power and $Q_{AC}$ is the reactive AC-bus power.

The error signals were input to the proposed control schemes. The control output obtained from Equation (103) of both real and reactive power was transformed from dqo to abc and was fed to the PWM generator. The control objective was achieved by the convergence of the following signals:

$$\begin{array}{c}\hfill \underset{t\to +\infty }{lim}\left\{\begin{array}{c}{P}_{AC}\left(t\right)\to P_{ref}\left(t\right)\hfill \\ Q_{AC}\left(t\right)\to Q_{ref}\left(t\right)\hfill \end{array}\right.\end{array}$$

(107)

The proposed control schemes achieved the objective and, thus, produced minimum THD and negligible fluctuations in frequency and voltage profiles.

14. Results and Discussion

The SMG-HPS testbed was prepared and simulated in Matlab/Simulink R2015a. The technical details of of SMG-HPS are discussed in Section 2. The real-time environmental data (wind speed, ambient temperature, and solar irradiance) were obtained from the Pakistan Meteorological Department (PMD) for a complete solar day at the Islamabad station. The case study taken was the Defense Housing Authority (DHA), Islamabad, Pakistan. The irradiation varied with the appearance of the sun. The average irradiation level during day time was about 1000 W/m${}^2$, while the average temperature level was 20 ${}^{°}$C, having a maximum peak of about 42.6 ${}^{°}$C during the day time. Figure 10 shows the irradiance profile on the left y-axis and the temperature profile on the right y-axis, while Figure 11 shows the wind speed profile used for this case study.

In this study, three intelligent hybrid adaptive NNQLNF control schemes were implemented on an interlinking converter in the proposed SMG-HPS. The performances of all the control schemes were compared with an aPID control scheme. Based on the simulation results, the best performing controller was identified.

For stability of the power system, the net power on the AC-bus should be equal to zero. Figure 12 shows the net active power profiles of the AC-bus for three NNQLNF-based controllers and an aPID control scheme. It was clear from the results that the net power, due to the aPID was of higher magnitude, as compared to the proposed control schemes. The negligible variations in the net real power on the AC-bus was due to the use of power electronic devices, converters, and nonlinear loads. However, the least magnitude was observed for the hybrid adaptive FRNF-MHW-based NNQLNF control scheme.

Figure 13 shows the net reactive power profile of the AC-bus for the proposed control schemes, compared with the aPID. The plots show the variation due to the aPID was of greater magnitude, as compared to the proposed control schemes. The variation in net reactive power of the AC-bus due to the proposed control schemes was of negligible magnitude. Little variation in the results was due to the use of multiple RESs that produced varying outputs, depending on weather conditions. Other factors of the variations in net active and reactive power profiles were due to the use of power electronic converters, CS, and nonlinear loads.

For the stability of the power system and life of various loads, the frequency of the AC-bus must remain close to 50 Hz. Figure 14 shows the frequency of the AC-bus maintained by the interlinking converter during various modes of operation in SMG-HPS. The results revealed that the frequency of the AC-bus for an aPID control scheme had greater variations, which could lead to instability of the power system. The larger magnitude of the frequency at the AC-bus reduced the life of power electronic devices and other sensitive loads. The frequency variations were responsible for imperfect charging/discharging of power storage devices, and, thus reduced their efficiency as well. It must be noted that the frequency maintained by using the proposed intelligent NNQLNF-based control schemes for interlinking converters was nearly 50 Hz all the time. However, negligible variations found in the simulation results were because of continuous switching of power between AC sub-grid and DC sub-grid. Other factors included the use of weather-dependent RES, power electronic converters, and nonlinear loads.

Table 2. Average values of $\mathrm{\Delta }{P}_{AC-bus}$ and $\mathrm{\Delta }{Q}_{AC-bus}$ variations for all controllers.

Control Scheme	P${}_{\mathit{AC}-\mathit{bus}}$	Q${}_{\mathit{AC}-\mathit{bus}}$
	($\mathit{\mu }$Watts)	($\mathit{\mu }$VARs)
aPID	1.0700	0.191100
FRNF-Leg wavelet based NNQLNF control	0.008600	0.008863
FRNF-Mor-W based NNQLNF control	0.008642	0.008794
FRNF-MHW based NNQLNF control	0.008472	0.008698

shows the average values of $\mathrm{\Delta }$P${}_{\mathrm{AC}-\mathrm{bus}}$ and $\mathrm{\Delta }$Q${}_{\mathrm{AC}-\mathrm{bus}}$ variations.

Figure 15 shows the %age change in load current frequency by various control schemes. The results revealed the performance of the proposed control schemes as being superior over the aPID control scheme. The negligible variation in the results for the proposed control schemes was due to sudden load changes, inverters, regulators, and charging/discharging operation of multiple devices. The variations were also because of power transfer to the utility grid during peak hours and bidirectional power transfer between the charging station and micro-grid. However, the results obtained from the proposed hybrid adaptive FRNF-MHW-based NNQLNF control scheme were more satisfactory and reliable, compared to the other control schemes.

Figure 16 shows the RMS voltage profiles for the various control schemes. The results showed that the RMS voltage for the aPID control scheme was not stable for any specific time interval and was continuously varying. This reduced the life of power electronic devices and other machines connected to the SMG-HPS. However, the RMS voltage for the proposed control schemes showed stable magnitude, compared with an aPID control scheme. The variations were because of switching of power between the AC sub-grid and the DC sub-grid, along with the varying output power of RES due to weather conditions.

Figure 17 shows the comparison of the percentage change in THD for the load current. The %$\mathrm{\Delta }$THD in load current due to the proposed control schemes was of lower magnitude, as compared to an aPID control scheme. The %$\mathrm{\Delta }$THD complied with IEEE standard 1547 [53]. However, the least magnitude was observed for the proposed adaptive FRNF-MHW-based NNQLNF control scheme, which proved its performance over other control schemes.

Table 3. Average values of %$\mathrm{\Delta }$f${}_{\mathrm{Load}}$ and %$\mathrm{\Delta }$THD${}_{\mathrm{Load}}$ variation in load current for all controllers.

Control Scheme	%$\mathbf{\Delta }$f${}_{\mathbf{Load}}$	%$\mathbf{\Delta }$THD${}_{\mathbf{Load}}$
aPID	0.0025480	1.29900
FRNF-Leg wavelet based NNQLNF control	0.0006112	0.06519
FRNF-Mor-W based NNQLNF control	0.0006130	0.06551
FRNF-MHW based NNQLNF control	0.0005992	0.06423

shows the average values of %$\mathrm{\Delta }$f${}_{\mathrm{Load}}$ and %$\mathrm{\Delta }$THD${}_{\mathrm{Load}}$ in load current.

Figure 18 shows the maximum output power obtained for the best performing controller given in the previously published article [28], with the same SMG-HPS. The interlinking converter in the research work referred to above was controlled by an aPID controller. However, in this research work, the maximum power output of PV array was obtained with different proposed controllers for the interlinking inverter. It is clear from the figure that the overall maximum power was obtained for the proposed controllers of the interlinking inverter. However, the maximum power was seen for he proposed adaptive FRNF-MHW-based NNQLNF control scheme, which proved its performance over other control schemes.

Figure 19 shows the maximum output power obtained for the best performing controller given in a previously published article [29], with the same SMG-HPS. The interlinking converter in the research work referred to above was controlled by an aPID controller. However, in this research work the maximum power output of the wind turbine was obtained with different proposed controllers for the interlinking inverter. It is clear from the figure that the overall maximum power was obtained for the proposed controllers of the interlinking inverter. However, the maximum power was observed for the proposed adaptive FRNF-MHW-based NNQLNF control scheme, which proved its performance over other control schemes.

Figure 20 shows a spider chart for comparable parameters of interlinking converter controllers. The values were scaled ($\mathrm{\Delta }$P${}_{AC}$ by 100, $\mathrm{\Delta }$Q${}_{AC}$ by 20, %$\mathrm{\Delta }$f by 10, and %$\mathrm{\Delta }$THD by 15) to have a clear picture. The observations and analysis revealed that the hybrid adaptive FRNF-MHW-based NNQLNF control scheme provided superior performance over all other control schemes.

15. Conclusions

In this article, mathematical modeling of the proposed hybrid adaptive FRNF, based on an NN Q-learning control scheme, was discussed in detail. The proposed control schemes used BP NN, FRNF, and AEM for estimation of theoptimal action-value function, parameters update and action exploration modifier. An intelligent supervisory control system was also described in detail with nine modes of operation. The supervisory control was responsible for optimal power flow and ensured the power balance between generation and load.

This research work also focused on three adaptive FRNF-based NN Q-Learning control schemes for the control of a bidirectional interlinking converter between bybrid AC and DC sub-grids in the proposed SMG-HPS. The results were compared with an aPID control scheme. The use of intelligent controllers ensured the following: (i) power system stability, (ii) power quality, and (iii) reliability. These benefits were due to reducing the following: (a) %age $\mathrm{\Delta }$THD and (b) %age $\mathrm{\Delta }$f in the load current for various modes of the power system operation. The proposed control schemes improved stability by reducing net real and reactive powers on the AC-buses. However, the overall best performance was observed for the proposed adaptive FRNF-MHW-based NN Q-Learning control scheme.

16. Future Work

Intelligent adaptive supervisory control based on NN/Fuzzy logic, advanced control schemes to schedule BSS/PHEV charging/discharging for profit gain, inclusion of priority-based sensitive loads, like data centers/hospitals, are some interesting and recommended future work.