Improvement of the Control of a Grid Connected Photovoltaic System Based on Synergetic and Sliding Mode Controllers Using a Reinforcement Learning Deep Deterministic Policy Gradient Agent

Round 1

Reviewer 1 Report

This paper has mainly studied improvement of the control of grid connected photovoltaic system based on synergetic and sliding mode controllers using reinforcement learning deep deterministic policy gradient agent. The authors presented presents the control with a cascade type structure of the grid connected PV array system, the control of a grid connected PV array system has been implemented in matlab/simulink. Numerical simulations prove the superiority of the control system that uses the RL-TD3 agent. This method is validated by the simulation. Generally, this paper was well organized and presented. It could be accepted to future publication if the authors can address some concerns as listed below: (1) This paper mainly discussed the improvement of the control of grid connected photovoltaic system, but what is the improvement performance? Please consider it and state them. (2) What was the working in the experimental test? If the author can, please consider it and state them. (3) When the state trajectory reaches the sliding mode surface, it is difficult to slide strictly along the sliding mode towards the equilibrium point, but cross back and forth on both sides to approach the equilibrium point, resulting in chattering, how to solve it?

Author Response

Dear reviewer, thanks for your recommendations.

           

            (1) This article is a follow-up on the comparison of the performances on a benchmark used in a series of ISI Web of Science articles (Conferences or Journals) [26,27,31,32]. In these articles, the following items are compared as performance indicators of the control system: response time, voltage ripple, overshooting, and steady-state error. We specify that the control systems are based on controllers ranging from simple PI-type to complex SMC-type controllers. In [32], we, the authors of the article under discussion (Nicola M. and Nicola C.-I.) proposed, to our best knowledge, for the first time a cascade structure in which the inner control loop is based on a SMC-type controller, and the inner control loop is based on a Synergetic-type controller. Moreover, with such a control structure, to their best knowledge, both for the benchmark under discussion in this article and for a benchmark on the control of a PMSM in [33], the same authors obtained the best performance of the control system. Indeed, in Articles [32] and [33], these control systems proposed by the authors also included the fractional order calculus, which brought extra performance by approximately 10% compared to the structure without the fractional order calculus, but together they provide peak performance of the control system. The main disadvantage of using the fractional order calculus consists in using very powerful DSPs in order to implement such control algorithms.

            Thus, this article presents the improvement of the performance of the PI or SMC type control system plus Synergetic type controllers (i.e. the simplest control systems with one of the best performance) by using an RL-TD3 agent, trained so as to provide correction signals to the primary control system. The improved performances are among those presented above and are summarized in Tables 1 and 2.

            It can be noted that, between PI-type the control systems and the PI-RL-TD3 agent, the response time is improved by approximately 7ms, and between the SMC plus Synergetic type control systems and the SMC-RL-TD3 agent and Synergetic-RL-TD3 agent, the response time is improved by approximately 1ms, i.e. a decrease of the response time by approximately 18% in the first case and by approximately 7% in the second case. However, between the simplest and most complex case, the decrease in the response time is about 27 ms, which means a decrease by about 70%. In the same way, the rest of the performances can be evaluated as relative or absolute units.

Table 1. Control of the grid connected PV array system based on PI-type controllers using RL-TD3 agent performances.

Controllers for the grid connected PV system	Response time [ms]	Voltage ripple [V]	Overshooting [%]	Steady-state error [%]
PI	40.4	57.87	<0.5%	0.2%
PI using RL-TD3 agent for the correction of idref.	37.1	57.23	<0.5%	0.2%
PI using RL-TD3 agent for the correction of udref and uqref	35.9	56.67	<0.5%	0.2%
PI using RL-TD3 agent for the correction of udref, uqref, and idref	33.8	56.22	<0.5%	0.2%

 

Table 2. Control of the grid connected PV system based on SMC and Synergetic controllers using RL-TD3 agent performances.

Controllers for the grid connected PV system	Response time [ms]	Voltage ripple [V]	Overshooting [%]	Steady-state error [%]
SMC and Synergetic	14.1	55.63	<0.2%	0.02%
SMC and Synergetic using RL-TD3 agent for the correction of idref.	13.7	55.12	<0.2%	0.02%
SMC and Synergetic using RL-TD3 agent for the correction of udref and uqref	13.5	54.58	<0.2%	0.02%
SMC and Synergetic using RL-TD3 agent for the correction of udref, uqref, and idref	13.2	54.03	<0.2%	0.02%

 

            (2) This article is a continuation regarding the comparison of performances on a benchmark used in a series of ISI Web of Science (Conferences or Journals) [26,27,31,32].  In these articles, they are compared as performance indicators of the control system: response time, voltage ripple, overshooting, and steady-state error. Also, all the elements shown in Figure 5 (the type of PV array, the type input irradiance and temperature signals, the type of DC-DC converters, and the load) were kept for comparison purposes precisely to present comparatively only the performance of the control systems. In none of these articles was presented a validation experiment performed exclusively by numerical simulations. In [33] the authors of the article in question for a similar control system, but for a benchmark on the control of a PMSM (with prices incomparably lower than the benchmark in the article under debate), the authors present in detail the implementation in real time using DSP.

            (3) Following [32], an article by the same authors, to improve convergence and reduce high frequency oscillations, the sgn function (blue line in the figure represented below) is replaced with the function below:

 (red line in the figure represented below)  

For a=4 and b=0, , and a smoothed transition is achieved between -1 and 1.

 

            This improvement was used by the authors in a series of ISI Web of Science indexed articles (Conference or Journals):

• Marcel NICOLA, Claudiu-Ionel NICOLA, Camelia MARINESCU, Sensorless Control of PMSM using FOC Strategy Based on PI-ILC Law and Sliding Mode Observer, Proceedings of the XXIst International Symposium on Electrical Apparatus and Technologies (SIELA 2020), Bourgas, Bulgaria, 3-6 June 2020, pp. 1-6, DOI: 10.1109/SIELA49118.2020.9167046;
• Marcel NICOLA, Claudiu-Ionel NICOLA, Sensorless Control for PMSM Using Model Reference Adaptive Control and back-EMF Sliding Mode Observer, Proceedings of the 12th International Conference and Exhibition on Electromechanical and Energy Systems (SIELMEN), Chișinău, Moldova, 10-11 October 2019, pp. 317-322, DOI: 10.1109/SIELMEN.2019.8905805;
• Marcel NICOLA, Claudiu-Ionel NICOLA, Adrian VINTILĂ, Sensorless Control of Multi-Motors PMSM using Back-EMF Sliding Mode Observer, Proceedings of the Electric Vehicles International Conference (EV2019), București, Romania, 3-4 October 2019, pp 1-6, DOI: 10.1109/EV.2019.8892950;
• Marcel NICOLA, Claudiu-Ionel NICOLA, Marian DUȚĂ, Adaptive Sensorless Control of PMSM using Back-EMF Sliding Mode Observer and Fuzzy Logic, Proceedings of the Electric Vehicles International Conference (EV2019), București, Romania, 3-4 October 2019, pp 1-6, DOI: 10.1109/EV.2019.8893070;

Author Response File: Author Response.pdf

Reviewer 2 Report

The presented manuscript paper proposes a benchmark within two different PV control grid with and without using RL based on TD. This manuscript has three parts, the first part consists in an introduction and theoretical background of the proposal method. Moreover, the RL is explained. The second part, they present the currents correction for PI based control and SMC based control using the RL-TD, and, finally, the numerical simulations are yield in the third part. The benchmark only gives two parameters response time and voltage ripple. Later, they stablish the discussion and conclusions.

The main conclusion is that the method proposed is superior. The results yielded better than the PI and SMC controllers.

The paper has "flaw" which must be commented:

• The figures are bad showed, some of them is not possible read nothing.
• Most of the figures are not necessary.
• The improvement is inappreciable, 7 seconds and less than 1 voltage for PI controller and 1 second and less again than 1 voltage for the SMC controller.

In the following list, I report issues that should be addressed:

1. Improve the figures.
2. Select the figures that are necessary.
3. What is the problem if the network response improves in 7 seconds? Or in other words, why is that important?
4. What is the contribution of this paper to the improvement?
5. Are there only these two benchmark parameters? Please, add more benchmark parameter.

Author Response

Dear reviewer, thanks for your recommendations.

1. We made the improvement of the quality for figures.
2. Your observation, in the opinion of the authors (who in turn are also reviewers at MDPI) is subjective. In this regard, with all due respect, we quote some elements by another reviewer of this article regarding the aspects indicated by you. “This article proposes an original cascade control structure to improve the performances of the grid connected PV array system starting from a benchmark. To achieve this goal the authors start with the presentation of main characteristics of the benchmark system. Then the structure of cascade control system is described, in which PI controllers for the inner control loop of currents i_dand i_q, respectively a SMC-type controller for the outer control loop of u_dc voltage are used. Furthermore the improvement of the grid connected PV array performances are obtained by using the elements regarding the RL-TD3 agent. A Matlab/Simulink application is implemented to prove the parametric robustness of the proposed control system based on SMC and Synergetic controllers in case of variation with 30% of the three-phase load. Moreover, the results of the numerical simulations are presented comparatively and the validation of control system superiority that uses the RL-TD3 agent is demonstrated.

The article is well structured; the proposed method is clear and correct presented. The simulation results seem promising.”

            3., 4., and 5. This article is a follow-up on the comparison of the performances on a benchmark used in a series of ISI Web of Science articles (Conferences or Journals) [26,27,31,32]. In these articles, the following items are compared as performance indicators of the control system: response time, voltage ripple, overshooting, and steady-state error. We specify that the control systems are based on controllers ranging from simple PI-type to complex SMC-type controllers. In [32], we, the authors of the article under discussion (Nicola M. and Nicola C.-I.) proposed, to our best knowledge, for the first time a cascade structure in which the inner control loop is based on a SMC-type controller, and the inner control loop is based on a Synergetic-type controller. Moreover, with such a control structure, to their best knowledge, both for the benchmark under discussion in this article and for a benchmark on the control of a PMSM in [33], the same authors obtained the best performance of the control system. Indeed, in articles [32] and [33], these control systems proposed by the authors also included the fractional order calculus, which brought extra performance by approximately 10% compared to the structure without the fractional order calculus, but together they provide peak performance of the control system. The main disadvantage of using the fractional order calculus consists in using very powerful DSPs in order to implement such control algorithms.

            Thus, this article presents the improvement of the performance of the PI or SMC type control system plus Synergetic type controllers (i.e. the simplest control systems with one of the best performance) by using an RL-TD3 agent, trained so as to provide correction signals to the primary control system. The improved performances are among those presented above and are summarized in Tables 1 and 2.

            It can be noted that, between PI-type the control systems and the PI-RL-TD3 agent, the response time is improved by approximately 7ms, and between the SMC plus Synergetic type control systems and the SMC-RL-TD3 agent and Synergetic-RL-TD3 agent, the response time is improved by approximately 1ms, i.e. a decrease of the response time by approximately 18% in the first case and by approximately 7% in the second case. However, between the simplest and most complex case, the decrease in the response time is about 27 ms, which means a decrease by about 70%. In the same way, the rest of the performances can be evaluated as relative or absolute units. Normally, “an improvement is an improvement”, regardless of the quantitative value added (which in this case is not quite negligible), and the qualitative improvement is the one which must prevail for the assessment.

Table 1. Control of the grid connected PV array system based on PI-type controllers using RL-TD3 agent performances.

Controllers for the grid connected PV system	Response time [ms]	Voltage ripple [V]	Overshooting [%]	Steady-state error [%]
PI	40.4	57.87	<0.5%	0.2%
PI using RL-TD3 agent for the correction of idref.	37.1	57.23	<0.5%	0.2%
PI using RL-TD3 agent for the correction of udref and uqref	35.9	56.67	<0.5%	0.2%
PI using RL-TD3 agent for the correction of udref, uqref, and idref	33.8	56.22	<0.5%	0.2%

 

Table 2. Control of the grid connected PV system based on SMC and Synergetic controllers using RL-TD3 agent performances.

Controllers for the grid connected PV system	Response time [ms]	Voltage ripple [V]	Overshooting [%]	Steady-state error [%]
SMC and Synergetic	14.1	55.63	<0.2%	0.02%
SMC and Synergetic using RL-TD3 agent for the correction of idref.	13.7	55.12	<0.2%	0.02%
SMC and Synergetic using RL-TD3 agent for the correction of udref and uqref	13.5	54.58	<0.2%	0.02%
SMC and Synergetic using RL-TD3 agent for the correction of udref, uqref, and idref	13.2	54.03	<0.2%	0.02%

 

 

Author Response File: Author Response.pdf

Reviewer 3 Report

This article proposes an original cascade control structure to improve the performances of the grid connected PV array system starting from a benchmark. To achieve this goal the authors start with the presentation of main characteristics of the benchmark system. Then the structure of cascade control system is described, in which PI controllers for the inner control loop of currents i_d and i_q, respectively a SMC-type controller for the outer control loop of u_dc voltage are used. Furthermore the improvement of the grid connected PV array performances are obtained by using the elements regarding the RL-TD3 agent. A Matlab/Simulink application is implemented to prove the parametric robustness of the proposed control system based on SMC and Synergetic controllers in case of variation with 30% of the three-phase load. Moreover, the results of the numerical simulations are presented comparatively and the validation of control system superiority that uses the RL-TD3 agent is demonstrated.

The article is well structured; the proposed method is clear and correct presented. The simulation results seem promising.

 

Comments to authors:

1. The format must be verified. Several examples:

• the font of equations 5,22,24,27,34 e.a.
• the references must be entered in numbering order on page 2 reference [33] appears before [32];

2. A verification of the entire text of article is required (example on page 2, line 49, e.a.)

 

Author Response

Dear reviewer, thanks for your recommendations.

We made the changes according to your requirements.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Dear Authors,

the article is 32 pages long, it is too long to propose two different types of control, looks like a book chapter. Most of the figures are of poor quality and most of them are not relevant and could be shown in a supplementary document.  The simulink models should also be in the secondary document.  It is the first time I see source code shown in an image. It is really shocking to see that. However, I leave it to the editor to decide to cut the article in half, explaining in a terse way the contribution to the scientific community by putting all the less relevant images in a supplementary document, as well as the source code and a repository with the developed simulink models. Focusing very well the context, the implementation, and the conclusions. However, I leave it to the editor to decide to cut the article in half, explaining in a terse way the contribution to the scientific community by putting all the less relevant images in a supplementary document, as well as the source code and a repository with the developed simulink models. Focusing very well the context, the implementation and the conclusions.

Author Response

Dear reviewer, thanks for your recommendations.

• The authors have a series of articles published in MDPI Energies, Electronics, and Automation, and the number of pages per article is on average of 38-40 pages;
• Short sequences of code were published in these articles and were appreciated because they help readers in the sense of clarifying specific aspects related to the implementation of the concepts presented;
• The authors are also Guest Editors for this Special Issue in which this article is presented and we insist on preserving the format and content of this article (which can be considered as a book chapter for this Special Issue);
• To summarize, this article presents the fact that the RL-TD3 agent improves the performance of the control system, both for the simplest PI-type control system, and for a system with top performance, such as the one proposed by the authors in the form of SMC and Synergetic type controllers;
• Indeed, the final decision rests with the editor.

Author Response File: Author Response.pdf