Resilient Reinforcement Learning for Voltage Control in an Islanded DC Microgrid Integrating Data-Driven Piezoelectric

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Dear authors,

Regarding the presented article, my main concern is that it seems the authors got lost between the article's proposal and what was presented during its development. The topic of the paper is very broad and quite complex to be addressed in a single article, as done here.

Some points related to hybrid microgrids that the authors did not address or, if they did, were addressed very poorly:

1) hybrid microgrids: hybrid microgrids refer to AC and DC systems, while in your work, "hybrid" refers to energy sources, so the concept was improperly applied;
2) parallel operation of converters: when it comes to microgrids, one issue is the parallel operation of converters. In the proposed scheme, there are three DC-DC converters, and the authors did not even mention how they operate (whether there is central, decentralized, or distributed control, load sharing, communication network, hierarchical control, etc.);
3) Converter efficiency: Since the authors devoted much space in the paper to piezoelectric energy harvesting, it lacked discussion on efficiency, power levels, and at what power levels it would be feasible to use this device;
4) Energy storage systems: There was no discussion on how the battery charging will be done, under what conditions, and how the state of charge control will be managed;
5) Design of PI compensators: Given that this is a comparison with this classical method, a section should have been dedicated to the design of these compensators;
6) Experiments: The authors did not clarify the microgrid characteristics (power levels, main bus voltage, converter values, etc.).

From my point of view and considering the initial proposal, the content of the paper begins in section 4. In its current form, the article is quite poor; however, the proposal is interesting, and I liked it. I believe that by revising the article and considering the following:

a) modeling of the converters (as they are well-known converters, cite papers that already offer a deeper modeling);
b) I do not see how the microgrid fits into this work. To me, they are just parallel converters to supply the load, so I suggest abandoning the microgrid concept;
c) It also does not make sense to focus on piezoelectric energy harvesting, as the proposal is about converter control;
d) Present more deeply how the proposed control was implemented in the converters, considering that they operate in parallel (Will you use communication? How does one converter "see" the other? Or, despite there being more than one converter, is everything done on the same microcontroller?).

Apologies for the lengthy comment, but I believe that a good revision of the article would raise it to another level.

Comments on the Quality of English Language

Correct a few spelling and spacing errors between cross-references and the end of words.

Author Response

Regarding the presented article, my main concern is that it seems the authors got lost between the article's proposal and what was presented during its development. The topic of the paper is very broad and quite complex to be addressed in a single article, as done here.

Authors’ Response:

Thank you for taking the time to review our manuscript. We acknowledge that the topic of voltage control in microgrids is indeed broad and complex; however, our intend was to provide a comprehensive overview while focusing specifically on the integration of piezoelectric harvesting within a microgrid and how to precisely control the voltage in such microgrids.

Some points related to hybrid microgrids that the authors did not address or, if they did, were addressed very poorly:

Authors’ Response:

Please see our responses below. All text changes are highlighted in the paper. Please accept our sincere appreciation for an outstanding review!

1) hybrid microgrids: hybrid microgrids refer to AC and DC systems, while in your work, "hybrid" refers to energy sources, so the concept was improperly applied;

Authors’ Response:

Thank you for pointing out the misuse of the term "hybrid microgrid" in our manuscript. After further review of existing literature, we found out that “hybrid microgrid” usually refers to systems where both AC and DC regulated buses exist. In our study, while we do integrate both AC (piezoelectric) and DC (solar and battery) energy sources, our configuration includes only a regulated DC bus without a separate regulated AC bus. To reflect the nature of our system, we removed the term hybrid in the revised manuscript. 

2) parallel operation of converters: when it comes to microgrids, one issue is the parallel operation of converters. In the proposed scheme, there are three DC-DC converters, and the authors did not even mention how they operate (whether there is central, decentralized, or distributed control, load sharing, communication network, hierarchical control, etc.);

Authors’ Response:

Thank you for the comment. In our microgrid, the focus is on the voltage control which is the first (local) control layer in the context of “hierarchical control”. Load sharing is managed under the secondary control which is not within the paper’s scope. In this study, we used a “decentralized” control scheme where each converter is responsible for managing it own energy source (solar PV, battery storage, and piezoelectric energy harvesting), independently. While our current implementation focuses on decentralized control, we recognize the potential benefits of hierarchical control, and load sharing for future scalability and integration with larger grid systems. We have added a discussion on how hierarchical control could be implemented in future iterations of our system to enhance coordination between converters and improve overall system resilience.

6. What is Next:

Distributed control and load sharing have the potential to greatly improve system resilience and operational efficiency in the context of islanded DC microgrids while they will be investigated in connection and disconnection from the main grid. Each power source, including photovoltaic, battery, and piezoelectric modules, can function independently through distributed control, responding to local conditions and reducing communication latency. This enables decentralized decision-making. By distributing the load evenly among different sources, load sharing avoids overloading and makes the best use of the available energy resources. This method increases fault tolerance, optimizes energy flow, and strengthens system stability in situations like unexpected load fluctuations or power supply failures.

3) Converter efficiency: Since the authors devoted much space in the paper to piezoelectric energy harvesting, it lacked discussion on efficiency, power levels, and at what power levels it would be feasible to use this device;

Authors’ Response:

Thank you for the insightful comment. We have added explanations on the converters’ efficiency as well as power ratings under each converter in the revised manuscript.

The system's DC-DC boost converter can function between 85% and 95% efficiently [16], depending on the load and switching frequency. The module for harvesting piezoelectric energy can generate up to 13.5V and 0.8A when working optimally; however, to effectively raise the gathered voltage to the required bus voltage for the system, a boost converter is needed. To fully utilize the energy that has been captured, even tiny power contributions from the piezoelectric source must be successfully incorporated into the microgrid through this high-efficiency conversion. A deeper modeling of the converter can be found in [17].

The PV system is equipped with a buck converter that can withstand variations in irradiance while preserving a high conversion efficiency of up to 90 %. It runs at a nominal voltage of 48V. A power switch (Q), switching controller, diode (D), inductor (L), capacitor (C), and DC load bus make up a DC-DC buck converter as shown in Figure 12. To get a lower output voltage, it steps down an input voltage. In this simulation, the input voltage is 48 V with 1000 irradiance. With the controller's assistance, the buck converter lowers the voltage to the load bus. Because the inductor stores energy when the switch is on and releases it to the load when it is off, smoothing the output, the output voltage grows linearly with the duty cycle. Performance can be impacted by non-ideal elements like as switching losses and inductor resistance  . [18] contains further modeling details.

A battery system's internal resistance, charging and discharging rates, and temperature all affect how efficient it is. For lithium-ion batteries, round-trip efficiency, or the ratio of energy production during discharge to energy input while charging, usually varies from 85% to 95%, contingent upon the operating circumstances and application.

The study skips over the technical details of SoC control and battery charging. Nonetheless, bidirectional DC-DC converters are usually employed in microgrid systems to manage battery charging and discharging. Algorithms such as constant current/constant voltage (CC/CV) could be used to provide charge control and guarantee ideal charging. The SoC, which is crucial for preserving battery health, is frequently represented mathematically using voltage thresholds and current integration. The following equation can be used to describe how quickly a battery charges:

(5)

where   is the charging current,   is the input power from the source (e.g., PV, piezoelectric, etc.), and   is the battery voltage. A battery's State of Charge (SoC) can be represented as follows:

(6)

where   is the battery capacity (in amp-hours, Ah),    is the charging or discharging current, and   is the initial state of charge at time . This equation integrates the current over time to determine the change in the state of charge. More modeling information about battery charging and discharging is provided in [20]. The converter parameters and power ratings of the planned DC MG are displayed in TABLE 1 along with its configuration.

4) Energy storage systems: There was no discussion on how the battery charging will be done, under what conditions, and how the state of charge control will be managed;

Authors’ Response:

Thank you for your feedback regarding battery charging and state of charge (SOC) control in our manuscript. The battery charging and SOC control have been discussed in response to the previous comment.

The study skips over the technical details of SoC control and battery charging. Nonetheless, bidirectional DC-DC converters are usually employed in microgrid systems to manage battery charging and discharging. Algorithms such as constant current/constant voltage (CC/CV) could be used to provide charge control and guarantee ideal charging. The SoC, which is crucial for preserving battery health, is frequently represented mathematically using voltage thresholds and current integration. The following equation can be used to describe how quickly a battery charges:

where   is the charging current,   is the input power from the source (e.g., PV, piezoelectric, etc.), and   is the battery voltage. A battery's State of Charge (SoC) can be represented as follows:

where   is the battery capacity (in amp-hours, Ah),    is the charging or discharging current, and   is the initial state of charge at time . This equation integrates the current over time to determine the change in the state of charge. More modeling information about battery charging and discharging is provided in [20].

Many battery management systems and industry standards recommend maintaining SOC within 10% to 90% limits to ensure a balance between performance, safety, and longevity. This practice is common in various applications, including electric vehicles and grid energy storage systems.

5) Design of PI compensators: Given that this is a comparison with this classical method, a section should have been dedicated to the design of these compensators;

Authors’ Response:

Thank you for highlighting the need for a more detailed discussion on the design of PI controller. We have added a paragraph, in the revised manuscript, under Results and Discussions to further discuss tuning of the PI controller.

Tests were conducted using a short-circuit fault state across the load on both the PI which has been tuned using the trial-and-error technique involves adjusting the proportional and integral gains based on system performance to minimize steady-state error and achieve desired response characteristics and the suggested technique.

6) Experiments: The authors did not clarify the microgrid characteristics (power levels, main bus voltage, converter values, etc.).

Authors’ Response:

Thank you for the comment. We have added a table in the revised manuscript to report the microgrid specifications.

7) From my point of view and considering the initial proposal, the content of the paper begins in section 4. In its current form, the article is quite poor; however, the proposal is interesting, and I liked it. I believe that by revising the article and considering the following:

Authors’ Response:

Thank you for your detailed and insightful comments. We appreciate your interest in our proposal and your constructive feedback, which will help us improve the quality of our manuscript. We aim to address all your concerns and improve the clarity of the paper.

1. a) modeling of the converters (as they are well-known converters, cite papers that already offer a deeper modeling);

Authors’ Response:

We appreciate your comment. We have shortened the section explaining modeling of converters by referring to the existing literature.

1. b) I do not see how the microgrid fits into this work. To me, they are just parallel converters to supply the load, so I suggest abandoning the microgrid concept;

Authors’ Response:

We appreciate your feedback. A microgrid typically consists of distributed energy resources and loads that can operate in parallel with or independently from the main grid. Our proposed system can be considered a microgrid since1) it includes multiple energy resources, 2) the system can be operated in the islanded mode, 3) the system involves advanced control schemes, and 4) the application (smart road) supports the notation of a microgrid, as it serves localized energy needs independently from the main grid.

1. c) It also does not make sense to focus on piezoelectric energy harvesting, as the proposal is about converter control;

Authors’ Response:

We appreciate your observation with respect to the focus on piezoelectric energy harvesting in our manuscript. The emphasis on this aspect is actually intentional and in alignment with the goals of our project, which was sponsored by the Louisiana Department of Transportation and Development (DOTD) and the Louisiana Transportation Research Center (LTRC). To further clarify the scope of work, we have added the following section in the revised manuscript:

The goal is to design and test a sustainable energy hub, with part of the energy sourced from piezo sensors embedded in asphalt or concrete. Due to the uncertainties associated with piezo harvesters, we propose and test a robust AI-driven controller that can handle these uncertainties and unexpected disturbances more effectively than traditional controllers.

1. d) Present more deeply how the proposed control was implemented in the converters, considering that they operate in parallel (Will you use communication? How does one converter "see" the other? Or, despite there being more than one converter, is everything done on the same microcontroller?).

Authors’ Response:

The control strategy presented in this study is centered on converter voltage regulation, which is crucial for preserving stability and controlling power flow in the islanded DC microgrid. It guarantees that voltage levels at every node stay within allowable bounds as a type of primary control, even in the event of variations in load or power generation from sources such as piezoelectric (PE), photovoltaic (PV), and battery systems. Voltage control serves as the primary control for ensuring stability in the microgrid. Regardless of changes in power output or load conditions, it keeps the DC bus voltage constant. Each converter dynamically adjusts its duty cycle to regulate the output voltage, either autonomously in a distributed control architecture or through global commands in a centralized system.

Apologies for the lengthy comment, but I believe that a good revision of the article would raise it to another level.

Authors’ Response:

Thank you for your thoughtful and comprehensive feedback. We are grateful for your insights and are committed to revising the manuscript to elevate its quality. Your comments have provided us with a clear path for improvement.

Correct a few spelling and spacing errors between cross-references and the end of words.

Thank you for pointing that out. The errors have been fixed in the revised manuscript.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

In this paper, the Authors suggest a novel control scheme for a hybrid DC microgrid (HDCMG) integrating solar photovoltaic (PV), battery storage (BESS), and piezoelectric (PE) energy harvesting modules in order to provide electricity for lighting systems in transportation, roads, and other infrastructure. Apart from the control scheme, the Authors assess their proposal against a conventional PI controller in four scenarios. The results of their simulation section are promising. Only small changes are required especially in the Section of conclusions so that the paper can be acceptable for publication. Indeed, it is an interesting paper for the potential reader. In accordance with the Authors, the further assessment of the economics, reliability, and durability of piezoelectric modules that could enhance their viability in practical applications can be a green solution for small municipalities and cities. At the moment, the paper is under minor revision and the Authors could take into account the following Reviewer’s remarks:

Section 1, Introduction: Are there in the literature piezoelectric applications that are in an operation mode in transportation, roads, and other infrastructure? Except for California’s study, please enrich your introduction with some relevant applications.

Section 2.0.1 and Figure 3: Very interesting piece of research and experimental implementation.

Figure 2: The first results of the experimental implementation. The voltage results of the concrete samples concerning the peak voltages (0.5V and 0.14V) can support the concept. In heavy traffic roads of big cities, such implementations can contribute a lot in their lighting systems and the municipals’ power needs.

Section 3: In this Section, the Authors demonstrate hybrid DC microgrid configurations and setup. Apart from the schematics for the hybrid DC microgrid and its components, a proper mathematical approach has been given. Here, a more detailed mathematic analysis could be given but it is OK since the paper mainly focuses on the operation rather than its theory.

Section 4 analyzes the controller and the algorithm that lies behind it. OK

Section 5: The results are based on the issues of short circuit across the load, converter failure, load variation and open circuit of the load. Reviewer is satisfied with the applied metrics, figures and Table 2 that synopsizes the findings and support the conclusions. The contribution of the paper is clear and justified in this Section. Since the analysis that explains the findings of voltage response figures is adequate and the control metrics comparison figures is acceptable (in the second type of figures, the Authors could adopt a different type of figures for presentation reasons), no changes are required here.

Conclusion Section: The Authors mention “Results show the effectiveness of the proposed controller in improving the resilience of the energy hub under test.” in line 381. They should add some of their findings of Section 5.

References: A good and recent list of references. No changes are required.

Author Response

In this paper, the Authors suggest a novel control scheme for a hybrid DC microgrid (HDCMG) integrating solar photovoltaic (PV), battery storage (BESS), and piezoelectric (PE) energy harvesting modules in order to provide electricity for lighting systems in transportation, roads, and other infrastructure. Apart from the control scheme, the Authors assess their proposal against a conventional PI controller in four scenarios. The results of their simulation section are promising. Only small changes are required especially in the Section of conclusions so that the paper can be acceptable for publication. Indeed, it is an interesting paper for the potential reader. In accordance with the Authors, the further assessment of the economics, reliability, and durability of piezoelectric modules that could enhance their viability in practical applications can be a green solution for small municipalities and cities. At the moment, the paper is under minor revision and the Authors could take into account the following Reviewer’s remarks:

Authors’ Response:

Thank you very much for taking the time to review our manuscript.

Section 1, Introduction: Are there in the literature piezoelectric applications that are in an operation mode in transportation, roads, and other infrastructure? Except for California’s study, please enrich your introduction with some relevant applications.

Authors’ Response:

Thank you for the comment. We have added two more references about how to capture piezoelectric energy from vehicles and pedestrians.

Authors in [9] present a dual-layer substrate piezoelectric transducer for harvesting energy from vehicle vibrations in asphalt, demonstrating that a finite element model predicts 132 V output and 3.56 mW power at 20 Hz with reduced stress concentration, enhancing durability and efficiency. To provide a workable harvested energy configuration for pedestrian deployment, the study in [10] highlighted the various strain elements that are impacted by the energy produced by a PZT (Lead Zirconate Titanate) energy-harvesting floor tile (EHFT).

Section 2.0.1 and Figure 3: Very interesting piece of research and experimental implementation.

Authors’ Response:

Thank you for your comment.  

Figure 2: The first results of the experimental implementation. The voltage results of the concrete samples concerning the peak voltages (0.5V and 0.14V) can support the concept. In heavy traffic roads of big cities, such implementations can contribute a lot in their lighting systems and the municipals’ power needs.

Authors’ Response:

Thank you for the positive observation regarding the impact of our experiments with piezo energy harvesters. We agree that implementations such as these could significantly contribute to the lighting systems and power needs of municipalities, especially in high-traffic urban areas.

Section 3: In this Section, the Authors demonstrate hybrid DC microgrid configurations and setup. Apart from the schematics for the hybrid DC microgrid and its components, a proper mathematical approach has been given. Here, a more detailed mathematic analysis could be given but it is OK since the paper mainly focuses on the operation rather than its theory.

Authors’ Response:

We appreciate your comment. In order to provide more theoretical background behind the operation of converters without making the manuscript overlength, we have included references in the revised manuscript for readers who are willing to know more about the mathematical approach.

Section 4 analyzes the controller and the algorithm that lies behind it. OK

Authors’ Response:

Thank you for your comment.

Section 5: The results are based on the issues of short circuit across the load, converter failure, load variation and open circuit of the load. Reviewer is satisfied with the applied metrics, figures and Table 2 that synopsizes the findings and support the conclusions. The contribution of the paper is clear and justified in this Section. Since the analysis that explains the findings of voltage response figures is adequate and the control metrics comparison figures is acceptable (in the second type of figures, the Authors could adopt a different type of figures for presentation reasons), no changes are required here.

Authors’ Response:

Thank you for your comment.

Conclusion Section: The Authors mention “Results show the effectiveness of the proposed controller in improving the resilience of the energy hub under test.” in line 381. They should add some of their findings of Section 5.

Authors’ Response:

Thank you for the comment. To highlight the findings, we have revised the conclusion by adding the following section that quantitatively compares our proposed control scheme with a traditional PI controller.

In various fault scenarios, the RL controller consistently demonstrates superior performance compared to the PI controller. In the short circuit across the load SCL scenario, the RL controller achieves a significantly lower overshoot of 0.63% compared to 6.82% with the PI controller. The undershoot is also reduced to 1.07% from 12.28%, and the peak-to-peak value is drastically lower at 40.62 compared to 458.3. For the open circuit load OCL scenario, the RL controller maintains better stability with an overshoot of 0.73% versus 1.51% and an undershoot of 2.83% compared to 5.33% for the PI controller. In the converter failure CF scenario, although both controllers perform closely, the RL controller still shows improved performance with a peak-to-peak value of 115.54 compared to 142.16. During load variation, the RL controller exhibits a much lower peak-to-peak value of 25.03 compared to 253.69 for the PI controller, demonstrating its superior ability to handle dynamic changes.

References: A good and recent list of references. No changes are required.

Authors’ Response:

Thank you for your comment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Dear authors, significant changes have been made to the work, there are still some corrections that need to be made, however there is no need to send it for a new review.

In Fig. 13, that topology is not buck-boost, although in some references, the correct buck-boost topology is this one

that topology has no specific name, I suggest calling it "bidirectional buck-boost", "buck-&-boost" or something similar, but not "buck-boost".

In Fig. 14, the converter in use does not invert the voltage polarity, I believe that it was when inserting the voltmeter that it was inverted, reversing the waveform.

Author Response

Dear authors, significant changes have been made to the work, there are still some corrections that need to be made, however there is no need to send it for a new review.

Authors’ Response:

Thank you for taking the time to review our responses and the revised manuscript. Please see our responses below. All text changes are highlighted in the paper.

1) In Fig. 13, that topology is not buck-boost, although in some references, the correct buck-boost topology is this one, that topology has no specific name, I suggest calling it "bidirectional buck-boost", "buck-&-boost" or something similar, but not "buck-boost".

Authors’ Response:

Thank you for your comment. We changed the name to “Bidirectional Buck-boost converter” as suggested by the reviewer.   

2) In Fig. 14, the converter in use does not invert the voltage polarity, I believe that it was when inserting the voltmeter that it was inverted, reversing the waveform.

Authors’ Response:

Thank you for the comment. In order to make the figure more clear and avoid change of polarity for the voltage, we revised the figure by splitting the modes (buck and boost) and change the  to the output voltage.

Author Response File: Author Response.pdf