Energy Management System for an Industrial Microgrid Using Optimization Algorithms-Based Reinforcement Learning Technique
Abstract
:1. Introduction
2. Microgrid Architecture
2.1. PV System
2.2. Battery Energy Storage System
3. Energy Management System
3.1. Data Acquisition and Processing
3.2. Data Analysis and Forecasting
- is the prediction function of the Random Forest.
- B is the number of trees.
- represents a single decision tree indexed by b, which is a function of the features, X, and random parameters, .
3.3. Logic-Based Optimization
3.4. Reinforcement Learning Algorithm
- is the probability ratio of the current policy, , to the old policy, .
- is an estimator of the advantage function at timestep t.
- is a small value (e.g., 0.1 or 0.2) that defines the clipping range to keep the updates stable [35].
3.5. Grid Pricing Scheme
3.6. Simulation Approach
4. Results and Discussion
4.1. Battery Scheduling with Peak Shaving
4.2. Results Comparison from Algorithms
4.3. Economic Optimization Based on a Peak Pricing Scheme
- TD3 Algorithm:
- PPO Algorithm:
- Spot price trading: both algorithms attempt to capitalize on price differentials by charging when prices are low and discharging when prices are high.
- Peak penalty avoidance: TD3, in particular, appears to prioritize reducing peak power draw from the grid, which helps minimize the monthly peak penalty.
- Battery utilization: the algorithms must balance the costs of battery degradation against the potential savings from energy arbitrage and peak shaving.
- Long-term vs. short-term optimization: the agents must weigh immediate gains from spot price trading against long-term benefits of peak shaving.
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
Abbreviations
BESS | Battery energy storage system |
EMS | Energy management system |
RES | Renewable energy sources |
PCS | Power Conversion System |
GPC | Grid Power Controller |
PPO | Proximal Policy Optimization |
TD3 | Twin-Delayed Deep Deterministic Policy Gradient |
DR | Demand response |
MG | Microgrid |
IMG | Industrial microgrid |
PV | Photovoltaics |
ESS | Energy Storage System |
DERs | Distributed energy resources |
RL | Reinforcement learning |
EV | Electric vehicle |
IoT | Internet of Things |
API | Application Programming Interface |
TCP | Transmission Control Protocol |
References
- Department of Energy, Office of Electricity Delivery and Energy Reliability. Summary Report: 2012 DOE Microgrid Workshop. 2012. Available online: https://www.energy.gov/oe/articles/2012-doe-microgrid-workshop-summary-report-september-2012 (accessed on 24 May 2022).
- Lu, R.; Bai, R.; Ding, Y.; Wei, M.; Jiang, J.; Sun, M.; Xiao, F.; Zhang, H.T. A hybrid deep learning-based online energy management scheme for industrial microgrid. Appl. Energy 2021, 304, 117857. [Google Scholar] [CrossRef]
- Wang, C.; Yan, J.; Marnay, C.; Djilali, N.; Dahlquist, E.; Wu, J.; Jia, H. Distributed Energy and Microgrids (DEM). Appl. Energy 2018, 210, 685–689. [Google Scholar] [CrossRef]
- Brem, A.; Adrita, M.M.; O’Sullivan, D.T.; Bruton, K. Industrial smart and micro grid systems—A systematic mapping study. J. Clean. Prod. 2020, 244, 118828. [Google Scholar] [CrossRef]
- Mehta, R. A microgrid case study for ensuring reliable power for commercial and industrial sites. In Proceedings of the 2019 IEEE PES GTD Grand International Conference and Exposition Asia (GTD Asia), Bangkok, Thailand, 19–23 March 2019; pp. 594–598. [Google Scholar]
- Roslan, M.; Hannan, M.; Ker, P.J.; Begum, R.; Mahlia, T.I.; Dong, Z. Scheduling controller for microgrids energy management system using optimization algorithm in achieving cost saving and emission reduction. Appl. Energy 2021, 292, 116883. [Google Scholar] [CrossRef]
- Roslan, M.; Hannan, M.; Ker, P.J.; Uddin, M. Microgrid control methods toward achieving sustainable energy management. Appl. Energy 2019, 240, 583–607. [Google Scholar] [CrossRef]
- Pourmousavi, S.A.; Nehrir, M.H.; Colson, C.M.; Wang, C. Real-time energy management of a stand-alone hybrid wind-microturbine energy system using particle swarm optimization. IEEE Trans. Sustain. Energy 2010, 1, 193–201. [Google Scholar] [CrossRef]
- Marzband, M.; Sumper, A.; Ruiz-Alvarez, A.; Domínguez-García, J.L.; Tomoiagă, B. Experimental evaluation of a real time energy management system for stand-alone microgrids in day-ahead markets. Appl. Energy 2013, 106, 365–376. [Google Scholar] [CrossRef]
- Choobineh, M.; Mohagheghi, S. A multi-objective optimization framework for energy and asset management in an industrial Microgrid. J. Clean. Prod. 2016, 139, 1326–1338. [Google Scholar] [CrossRef]
- Ding, Y.M.; Hong, S.H.; Li, X.H. A demand response energy management scheme for industrial facilities in smart grid. IEEE Trans. Ind. Inform. 2014, 10, 2257–2269. [Google Scholar] [CrossRef]
- Gholian, A.; Mohsenian-Rad, H.; Hua, Y. Optimal industrial load control in smart grid. IEEE Trans. Smart Grid 2015, 7, 2305–2316. [Google Scholar] [CrossRef]
- Huang, X.; Hong, S.H.; Li, Y. Hour-ahead price based energy management scheme for industrial facilities. IEEE Trans. Ind. Inform. 2017, 13, 2886–2898. [Google Scholar] [CrossRef]
- Youssef, T.A.; El Hariri, M.; Elsayed, A.T.; Mohammed, O.A. A DDS-based energy management framework for small microgrid operation and control. IEEE Trans. Ind. Inform. 2017, 14, 958–968. [Google Scholar] [CrossRef]
- Gutiérrez-Oliva, D.; Colmenar-Santos, A.; Rosales-Asensio, E. A review of the state of the art of industrial microgrids based on renewable energy. Electronics 2022, 11, 1002. [Google Scholar] [CrossRef]
- Correia, A.F.; Moura, P.; de Almeida, A.T. Technical and economic assessment of battery storage and vehicle-to-grid systems in building microgrids. Energies 2022, 15, 8905. [Google Scholar] [CrossRef]
- Hussain, A.; Bui, V.H.; Kim, H.M. Microgrids as a resilience resource and strategies used by microgrids for enhancing resilience. Appl. Energy 2019, 240, 56–72. [Google Scholar] [CrossRef]
- Sutton, R.S.; Barto, A.G. Reinforcement Learning: An Introduction; MIT Press: Cambridge, MA, USA, 2018. [Google Scholar]
- Arwa, E.O.; Folly, K.A. Reinforcement learning techniques for optimal power control in grid-connected microgrids: A comprehensive review. IEEE Access 2020, 8, 208992–209007. [Google Scholar] [CrossRef]
- Mughees, N.; Jaffery, M.H.; Mughees, A.; Ansari, E.A.; Mughees, A. Reinforcement learning-based composite differential evolution for integrated demand response scheme in industrial microgrids. Appl. Energy 2023, 342, 121150. [Google Scholar] [CrossRef]
- François-Lavet, V.; Taralla, D.; Ernst, D.; Fonteneau, R. Deep reinforcement learning solutions for energy microgrids management. In Proceedings of the European Workshop on Reinforcement Learning (EWRL 2016), Barcelona, Spain, 3–4 December 2016. [Google Scholar]
- Chen, P.; Liu, M.; Chen, C.; Shang, X. A battery management strategy in microgrid for personalized customer requirements. Energy 2019, 189, 116245. [Google Scholar] [CrossRef]
- Nakabi, T.A.; Toivanen, P. Deep reinforcement learning for energy management in a microgrid with flexible demand. Sustain. Energy Grids Netw. 2021, 25, 100413. [Google Scholar] [CrossRef]
- Ji, Y.; Wang, J.; Xu, J.; Fang, X.; Zhang, H. Real-time energy management of a microgrid using deep reinforcement learning. Energies 2019, 12, 2291. [Google Scholar] [CrossRef]
- Lee, S.; Seon, J.; Sun, Y.G.; Kim, S.H.; Kyeong, C.; Kim, D.I.; Kim, J.Y. Novel architecture of energy management systems based on deep reinforcement learning in microgrid. IEEE Trans. Smart Grid 2023, 15, 1646–1658. [Google Scholar] [CrossRef]
- Ahmed, I.; Pedersen, A.; Mihet-Popa, L. Smart Microgrid Optimization using Deep Reinforcement Learning by utilizing the Energy Storage Systems. In Proceedings of the 2024 4th International Conference on Smart Grid and Renewable Energy (SGRE), Doha, Qatar, 8–10 January 2024; pp. 1–7. [Google Scholar]
- ProSoft Technology. Introduction to Modbus TCP/IP; Acromag, Inc.: Wixom, MI, USA, 2024. [Google Scholar]
- EEM-MA771—Measuring Instrument. 2024. Available online: https://www.phoenixcontact.com/en-no/products/measuring-instrument-eem-ma771-2908286 (accessed on 10 March 2024).
- Hva Koster Strommen. What Does Strømmen.no Cost? 2024. Available online: https://www.hvakosterstrommen.no/ (accessed on 15 March 2024).
- ENTSOE. Entso-e Transparency Platform. 2024. Available online: https://transparency.entsoe.eu/ (accessed on 13 March 2024).
- RandomForestRegressor. Available online: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html (accessed on 30 May 2024).
- Breiman, L. Random forests. Mach. Learn. 2001, 45, 5–32. [Google Scholar] [CrossRef]
- Amazon Web Services. What Is Reinforcement Learning? Available online: https://aws.amazon.com/what-is/reinforcement-learning/ (accessed on 30 May 2024).
- OpenAI. Proximal Policy Optimization. Available online: https://spinningup.openai.com/en/latest/algorithms/ppo.html (accessed on 30 May 2024).
- Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; Klimov, O. Proximal policy optimization algorithms. arXiv 2017, arXiv:1707.06347. [Google Scholar]
- Nordpool Market Data. Available online: https://www.nordpoolgroupqa.com/en/trading/Market-data1/Intraday/Market-data1/Market-data1/Overview/ (accessed on 10 January 2023).
- Fujimoto, S.; Hoof, H.; Meger, D. Addressing function approximation error in actor-critic methods. In Proceedings of the International Conference on Machine Learning (PMLR), Stockholm, Sweden, 10–15 July 2018; pp. 1587–1596. [Google Scholar]
Parameters | Values |
---|---|
PV Generator Output | 200.88 kWp |
PV Generator Surface | 1059.6 m² |
Number of PV Modules | 648 |
Number of Inverters | 3 |
PV Module Used | JAM60S01-310/PR |
Speculated Annual Yield | 87,594 kWh/kWp |
Parameters | Values |
---|---|
Battery Type | LPF Lithium-ion |
Battery Capacity | 1105 kWh |
Rated Battery Voltage | 768 Vdc |
Battery Voltage Range | 672–852 Vdc |
Max. Charge/Discharge Current | 186 A |
Max. Charge/Discharge Power | 1000 kW |
Parameters | Values |
---|---|
Rated Voltage | 400 V (L-L) |
Rated Frequency | 50/60 Hz |
AC Connection | 3 W + N |
Rated Power | 2 × 500 kW |
Rated Current Imax | 2 × 721.7 A |
Power Factor | 0.8–1 (leading or lagging, load-dependent) |
Peak hour pricing scheme (taken from the highest peak in the month) |
Winter: November–March (84 NOK/kW/month) |
Summer: April–October (35 NOK/kW/month) |
Peak hour pricing scheme for reactive power (taken from the highest peak in the month) |
Winter: November–March (35 NOK/kVAr/month) |
Summer: April–October (15 NOK/kW/month) |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Upadhyay, S.; Ahmed, I.; Mihet-Popa, L. Energy Management System for an Industrial Microgrid Using Optimization Algorithms-Based Reinforcement Learning Technique. Energies 2024, 17, 3898. https://doi.org/10.3390/en17163898
Upadhyay S, Ahmed I, Mihet-Popa L. Energy Management System for an Industrial Microgrid Using Optimization Algorithms-Based Reinforcement Learning Technique. Energies. 2024; 17(16):3898. https://doi.org/10.3390/en17163898
Chicago/Turabian StyleUpadhyay, Saugat, Ibrahim Ahmed, and Lucian Mihet-Popa. 2024. "Energy Management System for an Industrial Microgrid Using Optimization Algorithms-Based Reinforcement Learning Technique" Energies 17, no. 16: 3898. https://doi.org/10.3390/en17163898