Deep Reinforcement Learning-Based Operation of Transmission Battery Storage with Dynamic Thermal Line Rating
Abstract
:1. Introduction
2. Background and Related Work
2.1. BESS Capacity Sizing
2.2. Deep Reinforcement Learning
2.2.1. DDPG and MADDPG
2.2.2. SAC and MASAC
2.2.3. PINNs
2.3. Load and Ampacity Forecasting
2.4. Related Work
3. Problem Formulation
3.1. Single-Agent Setting
3.2. Multi-Agent Setting
4. Experimental Setup
5. Results and Discussion
6. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Hussain, A.; Bui, V.H.; Kim, H.M. Deep reinforcement learning-based operation of fast charging stations coupled with energy storage system. Electr. Power Syst. Res. 2022, 210, 108087. [Google Scholar] [CrossRef]
- Cao, D.; Hu, W.; Zhao, J.; Zhang, G.; Zhang, B.; Liu, Z.; Chen, Z.; Blaabjerg, F. Reinforcement learning and its applications in modern power and energy systems: A review. J. Mod. Power Syst. Clean Energy 2020, 8, 1029–1042. [Google Scholar] [CrossRef]
- Zhang, B.; Hu, W.; Cao, D.; Li, T.; Zhang, Z.; Chen, Z.; Blaabjerg, F. Soft actor–critic–based multi-objective optimized energy conversion and management strategy for integrated energy systems with renewable energy. Energy Convers. Manag. 2021, 243, 114381. [Google Scholar] [CrossRef]
- Khalid, J.; Ramli, M.A.; Khan, M.S.; Hidayat, T. Efficient Load Frequency Control of Renewable Integrated Power System: A Twin Delayed DDPG-Based Deep Reinforcement Learning Approach. IEEE Access 2022, 10, 51561–51574. [Google Scholar] [CrossRef]
- Zhang, S.; May, D.; Gül, M.; Musilek, P. Reinforcement learning-driven local transactive energy market for distributed energy resources. Energy AI 2022, 8, 100150. [Google Scholar] [CrossRef]
- Lillicrap, T.P.; Hunt, J.J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa, Y.; Silver, D.; Wierstra, D. Continuous control with deep reinforcement learning. arXiv 2015, arXiv:1509.02971. [Google Scholar]
- Lowe, R.; Wu, Y.I.; Tamar, A.; Harb, J.; Pieter Abbeel, O.; Mordatch, I. Multi-agent actor–critic for mixed cooperative-competitive environments. Adv. Neural Inf. Process. Syst. (NIPS) 2017, 30, 1–12. [Google Scholar]
- Haarnoja, T.; Zhou, A.; Abbeel, P.; Levine, S. Soft actor–critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning, PMLR, Stockholm, Sweden, 10–15 July 2018; pp. 1861–1870. [Google Scholar]
- Cao, D.; Zhao, J.; Hu, W.; Ding, F.; Huang, Q.; Chen, Z.; Blaabjerg, F. Data-driven multi-agent deep reinforcement learning for distribution system decentralized voltage control with high penetration of PVs. IEEE Trans. Smart Grid 2021, 12, 4137–4150. [Google Scholar] [CrossRef]
- Cao, D.; Hu, W.; Zhao, J.; Huang, Q.; Chen, Z.; Blaabjerg, F. A multi-agent deep reinforcement learning based voltage regulation using coordinated PV inverters. IEEE Trans. Power Syst. 2020, 35, 4120–4123. [Google Scholar] [CrossRef]
- Barton, T.; Musilek, M.; Musilek, P. The Effect of Temporal Discretization on Dynamic Thermal Line Rating. In Proceedings of the 2020 21st International Scientific Conference on Electric Power Engineering (EPE), Prague, Czech Republic, 19–21 October 2020; pp. 1–6. [Google Scholar]
- Karimi, S.; Musilek, P.; Knight, A.M. Dynamic thermal rating of transmission lines: A review. Renew. Sustain. Energy Rev. 2018, 91, 600–612. [Google Scholar] [CrossRef]
- Faheem, M.; Shah, S.B.H.; Butt, R.A.; Raza, B.; Anwar, M.; Ashraf, M.W.; Ngadi, M.A.; Gungor, V.C. Smart grid communication and information technologies in the perspective of Industry 4.0: Opportunities and challenges. Comput. Sci. Rev. 2018, 30, 1–30. [Google Scholar] [CrossRef]
- Hussain, A.; Bui, V.H.; Kim, H.M. Impact analysis of demand response intensity and energy storage size on operation of networked microgrids. Energies 2017, 10, 882. [Google Scholar] [CrossRef] [Green Version]
- Wood, A.J.; Wollenberg, B.F.; Sheblé, G.B. Power Generation, Operation, and Control; John Wiley & Sons: New York, NY, USA, 2013. [Google Scholar]
- British Atmospheric Data Center (BADC). Available online: https://data.ceda.ac.uk/badc/ukmo-midas-open/data/ (accessed on 31 August 2022).
- Hesse, H.C.; Martins, R.; Musilek, P.; Naumann, M.; Truong, C.N.; Jossen, A. Economic optimization of component sizing for residential battery storage systems. Energies 2017, 10, 835. [Google Scholar] [CrossRef]
- Fujimoto, S.; Hoof, H.; Meger, D. Addressing function approximation error in actor–critic methods. In Proceedings of the International Conference on Machine Learning, PMLR, Stockholm, Sweden, 10–15 July 2018; pp. 1587–1596. [Google Scholar]
- Misyris, G.S.; Venzke, A.; Chatzivasileiadis, S. Physics-informed neural networks for power systems. In Proceedings of the 2020 IEEE Power & Energy Society General Meeting (PESGM), Virtual Event, 3–6 August 2020; pp. 1–5. [Google Scholar]
- Wang, Z.; Jia, L.; Ren, C. Attention-Bidirectional LSTM Based Short Term Power Load Forecasting. In Proceedings of the 2021 Power System and Green Energy Conference (PSGEC), Shanghai, China, 20–22 August 2021; pp. 171–175. [Google Scholar]
- Yan, Z.; Xu, Y.; Wang, Y.; Feng, X. Data-driven economic control of battery energy storage system considering battery degradation. In Proceedings of the 2019 9th International Conference on Power and Energy Systems (ICPES), Perth, Australia, 10–12 December 2019; pp. 1–5. [Google Scholar]
- Zhang, S.; Nandakumar, S.; Pan, Q.; Yang, E.; Migne, R.; Subramanian, L. Benchmarking Reinforcement Learning Algorithms on Island Microgrid Energy Management. In Proceedings of the 2021 IEEE PES Innovative Smart Grid Technologies-Asia (ISGT Asia), Brisbane, Australia, 5–8 December 2021; pp. 1–5. [Google Scholar]
- Bui, V.H.; Hussain, A.; Kim, H.M. Double deep Q-learning-based distributed operation of battery energy storage system considering uncertainties. IEEE Trans. Smart Grid 2019, 11, 457–469. [Google Scholar] [CrossRef]
- Metwaly, M.K.; Teh, J. Probabilistic Peak Demand Matching by Battery Energy Storage Alongside Dynamic Thermal Ratings and Demand Response for Enhanced Network Reliability. IEEE Access 2020, 8, 181547–181559. [Google Scholar] [CrossRef]
- IEEE Std 738-2012; Standard for Calculating the Current-Temperature Relationship of Bare Overhead Conductors. IEEE: Piscataway, NJ, USA, 2013; pp. 1–72. [CrossRef]
- Probability Methods Subcommittee. IEEE reliability test system. IEEE Trans. Power Appar. Syst. 1979, PAS-98, 2047–2054. [Google Scholar] [CrossRef]
Study | Algorithm | Control Area | Problem | Objective |
---|---|---|---|---|
[3] | SAC, DDPG, DDQN | Microgrid | Power plant control | Cost minimization |
[4] | TD3, DDPG | 2 area control | PID control | Area control error minimization |
[21] | SAC, DDPG, DDQN | 3 area control | Frequency control | Generation cost, battery aging cost, unscheduled interchange cost |
[22] | TD3, SAC, DDPG, DQN, A3C, PPO | Microgrid | Frequency control | Fuel consumption minimization |
[10] | MADDPG | IEEE 123-bus test grid | Voltage control | Voltage deviation minimization |
[9] | MASAC, MATD3, SAC | IEEE 123-bus test grid | Voltage control | Voltage deviation and curtailment minimization |
[1] | SAC, TD3, DDPG | Microgrid | EV charging | Load and peak load minimization |
[23] | DDQN | Microgrid | Microgrid management | Charging cost minimization |
SAC/MASAC | DDPG/MADDPG | |
---|---|---|
Exploration | 0.7 | 0.9 |
polyak | 0.1 | 0.1 |
Buffer size | 1,000,000 | 200,000 |
Batch size | 128 | 64 |
Actor learning rate | 0.001 | 0.002 |
Critic learning rate | 0.001 | 0.001 |
Optimizer | Adam | Adam |
Entropy | 0.9 | N/A |
Regularization | 0.001 | N/A |
No Forecasting [u] | Forecasting [u] | |||
---|---|---|---|---|
Algorithm | Best | Average | Best | Average |
DDPG | −7.66 | −9.89 | −609 | −708 |
SAC | −5.48 | −7.31 | −6.20 | −9.70 |
MADDPG | −73.6 | −83.9 | −72.3 | −77.2 |
MASAC | −96.4 | −132.8 | −241 | −289 |
Busses | 1 | 2 | 3 | 4 | 5 | 6 |
---|---|---|---|---|---|---|
Mean Power Balance [MW] | 9.29 | 9.19 | 9.35 | 9.38 | 0.36 | −5.28 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Avkhimenia, V.; Gemignani, M.; Weis, T.; Musilek, P. Deep Reinforcement Learning-Based Operation of Transmission Battery Storage with Dynamic Thermal Line Rating. Energies 2022, 15, 9032. https://doi.org/10.3390/en15239032
Avkhimenia V, Gemignani M, Weis T, Musilek P. Deep Reinforcement Learning-Based Operation of Transmission Battery Storage with Dynamic Thermal Line Rating. Energies. 2022; 15(23):9032. https://doi.org/10.3390/en15239032
Chicago/Turabian StyleAvkhimenia, Vadim, Matheus Gemignani, Tim Weis, and Petr Musilek. 2022. "Deep Reinforcement Learning-Based Operation of Transmission Battery Storage with Dynamic Thermal Line Rating" Energies 15, no. 23: 9032. https://doi.org/10.3390/en15239032