A Modified Quad Q Network Algorithm for Predicting Resource Management
Abstract
:1. Introduction
2. Literature Review: Related Work
2.1. The Latest Trends in Fuzzy Logic and Reinforcement Learning
2.2. Introduction to the Algorithms of Deep Q Learning
3. Description of the Quad Q Network Algorithms and Functionalities
4. Experiment and Results
4.1. Datasets of Industrial Google Resource Value
4.2. DQN, Dueling DQN, Double Dueling DQN, and QQN Learning Results on Google’s Total Resource
4.3. DQN, Dueling DQN, Double Dueling DQN, and QQN Learning Results on Apple’s Total Resource
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Alfaverh, F.; Denai, M.; Sun, Y. Demand Response Strategy Based on Reinforcement Learning and Fuzzy Reasoning for Home Energy Management. IEEE Access 2020, 8, 39310–39321. [Google Scholar] [CrossRef]
- Wu, Y.; Dai, H.N.; Wang, H. Convergence of Blockchain and Edge Computing for Secure and Scalable IIoT Critical Infrastructures in Industry 4.0. IEEE Internet Things J. 2020, 8, 2300–2317. [Google Scholar] [CrossRef]
- Vaio, A.D.; Boccia, F.; Landriani, L.; Palladino, R. Artificial Intelligence in the Agri-Food System: Rethinking Sustainable Business Models in the COVID-19 Scenario. Sustainability 2020, 12, 4851. [Google Scholar] [CrossRef]
- Agbehadji, I.E.; Awuzie, B.O.; Ngowi, A.B.; Millham, R. Review of Big Data Analytics, Artificial Intelligence and Nature-Inspired Computing Models towards Accurate Detection of COVID-19 Pandemic Cases and Contact Tracing. Int. J. Environ. Res. Public Health 2020, 17, 5330. [Google Scholar] [CrossRef] [PubMed]
- Lee, S.; Choi, D.-H. Energy management of smart home with appliances, energy storage system and electric vehicle: A hierarchical deep reinforcement learning approach. Sensors 2020, 20, 2157. [Google Scholar] [CrossRef] [Green Version]
- Athanassopoulos, E.; Voskoglou, M.G. A philosophical treatise on the connection of scientific reasoning with fuzzy logic. Mathematics 2020, 8, 875. [Google Scholar] [CrossRef]
- Chen, C.-H.; Jeng, S.-Y.; Lin, C.-J. Mobile Robot Wall-Following Control Using Fuzzy Logic Controller with Improved Differential Search and Reinforcement Learning. Mathematics 2020, 8, 1254. [Google Scholar] [CrossRef]
- Rustum, R.; Kurichiyanil, A.M.J.; Forrest, S.; Sommariva, C.; Adeloye, A.J.; Zounemat-Kermani, M.; Scholz, M. Sustainability Ranking of Desalination Plants Using Mamdani Fuzzy Logic Inference Systems. Sustainability 2020, 12, 631. [Google Scholar] [CrossRef] [Green Version]
- Gowida, A.; Elkatatny, S.; Al-Afnan, S.; Abdulraheem, A. New computational artificial intelligence models for generating synthetic formation bulk density logs while drilling. Sustainability 2020, 12, 686. [Google Scholar] [CrossRef] [Green Version]
- Yang, Q.; Jang, S.; Yoo, S.J. Q-Learning-Based Fuzzy Logic for Multi-objective Routing Algorithm in Flying Ad Hoc Networks. Wirel. Pers. Commun. 2020, 113, 115–138. [Google Scholar] [CrossRef]
- Chen, S.; Lin, T.; Jheng, K.; Wu, C. Application of Fuzzy Theory and Optimum Computing to the Obstacle Avoidance Control of Unmanned Underwater Vehicles. Appl. Sci. 2020, 10, 6105. [Google Scholar] [CrossRef]
- Gao, H.; Ran, L.G.; Wei, G.W.; Wei, C.; Wu, J. VIKOR Method for MAGDM Based on Q-Rung Interval-Valued Orthopair Fuzzy Information and Its Application to Supplier Selection of Medical Consumption Products. Int. J. Environ. Res. Public Health 2020, 17, 525. [Google Scholar] [CrossRef] [Green Version]
- Bylykbashi, K.; Qafzezi, E.; Ampririt, P.; Ikeda, M.; Matsuo, K.; Barolli, L. Performance Evaluation of an Integrated Fuzzy-Based Driving-Support System for Real-Time Risk Management in VANETs. Sensors 2020, 20, 6537. [Google Scholar] [CrossRef]
- Ahmad, M. Alshorman, Omar Alshorman. Fuzzy-Based Fault-Tolerant Control for Omnidirectional Mobile Robot. Machines 2020, 8, 55. [Google Scholar]
- Chen, L.; Hu, X.; Tang, B.; Cheng, Y. Conditional DQN-based motion planning with fuzzy logic for autonomous driving. IEEE Trans. Intell. Transp. Syst. 2020, 1–12. [Google Scholar] [CrossRef]
- Li, Y.; Zhang, W.; Wang, C.X.; Sun, J.; Liu, Y. Deep reinforcement learning for dynamic spectrum sensing and aggregation in multi-channel wireless networks. IEEE Trans. Cogn. Commun. Netw. 2020, 6, 464–475. [Google Scholar] [CrossRef]
- Xu, Y.; Yu, J.; Buehrer, R.M. The Application of deep reinforcement learning to distributed spectrum access in dynamic heterogeneous environments with partial observations. IEEE Trans. Wirel. Commun. 2020, 19, 4494–4506. [Google Scholar] [CrossRef]
- Raj, V.; Dias, I.; Tholeti, T.; Kalyani, S. Spectrum access in cognitive radio using a two-stage reinforcement learning approach. IEEE J. Sel. Top. Signal Process. 2018, 12, 20–34. [Google Scholar] [CrossRef] [Green Version]
- Luong, N.C.; Hoang, D.T.; Gong, S.; Niyato, D.; Wang, P.; Liang, Y.C.; Kim, D.I. Applications of deep reinforcement learning in communications and networking: A Survey. IEEE Commun. Surv. Tutor. 2019, 21, 3133–3174. [Google Scholar] [CrossRef] [Green Version]
- Wang, S.; Liu, H.; Gomes, P.H.; Krishnamachari, B. Deep reinforcement learning for dynamic multichannel access. In Proceedings of the International Conference on Computing, Networking and Communications (ICNC), Silicon Valley, CA, USA, 26–29 January 2017; pp. 599–603. [Google Scholar]
- Schaul, T.; Quan, J.; Antonoglou, I.; Silver, D. Prioritized experienc replay. arXiv 2016, arXiv:1511.05952. [Google Scholar]
- Ye, H.; Li, G.Y. Deep reinforcement learning for resource allocation in V2V communications. In Proceedings of the 2018 IEEE International Conference on Communications (ICC), Kansas City, MO, USA, 20–24 May 2018; pp. 1–6. [Google Scholar]
- Liu, S.; Hu, X.; Wang, W. Deep reinforcement learning based dynamic channel allocation algorithm in multibeam satellite systems. IEEE Access 2018, 6, 15733–15742. [Google Scholar] [CrossRef]
- Shi, Z.; Xie, X.; Lu, H.; Yang, H.; Kadoch, M.; Cheriet, M. Deep reinforcement learning based spectrum resource management for industrial internet of things. IEEE Internet Things J. 2020, 8, 3476–3489. [Google Scholar] [CrossRef]
- Zhu, J.; Song, Y.; Jiang, D.; Song, H. A new deep Q-learningbased transmission scheduling mechanism for the cognitive Internet of Things. IEEE Internet Things J. 2017, 5, 2375–2385. [Google Scholar] [CrossRef]
- Wang, S.; Liu, H.; Gomes, P.H.; Krishnamachari, B. Deep reinforcement learning for dynamic multichannel access in wireless networks. IEEE Trans. Cogn. Commun. Netw. 2018, 4, 257–265. [Google Scholar] [CrossRef] [Green Version]
- Zhong, C.; Lu, Z.; Gursoy, M.C.; Velipasalar, S. Actor-Critic deep reinforcement learning for dynamic multichannel access. In Proceedings of the 2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP), Anaheim, CA, USA, 26–29 November 2018; pp. 599–603. [Google Scholar]
- Chang, H.; Song, H.; Yi, Y.; Zhang, J.; He, H.; Liu, L. Distributive dynamic spectrum access through deep reinforcement learning: A reservoir computing-based approach. IEEE Internet Things J. 2019, 6, 1938–1948. [Google Scholar] [CrossRef] [Green Version]
- Naparstek, O.; Kobi, C. Deep multi-user reinforcement learning for dynamic spectrum access in multichannel wireless networks. In Proceedings of the Globecom 2017—2017 IEEE Global Communications Conference, Singapore, 4–8 December 2017; pp. 1–7. [Google Scholar]
- Huang, X.L.; Li, Y.X.; Gao, Y.; Tang, X.W. Q-learning based spectrum access for multimedia transmission over cognitive radio networks. IEEE Trans. Cogn. Commun. Netw. 2020, 7, 110–119. [Google Scholar] [CrossRef]
- Aref, M.A.; Jayaweera, S.K.; Machuzak, S. Multi-agent reinforcement learning based cognitive anti-jamming. In Proceedings of the 2017 IEEEWireless Communications and Networking Conference (WCNC), San Francisco, CA, USA, 19–22 March 2017; pp. 1–6. [Google Scholar]
- Zhang, Y.; Cai, P.; Pan, C.; Zhang, S. Multi-agent deep reinforcement learning-based cooperative spectrum sensing with upper confidence bound exploration. IEEE Access 2019, 7, 118898–118906. [Google Scholar] [CrossRef]
- Nabipour, M.; Nayyeri, P.; Jabani, H.; Mosavi, A.; Salwana, E.; Shahab, S. Deep Learning for Stock Market Prediction. Entropy 2020, 22, 840. [Google Scholar] [CrossRef]
- Gu, J.; Fang, Y.; Sheng, Z.; Wen, P. Double Deep Q-Network with a Dual-Agent for Traffic Signal Control. Appl. Sci. 2020, 10, 1622. [Google Scholar] [CrossRef] [Green Version]
- Polvara, R.; Patacchiola, M.; Hanheide, M.; Neumann, G. Sim-to-Real quadrotor landing via sequential deep Q-Networks and domain randomization. Robotics 2020, 9, 8. [Google Scholar] [CrossRef] [Green Version]
- Sun, Y.; Ran, X.; Zhang, G.; Xu, H.; Wang, X. AUV 3D Path Planning Based on the Improved Hierarchical Deep Q Network. J. Mar. Sci. Eng. 2020, 8, 145. [Google Scholar] [CrossRef] [Green Version]
- Li, D.; Xu, S.; Li, P. Deep reinforcement learning-empowered resource allocation for mobile edge computing in cellular v2x networks. Sensors 2021, 21, 372. [Google Scholar] [CrossRef]
Date | Price | Volume |
---|---|---|
28 March 2014 | 559.99 | 41,003 |
31 March 2014 | 556.97 | 10,772 |
1 April 2014 | 567.16 | 7932 |
2 April 2014 | 567.00 | 146,697 |
3 April 2014 | 569.74 | 5,087,500 |
4 April 2014 | 543.14 | 6,377,600 |
7 April 2014 | 538.15 | 4,368,717 |
8 April 2014 | 554.90 | 3,148,563 |
9 April 2014 | 564.14 | 3,323,579 |
⋮ | ⋮ | ⋮ |
3 November 2017 | 1032.48 | 1,076,350 |
6 November 2017 | 1025.90 | 1,124,765 |
7 November 2017 | 1033.33 | 1,112,146 |
8 November 2017 | 1039.85 | 1,088,395 |
9 November 2017 | 1031.05 | 1,244,886 |
10 November 2017 | 1028.07 | 720,674 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Kim, Y.; Park, J.; Kim, J.; Yoon, J.; Lee, S.; Kim, J. A Modified Quad Q Network Algorithm for Predicting Resource Management. Appl. Sci. 2021, 11, 5154. https://doi.org/10.3390/app11115154
Kim Y, Park J, Kim J, Yoon J, Lee S, Kim J. A Modified Quad Q Network Algorithm for Predicting Resource Management. Applied Sciences. 2021; 11(11):5154. https://doi.org/10.3390/app11115154
Chicago/Turabian StyleKim, Yeonggwang, Jaehyung Park, Jinyoung Kim, Junchurl Yoon, Sangjoon Lee, and Jinsul Kim. 2021. "A Modified Quad Q Network Algorithm for Predicting Resource Management" Applied Sciences 11, no. 11: 5154. https://doi.org/10.3390/app11115154