Mapless Path Planning for Mobile Robot Based on Improved Deep Deterministic Policy Gradient Algorithm
Abstract
:1. Introduction
2. DDPG Algorithm
2.1. DDPG Algorithm Flow
2.1.1. Initialize Networks
2.1.2. Interaction and Storage
2.1.3. Sampling and Training
3. Improved DDPG Algorithm
3.1. Designing Continuous State and Action Spaces
3.2. Multi-Step Update
3.3. Designing the Reward Function
3.3.1. Target Reward and Penalty
3.3.2. Stepwise Guided Immediate Rewards
3.4. Design of a Dual-Layer Noise Exploration Mechanism
4. Simulation and Experiment
4.1. Simulation Environment Setup
4.2. Analysis of Simulation Results
4.2.1. Static Obstacle Simulation
4.2.2. Dynamic Obstacle Simulation:
4.3. Real Mobile Robot Experiment
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Gao, H.; Liu, D.; Hu, J. A survey on path planning for mobile robot systems. In Proceedings of the 2023 IEEE 12th Data Driven Control and Learning Systems Conference (DDCLS), Xiangtan, China, 12–14 May 2023; pp. 1176–1181. [Google Scholar]
- Zhou, S.; Shan, L.; Chang, L.; Chen, J.; Liu, C.; Li, J. Robot path planning algorithm based on improved DDPG algorithm. J. Nanjing Univ. Sci. Technol. 2021, 45, 265. [Google Scholar]
- Chen, X.; Zhao, M.; Yin, L. Dynamic path planning of the UAV avoiding static and moving obstacles. J. Intell. Robot. Syst. 2020, 99, 909–931. [Google Scholar] [CrossRef]
- Gao, J.; Ye, W.; Guo, J.; Li, Z. Deep reinforcement learning for indoor mobile robot path planning. Sensors 2020, 20, 5493. [Google Scholar] [CrossRef] [PubMed]
- Liu, L.-S.; Lin, J.-F.; Yao, J.-X.; He, D.-W.; Zheng, J.-S.; Huang, J.; Shi, P. Path planning for smart car based on Dijkstra algorithm and dynamic window approach. Wirel. Commun. Mob. Comput. 2021, 2021, 8881684. [Google Scholar] [CrossRef]
- Zhang, Y.; Li, L.-l.; Lin, H.-C.; Ma, Z.; Zhao, J. Development of path planning approach using improved A-star algorithm in AGV system. J. Internet Technol. 2019, 20, 915–924. [Google Scholar]
- Mashayekhi, R.; Idris, M.Y.I.; Anisi, M.H.; Ahmedy, I. Hybrid RRT: A semi-dual-tree RRT-based motion planner. IEEE Access 2020, 8, 18658–18668. [Google Scholar] [CrossRef]
- Lambora, A.; Gupta, K.; Chopra, K. Genetic algorithm-A literature review. In Proceedings of the 2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon), Faridabad, India, 14–16 February 2019; pp. 380–384. [Google Scholar]
- Li, X.; Hu, X.; Wang, Z.; Du, Z. Path planning based on combinaion of improved A-STAR algorithm and DWA algorithm. In Proceedings of the 2020 2nd International Conference on Artificial Intelligence and Advanced Manufacture (AIAM), Manchester, UK, 15–17 October 2020; pp. 99–103. [Google Scholar]
- Bounini, F.; Gingras, D.; Pollart, H.; Gruyer, D. Modified artificial potential field method for online path planning applications. In Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA, 11–14 June 2017; pp. 180–185. [Google Scholar]
- Chang, L.; Shan, L.; Jiang, C.; Dai, Y. Reinforcement based mobile robot path planning with improved dynamic window approach in unknown environment. Auton. Robot. 2021, 45, 51–76. [Google Scholar] [CrossRef]
- Kaelbling, L.P.; Littman, M.L.; Moore, A.W. Reinforcement learning: A survey. J. Artif. Intell. Res. 1996, 4, 237–285. [Google Scholar] [CrossRef]
- Maoudj, A.; Hentout, A. Optimal path planning approach based on Q-learning algorithm for mobile robots. Appl. Soft Comput. 2020, 97, 106796. [Google Scholar] [CrossRef]
- Wen, S.; Jiang, Y.; Cui, B.; Gao, K.; Wang, F. A hierarchical path planning approach with Multi-SARSA based on topological map. Sensors 2022, 22, 2367. [Google Scholar] [CrossRef]
- Li, P.; Ding, X.; Ren, W. Research on path planning of cloud robot in dynamic environment based on improved ddpg algorithm. In Proceedings of the 2021 China Automation Congress (CAC), Beijing, China, 22–24 October 2021; pp. 3561–3566. [Google Scholar]
- Watkins, C.J.; Dayan, P. Q-learning. Mach. Learn. 1992, 8, 279–292. [Google Scholar] [CrossRef]
- Arulkumaran, K.; Deisenroth, M.P.; Brundage, M.; Bharath, A.A. Deep reinforcement learning: A brief survey. IEEE Signal Process. Mag. 2017, 34, 26–38. [Google Scholar] [CrossRef]
- Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A.A.; Veness, J.; Bellemare, M.G.; Graves, A.; Riedmiller, M.; Fidjeland, A.K.; Ostrovski, G. Human-level control through deep reinforcement learning. Nature 2015, 518, 529–533. [Google Scholar] [CrossRef]
- Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; Riedmiller, M. Playing atari with deep reinforcement learning. arXiv 2013, arXiv:1312.5602. [Google Scholar]
- Lillicrap, T.P.; Hunt, J.J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa, Y.; Silver, D.; Wierstra, D. Continuous control with deep reinforcement learning. arXiv 2015, arXiv:1509.02971. [Google Scholar]
- Dong, Y.; Zou, X. Mobile robot path planning based on improved DDPG reinforcement learning algorithm. In Proceedings of the 2020 IEEE 11th International Conference on Software Engineering and Service Science (ICSESS), Beijing, China, 16–18 October 2020; pp. 52–56. [Google Scholar]
- Zhang, H.; Xie, M.; Zhang, M.; Wu, N. Path Planning of Mobile Robot with Fusion DDPG Algorithm. Control Eng. China 2021, 28, 2136–2142. [Google Scholar] [CrossRef]
- Li, D.; Dong, C. Double-net DDPG with the Optimal Action Selection Mechanism. In Proceedings of the 2022 IEEE 11th Data Driven Control and Learning Systems Conference (DDCLS), Emei, China, 3–5 August 2022; pp. 1166–1170. [Google Scholar]
- Yuan, Y.; Yu, Z.L.; Gu, Z.; Yeboah, Y.; Wei, W.; Deng, X.; Li, J.; Li, Y. A novel multi-step Q-learning method to improve data efficiency for deep reinforcement learning. Knowl.-Based Syst. 2019, 175, 107–117. [Google Scholar] [CrossRef]
- Liu, J.; Qi, W.; Lu, X. Multi-step reinforcement learning algorithm of mobile robot path planning based on virtual potential field. In Proceedings of the International Conference of Pioneering Computer Scientists, Engineers and Educators, Changsha, China, 22–24 September 2017; pp. 528–538. [Google Scholar]
- Gu, W.; Wang, S. An Improved Strategy for Blood Glucose Control Using Multi-Step Deep Reinforcement Learning. arXiv 2024, arXiv:2403.07566. [Google Scholar]
Interaction and Storage |
---|
For each time step t: |
Based on current state using the policy network to generate action |
If Task Completed or FailureState: done = True |
Else: done = False |
into replay buffer |
End for |
Parameter | Value |
---|---|
Actor Network Learning Network | 0.0001 |
Critic Network Learning Rate | 0.0001 |
Target Network Soft Update Parameter | 0.005 |
Discount Factor | 0.99 |
Experience Pool Size | 100,000 |
Batch Size | 64 |
Collision Threshold | 0.35 m |
Distance to Target Threshold | 0.4 m |
Maximum Linear Speed | 1 m/s |
Maximum Angular Speed | 1 rad/s |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zhang, S.; Tang, W.; Li, P.; Zha, F. Mapless Path Planning for Mobile Robot Based on Improved Deep Deterministic Policy Gradient Algorithm. Sensors 2024, 24, 5667. https://doi.org/10.3390/s24175667
Zhang S, Tang W, Li P, Zha F. Mapless Path Planning for Mobile Robot Based on Improved Deep Deterministic Policy Gradient Algorithm. Sensors. 2024; 24(17):5667. https://doi.org/10.3390/s24175667
Chicago/Turabian StyleZhang, Shuzhen, Wei Tang, Panpan Li, and Fusheng Zha. 2024. "Mapless Path Planning for Mobile Robot Based on Improved Deep Deterministic Policy Gradient Algorithm" Sensors 24, no. 17: 5667. https://doi.org/10.3390/s24175667