Synergistic Pushing and Grasping for Enhanced Robotic Manipulation Using Deep Reinforcement Learning
Abstract
:1. Introduction
2. Related Works
Major Contributions
- Synergizing of pushing and grasping actions to enhance the overall grasping performance in cluttered environments.
- Usage of a dual-network approach for pushing actions (Push-Net) and for grasping actions (Grasp-Net)
- Self-supervised learning with grasp and push rewards
- Robust performance in diverse scenarios (randomly arranged clutter, well-ordered difficult configurations, and novel objects).
- Adopting of a unique technique called pixel-wise depth heightmap image differencing (PDD) for reward strategy.
3. Materials and Methods
- 1.
- Simulation Setup
- Overview
- Hardware components
- 1.
- UR5 Robotic Arm:Model: Universal Robots UR5.Degrees of Freedom: 6.Payload: 5 kg.Reach: 850 mm.Repeatability: ±0.1 mm.
- 2.
- End Effector:Type: RG2 parallel jaw gripper.Grip Force: 20–235 N.Grip Stroke: 0–85 mm.
- 3.
- Vision System:Camera: Intel RealSense Depth Camera D435.Resolution: 640 × 480 at 30 fps.Depth Range: 0.2–10 m.
- 4.
- Computing System:Processor: Intel Core i7.GPU: NVIDIA Quadro P2000 Intel(R) Xeon(R)RAM: 32 GB.
- Software Components
- 1.
- Operating System: Ubuntu 18.04 LTS.
- 2.
- Robotics Middleware: Virtual Robot Experimentation Platform (V-REP).
- 3.
- Control Interface: CoppeliaSim VR interface with the UR5 robot.
- 4.
- Simulation Environment: CoppeliaSim simulation.
- 5.
- Machine Learning Framework: PyTorch for model training and deployment.
- Simulation Procedure
- 1.
- Environment Setup:
- The robot is placed in a controlled test area with a flat, non-reflective surface to minimize visual noise.
- Objects for manipulation are randomly placed within the robot’s workspace to simulate cluttered environments.
- A mix of known and novel objects is used to evaluate the model’s adaptability.
- 2.
- Calibration:
- The vision system is calibrated using standard calibration techniques to ensure accurate depth perception.
- The end effector is calibrated to ensure precise gripping force and stroke control.
- 3.
- Task Execution:
- The robot is programmed to perform a series of grasping tasks, starting with randomly arranged clutter.
- The tasks are repeated with well-ordered object configurations and novel objects to test the model’s adaptability and performance consistency.
- Each task involves identifying the object, planning a grasp, and executing the grasp.
3.1. UR5 Mathematical Modeling and Deep Reinforcement Algorithm
3.1.1. UR5 Mathematical Modeling
3.1.2. D-H Representation of Forward Kinematic Equations of UR5
3.2. Data Collection
- ♦
- Randomly arranged clutter.
- ♦
- Challenging well-ordered configurations.
- ♦
- Novel object configurations.
3.3. Deep Reinforcement Learning
3.3.1. Densely Connected Convolutional Neural Networks
3.3.2. Pushing Primitive Actions
3.3.3. Grasping Primitive Actions
3.3.4. Reward Modeling
3.3.5. Training Procedures
3.3.6. Testing Procedures
- Average percent clearance: Over the n test runs, the average percentage clearance rate assesses the policy’s ability to complete the task by picking up all objects in the workspace without failing more than 10 times.
- The ratio of successful grasps in all grasp attempts per completion, which assesses the grasping policy’s accuracy, is called the average percent grasp success per clearance.
- The average grasp-to-push ratio is the number of successful grasps divided by the number of successful pushes in each test case’s complete run tests.
4. Results
4.1. Performance in Cluttered Environments
4.2. Randomly Arranged Objects
4.3. Challenging Well-Ordered Configurations
4.4. Generalization to Novel Objects
4.5. Discussion
- 1.
- Grasp Success Rate:
- 2.
- Action Efficiency:
- 3.
- Completion Rate:
- 4.
- Hardware and Computational Efficiency:
- 5.
- Training time and Epoch:
Year | Action | GPU | Epochs | Time | Work Area | Ref | Completion | Grasp Success | Action Efficiency |
---|---|---|---|---|---|---|---|---|---|
2021 | Grasping | Nvidia RTX 2080 | 2500 | N/A | Clutter | [6] | 53.1% | 62% | 51.4% |
2020 | Grasping | N/A | 7000 | N/A | Sparse | [7] | 100% | 87% | 95.2% |
2020 | Pushing and grasping | Titan X (12 GB) | 10,000 | 34 h | Clutter | [8] | 93.7% | 76% | 84% |
2020 | Pick and place | GeForce GTX-1660Ti | 3000 | 10 h | Clutter | [9] | 100% | 74.3% | 92% |
2018 | Pushing and grasping | NVIDIA Titan X | 2500 | 5.5 h | Clutter | [10] | 100% | 65.4% | 59.7% |
2022 | Pushing and grasping | Nvidia GTX 1080Ti | 250 | N/A | Clutter | [11] | 100% | 83.5% | 69% |
2023 | Pushing and grasping | N/A | 1000 | N/A | Clutter | [12] | 93.8% | 75.3% | 54.1% |
2024 | Stacking and grasping | NVIDIA GTX 2080TI | 10,000 | N/A | Clutter | [13] | 100% | 86% | 69% |
2024 | Proposed | DELL E2417H desktop | 3000 | 5 h | Clutter | N/A | 100% | 87% | 91% |
5. Conclusions and Future Work
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
Appendix A. The Proposed Model Training and Test Cases
References
- Liu, R.; Nageotte, F.; Zanne, P.; de Mathelin, M.; Dresp-Langley, B. Deep reinforcement learning for the control of robotic manipulation: A focussed mini-review. Robotics 2021, 10, 22. [Google Scholar] [CrossRef]
- Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A.A.; Veness, J.; Bellemare, M.G.; Graves, A.; Riedmiller, M.; Fidjeland, A.K.; Ostrovski, G.; et al. Human-level control through deep reinforcement learning. Nature 2015, 518, 529–533. [Google Scholar] [CrossRef] [PubMed]
- Sharma, V.; Rai, S.; Dev, A. A Comprehensive Study of Artificial Neural Networks. Int. J. Adv. Res. Comput. Sci. Softw. Eng. 2012, 2, 278–284. [Google Scholar]
- Sutton, R.S.; Barto, A.G.; Learning, R. No play, bad work, and poor health. Lancet 1998, 258, 675–676. [Google Scholar] [CrossRef]
- François-Lavet, V.; Henderson, P.; Islam, R.; Bellemare, M.G.; Pineau, J. An Introduction to Deep Reinforcement Learning. Found. Trends® Mach. Learn. 2018, 11, 219–354. [Google Scholar] [CrossRef]
- Al-Shanoon, A.; Lang, H.; Wang, Y.; Zhang, Y.; Hong, W. Learn to grasp unknown objects in robotic manipulation. Intell. Serv. Robot. 2021, 14, 571–582. [Google Scholar] [CrossRef]
- Joshi, S.; Kumra, S.; Sahin, F. Robotic Grasping using Deep Reinforcement Learning. In Proceedings of the 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), Hong Kong, China, 20–21 August 2020; pp. 1461–1466. [Google Scholar] [CrossRef]
- Hundt, A.; Killeen, B.; Greene, N.; Wu, H.; Kwon, H.; Paxton, C.; Hager, G.D. ‘Good Robot!’: Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer. IEEE Robot. Autom. Lett. 2020, 5, 6724–6731. [Google Scholar] [CrossRef]
- Mohammed, M.Q.; Chung, K.L. Pick and Place Objects in a Cluttered Scene Using Deep Reinforcement Learning. Int. J. Mech. Mechatron. Eng. 2020, 20, 50–57. [Google Scholar]
- Zeng, A.; Song, S.; Welker, S.; Lee, J.; Rodriguez, A.; Funkhouser, T. Learning Synergies between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning. In Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, 1–5 October 2018; pp. 4238–4245. [Google Scholar] [CrossRef]
- Peng, G.; Liao, J.; Guan, S.; Yang, J.; Li, X. A pushing-grasping collaborative method based on deep Q-network algorithm in dual viewpoints. Sci. Rep. 2022, 12, 3927. [Google Scholar] [CrossRef] [PubMed]
- Wu, L.; Chen, Y.; Li, Z.; Liu, Z. Efficient push-grasping for multiple target objects in clutter environments. Front. Neurorobot. 2023, 17, 1188468. [Google Scholar] [CrossRef] [PubMed]
- Wang, F.; Liu, Y.; Shi, M.; Chen, C.; Liu, S.; Zhu, J. Efficient Stacking and Grasping in Unstructured Environments. J. Intell. Robot. Syst. Theory Appl. 2024, 110, 57. [Google Scholar] [CrossRef]
- Yang, Y.; Liang, H.; Choi, C. A Deep Learning Approach to Grasping the Invisible. IEEE Robot. Autom. Lett. 2020, 5, 2232–2239. [Google Scholar] [CrossRef]
- Huang, G.; Liu, Z.; Van Der Maaten, L.; Weinberger, K.Q. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 2261–2269. [Google Scholar] [CrossRef]
- Rohmer, E.; Singh, S.P.N.; Freese, M. V-REP: A versatile and scalable robot simulation framework. In Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan, 3–7 November 2013; pp. 1321–1326. [Google Scholar] [CrossRef]
- Coumans, E. Bullet 2.82 Physics SDK Manual Table of Contents. 2013. Available online: https://wiki.blender.jp/images/9/95/Dev-Physics-bullet-documentation.pdf (accessed on 23 June 2022).
Link, i | Rot x | Disp x | Rot x | Rot z | Disp z |
---|---|---|---|---|---|
1 | 0 | 0.0891 | |||
2 | 0 | 0.425 | 0 | 0 | |
3 | 0 | 0.39225 | 0 | 0 | |
4 | 0 | 0.10915 | |||
5 | 0 | 0.09456 | |||
6 | - | - | - | 0.0823 |
Layers | Output Layer | DenseNet-121 |
---|---|---|
Convolution | 112 × 112 × 64 | BN, ReLU, 7 × 7 conv with stride = 2, 3 × 3 Zero padding |
Pooling | 56 × 56 × 64 | 3 × 3 max-pooling with stride = 2 |
Dense Block 1 (DB1) (6 layers) | 56 × 56 × 256 | BN, ReLU, BN, ReLU, |
Transition Layer 1 | 28 × 28 × 128 | BN, ReLU, 1 × 1 conv with , 2 × 2 average pooling with stride = 2 |
Dense Block 2 (DB2) (12 layers) | 28 × 28 × 512 | BN, ReLU, , BN, ReLU, |
Transition Layer 2 | 14 × 14 × 256 | BN, ReLU, 1 × 1 conv with , 2 × 2 average pooling with stride = 2 |
Dense Block 3 (DB3) (24 layers) | 14 × 14 × 1024 | BN, ReLU, BN, ReLU, |
Transition Layer 3 | 7 × 7 × 512 | BN, ReLU, 1 × 1 conv with , 2 × 2 average pooling with stride = 2 |
Dense Block 4 (DB4) (16 layers) | 7 × 7 × 1024 | BN, ReLU, , BN, ReLU, |
Output Layer | 1 × 1 × 1024 | 7 × 7 global average pooling |
Test Cases | Completion | Grasp Success | Grasp to Push Ratio |
---|---|---|---|
Random arrangements | 80% | 50.5% | 77.1% |
Test Cases | Completion | Grasp Success | Grasp-to-Push Ratio |
---|---|---|---|
Test case 00 | 96.8% | 84.7% | 99.5% |
Test case 01 | 86.7% | 61.4% | 95.3% |
Test case 02 | 96.8% | 75.1% | 100% |
Test Cases | Completion | Grasp Success Rate | Grasp-to-Push Ratio |
---|---|---|---|
Novel test case 00 | 74.1% | 73.9% | 93.9% |
Novel test case 01 | 80% | 65.4% | 93.2% |
Novel test case 02 | 100% | 95.8% | 100% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Shiferaw, B.A.; Agidew, T.F.; Alzahrani, A.S.; Srinivasagan, R. Synergistic Pushing and Grasping for Enhanced Robotic Manipulation Using Deep Reinforcement Learning. Actuators 2024, 13, 316. https://doi.org/10.3390/act13080316
Shiferaw BA, Agidew TF, Alzahrani AS, Srinivasagan R. Synergistic Pushing and Grasping for Enhanced Robotic Manipulation Using Deep Reinforcement Learning. Actuators. 2024; 13(8):316. https://doi.org/10.3390/act13080316
Chicago/Turabian StyleShiferaw, Birhanemeskel Alamir, Tayachew F. Agidew, Ali Saeed Alzahrani, and Ramasamy Srinivasagan. 2024. "Synergistic Pushing and Grasping for Enhanced Robotic Manipulation Using Deep Reinforcement Learning" Actuators 13, no. 8: 316. https://doi.org/10.3390/act13080316