Depth Image Completion through Iterative Low-Pass Filtering
Abstract
:1. Introduction
2. Approach
2.1. Extraction of Boundaries and Surfaces
2.2. Iterative Low-Pass Filter (ILF)
- (1)
- Multiplication with a low-pass filter in the Fourier domain, where the cut-off frequency changes from low to high in different iterations.
- (2)
- Updating of the expanded depth matrix with a raw depth matrix.
3. Experimental Results
3.1. Rate of the Maximum Defected Area
3.2. Denoising Ability
3.3. Real-Scene Results
3.4. Benchmark Results
4. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Geiger, A.; Ziegler, J.; Stiller, C. StereoScan: Dense 3d reconstruction in real-time. In Proceedings of the 2011 IEEE Intelligent Vehicles Symposium (IV), Baden-Baden, Germany, 5–9 June 2011. [Google Scholar]
- Rusu, R.B.; Marton, Z.C.; Blodow, N.; Dolha, M.; Beetz, M. Towards 3D point cloud-based object maps for household environments. Robot. Auton. Syst. 2008, 56, 927–941. [Google Scholar] [CrossRef]
- Marapane, S.B.; Trivedi, M.M. Region-based stereo analysis for robotic applications. IEEE Trans. Syst. Man Cybern. 1989, 19, 1447–1464. [Google Scholar] [CrossRef]
- Nalpantidis, L.; Gasteratos, A. Stereo vision for robotic applications in the presence of non-ideal lighting conditions. Image Vis. Comput 2010, 28, 940–951. [Google Scholar] [CrossRef]
- Murray, D.; Little, J.J. Using real-time stereo vision for mobile robot navigation. Auton. Robots 2000, 8, 161–171. [Google Scholar] [CrossRef]
- Huang, Y.K.; Liu, Y.C.; Wu, T.H.; Su, H.T.; Hsu, W.H. Expanding Sparse Guidance for Stereo Matching. arXiv 2020, arXiv:2005.02123. [Google Scholar]
- Shaked, A.; Wolf, L. Improved stereo matching with constant highway networks and reflective confidence learning. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017. [Google Scholar]
- Saito, Y.; Yamamoto, Y.; Kan, T.; Tsukagoshi, T.; Noda, K.; Shimoyama, I. Electrical detection SPR sensor with grating coupled backside illumination. Opt. Express 2019, 27, 17763–17770. [Google Scholar] [CrossRef] [PubMed]
- Dalla Betta, G.F.; Donati, S.; Hossain, Q.D.; Martini, G.; Pancheri, L.; Saguatti, D. Design and Characterization of Current-Assisted Photonic demodulators. IEEE Trans. Electron Devices 2011, 58, 1702–1709. [Google Scholar] [CrossRef]
- Khoshelham, K.; Elberink, S.O. Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 2012, 12, 1437–1454. [Google Scholar] [CrossRef] [PubMed]
- Mineo, C.; Cerniglia, D.; Ricotta, V.; Reitinger, B. Autonomous 3D geometry reconstruction through robot-manipulated optical sensors. Int. J. Adv. Manuf. Technol. 2021, 116, 1895–1911. [Google Scholar] [CrossRef]
- Mallick, T.; Das, P.P.; Majumdar, A.K. Characterizations of noise in Kinect depth images. IEEE Sens. J. 2014, 6, 1731–1740. [Google Scholar] [CrossRef]
- Matsuo, K.; Aoki, Y. Depth image enhancement using local tangent plane approximations. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015. [Google Scholar]
- Doria, D.; Radke, R.J. Filling large holes in lidar data by inpainting depth gradients. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops), Providence, RI, USA, 16–21 June 2012. [Google Scholar]
- Zhang, F.; Prisacariu, V.; Yang, R.; Torr, P.H. Ga-net: Guided aggregation net for end-to-end stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 16–20 June 2019. [Google Scholar]
- Park, S.; Guo, X.; Shin, H.; Qin, H. Shape and appearance repair for incomplete point surfaces. In Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV’05), Beijing, China, 17–21 October 2005. [Google Scholar]
- Kochanowski, M.; Jenke, P.; Straßer, W. Analysis of texture synthesis algorithms with respect to usage for Hole-Filling in 3D geometry. In Proceedings of the ITCS, Beijing, China, 12–17 October 2008. [Google Scholar]
- Hanocka, R.; Metzer, G.; Giryes, R.; Cohen-Or, D. Point2Mesh: A Self-Prior for Deformable Meshes. arXiv 2020, arXiv:2005.11084. [Google Scholar] [CrossRef]
- Watson, J.; Mac Aodha, O.; Prisacariu, V.; Brostow, G.; Firman, M. The temporal opportunist: Self-supervised multi-frame monocular depth. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021. [Google Scholar]
- Eigen, D.; Puhrsch, C.; Fergus, R. Depth map prediction from a single image using a multi-scale deep network. Adv. Neural Inf. Process. Syst. 2014, 27, 2366–2374. [Google Scholar]
- Godard, C.; Mac Aodha, O.; Brostow, G.J. Unsupervised monocular depth estimation with left-right consistency. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017. [Google Scholar]
- Hu, M.; Wang, S.; Li, B.; Ning, S.; Fan, L.; Gong, X. Towards precise and efficient image guided depth completion. In Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China, 30 May–5 June 2021. [Google Scholar]
- Casser, V.; Pirk, S.; Mahjourian, R.; Angelova, A. Unsupervised monocular depth and ego-motion learning with structure and semantics. In Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Long Beach, CA, USA, 15–20 June 2019. [Google Scholar]
- Park, J.; Joo, K.; Hu, Z.; Liu, C.K.; So Kweon, I. Non-local spatial propagation network for depth completion. In Proceedings of the European Computer Vision–ECCV 2020: 16th European Conference, Part XIII 16, Glasgow, UK, 23–28 August 2020. [Google Scholar]
- Zhang, Y.; Funkhouser, T. Deep depth completion of a single rgb-d image. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–23 June 2018. [Google Scholar]
- Van Gansbeke, W.; Neven, D.; De Brabandere, B.; Van Gool, L. Sparse and noisy lidar completion with rgb guidance and uncertainty. In Proceedings of the 2019 16th International Conference on Machine Vision Applications (MVA), Tokyo, Japan, 27–31 May 2019. [Google Scholar]
- Izadi, S.; Kim, D.; Hilliges, O.; Molyneaux, D.; Newcombe, R.; Kohli, P. KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of the UIST 2011—24th Annual ACM Symposium on User Interface Software and Technology, Santa Barbara, CA, USA, 16–19 October 2019. [Google Scholar]
- Cheng, X.; Wang, P.; Yang, R. Depth estimation via affinity learned with convolutional spatial propagation network. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018. [Google Scholar]
- Liu, S.; De Mello, S.; Gu, J.; Zhong, G.; Yang, M.H.; Kautz, J. Learning affinity via spatial propagation networks. Adv. Neural Inf. Process Syst. 2017, 30. [Google Scholar]
- Cheng, X.; Wang, P.; Guan, C.; Yang, R. Cspn++: Learning context and resource aware convolutional spatial propagation networks for depth completion. In Proceedings of the AAAI Conference on Artificial Intelligence, New York, New York, USA, 7–12 February 2020. [Google Scholar]
- Canny, J. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 1986, 6, 679–698. [Google Scholar] [CrossRef]
- Xie, S.; Tu, Z. Holistically-nested edge detection. In Proceedings of the IEEE International Conference on Computer vision, Santiago, Chile, 7–13 December 2015. [Google Scholar]
- Ganin, Y.; Lempitsky, V. -Fields: Neural network nearest neighbor fields for image transforms. In Asian Conference on Computer Vision; Springer International Publishing: Cham, Switzerland, 2014. [Google Scholar]
- Bertasius, G.; Shi, J.; Torresani, L. Deepedge: A multi-scale bifurcated deep network for top-down contour detection. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015. [Google Scholar]
- Hwang, J.J.; Liu, T.L. Pixel-wise deep learning for contour detection. arXiv 2015, arXiv:1504.01989. [Google Scholar]
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
- Martin, D.R.; Fowlkes, C.C.; Malik, J. Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE Trans. Pattern Anal. Mach. Intell. 2004, 26, 530–540. [Google Scholar] [CrossRef] [PubMed]
- Smith, S.W. The Scientist and Engineer’s Guide to Digital Signal Processing; California Technical Pub.: San Diego, CA, USA, 1997; pp. 32–34. [Google Scholar]
- Silberman, N.; Hoiem, D.; Kohli, P.; Fergus, R. Indoor segmentation and support inference from rgbd images. In Proceedings of the European Conference on Computer Vision, Florence, Italy, 7–13 October 2012. [Google Scholar]
- Ma, F.; Karaman, S. Sparse-to-dense: Depth prediction from sparse depth samples and a single image. In Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia, 21–25 May 2018. [Google Scholar]
- Franke, R. Scattered data interpolation: Tests of some methods. Math. Comput. 1982, 38, 181–200. [Google Scholar]
- Franke, R.; Nielson, G. Smooth interpolation of large sets of scattered data. Int. J. Numer. Methods Eng. 1980, 15, 1691–1704. [Google Scholar] [CrossRef]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wang, T.-K.; Yu, Y.-W.; Yang, T.-H.; Huang, P.-D.; Zhu, G.-Y.; Lau, C.-C.; Sun, C.-C. Depth Image Completion through Iterative Low-Pass Filtering. Appl. Sci. 2024, 14, 696. https://doi.org/10.3390/app14020696
Wang T-K, Yu Y-W, Yang T-H, Huang P-D, Zhu G-Y, Lau C-C, Sun C-C. Depth Image Completion through Iterative Low-Pass Filtering. Applied Sciences. 2024; 14(2):696. https://doi.org/10.3390/app14020696
Chicago/Turabian StyleWang, Tzu-Kai, Yeh-Wei Yu, Tsung-Hsun Yang, Pin-Duan Huang, Guan-Yu Zhu, Chi-Chung Lau, and Ching-Cherng Sun. 2024. "Depth Image Completion through Iterative Low-Pass Filtering" Applied Sciences 14, no. 2: 696. https://doi.org/10.3390/app14020696