Improved Unsupervised Stitching Algorithm for Multiple Environments SuperUDIS
Abstract
:1. Introduction
- The proposed algorithm applied the Superpoint feature point and LightGlue feature matching methods to the image stitching algorithm at first. The effect of Superpoint combined with LightGlue is significantly better than the distortion part of the ORB and UDIS algorithms on the HPatches dataset.
- In the proposed algorithm, we exploit a deep learning-based method of UDIS for seam splicing and use the chroma balance algorithm to preprocess the input image of UDIS to solve the problem of obvious color faults. At the same time, the loss function is replaced by the Laplacian operator. The difference in direction optimizes the splicing result.
- The proposed algorithm combines Superpoint, Lightglue, and the UDIS fusion algorithm. The three algorithms complement each other. Superpoint and Lightglue enhance the precision of image registration and solve the problem that the UDIS algorithm has limited resolution and poor effects in specific environments. In addition, the utilization of floating-point masks in the UDIS algorithm contributes to smoother stitching outcomes.
2. Related Work
2.1. Traditional Stitching Algorithm
2.2. Stitching Algorithm Based on Deep Learning
3. SuperUDIS
3.1. Superpoint
3.2. Lightglue
3.3. Chroma Balance Algorithm
3.4. UDIS Fusion Algorithm
3.5. Loss Function Optimization
4. Experiment and Result
4.1. Dataset and Details
4.2. Comparison to the Previous Methods
4.2.1. Image Registration Experiment
4.2.2. Image Fusion Experiment
4.3. Ablation Study
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Lyons, M.B.; Keith, D.A.; Phinn, S.R.; Mason, T.J.; Elith, J. A comparison of resampling methods for remote sensing classification and accuracy assessment. Remote Sens. Environ. 2018, 208, 145–153. [Google Scholar] [CrossRef]
- Kang, Z.; Zhang, L.; Zlatanova, S.; Li, J. An automatic stitching method for building facade texture mapping using a monocular close-range image sequence. ISPRS J. Photogr. Remote Sens. 2009, 65, 282–293. [Google Scholar] [CrossRef]
- Nie, L.; Lin, C.; Liao, K.; Liu, S.; Zhao, Y. Unsupervised deep image stitching: Reconstructing stitched features to images. IEEE Trans. Image Process. 2021, 30, 6184–6197. [Google Scholar] [CrossRef] [PubMed]
- Nie, L.; Lin, C.; Liao, K.; Liu, S.; Zhao, Y. Parallax-tolerant unsupervised deep image stitching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France, 2–6 October 2023; pp. 7399–7408. [Google Scholar]
- Moravec, H. Visual mapping by a robot rover. In Proceedings of the International Joint Conference on Artificial Intelligence, Tokyo, Japan, 20–23 August 1979; pp. 598–600. [Google Scholar]
- Harris, C.; Stephens, M. A combined corner and edge detector. In Proceedings of the Alvey Vision Conference, Manchester, UK, 31 August–2 September 1988; Volume 15, pp. 147–151. [Google Scholar] [CrossRef]
- Lowe, D.G. Object recognition from local scale-invariant features. In Proceedings of the Seventh IEEE International Conference on Computer Vision, Kerkyra, Greece, 20–27 September 1999; pp. 1150–1157. [Google Scholar]
- Lowe, D.G. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 2004, 60, 91–110. [Google Scholar] [CrossRef]
- Bay, H.; Tuytelaars, T.; Van Gool, L. Surf: Speeded up robust features. Lect. Notes Comput. Sci. 2006, 3951, 404–417. [Google Scholar]
- Bind, V.S.; Muduli, P.R.; Pati, U.C. A robust technique for feature-based image stitching using image fusion. Int. J. Adv. Comput. Res. 2013, 3, 263. [Google Scholar]
- Calonder, M.; Lepetit, V.; Ozuysal, M.; Trzcinski, T.; Strecha, C.; Fua, P. BRIEF: Computing a local binary descriptor very fast. IEEE Trans. Pattern Anal. Mach. Intell. 2011, 34, 1281–1298. [Google Scholar] [CrossRef]
- Biadgie, Y.; Sohn, K.A. Speed-up feature detector using adaptive accelerated segment test. IETE Tech. Rev. 2016, 33, 492–504. [Google Scholar] [CrossRef]
- Calonder, M.; Lepetit, V.; Strecha, C.; Fua, P. Brief: Binary robust independent elementary features. In Proceedings of the 11th European Conference on Computer Vision, Heraklion, Greece, 5–11 September 2010; pp. 778–792. [Google Scholar]
- Rublee, E.; Rabaud, V.; Konolige, K.; Bradski, G. ORB: An efficient alternative to SIFT or SURF. In Proceedings of the International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011; pp. 2564–2571. [Google Scholar]
- Leutenegger, S.; Chli, M.; Siegwart, R.Y. BRISK: Binary robust invariant scalable keypoints. In Proceedings of the International Conference on Computer Vision, Barcelona, Spain, 8–11 November 2011; pp. 2548–2555. [Google Scholar]
- Alcantarilla, P.F.; Bartoli, A.; Davison, A.J. KAZE features. In Proceedings of the European Conference on Computer Vision, Florence, Italy, 7–13 October 2012; pp. 214–227. [Google Scholar]
- Alcantarilla, P.F. Fast explicit diffusion for accelerated features in nonlinear scale spaces. In Proceedings of the British Machine Vision Conference, Bristol, UK, 9–13 September 2013; pp. 124–157. [Google Scholar]
- Weickert, J.; Romeny, B.H.; Viergever, M.A. Efficient and reliable schemes for nonlinear diffusion filtering. IEEE Trans. Image Process. 1998, 7, 353–398. [Google Scholar] [CrossRef] [PubMed]
- Zaragoza, J.; Chin, T.J.; Brown, M.S.; Brown, M.S.; Suter, D. As-projective-as-possible image stitching with moving DLT. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013; pp. 2339–2346. [Google Scholar]
- Li, S.; Yuan, L.; Sun, J.; Quan, L. Dual-feature warping-based motion model estimation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 7–13 December 2015; pp. 4283–4291. [Google Scholar]
- Liao, T.; Li, N. Single-perspective warps in natural image stitching. IEEE TIP 2019, 29, 724–735. [Google Scholar] [CrossRef] [PubMed]
- Jia, Q.; Li, Z.; Fan, X.; Zhao, H.; Teng, S.; Ye, X.; Latecki, L.J. Leveraging line-point consistence to preserve structures for wide parallax image stitching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021; pp. 12186–12195. [Google Scholar]
- Uyttendaele, M.; Eden, A.; Skeliski, R. Eliminating ghosting and exposure artifacts in image stitching. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Kauai, HI, USA, 8–14 December 2001; Volume 2. [Google Scholar]
- Davis, J. Mosaics of scenes with moving objects. In Proceedings of the 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Santa Barbara, CA, USA, 23–25 June 1998; pp. 354–360. [Google Scholar]
- Li, L.; Yao, J.; Lu, X.; Tu, J.; Shan, J. Optimal seamline detection for multiple image stitching via graph cuts. ISPRS J. Photogramm. Remote Sens. 2016, 113, 1–16. [Google Scholar] [CrossRef]
- Li, N.; Liao, T.; Wang, C. Perception-based seam cutting for image stitching. Signal Image Video Process. 2018, 12, 967–974. [Google Scholar] [CrossRef]
- Ai, Y.; Kan, J. Image Mosaicing Based on Improved Optimal Seam-Cutting. IEEE Access 2020, 8, 181526–181533. [Google Scholar] [CrossRef]
- Efros, A.A.; Freeman, W.T. Image quilting for texture synthesis and transfer. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, USA, 12–17 August 2001; pp. 341–346. [Google Scholar]
- Yu, L.; Holden, E.J.; Dentith, M.C.; Zhang, H. Towards the automatic selection of optimal seam line locations when merging optical remote-sensing images. Int. J. Remote Sens. 2012, 33, 1000–1014. [Google Scholar] [CrossRef]
- Herrmann, C.; Wang, C.; Bowen, R.S.; Keyder, E.; Zabih, R. Object-centered image stitching. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 821–835. [Google Scholar]
- Liao, T.; Chen, J.; Xu, Y. Quality evaluation-based iterative seam estimation for image stitching. Signal Image Video Process. 2019, 13, 1199–1206. [Google Scholar] [CrossRef]
- Lee, D.; Lee, S. Seamless image stitching by homography refinement and structure deformation using optimal seam pair detection. J. Electron. Imaging 2017, 26, 063016. [Google Scholar] [CrossRef]
- Ford, L.R., Jr.; Fulkerson, D.R. Flows in Networks; Princeton University Press: Princeton, NJ, USA, 2015. [Google Scholar]
- Boykov, Y.; Kolmogorov, V. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans. Pattern Anal. Mach. Intell. 2004, 26, 1124–1137. [Google Scholar] [CrossRef] [PubMed]
- Yuan, Y.; Fang, F.; Zhang, G. Superpixel-based seamless image stitching for UAV images. IEEE Trans. Geosci. Remote Sens. 2020, 59, 1565–1576. [Google Scholar] [CrossRef]
- Yi, K.M.; Trulls, E.; Lepetit, V.; Fua, P. Lift: Learned invariant feature transform. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 11–14 October 2016; Springer: Cham, Switzerland, 2016; pp. 467–483. [Google Scholar]
- DeTone, D.; Malisiewicz, T.; Rabinovich, A. Superpoint: Self-supervised interest point detection and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA, 18–22 June 2018; pp. 224–236. [Google Scholar]
- Sarlin, P.E.; DeTone, D.; Malisiewicz, T.; Rabinovich, A. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 4938–4947. [Google Scholar]
- Lindenberger, P.; Sarlin, P.E.; Pollefeys, M. Lightglue: Local feature matching at light speed. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France, 1–6 October 2023; pp. 17627–17638. [Google Scholar]
- Wu, H.; Zheng, S.; Zhang, J.; Huang, K. Gp-gan: Towards realistic high-resolution image blending. In Proceedings of the 27th ACM International Conference on Multimedia, Nice, France, 21–25 October 2019; pp. 2487–2495. [Google Scholar]
- Zheng, C.; Xia, S.; Robinson, J.; Lu, C.; Wu, W.; Qian, C.; Shao, M. Localin Reshuffle net: Toward naturally and efficiently facial image blending. In Proceedings of the Asian Conference on Computer Vision, Kyoto, Japan, 30 November–4 December 2020. [Google Scholar]
- Burt, P.J.; Adelson, E.H. A multiresolution spline with application to image stitchings. ACM Trans. Graph TOG 1983, 2, 217–236. [Google Scholar] [CrossRef]
- Zhang, H.; Zhang, J.; Perazzi, F.; Lin, Z.; Patel, V.M. Deep image compositing. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Virtual, 5–9 January 2021; pp. 365–374. [Google Scholar]
- Lu, C.N.; Chang, Y.C.; Chiu, W.C. Bridging the visual gap: Wide-range image blending. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 843–851. [Google Scholar] [CrossRef]
- Lin, K.; Jiang, N.; Cheong, L.-F.; Do, M.; Lu, J. SEAGULL: Seam-Guided Local Alignment for Parallax-Tolerant Image Stitching. In Proceedings of the 14th European Conference, Lecture Notes in Computer Science, Amsterdam, The Netherlands, 11–14 October 2016; pp. 370–385. [Google Scholar] [CrossRef]
- Balntas, V.; Lenc, K.; Vedaldi, A.; Mikolajczyk, K. HPatches: A benchmark and evaluation of handcrafted and learned local descriptors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 5173–5182. [Google Scholar]
- Brown, M.; Lowe, D.G. Automatic panoramic image stitching using invariant feature. Int. J. Comput. Vis. 2007, 74, 59–73. [Google Scholar] [CrossRef]
AutoStitch | UDIS | SuperUDIS | |
---|---|---|---|
PSNR↑ | 9.999 | 12.980 | 13.495 |
SSIM↑ | 0.4378 | 0.4362 | 0.4545 |
AutoStitch | UDIS | SuperUDIS | |
---|---|---|---|
PSNR↑ | 9.947 | 10.091 | 10.200 |
SSIM↑ | 0.4174 | 0.4424 | 0.4430 |
Method | PSNR | SSIM | ||
---|---|---|---|---|
Superpoint and Lightglue | Chroma Balance | Improved UDIS | ||
√ | 13.291 | 0.4417 | ||
√ | 12.960 | 0.4399 | ||
√ | 13.044 | 0.4426 | ||
√ | √ | 13.475 | 0.4541 | |
√ | √ | √ | 13.495 | 0.4545 |
Method | PSNR | SSIM | ||
---|---|---|---|---|
Superpoint and Lightglue | Chroma Balance | Improved UDIS | ||
√ | 9.991 | 0.4294 | ||
√ | 10.060 | 0.4365 | ||
√ | 10.086 | 0.4423 | ||
√ | √ | 10.054 | 0.4383 | |
√ | √ | √ | 10.200 | 0.4430 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wu, H.; Bao, C.; Hao, Q.; Cao, J.; Zhang, L. Improved Unsupervised Stitching Algorithm for Multiple Environments SuperUDIS. Sensors 2024, 24, 5352. https://doi.org/10.3390/s24165352
Wu H, Bao C, Hao Q, Cao J, Zhang L. Improved Unsupervised Stitching Algorithm for Multiple Environments SuperUDIS. Sensors. 2024; 24(16):5352. https://doi.org/10.3390/s24165352
Chicago/Turabian StyleWu, Haoze, Chun Bao, Qun Hao, Jie Cao, and Li Zhang. 2024. "Improved Unsupervised Stitching Algorithm for Multiple Environments SuperUDIS" Sensors 24, no. 16: 5352. https://doi.org/10.3390/s24165352