Local-Peak Scale-Invariant Feature Transform for Fast and Random Image Stitching
Abstract
:1. Introduction
2. Materials and Methods
2.1. Image Preprocessing
2.2. Feature Point Detection
2.3. Feature Point Description
2.4. Feature Point Matching
2.5. Image Stitching
3. Results of Stitching on Two Images
3.1. Datasets
3.2. Evaluation Metrics
3.3. Images of Small Size
3.4. Images of Medium Size
3.5. Images of Large Size with Translational Displacement
3.6. Images of Large Size with Rotational Displacement
3.7. Discussion
4. Mosaic of Multiple Images without Prior Knowledge
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Deshmukh, P.; Paikrao, P. A Review of Various Image Mosaicing Techniques. In Proceedings of the 2019 Innovations in Power and Advanced Computing Technologies (i-PACT), Vellore, India, 22–23 March 2019. [Google Scholar] [CrossRef]
- Lee, W.T.; Chen, H.I.; Chen, M.S.; Shen, I.C.; Chen, B.Y. High-resolution 360 Video Foveated Stitching for Real-Time VR. Comput. Graph. Forum 2017, 36, 115–123. [Google Scholar] [CrossRef]
- Saeed, S.; Kakli, M.U.; Cho, Y.; Seo, J.; Park, U. A High-Quality Vr Calibration and Real-Time Stitching Framework Using Preprocessed Features. IEEE Access 2020, 8, 190300–190311. [Google Scholar] [CrossRef]
- Dang, P.; Zhu, J.; Zhou, Y.; Rao, Y.; You, J.; Wu, J.; Zhang, M.; Li, W. A 3D-Panoramic Fusion Flood Enhanced Visualization Method for VR. Environ. Model. Softw. 2023, 169, 105810. [Google Scholar] [CrossRef]
- Liu, Z.; Chang, S. A Study of Digital Exhibition Visual Design Led by Digital Twin and VR Technology. Meas. Sens. 2024, 31, 100970. [Google Scholar] [CrossRef]
- Greibe, T.; Anhøj, T.A.; Johansen, L.S.; Han, A. Quality Control of Jeol Jbx-9500fsz E-Beam Lithography System in a Multi-User Laboratory. Microelectron. Eng. 2016, 155, 25–28. [Google Scholar] [CrossRef]
- Pan, J.; Liu, W.; Ge, P.; Li, F.; Shi, W.; Jia, L.; Qin, H. Real-Time Segmentation and Tracking of Excised Corneal Contour by Deep Neural Networks for Dalk Surgical Navigation. Comput. Methods Programs Biomed. 2020, 197, 105679. [Google Scholar] [CrossRef]
- Wang, Y.; Yang, J. Origin of Organic Matter Pore Heterogeneity in Oil Mature Triassic Chang-7 Mudstones, Ordos Basin, China. Int. J. Coal Geol. 2024, 283, 104458. [Google Scholar] [CrossRef]
- Zhou, Q.; Zhou, Z. Web-Based Mixed Reality Video Fusion with Remote Rendering. Virtual Real. Intell. Hardw. 2023, 5, 188–199. [Google Scholar] [CrossRef]
- He, Z.; He, Z.; Li, S.; Yu, Y.; Liu, K. A Ship Navigation Risk Online Prediction Model Based on Informer Network Using Multi-Source Data. Ocean Eng. 2024, 298, 117007. [Google Scholar] [CrossRef]
- Wang, B.; Gou, S.; Di, K.; Wan, W.; Peng, M.; Zhao, C.; Zhang, Y.; Xie, B. Rock Size-Frequency Distribution Analysis at the Zhurong Landing Site Based on Navigation and Terrain Camera Images along the Entire Traverse. Icarus 2024, 413, 116001. [Google Scholar] [CrossRef]
- Cao, M.; Zheng, L.; Jia, W.; Liu, X. Constructing Big Panorama from Video Sequence Based on Deep Local Feature. Image Vis. Comput. 2020, 101, 103972. [Google Scholar] [CrossRef]
- Lyu, W.; Zhou, Z.; Chen, L.; Zhou, Y. A Survey on Image and Video Stitching. Virtual Real. Intell. Hardw. 2019, 1, 55–83. [Google Scholar] [CrossRef]
- Wang, Q.; Reimeier, F.; Wolter, K. Efficient Image Stitching through Mobile Offloading. Electron. Notes Theor. Comput. Sci. 2016, 327, 125–146. [Google Scholar] [CrossRef]
- Torres, R.; Mahalingam, G.; Kapner, D.; Trautman, E.T.; Fliss, T.; Seshamani, S.; Perlman, E.; Young, R.; Kinn, S.; Buchanan, J.; et al. A Scalable and Modular Automated Pipeline for Stitching of Large Electron Microscopy Datasets. eLife 2022, 11, e76534. [Google Scholar] [CrossRef]
- Ma, B.; Zimmermann, T.; Rohde, M.; Winkelbach, S.; He, F.; Lindenmaier, W.; Dittmar, K.E.J. Use of Autostitch for Automatic Stitching of Microscope Images. Micron 2007, 38, 492–499. [Google Scholar] [CrossRef]
- Yang, F.; Deng, Z.-S.; Fan, Q.-H. A Method for Fast Automated Microscope Image Stitching. Micron 2013, 48, 17–25. [Google Scholar] [CrossRef] [PubMed]
- Yang, F.; He, Y.; Deng, Z.S.; Yan, A. Improvement of Automated Image Stitching System for DR X-ray Images. Comput. Biol. Med. 2016, 71, 108–114. [Google Scholar] [CrossRef] [PubMed]
- Seo, J.-H.; Yang, S.; Kang, M.-S.; Her, N.-G.; Nam, D.-H.; Choi, J.-H.; Kim, M.H. Automated Stitching of Microscope Images of Fluorescence in Cells with Minimal Overlap. Micron 2019, 126, 102718. [Google Scholar] [CrossRef] [PubMed]
- Lei, Z.; Liu, X.; Zhao, L.; Chen, L.; Li, Q.; Yuan, T.; Lu, W. A Novel 3D Stitching Method for WLI Based Large Range Surface Topography Measurement. Opt. Commun. 2016, 359, 435–447. [Google Scholar] [CrossRef]
- Yang, P.; Ye, S.-w.; Peng, Y.-f. Three-Dimensional Profile Stitching Measurement for Large Aspheric Surface during Grinding Process with Sub-Micron Accuracy. Precis. Eng. 2017, 47, 62–71. [Google Scholar] [CrossRef]
- Kim, W.Y.; Seo, B.W.; Lee, S.H.; Lee, T.G.; Kwon, S.; Chang, W.S.; Nam, S.-H.; Fang, N.X.; Kim, S.; Cho, Y.T. Quasi-Seamless Stitching for Large-Area Micropatterned Surfaces Enabled by Fourier Spectral Analysis of Moiré Patterns. Nat. Commun. 2023, 14, 2202. [Google Scholar] [CrossRef]
- Feng, A.; Vong, C.N.; Zhou, J.; Conway, L.S.; Zhou, J.; Vories, E.D.; Sudduth, K.A.; Kitchen, N.R. Developing an Image Processing Pipeline to Improve the Position Accuracy of Single UAV Images. Comput. Electron. Agric. 2023, 206, 107650. [Google Scholar] [CrossRef]
- Feng, S.; Gao, M.; Jin, X.; Zhao, T.; Yang, F. Fine-Grained Damage Detection of Cement Concrete Pavement Based on UAV Remote Sensing Image Segmentation and Stitching. Measurement 2024, 226, 113844. [Google Scholar] [CrossRef]
- Wang, X.; He, N.; Hong, C.; Wang, Q.; Chen, M. Improved Yolox-X Based Uav Aerial Photography Object Detection Algorithm. Image Vis. Comput. 2023, 135, 104697. [Google Scholar] [CrossRef]
- Zeng, W.; Deng, Q.; Zhao, X.; Li, D.; Min, X. A Method for Stitching Remote Sensing Images with Delaunay Triangle Feature Constraints. Geocarto Int. 2023, 38, 2285356. [Google Scholar] [CrossRef]
- Rui, T.; Hu, Y.; Yang, C.; Wang, D.; Liu, X. Research on Fast Natural Aerial Image Mosaic. Comput. Electr. Eng. 2021, 90, 107007. [Google Scholar] [CrossRef]
- Ghosh, D.; Kaabouch, N. A Survey on Image Mosaicing Techniques. J. Vis. Commun. Image Represent. 2016, 34, 1–11. [Google Scholar] [CrossRef]
- Ma, Z.; Liu, S. A Review of 3D Reconstruction Techniques in Civil Engineering and Their Applications. Adv. Eng. Inform. 2018, 37, 163–174. [Google Scholar] [CrossRef]
- Bonny, M.Z.; Uddin, M.S. Feature-Based Image Stitching Algorithms. In Proceedings of the 2016 International Workshop on Computational Intelligence (IWCI), Dhaka, Bangladesh, 12–13 December 2016; pp. 198–203. [Google Scholar]
- Harris, C.; Stephens, M. A Combined Corner and Edge Detector. In Proceedings of the Alvey Vision Conference 1988, Manchester, UK, 31 August–2 September 1988; pp. 23.21–23.26. [Google Scholar]
- Lowe, D.G. Distinctive Image Features from Scale-Invariant Keypoints. Int. J. Comput. Vis. 2004, 60, 91–110. [Google Scholar] [CrossRef]
- Rosten, E.; Drummond, T. Machine Learning for High-Speed Corner Detection. In Proceedings of the Computer Vision–ECCV 2006: 9th European Conference on Computer Vision, Graz, Austria, 7–13 May 2006; Part I 9. pp. 430–443. [Google Scholar]
- Bay, H.; Ess, A.; Tuytelaars, T.; Van Gool, L. Speeded-Up Robust Features (SURF). Comput. Vis. Image Underst. 2008, 110, 346–359. [Google Scholar] [CrossRef]
- Rublee, E.; Rabaud, V.; Konolige, K.; Bradski, G. ORB: An Efficient Alternative to SIFT or SURF. In Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011; pp. 2564–2571. [Google Scholar]
- Leutenegger, S.; Chli, M.; Siegwart, R.Y. Brisk: Binary Robust Invariant Scalable Keypoints. In Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011; pp. 2548–2555. [Google Scholar]
- Alcantarilla, P.F.; Bartoli, A.; Davison, A.J. Kaze Features. In Proceedings of the Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, 7–13 October 2012; Part VI 12. pp. 214–227. [Google Scholar]
- Pablo, F.; Alcantarilla, J.N.; Bartoli, A. Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces. In Proceedings of the British Machine Vision Conference, Bristol, UK, 9–13 September 2013. [Google Scholar]
- Li, G.; Li, T.; Li, F.; Zhang, C. NerveStitcher: Corneal Confocal Microscope Images Stitching with Neural Networks. Comput. Biol. Med. 2022, 151, 106303. [Google Scholar] [CrossRef] [PubMed]
- Zhu, F.; Li, J.; Zhu, B.; Li, H.; Liu, G. UAV Remote Sensing Image Stitching via Improved VGG16 Siamese Feature Extraction Network. Expert Syst. Appl. 2023, 229, 120525. [Google Scholar] [CrossRef]
- ul-Huda, N.; Ahmad, H.; Banjar, A.; Alzahrani, A.O.; Ahmad, I.; Naeem, M.S. Image Synthesis of Apparel Stitching Defects Using Deep Convolutional Generative Adversarial Networks. Heliyon 2024, 10, e26466. [Google Scholar] [CrossRef]
- Wu, Z.; Wu, H. Improved Sift Image Feature Matching Algorithm. In Proceedings of the 2022 2nd International Conference on Computer Graphics, Image and Virtualization (ICCGIV), Chongqing, China, 23–25 September 2022; pp. 223–226. [Google Scholar]
- Gan, W.; Wu, Z.; Wang, M.; Cui, X. Image Stitching Based on Optimized SIFT Algorithm. In Proceedings of the 2023 5th International Conference on Intelligent Control, Measurement and Signal Processing (ICMSP), Chengdu, China, 19–21 May 2023; pp. 1099–1102. [Google Scholar]
- Li, X.; Li, S. Image Registration Algorithm Based on Improved SIFT. In Proceedings of the 2023 4th International Conference on Electronic Communication and Artificial Intelligence (ICECAI), Guangzhou, China, 12–14 May 2023; pp. 264–267. [Google Scholar]
- Wang, L.; Peters, N. The Length-Scale Distribution Function of the Distance between Extremal Points in Passive Scalar Turbulence. J. Fluid Mech. 2006, 554, 457–475. [Google Scholar] [CrossRef]
- Peters, N.; Wang, L. Dissipation Element Analysis of Scalar Fields in Turbulence. Comptes Rendus Mécanique 2006, 334, 493–506. [Google Scholar] [CrossRef]
- Wang, L.P.; Huang, Y.X. Multi-Level Segment Analysis: Definition and Application in Turbulent Systems. J. Stat. Mech. Theory Exp. 2015, 2015, P06018. [Google Scholar] [CrossRef]
- Lowe, D.G. Object Recognition from Local Scale-Invariant Features. In Proceedings of the Seventh IEEE International Conference on Computer Vision, Kerkyra, Greece, 20–27 September 1999; pp. 1150–1157. [Google Scholar]
- Zhang, Y.; Xie, Y. Adaptive Clustering Feature Matching Algorithm Based on Sift and Ransac. In Proceedings of the 2021 2nd International Conference on Electronics, Communications and Information Technology (CECIT), Sanya, China, 27–29 December 2021; pp. 174–179. [Google Scholar]
- Fischler, M.A.; Bolles, R.C. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Commun. ACM 1981, 24, 381–395. [Google Scholar] [CrossRef]
- Wu, H. Image Stitching. Available online: https://github.com/haoningwu3639/ImageStitching (accessed on 22 June 2021).
- Zaragoza, J.; Chin, T.-J.; Brown, M.S.; Suter, D. As-Projective-as-Possible Image Stitching with Moving DLT. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013; pp. 2339–2346. [Google Scholar]
- Hernandez Zaragoza, J.C. As-Projective-as-Possible Image Stitching with Moving DLT. Ph.D. Thesis, The University of Adelaide, Adelaide, Australia, 2014. [Google Scholar]
- Vedaldi, A.; Fulkerson, B. Vlfeat: An Open and Portable Library of Computer Vision Algorithms. In Proceedings of the 18th ACM International Conference on Multimedia (MM ‘10), Firenze, Italy, 25–29 October 2010. [Google Scholar]
- Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef] [PubMed]
- Hore, A.; Ziou, D. Image Quality Metrics: PSNR vs. SSIM. In Proceedings of the 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey, 23–26 August 2010; pp. 2366–2369. [Google Scholar]
- Xu, S.; Xu, Y. Objective Evaluation Method of Fusion Performance for Remote Sensing Image Based on Matlab. Sci. Surv. Mapp. 2008, 33, 143–145. [Google Scholar] [CrossRef]
- Heilbronner, R.; Barrett, S. Image Analysis in Earth Sciences: Microstructures and Textures of Earth Materials; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2013; Volume 129. [Google Scholar]
- Zhang, D.; Jackson, W.; Dobie, G.; West, G.; MacLeod, C. Structure-from-Motion Based Image Unwrapping and Stitching for Small Bore Pipe Inspections. Comput. Ind. 2022, 139, 103664. [Google Scholar] [CrossRef]
- Chatterjee, S.; Issac, K.K. Viewpoint Planning and 3D Image Stitching Algorithms for Inspection of Panels. NDT E Int. 2023, 137, 102837. [Google Scholar] [CrossRef]
- Popovych, S.; Macrina, T.; Kemnitz, N.; Castro, M.; Nehoran, B.; Jia, Z.; Bae, J.A.; Mitchell, E.; Mu, S.; Trautman, E.T.; et al. Petascale pipeline for precise alignment of images from serial section electron microscopy. Nat. Commun. 2024, 15, 289. [Google Scholar] [CrossRef]
- Xie, R.; Yao, J.; Liu, K.; Lu, X.; Liu, Y.; Xia, M.; Zeng, Q. Automatic Multi-Image Stitching for Concrete Bridge Inspection by Combining Point and Line Features. Autom. Constr. 2018, 90, 265–280. [Google Scholar] [CrossRef]
- Zhu, W.; Liu, L.; Jiang, G.; Yin, S.; Wei, S. A 135-Frames/s 1080p 87.5-mw Binary-Descriptor-Based Image Feature Extraction Accelerator. IEEE Trans. Circuits Syst. Video Technol. 2015, 26, 1532–1543. [Google Scholar] [CrossRef]
- Zhang, X.; Sun, H.; Chen, S.; Zheng, N. VLSI Architecture Exploration of Guided Image Filtering for 1080P@ 60Hz Video Processing. IEEE Trans. Circuits Syst. Video Technol. 2016, 28, 230–241. [Google Scholar] [CrossRef]
- Bordallo-Lopez, M.; Silvén, O.; Tico, M.; Vehviläinen, M. Creating Panoramas on Mobile Phones. In Proceedings of the Computational Imaging V, San Jose, CA, USA, 29–31 January 2007; pp. 54–63. [Google Scholar]
- Xiong, Y.; Pulli, K. Fast Panorama Stitching for High-Quality Panoramic Images on Mobile Phones. IEEE Trans. Consum. Electron. 2010, 56, 298–306. [Google Scholar] [CrossRef]
- Wang, L.; Zhang, Y.; Wang, T.; Zhang, Y.; Zhang, Z.; Yu, Y.; Li, L.J.R.S. Stitching and Geometric Modeling Approach Based on Multi-Slice Satellite Images. Remote Sens. 2021, 13, 4663. [Google Scholar] [CrossRef]
- Huang, B.; Collins, L.M.; Bradbury, K.; Malof, J.M. Deep Convolutional Segmentation of Remote Sensing Imagery: A Simple and Efficient Alternative to Stitching Output Labels. In Proceedings of the IGARSS 2018—2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain, 22–27 July 2018; pp. 6899–6902. [Google Scholar]
- Ren, M.; Li, J.; Song, L.; Li, H.; Xu, T. MLP-Based Efficient Stitching Method for UAV Images. IEEE Geosci. Remote Sens. Lett. 2022, 19, 2503305. [Google Scholar] [CrossRef]
- Sansoni, G.; Trebeschi, M.; Docchio, F.J.S. State-of-the-Art and Applications of 3D Imaging Sensors in Industry, Cultural Heritage, Medicine, and Criminal Investigation. Sensors 2009, 9, 568–601. [Google Scholar] [CrossRef]
Hardware | Operation system | Windows 11 64-bit operating system |
Processor | Intel Core i9-12900 | |
Memory | 64 GB | |
Graphics card | NVIDIA GeForce RTX 3090 | |
Software | Platform | MATLAB 2021a |
Library | Computer Vision Toolbox 10.0 | |
Development environment | SIFT | MATLAB 2021a |
ORB | MATLAB 2021a | |
BRISK | MATLAB 2021a | |
SURF | MATLAB 2021a | |
LP-SIFT | MATLAB 2021a |
Name | Image Size | Algorithm | Number of Feature Points | Number of Matched Pairs | Stitching Time (s) | AG | SF | ||
---|---|---|---|---|---|---|---|---|---|
Mountain | Small image | 602 × 400 | SIFT | 1496 | 939 | 15 | 101.21 | 6.68 | 27.73 |
ORB | 10,505 | 6926 | 1132 | 0.71 | 6.60 | 27.34 | |||
BRISK | 1416 | 1122 | 75 | 1.30 | 6.51 | 27.20 | |||
SURF | 638 | 490 | 216 | 0.85 | 6.55 | 27.28 | |||
LP-SIFT | 487 | 493 | 31 | 1.16 | 6.54 | 27.28 | |||
Street view | Small image | 653 × 490 | SIFT | 1948 | 2726 | 23 | 226.62 | 6.58 | 23.96 |
ORB | 11,363 | 15,597 | 541 | 2.27 | 7.17 | 25.85 | |||
BRISK | 2523 | 4104 | 59 | 2.22 | 6.70 | 24.39 | |||
SURF | 933 | 1149 | 137 | 2.54 | 6.63 | 24.10 | |||
LP-SIFT | 811 | 812 | 59 | 2.05 | 7.00 | 25.23 | |||
Terrain | Medium image | 1024 × 768 | SIFT | 9495 | 10,368 | 134 | 1674.87 | 5.41 | 17.33 |
ORB | 3182 | 3182 | 2224 | 15.77 | 5.48 | 17.36 | |||
BRISK | 8149 | 8306 | 95 | 3.20 | 5.39 | 17.11 | |||
SURF | 2883 | 3037 | 204 | 5.16 | 5.49 | 17.75 | |||
LP-SIFT | 1847 | 1837 | 29 | 4.47 | 5.50 | 17.78 | |||
Building | Medium image | 1080 × 1920 | SIFT | × | × | × | >104 | × | × |
ORB | 107,612 | 108,452 | 9720 | 327.25 | 6.57 | 24.29 | |||
BRISK | 14,660 | 15,428 | 605 | 4.08 | 6.56 | 24.33 | |||
SURF | 6123 | 5985 | 1780 | 1.28 | 6.58 | 24.35 | |||
LP-SIFT | 532 | 484 | 17 | 2.03 | 6.47 | 24.11 | |||
Campus view (translation) | Large image | 3072 × 4096 | SIFT | × | × | × | >104 | × | × |
ORB | 1,025,750 | 927,050 | Over size | × | × | × | |||
BRISK | 104,981 | 94,657 | 3299 | 195.44 | 9.37 | 24.87 | |||
SURF | 27,790 | 25,465 | 6056 | 6.52 | 9.44 | 24.96 | |||
LP-SIFT | 418 | 403 | 29 | 4.49 | 9.37 | 24.88 | |||
Campus view (rotation) | Large image | 3072 × 4096 | SIFT | × | × | × | >104 | × | × |
ORB | 1,326,389 | 1,332,929 | Over size | × | × | × | |||
BRISK | 158,247 | 164,035 | Over size | × | × | × | |||
SURF | 47,568 | 47,678 | 11,293 | 11.42 | 10.42 | 24.26 | |||
LP-SIFT | 422 | 429 | 22 | 4.58 | 11.74 | 27.37 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Li, H.; Wang, L.; Zhao, T.; Zhao, W. Local-Peak Scale-Invariant Feature Transform for Fast and Random Image Stitching. Sensors 2024, 24, 5759. https://doi.org/10.3390/s24175759
Li H, Wang L, Zhao T, Zhao W. Local-Peak Scale-Invariant Feature Transform for Fast and Random Image Stitching. Sensors. 2024; 24(17):5759. https://doi.org/10.3390/s24175759
Chicago/Turabian StyleLi, Hao, Lipo Wang, Tianyun Zhao, and Wei Zhao. 2024. "Local-Peak Scale-Invariant Feature Transform for Fast and Random Image Stitching" Sensors 24, no. 17: 5759. https://doi.org/10.3390/s24175759