An Improved 3D Reconstruction Method for Satellite Images Based on Generative Adversarial Network Image Enhancement
Abstract
:1. Introduction
- A new improvement method for automatic 3D reconstruction of satellite images is proposed, which can generate high-quality reconstruction results without any ground control points.
- The perceptual loss function is applied to GAN image enhancement to improve the clarity of satellite images, further improving the quality of 3D reconstruction results of satellite images.
2. RPC Model and Generative Adversarial Network
2.1. RPC Model
2.2. Generative Adversarial Network
3. The Improved 3D Reconstruction Method of Satellite Images
3.1. GAN-Based Image Enhancement
3.2. Stereo Rectification
3.3. Stereo Matching
3.4. Altitude Estimation
4. Experimental Results and Discussion
4.1. Dataset and Metrics
4.2. Performance Analysis of the Proposed Method
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Zhang, K.; Snavely, N.; Sun, J. Leveraging vision reconstruction pipelines for satellite imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, Seoul, Republic of Korea, 27–28 October 2019. [Google Scholar]
- Qin, R.; Song, S.; Ling, X.; Elhashash, M. 3D reconstruction through fusion of cross-view images. In Proceedings of the Recent Advances in Image Restoration with Applications to Real World Problems, London, UK, 4 November 2022. [Google Scholar]
- Wang, P.; Shi, L.; Chen, B.; Hu, Z.; Qiao, J.; Dong, Q. Pursuing 3-D scene structures with optical satellite images from affine reconstruction to Euclidean reconstruction. IEEE Trans. Geosci. Remote 2022, 60, 1–14. [Google Scholar] [CrossRef]
- Facciolo, G.; De Franchis, C.; Meinhardt-Llopis, E. Automatic 3D reconstruction from multi-date satellite images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA, 21–26 July 2017. [Google Scholar]
- Wang, M.; Hu, F.; Li, J. Epipolar arrangement of satellite imagery by projection trajectory simplification. Photogramm. Rec. 2010, 25, 422–436. [Google Scholar] [CrossRef]
- Oh, J.; Lee, W.H.; Toth, C.K.; Grejner-Brzezinska, D.A.; Lee, C. A piecewise approach to epipolar resampling of pushbroom satellite images based on RPC. Photogramm. Eng. Rem. Sens. 2010, 76, 1353–1363. [Google Scholar] [CrossRef]
- De Franchis, C.; Meinhardt-Llopis, E.; Michel, J.; Morel, J.-M.; Facciolo, G. Automatic sensor orientation refinement of Pléiades stereo images. In Proceedings of the IEEE Geoscience and Remote Sensing Symposium, Quebec City, QC, Canada, 13–18 July 2014. [Google Scholar]
- Ghuffar, S. Satellite stereo based digital surface model generation using semi global matching in object and image space. ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. 2016, 3, 63–68. [Google Scholar] [CrossRef]
- Hirschmuller, H. Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. 2007, 30, 328–341. [Google Scholar] [CrossRef] [PubMed]
- Tatar, N.; Saadatseresht, M.; Arefi, H.; Hadavand, A. Quasi-epipolar resampling of high resolution satellite stereo imagery for semi global matching. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2015, 40, 707–712. [Google Scholar] [CrossRef]
- Ye, L.; Peng, M.; Di, K.; Liu, B.; Wang, Y. Lunar terrain reconstruction from multi-view Lroc Nac images based on semi-global matching in object space. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2020, 43, 1177–1183. [Google Scholar] [CrossRef]
- Facciolo, G.; De Franchis, C.; Meinhardt, E. MGM: A significantly more global matching for stereovision. In Proceedings of the British Machine Vision Conference 2015, Swansea, UK, 7–10 September 2015. [Google Scholar]
- Qayyum, A.; Malik, A.S.; Saad, M.N.B.M.; Abdullah, F.; Iqbal, M. Disparity map estimation based on optimization algorithms using satellite stereo imagery. In Proceedings of the IEEE International Conference on Signal and Image Processing Applications, Kuala Lumpur, Malaysia, 19–21 October 2015. [Google Scholar]
- Bleyer, M.; Rhemann, C.; Rother, C. Patchmatch stereo-stereo matching with slanted support windows. In Proceedings of the British Machine Vision Conference 2011, Dundee, UK, 29 August–2 September 2011. [Google Scholar]
- Zbontar, J.; LeCun, Y. Computing the stereo matching cost with a convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015. [Google Scholar]
- Poggi, M.; Mattoccia, S. Learning a general-purpose confidence measure based on O(1) features and a smarter aggregation strategy for semi global matching. In Proceedings of the Fourth International Conference on 3D Vision, Stanford, CA, USA, 25–28 October 2016. [Google Scholar]
- Bosch, M.; Kurtz, Z.; Hagstrom, S.; Brown, M. A multiple view stereo benchmark for satellite imagery. In Proceedings of the IEEE Applied Imagery Pattern Recognition Workshop, Washington, DC, USA, 18–20 October 2016. [Google Scholar]
- Marí, R.; De Franchis, C.; Meinhardt-Llopis, E.; Anger, J.; Facciolo, G. A generic bundle adjustment methodology for indirect RPC model refinement of satellite imagery. Image Process. Line 2021, 11, 344–373. [Google Scholar] [CrossRef]
- Qin, R. A critical analysis of satellite stereo pairs for digital surface model generation and a matching quality prediction model. ISPRS J. Photogramm. 2019, 154, 139–150. [Google Scholar] [CrossRef]
- Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, PQ, Canada, 8–13 December 2014. [Google Scholar]
- Ledig, C.; Theis, L.; Huszár, F.; Caballero, J.; Cunningham, A.; Acosta, A.; Aitken, A.; Tejani, A.; Totz, J.; Wang, Z.; et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017. [Google Scholar]
- Ignatov, A.; Kobyshev, N.; Timofte, R.; Vanhoey, K.; Van Gool, L. WESPE: Weakly supervised photo enhancer for digital cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA, 18–22 June 2018. [Google Scholar]
- Sun, X.; Li, M.; He, T.; Fan, L. Enhance images as you like with unpaired learning. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, Online, 19–26 August 2021. [Google Scholar]
- Jiang, Y.; Gong, X.; Liu, D.; Cheng, Y.; Fang, C.; Shen, X.; Yang, J.; Zhou, P.; Wang, Z. EnlightenGAN: Deep light enhancement without paired supervision. IEEE Trans. Image Process. 2021, 30, 2340–2349. [Google Scholar] [CrossRef] [PubMed]
- Jiang, L.; Zhang, C.; Huang, M.; Liu, C.; Shi, J.; Loy, C.C. TSIT: A simple and versatile framework for image-to-image translation. In Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2020. [Google Scholar]
- Ni, Z.; Yang, W.; Wang, S.; Ma, L.; Kwong, S. Towards unsupervised deep image enhancement with generative adversarial network. IEEE Trans. Image Process. 2020, 29, 9140–9151. [Google Scholar] [CrossRef] [PubMed]
- Fife, W.S.; Archibald, J.K. Improved census transforms for resource-optimized stereo vision. IEEE Trans. Circ. Syst. Vid. 2012, 23, 60–73. [Google Scholar] [CrossRef]
- Jie, Z.; Wang, P.; Ling, Y.; Zhao, B.; Wei, Y.; Feng, J.; Liu, W. Left-right comparative recurrent model for stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018. [Google Scholar]
- Bosch, M.; Leichtman, A.; Chilcott, D.; Goldberg, H.; Brown, M. Metric evaluation pipeline for 3D modeling of urban scenes. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2017, 42, 239–246. [Google Scholar] [CrossRef]
Site | Method | CP (%) | RMSE (m) | ME (m) |
---|---|---|---|---|
1 | S2P method [7] | 73.98 | 2.60 | 0.39 |
BA method [18] | 72.60 | 2.68 | 0.65 | |
Proposed method | 74.69 | 2.47 | 0.42 | |
2 | S2P method [7] | 60.77 | 2.74 | 0.57 |
BA method [18] | 60.66 | 2.64 | 0.55 | |
Proposed method | 65.08 | 2.22 | 0.50 | |
3 | S2P method [7] | 67.21 | 8.87 | 0.35 |
BA method [18] | 66.91 | 3.99 | 0.47 | |
Proposed method | 68.88 | 5.58 | 0.34 | |
4 | S2P method [7] | 50.21 | 11.14 | 0.98 |
BA method [18] | 42.04 | 9.34 | 1.45 | |
Proposed method | 51.50 | 10.74 | 0.89 | |
5 | S2P method [7] | 71.19 | 1.92 | 0.52 |
BA method [18] | 68.61 | 2.31 | 0.46 | |
Proposed method | 71.45 | 1.88 | 0.51 | |
6 | S2P method [7] | 68.57 | 2.17 | 0.53 |
BA method [18] | 53.02 | 3.74 | 0.90 | |
Proposed method | 68.74 | 2.17 | 0.54 | |
7 | S2P method [7] | 59.20 | 7.56 | 0.74 |
BA method [18] | 54.71 | 6.37 | 0.82 | |
Proposed method | 58.67 | 4.97 | 0.74 | |
8 | S2P method [7] | 63.12 | 5.03 | 0.47 |
BA method [18] | 62.66 | 3.80 | 0.66 | |
Proposed method | 63.34 | 4.85 | 0.48 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Li, H.; Yin, J.; Jiao, L. An Improved 3D Reconstruction Method for Satellite Images Based on Generative Adversarial Network Image Enhancement. Appl. Sci. 2024, 14, 7177. https://doi.org/10.3390/app14167177
Li H, Yin J, Jiao L. An Improved 3D Reconstruction Method for Satellite Images Based on Generative Adversarial Network Image Enhancement. Applied Sciences. 2024; 14(16):7177. https://doi.org/10.3390/app14167177
Chicago/Turabian StyleLi, Henan, Junping Yin, and Liguo Jiao. 2024. "An Improved 3D Reconstruction Method for Satellite Images Based on Generative Adversarial Network Image Enhancement" Applied Sciences 14, no. 16: 7177. https://doi.org/10.3390/app14167177