Hyperspectral Image Super-Resolution Based on Spatial Group Sparsity Regularization Unmixing
Abstract
:1. Introduction
- We propose an HSI super-resolution approach by simultaneously unmixing the two input images into their endmembers and associated abundances.
- The advantage of the proposed unmixing lies in taking the spatial correlation, as well as noise and outliers, into consideration and providing a solution that combines SLIC superpixels and robust sparse unmixing.
- We test our approach with a widely used standard benchmark, the “Harvard data set”, and several remotely sensed hyperspectral images. The results of the experiments demonstrate that the proposed approach is superior to other related state-of-the-art methods.
2. Related Work
3. Proposed Method
3.1. Problem Formulation
3.2. Problem Solution
4. Experiments
4.1. Data Sets and Quantitative Metrics
4.2. Experimental Setting
5. Results and Analysis
5.1. Hyperspectral Unmixing
5.2. Super-Resolution
5.3. Impact on Classification
5.4. Computational Cost
6. Conclusions
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
Abbreviations
HSI | Hyperspectral image |
MSI | Multispectral image |
HS-MS image fusion | Hyperspectral and multispectral image fusion |
SLIC | Simple linear iterative clustering |
ADMM | Alternative direction method of multipliers |
MRA | Multi-resolution analysis |
NMF | Non-negative matrix factorization |
SRF | Spectral response functions |
PSF | Point spread functions |
LTTR | Low tensor-train rank |
CNN | convolutional neural network |
KNN | K-nearest neighbors |
References
- Yokoya, N.; Grohnfeldt, C.; Chanussot, J. Hyperspectral and multispectral data fusion: A comparative review of the recent literature. IEEE Geosci. Remote Sens. Mag. 2017, 5, 29–56. [Google Scholar] [CrossRef]
- Sankararaman, S.P. Tissue Characterization by Deep Learning in Medical Hyperspectral Images; Delft University of Technology: Delft, The Netherland, 2019. [Google Scholar]
- Liu, Y.; Tao, Z.; Zhang, J.; Hao, H.; Peng, Y.; Hou, J.; Jiang, T. Deep-Learning-Based Active Hyperspectral Imaging Classification Method Illuminated by the Supercontinuum Laser. Appl. Sci. 2020, 10, 3088. [Google Scholar] [CrossRef]
- Liu, Y.; Su, M.; Liu, L.; Li, C.; Peng, Y.; Hou, J.; Jiang, T. Deep residual prototype learning network for hyperspectral image classification. In Proceedings of the Second Target Recognition and Artificial Intelligence Summit Forum, Tirana, Albania, 11–13 December 2020; Volume 11427, p. 1142705. [Google Scholar]
- Uzkent, B.; Rangnekar, A.; Hoffman, M. Aerial vehicle tracking by adaptive fusion of hyperspectral likelihood maps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA, 21–26 July 2017; pp. 39–48. [Google Scholar]
- Bitar, A.W.; Cheong, L.F.; Ovarlez, J.P. Sparse and low-rank matrix decomposition for automatic target detection in hyperspectral imagery. IEEE Trans. Geosci. Remote Sens. 2019, 57, 5239–5251. [Google Scholar] [CrossRef] [Green Version]
- Akgun, T.; Altunbasak, Y.; Mersereau, R.M. Super-resolution reconstruction of hyperspectral images. IEEE Trans. Image Process. 2005, 14, 1860–1875. [Google Scholar] [CrossRef] [PubMed]
- Li, C.; Sun, T.; Kelly, K.F.; Zhang, Y. A compressive sensing and unmixing scheme for hyperspectral data processing. IEEE Trans. Image Process. 2011, 21, 1200–1210. [Google Scholar]
- Gomez, R.B.; Jazaeri, A.; Kafatos, M. Wavelet-based hyperspectral and multispectral image fusion. In Proceedings of the Geo-Spatial Image and Data Exploitation II, Orlando, FL, USA, 1 June 2001; Volume 4383, pp. 36–42. [Google Scholar]
- Zhang, Y.; He, M. Multi-spectral and hyperspectral image fusion using 3D wavelet transform. J. Electron. (China) 2007, 24, 218–224. [Google Scholar] [CrossRef]
- Chen, Z.; Pu, H.; Wang, B.; Jiang, G.M. Fusion of hyperspectral and multispectral images: A novel framework based on generalization of pan-sharpening methods. IEEE Geosci. Remote Sens. Lett. 2014, 11, 1418–1422. [Google Scholar] [CrossRef]
- Berné, O.; Helens, A.; Pilleri, P.; Joblin, C. Non-negative matrix factorization pansharpening of hyperspectral data: An application to mid-infrared astronomy. In Proceedings of the IEEE 2010 2nd Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing, Amsterdam, The Netherlands, 30 September–2 October 2010; pp. 1–4. [Google Scholar]
- Yokoya, N.; Yairi, T.; Iwasaki, A. Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Trans. Geosci. Remote Sens. 2011, 50, 528–537. [Google Scholar] [CrossRef]
- Dian, R.; Fang, L.; Li, S. Hyperspectral image super-resolution via non-local sparse tensor factorization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 5344–5353. [Google Scholar]
- Li, S.; Dian, R.; Fang, L.; Bioucas-Dias, J.M. Fusing hyperspectral and multispectral images via coupled sparse tensor factorization. IEEE Trans. Image Process. 2018, 27, 4118–4130. [Google Scholar] [CrossRef]
- Qu, Y.; Qi, H.; Kwan, C. Unsupervised sparse dirichlet-net for hyperspectral image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 2511–2520. [Google Scholar]
- Xie, Q.; Zhou, M.; Zhao, Q.; Meng, D.; Zuo, W.; Xu, Z. Multispectral and hyperspectral image fusion by MS/HS fusion net. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Alexandria, Egypt, 10 January 2019; pp. 1585–1594. [Google Scholar]
- Dobigeon, N.; Tourneret, J.Y.; Richard, C.; Bermudez, J.C.M.; McLaughlin, S.; Hero, A.O. Nonlinear unmixing of hyperspectral images: Models and algorithms. IEEE Signal Process. Mag. 2013, 31, 82–94. [Google Scholar] [CrossRef] [Green Version]
- Imbiriba, T.; Borsoi, R.A.; Bermudez, J.C.M. Generalized linear mixing model accounting for endmember variability. In Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada, 15–20 April 2018; pp. 1862–1866. [Google Scholar]
- Huang, B.; Song, H.; Cui, H.; Peng, J.; Xu, Z. Spatial and spectral image fusion using sparse matrix factorization. IEEE Trans. Geosci. Remote Sens. 2013, 52, 1693–1704. [Google Scholar] [CrossRef]
- Selva, M.; Aiazzi, B.; Butera, F.; Chiarantini, L.; Baronti, S. Hyper-sharpening: A first approach on SIM-GA data. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2015, 8, 3008–3024. [Google Scholar] [CrossRef]
- Akhtar, N.; Shafait, F.; Mian, A. Bayesian sparse representation for hyperspectral image super resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 3631–3640. [Google Scholar]
- Lanaras, C.; Baltsavias, E.; Schindler, K. Hyperspectral super-resolution by coupled spectral unmixing. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 3586–3594. [Google Scholar]
- Akhtar, N.; Shafait, F.; Mian, A. Sparse spatio-spectral representation for hyperspectral image super-resolution. In Proceedings of the European Conference on Computer Vision, Zurich, Switzerland, 6–12 September 2014; Springer: Berlin/Heidelberg, Germany, 2014; pp. 63–78. [Google Scholar]
- Wei, Q.; Bioucas-Dias, J.; Dobigeon, N.; Tourneret, J.Y.; Chen, M.; Godsill, S. Multiband image fusion based on spectral unmixing. IEEE Trans. Geosci. Remote Sens. 2016, 54, 7236–7249. [Google Scholar] [CrossRef] [Green Version]
- Zhang, L.; Wei, W.; Bai, C.; Gao, Y.; Zhang, Y. Exploiting clustering manifold structure for hyperspectral imagery super-resolution. IEEE Trans. Image Process. 2018, 27, 5969–5982. [Google Scholar] [CrossRef] [PubMed]
- Yi, C.; Zhao, Y.Q.; Chan, J.C.W.; Kong, S.G. Joint Spatial-spectral Resolution Enhancement of Multispectral Images with Spectral Matrix Factorization and Spatial Sparsity Constraints. Remote Sens. 2020, 12, 993. [Google Scholar] [CrossRef] [Green Version]
- Saragadam, V.; Sankaranarayanan, A.C. KRISM—Krylov subspace-based optical computing of hyperspectral images. ACM Trans. Graph. (TOG) 2019, 38, 1–14. [Google Scholar] [CrossRef] [Green Version]
- Zhang, K.; Wang, M.; Yang, S.; Jiao, L. Spatial–spectral-graph-regularized low-rank tensor decomposition for multispectral and hyperspectral image fusion. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2018, 11, 1030–1040. [Google Scholar] [CrossRef]
- Bengua, J.A.; Phien, H.N.; Tuan, H.D.; Do, M.N. Efficient tensor completion for color image and video recovery: Low-rank tensor train. IEEE Trans. Image Process. 2017, 26, 2466–2479. [Google Scholar] [CrossRef] [Green Version]
- Dian, R.; Li, S.; Fang, L. Learning a low tensor-train rank representation for hyperspectral image super-resolution. IEEE Trans. Neural Netw. Learn. Syst. 2019, 30, 2672–2683. [Google Scholar] [CrossRef]
- Zhang, T.; Fu, Y.; Wang, L.; Huang, H. Hyperspectral Image Reconstruction Using Deep External and Internal Learning. In Proceedings of the IEEE International Conference on Computer Vision, Tokyo, Japan, 12–15 April 2019; pp. 8559–8568. [Google Scholar]
- Ma, Y.; Li, C.; Mei, X.; Liu, C.; Ma, J. Robust Sparse Hyperspectral Unmixing With ell_{2, 1} Norm. IEEE Trans. Geosci. Remote Sens. 2016, 55, 1227–1239. [Google Scholar] [CrossRef]
- Achanta, R.; Shaji, A.; Smith, K.; Lucchi, A.; Fua, P.; Süsstrunk, S. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 2012, 34, 2274–2282. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Wei, Q.; Dobigeon, N.; Tourneret, J.Y. Fast fusion of multi-band images based on solving a Sylvester equation. IEEE Trans. Image Process. 2015, 24, 4109–4121. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Chakrabarti, A.; Zickler, T. Statistics of real-world hyperspectral images. In Proceedings of the IEEE CVPR 2011, Seattle, WA, USA, 21–23 June 2011; pp. 193–200. [Google Scholar]
- Garzelli, A.; Nencini, F. Hypercomplex quality assessment of multi/hyperspectral images. IEEE Geosci. Remote Sens. Lett. 2009, 6, 662–665. [Google Scholar] [CrossRef]
- Bioucas-Dias, J.M.; Figueiredo, M.A. Alternating direction algorithms for constrained sparse regression: Application to hyperspectral unmixing. In Proceedings of the IEEE 2010 2nd Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing, Reykjavik, Iceland, 14–16 June 2010; pp. 1–4. [Google Scholar]
- Iordache, M.D.; Bioucas-Dias, J.M.; Plaza, A. Collaborative sparse regression for hyperspectral unmixing. IEEE Trans. Geosci. Remote Sens. 2013, 52, 341–354. [Google Scholar] [CrossRef] [Green Version]
- Iordache, M.D.; Bioucas-Dias, J.M.; Plaza, A. Total variation regulatization in sparse hyperspectral unmixing. In Proceedings of the IEEE 2011 3rd Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS), Cambridge, MA, USA, 6–9 June 2011; pp. 1–4. [Google Scholar]
- Wang, X.; Zhong, Y.; Zhang, L.; Xu, Y. Spatial group sparsity regularized nonnegative matrix factorization for hyperspectral unmixing. IEEE Trans. Geosci. Remote. Sens. 2017, 55, 6287–6304. [Google Scholar] [CrossRef]
- Ma, L.; Crawford, M.M.; Tian, J. Local manifold learning-based k-nearest-neighbor for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 2010, 48, 4099–4109. [Google Scholar] [CrossRef]
Data Sets | CUPRITE | CHIKUSEI | INDIAN PINES | PAVIA | HOUSTON |
---|---|---|---|---|---|
Spectral range () | 0.4–2.5 | 0.36–1.02 | 0.4–2.5 | 0.43–0.84 | 0.36–1.05 |
Total bands | 224 | 128 | 224 | 115 | 144 |
Used bands | 185 | 128 | 192 | 103 | 144 |
Total image size | 512 × 614 | 2517 × 2335 | 512 × 614 | 610 × 340 | 349 × 1905 |
Used image size | 420 × 360 | 540 × 320 | 360 × 360 | 560 × 320 | 320 × 540 |
Ground Sampling Distance | 20 | 2.5 | 20 | 1.3 | 2.5 |
SNR | 35 | 35 | 35 | 35 | 35 |
Data Sets | CUPRITE | CHIKUSEI | INDIAN PINES | PAVIA | HOUSTON |
---|---|---|---|---|---|
Bands of ground truth images | 185 | 128 | 192 | 103 | 144 |
Bands of conventional images | 16 | 8 | 16 | 4 | 4 |
Size of ground truth images | 420 × 360 | 540 × 420 | 360 × 360 | 560 × 320 | 320 × 540 |
Size of hyperspectral images /Scale factor | 84 × 72 /5 | 90 × 70 /6 | 90 × 90 /4 | 70 × 40 /8 | 64 × 108 /5 |
Methods | FUSE | BSR | CSU | CMS | LTTR | Ours |
---|---|---|---|---|---|---|
SSIM | 0.99902 ± 7.74 × 10 | 0.99943 ±6.9 | 0.99989 ±1.84× 10 | 0.99995 ±1.13 | 0.99991 ±1.23 | 0.99996 ±3.61 |
RMSE | 0.48136 ±0.13070 | 0.28558 ±0.078466 | 0.23279 ±0.07063 | 0.22475 ±0.06109 | 0.24432 ±0.06109 | 0.21031 ±0.02689 |
PSNR | 29.56316 ±4.23725 | 38.62009 ±5.45638 | 41.01761 ±5.27647 | 41.81752 ±5.81294 | 40.41452 ±5.52129 | 42.74880 ±2.89850 |
SAM | 2.95474 ±0.793651 | 4.21175 ±1.31910 | 2.62583 ±0.86600 | 2.44085 ±0.75043 | 2.69322 ±0.76777 | 2.25826 ±0.58335 |
ERGAS | 2.78576 ±1.21321 | 1.38858 ±0.69591 | 1.08028 ±0.71299 | 0.91274 ±0.47246 | 1.02112 ±0.49095 | 0.86787 ±0.52625 |
0.53836 ±0.10412 | 0.75601 ±0.09046 | 0.81238 ±0.06793 | 0.80453 ±0.06697 | 0.78496 ±0.07007 | 0.82477 ±0.05305 | |
Time (s) | 10.66 | 8157.12 | 1358.69 | 1388.09 | 1631.74 | 2369.83 |
Methods | FUSE | BSR | CSU | CMS | LTTR | Ours | |
---|---|---|---|---|---|---|---|
CUPRITE | SSIM | 0.83747 | 0.98963 | 0.99192 | 0.97348 | 0.95545 | 0.99238 |
RMSE | 4.9375 | 1.15290 | 1.0562 | 2.08420 | 2.15130 | 0.98903 | |
PSNR | 31.6723 | 41.1775 | 41.8221 | 37.0663 | 40.8065 | 42.6838 | |
SAM | 3.635 | 0.76047 | 0.65195 | 0.88845 | 0.93718 | 0.62051 | |
ERGAS | 1.2788 | 0.3173 | 0.29971 | 0.54418 | 0.55328 | 0.26375 | |
0.80201 | 0.97817 | 0.97233 | 0.89626 | 0.97452 | 0.97993 | ||
Time (s) | 1.29 | 2022.60 | 111.01 | 206.09 | 340.67 | 131.65 | |
CHIKUSEI | SSIM | 0.99651 | 0.99575 | 0.99603 | 0.98038 | 0.99215 | 0.99706 |
RMSE | 1.1972 | 1.2076 | 1.1823 | 2.9822 | 1.7675 | 1.0371 | |
PSNR | 45.4159 | 43.9835 | 42.4042 | 38.7097 | 42.6187 | 44.4753 | |
SAM | 1.4699 | 1.5546 | 1.3871 | 2.2058 | 2.0718 | 1.2426 | |
ERGAS | 1.6222 | 1.8631 | 1.7196 | 1.7487 | 1.6666 | 1.6081 | |
0.91975 | 0.94963 | 0.91864 | 0.86428 | 0.87594 | 0.92592 | ||
Time (s) | 2.91 | 2235.74 | 233.58 | 203.61 | 720.26 | 265.67 | |
INDIAN PINES | SSIM | 0.90796 | 0.98916 | 0.98807 | 0.97736 | 0.97826 | 0.99075 |
RMSE | 9.5287 | 1.3632 | 1.5157 | 2.2509 | 1.6519 | 1.3892 | |
PSNR | 35.5488 | 40.4208 | 41.3034 | 40.1867 | 41.0773 | 42.9987 | |
SAM | 6.4675 | 0.79846 | 0.88154 | 0.96104 | 0.83385 | 0.82111 | |
ERGAS | 1.7171 | 0.4197 | 0.41053 | 0.55839 | 0.58602 | 0.33396 | |
0.38834 | 0.76825 | 0.70134 | 0.81233 | 0.88700 | 0.71257 | ||
Time (s) | 2.80 | 1924.67 | 157.41 | 280.83 | 570.40 | 173.08 | |
UNIVERSITY OF PAVIA | SSIM | 0.9805 | 0.98122 | 0.98029 | 0.94498 | 0.93797 | 0.98206 |
RMSE | 2.0836 | 2.1192 | 2.3904 | 4.3442 | 4.5258 | 2.0834 | |
PSNR | 41.6695 | 41.6288 | 39.5331 | 35.8998 | 35.9253 | 41.0519 | |
SAM | 2.8931 | 3.0125 | 2.7325 | 4.3366 | 5.1832 | 2.6367 | |
ERGAS | 0.83523 | 0.86948 | 0.97139 | 1.5170 | 1.8383 | 0.8492 | |
0.88702 | 0.87856 | 0.80358 | 0.62222 | 0.59185 | 0.84299 | ||
Time (s) | 1.98 | 1648.0 | 175.0 | 210.96 | 424.48 | 203.73 | |
UNIVERSITY OF HOUSTON | SSIM | 0.95808 | 0.95452 | 0.96435 | 0.76379 | 0.64306 | 0.96456 |
RMSE | 7.6814 | 8.5580 | 7.3756 | 20.9834 | 24.4054 | 7.3900 | |
PSNR | 32.155 | 31.574 | 32.7052 | 23.8961 | 28.877 | 32.7105 | |
SAM | 2.2091 | 2.2500 | 2.0754 | 4.7053 | 6.2191 | 2.0917 | |
ERGAS | 3.4876 | 3.9151 | 3.255 | 8.2063 | 5.1517 | 3.2605 | |
0.90252 | 0.88463 | 0.90889 | 0.69074 | 0.65189 | 0.91488 | ||
Time (s) | 1.47 | 1904.26 | 145.68 | 93.79 | 638.94 | 114.72 |
Methods | Reference | FUSE | BSR | CSU | CMS | LTTR | Ours |
---|---|---|---|---|---|---|---|
OA | 72.06% | 70.39% | 70.07% | 71.43% | 69.84% | 70.82% | 71.84% |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Li, J.; Peng, Y.; Jiang, T.; Zhang, L.; Long, J. Hyperspectral Image Super-Resolution Based on Spatial Group Sparsity Regularization Unmixing. Appl. Sci. 2020, 10, 5583. https://doi.org/10.3390/app10165583
Li J, Peng Y, Jiang T, Zhang L, Long J. Hyperspectral Image Super-Resolution Based on Spatial Group Sparsity Regularization Unmixing. Applied Sciences. 2020; 10(16):5583. https://doi.org/10.3390/app10165583
Chicago/Turabian StyleLi, Jun, Yuanxi Peng, Tian Jiang, Longlong Zhang, and Jian Long. 2020. "Hyperspectral Image Super-Resolution Based on Spatial Group Sparsity Regularization Unmixing" Applied Sciences 10, no. 16: 5583. https://doi.org/10.3390/app10165583
APA StyleLi, J., Peng, Y., Jiang, T., Zhang, L., & Long, J. (2020). Hyperspectral Image Super-Resolution Based on Spatial Group Sparsity Regularization Unmixing. Applied Sciences, 10(16), 5583. https://doi.org/10.3390/app10165583