Hybrid Transformer and Convolution for Image Compressed Sensing
Abstract
:1. Introduction
- Combining Transformer and large kernel convolution, an optimization-inspired TC module is designed and its iterative process is used to construct a novel image CS unfolding framework, TCR-Net, which realizes the joint optimization of image sparse sampling and reconstruction.
- In order to extract feature information more completely, an information transfer path is built between neighboring TC modules. Meanwhile, a new dual-channel large kernel attention mechanism (Dual-LKA) is proposed to deal with contextual information efficiently while retaining local descriptions, which integrates the advantages of convolution and self-attention, and avoids their drawbacks, and makes TCR-Net more suitable for CS reconstruction of images.
- Extensive experiments demonstrate that our proposed TCR-Net outperforms existing state-of-the-art CS methods while maintaining fast computational speed.
2. Related Work
2.1. Deep Unfolding Network
2.2. Transformer
3. Proposed Method
3.1. Overall Architecture
3.2. Architecture Design of TCR-Net
3.2.1. Sampling and Initial Reconstruction
3.2.2. Deep Reconstruction
Algorithm 1: TCR-Net for Image Compressed Sensing. |
|
3.3. Loss Function and Network Parameters
4. Experiment
4.1. Experimental Settings
4.1.1. Datasets and Performance Measures
4.1.2. Implementation Details
4.2. Comparisons with State-of-the-Art Methods
4.3. Ablation Studies and Discussions
4.3.1. Measurement Matrix
4.3.2. Dual-LKA
4.3.3. Sensitivity to Noise
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Donoho, D.L. Compressed sensing. IEEE Trans. Inf. Theory 2006, 52, 1289–1306. [Google Scholar] [CrossRef]
- Tian, X.; Wei, G.; Wang, J. Target location method based on compressed sensing in hidden semi Markov model. Electronics 2022, 11, 1715. [Google Scholar] [CrossRef]
- Fei, T.; Feng, X. Learing sampling and reconstruction using Bregman iteration for CS-MRI. Electronics 2023, 12, 4657. [Google Scholar] [CrossRef]
- Abedi, M.; Sun, B.; Zheng, Z. Single-pixel compressive imaging based on random DoG filtering. Signal Process. 2021, 178, 107746. [Google Scholar] [CrossRef]
- Zhao, C.; Ma, S.; Zhang, J.; Xiong, R.; Gao, W. Video compressive sensing reconstruction via reweighted residual sparsity. IEEE Trans. Circuits Syst. Video Technol. 2017, 27, 1182–1195. [Google Scholar] [CrossRef]
- Zhao, C.; Ma, S.; Gao, W. Image compressive-sensing recovery using structured laplacian sparsity in DCT domain and multi-hypothesis prediction. In Proceedings of the 2014 IEEE International Conference on Multimedia and Expo (ICME), Chengdu, China, 14–18 July 2014. [Google Scholar] [CrossRef]
- Zhao, C.; Zhang, J.; Ma, S.; Gao, W. Non-convex Lp Nuclear Norm based ADMM Framework for Compressed Sensing. In Proceedings of the 2016 Data Compression Conference (DCC), Snowbird, UT, USA, 30 March–1 April 2016; pp. 161–170. [Google Scholar] [CrossRef]
- Wang, B.; Lian, Y.; Xiong, X.; Zhou, H.; Liu, Z.; Das, M. Progressive feature reconstruction and fusion to accelerate MRI imaging: Exploring insights across low, mid, and high-order dimensions. Electronics 2023, 12, 4742. [Google Scholar] [CrossRef]
- Xie, Y.; Li, Q. A review of deep learning methods for compressed sensing image reconstruction and its medical applications. Electronics 2022, 11, 586. [Google Scholar] [CrossRef]
- Shi, W.; Jiang, F.; Liu, S.; Zhao, D. Scalable convolutional neural network for image compressed sensing. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15–20 June 2019; pp. 12282–12291. [Google Scholar] [CrossRef]
- You, D.; Zhang, J.; Xie, J.; Chen, B.; Ma, S. COAST: Controllable arbitrary-sampling network for compressive sensing. IEEE Trans. Image Process. 2021, 30, 6066–6080. [Google Scholar] [CrossRef]
- Luo, W.; Li, Y.; Urtasun, R.; Zemel, R. Understanding the effective receptive field in deep convolutional neural networks. In Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain, 5–10 December 2016; pp. 4905–4913. [Google Scholar]
- Prakash, A.; Storer, J.; Florencio, D.; Zhang, C. RePr: Improved training of convolutional filters. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15–20 June 2019; pp. 10658–10667. [Google Scholar] [CrossRef]
- Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv 2021, arXiv:2010.11929v2. [Google Scholar]
- Donoho, D.L. De-noising by soft-thresholding. IEEE Trans. Inf. Theory 1995, 41, 613–627. [Google Scholar] [CrossRef]
- Donoho, D.L.; Maleki, A.; Montanari, A. Message-passing algorithms for compressed sensing. Proc. Natl. Acad. Sci. USA 2009, 106, 18914–18919. [Google Scholar] [CrossRef]
- Zhang, J.; Zhao, C.; Gao, W. Optimization-inspired compact deep compressive sensing. IEEE J. Sel. Top. Signal Process. 2020, 14, 765–774. [Google Scholar] [CrossRef]
- Zhang, Z.; Liu, Y.; Liu, J.; Wen, F.; Zhu, C. AMP-Net: Denoising-based deep unfolding for compressive image sensing. IEEE Trans. Image Process. 2021, 30, 1487–1500. [Google Scholar] [CrossRef] [PubMed]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.; Polosukhin, I. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 5999–6099. [Google Scholar]
- Zhang, J.; Huang, Y.; Wu, W.; Lyu, M.R. Transferable adversarial attacks on vision transformers with token gradient regularization. In Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 17–24 June 2023; pp. 16415–16424. [Google Scholar] [CrossRef]
- Wang, Y.; Xu, Z.; Wang, X.; Shen, C.; Cheng, B.; Shen, H.; Xia, H. End-to-end video instance segmentation with Transformers. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021; pp. 17378–17389. [Google Scholar] [CrossRef]
- Zhu, X.; Su, W.; Lu, L.; Li, B.; Wang, X.; Dai, J. Deformable DETR: Deformable transformers for end-to-end object detection. In Proceedings of the 9th International Conference on Learning Representations, Virtual, 3–7 May 2021. [Google Scholar]
- Wang, Z.; Cun, X.; Bao, J.; Zhou, W.; Liu, J.; Li, H. Uformer: A general U-shaped Transformer for image restoration. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 17662–17672. [Google Scholar] [CrossRef]
- Zhang, J.; Huang, J.T.; Wang, W.; Li, Y.; Wu, W.; Wang, X.; Su, Y.; Lyu, M.R. Improving the transferability of adversarial samples by path-augmented method. In Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 17–24 June 2023; pp. 8173–8182. [Google Scholar] [CrossRef]
- Hu, J.; Shen, L.; Sun, G. Squeeze-and-excitation networks. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 7132–7141. [Google Scholar] [CrossRef]
- Liu, H.; Dai, Z.; So, D.R.; Le, Q.V. Pay attention to MLPs. In Proceedings of the Advances in Neural Information Processing Systems, Virtual, 6–14 December 2021; pp. 9204–9215. [Google Scholar]
- Lefkimmiatis, S. Universal denoising networks: A novel CNN architecture for image denoising. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 3204–3213. [Google Scholar] [CrossRef]
- Parikh, N.; Boyd, S. Proximal algorithms. Found. Trends Optim. 2014, 1, 127–239. [Google Scholar] [CrossRef]
- Xu, Y.; Wei, H.; Lin, M.; Deng, Y.; Sheng, K.; Zhang, M.; Tang, F.; Dong, W.; Huang, F.; Xu, C. Transformers in computational visual media: A survey. Comput. Vis. Media 2022, 8, 33–62. [Google Scholar] [CrossRef]
- Bello, I.; Zoph, B.; Vaswani, A.; Shlens, J.; Le, Q.V. Attention augmented convolutional networks. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea, 27 October–2 November 2019; pp. 3285–3294. [Google Scholar] [CrossRef]
- Woo, S.; Park, J.; Lee, J.Y.; Kweon, I.S. CBAM: Convolutional block attention module. In Proceedings of the European Conference Computer Vision, Munich, Germany, 8–14 September 2018; pp. 3–19. [Google Scholar] [CrossRef]
- Park, J.; Woo, S.; Lee, J.Y.; Kweon, I.S. BAM: Bottleneck attention module. arXiv 2018, arXiv:1807.06514. [Google Scholar]
- Guo, M.H.; Lu, C.Z.; Liu, Z.N.; Cheng, M.M.; Hu, S.M. Visual attention network. Comput. Vis. Media 2023, 9, 733–752. [Google Scholar] [CrossRef]
- Arbelaez, P.; Maire, M.; Fowlkes, C.; Malik, J. Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 2011, 33, 898–916. [Google Scholar] [CrossRef]
- Kulkarni, K.; Lohit, S.; Turaga, P.; Kerviche, R.; Ashok, A. ReconNet: Non-iterative reconstruction of images from compressively sensed measurements. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 449–458. [Google Scholar] [CrossRef]
- Bevilacqua, M.; Roumy, A.; Guillemot, C.; Alberi-Morel, M.L. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In Proceedings of the 23rd British Machine Vision Conference (BMVC), Surrey, UK, 3–7 September 2012; p. 135. [Google Scholar] [CrossRef]
- Zhang, L.; Wu, X.; Buades, A.; Li, X. Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. J. Electron. Imag. 2011, 20, 023016. [Google Scholar] [CrossRef]
- Dong, C.; Loy, C.C.; Tang, X. Accelerating the super-resolution convolutional neural network. In Proceedings of the 14th European Conference, Amsterdam, The Netherlands, 11–14 October 2016; pp. 391–407. [Google Scholar] [CrossRef]
- Kingma, D.; Ba, J. Adam: A method for stochastic optimization. arXiv 2015, arXiv:1412.6980. [Google Scholar]
- Loshchilov, I.; Hutter, F. SGDR: Stochastic gradient descent with warm restarts. In Proceedings of the 5th International Conference on Learning Representations, Toulon, France, 24–26 April 2017. [Google Scholar]
- Zhang, J.; Ghanem, B. ISTA-Net: Interpretable optimization-inspired deep network for image compressive sensing. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 1828–1837. [Google Scholar] [CrossRef]
- Mou, C.; Wang, Q.; Zhang, J. Deep generalized unfolding networks for image restoration. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 17378–17389. [Google Scholar] [CrossRef]
- Shen, M.; Gan, H.; Ning, C.; Hua, Y.; Zhang, T. TransCS: A Transformer-based hybrid architecture for image compressed sensing. IEEE Trans. Image Process. 2022, 31, 6991–7005. [Google Scholar] [CrossRef] [PubMed]
- Gan, H.; Shen, M.; Hua, Y.; Ma, C.; Zhang, T. From patch to pixel: A Transformer-based hierarchical framework for compressive image sensing. IEEE Trans. Comput. Imaging 2023, 9, 133–146. [Google Scholar] [CrossRef]
Dataset | CS ratios | ISTA-Net+ | OPINE-Net+ | AMP-Net-BM | DGU-Net+ | TransCS | TCS-Net | TCR-Net |
---|---|---|---|---|---|---|---|---|
Set5 | 0.01 | 18.5225/0.4408 | 21.8914/0.6101 | 22.4254/0.6185 | 22.4190/0.6237 | -/- | 22.7494/0.6003 | 23.0929/0.6367 |
0.04 | 23.4528/0.6619 | 27.9457/0.8209 | 27.8246/0.8179 | 28.3861/0.8318 | 27.9142/0.8262 | 27.5483/0.8173 | 28.5561/0.8427 | |
0.10 | 28.6065/0.8315 | 32.5102/0.9058 | 32.1392/0.9031 | 32.8441/0.9111 | 32.2531/0.9138 | 31.4809/0.9067 | 33.1097/0.9244 | |
0.25 | 34.1672/0.9272 | 36.7785/0.9510 | 36.9258/0.9541 | 37.3302/0.9558 | 36.9154/0.9594 | 35.8560/0.9559 | 37.6505/0.9630 | |
0.50 | 39.4886/0.9706 | 41.6234/0.9779 | 42.1352/0.9804 | 42.4728/0.9809 | 42.1788/0.9825 | -/- | 42.7052/0.9842 | |
McM18 | 0.01 | 19.9893/0.4942 | 23.4088/0.6316 | 23.7917/0.6431 | 23.0500/0.6372 | -/- | 23.6266/0.6144 | 24.0858/0.6427 |
0.04 | 24.2732/0.6577 | 27.9489/0.7891 | 27.9164/0.7887 | 28.1609/0.7998 | 28.0115/0.7943 | 27.5373/0.7907 | 28.3934/0.8126 | |
0.10 | 28.5360/0.8104 | 31.9249/0.8878 | 31.7231/0.8869 | 32.3243/0.8977 | 31.8816/0.8964 | 30.9669/0.8913 | 32.5369/0.9103 | |
0.25 | 33.9880/0.9237 | 36.9213/0.9537 | 37.0400/0.9570 | 37.7359/0.9614 | 37.1519/0.9605 | 35.8945/0.9579 | 37.9561/0.9668 | |
0.50 | 39.5162/0.9728 | 42.2930/0.9834 | 43.0616/0.9866 | 43.6171/0.9875 | 42.9903/0.9872 | -/- | 43.8347/0.9894 | |
BSD100 | 0.01 | 19.1925/0.4056 | 21.8885/0.5103 | 22.2577/0.5225 | 22.1172/0.5126 | -/- | 22.22/0.5029 | 22.7562/0.5248 |
0.04 | 22.2471/0.5411 | 25.0002/0.6516 | 25.0882/0.6593 | 25.2757/0.6653 | 25.0518/0.6690 | 24.9076/0.6665 | 25.4058/0.6839 | |
0.10 | 25.0920/0.6841 | 27.5465/0.7715 | 27.6100/0.7786 | 27.8919/0.7847 | 27.5527/0.7937 | 27.1727/0.7903 | 28.0507/0.8062 | |
0.25 | 29.0354/0.8402 | 31.2010/0.8870 | 31.4876/0.8974 | 31.6787/0.8984 | 31.3815/0.9039 | 30.6594/0.9009 | 31.9835/0.9121 | |
0.50 | 33.7145/0.9371 | 36.0208/0.9582 | 36.7344/0.9658 | 36.7421/0.9656 | 36.4358/0.9668 | -/- | 37.1064/0.9704 | |
General100 | 0.01 | 18.9989/0.4700 | 22.5268/0.6229 | 22.9131/0.6321 | 22.8558/0.6276 | -/- | 22.9139/0.6018 | 23.4813/0.6374 |
0.04 | 23.7578/0.6549 | 27.6234/0.7865 | 27.4456/0.7846 | 27.9241/0.7969 | 27.6451/0.7947 | 27.2431/0.7888 | 28.2144/0.8104 | |
0.10 | 28.5443/0.8104 | 32.0279/0.8863 | 31.5631/0.8838 | 32.4102/0.8968 | 31.7109/0.8964 | 30.8719/0.8895 | 32.7478/0.9093 | |
0.25 | 34.3164/0.9250 | 37.1454/0.9530 | 36.9876/0.3552 | 37.5467/0.9598 | 37.2737/0.9614 | 35.7811/0.9568 | 38.2174/0.9667 | |
0.50 | 39.9733/0.9740 | 42.5183/0.9835 | 42.8420/0.9857 | 43.2621/0.9869 | 42.9651/0.9874 | -/- | 43.8010/0.9891 | |
Avg. | 0.01 | 19.1550/0.4424 | 22.2975/0.5728 | 22.6792/0.5835 | 22.5305/0.5767 | -/- | 22.6566/0.5584 | 23.1962/0.5873 |
0.04 | 23.1151/0.6043 | 26.4806/0.7270 | 26.4350/0.7295 | 26.7659/0.7389 | 26.5178/0.7390 | 26.2264/0.7347 | 26.9770/0.7546 | |
0.10 | 26.9969/0.7542 | 30.0208/0.8354 | 29.8162/0.8373 | 30.3869/0.8469 | 29.8722/0.8507 | 29.2344/0.8455 | 30.6326/0.8635 | |
0.25 | 31.9184/0.8869 | 34.4534/0.9234 | 34.5241/0.6603 | 34.9257/0.9323 | 34.6135/0.9354 | 33.4952/0.9318 | 35.3881/0.9421 | |
0.50 | 37.1189/0.9573 | 39.5664/0.9720 | 40.1050/0.9767 | 40.3492/0.9772 | 40.0215/0.9780 | -/- | 40.7771/0.9806 |
ISTA-Net+ | OPINE-Net+ | AMP-Net-BM | DGU-Net+ | TransCS | TCS-Net | TCR-Net | |
---|---|---|---|---|---|---|---|
Time (s) | 0.0071 | 0.0088 | 0.0383 | 0.0278 | 0.0360 | 0.0212 | 0.0128 |
Parameters (M) | 0.34 | 0.62 | 0.58 | 6.92 | 1.44 | 0.52 | 0.37 |
Size (MB) | 1.4 | 2.5 | 2.4 | 9.6 | 21.4 | 2.1 | 4.7 |
Datasets | TCR-Net-(7, 7, 1) | TCR-Net-(3, 5, 1) | TCR-Net |
---|---|---|---|
Set5 | 37.5737 | 37.5016 | 37.6505 |
McM18 | 37.8523 | 37.7893 | 37.9561 |
BSD100 | 31.9179 | 31.8828 | 31.9835 |
General100 | 38.0839 | 38.0262 | 38.2174 |
Avg. | 35.2887 | 35.2404 | 35.3881 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Nan, R.; Sun, G.; Zheng, B.; Zhang, P. Hybrid Transformer and Convolution for Image Compressed Sensing. Electronics 2024, 13, 3496. https://doi.org/10.3390/electronics13173496
Nan R, Sun G, Zheng B, Zhang P. Hybrid Transformer and Convolution for Image Compressed Sensing. Electronics. 2024; 13(17):3496. https://doi.org/10.3390/electronics13173496
Chicago/Turabian StyleNan, Ruili, Guiling Sun, Bowen Zheng, and Pengchen Zhang. 2024. "Hybrid Transformer and Convolution for Image Compressed Sensing" Electronics 13, no. 17: 3496. https://doi.org/10.3390/electronics13173496