Next Article in Journal
A New Regularization for Deep Learning-Based Segmentation of Images with Fine Structures and Low Contrast
Next Article in Special Issue
Compressive Sensing of Medical Images Based on HSV Color Space
Previous Article in Journal
Controlling Upper Limb Prostheses Using Sonomyography (SMG): A Review
Previous Article in Special Issue
Sidelobe Suppression Techniques for Near-Field Multistatic SAR
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

IEF-CSNET: Information Enhancement and Fusion Network for Compressed Sensing Reconstruction

College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou 310058, China
*
Author to whom correspondence should be addressed.
Sensors 2023, 23(4), 1886; https://doi.org/10.3390/s23041886
Submission received: 9 December 2022 / Revised: 4 February 2023 / Accepted: 6 February 2023 / Published: 8 February 2023
(This article belongs to the Special Issue Compressed Sensing and Imaging Processing)

Abstract

:
The rapidly growing requirement for data has put forward Compressed Sensing (CS) to realize low-ratio sampling and to reconstruct complete signals. With the intensive development of Deep Neural Network (DNN) methods, performance in image reconstruction from CS measurements is constantly increasing. Currently, many network structures pay less attention to the relevance of before- and after-stage results and fail to make full use of relevant information in the compressed domain to achieve interblock information fusion and a great receptive field. Additionally, due to multiple resamplings and several forced compressions of information flow, information loss and network structure redundancy inevitably result. Therefore, an Information Enhancement and Fusion Network for CS reconstruction (IEF-CSNET) is proposed in this work, and a Compressed Information Extension (CIE) module is designed to fuse the compressed information in the compressed domain and greatly expand the receptive field. The Error Comprehensive Consideration Enhancement (ECCE) module enhances the error image by incorporating the previous recovered error so that the interlink among the iterations can be utilized for better recovery. In addition, an Iterative Information Flow Enhancement (IIFE) module is further proposed to complete the progressive recovery with loss-less information transmission during the iteration. In summary, the proposed method achieves the best effect, exhibits high robustness at this stage, with the peak signal-to-noise ratio (PSNR) improved by 0.59 dB on average under all test sets and sampling rates, and presents a greatly improved speed compared with the best algorithm.

1. Introduction

At present, the acquisition, transmission, and processing of information are proliferating, which brings great challenges to data storage and transmission. Meanwhile, the amount of sampled data is greatly restricted due to limitations in power, bandwidth, or sampling time in some extreme cases. Fortunately, the Compressed Sensing (CS) [1] theory has partially broken through the limitations of the traditional Nyquist sampling theory [2]. A reduced sampling rate can achieve low-cost and efficient data compression and is beneficial for decreasing the burden of storage and transmission.
In recent years, a tremendous number of algorithms have been proposed to address CS reconstruction, including two main categories: traditional methods and Deep Learning (DL) methods.
Traditional methods are usually based on theoretical guarantees to solve undetermined equations and obtain optimal results through gradual iteration, so they inevitably suffer from the high computational cost dictated by interactive calculations [3] and are hard to execute by parallel computing.
Different from traditional methods, DL methods have been applied to solve image CS reconstruction problems. They can map from compressed data to original signals by training a larger number of parameters in Deep Neural Networks (DNNs) with superior inference time and reconstruction quality. Of course, they are subject to some defects.
First, few methods, such as AMPNet [4], can make ultimate use of information in the compressed domain. However, the current region of compressed information will be applied to reconstruct the corresponding image blocks only, leading to a small receptive field [3]. Second, the intermediate features of previous iterations are completely ignored during reconstruction, although they can guide the recovery of the following iterations effectively. Last but not least, most relevant works inevitably suffer from a large amount of information loss caused by resampling and channel compression. The intermediate results should be compressed and resampled to obtain error information for supplementing the iterative reconstruction.
In view of the previous studies and limitations mentioned above, an IEF-CSNET is proposed here. The main contributions of this work are listed as follows:
  • Based on the Compressed Information Extension (CIE) module, information in the compressed domain is fully utilized for high-dimensional fusion, greatly expanding the receptive field of DNN methods.
  • In consideration of the initial image and the error enhancement image recovered by previous iterations, the Error Comprehensive Consideration Enhancement (ECCE) module can incorporate the enhancement information into the output flow more efficiently.
  • To solve the information compression due to obtaining errors, an Iterative Information Flow Enhancement (IIFE) module is proposed to complete iterative and progressive recovery during loss-less information transmission.
  • Combined with CIE, ECCE, and IIFE, the IEF-CSNET is proposed. On this basis, several experiments and visual analyses of its effectiveness are performed. Under all test sets and sampling rates, the average increase is approximately 0.59 dB, and the operating speed is improved by orders of magnitude from the state-of-the-art (SOTA) method.
The rest of this paper is organized as follows. In Section 2, the CS and some related works are introduced and analyzed. The proposed IEF-CSNET is elaborated on in Section 3. The settings and substantial results of the experiments are provided in Section 4. Finally, the conclusion is summarized in Section 5.

2. Related Works

In this section, a brief review of CS and some benchmark works are provided, which can be generally divided into traditional methods and DL methods.

2.1. Compressed Sensing and Traditional Methods

CS can sample and compress the signals simultaneously, breaking through the limitation of the Shannon Nyquist sampling theorem [1]. Mathematically, CS reconstruction aims to infer the original signal X R N × 1 from the CS measurements Y R M × 1 .  Measurement Y can be obtained by a nonadaptive linear projection Y = Φ X , Φ R M × N , whereby the CS sampling ratio is defined as R = M / N . However, if there are sparse representations Ψ X of X in domain Ψ , the typical ill-posed problems can be solved by measurement Y correctly with high probability because it can be transformed into the following expression, which is an optimization:
min X Ψ X p , s . t . Y = Φ X
where p means the p norm of vector * to characterize the sparsity of the vector. Thanks to CS theory, the loads of data acquisition, transmission, and storage can be greatly reduced.
Many traditional works have been performed to design the sampling matrix [5] and reconstruct X from the CS measurement Y . Convex optimization methods achieve accurate and robust recovery results by translating the nonconvex constraint into a convex constraint [6]. Greedy algorithms, such as Matching Pursuit (MP) [7], Orthogonal MP (OMP) [8], and stagewise OMP [9], generally obtain results based on the iterative residual, achieving lower reconstruction quality but sharing low computational complexity [10]. Refs. [11,12,13,14] take the source at Projected Landweber [15] and reconstruct by successively projecting and thresholding. The Total Variation (TV) [16] establishes more sophisticated models and focuses more on image priors. Nevertheless, details could be lost due to the too-smooth constraint.
Although some traditional methods have been widely applied practically, such as wireless sensor network [17], hyperspectral imaging [18], Magnetic Resonance Imaging (MRI) [19,20], underwater sensors [21], and aerospace [22], they usually suffer from too slow a running speed, due to the inevitable large numbers of iterations and heavy matrix operations, which must be executed in a nonparallel way in each iteration. In addition, it is difficult to draw enough prior knowledge from big data, causing performance bottlenecks.

2.2. Deep Learning Methods

By exploiting massive paralleled Graphic Processing Unit (GPU) processing architectures and large amounts of training data, DL methods are mainly composed of various high-density parallel computing processes and can achieve superior reconstruction quality and computational complexity when compared to traditional methods. ReconNet  [23], the first DL method, presents a noniterative and extremely fast algorithm to reconstruct images from CS measurements. Block Compressed Sensing (BCS) is suitable for image sampling and compression in resource-constrained applications [24]. However, some methods are prone to suffering from blocking artifacts due to the independent recovery among blocks, so it is necessary to cascade a time-consuming filter, BM3D. Instead of the fixed sampling matrix, DR2-Net [25], CSNET [10], and CSNET+ [26] implement the learnable fully connected layer and convolution layers for sampling. They all improve the reconstruction quality by stacking several residual learning blocks. NLR-CSNET [27] and DPA-Net [28] introduce a 3D encoder/decoder and a dual-path network based on the attention mechanism for better reconstruction quality. The encoder and decoder in [27] with channel attention motivate the effective skip links. However, these straightforward methods are largely unprincipled black boxes that are difficult to train and often-times specific to a single measurement matrix [29].
Deep unfolding methods incorporate traditional iterative reconstruction and DNNs, and they can map each iteration into a series of network layers that preserve interpretability and performance. Learned Denoising-based Approximate Message Passing (LDAMP) [29] combines the Denoising-Based Approximate Message Passing (D-AMP) algorithm and neural network and runs over 50 times faster than BM3D-AMP. Inspired by the Iterative Shrinkage-Thresholding Algorithm (ISTA), ISTA-Net+ [30], and ISTA-Net++ [31] design deep networks,  all parameters are learned end-to-end rather than hand-crafted. The difference is that ISTA-Net++ uses a cross-block learnable sampling strategy and achieves multi-ratio sampling and reconstruction in one model but leads to a low robustness of all compression ratios. Moreover, OPINE-Net [32] is a variant of ISTA-Net+ and adopts end-to-end training and learning to explore adaptive sampling and reconstruction. An unfolded LSTM network is utilized in video compression sensing, deeply fusing the intra- and interframe spatial–temporal information [33]. Finally, AMPNet [4] unfolds the iterative denoising of the AMP algorithm and shows a better reconstruction accuracy with high reconstruction speed.
Generally, due to the memory limitation and compatibility with input size, the sampling matrix can not share the same size as the original image. Thus, almost all images are initially reconstructed based on blocks and then denoised, leading to insufficient information integration in the compressed domain and small receptive fields. Some valuable compressed information from the other adjacent blocks can be extracted to assist in the reconstruction of the current block. This is in line with the data-oriented training mechanism because DL methods are good at high-dimensional mapping and learning autonomous knowledge. The methods mentioned above employ the solution of iterative progressive reconstruction for better performance. In this case, the processed results of intermediate iterations are considered to contain a wealth of information. In the repeated process, the data reconstructed painstakingly will be forcibly compressed or discarded and are expected to be resampled to the same size as input Y to obtain the difference in the compressed domain. Meanwhile, it is inconsistent with the advantage of the parallel computing of DL methods. Moreover, the generous results underutilized may cause many network redundancies. In the meantime, it is frustrating that the results calculated in previous iterations are ignored in most methods, and the previous features are supposed to be better used to explore and analyze which modes of information are difficult to recover. In this way, they can effectively strengthen the subsequent reconstruction.

3. Methods

3.1. Overview of Proposed Method

This subsection introduces the overall structure and exhibits the whole prediction pipeline of the proposed method. Figure 1 shows how the submodules are closely combined with each other and organized into a more effective whole. The complete method is implemented by the following detailed submodules:
  • The CIE module expands and integrates the information elements in the compressed domain to output CR 1 and the Compressed-domain Fusion Error Image CFEI i , which can take greater advantage of the measurements in each iteration and achieve a larger receptive field (Section 3.2).
  • The ECCE module outputs the Enhanced Error Image EEI i by taking CR i , CFEI i , and EEI 2 ( i 1 ) of the previous iterations into consideration. In this way, the error and residual can be accurately predicted with high robustness to supplement the following reconstruction more efficiently (Section 3.3).
  • Based on the IIFE module, the Intermediate Features IF i and EEI i can be supplemented progressively and fused more smoothly under loss-less information transmission while the sampling is repeated in the iterative reconstruction process (Section 3.4).
Meanwhile, pseudo code matching with the structure diagram (Figure 1) is provided as follows (Algorithm 1) to explain the inference more intuitively.
Algorithm 1 Prediction of IEF-CSNET.
1:
PREDICT (Input: SD = Φ ( X ) )
2:
for each i [ 1 , N I ]  do
3:
  if  i = = 1  then
4:
    CR 1 = CIE 1 ( SD )
5:
    IF 1 = BCNN 1 ( CR 1 )
6:
  else
7:
    CR i = IRE i ( IF i 1 )
8:
    SSD i = S ( CR i )
9:
    CFEI i = CIE i ( SSD i - SD )
10:
    EEI i = ECCE i ( CR i , EEI ( i 1 ) 2 , CFEI i )
11:
    IF i = ES i ( IF i 1 , IEE i ( EEI i ))
12:
    IF i = BCNN i ( IF i )
13:
  end if
14:
end for
15:
return  IF N I

3.2. Compressed Information Extension (CIE)

After analyzing and considering most of other related works, it is realized that the images are processed and divided into many blocks, which will be compressively sampled and independently reconstructed as the CR blocks. In this process, only measurements of the corresponding region block are employed for reconstruction, which are shown and represented by the red dotted boxes in Figure 2. Under these circumstances, the reconstruction of each block is competed independently. It can be summarized as follows:
CR ( i , j ) = P S u p s a m p l i n g SD ( i , j )
where SD ( i , j ) and CR ( i , j ) mean the sampled data and the coarse reconstruction of block ( i , j ) , respectively. The 1 × 1 convolution or full connection layer is adopted for upsampling, and P S ( · ) means the Pixelshuffle option [34].
Some methods cannot prevent reconstruction from blocking artifacts under a low sampling rate, such as [23]. Because the number of measurements in SD ( i , j ) is severely insufficient when extremely compressed, small disturbances of measurements from different blocks may cause overall offsets of each CR ( i , j ) after the reconstruction. Thus, the helpful information in the compressed domain should be drawn from related blocks and fused into the current block to obtain more valuable guidance for reconstruction.
Meanwhile, it is found that only a few methods can achieve a relatively large receptive field while reconstructing. During upsampling, other methods [4,23,26,31,35] only focus on the corresponding measurements compressed from the target single block while reconstructing. Under normal conditions, a larger receptive field tends to bring better performance. For example, CSformer [3] integrates the characteristics of Transformer [36] to obtain a large receptive field in theory and obtain SOTA performance. However, an inference speed that is too low may be very fatal. It is even slower than some traditional algorithms, so the advantage of DL methods cannot be exhibited. Currently, GPU devices are good at parallel computing with high throughput, which makes the calculation of multichannel feature maps possible, promoting the feasibility of the methods introduced later.
For the reasons and analysis mentioned above, the CIE module is developed as a solution. It can take full advantage of measurements in the compressed domain and share a super large receptive field. In addition, it is suitable for GPU devices. To our knowledge, a similar design has not been used in relevant works at this stage. The pipeline is illustrated in Figure 3.
EFSD and CR 1 at the 1st iteration can be calculated as follows:
ESD = SD Θ ( W ( 3 × 3 ) , b ) EFSD = SD , ESD CR 1 = P S u p s a m p l i n g EFSD
where ( · ) Θ ( W , b ) represents the convolution option through parameter groups W and b .
In the related works, each block with 32 × 32 pixels is compressed into 1 × 1 pixels with 1024 × R channels in the compressed domain. Thus, the receptive field may be limited to the current block. However, the fusion of SD and ESD can easily achieve three or even more times the effect of the receptive field from other methods. The CIE modules can ensure the information perception for at least 32 × 3 receptive fields at each iteration. In addition, the checking, judging, and consulting of the surrounding blocks can be realized in advance at the initial reconstruction of CR 1 for better reconstruction of the current block.
Similar to Equation (3), when the iteration i > = 2 , the CFEI i can be calculated by the CIE module from SSD i SD , as follows:
ESD = ( SSD i SD ) Θ ( W ( 3 × 3 ) , b ) EFSD = ( SSD i SD ) , ESD CFEI i = P S u p s a m p l i n g EFSD
where SSD i SD is noted as the error between SD and SSD i . CFEI i is the image of the residual error after more comprehensive consideration in the compression domain.
In this way, EFSD and EFSD can be obtained as more effective information in the compressed domain without damaging or impacting original measurements. In the meantime, CIE modules make larger receptive fields come true, tending to better reconstruction performance.

3.3. Error Comprehensive Consideration Enhancement (ECCE)

Most related works fail to make full use of the previous iteration results and may ignore the connections during progressive reconstruction to some extent. First, the task in each iteration is consistent during the entire process, and the errors between CR i and X can be predicted in each iteration. The residual error is gradually reduced during the iteration and shares the same target so that the previous residuals are valuable to guide the next stage of error prediction. Based on this, an ECCE module is proposed here to refine the reconstruction, the pipeline of which is shown in Figure 4.
The input of ECCE i of iteration i can be achieved as follows:
Input i = CR 2 , CFEI 2 , i = 2 CR i , EEI 2 ( i 1 ) , CFEI i , 2 < i N I
where CFEI i represents the output of CIE i (Equation (4)). CFEI i and CR i can be understood as the abstract summary and the prediction of target error in iteration i, respectively. The set of EEI 2 ( i 1 ) means that ECCE i considers the previous results of iteration [ 2 , i 1 ] . They are all employed as the input of ECCE i to predict the EEI i . ECCE i makes the fusion of multiple pieces of information more sufficient by coding the input as follows:
EEI i = C N N 32 1 C N N i 32 Input i
where C N N i j ( · ) represents the 3 × 3 convolution option and one ReLU layer, with i input channels and j output channels.
The ECCE module has the following advantages. First, based on EEI 2 ( i 1 ) from previous stages, the proposed module predicts the EEI i more accurately and realizes more adaptive reconstruction. Under the training of a large amount of data, the network can remember and even judge which information mode is difficult to reconstruct. Because of paying more attention to the connection of previous results, some components are always maintained throughout the whole process, and then the network will focus more on these stubborn questions to strengthen the final output. Second, different from other methods, ECCE receives the CR i and CFEI i as parts of the input at each iteration (as illustrated in Figure 4 and Equation (5)). In addition, it aims to combine the intermediate results with errors to analyze the targeted shortcomings in CR i in the current situation to better integrate the errors later. Finally, two-layer CNN is employed for dimension expansion and compression to achieve a similar effect as the autoencoder for deeper information fusion.

3.4. Iterative Information Flow Enhancement Module (IIFE)

The existing hardware system performs poorly in accelerating the large kernel convolution, so images or feature maps are no longer sampled by convolution options with a large kernel size. Instead, the sampling module is completed by multichannel parallel multiplication due to no overlap among different blocks during sampling. It is noted as S ( · ) and shown in Figure 5.
First, the image is divided into many blocks by B = 32 .
I ( i , j ) = I [ ( i 1 ) B : i B , ( j 1 ) B : j B ] I B = B ( I ) = I ( 1 , 1 ) , I ( 1 , 2 ) , I ( 1 , w ) I ( 2 , 1 ) , I ( 2 , 2 ) , I ( 2 , w ) I ( h , 1 ) , I ( h , 2 ) , I ( h , w )
where w = W B and h = H B represent the numbers of blocks in width and height, respectively. Then the blocks are concatenated as a whole feature map at dimension C.
I C = C ( I B ) = I ( 1 , 1 ) , I ( h , w )
where C ( · ) represents the option of concatenating. In this way, I C R 32 32 ( w h ) becomes the set of blocks being sampled, and the SSD i R w h ( 1024 R ) in iteration i can be achieved as follows:
SSD i = S ( CR i ) = r e s h a p e ( C ( B ( CR i ) ) · Φ )
where r e s h a p e ( · ) represents the reverse operation of B ( · ) , which aims to organize C ( B ( CR i ) ) · Φ R ( w h ) ( 1024 R ) into R w h ( 1024 R ) (Figure 5). It is noted that S ( · ) should be executed repeatedly with the same sampling matrix Φ in each IIFE.
In other related methods, IF i is normally compressed into one-channel CR i by C E i and sampled to achieve the error in the compressed domain, as shown in Figure 6. The residual error is upsampled into another error image EI i with the same size as X . Then, EI i is added to the main branch directly. Following that, the one-channel feature CR i is augmented in the channel dimension to generate IF i + 1 for the next step. In such a way, most intermediate results will be lost during the forced compression, which is indicated by the lighter and smaller green arrows in Figure 6. This is a large bottleneck of the network performance.
Within this work, the IIFE module is proposed to make full use of IF i , which is shown in Figure 7. Based on SSD i SD in the compressed domain, a relatively complete EI i can be predicted by upsampling. Meanwhile, the EI i can be expanded to enrich information EF i by the IEE and to adjust the main branch features in all channels. In this case, the fusion of EF i and IF i can be smoother without any information being lost. Therefore, IF i with diverse information in different channels can be corrected in parallel to avoid wasting hard-earned data from previous heavy channel recovery.
The IIFE can be calculated by the following equations:
CR i = IRE i IF i 1 SSD i = S ( CR i ) EF i = IEE i ( u p s a m p l i n g ( SD SSD i ) ) IF i = ES i ( IF i 1 , EF i )
where IRE i is realized by two 3 × 3 convolution and ReLU layers to extract a one-channel CR i . Then, SSD i can be achieved by module S ( · ) defined in Equation (9). IEE i also consists of two 3 × 3 convolution and ReLU layers to augment EI i into EF i that contains more helpful information. Then, IF i is completely revised as a whole through ES i , by incorporating EF i and IF i 1 .
Therefore, it is believed that information flow in the main branch is protected and enhanced. To perfectly use the IF i continuously produced by the previous network, IIFE shows great advantages in the resampling process of each iteration. It is emphasized that there are no steps of forced channel compression that will cause information loss. Under these circumstances, the method of effectively retaining and recovering more information can be found.
In the meantime, a mechanism of error compensation is usually adopted through the simple pointwise addition of the two one-channel images, CR i and EI i . It is believed that nonlinear mapping will be helpful for image reconstruction. Therefore, the ES module is designed and inspired by the encoder-decoder, as shown in Figure 8. Instead of simple addition, it is efficient to make the network learn which information to absorb and how to integrate due to the more adaptive and diversified fusion than addition. The performance of IIFE is far better than that shown in Figure 6 because the Φ , IRE, IEE, and ES modules are combined effectively, realizing outstanding information collection, transmission, supplementation, and fusion.
Finally, IIFE can cooperate with both ECCE and CIE to form a tight structure as IEF-CSNET. The upsampling module is replaced with CIE and ECCE to generate high-quality EEI i . For specific details of the combination, please refer to the overview of the pipeline in Figure 7 and the pseudocode in Algorithm 1 under Section 3.1.

4. Experiment

4.1. Settings

Datasets: The datasets are prepared for training and testing in the same way as the experimental details in [3]. COCO 2017 [37] is a large-scale dataset and is applied as the training set in this work by gathering images of complex everyday scenes containing common objects in their natural context. The patches with 128 × 128 pixels are cropped randomly without any data augmentation during the training. In addition, Set5 [38], Set11 [23], Set14 [39], BSD68 [40], and Urban100 [41] are employed as testing sets to evaluate the performance and robustness more comprehensively because they are widely applied in image reconstructions. Their specific information is listed in Table 1.
Training and Test Details: During the training, the batch size, N I , and learning rate L r are set as 64, 12, and 0.0001, respectively. All images in the datasets are transferred into YCbCr format, and the luminance components ( Y channel) are utilized for both training and testing, similar to what the reference papers performed. Because of the different resolutions, images in the test sets are processed with batch size = 1 one by one. Peak Signal-to-Noise Ratio ( PSNR ) and Structural Similarity ( SSIM ) [42] are employed to quantitatively evaluate the performance of the reconstructed images. The larger the PSNR and SSIM values are, the better the performance is [43].
All the implementations and experiments are deployed under the environment of open-source framework Pytorch 1.8.0 and CPU (Intel Xeon CPU E5-2678 v3 @ 2.50 GHz) with GPU (GeForce RTX 2080 Ti).

4.2. Quantitative Evaluation

The quantitative analysis results of all methods are provided in Table 2. PSNR and SSIM are tested on five testing sets and five different sampling rates ( 1 % , 4 % , 10 % , 25 % , and 50 % ) so that the effects of all methods can be objectively compared under different conditions. The results indicate that the proposed method achieves the best results under different sets or sampling rates. In addition, the smaller standard deviation than others reflects the higher robustness. As listed in Table 2, the average PSNR values of all testing sets are improved by 0.62, 0.6, 0.95, 1.32, and 0.99 dB under the five sampling rates. The absolute improvement under all sampling rates is helpful for practically applying the CS. Meanwhile, the inference speed is much higher than that of the SOTA methods [3] (see Section 4.4). The better reconstruction performance is attributed to the fact that the proposed method can match the characteristics of CS well and realize optimization avoiding the forced loss of intermediate results encountered by other methods. Meanwhile, it benefits from making full use of information in the compressed domain and intermediate results of previous and subsequent iterations to assist the reconstruction.

4.3. Qualitative Evaluation

For the qualitative evaluation, the performances of different methods can be compared based on the visual perception of the final output images. In Figure 9, three result sets ( R = 0.04 % , 0.10 % , and 0.25 % ) are randomly selected to fully demonstrate the intuitive performance of reconstruction. The full images and the enlarged parts are displayed simultaneously to show the texture and edge more clearly. In addition, the PSNR and SSIM of the images and enlarged parts are both calculated and listed. The comparison shows that there are much fewer artifacts or blurred parts in the results from the proposed method in this work than that of the other counterparts. The comparison among different methods can fully prove that the proposed method shows greater advantages in processing texture details and high-quality images with vivid and sharp edges.

4.4. Inference Speed

The inference speed experiments are set as in reference [10] because detailed descriptions of the settings and results are provided. The number of images that can be processed by different methods per second are listed in Table 3, based on which the running speed can be compared more easily on the same hardware system. On the one hand, the inference speed of this work is obviously superior to that of the SOTA method [3] by orders of magnitude. On the other hand, the proposed method is slightly slower than the fastest method, Reconnet [23], but an additional BM3D denoiser must be cascaded after Reconnet, which will take more than 10 s for each 256 × 256 image in use and cannot be parallelized among images. Finally, it needs to be noted that the proposed method greatly improves the reconstruction performance compared with all other methods of approximate inference speed. The overall analysis results suggest that the proposed method realizes a stable and outstanding reconstruction and shows a speed advantage.

4.5. Ablation Experiment

For the ablation study, the effectiveness of the three designed submodules is explored and analyzed. To illustrate their improvement effects in CS reconstruction separately, four different configurations of IEF-CSNET are implemented, which are introduced as follows:
  • W / O IIFE: No IIFE is set, but ECCE, CIE, and the base model in Figure 6 are a part of the network.
  • W / O ECCE: No ECCE works, but the other two modules are employed.
  • W / O CIE: No CIE is added, but the other two are considered.
  • ALL: CIE, ECCE, and IIFE act with united strength.
The average PSNR and SSIM values of the five datasets under these settings are calculated. To evaluate the function of each module more comprehensively, two sampling rates, R = 1 % and 50 % , are employed for testing, and the results are tabulated in Table 4. The IIFE module improves the performance most greatly, almost close to 1 dB higher than the base module in Figure 6. After supplementation with ECCE and CIE, the performance is still improved under both sampling ratios, even with the help of IIFE. In the case of an extremely low compression ratio, the absence of CIE ( W / O CIE) will lead to a larger loss because the FESD obtained by CIE is critical in the reconstruction.
In addition, the average weights in the convolution layers of ECCE from different iterations are calculated to analyze the internal interpretability, as visualized in Figure 10. In the figure, the two sampling rates, R = 1 % and 50 % , are employed to show the internal interpretability. On the one hand, all polylines I i from different ratios are basically in an upwards trend. This is because the closer the iteration is, the more contributive it will be to the current iteration results. The EEI s in previous iterations will guide the inference of EEI i at this stage because their corresponding weights cannot be ignored by comparison. The network will be committed to recovering the problems not solved by previous iterations. However, the CFEI i produced in the current iteration is decisive because the weight of this input channel is the largest.
Finally, all the submodules will be composed of the complete network structure of IEF-CSNET, and the best performance will be achieved under all compression ratios and different datasets.

5. Conclusions

In this work, a novel network architecture IEF-CSNET is proposed for high-performance image reconstruction based on CS. The IIFE to strengthen the information flow can enhance the efficiency of the whole recovery network and reduce the loss of information. The ECCE module, which closely connects the whole network, purposefully enhances the prediction of error images for higher performance in image restoration. The sensing module CIE allows the network to obtain a larger receptive field and can make full use of the information in the compressed domain. In this way, IEF-CSNET achieves the best reconstruction performance at this stage with the help of the above submodules and exhibits an improved operating speed by orders of magnitude from the SOTA method. Finally, these modules may be applied to other networks for image restoration networks and provide some reference for future work.

Author Contributions

Methodology, Z.Z.; software, Z.Z.; investigation, data curation, F.L.; data curation, F.L.; writing—original draft preparation, Z.Z.; writing—review and editing, H.S.; visualization, F.L.; supervision, F.L. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Donoho, D. Compressed sensing. IEEE Trans. Inf. Theory 2006, 52, 1289–1306. [Google Scholar] [CrossRef]
  2. Shannon, C. Communication in the Presence of Noise. Proc. IRE 1949, 37, 10–21. [Google Scholar] [CrossRef]
  3. Ye, D.; Ni, Z.; Wang, H.; Zhang, J.; Wang, S.; Kwong, S. CSformer: Bridging Convolution and Transformer for Compressive Sensing. arXiv 2021, arXiv:2112.15299. [Google Scholar]
  4. Zhang, Z.; Liu, Y.; Liu, J.; Wen, F.; Zhu, C. AMP-Net: Denoising-Based Deep Unfolding for Compressive Image Sensing. IEEE Trans. Image Process. 2021, 30, 1487–1500. [Google Scholar] [CrossRef] [PubMed]
  5. Liang, J.; Peng, H.; Li, L.; Tong, F. Construction of Structured Random Measurement Matrices in Semi-Tensor Product Compressed Sensing Based on Combinatorial Designs. Sensors 2022, 22, 8260. [Google Scholar] [CrossRef] [PubMed]
  6. Chen, S.S.; Donoho, D.L.; Saunders, M.A. Atomic decomposition by basis pursuit. SIAM Rev. 2001, 43, 129–159. [Google Scholar] [CrossRef]
  7. Mallat, S.G.; Zhang, Z. Matching pursuits with time-frequency dictionaries. IEEE Trans. Signal Process. 1993, 41, 3397–3415. [Google Scholar] [CrossRef]
  8. Tropp, J.A.; Gilbert, A.C. Signal recovery from random measurements via orthogonal matching pursuit. IEEE Trans. Inf. Theory 2007, 53, 4655–4666. [Google Scholar] [CrossRef]
  9. Donoho, D.L.; Tsaig, Y.; Drori, I.; Starck, J.L. Sparse solution of underdetermined systems of linear equations by stagewise orthogonal matching pursuit. IEEE Trans. Inf. Theory 2012, 58, 1094–1121. [Google Scholar] [CrossRef]
  10. Shi, W.; Jiang, F.; Zhang, S.; Zhao, D. Deep networks for compressed image sensing. In Proceedings of the 2017 IEEE International Conference on Multimedia and Expo (ICME), Hong Kong, China, 10–14 July 2017; pp. 877–882. [Google Scholar]
  11. Mun, S.; Fowler, J.E. Residual reconstruction for block-based compressed sensing of video. In Proceedings of the 2011 Data Compression Conference, Palinuro, Italy, 21–24 June 2011; pp. 183–192. [Google Scholar]
  12. Haupt, J.; Nowak, R. Signal reconstruction from noisy random projections. IEEE Trans. Inf. Theory 2006, 52, 4036–4048. [Google Scholar] [CrossRef]
  13. Chen, C.; Tramel, E.W.; Fowler, J.E. Compressed-sensing recovery of images and video using multihypothesis predictions. In Proceedings of the 2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR), Pacific Grove, CA, USA, 6–9 November 2011; pp. 1193–1198. [Google Scholar]
  14. Gan, L. Block compressed sensing of natural images. In Proceedings of the 2007 15th International Conference on Digital Signal Processing, Cardiff, UK, 1–4 July 2007; pp. 403–406. [Google Scholar]
  15. Bertero, M.; Boccacci, P.; De Mol, C. Introduction to Inverse Problems in Imaging; CRC Press: Boca Raton, FL, USA, 2021. [Google Scholar]
  16. Chengbo Li, W.Y.; Zhang, Y. TVAL3: TV Minimization by Augmented Lagrangian and Alternating Direction Agorithm 2009. 2013. Available online: https://nuit-blanche.blogspot.com/2009/06/cs-tval3-tv-minimization-by-augmented.html (accessed on 19 June 2009).
  17. Huang, Y.; Li, H.; Peng, J. A Non-Convex Compressed Sensing Model Improving the Energy Efficiency of WSNs for Abnormal Events’ Monitoring. Sensors 2022, 22, 8378. [Google Scholar] [CrossRef]
  18. Wang, Z.; Gao, Y.; Duan, X.; Cao, J. Adaptive High-Resolution Imaging Method Based on Compressive Sensing. Sensors 2022, 22, 8848. [Google Scholar] [CrossRef]
  19. Guruprasad, S.; Bharathi, S.; Delvi, D.A.R. Effective compressed sensing MRI reconstruction via hybrid GSGWO algorithm. J. Vis. Commun. Image Represent. 2021, 80, 103274. [Google Scholar] [CrossRef]
  20. Schork, N.; Schuhmann, S.; Nirschl, H.; Guthausen, G. Compressed sensing MRI to characterize sodium alginate deposits during cross-flow filtration in membranes with a helical ridge. J. Membr. Sci. 2021, 626, 119170. [Google Scholar] [CrossRef]
  21. Zhou, Y.H.; Tong, F.; Zhang, G.Q. Distributed compressed sensing estimation of underwater acoustic OFDM channel. Appl. Acoust. 2017, 117, 160–166. [Google Scholar] [CrossRef]
  22. Daponte, P.; De Vito, L.; Picariello, F.; Rapuano, S.; Tudosa, I. Compressed Sensing Technologies and Challenges for Aerospace and Defense RF Source Localization. In Proceedings of the 2018 5th IEEE International Workshop on Metrology for AeroSpace (MetroAeroSpace), Rome, Italy, 20–22 June 2018; pp. 634–639. [Google Scholar] [CrossRef]
  23. Kulkarni, K.; Lohit, S.; Turaga, P.; Kerviche, R.; Ashok, A. ReconNet: Non-Iterative Reconstruction of Images From Compressively Sensed Measurements. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016. [Google Scholar]
  24. Chen, Q.; Chen, D.; Gong, J. Low-Complexity Adaptive Sampling of Block Compressed Sensing Based on Distortion Minimization. Sensors 2022, 22, 4806. [Google Scholar] [CrossRef]
  25. Yao, H.; Dai, F.; Zhang, S.; Zhang, Y.; Tian, Q.; Xu, C. Dr2-net: Deep residual reconstruction network for image compressive sensing. Neurocomputing 2019, 359, 483–493. [Google Scholar] [CrossRef]
  26. Shi, W.; Jiang, F.; Liu, S.; Zhao, D. Image Compressed Sensing Using Convolutional Neural Network. IEEE Trans. Image Process. 2020, 29, 375–388. [Google Scholar] [CrossRef]
  27. Sun, Y.; Yang, Y.; Liu, Q.; Chen, J.; Yuan, X.T.; Guo, G. Learning non-locally regularized compressed sensing network with half-quadratic splitting. IEEE Trans. Multimed. 2020, 22, 3236–3248. [Google Scholar] [CrossRef]
  28. Sun, Y.; Chen, J.; Liu, Q.; Liu, B.; Guo, G. Dual-path attention network for compressed sensing image reconstruction. IEEE Trans. Image Process. 2020, 29, 9482–9495. [Google Scholar] [CrossRef]
  29. Metzler, C.; Mousavi, A.; Baraniuk, R. Learned D-AMP: Principled neural network based compressive image recovery. Adv. Neural Inf. Process. Syst. 2017, 30, 1770–1781. [Google Scholar]
  30. Zhang, J.; Ghanem, B. ISTA-Net: Interpretable optimization-inspired deep network for image compressive sensing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 1828–1837. [Google Scholar]
  31. You, D.; Xie, J.; Zhang, J. ISTA-NET++: Flexible Deep Unfolding Network for Compressive Sensing. In Proceedings of the 2021 IEEE International Conference on Multimedia and Expo (ICME), Shenzhen, China, 5–9 July 2021; pp. 1–6. [Google Scholar] [CrossRef]
  32. Zhang, J.; Zhao, C.; Gao, W. Optimization-inspired compact deep compressive sensing. IEEE J. Sel. Top. Signal Process. 2020, 14, 765–774. [Google Scholar] [CrossRef]
  33. Xia, K.; Pan, Z.; Mao, P. Video Compressive Sensing Reconstruction Using Unfolded LSTM. Sensors 2022, 22, 7172. [Google Scholar] [CrossRef] [PubMed]
  34. Shi, W.; Caballero, J.; Huszár, F.; Totz, J.; Aitken, A.P.; Bishop, R.; Rueckert, D.; Wang, Z. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. arXiv 2016, arXiv:1609.05158. [Google Scholar]
  35. Li, N.; Zhou, C.C. AMPA-Net: Optimization-Inspired Attention Neural Network for Deep Compressed Sensing. In Proceedings of the 2020 IEEE 20th International Conference on Communication Technology (ICCT), Nanning, China, 28–31 October 2020; pp. 1338–1344. [Google Scholar]
  36. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. Adv. Neural Inf. Process. Syst. 2017, 30, 6000–6010. [Google Scholar]
  37. Lin, T.Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; Zitnick, C.L. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision, Zurich, Switzerland, 6–12 September 2014; pp. 740–755. [Google Scholar]
  38. Bevilacqua, M.; Roumy, A.; Guillemot, C.; Alberi Morel, M.L. Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding. In Proceedings of the British Machine Vision Conference 2012, Surrey, UK, 3–7 September 2012; pp. 135.1–135.10. [Google Scholar] [CrossRef]
  39. Zeyde, R.; Elad, M.; Protter, M. On single image scale-up using sparse-representations. In Proceedings of the International Conference on Curves and Surfaces, Avignon, France, 24–30 June 2010; pp. 711–730. [Google Scholar]
  40. Martin, D.; Fowlkes, C.; Tal, D.; Malik, J. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the Eighth IEEE International Conference on Computer Vision. ICCV 2001, Vancouver, BC, Canada, 7–14 July 2001; Volume 2, pp. 416–423. [Google Scholar]
  41. Huang, J.B.; Singh, A.; Ahuja, N. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 5197–5206. [Google Scholar]
  42. Wang, Z.; Bovik, A.; Sheikh, H.; Simoncelli, E. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef]
  43. Wang, M.; Wei, S.; Liang, J.; Zhou, Z.; Qu, Q.; Shi, J.; Zhang, X. TPSSI-Net: Fast and Enhanced Two-Path Iterative Network for 3D SAR Sparse Imaging. IEEE Trans. Image Process. 2021, 30, 7317–7332. [Google Scholar] [CrossRef]
  44. You, D.; Zhang, J.; Xie, J.; Chen, B.; Ma, S. Coast: Controllable arbitrary-sampling network for compressive sensing. IEEE Trans. Image Process. 2021, 30, 6066–6080. [Google Scholar] [CrossRef]
  45. Song, J.; Chen, B.; Zhang, J. Memory-Augmented Deep Unfolding Network for Compressive Sensing. In Proceedings of the 29th ACM International Conference on Multimedia, Virtual Event, 20–24 October 2021; Association for Computing Machinery: New York, NY, USA, 2021; pp. 4249–4258. [Google Scholar] [CrossRef]
Figure 1. Overview of the proposed IEF-CSNET. The real image X to be sampled and the final output O are marked by solid black and red wireframes, respectively. X also plays the role of ground truth for training. (1) The CIE modules (displayed as purple blocks and introduced in Section 3.2) receive information in the compressed domain (Sampled Data ( SD ) or the error between SD and Stage Sampled Data ( SSD i )) as input. Then, CIE modules output the 1st iteration Corase Result ( CR 1 ) or Compressed-domain Fusion Error Image CFEI i . (2) ECCE modules (displayed as green blocks and introduced in Section 3.3) achieve EEI i by summarizing CR i , CFEI i , and EEI 2 ( i 1 ) from ECCE modules in previous iterations. (3) The N I IIFE modules (displayed as blue blocks) aim to reconstruct images more effectively stage by stage. Each IIFE module can be transmitted with little information loss. It is composed of the Sampling S ( · ) , Iterative Result Extraction (IRE), Iterative Error Extension (IEE), and Error Supplement (ES) modules, which are all introduced in Section 3.4. (4) A total of 3 3 × 3 convolution layers with 32 channels and ReLU layers are employed as the Basic CNN (BCNN) module for nonlinear mapping and output the Intermediate Feature IF i .
Figure 1. Overview of the proposed IEF-CSNET. The real image X to be sampled and the final output O are marked by solid black and red wireframes, respectively. X also plays the role of ground truth for training. (1) The CIE modules (displayed as purple blocks and introduced in Section 3.2) receive information in the compressed domain (Sampled Data ( SD ) or the error between SD and Stage Sampled Data ( SSD i )) as input. Then, CIE modules output the 1st iteration Corase Result ( CR 1 ) or Compressed-domain Fusion Error Image CFEI i . (2) ECCE modules (displayed as green blocks and introduced in Section 3.3) achieve EEI i by summarizing CR i , CFEI i , and EEI 2 ( i 1 ) from ECCE modules in previous iterations. (3) The N I IIFE modules (displayed as blue blocks) aim to reconstruct images more effectively stage by stage. Each IIFE module can be transmitted with little information loss. It is composed of the Sampling S ( · ) , Iterative Result Extraction (IRE), Iterative Error Extension (IEE), and Error Supplement (ES) modules, which are all introduced in Section 3.4. (4) A total of 3 3 × 3 convolution layers with 32 channels and ReLU layers are employed as the Basic CNN (BCNN) module for nonlinear mapping and output the Intermediate Feature IF i .
Sensors 23 01886 g001
Figure 2. The base upsampling module in most related works. SD is displayed as a feature map with 4 channels to represent that each block is sampled into 4 measurements, and then it is upsampled and expanded into 16 channels, which is marked as the purple arrow. After that, the feature map is stretched into the same shape as the ground truth X by pixelshuffle [34]. The image is processed separately block by block, which is shown and divided by the black dotted line. For example, the SD marked by the red dotted line is just processed and stretched into the block in the upper left corner of the image by itself.
Figure 2. The base upsampling module in most related works. SD is displayed as a feature map with 4 channels to represent that each block is sampled into 4 measurements, and then it is upsampled and expanded into 16 channels, which is marked as the purple arrow. After that, the feature map is stretched into the same shape as the ground truth X by pixelshuffle [34]. The image is processed separately block by block, which is shown and divided by the black dotted line. For example, the SD marked by the red dotted line is just processed and stretched into the block in the upper left corner of the image by itself.
Sensors 23 01886 g002
Figure 3. The pipeline of the CIE module. First, the fusion of measurements in the compressed domain is realized by 3 × 3 convolution (indicated by the purple arrow with tail) to obtain another Expanded Sampled Data ( ESD ), which is marked in a purple solid wireframe and is concatenated to the SD to maintain that the original measurements will not be averaged. Then, this fusion of SD and ESD , which is noted as EFSD , can be upsampled and reshaped in a way similar to that given in Figure 2.
Figure 3. The pipeline of the CIE module. First, the fusion of measurements in the compressed domain is realized by 3 × 3 convolution (indicated by the purple arrow with tail) to obtain another Expanded Sampled Data ( ESD ), which is marked in a purple solid wireframe and is concatenated to the SD to maintain that the original measurements will not be averaged. Then, this fusion of SD and ESD , which is noted as EFSD , can be upsampled and reshaped in a way similar to that given in Figure 2.
Sensors 23 01886 g003
Figure 4. The pipeline of the ECCE module. In the iteration i, the module CIE i and its output CFEI i are both marked in purple. The CR i and the previous EEI s output by ECCE s are marked as blue solid wireframes and green dotted boxes, respectively. ECCE i takes CR i , CFEI i , and EEI ( 2 ( i 1 ) ) as input and outputs EEI i . The green arrows represent the 3 × 3 convolution and activation options. I/O channels of the two convolution layers are i/32 and 32/1, respectively, for nonlinear mapping and comprehensive enhancement of EEI .
Figure 4. The pipeline of the ECCE module. In the iteration i, the module CIE i and its output CFEI i are both marked in purple. The CR i and the previous EEI s output by ECCE s are marked as blue solid wireframes and green dotted boxes, respectively. ECCE i takes CR i , CFEI i , and EEI ( 2 ( i 1 ) ) as input and outputs EEI i . The green arrows represent the 3 × 3 convolution and activation options. I/O channels of the two convolution layers are i/32 and 32/1, respectively, for nonlinear mapping and comprehensive enhancement of EEI .
Sensors 23 01886 g004
Figure 5. Parallel sampling module S ( · ) . The dimension of the sampling matrix Φ is set as 32 × 32 × ( 1024 × R ) . The input image of S ( · ) is divided into blocks with 32 × 32 pixels, which is the same size as one channel of Φ . The dividing line is shown in red, where w = W 32 and h = H 32 represent the number of blocks in the width and height, respectively. The block set will be sampled as a 1 × 1 × ( w h ) measurement for a total of 1024 × R parallel executions. Therefore, the sampling rate can be understood as 1024 × R 32 32 = R .
Figure 5. Parallel sampling module S ( · ) . The dimension of the sampling matrix Φ is set as 32 × 32 × ( 1024 × R ) . The input image of S ( · ) is divided into blocks with 32 × 32 pixels, which is the same size as one channel of Φ . The dividing line is shown in red, where w = W 32 and h = H 32 represent the number of blocks in the width and height, respectively. The block set will be sampled as a 1 × 1 × ( w h ) measurement for a total of 1024 × R parallel executions. Therefore, the sampling rate can be understood as 1024 × R 32 32 = R .
Sensors 23 01886 g005
Figure 6. Base module in related works. CR i and error images EI i are represented as blue solid wireframes and green dotted wireframes, respectively. The IF i is marked as blue cubes. The green double arrows mark the total number of feature maps, which can also be understood as the reconstructed information flow contained in the network.
Figure 6. Base module in related works. CR i and error images EI i are represented as blue solid wireframes and green dotted wireframes, respectively. The IF i is marked as blue cubes. The green double arrows mark the total number of feature maps, which can also be understood as the reconstructed information flow contained in the network.
Sensors 23 01886 g006
Figure 7. IIFE module. IRE, IEE, and ES modules, marked with dark blue boxes, aim to extract the CR i from IF i , encode and extend the EI i to output abundant information of Error Features EF i , and supplement the main branch IF i by EF i , respectively. Additionally, IF i and EF i are represented as blue and gray cubes, respectively. Compared with Figure 6, there is no forced waste and compression of the information flow marked by green arrows, thus achieving a smoother reconstruction.
Figure 7. IIFE module. IRE, IEE, and ES modules, marked with dark blue boxes, aim to extract the CR i from IF i , encode and extend the EI i to output abundant information of Error Features EF i , and supplement the main branch IF i by EF i , respectively. Additionally, IF i and EF i are represented as blue and gray cubes, respectively. Compared with Figure 6, there is no forced waste and compression of the information flow marked by green arrows, thus achieving a smoother reconstruction.
Sensors 23 01886 g007
Figure 8. The fusion of errors in ES. At first, the two input feature maps IF i 1 and EF i , which are represented by blue and gray cubes, respectively, are concatenated. Then, the features are encoded into a hidden feature with 32 channels (shown as the green cube) and decoded to achieve the final result IF i . In this way, the full fusion between errors and intermediate features can be achieved. The encoder and decoder are realized by two 3 × 3 convolution and ReLU layers of C N N 64 32 and C N N 32 32 , respectively.
Figure 8. The fusion of errors in ES. At first, the two input feature maps IF i 1 and EF i , which are represented by blue and gray cubes, respectively, are concatenated. Then, the features are encoded into a hidden feature with 32 channels (shown as the green cube) and decoded to achieve the final result IF i . In this way, the full fusion between errors and intermediate features can be achieved. The encoder and decoder are realized by two 3 × 3 convolution and ReLU layers of C N N 64 32 and C N N 32 32 , respectively.
Sensors 23 01886 g008
Figure 9. The results for qualitative evaluation. (a): Ground truth; (b): ReconNet; (c): ISTANet++; (d): CSNET+; (e): AMPNet; (f): COAST; (g): MADUN; (h): Proposed method. The images are randomly selected for comparison under the three sampling ratios of ( 0.04 % , 0.10 % , and 0.25 % ). The detailed parts of the whole image are marked with a red box and shown in an enlarged view below the corresponding image. The indicators of both the complete and enlarged images are calculated and listed.
Figure 9. The results for qualitative evaluation. (a): Ground truth; (b): ReconNet; (c): ISTANet++; (d): CSNET+; (e): AMPNet; (f): COAST; (g): MADUN; (h): Proposed method. The images are randomly selected for comparison under the three sampling ratios of ( 0.04 % , 0.10 % , and 0.25 % ). The detailed parts of the whole image are marked with a red box and shown in an enlarged view below the corresponding image. The indicators of both the complete and enlarged images are calculated and listed.
Sensors 23 01886 g009
Figure 10. The internal interpretability of ECCE. Each line denoted as I i represents the weights of the convolution layer from ECCE i in iteration i. On a specific polyline I i , there are different weight responses for different input channels. Based on these weights, the importance of each channel in the whole task can be measured because the larger the weight is, the higher the proportion of information extracted from the corresponding input channel. Point ( x , y ) indicates how valuable the specific input channel ( EEI x , x [ 2 , i 1 ] or CFEI i , x = i ) is in the calculation of ECCE i .
Figure 10. The internal interpretability of ECCE. Each line denoted as I i represents the weights of the convolution layer from ECCE i in iteration i. On a specific polyline I i , there are different weight responses for different input channels. Based on these weights, the importance of each channel in the whole task can be measured because the larger the weight is, the higher the proportion of information extracted from the corresponding input channel. Point ( x , y ) indicates how valuable the specific input channel ( EEI x , x [ 2 , i 1 ] or CFEI i , x = i ) is in the calculation of ECCE i .
Sensors 23 01886 g010
Table 1. Summary of datasets.
Table 1. Summary of datasets.
DatasetNumberComments
Set55Red-Green-Blue (RGB), unfixed resolutions
Set1111Gray, unfixed resolutions
Set14142 Gray, 12 RGB, unfixed resolutions
BSD6868RGB, fixed resolution
Urban100100RGB, unfixed high-resolution city images
Table 2. The performances of different methods. All methods are tested under five testing datasets and five sampling ratios R. The indicators are shown in PSNR / SSIM format.
Table 2. The performances of different methods. All methods are tested under five testing datasets and five sampling ratios R. The indicators are shown in PSNR / SSIM format.
MethodsRSet5Set11Set14BSD68Urban100Avg ± Std
Reconnet
[23]
1%20.66/0.521119.34/0.471620.15/0.465021.16/0.481618.32/0.426119.92 ± 1.00/0.4731 ± 0.0305
4%24.45/0.659922.63/0.611523.16/0.581323.58/0.576020.82/0.542622.93 ± 1.21/0.5943 ± 0.0394
10%27.82/0.782425.87/0.745925.90/0.693725.79/0.676323.38/0.669725.75 ± 1.41/0.7136 ± 0.0436
25%31.93/0.879629.80/0.857829.28/0.813728.74/0.796526.84/0.802029.32 ± 1.64/0.8299 ± 0.0329
50%35.80/0.935033.89/0.926032.96/0.901332.22/0.893230.69/0.895433.11 ± 1.70/0.9102 ± 0.0170
Avg.28.13/0.755626.31/0.722526.29/0.691026.30/0.684724.01/0.667126.21 ± 1.31/0.7042 ± 0.0313
ISTA-Net++
[31]
1%22.21/0.587220.43/0.523521.24/0.511822.09/0.509519.27/0.468221.05 ± 1.10/0.5200 ± 0.0384
4%26.53/0.796824.85/0.752824.79/0.685824.80/0.655722.71/0.676824.74 ± 1.21/0.7136 ± 0.0528
10%31.47/0.911129.82/0.897228.63/0.822027.64/0.785827.53/0.851329.02 ± 1.48/0.8535 ± 0.0465
25%36.09/0.957734.78/0.956933.03/0.914631.23/0.893932.48/0.939333.52 ± 1.72/0.9325 ± 0.0248
50%41.43/0.982440.19/0.983338.28/0.967236.08/0.961538.14/0.979438.82 ± 1.84/0.9747 ± 0.0088
Avg.31.55/0.847030.02/0.822729.19/0.780328.37/0.761328.03/0.783029.43 ± 1.26/0.7989 ± 0.0313
CSNET+
[26]
1%24.57/0.685322.70/0.625723.20/0.602723.94/0.587621.03/0.559123.09 ± 1.21/0.6121 ± 0.0425
4%29.20/0.879926.78/0.842126.72/0.781626.58/0.755524.26/0.765826.71 ± 1.56/0.8050 ± 0.0480
10%32.97/0.941830.38/0.918829.68/0.874028.93/0.851927.26/0.868729.84 ± 1.88/0.8910 ± 0.0337
25%37.35/0.972135.00/0.962933.69/0.940732.55/0.932031.56/0.942334.03 ± 2.02/0.9500 ± 0.0150
50%42.47/0.987940.77/0.987638.75/0.976837.56/0.977236.96/0.979839.30 ± 2.05/0.9819 ± 0.0049
Avg.33.31/0.893431.13/0.867430.41/0.835229.91/0.820928.21/0.823230.59 ± 1.66/0.8480 ± 0.0281
AMPNet
[4]
1%24.74/0.698921.61/0.620123.41/0.615324.10/0.596721.34/0.580323.04 ± 1.35/0.6222 ± 0.0408
4%29.44/0.887826.13/0.843327.14/0.788426.82/0.759324.89/0.784226.88 ± 1.49/0.8126 ± 0.0465
10%33.84/0.948030.01/0.920230.43/0.880129.37/0.855128.67/0.889230.46 ± 1.79/0.8985 ± 0.0324
25%38.31/0.975035.12/0.967634.93/0.947033.20/0.933733.88/0.956635.09 ± 1.75/0.9560 ± 0.0147
50%43.53/0.989240.56/0.986840.08/0.978738.26/0.977439.34/0.984840.35 ± 1.77/0.9834 ± 0.0046
Avg.33.97/0.899830.68/0.867631.20/0.841930.35/0.824429.63/0.839031.17 ± 1.49/0.8545 ± 0.0266
COAST
[44]
1%24.05/0.663720.87/0.583622.70/0.584723.62/0.574920.74/0.547322.40 ± 1.37/0.5908 ± 0.0388
4%29.16/0.881325.55/0.833326.71/0.781626.56/0.753724.45/0.773826.49 ± 1.56/0.8048 ± 0.0464
10%33.36/0.944529.45/0.915929.99/0.876129.11/0.851728.06/0.881129.99 ± 1.80/0.8938 ± 0.0326
25%38.20/0.974235.03/0.968034.72/0.946533.08/0.933833.65/0.956534.94 ± 1.78/0.9558 ± 0.0145
50%42.81/0.987939.58/0.985739.13/0.977037.66/0.976037.96/0.982039.43 ± 1.83/0.9817 ± 0.0047
Avg.33.52/0.890330.10/0.857330.65/0.833230.00/0.818028.97/0.828130.65 ± 1.53/0.8454 ± 0.0259
MADUN
[45]
1%24.91/0.716121.80/0.641223.46/0.626924.17/0.604221.56/0.604423.18 ± 1.31/0.6386 ± 0.0412
4%29.94/0.898426.56/0.859527.41/0.798527.03/0.768225.56/0.809427.30 ± 1.46/0.8268 ± 0.0463
10%34.19/0.950330.42/0.926130.66/0.885629.59/0.861229.54/0.905230.88 ± 1.71/0.9057 ± 0.0310
25%38.82/0.975735.88/0.971435.42/0.950933.52/0.937834.85/0.963435.70 ± 1.75/0.9599 ± 0.0139
50%42.36/0.986239.31/0.984938.93/0.974636.99/0.971738.63/0.983939.25 ± 1.75/0.9802 ± 0.0059
Avg.34.04/0.905330.79/0.876631.18/0.847330.26/0.828630.03/0.853331.26 ± 1.45/0.8622 ± 0.0264
CSformer
[3]
1%25.22/0.719721.95/0.624123.88/0.614623.07/0.559121.94/0.588523.21 ± 1.24/0.6212 ± 0.0542
4%30.31/0.868626.93/0.825127.78/0.758125.91/0.704526.13/0.780327.41 ± 1.59/0.7873 ± 0.0562
10%34.20/0.926230.66/0.902730.85/0.851528.28/0.807829.61/0.876230.72 ± 1.97/0.8729 ± 0.0411
25%38.30/0.961935.46/0.957035.04/0.931631.91/0.910234.16/0.947034.97 ± 2.07/0.9415 ± 0.0188
50%43.55/0.984541.04/0.983140.41/0.973037.16/0.971439.46/0.981140.32 ± 2.08/0.9786 ± 0.0054
Avg.34.32/0.892231.21/0.858431.59/0.825829.27/0.790630.26/0.834631.33 ± 1.70/0.8403 ± 0.0339
IEF-CSNET1%25.26/0.728522.21/0.653323.88/0.636324.33/0.609022.04/0.627523.54 ± 1.24/0.6509 ± 0.0414
4%30.31/0.901626.98/0.865627.82/0.803327.17/0.770626.27/0.824727.71 ± 1.39/0.8332 ± 0.0461
10%34.64/0.952231.03/0.932431.09/0.888429.78/0.862630.29/0.913331.37 ± 1.71/0.9098 ± 0.0316
25%39.00/0.975836.20/0.972135.71/0.951933.65/0.938135.36/0.965635.99 ± 1.73/0.9607 ± 0.0139
50%44.17/0.989341.18/0.987740.65/0.979938.67/0.979140.29/0.987040.99 ± 1.80/0.9846 ± 0.0042
Avg.34.68/0.909531.52/0.882231.83/0.851930.72/0.831930.85/0.863631.92 ± 1.44/0.8678 ± 0.0265
Table 3. The number of images that can be processed by different methods per second. The inference of 256 × 256 images is executed 105 times, and the average running time of the next 100 times is taken as the final result of the running time t of each image. All images are processed with B = 1 . The number of pictures that can be processed per second is 1 / t .
Table 3. The number of images that can be processed by different methods per second. The inference of 256 × 256 images is executed 105 times, and the average running time of the next 100 times is taken as the final result of the running time t of each image. All images are processed with B = 1 . The number of pictures that can be processed per second is 1 / t .
MethodsRatio = 0.01Ratio = 0.01
Reconnet137.17132.62
ISTA-Net++44.8044.84
CSNET+93.0291.32
AMPNet39.9537.52
COAST24.7624.87
MADUN16.0016.02
CSformer-0.20
IEF-CSNET36.1135.71
Table 4. The Ablation Experiment.
Table 4. The Ablation Experiment.
R = 0.01R = 0.5
PSNRSSIMPSNRSSIM
W / O IIFE23.400.629140.280.9833
W / O ECCE23.770.651941.180.9848
W / O CIE23.700.647941.240.9849
ALL23.830.655141.310.9850
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Zhou, Z.; Liu, F.; Shen, H. IEF-CSNET: Information Enhancement and Fusion Network for Compressed Sensing Reconstruction. Sensors 2023, 23, 1886. https://doi.org/10.3390/s23041886

AMA Style

Zhou Z, Liu F, Shen H. IEF-CSNET: Information Enhancement and Fusion Network for Compressed Sensing Reconstruction. Sensors. 2023; 23(4):1886. https://doi.org/10.3390/s23041886

Chicago/Turabian Style

Zhou, Ziqun, Fengyin Liu, and Haibin Shen. 2023. "IEF-CSNET: Information Enhancement and Fusion Network for Compressed Sensing Reconstruction" Sensors 23, no. 4: 1886. https://doi.org/10.3390/s23041886

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop