DAMNet: A Dual Adjacent Indexing and Multi-Deraining Network for Real-Time Image Deraining

Abstract

Image deraining is increasingly critical in the domain of computer vision. However, there is a lack of fast deraining algorithms for multiple images without temporal and spatial features. To fill this gap, an efficient image-deraining algorithm based on dual adjacent indexing and multi-deraining layers is proposed to increase deraining efficiency. The deraining operation is based on two proposals: the dual adjacent method and the joint training method based on multi-deraining layers. The dual adjacent structure indexes pixels from adjacent features of the previous layer to merge with features produced by deraining layers, and the merged features are reshaped to prepare for the loss computation. Joint training method is based on multi-deraining layers, which utilise the pixelshuffle operation to prepare various deraining features for the multi-loss functions. Multi-loss functions jointly compute the structural similarity by loss calculation based on reshaped and deraining features. The features produced by the four deraining layers are concatenated in the channel dimension to obtain the total structural similarity and mean square error. During the experiments, the proposed deraining model is relatively efficient in primary rain datasets, reaching more than 200 fps, and maintains relatively impressive results in single and crossing datasets, demonstrating that our deraining model reaches one of the most advanced ranks in the domain of rain-removing.

1. Introduction

Rain traces or drops caused by outdoor weather are nonnegligible to computer vision, especially when capturing images or videos in the outdoor environment [1,2], because of the image quality degradation, leading to the unstable performance of other computer version algorithms [3,4,5,6]. Rain removal task is a primary but important component in basic algorithms, whose performance deeply affects the underlying algorithms based on rain removal models, such as target and segmentation detectors in camera-based autonomous driving or maritime video surveillance systems [7,8,9]. Although some deraining methods maintain high values of peak signal-to-noise ratio ($PSNR$) and structural similarity index measure ($SSIM$), corresponding algorithms are suitable for rain streak removal of a single image, even achieving only elementary real-time speed when an Nvidia Graphics Processing Unit (GPU) with high performance participates. To capture the high real-time speed for multi-image deraining, efficient and robust deraining structures are critical to enabling the efficient deraining ability to the rain removal models, which is also fundamental for advanced computer vision processing.

Witnessing the impressive benefits of deep learning, several state-of-the-art deraining methods have been proposed. Regardless of the computational cost, various deraining methods have high deraining performance for a single image [10,11,12,13,14,15,16] which is also practical for image reconstruction and superresolution. The well-designed iterative multi-stage processes have a nonnegligible impact on anti-degradation. Attention methods and residual structures have accompanied the multi-stage processing, which is based on carefully designed assumptions. Attention methods are conducive to solving “what” or “where” pixels and channels are relatively critical with affordable computational costs [11,17,18]. Residual structures prompt the cooperation of different layers [10,11,12,13,14,15,16] with bearable GPU memory or computational burdens. U-Net, containing global down-up sampling processes, is commonly used to combine detail and abstract features [19]. However, we find that the existing deraining algorithms mainly pay attention to physical methods of rain removal or rely on repetitive structures based on attention blocks, residual connections, and local-global U-Net, lacking the commonality and universality and limiting the freedom of the deraining model. In addition, elaborate designs based on seemingly natural assumptions lead to visible restrictions for real-time requirements. Deraining models based on the multi-strategy of elaborate designs excessively focus on local patterns of structural similarity [10,11,12,13,14,15,16,17,18,20]. Although elaborate designs and assumptions based on local patterns deserve impressive deraining results for a single image, the corresponding deraining methods inappropriately meet the real-time deraining for multiple images even after GPU prompt. Comprehensively, the single image deraining models mainly utilise the repetitive multi-stage or multi-scale operations [10,11,12,13,14,15,21,22] which are designed based on locally designed features and have visible deraining effects. However, the computational burden and inefficiency are nonnegligible, leading to a low inference speed. The model-free method that EfDeRain [23] adopts is a progressive attempt, which abandons the multi-stages or multi-strategies based on locally enhanced features and releases the following optimisation and computational burdens by end-to-end networks. Due to the concise networks of model-free methods, the requirement of practical real-time inference becomes possible. What the deraining algorithms above have in common is the loss calculation method, which is performed by calculating the loss function based on one deraining layer or fused features of multiple layers. Although deraining loss calculation depends on the composite function, the inputs for the loss functions are the same or derived from single features. We regard this loss computation as a “one-off” loss. The one-off loss can measure the global differences in total inputs while ignoring the differences between local features. The direct influence of “one-off” loss on model-free deraining algorithms is the degradation of the generalisation in crossing public datasets in which the training and testing images are derived from different rain datasets. Therefore, a novel model-free deraining method, further balancing inference and performance, is of great importance for deraining single images.

This paper approaches the highly real-time deraining method for multi-image deraining while achieving impressive performance in rainy public datasets. Our proposed method is a model-free algorithm of multi-deraining layers without any rain streak assumptions and subsequent optimisation. The proposed multi-deraining layers are totally different from the multi-stages or multi-strategies utilised by deraining models for a single image [10,11,12,13,14,15,21,22], in which multiple input scales are adopted to enhance rain patterns or multiple stages of local dimension reduction and increase are built based on additional assumptions. Comprehensively, our major contributions are as follows:

• We propose a dual adjacent method that produces new pixels to form a container by indexing pixels from two adjacent features in a periodic interval step. The container stores the new pixels before the end of all new pixel production and is reshaped to prepare for the next fusion with deraining features derived from multi-deraining layers, which is conducive to the diversity and efficiency of subsequent pixel fusion. The dual adjacent method realises the avoidance of utilisation of repetitive multi-strategy methods and bridges the efficient connection between existing U-Net and deraining layers.
• We offer a joint training method for computing the loss based on five deraining layers. The multi-deraining layers utilise the pixelshuffle operations to produce deraining features by maintaining the original volume of pixels. The multi-loss functions jointly compute the losses between the ground truth (GT) and each merged feature generated based on features of multi-deraining layers and the reshaped image of dual adjacent operations. The joint training method utilising pixelshuffle operation avoids pixel multiplication caused by common up-sampling operations of Pytorch (nn.Interpolate or nn.Upsample) while avoiding the defect of “one-off” loss.

2. Related Works

The deraining methods consist of two components based on the processed data: video and single-image deraining models. The video deraining models mainly focus on space-time features [5,24,25,26], intrinsic characteristics or repetitive local patterns [27,28,29], and deep learning networks [30,31,32]. Although video deraining methods, especially the models utilising deep learning, have high-performance for video deraining, it is inappropriate to immediately use them to remove rain streaks from separate images, because of the absence of temporal features in multiple images or single images. The removal of separate images is more complicated than the deraining of sequence images and videos. In this paper, we confine our discussion to several classic deraining methods of separate images based on deep learning.

Various image-deraining methods have been proposed with the development of deep learning methods. Pioneer attempts [33,34] tend to utilise the advantages of the automatic extraction of the convolutional neural network (CNN), outperforming the traditional deraining methods based on low-frequency part (LFP) and high-frequency part (HFP) [35,36,37]. Recently, multi-stage or multi-scale strategies have been proposed and carefully designed to prompt the performance of deraining models for a single image. The attentive spatial network (SPANet) [14] is introduced based on a multi-directional recurrent neural network (RNN), in which several residual structures and attention methods are applied to extract useful features and iteratively identify rain traces during multi-stage processing. A semi-supervised rain removal method (SIRR) [16] is provided based on a semi-supervised method that elaborately calculates residuals and correlations between inputs and derained images. A rain convolutional dictionary network (RCD-Net) [10] is introduced to remove the degradation of a single image caused by rainy weather, in which inner dictionaries based on CNN encode rain traces, and then the encoded features are processed by the multi-stage structures to iteratively remove the degradation influence. Joint rain detection and removal (JORDERE) [21] is designed to release the adverse impact of rain by multi-task networks, which jointly process features extracted from binary rain images and clean images without rain streaks. A condition variational image deraining (CVID) [12] is offered for a single image deraining based on a conditional variational autoencoder (CVAE), which generates multiple predictions of the derained image to produce the corresponding clean backgrounds and rain traces. A recurrent context aggregation network (RESCAN) [20] is proposed to solve the inverse effects of rain by similarly multi-stage, in which RNN layers are utilised to extract and purify rain patterns to build a memory container aggregating useful features. A progressive recurrent network (PReNet) [15] is proposed by constructing repetitive residual networks in which a recurrent structure is further utilised to approach the correlations of features from multiple stages. More recently, an attentive image deraining network (AIDNet) [11] is offered based on residual attention modules and correlations between ground truth and derained outputs, which are realised by imposing adversarial supervision and loss of wavelet space. A multi-scale hourglass extraction block (MHEB) [13] is proposed to derain single images by local and global features extracted from the multi-scale processing, in which a hierarchical attentive distillation block (HADB) and residual projected feature fusion (RPFF) are utilised to exploit and aggregate the features. CKT&MCR [22] is offered with an elaborately designed multi-strategy structure, which furtherly pushes the rain removal performance to practical usage again. Efficientderain (EfDeRain) [23] is proposed based on a model-free method, with relatively few human interventions that models should focus on.

Comprehensively, these multi-strategy methods consist of three kinds, as shown in Figure 1, containing multi-sampling networks, multi-scale input networks, and multi-residual networks. One common characteristic is the repetitive stages or multiple operations in the middle or global processing of algorithms. Although multi-strategy methods and local pattern assumptions deserve impressive performance for a single image, the corresponding deraining methods pay more attention to repetitive structures, which are inserted in the middle or global processes of algorithms by several times and improperly meet efficient and high real-time requirements for multiple images. Computational costs and inefficiencies are nonnegligible, resulting in a low inference speed even after the accelerating prompts from the GPU. We argue that elaborate prior knowledge and assumptions potentially lead to barriers that hinder the balance between inference speed and performance. Although EfDeRain [23] is a model-free model, the composite loss function is used to calculate the features from different operations at one time(one-off loss), leading to a performance that slightly mismatches the inherent advantage of high-speed processing. In this paper, we exploit a seemingly simple deraining method for multiple images while utilising the pixelshuffle [38] to maintain the number of original pixels during upsampling stages which can avoid repetitive designs based on prior assumptions.

3. Methodologies

The dual adjacent indexing operation indexes and fuses the adjacent degraded pixels to form a container, reshaped to the compatible size for deraining features. Multi-deraining layers are performed to generate deraining features to prepare for the fusion with reshaped features. The joint training method of the multi-loss prompts the training of DAMNet and considers the relationship between local-global features and GT. DAMNet is a model-free deraining method that has a seemingly simple structure without elaborate designs and assumptions.

3.1. Structures

The point of the deraining method is the reasonable suggestion of the learning between the ground truth and degradation, which should be presented by “$DF\left(X\right)$ versus $GT$”, where $GT$ is the ground truth or clear images, and the outputs of $DF\left(X\right)$ are images predicted based on the degraded images X. The function “$DF$” is the platform to realise the deraining method. To construct efficient and concise networks, we first introduce the formalisation of our methods for rain removal to construct our “$DF$”, as shown in Figure 2. Figure 2a is the dual adjacent method consisting of dual adjacent indexing and reshaped operations. Dual adjacent indexing operations fuse the pixels from adjacent channels to prepare the fusion with the corresponding deraining layer. Figure 2b is the training feedback mechanism, in which five deraining layers join the loss calculations based on each deraining layer and the reshaped feature. $SSI{M}_i$ and $SSI{M}_{total}$ are the loss computations based on the criterion of the structural similarity index measure ($SSIM$), and $MSE$ is the global loss computation based on the criterion of the mean square error ($MSE$). Based on the formalisation of Figure 2, we utilise relatively simple CNN layers to build the backbone and propose a dual adjacent indexing and multi-deraining network for real-time image deraining(DAMNet). The structure of DAMNet is shown in Figure 3, representing the mechanism of forwarding propagation.The underlying and head networks of DAMNet consist of 9 CNN layers, respectively. “×” means the multiplying operation of scalars and matrices, “3 × Conv2d” represents three layers of CNN, “Pixelshuffle 2” denotes the “nn.PixelShuffle” [38] of PyTorch and the upsampling fold is “2” whose denotation is similar to other “PixelShuffle” above. DAMNet utilizes the existing upsampling of the head network to perform the multi-deraining works. The container stores the new pixels produced by the dual adjacent method shown in Figure 4, and the multi-deraining operations process the pixels sampled from the container. Figure 4 is the processing of new pixel production, in which new pixels are produced by isometric trilinear interpolation based on periodical and previous pixels. $Origi{n}_F$, $D{A}_{F}1$, … are the basement for the production of the next new pixels. The number of $D{A}_{F}i$ depends on the size severalfold between original images and features. Before the production of each $D{A}_{F}i$, new pixels are stored in a pixel container with a periodical location as the red frame of Figure 2a. We use the pixelshuffle operations to upsample the features that are helpful to maintain the original number of pixels [38], while indirectly compressing the channels compared with nn.Upsample [23] and deconvolution. Each deraining layer uses the pixelshuffle [38] to generate the compressed and deraining features that combine the reshaped features derived from dual adjacent operations. The final deraining layer is the combination of $derain0$ to $derain3$ in the channel dimension.

Multiple loss calculation based on multi-deraining layers is totally different from “one-off” loss utilised in recent deraining models [11,12,13,14,15,21,39] which only calculate the loss based on single features or similar inputs of composite functions. Multiple loss computations based on different deraining features suggest that not only the global difference between GT and combined deraining features can be focused on, but also the contrast between GT and local deraining features can be considered. Multi-deraining layers can avoid the simple and crude learning method of one composite function, such as the loss calculation of EfDeRain [23]. In addition, the combination of multi-deraining layers helps combine the detailed and abstract information in the underlying and head networks and prompts the inherent advantages of CNN. Comprehensively, our deraining method is a multi-deraining structure that utilises the pixelshuffle upsampling structure of head networks and relatively simple underlying networks to avoid the multi-stage strategies [11,12,13,14,15,21,22,39]. The multi-deraining structure is different from the repetitive multi-strategy methods, our multiple deraining layers are directly used for the deraining performance at the end of global networks, while repetitive multi-strategy methods are utilised for the enhancement or information of the features accumulated in the middle or global processing of algorithms, as shown in Figure 1. The primary workflow of DAMNet is shown in Algorithm 1 which, from another angle, directly shows the avoidance of the repetitive multi-stage methods, containing repetitively local down-up sampling processes and multiple information accumulations.

Algorithm 1: The deraining processing of DAMNet.

3.2. The Dual Adjacent Method

The dual adjacent method is the preprocessing for the degraded images, preparing the fusion with the deraining features produced by each deraining layer. The merge operation, which is the multiplication mentioned in Figure 2, informs the adjustable space between degraded pixels and predicted pixels of deraining layers and releases the changing freedom of deraining models. The overall workflow of the dual adjacent method is shown as the red frame in Figure 2a, consisting of dual adjacent indexing and reshaped operations. The dual adjacent indexing operations fuse the pixels from adjacent channels to prepare the processing of the corresponding deraining layer. New pixels are produced by dual adjacent indexing interpolation based on periodical and previous pixels from dual adjacent channels, as shown in Figure 4. Before the production of each $D{A}_{F}i$, the fused pixels are stored in the container by the periodic interval step based on the previous features. The container is indexed to sample the pixels in similar steps, and new features are formed based on the fused pixels, increasing the diversity of indexing pixels and deep fusion capability. The initial location of periodic interval steps is from 0 to $step-1$, which gains the dilation attribution and squeezes the computations indirectly. Finally, the features constructed on new pixels are reshaped to be compatible with the multiplication between multi-deraining layers.

Isometric interpolation, used in the dual adjacent method based on periodical and previous pixels, is different from “nn.Interpolate” and “nn.Upsample”(upsampling operations of Pytorch). “nn.Interpolate” and “nn.Upsample” upsample features to twice or several fold sizes by interpolation, while the isometric interpolation we used is not to upsample features. We utilise isometric interpolation to produce new pixels based on periodical and previous pixels, as shown in Figure 4. The specific interpolation of each generated new pixel is shown in Equations (1) and (2), displaying the dual adjacent operations to generate new pixels.

Comprehensively, the dual adjacent method realises the avoidance of utilisation of repetitive down-up sampling structures or multi-scale operations [10,11,21,22], in which CNN blocks containing repetitive down-up sampling operations iteratively perform features, and enhanced images join the accumulation processes. The dual adjacent method acts as a middle operation to bridge the efficient connection between existing U-Net and deraining layers.

$$\left\{\begin{array}{cc}\hfill f\left(R_1\right)& =\frac{x_2-x}{x_2-x_1}{V}_{11}+\frac{x-x_1}{x_2-x_1}{V}_{21}\hfill \\ \hfill f\left(R_2\right)& =\frac{x_2-x}{x_2-x_1}{V}_{12}+\frac{x-x_1}{x_2-x_1}{V}_{22}\hfill \\ \hfill f\left(R_1^{\prime }\right)& =\frac{x_2^{\prime }-x^{\prime }}{x_2-x_1}{V}_{11}^{\prime }+\frac{x^{\prime }-x_1^{\prime }}{x_2-x_1}{V}_{21}^{\prime }\hfill \\ \hfill f\left(R_2^{\prime }\right)& =\frac{x_2^{\prime }-x^{\prime }}{x_2^{\prime }-x_1^{\prime }}{V}_{12}^{\prime }+\frac{x^{\prime }-x_1^{\prime }}{x_2^{\prime }-x_1^{\prime }}{V}_{22}^{\prime }\hfill \end{array}\right.$$

(1)

where $V_{11-22}$ and $V_{11-22}^{\prime }$ represent eight pixel values in the dual adjacent features. $x_{1-2}$ and $x_{1-2}^{\prime }$ denote the locations of corresponding eight pixels. x and $x^{\prime }$ mean the position of new pixels periodically projected on the dual adjacent features.

$$\mathrm{Container}\stackrel{\mathrm{indexing}}{\Leftarrow }d\frac{\left(y_2-y\right)f\left(R_1\right)+\left(y-y_1\right)f\left(R_2\right)}{y_2-y_1}+d^{\prime }\frac{\left(y_2^{\prime }-y^{\prime }\right)f\left(R_1^{\prime }\right)+\left(y^{\prime }-y_1^{\prime }\right)f\left(R_2^{\prime }\right)}{y_2^{\prime }-y_1^{\prime }}$$

(2)

where $y_{1-2}$ and $y_{1-2}^{\prime }$ denote the normalised position of new pixels projected on the dual adjacent features. y is the location of new pixels planned to generate. d and $d^{\prime }$ are adjusting parameters that measure the scales between dual adjacent features, usually set to 0.5 and 0.5. Each generated pixel is temporarily stored in a “Container”.

3.3. Joint Training Method

The loss calculation of DAMNet consists of two components, $MSE$ and $SSIM$. $PSNR$ and $SSIM$ are popular criteria for measuring performance. As shown in Equation (3), $PSNR$ is the function of $MSE$, which both measures the difference between individual pixels, while the computational complexity of $PSNR$ is O($log{N}^2$) times higher than $MSE$, leading to relatively low training speed. In the case of numerical normalisation, $PSNR$ tends to encounter non-differentiable cases. Therefore, we choose the $MSE$ as the loss function tending to measure the changes in individual pixels. Considering the inconstancy of $MSE$ during the initial training steps, we utilise $MSE$ only in the final deraining layer. $SSIM$ is the criterion measuring the global difference between degraded images and inputs, whose relative stability is conducive to combining each deraining layer. We choose $SSIM$ and $MSE$ as the two components of the joint training method, shown in Figure 2b, multi-loss functions computing losses of 5 deraining layers.

$$PNSR\iff f\left(MSE\right)=10{log}_{10}\frac{N{L}^2}{{\sum }_i^N{\left(x_i-y_i\right)}^2}$$

(3)

where N is the number of pixels, L is the maximum value of pixels, and $x_i$ and $y_i$ are the corresponding pixel values. $PSNR$ is the peak signal-to-noise ratio, and $SSIM$ is the structural similarity index measure.

The elements of the multi-loss function we directly use in each deraining layer are the structural similarity index measure ($SSIM$), as shown in Equation (4), whose function is the simplified formula under the normalisation of pixel values. Due to the exchanges of data types, it is inevitable to influence individual pixel values and make non-negligible changes for “$c_1$” and “$c_2$”. Therefore, $MSE$ is utilised in the final deraining layer to weaken the adverse effect of changing datatypes and offset the overall computational complexity [23].

The deraining layers, utilising pixelshuffle, and the joint multi-loss function, combining the loss functions of Figure 2, are indivisible. The joint training method is based on the joint multi-loss function and deraining layers. Deraining layers use pixelshuffle [38] to generate various deraining features, as shown in Figure 3. If we adopt the common up-sampling approach that recent deraining models have taken, the number of pixels will be multiplied, leading to multiple computational burdens for the following structures. Pixelshuffle orderly dislocates the original pixels to gain deraining features with a severalfold size while maintaining the original number of previous pixels, which makes the joint training method efficient. The calculation of producing deraining features is Equation (5), in which pixels are orderly dislocated in the dimensions of height and width. $\mathcal{DR}$ is the generated deraining features prepared for multi-loss computations.

EfDeRain [23] has an elaborate network and fast inference speed, while the feedback loss computing is only dependent on the final layer of head networks. Although it has a relatively fast training speed, this “one-off” loss method ignores the local patterns stemming from the different features. The multi-loss function of the joint training method is conducive to training the overall network and combining abstract and detailed features. The multi-loss function is directly defined as Equation (6), in which we keep the sum of the coefficients of all $SSIM$ [40] computations as one, and the weights of $SSIM$ and $MSE$ [41] are globally equal. Although the multi-loss function is based on multiple deraining layers, leading to more computational burdens than EfDeRain, we utilise pixelshuffle [38] operations to maintain the volume of the original pixel during the upsampling processes, denoting the following processings have a quarter of the computation, compared with nn.Upsampling used in EfDeRain [23]. The following experiments show that multi-loss function and joint training based on multi-deraining layers improve the performance in the single and crossing datasets while maintaining a high real-time speed.

$$\left\{\begin{array}{c}SSI{M}_x=\frac{\left(2D{F}_x\times GT+c_1\right)\left({\sigma }_{D{F}_{x}GT}+c_2\right)}{\left(D{F}_x^2+G{T}^2+c_1\right)\left({\sigma }_{D{F}_x}^2+{\sigma }_{GT}^2+c_2\right)}\hfill \\ MSE=\frac{1}{M\times N}\sum _{0\le i\le N}\sum _{0\le j\le M}{\left(f_{ij}-f_{ij}^{\prime }\right)}^2\hfill \end{array}\right.$$

(4)

where x∈ { $0,1,2,3,4$ }, $SSI{M}_x$ is the $SSIM$ between the $x_{th}$$DF$ and inputs, the $x_{th}$$DF$ is the prediction from one of 5 deraining layers. $D{F}_x$ and $GT$ are the average values of $x_{th}DF$ and inputs, while ${\sigma }_{D{F}_x}^2$ and ${\sigma }_{GT}^2$ are the corresponding variance. ${\sigma }_{D{F}_{x}GT}$ is the covariance of $x_{th}DF$ and Inputs. $c_1$ and $c_2$ are two constants utilised to maintain stability. N and M are the height and width of derained images or inputs, while $f_{ij}-f_{ij}^{\prime }$ is the pixel preparation to mean square error.

$$\left\{\begin{array}{c}\mathcal{F}=W_l\times F_l\left(W_{l-1}\times F_{l-1}+b_{l-1}\right)+b_l\hfill \\ \\ \mathcal{DR}=\mathcal{F}[\dots ,::2,1::2]{\cup }^{\prime }\mathcal{F}[\dots ,1::2,::2]{\cup }^{\prime }\mathcal{F}[\dots ,::2,::2]{\cup }^{\prime }\mathcal{F}[\dots ,1::2,1::2]\hfill \end{array}\right.$$

(5)

where $W_l$, $F_l$, and $b_l$ represent the trainable kernels weights, previous features, and adjustable constants whose denotations are the same as $W_{l-1}$, $F_{l-1}$, and $b_{l-1}$. ${\cup }^{\prime }$ denotes the union set(∪) with interval steps in which pixels are orderly dislocated in the dimensions of height and width. $\mathcal{F}[\dots ,::2,1::2]$ and related presentations are the Python index method for the eigenmatrix. $\mathcal{DR}$ is the generated deraining features prepared for loss computations.

$$Loss=\sum _{x\in 0,1,2,3}-0.1\times SSI{M}_x({\mathcal{DR}}_x\times \mathcal{RS},GT)-0.5\times SSI{M}_{total}({\mathcal{DR}}_{total}\times \mathcal{RS},GT)+MSE$$

(6)

where $SSI{M}_{total}$ is the $SSIM$ of the $derain$ 4 and $Inputs$, $SSI{M}_x$ is the structural similarity of the x$th$ deraining layer and x has four values representing the location of four deraining layers. ${\mathcal{DR}}_x$ and ${\mathcal{DR}}_{total}$ represent each deraining feature and the final concatenated deraining feature. $GT$ is the clear and undegraded images.

4. Experiments and Results

4.1. Settings

Experimental condition: The experimental core software contains PyTorch 1.71, and CUDA10.2, and the core hardware contains Nvidia Titan V, whose performance is comparable to that of the NVIDIA Quadro P6000 GPU.

Datasets: Rain1400 [34], Rain100H [21], Real-world SPA dataset [14], and Raindrop dataset [42] are used to train and test related models, while each high real-time model trained in one dataset above is retested in the other three datasets, whose results directly demonstrate the generalization ability.

Training tricks: We train our model proposed 5000 epochs with a batch size of 16 in Rain100H, 1200 epochs with a batch size of 16 in Rain1400, 5000 epochs with a batch size of 16 in Raindrops, 15 epochs with a batch size of 12 in Real-world dataset, respectively. The optimiser is Adam, and the learning rate is controlled by an adjusting function, in which no changes happen before the half-training. Then the learning rate slowly decreases in a straight line to zero at the end. There is no data augmentation and fancy training tricks, and the training tricks are simple.

Metrics: The metrics contain objective and subjective evaluations. The objective metrics are $SSIM$ and $PSNR$, representing the performance of deraining models in specific numerical quantification. $SSIM$ and $PSNR$ are popular and accepted, and the larger the two values are, the better the comparison results. The $PSNR$ calculation is shown in Equation (4). The subjective metrics [43], taken as an assistant way to further distinguish the results by human eyes, are evaluations of colour changes, preserved detailed features, and deformations, which should be considered because adverse effects should be carefully restrained accompanying the deraining processing.

$$PSNR=10·{log}_{10}\left(\frac{MA{X}_I^2}{MSE}\right)=20·{log}_{10}\left(\frac{MA{X}_I}{\sqrt{MSE}}\right)$$

(7)

where $MA{X}_I$ is the maximum value of the pixels.

4.2. Comparison on Public Datasets

We have compared DAMNet with several classic deraining methods on Rain1400 [34], Rain100H [21], and Real-world SPA [14]. All testing results are obtained after acceleration from a single GPU with comparable performance. We divide these algorithms into three categories, high real-time (1000/fps $<=$ 10 ms), real-time (10 < 1000/fps $<=$ 40 ms), and low real-time (1000/fps > 40 ms) kinds, based on the average time used of all derained images.

The results comparison on Rain100H is shown in

Table 1. Experimental comparison on Rain100H [21].

Rain100H
Derainor	$PSNR$	$SSIM$	Speed (ms)
DAMNet	31.49	0.914	4.16
EfDeRain [23]	30.35	0.883	4.05
AIDDWT [11]	28.90	0.910	504.12
RCDNet [10]	31.50	0.910	424.80
CVID [12]	27.90	0.855	284.80
JORDERE [21]	30.10	0.890	14.40
SPANet [14]	25.01	0.825	10.05
PReNet [15]	30.12	0.907	50.01
SIRR [16]	22.50	0.725	354.80

. Although our deraining model is a relatively simple and model-free network, DAMNet gains parallel performance compared with the RCDNet [10] which is an elaborate and intricate model built on inner dictionaries of the rain traces. Owing to avoiding the repetitive multi-stages and utilising the pixelshuffle structure of head networks, DAMNet achieves a high real-time inference, far faster than RCDNet, while maintaining impressive deraining effects. In contrast, EfDeRain [23] has a comparable inference speed compared with our model, while DAMNet achieves better performance because of the adoption of joint training based on multi-loss functions. Although the five deraining layers lead to more inference burdens than the one deraining layer of EfDeRain, our deraining models maintain a high real-time speed by taking advantage of the dual adjacent method. In addition, we confirm that deraining models for a single image have a low inference speed even after GPU prompts. Because deraining methods utilised for a single image belong to the low real-time series, when the specific inference speed is not confirmed, the spent time per image is approximately assessed by “>40 ms”. Compared with the deraining models for a single image [11,12,14,21], DAMNet achieves the fastest inference speed while gaining impressive results, suggesting our muti-deraining method has practical effects despite the adoption of relatively simple networks. The visible comparison is shown in Figure 5, DAMNet achieves sufficient deraining effects while persevering visible detailed features. Overall, our deraining model better balances the inference speed and performance on Rain100H [21], whose rain backgrounds are more complex than those of Rain100L [21].

The performances on Rain1400H are shown in

Table 2. Experimental comparison on Rain1400 [34].

Rain1400
Derainor	$PSNR$	$SSIM$	Speed (ms)
DAMNet	32.40	0.931	4.40
EfDeRain [23]	32.29	0.927	4.06
MHEB [13]	33.51	0.954	>40
CKT&MCR [22]	33.13	0.930	>40
AIDDWT [11]	31.00	0.925	580.21
RCDNet [10]	33.14	0.947	549.00
CVID [12]	29.10	0.937	240.90
JORDERE [21]	32.04	0.930	11.22
SPANet [14]	29.80	0.914	8.12
PReNet [15]	32.34	0.945	60.31
SIRR [16]	28.51	0.890	400.00

. MHEB [13] achieves the best performance in Table 2, while the corresponding network neglects the computational burden and only considers the deraining effect for a single image, leading to a low inference speed. Conversely, DAMNet better balances deraining effects and inference speed with a model-free structure. Although the performance of our model outperforms EfDeRain by a narrow margin on the Rain1400 [34], the practical deraining effect of DAMNet is more visible and the restriction for adverse effects is recognisable, as shown in Figure 6. Ears colour changes and deformation appear in the results of most deraining models, but the deraining effect of DAMNet is satisfactory while the adverse effects are restrained very well.

The demonstrations on the Realworld dataset are shown in

Table 3. Experimental comparison on Realworld [14].

Realworld
Derainor	$PSNR$	$SSIM$	Speed (ms)
DAMNet	42.48	0.984	3.70
EfDeRain [23]	41.09	0.982	3.62
MHEB [13]	42.72	0.987	>40
RCDNet [10]	41.20	0.982	425.50
JORDERE [21]	40.80	0.981	18.10
SPANet [14]	40.22	0.980	12.23
PReNet [15]	40.10	0.981	75.30
SIRR [16]	35.50	0.943	362.00

. Although achieving the best deraining effect, RCDNet [10] removes the rain streaks with a low inference speed even after the acceleration of the GPU. DAMNet achieves a parallel performance compared with MHEB [13] and balances the performance and inference speed, demonstrating the advantage of the structure of DAMNet. Although outperforming EfDeRain [23] by a narrow margin, only considering the values of $PSNR$ and $SSIM$, DAMNet performs better in the constraint of adverse deraining effects, as shown in Figure 7. Although relatively thicker greens are visually better, whether to add a colour attribute that does not exist is another manifestation of the algorithm’s ability. From Figure 7, DAMNet best maintains the original colour.

The practical effects of raindrop removal are shown in

Table 4. Experimental comparison on Raindrops [42].

Raindrops
Methods	$PSNR$	$SSIM$	Speed (ms)
DAMNet	29.69	0.911	4.10
EfDeRain [23]	28.48	0.897	4.06
HSNet [45]	26.73	0.831	>40
DAM [46]	30.26	0.914	>40
ARNet [42]	31.57	0.902	>40
JORDERE [21]	27.52	0.824	12.45
EIGEN [44]	28.59	0.673	>40
Pix2Pix [47]	30.14	0.829	>40

. Raindrops are different from Rain100H, Rain1400, and Realworld. Except for the DAMNet we proposed and EfDeRain [23], the models in Table 4 are specific algorithms for removing raindrops. DAMNet and DfDeRain are directly applied and validated in Raindrops while achieving satisfactory effects, even better than the deraining effects of JORDERE [21] and EIGEN [44]. Although the specific inference speed in our platform can not be confirmed, we assign them to “>40 ms” considering their complex structures and the time consumed in public papers. Compared with EfDeRain, one of the high real-time kinds of Table 4, DAMNet achieves a better balance between performance and inference speed. The removal raindrop effects of DAMNet and EfDeRain [23] can demonstrate the model-free structures, avoiding multi-strategies for elaborate assumptions, have a strong generalisation ability and enough free adjustable space to fit the degraded pattern. Although belonging to the model-free method, DAMNet achieves higher values of $SSIM$ and $PSNR$ than EfDeRain [23] which adopts the “one-off” loss method. Besides, DAMNet curbs the side effects at full steam and demonstrates satisfactory generalisation ability, as shown in Figure 8.

Comprehensively, DAMNet reaches the degree of high real-time speed in all four datasets while maintaining a stable performance. Although five deraining layers acquire more computational cost than EfDeRain, which only adopts one deraining layer, DAMNet utilises the pixelshuffle operation to form a fast U-Net to maintain the parallel inference speed with EfDeRain. Our deraining model reaches one of the most efficient models in single datasets in which the training and testing images are different but stem from the same datasets. The comparison of visualisation on single datasets is also given in Figure 9. The deraining effects of DAMNet outperform EfDeRain, while the adverse deraining effects of DAMNet are better restrained.

Different datasets have different properties, and the method of training and testing in different or crossing datasets is a practical way to demonstrate the generalisation of the high real-time deraining models. Therefore, we compare DAMNet with EfDeRain in the crossing datasets in which the training and testing images are derived from different datasets. The generalisation of our model outperforms EfDeRain [23] by an impressive margin, as shown in

Table 5. The comparison on crossing-rain datasets. The models’ suffixes are datasets for training themselves. DAMNet is the multi-deraining model proposed, and EfDeRain [23] is the model with only one deraining layer. The numbers highlighted in bold denote that DAMNet outperforms EfDeRain on the same crossing datasets under the same conditions.

	Rain100H ($\mathrm{PSNR}$)	Rain1400 ($\mathrm{PSNR}$)	SPA ($\mathrm{PSNR}$)	Raindrops ($\mathrm{PSNR}$)	$\mathrm{MEAN}$
DAMNet_rain100H	31.49	26.36	32.89	23.43	28.543
EfDeRain_rain100H	30.35	25.55	31.83	23.18	27.728
DAMNet_rain1400	15.06	32.38	30.71	23.46	25.403
EfDeRain_rain1400	14.73	32.29	31.08	23.52	25.405
DAMNet_SPA	12.96	25.48	42.48	23.5	26.105
EfDeRain_SPA	12.97	25.29	41.09	24.13	25.870
DAMNet_raindrops	12.53	23.05	27.21	29.69	23.120
EfDeRain_raindrops	11.89	21.6	26.3	28.48	22.068
	Rain100H ($\mathit{SSIM}$)	Rain1400 ($\mathit{SSIM}$)	SPA ($\mathit{SSIM}$)	Raindrops ($\mathit{SSIM}$)	$\mathit{MEAN}$
DAMNet_rain100H	0.914	0.865	0.933	0.823	0.884
EfDeRain_rain100H	0.8834	0.854	0.93	0.818	0.871
DAMNet_rain1400	0.46	0.931	0.924	0.817	0.783
EfDeRain_rain1400	0.444	0.927	0.926	0.82	0.779
DAMNet_SPA	0.398	0.82	0.984	0.822	0.756
EfDeRain_SPA	0.399	0.811	0.9825	0.821	0.753
DAMNet_raindrops	0.353	0.752	0.877	0.911	0.723
EfDeRain_raindrops	0.352	0.741	0.873	0.8971	0.716

DAMNet_rain100H and EfDeRain_rain100H mean that the training images of DAMNet and EfDeRain are derived from the Rain100H. Raindrops (PSNR) and Raindrops (SSIM) denote that the performance results are achieved when testing on the Raindrop [42]. Other similar naming ways are the same as the interpretation above.

. The models’ suffixes are datasets for training themselves. The values highlighted in black represent that DAMNet outperforms EfDeRain under correspondingly identical training conditions. DAMNet achieves the higher mean value of $SSIM$ in all crossing datasets while keeping an overall advantage by measuring $PSNR$. Comprehensively, our deraining model performs relatively stably, which is why we adopt the dual adjacent indexing and the joint training methods. Although the multi-deraining layers lead to affordable computing burdens compared with one deraining layer of EfDeRain, the generalisation ability of DAMNet has practically improved, demonstrating our deraining method is a more efficient model-free algorithm. The visualisation of the results on crossing datasets is shown in Figure 10. Although the derained images have visible rain streaks, the more obvious tendency to remove rain tracks is visible.

4.3. Ablation Study

Rain100H is the dataset most severely disturbed by rain streaks, compared with Rain1400 and Realworld, which can be reflected in Figure 5, Figure 6 and Figure 7. The ablations of DAMNet on Rain100H can evaluate the relationship between the dual adjacent method and the joint training method of computing the loss of each deraining layer. Experimental ablations on Rain100H are shown in

Table 6. The ablation of DAMNet on Rain100H, approaching the relationship of the proposals.

	DAMNet	DAMNetfinalLoss	DAMNet_noDual	DAMNet_noMSE	DAMNet_noSSIMtotal	DAMNet1234
$PSNR$	31.49	30.5	30.36	29.5	30.1	18.28
$SSIM$	0.914	0.901	0.885	0.88	0.89	0.565
	DAMNet34	DAMNet0123	DAMNet012	DAMNet01	DAMNet04	DAMNet234
$PSNR$	18.12	18.46	18.3	17.912	26.65	18.26
$SSIM$	0.501	0.588	0.5838	0.5285	0.8403	0.55

DAMNet_noDual is the model that eliminates the dual adjacent indexing operation and is directly reshaped to the next merged stage. DAMNet_noMSE and DAMNet_noSSIMtotal represent the loss calculation eliminating MSE and SSIM_total in the loss computation of the final deraining layer. DAMNetfinalLoss is the deraining model proposed with one-off loss computation, whose loss computation is the same as the EfDeRain. The models with number affixes are the version of DAMNet maintaining the corresponding deraining layers.

. The results of DAMNetfinalLoss are lower than DAMNet, which represents the “one-off” loss computation that definitely weakens the deraining effects and is also why DAMNet outperforms EfDeRain in single and crossing datasets. Comparing the results of DAMNet_noDual and DAMNet, we can confirm that the dual adjacent method can offer more useful information than the direct reshaped operations. The only participation of all deraining layers rarely maintains DAMNet performance when the dual adjacent method is absent. Compared with DAMNet04 saving both the first and final deraining layers, the models just maintain first or final deraining layers have an impressive decrease in deraining effects, dropping approximately 38.2% for $PSNR$, which denotes the combination of first and final deraining layers is critically important for the deraining effect of DAMNet. Considering the comparison of DAMNet04 and DAMNet, the abandonment of middle deraining layers weakens the performance, whose phenomenon occurs in DAMNet_noMSE and DAMNet_noSSIMtotal. The $PSNR$ gap between DAMNet_noDual and DAMNet reaches more than 1%, which is larger than the comparison in Table 1, Table 2, Table 3 and Table 4, denoting the dual adjacent method is also nonnegligible for improving the performance. Comprehensively, DAMNet has the best performance, indicating that each deraining layer and the dual adjacent method are indispensable to gaining the efficiency and robustness of DAMNet, and suggesting the joint training method of multi-loss on the combination of multi-deraining layers is critical to prompt the performance.

5. Discussion

Rain streaks are a nonnegligible degradation factor to computer vision, because of the image quality degradation, leading to the unstable performance of advanced algorithms, such as camera-based autonomous driving or maritime video surveillance systems. Although some deraining methods maintain satisfactory deraining effects and multi-strategies utilised by deraining models for a single image deserve impressive results, corresponding algorithms are suitable for rain streak removal of a single image, which pays more attention to the performance and neglects the computational burdens. Model-free methods abandon the seemingly elaborate designs and release the constraints raised by those multi-strategies. To capture the high real-time speed for multi-image deraining, not just a single image, DAMNet is proposed based on the dual adjacent method and the joint training of multi-deraining layers. DAMNet improves the deraining effects while maintaining a high real-time speed. However, DAMNet bears a bit more computational burden than EfDeRain, adopting only one deraining layer. Although achieving a high real-time speed, DAMNet is not a lightweight deraining model, which is not totally compatible with edge devices. Although seemingly simple structures of DAMNet achieve high real-time inference speed compared with those deraining models with elaborate designs, the container prebuilt for the dual adjacent method trades additional spaces for the avoidance of multiple upsampling operations. We have decided to prepare more lightweight models compatible with edge devices while maintaining the balance between performance and inference speed. In addition, the side-effect raised by over-deraining processing is non-negligible when enhancing the image quality. Therefore, a better balance between rain removal and corresponding side effects is our further attempt.

6. Conclusions

To capture the high real-time speed for multi-image deraining, not just a single image, DAMNet is an efficient deraining algorithm for multiple separate images that avoids the iterative multi-strategy methods used for single-image deraining methods. The dual adjacent indexing is introduced to prompt the efficiency of new pixel production and match the number of deraining features. joint training methods is proposed to jointly train DAMNet and improve deraining efficiency while ensuring high speed. Experiments show that DAMNet has a high real-time speed of more than 200 fps and maintains an immediate and stable deraining effect in single and crossing datasets. In addition, DAMNet has a practical rain removal effect, while suppressing the side effects of rain removal to the relative best level. Comprehensively, DAMNet has reached the advanced level of high-efficient deraining models.