Analysis of Blur Measure Operators for Single Image Blur Segmentation

Abstract

Blur detection and segmentation for a single image without any prior information is a challenging task. Numerous techniques for blur detection and segmentation have been proposed in the literature to ultimately restore the sharp images. These techniques use different blur measures in different settings, and in all of them, blur measure plays a central role among all other steps. Blur measure operators have not been analyzed comparatively for both of the spatially-variant defocus and motion blur cases. In this paper, we provide the performance analysis of the state-of-the-art blur measure operators under a unified framework for blur segmentation. A large number of blur measure operators are considered for applying on a diverse set of real blurry images affected by different types and levels of blur and noise. The initial blur maps then are segmented into blurred and non-blurred regions. In order to test the performance of blur measure operators in segmentation process in equal terms, it is crucial to consider the same intermediate steps involved in the blur segmentation process for all of the blur measure operators. The performance of the operators is evaluated by using various qualitative measures. Results reveal that the blur measure operators perform well under certain condition and factors. However, it has been observed that some operators perform adequately overall well or worse against almost all imperfections that prevail over the real-world images.

1. Introduction

Blur in images is considered as an undesirable effect because it leads to the loss of the necessary details required for the scene interpretation. Automatic detection of blurred and sharp pixels in an image and their classification into respective regions are very important for different image processing and computer vision applications [1]. The benefits of blur segmentation are exhibited in many applications including but not limited to object detection [2], scene classification [3], image segmentation [4], background blur magnification [5], depth of field extension [6] and depth estimation [7,8]. Blur detection and segmentation for a single image without any prior information is a challenging task.

For blur detection and segmentation, a large number of techniques have been proposed in the literature [9,10,11,12,13,14,15,16,17]. The blur segmentation techniques comprise of a number of key steps. The first and the fundamental step is to employ a blur measure (BM) operator which measures the level of blurriness associated with the pixels or regions of the image and realizes discrimination between sharp and blurred pixels or regions of the image is realized by applying an operator, known as blur measure (BM) operator. In the second step, the features computed through the BM operator(s) are provided to a classifier for segmentation of the blurred and sharp regions. Finally, various postprocessing techniques are deployed for improve the initial segmentation results. In each of the techniques, the quality of the underlying blur measure operator plays a distinguishing role. Recently, Lai et al. [18] reported that the existing techniques perform well on images with certain properties however, their performance is deteriorated in real images. As different blur measure operators recognize and exploit different features of the input blurred image, it is difficult to determine which operator has the best performance under which conditions. Additionally, due to the diversity in the subsequent steps of segmentation process, it becomes unjustifiable to compare the output of one BM operator with that of another one.

In this paper, we aim for a comprehensive study to analyze the performance of the state-of-the-art blur measure operators used to detect and segment blurred and sharp regions of the image. In our study, the performance of recently suggested BM’s is investigated. In addition, a number of operators which have also been studied in this work were originally developed for autofocus and shape from focus (SFF) techniques [19]. They have been brought into this study because their working principle is similar to that of blur measure operators i.e., each of them can distinguish between sharp and blurred regions of the image by manifesting different responses for these different regions. For experiments, we have considered a dataset of real-world blurred images with the diverse levels and types of blur. A number of experiments have been conducted on the sample images for qualitative and quantitative results which have been presented accordingly. The statistical evaluation of different blur measure operators helps to understand their prospective performance attributes.

The rest of the paper is organized as follows. The Section 2 sheds light on the related work done in the past. Section 3 explains the methodology for the blur segmentation, including image data sets, blur measure operators and the evaluation measures. Experimental results and analysis are presented in Section 4. Finally, the work is concluded by Section 5.

2. Related Work

Most of the images captured using optical imaging systems usually contain two types of the regions: blurred and sharp. Blur can be categorized mainly into two types: (a) defocus blur, which is caused by the optical imaging system and (b) motion blur, which is caused by the relative motion between camera and scene objects. Blur in the image causes the deterioration of the image quality in that region. Therefore, it is important to detect and eliminate the blur from the images.

Blur detection techniques can be divided broadly into two classes: (1) techniques that need only one image [9,10,11,12,13,14,15] and (2) techniques that use multiple images [20,21]. Though, high-quality image can be recovered through multi-image scheme, yet it is a very challenging task as it requires to align the images, and thus it poses concerns on its applicability. Moreover, our study is restricted to techniques for single image. Shi et al. differentiate between blurred and sharp regions based on the features like image gradient, spectra in frequency domain and data driven local filters in a multiscale scheme [11]. Bae and Durand focused on the blurriness of the edge pixels by employing multiscale edge detector and then propagated this measure to the rest of the image [5]. Elder and Zucker measured the image blur by calculating first and second order gradients on the edges [22]. Namboodiri and Chaudhuri employed the inhomogenous inversion heat diffusion to measure the defocus blur on the edges and propagated them through graph-cuts [23]. Tai and Brown estimated the defocus blur in [24] by using a local contrast prior on edge gradient magnitudes and then propagated through Markov random field (MRF). Zhuo and Sim looked for the Gaussian blur on edge pixels and then interpolated through matting Laplacian for non-edge pixels [25]. Peng et al. measured the pixel blur by calculating the difference between before and after multi-scale Gaussian smoothing [26]. Then, blur map is refined by employing morphological reconstruction and guided filtering. Zhang and Hirakawa measured the blurriness by exploiting the double discrete wavelet transform [27]. Zhu et al. used localized Fourier spectrum to measure the probability of blur scale for each pixel and then implemented a constrained pairwise energy function [28]. Shi et al. estimated the slight noticeable blur by utilizing sparse representation of image neighborhood windows in dictionary method and this blur is further smoothed by an edge-preserving fitler [12]. Lin et al. estimated local blur by analyzing local and global gradient statistics [29]. Oliveira et al. restored the natural images through their spectrum by estimating the parameters for defocus and motion blur kernels [30]. Tang et al. proposed a general framework to retrieve a course blur map for both defocus and motion blurred images by using the log averaged spectrum residual of the image, and then updated it iteratively to achieve the fine map by exploiting the intrinsic relevance of the neighboring similar regions of the image [13]. Yi and Eramian proposed a blur measure operator based on the difference between the distributions of uniform local binary patterns (LBP) of the blurred and proposed a robust algorithm to segment the defocus blur [14]. Golestaneh and Karam detected the spatially varying blur by applying multiscale fusion of the high frequency Discrete Cosine Transform (DCT) coefficients [31]. Chakrabarti et al. used a mixture of Gaussians to model the heavy-tailed natural image gradients and estimated the likelihood of spatially varying directional motion blur in the image using local Fourier transform [32].

Spatially uniform motion deblurring techniques are discussed in [33,34,35,36,37,38]. Most of the uniform deblurring techniques have difficulty in handling non-uniform blurred images. Pan et al. and Hyun et al. have discussed the problem of motion deblurring for general dynamic scenes in which object motion and camera shake both are involved [33,34]. Now, we mention spatially non-uniform camera motion deblurring techniques. Whyte et al. simplified a general projective model proposed by [39] and employed a variational Bayesian framework for image deblurring [35]. Gupta et al. utilized motion density functions to estimate the camera motion trajectory [36]. Shan et al. analyzed the rotational motion blur using a transparency map [38]. Generally, it is computationally expensive if optimization is to be applied in any of these non-uniform deblurring techniques. To tackle it, neighborhood window-based locally uniform techniques are developed which involve fast Fourier transform to provide computationally efficient results. These techniques are to counter for the camera motion [37].

In the following paragraphs, we highlight the spatially non-uniform object motion deblurring techniques. Raskar et al. [40] removed the motion blur by preserving the high frequency content of the imaged scene by fluttering the camera’s shutter open and close during the exposure time according to a well chosen binary sequence. Tai et al. [41] coupled the standard high resolution camera with an auxiliary low resolution camera to combine their data streams for deblurring the motion blur efficiently. The alpha matte solution proposed by Levin et al. [42] has been utilized by [43,44] to segment the image into two layers foreground and background for deblurring. The $L_0$ intensity prior used in [45] favors the images having more pixels with zero intensity and thus can reduce light streaks and saturated regions. [46] can deblur images with large motion blur in the presence of noise. Its refinement phase helps reduce noise in blur kernels and leads to robust deblurring results.

Spatially uniform defocus blur is studied in [47,48,49], while partially non-uniform defocus blur is addressed by [9,15,50]. Cheong et al. [51] obtained the blur amount by estimating the parameters of space-variant PSF using the local variance of first and second order derivatives. Chan and Nguyen separated a defocused background from the focused foreground in an image using a matting method, implied blind deconvolution to learn uniform blur kernel, and recovered the background using total variation minimization [52]. This produced good results for two-layer, foreground and background, defocus images but couldn’t perform well for multi-depth-layer images. Pan J. et al. proposed the maximum a posterior model based method which jointly estimates the object layer and camera shake under the guidance of soft-segmentation [33]. Hyun et al. have proposed an energy model consisting of weighted sum of multiple blur data models to estimate different motion blurs. Adopting non-local regularization of weights, a convex optimization approach is followed to solve the energy model [34].

3. Material and Method

3.1. Image Dataset

Most of the real-world images are usually affected by the complex and varied nature of the blur. In such cases, blur assessment and segmentation turns out to be a challenging task. Generally, different blur measure operators proposed in the literature are sensitive/receptive to different type of features/characteristics/attributes of the blurred image. Consequently, if the performance of these operators is to be compared, a large dataset is imperative that strives to incorporate broadly varied nature of the blur. Out of the available on-line resources, the dataset of [11] seems an appropriate choice and has been considered in this comprehensive study. This dataset contains 1000 real-world partially blurred images. These are of different resolutions and are collected from the Internet. As humans are the ultimate objects to perceive the images, the human labeled ground-truths are also provided for the blurred and sharp regions. There are 704 images corresponding to defocus blur and 296 images representing the motion blur with different magnitudes of defocus and motion respectively. These images cover various scenarios and comprise of numerous attributes like nature, people, vehicles, man-made structure and other living beings. This dataset provides an ample test-bench to evaluate different blur measures. Few images with ground-truths are shown in Figure 1.

3.2. Methodology

In this section, we present a unified framework for the blur detection and segmentation. The blur measure operators investigated in this study have been mentioned in the next section. We have used the publicly available code of the blur segmentation method originally proposed by Yi and Eramian [14]. However, we have introduced a little modification in the blur map classification step of this framework such that we have employed Interquartile Range (IQR) of the initial blur map for its classification instead of some fixed thresholds. This is more compelling choice and its advantage has been proven by better results. All of the steps of the blur segmentation methodology have been performed at three different scales to account for the varying sizes of the blur as suggested in [11] and followed in [14]. Figure 2 depicts the key steps involved in the blur segmentation process by taking the first image of Figure 1 as an example.

3.2.1. Blur Measures

Let $I_b(x,y)$ be the blurred input image for which the amount of blurriness is to be computed at each pixel location. This can be achieved by the application of a blur measure operator B in the local neighborhood window-wise environment around a pixel $\mathsf{\Omega }(x,y)$. For this, B is applied on a local image neighborhood window $\mathsf{\Omega }(x,y)$ centered at pixel $\mathsf{\Omega }(x,y)$ to provide the blur measurement for this pixel. By sliding this neighborhood window for all the image pixels to be at the center of the neighborhood window one-by-one, the whole image can be traversed to result into the generation of the initial blur map $M(x,y)$.

$$M(x,y)=B\left(I_b(x,y)\right).$$

(1)

Then, this initial blur map is normalized linearly so that $M(x,y)\in [0,1]$. The resultant normalized blur map can be expressed as

$$\widehat{M}(x,y)=\frac{M(x,y)-min\left(M\right)}{max\left(M\right)-min\left(M\right)},$$

(2)

where $min\left(M\right)$ and $max\left(M\right)$ are the minimum and maximum measurements in the initial blur map. In the literature, a wide variety of blur detection techniques and blur measure operators have been proposed to measure the level of blur of either the whole image or neighborhood window or individual pixels. In this work, blur measure operators have been grouped into four broad categories according to their working principle. This categorization is to recognize the similarities and differences, if any, in their performances and then rank them accordingly. A brief description of each of the categories is presented in this section. The interested reader is referred to Appendix A for more detailed description of the blur measure operators which have been analyzed in this work. Abbreviations have been used for the blur measure operators as described in

Table 1. Serial numbers and abbreviations to refer the blur measure operators.

Sr. No.	Blur Operator	Abbr.	Sr. No.	Blur Operator	Abbr.
1	Gradient Histogram Span	DER01	17	Gray-level local variance	STA06
2	Kurtosis	DER02	18	Normalized Gray-level variance	STA07
3	Gaussian derivative	DER03	19	Histogram entropy	STA08
4	Gradient energy	DER04	20	DCT energy ratio	STA09
5	Squared gradient	DER05	21	DCT reduced energy ratio	STA10
7	Tenengrad variance	DER07	23	Power spectrum	TRA02
7	Tenengrad variance	DER07	23	High-frequency multiscale Fusion and Sort Transform (HiFST)	TRA02
8	Energy of Laplacian	DER08	24	Sum of wavelet coefficients	TRA03
9	Modified Laplacian	DER09	25	Variance of wavelet coefficients	TRA04
10	Diagonal modified Laplacian	DER10	26	Ratio of wavelet coefficients	TRA05
11	Variance of Laplacian	DER11	27	Brenner’s measure	MIS01
12	Singular value decomposition	STA01	28	Image contrast	MIS02
13	Sparsity of dark channel	STA02	29	Image curvature measure	MIS03
14	Total variation	STA03	30	Steerable filters-based	MIS04
15	Local binary pattern	STA04	31	Spatial frequency	MIS05
16	Gray-level variance	STA05	32	Vollath’s autocorrelation	MIS06

. This is to refer them conveniently, as well as to signify the category to which they belong.

The four categories of blur measure operators analyzed in this work are:

• Derivative-based operators [DER*]: The blur measure operators in this category are based on the derivative of the image. These operators are based on the assumption that non-blurred images present sharp edges as compared to blurred images. First and second order derivatives of the image neighborhood windows provide the base to distinguish between blurred and non-blurred regions of the image.
• Statistical-based operators [STA*]: The blur measure operators of this category utilize several statistical measures which are computed on image neighborhood windows to differentiate between blurred and non-blurred neighborhood windows in the image.
• Transform-based operators [TRA*]: The blur measure operators within this category are based on the transform domain representations of the image content. These frequency domain representations offer to be the true replica of the same information as in the spatial domain and thus this frequency content of the image can be utilized to differentiate between blurred and non-blurred regions of the image.
• Miscellaneous operators [MIS*]: These operators do not belong to any of the previously mentioned categories.

3.2.2. Blur Classification

The mentioned blur measure operators have been applied on the images of the data set in order to obtain their respective blur maps. After acquiring the initial blur map, pixels need to be declared sharp or blurred, and they need to be separated into blurred and sharp regions respectively. This blur classification phenomenon has been carried out in two steps. In the first step, the initial normalized blur map has been divided into three classes by applying a double threshold $T_1\mathrm{and}{T}_2$, as given by,

$$\alpha (x,y)=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}\widehat{M}(x,y)<T_1\hfill \\ 1,\hfill & \mathrm{if}\widehat{M}(x,y)>T_2\hfill \\ I_b(x,y),\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(3)

where $\alpha (x,y)$ is the initial alpha map. Authors of [14] have used some fixed precomputed thresholds. However, since different blur measure operators give different blur measures for the same pixels, the classifying thresholds cannot be held fixed to some aprior. In fact, to acknowledge the diversity of each image and each blur measure operator, these thresholds are to be computed adaptively, for each image—operator pair so as to provide uniform and comparable results for the final blur maps. Therefore, we propose to apply thresholds by computing them adaptively. For this, interquartile range (IQR) of the initial normalized blur map of the image has been considered. IQR is a measure of statistical dispersion and variability, and it divides the data into four equal parts. Here, $Q_1$ and $Q_2$ are taken respectively as the first and third quartiles of the pixel intensity distribution of the image under consideration. By applying this IQR thresholding, pixels in the alpha matting step have been classified initially into three categories: (1) 1 s (2) 0 s and (3) yet to be decided, as expressed in Equation (3). The improved alpha map initialization through IQR can be seen in Figure 3 for the second image of Figure 1.

The pixels which could not have been classified as either 1 or 0 are refined through the optimization in the second step. This refinement of the pixels is achieved through the minimization of the cost function as proposed by [42]

$$E\left(\widehat{\alpha }\right)={\widehat{\alpha }}^T\mathbf{L}\widehat{\alpha }+\lambda {(\widehat{\alpha }-\alpha )}^T(\widehat{\alpha }-\alpha ),$$

(4)

where $\mathbf{L}$ is the matting Laplacian matrix, $\widehat{\alpha }$ is the vectorized alpha map for the pixels in the IQR and $\alpha$ is the vectorized alpha map as computed in Equation (3).

3.2.3. Multiscale Inference

Usually, the blurred images have spatially varying blur sizes. In such a case, the consideration of only one size for the local image neighborhood window may not help in inferring about the blurriness accurately [53]. Therefore, three sizes for the local neighborhood window are considered for blur map generation which correspond to three scales, $s=1,2,3$.

Given an input image, the blur measures are computed at three different scales by applying an operator B and then alpha matting is applied at each scale for classification. After acquiring $\alpha$-maps at three different scales, these are fused together using a multi-scale graphical model as proposed by [11]. The total energy on the graphical model is expressed as

$$E\left(\overline{\alpha }\right)=\sum _{s=1}^3\sum _i|{\overline{\alpha }}_i^{\left(s\right)}-{\widehat{\alpha }}_i^{\left(s\right)}|+\beta \left(\sum _{s=1}^3\sum _i\sum _{j\in {\mathcal{N}}_i^{\left(s\right)}}|{\overline{\alpha }}_i^{\left(s\right)}-{\overline{\alpha }}_j^{\left(s\right)}|+\sum _{s=1}^2\sum _i|{\overline{\alpha }}_i^{\left(s\right)}-{\overline{\alpha }}_i^{(s+1)}|\right),$$

(5)

where ${\widehat{\alpha }}_i^{\left(s\right)}$ represents the optimized alpha map for a pixel i at a particular scale s and ${\overline{\alpha }}_i^{\left(s\right)}$ is the alpha map to be inferred. This multi-scale inference makes it feasible to extract small and large sized blur, which makes our methodology more effective. The first data term on the right is the cost of assigning the alpha map values. The second term consists of two parts and it enforces smoothness in the same scale and across different scales. The first part represents the spatial affinity in the four neighbor set ${\mathcal{N}}_i^{\left(s\right)}$ for pixel i in the s scale. The second part accounts for the inter-scale affinity. The weight $\beta$ steers the relative importance of these two terms. Equation (5) can be optimized using loopy belief propagation [54].

Finally, the output of the proposed methodology is the gray scale image ${\overline{\alpha }}_i^{\left(3\right)}$ which is the inferred blur map at the largest scale. However, as the ground truth is in binary (1 and 0), so ${\overline{\alpha }}_i^{\left(3\right)}$ has also been binarized into 1 and 0 s accordingly by simply applying an adaptive threshold $T_{seg}$,

$${\overline{\alpha }}_i^{\left(3\right)}=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}{\overline{\alpha }}_i^{\left(3\right)}<T_{seg}\hfill \\ 1,\hfill & \mathrm{if}{\overline{\alpha }}_i^{\left(3\right)}>T_{seg}.\hfill \end{array}\right.$$

(6)

3.3. Evaluation Measures

The relative performance of the blur measure operators can be analyzed quantitatively as they are being tested under the same blur segmentation framework. All of the stated blur measure operators have been applied on the data set for which ground truth depth maps are also available. In this case, ground-truth is our actual observation in which the pixels are clearly labeled as either blurred or sharp. The blur maps are our retrieved results which are obtained by applying blur measure operators, and these can be compared against the ground-truth to provide the relative performance measurements. We have used Precision, recall and F-measure to determine the quality of the retrieved blur maps. Precision is a measure of relevance between the retrieved result and the observation. In our case it refers to the fraction of the detected blurred (sharp) pixels which are actually blurred (sharp) in the ground-truth.

$$Precision,P=\frac{T_p}{T_p+F_p},$$

(7)

where $T_p$ means that a blurred (sharp) pixel has been correctly detected as blurred (sharp) pixel and $F_p$ expresses that a pixel has been inaccurately detected as blurred (sharp) but it was sharp (blurred) actually.

Recall, also called as sensitivity in binary classification, is a measure of ability to retrieve the relevant results. In our case, it depicts the fraction of the actual blurred (sharp) pixels which are detected.

$$Recall,R=\frac{T_p}{T_p+F_n},$$

(8)

where $F_n$ means that a pixel has been inaccurately detected as sharp (blurred) but it was blurred (sharp) actually.

Being based on different working principles, different blur measure operators may show different capabilities with respect to precision and recall evaluations. A more suitable measure in this situation is F-measure which takes both precision and recall into consideration. The F-measure computes the accuracy of retrieved result by comparing it with the observation. In our case, segmented blur map is compared with the ground truth. The general form of F-measure is $F_{\beta }$ which is based on Van Rijsbergen’s effectiveness measure [55]. It is the weighted harmonic mean of precision and recall such that recall gets $\beta$ times more importance as compared to precision. If the performance objective is to be considered, this allows to place more emphasis on recall or precision [56] as might be required or intended.

$$F-\mathrm{measure},F_{\beta }=\frac{(1+{\beta }^2)\times P\times R}{{\beta }^2\times P+R}.$$

(9)

These quantitative measures provide an appropriate tool for analysis and evaluation.

4. Results and Discussion

In this section, we describe a number of experiments that have been conducted aiming to analyze the comparative performance of blur measure operators. First of all, we intend to demonstrate that blurred and sharp regions of an image produce clearly distinct responses when a suitable blur measure operator is applied to the image. The essence of a well crafted blur measure operator is that it can help clearly differentiate between blurred and sharp regions. For this objective, two neighborhood windows each of size $30\times 30$ have been selected as shown in Figure 4a for the third image of Figure 1. The red colored window patch (A) is on the defocused area while the green window patch (B) is placed on the focused region, and these areas have been selected to reveal their potential difference with respect to blur measure operators. Figure 4b displays the image responses when arbitrarily chosen five blur measure operators DER05, DER10, STA01, STA04 and MIS03 have been applied on these selected areas. It clearly indicates that pixels in the blurred area yield small values (black) of blur measures as compared to those of sharp areas. (c) and (d) give the numerical values of blur measurements for the vectorized pixels of the blur and sharp areas, and the marginal gap between them can be clearly identified. More specifically, the blur measures for the pixels in the blurred (defoused) area are less than 0.1 while they all are greater than 0.2 in the case of sharp (focused) area. Moreover, blur measures in the sharp area manifest a lot of fluctuations depicting the high frequency content there.

Regarding the blur detection and segmentation, all of the 32 blur measure operators described in Table 1 have been applied on the randomly selected images of the data set mentioned in Section 3.1. To test each of the blur measure operator under the same framework, the same blur segmentation methodology of Section 3.2 has been followed for all. Even then, the difference in image quality, degree of blur, content and nature of the image can favor some operators. Therefore, the results for images from defocus and motion blur are presented separately. Both qualitative and quantitative results are drawn and shown accordingly. The three scales chosen for local neighborhood patches were $11\times 11$, $15\times 15$ and $21\times 21$ pixels. The threshold $T_{seg}$ for the binary classification of blur map at the last step of blur segmentation has been taken as the mean of the final blur map, i.e., $T_{seg}=\frac{1}{W\times H}{\sum }_i{\overline{\alpha }}_i^{\left(3\right)}$, where W and H are the dimensions of the final blur map. Firstly, in the qualitative analysis, we present the blur maps obtained by applying the mentioned blur measure operators on the images. These blur maps when compared with the ground-truth give a fair idea about the competency of these operators. In the second case, for quantitative analysis, the evaluation measures mentioned in Section 3.3 have been used to evaluate the relative performance of blur measure operators. A number of experiments have been conducted for this and their results have been presented accordingly. The rankings of the performance of blur measure operators, as learnt through the experiments, are also mentioned for comparative analysis.

4.1. Qualitative Analysis

In this subsection, we conduct two types of experiments and exhibit their results through images. In both type of experiments, we randomly select one image from each of the sets of defocus and motion blurred images as their representative example and apply blur measure operators to retrieve their blur maps.

In the first experiment, we intend to assess the quality of the responses of blur measure operators at the intermediate steps of blur segmentation process. In addition, in order to evaluate the performances relatively, responses of one blur measure operator can be compared with that of another one at all the intermediate steps. For the defocus blur case, the fourth image of Figure 1 has been selected. The responses of blur measure operators for this image have been shown in Figure 5. It can be observed that the performances for DER08 and DER09 are the best while DER02, DER03, STA02, MIS05 and MIS06 are the worst at all the key steps. The highest scale i.e., $21\times 21$ has been shown for the initial blur and alpha maps.

For the motion blurred case, the fifth image of Figure 1 has been selected and Figure 6 presents its behavior at the steps of segmentation algorithm. It can be noticed that the STA06 produces better results at all the steps while MIS06’s performance is again the worst along with DER01, DER05, DER08 and STA02.

In addition to the blurriness, images in real scenarios may also get corrupted by the noise too. Therefore, to replicate such a common and vulnerable scenario, we conduct the second experiment to explore the effect of noise along with the blur. For this, two types of noise, salt and pepper and random impulse noise have been considered. This constitutes four scenarios: (1) defocus blur with salt and pepper noise, (2) defocus blur with random impulse noise, (3) motion blur with salt and pepper noise, and (4) motion blur with impulse noise. Further, to explore the robustness of blur measure operators against different levels of noise, three density levels $1\%$, $10\%$ and $20\%$ for both types of noise have been considered. One image from focus blurred and one image from motion blurred image sets has been selected randomly and has been garbled with the noise. The blur maps produced by the blur measure operators for the four scenarios have been shown in Figure 7, Figure 8, Figure 9 and Figure 10, respectively.

For the first case, Figure 7a displays the selected clean image, its ground truth and its three copies corrupted by salt and pepper noise of density $1\%$, $10\%$ and $20\%$ respectively. In (b), the blur maps obtained by applying blur measure operators have been shown. The best results are manifested by DER09, DER10; while the worst responses are from DER01 and STA02. Not surprisingly, majority of the operators adapted from the autofocus and SFF techniques demonstrated appreciable results, like DER04 to DER11 and STA06. Further, among different categories, transform-based operators show degraded results in case of noise.

In the second case, Figure 8 shows the responses of blur measure operators for the defocus blur and random impulse noise. Figure 9 and Figure 10 display the responses of blur measure operators for the motion blurred image by considering the effect of salt and pepper and impulse noise, respectively.

4.2. Quantitative Analysis

In the previous subsection, blur measure operators have been described according to their performance when applied on a randomly selected blurred image. However, it does not seem qualified to compare the performance of the blur measure operators on the basis of just a single image. As it is quite possible that the evaluation performances might have been influenced by the particular choice of the image, such that, if some other image might had been selected for analysis, performance ranking of operators might had been different. Therefore, now, we intend to investigate the possibility that whether the ranking of operators is dependent on the choice of the image or not. Put another way, this is to explore the validity of general notion that diverse nature of image content and the type and degree of blur may favor certain type of operators. We select considerable number of images from each of the defocus and motion blurred image sets separately and consider them to be the representative subset of their category. These subsets instead of a single image serve as a better instance to integrate and comprehend the diverse range of image content and level of blurriness. The blur segmentation methodology of Section 3.2 has been applied on these subsets and their results have been compared with their respective ground-truths. The quantitative measures used for the evaluation of performance have been described in Section 3.3. The value of $\beta$ in Equation (9) for the computation of f-measure has been set to be as 0.3. The evaluation measures for all the images in the subset, in both of the defocus and motion blur cases separately, have been combined together to give their average values. These average values are displayed in the bar graphs for the comparison of blur measure operators. The responses of the categories of blur measure operators are also shown in the sub-figures. These category evaluation measures are computed such that the responses of blur measure operators belonging to the same category have been combined together to give the average response for that category.

Figure 11 represents the average values of evaluation measures for blur measure operators for the set of 303 images affected by the defocus blur. Highest precision and F-measure are achieved by TRA04 and then TRA05. Those who got highest recall are DER09 and then DER10. Lowest values of precision, and F-measure are attained by DER05 and then DER01.

Figure 12 shows the average performance for the 204 defocused blurred images when salt and pepper noise with $1\%$ density is added to them. Highest precision and F-measure are achieved by STA04 and then STA01. Highest recall is for DER09 and then DER08. While the lowest precision, recall and F-measure are exhibited by DER02. While comparing the average responses of categories of blur measure operators in Figure 11 and Figure 12, it reveals that miscellaneous operators show robustness against noise while the derivative-based operators get degraded highly in case of noise.

While, Figure 13 demonstrates the average performance of blur measure operators for the set of 235 images affected by motion blur, Figure 14 shows the average values of evaluation measures for the 296 motion blurred images corrupted by salt and pepper noise of density $1\%$. It can be seen that STA01 achieves the best results in terms of precision and F-measure, while DER08 is at the top and DER09 is second for recall. The worst values for all three evaluation measures are exhibited by DER02. However, again on average, the best performance is shown by transform-based operators.

In Figure 12 and Figure 14 it can be easily identified that DER02 offers the least robustness against noise.

Hence, it is important to mention that the ranking for the performance of blur measure operators is highly influenced by the choice of the image under consideration. That is, if a different image is selected, the operator which appeared somewhere down in the previous performance ranking may now surface up, even to the top, in the new ranking.

The responses of blur measure operators have been displayed through Figure 11, Figure 12, Figure 13 and Figure 14 for defocus and motion blurred images in clean and noisy scenarios separately. Now, we combine all those results together to have comprehensive evaluation for the performance of all of the blur measure operators. We show average values for all three evaluation measures, precision, recall and f-measure, jointly for the clean and noisy cases in Figure 15 for defocus blur and in Figure 16 for motion blurred images. This is to deduce about the overall response of each of the blur measure operators in defocus and motion blur cases.

5. Conclusions

In this work, we analyze the relative performance of a number of blur measure operators for a single image through the blur segmentation process. A unified framework that treats all of the operators in equal terms has been implemented to compute blur maps and evaluate measures. Few of the studied operators belong to the state-of-the-art blur measure operators proposed in the literature for blur measurement while others have been adopted from the autofocus and shape from focus techniques.

The real world blurred images portray very complex characteristics of blurriness as these images are affected by a number of factors like lens distortion, sensor noise, poor illumination, saturation, nonlinear camera response function and compression in camera pipeline. It has been observed that, on average, STA01, STA03, STA04 and TRA02 to TRA05 exhibit comparatively better results among the blur measure operators considered in this study. It has been noticed that derivative-based operators, like DER08, DER09 and DER10, show highest values for recall. Further, the category of miscellaneous operators show highest robustness against noise, while derivative-based operators show the least robustness, specifically DER02 has been found to offer least robustness. The blur measure operators proposed in the literature and those as discussed in this study seem to be efficient against one or only a few of the factors, but not all; more precisely, they cannot handle all the imperfections in equal terms. However, it has been observed that some operators perform adequately well or worse against almost all imperfections that exist in the real-world images.