A Stronger Aadaptive Local Dimming Method with Details Preservation

Abstract

Local dimming technology focuses on improving the contrast ratio of the displayed images for a great visual perception. It consists of backlight extraction and pixel compensation. Considering a single existing backlight extraction algorithm can hardly adapt to images with diverse characteristics and rich details, we propose a stronger adaptive local dimming method with details preservation in this paper. This method, combining the advantages of some existing methods and introducing the combination of the subjective evaluation and the objective evaluation, obtains a stronger adaptation compared with others. Besides, to offset the luminance reduction caused in the backlight extraction process, we improve the bi-histogram equalization algorithm and propose a new pixel compensation method. To preserve image details, the Retinex theory is adopted to separate details. Experimental results demonstrate the effect of the proposed method on contrast ratio improvement and details preservation.

1. Introduction

High Dynamic Range (HDR) display is developed for HDR images and videos that convey vastly more color shades and nuances than previous standards. However, these devices are expensive for their complex technology, which limits their promotion. Liquid Crystal Display (LCD) is still the current technology for most devices such as computers and TVs.

LCD consists of a Liquid Crystal (LC) panel and a backlight panel with arrayed Back-Light Units (BLUs). The LC panel is light-modulated instead of self-luminous directly. Hence, an image is displayed by it with the backlight produced by BLUs. In early LCD technology, BLUs are always-on with the maximum luminance level, leading to high power consumption and low contrast ratio. In addition, the image quality is deteriorated due to the light leakage problem [1] in the dark state. Local dimming technology is developed to alleviate these weaknesses. As shown in Figure 1, the technology consists of the backlight extraction and the pixel compensation, which are, respectively, used in obtaining the luminance level for each BLU in backlight panel and the compensated image for the LC panel. In the process of backlight extraction, the luminance of each BLU is controlled dynamically according to the corresponding image content. The power consumption is reduced while the contrast ratio is improved effectively. Backlight smoothing is used to simulate the process of light diffusion, which is a solution to alleviate block artifacts [2,3]. Pixel compensation offsets the luminance reduction caused by the backlight dimming in the backlight extraction process.

The resolution of the displayed image in LC panel is larger than that of the backlight array in backlight panel. The diagram of backlight extraction is shown in Figure 2; the luminance of a BLU is determined by the corresponding block of the input image.

Many approaches for backlight extraction have been proposed. They determine the luminance level for BLUs from different characteristics of the image. The early method in [4] explores the maximum and the average luminance of the corresponding image block to determine the luminance level of each BLU. The following methods [5,6,7,8,9,10,11,12,13,14,15,16] extract luminance for BLUs from many other perspectives, such as image histogram, image details, and image quality. However, each method can hardly handle images with diverse and complicated contents. It makes sense to broaden the adaptive scope for a single backlight extraction method. To this end, we propose a method to extract backlight that adapts to images with diverse contents by combining the advantages of some existing methods [4,6,7,8,10,11,12], to which previous approaches have paid rare attention. Besides, we introduce subjective evaluation for a better visual perception. Specifically, our method takes three steps to obtain backlight luminance. First, a target backlight is selected from base backlights generated by existing methods. Second, we design a group of constraint conditions and adjust the target backlight under them to obtain several alternative backlights. Finally, the optimal backlight is determined by both the objective evaluation and the display quality of subjective evaluation.

Luminance overcompensation in pixel compensation process will cause image distortion, decreasing its contrast ratio and visual perception. Hence, we take both the backlight information and the luminance information of original image into account to address the overcompensation. Besides, we propose an Improved Bi-Histogram Equalization (IBHE) to further enhance the image. Specifically, Bi-Histogram Equalization (BHE) [17] applies a histogram equalization on two sub-images segmented from one image, obtaining a tradeoff between brightness enhancement and details preservation. Kim, Y.T. [17] adopted the mean luminance of the image as the breakpoint to make segmentation. However, the method lacks effectiveness in details preservation as it simply adopts the image mean luminance as the breakpoint for segmentation. To this end, we improve the method from the view of taking more image details in breakpoint selection. We make the IBHE a part of pixel compensation.

Combing the proposed backlight extraction with pixel compensation methods, a stronger adaptive local dimming method with details preservation is proposed. Our contributions can be summarized as:

• A backlight extraction method is proposed to acquire a stronger adaption in processing images with diverse contents by combining advantages of several existing methods.
• An IBHE method with Retinex theory is proposed to enhance image quality by preserving abundant details.
• A pixel compensation method is proposed to alleviate overcompensation by leveraging information of both the extracted backlight and the original image based on IBHE.

Experimental results demonstrate the effect of the proposed approach in improving contrast ratio, preserving image details, and enhancing image quality in the real display.

The rest of the paper is structured as follows. Section 2 details the related work of local dimming. Section 3 documents the specific process of the proposed method elaborately. Section 4 describes the experiments and results, and analysis of the results is made in this section. Finally, we conclude the paper in Section 5.

2. Related Work

In this section, existing local dimming approaches are described in detail. For clarity, we organize this section into backlight extraction and pixel compensation, as they are the two parts of local dimming.

2.1. Backlight Extraction

As an early and fundamental approach, the max method [4] employs the maximum luminance of an image block to determine the luminance level of the corresponding BLU. It also explores replacing the maximum luminance with the average luminance. However, the former is sensitive to noise and suffers from light leakage [1] problem. The latter reduces image luminance and suffers from losing details in bright areas. To achieve a trade-off between the above two methods, the methods proposed in [1,5,6] consider both the maximum luminance and the average luminance to improve display quality. In  [1], a decision rule is proposed to determine optimal backlight by comparing the light leakage and the clipping of image blocks. This method is effective for images with bright objects in a dark area. In  [5,6], the difference between the maximum and the average luminance value of a block is used to adjust backlight based on the average value. In [7], the global information of the image is used to extract backlight. It proposes a threshold method using a Cumulative Distribution Function (CDF) to ensure the distortion of the compensated image within a certain range. Besides, the methods proposed in [8,9] are effective to reduce power consumption. Based on Otsu [18], Zhang, T.; Wang, Y.F. [10] introduced a local dimming algorithm to separate foreground and background pixels for backlight extraction. In [11], the Peak Signal to Noise Ratio (PSNR) = 30 is considered as the lowest standard to guarantee the image quality. In [12], a Gaussian distribution model is proposed to reduce power consumption and improve image quality. In Swarm Intelligence (SI), the authors of [13,14] transformed the local backlight dimming to an optimization problem to preserve the image quality with low power consumption. A guided firework algorithm proposed in [14] achieves higher performance than the one in [13]. Although there are other local dimming algorithms [15,16], most of them favor only specific characteristic of an image. Therefore, we propose a backlight extraction method to adapt to images with diverse characteristics, acquiring preferable display quality.

2.2. Pixel Compensation

A backlight extraction method is commonly followed with a corresponding pixel compensation method to offset the luminance reduction. In this section, we document the representative method [5] and the closest related method [10] to ours for readability. In [5], the compensated luminance is obtained using the nonlinear relationship between the maximum backlight and the extracted backlight. However, image distortion caused by overcompensation in this method decreases the image quality. In [10], the logarithm function is used to compensate for luminance based on the input image and the smoothed backlight image. It is effective to prevent overcompensation but less effective for bright images.

Our pixel compensation method aims to alleviate the overcompensation problem by adjusting the luminance of the pixel in the input image according to the backlight value. Besides, by the proposed IBHE, this method is effective for improving the quality of display images.

3. Method

In this section, we document our local dimming method elaborately. We first introduce the holistic structure of the method. Then, the proposed backlight extraction method and the compensation method are described, respectively.

The diagram of the proposed method is shown in Figure 3. The whole architecture consists of two modules: an Adjustable Backlight Extraction (ABE) module and a pixel compensation module. Furthermore, the ABE module consists of base backlights extraction and optimal backlight selection.

To avoid color distortion, most of the existing local dimming algorithms are performed on the luminance rather than the chroma component. Following this, we separate the luminance information by converting the color space from $RGB$ into $YCbCr$ [19] before all of the succeeding operations. The conversion formula is

$$\left\{\begin{array}{c}{\left[\mathbf{Y}\mathbf{Cb}\mathbf{Cr}\right]}^T={\mathbf{M}}_{\mathbf{sRGB}}\times {\left[\mathbf{R}\mathbf{G}\mathbf{B}\right]}^T+{\left[16128128\right]}^T\hfill \\ {\mathbf{M}}_{\mathbf{sRGB}}=\left[\begin{array}{ccc}0.257& 0.564& 0.098\\ -0.148& -0.291& 0.439\\ 0.439& -0.368& -0.071\end{array}\right]\hfill \end{array}\right.$$

(1)

According to the Retinex theory [20], which is widely used in image processing [21], an image is composed of reflectance and illuminance. The former presents the detail information and the latter determines the dynamic range. This is represented as:

$$\mathbf{S}(x,y)={\mathbf{R}}_{\mathbf{c}}(x,y)\times \mathbf{I}(x,y)$$

(2)

where $(x,y)$ is the coordinate of the pixel in the image, $\mathbf{S}$ is the image perceived by human eyes, ${\mathbf{R}}_{\mathbf{c}}$ is the reflectance, and $\mathbf{I}$ is the illuminance. In our method, $\mathbf{Y}$ component in $YCbCr$ color space is considered as $\mathbf{S}$. $\mathbf{I}$ is obtained by:

$$\mathbf{I}(x,y)=\mathbf{F}(x,y)\otimes \mathbf{Y}(x,y)$$

(3)

where $\mathbf{F}$ is the Weighted Least Squares (WLS) filter [22], which is known as an edge-preserving filter; ⊗ is the convolution operation; and $\mathbf{Y}$ is the $\mathbf{Y}$ component in $YCbCr$ color space.

Image edge is the most concentrated part of image information such as the change of gray level and the mutation of texture structure, which contains rich details. Therefore, the edge information must be decomposed and kept to improve image quality in image processing. An alternative filter is bilateral filtering, which has been used in many previous works as a base-detail decomposition technique. However, WLS filter is chosen in this paper because of its better performance, especially for increased blur level compared with bilateral filtering. WLS filter is well suited for progressive coarsening of images and for multi-scale detail extraction. For an input image g, an image u is expected to be as close to g as possible and be smoother except for some places where the gradient of the edge of g changes greatly. Formally, the solution to minimize the objective function in Equation (4) is the result of filtering u.

$$\sum _p\left({\left(u_p-g_p\right)}^2+\lambda \left(a_{x,p\left(g\right)}{\left(\frac{\partial u}{\partial x}\right)}^2+a_{y,p\left(g\right)}{\left(\frac{\partial u}{\partial y}\right)}^2\right)\right)$$

(4)

where the subscript p represents the coordinate of the pixel. The first term ${\left(u_p-g_p\right)}^2$ is used to measure the similarity between g and u. The second term is a regular term, and $\lambda$ is a weight coefficient of the regular term. The larger $\lambda$ is, the smoother the image will be. The image g is smoothed by minimizing the partial derivative of u, and the weights of smoothing terms are $a_{x,p\left(g\right)}$ and $a_{y,p\left(g\right)}$, respectively. The definitions of $a_{x,p\left(g\right)}$ and $a_{y,p\left(g\right)}$ are shown in Equation (5).

$$\left\{\begin{array}{c}\hfill a_{x,p\left(g\right)}={\left({\left|\frac{\partial l}{\partial x}\left(p\right)\right|}^{\alpha }+ϵ\right)}^{-1}\\ \hfill a_{y,p\left(g\right)}={\left({\left|\frac{\partial l}{\partial y}\left(p\right)\right|}^{\alpha }+ϵ\right)}^{-1}\end{array}\right.$$

(5)

where l is the logarithmic transformation of g, the exponential parameter $\alpha$ is used to determine the gradient sensitivity, and $ϵ$ is the offset for avoiding invalid division when $\frac{\partial l}{\partial x}\left(p\right)$ or $\frac{\partial l}{\partial y}\left(p\right)$ is zero. From the above equations, $a_{x,p\left(g\right)}$ and $a_{y,p\left(g\right)}$ decrease when the gradient of l increases, by which the edge information is kept and unnecessary details is smoothed.

Backlight extraction and pixel compensation are processed on the illuminance $\mathbf{I}$. Then, the logarithm of ${\mathbf{R}}_{\mathbf{c}}$ is defined in Equation (6).

$$\mathbf{r}(x,y)=log\left({\mathbf{R}}_{\mathbf{c}}\left(x,y\right)\right)=log\left(\mathbf{Y}\left(x,y\right)\right)-log\left(\mathbf{I}\left(x,y\right)\right)$$

(6)

where $\mathbf{r}$ is the logarithmic result of ${\mathbf{R}}_{\mathbf{c}}$, and it is used in IBHE in Section 3.2.

3.1. Adjustable Backlight Extraction Module

To respect the image content and avoid drawbacks of a single algorithm, we propose a three-step backlight extraction method to determine an optimal backlight, making displayed images perceived vividly. We document the first step in Section 3.1.1 and the second and the third steps in Section 3.1.2.

3.1.1. Base Backlights Extraction

The first step is to extract base backlights. As mentioned above, each single existing backlight extraction method is not enough to adapt to images with diverse characteristics and contents. However, their respective strengths are complementary and compatible. Each of them can be a base backlight from which we absorb advantages. Assuming that N is the number of base backlights, the base backlights extraction in Figure 3 is defined as follows.

$${\mathbf{BL}}^t=f^t\left(I\right)t=1,2,\cdots ,N$$

(7)

where $f^t$ is the $t_{th}$ base backlight algorithm and ${\mathbf{BL}}^t$ is the $t_{th}$ base backlight.

3.1.2. Optimal Backlight Selection

Optimal backlight constraint conditions are constructed by all of the base backlights extracted above. One of the base backlights is selected as the target backlight for its ability in reducing power consumption and improving the contrast ratio.

The second step of our method is to adjust the selected target backlight. Specifically, we adjust it based on the segmentation method in [23] and the obtained constraint conditions.

Based on our self-developed LCD-LED dual modulation display [10], the optimal backlight is selected from adjusted backlights by objective evaluation as well as subjective evaluation in the third step.

• Backlight constraint conditions

We change specific values of the target backlight to obtain several adjusted backlights as alternations of the optimal backlight. The change needs to be within the effective range of backlight to prevent the deterioration of image quality. For an image block, we go through all base backlights for its corresponding maximum and minimum values. The maximum and minimum matrices are denoted as ${\mathbf{P}}_{\mathbf{max}}$ and ${\mathbf{P}}_{\mathbf{min}}$. This process is defined as:

$$\left\{\begin{array}{c}\hfill {\mathbf{P}}_{\mathbf{max}}=max\left({\mathbf{BL}}^t(m,n)\right)\\ \hfill {\mathbf{P}}_{\mathbf{min}}=min\left({\mathbf{BL}}^t(m,n)\right)\end{array}\right.$$

(8)

where $(m,n)$ is the coordinate of each backlight value in the backlight image.

Considering that limited backlight extraction algorithms are used to construct backlight constraint conditions, ${\mathbf{P}}_{\mathbf{max}}$ is increased by 10% with an upper boundary 255 and ${\mathbf{P}}_{\mathbf{min}}$ is decreased by 10% with a lower boundary 0. The adjusted ${\mathbf{P}}_{\mathbf{max}}$ and ${\mathbf{P}}_{\mathbf{min}}$ are represented as ${\mathbf{PA}}_{\mathbf{max}}$ and ${\mathbf{PA}}_{\mathbf{min}}$, and they form the constraint conditions for obtaining the optimal backlight.

• Backlight adjustment and optimal backlight selection

Just Noticeable Difference (JND) [24] reflects the sensitivity of human vision. As shown in Figure 4, under different background luminance, JND is different.

In image quality evaluation, if the luminance of details is too close to the neighboring pixels, that is, the difference is less than JND, then the details of this image are not well-displayed. In real display, the background luminance may be changed due to light diffusion between different backlights, leading to the change of JND. Therefore, the image quality in real display may be degraded. To this end, we adjust the target backlight to change the background luminance and select the backlight that presents the details effectively based on display quality as the optimal backlight.

Assume that target backlight is the $k_{th}$ base backlight denoted as ${\mathbf{BL}}_{\mathbf{0}}$.

$${\mathbf{BL}}_{\mathbf{0}}=f^k\left(I\right)k\in \left\{1,2,\cdots ,N\right\}$$

(9)

where $f^k$ is the $k_{th}$ base backlight extraction method. Improving the dynamic range is important in local dimming. Hence, for ${\mathbf{BL}}_{\mathbf{0}}$, we strengthen the luminance in bright area and weaken it in dark area to improve its dynamic range. Specifically, the bright and the dark area are selected by mean and variance of the backlight image. The process is expressed as Equations (10) and (11).

$$\left\{\begin{array}{c}M=\left(\sum _{m=1}^W\sum _{n=1}^H{\mathbf{BL}}_{\mathbf{0}}\left(m,n\right)\right)÷\left(W\times H\right)\hfill \\ V=\left(\sum _{m=1}^W\sum _{n=1}^H{\left({\mathbf{BL}}_{\mathbf{0}}\left(m,n\right)-M\right)}^2\right)÷\left(W\times H\right)\hfill \end{array}\right.$$

(10)

$$\left\{\begin{array}{c}\hfill P_1=M-V\\ \hfill P_2=M+V\end{array}\right.$$

(11)

where W, H, M, and V are the width, height, mean, and variance of ${\mathbf{BL}}_{\mathbf{0}}$, respectively. $P_1$ and $P_2$ are the breakpoints to partition areas of different luminance. The pixel with luminance less than $P_1$ is considered as dark area and the pixel with luminance larger than $P_2$ is considered as bright area,

Since the backlight value ranges in 0–255, we use the exponent of 2 as the adjustment step to adjust the target backlight ${\mathbf{BL}}_{\mathbf{0}}$. The process is expressed as Equation (12).

$$\left\{\begin{array}{c}{\mathbf{BL}}_i\left(m,n\right)=\left\{\begin{array}{c}\hfill {\mathbf{BL}}_{\mathbf{0}}\left(m,n\right)-2^i{\mathbf{BL}}_{\mathbf{0}}\left(m,n\right)<P_1\\ \hfill {\mathbf{BL}}_{\mathbf{0}}\left(m,n\right)+2^i{\mathbf{BL}}_{\mathbf{0}}\left(m,n\right)>P_2\end{array}\right.\hfill \\ {\mathbf{BL}}_i\left(m,n\right)=\left\{\begin{array}{c}\hfill min\left({\mathbf{PA}}_{\mathbf{max}}\left(m,n\right),{\mathbf{BL}}_i\left(m,n\right)\right){\mathbf{BL}}_i\left(m,n\right)>{\mathbf{PA}}_{\mathbf{max}}\left(m,n\right)\\ \hfill max\left({\mathbf{PA}}_{\mathbf{min}}\left(m,n\right),{\mathbf{BL}}_i\left(m,n\right)\right){\mathbf{BL}}_i\left(m,n\right)<{\mathbf{PA}}_{\mathbf{min}}\left(m,n\right)\end{array}\right.\hfill \end{array}\right.$$

(12)

where $i=1,2,\cdots ,8$, ${\mathbf{BL}}_i$ means the $i_{th}$ adjusted backlight based on ${\mathbf{BL}}_{\mathbf{0}}$. ${\mathbf{PA}}_{\mathbf{max}}$ and ${\mathbf{PA}}_{\mathbf{min}}$ are used to prevent the adjusted backlight from making poor display quality.

Contrast Ratio (CR) and Dynamic Range (DR) are two objective indicators of display quality. Both reflect the change of brightness ranging from dark to bright, and the higher are CR and DR, the wider is the brightness range. To select the optimal backlight, the display luminance is first measured under all adjusted backlights. Then, CR and DR of each measured luminance are calculated. The adjusted backlights with the top three performances based on the results of CR and DR are selected. Finally, the optimal backlight is selected from the three adjusted backlights by display quality subjectively. The process is shown in Figure 5.

The subjective evaluation is set as follows. The three selected adjusted backlights are demonstrated on the display prototype. Observers are asked to vote for the optimal backlight based on visual perception over details, contrast, and brightness. The backlight with the largest number of votes is determined to be the optimal backlight. Given that subjective feeling is susceptible to factors such as gender, age, occupation, and surroundings, the selection was done by 16 observers who are non-experts in image and video processing field. Their ages range from 22 to 30, with eight males and eight females. All of them have normal visual ability, that is, none of them have eye problems such as color blindness, color weakness, shortsighted, etc.

3.2. Pixel Compensation Module

In this work, we compensate luminance according to the optimal backlight described as ${\mathbf{BL}}_{\mathbf{op}}$ and the luminance of the input image. Before the compensation, we use Improved Blur Mask Approach (IBMA) [10] to smooth optimal backlight to removes the block artifacts. Specifically, Zhang, T.; Wang, Y.F. [10] divided the points of ${\mathbf{BL}}_{\mathbf{op}}$ into three categories. The first category includes the corner points, the second category includes the peripheral points except for the cornet points, and the third category includes the internal points of ${\mathbf{BL}}_{\mathbf{op}}$. Different Low Pass Filter (LPF) templates are used to smooth points of ${\mathbf{BL}}_{\mathbf{op}}$ in different categories. IBMA uses the smoothing process to simulate the light diffusion, and ${\mathbf{BL}}_{\mathbf{op}}$ is resized after each smoothing operation. By several smoothing operations, the smoothed backlight has the same size as the input image. The process is expressed in Equation (13).

$${\mathbf{BL}}_{\mathbf{sm}}=IBMA\left({\mathbf{BL}}_{\mathbf{op}}\right)$$

(13)

where ${\mathbf{BL}}_{\mathbf{sm}}$ represents the backlight after IBMA and ${\mathbf{BL}}_{\mathbf{op}}$ is the optimal backlight. The comparison results of using or not using IBMA are shown in Figure 6.

Compared with Figure 6a, artifacts are removed obviously in Figure 6b by applying IBMA.

We use a compensation coefficient k to control the compensation degree, which is determined by the smoothed backlight and the luminance of the input image. The process is formulated as Equations (14) and (15).

$$\mathbf{k}\left(x,y\right)={\left({\mathbf{BL}}_{\mathbf{sm}}\left(x,y\right)÷\mathbf{I}\left(x,y\right)\right)}^{\gamma }$$

(14)

$${\mathbf{I}}_{\mathbf{p}}\left(x,y\right)=\mathbf{k}\left(x,y\right)\times \mathbf{I}\left(x,y\right)+\left(1-\mathbf{k}\left(x,y\right)\right)\times {\mathbf{BL}}_{\mathbf{sm}}\left(x,y\right)$$

(15)

where $\gamma$ = 0.125 is selected from multiple experimental results to prevent overcompensation problem and enhance the overall luminance of the image effectively. ${\mathbf{I}}_{\mathbf{P}}$ is the compensated luminance.

Next, IBHE is used to further enhance the compensated luminance. In BHE, an image is decomposed into two sub-images based on its mean, and then the sub-images are equalized independently to improve CR while maintaining the luminance of the image. Different from this, we rely on the Otsu method and the histogram of r in Equation (6) to obtain two sub-images. Algorithm 1 is devised to acquire the breakpoint for the image segmentation.

Algorithm 1 Proposed algorithm for breakpoint acquisition.

Input:r, $I_P$;

Output: the breakpoint T;

1: $\left[z_1,z_2\right]$ = size $\left(r\right)$; 2: $num$ = 0; 3: $I_m$ = $I_P$; 4: for$i=1$ to $z_1$ do 5:     for $j=1$ to $z_2$ do 6:         if $r\left(i,j\right)==0$ then 7:            $num+=1$; 8:         else 9:            $I_m\left(i,j\right)=0$; 10:         end if 11:     end for 12: end for 13: $T1=\left(sum\left(sum\left(I_m\right)\right)\right)÷num$; 14: $T2=Otsu\left(\left(I_m\right)\right)$;       the breakpoint obtained by Otsu method 15: $T=floor\left(\left(T1+T2\right)÷2\right)$;       the average value of $T_1$ and $T_2$ 16: returnT;

By the breakpoint T, the CDF curves of the two sub-images are obtained to perform BHE [17]. The process is expressed as Equation (16).

$${\mathbf{I}}_{\mathbf{out}}\left(k\right)=\left\{\begin{array}{c}\left(T-I_{pmin}\right)\times CD{F}_1\left(k\right)+I_{pmin}0<k<T\hfill \\ \left(I_{pmax}-T\right)\times CD{F}_2\left(k\right)+TT+1<k<255\hfill \end{array}\right.$$

(16)

where $I_{pmin}$ and $I_{pmax}$ are the minimum and the maximum of compensated luminance ${\mathbf{I}}_{\mathbf{p}}$, respectively. $CD{F}_1$ and $CD{F}_2$ are respective CDF curves of the two sub-images.

Finally, ${\mathbf{I}}_{\mathbf{out}}$ and $\mathbf{r}$ in Equation (6) are combined to reconstruct final luminance image by Equation (17), and color transformation from $YCbCr$ to $RGB$ [25] is employed to generate the final image.

$${\mathbf{Y}}_{\mathbf{out}}\left(x,y\right)={\mathbf{I}}_{\mathbf{out}}\left(x,y\right)\times e^{\mathbf{r}\left(x,y\right)}$$

(17)

4. Experiment

In this section, we describe the settings and the results of our experiments. In backlight extraction, the base backlight extraction methods in our experiment consisted of the max method [4], the average method [4], LUT method [6], CDF method [7], IMF method [8], the method based on Otsu [10], PSNR method [11], and Gaussian method [12]. The method based on Otsu in [10] was considered as the target algorithm, extracted by which the backlight is taken as the target backlight. The sizes of the input image and the backlight are 1920 × 1080 and 66 × 36, respectively.

4.1. Hardware

A self-designed LED-LCD prototype display [10] was adopted to verify display quality with the optimal backlight and compensated image that is the output in Figure 3. The display principle of the prototype is shown in Figure 7a. The measure environment in our experiment was a dark room to prevent interference from light. The luminance meter used in our experiment is CX-2B color brightness meter, for which the luminance ranges in 0.001–200 kcd/m².

4.2. Experiment of Improved Bi-Histogram Equalization

To illustrate the effectiveness of the proposed IBHE method in segmenting image, the breakpoint by Kim, Y.T. [17] was used to make comparison. The results are shown in Figure 8.

As shown in the red square of Figure 8b,c, the segmentation effect is basically the same, while, in the yellow as well as green squares, the segmentation is more accurate by the proposed IBHE method than by the method in [17] according to the test image.

4.3. Experiment of Adjustable Backlight Extraction

4.3.1. Subjective Experiment

The comparison of image display quality under the target backlight and the optimal backlight is shown in Figure 9.

The images were taken by an optical recorder. Obviously, the image recorders can hardly reproduce the visual perception of eyes. However, we can still distinguish that the visual perception of the optimal backlight is more vivid than that of the target backlight. For clarity, the detail with obvious difference is marked. In the above two images in Figure 9, we observe that the dark areas are enhanced under optimal backlight to improve CR of the displayed image. In the other two images, the marked areas illustrate a higher color saturation to improve display quality under optimal backlight. These demonstrate that the proposed backlight extraction method has stronger adaption in an image with different brightness and rich details in the real display.

4.3.2. Objective Experiment

Tong, H. [26] defined a group of conditions to separate low/high luminance and low/high CR images with CDF curve. We selected one image from each category to evaluate the proposed method according to the conditions. The images chosen for the experiment and their corresponding CDF curves are illustrated in Figure 10.

Assuming that h denotes the histogram of the measured luminance using luminance meter, $h\left(x\right)$ denotes the number of pixels whose luminance is x. Inspired by the method of calculating CR in [10], we calculated the average values of luminance that are greater than $P_{90}$ and lower than $P_{10}$, respectively, to calculate CR to reduce the influence of the measurement error. $P_{10}$ and $P_{90}$ are the luminance of which the cumulative numbers account for 10% and 90% of the total pixels, respectively. $M_{max}$ and $M_{min}$ are the maximum and minimum of the measured luminance. CR and DR are calculated as follows.

$$\left\{\begin{array}{c}Av{g}_{10}=\sum _{x<P_{10}}\left(h\left(x\right)\times x\right)÷\sum _{x<P_{10}}h\left(x\right)\hfill \\ Av{g}_{90}=\sum _{x>P_{90}}\left(h\left(x\right)\times x\right)÷\sum _{x>P_{90}}h\left(x\right)\hfill \\ CR=Av{g}_{90}÷Av{g}_{10}\hfill \\ DR=M_{max}÷M_{min}\hfill \end{array}\right.$$

(18)

The objective evaluations are shown in

Table 1. Objective evaluation comparisons of CR and DR. ${\mathbf{BL}}_{\mathbf{0}}$, the target backlight; ${\mathbf{BL}}_i$, the optimal backlight; (a)–(d), images in Figure 10. The better performance is marked in bold.

Image	Backlight	CR	DR
(a)	$B{L}_0$	11,209.24	1,630,700
	$B{L}_3$	11,284.18	1,631,900
(b)	$B{L}_0$	8.97	2,059,400
	$B{L}_4$	9.14	1,974,200
(c)	$B{L}_0$	3228.41	1,056,333
	$B{L}_4$	3272.41	1,156,091
(d)	$B{L}_0$	1913.71	1,659,500
	$B{L}_3$	5464.29	1,656,385

. We can observe that all CR values of optimal backlights are better than that of the target backlights. DR values in Images (b) and (d) under optimal backlights are slightly lower than those under target backlights. The reason lies in that the luminance of the bright area is almost 255, which cannot be increased. In contrast, the luminance of the dark area is still decreased, resulting in a reduction in overall luminance. For most cases, the comparisons confirm the effectiveness of the proposed method.

4.4. Experiment of Simulated Images

4.4.1. Subjective Experiment

The simulated comparisons of the proposed local dimming method with LUT method, CDF method, and the method based on Otsu [10] are shown in Figure 11.

For Figure 11a, the dark areas in red rectangles of CDF and LUT algorithms are brighter, leading to a lower CR than the other two methods. In contrast, the method in [10] missed details caused by reducing luminance. The proposed method is a balance between improving CR and preserving details.

For Figure 11b, the clouds in the red circles of CDF and LUT methods are brighter than Figure 10b, which should be darker. For the clouds in the red rectangle by the method in [10], the image distortion is caused by overcompensation. For the image by the proposed method, the image contents in both the red circle and the red rectangle are well-compensated to improve CR and preserve details.

For Figure 11c, CDF and LUT methods simply improve the overall image luminance but without improvement of CR and image quality. The method in [10] improved the luminance of bright areas and decreased that of dark areas, which shows a preferable image. The image of the proposed method shows a higher CR with a higher saturation. However, the mountain part is slightly unsatisfactory compared with the image obtained by the method in [10]. This may be because the low luminance is mapped to smaller by histogram equalization.

For Figure 11d, the images of CDF and LUT methods are bright to show rich details. In contrast, the image obtained by the method in [10] is distorted. Note that the red rectangle, seen as a high spatial-frequency part, retains more details by the proposed method compared with the image obtained by the method in [10].

4.4.2. Objective Experiment

In our experiments, in addition to for CR [10], Peak Signal-to-Noise Ratio (PSNR) [27], Structural Similarity Index (SSIM) [14], and Color Difference (CD) [28] were further applied to evaluate the simulated image comprehensively.

CR, used to evaluate the dynamic range of luminance, is an important metrics in image processing. Generally, an image with a high CR presents vivid and rich colors. Note that the CR used to evaluate the simulated images is calculated differently compared with the CR in Section 4.3.2. To distinguish them, the CR calculated by the simulated image is defined as $C{R}_{SI}$ and obtained by Equation (19).

$$C{R}_{SI}=P_{90}/P_{10}$$

(19)

where $P_{10}$ and $P_{90}$ are the luminance of which the cumulative numbers account for 10% and 90% of the total number of pixels in simulated image, respectively.

PSNR was employed to evaluate the distortion between signal and noise. A higher PSNR indicates a lower distortion. The definition of PSNR is described in Equation (20).

$$\left\{\begin{array}{c}PSNR\left(C_1,C_2\right)=10\times log\left(\frac{{255}^2}{MSE\left(C_1,C_2\right)}\right)\hfill \\ MSE\left(C_1,C_2\right)=\frac{1}{w\times h}\sum _{i=1}^w\sum _{j=1}^h{\left(C_1\left(i,j\right)-C_2\left(i,j\right)\right)}^2\hfill \end{array}\right.$$

(20)

where $C_1$ and $C_2$ are the original image and the simulated image, respectively, while w and h mean the width and the height of the simulated image.

SSIM is widely used in realizing structural similarity theory. SSIM ranges from 0 to 1 and a better image quality leads to a higher SSIM. The definition of SSIM is described in Equation (21)

$$SSIM\left(C_1,C_2\right)=\frac{\left(2{\mu }_{C_1}{\mu }_{C_2}+{ϵ}_1\right)\left(2{\sigma }_{C_1{C}_2}+{ϵ}_2\right)}{\left({\mu }_{C_1}^2+{\mu }_{C_2}^2+{ϵ}_1\right)\left({\sigma }_{C_1}^2+{\sigma }_{C_2}^2+{ϵ}_2\right)}$$

(21)

where ${\mu }_{C_1}$ and ${\mu }_{C_2}$ are the mean values of $C_1$ and $C_2$ respectively; ${\sigma }_{C_1}$ and ${\sigma }_{C_2}$ are the variance of $C_1$ and $C_2$, respectively; ${\sigma }_{C_1{C}_2}$ is the covariance of $C_1$ and $C_2$; and ${ϵ}_1$ and ${ϵ}_2$ are two constants to avoid invalid division.

Besides, from the perspective of color information, we adopted CD to evaluate the color distortion by applying weighted Euclidean distance in RGB color space. The process to obtain the CD is expressed in Equation (22).

$$\left\{\begin{array}{c}\mathsf{\Delta }C=\sqrt{\left(2+\frac{\overline{r}}{256}\right)\times \mathsf{\Delta }{R}^2+4\times \mathsf{\Delta }{G}^2+\left(2+\frac{255-\overline{r}}{256}\right)\times \mathsf{\Delta }{B}^2}\hfill \\ CD=\frac{\mathsf{\Delta }C}{w\times h}\hfill \end{array}\right.$$

(22)

where $\overline{r}=\left(C_{1,R}+C_{2,R}\right)/2$, $\mathsf{\Delta }R=C_{1,R}-C_{2,R}$, $\mathsf{\Delta }G=C_{1,G}-C_{2,G}$, and $\mathsf{\Delta }B=C_{1,B}-C_{2,B}$; $C_{1,R}$, $C_{1,G}$, and $C_{1,B}$ represent the normalized components of original image, respectively; and $C_{2,R}$, $C_{2,G}$, and $C_{2,B}$ represent the normalized components of simulated image; respectively. The comparisons of the above four metrics are shown in

Table 2. Comparisons of algorithmic processing. The best performance is marked in bold.

Image	Evaluation Metrics	CDF Method	LUT Method	[10]	the Proposed Method
(a)	$C{R}_{SI}$	4.88	4.93	7.00	7.50
	$PSNR$	18.40	19.43	24.78	33.19
	$SSIM$	0.87	0.89	0.94	0.97
	$CD$	0.32	0.28	0.15	0.06
(b)	$C{R}_{SI}$	4.32	4.28	7.68	8.42
	$PSNR$	24.10	25.49	24.24	27.00
	$SSIM$	0.98	0.99	0.97	0.98
	$CD$	0.13	0.12	0.16	0.12
(c)	$C{R}_{SI}$	5.88	5.76	6.08	7.71
	$PSNR$	20.32	21.49	23.44	24.39
	$SSIM$	0.91	0.92	0.93	0.90
	$CD$	0.26	0.23	0.19	0.19
(d)	$C{R}_{SI}$	5.69	6.00	6.50	8.00
	$PSNR$	19.81	20.49	25.85	24.80
	$SSIM$	0.83	0.84	0.90	0.93
	$CD$	0.23	0.21	0.13	0.14

.

In Table 2, $C{R}_{SI}$ obtained by the proposed method is higher than that of the three other top algorithms by 7.0%, 10.0%, 26.8%, and 23.1%, respectively. PSNR is improved by 33.9%, 11.4%, 4.1%, and −4.1%, respectively. PSNR of high contrast ratio image by the proposed method follows the highest one by the method in [10] closely. For SSIM, the performance of the proposed method is slightly inferior for low contrast ratio image. However, it is still competitive to other algorithms, especially for low luminance image. For CD, images processed by the proposed method reduce the distortion of chroma information effectively, especially for the low luminance image and the high contrast ratio image. In other words, the objective evaluation values are consistent with the subjective quality of the simulated images.

5. Conclusions

In this paper, a stronger adaptive local dimming method with details preservation is proposed to alleviate the disadvantage of a single algorithm. A three-step backlight extraction method is applied to determine the optimal backlight to improve display quality. In the pixel compensation, we compensate the luminance of the input image according to the smoothed backlight information. In addition, IBHE is proposed to enhance the luminance of an image and realize details preservation. Both the objective and subjective evaluation results demonstrate the effectiveness of the proposed local dimming method in keeping chroma information, and improving CR as well as PSNR, SSIM.