1. Introduction
Nature scenes have a very wide dynamic range. However, because the existing display equipment has a narrow dynamic range, it cannot express all the information, such as the brightness and color, of a natural scene. To reduce the loss of information, it is necessary to increase the dynamic range of the camera sensor and display equipment or to compress the information in a wide dynamic range without loss. However, since increasing the dynamic range of a physical sensor is expensive, HDR imaging has been developed to reduce information loss by synthesizing several narrow dynamic images.
An HDR image can be created using multiple low dynamic range (LDR) images with different exposure values [
1]. The exposure value of the camera is expressed as the amount of light incident on the sensor. An image with a low exposure value is dark, but details in a bright area can be obtained. An image with a high exposure value is brightly saturated, but the detail in the dark area increases. HDR imaging is a method of acquiring all the details from dark areas to bright areas by synthesizing multiple images with different exposure values. However, since the HDR image has a wider dynamic range than the existing LDR display device, all information included in the HDR image cannot be displayed on the LDR display device. To solve this problem, tone mapping operators (TMOs) are used. TMOs are a method of mapping the dynamic range of the HDR image to a range that can be expressed on a display device. The goal of TMOs is to reproduce accurate HDR scenes by reducing information loss during mapping.
Existing TMOs can be divided into global and local methods [
2]. Global TMO is a method of mapping all pixels using the same function. Global TMO has the advantages of simple structure and fast processing speed but has the disadvantage of the low contrast of the rendered image. Local TMO is a mapping method with reference to information of the adjacent pixels during mapping. The mapping value of the local TMO is changed according to surrounding information even if the value of the target pixel is the same. Therefore, the rendered images of local TMO have high contrast, but the processing speed is slower than global TMO, and halo artifacts appear.
Reinhard et al. [
3] proposed global TMO based on a zone system for LDR display devices or photo printing for HDR images, and local TMO using dodging-and-burning to reduce the detail loss that occurs in very high dynamic range images. Mantiuk et al. [
4] proposed a method to reduce the visible contrast distortion that occurs depending on the range of the output device. The method consists of a contrast distortion weight based on the human visual system and constraints to reduce distortion according to the display device. Banterle et al. [
5] proposed a TMO based on differential zone mapping. The method consists of image decomposition using zone mapping, optimal TMO application for each zone, and Laplacian pyramid fusion. Zone mapping applies the optimal TMO selected through physical experiments to each zone. Laplacian pyramid fusion was used to combine tone-mapped zones into one image without discontinuity and seams in the edge region. Kwon et al. [
6] proposed a luminance adaptation transform (LAT) based on human visual function and retinex. The LAT is composed of a local TMO and chrominance scaling part to improve the local contrast and prevent saturation from occurring in the highlighted area. The local TMO part uses the luminance threshold and visual gamma to map the image to the optimal tone according to the local adaptation luminance level. The chrominance scaling part is used to compensate for the chrominance channel after tone mapping. Then, a multi-scale method was applied to balance local and global contrast.
Some studies apply the image appearance model as a local TMO. The color appearance model is a model that can predict color appearance attributes such as lightness, brightness, colorfulness, chroma, and hue in given stimulus and viewing conditions [
7,
8]. The image appearance model is an extended model of the color appearance model and was developed to predict the perceptual attributes of spatially and temporally complex stimuli such as images. Fairchild and Johnson developed an image color appearance model (iCAM) to predict the appearance of an image with spatially complex color stimuli by extending the traditional color appearance model and rendering the prediction result as an image [
9]. Kaung et al. [
10] proposed an iCAM06 to deal with HDR images by extending the iCAM. The iCAM06 adopts two-scale decomposition to reduce halo artifacts. The iCAM06 uses cone and rod photoreceptor response functions to process a wide dynamic range from scotopic to photopic and applies various visual modeling functions to express visual attributes according to the surrounding changes. Chae et al. [
11] proposed a compensating method to reduce the white point shift that appears during tone compression in the iCAM06. Kwon et al. [
12] proposed global chromatic adaptation to improve the saturation degradation caused by incorrect lighting prediction during HDR rendering. The global chromatic adaptation consists of illuminant estimation and an adaptive degree function that can adjust the adaptation level. The illumination estimation is calculated as the average value of pixels distributed near the black body locus in the
xy color space. The adaptive degree function was modeled to control the degree of chromatic adaptation according to the intensity of illumination and color temperature. Kwon et al. [
13] proposed decoupled processing and image appearance mapping to preserve color components and reduce color distortion. The decoupled processing is used to reduce interference between chrominance and luminance channels by processing the chromatic adaptation and tone compression in parallel. The image appearance mapping is used to preserve visual attributes such as lightness, chroma, and hue when recombining detail and base images.
In this study, we proposed color compensation and enhancement methods based on a multi-layer decomposition to reduce the color distortion in the boundary region and improve the decrease in saturation in the low-luminance region after tone compression. The multi-layer decomposition consists of a base layer (BL) and a detail layer (DL) separated with bilateral filtering, and an original layer (OL) without bilateral filtering. BL and DL are used for tone compression, and OL is used to acquire information to reduce the color distortion that occurs during tone compression. The color compensation and enhancement are processed in the IPT color space, and only the chrominance components are used. The IPT color space has better hue linearity than other color spaces. In IPT color space, I is the luminance channel, and PT are the chrominance channels. The color compensation includes a color difference map and a color scaling degree. The color difference map is used to detect a color distortion region that occurs in a strong boundary region. The color scaling degree is used to compensate for the chrominance distortion that occurs between the chromatic adaptation and tone compression. The color enhancement consists of an enhancement stop mask, an enhancement gain, and a color difference vector. The enhancement stop mask is used to prevent white point shift or hue shift at monotone or low saturation boundaries. The enhancement gain is used to compensate for the chrominance components according to the luminance level conversion between chromatic adaptation and tone compression. The color difference vector is used to improve the saturation degradation that occurs in the color boundary area. Finally, the chrominance components processed by the proposed method are combined with the luminance component of the existing tone-mapped image. The main contributions of the study are as follows:
The goal of the proposed method is compensating the color distortion around the color edge region and enhancing the desaturation in the low luminance and chroma areas after the color compensation.
The multi-layer decomposition is used to divide the image into base, detail, and original layers. The base layer is used for the chromatic adaptation and tone compression. The detail layer is used to enhance the detail information. The original layer is used to obtain the color distortion region and the necessary information for the color compensation and enhancement.
The color compensation is used to find the color edge region and reduce the color distortion around the color edge region. The color enhancement is used to reduce the desaturation of the low luminance and low saturation area after the color compensation.
The color compensation and color enhancement are processed in the IPT color space to preserve the luminance and chrominance.
3. Proposed Methods
To reduce the halo effect, the iCAM06 uses a bilateral filter to separate the image into base and detail layers. It preserves the detail layer with local contrast components sensitive to human vision while compressing the dynamic range of the base layer with global contrast components. The base layer sequentially performs chromatic adaptation and tone compression. The tone compression is performed on each CIEXYZ channel, which distorts the color balance of each channel. The tone-compressed image and the detail layer are merged, thereby amplifying the color distortion in the boundary area.
In this study, we propose multi-layer decomposition-based color compensation and enhancement algorithms to improve the color distortion and global saturation degradation in the boundary region after tone compression.
Figure 2 shows the flow chart of the proposed algorithm. The multi-layer decomposition block consists of a BL and a DL to which the bilateral filter is applied to obtain the color components necessary for color compensation and enhancement, and an OL to which the bilateral filter is not applied. The color compensation block deals with the detection of a color distortion area and a method for compensating for the distorted color. The color enhancement block deals with a method for improving the saturation degradation of the low luminance area that occurs in the tone compression image. The color compensation and enhancement algorithms are processed in
IPT color space, which has superior hue linearity to other color spaces.
3.1. Color Compensation
The proposed color compensation is used to compensate the color boundary region using the color difference map and color scaling degree. The color difference map is used to find the color distortion area that occurs after tone compression and compensate for the color distortion area. BL and OL images are used to find color distortion regions. The BL image represents the image generated from the bilateral filter, and the OL image represents the input image. The color distortion region that occurs after tone compression can be found by using the difference between the chrominance channels of the BL and OL images. The process is as follows:
Apply chromatic adaptation and tone compression sequentially to the BL and OL images.
Convert two tone-compressed images from XYZ color space to IPT color space.
Calculate the difference between the chrominance channels (P and T) of the BL and OL images as L2-norm.
Figure 3 shows the tone compression result of the test image.
Figure 3a is the input image. Unlike this pattern image, a general image has detailed information. Therefore, tone compression is applied only to the separated base image for preserving the detail information. The test image in
Figure 3 shows the problem of reproduction of the boundary of tone processing for the separated base image.
Figure 3b,c show the tone-compressed BL and OL images, respectively. A halo effect due to color distortion appears around the four square patterns in
Figure 3b. In particular, the red halo effect is displayed at the border of the blue rectangle. On the other hand, in the image of
Figure 3c, the halo effect does not appear around the square pattern.
Figure 3d is the color difference map and shows the area where the color distortion appears in
Figure 3b. The color distortion occurs around the edge of the rectangle. The larger the color distortion, the larger the color difference value and the brighter the pixel value in the corresponding area. The color difference map ranges from 0 to 1. The large color distortion areas have values close to 1. The equations of the color difference map are as follows:
where
i,
j represent the coordinates of the image.
Pd and
Td represent the color difference between
P and
T channels. Subscripts
tc_BL and
tc_OL represent the tone-compressed BL and OL images, respectively.
PTd is the L2-norm of
Pd and
Td, and
PTd_max is the maximum value of
PTd.
Md represents the color difference map, and the maximum value is 1.
The color scaling degree function is a weight that represents the color distortion ratio of chromatic adaptation and tone compression images. The chrominance component of the image is affected by the luminance component. During tone compression, a difference in the chrominance component of the chromatic adaptation image and the tone compression image occurs according to a change in the dynamic range. The color scaling degree is used to measure the degree of change in the chrominance component of the chromatic-adapted and tone-compressed images that occurs after tone compression. It is calculated using the scalar ratio of the
PT-axis chromatic response of the chromatic-adapted and tone-compressed images. The OL image is used to calculate the color scaling degree. The chromatic response can be calculated as the L2-norm of the
P and
T channels. The following equation shows the proposed color scaling degree:
where subscript
tc_OL and
ca_OL represent the tone-compressed and chromatic-adapted OL images, respectively.
Cs represents the color scaling degree.
The boundary color compensation uses a color difference map, color scaling degree, chromatic-adapted image, and tone-compressed images to reduce color boundary error. A value of the color difference map close to 0 indicates a small color boundary error area, while a value of the color difference map close to 1 indicates a larger color boundary error area. The ratio of the tone-compressed OL image is increased in the area with a small boundary error, and the ratio of the chromatic-adapted OL image is increased in the area with a large boundary error. The color scaling degree is used to compensate for the ratio of the chromatic-adapted OL image to the tone-compressed OL image when compensating for the boundary region. The proposed color compensation algorithm formula is as follows, and
Ps and
Ts represent standard color channels.
3.2. Color Enhancement
In general, since tone compression reduces the dynamic range to a locally different level, irregular saturation degradation occurs. The decrease in saturation is conspicuously displayed at the low-luminance monotone or low-saturation boundary. Color enhancement was proposed to reduce the saturation degradation occurring in the low luminance and low chroma areas. Color enhancement consists of an enhancement stop mask, an enhancement gain, and a color difference vector.
An enhancement stop mask is used to prevent white point shift or hue shift at monotone or low saturation boundaries. In the
IPT color space, saturation can be calculated using the L2-norm of the chrominance channels. The formula for the enhancement stop mask is as follows:
where
PTtc_OL is the L2-norm of
Ptc_OL and
Ttc_OL, and
Me represents the enhancement stop mask. The constant
c was set to 1 as a threshold value defining the low-saturation region.
Figure 4 shows the enhancement stop mask. In the input image, flowers, fruits, and cookie boxes with high saturation values have bright values in the high-saturation region of the color check.
The enhancement gain is used to compensate for the chrominance component according to the brightness change after the tone-compressed image. As the change in brightness affects the chroma component, it is necessary to compensate for the saturation corresponding to the change in brightness. The change in brightness can be calculated using the ratio of the luminance channels before and after tone compression. The compensation of saturation corresponding to brightness change is described by Bae et al. [
17], who modeled the chroma change rate corresponding to the luminance change of the image by changing the exposure value of the input image, and the formula is as follows:
where
Itc_OL and
Ica_OL represent the tone-compressed luminance channel and the chromatic-adapted luminance channel of each OL image. If the enhancement gain is less than 1, the minimum value of the enhancement gain is set to 1 because the saturation is further reduced.
Color difference vectors (
Vp and
Vt) are used to improve the saturation of the color boundary area. The color difference vector divided the color difference channels
Pd and
Td by
PTd_max to indicate the direction of change of the
P and
T channels.
Color enhancement uses standard color channels (
Ps and
Ts), enhancement gain
α, enhancement stop mask
Me, and color difference vectors (
Vp and
Vt).
Pe and
Te represent color-enhanced
PT channels.
Finally, the Pe and Te channels and luminance channel (I’) of the detail-combined image (IPT’) are synthesized. The detail-combined image is an image that combines the result of a tone-compressed BL image and detail adjustment.
Figure 5 shows the proposed color compensation and color enhancement. Distorted colors (
Ptc_BL,
Ttc_BL) are transformed into standard colors (
Ps,
Ts) using the proposed color compensation. The hue of the distorted color shifts from
θd to
θs. In the
IPT color space, the angle on the
PT-axis represents the hue. The saturation of the color boundary area is improved by using a color difference vector for the standard color. The enhancement stop mask is used to adjust the size of the color difference vector to prevent white point shift or hue shift in the low-saturation boundary area.
5. Conclusions
In this paper, multi-layer decomposition, color compensation, and color enhancement were proposed to improve color distortion and saturation degradation after tone compression. Multi-layer decomposition was processed separately into BL, DL, and OL to extract and compensate for the color distortion region that occurs during tone compression. The color distortion region was extracted using BL and OL images, and color information for the original image was obtained from the OL image. Color compensation and enhancements were processed in IPT color space with excellent hue linearity to reduce interference between luminance and chrominance components. The proposed color compensation used the color difference map to find the region where color distortion occurs and effectively compensated for the difference in chroma between tone compression and chromatic adaptation using the color scaling value. For color enhancement, it was possible to prevent white point shift or hue shift occurring in a low-saturation region by using an enhancement stop mask and to improve saturation by using an enhancement gain and a color difference vector. To confirm the performance of the proposed method, test patterns and various HDR scenes were compared with existing methods. IQA metrics such as BRSIQUE, TMQI, FSITM, and FSTIM_TMQI were used to compare the naturalness, distortion, and tone-mapped image quality of the image. The results showed that the proposed method has better performance in terms of naturalness, distortion, and tone-mapped image quality than the existing methods.