HSVNet: Reconstructing HDR Image from a Single Exposure LDR Image with CNN

Abstract

Most photographs are low dynamic range (LDR) images that might not perfectly describe the scene as perceived by humans due to the difference in dynamic ranges between photography and natural scenes. High dynamic range (HDR) images have been used widely to depict the natural scene as accurately as possible. Even though HDR images can be generated by an exposure bracketing method or HDR-supported cameras, most photos are still taken as LDR due to annoyance. In this paper, we propose a method that can produce an HDR image from a single arbitrary exposure LDR image. The proposed method, HSVNet, is a deep learning architecture using a Convolutional Neural Networks (CNN) based U-net. Our model uses the HSV color space that enables the network to identify saturated regions and adaptively focus on crucial components. We generated a paired LDR-HDR image dataset of diverse scenes including under/oversaturated regions for training and testing. We also show the effectiveness of our method through experiments, compared to existing methods.

1. Introduction

High dynamic range (HDR) photographs include rich color details by using higher color depths to encode a broad range of real-world luminance. Conventional color images and displays using the 8-bit depth for each color channel can only convey a limited range of color, which is called a low dynamic range (LDR), compared to the HDR. Especially when we take a picture in extremely bright scenery with some dark areas such as shadows under the sunlight or dark scenery with some bright areas such as the moon at night, the picture may partially contain very bright or dark areas. This results in missing details in over/underexposed areas of LDR images. However, creating a sound-quality HDR image with a standard camera setting is bothersome. We usually use exposure bracketing, which shoots multiple exposure images and merges them into an HDR image.

There has been active research towards developing the reconstructing process of HDR images from LDR images to be more universal. Thanks to advances in hardware technology, high-end cameras can capture 3 or 5 different exposure images at the same time and fuse them into an HDR image automatically. However, there are still a few unsolved problems: the price of specialized cameras, moving objects, and camera angle stabilization [1,2]. Many researchers have also developed software solutions to create an HDR image from a single LDR image. Inverse tone mapping (iTMO) [3] is a process that expands an LDR image to an HDR image for visualizing on a high dynamic range display. Several iTMO methods have been proposed based on mathematical models which infer color details in missing areas from LDR images [4,5,6]. However, those models have limitations that could not consider each image’s individual characteristics and relationships between channels. Recent advances in deep learning neural networks for the computer vision field enable numerous approaches to minimize these limitations. Deep Neural Networks, especially convolutional neural networks, help models to comprehend the complex nonlinearity between the input image and the target image, which is arduous for humans to detect [7,8,9,10]. However, there are existing limitations. Expanding an 8-bit LDR image to a 32-bit HDR image is still challenging because of the lost details in over/undersaturated regions and too low or high exposure images [10,11,12,13]. To solve these problems, we propose a method that lets a model find and concentrate on over/undersaturated regions. Additionally, the proposed method is able to convert the images which are captured by not only middle exposure but low or high exposure to HDR images.

In this paper, we propose a convolutional neural network architecture, HSVNet, for reconstructing an HDR image from a single-exposure LDR input image. We utilize U-net with a mask and a customized loss function. For training, we make our own paired LDR-HDR image dataset by taking multiple-exposure LDR images and fusing them into an HDR image. We also gather a paired LDR-HDR image dataset from the image source [14]. In the network, the color space of the input image, which is usually RGB (Red, Green, and Blue), is converted to HSV (Hue, Saturation, and Value) color space. We use HSV color space in this paper because the HSV color space is designed to more closely represent the human perception of color. Since each channel in the HSV space represents a perceptual attribute of color, we believe that deep neural networks can learn the ground truth more concisely. Specifically, the V channel represents the brightness which is directly related to the high dynamic range of natural lights. Additionally, the hue of color in RGB space is determined by the ratio between three channels, which can make it difficult to reconstruct them. A better estimation is expected by separating these attributes. Reconstructing HDR in RGB color space may feature a weakness such as severe local inversion artifacts on reconstructed HDR images and insufficient contrast [2,15,16]. This is partly because loss functions were simple mean-absolute error (MAE) or mean-squared error (MSE) between the predicted and target images. Therefore, we propose a customized loss function for HSV color space which allows the network to output reliable and natural color.

This paper presents the following three main contributions.

1

We propose a deep learning system that can reconstruct a high-quality HDR image (32-bit) from an arbitrary single-exposed LDR image (8-bit) which may include both over- and undersaturated regions;

2

We present a mask function derived from Saturation (S) and Value (V) information in HSV color space which can detect both over- and underexposed areas and suppress unnecessary information for each channel;

3

We propose a customized loss function appropriate for HSV color space and choose an optimal loss function by experiments.

2. Related Works

There has been extensive research on reconstructing HDR images from LDR images, which is also known as inverse tone mapping [3]. This problem is still challenging because of the missing information on over/undersaturated regions. In this section, we summarize previous research.

2.1. Non-Learning Methods

In the early years, non-learning methods based on mostly mathematical models were proposed. Banterle et al. [3] first presented inverse tone mapping operations and utilized the Median Cut algorithm [17] to find high luminance areas and generate an expand-map from the result of density estimation of the areas. Rempel et al. [4] applied a Gaussian function to an expand map. They then used an edge stopping function to enhance the brightness of saturated areas. Kovaleski and Oliviera [5] exploited analogous steps to Rempel et al. by using an inverse tone mapping based on cross bilateral filtering. Wang et al. [6] presented a hallucination method that requires user interaction. The saturated region could be re-filled with texture information from a user-selected source region. Meylan et al. [18] applied a piece-wise linear tone scale function to expand the specular and diffuse parts in the image. The unnatural artifacts from discontinuity in the tone scale function were suppressed by adding a smoothing step. Masia et al. improved on their previous methods [19], which had the shortcoming of insufficient performance for the large oversaturated input images, by proposing a global iTMO based on a simple gamma expansion [20]. Huo et al. [21] proposed a psychological iTMO based on the HVS (Human Visual System) using the local retina response for minimizing artifacts. Sen et al. [22] proposed a patch-based approach. They generated highly related local features of the result with all LDR images directly. The alignment and merging process could be completed at the same time.

2.2. Learning Methods

In recent years, methods utilizing deep learning from input to targeting ground truth have been actively presented, due to the excellent performance on analytic training. Most of them were based on a convolutional neural network (CNN). Chen et al. [7] expanded an 8-bit LDR image to a 16-bit image by utilizing a U-net-based network with a spatial feature transform (SFT) layer. Eilertsen et al. [8] also used a U-net with gamma corrected skip connections. They proposed a masking method to find overexposed regions in images. The masks were created by the maximum value among red, green, and blue channels and used for blending an input image and a predicted image. Marnerides et al. [9] attempted to capture different levels of features in input LDR images. They proposed 3 different branches—local, dilated, and global branches. Local/global branches have a small/big receptive field for high-level features/low-level features. The dilation branch captures mid-level features. Yang et al. [23] introduced a U-net-based image correction method which reconstructs an HDR image from an LDR image first and then renders a color-corrected LDR image from the HDR domain. Khan et al. [24] suggested an end-to-end feedback network that was effective for rendering a coarse-to-fine representation. They learned the feedback network using the recurrent neural network multiple times resulting in a deeper network for better reconstruction quality. Liu et al. [10] designed a pipeline in a sequence of clipping, a non-linear mapping, and quantization for HDR images to LDR images. They then trained the CNN model to learn each reversed step of the pipeline. For loss functions, MAE, MSE, and perceptual loss from the VGG network [25] were employed. Additionally, Santos et al. [26] utilized perceptual loss. They applied the sum of MAE loss and perceptual loss as a loss function. They used a U-net architecture with masks kept updated every step. Li et al. [27] suggested the attention approach. They devised a network with two different branches, which are the attention branch and the dilation branch, for finding interdependencies between channels.

After generative adversarial networks (GAN) [28] were presented, GAN demonstrated its exceptional performance in image translation. Researchers applied the concept of GAN to inverse tone mapping since it also maps input images to target images. A GAN-based inverse tone mapping approach was first proposed by Niu et al. [11]. They suggested the model with adversarial learning to eliminate the misalignment from large motions. Moriwaki et al. [16] proposed a hybrid loss function. The proposed loss function was the sum of reconstruction loss (MAE), adversarial loss (GAN loss), and perceptual loss (VGG loss). Kim et al. [29] also applied the GAN-based framework to their three subnetworks for image reconstruction, a detail restoration, and a local contrast enhancement. Marnerides et al. [30] used a GAN-based architecture with a Gaussian distribution to transform the density of HDR pixels.

There has been some research that attempted to use the HSV color space. Ye et al. [12] applied a Hue loss function with a simple MAE. They proposed a network with two branches for overexposed and underexposed areas, respectively. Two kinds of masks, light and gray masks, were employed for training, which was calculated by the gray scale of an LDR image. Wang et al. [13] also employed an HSV loss function. They utilized saturation information for their loss function with a simple MAE. Their network predicted different exposure images from an input image and then merged all of them into an HDR image.

3. Method

We propose a method to generate an HDR image from an LDR image with a deep learning architecture. Our proposed method first (1) pr3processes the data by converting color space into HSV, normalizing H, and a gamma correction on V channel of ground truth for the normalized range in $[0,1]$, (2) creates a mask for each input LDR image, (3) learns the U-net based network, HSVNet, to generate an HDR image by uncovering the relationship between H, S, and V color components, and (4) postprocesses the prediction to recover the original range of V with an inverse gamma correction ($f^{-1}$) on the V channel,

$$H_{i,v}=f^{-1}\left({\widehat{y}}_{i,v}\right)$$

(1)

where H is the reconstructed HDR, i is the index of image, v is V channel, and $\widehat{y}$ is the network output. The overall process of our framework is shown in Figure 1. First, we convert the input and the corresponding ground truth image to HSV color. Then, we normalize all values which are out of the range $[0,1]$ and create a mask of the normalized input. We transfer the normalized input, mask, and ground truth to our network. The mask is used as a weight in a loss function. Finally, we map the values of the estimated image to the original range of the ground truth to obtain a genuine HDR predicted image.

3.1. Architecture

An autoencoder is a feedforward neural network that is trained to attempt to copy its input to output by reducing the input to low-dimensional data and reconstructing high-dimensional data as the output [31,32]. The autoencoder consists of an encoder part (contracting path) and a decoder (expansive path). U-net is an autoencoder with skip connections between encoder and decoder [33]. The skip connections help rebuild the high-frequency features, such as border pixels, which could be lost from the deep convolution layer [34]. We use the same concept for the reconstruction of HDR images. Our proposed network is shown in Figure 2. The proposed network is based on the well-known VGG-16 network [25].

In the encoder layer, the spatial feature resolution is gradually reduced by half from 256 to 4, applying $3\times 3$ convolutions with stride 1. After a bottleneck layer, the decoder reconstructs full-dimensional data from the latent data by upsampling. The decoder process is the reversed encoder process. We use $4\times 4$ deconvolution for upsampling with stride 2. There are skip connections between encoder and decoder to preserve the detailed information. Each skip connection concatenates the encoder layer and the decoder layer corresponding to the resolution. Then, the concatenated layer is downsampled with $1\times 1$ convolutions with stride 1 to reduce the number of feature channels by half. We use ReLU activation function at every layer.

3.2. Loss Function

A normal Mean Absolute Error (MAE) is applied to S and V channels. Unlike Saturation and Value, Hue is angular data. When MAE is directly applied to the H channel, the model cannot predict the color around the red correctly. Even though both $0^{°}$ and ${360}^{°}$ indicate the same red color, MAE gives the highest penalty when predicted H is $0^{°}$ and target H is ${360}^{°}$ or vice versa. Therefore, the red color might converge to the average value of $0^{°}$ and ${360}^{°}$. As shown in Figure 3, the predicted images with a MAE H loss may have local artifacts, e.g., the red color is predicted as blue.

Therefore, we develop a customized loss function, $L_h$, for the H channel. First, we define a simple wrap-around distance, $d_h$, which is the difference between the predicted value and the target value in polar coordinates. $d_h$ is the angle of the shortest arc between two H values on a unit circle. As shown in Figure 4, the shortest arc angle between $H_1$ and $H_2$ can be represented as $|H_2-H_1|$ or $1-|H_2-H_1|$. Since we take the shortest path, the minimum value is selected.

$$d_h(H_1,H_2)=Min\left(\right|H_2-H_1|,1-|H_2-H_1\left|\right)$$

(2)

Figure 5a shows a Hue space of HSV color [35], where the angle presents the Hue. In this space, three primary colors (Red, Green, and Blue) are separated by and three secondary colors (Yellow, Cyan, and Magenta) are located ${60}^{°}$ from primaries. As a result, color changes largely every ${60}^{°}$. For this reason, if the wrap-around distance is over 60 degrees, we assign the maximum penalty to prediction in the H loss function. In human eyes, we perceive that both cyan and green are far from red to a similar extent, even though green is closer to red in the Hue space model theoretically. Therefore, not directly applying wrap-distance to $L_h$, we set the maximum penalty if $d_h$ is over $1/6$ (=${60}^{°}/{360}^{°}$) (see Figure 5b), which is a modified ramp function.

Finally, $L_h$, $L_s$, and $L_v$ are defined as

$$\begin{array}{cc}\hfill L_h& =\frac{1}{N}\sum M_{h}Min(6d_h({\widehat{y}}_h,y_h),1)\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill L_s& =\frac{1}{N}\sum M_s|{\widehat{y}}_s-y_s|\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill L_v& =\frac{1}{N}\sum M_v|{\widehat{y}}_v-y_v|\hfill \end{array}$$

(5)

$$L=L_h+L_s+L_v$$

(6)

where L is a loss function with channel h, s, and v, M is a mask with channel h, s, and v, N is the number of pixels, $\widehat{y}$ is estimation, and y is ground truth. The total loss function L is the sum of $L_h$, $L_s$, and $L_v$. Each mask is applied to the corresponding channel in the loss function. We recognize that there are two distinct regions where H, S, or V are differently important to the training. Therefore, the mask applied to the loss function contributes to the better quality of output by adaptively focusing on crucial regions during the training. As Figure 6 shows, the HDR images predicted with the mask are more similar to the ground truth than those without the mask. The details about the mask are discussed in Section 3.3.

3.3. Mask

One of our main objectives is to infer color details in over/underexposed regions from the LDR image. It is essential to understand the characteristics of saturated regions in the HSV color space. H is relatively irrelevant for finding saturated pixels since the over/underexposed regions hold little information on the hue of the color. Therefore, we design a mask with S and V values that can separate saturated regions and unsaturated regions. We consider saturated areas to have the following characteristics: saturated areas have small S and high or low V (see Figure 7). We first build a saturation mask $M_d$ to differentiate between high saturation values and a value mask $M_u$ to distinguish bright and dark values. Then, we combine them into the final mask, $M_f$, as shown in Equation (9):

$$\begin{array}{cc}\hfill M_d& ={\left(\frac{c}{x_s+c}\right)}^{s^{\prime }}\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill M_u& =4(1-v_{min}){(x_v-0.5)}^2+v_{min}\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill M_f& =M_d{M}_u\hfill \end{array}$$

(9)

where x is input image, $x_s$ and $x_v$ are input image’s S and V, values respectively, and c, $s^{\prime }$, and $v_{min}$ are parameters for the mask. For the saturation mask $M_d$, the weight is inversely proportional to the pixel’s S value, which means a pixel has a higher weight for a lower S (see Figure 8). The weight for the saturation mask is determined by adjusting c and $s^{\prime }$. The weight is always 1 for the smallest S, when $x_s=0$. Increasing c allows a more gradual descent of $M_d$, where high S pixels are not sifted much. Increasing $s^{\prime }$ results in a more curved and steeper function. Therefore, the network focuses more on the low saturated regions with smaller c and bigger $s^{\prime }$. For the value mask $M_u$, a function is U-shaped between 0 and 1, where a pixel has a higher weight as V approaches two borders, 0 and 1, as shown in Figure 8. $v_{min}\in [0,1]$ determines the weight for the value mask. The weight is 1 for the smallest and the biggest value, when $x_v=0$ and $x_v=1$. Since lower $v_{min}$ makes the lower midpoint of $M_u$, the network focuses more on bright or dark areas with lower $v_{min}$. By the experiments on different parameters for the mask, in ranges $s^{\prime }\in [1,1.5]$, $c\in [0.5,1.5]$, and $v_{min}\in [0.1,0.9]$, we use $s^{\prime }=1.0$, $c=1.0$, and $v_{min}=$$0.7$ in this paper.

Then, we use $M_f$ for S and V masks, $M_s=M_v=M_f$, and $1-M_f$ for H, $M_h=1-M_f$. The mask applied to our proposed loss function is as follows.

$$\begin{array}{cc}\hfill L_h& =\frac{1}{N}\sum (1-M_f)Min(6d_h({\widehat{y}}_h,y_h),1)\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill L_s& =\frac{1}{N}\sum M_f|{\widehat{y}}_s-y_s|\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill L_v& =\frac{1}{N}\sum M_f|{\widehat{y}}_v-y_v|\hfill \end{array}$$

(12)

We suppress the influence of the H channel on the saturated regions when learning the network since Hue is unreliable in these regions. Therefore, we use the inverted $M_f$ for the mask of hue, $M_h=1-M_f$. Figure 9 shows that the predicted HDR images could have local artifacts when the identical mask is applied to all H, S, and V channels.

As shown in Figure 6, the result tends to be similar to the input image when mask is not applied. Additionally, when the same mask is applied to H, S, V, many results show artifacts (second column of Figure 9).

Table 1. Comparison of the network with no mask, the network with the identical mask, and the network with the inverted mask. The values in bold indicate the best scores for each metric.

	L1	HDR-VDP-2.2	PU21-SSIM	PU21-PSNR
No mask	0.0697	50.615	0.7073	9.8803
Identical mask	0.0631	52.3293	0.7493	15.4056
Inverted mask	0.0672	52.5142	0.7761	16.6479

shows that the result with the inverted mask performs better than the result with no mask or with the identical mask.

We adopt 4 full-reference metrics to compare the performances of loss functions: L1 loss (Mean Absolute Error) [36], HDR-VDP-2.2 Q score [37], PU21-PSNR [38], and PU21-SSIM [38]. The HDR-VDP Q score suggested by Nawaria et al. is a visual metric especially for HDR images. Q score compares the quality of a test image to that of a reference image based on the human perception. The high score indicates that the test image is close enough to the reference image. PU21 is a metric for HDR images using perceptually uniform transform while overcoming the nonlinearity of PSNR and SSIM.

4. Analytical Study

To determine the optimal architecture, loss function, and masks, we tested our model in different settings including (1) the sequence of layers and (2) loss functions.

4.1. Network Variations

Our proposed network is a VGG-16-like network and we ran ablations on different layer combinations. We experimented in order to confirm the optimal network for our dataset. We stacked up more layers than VGG-16 for expanding the receptive field. We experimented with variations in the VGG-16-like network (see Figure 10): (a) a network with convolution layers always followed by a pooling layer, (b) a network with consecutive blocks, each with one non-pooling convolution layer and one convolution layer followed by a pooling layer, and (c) a network with blocks with two non-pooling convolution layers and one convolution layer, followed by a pooling layer.

The performances of the second and the third network were meaningfully better than the first one. Especially, the result of the third network is slightly better than that of the second network (see

Table 2. The test results for networks (1), (2), and (3) shown in Figure 10. The values in bold indicate the best scores for each metric.

Network Variations	L1	HDR-VDP-2.2	PU21-SSIM	PU21-PSNR
network (1)	0.0602	51.8483	0.7416	15.4565
network (2)	0.0605	51.8748	0.7288	15.7509
network (3)	0.5770	52.1077	0.7330	15.1419

). However, the training time of the third network is much longer than the second one. The training times of network (1), (2), and (3) are 2 h 30 min, 3 h, and 4 h, respectively. Based on these results, we decided to use the second network.

4.2. Loss Function Variations

We ran variations on different loss functions.

1

Cone space distance: Since HSV color space can be represented as a cone model, the most intuitive difference between two HSV colors will be a Euclidean distance in a cone space. In this method, we convert HSV colors of H, S, and V to points in a cone space and calculate a Euclidean distance between the prediction and the target (Figure 11 left).

2

Wrap-around distance ($d_h$): In this method, we use mean absolute error (MAE) for each channel difference. Since H values are angles, we compute wrap-around distance, $d_h$, for H loss as in Equation (2), which is the angle of the shortest arc between two H values (Figure 4).

3

Square root of wrap-around distance ($\sqrt{d_h}$): We tried using the squared root of the wrap-around distance, $\sqrt{d_h}$, for H loss because we believe the penalty should be similar for big errors as described in Section 3.2. MAE was applied to S and V loss.

4

Circle plane distance: We converted H values to the points on a unit circle in polar coordinates and calculated the Euclidean distance between two points for H loss as shown in the right image of Figure 11. This function also shows steeper slopes for smaller distances as the square root function. MAE was applied to S and V loss.

5

Modified ramp: We cropped wrap-around distance $d_h$ and set the maximum value 1 when $d_h$ is over ${60}^{°}$, as described in Section 3.2 (Figure 5b). MAE was applied to S and V loss. We tested the network with various degrees for the penalty threshold, and ${60}^{°}$ performs better than others, as shown in

Table 3. Test results with setting the maximum penalty over different degrees in the H loss function. The H loss function with degree $\theta$ is $Min(\frac{360}{\theta }{d}_h,1)$. The values in bold indicate the best scores for each metric.

	L1	HDR-VDP-2.2	PU21-SSIM	PU21-PSNR
${30}^{°}$	0.0647	51.9500	0.7542	15.4394
${45}^{°}$	0.0583	52.0048	0.7280	15.2573
${60}^{°}$	0.0672	52.5142	0.7761	16.6479
${75}^{°}$	0.0713	47.7744	0.6498	12.5720
${90}^{°}$	0.0617	52.5844	0.7608	16.4168

.

Table 4. The test results for different loss functions: The same mask parameters are applied ($s^{\prime }=1.0$, $c=1.0$, $v_{min}=$$0.7$). The values in bold indicate the best scores for each metric.

Loss Function	L1	HDR-VDP-2.2	PU21-SSIM	PU21-PSNR
cone space	0.0624	52.5526	0.7601	15.9850
$d_h$ + $MA{E}_{s,v}$	0.0583	52.0549	0.7341	13.6357
$\sqrt{d_h}$ + $MA{E}_{s,v}$	0.0585	52.7337	0.7635	16.0414
unit circle${}_h$ + $MA{E}_{s,v}$	0.0760	52.0152	0.7603	16.2927
$Min(6d_h,1)$ + $MA{E}_{s,v}$	0.0672	52.5142	0.7761	16.6479

presents test results for different loss functions with the same mask parameters. Although there is no single method which outperforms others in all factors, the last method of modified ramp function, $Min(6d_h,1)+MA{E}_{s,v}$, produces visually plausible results, as well as the satisfactory test results in general.

5. Training

5.1. Dataset

A major challenge for successful development is to gather a sufficiently large paired LDR-HDR dataset which has multiple exposures of LDR images. Even though many HDR datasets are available on the web, most of them do not provide corresponding LDR images. FHDR [24] and HDRCNN [8] generated LDR images from HDR images to obtain training data. However, our goal for the dataset is to create authentic LDR images in multiple exposures paired with each HDR image. To use genuine LDR and HDR paired images, we use Fairchild database [14] which provides 7 or 9 multiple-exposure LDR images for each HDR image. We also generated our own paired dataset. We captured 504 scenes using a Canon Mark 3 in 5, 7, or 9 different exposures. The resolution of the images is $5760\times 3840$ pixels. The dataset comprises various categories: landscapes, outdoor, indoor, buildings, objects, daytime, and nighttime. We merged these LDR images (8-bit) with HDR images (32-bit) using the HDR Pro process built-in Photoshop. The number of total HDR images is 608.

5.2. Preprocess

In order to build a network for a broader application, we do not use fixed exposures, but a random exposure among −1, 0, +1 EV for the input dataset. For the input dataset, we first choose a single image among 3 multiple-exposure LDR images randomly. We resize and crop the chosen image’s short side into $500\times 500$ resolution. Then, the image is randomly cropped into 4 pieces with $256\times 256$ resolution to solve a small dataset problem. For random cropping, we use the low-discrepancy sequence proposed by Hammersley [39] to avoid overlapping. The ground truth dataset is perfectly paired with the input dataset. Finally, we attain 2432 images of input and ground truth images. Train and test datasets are separated from 2432 images by randomly selecting the group of 4 cropped images which are from the same original image. Consequently, a train dataset has 2000 images from 500 original images and a test dataset has 432 images from 108 original images. All images are converted to HSV color space before training. H is divided by 360 to be normalized in the range $[0,1]$. S of input and ground truth images are already well normalized in the range $[0,1]$ when converted to HSV color space. V of ground truth images is not in the range $[0,1]$, while V of input images is in the range. To normalize V of ground truth images, we use a gamma correction, $f\left(x\right)={\frac{x}{x+1}}^{1/\gamma }$, where $\gamma =2.2$.

The network has two inputs (the normalized input images in HSV color and the corresponding mask) and one target input (the normalized target images in HSV color). Through the network, the mask is multiplied to the loss function as a weight.

5.3. Implementation and Details

The total number of parameters of the proposed network is 49,590,945, taking approximately 3 h on an Nvidia GeForce RTX 2070 GPU and 32-GB RAM. We implemented the proposed network in Python with Keras using Tensorflow library. We use Adam optimizer with a learning rate of $1\times {10}^{-4}$. The number of epochs is 100.

6. Results and Evaluation

We compared our methods with four other state-of-the-art CNN-based methods: SingleHDR [10], HDRCNN [8], Expand-Net [9], FHDR [24], and HDRUNet [7]. We fed our test dataset to each network and obtained results. It can be seen from

Table 5. The test results of our HSVNet and existing methods in four metrics. The value in bold indicates the best score for each metric.

	L1	HDR-VDP-2.2	PU21-SSIM	PU21-PSNR
SingleHDR [10]	0.1789	50.7160	0.7433	13.8755
HDRCNN [8]	0.1548	48.7974	0.6497	11.3025
Expand-Net [9]	0.1099	49.055	0.6731	11.1831
FHDR [24]	0.1031	50.1611	0.7328	14.5347
HDRUNet [7]	0.3400	47.5183	0.6596	9.3127
HSVNet(ours)	0.0672	52.5142	0.7761	16.6479

that our proposed method performs quantitatively better than other methods.

Figure 12 shows the visual comparison of result images between our method, HSVNet, and others. In most cases, HSVNet produces satisfying results which are closest to ground truth images than others. Missing details in both undersaturated and oversaturated areas, as well as the rest areas of input LDR images, are fairly well reconstructed by HSVNet. Especially in the color checkerboard in the fifth and the sixth row, most of colors are restored well even though the checkerboard in the input LDR image is partially undersaturated due to shadows or oversaturated due to sunlight. In addition to the hue of the checker board, saturation and brightness are well recovered by our method.

Our approach has some limitations, as well. Our method might fail to infer missing detailed information, especially when the majority of the input image is oversaturated. When the input is extremely overexposured (see the first and second row of Figure 13), missing details in input LDR images are not reconstructed well with our method as with all other methods. However, our HSVNet predicts brightness and saturation better than other methods even in extremely oversaturated regions. Additionally, as the input image of the third and the fourth row in Figure 13 is largely overexposed, there is a huge gap between the result images and the ground truth image. There is also a limitation on the resolution of images. Our model is developed not for high resolution, but for relatively low resolution ($256\times 256$ pixels). The performance might be debased when the size of the input image is large.

7. Conclusions

It is challenging to reconstruct an HDR image from a single random exposure LDR image. To solve this problem, we propose an end-to-end model, HSVNet, which is a U-net-based network. The main difference from previous HDR reconstruction models is using HSV information of images to allow the network to learn the relationships between saturated regions and colors. We develop a customized loss function that is suitable for HSV color space based on characteristics of Hue space to produce satisfying optimal results for human eyes. By applying masks created from S and V information, we restrain the impact of H on saturated regions. The performance quality is proven through comparisons to other existing inverse tone mapping models after experiments.