3.1. Single Camera Module Time-Division Image Acquisition System
In this study, daytime images were obtained by temporally multiplexed capturing using an IR cut filer, and in the case of night images, a separately manufactured imaging device was used. A time-division image acquisition system using a single CMOS broadband camera module was applied to obtain visible and near-infrared images, thereby solving the problem of matching errors due to the optical path difference occurring in a dual camera system. However, if a time-sharing system is used, motion blur may occur because of a time-shifting error. To address this issue, different exposure times were set for the visible-light image and the near-infrared image, and when the amount of light was insufficient owing to a short exposure time, the amount of light was secured by using the IR auxiliary light. The decision to use the IR auxiliary light was made according to the surrounding environment.
The time-shifting error and motion blur both increase if the exposure time is increased to secure sufficient light while acquiring low-light images with minimized exposure time. The increased noise is unfavorable for high-quality image acquisition. Therefore, an image is obtained with minimized time-shifting error, motion blur, and noise by supplying a sufficient amount of light in a short time using the IR auxiliary light. In the case of daytime images, which experience luminance saturation when using IR-assisted light, images are acquired while limiting the use of IR-assisted light [
11].
When acquiring visible-light and near-infrared images, it is difficult to adaptively adjust the brightness of the lighting according to environmental changes. Therefore, in this study, we proposed a time-division image acquisition system using a camera trigger signal and a flash-type IR auxiliary light switching signal, as shown in the flowchart in
Figure 3. As shown in
Figure 4b,c, by repeatedly operating the auxiliary light with the IR auxiliary light switching signal, a sufficient amount of light is first secured in the near-infrared image using a fixed IR flash of sufficient brightness, and the luminance channel visibility is then secured by adjusting the synthesis ratio with the visible-light image for the saturated area.
3.3. Base Layer Synthesis
The base layers of the separated visible-light and near-infrared images are synthesized by forming different weight maps according to the infrared natural and auxiliary light characteristics. In the case of daytime images, the intensity and diffusion of IR natural light in the near-infrared images are secured compared to visible images in distant, fog, and shade areas. In contrast, at night, IR auxiliary light is used to compensate for the low-light conditions, long-range IR infiltration is weak in the nighttime infrared images, and the near-distance luminance saturation is relatively strong due to the IR flash.
The main purpose of base layer synthesis is local background tone compression through synthetic ratio control of visible-light images and near-infrared images. This compresses the local tone range of the base layer within the limited luminance range of the display, thereby improving the expression performance of the detailed layer and enhancing the clarity. Therefore, in low-light areas, such as backlight and shade, which inhibit visibility, the synthesis ratio of an infrared image with a relatively large luminance value must be increased, and in a high-light area, such as the anterior saturation area caused by night auxiliary lights, the ratio of visible images with a relatively small luminance value should be increased.
As shown in
Figure 6a,b, the left-side visible-light luminance image has low-level shading information on average, and the right-side near-infrared luminance image is bright on average and has many saturated regions owing to the extended band component. The image synthesis ratio needs to be adjusted for these regions. A bilateral filter with edge preservation is used to divide local regions according to strong boundary components and compress tones from the synthetic map, from which details are removed. The structural formula of the bilateral filter used in each synthesis map and the VIS-IR joint bilateral filter blurred by the boundary information of the near-infrared image is as follows:
where
is a bilateral filter map, which is an image of applying a bilateral filter by substituting visible-light luminance images
into the input image
I in Equation (1). In addition, in Equation (2),
is a joint bilateral filter map obtained by substituting a visible-light luminance image
into the input image
I and a near-infrared luminance image into the input image
to determine the blur weight using boundary information.
is a min–max normalization function that normalizes the image to a value between 0 and 1.
However, when the two images are synthesized using a bilateral filter map to preserve the strong boundaries of the object, the results include distorted objects because the object details are reflected in the synthetic map, as shown in
Figure 6c. In the low-brightness areas of visible light, the synthetic ratio of near-infrared images is increased, and the high-brightness areas of near-infrared images are excessively synthesized in dark letters. When creating a synthetic map using a joint bilateral filter, the strength of near-infrared images is reflected in the visible-light low-brightness areas, which increases the ratio of near-infrared images. Therefore, the VIS-IR joint bilateral filter, which includes the boundary information of the near-infrared image of Equation (7), was used.
Figure 6d shows the synthesis results using the joint bilateral filter map. It is confirmed that the internal area of the object of the synthetic map is blurred according to the strength of the neighboring pixel image of the near-infrared image, and the details of the information are well-synthesized as the local tone increases in the resulting image.
Figure 7 shows the average synthesis results of the visible-light and near-infrared luminance base layer for comparison with the synthesis results of the visible-light and near-infrared image base layer.
Next, synthetic maps were added to reflect the characteristics of the nearest infrared images that appear differently between day and night, along with the joint bilateral filter. In the daytime visible-light image, a low-brightness area that inhibits visibility due to shade or backlight, such as the center of the tree in
Figure 8b, is generated, and the boundary information is not expressed. In contrast, in the daytime near-infrared image, light penetration and diffusion phenomena caused by IR natural light in the tree shade area, as shown in
Figure 8c, secures a large amount of light compared to visible light and increases visibility, including boundary information. The characteristics of this daytime near-infrared image are visible through the car image of visible-light and near-infrared rays in
Figure 8d, and it can be seen that a large difference occurs in the area where IR penetration occurs.
In the nighttime visible-light image, the border information is weak for shaded areas, such as the box in
Figure 9b, and has a low luminance value on average. In contrast, in the nighttime near-infrared
Figure 9c, IR auxiliary light complements the lack of light in the box and secures the boundary information of the color chart in the box. However, unlike natural light, auxiliary light is weak in long-range IR infiltration and diffusion, and short-range luminance saturation caused by the front filter of the IR flash is relatively strong. This can be confirmed by the characteristics of the night image using the auxiliary light and the change in the luminance in the cup area where saturation occurs when comparing the box and the cup area in
Figure 9d. Therefore, brighter daytime visibility and near-infrared car images reflect IR light penetration, and brighter visible-light and near-infrared car images show anterior luminance saturation by IR auxiliary light. To synthesize the characteristics of these day and night images, an IR depth compensation map was applied at night, and a penetration compensation map was applied during the day.
Because the night near-infrared image uses the IR assist light from the front, the difference between the visible-light image and the luminance value is oversaturated in the objects closer to the camera and shows the characteristics of the depth map, where the difference becomes weaker as the distance increases. Therefore, in the near-infrared image, it is necessary to increase the synthesis ratio of the non-saturated visible-light image so that the boundary information is well-preserved in the forward saturation region owing to the auxiliary light projection. In this case, the halo phenomenon that may occur during local tone mapping is minimized by using the difference image of the base layer image to which the bilateral filter is applied. The depth compensation map is obtained as follows:
where, as the value of the depth compensation map
increases, it becomes a near-saturation region and increases the ratio of the visible-light base layer.
and
are the base layer images of the visible and near-infrared images, respectively.
Unlike nighttime images using auxiliary light, light diffusion and penetration by near-infrared natural light occur in daytime images. A visible-light image may have a high brightness level in the distant region, and an absolute value is used for the difference between visible light and near-infrared light. Because the regions with a large difference value are ones in which visibility can be improved by IR light penetration, it is necessary to increase the synthesis weight of the near-infrared image to secure boundary information. However, in the case of the daytime image, because the average luminance of the bright region of the visible-light image is higher than that of the night image, synthesis saturation may occur in the high-luminance region of the visible-light image. Therefore, the daytime base layer synthesis increases the ratio of the near-infrared base layer image as the difference value in the visible-light and near-infrared difference images increases, but this should be limitedly applied to the low-luminance region of the visible light image. The penetration compensation map is obtained as follows:
where, as the penetration compensation map
value increases, it becomes a near-infrared penetration compensation region and increases the ratio of the base layer of the near-infrared image. The
term indicates the IR light penetration region, and the
term is a function that limits the low-luminance region of the visible-light image. Using these two characteristics as an AND condition, the ratio of the near-infrared image was increased.
Using the previously obtained VIS-IR joint bilateral filter and the additional compensation map according to the day and night images, the final base layer synthesis map and synthesized image can be obtained as follows:
where the larger the value of the base layer synthesis map
, the higher the visible light base layer synthesis ratio.
is the luminance base layer synthesis result image, and the base layers of the day and night images generated above are synthesized using the alpha blending method with the base layer synthesis map as a weight.
3.4. Detail Layer Synthesis
In the synthesis of detail layers, the main goal of synthesizing sublayers is to include and synthesize as much boundary information as possible according to the activity of the visible and near-infrared images. In the visible-light image, boundary information is lost in distant, foggy, or shaded areas. In the near-infrared image, the luminance saturated area due to the auxiliary light, or the boundary information below the water surface through which the near-infrared light does not pass, cannot be expressed. Therefore, for synthesis, the activity of detailed layers of visible-light and near-infrared images is calculated through local variance, and a variance map that can adjust the synthesis ratio according to the difference is used.
In this study, the mean deviated local variance, which has low computational complexity, was used to measure the activity of images. Because the mean deviated local variance uses the difference from the mean values of the pixels around the central pixel without considering each pixel value of the kernel, the distribution of changes in the pixel values in the kernel can be well-represented. The mean deviation local variance is calculated as follows:
where
is the mean deviated local variance, and
is the local mean.
is the size of the kernel, and a value of
was used in the proposed method.
The local variance image contains detailed boundary information for each visible-light and near-infrared image. Because the square term of the luminance channel is included in the calculation of the local variance, the variance may have a value of 0 to
. In this case, because the range of values is very wide, the min–max normalization method is used, and the case where the variance value is small can be ignored. Therefore, we use a log function to adjust the scale. The difference image between the two dispersed images is then obtained and used as an index for determining which image among the visible-light and near-infrared images contains more detailed information in a specific area. The following equation shows the local variance log-scale difference images of the visible and near-infrared images.
where
represents the log-scale local variance difference image.
and
are the mean deviated local variance images of the visible and near-infrared regions, respectively.
The log-scale difference image including negative values is then converted to a value between 0 and 1 and normalized so that it can be used as a synthesis map. If the local variance difference image (
) is negative, it indicates an area with a relatively large amount of detail in the near-infrared image, and if it is positive, it indicates an area with a relatively large amount of detailed information in the visible-light image. For this purpose, when the variance difference image is negative, it is normalized to a value between 0 and 0.5, and when it is positive, it is normalized to a value between 0.5 and 1.
where
is a distributed map in which
has a value between 0 and 1, and the proposed regularization expression is applied.
A final detailed layer composite map is created by applying a blur function to the variance map, as follows:
where the
function represents a two-dimensional Gaussian filter and
,
are the standard deviations in the x and y directions, respectively.
is a detailed layer synthesis map that is blurred by applying a Gaussian filter to
for natural synthesis. The larger the kernel size of Gaussian function, the more natural detail layer fusion is possible, but a tradeoff occurs due to a decrease in detail contrast and an increase in the amount of computation. Thus, in this study, the kernel size (7, 7) was applied for this Gaussian filter.
Using the generated detailed layer synthesis map (
), the synthesis is performed as shown in the following equation:
where
is the luminance detailed layer synthesis result, and the detailed layers of the day and night images generated above are synthesized using the alpha blending method using the detailed layer synthesis map as a weight.
and
are detail layer images of visible and near-infrared images, respectively.
Figure 10 shows the sublayers of the visible and near-infrared images and the results of the synthesis of the sublayers. The details of the cup and box parts are well-preserved.
The final synthesized luminance channel
is generated as follows by synthesizing the independently synthesized base layer and detailed layer using the above process.
3.5. Color Compensation and Adjustment
In general, color compensation is required to maintain the same color information when the luminance of an image is changed. The color information uses the
ab channel of the visible-light image. By synthesizing the luminance channels of the two images, the luminance channel
L of the existing visible-light image is changed, resulting in an imbalance with the color channel
ab, making the color expression unnatural. Therefore, for the color channels (
of the visible-light image, saturation compensation using the ratio of the luminance channel (
) of the visible-light image to the luminance (
) after synthesis is required, and the saturation compensation function is calculated as follows:
where
is the chroma compensation ratio, and the chroma compensation is changed according to the change range of the luminance value through the change ratio between the visible-light image and the luminance synthesis result. At this time, 0.001 is added to the denominator to prevent divergence of the
compensation gain. In addition, the blending process is performed through the base layer synthesis map to prevent excessive saturation compensation at the boundary of the image.
is a constant value that controls the amount of color compensation, and in this study,
is used.
In Equation (21), and are color channels for which saturation is compensated, and and are color channels before saturation compensation of a visible-light image. and are multiplied by to perform saturation compensation for luminance modulation. At this time, to make the median of the chroma channel range 0, 128 is subtracted, values are multiplied by , and 128 is added again to obtain a chroma channel with a median of 128 that is uniformly distributed. Using the obtained chroma and and the synthesized luminance , the final synthesized image is acquired through a color-space conversion process.