Fog Density Evaluation by Combining Image Grayscale Entropy and Directional Entropy

Abstract

The fog density level, as one of the indicators of weather conditions, will affect the management decisions of transportation management agencies. This paper proposes an image-based method to estimate fog density levels to improve the accuracy and efficiency of analyzing fine meteorological conditions and validating fog density predictions. The method involves two types of image entropy: a two-dimensional directional entropy derived from four-direction Sobel operators, and a combined entropy that integrates the image directional entropy and grayscale entropy. For evaluating the performance of the proposed method, an image test set and an image training set are constructed; and each image is labeled as heavy fog, moderate fog, light fog, or fog-free according to the fog density level of the image based on a user study. Using our method, the average accuracy rates of image fog level estimation were 77.27% and 79.39% on the training set using the five-fold cross-validation and the test set, respectively. Our experimental results demonstrate the effectiveness of the proposed combined entropy for image-based fog density level estimation.

1. Introduction

Fog density forecasting and early warning is a crucial meteorological indicator that helps minimize fog’s impact on traffic safety [1]. The image-based fog density estimation method is becoming a low-cost and convenient means of fog density analysis due to the availability of video surveillance. Real-world images can provide a good indication of current visibility information, which makes it possible to infer the fog density level based on the image. With the increasing use of autonomous driving and intelligent monitoring, obtaining foggy image data under different weather conditions and measurement locations has become more convenient. These foggy images provide reliable experimental data for image-based fog density estimation research. In addition to providing fog forecasting and early warning, fog density estimation can also be used in image-defogging applications [2,3]. However, more research still needs to be done on classifying images based on fog density levels. An automatic and efficient method for evaluating fog density based on images would benefit various applications, including transportation, surveillance, and environmental monitoring.

We in this paper propose a new method by combining grayscale entropy and directional entropy to evaluate fog density levels and improve foggy image classification. Image entropy is a valuable tool for evaluating the degree of disorder and information content in images. At the same time, fog density can directly affect image color and shape information acquisition. By measuring changes in both color and shape influenced by fog, the new image entropy method helps improve the accuracy of foggy image classification and fog density evaluation.

2. Related Work

The literature related to this study can be broadly classified into two categories: image entropy and fog density evaluation.

2.1. Image Entropy

Entropy has a broad spectrum of applications in image processing. Conventional techniques compute image entropy using a solitary index, usually by extracting a specific attribute from the image and transforming pixel-level attributes into probability distributions. Probability distributions can be attained by computing a histogram of pixel intensity and normalizing it. The entropy is ultimately defined based on this probability distribution. Classical one-dimensional image entropy includes fuzzy entropy [4,5], Kapur entropy [6], cross entropy [7,8], and Shannon entropy [9]. In addition to being applied to image segmentation or classification, image entropy has also been applied to image filtering and denoising [10,11]. There are relatively few studies on two-dimensional or multidimensional entropy. Jena et al. [12] proposed a multi-threshold segmentation algorithm with three-dimensional Tsallis entropy as an objective function. Robin et al. [13] combined two-dimensional grayscale entropy with other features to achieve the binary classification of normal and cirrhotic liver ultrasound images.

Shape entropy measures the amount of information associated with an image’s shape. Compared to grayscale entropy, it describes an image’s texture and shape characteristics more efficiently. Typical shape features include image edges, line directions, boundary features, curvature functions, centroid distances, and geometric parameters such as roundness, eccentricity, and principal axis direction. Extracting shape features is a crucial step in studying shape entropy. Typical shape entropies include curvature entropy based on curvature function, edge entropy based on edge features, and directional entropy based on line directions. Shape entropy is widely used in image processing, including, but not limited to fingerprint and palm print recognition, leaf vein recognition, and wood texture recognition.

Using edge or contour information from images to define shape entropy is an effective method. Louis Oddo [14] proposed the concept of global and local shape entropy by introducing curvature as a measure of boundary information and calculating the first-order global shape entropy of boundary curvature, and used it for the automatic extraction of building outlines in aerial images. Briguglio et al. [15] calculated shape entropy based on the difference in length, middle axis length, and minor axis length of the particles; and used them to estimate the sedimentation velocity of the particles. Anders et al. [16] referred to the entropy effect generated by the geometric structure of shape in the particle model system as shape entropy, and proved that shape entropy drives the phase behavior of anisotropic shape systems through directional entropy force. Zhu [17] used the length and the number of breakpoints of the coastline to define shape entropy, which was then used to describe changes in the shoreline of the Yangtze River estuary. Lee et al. [18] defined the branch length similarity entropy (BLS entropy) to quantify the self-similarity of shape to solve shape problems in image retrieval. Hossein et al. [19] defined shape entropy based on the transverse slope of the cross-section of the riverbank, and used it to predict the shape trend of the riverbank profiles and the free water surface in the channel. Lu et al. [20] constructed the quadratic curvature entropy based on the Markov process, using it as macroscopic shape information of the curve profile of the target product to evaluate whether the product form conforms to consumers’ aesthetic preferences. Additionally, Sziová et al. [21] adopted structural Rényi entropy, based on the entropy definition, as one of the indexes to deal with the problem of insufficient data in colonoscopic polyp images.

Researchers have recently utilized different methods, such as particle system motion analysis and image contour line analysis, to design shape entropy for different research purposes. Despite these advancements, research on image shape entropy is still relatively limited. Therefore, further research and development are necessary to improve the accuracy and applicability of image shape entropy.

2.2. Fog Density Estimation

Fog density is traditionally measured by sensors, but this method is expensive, has limited coverage, and requires professional operators. In recent years, due to the rapid development of computer vision technology and the widespread application of monitoring equipment, research on the use of images for fog density estimation has received significant attention [3,22].

One commonly used approach is to utilize the dark channel of an image to determine fog density. Wan et al. [23] proposed an unsupervised learning method based on a five-dimensional feature vector consisting of edge strength, discrete cosine transform (DCT) ratio, dark channel (DC) ratio, average local contrast, and average standard deviation, as well as Gaussian mixture model. This method can classify fog images into three categories: fog-free, foggy, and dense fog. Li et al. [24] developed a foggy image classification algorithm that uses a linear SVM classifier and a hybrid feature composed of the dark channel, wavelet features, and mean normalization. Jiang et al. [25] employed a proxy-based method to learn a refined optical depth PRG model and selected features related to fog density, such as the dark channel, saturation value, and chrominance, for fog density estimation and image defogging. Ju et al. [26] defined a fog density index model to guide image defogging. This model utilizes the positive correlation between fog density’s minimum and range values, and the dark channel.

In addition, some literature employs image features, such as global contrast, color saturation, and image gradient, to estimate fog density. For example, in reference [27], foggy images were analyzed by maximizing three features relevant to fog density, including image saturation, brightness, and clarity, when performing foggy image classification and recognition. Lou et al. [28] defined fog density by brightness, saturation, and flatness; and established a linear model of transmittance and fog density based on these features. Literature [29] mainly focused on evaluating the defogging effect, subjectively dividing the fog image dataset into three levels: light fog, moderate fog, and heavy fog; and then comparing and analyzing the defogging results. The selection of features used depends on the specific application scenario and the image content that is prone to be affected by fog.

The two methods close to our research objectives are FADE (fog aware density evaluator) [2,30,31] and SFDE (a simple fog density evaluator) [32]. FADE extracts 12 dimensional features, such as MSCN coefficient variance, contrast, brightness fog perception statistical features; and uses a Mahalanobis-like distance measure between multivariate Gaussian fitting to predict the fog density. SFDE establishes a linear combination of three fog-related statistical features, namely saturation, Weber contrast of haze, and variance of chrome, for fog density estimation. Both methods mainly utilize color features, while our method utilizes color features and shape features.

In this paper, we propose a new shape entropy based on the two-dimensional angle characteristics of the edge points in the image. This shape entropy is combined with the gray entropy to estimate the fog density level in images. Referring to China’s national haze warning levels, yellow, orange, and red, which are three foggy cases, our image-based fog density classification results consist of four levels: fog-free, light fog, moderate fog, and heavy fog. The following sections will detail our methods, experiments, and conclusions.

3. Proposed Method

In this section, we propose a method for estimating fog density in images using a combined entropy approach. Our method first uses the Sobel operator in four directions to extract two-dimensional angular features from each edge point. We then calculate the binary probability and directional entropy, and construct a new type of combined entropy that integrates the image directional entropy and grayscale entropy to evaluate fog density and classify the image. The algorithm flow is as follows:

• Step 1. Convert the original image to grayscale and perform pseudo-edge detection;
• Step 2. Calculate two-dimensional grayscale entropy and directional entropy using the pseudo-edge image;
• Step 3. Define a piecewise function to construct the combined entropy based on the fog density discrimination capability of the two entropies;
• Step 4. Conduct experiments on both synthetic and real fog image datasets to evaluate the fog density level recognition performance of the combined entropy.

3.1. Two-Dimensional Grayscale Entropy

To comprehensively describe the spatial characteristics of the grayscale distribution of an image, an additional feature quantity is introduced based on the one-dimensional (1D) entropy of the image ($H_{gray1})$ to form a two-dimensional (2D) entropy. The image is converted into a grayscale image. If the gray value at pixel point $\left(i,j\right)$ is $x_1$, and the average gray value of its 8-neighborhood is $x_2$, the probability of the occurrence of the binary tuple$\left(x_1,x_2\right),$ represented as $p\left(x_1,x_2\right),$can be calculated. Based on this probability, the 2D entropy is calculated as:

$$H_{gray2}\left(I\right)=-\sum _{x_1=0}^{255}\sum _{x_2=0}^{255}p\left(x_1,x_2\right){\log }_{2}p\left(x_1,x_2\right),$$

(1)

The image entropy of a color image, denoted as $H_{RGB}$, can be directly defined as the sum of the grayscale entropy ($H_{gray1})$ of the image in the red, green, and blue color channels; and its calculation formula is as follows:

$$H_{RGB}\left(I\right)=-\sum _{C\in \left\{R,G,B\right\}}\sum _{x=0}^{255}{p}_C\left(x\right){\log }_2{p}_C\left(x\right),$$

(2)

Three foggy images listed in the top-left of Figure 1 are used to illustrate the values of images with different fog densities. The fog density of these images decreases gradually from left to right and exhibits significant visual variation.

The calculation results of three methods, $H_{gray1}$, $H_{gray2}$, and $H_{RGB}$, are shown on the right of Figure 1 as the line charts of three types of entropy for three foggy images with different fog densities. It is observed that $H_{gray2}$ has the widest range of $3.3091$ (i.e.,$12.0194-8.7103=3.3091$), while the other two entropies (i.e., $H_{gray1}$ and $H_{RGB}$) have significantly smaller ranges of $0.6168$ and $0.5085$, respectively. A larger range indicates a more distinctive contrast, which effectively distinguishes the fog density. Therefore, the two-dimensional grayscale entropy $H_{gray2}$ is integrated into constructing our proposed combined entropy.

3.2. Two-Dimensional Directional Entropy

The shape features in image content can be reflected to some extent by the image edges. There are many methods for calculating image edges; and we here use the Sobel operator in four directions, including 0° horizontal, 90° vertical, 45° diagonal, and 135° anti-diagonal, to calculate the gradients. The horizontal gradient $g_x$ and vertical gradient $g_y$ are calculated as follows:

$$\begin{array}{cc}\hfill g_x\left(i,j\right)=& I\left(i+1,j-1\right)+2\times I\left(i+1,j\right)+I\left(i+1,j+1\right)\hfill \\ & -I\left(i-1,j-1\right)-2\times I\left(i-1,j\right)-I\left(i-1,j+1\right)\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill g_y\left(i,j\right)=& I\left(i-1,j-1\right)+2\times I\left(i,j-1\right)+I\left(i+1,j-1\right)\hfill \\ & -I\left(i-1,j+1\right)-2\times I\left(i,j+1\right)-I\left(i+1,j+1\right)\hfill \end{array}$$

(4)

In Formulas (3) and (4), $I\left(i,j\right)$ is the gray value at pixel point $\left(i,j\right)$ in the edge image. The gradient magnitude $Ma{g}_1$and direction ${\theta }_1$of edge points$\left(i,j\right)$ are computed as:

$$Ma{g}_1\left(i,j\right)=\sqrt{g_x{\left(i,j\right)}^2+g_y{\left(i,j\right)}^2},$$

(5)

$${\theta }_1\left(i,j\right)=atan2 \frac{g_y\left(i,j\right)}{g_x\left(i,j\right)}\times \frac{180°}{\pi }+180°,$$

(6)

Similarly, using 45° and 135° directional templates, compute another gradient direction ${\theta }_2\left(i,j\right)$. With the two direction values, we generate a two-dimensional random variable $\left({\theta }_1\left(i,j\right),{\theta }_2\left(i,j\right)\right)$, which represents the directional feature of the edge pixel$\left(i,j\right)$ in the image.

Then, the marginal directional angle values of${\theta }_1$ and ${\theta }_2$ are discretized by dividing interval $\left[0^o, {360}^o\right)$ into n subintervals of equal size, which are$\left[360°\left(l-1\right)/n, 360°\times l/n\right),l=1,2,\cdots ,n$. These subintervals are labeled in ascending order as 1 to$n$. Generally, a larger $n$ results in a greater difference in entropy values, providing better differentiation effects. However, higher n-values also increase the computational complexity since the number of binary tuples that require counting rises, leading to a surge in events within the probability space.

Finally, each angle value of ${\theta }_1$and ${\theta }_2$ is assigned a corresponding quantization value between 1 and n.

$${\mathsf{\Theta }}_k\left(i,j\right)=l,\hspace{1em}{\theta }_k\in \left[\frac{360°}{n}\left(l-1\right), \frac{360°}{n}\times l\right), l=1,2,\cdots ,n,$$

(7)

In this way, each pixel is mapped to a certain angle group, forming a new two-dimensional discrete random variable $\left({\mathsf{\Theta }}_1\left(i,j\right),{\mathsf{\Theta }}_2\left(i,j\right)\right)$. The edge points on each direction group are counted, and the resulting edge direction histogram is used as the image’s shape feature. After normalization, a probability distribution is obtained for each direction interval, and this distribution is used to calculate the two-dimensional directional entropy $H_{sobel2}\left(I\right)$.

$$H_{sobel2}\left(I\right)=-\sum _{{\mathsf{\Theta }}_1=1}^n\sum _{{\mathsf{\Theta }}_2=1}^{n}p\left({\mathsf{\Theta }}_1,{\mathsf{\Theta }}_2\right){log}_{2}p\left({\mathsf{\Theta }}_1,{\mathsf{\Theta }}_2\right).$$

(8)

In the Formula (8), $p\left({\mathsf{\Theta }}_1,{\mathsf{\Theta }}_2\right)=f\left({\mathsf{\Theta }}_1,{\mathsf{\Theta }}_2\right)/N_{edge}$, which represents the probability of occurrence of the two-dimensional discrete variable, where$f\left({\mathsf{\Theta }}_1,{\mathsf{\Theta }}_2\right)$ is the statistical frequency of$\left({\mathsf{\Theta }}_1,{\mathsf{\Theta }}_2\right)$for all edge points in the image I, and $N_{edge}$is the number of edge points.

To simplify the notation, we denote the probability $p\left({\mathsf{\Theta }}_1,{\mathsf{\Theta }}_2\right)$ as $p_m$, where $m=\left({\mathsf{\Theta }}_1-1\right)n+{\mathsf{\Theta }}_2$, and m varies between 1 and $n^2$. The probability vector is represented as$\overrightarrow{P}=\left(p_1,p_2,\cdots p_{n^2}\right)$. We can rewrite Formula (8) as follows:

$$H_{sobel2}\left(I\right)=-\sum _{m=1}^{n^2}{p}_{m}log{p}_m=H_{sobel2}\left(\overrightarrow{P}\right),$$

(9)

where $H_{sobel2}\left(\overrightarrow{P}\right)$ is the entropy function of the probability vector$\overrightarrow{P}$, another expression of $H_{sobel2}\left(I\right)$. It has been proven that the constructed 2D directional entropy $H_{sobel2}\left(I\right)$ satisfies the four properties of information entropy: non-negativity, symmetry, extremum, and additivity.

If only the horizontal and vertical gradients are preserved, Equation (8) degenerates into a 1D directional entropy formula, that is

$$H_{EDH}\left(I\right)=-\sum _{i=1}^{n}p\left({\mathsf{\Theta }}_i\right)log\left(p\left({\mathsf{\Theta }}_i\right)\right)$$

(10)

So, the 2D directional entropy $H_{sobel2}\left(I\right)$ is an extension of 1D directional entropy.

We use three images in the top-left of Figure 1 again to illustrate the calculation results of $H_{sobel2}$ and $H_{EDH}$. At first, three pseudo-edge images are generated using the Canny operator and shown in the third row of Figure 1. Let $n=72$. After calculating with Formulas (9) and (10), two line charts are shown in the right sub-figure of Figure 1, which uncovers the relationship between 1D direction entropy ($H_{EDH}$) and 2D direction entropy ($H_{sobel2}$) with respect to the fog density. It could be found that both methods effectively highlight the differences in fog density, with the range of $H_{sobel2}$ being 2.6177, and$H_{EDH}$ being 0.9127.

Obviously, the range of $H_{sobel2}$is significantly higher than$H_{EDH}$, indicating that the proposed $H_{sobel2}$can better distinguish the fog density. Thus, it can be used as an appropriate index to distinguish fog density within images. However, the value of $H_{sobel2},$like grayscale entropy, is not solely affected by fog density, but also by the scene within the image.

3.3. The Combined Entropy

Figure 1 shows that $H_{gray2 }$ in red line changes less at high fog density, but significantly at low fog density, while the $H_{sobel2}$ in the purple line has the opposite performance. To utilize the advantages of both 2D grayscale entropy and 2D directional entropy in fog density estimation, we construct a piecewise function:

$$H_{com}=\left\{\begin{array}{c}{H}_{gray2 }, H_{gray2 }>\delta \\ H_{sobel2}, H_{gray2 }\le \delta \end{array}\right.$$

(11)

In Formula (11), the threshold $\delta$ in the function is an experimental value. In subsequent experiments, we provide a method to determine its value.

3.4. Algorithm Evaluation Indexes

To assess the effectiveness of the classification model, we create a confusion matrix that compares the preset fog density labels (true labels) with the model’s predicted labels. Using this matrix, four metrics, including precision, recall, f1 value, and accuracy, are calculated to objectively evaluate the effectiveness of the fog density level classification model.

If we divide fog density into K levels, each level is given a corresponding label between 1 and K. The resulting confusion matrix is represented as:

$$CM={\left(m_{ij}\right)}_{K\times K}$$

(12)

In Formula (12), $m_{ij}$ represents the number of samples with preset label i and algorithmic estimated label j, where $i,j=1,2,\cdots ,K$. The precision, recall, and score $f1$ of each category are:

$$Precision\left(j\right)=\frac{m_{jj}}{{\sum }_{i=1}^K{m}_{ij}}$$

(13)

$$Recall\left(i\right)=\frac{m_{ii}}{{\sum }_{j=1}^K{m}_{ij}}$$

(14)

$$f1\left(i\right)=\frac{2\times precision\left(i\right)\times recall\left(i\right)}{precision\left(i\right)+recall\left(i\right)}$$

(15)

The overall index of the classification model is calculated by taking the weighted average of all the indexes.

Furthermore, accuracy, the overall evaluating index of classification tasks, can be calculated using Formula (16), as

$$Accuracy=\frac{{\sum }_{i=1}^K{m}_{ii}}{{\sum }_{i=1}^K{\sum }_{j=1}^K{m}_{ij}}$$

(16)

4. Experiments and Results

It is necessary to verify the effectiveness of the proposed method for foggy images under different scenes. The experimental process for this section is illustrated in Figure 2.

4.1. Datasets and Preprocessing

Three datasets, as the following three subsections, are used to verify our method and to analyze the fog density levels.

4.1.1. Color Hazy Image Database

The Color Hazy Image Database (CHIC, http://chic.u-bourgogne.fr/chicpage.php, accessed on 16 August 2022) is opened by El Khoury et al. [33,34].

This dataset comprises CHIC_Static_scenes and CHIC_Dynamic_scenes. In CHIC_Static_scenes, there are two indoor scenes, named Scene 1 and Scene 2, in a controlled environment. Each scene consists of 10 images with different fog densities, from heavy fog (Level 1) to haze-free (Level 10). Ten images consisting of Scene 2 are displayed in Figure 3. We use the resized images from Scene 1 and Scene 2, and the size of each image is $1800\times 1200.$

4.1.2. Haze Groups Training Set

Our experimental dataset, Haze groups, is divided into training and test sets. The training set includes 194 images that are randomly selected from three publicly available datasets, O-HAZE, Dense_Haze, and D-HAZY_DATASET, as well as an image website, Ooopic (https://www.ooopic.com, accessed on 29 September 2022).

O-HAZE (https://data.vision.ee.ethz.ch/cvl/ntire18//o-haze/, accessed on 23 September 2022) is initially released by [35] and employed in the dehazing challenge of the NTIRE 2018 CVPR workshop, which contains 45 different outdoor scenes composed of pairs of real hazy and corresponding haze-free images. The fog with foggy images is relatively light.

Dense_Haze [36,37] contains 33 pairs of real hazy and corresponding haze-free images of various outdoor scenes, which are characterized by dense and homogeneous hazy scenes (https://data.vision.ee.ethz.ch/cvl/ntire19//dense-haze/, accessed on 25 September 2022).

HAZY, initially released by [38], contains more than 1400 pairs of images with ground truth reference images and synthetic hazy images of the same scene (https://dial.uclouvain.be/pr/boreal/object/boreal:175854/datastream/, accessed on 23 September 2022).

4.1.3. Haze Groups Test Set

The test set includes 131 images with different fog density levels. Some of those images are also randomly selected from O-HAZE, Dense_Haze, D-HAZY_DATASET, and Ooopic website. Other images are from another publicly available dataset, Foggy Driving (https://people.ee.ethz.ch/~csakarid/SFSU_synthetic/, accessed on 21 June 2023), which is released by [39,40], mainly composed of foggy images of traffic roads. The test set does not intersect with the training set.

4.1.4. Preprocessing

Images in the test and training sets are classified into four groups: heavy fog, moderate fog, light fog, and fog-free, with a questionnaire. According to the user study results, the number of images in each class is listed in

Table 1. The number of images in each class.

Set	Heavy Fog	Moderate Fog	Light Fog	Fog-Free	Total N
Training set	55	51	36	52	194
Test set	38	34	24	35	131

. In this table, “Total N” is the number of images in the corresponding image set.

We then applied Formula (17) to assign appropriate labels for each image.

$$Label=\left\{\begin{array}{c}1,\hspace{1em}heavy fog\hfill \\ 2,\hspace{1em}moderate fog\hfill \\ 3,\hspace{1em}light fog\hfill \\ 4,\hspace{1em}fog-free\hfill \end{array}\right.$$

(17)

For the sake of computational efficiency in engineering applications, we downsized all images in the dataset by reducing them to 50% of their original size while preserving their initial aspect ratios. Figure 4 shows some examples of images from different fog density groups.

4.2. Experimental Results

4.2.1. The Threshold of Combinatorial Entropy

To find the threshold $\delta$ of combinatorial entropy (Formula (11)), images in Scene 2 in the CHIC dataset, as shown in Figure 3, are used to calculate their 2D grayscale entropy (Formula (1)), and 2D directional entropy (Formula (9)). The calculated results are shown as two line charts in Figure 5.

From Figure 5, it can be concluded that both $H_{gray2 }$and $H_{sobel2 }$are effective in distinguishing the ten different levels of fog density. From the first image to the fifth one, the difference in directional entropy ($H_{sobel2 }$) is more significant, making it easier to distinguish fog density levels. Conversely, grayscale entropy ($H_{gray2})$ shows a relatively flat change and is not as effective in distinguishing fog density levels as the directional entropy. On the other hand, from the fifth image to the tenth one, the slope of the grayscale entropy curve becomes a lot bigger, highlighting a more apparent change in grayscale entropy. The shape entropy, however, remains relatively stable, showing minimal changes. These observations suggest that directional entropy is more effective in distinguishing higher fog density levels, while grayscale entropy is more effective in situations with lower fog density.

The threshold $\delta$ in the function (11) can be determined based on the experimental results (Figure 5) obtained from Scene 2. Figure 5 shows that the grayscale entropy of images at different fog density levels varied significantly, while the difference in directional entropy was relatively small. Hence, it is more reasonable to use the grayscale entropy value as a threshold. In this paper, the threshold $\delta$ was determined as 7.3 based on the mean grayscale entropy values of the fourth ($H_{gray2}=6.8142)$ and fifth images $(H_{gray2 }=7.8099$5).

By setting $\delta =7.3$ and using Formula (11), the experiment results on two data sets, Scene 1 and Scene 2, are displayed as line charts in Figure 6. The curves in the figure display the relationship between the 2D grayscale entropy $H_{gray2 }$, the 2D directional entropy $H_{sobel2}$, and the combined entropy $H_{com}$ concerning fog density. Two sub-figures clearly demonstrate that the proposed combined entropy $H_{com}$ is more effective than the other entropies in distinguishing the ten levels of fog density in the view of the strictly monotone increasing and maximum level difference.

4.2.2. Training and Analysis

For classifying foggy images into four categories, heavy fog, moderate fog, light fog, and fog-free, using combination entropy, the training set of “Haze groups” is used to determine inter-class segmentation parameters.

First, the 2D grayscale entropy, 2D directional entropy, and combined entropy for all images in the training set are calculated. Next, all images were classified into four categories with respect to their fog density levels, which were estimated using the combined entropy value (Figure 7).

Figure 7 indicates that the curve of the combined entropy displays an overall increasing trend from thick fog to fog-free. The thick fog image exhibits a relatively low entropy value, with directional entropy playing a significant role. On the other hand, for images with lower fog density, grayscale entropy becomes the dominant contributing factor. This observation aligns with the findings in Section 3.3, where the fog-free image shows the highest entropy value. However, distinguishing between light and moderate fog data is challenging based on the graph. Next, we will classify the images based on their density levels, evaluated by their combined entropy values.

We first conduct a statistical analysis of the obtained entropy data. According to the pre-set labels (Section 4.1.4), the data are grouped and tested for normality. The test results showed that the p-value of the fog group is 0.0045, which is less than 0.05, indicating that the data in this group does not conform to the normal distribution. Therefore, the nonparametric Kruskal-Wallis (K-W) method [41] is selected for significance difference testing, with a significant level set to 0.05.

The null hypothesis $H_{0 }$is that there is no significant difference between these four sets of data, and the alternative hypothesis$H_1$ is that that these four sets of data are significantly different.

The K-W test results show that the p-value is $6.3186\times {10}^{-32}$, less than the significance level of 0.05, which indicates a significant difference in the entropy results of the four groups.

Based on the results shown in Figure 7 and Figure 8, it can be observed that the third group exhibits outliers, and the distance between the upper or lower quartile values and the median is significant for both the first and third groups. These factors can potentially affect the normal distribution fitting effect. To eliminate the impact of outliers, we employ the 1sigma criterion (as the 3sigma criterion is not applicable in this experiment). Consequently, we obtained a final dataset that includes 37 images with heavy fog, 34 with moderate fog, 28 with light fog, and 33 without fog.

For the entropy data of the processed image group, we perform a new normality test. The results reveal that the p-values for the four groups of data are 0.2905, 0.3960, 0.2405, and 0.2521, all greater than 0.05; so, it can be considered that they all conform to normal distribution. Next, we conduct a homogeneity test for variance, and the result shows a p-value of 0.0002, which does not meet the homogeneity of variance condition. However, since the sample size of each dataset is small, the violation of the homogeneity of variance condition is temporarily overlooked. We will subsequently proceed with a one-factor ANOVA analysis.

Null hypothesis$H_{0 }$: it is believed that the mean values of all four dataset are the same; alternative hypothesis$H_1$: it is believed that the mean values of the four datasets are different, and there are significant differences.

The p-value of the test result is $6.5322\times {10}^{-56}$, which is less than 0.05. Thus, the null hypothesis is rejected, and it is concluded that the mean values of the four sets of entropy values are significantly different. The boxplot shown in Figure 9 confirms the significant difference in the combined entropy of images with different fog density levels, which supports using this entropy value for fog density level evaluation.

In order to determine the fog density level based on the combined entropy, the first step is to determine the division thresholds (${\delta }_1$, ${\delta }_2$, ${\delta }_3$, and${\delta }_4$) for the four categories using the training data. To achieve this, we perform a normal distribution fitting for each group of entropy data and present the fitting curves in Figure 10.

The threshold value for any two adjacent categories is determined by calculating the X value of their intersection within the fitted curve and the labeled adjacent curve. By utilizing these thresholds, it becomes feasible to construct a fog density level estimation model with the following structure:

$$Labe{l}_{predict}=\left\{\begin{array}{c}1,\hspace{1em}{H}_{com}\le {\delta }_1\hfill \\ 2,\hspace{1em}{\delta }_1<H_{com}\le {\delta }_2\hfill \\ 3,\hspace{1em}{\delta }_2<H_{com}\le {\delta }_3\hfill \\ 4,\hspace{1em}{H}_{com}>{\delta }_3\hfill \end{array}\right.$$

(18)

Utilize the five-fold cross-validation approach to train and fine-tune the model, enhancing its capacity for generalization.

Table 2. Experimental results on the training set using five-fold cross-validation.

Index	${\mathit{\delta }}_1$	${\mathit{\delta }}_2$	${\mathit{\delta }}_3$	Training Accuracy	Testing Accuracy
1	9.106	9.952	11.2	0.8000	0.7692
2	9.02	9.777	11.21	0.7736	0.6154
3	9.036	9.913	11.31	0.7642	0.8077
4	9.06	9.829	11.3	0.7238	0.8519
5	9.178	9.998	11.26	0.8396	0.6538

shows the three thresholds obtained from five experiments along with the classification accuracy on both the training and testing sets using those thresholds. By averaging the thresholds obtained from the five experiments, the final thresholds are: ${\delta }_1=$9.08, ${\delta }_2=$9.8938, and ${\delta }_3=$11.256. The fog density evaluation model with determined thresholds is:

$$Labe{l}_{predict}=\left\{\begin{array}{c}1,\hspace{1em}{H}_{com}\le 9.08\hfill \\ 2,\hspace{1em}9.08<H_{com}\le 9.8938\hfill \\ 3,\hspace{1em}9.8938<H_{com}\le 11.256\hfill \\ 4,\hspace{1em}{H}_{com}>11.256\hfill \end{array}\right.$$

(19)

Based on the calculation results of the above model, all images in the dataset have been assigned new fog density levels, as depicted in Figure 11. The red dots indicate the preset fog density labels, while the yellow dots show the predicted labels. It is apparent from Figure 11 that the training performance is optimal for the set of clear images since they typically exhibit bright colors, clarity, brightness, and prominent edge details, leading to higher entropy values. Following this, the group of images with heavy fog has a better outcome because the images possess diminished edge information and low entropy values. On the other hand, the categorization effect for the groups containing light and medium fog images is less effective due to imprecise preset labels and difficulties in defining them. Additionally, the scene content of the images is obscured by thin fog, making it challenging to extract clear edge information.

4.3. Evaluation

To quantitatively evaluate the accuracy of our proposed method, we built the confusion matrix and compared our results to those estimated by FADE [2,30,31] and SFDE [32]. The three confusion matrices are displayed in Figure 12. The precision, recall, f1 value for each category, and classification accuracy are described in

Table 3. Accuracy comparison of fog density estimated on the training set by three methods.

	SFDE			FADE			Our Method
Label	Precision	Recall	f1	Precision	Recall	f1	Precision	Recall	f1
1	0.5854	0.4364	0.5000	0.7556	0.6182	0.6800	0.8250	0.8919	0.8571
2	0.4412	0.3000	0.3571	0.3968	0.5000	0.4425	0.5758	0.5588	0.5672
3	0.2540	0.4324	0.3200	0.2917	0.3784	0.3294	0.6800	0.6071	0.6415
4	0.6429	0.6923	0.6667	0.8684	0.6346	0.7333	0.9706	1.0000	0.9851
weighted avg	0.5004	0.4091	0.4735	0.6049	0.5464	0.5662	0.7764	0.7727	0.7787
Accuracy	0.4691			0.5464			0.7727

. In this table, “weighted-avg” is the weighted average.

From Table 3, it can be seen that among the three methods, our method attains the best performance according to all four indexes, precision, recall, f1, and overall accuracy; compared with the other two methods, SFDE and FADE, our method has improved overall accuracy by 64.7% and 41.4%, respectively. In addition, compared to the first category (heavy fog) and fourth category (fog-free), the three methods have relatively low recognition accuracy for the second and third categories, which may be a key point for constructing methods with higher accuracy in the future.

4.4. Experimental Results on Test Set

In this sub-section, the proposed method is validated on the test set and compared with FADE and SFDE. Three confusion matrices are built, similar to what has been conducted in Section 4.3. Then four metrics, precision, recall, f1 value, and accuracy, are calculated to objectively evaluate the three models’ effectiveness objectively.

The visual comparison of classification results can be seen in Figure 13. The classification performance of our model on the test is similar to that on the training set, with better performance in the heavy-fog and fog-free groups, and worse performance in the light-fog and moderate-fog groups. It is noteworthy that there is an increase in controversial images between the light-fog and fog-free groups because the images labeled as ‘3’ in fog-free groups mostly have monotonous colors, while some mist images are labeled as ‘4’, which are colorful and with complex scenes.

Then, the confusion matrices of the three methods assessed on the test set are displayed in Figure 14. Moreover, the precision, recall, f1 value for each category, and classification accuracy are described in

Table 4. Accuracy comparison of fog density estimated on the test set by three methods.

	SFDE			FADE			Our Method
Label	Precision	Recall	f1	Precision	Recall	f1	Precision	Recall	f1
1	0.6415	0.8947	0.7473	0.6271	0.9737	0.7629	0.8571	0.9474	0.9000
2	0.5652	0.3824	0.4561	0.5714	0.3529	0.4364	0.8077	0.6176	0.7000
3	0.4074	0.4583	0.4314	0.6000	0.5000	0.5455	0.5862	0.7083	0.6415
4	0.9286	0.7426	0.8254	0.9355	0.8286	0.8788	0.8824	0.8571	0.8696
weighted avg	0.6555	0.6412	0.6347	0.6901	0.6870	0.6693	0.8014	0.7939	0.7926
Accuracy	0.6412			0.6870			0.7939

.

As shown in Table 4, the proposed method obtains the best performance among the three methods, according to all four indexes, precision, recall, f1, and overall accuracy; compared with the other two methods, SFDE and FADE, our method has improved overall accuracy by 23.8% and 15.6%, respectively. Based on the classification results of the training and the test sets, our method performs the best among the three methods compared.

4.5. Discussion

When assessing the level of fog density in images, 2D grayscale entropy provides better results than 1D grayscale entropy. Additionally, 2D directional entropy based on the Sobel operator in four directions can better distinguish the level of fog density than 1D directional entropy based on the edge direction histogram. The results of 2D grayscale and 2D directional entropies on the same data set show they have different application ranges. As a result, a new quantitative indicator, combined entropy, is proposed.

Although the CHIC dataset’s combined entropy efficiently distinguishes ten fog density levels in both scenarios, the piecewise function thresholds need experimental derivation using diverse image sets with varying density levels within the same scene. We trained the threshold on scene 2 of the CHIC dataset, which yielded satisfactory outcomes when validated on scene 1. Nevertheless, this threshold has some restrictions due to the limited datasets of the same type. A more comprehensive dataset of the same type could lead to more precise thresholds.

Furthermore, the pre-labeling of fog density is based on the subjective visual perception of an individual, which can be influenced by personal observation and emotions. It is challenging to ensure the accuracy of the pre-labeling, potentially impacting the calculation of the final classification accuracy.

In summary, the proposed method has several limitations. On the one hand, the level of image fog density is labeled through the user study, which is a subjective way. A better approach is to build the relationship between the image entropy value and the measurable property of fog. The property can be measured through fixed-point scene sensing equipment and visibility measurements. On the other hand, the threshold δ in combined entropy is an experimental value. Although its value obtained in the experiment in this article showed a suitable partition, it still needs more experiments on a large number of datasets to test. Similarly, the thresholds of our classification model are also data-driven, and the generalizability of the proposed method still needs further verification.

5. Conclusions and Future Work

This study proposes a formula for calculating a two-dimensional directional entropy based on the four-direction Sobel gradient algorithm. Additionally, a combined entropy is constructed by merging the 2D grayscale entropy and 2D directional entropy to distinguish between various levels of fog density. This method was used to effectively differentiate between ten levels of fog density in images from the CHIC dataset. In the multi-scenario Haze-groups dataset, the study employed normal fitting to obtain related thresholds and utilized five-fold cross-validation for training the model. After obtaining the optimal threshold, the fog density was classified into four levels: heavy fog, medium fog, light fog, and fog-free. The four indicators calculated using the confusion matrix demonstrated that the classification accuracy rate of the combined entropy was larger than 77.2%, indicating effective differentiation of fog density levels.

There are only a few research results on image-based estimation of fog density, and there are still many related topics worth further research. For example, deep-learning methods have been widely applied in various fields recently, especially in image classification and restoration [42]. A natural extension is the image classification based on the image fog density level. Therefore, using deep-learning methods to estimate fog density after collecting labeled data [43] should be a promising research direction in the future. In addition to the directional entropy and combined entropy proposed in this paper, we can also extract other fog density-related features, such as brightness, saturation, dark channel, to form a multi-dimensional feature group, input the convolutional neural network encoder for better representation, and then use the classifier for training to judge the image fog concentration level. Selecting features, determining feature dimensions, and balancing classification accuracy and computational efficiency are all critical issues worth studying.