Fire Segmentation with an Optimized Weighted Image Fusion Method

Abstract

In recent decades, earlier fire detection has become a research priority. Since visible and infrared images cannot produce clear and complete information, we propose in this work to combine two images with an appropriate fusion technique to improve the quality of fire detection, segmentation, and localization. The visible image is at first weighted before being used in the fusion process. The value of the optimal weight is estimated from the mean of the visible image with a second-order polynomial model. The parameters of this model are optimized with the least squares method from the curve of optimal weights according to the mean of visible images. Finally, a major voting method based on deep learning models is used. Experiments include an assessment of the framework’s performance not only with respect to its visual appearance but also across a spectrum of predefined evaluation criteria. The experiments show that the proposed model, which includes an optimized weighted image fusion stage before segmentation, has a high Intersection over Union (IoU) score of more than 94%.

1. Introduction

One of the most catastrophic natural disasters that has occurred over the past few years is forest fires. The burning of forests not only contributes to air pollution but also puts people’s lives and the lives of wild animals in jeopardy [1]. Each year, a staggering 60,000 to 80,000 forest fires have ravaged the landscape, resulting in damage spanning from 3 to 10 million hectares. The environmental impact of these fires is highly contingent on their scale and frequency, but what remains consistent is the diverse array of causes behind these destructive blazes. In the past, forest fires were predominantly a natural occurrence, primarily ignited by uncommon events like volcanic eruptions or earthquakes, and typically confined to highly specific geographic regions [2]. Today, natural causes are rarer than human activity, whether voluntarily or not. Human imprudence causes 43% of forest fires (garbage deposits, burning cigarette butts). Surges, electricity line damage, or military incidents can also produce them, as in 2016 and 2017 at Captieux military base in Gironde (1300 hectares of pines destroyed by military fire) [3]. Drawing from comprehensive research conducted by the National Interagency Fire Center (NIFC) in the United States of America, a concerning trend emerges. Over the decade encompassing 2010 to 2019, an alarming annual average of 51,296 fires raged across the nation’s landscapes. These data underscore the urgent need for proactive measures and strategies to mitigate the escalating impact of forest fires on both the natural environment and the economy [4]. As a result of the destructive impacts of fire, various fire detection methods have developed today. In earlier times, the detection of forest fires relied on the watchful eyes of human watchers or the strategic placement of watchtowers [5]. However, these methods included several high-risk elements as well as a significant amount of physical work. Low temporal and spatial resolution are an issue with satellite-based remote sensing technologies [6,7]. These worrying numbers encourage academics to seek innovative strategies for the early detection and control of fires.

Video surveillance has become the primary method for capturing images of forest fires. Forest fire detection methods have evolved into three primary categories: ground-based, aviation-based, and satellite-based detection. Recent advancements in technology have significantly improved both hardware and software. Deep learning techniques, including Convolutional Neural Networks (CNNs) and hybrid models [8,9,10], are now widely used for precise segmentation, often enhanced by transfer learning [11,12]. Multi-spectral and hyperspectral imaging has increased accuracy in challenging conditions [13], while Generative Adversarial Networks (GANs) create robust training datasets through data augmentation and synthesis [14]. Sensor fusion, which integrates data from optical and thermal cameras, enhances detection capabilities [15]. A notable trend is the fusion of visible and infrared images as a pre-segmentation step, combining information from both spectra to improve the accuracy and robustness of flame detection. Infrared images are particularly useful in low-visibility conditions like smoke or night-time, while visible images provide detailed spatial information. This combined approach allows segmentation algorithms to more accurately identify flames in diverse conditions [16]. The choice between visible and infrared systems depends on the specific requirements of the fire detection task and environmental conditions. Visible systems, which capture wavelengths perceivable by the human eye, are effective during daylight hours and for detecting open flames. In contrast, infrared systems detect heat emissions and variations, excelling in low-light conditions, during the night, or when smoke obscures visibility. Infrared cameras can identify fires by their heat signatures even before visible flames emerge, enhancing their effectiveness in swiftly identifying potential fire outbreaks [17,18]. In some situations, a combination of both systems, known as multi-spectral or fused imaging, may be employed to harness the strengths of each, providing a comprehensive and robust fire detection solution. This adaptability underscores the dynamic nature of vision-based fire detection and its capacity to meet the diverse challenges posed by different scenarios, lighting conditions, and environmental factors [19]. An advanced and highly effective technique in the field of fire detection involves the fusion of infrared and visible images, representing a method that goes beyond mere image combination. This sophisticated approach leverages the unique strengths of both infrared and visible imaging to create a synergistic process that yields images of exceptional robustness and informativeness. Image fusion has several uses in pattern recognition, remote sensing, medical image processing, and current military technology. The capacity of the human visual system to detect and identify targets can be considerably enhanced by the fusion of visible and infrared images. Visible images have rich appearance information; however, infrared images lack texture and detail. Infrared images reflect the heat radiation produced by objects and are less influenced by illumination fluctuations or artifacts, allowing for night-time target recognition. Infrared images have poorer spatial resolution than visible ones. As a result, the fusion of thermal radiation data with texture detail information within a single image yields a powerful capability for automated target detection and precise target localization. In the contemporary landscape of image fusion techniques that combine visible and infrared imagery, there are four broad categories that provide diverse approaches to this process. These categories encompass sparse representation, multi-scale transformation, saliency-based methods, and subspace methods. Each of these approaches contributes uniquely to the field, offering a range of tools and methodologies to enhance the accuracy and efficacy of image fusion across various applications, including the vital domain of fire detection and surveillance [20,21,22]. In the latent low-rank representation (LatLRR) technique, decomposing source images at multiple scales enhances resistance to noise and anomalies. The GFCE approach allows for the incorporation of the most pertinent IR spectral characteristics into the visible image.

The LatLRR is a powerful method for image fusion because it can handle multiple sources of data, different types of data, noise, blur, and missing data, and it has been found to have better performance in image fusion tasks when compared to traditional methods. Traditional methods such as IHS and PCA [23,24] may not be suitable for different types of images and may not be able to effectively fuse images with different characteristics and do not fully exploit the spectral information in images, resulting in a lack of data in the fused image that may not be adaptable to changing conditions, such as changes in lighting or image resolution. Image segmentation has many applications in various fields, including medical imaging and computer vision [25,26,27]. In the second stage, the goal of fire image segmentation is to accurately identify and locate fire in images and videos, which can be used for applications such as firefighting and wildfire management. A fire detection model has been developed to improve our ability to identify fire zones. This model achieves this by fusing visible and infrared images using the innovative LatLRR-based approach. Through this fusion process, it strategically combines the unique strengths of these two image types, effectively using thermal data and visual information to significantly improve fire detection accuracy. Following the fusion step, the resulting images undergo a segmentation phase using a major voting method based on deep learning models, specifically utilizing the UNet-ResNet50, UNet-VGG16, and UNet-EfficientNet models. Furthermore, since visible images vary in brightness, depending on the acquisition period, we propose to weigh the visible image in grayscale before using it in the fusion process with the infrared image, with an optimal weight, which is variable according to the brightness mean of the image. The visible image is at first weighted before being used in the fusion process. The value of the optimal weight is estimated from the mean of the visible image with a second-order polynomial model. The parameters of this model are optimized with the least-squares method from the curve of optimal weights according to the mean of visible images. Finally, the segmentation procedure makes it possible to further refine the detection process by precisely delimiting the boundaries of the fire zones. The combination of these techniques represents a significant advancement in fire detection technology, where the fusion of multiple imaging modalities and the application of advanced segmentation models collaboratively contribute to increased accuracy and efficiency in the early fire detection and mitigation of potential fires.

This paper is structured as follows: in Section 2, we introduce our proposed framework for fire segmentation; then, in Section 3 we give a detailed description of our proposed approach to improve the latLRR. In Section 4, we show how to use the major voting mechanism to refine the segmentation process. The paper concludes with Section 5.

2. Overview of the Proposed Segmentation Framework

2.1. Introduction

In this section, we begin by introducing the two methods chosen for our study. First, we explore the LatLRR technique developed by Hui et al. [28], which leverages multi-scale decomposition to enhance robustness against noise and anomalies in source images. The second approach is the infrared and visible image fusion method (GFCE) [29]. This technique adeptly integrates crucial IR spectral features into the visible spectrum, ensuring that essential perceptual cues from the background scenery and visible image details are either retained or suitably enhanced. Following this, we detail the evaluation metrics used in our assessment. We conclude with an overview of the Corsican fire database, as documented by Toulouse et al. [30], is necessary and Figure 1 provides an outline of our proposed fire semantic segmentation structure.

2.2. Presentation of the Used Image Fusion Methods

2.2.1. GFCE Fusion-Based Method

Zhiqiang Zhou and his team [29] introduced a unique algorithm for enhancing the night-vision context by fusing infrared and visible images through a guided filter (see Figure 2). The process begins with the development of an adaptive enhancement technique. Following that, a hybrid multi-scale decomposition method, rooted in the guided filter, is utilized to integrate infrared data into the visible image using a multi-scale fusion strategy. In the concluding step, a perceptual-based regularization parameter is employed to ascertain the proportion of infrared spectral features to be incorporated, which is achieved by assessing the perceptual saliency of both infrared and visible image data. One advantage of a guided filter-based context enhancement method is that it can effectively preserve fine details and edges in an image while also enhancing the overall contrast and brightness.

This can be useful for improving the visibility and interpretability of an image, particularly in situations where the original image is of low quality or has poor lighting conditions. Additionally, it is simple to implement and computationally efficient. One disadvantage of a guided filter-based context enhancement method is that it can introduce artifacts or distortions in the enhanced image, particularly in areas with a high texture or fine details. Additionally, it may not perform as well on images with low contrast or under-exposed areas.

2.2.2. LatLRR Fusion-Based Method

In their work, Hui et al. [28] introduced a straightforward yet robust image fusion technique based on LatLRR (see Figure 3). Initially, input images undergo decomposition into low-rank segments, denoted as $I_{C\_lrr}$ (representing global structure), and saliency segments, represented as $I_{C\_s}$ (indicative of local structure) using LatLRR, with $C\in \left(\mathrm{1,2}\right)$. The researchers utilized a weighted-average fusion approach and a sum strategy for the amalgamation of low-rank and saliency segments. This results in the formation of the fused low-rank segment, $F_{lrr,}$ and the fused saliency segment, $F_s$. The ultimate fused image is the summation of $F_{lrr}$ and $F_s$.

LatLRR has several advantages. One advantage is that LatLRR can effectively capture the underlying low-dimensional structure of the data. This allows for the fusion of infrared and visible images to be performed in a compact and efficient manner without the need for excessive computational resources or large amounts of training data. Additionally, LatLRR can be used to perform feature extraction and dimensionality reduction, which can be useful for image classification and detection tasks. Another advantage of LLRR is that it can preserve the spectral and spatial information of the original images, which can be useful for applications such as target recognition and scene analysis. One disadvantage of fusion using LatLRR is that it can be sensitive to the initial estimate of the low-rank matrix and the sparse matrix, which can affect the final fused image. Another disadvantage of LatLRR is that it requires many images to produce a good result, and it may not be suitable for small datasets. Finally, it may be sensitive to the choice of parameters, such as the rank of the low-rank matrix and the sparsity constraint and finding the optimal values for these parameters can be challenging.

2.3. Reminder of the Used Fusion Evaluation Criteria

The quality of the fused images reflects the success of the fusion methods. Consequently, the evaluation of the quality of the fused images is a crucial issue in this area. The quality of the fused images can be assessed using either subjective or objective evaluations. Most existing evaluation techniques rely on human intervention, which comes with the drawbacks of being slow, not conducted in real-time, costly, and difficult to replicate. As a result, a metric for objective and automatic evaluation is necessary. While all these metrics take a variety of approaches, they all have the goal of imitating human assessment as precisely as possible. Various quality metrics have been developed in the literature. By assessing the fused image, quality indicators such as “standard deviation” and “spatial frequency” estimate the image’s quality. Typically, these metrics evaluate the intensity dispersion of the fused image. Furthermore, the source images can affect the quality of the fused image. Therefore, these methods are typically unable to offer sufficient information about product quality.

Table 1. Commonly used fusion evaluation criteria, their references and formula.

Used Evaluation Criteria	Definition and Formula
SSIM: mean of the structural similarity index measure between fused and source images [31,32,33]	$SSIM(x,y)={\left({\displaystyle \frac{2{\mu }_x{\mu }_y+c_1}{{\mu }_x^2+{\mu }_y^2+c_1}}\right)}^{\alpha }\times {\left({\displaystyle \frac{2{\sigma }_x{\sigma }_y+c_2}{{\sigma }_x^2+{\sigma }_y^2+c_2}}\right)}^{\beta }\times {\left({\displaystyle \frac{{\sigma }_{xy}+c_3}{{\sigma }_x{\sigma }_y+c_3}}\right)}^{\gamma }$ $x$: the reference image. $y$: the fused image. ${\mu }_x,{\mu }_y$$: \mathrm{mean} \mathrm{values} \mathrm{of} \mathrm{images} x$$\mathrm{and} y$. ${\sigma }_x^2,{\sigma }_y^2$$: \mathrm{variances} \mathrm{of} \mathrm{images} x$$\mathrm{and} y$. ${\sigma }_{xy}$$: \mathrm{covariance} \mathrm{of} \mathrm{images} x$$\mathrm{and} y$. $\mathrm{To} \mathrm{prevent} \mathrm{division} \mathrm{by} \mathrm{zero}, \mathrm{small} \mathrm{constants}, c_1,c_{2,}$$\mathrm{and} c_3$ are used. $\mathrm{Parameters} \alpha ,\beta ,\gamma$ are used to adjust the proportions.
FMI: feature mutual information including both fused and source images [34,35]	$FMI={\displaystyle \frac{MI\left(A,F\right)+MI\left(B,F\right)}{2}}$ $MI(F,A)$ $\mathrm{a}\mathrm{n}\mathrm{d} MI(F,B)$$\mathrm{are} \mathrm{the} \mathrm{mutual} \mathrm{information} \mathrm{between} \mathrm{the} \mathrm{fused} \mathrm{image} F$$\mathrm{and} \mathrm{source} \mathrm{images} A$ and B, respectively.
MCC: mean of the correlation coefficient between fused and source images [36,37,38,39]	$MCC={\displaystyle \frac{\left(r_{I,F}+r_{V,F}\right)}{22}}$ where $r_{XP}={\displaystyle \frac{\sum _{i=1}^M \sum _{j=1}^N \left(X\right(i,j)-\bar{X})\left(F\right(i,j)-\mu )}{\sqrt{\sum _{i=1}^M \sum _{j=1}^N \left(X\right(i,j)-\bar{X}{)}^2\left(\sum _{i=1}^M \sum _{j=1}^N \left(F\right(i,j)-\mu {)}^2\right)}}}$
PSNR: peak signal-to-noise ratio, including both fused and source images [40,41,42]	$PSNR=10{\mathrm{l}\mathrm{o}\mathrm{g}}_{10}{\displaystyle \frac{r^2}{MSE}}$ where r is the correlation coefficient between fused and source images and MSE is the mean squared error between fused and fused images
MSS: multi-scale structural similarity between fused and source images [43,44]	$MSSIM(X,Y)={\displaystyle \frac{1}{M}}\sum _{j=1}^{M}SSIM(x_j,y_j)$
PM: petrovic metric (or edge preservation measure) including both fused and source images [45,46,47]	$Q^{AB/F}={\displaystyle \frac{\sum _{\mathrm{n}=1}^N \sum _{m=1}^M Q^{AF}(\mathrm{n},\mathrm{m})w_A(\mathrm{n},\mathrm{m})}{\sum _{i=1}^N \sum _{\mathrm{j}=1}^M \left(w_A(\mathrm{i},\mathrm{j})+w(\mathrm{i},\mathrm{j})\right)}}$
SCD: sum of the correlation differences between fused and source images [48,49]	$\begin{array}{c}SCD=r\left(D_1,S_1\right)+r\left(D_2,S_2\right)\end{array}$ with $\begin{array}{c}r\left(D_k,S_k\right)={\displaystyle \frac{\sum _i \sum _j \left(D_k(i,j)-\overline{D_k}\right)\left(S_k(i,j)-\overline{S_k}\right)}{\sqrt{\left(\sum _i \sum _j {\left(D_k(i,j)-{\bar{D}}_k\right)}^2\right)\left(\sum _i \sum _j {\left(S_k(i,j)-{\bar{S}}_k\right)}^2\right)}}}\end{array}$

outlines details of the objective and commonly used fusion evaluation metrics. A higher value of each criterion indicates superior fusion performance.

2.4. Data Presentation

For this study, we made use of the visible-infrared image pairs sourced from the Corsican Fire Dataset. We carried out a series of experiments to evaluate the efficacy of our fusion technique. Drawing from the Corsican fire database, we analyzed 640 reference image pairs to evaluate our method’s potential. Each set within this collection comprised three related images: a visible domain image, its manually curated ground truth provided by a specialist, and a corresponding infrared image obtained concurrently.

Table 2. Example of visible images captured in different periods of the day and their IR and ground truth corresponding images.

Captured Visible Images	IR Corresponding Images	Ground Truth Corresponding Images
Bright daylight period (Image 33) Mean of the image = 126
Low-light period (Image 24) Mean of the image = 66
Night period (Image 42) Mean of the image = 21

displays a couple of such image trios from the dataset. These images were captured by the UMR CNRS 6134 SPE—University of Corsica team and their collaborators— [30], which include researchers, foresters, civil security personnel, and firefighters during their studies on fire spread, controlled burns, and actual forest fires.

These images were captured across diverse locations, at varying times of the day, from multiple perspectives, and under different environmental conditions and lighting intensities. These multi-modal images sourced directly from the JAI AD-080GE camera manufactured by JAI A/S, a company based in Skovlunde, Denmark are not in alignment, primarily because the visible and NIR sensors do not perfectly overlap. To bring these multi-modal images into alignment within the database, an image registration process using the homography matrix transform was employed. All images in the database are preserved in a lossless PNG format.

2.5. Fusion Experimental Results and Discussions

In this section, we assess the chosen fusion method. For this evaluation, we gathered 640 image pairs, with

Table 3. Examples of some fusion results and criteria. (a) Visible images, (b) IR images, (c) result of GFCE method, and (d) result of LatLRR method.

	Input Images	Fused Images	Fusion Evaluation Metrics
Image 1	(a) Visible	(c) GFCE-based fusion	SSIM = 0.47 FMI = 0.89 MCC = 0.68 PSNR = 11.29 MSS = 0.76 PM = 0.48 SCD = 0.66
	(b) IR	(d) LatLRR–based fusion	SSIM = 0.87 FMI = 0.93 MCC = 0.79 PSNR = 19.40 MSS = 0.96 PM = 0.40 SCD = 0.71
Image 2	(a) Visible	(c) GFCE-based fusion	SSIM = 0.47 FMI = 0.88 MCC = 0.70 PSNR = 12.6 MSS = 0.89 PM = 0.49 SCD = 0.67
	(b) IR	(d) LatLRR–based fusion	SSIM = 0.86 FMI = 0.91 MCC = 0.78 PSNR = 16.05 MSS = 0.94 PM = 0.48 SCD = 0.68
Image 3	(a) Visible	(c) GFCE-based fusion	SSIM = 0.63 FMI = 0.88 MCC = 0.72 PSNR = 12.85 MSS = 0.86 PM = 0.49 SCD = 0.45
	(b) IR	(d) LatLRR–based fusion	SSIM = 0.82 FMI = 0.92 MCC = 0.81 PSNR = 16.44 MSS = 0.91 PM = 0.44 SCD = 0.68
Image 4	(a) Visible	(c) GFCE-based fusion	SSIM = 0.63 FMI = 0.88 MCC = 0.7 PSNR = 12.52 MSS = 0.87 PM = 0.5 SCD = 0.44
	(b) IR	(d) LatLRR–based fusion	SSIM = 0.82 FMI = 0.91 MCC = 0.78 PSNR = 15.55 MSS = 0.92 PM = 0.44 SCD = 0.64

a,b showcasing four examples. The resultant fused images can be observed in Table 3c,d. When VIS/IR images are fused, the resulting fire images closely resemble the VIS image. A comprehensive analysis of the results produced by the chosen fusion method is presented. To evaluate the effectiveness of these fusion methods, we analyzed various performance metrics to determine the efficacy of the fusion techniques. The fused images from both methods are illustrated in Table 3c,d.

The results presented in

Table 4. Averaged values of each criterion for each fusion approach using all database.

Fusion Methods	SSIM	FMI	MCC	PSNR	MSS	PM	SCD
GFCE	0.64	0.90	0.71	13.23	0.85	0.44	0.7
LatLRR	0.85	0.92	0.79	17.96	0.93	0.46	0.88

show that LatLRR consistently outperforms GFCE across all evaluated criteria. This indicates that LatLRR is more effective at maintaining structural integrity, preserving feature information, and producing higher quality-fused images. These advantages make LatLRR a superior choice for applications requiring precise and reliable image fusion, such as forest fire detection and segmentation.

3. Improvement of LatLRR Fusion Method with Optimal Weighting of the Visible Source Image

3.1. Introduction

Since visible images vary in brightness, and depending on the acquisition period, we proposed weighting the visible images in grayscale before using them in the fusion process with the infrared image, and a weight noted α. The value of this weight affects the fusion result. We chose the SCD criterion to quantify this effect. For each one of the 640 used visible images, the SCD criterion was plotted versus the mean of the visible image brightness. The optimal value of the weight was calculated. This operation is summarized in the framework of Figure 4. Some examples of visible images captured in different day periods and the plot of their SCD criteria versus applied weight α are presented in

Table 5. Examples of visible images captured in different day periods and the plot of their SCD criteria versus α.

Captured Visible Images	SCD Creterion Value versus the Weight α	${\mathit{\alpha }}_{\mathit{o}\mathit{p}\mathit{t}}$
Bright daylight period (Image 33) Mean of the image = 126		0.7
Low-light period (Image 24) Mean of the image = 66		1
Night period (Image 42) Mean of the image = 21		1.5

. The optimal value of α is also shown. In this Table, three periods of the day are considered: the bright daylight period, the low-light period, and the night period. We remarked that this optimal value of α depends on the brightness of the visible image.

3.2. Estimation of the Optimal Weight α with the Least Mean Squares Method Source Image

In Figure 5, we plot all the obtained optimal weights versus the mean of the corresponding visible image. The shape of this curve is modeled by a polynomial function of order two. So, we propose estimating the optimal value of the weight α with a second-order polynomial model whose parameters are optimized with the least squares method from the curve of optimal weights with respect to the mean values of visible images. The second-order polynomial model can be expressed with the following equation: ${\alpha }_{\mathrm{opt} }{=a\times \mu }^2+b\times \mu +c$, where µ is the mean value of the visible image. We note that x_i is the mean of the i^th visible image, y_i is the corresponding i^th optimal value of α, and n is the number of used images (640). The image index, i, varies from 1 to n. The optimal estimated values of coefficients a, b, and c can be obtained by the resolution of the following linear system:

$$\left\{\begin{array}{c}na+b{\displaystyle \sum _{i=1}^n} x_i+c{\displaystyle \sum _{i=1}^n} x_i^2={\displaystyle \sum _{i=1}^n} y_i\\ a{\displaystyle \sum _{i=1}^n} x_i+b{\displaystyle \sum _{i=1}^n} x_i^2+c{\displaystyle \sum _{i=1}^n} x_i^3={\displaystyle \sum _{i=1}^n} x_i{y}_i\\ a{\displaystyle \sum _{i=1}^n} x_i^2+b{\displaystyle \sum _{i=1}^n} x_i^3+c{\displaystyle \sum _{i=1}^n} x_i^4={\displaystyle \sum _{i=1}^n} x_i^2{y}_i\end{array}\right.$$

The obtained results show that the estimated values of coefficients a, b, and c are, respectively, a = 24, b = −0.12, and c = 0.02. The use of the obtained optimal weight ${ \alpha }_{\mathit{opt} }$ to improves the LalLRR fusion technique depending on the mean of the gray level visible image is described in Figure 6. The improvement of the SCD criterion for all tested images is illustrated in Figure 7. The mean of the SCD criterion improvement is 0.05, as shown in

Table 6. Mean of the SCD fusion criterion value of all fused images.

	Without Visible Image Optimization Weighting	With Visible Image Optimization Weighting
Mean of the SCD fusion creterion value	0.88	0.93

.

4. Segmentation of Fire Images Using the Obtained Fused Images by a Major Voting Approach

4.1. Introduction

In this study, we combined three UNet backbones using a major voting scheme to process the segmentation task, as illustrated in Figure 8. Among various architectures, the UNet model, combined with powerful backbones like ResNet50, VGG16, and EfficientNet, showed promising results. This study explores these combinations, utilizing a major voting mechanism to assess their effectiveness in fire segmentation tasks.

4.2. Recall of Segmentation Method Using the Major Voting

The UNet-ResNet50 model integrates the UNet architecture with a ResNet50 backbone. Known for its deep residual learning framework, ResNet50 serves as the encoder, enhancing the model’s ability to learn complex features from fire images. Special adaptations for fire segmentation include the use of pre-trained weights on ImageNet for faster convergence and the introduction of dropout layers to prevent overfitting. The UNet-VGG16 architecture utilizes VGG16 as its backbone. VGG16’s architecture is appreciated for its simplicity and depth, providing robust feature extraction capabilities. The model incorporates batch normalization after each convolutional layer to stabilize learning and improve generalization. Combining UNet with EfficientNet, the UNet-EfficientNet model benefits from EfficientNet’s scalable architecture, which balances network depth, width, and resolution, making it computationally efficient. EfficientNet’s compound scaling method is fine-tuned to optimize performance on the fire dataset, ensuring effective feature extraction at various scales. The major voting scheme is a decision-making process that aggregates predictions from the three models to improve the reliability and accuracy of fire segmentation. For each pixel in the output mask, if at least two models predict the pixel as ‘fire’, the pixel is classified as ‘fire’. If the models do not agree, the pixel classification is deferred to the model with the highest confidence score for that pixel. This scheme leverages the strengths of each model, mitigating individual weaknesses and improving the overall segmentation performance.

4.3. Data Training

Within this section, we provide a detailed assessment of our selected segmentation strategy, informed by a succession of systematically designed experiments. To quantify the effectiveness of this segmentation technique, we employed three models, UNet-ResNet50, UNet-VGG16, and UNet-EfficientNet, followed by a major voting scheme for final segmentation. Implemented using the Keras library with TensorFlow as the backend, the models were trained on Google Colab Pro with a powerful GPU. Our dataset was partitioned into 80% for training, 10% for validation, and 10% for testing.

The training process involved batch sizes of 16 and 150 epochs, optimized using the Adam optimizer with a learning rate of 0.0001. After training each model individually, we employed a major voting scheme to enhance segmentation accuracy. This process aggregates predictions from the three models to improve reliability and accuracy. For each pixel in the output mask, if at least two models predict ‘fire’, the pixel is classified as ‘fire’. If the models do not agree, the pixel classification is deferred to the model with the highest confidence score. This scheme leverages the strengths of each model, mitigating individual weaknesses and improving overall segmentation performance. The inference time for a single image was about 100 ms.

4.4. Segmentation Results

After devising a semantic segmentation method, the predominant question was, “How can we determine if our segmentation is accurately executed?” In the realm of image segmentation, evaluating performance using metrics is vital to quantifiably assess the classification outcomes [50,51,52].

4.4.1. Presentation of the Segmentation Evaluation Criteria: Accuracy, Precision, Specificity, Recall, F1 Score, and IoU

We classified data into two classes, labeled as positive class and negative class (ground truth). A classification method assigned a label to each data point, either estimated positive (predicted as positive) or estimated negative. The evaluation of the performance of the classification method was linked to the following metrics:

• TP (True Positive): cases where the assigned class is positive, knowing that the actual value (ground truth) is indeed positive.
• TN (True Negative): cases where the assigned class is negative, and the actual value is indeed negative.
• FP (False Positive): cases where the assigned class is positive, but the actual value is negative.
• FN (False Negative): cases where the assigned class is negative, but the actual value is positive.

Based on these four quantities, we defined the values of the most used classification performance criteria, as shown in

Table 7. Formulas of the used classification performance metrics.

$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$

$\mathit{Precision}={\displaystyle \frac{TP}{TP+FP}},$.

$Specificity={\displaystyle \frac{TN}{FP+TN}}$

$\mathit{Recall}={\displaystyle \frac{TP}{TP+FN}}$

$F1 \mathit{Score}=2\times {\displaystyle \frac{\mathit{Precision}\times \mathit{Recall}}{\mathit{Precision}+\mathit{Recall}}}$

$IoU={\displaystyle \frac{TP}{TP+FP+FN}}$

.

4.4.2. Details and Discussion of the Obtained Segmentation Results

The results from both

Table 8. Some examples of segmentation results and performances for four images.

(a)	(b)	(c)	(d)	(e)
Visible images	Segmentation of visible images only	Segmentation of IR images only	Segmentation of the fused images with classical LatLRR	Segmentation of the fused images with LatLRR after image optimization weighting
IOU	89.34%	87.43%	92.22%	96.88%
Accuracy	99.78%	99.74%	99.83%	99.94%
Precision	96.85%	97.25%	92.22%	96.88%
Specificity	99.94%	99.95%	99.83%	99.93%
Recall	92.02%	89.64%	94.05%	97.64%
F1 score	94.37%	93.29%	95.95%	98.41%
IOU	88.42%	84.21%	93.02%	95.66%
Accuracy	99.36%	98.98%	99.62%	99.76%
Precision	99.38%	84.40%	99.14%	99.08%
Specificity	99.97%	98.93%	99.95%	99.95%
Recall	88.92%	99.72%	93.77%	96.51%
F1 score	93.86%	91.43%	96.38%	97.78%
IOU	86.45%	83.40%	89.51%	90.68%
Accuracy	99.11%	98.93%	99.21%	99.40%
Precision	88.32%	89.51%	89.41%	90.94%
Specificity	99.20%	99.33%	99.28%	99.39%
Recall	97.61%	92.43%	98.06%	99.68%
F1 Score	92.73%	90.95%	93.53%	95.11%
IOU	82.93%	76.93%	90.86%	94.16%
Accuracy	98.97%	98.55%	99.49%	99.69%
Precision	86.69%	82.99%	94.56%	99.04%
Specificity	99.19%	98.96%	99.69%	99.95%
Recall	95.03%	91.33%	95.87%	95.03%
F1 score	90.67%	86.96%	95.21%	96.99%

The value displayed in bold denotes the method with the highest score.

and

Table 9. Mean of the segmentation criteria values computed for all test database images.

Segmentation Criteria	Segmentation of Visible Image Only	Segmentation of IR Images Only	Segmentation of the Fused Images with Classical LatLRR	Segmentation of the Fused Images with LatLRR after Image Optimization Weighting
IOU	88.81%	86.42%	92.05%	94.52%
Accuracy	99.53%	99.44%	99.61%	99.84%
Precision	93.55%	92.30%	95.12%	96.62%
Specificity	99.70%	99.63%	99.77%	99.88%
Recall	94.41%	93.06%	95.84%	97.44%
F1 score	94.06%	92.74%	95.75%	97.09%

clearly demonstrate that the segmentation of fused images with LatLRR after image optimization weighting outperforms other methods across all evaluated criteria. This method achieves higher IOU, accuracy, precision, specificity, recall, and F1 scores, indicating its effectiveness in accurately identifying fire regions and minimizing false positives and negatives. The experiments show that the proposed model, which includes an optimized weighted image fusion stage before segmentation, provides a high Intersection over Union (IoU) score of more than 94% and an F1 score and recall of more than 97%.

The optimized fusion method shows the strengths of both visible and IR images, providing a robust solution for fire detection. By combining the high-resolution details from visible images with the thermal information from IR images, the optimized fusion method offers a comprehensive and reliable approach to fire segmentation.

5. Conclusions and Perspectives

We propose in this work to combine both visible and infrared images with an appropriate fusion technique to improve the quality of fire detection, segmentation, and localization. The visible image is weighted before being used in the fusion process. The value of the optimal weight is estimated from the mean of the visible image with a second-order polynomial model. We optimized the model’s parameters using the least-squares method based on the relationship between optimal weights and the mean of visible images. Finally, a major voting method based on deep learning models was used. By combining the rich details and sharp contrasts of visible images with the precise thermal information provided by IR images, this optimized fusion method offered a comprehensive and reliable approach for detecting and segmenting forest fires. The results highlight the significant potential of advanced image fusion techniques, as well as optimization processes, to enhance the performance of automated forest fire segmentation systems. The substantial improvements observed across all evaluation metrics underscore the importance of integrating multiple image sources and applying sophisticated optimization techniques to achieve significantly superior segmentation results. This method could thus revolutionize forest fire monitoring and management systems, providing increased accuracy and reliability, which is essential in critical applications such as fire safety and disaster management. More precisely, in cases of forest fires spreading rapidly, with our optimized fusion method, emergency responders can quickly identify the fire’s location and intensity, enabling faster and more effective action.