Real-Time Wildfire Monitoring Using Low-Altitude Remote Sensing Imagery

Abstract

With rising global temperatures, wildfires frequently occur worldwide during the summer season. The timely detection of these fires, based on unmanned aerial vehicle (UAV) images, can significantly reduce the damage they cause. Existing Convolutional Neural Network (CNN)-based fire detection methods usually use multiple convolutional layers to enhance the receptive fields, but this compromises real-time performance. This paper proposes a novel real-time semantic segmentation network called FireFormer, combining the strengths of CNNs and Transformers to detect fires. An agile ResNet18 as the encoding component tailored to fulfill the efficient fire segmentation is adopted here, and a Forest Fire Transformer Block (FFTB) rooted in the Transformer architecture is proposed as the decoding mechanism. Additionally, to accurately detect and segment small fire spots, we have developed a novel Feature Refinement Network (FRN) to enhance fire segmentation accuracy. The experimental results demonstrate that our proposed FireFormer achieves state-of-the-art performance on the publicly available forest fire dataset FLAME—specifically, with an impressive 73.13% IoU and 84.48% F1 Score.

1. Introduction

In recent years, propelled by advancements in sensor technology, the acquisition of remote sensing (RS) images has become increasingly convenient. In the field of remote sensing, tasks such as ground crop detection [1,2], change detection [3,4,5], and urban scene segmentation [6,7,8] have attracted increasing attention. For a more detailed field of remote sensing, remote sensing-based fire detection or segmentation can be important to reduce damage and analyze carbon emissions [9].

Thanks to the availability of publicly accessible satellite data, hyperspectral images (HSIs) acquired through satellites have been widely employed for fire segmentation [10,11,12]. Some works [13,14] use satellite post-disaster images as data and focus on detecting burned areas after wildfires. The reasons why these satellite images are not convenient for direct real-time wildfire detection are as follows: the inherent characteristics of satellites, such as low revisit rates and limited spatial resolution, significantly impede their utility in emergency situations, including fire disaster, as shown in Figure 1. Consequently, relying solely on satellite remote sensing images proves challenging in meeting the real-time demands necessitated by such unforeseen events.

Unmanned aerial vehicles (UAVs) offer a flexible and cost-effective solution for capturing high-resolution remote sensing images at low altitudes, addressing the challenges associated with satellite imagery. Low-altitude UAV-based wildfire detection algorithms have been explored previously [15,16,17]; these early methods leverage enhanced CNN architectures to extract information from low-altitude remote sensing images [18,19]. However, due to the inherent characteristics of CNNs, i.e., locality and translation invariance, they struggle to effectively capture global information within the images.

Visual Transformer (ViT) can obtain higher accuracy rates than CNNs in tasks, including image segmentation, for its effective attention mechanism [20,21,22,23,24]. However, it requires a higher computational load. Even though several approaches have been proposed addressing this issue, e.g., the shifted window attention mechanism introduced in [25], the existing methods [26,27] are still limited in fulfilling the task of real-time precise fire segmentation for low-altitude remote sensing images. Since images captured by UAV usually possess higher resolutions (typically 3840 × 2140) and encompass more advanced semantic information than common images, this is particularly true for UAV fire images. In order to achieve accurate segmentation results in RS images, the connection between the small fire point region and the overall image is indispensable, just like the annotated local information and global information in Figure 2.

This paper presents a novel network named FireFormer for the real-time wildfire segmentation of UAV RS images. FireFormer adopts a hybrid structure, comprising a ResNet18-based encoder [28] and Transformer-based decoder. In the decoder, we introduce the Forest Fire Transformer Block (FFTB) as a key component to effectively integrate global and local contexts through a lightweight dual-branch structure. Moreover, to overcome the issue of blurred-edge features caused in a series of downsampling and upsampling operations in the network, we propose the Feature Refinement Net (FRN).

The main contributions of this research are outlined as follows:

(1)

We propose the FFTB, which incorporates two parallel branches. These branches focus on capturing both the global and local information of the RS image.

(2)

To acquire global information of an image with a lower computational cost, we have devised a Cross-Scaled Dot-Product attention mechanism. This mechanism captures global information at different scales and performs feature fusion.

(3)

We propose the FRN to refine the output feature maps with finer details. This structure addresses the challenge of losing small objects during the segmentation process by leveraging parallel spatial and channel operations.

2. Related Work

2.1. Semantic Segmentation of Fire Remote Sensing Images

Forest fire is a significant and non-negligible contributor to carbon emissions, and as a result, many researchers in remote sensing have been dedicated to fire detection [29,30] and segmentation [31,32,33]. These works can be divided into two types based on the image data they used.

The first type of works use satellite remote sensing images, which employ semantic segmentation to delineate fire areas in the satellite images and conduct a series of downstream studies such as fire damage assessment [21,34] and carbon emissions [35,36,37] analysis based on the delineated areas. Reference [21] introduces a novel Transformer network that segments active fire pixels from VIIRS satellite time series, focusing on temporal information for improved detection accuracy. It identifies the limitations of the current methods, including low precision and the underutilization of temporal data in active fire detection from satellite imagery. The method increases the accuracy of active fire detection to 53.1%. In reference [34], the study refines a Double-Step U-Net model using deep learning to enhance wildfire severity classification from satellite images, emphasizing the optimization of loss functions for better accuracy. It tackles the challenge of a precise wildfire damage estimation by evaluating these loss functions’ impact on model performance. And it improves the accuracy of wildfire detection to 54%.

The second type of works use UAV images [38,39]. The semantic segmentation work based on UAV fire images mainly detects small fire points and smoke in the early stages of forest fires. Ref. [40] proposes a real-time fire segmentation method that improves upon deeplabv3+ by using MobileNetV3 for faster processing and adding shallow features to ensure accuracy, helping firefighters quickly assess the fire extent and plan responses. This method increases the recognition accuracy of the FLAME wildfire dataset to 72.87%. Ref. [41] introduces the FLAME dataset for aerial imagery pile burn detection, utilizing deep learning methods for the binary classification of fire presence and segmentation to delineate fire borders. The paper addresses the challenges in early fire detection by developing an Artificial Neural Network (ANN) for classification accuracy and a U-Net-based deep learning approach for precise fire segmentation, enhancing fire management strategies. These works greatly promote the combination of semantic segmentation and fire detection. However, it is worth noting that due to the particularity of fire detection, segmentation methods based on UAV fire images usually focus more on segmentation accuracy (metrics such as the IoU, OA and F1 score) but overlook the real-time requirements of responding to fire scenes (metrics such as FPS and params).

2.2. Semantic Segmentation Based on Transformer

The Transformer [42,43] was first introduced in the field of natural language processing and showcased its remarkable ability to integrate global information. The standard Transformer Block is depicted in Figure 3a. Leveraging the Transformer’s impressive performance in global information modeling, the Visual Transformer (ViT) brought the Transformer into the realm of computer vision. TransUNet [44] and TransFuse [45] combined CNN and Transformer in an encoder to achieve a balance between local and global information. In the field of remote sensing, the UNetformer [8] is a novel hybrid network that combines a CNN encoder with a Transformer decoder for efficiently segmenting urban scenes in remote sensing imagery, capturing both global and local contexts. The paper proposes an efficient global–local attention mechanism and a feature refinement head to enhance semantic segmentation performance, achieving a 71.33% IoU on the FLAME dataset. However, researchers recognized the high memory requirements behind ViT’s exceptional long-range modeling capabilities [46,47,48]. Researchers have generally focused on two directions for improving semantic segmentation networks based on Transformers. The first direction is to continue exploring ViT’s ability to represent deeper image features. For instance, refs. [49,50] introduced the “cloze test” idea into images, forcing ViT to learn deeper image features from damaged images. In the field of remote sensing, SS-MAE [51] innovatively introduces spatial masks and multispectral masks, forcing the network to learn detailed features from remote sensing images from both the spatial and multi-band perspectives. This framework achieves a higher accuracy in remote sensing images.

The second direction is to reduce ViT’s computational complexity and memory consumption. One well-known work is Swin Transformer [25], which proposed the shifted window attention mechanism (SWMSA), as shown in Figure 3b. These works [27,52] have promoted the lightweighting of semantic segmentation networks based on Transformer to varying degrees, allowing the networks to adapt to real-time requirements in specific scenarios. However, it should be noted that remote sensing images have larger resolutions and richer semantic information. This makes it difficult for common general segmentation networks to perform well in real-time when facing remote sensing images.

To address this issue, we designed the Forest Fire Transformer Block (FFTB) in FireFormer, which is a lightweight module composed of parallel global and local branches, as shown in Figure 3c. To reduce the computational complexity of FFTB, we designed a Cross-Scaled Dot-Product attention mechanism, enabling FFTB to obtain global information while meeting real-time requirements. Our experiments have shown that FFTB enables FireFormer to achieve the highest IoU metric (73.13%) while maintaining strong real-time inference capabilities, reducing the network parameter count to 11.7 M and reaching a frame rate of 38.29 FPS.

3. Methods

In this section, we begin by presenting an overview of the FireFormer model. Subsequently, we delve into the introduction of two pivotal modules within FireFormer, namely, the Forest Fire Transformer Block (FFTB) and the Feature Refinement Network (FRN).

3.1. Network Structure

The FireFormer model’s overall architecture, depicted in Figure 4, is introduced. FireFormer adopts a hybrid structure comprising a CNN-based encoder and a Transformer-based decoder. By employing Weighted Feature map fusion (WF), FireFormer effectively obtains the discriminative features that are crucial for identifying forest fires in RS images. The formula denoting this feature map fusion is represented as Equation (1).

$$FF=\alpha \times {Res}_n+(1-\alpha )\times S_n$$

(1)

where ${Res}_n$ represents the output feature map of the Resnet encoder and $S_n$ represents the output feature map of the decoder, and $\alpha$ is the a learnable weight value. Furthermore, the FRN is introduced to further enhance the segmentation accuracy of FireFormer.

In the CNN-based encoder of FireFormer, we have chosen to use a pre-trained ResNet18 as the encoder. ResNet18 is composed of four stages of residual blocks (Res Blocks), where each block downsamples the feature maps by a factor of two and doubles the number of channels. Assuming the size of the given RS image is $\mathrm{f}\in {\mathrm{R}}^{3\times 1024\times 1024}$, the output defined for each stage when passing the RS image through the encoder is ${\mathrm{R}\mathrm{e}\mathrm{s}}_{\mathrm{n}} \in {\mathrm{R}}^{64\cdot 2^{(\mathrm{n}-1)}\times 256\cdot 2^{(1-\mathrm{n})}\times 256\cdot 2^{(1-\mathrm{n})}}$, where $\mathrm{n}=\mathrm{1,2},\mathrm{3,4}$.

In the Transformer-based decoder of FireFormer, we have designed a parallel dual-branch (e.g., global branch and local branch) FFTB to enhance the overall representation of the RS image features extracted by the CNN-based encoder, where each block upsamples the feature maps by a factor of two and halves the number of channels. When the given feature maps pass through the global branch of FFTB, Cross-Scaled Dot-Product attention (CDA) is employed to aggregate information from both the local details and the global context. Specifically, CDA divides the feature maps into patches, with each patch having a size of 2 × 2. These patches are then flattened and separated into Key (K), Query (Q) and Value (V) through linear operations. The aggregation of global information in the feature maps is achieved through Equation (2).

$$S{A}_h(x)=Softmax({\displaystyle \frac{Q\cdot K^T}{\sqrt{D_h}}})\cdot V$$

(2)

In the local branch of the FFTB, FireFormer incorporates a simple yet effective Local Feature Enhancement (LFE) block. This block is specifically designed to enhance FireFormer’s capability to accurately identify fire points in vast forests and improve its ability to discern fire edges. Lastly, the FFTB performs a summation operation on the output feature maps of both the global and local branches. This operation ensures that the output feature maps maintain their original resolution. The output of the each FFTB is defined as ${\mathrm{S}}_{\mathrm{n}}\in {\mathrm{R}}^{512\cdot 2^{(1-\mathrm{n})}\times 32\cdot 2^{(\mathrm{n}-1)}\times 32\cdot 2^{(\mathrm{n}-1)}}$, where $\mathrm{n}=\mathrm{1,2},\mathrm{3,4}$.

After the four encoding and decoding stages, we obtain the encoded feature maps ${\mathrm{R}\mathrm{e}\mathrm{s}}_{\mathrm{n}}$ and perform a weighted summation with the corresponding ${\mathrm{S}}_{\mathrm{n}}$ in skip connection. This process ensures a balance between local and global information in the feature maps. After four iterations, the final output feature maps are denoted as $F\in R^{64\times 256\times 256}$.

Segmenting small fire points in high-resolution RS images is challenging due to factors like tree occlusion and variations in fire intensity. To address this, we introduce the FRN at the end of FireFormer. The FRN refines the feature maps spatially and channel-wise, and the feature maps are upsampled four times to restore the original resolution. The refined feature maps obtained through the FRN are represented as $F\in {\mathrm{R}}^{n\times 256\times 256}$, where $\mathrm{n}$ denotes the number of classes. These refined feature maps play a crucial role in achieving accurate segmentation results.

3.2. Forest Fire Transformer Block

The semantic segmentation of RS images requires a network with robust information extraction capabilities. This capability should prioritize the extraction of local information, while also considering the extraction of global information. To address these challenges, we have developed the FFTB. The structure of the FFTB is visually depicted in Figure 5.

The FFTB global branch is constructed with Transformer Blocks that incorporate the CDA mechanism. This branch is specifically designed to extract global information from the feature maps, while preserving their resolution and channel size. Suppose the given feature map as $f^{l-1}\in R^{c_1\times h\times w}$, then the output feature map after passing through the global branch can be represented as $G\in R^{c_1\times h\times w}$. The entire process can be succinctly summarized by the following:

$$G=MLP(BPA(f^{l-1}))$$

(3)

The FFTB local branch consists of two parallel depthwise separable convolutions (DWconv). Suppose the given feature map as $f^{l-1}\in R^{c_1\times h\times w}$. To ensure the preservation of features across channels, we have maintained the same number of channels in the feature map without any alterations. The outputs of the parallel depthwise separable convolutions applied to the feature map ${ f}^{l-1}$ are denoted as $L_1$ and $L_2$. Subsequently, we fuse $L_1$ and $L_2$ along the channel dimension to obtain the final output feature map $L$ of the local branch. Lastly, we merge the feature map L with the corresponding output $G$ from the global branch. Additionally, we employ the $GELU$ activation function to eliminate information redundancy within the fused feature map. The entire process can be concisely summarized as follows:

$$\begin{array}{cc}& L_1=\sigma \left(f^{l-1}\right)\\ & L_2=\varphi \left(f^{l-1}\right)\\ & L=GELU(G+LN(L_1+L_2\left)\right)\end{array}$$

(4)

3.3. Cross-Scaled Dot-Product Attention Mechanism

After conducting our observations, we have discovered that the feature maps derived from RS images during the process of feature extraction contain a wealth of information. These include intricate texture details that exhibit strong local characteristics, as well as semantic information that showcases robust global characteristics.

In order to fully exploit the detailed texture information in remote sensing images, we have opted for a straightforward non-overlapping window partitioning approach (specifically, we have utilized 2 × 2 windows in our experiments).

The advanced semantic information in remote sensing images is crucial for semantic segmentation. Therefore, we propose the Cross-Scaled Dot-Product Attention Mechanism to globally model the feature maps of remote sensing images, as illustrated in Figure 6. In order to enable the exchange of information between windows, we employ convolution in both the horizontal and vertical directions. The cross-shaped window interaction module merges features from both the vertical and horizontal directions, thereby achieving global information interaction. Specifically, in the horizontal direction, the information interaction between any point $P_1^{(x,y)}$ in window 1 and any point $P_2^{(x+s,y)}$ in window 2 can be effectively modeled using Equation (5).

$$P_1^{(x,y)}={\displaystyle \frac{{\sum }_{i=0}^{s-x-1}{P}_1^{(x+i,y)}+{\sum }_{j=0}^s{P}_2^{(x+w-j,y)}}{s}}$$

(5)

$$P_1^{(m+i,n)}=WSA\left(P_1^{(m,n)}\right)$$

(6)

where $s$ represents the size of the windows. Simultaneously, the relationships between points within individual windows, as exemplified by the red path within window 1, have already been established during the window self-attention process, as Equation (6). Consequently, in the horizontal direction, the interaction of information between windows is facilitated by the convolution operation.

Similarly, in the vertical direction, inter-window information exchange is established through vertical convolution. The relationship between window 1 and window 3 can be described using Equation (5). Information exchange can be achieved by connecting multiple intermediate windows; for instance, points $P_4^{(x+s,y+s)}$ in window 4 can interact with points $P_2^{(x+s,y)}$ in window 2, and subsequently, points $P_2^{(x+s,y)}$ in window 2 can interact with points $P_1^{(x,y)}$ in window 1, thereby facilitating information exchange between windows in different directions. Thus, the cross-shaped window interaction method effectively simulates global relationships in window directions, thereby capturing the global information present in the feature map.

3.4. Feature Refinement Net

In the context of semantic segmentation tasks involving RS images, feature map fusion plays a pivotal role within the U-Net-like framework. The utilization of skip connections has a limited impact on enhancing the segmentation accuracy of the final fused feature map. Hence, we propose the Feature Refinement Net, which refines the ultimate fused feature map from two distinct perspectives: the Spatial Path and the Channel Path, as depicted in Figure 7.

We commence by upsampling the fused feature map by a factor of 2, denoting it as $\mathrm{f}\in {\mathrm{R}}^{\mathrm{c}\times \mathrm{h}\times \mathrm{w}}$. In the Channel Path, we employ Adaptive Avgpool to diminish the spatial dimensions of the feature map to 1 × 1, which we define as $\mathrm{f}\in {\mathrm{R}}^{\mathrm{c}\times 1\times 1}$. To ensure a precise recovery of the channel information in the FRN, we allocate the channel numbers in a ratio of 2:1:1, represented as ${\mathrm{f}}_{\mathrm{c}}^1\in {\mathrm{R}}^{{\displaystyle \frac{\mathrm{c}}{2}}\times 1\times 1}$, ${\mathrm{f}}_{\mathrm{c}}^2\in {\mathrm{R}}^{{\displaystyle \frac{\mathrm{c}}{4}}\times 1\times 1}$ and ${\mathrm{f}}_{\mathrm{c}}^3\in {\mathrm{R}}^{{\displaystyle \frac{\mathrm{c}}{4}}\times 1\times 1}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{l}\mathrm{y}$. Subsequently, within each channel branch, we apply the reduce and expand operation, comprised of four 1×1 convolutional layers. This operation reduces the channel count by half, then by a quarter, before restoring them to their original quantity. Following this, the outcomes from the three channel branches are concatenated along the channel dimension, resulting in the final channel-wise recovered feature map denoted as $C_0\in R^{c\times 1\times 1}$. Finally, the spatial recovery is achieved through the matrix multiplication between $C_0$ and $f$.

To refine spatial features at different levels in the Spatial Path, we divide the feature map $\mathrm{f}\in {\mathrm{R}}^{\mathrm{c}\times \mathrm{h}\times \mathrm{w}}$ into two parts along the channel dimension, referred to as $f_s^1\in R^{{\displaystyle \frac{c}{2}}\times h\times w}$ and $f_s^2\in R^{{\displaystyle \frac{c}{2}}\times h\times w}$. In the first branch, we apply deep separable convolution layers twice consecutively to segment $f_s^1$, using convolution kernels of size 7 × 7 and 5 × 5. The use of larger convolution kernels allows for a wider receptive field, thereby enhancing the global semantic information of the feature map. In the second branch, we perform a depth-separable convolution on segment ${\mathrm{f}}_{\mathrm{s}}^2$, utilizing a 3 × 3 convolution kernel to increase the spatial texture details of the feature map. Finally, ${\mathrm{f}}_{\mathrm{s}}^1$ and ${\mathrm{f}}_{\mathrm{s}}^2$ are merged along the channel dimension and added together to obtain the refined feature map. It is important to note that a residual module has been incorporated to prevent any degradation in network performance.

3.5. Datasets

3.5.1. FLAME Dataset

The FLAME dataset was gathered in northern Arizona, USA, utilizing drones for controlled pile burning. This dataset comprises multiple collections, encompassing aerial videos captured by drone cameras and thermal images recorded by infrared cameras. Within this dataset, there are 2003 photos annotated with fire areas, each with a size of 3840 × 2140. These annotated images serve as valuable resources for conducting fire semantic segmentation research. To facilitate training, we allocated 60% of the images for training purposes, 20% for validation and the remaining 20% for testing. Prior to training, each image is cropped to a size of 1024 × 1024, ensuring consistency across the dataset.

3.5.2. FLAME2 Dataset

To assess the generalization capability of FireFormer and mitigate the impact of similar images during training or segmentation results, we conducted segmentation tests on the FLAME2 dataset [53]. Additionally, based on the obtained test results, we performed an analysis of carbon emissions during forest fire burning processes. The FLAME2 dataset primarily comprises original and manipulated aerial images captured during a controlled fire in November 2021 in an open canopy pine forest in northern Arizona. It includes 7 sets of original RGB and infrared (IR) videos that have not been annotated. For the experiment, we primarily selected a portion of the third fire’s RGB video for segmentation testing. We extracted frames from the RGB video and subsequently cropped the extracted frame images to a size of 1024 × 1024.

3.6. Implementation Details

3.6.1. Training and Testing Settings

In the experiment, all models were implemented using the PyTorch framework on an NVIDIA GTX 3060 GPU. To achieve fast convergence, we used the AdamW optimizer to train all models. The base learning rate was set to 6 × 10⁻⁴. The learning rate was adjusted using the cosine annealing strategy. The maximum number of epochs during training was set to 40, and the batch size was set to 4. During the training process, we randomly cropped the FLAME dataset to a size of 1024 × 1024.

3.6.2. Evaluation Metrics

In our experiment, we employed two main categories of evaluation metrics. The first category focuses on assessing the accuracy of the network and includes the following metrics: overall accuracy (OA), average F1 score (F1) and mean intersection over union (mIoU).

The second category evaluates the efficiency of the network and includes the following metrics: floating-point operations (Flops), frames per second (FPS), memory usage in megabytes (MB) and the number of model parameters (M). By considering both accuracy and efficiency metrics, we can comprehensively evaluate the performance of the network in terms of its accuracy, computational complexity, speed and memory usage.

3.6.3. Loss Functions

In order to expedite the convergence of the network during the training process, we employ a unified loss function that amalgamates the Dice loss $L_{dice}$ and cross-entropy loss $L_{ce}$ to oversee the model’s progress. The calculation of this joint loss function is as follows:

$$\begin{array}{cc}& L_{dice}=1-{\displaystyle \frac{2}{N}}{\sum }_{n=1}^N{\sum }_{k=1}^K{\displaystyle \frac{{\widehat{y_k}}^{\left(n\right)}{y}_k^{\left(n\right)}}{{\widehat{y_k}}^{\left(n\right)}+y_k^{\left(n\right)}}}\\ & L_{ce}=-{\displaystyle \frac{1}{N}}{\sum }_{n=1}^N{\sum }_{k=1}^K{y}_k^{\left(n\right)}log{\widehat{y_k}}^{\left(n\right)}\end{array}$$

(7)

The Dice loss function $L_{dice}$ has a robustness to class imbalance and higher sensitivity to the pixel overlap of small targets. Therefore, the Dice loss function performs better in segmenting small targets. The cross-entropy loss function $L_{ce}$ penalizes misclassifications more heavily during training, promoting the model to learn more classification boundaries. To improve the accuracy of the segmentation results, we used a combination of these two loss functions.

3.7. Hardware

The FLAME dataset used in this study was collected using various drones and cameras.

Table 1. Technical specification of hardware and dataset information.

	Phantom 3 Professional, DJI, 1280 g, diagonal size = 350 mm, max speed = 16 m/s (~57 kph), max flight time is 23 min, flight time is reduced to 18 min due to additional weight
	Matrice 200, DJI, 3.80 kg, size: 71 6 mm × 220 mm × 236 mm, payload up to 2 kg, 16 m/s (~61 kph), batteries: (TB50) and TB55a, max flight time: 38 min, operation range of 7 km
Type	Camera	Palette	Duration	Resolution	FPS	Size	Application	Labeled
Video	Phantom	Normal (.JPEG)	2003 frames	3480 × 2160	-	5.3 GB	Segmentation	Y (Fire)

describes the technical specification of the utilized drones and cameras and the specific information obtained from the FLAME dataset.

4. Results

4.1. Ablation Study

In order to assess the efficacy of the proposed network architecture and crucial modules, we conducted ablation experiments on the FLAME dataset, utilizing U-Net as the baseline network, The parameters and structure used in the baseline model ResNet18 are consistent with the original paper. In this experiment, we kept the parameters of the three blocks fixed. In the FFTB, the window size and number of layers are set to 2 × 2 and 4, respectively. In WF, the encoding and decoding feature maps have a ratio of 2:8. In the FRN, the spatial convolution kernel sizes are {7,7}; {5,5}; {3,3}, with a channel ratio of 2:1:1. To demonstrate that our parameter settings achieve the optimal state for each network, we configured different parameters for each network, as shown in

Table 2. Experiment of network parameter settings.

Modules	Window Size	Layers	Feature Map Ratio	Spatial Convolution Kernel	Channel Ratio	IoU
FFTB	2 × 2	2	Nan			70.93
	4 × 4	4				71.18
	2 × 2	4				71.34
	4 × 4	2				70.81
WF	Nan		1:9	Nan		71.67
			2:8			71.83
			7:3			70.64
			8:2			70.49
FRN	Nan			{7,7};{5,5};{3,3}	1:1:2	73.04
				{7,7};{5,5};{3,3}	2:1:1	73.13
				{7,7};{7,7};{5,5}	2:1:1	72.96

. To ensure a fair and unbiased comparison, all ablation studies were carried out without employing any test-time augmentation strategies. The resulting findings are presented in

Table 3. Ablation experiment of FireFormer on the FLAME dataset.

Model’s Name	FFTB	WF	FRN	IoU	F1	OA
Baseline	F	F	F	67.18	80.37	99.76
Baseline + FFTB	T	F	F	71.34	83.27	99.82
Baseline + FFTB + WF	T	T	F	71.83	83.61	99.82
Baseline + FFTB + WF + FRN	T	T	T	73.13	84.48	99.83

.

4.1.1. Effect of Forest Fire Transformer Block (FFTB)

By incorporating three FFTBs into the baseline architecture, we created Baseline + FFTB. As shown in Table 1, we can observe that the introduction of the FFTB effectively enhances the segmentation capability of the baseline. Specifically, the IoU is improved by 4.16%, the F1 score is improved by 2.9%, and the OA is improved by 0.06%. This improvement demonstrates that the network can learn more intricate image details in the FFTB dual-branch mode. Figure 8c demonstrates that the FFTB module significantly enhances the baseline’s ability to perceive edges, resulting in more accurate segmentation results at the edges.

4.1.2. Effect of Weight Fuse (WF)

The data presented in Table 1 indicates that the inclusion of WF into the Baseline + FFTB model, namely, Baseline + FFTB + WF, leads to an increase of 0.49% in the IoU and a 0.34% improvement in the F1 score, while the OA remains unchanged. The objective of WF is to fuse feature maps of varying granularities through Weighted Fusion. In Figure 8d, the upper right corner is particularly significant, indicating that through the WF module, the model can selectively learn more accurate features from multiple scales of feature maps.

4.1.3. Effect of Feature Refinement Net (FRN)

During the ablation experiment on the FRN, we incorporated it into Baseline + FFTB + WF, and created Baseline + FFTB + WF + FRN; it was observed that compared to the former, there was a 1.3% increase in IoU, a 0.87% increase in the F1 score, and a 0.01% increase in OA. As depicted in Figure 8e, the network demonstrates an enhanced edge segmentation capability for objects when FRN blocks are incorporated, as opposed to the results obtained without FRN blocks.

4.2. Comparison with Other Semantic Segmentation Methods

In our study, we conducted a comparison between our proposed FireFormer model and several existing methods, namely, FCN [54], U-Net [55], DeeplabV3 [56], TransUNet [44] and SwinUNet [57]. The initial methods mentioned are based on conventional CNN networks, while SwinUNet solely utilizes the ViT architecture. On the other hand, TransUNet represents a hybrid structure that combines both CNN and ViT in the encoder stage. In contrast, FireFormer, our proposed method, adopts a hybrid architecture comprising a CNN-based encoder and a Transformer-based decoder.

4.2.1. Comparison with Different Encoder–Decoder Combinations on FLAME Dataset

In order to highlight the advantages of our hybrid architecture, FireFormer, in efficient semantic segmentation, we conducted comparative experiments on the FLAME dataset, comparing it with UNet [55], SwinUNet [57] and TransUNet [44]. As depicted in

Table 4. Comparison experiment of different encoder–decoders on the FLAME dataset.

Method	Backbone	Encoder	Decoder	FLOPs (G)	Memory (MB)	IoU
UNet	-	CNN	CNN	792.72	3180.00	62.63
SwinUNet	Swin-T	Trans	Trans	123.69	3355.28	53.50
TransUNet	ViT-R50	Trans	CNN	512.51	4880.33	74.13
FireFormer(ours)	Res18	CNN	Trans	42.01	1203.92	73.13

, our proposed FireFormer exhibits competitive accuracy and complexity. Specifically, FireFormer achieves a remarkable 10.5% improvement in IoU compared to the pure CNN-based U-Net. When compared to the pure ViT-based SwinUNet, FireFormer attains a higher accuracy, while reducing memory consumption by 65%. Although the hybrid structure of TransUNet slightly surpasses FireFormer by 1% in accuracy, FireFormer consumes only 0.24 times the memory of TransUNet and exhibits a nine times faster inference speed.

4.2.2. Comparison with Other Methods on FLAME Dataset

The assessment of network complexity and speed is of utmost importance, particularly in real-time fire segmentation applications. In the official FLAME dataset evaluation, we compared FireFormer with other networks in terms of IoU, GPU memory consumption, complexity, parameters, and speed. The comparative results are depicted in

Table 5. Comparison of models memory, parameters, Flops, speed and FPS.

Method	Backbone	Memory (MB)	Params (M)	FLOPs (G)	Speed (ms)	FPS
U-Net	-	3184.00	24.79	792.82	211.95	4.72
FCN	Res50	5132.12	32.95	555.45	150.92	6.63
SegNet	VGG16	5384.00	29.44	642.70	104.83	5.46
RefineNet	Res50	2876.00	27.31	125.97	70.77	14.13
DeconvNet	VGG16	-	251.80	866.61	256.48	3.92
BiseNet	Res18	1088.15	27.01	244.15	123.10	8.12
DeepLabV3	VGG16	2122.04	33.49	100.81	85.11	11.75
HRNet	W32	4278.75	29.54	181.85	109.77	9.11
MANet	Res18	4245.25	35.86	309.77	120.48	8.31
SegFormer	MiT-B1	2690.60	13.68	52.97	104.38	8.58
BuildFormer	Res50	6377.00	40.52	466.36	159.24	6.28
A2FPN	-	985.00	12.16	52.97	31.16	32.09
UNetFormer	Res18	1124.25	11.72	46.95	26.15	38.24
PIDNet	-	1206.31	28.54	89.15	69.16	14.46
FireFormer	Res18	1203.92	11.66	42.01	26.11	38.29

. In comparison to A2FPN [52], which exhibits the lowest GPU memory consumption, FireFormer achieves a faster inference speed by 5.05 ms. When compared to the leading PIDNet [58], FireFormer demonstrates fewer parameters and more than twice the inference speed. While maintaining a high level of accuracy, FireFormer showcases commendable performance in terms of network inference speed and complexity. This balance between accuracy and network performance serves as a validation for the remarkable effectiveness of our proposed FFTB and FRN network modules. Indeed, real-time fire segmentation applications necessitate not only high network complexity and speed but also precise segmentation accuracy. To address this requirement, we conducted meticulous comparative experiments on the FLAME dataset. FLAME is a comprehensive forest fire scene segmentation dataset, comprising forest fire images captured from various heights and angles. Consequently, achieving high scores on this dataset presents certain challenges. We trained multiple advanced and efficient segmentation networks and conducted an extensive comparison of the results on the official FLAME test set. The detailed results are presented in

Table 6. Comparison of segmentation accuracy on FLAME dataset.

Method	Backbone	F1	IoU	OA
U-Net	-	77.02	62.63	99.71
FCN	Res50	84.37	72.97	99.82
SegNet	VGG16	77.27	62.95	99.71
RefineNet	Res50	84.35	72.93	99.83
DeconvNet	VGG16	78.51	64.62	99.75
BiseNet	Res18	83.74	72.03	99.82
DeepLabV3	VGG16	84.31	72.87	99.83
HRNet	W32	82.68	70.47	99.91
MANet	Res18	83.91	72.28	99.83
SegFormer	MiT-B1	79.95	66.60	99.77
BuildFormer	Res50	84.18	72.68	99.82
A2FPN	-	81.78	69.18	99.78
UNetFormer	Res18	83.26	71.33	99.81
PIDNet	-	83.74	72.03	99.82
FireFormer	Res18	84.48	73.13	99.83

. From the table, it can be observed that our proposed FireFormer achieves the highest IoU and F1 score, indicating its superior performance in accurately segmenting forest fire scenes. The visualization of the segmentation results is shown in Figure 9, from which it can be construed that FireFormer’s segmentation results are more accurate, especially at the edges of small fire points.

Our proposed FireFormer achieved exceptional performance in the accurate segmentation of forest fire scenes. With an impressive IoU of 73.13%, FireFormer also attained the highest scores in terms of F1 (84.48%). Comparatively, FireFormer outperformed traditional CNN-based segmentation networks such as U-Net, FCN and SegNet by 10.5%, 0.16% and 10.18% in IoU, and by 7.46%, 0.11% and 7.21% in F1, respectively. Furthermore, FireFormer surpassed the recently proposed efficient ViT-based network, PIDNet, by 1.1% in IoU and 0.74% in F1. Even in challenging scenarios where the segmentation of small fire spots is difficult, FireFormer demonstrated superior performance, solidifying its crucial significance in addressing fire segmentation applications.

4.3. Test the Scene Adaptability of FireFormer and Explore Carbon Emissions

To test the scene adaptability of FireFormer, we applied the FireFormer model trained on the FLAME dataset to the FLAME2 dataset. Based on the obtained detection results, we conducted an interesting exploration to estimate the carbon emissions caused by flames using FireFormer’s flame activity detection capability.

As shown in Figure 10, FireFormer detects four active fire points in each sample of the FLAME2 dataset. We explored the carbon emissions of these fire points based on the results detected by FireFormer. Since the images in the FLAME2 dataset are captured from a vertical perspective, we approximated the segmentation result of the flames to be the size of the burning area and calculated the carbon emissions for this perspective using Formula (8).

$$E=A\times B\times R\times F\times O$$

(8)

where E represents the carbon emissions resulting from the fires, $A$ represents the area of the burned forest land (this value will change as the flame burns), and $B$ represents the biomass per unit area of the burning area. After carefully reviewing the official website of Kaibab Forest, we have found the officially released data. Therefore, we have determined that the biomass per unit area is 35.49. $R$ represents the carbon content in the biomass. This value should vary depending on the composition of the forest. Due to variations in vegetation types, forest age and composition, the conversion rates exhibit significant disparities. Moreover, the availability of conversion rates for various vegetation types is limited. Therefore, it is common practice to adopt the internationally recognized conversion rate of 0.5. $F$ represents the combustion ratio, which we have set at 0.15 for our calculations. Lastly, $O$ represents the oxidation coefficient, which we have taken as 0.9.

An important point to note is that in exploring the carbon emissions of the fire points, it is inevitable that errors will be introduced. The reason for these errors lies in our use of fixed average values, rather than on-site measurements, when calculating carbon emissions. Our calculation of $\mathrm{A}$ will introduce errors as we use the segmented result of the flame as the burning area, and due to the influence of factors such as wind and smoke during the combustion process, the value of A will be underestimated. Furthermore, during the combustion process of the flame, it is difficult to determine the $R$, $F$ and $O$ in real time. While we have adopted internationally recommended values in our calculations, these values should be specifically measured for the actual conditions of forest fires. Based on the detection results of FireFormer on the FLAME2 dataset and the calculation of carbon emissions from active fire points using Formula (8), we obtained Figure 11, which illustrates the trends in carbon emissions for four active fire points. We plotted the carbon emission data generated by the four fire sources in Figure 11 into

Table 7. The emissions from each fire point and estimation accuracy.

Frame	0	5	10	15	20	25	30	35	40	45	50	55	60
Point 1 Carbon emissions (kgC)	23.86	24.69	25.61	25.41	23.09	24.49	24.91	23.88	27.96	26.01	25.154	26.91	27.31
Point 2 Carbon emissions (kgC)	30.81	18.86	26.86	24.92	24.98	21.94	22.85	24.40	31.44	26.38	32.77	22.04	15.93
Point 3 Carbon emissions (kgC)	12.77	12.16	9.53	4.21	5.13	5.63	0.304	0	4.59	2.96	4.61	7.42	9.01
Point 4 Carbon emissions (kgC)	1.90	1.69	2.13	2.57	2.58	2.42	2.18	2.45	3.06	2.45	2.11	2.45	2.90
Acc (%)	87.19	91.02	84.36	89.92	85.11	84.70	66.21	71.83	65.35	83.96	92.63	86.29	82.04

and evaluated our experimental data using the accuracy metric. An analysis of the accuracy in the table reveals that our proposed method performs well when the UAV’s vision is unobstructed by smoke during fire incidents. However, significant estimation errors in carbon emission occur when the UAV’s vision is obscured by smoke from the fire. We attribute this issue to incorrect identification by the semantic segmentation algorithm when smoke obscures the fire sources, thereby leading to errors in carbon emission estimation.

5. Discussion

In this work, we used low-altitude remote sensing images of early forest fires based on unmanned aerial vehicles as data to develop a novel CNN–Transformer hybrid forest fire identification algorithm.

Firstly, the impetus for this work originated from our observation that within the field of remote sensing, there is a limited but significant amount of research on monitoring fires using UAV remote sensing imagery [59,60]. These efforts have advanced fire detection methodologies within the domain of remote sensing. LUFFD-YOLO [61] introduced an algorithm for wildfire detection that is based on attention mechanisms and the fusion of multi-scale features, enabling the automated identification of fire regions in remote sensing images. The authors of [56], building upon the Deeplabv3+ framework, have achieved the detection of fires in UAV imagery. However, we have noted that the aforementioned methods are predominantly founded on visual models for object recognition, which are only capable of outlining the fire areas within the remote sensing images. This can lead to the inclusion of pixels that do not actually belong to the fire region. Therefore, the FireFormer we propose is constructed on a semantic segmentation visual model, facilitating pixel-level detection.

Secondly, the reason for proposing FireFormer is that most current fire detection algorithms still utilize satellite imagery as data [62,63]. Rida Kanwal and colleagues [64] employed a CNN neural network to achieve fire segmentation based on satellite remote sensing images. Mukul Badhan and colleagues [65] used VIIRS satellite remote sensing images to implement more accurate large-scale fire identification. We believe that satellite images have a relatively low spatial resolution and long revisit cycles, making it difficult to clearly and promptly reflect fire conditions on the ground. For the early small flames of forest fires, relying solely on satellite images is even more challenging. Unmanned aerial vehicles, as low-altitude remote sensing image acquisition devices, can more quickly and rapidly obtain real-time fire conditions.

To address the shortcomings we identified in the existing work during our research process, we proposed FireFormer. Considering the high resolution of remote sensing images, we introduced a lightweight Transformer framework, the Forest Fire Transformer Block. The experimental results have demonstrated that this module enhances the network’s ability for real-time fire monitoring. We believe that FireFormer can be deployed on unmanned aerial vehicle systems, enabling ground detection personnel to respond to forest fires more conveniently and accurately.

However, our work still has two areas of insufficiency. First, the algorithm we proposed is a supervised algorithm, which requires a significant amount of human labor to annotate remote sensing images in order to successfully complete the model training. Second, we believe that there is still room for improvement in the feature extraction module of FireFormer, and more advanced feature extraction methods can achieve more accurate recognition accuracy. In the future, we are considering using self-supervised learning methods for model training to alleviate the predicament that fire recognition algorithms require a large amount of annotated data for training. In addition, we will continue to develop new feature extraction modules to achieve more accurate fire detection.

6. Conclusions

This work aims to extract valuable information from early wildfire images obtained from low-altitude unmanned aerial vehicles (UAVs), to achieve higher flame segmentation accuracy with a smaller computational cost. Based on our findings, traditional CNN segmentation methods and Transformer-based segmentation methods struggle to strike a good balance between real-time performance and accuracy when faced with wildfire images captured by UAVs. To address this problem, we designed a novel real-time semantic segmentation network with a hybrid structure that incorporates our innovative Forest Fire Transformer Block (FFTB) and Feature Refinement Net (FRN). Specifically, in the Lo-Fi branch of FFTB, we designed a bi-paths attention (BPA) mechanism that allows the network to perform the global modeling of feature maps with a lower computational cost, effectively reducing the computational load and improving real-time performance. Additionally, in segmentation tasks, feature maps undergo multiple downsampling and upsampling, resulting in the loss of edge details. Therefore, we designed the FRN, which was validated through experiments and result visualization. Finally, we proposed an approximate calculation method for estimating carbon emissions from early forest fires from the perspective of low-altitude UAVs based on the segmentation results of FireFormer. In future work, we will continue to research better image feature encoders to replace the concise encoding in FireFormer, to achieve higher segmentation accuracy.