Object Extraction-Based Comprehensive Ship Dataset Creation to Improve Ship Fire Detection

Abstract

The detection of ship fires is a critical aspect of maritime safety and surveillance, demanding high accuracy in both identification and response mechanisms. However, the scarcity of ship fire images poses a significant challenge to the development and training of effective machine learning models. This research paper addresses this challenge by exploring advanced data augmentation techniques aimed at enhancing the training datasets for ship and ship fire detection. We have curated a dataset comprising ship images (both fire and non-fire) and various oceanic images, which serve as target and source images. By employing diverse image blending methods, we randomly integrate target images of ships with source images of oceanic environments under various conditions, such as windy, rainy, hazy, cloudy, or open-sky scenarios. This approach not only increases the quantity but also the diversity of the training data, thus improving the robustness and performance of machine learning models in detecting ship fires across different contexts. Furthermore, we developed a Gradio web interface application that facilitates selective augmentation of images. The key contribution of this work is related to object extraction-based blending. We propose basic and advanced data augmentation techniques while applying blending and selective randomness. Overall, we cover eight critical steps for dataset creation. We collected 9200 ship fire and 4100 ship non-fire images. From the images, we augmented 90 ship fire images with 13 background images and achieved 11,440 augmented images. To test the augmented dataset performance, we trained Yolo-v8 and Yolo-v10 models with “Fire” and “No-fire” augmented ship images. In the Yolo-v8 case, the precision-recall curve achieved 96.6% (Fire), 98.2% (No-fire), and 97.4% mAP score achievement in all classes at a 0.5 rate. In Yolo-v10 model training achievement, we got 90.3% (Fire), 93.7 (No-fire), and 92% mAP score achievement in all classes at 0.5 rate. In comparison, both trained models’ performance is outperforming other Yolo-based SOTA ship fire detection models in overall and mAP scores.

1. Introduction

The detection and management of fires aboard ships is a critical aspect of maritime safety. With the increasing reliance on automated systems for monitoring and surveillance, the development of robust ML models for ship fire detection has become essential. However, the effectiveness of these models is often constrained by the availability and diversity of training data. In ocean environments, ship fire incidents are rare and challenging to capture, due to several factors related to both the nature conditions of the maritime environment and the operational characteristics of ships, resulting in a scarcity of labeled data for model training. These factors include the hazy ocean scenes and relatively low occurrence of ship fires, from which we can mostly gather certain amounts of ship fire images from internet sources. Fortunately, ship fires are relatively rare due to the extensive safety measures implemented on board. In addition, when ship fires occur, access to the site is often heavily restricted. For safety reasons, only trained personnel are allowed near the incident. This limitation on access reduces the likelihood of images being taken and subsequently shared. Furthermore, many ship fires occur in remote areas of the ocean, far from the reach of media and casual observers. Therefore, the remote location of many ships, such as in the middle of the ocean, makes it difficult to capture images and to reach the site promptly. However, it cannot be denied that there are a certain amount of ship and ship fire images, such as in the Google images search engine. Regarding the World Shipping Council statistics (WSC) [1], the number of container ship fires are in an upward trend, and the estimation of ship fire occurrences is averaged every 60 days across. For the reasons above, and in order to address the scarcity, we aimed to create a ship fire dataset by augmentation techniques. Numerous studies have assessed the effectiveness of data augmentation to leverage well-known academic image datasets.

The success of ML models, especially in computer vision (CV) tasks, is heavily dependent on the quality and quantity of training data. It is widely recognized that larger datasets tend to enhance the performance of deep learning (DL) models. Many researchers [2,3] highlight that models trained on extensive datasets generally exhibit superior accuracy and robustness compared to those trained on smaller datasets. This is because larger datasets provide a broader representation of possible scenarios, reducing the likelihood of overfitting and improving the model’s ability to generalize to new data.

A common issue related to small datasets in CV is that trained models struggle to generalize data from validation and test sets [4]. This lack of generalizations is often due to overfitting, where the model learns to perform well on the training data but fails to adapt to new, unseen data. This issue is exacerbated in domains where data collection is challenging or expensive, leading to a reliance on small datasets. Several advanced techniques have been developed to address the limitations of smaller datasets when developing DL models [5,6,7,8,9]. These include techniques like dropout regularization [10,11,12], batch normalization [13,14] and transfer learning [15,16,17]. Dropout involves randomly dropping units during training, which helps to prevent overfitting by ensuring that the model does not rely too heavily on any single node. In batch normalization, the inputs of each layer are normalized to have a mean of zero and a standard deviation of one, which helps to stabilize the training process and improve the convergence speed. Transfer learning involves using a pre-trained model on a large dataset and fine-tuning it on a smaller set of task-specific datasets. This approach leverages the features learned by the model on the large dataset, which can significantly benefit when labeled data are scarce. Wang et al. [18], for example, explored and compared multiple solutions to the problem of data augmentation, focusing on image classification tasks and experiments. Their study underscores the effectiveness of data augmentation techniques in enhancing model performance. However, assembling extensive datasets presents a formidable challenge, primarily due to the substantial manual effort required for data collection and annotation. This is particularly true for specialized tasks, such as detecting ship fires, where images are rare and often difficult to capture. To evaluate the impact of data augmentation on classification accuracy, it is beneficial to perform a comparative analysis of widely recognized image classification architectures [19,20,21,22,23].

The selected datasets, CIFAR-10/100 and SVHN, are widely applied benchmarks in the field of CV, providing a comprehensive basis for evaluating the effectiveness of data augmentation, as can be seen in

Table 1. Comparative analysis of image classification enhancement by data augmentation.

Dataset	Model Name	Before Augmentation (%)	After Augmentation (%)	Accuracy Improvement by (%)
CIFAR-10	DenseNet	94.15	94.59	0.44
	Wide-ResNet	93.34	93.67	1.33
	Shake-ResNet	93.7	94.84	1.11
CIFAR-100	DenseNet	74.98	75.93	0.95
	Wide-ResNet	74.46	76.52	2.06
	Shake-ResNet	73.96	76.76	2.80
SVHN	DenseNet	97.91	97.98	0.07
	Wide-ResNet	98.23	98.31	0.80
	Shake-ResNet	98.37	98.40	0.30

.

These datasets have given many researchers a good advantage to perform many experiments and to compare the performance results of data augmentation techniques. Therefore, data augmentation is a powerful technique in ML to train DL models. This process enhances the diversity and volume of the training dataset without the need for additional manual labeling. By having a bigger dataset, the model can learn better and avoid overfitting. In many cases, especially in fields like autonomous driving and medical imaging, collecting data from all possible real-world conditions is impractical. In addition, by augmentation, we can address class imbalance by generating more examples of not represented classes.

For instance, Esteva et al. [24] mention that D-CNN can perform significantly well for medical image analyses, such as skin lesion classification tasks [25]. Research experiments show that in the case of CIFAR-100, they achieved an increase in performance from 66% to 73%. As mentioned in

Table 2. Experimental performance based on feature vs. input space extrapolation on MNIST and CIFAR-10 datasets [26].

Model	MNIST	CIFAR-10
Baseline	1.093 ± 0.057	30.65 ± 0.27
Baseline + input space affine transformations	1.477 ± 0.068	-
Baseline + input space extrapolation	1.010 ± 0.065	-
Baseline + feature space extrapolation	0.950 ± 0.036	29.24 ± 0.27

, augmentation techniques applied on feature and input space have increased the accuracy in the application of MNIST and CIFAR-10 datasets.

Several studies have proved the advantage of data augmentation in ML to train DL models. In this paper, we propose comprehensive data augmentation techniques based on image blending. By automatically removing the background from source images and integrating these into various target image backgrounds, we can create a vast and diverse dataset that simulates numerous real-world conditions. Figure 1 is an example of a data augmentation illustration with basic and advanced data augmentation techniques combined, which is our proposed approach.

As previously mentioned, D-CNNs perform well in CV tasks with big data applications. This research focused on handling data scarcity related to ocean vessels. We augmented ship fire images by blending source and target images, where the source image is a ship on fire and the target images are various ocean environment scenes. To augment, backgrounds of ship fire images are removed and then attached to source images. During blending, we applied basic and advanced augmentation approaches. In Section 3, we describe in more detail the methods that we utilized in this work. We anticipate that applying augmentation techniques, such as noise injection and scaling methods, for example, will significantly enhance the robustness of our generated dataset. These techniques enable us to create hazy ocean environmental images and simulate faraway distance localization, which in turn aid in improving the detection and recognition of small objects. To see the proposed method’s assistance in better image classification, especially ship fire detection, we fine-tuned Yolo (You Only Look Once) models, such as Yolo-v8 and Yolo-v10, and compared our achievements with other Yolo-based methods in Section 5.

The main contributions of this research are as follows:

• Collection of ship and ship fire images from internet sources.
• Object-extracted blending of collected images with basic and advanced approaches. To note, our study will not cover all techniques of basic and advanced approaches.
• Open-source Gradio web application for data augmentation that we applied in our work. Application can be used for the creation of other limited datasets.

To provide a comprehensive understanding of our research on data augmentation for the ship fire dataset, we have structured the sections of this paper accordingly as follows:

Section 2 is about related research works. This section provides a review of existing literature on data augmentation, focusing on various methods, datasets, and algorithms that have been studied. Section 3 explains the contributions of this study. This part details the unique contributions of our research, specifically focusing on the techniques we employed for data augmentation in the context of ship fire images with blending to various ocean environment images. The key point is to include relevant data and techniques, such as background removal and blending with various ocean images by random location of the source image in the target image. Blending methods include random location, noise injection, rotation, cropping, scaling, and color and contrast change applications. In Section 4, we highlight the development and functionality of the Gradio web application, created to facilitate the testing and visualization of our data augmentation techniques. The final section, Section 5, summarizes the potential research directions and key contributions.

2. Related Work

As illustrated in Figure 1 above, image augmentation techniques are broadly categorized into two main types: basic and advanced approaches. Traditional methods fall under the image manipulation category, encompassing basic augmentation methods. Among these, geometric distortions or deformations, such as affine transformations, are widely used to increase the number of training samples for DNNs [27]. These transformations help to balance sizes [28] and improve the efficiency of the models [29,30,31,32,33]. The evolution of data augmentation techniques in the context of ML, particularly DL, has been a dynamic journey marked by significant innovations [34,35]. Data augmentation involves creating additional training data from existing data to improve model performance, robustness, and generalization. The techniques have evolved from simple transformations to sophisticated generative methods.

2.1. Basic Image Manipulation

Flipping: Horizontally or vertically flipping images to create mirror images, helps the model recognize objects from different perspectives. This approach can help the model recognize objects from different perspectives. For example, a horizontally flipped image can simulate a left-hand view of an object originally viewed from right. Moreover, many objects and scenes in real-world applications have symmetrical properties. By training on flipped images, the model becomes better at recognizing these symmetrical patterns [35,36,37,38].

Rotation: Rotation involves turning the image around its center by a certain angle. Common angles used for rotation are commonly 90, 180, and 270 angles. This approach helps models learn to recognize objects regardless of their orientation. Therefore, we also applied this technique because ships can appear at varied angles in sea waves [39,40].

Scaling: Scaling changes the size of the image by either enlarging or reducing it while maintaining the aspect ratio. Objects in images can appear at various scales due to differences in distance or zoom level. Training on scaled images ensures that the model performs well across different resolutions, which is crucial for applications like image recognition on devices with varying screen sizes [41,42].

Color space transformations: These techniques involve modifying the color properties of images by enhancing the contrast of images by spreading out the most frequent intensity values, modifying the brightness of images to simulate different lighting conditions, and altering the intensity of colors to simulate different environmental conditions [40]. Chatfield et al. [43] observed an approximate 3% decrease in classification accuracy when using grayscale images compared to Red Green Blue (RGB) images in their experiments on the ImageNet [44] and PASCAL VOC [45] datasets. This finding underscores the significance of color information in image classification tasks. Their research highlights that while grayscale images simplify the data and reduce computational requirements, they also lead to a loss of critical color-related features that can be essential for accurate classification. This accuracy drop is particularly relevant in datasets like ImageNet and PASCAL VOC, which contain diverse and complex images where color information plays a crucial role in distinguishing between different classes.

2.2. Generative Models for Data Synthesis

The introduction of generative models, particularly generative adversarial networks (GANs), marked a significant advancement in DL frameworks. Developed by Goodfellow et al. [46] in 2014, GANs comprise tow primary neural networks: the generator and the discriminator. The generator network is designed to produce synthetic data samples that closely mimic real data. By augmenting the training dataset with varied synthetic samples, GANs help mitigate issues related to class imbalance and limited data availability. For example, in medical imaging, GANs are used to generate synthetic scans that mimic real patient data, aiding in the development of diagnostic models. It achieves this by inputting random noise and transforming it into plausible data samples. During the training process, the generator’s objective is to create samples that can deceive the discriminator into recognizing them as genuine. Concurrently, the discriminator’s role is to distinguish accurately between real and synthetic samples. The core aim of GANs is to train the generator’s distribution, denoted as $P_g\left(x\right),$ to closely approximate the real data distribution $P_{reah}\left(x\right)$ over the dataset $\left(x\right)$ within a minimax adversarial framework. Applications of GANs have found numerous applications across various domains, all related to the generation of new forms of data. For example, GANs have found numerous applications across various domains, including image generation, data augmentation, image-to-image translation, super resolution, and text-to-image synthesis [47].

The remarkable performance of GANs has led to heightened interest in their application for data augmentation. The GAN framework has been extended to enhance the quality of samples produced, particularly when integrated with variational auto-encoders (VAEs) [48]. While GANs have traditionally been employed to generate realistic 2D visual content, innovative applications have emerged. For instance, a study successfully trained a GAN model directly on 3D shapes represented as point clouds, allowing the generator networks to comprehend the inherent structure of 3D shapes and generate complete forms from latent codes. In the domain of text-to-image generation, Zhang et al. [49] were pioneers, demonstrating the capability of GANs to generate images of varying sizes from textual descriptions. Xu et al. [50] advanced this concept by incorporating an attention mechanism within GANs, facilitating the generation of high-resolution images guided by word-level conditioning inputs. This attention mechanism ensures that the generated images are more closely aligned with the given textual descriptions, thereby enhancing the fidelity and relevance of the synthesized images. Additionally, Zhang et al. [51] introduced StackGAN, a framework that progressively refines the resolution of synthesized images through a series of stacked GANs. This iterative enhancement approach enables the generation of high-resolution images with increasingly fine details, addressing the challenge of producing visually coherent and detailed images at higher resolutions.

GANs also facilitate semi-supervised learning by generating labeled data from a small set of real samples. This approach reduces the dependency on large, labeled datasets, making it feasible to develop accurate models with limited labeled data [52,53,54]. GANs can effectively serve as an oversampling technique to address class imbalance issues. Class imbalance, where certain classes are underrepresented in a dataset, poses a significant challenge in ML. Traditional oversampling techniques, such as SMOTE (synthetic minor oversampling technique), create new samples by interpolating between existing ones. However, GANs offer a more sophisticated approach by generating entirely new, realistic samples for the minority class. For example, Lim et al. [55] demonstrated the utility of GAN-generated samples in unsupervised anomaly detection. By oversampling rare normal samples, those with a low occurrence probability, GANs can reduce the false positive rate in anomaly detection tasks. This is achieved through the use of the adversarial auto-encoder (AAE) framework, as proposed by Makhzani et al. [56].

2.3. Style Transfer Approaches

GANs have been employed to transfer styles (ST) between domains, such as converting photos to paintings or transforming daytime images to nighttime scenes. While the neural style transfer (NST) [57] is widely recognized for its artistic applications, it also proves to be an effective tool for data augmentation. NST stands out as one of the most visually striking demonstrations of DL capabilities. The core concept involves manipulating the representations of images created by CNNs. CV models trained on large-scale image datasets often exhibit biases towards local object features such as color and texture [58]. To mitigate this limitation, several ST approaches have been developed to encourage models to focus more on shape information [59,60]. These techniques have been employed to tackle domain adaptation challenges. In such cases, the NST network extracts useful visual features from images with the desired target appearance and applies them to target images that lack this appearance. For example, a study [61] utilized the NST model to transfer images taken during the daytime to appear as nighttime images, a scenario where labeled data are typically scarce. Introducing a fixed set of styles can lead to a new form of bias within the augmented dataset. To address this, methods like StyleAugment and StyleGan [62,63] advocate for the random selection or target selection of styles. This approach either ensures that the augmented images reflect a broader range of variations or are target-oriented, thereby enhancing the model’s ability to generalize.

3. Proposed Methods, Model Architecture

3.1. Background Removing

From Section 2, we have seen that by artificially enlarging the training dataset through various transformations, the application of data augmentation is becoming significantly helpful to develop a robust model by mitigating overfitting, covering scarce datasets, improving model generalization, and boosting overall accuracy. In this research, we propose a data augmentation approach that involves blending source and target images, applying a suite of augmentation techniques during the blending process. These techniques include random application of background-removed ship fire images while blending to source ocean images. Random application is crucial in every technique that we use in blending. These techniques include random resizing, cropping, blurring, positioning, color change, and usual flipping approaches.

Background removal, a specific form of augmentation, has gained prominence for its ability to isolate objects of interest from non-essential elements in an image. The image background removal library we use in our project was developed by Daniel Gatis. Removing the background from images is a technique used in image processing where the non-essential parts of an image are excluded, leaving only the foreground objects of interest. This process typically involves distinguishing the primary subject from the background using algorithms such as thresholding, edge detection, segmentation, and deep learning-based models. In our approach, the initial step was set to target image, where we focused on the primary subject and removed its background. For a given image I, represented as a matrix of pixel values:

$$\mathrm{I}=\left\{{\mathrm{I}}_{\mathrm{i},\mathrm{j}}\right|\mathrm{i}\in \left[1,\mathrm{M}\right],\mathrm{j}\in [1,\mathrm{N}]\}$$

(1)

where M and N are the dimensions of the image, and ${\mathrm{I}}_{\mathrm{i},\mathrm{j}}$ denotes the pixel value at position ($\mathrm{i},\mathrm{j}$).

Next, a segmentation algorithm is applied to divide the image into regions, producing a segmentation mask S:

$$\mathrm{S}=\left\{{\mathrm{S}}_{\mathrm{i},\mathrm{j}}\right|\mathrm{i}\in \left[1,\mathrm{M}\right],\mathrm{j}\in [1,\mathrm{N}]\}$$

(2)

where ${\mathrm{S}}_{\mathrm{i},\mathrm{j}}$ is a binary value indicating whether pixel ($\mathrm{i},\mathrm{j}$) belongs to the foreground (1) or the background (0).

After that, the processed image I’, after background removal is obtained by element-wise multiplication of the original image I and the segmentation mask S:

$${\mathrm{I}}^{\prime }=\mathrm{I}\circ \mathrm{S}$$

(3)

where, ∘ denotes the Hadamard product (element-wise multiplication).

The blending image function is designed to seamlessly overlay a ship image onto a background image at a specified position, ensuring proper handling of transparency. This function accepts three parameters, as can be seen in

Table 3. OE source image blending with target image formulation.

Input	background, ship_image, position
Output	blended_image
background	background.convert(“RGBA”)
mask	ship_image.split() [3]
background.paste(ship_image, position, mask)
return background

.

The RGBA conversion ensures whether the background image supports transparency by including the alpha channel. The alpha channel acts as a mask in its represented channels to delineate the transparent and opaque regions of the ship image. The mask confirms that only the non-transparent parts of the ship image are pasted onto the background, allowing for smooth blending and preserving the visual integrity of both images.

3.2. Image Resizing and Positioning

The input image is converted to the RGBA mode to ensure the image has four channels, such as red, green, blue, and alpha—transparency. The inclusion of the alpha channel is particularly important for preserving transparency information, which is crucial when compositing the resized image onto different backgrounds. The dimensions of the resized image are computed by multiplying the original width and height of the ship image by the scale factor value. This step involves the use of integer typecasting to ensure the new dimensions are whole numbers. The image is resized to the newly calculated dimensions using the LANSZOS resampling algorithm, which is a popular method for image scaling.

To simulate different scales and perspectives, the isolated object is resized using a scaling factor. This is randomly selected within a predefined range to ensure variability. The resized object is then positioned onto the source image at random coordinates, creating a natural appearance of the object within different contexts.

Scaling by a factor of s:

$${\mathrm{I}}_{\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{e}\mathrm{d}}\left({\mathrm{x}}^{\prime },{\mathrm{y}}^{\prime }\right)=\mathrm{I}\left({\displaystyle \frac{\mathrm{x}}{\mathrm{s}}},{\displaystyle \frac{\mathrm{y}}{\mathrm{s}}}\right)$$

(4)

where ($\mathrm{x},\mathrm{y}$) are the coordinates in the original image, and (${\mathrm{x}}^{\prime },{\mathrm{y}}^{\prime }$) are the coordinates in the scaled image.

A more formal representation of the LANCZOS resampling process for a discrete set of pixels as follows in the equation below:

$${\mathrm{I}}^{\prime }\left({\mathrm{x}}^{\prime }\right)={\sum }_{\mathrm{x}}\mathrm{I}\left(\mathrm{x}\right)·\mathrm{L}({\mathrm{x}}^{\prime }-\mathrm{x})$$

(5)

-

$\mathrm{I}\left(\mathrm{x}\right)$ is the original pixel value at position $x$.

-

${\mathrm{I}}^{\prime }\left({\mathrm{x}}^{\prime }\right)$ is the resampled pixel value at the new position $x^{\prime }$.

-

$\mathrm{L}({\mathrm{x}}^{\prime }-\mathrm{x})$ is the Lanczos kernel applied to the distance between the original and new pixel positions.

3.3. Random Flip, Rotation and Combination

The input image was rotated at random angles to simulate different orientations. It was then combined with the target background image using blending techniques that ensure smooth integration and natural appearance. This step resulted in a final augmented image that was saved for further usage.

Horizontal flip:

$${\mathrm{I}}_{\mathrm{f}\mathrm{l}\mathrm{i}\mathrm{p}\mathrm{p}\mathrm{e}{\mathrm{d}}_{\mathrm{h}}}\left(\mathrm{x},\mathrm{y}\right)=\mathrm{I}(\mathrm{W}-\mathrm{x}-1,\mathrm{y})$$

(6)

Vertical flip:

$${\mathrm{I}}_{\mathrm{f}\mathrm{l}\mathrm{i}\mathrm{p}\mathrm{p}\mathrm{e}{\mathrm{d}}_{\mathrm{v}}}\left(\mathrm{x},\mathrm{y}\right)=\mathrm{I}(\mathrm{x},\mathrm{H}-\mathrm{y}-1)$$

(7)

where I($\mathrm{x},\mathrm{y}$) is the pixel at coordinates ($\mathrm{x},\mathrm{y}$) in the original image, W is the width, and H is the height.

Rotation by and angle $\mathsf{\theta }$:

$${\mathrm{I}}_{\mathrm{r}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{d}}\left({\mathrm{x}}^{\prime },{\mathrm{y}}^{\prime }\right)=\mathrm{I}\left(\cos (\mathsf{\theta }\right)·\mathrm{x}+\sin \left(\mathsf{\theta }\right)·\mathrm{y})-(\sin \left(\mathsf{\theta }\right)·\mathrm{x}+\cos \left(\mathsf{\theta }\right)·\mathrm{y})$$

(8)

where ($\mathrm{x},\mathrm{y}$) are the coordinates in the original image, and (${\mathrm{x}}^{\prime },{\mathrm{y}}^{\prime }$) are the coordinates in the rotated image.

3.4. Blur Effects

We applied the Gaussian blur effect to simulate various levels of atmospheric effects and depth of field. Noise injection is a matrix value usually drawn from a Gaussian distribution [4], and it can help networks learn more robust features. Our work covers noise injection to prevent the model from learning the noise in the training data, which can lead to overfitting. By making the model robust to noise, it learns to better generalize unseen data and reach training stability. Our noise injection equation is as follows:

$${\mathrm{I}}_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{y}}\left(\mathrm{x},\mathrm{y}\right)=\mathrm{I}\left(\mathrm{x},\mathrm{y}\right)+\mathrm{N}(\mathrm{x},\mathrm{y})$$

(9)

where, $\mathrm{N}(\mathrm{x},\mathrm{y})~{\mathrm{N}}^{\prime }(\mathsf{\mu },{\mathsf{\sigma }}^2)$ is Gaussian noise with mean $\mathsf{\mu }$ and variance ${\mathsf{\sigma }}^2$.

In the next step, we applied a blur filter to the image. The Gaussian blur is a smoothing operation that reduces the high-frequency components in the image, effectively introducing an atmospheric effect. We set the blur radius from 1 to 20 and applied it to the blended ship image at random values. From the Gradio web app, the user can set blur ratio to zero, depending on their purpose [64,65].

3.5. Change Color Channels and White Balance Adjustment

Lighting biases are one of the most frequently facing challenges in image recognition tasks. Digital image data are typically represented as a tensor with dimensions corresponding to height, width, and color channels [4]. Chatifled et al. [43] mention that there is a 3% accuracy drop between grayscale and RGB image classification.

Implementing augmentations within the color channel space is an effective and straightforward form for individual color channels.

$${\mathrm{I}}_{\mathrm{c}\mathrm{o}\mathrm{l}\mathrm{o}\mathrm{r}\_\mathrm{c}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{d}}\left(\mathrm{x},\mathrm{y}\right)=(\mathrm{R}\left(\mathrm{x},\mathrm{y}\right),\mathrm{G}\left(\mathrm{x},\mathrm{y}\right),\mathrm{B}(\mathrm{x},\mathrm{y}\left)\right)$$

(10)

where $\mathrm{R},\mathrm{G}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{B}$ represent the red, green, and blue channels, respectively.

The white balance function is designed to correct color imbalances between target ship images and source ocean background images by performing white balance adjustment. With this, we ensure that colors in the image appear natural and accurate under varying lighting conditions. Proper white balance adjustment is crucial for maintaining color fidelity, especially in applications such as digital photography and image analysis or visual perception studies.

To calculate average intensities of each color channel, we have $\mathrm{c}\in \{\mathrm{R},\mathrm{G},\mathrm{B}\}$:

$${\mathrm{c}}_{\mathrm{a}\mathrm{v}\mathrm{g}}={\displaystyle \frac{1}{\mathrm{N}}}{\sum }_{\mathrm{i}=1}^{\mathrm{N}}{\mathrm{I}}_{\mathrm{c},\mathrm{i}}$$

(11)

where, ${\mathrm{I}}_{\mathrm{c},\mathrm{i}}$ represents the intensity of the i-th pixel in channel c, and N is the total number of pixels in the channel.

Afterwards, we compute scaling factor of channel c as:

$${\mathrm{S}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{n}{\mathrm{g}}_{\mathrm{f}\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{o}\mathrm{r}}}_{\mathrm{c}}={\displaystyle \frac{\mathrm{a}\mathrm{v}\mathrm{g}}{{\mathrm{c}}_{\mathrm{a}\mathrm{v}\mathrm{g}}}}$$

(12)

where avg is the average intensity computed across all channels.

Last, we adjust pixel intensities of ${{\mathrm{I}}^{\prime }}_{\mathrm{c},\mathrm{i}}$ in channel c using:

$${{\mathrm{I}}^{\prime }}_{\mathrm{c},\mathrm{i}}={\mathrm{I}}_{\mathrm{c},\mathrm{i}}·{\mathrm{S}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{n}{\mathrm{g}}_{\mathrm{f}\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{o}\mathrm{r}}}_{\mathrm{c}}$$

(13)

3.6. Change Contrast Button and Equalize Histogram

The contrast of the image is randomly adjusted to simulate different lighting conditions. This technique is used to improve the contrast of the image, making the objects more distinct.

$${\mathrm{I}}_{\mathrm{c}\mathrm{o}\mathrm{l}\mathrm{o}\mathrm{r}\_\mathrm{c}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{d}}\left(\mathrm{x},\mathrm{y}\right)=\mathsf{\alpha }·\mathrm{I}\left(\mathrm{x},\mathrm{y}\right)+\left(1-\mathsf{\alpha }\right)·{\mathrm{I}}_{\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}}$$

(14)

where $\mathsf{\alpha }$ is the contrast factor, and ${\mathrm{I}}_{\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}}$ is the mean intensity of the image.

The histogram equalization function is pivotal in preprocessing images for various computer vision tasks. Histogram equalization adjusts the cumulative distribution function (CDF) of pixel intensities to enhance contrast. By enhancing the contrast through HE, the function ensures that the images exhibit a more uniform distribution of pixel intensity. The process can be described mathematically as follows:

First, we compute the CDF

$$\mathrm{c}\left(\mathrm{i}\right)={\sum }_{\mathrm{j}=0}^{\mathrm{i}}{\displaystyle \frac{\mathrm{h}\left(\mathrm{j}\right)}{\mathrm{N}}}$$

(15)

where N is the total number of pixels in the image.

Next, we compute the map intensity values by transforming each pixel intensity $I_{orig}\mathrm{u}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{t}\mathrm{h}\mathrm{C}\mathrm{D}\mathrm{F}$ as:

$${\mathrm{I}}_{\mathrm{n}\mathrm{e}\mathrm{w}}=\mathrm{f}\mathrm{l}\mathrm{o}\mathrm{o}\mathrm{r}\left(\right(\mathrm{L}-1)·\mathrm{c}({\mathrm{I}}_{\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{g}}\left)\right)$$

(16)

where L is the number of possible intensity levels, and $I_{new}$ is the new intensity value.

3.7. Change Sharpness Button

Then, we enhance image sharpness in a random range as we applied to other augmentation methods. The sharpness of the image is varied to simulate different camera focus settings.

$${\mathrm{I}}_{\mathrm{s}\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{p}\mathrm{n}\mathrm{e}\mathrm{s}\mathrm{s}}\left(\mathrm{x},\mathrm{y}\right)=\left(1-\mathsf{\beta }\right)·\mathrm{I}\left(\mathrm{x},\mathrm{y}\right)+\mathsf{\beta }·{\mathrm{I}}_{\mathrm{d}\mathrm{e}\mathrm{t}\mathrm{a}\mathrm{i}\mathrm{l}}(\mathrm{x},\mathrm{y})$$

(17)

where $\mathsf{\beta }$ is the sharpness factor, and ${\mathrm{I}}_{\mathrm{d}\mathrm{e}\mathrm{t}\mathrm{a}\mathrm{i}\mathrm{l}}$ is the detailed enhanced image.

By iterating the above steps multiple times with different sources and target images, a massive dataset can be generated. Each iteration introduces random variations in scaling, positioning, rotation, blurring, color change, noise injection, and other OE-focused methods, resulting in a highly diverse and comprehensive dataset [66,67,68].

3.8. Random Crop Function

We aimed to perform a random cropping (erasing) operation in an input image. This technique is a common data augmentation method used to increase the diversity of training datasets by generating multiple variations of the original image. Random cropping can help improve the robustness and generalization of ML models by providing a wider range. In this approach, we randomly cropped sections from background images after both target and source images were blended. This cropping might even crop ship fires images as well. The rectangular region is set to 500 by 500 in height and width, respectively. From the Gradio web app, it is possible to specify the height and width of a random rectangle area.

Image cropping (random erasing [69]) is a common technique in image processing and data augmentation. In image recognition, this method mainly focuses on overcoming occlusion-related challenges, such as unclear part representation on objects. Mathematically, $\mathrm{I}$ is the image dimensions of width ($\mathrm{W}$) and height ($\mathrm{H}$) of the image. The goal is to crop a region of size (${\mathrm{C}}_{\mathrm{w}}$,${\mathrm{C}}_{\mathrm{h}}$) randomly from $\mathrm{I}$, where ${\mathrm{C}}_{\mathrm{w}}$ and ${\mathrm{C}}_{\mathrm{h}}$ are the width and height of the cropping image region. For cropping coordinate detection, we randomly selected the top-left corner of the cropping region. Let $(\mathrm{x},\mathrm{y})$ be the coordinates, then:

$$\mathrm{x}=\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{d}(0,\mathrm{W}-{\mathrm{C}}_{\mathrm{w}}),$$

(18)

$$\mathrm{y}=\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{d}(0,\mathrm{H}-{\mathrm{C}}_{\mathrm{h}}),$$

(19)

where, $\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{d}(\mathrm{a},\mathrm{b})$ is a function that returns a random integer between $\mathrm{a}$ and $\mathrm{b}$.

We then set a transparency to that cropped area, denoted as ${\mathrm{I}}_{\mathrm{c}},$ in transparent image size ${(\mathrm{C}}_{\mathrm{w}}$,${\mathrm{C}}_{\mathrm{h}})$. The transparent region ${\mathrm{I}}_{\mathrm{c}}$ is then pasted onto the original image $\mathrm{I}$ with transparency. This result in ${\mathrm{I}}^{\prime }$ is given by:

$${{\mathrm{I}}^{\prime }}_{(\mathrm{x}:\mathrm{x}+{\mathrm{C}}_{\mathrm{w}}\mathrm{y}:\mathrm{y}+{\mathrm{C}}_{\mathrm{h}})}={\mathrm{I}}_{\mathrm{c}}$$

(20)

This equation indicates that the region starting at (x,y) and extending ${\mathrm{C}}_{\mathrm{w}}$ pixels in width and ${\mathrm{C}}_{\mathrm{h}}$ pixels in height is replaced by the transparent region ${\mathrm{I}}_{\mathrm{c}}.$ This process effectively creates an occlusion in the image, simulating scenarios where parts of the object are hidden. As we mentioned above, we have developed a Gradio web app, an interface to facilitate an interactive platform for performing and visualizing various image augmentation operations.

This system enables users to upload a target and a background image to generate a set of augmented images with varying parameters. In

Table 4. Gradio web app interface parameters.

Description	Values (Range)
Number of images	1–1000
Target image x—axis	0–1500
Target image y—axis	0–1500
Output folder	Specify
Contrast factor	0.5–3
Sharpness factor	0.5–3
Blur radius	0–20
Rotation angle	0–360
Equalize histogram	yes/no
Apply white balance	yes/no
Random crop width	10–500
Random crop height	10–500
Apply random crop	yes/no

, we can see the selection values of data augmentation. Based on the given features, data augmentation parameters can be customized. For example, if you want to have a bigger cropped area on the image by width or height, it is adjustable, ranging from 10 to 500. Similarly, other parameters are also available to select certain range values.

3.9. Dataset Collection

Figure 2 depicts the Gradio web interface, where we have an input target and source image. By sliding the OE augmentation method included functions, we can adjust data generation into our project.

Table 5. Internet source and augmented datasets.

Internet Source Images		Total
Ship Fire		9200
Ship No—Fire		4100
Internet Source (Sample)		Augmented
Ship fire	90	11,440
Background	13

provides a detailed summary of the image dataset utilized for the task of ship fire detection, including both original internet source images and their augmented counterparts. The datasets are categorized into two primary classes, such as “Fire” and “No-fire”, with a further distinction between internet-sourced images and augmented samples in combination with ocean background images. For blending, the background should be a kind of image that represents all kinds of ocean scenarios. Similarly, sea vessels also should be varied, as the ocean environment covers many kinds of sea vessels. The initial dataset comprises 9200 images of ship fires and 4100 images of ships without fire. To increase the dataset size, a sample of 90 ship fire images was subjected to augmentation, resulting in an expanded dataset of 11,440 images. The extended dataset is generated by blending with 13 background ocean images.

Figure 3 depicts a sample of ship images in an ocean environment. Ships in the image are considered as main objects that are extracted from the background and combined with other ocean images. Ocean and ship images are target and source images, respectively. Of these images, we augment one single-source ship image with another ocean target image. However, during blending source and target image, we can increase the number of images.

In Figure 4, various different ocean environments are represented. An extracted ship image can represent various parts of an ocean image in applied augmentation techniques, such as noise injection (blur effect), rotation scaling, and random cropping techniques.

4. Experimental Results

In this research, we proposed several data augmentation techniques to source and target images for the purpose of ship fire image classification improvement. In Figure 5a, a sample ship fire (boat) image is represented. From Figure 5a, the main target image is extracted from the ocean environment and shown in Figure 5b. This is the initial step for data augmentation.

In Figure 6, OE target images are attached to source ocean background images. In the red circles, we highlighted the source image location in the ocean image. As can be seen, augmentation techniques change the ocean image blur effect, sharpness, and a ship image’s random locations. Using the Gradio web application, we can change the source image location in the x-y axis and all other mentioned metrics listed in Table 4. Finally, in Figure 7, we see the OE ship fire dataset creation process beginning with the input ship fire image as the source and progressing to the extraction step, blending to target the ocean background image with the application of the data augmentation techniques.

To compare the augmented dataset performance, we labeled our dataset and trained the labeled dataset in Yolo-v8 and Yolo-v10 versions. Both models’ classification accuracy was high; therefore, we chose these models to fine-tune for ship fire detection purposes. Figure 8 and Figure 9 show training image examples with bounding box information on labeled classes. From figures, we can see that ship images are relatively clear to distinguish and identify, which is the “No-fire” class dataset. However, in the “Fire” class, we achieved our proposed style of dataset and type. In other words, we obtained hazy ship fire images with a different look, scale, size, and other applied augmentation technique results. From the comparative analysis of Table 1, it can be concluded that having a large variety of datasets increases the model’s detection and classification accuracy.

We trained both models on 100 epochs (Figure 10). The training and validation loss plots show a decreasing trend, indicating that the models are improving as training progresses. The loss curves appear smoother, but there are some fluctuations in both models’ cases, possibly due to challenges in model optimization. In comparison, training and validation losses for the Yolo-v10 fine-tuned model for ship fire detection are consistently lower and smoother compared to Yolo-v8.

Metrics, such as precision, recall, and mean average precision (mAP), improve over epochs but show irregular behavior, especially after 50 epochs. These fluctuations are considered as our future research work that we focus on to optimize model training and solve instability in training or validation performances. The mAP score is a key metric for object detection tasks, measuring the accuracy of the bounding boxes. Yolo-v10 showed a higher and more consistent mAP than Yolo-v8, which demonstrates that the fine-tuned Yolo-v10′s performance is higher at detecting and classifying ship fires.

In Figure 11, precision-recall (PR) curves are presented to illustrate the performance of both models’ training. In Yolo-v8, the mAP at the intersection over union (IoU) threshold of 0.5 reached 0.91% accuracy, which demonstrates the model’s relatively high accuracy in detecting and classifying ship fires. The PR curve for Yolo-v10 demonstrates slightly improved precision at 0.913 and recall of 0.937, indicating a better balance between precision and recall compared to Yolo-v8. However, in terms of Yolo-v8, the exact scores were 0.866 and 0.912 in precision and recall metrics. The mAP score of Yolo-v10 at the IoU ([email protected]) is also higher for Yolo-v10 at 0.937, as can be seen in Figure 11 and Figure 12.

Table 6. Comparative analysis of ship fire detection model metrics with other Yolo-based architectures.

Models	Precision	Recall	F1	[email protected]
Yolo-v3-Tiny [70]	0.797	0.821	0.81	0.745
Yolo-v5s [71]	0.901	0.855	0.88	0.814
Yolo-v7-tiny [72]	0.857	0.7921	0.82	0.758
Yolo-v8n (custom data)	0.866	0.912	0.888	0.910
Yolo-v10n (custom data)	0.913	0.937	0.925	0.937

compares the ship fire detection models’ achievements. Applied data augmentation and blending techniques for ship fire image, the custom fine-tuned models, such as Yolo-v8 and Yolo-10, clearly outperformed the more generalized models like, Yolo-v3-Tiny, Yolo-v5s, and Yolo-v7-Tiny. The fine-tuned Yolo-v10 model stands out, delivering the best precision, recall, F1 score, and [email protected], which indicates proposed method superiority for accurately detecting ship fires in diverse ocean environments.

5. Limitations

One limitation of this research is the reliance on synthetic data augmentation to address the scarcity of real-world ship fire images. While techniques like blending, noise injection, and scaling improve the diversity of the dataset, they may not fully capture the complexity of natural ocean environments, such as varying weather conditions, lighting, or ship movements. This could limit the generalization of the model when applied to real-world scenarios, particularly in extreme or less frequently encountered conditions. Additionally, in our study we focused on a subset of augmentation techniques, leaving unexplored methods that might further enhance model robustness, such as more sophisticated image generation approaches.

6. Conclusions

In this study, we presented an OE data augmentation technique based on image blending, which involves automatically removing backgrounds from source images and combining them with target images. This method allows for the generation of massive and diverse datasets, enhancing the training and generalization capabilities of ML models. Nowadays, many researchers focus on GAN, or style transfer architectures, as a focus area of interest. However, GANs can produce synthetic data with artifacts and noise that do not appear in real data and require significant computational resources. Perez et al. [73], for example, proposed a neural style transfer algorithm that required two images from samples. However, the style transfer augmentation approach did not have any impact on the MNIST problem, which is the generation of a numerical dataset. Our approach is superficial with keeping OE identity by avoiding unrealistic subtlety, which cannot mislead the training process and degrade model performance. Moreover, our proposed OE data augmentation methods can iteratively generate huge amount of data, so that training of ML models does not suffer from a lack of data and diversity.

We used a dataset of ship images and an ocean image as a background to blend ship images in ocean environments. The ship images were processed using the background removing algorithm. The resulting isolated ship images were then merged with the source background image and blended into the OE augmentation pipeline.

In our future work, we will focus on optimizing data augmentation functionalities even further and wish to combine with Yolo (You Only Look Once) models to automate CV model training processes in a more straightforward manner.