YOLOv8-TF: Transformer-Enhanced YOLOv8 for Underwater Fish Species Recognition with Class Imbalance Handling

Abstract

In video-based fish surveys, species recognition plays a vital role in stock assessments, ecosystem analysis, production management, and protection of endangered species. However, implementing fish species detection algorithms in underwater environments presents significant challenges due to factors such as varying lighting conditions, water turbidity, and the diverse appearances of fish species. In this work, a transformer-enhanced YOLOv8 (YOLOv8-TF) is proposed for underwater fish species recognition. The YOLOv8-TF enhances the performance of YOLOv8 by adjusting depth scales, incorporating a transformer block into the backbone and neck, and introducing a class-aware loss function to address class imbalance in the dataset. The class-aware loss considers the count of instances within each species and assigns a higher weight to species with fewer instances. This approach enables fish species recognition through object detection, encompassing the classification of each fish species and localization to estimate their position and size within an image. Experiments were conducted using the 2021 Southeast Area Monitoring and Assessment Program (SEAMAPD21) dataset, a detailed and extensive reef fish dataset from the Gulf of Mexico. The experimental results on SEAMAPD21 demonstrate that the YOLOv8-TF model, with a mean Average Precision (mAP)_0.5 of 87.9% and mAP_0.5–0.95 of 61.2%, achieves better detection results for underwater fish species recognition compared to state-of-the-art YOLO models. Additionally, experimental results on the publicly available datasets, such as Pascal VOC and MS COCO datasets demonstrate that the model outperforms existing approaches.

1. Introduction

Accurate fish species identification holds significant importance in fisheries management and environmental monitoring. It plays a vital role in various aspects, including the identification of endangered species, optimizing harvesting timing and size, ecosystem monitoring, and establishing efficient production management systems [1,2,3]. Accurate fish species recognition becomes even more crucial due to legal constraints on fishing methods, particularly for threatened or endangered species. Traditional approaches for identifying fish species often rely on manual, labor-intensive processes that consume substantial time. Furthermore, hook-based sampling can disturb the natural behavior of fish. Conventional methods present difficulties in ensuring the availability of reliable data to effectively manage sustainability in fisheries, monitoring federal fisheries, evaluating fish populations, and distinguishing between various species of fish. In contrast, leveraging computer vision, deep learning (DL) technology, and annotation tool offers the potential for robust models in the identification of fish species, leading to cost and time savings while enhancing the accuracy of identification [4,5,6,7].

Machine vision solutions have the potential to replace manual counting and species identification by offering comparable or even superior precision. Several methods for fish detection, including lidar [8,9], sonar [10], and RGB imaging [11], are available. When it comes to species identification in clear water, RGB imaging stands out as the favored choice. It enables a straightforward identification of fish using their color, texture, and geometry. Additionally, RGB imaging is not only cost-effective and lightweight, but is also environmentally friendly as it does not disrupt fish habitat. In earlier studies, the analysis of video frames [12] was performed on an individual basis to detect objects present in each frame. Recently, multiple camera systems have been employed to monitor fish stocks and assess the sustainability of marine ecosystems. Nevertheless, these systems face a common challenge of time-consuming manual processing bottlenecks [13]. Recent advancements in deep learning have demonstrated significant potential for extracting valuable information about marine ecology through object detection and classification [14,15,16]. Though RGB imaging offers advantages in clear water conditions, its effectiveness is significantly hindered by the challenges posed by underwater environments, such as low light and turbid conditions [17,18], occlusion [19], low-resolution images and videos, and the inherent difficulty in distinguishing fish from the background [20]. These factors can lead to blurry images and noisy data, making it difficult to accurately detect and classify objects, particularly smaller fish species. To address these issues, innovative approaches such as the SWIPENET+CMA framework have been proposed in [21]. By leveraging advanced DL techniques, this framework aims to improve the accuracy of object detection and classification in challenging underwater conditions. While optical cameras struggle in underwater environments characterized by poor lighting and turbidity, sonar cameras [22] offer a more reliable solution. Sonar systems can cover a wider area and penetrate deeper into the water column, making them suitable for large-scale underwater surveys. The integration of machine learning (ML) techniques with sonar data [23] has enabled the development of automated systems for analyzing sonar data, facilitating precise fish detection and classification [24]. However, the high cost of high-resolution 3D sonar cameras remains a significant barrier to their widespread adoption. The dynamic movement of fish introduces complexities such as shape variations and object overlaps, which pose challenges in accurately identifying and detecting underwater fish species. Another significant challenge for marine applications is marine snow, comprised of drifting organic and inorganic particles, significantly degrades underwater image clarity. Solutions targeting this issue include: end-to-end dual-channel frameworks that address multiple degradations including marine snow removal [25], deep learning approaches fusing spatial and Fourier domain information for effective marine snow restoration [26], and the introduction of a dedicated benchmarking dataset to facilitate further research in this area [27]. In addition, DL is a prominent field within computer vision, has found widespread application in addressing diverse challenges such as detection, localization, estimation, and classification [4,28,29,30,31]. Numerous ML and DL algorithms have been created for the purpose of categorizing fish species. For example, Jager et al. [32] utilized the AlexNet architecture for feature extraction and employed multiclass support vector machine (SVM) for classification. Similarly, hierarchical features combined with SVM have been utilized for fish classification. Carion et al. [33] introduced end to end object detection with transformers (DETR) and the experiments are conducted in challenging MSCOCO object detection datasets. Fang et al. [34] presented You Only Look Once (YOLO) model based on transformer for object detection on public MSCOCO dataset.

The You Only Look Once (YOLO) model [35,36,37,38] widely used for object detection, is effective in identifying fish, especially in videos. YOLO operates as a single-shot detection model [39], processing the entire image or video frame in one go to accurately predict object locations and classes. The YOLO model is specifically designed for efficient processing, making it ideal for real-time use. With extensive training on large datasets, it can achieve impressive levels of accuracy. The YOLO model utilizes a convolutional neural network (CNN) architecture and is trained using a set of labeled images. Through training, the model becomes adept at predicting bounding boxes that represent the location of fish in an image or video, along with the associated class probabilities (i.e., the fish species). Several loss functions are employed during training to penalize inaccurate predictions and encourage continuous improvement in the model’s prediction capabilities over time.

YOLO provides a unified framework for object detection, allowing it to identify multiple objects in one traversal of the network. This is advantageous for tasks that require detecting and recognizing multiple objects. YOLO offers the advantage of efficient processing by being highly parallelizable, allowing it to effectively utilize multiple GPUs simultaneously. However, due to its single-stage detection network architecture, YOLO sacrifices some precision compared to two-stage networks like Faster R-CNN [40]. Underwater images often contain numerous small objects, which are typically overlooked by the pooling layers in CNNs due to their size, making detection and recognition difficult. Moreover, similar to other object detection models, YOLO encounters challenges in accurately detecting fish in situations of low lighting or partial obstructions. Despite these limitations, YOLO continues to be widely favored and proven effective for tasks involving fish classification and detection [41]. The YOLOv5 technique for object detection was initially introduced by Jocher et al., 2021 [42], who utilized it on the MS COCO public datasets [43]. Subsequently, Jung et al., 2022 [44] implemented YOLOv5 specifically for object detection in drone images. Wang et al. [45] presented an optimized nighttime nail detection with an advanced YOLOv5 model for improved road safety. In our previous work [5], we enhanced YOLOv5 by modifying its backbone to improve fish species recognition.In addition, Varghese et al., 2024 [46] introduced a novel YOLOv8 model for object detection in the public MS COCO dataset. When compared to the aforementioned algorithms, YOLOv8 was a significant advancement in the YOLO series, known for its high accuracy, fast processing speed, and capacity to detect multiple objects within an image [47]. Following that, Li et al. [48] introduced a modified YOLOv8-based technique for recognizing objects in images collected from the air. Ansah et al. [49] proposed a SB-YOLO-V8 technique layered with deep Learning framework for real-time detection of humans. Bi et al. [50] introduced a streamlined detection network based on an improved YOLOv8 model, tailored for identifying small targets in drone-captured aerial images. The modifications aim to enhance detection accuracy for objects observed from high altitudes. None of the aforementioned techniques have been adequately implemented for recognizing fish species in underwater environments with highly imbalanced species distributions. In this study, we have proposed YOLOv8-TF: Transformer-Enhanced YOLOv8 for Underwater Fish Species Recognition with Class Imbalance Handling to better handle the challenges posed by class imbalance in underwater fish detection, further improving its effectiveness in these complex environments.

The primary contributions of this research are as follows:

• To enhance fish species recognition, we made modifications to the depth scale of various layers within the backbone of the YOLOv8 model. These modifications resulted in improved performance and accuracy in identifying fish species.
• We incorporated the transformer block into both the backbone and neck networks of the YOLOv8-based approach. This enhancement boosts the model’s capabilities by enabling it to capture more contextual and global information, thereby improving its performance.
• To tackle the challenges posed by a highly imbalanced dataset, we introduced a class-aware loss function along with Wise-IoUv3 [48]. This approach enhances both classification and localization accuracy, leading to more precise and reliable results.

2. Related Works

The rapid rise in popularity of DL techniques has led to their successful implementation across various industries, including the fishery industry [51]. DL techniques have greatly influenced information and data processing in the realm of smart fish farming, presenting both new opportunities and challenges. Within aquaculture, DL is extensively utilized for various tasks, including live fish identification [52], species classification [53], behavioral analysis [54], feeding selection [55], size or biomass estimation [56], and water quality forecasting [57]. Several DL approaches exist that are particularly well-suited for fish datasets, allowing for the development of tailored DL models specific to each task.

The YOLO algorithm has gained significant popularity in fish identification and detection tasks. A notable early study conducted by Xu et al. (2018) [29] employed YOLO to develop a model capable of identifying fish in underwater videos. The study utilized three diverse datasets captured at water power sites to train the YOLO model specifically for this purpose. In a recent study, the Southeast Area Monitoring and Assessment Program Dataset (SEAMAPD21) was utilized to examine the effectiveness of YOLOv4 in the identification and categorization of different fish species. The results showcased the successful localization and identification of various fish species within the SEAMAPD21 dataset [58] using the YOLOv4 model. Alaba et al. [4] employed two feature extraction networks, namely MobileNetv3-large [59] and VGG16 [60], to extract relevant features from underwater images. They then utilized a single-shot multi-box detector (SSD) to classify fish species and detect the precise location of the fish within the images from the SEAMAPD21 dataset. Wang et al. [61] introduced a novel attention mechanism, Channel and Spatial Fusion Attention (CSFA), which integrates channel and spatial attention into the YOLOv5 architecture. By combining these attention mechanisms, the network can effectively focus on both the salient features and spatial context of detected objects, leading to improved detection performance. To overcome challenges like turbidity and low lighting in underwater object detection, Pachaiyappan et al. [62] worked on including attention mechanisms, transformers, and diffusion models. These approaches enhance image quality, enabling more accurate detection and classification by using advanced imaging technique YOLO network (AIT-YOLOv7), supporting Sustainable Development Goal (SDG) 14: Life Below Water, which aims to conserve and sustainably use the oceans, seas, and marine resources.

Sung et al. [63] introduced a real-time fish detection model based on the YOLO algorithm, specifically designed for underwater vision. The effectiveness and precision of the proposed method were evaluated using real fish video images. The CNN model achieved an impressive classification accuracy of 93%, a predicted bounding box intersection over a union of 0.634, and a fish detection rate of 16.7 frames per second. Notably, it outperformed a fish detector that utilized a sliding window algorithm and a classifier trained with a histogram of oriented gradient features and a SVM. Jalal et al. [64] presented a two-step DL approach for identifying and classifying temperate fishes. In the initial step, they employed the YOLO object detection method to detect each fish in the image, disregarding its species or gender. In the subsequent step, a CNN with the Squeeze-and-Excitation structure was utilized to detect each individual fish in the image without any preliminary filtering. To address the limited samples issue, transfer learning was employed to improve the overall classification accuracy.

Collectively, these studies highlight the potential of DL, specifically the YOLO-based approach, in fish identification and detection tasks. They showcase promising outcomes in terms of accuracy and processing speed. However, challenges remain in addressing issues related to the complex and diverse underwater environment, as well as improving the models’ ability to generalize and handle new and unseen fish species effectively.

3. Proposed Approach

3.1. A YOLOv8 Technique for the Recognition of Fish Species in Underwater Environments

In YOLOv8, both the conventional convolution module and the C2f module [48,65] were employed to accomplish effective feature extraction and image downsampling, resulting in high-quality outputs, as shown in Figure 1. The C2f module integrates high-level features with contextual data to enhance detection performance. To gain richer gradient flow information while maintaining low weight, this module expands the gradient branch by reducing one conventional convolutional layer and fully utilizing the bottleneck module. YOLOv8 offers a range of models, including YOLOv8n, YOLOv8s, YOLOv8m, and YOLOv8l, each with varying architectural complexities and performance characteristics. While the underlying principle remains consistent, the classification of these models is based on their respective memory usage. For YOLOv8l, its backbone consists of C2f modules and conventional convolutional modules represented as $3\times \mathrm{C}2\mathrm{f}-6\times \mathrm{C}2\mathrm{f}-6\times \mathrm{C}2\mathrm{f}-3\times \mathrm{C}2\mathrm{f}$. Figure 2 shows the detailed diagram for C2f module. Within the YOLOv8 architecture, feature maps are segregated into several distinct scale features in descending order. These features are denoted as C2f layers in the backbone, the FPN (Feature Pyramid Network) [66] structure and the PAN (Path Aggregation Network) [67] structure in the neck. The PAN-FPN structure employed in the YOLOv8 model serves as a complement to the conventional FPN. It utilizes a top-down approach to effectively transfer deep semantic features. In the final stage, the head component utilizes the image features obtained from the neck module to generate predictions, involving steps for both class and box predictions.

3.2. YOLOv8-TF: Transformer-Enhanced YOLOv8 for Underwater Fish Species Recognition with Class Imbalance Handling

The YOLOv8 backbone consists of C2f layers [48,65]. The C2f module establishes connections across layers through additional branches and adjusts varying channel numbers for different scale models. This design not only ensures a lightweight model but also captures richer gradient flow information. For the improvement in the backbone, we have modified the width and depth scales of the C2f layers. As shown in Figure 1, C2f modules for YOLOv8-TF model are represented as $6\times \mathrm{C}2\mathrm{f}-12\times \mathrm{C}2\mathrm{f}-12\times \mathrm{C}2\mathrm{f}-6\times \mathrm{C}2\mathrm{f}$.

3.2.1. Transformer Block

We have introduced a transformer (trans) block [5,69,70], as shown in Figure 3, in the backbone network of YOLOv8-TF resulting in C2f and transformer (trans) modules as $6\times \mathrm{C}2\mathrm{f}-12\times \mathrm{C}2\mathrm{f}-12\times \mathrm{C}2\mathrm{f}-6\times trans$. For the improvements in the Neck, C2f modules are replaced with transformer (trans) modules. The transformer layers help capture long-range dependencies and global contextual information, which enhances the model’s ability to detect fish species more accurately by effectively distinguishing subtle visual patterns across different scales. For the processing of 2D images in trans layer, we flatten the spatial dimensions of the 2D feature map $xϵ{\mathbb{R}}^{h\times w\times d}$ into a sequence $x_pϵ{\mathbb{R}}^{h\times w\times d}$, where (h,w) represents the original feature map resolution, d denotes the number of channels, and $h\times w$ functions as the effective input sequence length for the transformer layer. To imbue the attention operation with positional awareness, transformer-based architectures commonly employ position encoding. We employ standard learnable 1D positional embeddings through a linear layer to maintain positional information.

3.2.2. Wise-IoU Loss Function

YOLOv8 employs the anchor-free concept such that the Binary Cross Entropy (BCE) loss is used for the classification loss, and Bounding Box Regression (BBR) loss and distribution focal (DF) loss are used for the regression portion [48,65].

$$L_{loss}={\alpha }_1{L}_{BC}+{\alpha }_2{L}_{DF}+{\alpha }_3{L}_{BBR}$$

(1)

where $L_{BC}$ denotes classification loss, $L_{BBR}$ represents regression loss, and $L_{DF}$ defines the distribution focal loss with regularization coefficients ${\alpha }_1$, ${\alpha }_2$, and ${\alpha }_3$.

Wise Intersection over Union (WIoU) [71,72] introduces a dynamic, non-monotonic focus mechanism that evaluates anchor box quality through dissimilarity rather than traditional IoU. Then, the cost function can be estimated as:

$$L_{loss}={\alpha }_1{L}_{BC}+{\alpha }_2{L}_{DF}+L_{WIoUv3}$$

(2)

where $L_{WIoUv3}$ denotes the wise-IoU loss term, which can be estimated as below:

$$L_{WIoUv3}=\left(1-{\displaystyle \frac{\mathrm{bbox}\cap \mathrm{wbox}}{\mathrm{bbox}\cup \mathrm{wbox}}}\right)exp\left({\displaystyle \frac{{(x_{\mathrm{bbox}}-x_{\mathrm{wbox}})}^2+{(y_{\mathrm{bbox}}-y_{\mathrm{wbox}})}^2}{{(w^2+h^2)}^{★}}}\right)\kappa$$

(3)

$$\kappa =\theta /\tau {\beta }^{\theta -\tau }$$

(4)

The $L_{WIoUv3}$ loss evaluates the quality of an anchor box by considering both the spatial overlap and the distance between the centers of the predicted and ground truth boxes. Unlike traditional Intersection over Union (IoU) metrics, which only measure overlap, wise IoU introduces a distance-based penalty for more insightful gradient gain allocation. In this formulation, $x_{bbox},y_{bbox}$ represent the center coordinates of the predicted anchor box, while $x_{wbox},y_{wbox}$ denote the center coordinates of the ground truth anchor box. The terms $bbox$ and $wbox$ correspond to the predicted and ground truth bounding boxes, respectively, with w and h representing the width and height of the reference prediction box. The term $\theta$ represents the abnormality degree for prediction box, $\beta$ and $\theta$ are hyperparameters. By incorporating the center distance into the loss calculation, wise IoU dynamically adjusts gradients to prioritize anchor boxes closer to the ground truth, ultimately improving object detection performance.

3.2.3. Class-Aware Loss Function

To improve the loss function, we have introduced class-aware (ca) regularization [4] terms in classification, regression, and distribution focal loss terms of the YOLOv8 model.

$$L_{loss}={\alpha }_1{L}_{BCca}+{\alpha }_2{L}_{DFca}+L_{WIoUv3ca}$$

(5)

The proposed class aware classification, distribution focus, and regression loss terms can be defined as:

$$\begin{array}{cc}\hfill L_{BCca}& ={\displaystyle \frac{1-\left(\frac{n_s}{n}\right)}{1-{\left(\frac{n_s}{n}\right)}^{\eta }}}{L}_{BC},\hfill \\ \hfill L_{DFca}& ={\displaystyle \frac{1-\left(\frac{n_s}{n}\right)}{1-{\left(\frac{n_s}{n}\right)}^{\eta }}}{L}_{DF},\hfill \\ \hfill L_{WIoUv3ca}& ={\displaystyle \frac{1-\left(\frac{n_s}{n}\right)}{1-{\left(\frac{n_s}{n}\right)}^{\eta }}}{L}_{WIoUv3},\hfill \end{array}$$

(6)

where, $L_{BCca}$ is class-aware classification loss and $L_{WIoUv3ca}$ is class-aware wise IoU loss, and $L_{DFca}$ is the class-aware distribution focal loss. The quantity $n_s$ represents the number of training instances associated with each species, while n denotes the total number of training samples and $\eta$ is a tunable hyper-parameter described below. It is important to note that in this context, $n_s$ is considerably smaller than n.

When $n_s$ is characterized by a small value, and the parameter $\eta$ is increased, the multiplying term’s value $\left({\displaystyle \frac{1-\left(\frac{n_s}{n}\right)}{1-{\left(\frac{n_s}{n}\right)}^{\eta }}}\right)$ increases as well. This increase effectively amplifies the weight assigned to the less-represented class. To achieve optimal performance, it is essential to fine-tune the hyper-parameter values. In most cases, raising the $\eta$ value results in an increase in the value of the multiplying term within the loss functions. However, this alone may not lead to an overall performance improvement, as it can have repercussions on other classes. Consequently, the ideal hyper-parameter value for maximizing performance on a specific dataset must be systematically adjusted, beginning with an initial setting of parameter $\eta$ = 1. For this specific training scenario, we set $\eta$ to the value of 8.

3.3. Dataset Description

The Southeast Area Monitoring and Assessment Program Dataset 2021 (SEAMAPD21) [58] provides a large-scale collection of reef fish data from the Gulf of Mexico. This dataset encompasses 130 distinct classes of fish species captured in underwater environments, with a total of 28,319 images. However, certain species have limited representation in the dataset as shown in Figure 4, and the model’s performance is influenced by samples with a higher number of species per class. Furthermore, detecting fish in low-resolution underwater environments presents challenges due to the difficulty in distinguishing between images and the background. Images from the dataset are illustrated in Figure 5. The task of detecting fish in certain images presents challenges, even for a human. For the experimental setup, a train-validation-test split ratio of 70/15/15 is utilized. The mean average precision (mAP) is then evaluated on the test set to assess the model’s performance.

4. Experimental Results

For the methods in comparison, parameters were tuned according to MobileNetv3-large [4], VGG300 [4], VGG512 [4], YOLOV5s [5], YOLOv5m [5], YOLOV5l [5], YOLOV8s [65], YOLOV5enh [5], YOLOv8m [65], and YOLOV8l [65].

4.1. Implementation Details

For training the proposed YOLOv8-TF, we utilized PyTorch 1.13.0 and employed an NVIDIA A100-SXM GPU. The models were trained and tested using this setup. The approach was trained from scratch, and Stochastic Gradient Descent (SGD) was employed as the optimizer for the model. A total of 300 epochs were conducted during the training process. The values for $\eta$ were varied from 1 to 16 and optimal performance was found at $\eta$ = 8. Optimal values of ${\alpha }_1$ = 0.5, ${\alpha }_2$ = 1.5, ${\alpha }_3$ = 7.5, $\theta$ = 2, $\beta$ = 1.8, and $\tau$ = 3 were used.

4.2. Performance

To evaluate the performance of the proposed YOLOv8-TF and other existing versions of YOLOv8, the commonly used metric mean average precision (mAP) is employed. Two variants of mAP are utilized: mAP_0.5:0.95, which calculates the average precision over a range of Intersection Over Union (IoU) values from 0.5 to 0.95, and mAP_0.5, which focuses on IoU at 0.5. These metrics provide a comprehensive assessment of the detection performance across different IoU thresholds.

Table 1. The mean average precision (mAP%) was computed for 121 species using the SEAMAPD21 dataset (see Table A1).

Method	mAP0.5	mAP0.5:0.95	Parameters	GFLOPS	FPS
MobileNetv3-large	-	32.51	-	-	105
VGG300	-	48.99	-	-	67
VGG512	-	52.75	-	-	54
YOLOv5s	71.9	43.9	7.37 M	17.1	117
YOLOv5m	75.6	47.9	21.37 M	49.5	110
YOLOv5l	78.5	50.6	46.80 M	109.9	103
YOLOv5enh	81.1	53.0	61.30 M	151.0	99
YOLOv8n	72.1	45.4	3.66 M	11.2	146
YOLOv8s	76.1	49.6	11.21 M	29.1	137
YOLOv8m	80.3	52.7	25.91 M	79.1	128
YOLOv8l	81.1	53.4	43.70 M	151.0	120
YOLOv10l	84.2	58.5	25.92 M	127.4	125
YOLOv8-TF	87.9	61.2	30.56 M	195.7	116

presents the mAP for the YOLOv8enh-based approach in contrast to alternative methods. Certain species exhibit a limited number of samples, such as fewer than 10, which may result in inadequate samples for training, validation, and testing sets. As a result, 121 species are employed to gauge the mAP. The results indicate that the proposed YOLOv8-TF surpasses other versions, including YOLOv8n, YOLOv8s, YOLOv8m, YOLOv8l, YOLOv5s, YOLOv5m, YOLOv5l, MobileNetv3-large, VGG300, and VGG512 in the detection of fish species in underwater environments. Specifically, YOLOv8-TF exhibits superior performance with a mAP_0.5 of 87.9% and mAP_0.5:0.95 of 61.2%.

Additionally, Table 1 displays the network size, measured in terms of the number of parameters in millions (M), the number of calculations in GFLOPS, and speed measured in terms of Frames Per Second (FPS). It displays the inference latency on an NVIDIA A100-SXM GPU. This facilitates a comparison of the impact of the proposed approach with existing YOLOv8-based techniques. It is noticeable that YOLOv8-TF exhibits a higher number of parameters (51.08 M) and GFLOPS (165.4) compared to YOLOv8n, YOLOv8s, YOLOv8m, YOLOv8l, YOLOv5s, YOLOv5m, and YOLOv5l. YOLOv8 outperforms YOLOv5 in terms of speed measured with FPS for fish species recognition in underwater environments. YOLOv8-TF provides a competitive FPS of 116 compared to other methods. When considering accuracy measured in terms of mAP, YOLOv8-TF also demonstrates superiority compared to other methods.

Figure 6 presents a comparison between YOLOv8-TF and other existing versions like YOLOv5n, YOLOv5s, YOLOv5m, YOLOv5l, YOLOv5enh, YOLOv8s, YOLOv8m, and YOLOv8l focusing on metrics such as mAP, frames per second (FPS), and GFLOPS. The GFLOPS is depicted through the radii of the circles. While YOLOv8-TF demands higher GFLOPS compared to YOLOv8m, the performance improvement, measured in terms of mAP, makes up for this difference. In terms of speed, YOLOv8-TF with 116 FPS outperforms YOLOv5m, YOLOv5l, YOLOv5enh although it is slower compared to YOLOv8n, YOLOv8s, YOLOv8m, and YOLOv8l. The trade-off between detection accuracy and speed provided by YOLOv8-TF makes it superior in many cases compared to other versions.

Table 2. The mean average precision (mAP%) was computed for 20 classes using the pascal VOC dataset.

Method	mAP0.5	mAP0.5:0.95	Parameters	FPS
YOLOv8n	83.90	53.70	3.93 M	194
YOLOv8s	87.65	54.80	14.81 M	168
YOLOv8m	89.10	54.10	26.49 M	155
YOLOv8l	91.70	56.40	47.31 M	141
YOLOv10l	92.40	57.12	25.79 M	147
YOLOv8-TF	94.60	58.21	30.51 M	130

shows that the proposed YOLOv8-TF with a mAP_0.5 of 94.30% and mAP_0.5:0.95 of 58.10% outperforms all other compared methods for the publicly available Pascal VOC dataset. Additionally, parameters (in millions) and speed measured in FPS (frames per second) are reported. Proposed YOLOv8-TF shows better accuracy compared to other methods in expense of higher parameters 30.51M and lower FPS of 130. Similarly,

Table 3. The mean average precision (mAP%) was computed for 80 classes using the MSCOCO dataset.

Method	mAP0.5	mAP0.5:0.95	Parameters	FPS
YOLOv8n	47.70	32.40	4.07 M	242
YOLOv8s	59.44	42.43	14.83 M	216
YOLOv8m	62.90	45.70	26.86 M	190
YOLOv8l	67.20	49.60	47.36 M	153
YOLOv10l	68.20	50.50	25.88 M	162
YOLOv8-TF	69.50	51.80	30.54 M	139

shows that the proposed YOLOv8-TF with a mAP_0.5 of 69.50% and mAP_0.5:0.95 of 51.80% outperforms all other compared methods for the publicly available MSCOCO dataset. Moreover, proposed YOLOV8-TF outperforms other compared methods with high mean average precision although it introduced complexity with higher parameters of 30.54M and slower FPS of 139.

Figure 7, Figure 8, Figure 9 and Figure 10 depict detection maps for fish species in underwater environments using SEAMAPD21. The images are generated using the established YOLOv8s, YOLOv8m, YOLOv8l, and the proposed YOLOv8-TF. The numbers after the names refer to the version of the YOLO model. Notably, YOLOv8-TF as shown in Figure 10 exhibits superior detection performance compared to other models, as shown by the higher mAP values in Table 1. In certain images, even humans may find it challenging to detect missed fish. For instance, the bottom two images in Figure 8 and the upper left corner of Figure 9 may not be identifiable or verifiable by a human.

Table 4. Ablation analysis.

YOLOv8-TF				mAP0.5	mAP0.5:0.95	Params	FPS
Depth Scale	CA Loss	Trans	Wise IoU v3 Loss
✓	✕	✕	✕	81.8	53.2	26.87 M	126
✓	✓	✕	✕	82.8	54.8	27.12 M	123
✓	✓	✓	✕	86.2	59.2	30.52 M	118
✓	✓	✓	✓	87.9	61.2	30.56 M	116

presents an ablation study conducted on the proposed YOLOv8-TF technique. The removal of the transformer block, depicted in the backbone and neck sections of Figure 1, results in reduced accuracy for YOLOv8-TF with class-aware (ca) loss and modified depth scale of backbone in C2f structure. Specifically, the mAP_0.5 is 82.8%, and mAP_0.5–0.95 is 54.8%. Furthermore, upon removing the class aware (ca) loss function from YOLOv8-TF, the accuracy of YOLOv8-TF with modified depth scale of backbone in C2f structure, without ca loss and trans block decreases to an mAP_0.5 of 81.8% and mAP_0.5–0.95 of 53.2%. Furthermore, parameters in million (params) and FPS (frames per second) are reported for ablation analysis. It can be observed that proposed YOLOV8-TF shows higher mean average precision compared to other versions consuming more parameters of 30.56 M and slower FPS of 116. It is because of involvement of transformer block in backbone and neck, using class aware loss function along with wise iou v3 loss, and implementing increased depth scale in the backbone of proposed YOLOv8-TF.

5. Discussion

For proposed YOLOv8-TF on SEAMAPD21, we have listed estimation of parameter count, GFLOPS, and FPS in Table 1. Similarly, Table 4 shows estimation of params (parameters in millions) and FPS (frames per second) for ablation analysis on YOLOv8-TF. It can be observed that proposed YOLOv8-TF outperforms other compared methods in terms of mean average precision (mAP). However, it suffers from increased parameter counts and slower FPS. Due to attention mechanism involved in transformer, addition of increased depth scale, involvement of class imbalance loss along with wise iou v3 loss, proposed YOLOv8-TF is influenced by increased number of parameters. An increase in the number of parameters leads to slower recognition speed because more resources are needed for each inference. Our model shows a higher accuracy for increase in the number of parameter counts and a decrease in the FPS, which result in an higher inference time. This model requires substantial memory because of higher number of parameters compared to other baseline models, such asYOLOV8m, YOLOv8s, and YOLOv8n. It may limit its use in resource-constrained environments. Our model may need optimization to address issues of slower inference time and higher memory usage in resource-constrained environments like embedded systems.

6. Conclusions

This study presents an enhanced approach, YOLOv8-TF, for fish species recognition in underwater environments. The effectiveness of the proposed YOLOv8-TF is evaluated through experiments conducted on the Southeast Area Monitoring and Assessment Program Dataset 2021 (SEAMAPD21), which comprises an underwater fish species dataset obtained from the Gulf of Mexico. The experimental results highlight the superiority of YOLOv8-TF compared to existing YOLOv8-based techniques. The method’s effectiveness is improved by adjusting the backbone’s depth scale, and adding a transformer block to both the backbone and neck sections further enhances the recognition of different fish species. Moreover, class-aware (ca) loss further enhances the detection performance by helping mitigate the class imbalance issue in the SEAMAPD21 dataset. We compare different YOLO models to illustrate the performance improvements achieved by YOLOv8-TF.

Future research directions will concentrate on building a lightweight model that can be deployed in resource-constrained environments and devices, aiming to enhance real-time processing capabilities and achieve higher accuracy. Furthermore, we plan to significantly ramp up the complexity and scope of our training data by incorporating a full training library that includes all available video annotations. This expansion will allow YOLOv8-TF to leverage temporal information inherent in video data and enrich the model’s understanding of fish behavior and movement patterns, significantly enhancing our model’s capabilities in fish tracking and counting across multiple images or videos. Such advancements will propel our ability to monitor and understand aquatic ecosystems to new heights and open the way for groundbreaking marine biology and conservation applications.