Computer Vision Techniques for Morphological Analysis and Identification of Two Pseudo-nitzschia Species

Abstract

The diversity of phytoplankton influences the structure of and processes that occur in marine ecosystems, with size and other morphological traits being crucial for nutrient uptake and retention in the euphotic zone. Our research introduces a machine learning method that can facilitate the analysis of phytoplankton functional traits from image data. We use computer vision to identify and quantify phytoplankton species and estimate size-related traits based on cell morphology. The study uses transfer learning, where generic, pre-trained YOLOv8 computer vision models are fine-tuned with microscope image data from the Adriatic Sea. The study shows that, for this task, it is possible to effectively fine-tune models trained on out-of-domain images and that this is possible with a small training dataset. The results show high accuracy in detecting and segmenting phytoplankton cells from the microscopic images of the two selected phytoplankton taxa. For detection, the model achieves AP scores of 88.1% for Pseudo-nitzschia cf. delicatissima and 90.9% for Pseudo-nitzschia cf. calliantha, while for segmentation, the scores are 88.4% for Pseudo-nitzschia cf. delicatissima and 91.2% for Pseudo-nitzschia cf. calliantha. Compared to manual image analysis, the developed automatic method significantly increases the number of samples that can be processed.

1. Introduction

Phytoplankton are marine microorganisms that are crucial for the production of organic matter and the carbon cycle in the oceans and account for about 50% of global primary production [1]. These microorganisms play an important role in structuring pelagic food webs and the ocean’s biological carbon pump [2] and their diversity and community structure are important indicators of the health of marine ecosystems. The diversity of phytoplankton influences the structure of and processes that occur within marine ecosystems, with cell size and size-related traits being critical factors for nutrient uptake and retention within the euphotic zone. Climate change, which is characterised by increased seawater temperatures and stratification, is leading to a trend towards smaller phytoplankton cells [3].

It is important to understand how changing physical and chemical conditions in the sea affect phytoplankton physiology and community structure, and how these changes disrupt marine ecosystems and their functions. Understanding these impacts is critical for assessing the health of aquatic ecosystems and predicting future changes in primary production, biomass and carbon export to the seafloor. For this reason, information on phytoplankton functional traits is crucial for aquatic ecologists to assess trait variability and link diversity to ecosystem functioning [4]. The body or cell size of microorganisms is considered a master trait, not only because other morphological traits can be derived from it, but also because it influences other general traits such as resource acquisition and defence [5].

Conventional microscopy techniques for determining the abundance, diversity and cell size of phytoplankton are time consuming and require experienced taxonomists. These limitations hinder the processing of the large amount of samples required for a comprehensive analysis.

Recently, the use of machine learning (ML) in aquatic ecology and marine biology has evolved significantly due to the need to process large-scale ecological data efficiently and accurately. Automated computer vision systems can now process large datasets of microscope images quickly and with high precision, reducing the reliance on human experts and making them invaluable tools for ecological monitoring. These advances have been made possible by deep neural networks (DNNs), which are able to learn complex patterns and representations from large datasets [6]. Deep learning, which encompasses a range of deep neural network architectures, has been critical to advances in computer vision. DNN architectures such as convolutional neural networks (CNNs) [7,8] have demonstrated exceptional capabilities in image analysis, enabling the automatic identification, classification and segmentation of objects. Transfer learning, a technique in which a model trained on a large dataset is fine-tuned on a smaller, domain-specific dataset (see [9,10] for reviews), has further increased the utility of DNNs in specialised domains where training data are scarce.

These technologies have been used for various tasks such as species identification, behavioural analysis and monitoring populations of marine organisms [11]. These applications have significantly expanded the scope and scale of marine research, allowing scientists to gain insights that were previously difficult to obtain manually. An earlier example of an ML approach comes from [12], who developed an automated classification system for phytoplankton using imaging-in-flow cytometry that combined various image features and a support vector machine and achieved a high classification accuracy. Recent studies have shown the effectiveness of deep learning models such as CNNs in analysing phytoplankton images [13]. These models can be trained on large datasets of labelled images to detect different phytoplankton species with high accuracy.

Ciranni et al. [14] provide a comprehensive analysis of the wide range of computer vision techniques and methods that have emerged to facilitate the automated analysis of small- to large-scale datasets of plankton images. One of the studies presented is by the authors of [15], who applied a deep learning-based object detection system for specimen classification on their PMID2019 dataset (10,819 microscopic phytoplankton images with 24 different categories) and compared baseline performances with state-of-the-art object detection models such as Faster R-CNNs, an FPN, a Single-Shot-Detector (SSD) [16], a YOLOv3 [17] and a RetinaNet [18]. When training and testing on the six most common classes and with IoU thresholds of 0.5 and 0.75, they achieved an average precision (AP) score of 90% regardless of the model, with the best performance achieved with YOLOv3 at 93.10%. When evaluated at an IoU of 0.75, YOLOv3 was the worst model with a score of 82.81%, while FPN was the best model with 89.19%. In another study, Luo et al. [13] used CNNs to analyse plankton images (23.4 million images with 108 classes divided into 38 groups) by performing segmentation and classification, for which they achieved an average precision of 84% and recall of 40% for all groups.

Despite the successes, there are still some challenges in the application of ML and computer vision in aquatic ecology. One of the biggest challenges is the need for large, annotated datasets to effectively train deep learning models. Annotating images is a time-consuming process that requires expert knowledge. Transfer learning can be a solution to this challenge as it can significantly reduce the training data requirements. Lysenko et al. [19] focused on the classification of Baikal phytoplankton using CNNs and achieved significant accuracy by using transfer learning.

As emphasised by [20], the ability of today’s imaging systems can go well beyond only examining abundances of different taxa. According to them, the extraction of functional traits will be the next development in ML-based analysis of ecological images. They therefore suggest promising avenues for ML and computer vision approaches to extract information about functional traits from the images, such as biovolume estimation, which is of interest to our study.

The aim of this work is to develop a dedicated computer vision method that can accelerate and automate the monitoring and analysis of microscopic images of phytoplankton taxa. Besides the identification and localisation of phytoplankton species in image data, this method can also provide information on important traits such as cell size by predicting the cell area via instance segmentation with ML. The selection of the model objects—two species of Pseudo-nitzschia—was based on their ecological importance. These species, along with others from the same genus, are an important component of the phytoplankton community in the Gulf of Trieste [21,22]. Additionally, some species of this genus are potentially toxic, including the two selected taxa [23,24]. Therefore, improving methods for identifying species and distinguishing between potentially toxic and non-toxic species can significantly contribute to controlling seawater quality in designated aquaculture sites.

2. Materials and Methods

To train the computer vision model, a training dataset with labelled images had to be created. The microscopic images for this study were obtained from seawater samples collected in May/June 2023 from the southeastern part of the Gulf of Trieste (Adriatic Sea). A total of 306 images of phytoplankton were taken at 400× magnification using the Zeiss AxioObserver Z1 microscope with an integrated AxioCam Mrc5 digital camera. These images contained two species of interest, Pseudo-nitzschia cf. delicatissima (167 images) and Pseudo-nitzschia cf. calliantha (139 images), with the identification constrained by the limitations of light microscopy (Figure 1). For annotation, the distinction between the two taxa was based on the width of the transapical axis (smaller in P. cf. delicatissima) and on the characteristic overlapping of the cell ends in P. cf. delicatissima.

A total of 265 of the 306 microscopic images were manually annotated by identifying individual cell species, and drawing bounding boxes and segmentation masks. The open-source tool Computer Vision Annotation Tool (CVAT) [25] was used for this. These steps were crucial for the creation of a high-quality dataset for training the computer vision models. For cell area estimation, the model must predict a mask that covers the cell’s surface, which requires accurate annotation masks. To evaluate its performance, the labelled dataset was split into a training (80%, 211 images) and a test dataset (20%, 54 images) using stratified random sampling to maintain the class distribution in both sets. In addition, the model internally performed data augmentation that included the following operators: hue, saturation, brightness, translation, scaling, flipping, and mosaic. Due to the small amount of labelled data, no tuning of the hyperparameters was performed.

To achieve the goal of the study, the computer vision system had to perform object detection and instance segmentation tasks. In object detection, objects in an image are classified and localised by drawing a bounding box around the objects, which facilitates the counting and identification of objects in an image. Instance segmentation involves detecting and delimiting the individual objects in an image, which enables precise measurement of the individual objects.

Due to the limited amount of training data (only 211 images for the two phytoplankton species), the approach involved fine-tuning two generic pre-trained models: one for object detection and one for instance segmentation. The models from the You Only Look Once (YOLO) series, in particular YOLOv8 (https://yolov8.com/, accessed on 29 July 2024), are among the most effective for these tasks. YOLO models are known for their speed and accuracy, making them suitable for real-time applications [17]. The architecture of the YOLOv8 model consists of (1) Backbone (CSPDarknet53, a CNN responsible for feature extraction), (2) Neck (C2f module that combines high-level semantic features with low-level spatial information) and (3) Head (making predictions) [26].

For this study, we selected the smallest Nano YOLOv8 models with 3.2 million (detection) and 3.4 million (segmentation) parameters. Both were pre-trained on the Common Objects in Context (COCO) dataset for large-scale object detection, segmentation and labelling [27]. Then, transfer-learning was used to fine-tune the pre-trained YOLOv8 model by re-training it on our annotated training dataset. This adapted the model to the specific characteristics of the selected phytoplankton species and improved its ability to accurately segment and detect individual phytoplankton cells. Key hyperparameters for training included a learning rate of 0.01, an image size of 1920 × 1920 pixels and a batch size of 16. The model was trained for 70 epochs, with training loss monitored to ensure convergence.

The performance of the model was evaluated using standard metrics for object detection and instance segmentation, including mean average precision (mAP) and intersection over union (IoU). IoU is a metric used to evaluate the accuracy of an object detector by comparing the overlap between the predicted bounding box and the ground truth bounding box. mAP is a crucial performance measure that captures the trade-off between precision at different levels of recall to evaluate the overall accuracy of object detection or segmentation models across multiple classes and selected IoU thresholds. These metrics provide a measure of the model’s accuracy and precision in phytoplankton cell detection and segmentation. For an additional qualitative inspection of the results, a limited random sample (10% of the test set) of the predicted bounding boxes and segmentation masks was visualised and manually inspected to improve understanding of the models’ results and trustworthiness with domain experts (see Figure 3).

The development was performed in Python within JupyterLab, the latest interface of the Jupyter project [28], YOLOv8 computer vision library, the scikit-learn machine learning library [29], pandas [30] and, for visualisations, seaborn [31] and Matplotlib [32].

3. Results

After fine-tuning, the YOLOv8 Nano models showed high accuracy in the detection and segmentation of phytoplankton cells from the microscopic images of the two species. For the detection, at an IoU of 0.5, the mAP for both species was 89.5%, and the AP was 88.1% for Pseudo-nitzschia-delicatissima and 90.9% for Pseudo-nitzschia-calliantha. For segmentation, the values were similar. The mAP for both species was 89.8%, and the AP was 88.4% for Pseudo-nitzschia-delicatissima and 91.2% for Pseudo-nitzschia-calliantha. At higher IoU thresholds (0.50 to 0.95), the mAP scores for both classes were 71.3% for detection and 51% for segmentation, respectively. Such mAPs indicate that the model could accurately localise and classify phytoplankton species and that the segmentation masks generated by the model were precise and provided an accurate estimate of cell area. Figure 2 shows the performance metrics for evaluating the model on the test set.

Figure 2. Performance metrics of the model’s evaluation on the test set for both tasks, (a) object detection and (b) instant segmentation. The area under the curve is the AP for each class and mAP for both classes.

Figure 2. Performance metrics of the model’s evaluation on the test set for both tasks, (a) object detection and (b) instant segmentation. The area under the curve is the AP for each class and mAP for both classes.

Figure 3. Visual inspection of the predicted classes, bounding boxes and segmentation masks. Separately drawn masks show the precision of the masks and how the model was able to discern difficult areas where cells chain together. While a few pixels in the difficult areas at the beginning and end of the cells are not correctly predicted, the most of the cells’ bodies are drawn out correctly.

Figure 3. Visual inspection of the predicted classes, bounding boxes and segmentation masks. Separately drawn masks show the precision of the masks and how the model was able to discern difficult areas where cells chain together. While a few pixels in the difficult areas at the beginning and end of the cells are not correctly predicted, the most of the cells’ bodies are drawn out correctly.

Furthermore, a visual inspection of the predicted bounding boxes and segmentation masks confirmed the precision of the models’ results (Figure 3). It is important to note that the chaining of the cells of P. cf. calliantha and especially P. cf. delicatissima makes it difficult even for human annotators to distinguish one cell from another at the contact points where the cells chain together (Figure 3). This also makes the segmentation task very difficult for the computer vision model. Nevertheless, the model also performed quite well on this task, less optimal performance can be observed in the areas where the cells overlap since the model is struggling to discern to which cell instance a pixel belongs (Figure 3).

To investigate how the distributions of the actual cell area for both taxa correspond to the predicted ones, we superimposed the ground truth and predicted cell area distributions (Figure 4). The distributions of the ground truth (Figure 5) and the predicted cell area (Figure 4) show a similar pattern, indicating that the model is able to generalise well to new data. This indicates that the model can reliably estimate the cell area of phytoplankton.

4. Discussion

The results of this study demonstrate the effectiveness of using computer vision techniques to identify phytoplankton and estimate important functional traits, such as cell size and biovolume. Our methodology closely matches the proposed protocols for characterising the functional traits of plankton from image data as described in a recent comprehensive review of ML techniques in [20]. The authors point out that biovolume and biomass can be estimated from the measured area of an organism or cell, and mention the application of segmentation algorithms when research requires pixel-level analysis. They also recommend evaluating the performance of the algorithms with the mAP score.

One of the challenges the study faced was the limited number of labelled images. This problem was addressed using transfer learning which allowed the model to leverage knowledge gained from pre-training on the COCO dataset. Our results show that fine-tuning of a generic pre-trained computer vision model is possible with a relatively small number of images, even if these images are from different domains (e.g., the COCO dataset does not contain microscopic images). This is in line with similar studies on plankton ecology where the benefits of using such approaches for classification have been reported (e.g., [33,34]).

Most similar published work is on marine plankton differentiation (classification) and detection; however, for comparisons to our study, the most suitable are those dealing with instance segmentation on plankton images, which are less abundant. The recent comprehensive review on automatic plankton image recognition by [35] only mentions the comparison in [36] of a semantic segmentation model SegNet [37], and an instance segmentation model Mask R-CNN [38] for algae detection and recognition.

Ruiz-Santaquiteria et al. [36] used Mask R-CNN, which is based on ResNet101 and was pre-trained on COCO for the instance segmentation of 126 plankton images (10 different taxa). Their best model for instance segmentation achieved an average precision of 85% with 86% recall. Unfortunately, the authors do not provide the mAP and IoU thresholds that would allow a more direct comparison. Ciranni et al. [14] summarise the work of [39] who compare three alternative approaches for plankton entity segmentation in flow-cytometry images. Similar to our approach, their Mask R-CNN model was pre-trained on COCO and they manually annotated over 3000 images. The method relying on the Mask R-CNN showed the best trade-off in terms of precision (16.3%) and recall (91.9%). In their study, Bergum et al. [40] also used a Mask R-CNN pre-trained on the COCO dataset and fine-tuned on their own dataset of 126 images of planktonic organisms (copepods). With their fine-tuned X101-32x8d-FPN-3x model, they reported an AP of 41.1% for instance segmentation.

Another study dealing with similar tasks of phytoplankton detection and instance segmentation in microscope image data from water samples was carried out by [41], who used classical ML algorithms and exploited texture and colour features in the images. Similarly to our approach, they used an IoU of 50% for a detection as a true positive. For phytoplankton detection, they reported that the best model was the random forest classifier using texture features with a precision of 77.2% and a recall of 90%.

In comparison, our model was fine-tuned with only 211 images and also included only two classes, which is less than most of the similar studies. Our model achieved an mAP score of 89.5% for detection and 89.8% for instance segmentation at an IoU of 0.5. At higher IoU thresholds (0.50 to 0.95), mAP scores were 71.3% for detection and 51% for segmentation, respectively. Although our main evaluation metric was mAP, we also report a precision of 84.9% and recall of 84.2% for instance segmentation to facilitate comparison with studies that reported these metrics, although they refer to classification evaluation, which is a simpler task than instance segmentation.

These comparisons underscore the effectiveness of our approach in processing instance segmentation tasks for extracting information about phytoplankton functional traits from microscopic images. The results show that while larger annotated datasets generally contribute to a higher accuracy, the combination of transfer learning and accurate annotation can yield competitive results even with a small number of annotated images. Furthermore, it was observed that, during training, the higher-resolution images improved the precision of segmentation tasks, which is likely applicable to real-world scenarios. The model’s strong performance on these challenging tasks can also be attributed to the accurate mask annotations in the training set, though creating a large number of such annotations can be costly.

In our future work, we will focus on refining the model to solve difficult cases, such as overlapping cells, and improving the precision of segmentation masks, which is particularly relevant for cell area estimation. To this end, performance improvements should be sought at higher IoU settings during training, as this will ensure that more accurate segmentation masks are generated. Higher IoUs would place higher demands on the precision of the model in capturing the exact contours of thin and elongated objects such as the studied Pseudo-nitzschia species. However, this could also come at the cost of false negatives, meaning that some cells are missed altogether.

We also plan to expand the dataset by annotating more images, as this would likely further improve the performance of the model. We would also like to include other important phytoplankton species. Testing the model in real environments, will allow further validation and insight into its practical applications, where the approach can also be adapted for optimal cost–benefit.

5. Conclusions

Our study demonstrates the potential of computer vision techniques, particularly instance segmentation, in analysing phytoplankton functional traits, enabling aquatic ecologists to better understand their community structure. Our object detection model accurately detects and identifies the two phytoplankton from microscopic images. The instance segmentation model provides a good estimate of individual cell area, which is necessary for estimating another important functional trait, biovolume. The study shows that it is possible to fine-tune a generic pre-trained model trained on out-of-domain images, and that this is possible even with a relatively small training dataset.

Compared to manual analysis of phytoplankton images, the automated method significantly increases the number of samples to be processed. Such automation could provide a scalable solution to the spatially and temporally extensive monitoring programs.