An Automated Framework Based on Deep Learning for Shark Recognition

Abstract

The recent progress in deep learning has given rise to a non-invasive and effective approach for animal biometrics. These modern techniques allow researchers to track animal individuals on a large-scale image database. Typical approaches are suited to a closed-set recognition problem, which is to identify images of known objects only. However, such approaches are not scalable because they mis-classify images of unknown objects. To recognize the images of unknown objects as ‘unknown’, a framework should be able to deal with the open set recognition scenario. This paper proposes a fully automatic, vision-based identification framework capable of recognizing shark individuals including those that are unknown. The framework first detects and extracts the shark from the original image. After that, we develop a deep network to transform the extracted image to an embedding vector in latent space. The proposed network consists of the Visual Geometry Group-UNet (VGG-UNet) and a modified Visual Geometry Group-16 (VGG-16) network. The VGG-UNet is utilized to detect shark bodies, and the modified VGG-16 is used to learn embeddings of shark individuals. For the recognition task, our framework learns a decision boundary using a one-class support vector machine (OSVM) for each shark included in the training phase using a few embedding vectors belonging to them, then it determines whether a new shark image is recognized as belonging to a known shark individual. Our proposed network can recognize shark individuals with high accuracy and can effectively deal with the open set recognition problem with shark images.

1. Introduction

The ability to reliably identify and track individuals within a population or habitat is essential for understanding population dynamics, behavioral patterns and ecosystem function. Historically, animal marking (e.g., branding, notching, external tags), which began in the 1700s, has been used as a tool for quantifying behavior and identifying individuals previously sampled and counted. In the 1960s, electronic tags such as acoustic, radio and satellite transmitters were developed that could be attached to individuals, allowing researchers to track their movement and habitat use [1]. However, these human-made markings can fade and tags can be shed over time, making their utility less valuable for population estimation and long-term behavioral-pattern interpretation [2]. While applying these technologies to large, highly mobile species, particularly, aquatic species provided some of the only approaches for studying populations; they require the capture of individuals and have been limited in sample size due to costs. With improved camera technology, other techniques such as photogrammetry have grown in popularity, providing researchers a permanent record for identifying and counting individuals with unique natural marks, scars and features that are less likely to fade or be lost over time [3,4]. While this technology is now becoming commonplace and datasets continue to grow, data processing, analysis and categorization of unique features has been time consuming and labor intensive [5]. Recently, advances in computer-vision techniques combined with animals’ unique characteristics, animal biometrics, have introduced a more effective and non-invasive approach to recognize animal individuals. These approaches rely on large-scale datasets and involve manual or semi-manual work to identify individuals [6,7,8].

A disadvantage of these above approaches is that they require feature engineering. For example, users have to select reference points for each image manually [7,9]. Therefore, the recent research has been focused on automating the recognition process to locate, identify and classify images. In general, an automated recognition process includes methods to detect animals from backgrounds and to extract unique characteristics of the detected animals. For example, a deep-learning network model was able to extract unique features from cattle’s muzzle images, and a matching scheme based on the extracted features was used for cattle recognition [10,11]. This approach can reduce human biases and help us conduct research on large scale. When it comes to shark recognition, there were approaches using sharks’ dorsal fin as the animal’s unique characteristics. For example, Hughes and Burghardt [12] developed the partially automated contour-based visual ID system in shark biometrics. They focused on the contour information of the sharks as their biometric signature. First, they developed an open-contour stroke model to achieve robust fin detection. Second, they used scale-space selective fingerprinting to encode dorsal-fin patterns individually. Finally, they proposed a non-linear model for individual shark recognition.

This paper aims to provide a fully automated and vision-based framework to recognize shark individuals from underwater images. Our framework has two main components: shark detection and shark recognition. For the shark-detection phase, we develop a system to capture the shark body from underwater images. Given a training set including images and shark bodies, we train a network capable of predicting the shark-body region from a new image. Once the trained network predicts the region from a new image, we crop out the region and get rid of the background using Gaussian smoothing. These images are the input for the next phase. For the shark-recognition phase, we design a deep neural network that maps an image of the lateral view of a shark’s body to an embedding vector in latent space. To train the network, we use the triplet loss function, which enforces that the distances between the same sharks’ embedding vectors are smaller than the distances between the embedding vectors of different sharks. The embedding vector is a representation of each unique shark.

To adapt our recognition system to the open-set recognition scenario, we split the data into training, known test, and unknown test sets. The training set includes images of sharks and their labels, which is provided to train the network with the triplet loss. After the training phase, a few ($k\le 20$) images for every shark in the known test set are selected and their embedding vectors are stored. Using these embedding vectors as inputs into OSVM [13], we compute the decision function for the k embedding vectors of these known sharks. During testing, an image in either the known test set (excluding the ones used to build the decision functions) or unknown test set is given to the system, and the system should be able to answer whether the image belongs to a shark included in the known test set or is a never before seen shark. The system achieves this by presenting an image as an embedding vector and checking whether it is within the decision boundary of the closest known shark’s centroid.

Our contributions are as follows. (1) We develop an automated system to capture shark-body images from underwater images. (2) We propose a network that transforms a shark-body image to an embedding vector. The same sharks’ embedding vectors in latent space have smaller distances than those of different sharks. (3) We introduce a system that recognize the images of known objects as ‘identified whom’ or unknown objects as ‘unknown’ with 81% accuracy.

2. Materials and Methods

2.1. Shark Recognition

We propose an encoder architecture. The encoder $f(·)$ maps images of sharks to embedding vectors in a latent space such that the distance between same sharks is smaller than the distance between different sharks. The encoder includes the VGG-UNet network and VGG-16 network. The encoder maps an image $\mathbf{s}$ to an embedding vector $\mathbf{v}$:

$$\mathbf{v}=f\left(\mathbf{s}\right).$$

(1)

The dimension of the embedding vector is 256 and the vectors are normalized, i.e., $\mathbf{v}\in {\mathbb{R}}^{256}$ and ${\left|\right|f(·)\left|\right|}_2=1$. Let ${\mathbf{s}}_{i,a}$ and ${\mathbf{s}}_{j,a}$ be two images ($i\ne j$) of the shark $id=a$ and let ${\mathbf{s}}_{k,b}$ be an image of the shark $id=b$. Adapting the triplet loss [14], the loss of the proposed network is defined by

$$L=\left|\right|{\mathbf{v}}_{i,a}-{\mathbf{v}}_{j,a}{\left|\right|}_2^2-\left|\right|{\mathbf{v}}_{i,a}-{\mathbf{v}}_{k,b}{\left|\right|}_2^2+\alpha ,$$

(2)

where ${\mathbf{v}}_{i,a}=f\left({\mathbf{s}}_{i,a}\right)$, ${\mathbf{v}}_{j,a}=f\left({\mathbf{s}}_{j,a}\right)$, ${\mathbf{v}}_{k,b}=f\left({\mathbf{s}}_{k,b}\right)$, and $\alpha$ is a margin. We determined the margin $\alpha$ after comparing the performances of the system with different margin values. We evaluated the system with $\alpha =0,0.5,1.0,1.5,\mathrm{and}2.0$. We set $\alpha =1.0$ because the systems’ performance is best with $\alpha =1.0$. The loss tries to ensure the Euclidean distance between ${\mathbf{v}}_{i,a}$ and ${\mathbf{v}}_{j,a}$ is smaller than the distance between ${\mathbf{v}}_{i,a}$ and ${\mathbf{v}}_{k,b}$ for all possible triplets in the training set. The encoder architecture is illustrated in Figure 1.

2.1.1. VGG-UNet

The input of the network is a three channel image (RGB) of shape $224\times 224$ [15]. As shown in Figure 2, there are two main paths in the network: The contracting path (left side) and the expansive path (right side). The contracting path consists of four blocks; each block includes two $3\times 3$ convolutional layers of stride 2 without padding followed by a rectified linear unit (ReLU) and a $2\times 2$ max-pooling operation. At each block, the number of channels are doubled from 64. The expansive path includes three blocks; each block consists of $2\times 2$ up-convolution that halves the number of channels followed by a concatenation with the corresponding cropped feature map from the contracting path and two $3\times 3$ convolution operation. At the final layer, we use $1\times 1$ convolution operation to obtain the 2 channel segmentation map from 64 channel feature map. The output of the network is a two channel image of shape $112\times 112$.

We trained the network capable of partitioning images into two regions: the shark and the surrounding area from original images. The training images consist of the images captured from the videos and the corresponding segmentation maps which were manually labeled by human. Once the network is trained, we predicted the masks using our images captured from the videos as input. During the test phase, for an image of arbitrary shape, the image was reshaped to be $224\times 224$. For each captured image, the network’s output is a tensor of shape $112\times 112\times 2$. For $0\le i<112$ and $0\le j<112$, the probability that the pixel is part of the shark region is denoted by $seg\left[i\right]\left[j\right]\left[1\right]$, and the probability that the pixel belongs to the background is denoted by $seg\left[i\right]\left[j\right]\left[0\right]$. After that, the annotation mask $pr$ was created such that $pr\left[i\right]\left[j\right]=1$ if $seg\left[i\right]\left[j\right]\left[0\right]\le seg\left[i\right]\left[j\right]\left[1\right]$ and 0 otherwise. The prediction map was then reshaped to be $224\times 224$ using nearest neighbor interpolation. Combining the initially captured image and the annotation mask, we cropped the region occupied by sharks. We then used a Gaussian filter to generate a smoothing boundary between the shark and background as the final output image.

2.1.2. VGG-16 Network

As our dataset is relatively small, we used the pre-trained VGG-16 as our base model. The pre-trained VGG-16 is a deep convolutional neural network (CNN) [16] that has shown significant results for large-scale image-recognition tasks. The network was trained using the ImageNet database [17], which includes 1.3 training million images of 1000 classes [18]. The pre-trained VGG-16 includes five blocks of convolutional layers and three fully connected layers. We adapted the pre-trained weights of five blocks of convolutional layers and added four fully connected layers to extract the features of the sharks. The input to the network was a three-channel image of shape $224\times 224$. The network includes five blocks of convolutional layers; each block includes $3\times 3$ convolutional layers of stride 1 with padding 1 followed by a ReLU and a $2\times 2$ max-pooling operation. At each block, the number of channels was doubled from 64. The number of convolutional layers for the first two blocks are two, and the last three blocks are three. The output of the last maxpooling layer is a matrix of shape $7\times 7\times 512$. To extract the unique features of the shark bodies, we flattened this output to make it a $25,088\times 1$ feature vector, then we added four fully connected layers with 4096, 1024, 512, and 256 units in that order. In addition, we added a dropout layer with the dropout rate of 0.3 after each fully connected layer, except the last one. Figure 3 shows the proposed network model.

2.1.3. Few-Shot Learning

Figure 4 shows the shark-recognition process using the trained model. For $n\ge 1$, let $\left\{{\mathbf{s}}_{i,a}\right|1\le i\le n\}$ be the set of the randomly selected images of the sharks $id=a$ in the known dataset and $\{{\mathbf{v}}_{i,a}=f\left({\mathbf{s}}_{i,a}\right)|1\le i\le n\}$ be the set of corresponding embedding vectors using the trained model. For each shark $id=a$ in the known dataset, the system computes the centroid of n embedding vectors, defined by ${\mathbf{M}}_a=\frac{1}{n}{\sum }_{i=1}^n{\mathbf{v}}_{i,a}$. The system learns decision functions in the latent space by using the OSVM algorithm [13] for all shark individuals. The algorithm obtains $\left\{{\mathbf{v}}_{i,a}\right|1\le i\le n\}$ as an input and solves the following optimization problem

$$\left\{\begin{array}{cc}\hfill & \underset{\alpha }{min}\frac{1}{2}\sum _i^n\sum _{i^{\prime }}^n{\alpha }_i{\alpha }_{i^{\prime }}𝒦({\mathbf{v}}_{i,a},{\mathbf{v}}_{i^{\prime },a})\hfill \\ \hfill & \mathrm{subject}\mathrm{to}:0\le {\alpha }_i\le \frac{1}{\nu n},\sum _{i=1}^n{\alpha }_i=1\hfill \end{array}\right.$$

(3)

where $\mathcal{K}(\mathbf{v},{\mathbf{v}}^{\prime })=e^{-\gamma \left|\right|\mathbf{v}-{\mathbf{v}}^{\prime }{\left|\right|}_2^2}$ is a radial bias kernel function, ${\alpha }_i$ are the Lagrange multipliers, $\gamma$ and $\nu$ are among the hyper-parameters of the system.

Let ${\mathbf{s}}_{*,u}$ be an image of an unknown shark u. For each shark, the decision function of ${\mathbf{v}}_{*,u}$ for the participant $id=a$ is defined by $h_a\left({\mathbf{v}}_{*,u}\right)={\sum }_i^n{\alpha }_i\mathcal{K}({\mathbf{v}}_{i,a},{\mathbf{v}}_{*,u})-{\rho }_a$ where ${\rho }_a={\sum }_i^n{\alpha }_i\mathcal{K}({\mathbf{v}}_{i,a},{\mathbf{v}}_{l,a})$ for any l such that $0<{\alpha }_l<\frac{1}{\nu n}$ and $1\le l\le n$. Considering that an unknown participant can be either one who was included in the training dataset or one who was not, the system determines the prediction of u as follows;

1.

Compute ${\mathbf{v}}_{*,u}=f\left({\mathbf{s}}_{*,u}\right)$;

2.

Find provisional shark $p=arg{min}_a\left|\right|{\mathbf{M}}_a-{\mathbf{v}}_{*,u}{\left|\right|}_2$;

3.

If $h_p\left({\mathbf{v}}_{*,u}\right)\ge \tau$, then “u is recognized as p”;

4.

Otherwise, “u is not recognized as any known shark”;

where $\tau$ is a system’s hyper-parameter.

In addition, we implemented a simple Euclidean-distance-based recognition framework and evaluated its performance. To determine whether a test image belongs to known objects, we compared the Euclidean distances between the test image and the centroids of the known objects in the latent space. By setting a threshold for the Euclidean distance as a hyper-parameter, we evaluated the performance of the system. However, because its performance is lower than our proposed method, we did not include the comparison result here.

3. Results

Datasets and Evaluation Metric

The original videos (4k, 30 fps) were recorded using GoPro Hero 8 underwater cameras at Guadalupe Island, off the west coast of Baja, Mexico. Figure 5 shows how the original videos were collected. From the videos, 12-megapixel frame images were generated and we manually removed images that did not include shark bodies. Our dataset includes images of 13 different shark individuals (subjects).

The data were divided into three sets: training, known test, and unknown test. First, we sampled six subjects from 13 subjects; for each subject, 75% of its images were allocated into the training set and the other 25% were allocated into the known test set. Second, for each subject in the known test set, we randomly selected 20 images to determine the decision boundary in latent space using OSVM algorithm. Finally, all images of the remaining 7 subjects were assigned to the unknown test set. The known test, unknown test, and training sets contain approximately 1527, 455, and 456 images, respectively. Such datasets were generated 20 times repeatedly and independently. For each data generation, we train and test the proposed method, and the averaged evaluation metrics are summarized.

We define an image to be true positive (TP) if it belongs to an individual in the known test set and the network correctly identifies it, and a false positive otherwise. On the other hand, we define an image to be true negative if it belongs to the unknown test set and the network correctly classifies it as such, and false negative otherwise. We report the true positive rate $TPR=\frac{TP}{TP+FN}$, the true negative rate $TNR=\frac{TN}{TN+FP}$, and the accuracy $ACC=\frac{TP+TN}{TP+FN+TN+FP}$.

The contour plots of ACC, TPR, and TNR as a function of hyper-parameters $\gamma$ and $\nu$ are shown in Figure 6. All of the evaluation metrics are correlated with the selection of $\gamma$ and $\nu$, especially the selection of $\nu$ is for the system’s performance. The setting of hyper-parameters needs to be tuned considering both TPR and TNR simultaneously. If we select $\gamma$ and $\nu$ values to obtain high TPR, in general, those selection results in low TNR. More concretely, if the system rejects all images, then we could obtain a perfect score for TNR, but obtain a zero score for TNR. Therefore, we selected $\gamma$ and $\nu$ values to minimize the difference between TPR and TNR.

The effect of $\tau$ for fixed $\gamma$ and $\nu$ is shown in Figure 7. We can see that the difference between TPR and TNR gets considerably reduced when we select $\tau$ which is smaller than 0. From this, we chose a negative $\tau$ for the decision boundary of OSVM in the latent space. Overall, the best TPR, TNR, and ACC of the proposed system are all 81% with the hyper-parameter setting of $\gamma =1.9,\nu =0.1,\tau =-0.15$.

4. Discussion

To verify the system’s discriminative performance, we computed the Euclidean distances between genuine images (images of same shark) and between imposter images (images of different sharks). Figure 8A shows the genuine–imposter distributions of shark images. It shows that two distribution curves are clearly separated, which means that the proposed system distinguishes the images of already seen shark individuals.

To verify that the proposed method is correctly separating images from different subjects while clustering all images from the same subject, we utilized the UMAP [19] plot of embedding vectors from the known test dataset. Figure 8B shows that the proposed system can learn and distinguish the characteristics of the images of sharks. In this study, our proposed method maps images of sharks to high dimensional vectors in a latent space first then sets the decision boundaries of the subjects in the latent space. Any latent vector that does not belong to those boundaries is recognized as an unknown subject. Considering the dimension of the latent vector, the proposed system is highly scalable in terms of the number of subjects.

Figure 9 shows the optimized images computed for a selection of units in VGG-16 network. To unveil and understand that the activation of neurons at intermediate layers are correlated with input images, we adapted the method for visualizing and interpreting neural networks by Yosinski et al. [20]. For a given input image x and the activation of a neuron a, we obtained an optimized image using a gradient descent algorithm. With a gradient descent step size $\eta$, a single step updates the image

$$x\leftarrow x+\eta \frac{\partial a}{\partial x}.$$

(4)

The optimized images in Figure 9 show that some neurons in the network are correlated with the meaningful features of a shark body such as dorsal fin, tail, head, and belly. Therefore, we claim that the proposed system recognizes shark individuals using such important features of shark bodies.

Figure 10 shows the visualizations of highly activated regions in input images using gradient-weighted class activation mapping (Grad-CAM) [21]. The VGG-16 was pre-trained with the ImagNet dataset, and the dataset includes the classes of sharks including white shark, tiger shark, and hammerhead shark [17]. Given an input image, we forward propagated the image through VGG-16 network and computed the probabilities of every classes; when the predicted class is a shark, then the gradient of the predicted class is back-propagated to the input layer to get the visualization. These visualizations show that VGG-16 is activating meaningful regions of shark body well.

5. Conclusions

We proposed a fully automated and vision-based framework to recognize shark individuals. Our framework is capable of extracting shark images from snapshots of underwater videos. The framework then learns to map each shark image into an embedding vector in latent space such that the Euclidean distance between images of the same shark is smaller than that between images of different sharks. During the testing phase, the framework can identify whether the new image belongs to a shark individual encountered during the training phase or belongs to an unknown shark with 81% accuracy. When we have a database with more named shark individuals, we will be able to utilize the system to recognize and trace them very efficiently with low cost.