Classification of Salmon Freshness In Situ Using Convolutional Neural Network ^†

Abstract

Fish is an important food resource, an economic contributor, and a staple food for Filipinos. For the safety and satisfaction of consumers, fish freshness must be determined. Using the convolutional neural network (CNN) algorithm, we determined salmon fillet freshness in this study. In total, 7000 images were used for training and 40 for testing the CNN model. The deep learning technique, specifically ResNet50 architecture, was used with Raspberry Pi 4B, and Raspberry Pi camera V₂ was employed to take images of fish. The model showed a 92.5% accuracy, highlighting the CNN model’s accurate evaluation of seafood quality.

1. Introduction

Fish is an essential food and a source of livelihood in the Philippines. Fishing has significantly contributed to the economy of the Philippines by creating income and employment [1]. Fish is extremely perishable without the proper conditions, as the quality of the fish deteriorates and eventually spoils easily. Fish freshness is important to monitor for safe consumption. Fish freshness is commonly determined by sensory methods, such as sight, smell, and touch. Such traditional methods are used to determine the freshness of fish but are only conducted by trained and experienced people. Training for the traditional method is a long process, and it is difficult to make quantitative sensory assessment [2]. Therefore, we developed a straightforward and effective method for assessing fish freshness using the convolutional neural network (CNN) algorithm.

Salmon freshness is determined usually by its appearance. The eyes of the salmon must be clear and bulge a little, the gills must be bright red, the flesh must be firm and shiny, and the colors of the edges of the fish must not be brown or orange. These are ways used to determine salmon freshness. Other than the sensory methods, a variety of methods have been proposed to determine the freshness of fish using image processing techniques, a complementary metal-oxide-semiconductor (CMOS) fish freshness detector, and near-infrared (NIR) spectroscopy. These are viable methods for determining fish freshness.

In Ref. [3], a fish freshness monitoring system was designed using a CMOS fish freshness detector in the Internet of Things (IoT) applications. A chip fabricated with CMOS technology was used in the continuous-time incremental sigma-delta modulator. The system used MATLAB software version 7.0.4 and SPICE simulation software for simulations in fish freshness monitoring. NIR was also used for determining the freshness of a Rohu fish [4]. The data were analyzed and clustered using principal component analysis (PCA). Reference [5] proposed a nondestructive framework for determining fish freshness using image processing techniques. The freshness of fish was determined using its skin tissue. The accuracy of the method reached 96.66%. In Ref. [6], k-nearest neighbor (KNN) was used as the classification algorithm in determining the freshness of fish using the image color summarization of fish skin. The method presented an accuracy of 91.36%. The traditional salmon freshness detection methods depend on the observation of experts.

The salmon’s appearance was observed for the determination of its freshness. NIR spectroscopy, gas sensors, and the radioactive area increase (RAI) were used in determining fish freshness based on the colors and appearances of eyes, gills, and/or skin. There are scarce studies that used CNN to categorize salmon freshness. In this study, we created a system to classify salmon freshness. We collected images of salmon to train CNN to classify its freshness levels. The performance of the CNN model was evaluated using its accuracy using a confusion matrix.

The model and its results benefit the fish industry, as it provides an accurate method to assess the freshness of salmon with convenience as the freshness of fish is a significant factor that affects consumer’s willingness to pay. Using the model, the food safety and consumer’s willingness to buy are ensured.

2. Literature Review

You Look Only Once (YOLO) V₃ and CNN algorithms were used to classify the type of otitis media infection [7]. The CNN model classified the detected tympanic membrane of acute otitis media and chronic suppurative otitis media (AOM/CSOM). The detection accuracy was 75%. A CNN algorithm was also used in estimating chicken and fish weight and calorie content. In determining the parts of food in an image, the graph-cut image segmentation technique was used. The model’s food detection accuracy was 91.82% and the overall accuracy for fried or grilled chicken and fish was 88.18% [8]. Abaca plant diseases were also identified using the CNN-VGGNet-16. A total of 300 training datasets were used to train the model. The training data were split to 80 and 20% for training and validation. The model predicted with an accuracy of 88.9% [9]. Reference [10] determined corn damages using CNN and colored image edge detection. Image processing reduced human errors in corn quality assessment and detected the quality-related factors of corn kernels efficiently. The model’s accuracy was 96.66% in determining different corn kernel damages [10]. Avocado ripeness was assessed using a graph neural network [11]. A total of 400 avocado images were used in training the algorithm, with 200 images per class. In total, 400 avocadoes were experimented with to classify their ripeness. The accuracy of the avocado ripeness classification with GNN was 97.75%. OpenMP and CNNs were used to detect corn leaf diseases [12]. The accuracy of detecting leaf rust was 89%, while that for leaf spot and leaf blight was 89 and 93%. Romaine lettuce health was assessed using CNN. The overall accuracy was 90%, and the model classified healthy Romaine lettuce successfully [13]. Reference [14] created a computer vision system to classify cocoa beans. The model’s accuracy was 90.67%. A system to classify the freshness of chicken meat using VGG 16 CNN showed an accuracy of 94.12% [15]. An electric nose and a deep-learning model were used to classify tomato ripeness in Ref. [16]. The system correctly identified 43 out of 50 samples, with an accuracy of 86%. The freshness and food spoilage inside a refrigerator were determined using an electric nose and PCA-KNN. The accuracy was 92% [17].

3. Methodology

In this study, the freshness of Salmo Salar sold in fish markets in the Philippines was determined using CNN. A total of 40 images of salmon fillets were used for testing, and 7000 were used for training. In total, 1750 images belonged to the classes of “Fresh Fish (FF)”, “Stale Fish (SF)”, “RF (RF)”, and “Unknown Fish (UF)”. ResNet50 was used with 50 layers. A Raspberry Pi 4, Raspberry Pi camera Module V₂, and an LCD were included in the hardware (Figure 1).

Figure 1 shows the flowchart of this study. We designed the system hardware and software. The additional layer in the ResNet50 was added for the output of the model. The parameters in the model included image dimensions, batch size, data split for training and validation, random seed for reproducibility, model architecture, optimizer, loss function, metrics, and number of epochs. The image height and width were set to 180 pixels. The batch size was 32. The data were split for training and validation in a ratio of 0.8:0.2. The random seed was set to 123.

The input image had the dimensions of (180, 180, 3), and the added layers were classified into Flatten (which does not take any paremeters), Dense (16, activation = “relu”), and Dense (4, activation = “softmax”). For the loss function, sparse categorical crossentropy was used. The Adam optimizer was used with a learning rate of 0.001. The number of epochs was 10. The hardware consisted of the Raspberry Pi 4 and the camera module. Gaussian filtering was used for the noise reduction of the image. The model classified the freshness of the salmon using CNN with the trained ResNet50 and the imageAI library. The performance of the system was evaluated using a confusion matrix.

We took the images of salmon at varying states of freshness. The salmon images were captured in Cubao Fish Market, the Phillippines. The acquired images were processed to obtain the best output. A total of 7000 salmon fillet images were used for training, of which 1750 images were used for each class. For image processing, ResNet50 was used with an additional layer. The model was trained with the parameters for image augmentation. Using the imageAI library, the freshness of the salmon was classified. The trained model determined the freshness of the salmon and displayed the results in each class (Figure 2).

3.1. Hardware

Figure 3 shows the hardware block diagram of the study. The main hardware in the study is the Raspberry Pi 4. The Raspberry Pi Camera V2 is the camera module that has been implemented and linked to the Raspberry Pi 4. The Raspberry Pi Camera V2 is an 8-megapixel sensor capturing the salmon samples that will be implemented in the study. A Raspberry Pi 4 Power Supply is also connected to the module for powering the device. Lastly, a Liquid Crystal Display (LCD) is used to output the freshness level of the salmon sample.

3.2. Software

The project was implemented using Python 3.7. A Python package called “ImageAI” was imported with the ImagePrediction for the ImageAI Library for object detection. An eight-megapixel camera was used to take images. After the image was preprocessed, the ImagePrediction object loaded the model and then categorized the images. The level of freshness was determined based on probability. The output was shown on the LCD. The image preprocessing was conducted to resize the images to 180 × 180 pixels to match the dataset size. The noise was reduced using the Gaussian Blur filter. Next, the image dimensions were expanded because the dataset was trained on different images. The ResNet model predicted the level of freshness (Figure 4).

The ResNet50 algorithm’s model accuracy for 10 training epochs is displayed in Figure 5.

3.3. Experiment

The system for the determination of the freshness of salmon was constructed as shown in Figure 6. Its dimensions were 30 × 30 × 30 cm (length, width, and height), and it was made of acrylic. On the top of the case, the camera module was installed in a box of 30 × 20 × 10 cm. The salmon sample was placed inside the box, and the Raspberry Pi 4 camera captured images. The seven-inch LCD was placed at the top of the prototype, facing the front. Lighting equipment was placed to light the salmon fillet. The images were taken at least 35 cm from the camera. The LCD displayed the freshness level of the salmon. The output data were recorded to determine the accuracy using the confusion matrix.

3.4. Data Collection

The salmon samples were bought in a local fish market “Ging Fresh Seafood” in Cubao, Phillippines. Each fillet was observed and classified by an expert. The salmon fillet images were then captured. An initial 600 images were obtained. After data augmentation, the dataset contained 7000 images. In the classes of FF, SF, RF, and UF, 1750 images were contained for training the model.

4. Results and Discussion

A total of 40 images were used in testing the system. In total, 10 images were assigned to fresh, stale, rotten, and unknown classes. A multi-class confusion matrix was created to determine the classification accuracy. FF represented the salmon in its best condition, sold after a certain amount of time stored in appropriate conditions. SF stood for a longer period of storage, with a subtle color difference compared with FF, but was still safe to eat. RF presented fish fillets deteriorated to not safe to eat and with noticeable differences in color and texture. UF showed a different species of fish.

The color and texture of the salmon fillet were used to classify the fish fillet. FF showed a bright and firm texture. SF showed a slightly faded color, a softer texture, and small gaps visible on the surface. RF showed a faded color, the texture crumbling upon touch, and big gaps on the surface of the fish.

Table 1. Confusion matrix.

Actual Classification	Predicted Classification
		FF	SF	RF	UN	Total
	FF	10	0	0	0	10
	SF	0	8	1	1	10
	RF	0	0	9	1	10
	UN	0	0	0	10	10
	Total	10	8	10	12	40

shows the confusion matrix displaying the actual and the predicted classification of the system. For FF, all 10 images were correctly classified. For SF, eight were classified correctly. For RF, nine images were accurately classified. For UF, all 10 images were correctly classified as unknown.

In calculating the performance of the classification, different metrics were used: true positive (TP), false positive (FP), true negative (TN), and false negative (FN).

The accuracy was calculated by dividing the sum of true positives by the sum of all classifications. The sum of true positives and true negatives was 37, which was divided by the total number of classifications. The overall accuracy was 92.5%.

$$\mathrm{ACC}={\displaystyle \frac{{{\displaystyle \sum }}_{n=1}^4{\mathrm{A}}_{\mathrm{nn}}}{{{\displaystyle \sum }}_{\begin{array}{c}i=1\\ j=1\end{array}}^4{\mathrm{A}}_{\mathrm{ij}}}}$$

(1)

ACC = 92.5%

5. Conclusions

We created a model and a system for classifying the freshness of salmon fillets. CNN with ResNet50 was used with the Raspberry Pi 4B and a Raspberry Pi camera module V₂, respectively. The system successfully classified 37 out of 40 salmon sample images with an accuracy of 92.5%. The system successfully classified salmon meat freshness.

More training of the ResNet50 algorithm is required by adding more salmon images for each class at different orientations. Integrating an electric nose enhances the accuracy of the model. By adding a deep learning algorithm, a hybrid model can be created to enhance the accuracy of classification. Also, adding a cooling unit to the prototype is needed to keep the freshness of the fish and avoid degradation while testing.