Prediction of Fruit Maturity, Quality, and Its Life Using Deep Learning Algorithms

Abstract

Fruit that has reached maturity is ready to be harvested. The prediction of fruit maturity and quality is important not only for farmers or the food industry but also for small retail stores and supermarkets where fruits are sold and purchased. Fruit maturity classification is the process by which fruits are classified according to their maturity in their life cycle. Nowadays, deep learning (DL) has been applied in many applications of smart agriculture such as water and soil management, crop planting, crop disease detection, weed removal, crop distribution, strong fruit counting, crop harvesting, and production forecasting. This study aims to find the best deep learning algorithms which can be used for the prediction of fruit maturity and quality for the shelf life of fruit. In this study, two datasets of banana fruit are used, where we create the first dataset, and the second dataset is taken from Kaggle, named Fruit 360. Our dataset contains 2100 images in 3 categories: ripe, unripe, and over-ripe, each of 700 images. An image augmentation technique is used to maximize the dataset size to 18,900. Convolutional neural networks (CNN) and AlexNet techniques are used for building the model for both datasets. The original dataset achieved an accuracy of 98.25% for the CNN model and 81.75% for the AlexNet model, while the augmented dataset achieved an accuracy of 99.36% for the CNN model and 99.44% for the AlexNet model. The Fruit 360 dataset achieved an accuracy of 81.96% for CNN and 81.75% for the AlexNet model. We concluded that for all three datasets of banana images, the proposed CNN model is the best suitable DL algorithm for bananas’ fruit maturity classification and quality detection.

1. Introduction

Fruit maturity is the completeness of development, which can only take place while the fruit is still attached to the tree, and is shown by the halt of cell growth and the buildup of dry matter. Fruit maturity at harvest has a significant impact on all fruits and vegetables along the postharvest value chain in terms of quality [1]. An efficient and effective automatic model is very much needed that can identify and classify the fruits according to their maturity level within a limited time. DL technology has emerged with big data technologies and high-efficiency computing that creates new opportunities for crop harvesting and crop management in agricultural operations environments. In this study, a system is developed that can help farmers manage their fruit harvests and reduce harvest losses. The “Banana fruit” is used for the fruit maturity classification and quality identification system. In order to prevent harvesting either under- or over-matured bananas, farmers will greatly benefit from being able to recognize the age of fresh banana fruit [2]. A banana provides about 112 calories and contains water, protein, fiber, and carbohydrates. It improves kidney health and reduces amounts of fat. To reduce human efforts and automate rotten fruit identification, the authors proposed a model [3]. Apple, banana, and orange image datasets were used, and input image features were integrated using the CNN algorithm, and images were classified by max pooling, average pooling, and MobileNetV2 techniques. The CNN-based model was proposed by authors [4] to identify the rottenest bananas and classify five types of bananas: cavendish, ladyfinger, Sabri, green, and red bananas. The authors created their dataset, where 2000 banana images of each type and a total of 10,000 images are available in the dataset. Authors [5] proposed a unique CNN-based model for apple fruit categorization and quality identification. The dataset was created by the authors for this implementation, and it contains a total of 36,000 images of apples, in which it contains three classes of apples for training: (1) Premium, (2) Middle, and (3) Poor. The suggested model collected distinct, complicated, and relevant image properties for fruit identification and fruit classification. As compared with previous methods, their current proposed model has learned the two adjacent layers’ high-order features of different channels with a strong connection. Figure 1 shows the basis for the CNN model for fruit maturation classification, where each input image is sent to the first, second, and third fully connected layer and finally to output layer.

The main contributions of this research are given below:

• Augmentation methods have been applied to enhance the size of the dataset.
• Developed a deep CNN to identify and classify bananas’ maturity range.

The rest of the paper is organized as follows: Related work is presented in Section 2. Section 3 contains the data collection. The proposed methodology is given in Section 4. Results and discussions are explained in Section 5. Finally, Section 6 contains the conclusion and future scope.

2. Related Work

Various methods for automatic fruit quality inspection and grading have been proposed to solve problems in different areas. In this paper, we discuss the evolution of fruit maturation classification and quality detection models using DL techniques such as CNN and AlexNet. The authors [4] have developed two DL-based CNN models, where the second performed well with 93.4 ± 0.8% accuracy for classification and achieved 98.3 ± 0.8% accuracy for rotten banana identification. The authors [6] proposed a five-layer CNN containing a convolution layer, a pooling layer, and a fully connected layer model to identify bananas. Features were extracted using CNN by analyzing fruits such as apples, strawberries, oranges, mangoes, and bananas. Classification algorithms such as Random Forest (RF) and K-Nearest Neighbor (KNN) were used to identify fruits. These DL-based RF and CNN algorithms were compared with current systems, where the deep feature RF combination algorithms achieved 96.98% accuracy over others. The model proposed by the authors [3] achieved 99.46% accuracy on training and 99.61% accuracy on the validation set using the MobileNetV2 method. Max pooling and average pooling achieved 94.49%, and 93.06% accuracy for training, respectively, and 94.97% and 93.72% accuracy for validation, respectively. According to the authors [5], the proposed model provided an accuracy of 99% for training, 98.8% for validation, and an overall accuracy of 95.33% when it was tested using an independent 300 apple dataset. Authors [7] proposed an automatic mango sorting and grading model using a DL technique, where eight types of harvested mango features such as size, shape, color, and texture were considered. Image rotation, image translation, image zooming, image sharing, and image horizontal flip data augmentation methods are used. VGG16, ResNet152, and Inception v3 methods were compared using augmented data, where Inception v3 CNN architecture achieved 99.2% accuracy for sorting and 96.7% average accuracy for grading. A computer-vision-based application was designed using CNN and tested for the classification of the ripening stages of the mulberry fruits [8]. The CNN classification model was fine-tuned using transfer learning to improve accuracy and reduce training costs. Different CNN models such as AlexNet, ResNet18, ResNet50, DenseNet, and Inception-v3 were used for testing. AlexNet and ResNet18 achieved the highest accuracy of 98.32% and 98.65% for white and black mulberry’s ripeness classification. ResNet18 was able to classify both genotype and maturity from 600 fruit images with a total accuracy of 98.03%.

The quality of strawberries was assessed using an extended Alexnet (E-Alexnet) [9]. Laboratory and field images of strawberries were collected and pre-processed using image augmentation and image balancing. At last, to process further training, images were imported for the E-AlexNet model. To improve the network accuracy and magnitude order, they changed the convolutional kernel size; three convolutional layers were created from one convolutional layer, and the L2 regularization with the batch normalization layer was used. They obtained 84.50% and 90.70% accuracy before data augmentation and 89.34% and 95.75% accuracy after data augmentation for AlexNet and E-AlexNet techniques, respectively. Transfer learning utilized a pre-trained network AlexNet using tomato maturity classification in terms of colors (green, yellow, and red) for an automatic grading method. Authors [10] aimed to provide a low-cost solution for tomato maturity grading which gives the best performance and accuracy. According to their results, the proposed model received 100% accuracy for the work of tomato classification, which had been performing better than other DL and machine learning (ML) techniques over the years.

The article aimed to classify the papaya fruit according to its maturity level, whether it was ripe, partially ripe, or unripe [11]. Extensive DL techniques were used to identify the papaya fruit images. The trained model achieved 100% accuracy on the test dataset, explaining the feasibility of the proposed approach. The classification model of VGG16 achieved 100% accuracy and a training time of 112 s. The ripeness status and variety of plum fruit were identified by analyzing the plum images using a DL-based tool proposed by the author in [12]. The authors created an uncontrolled image acquisition technique, and images were taken directly in the field using a mobile phone and camera, considering necessary parameters such as light conditions, focus, etc. The proposed classification system used Angeleno, black diamond, and red beaut, three varieties of plum fruit with different ripening stages. Using this robust system, any user could identify and distinguish the types of plum fruits. The system achieved 92.83% accuracy for the given three types of palm fruits, and the average ripening accuracy of palm fruits was 95.5%.

Authors [13] presented a fruit ripeness classification paper, where 9000 training images data of 4 fruits, apple, orange, mango, and tomatoes, were used, and data were trained using VGG16 models with a transfer learning technique using 200 epochs. Data augmentation techniques were used to generate more data to avoid over-fitting. Four frameworks with different techniques were used with a dropout of 0.5, and the average accuracy rate of all four frameworks was 92%, showing the best performance for all. Authors [14] presented a non-invasive banana classification using deep learning algorithms to grade the banana layers into distinct categories. It was a tier-based sorting system, where fruits were graded based on quality, maturity, and size. The quality was having export class, middle class, and rejects class. Maturity had green, yellowish, yellow, and over-ripe, whereas size had small, medium, and large parameters for the fruit grading system. On an own-created banana images dataset, the VGG16 architecture with the transfer learning method was used for grading systems, and it achieved 98% accuracy for training and 92% accuracy for validation. Authors [15] used transfer learning to evaluate the changing process of fruit freshness and developed storage dates and freshness relationships. The GoogleNet model was used to automatically extract features from banana photos and then sorted them by classification module. Using this model, 98.02% accuracy was achieved in detecting the freshness of bananas. Based on the results, they concluded that transfer learning is a perfect, productive, and automatic fruit freshness monitoring technique.

Authors [16] used the CNN algorithm to classify bananas as unripe, yellowish-green, medium ripe, and over-ripe. A bilateral filter was used for image noise cancelation before they were sent for training, and data augmentation was used for variations. The proposed model achieved the highest accuracy of 96.18% and a faster execution time than NASNetMobile. Authors [17] experimented with different hyper-parameters of CNN to sort mature Medjool dates and used ResNet50, ResNet101, ResNet152, VGG16, VGG19, InceptionV3, AlexNet, and CNN from scratch CNN architectures. In Medjool date ripeness classification, processing time and accuracy were utilized to calculate the performance of CNN architectures. The VGG19 model achieved the highest accuracy of 99.32% among all others, with 128 batch sizes and a 0.01 learning rate with the Adam optimizer. The authors concluded that the computer vision system could be created using the VGG19 model to improve the discovery of the Medjool dates’ Tamar stage. Mango was classified into seven categories, namely dahseri, Kesar, jamadar, hafush, totapuri, langado, and rajapuri, using the CNN technique.

The authors in [18] created their dataset and classified mangoes using linear classifiers such as KNN, SVM, and a multilayer perceptron neural network (MLP). The texture was identified using a combination of Local Binary Pattern (LBP), Scale-Invariable Features Transform (SIFT), and Gray-Level Co-Occurrence Matrix (GLCM), where shape identification used a combination of chain code and size parameters. Features were extracted using CNN models, namely MobileNet, ResNet, Inception v3, Xception, and DenseNet. The system achieved 98.50% high accuracy and 94% average accuracy for the combination of InceptionV3 features and MLP classifier. Mangosteen fruit maturation was classified into different market segments using deep CNN. The export market, domestic market, local market, and ungraded mangosteen were different segments. Authors [19] used different CNN architecture on the Mangosteen dataset, they achieved the following accuracy: VGG16 (77.50%), AlexNet (75%), ResNet50 (79%), and InceptionV3 (78%). However, the algorithms mentioned above were over-fitted on the mangosteen dataset, except for ResNet50, so the user concluded that ResNet50 was the most suitable model for a given dataset. With this in mind, the authors developed a modified efficient maturity classification model for the mangosteen fruit. The author in [20] presented a categorization system of palm fruit by designing a model to differentiate three maturity classes: unripe, under-ripen, and ripened. The aim of the research began with the segmentation approach, where thresholding was applied using the Otsu method. Mean- and standard-deviation-based features were calculated by the color extraction functions. Following that, the color extraction characteristics were used by measuring two kinds of characteristics, considering the mean and standard deviation, based on four color parameters: red, green, blue, and grey, yielding eight attributes. At last, the SVM approach was used for classification. A total of 160 images were used for testing and achieved 92.5% accuracy. Authors [21] proposed a framework that works on mobile phones to detect the four maturity levels of tomatoes and chilies. For training, data images were captured continuously before harvesting-to-harvesting time. The k-means clustering method was used for picture segmentation and fuzzy logic for tomato and chili maturity detection. Four categories for ripeness are unripe I, unripe II, medium, and ripe. Image segmentation achieved an accuracy for tomato of 80% and for chili of 100%, as well as ripeness detection achieved an accuracy for tomato of 80% and for chili 90%. Overall detection of ripeness accuracy was 85% by training each category of data. Authors [22] used a pre-trained CNN to classify mangoes into different categories: MobileNet, DensNet, Inception, and Xception are deep learning models used for training and comparison. The highest accuracy of 91.42% was achieved by using the DenseNet and Xception models.

Table 1. Summary of already existing deep learning methods.

Year and Ref.	Fruits	Applications	Techniques	Remarks and Result
2022 [7]	Mango	Fruit Grading and Automatic Classification	CNN, VGG16, InceptionV3	To stop the spoilage of this seasonal fruit and to remove the manual process. Classification accuracy was 99.2%, and grading accuracy was 96.7%
2021 [3]	Banana, Apple, and Orange	Ripeness Identification and Maturity Classification	CNN	To reduce harvest losses. Provides good-quality fruits to farmers and people who used this system. Training accuracy was 99.46%, and testing accuracy was 99.6%
2021 [5]	Apple	Quality Identification and Maturity Classification	CNN	For speed and precise apple fruit grading. Achieved high accuracy for grading on maturity level. Grading accuracy was 98.98%
2021 [6]	Banana	Fruit Recognition and Classification	CNN, Alex-Net	To help industrial applications and automate the process of recognition and classification. Classification accuracy was 96.98%
2021 [16]	Banana	Ripeness Classification	CNN	To increase farmers’ income by reducing harvesting loss. To obtain a good-quality banana. Using a pre-trained model for ripeness classification. Classification accuracy was 96.18%
2021 [15]	Banana	Freshness Detection	CNN, GoogLeNet	Providing fresh fruit to customers and automating this task. Bananas are good for bone health, weight loss, heart, and to prevent cancer. Detection accuracy was 98.02%
2021	Strawberry	Quality Evaluation	Alex-Net	To obtain worldwide cultivated quality strawberry fruit. AlexNet was the best-suited algorithm for this work. Accuracy was 95.75%
2021	Tomato	Maturity Grading	Alex-Net	To reduce pre- and post-harvesting loss. Accurately and efficiently recognized the maturity of tomato. Detection Accuracy was 100%

shows the summary of already existing deep learning approaches.

3. Data Collection

For this research, we created our dataset and used a mobile phone camera to capture images of banana fruit. We used white background (i.e., white paper in the background) to capture the right image with different angles and directions.

The Redmi Note 7 Pro mobile phone with a camera of 48 megapixels, 4 GB RAM, and 64 GB internal storage was used for dataset collection. Two datasets of banana images were used. The first dataset is created by us, and it contains images of raw, ripe, and over-ripe bananas, with 700 images in each category. The second dataset is available on Kaggle, which has 81 types of fruits, but we only used two fruits: banana and red banana, and both fruits contain 1312 banana images of size 100 × 100 × 3. Figure 2 shows the different categories of banana images from the original dataset and Fruit 360 images.

First, we resized the captured images into 227 × 227 pixels and 112 × 112 pixels, and the Fruit 360 dataset is already resized to 100 × 100. DL-based CNN and AlexNet techniques were used for building the model for all three datasets that are the custom dataset, the augmented custom dataset, and the Fruit 360 datasets. Figure 3 shows the images of an original augmented dataset and the Fruit 360 dataset.

Table 2. Original, augmented, and Fruit 360 dataset information.

Title 1	Original Dataset		Augmented Dataset		Fruit 360 Dataset
	Training	Testing	Training	Testing	Training	Testing
Ripe (0)	490	210	4410	1890	460	196
Unripe (1)	490	210	4410	1890	-	-
Overripe (2)	490	210	4410	1890	460	196
Total Images	1470	630	13,230	5670	920	392
	2100		18,900		1312

shows the original, augmented and Fruit 360 dataset information.

The authors (Asep Nana Hermana et al. [13]) also suggested by their experiment that by using image augmentation, over-fitting can be avoided. So, image augmentation methods are used such as affine transformation, compose transformation of 450 rotation, horizontal flip, random crop, and the different degree rotation in position augmentation to increase the original dataset size, where the augmented dataset contains 18,600 banana fruit images of 227 × 227 pixels. Three categories of ripening stages are considered for the created dataset and two categories for the Fruit 360 dataset, as images of unripe banana fruits are not available. Figure 4 shows the images after applying different augmentation techniques to the original dataset.

4. Proposed Methodology

DL with the CNN is commonly used for image processing tasks, as convo-Nets/conv2D/cv2 can learn international symmetrical structures, find objects anywhere in the image, and retrieve abstract visual concepts by capturing increasingly difficult hierarchies. So, in this study, two DL architectures are utilized: the CNN and the AlexNet. The proposed approach is implemented using Keras and TensorFlow. For initial training and validation, common datasets are utilized. The layers of pre-processing for frequent input changes are also very scalable, as are the data loading pipelines [24,25].

4.1. AlexNet

Eight layers make up the architecture of AlexNet: three fully linked layers and five convolutional layers [26]. The network’s primary building block, the convolution layer, makes up the first layer [27]. The convolution window shape in AlexNet’s first layer is 11 by 11. The objects in ImageNet data often occupy more pixels with greater visual information, since the images in ImageNet are eight times higher and broader than the MNIST images. Therefore, to catch the object, a bigger convolution window is required. The second layer’s convolution window form is altered to 5 × 5, then 3 × 3. In addition, the network adds max-pooling layers with a window shape of 3 × 3 and a stride of 2 after the first, second, and fifth convolutional layers. There are two enormous fully linked layers with a total of 4096 outputs following the final convolutional layer [28]. The input image is given to the input layer below, as shown in Figure 5, and its pixel value is 227 as width, 227 as height, and 3D colors, which are stored as RGB. The image is then sent to the hidden layers because it contains convolutional layers of different filters, and then after processing the image is transferred to the fully connected layers and, finally, to the output layer.

When the image is transferred from the input to the first convoluted layer, it is resized depending on whether the padding is applied or not.

The size of the filter for the first layer is 96, so the size of the input image of the pooling layer will be 55 × 55 × 96 and as given in the below

Table 3. The proposed CNN model architecture.

Layers (Type)	Image Size	Filters	Filter Size	Pooling Size	Output Shape	Params#
Conv2d	227 × 227	32	3 × 3	-	(None, 225, 225, 32)	896
Max pooling2d	112 × 112	-	-	2 × 2	(None, 112, 112, 32)	0
Conv2d_1	112 × 112	64	3 × 3	-	(None, 110, 110, 64)	18,496
Max_pooling2d_1	56 × 56	-	-	2 × 2	(None, 55, 55, 64)	0
Conv2d_2	56 × 56	64	3 × 3	-	(None, 53, 53, 64)	36,928
Flatten	Fully Connected Layer				(None, 179776)	0
Dropout	20%				(None, 179776)	0
Dense	“Units = 64, Activation = relu”				(None, 64)	11,505,728
Dense_1	“Units = 3 (Categorical), Activation = softmax”				(None, 3)	195

. All input size values are calculated using the above two formulas. When the augmented dataset is used for training, the image size is resized to 112 x 112 pixels, because when training a model with a large dataset, its RAM is insufficient for performance. So, the AlexNet model is trained, validated, and tested by reducing the image size to “112 × 112 × 3”.

4.2. Proposed CNN Model

A CNN is a kind of DL method made up of neurons that takes input images and assigns trainable weights and biases. Tens or even hundreds of layers can be present in a CNN, and each layer can be trained to recognize various aspects of an image. A CNN is built to automatically and adaptively learn spatial hierarchies of data by backpropagation using a number of building blocks, including convolution layers, pooling layers, and fully connected layers [29]. Each training image is subjected to filters at various resolutions, and the result of each convolved image is utilized as the input to the following layer. Beginning with relatively basic properties such as brightness and borders, the filters can get more complicated until they reach characteristics that specifically identify the object [30]. The maximum value from the area of the image that the Kernel has covered is returned by max pooling layers [29,31].

In this work, 3 convolutional layers with 2 max-pooling layers are used, where the input image size is set to 227 × 227 × 3 for the original dataset and 112 × 112 × 3 for the augmented dataset. The loss function is chosen as the cross-entropy function and Adam is chosen as the optimizer. This is because the Adam’s algorithm makes weight and offset updates more stable. A 20% dropout was used to balance training and validation accuracy and loss. Finally, the softmax activation function is applied at the output layer. The architecture of the proposed CNN model is shown in Figure 6. Table 3 shows the configuration of the proposed CNN architecture.

5. Results and Discussion

Extensive research on fruit maturity classification and quality assessment is presented to identify fruit shelf life. This study uses two DL methods (CNN and AlexNet) with the image augmentation technique. A slight drop in accuracy is experienced at epoch 5 for the original dataset, which is removed by increasing the dataset size using the image augmentation technique. The results are shown in Figure 7, where the X-axis represents epoch number, and the Y-axis represents accuracy in the accuracy graph and loss in the loss graph.

The AlexNet, achieves 98.18% training accuracy and 81.75% validation accuracy for the original dataset with a “227 × 227 × 3” dataset image size. The augmented dataset achieved 99.80% training accuracy and 99.44% validation accuracy with an image size of “112 × 112 × 3”. The Fruit 360 dataset, which has an image size of “100 × 100 × 3”, achieved 100%, the highest training accuracy, and 89.84% validation accuracy when the proposed methodology is applied. The results of the AlexNet architecture are shown in Figure 8, where the X-axis represents epoch number, and the Y-axis represents accuracy in the accuracy graph and loss in the loss graph.

The proposed CNN model obtained a training accuracy of 99.18% and a validation accuracy of 98.25% on the original dataset with a 227 × 227 × 3 image size. The augmented dataset has an image size of 112 × 112 × 3 and achieved 99.42% training accuracy and 99.36% validation accuracy. The Fruit 360 dataset reached 81.75% validation accuracy, where the training accuracy is 100% and has properly classified the unripe, ripe, and over-ripe fruits, as shown in Figure 9.

Table 4. Training and validation accuracy for CNN and AlexNet model.

Datasets	CNN		AlexNet
	Training	Validation	Training	Validation
Original Dataset	99.18%	98.25%	99.18%	81.75%
Augmented Dataset	99.42%	99.36%	99.80%	99.44%
Fruit 360 Dataset	100%	81.96%	100%	81.75%

describes the training accuracy and validation accuracy for the three datasets for the proposed CNN model and AlexNet model. The images in the test dataset are shown in Figure 9, (a) the 3rd image is classified as unripe, (b) the 56th image is classified as ripe, and (c) the 89th image is classified as over-ripe when tested. We used bar graphs to plot 0 to unripe, 1 to ripe, and 2 to over-ripe. In Figure 9, we plotted the image and predicted values bar graph of that image.

6. Conclusions and Future Work

We studied 35 different papers (research_articles) in our review paper [32] based on ML and DL techniques for fruit maturation classification, and we concluded that a CNN is the best suitable algorithm for fruit maturation classification and quality evaluation. Fruit ripening classification is a very crucial function in different fields such as the food industry, agriculture, and other industries.

In this study, we proposed two approaches using DL methods (CNN and AlexNet) for banana fruit images on our original dataset, augmented dataset, and available dataset (Fruit 360). Training and validation were conducted for both the proposed models and tested their working performance, and the accuracy rate we obtained for the original dataset is 98.25% for the CNN and 81.75% for the AlexNet model, while the augmented dataset achieved 99.36% accuracy for the CNN and 99.44% for the AlexNet model. This indicates that the classification and prediction accuracy for fruits at the ripening stage is well attained by our suggested models. The proposed methodology is limited to banana, in the future, we may use some different fruits such as mango, orange, and papaya, which are similar to bananas, as they are green when not ripe, yellow when ripe, and brown or black when over-ripe. Moreover, we will try to use our dataset for other deep learning methods for fruit maturity classification and quality assessment.