A Maturity Estimation of Bell Pepper (Capsicum annuum L.) by Artificial Vision System for Quality Control

Abstract

Sweet bell peppers are a Solanaceous fruit belonging to the Capsicum annuum L. species whose consumption is popular in world gastronomy due to its wide variety of colors (ranging green, yellow, orange, red, and purple), shapes, and sizes and the absence of spicy flavor. In addition, these fruits have a characteristic flavor and nutritional attributes that include ascorbic acid, polyphenols, and carotenoids. A quality criterion for the harvest of this fruit is maturity; this attribute is visually determined by the consumer when verifying the color of the fruit’s pericarp. The present work proposes an artificial vision system that automatically describes ripeness levels of the bell pepper and compares the Fuzzy logic (FL) and Neuronal Networks for the classification stage. In this investigation, maturity stages of bell peppers were referenced by measuring total soluble solids (TSS), ° Brix, using refractometry. The proposed method was integrated in four stages. The first one consists in the image acquisition of five views using the Raspberry Pi 5 Megapixel camera. The second one is the segmentation of acquired image samples, where background and noise are removed from each image. The third phase is the segmentation of the regions of interest (green, yellow, orange and red) using the connect components algorithm to select areas. The last phase is the classification, which outputs the maturity stage. The classificatory was designed using Matlab’s Fuzzy Logic Toolbox and Deep Learning Toolbox. Its implementation was carried out onto Raspberry Pi platform. It tested the maturity classifier models using neural networks (RBF-ANN) and fuzzy logic models (ANFIS) with an accuracy of 100% and 88%, respectively. Finally, it was constructed with a content of ° Brix prediction model with small improvements regarding the state of art.

1. Introduction

Native to the Americas, sweet bell peppers are a Solanaceous fruit belonging to the Capsicum annuum L. species. It is a non-pungent fruit that is valued for its color, flavor, and nutritional attributes including ascorbic acid, polyphenolics, and various carotenoids. It comes in a wide variety of colors (ranging green, yellow, orange, red, and purple), shapes, and sizes, as well as because it has a high content of ascorbic acid, polyphenols, and other antioxidants. Nowadays, bell peppers are widely consumed in various ways, dehydrated, preserved, frozen, or raw for packaged salads. Generally, the harvest of bell peppers is determined by the size, color, and texture of the fruit. Traditionally, the harvest of this fruit is done by reaching physiological mute when the pericarp becomes thick and the fruit reaches the typical size. However, estimating pepper maturity at the green stage can be difficult even for fruit with similar physical attributes [1]. Under certain conditions, bell peppers can begin to ripen during shipping. Partially ripened fruit, classified as chocolate or suntan, have lower market values than at the solid color stage.

The bell peppers reach their optimum state of maturity for use in the kitchen when they are in solid color. Consumers prefer this fruit in its best stage of maturity more so than its physical appearance and nutritional content [2]. Dutch researchers specializing in the sensory area, reported that study groups have considered that more ripeness of bell peppers is sweeter and has a red pepper aroma, while those in the green stage were rated for bitterness and aroma of herbs and cucumber [3]. On the other hand, ripe peppers are more expensive to produce, due to the longer time required for ripening and the greater likelihood of damage from insects or disease. Furthermore, ripe peppers are more susceptible to physical damage during transport. In addition, they have a shorter shelf life due to the stage of maturity at which they are harvested. These fruits are non-climacteric regarding postharvest respiratory patterns. At the mentioned stage, bell peppers will progress through the normal ripening process to degrade chlorophyll while simultaneously synthesizing a variety of red and yellow carotenoids. The red and yellow varieties are the peppers most on demand by consumers, followed by the orange and purple varieties [4].

Bell pepper contains provitamin A, carotenoids, and xanthophylls. Many studies have focused on improving retention of these compounds during processing and storage [5,6,7,8,9,10] that increase their concentration as the fruit reaches a major state of maturity. Bell peppers also contain high concentrations of ascorbic acid (0.15–2.0 mg·g⁻¹ fresh weight) compared to other fruits and vegetables [11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]. The production of ascorbic acid in peppers and other fruits are related to glucose metabolism and light exposure, and concentrations of both ascorbic acid and sugar reduction typically increases with the stage of maturity [14]. Polyphenolics are also an important chemical component in bell peppers and impart functional properties to the plant such as disease resistance and potential health benefits to consumers. Some studies found that total phenolics, including the flavonoid quercetin, decrease with increased maturity for yellow bell peppers, but increased for other pepper varieties [6]. Estrada et al. [31] also demonstrated a decrease in free phenolic concentrations in peppers over five stages of maturity. Pepper growers could reduce field production costs by hastening the fruit ripening rate on the plant or by harvesting the fruit before attaining full color and completing the ripening process during storage without appreciable loss in quality or phytochemical attributes. An important criterion for consumers of sweet pepper is its sweetness, commonly estimated with the content of soluble solids (SSC) [22]. This parameter is associated with the different stages of fruit maturity [24,25,26,27,28]. The measurement of SSCs is traditionally a destructive test, to perform it, the most used instrument is the refractometer. This is an optical instrument that measures the refractive index of the juice from the sample [29,30].

At present, the food industry has demanded the use of non-invasive high precision measuring equipment that determines the external and internal quality parameters of the fruits [32]. In this respect, one of the disciplines that has had a great impact on fruit quality control is computational vision (VC). This has the characteristic that emulates the functionality of human vision and allows spatial and optical information from the captured image of the sample. Different investigations have been reported with VC focused on determining the degrees of ripeness of various fruits such as persimmon, strawberries, pomegranate, and tomato [33,34,35]. The present work aims to implement an artificial vision system that automatically describes the ripeness levels of the bell pepper.

More recently, total soluble solids (TSS) or Brix grades are a classical tool to determine the maturity of the fruits in the food industry even though it is a destructive technique. The content of TSS consists of 80–95% sugars and the measure of TSS is associated with the dissolved sugars in cell juice [36]. These authors affirmed that the quantity of sugars in the fruit depends mainly on the variety, the assimilatory yield of the leaves, the leaf/fruit ratio, the climatic conditions during the development of the fruit, the state of development, and the maturity. The accumulation of sugars is associated with the development of optimum quality for consumption. Although the sugars can be transported to the fruit by the sap, they are also contributed by the splitting of the starch reserves of the fruits [36]. When the fruits in general have their highest sugar content, they have reached their physiological maturity, which coincides with what was investigated in relation to the bell peppers and their greater ripeness [37]. The TSS content showed a constant increase as the fruit maturity status increased, which could be seen with an increase in color fastness in the sample bell peppers. The ascending behavior of TSS content is consistent with that reported in the literature [37]. Kays [38] explained that, when the fruit is ripening in the plant, the sugars increase their concentration by the translocation of sucrose from the leaves, which occurs in most species. However, there is also the recycling of the respiratory substrate from the carbon stored in the fruit. Overall, highly significant positive correlations of soluble solids content were found with this research. The measurement of the TSS was done in triplicate with two different refractometers, an ATAGO Refractometer (MASTER-M model, Tokyo, Japan), Brix measurement scale of 0.0–33.0%, with a minimum of 0.2%, an accuracy of ±0.2%, and repeatability of ±0.1%; and a HANNA brand refractometer, model HI 96802 (Woonsocket, RI, USA), Brix measurement scale of 0.0–85.0%, with a temperature range of 0–80 °C, an accuracy of ±0.2%, and repeatability of ±0.1%.

Figure 1 shows the distribution of Brix grades of soluble solids content in the four stages of maturity of the fruit. This allowed us to establish that the grouping of the samples contemplates different stages of maturity of the bell pepper. This describes an increase in sugars from maturity Stage 1 to Stage 3 and the maximum sugar content is in State 4.

2. Related Work

Currently, the high demand in the food industry has created the need for a fast and effective way to control the quality of products, which has led to the application of intelligent systems such as artificial vision that helps determine the internal and external properties of the fruits resulting in the stage of maturity. Different authors have focused on determining the maturity of various fruits such as apricot, nightshade, citrus, coffee, and mango. It is worth mentioning several studies that have focused on determining the maturity of bell peppers and estimating total soluble solids [39].

Harel et al. [40] classified their maturity using the random forest algorithm, which identified four classes: fully green, partially colored immature, mature partially colored, and colored.

Elhariri et al. [41] used an algorithm based on a support vector machine (SVM) for classification. This identified five maturity classes associated with green and red shades.

Shamili et al. [42] proposed a polynomial regression model that estimated the soluble solids content of mango. The descriptor that he proposed was the normal values of * a of the images captured from five types of mangoes; the samples he used had different percentages of skin coloration of 0%, 20%, 25%, 50%, and more than 50%.

Li et al. [43] used hyperspectral imaging in the visible and near-infrared (VNIR) and short-wave near-infrared (SWIR) regions focused on measuring maturity, firmness, and suspended sediment concentration (SSC). This research showed that there is a strong correlation between the SSC and the average spectra obtained from one or two opposite sides of the fruit in the SWIR region.

Another similar work was carried out by Teerachaichayut et al. [44], who proposed several models for limes to determine total soluble solids (TSS) and titratable acidity (TA) using partial least square regression.

3. Materials and Methods

3.1. Samples

A sample set of 50 bell peppers produced in the Laja-Bajió region, Guanajuato, Mexico was analyzed. The selected sample attributes consider different maturity degrees and homogeneous size [45]. For the training of the vision system, fruits with homogeneous and defined colors were chosen. Therefore, samples showing heterogeneous colors with various spots in the pericarp were excluded for this stage but were included for general testing. The bell peppers were classified into four classes, as shown in Figure 2. Class 1 grouped ten green samples. Class 2 was made up of seven yellow fruits. Class 3 was fourteen orange pigmented fruits. In Class 4, nineteen samples with a red hue were grouped.

3.2. General Structure of Artificial Vision System

Figure 3 shows the proposed method for predicting the sugar content of bell pepper. The first step of the proposed method was the acquisition of the image samples using computer vision system (VCS). The second step was the segmentation of the fruit in the image, where the background and the supersaturated pixels were removed. The third step was the masking step to identify the regions of interest with green (GPOI), yellow (YPOI), orange (OPOI), and red (RPOI) pixels. These were obtained by means of four masks that use the red, green, blue (RGB) components of the segmented images. The fourth step was to obtain the areas of each region of interest (GAROI, YAROI, OAROI and RAROI). The fifth step was the classification of the fruit with the areas from the filters. The last step was the prediction of ° Brix, which was done with the class and the areas determined by the masks.

3.3. Image Acquisition

Five images per fruit were captured, as shown in Figure 4. The image format was JPG with a resolution of 768 × 1366 × 3. The acquisition was made with a computational vision system (VSC) that operates with OpenCV-Python language. This was integrated by the isolation, lighting, image capture, and image processing subsystems. The insulation subsystem consisted of a black cabinet of dimensions 38 cm × 38 cm × 43 cm, which allowed the reduction of variations in lighting. The architecture used by the lighting subsystem was a 5.4 W led ring configuration placed 30 cm above sample. The image capture was performed with Raspberry camera module (8 megapixels), and the processing platform was Raspberry Pi 3 card [33].

3.4. Automatic Sample Classification

In the second phase, the degrees and averages of the RGB channels of the segmented images of each fruit were mapped, as shown in Figure 5. G (green) corresponds to samples belonging to Class 1, Y (yellow) label to samples of Class 2, O (orange) label to samples of Class 3, and R (red) label to the samples of Class 4. In the last phase, the regions of interest corresponding to the green, yellow, orange, and red tones were carried out.

3.5. Obtaining Regions of Interest

Obtaining regions of interest used the methodology proposed by Goel et al [32], as shown in Figure 6. (i) The first step required the binarization of the captured images, which was done using the Hue, Saturation, Value (HSV) color space model. This change in the color space model allowed obtaining the frequency of each color in the visible spectrum with component H, the purity of color with component S, and the proximity of the pixel to black and white with component V. The thresholds used to segment the background of the samples were 122 ≤ H ≤ 255, 127 ≤ S ≤ 255, and −0 ≤ V ≤ 255 using a range from 0 to 255. (ii) The second step was segmentation of images of each view of bell pepper using a component connection algorithm, and later all the small regions smaller than 400 pixels that did not correspond to the binarized image of the sample were discriminated. (iii) The last step was to use respective masks to obtain regions of interest due to its color.

Table 1. Segmentation mask thresholds.

Green Mask
H min	H max	S min	S max	V min	V max
47	118	0	255	0	255
Yellow Mask
H min	H max	S min	S max	V min	V max
32	46	0	255	0	255
Orange Mask
H min	H max	S min	S max	V min	V max
15	32	0	255	0	255
Red Mask
L min	L min	L min	L min	L min	L min
241	155	11	247	0	255

shows the four ranges of the Hue parameters to identify the pixel areas with green (47 ≤ H ≤ 118), yellow (32 ≤ H ≤ 46), orange (15 ≤ H ≤ 32), and red (155 ≤ H ≤ 241) hue of the images. Most of the masks except the red mask one used the range from 0 to 255 for the S and V components to identify the different color regions. The red mask used a smaller range of the S component that allowed the areas with orange and red pixels to be correctly identified. These thresholds were obtained using the 50 samples where each of its segmented images was analyzed. These masks identify the pixels corresponding to the green, yellow, orange, and red shades.

3.6. Maturity Status Estimator

3.6.1. Artificial Neural Network (ANN)

Artificial neural networks (ANN) are mathematical models that emulate the functioning of biological neural networks. This was used in this study for its structural simplicity, learning skills, and its application for the approach, classification, and pattern recognition. Figure 7 presents the ANN architecture that was used to estimate the ° Brix content of the fifty samples presented in Figure 1. Its architecture consisted of an input layer, three hidden layers, and an output layer. The input layer is used to present the training and test patterns that correspond to the GAROI, YAROI, OAROI, and RAROI regions of interest. The hidden first layer used radial base-type activation functions, each of its neurons calculating the similarity between the input and its training set. The second hidden layer adds the values of each neuron from the first hidden layer that are multiplied by their weight associated with each neuron to obtain the class that the sample belongs to. The third layer of neurons employs neurons with sigmoidal-type firing functions where the sample classification information and the four regions of interest were weighted. The output layer is of the linear type, allowing the Brix content to be estimated using the weights associated with the neurons of the third hidden layer.

One of the ANNs most used in classification tasks is radial basis function ANN (RBF-ANN), as shown in Figure 8. Its architecture uses an input layer, hidden layers, and output layer. The input layer is used to present the training and test patterns. The hidden layer is made up of radial (Gaussian) functions that are completely interconnected between all its nodes with the input layer, which are activated by this function. The output layer is activated by the linear functions that determine the classification and is interconnected with all the nodes of the hidden layer.

The proposed RBFNN classification models were developed with Matlab’s Deep Learning Toolbox. The main difference between them is the number of neurons that the hidden layer contains, as shown in

Table 2. RBF-ANN maturity classifier.

	Inputs	Number of Neurons in the Hidden Layer	Output	Epochs	Accuracy
Model 1	4	4	4	10	92%
Model 2	4	5	4	10	98%
Model 3	4	8	4	10	100%
Model 4	4	10	4	10	100%
Model 5	4	15	4	10	100%

. The inputs used by each classifier are the interest regions with shades of green, yellow, orange, and red and the outputs are the four classes. The model design used 70% of the data for training, 20% for validation, and the rest for the testing stage.

Together, three models of two-layer feed-forward network with sigmoid hidden neurons and linear output neurons (Fitnet) are proposed. Figure 9 presents the architecture of the implemented model to predict the ° Brix content of bell peppers. Three models were proposed where their main difference is the number of neurons in the hidden layer, as shown in

Table 3. Models using neural network Feedback prediction of Brix degrees.

	Inputs	Number of Neurons in the Hidden Layer	Output	Epochs	Mean Squared Error (MSE)	Pearson Correlation Coefficient (R)
Model 6	4	4	1	10	0.5483	0.68659
Model 7	4	5	1	10	0.5013	0.50130
Model 8	4	8	1	10	0.5016	0.73176
Model 9	4	10	1	10	0.4676	0.75910
Model 10	4	15	1	10	0.3888	0.79543

; the precision of each model to predict the ° Brix was different.

3.6.2. Fuzzy Logic

Fuzzy Logic (FL) is a discipline of Artificial Intelligence that analyzes real-world information on a scale between true and false. This is integrated by three stages: fuzzification, the inference that uses a series of linguistic rules, and defuzzification. Figure 10 shows a structure of the fuzzy classification system proposed to classify bell peppers. It has four inputs and one output.

The regions of interest of each shade correspond to the input and are used to identify the degree of maturity associated with the output fuzzy system. It is used to identify segments or regions of interest with shades of green, yellow, orange, and red. The output has different ranges for each class.

Fuzzification of the input variables was carried out with membership functions of triangular type, which are characterized by their easy implementation in hardware. The linguistic variables used were low, medium, and high, as shown in Figure 11, Figure 12, Figure 13 and Figure 14. The mathematical functions of each membership function are described in Equations (1)–(12). Together, the different pixel areas were used to increase the sensitivity of the fuzzy system to infer maturity changes.

$$LowGARO=\{\frac{50-GAROI}{50}0<GAROI\le 50 050<GAROI\le 100$$

(1)

$$MediumGAROI\{\frac{GAROI}{50}0<GAROI\le 50 \frac{100-GAROI}{50}50<GAROI\le 100$$

(2)

$$HighGAROI=\{00<GAROI\le 50 \frac{GAROI-50}{50}50<GAROI\le 100$$

(3)

$$LowYAROI=\{\frac{29.09-YAROI}{29.09}0<YAROI\le 16.48 016.48<YAROI\le 34.73$$

(4)

$$MediumYAROI=\{\frac{YAROI}{29.09}0<YAROI\le 29.09 \frac{58.17-YAROI}{29.08}V<YAROI\le 58.17$$

(5)

$$HighYAROI=00<YAROI\le 29.09 \frac{YAROI-29.09}{29.09}29.09<YAROI\le 58.17$$

(6)

$$LowOAROI=\{\frac{16.48-OAROI}{16.48}0<OAROI\le 16.48 \frac{OAROI-34.73}{18.25}16.48<OAROI\le 34.37$$

(7)

$$MediumOAROI=\{\frac{16.48-OAROI}{16.48}0<OAROI\le 16.48 \frac{OAROI-34.73}{18.25}16.48<OAROI\le 34.37$$

(8)

$$HighOAROI=\{\frac{16.48-OAROI}{16.48}0<OAROI\le 16.48 \frac{OAROI-34.73}{18.25}16.48<OAROI\le 34.37$$

(9)

$$Low RROI=\{\begin{array}{l}\frac{16.48-RROI}{16.48} 0<RROI\le 16.48\\ \frac{RROI-34.73}{18.25} 16.48<RROI\le 34.37\end{array}$$

(10)

$$Medium RROI=\{\begin{array}{l}\frac{16.48-RROI}{16.48} 0<RROI\le 16.48\\ \frac{RROI-34.73}{18.25} 16.48<RROI\le 34.37\end{array}$$

(11)

$$High RROI=\{\begin{array}{l}\frac{16.48-RROI}{16.48} 0<RROI\le 16.48\\ \frac{RROI-34.73}{18.25} 16.48<RROI\le 34.37\end{array}$$

(12)

Figure 15 shows the fuzzy Takagi–Sugeno model used for the development of the classifier; this was selected for its low computational cost unlike the Mandami model. It uses 81 inference rules obtained from using fuzzy values of interest regions. The transforming fuzzy values of classification output was carried out using Equation (13). The final output is determined by rules using Z_i levels of output and weight $w_i$ of the rule.

$$Final Output=\frac{{\sum }_{i=1}^N{w}_i{Z}_i}{{\sum }_{i=1}^{18}{w}_i}$$

(13)

Five fuzzy classification models were designed using the Matlab R2014a Fuzzy Logic Toolbox. Its training used a set of 50 vectors consisting of areas the interest regions in shades of green, yellow, orange, and red and their label corresponding to their maturity. The training used 10 epochs. The main difference between the models is the number of membership functions.

Table 4. Classification models.

	Number of GROI Membership Functions	Number of YROI Membership Functions	Number of OROI Membership Functions	Number of RROI Membership Functions	Training Error RMSE 1 × 10−6
Model 11	2	2	2	2	930.46
Model 12	3	2	2	3	479.86
Model 13	2	3	3	2	19.466
Model 14	3	3	3	3	9.8679
Model 15	4	4	4	4	2.0339

shows the number of membership functions used by each model and their training error.

Three models were proposed where their main difference is the number of neurons in the hidden layer, as shown in

Table 5. Models using FL prediction of ° Brix.

	Number of GROI Membership Functions	Number of YROI Membership Functions	Number of OROI Membership Functions	Number of RROI Membership Functions	Mean Squared Error (MSE)	Root Mean Squared Error (RMSE)	Pearson Correlation Coefficient (R)
Model 16	3	3	3	3	5.923	2.433	0.499
Model 17	4	3	3	4	0.891	0.944	0.696
Model 18	3	4	4	3	1.645	1.282	0.424

; the precision of each model to predict the Brix degrees was different.

4. Results

Figure 16 shows the accuracy of the 10 maturity classifier models using ANN and FL. The first five models are type ANN (blue circles) and the next five models are type FL (red squares). It can be seen that the proposed models which used ANN achieved greater than 90% accuracy, unlike those that used FL. Together, the models that had good precision are Models 13–15. Finally, the worst models were Models 11 and 12 with an accuracy of less than 70%. In Figure 17, the prediction error of the model is presented, where it can be highlighted that the models which use ANN have lower least squared medium error. Five ANN models (blue circles) and three FL models (red squares) are shown.

5. Discussion

The main contribution of this work is the development of a ° Brix content prediction system for bell pepper maturity. According to the results, the models with FL achieved a maximum precision of 88% in identifying the four stages of maturity corresponding to the shades of green, yellow, orange, and red. The models with ANN have 100% precision to identify samples of green and red color similar to the results reported by Elhariri et al. [41]. Of the results obtained, Model 8 was the one that presented a correlation of R = 0.79543 between the green, yellow, orange, and red regions of interest and ° Brix. This result is similar to that reported by Shamili [42] and is slightly superior to those reported by Leiva-Valenzuela et al. [46] (R = 0.788) and Rahman et al. [47] (R = 0.74). The advantage is that our proposal uses a visible RGB camera and they used a high-cost multispectral camera. Figure 18 shows the estimation.

6. Conclusions

In this work, a new classification system is proposed to evaluate the bell peppers maturity using an artificial vision system. According to the results obtained, it can be concluded that the proposed architecture improves the ° Brix prediction using the regions of interest associated with the color of the pericarp. The correlation obtained was R = 0.7929 with the use of the NETFIT-ANN, surpassing the model designed with FL with a correlation of R = 0.696. Together, it was possible to classify 100% of the classes of the samples with the use of a model that uses the RBF-ANN architecture. This architecture presented a better result than the FL models that obtained a maximum accuracy of 88%. The present work demonstrated that it is possible to identify the degrees of maturity of the peppers using an artificial vision system that is sensitive to the total soluble solids content of the fruit. It has the advantage of using a low-cost RGB camera instead of a multispectral one, and it is a non-destructive technique to estimate ° Brix. This vision system model is applicable to real scenarios in the industrial sector such as online processes. Furthermore, it is a system that can be used to predict the ° Brix content in real time. As further research, the development of a prediction system for ascorbic acid, polyphenolics, and various carotenoids has been completed.