Towards a Real-Time Oil Palm Fruit Maturity System Using Supervised Classifiers Based on Feature Analysis

Abstract

Remote sensing sensors-based image processing techniques have been widely applied in non-destructive quality inspection systems of agricultural crops. Image processing and analysis were performed with computer vision and external grading systems by general and standard steps, such as image acquisition, pre-processing and segmentation, extraction and classification of image characteristics. This paper describes the design and implementation of a real-time fresh fruit bunch (FFB) maturity classification system for palm oil based on unrestricted remote sensing (CCD camera sensor) and image processing techniques using five multivariate techniques (statistics, histograms, Gabor wavelets, GLCM and BGLAM) to extract fruit image characteristics and incorporate information on palm oil species classification FFB and maturity testing. To optimize the proposed solution in terms of performance reporting and processing time, supervised classifiers, such as support vector machine (SVM), K-nearest neighbor (KNN) and artificial neural network (ANN), were performed and evaluated via ROC and AUC measurements. The experimental results showed that the FFB classification system of non-destructive palm oil maturation in real time provided a significant result. Although the SVM classifier is generally a robust classifier, ANN has better performance due to the natural noise of the data. The highest precision was obtained on the basis of the ANN and BGLAM algorithms applied to the texture of the fruit. In particular, the robust image processing algorithm based on BGLAM feature extraction technology and the ANN classifier largely provided a high AUC test accuracy of over 93% and an image-processing time of 0,44 (s) for the detection of FFB palm oil species.

1. Introduction

Usually, information that can be obtained from a distance about objects or areas is a science called remote sensing [1]. Remote sensing is a procedure that can be used to measure the external physical properties of an area by receiving the energy reflected and emitted from the target area [2].

Further, an introduction and development of remote sensing was published by the authors of Refs. [3,4] using various sensors, image processing tools and techniques for remote sensing applications. In fact, the most common sensors used in remote sensing are cameras and solid-state scanners, such as CCD (charge coupled device) images, which are available in 2D matrices for the application, and satellite image sensors [5,6].

Bakker et al. [7] indicated that a digital CCD camera is an electro-optical remote sensor made of semiconductor material, which is the most common type of detection nowadays in the range from visible to very near IR; it is used to provide area information in aerial Earth observation application for low-cost imagery. Likewise, bi-directional reflectance measurements and the method for determining the suppression of atmospheric MTF for the CCD camera on board the Huanjing 1A (HJ-1A) satellite and the digital CCD camera were designed and studied by the authors of Refs. [8,9].

In addition, a standard digital camera used to automatically monitor snow cover with high accuracy in terms of time and location as a unified index based on red, green and blue (RGB) values was developed to control errors due to lighting effects [10]. On the other hand, microcomputers have increased power as an advantage of remote sensing image processing technology. Based on their study [11], researchers used the BRIAN micro CSIRO system as an example of how to access methods for image processing, which contributed to the acceptance of remote sensing technology.

Machine learning is an evolutionary area of algorithms, hardware and storage systems working in smarter ways for several applications, such as (a) abnormal behavior proactive detection for reasonable solutions in advance; (b) creating events models based on system training in order to forecast the values of a future inquiry; (c) testing the future inquiry based on the understating of the created event model and (d) computing the individual loss reserve [12]. Thus, different researchers have used the advantage of machine learning for automated wheat diseases classification, estimation of the long-term agricultural output and prediction of soil organic carbon and available phosphorus [13,14,15]. Therefore, there are many benefits and advantages to using machine learning methods in computing the individual loss reserve regarding ML techniques, making such methods more feasible, with more accurate pricing, claims triage, loss prevention, a deep dive in changes in loss reserves and frequent monitoring to calculate claims reserves on individual claims data (ICR) [16,17,18,19].

In addition, Refs. [20,21] reported that machine learning techniques provide the possibility to activate and control the classification of images by remote sensing. However, multi-band deep learning, deep convolutional neural networks and modular features were implemented using limited training samples by the authors of Refs. [22,23,24,25,26,27,28] to classify the hyperspectral, hyperparameter, spectroradiometer and spectrometer images as those remotely detected data.

As a result, a variety of fields have been successfully populated with numerous remotely sensed images with high spectral–spatiotemporal resolution in order to identify important acoustic processes in agricultural applications [29,30]. The old method of assessing the quality of agricultural products, in general, is tedious and expensive [31]. Traditional techniques have been in use for a long time, but they are extremely tedious, expensive and out of control over time. In this context, high-tech switches are needed to use machine vision to classify the quality of agricultural food products and to assess timely and accurately [32,33,34,35,36,37,38,39,40,41,42].

This technique is suitable for surface or sub-surface imaging due to the incomplete penetration depth of the interrogation source. In addition, researchers from all over the world have contributed and developed an automated internal classification system as a solution for screening agricultural crops based on internal characteristics, such as sugar, moisture and acid [43,44,45,46].

Equally, the authors of Refs. [47,48,49,50,51] applied remote sensing and image processing technology based on texture and color measurement methods to classify images in a different application. In this paper, the remote sensing (CCD camera) and image processing (Gabor waves, GLCM and BGLAM) sensor technologies as texture property extraction techniques are based on supervised machine learning classifiers (SVMs, KNN and ANN) so as not to distract the assets and to assess the FFB quality inspection on a real-time system.

This article will review in Section 2 the fruit ripeness classification aspect, and the discussion will expand on relevant works regarding methodologies and strategies for an automated FFB grading and sorting system using different approaches, including data collection, system material, image processing, classification system and other available tools for system evaluation and assessment. Next, Section 3 will entail the results and discussion, including obtained results by modeling the different feature algorithms with classification modules. Section 4 will be the conclusion and discuss the challenges and future direction.

2. Fruit Ripeness Classification

Currently, different computer vision systems have been invented and applied to assess the quality of agricultural crops based on different color spaces through machine learning techniques. The use of such systems for a variety of fruit ripening processes based on different color spaces and classification techniques resulted in varying research accuracy, as shown in

Table 1. The literature on accurately assessing crop quality using machine learning techniques based on a variety of color spaces. Table from Ref. [51] is cited and updated with new implementation results.

Item	Color Space	Classification Technique	Accuracy	Ref.
Oil palm	UV + RGB + NIR	KNN and SVM	93.80	[25]
Dates	JPG	CNN	99.32	[26]
Banana	RGB and GLCM	CNN and MLP	98.45	[27]
Apple	HSI	SVM	95.00	[52]
Apple	L*a*b*	MDA	100.00	[53]
Apple	RGB	SVM	96.81	[54]
Apple, pears and peaches	RGB	ANN	98.90	[55]
Papaya	LBP, HOG and GLCM	KNN, SVM and Naıve Bayes	100.00	[56]
Avocado	RGB	K-Means	82.22	[57]
Dragon Fruit	HSV + RGB	Naive Bayes	86.60	[58]
Banana	L*a*b*	LDA	98.00	[59]
Banana	RGB	ANN	96.00	[60]
Blueberry	RGB	KNN and SK-Means	85.00–98.00	[61]
Date	RGB	K-Means	99.60	[62]
Lime	RGB	ANN	100.00	[63]
Mango	RGB	SVM	96.00	[64]
Mango	L*a*b*	MDA	90.00	[65]
Mango	L*a*b*	LS-SVM	88.00	[66]
Oil palm	L*a*b*	ANN	91.67	[67]
Pepper	HSV	SVM	93.89	[68]
Paper	HIS + RGB	SIS	99.00	[69]
Persimmon	RGB + L*a*b*	QDA	90.24	[70]
Tomato	HSV	SVM	90.80	[71]
Tomato	RGB	DT	94.29	[72]
Tomato	RGB	LDA	81.00	[73]
Tomato	L*a*b*	ANN	96.00	[74]
Rice	Texture Features (Gray)	SVM	86.00	[75]
Soya	HSI	ANN	95.70	[76]
Banana	RGB	Fuzzy logic	NA	[77]
Banana	RGB + CIE L*a*b*	ANN	NA	[78]
Banana	RGB	CNN	87.00	[77]
Watermelon	YCbCr	ANN	86.51	[79]
Watermelon	VIS/NIR	ANN	80.00	[80]
Watermelon	RGB	ANN	73.33	[81]
Tomato	FTIR	SVM	99.00	[82]
Kiwi	Chemometrics MOS E-nose	PLSR, SVM and RF	99.40	[83]
Coffee	RGB + L*a*b* + Luv + YCbCr + HSV	SVM	92.00	[84]
Coffee	RGB, HIS and L*a*b*	PCA and K-Means	100.00	[85]
Cape Gooseberry	RGB + HSV + L*a*b*	ANN, DT, SVM and KNN	93.02	[86,87]

L* indicates lightness, and a* and b* are chromaticity coordinates.

.

As shown in Table 1, the largest generic classifier technologies are SVM, ANN, K-Means and KNN at 34%, 31%, 11% and 9%, respectively, whereas the most used color spaces in Table 1 are RGB, LAB, HSV, HIS and YCbCr with 57%, 31%, 14%, 9% and 6%, respectively, with high output resolution. Thus, RGB color space and SVM classifier are the most popular technologies that achieved higher resolution.

To increase the production of high-quality crude palm oil, one of the challenges is to harvest the fresh fruit bunches (FFB) of oil palm at the optimal stage of ripeness. Actually, the current methods used to determine the optimal ripened stage are based on color and loose fruits observation. This traditional method relies heavily on the undiscovered technique of palm fruit size experimentation and intuition to accurately determine ripeness that cannot be easily replicated and is subject to significant human error. To address this issue and find a systematic solution to determine the oil palm fruit ripeness that is cost-efficient, fast, non-invasive, reliable and precise, researchers contributed to developing a tech-based solution using computer vision that enables auto-grading and sorting of the optimal ripened stage by integrating software (image processing, robust datasets, AI decision-making) and hardware systems (lighting system, grading and sorting system). The advancement in methods and techniques for FFB classification and grading has resulted in the development of automated computer analysis, which will aid farmers significantly in obtaining good quality in crude palm oil production, particularly in rural areas with limited access to automation facilities.

2.1. Data Collection

According to confirmation between the scientific teams of the Universiti Putra Malaysia (UPM) and Palm Oil Board of Malaysia (MPOB), knowledge and experience were shared to study the properties and future of FFB palm oil at different stages of maturity to collect valuable information. Thus, the study began with a field visit, as shown in Figure 1.

The purpose of the visit was to select the study area and FFB types of oil palm according to the research needs. Accordingly, the preparation of the survey and verification of the methods and techniques for the palm oil fruit maturity grading system involved collecting 270 fruit images for each of the three types of palm oil fruit, which are (i) Nigrescens, (ii) Oleifera and (iii) Virescens, as shown in Figure 2. Each harvested fruit received a specific sheet containing its name, number, type and ripeness class. The data collection process for the oil palm system is as follows:

• An expert in the classification of palm oil fruit maturity was appointed. The expert classified the fruits based on three grades, namely under-ripe, ripe and over-ripe;
• A specified number of fruits per day were collected. The collection ranged from 15 to 20 fruits based on the ability of the lab capacity and the quantity available in the field;
• Give the physical image of each fruit the name and number of the organization using the computer or during laboratory analysis;
• Third item.

2.2. System Material

In general, the material and process of the FFB palm oil maturity classification system is shown in Figure 3. Accordingly, the fruit ripeness grading system used computer vision application in agricultural quality inspection to ensure ripeness category of fruit. The system includes: (a) a housing having an enclosure for scanning process; (b) defused tubes of LED illumination means with optical lens illumination filter provided at the enclosure of the housing; (c) preferably, a suitable charge coupled device (CCD) digital camera DFK 41BF02.H FireWire CCD color camera is used to capture fruit sample’s image, provided at top portion of the enclosure of the housing; (d) a feeding device for conveying fruit samples to the housing; (e) a processing unit to process and analyze the fruit sample image; (f) a data acquisition interface provided in between the camera and the processing unit and wherein the processing unit further provided with a disk top computational unit serves to transfer data to a computer. In fact, the fruit was obtained in real time with a controlled indoor lighting system.

2.3. Image Processing Approach

In general, image processing and analysis using computer vision and external file systems were performed with general and standard steps, as shown in Figure 4 [31,88,89]. Image acquisition and pre-processing include low-level processing, segmentation, representation and description as mid-level operations, while higher-level operations include object recognition and image classification.

As a result, the group of oil palm fruits went through fruit image processing stages based on various steps, as shown in Figure 5. The steps included fruit image acquisition, pre-processing and processing, treatment, segmentation and extraction of features as well as applying the retrieval methods and techniques as a decision-making system based on the similarity calculation as proposed in the future work. All images were related to the training model and a fresh fruit bunch was evaluated. The decision-making process was based on the training model.

Several experiments were performed with different models (color, texture and thorns) of the FFB palm oil classification system. The three different regions of interest (ROI1, ROI2, ROI3) were verified for the FFB maturity of the oil palm, as shown in Figure 6, using various feature extraction techniques (color feature extraction, such as mean, standard deviation and color histogram techniques) as well as texture extraction techniques (Gabor wavelet (GW), gray level co-occurrence matrix (GLCM) and basic gray level halo matrix (BGLAM)).

2.4. Classification System

Decision-making based on image classification through supervised machine learning classifiers is the last step in the process, which is a method of learning a set of rules from cases called a training set to create a classifier that can be used to create a great presentation using new cases for tests [90,91,92]. The classification system defines objects by classifying them in a limited set of categories [93,94,95]. As noted at the beginning of this article in Table 1, the most popular supervised classifiers in fruit categorization are SVM, ANN and KNN. These classifiers were used in this article for the experimental parameters.

2.4.1. Artificial Neural Network (ANN)

An artificial neural network (ANN) provides an efficient alternative for mapping complex nonlinear relationships between input and output datasets without the need for detailed knowledge of the underlying physical relationships [96]. ANNs contain connected nerve cells that mimic the work of the brain. ANN differs significantly from algorithm software due to its ability to disseminate knowledge about new data unearthed. Expert systems must collect real knowledge about the specific area. Multi-layered direct feedback neural networks are grouped into input, output and hidden layers and are used with the FFB oil palm classification system.

Each layer comprises several neurons, which are known as processing elements (PE), as illustrated in Figure 7 [67,97,98,99]. No pre-defined rules were needed to be set for ANN because it is able to learn and generalize from “experience” or a set of presented examples, which is called a training set. The number of optimum hidden neurons was determined experimentally from the training processes of the MLP classifiers. An in-depth description of the MLP concept was addressed by the authors of Ref. [100].

Figure 6 illustrates the construction of a three-layer MLP building. The general task of the PEs in the input layer of an MLP is to buffer the input signals to the PEs in the hidden layer. This step collects the products of input signals with their weighted connections by each PE.

Artificial Neural Network (ANN)

Varying the weights given to neural connections is a process of training a neural network to achieve a satisfactory result. The supervised learning procedure for multi-layered front-end power systems provides a recipe for changing the weight of elements in adjacent layers. This algorithm reduces the sum of squares errors, which have been identified as least squares.

The mean square errors (MSE) and the efficiency (EFF) of the training and testing for each classifier are calculated.

During the training phase, data were used to fit the system using the ANN model. Each category in the dataset was presented as an input sample for ANN–MLP for training assignments. In order to reduce the mean square error (MSE) between goals and outputs, a trial and error trial [100] was performed. Under-ripe, ripe and over-ripe were determined using the desired outputs as 0.5, 0 and 1, respectively, while the input characteristics were normalized within the range [0, 1]. Training effectiveness was used as an important indicator of the accuracy of rating evaluation. However, each method used different ANN constructs to result in inefficiency. The commonly used backpropagation networks were selected for the FFB classification system for oil palm trees due to their success with a variety of image processing applications in agriculture [101,102,103,104].

2.4.2. K-Nearest Neighbor (KNN)

KNN is another supervised classifier used in this work based on the concept that observations in a dataset are, in general, close to other observations with similar properties. Additionally, the metric distance and k-value play a major role in the KNN classification algorithm [105], although Ref. [106] notes that the KNN classifier is not a pre-classifier; KNN determines their location. kNN is used to query the new training space model based on the appropriate similarity distance scale.

KNN Performance

KNN regulation is one of the largest algorithms for classifying attractive patterns. In this work, different k-values and distance measurement methods were adapted to balance the trade-off of the FFB maturity classification by excluding values and methods having low confidence accuracy levels, as shown in Figure 8.

Moreover, an experimental investigation was carried out based on the values of K, which are 1, 3, 5, 7 and 9, as well as methods for measuring metric distance, namely: Euclidean, City, Cosine and Correlation, as in “Equations” by Refs. [105,107]. The research aims to determine the appropriate KNN classification coefficient for the high-precision FFB palm oil maturity classification system. The study showed that the appropriate distance measure that reduced the distance between two similar classified examples is the city-block metric. The value of k = 1 affects the performance of the KNN procedure. The results of the evaluation can be obtained next regarding applications in agriculture [101,102,103,104].

2.4.3. Support Vector Machine (SVM)

KNN SVM is a supervised machine learning classifier developed by the authors of Ref. [108] based on constructing hyper-plane as a decision line separating Class 1 from Class 2, as shown in Figure 9 [109]. A special characteristic of SVMs is that they simultaneously reduce experimental classification error and maximize geometric boundary by optimizing the superlative level of linear separation and converting the nonlinear data model into a linearly separable format in a feature space with high-dimensional [110].

In the FFB maturity classification, there are three different target classes (under-ripe, ripe and over-ripe) and one against all approach (OAA), which subdivides each class and merges all the others [111]. Due to the performance efficiency and less processing time than the multi-class SVM classifier, the OAA method was used to perform the FFB ripeness classification of oil palm.

SVM Performance

To improve the classification result for specific models, special classes of FFB palm oil had to be learned according to linear, non-linear and four-step basis. First, the input data comprise two sets of vectors in an n-dimensional space. SVM will build a separate hyperplane in that space that increases the “margin” between the two datasets. Second, when calculating the margin, we construct two parallel planes parallel, one on each side of the separator planes, that are “pushed up” for the two datasets. Third, instinctively, a fine separation is reached by means of the hyper-plane that has the largest distance to the data points adjacent to both classes. Finally, the classifier’s best generalization error will depend on the largest margin or distance between these parallel hyperplanes.

The parameter tuning is the most important factor in the SVM model-building process. In SVM, tests were accomplished with different kernel types, such as linear, polynomial and radial basis function kernels, to achieve the classification task. Furthermore, to control the trade-off between maximizing the margin and minimizing the training error, the sigma of RBF was tuned from 1 to 100 and the polynomial distance was also tuned from 1 to 4. The regularization parameter magnitude C was tuned from 1 to 1000 for both polynomial and RBF kernels.

2.5. Training and Testing

As with Kotsiantis, three techniques are used to estimate the accuracy of the classifier [105]. First is the cross-validation technique, by dividing the training set into subgroups of equal selection and size. For each subgroup, the classifier is trained on one of all other subgroups. The second is the leave-one-out validation. The third is the most common one, which is used in this work with the FFB palm oil grading system. Two-thirds of the data are for training and the remaining is for performance appraisal.

Numerous statistical measurements of efficiency and mean square error (MSE) were applied as indexes to validate the performance of the classifier. In particular, an automatic parameter tuning procedure as in Ref. [112] is implemented for the system to dynamic adaptive thresholding algorithm for the oil palm FFB ripeness grading. The objective of supervised learning is to create a concise model of the distribution of class labels in terms of predictor features.

Training and Testing Stage

The training stage includes data collection, data analysis and a training model analyzing 270 fruit samples of three different ripeness categories for the three different oil palm FFB types that were collected, analyzed and then a training model for fruit image type and ripeness classification was created. Meanwhile, the testing stage included testing the grading system initially in the lab. Testing the grading system in the field ensured that the system provided a high percentage of internal validity for findings obtained using the system design. Furthermore, 90 samples for each class were used to test the oil palm FFB ripeness grading system. Figure 10 illustrates the main approaches considered in the classification module for oil palm FFB types and ripeness.

In general, the classification of FFB type and ripeness of oil palm was successfully carried out based on the performance of three levels of image processing and subsequent analysis, as shown in Figure 11.

2.6. Classifier Performance Evaluation

The performance measurement of a classifier independently is conducted according to its sensitivity and specificity. The analysis of the ROC of a classifier is a solution to limit the empirical precision of binary classification. Results significantly greater than 50% could be due to a biased classifier tested on an unbalanced dataset, and overall precision does not differentiate between forms of error [113]. The experiments aimed to infer the crucial architecture with the selected color, texture and spine models using the ROC as a statistical measurement analysis. This analysis provides a quantitative assessment using AUC.

Receiver Operating Characteristic Curve

Figure 12 shows the ROC curve, which has become the standard tool for evaluating predictive accuracy to evaluate and compare models and prediction algorithms. ROC analysis offers a methodical analysis of the sensitivity and specificity of judgment [114,115].

Sensitivity is the capability of the classifier to recognize the positive pattern amongst the truly positive patterns. Specificity is the ability of the classifier to recognize the negative patterns amongst the truly negative patterns. Figure 12 shows that point (0,1) is the ideal classifier, which categorizes all the positive and negative cases appropriately. In this instance, the false positive rate is none (0), and the true positive rate is all (1). In addition, point (0, 0) indicates that the classifier predicted all the cases to be negative, while point (1, 1) matches a classifier with all the cases that are positive. Point (1,0) means the classifier fails to implement the correct classification for all the cases, as shown in Figure 9. The given n test samples are constructed according to the confusion matrix as illustrated in

Table 2. Confusion matrix.

Test	Actual Positive	Actual Negative
Predicted Positive	TP	FP
Predicted Negative	FN	TN
Total	P	N

that resulted from classification [113,115,116,117]. The calculation of accuracy, sensitivity or true positive rate (TPR) and 1- specificity or false positive rate (FPR) are given by ‘Equations (1)–(3)’, respectively.

$$Accuarcy=\frac{TP+TN}{n}$$

(1)

$$TPR=Sensitivity=\frac{{\displaystyle \sum TP}}{{\displaystyle \sum TP}+{\displaystyle \sum FN}}$$

(2)

$$FPR=Specificity=\frac{{\displaystyle \sum TN}}{{\displaystyle \sum TN}+{\displaystyle \sum FP}}$$

(3)

where the true TP positives are the number of correctly classified maturities; true negatives TN is the number of incorrectly classified maturities; false positives FP is the number of maturities classified as non-maturities and false negative FN is the number of non-maturities classified as maturities. Finally, the performance evaluation of the oil palm FFB maturity classification system classifier typically includes the measurement of sensitivity and specificity as performance results based on the ROC curve and measurement of the area under the ROC curve (AUC).

3. Results and Discussion

The FFB characteristics of the oil palm (color, texture and thorns) were extracted using the algorithms of the color model, the texture model and the thorn model. Three different supervised machine learning techniques, ANN, KNN and SVM, were incorporated into the extracted features based on the three different models to make decisions regarding FFB type and maturity. Experiments were carried out on the classifiers to select the appropriate model for the FFB oil palm grading system and to ensure high-quality grading results. The best possible classification accuracy can be achieved by selecting the highest AUC measured from the ROC curve.

3.1. Classification Based on ANN–MLP

This section discusses MLP models as classifying FFB maturity of oil palm based on statistical color function, color histogram, Gabor wavelets, GLCM and BGLAM functionality. The different ANN models selected on the basis of the experimental results were performed with different feature extraction techniques implemented in the oil palm grading system, as shown in Figure 13. A comparison between the MSE and the effectiveness of the training results and test steps was performed to validate the parameters of the ANN supervised learning classifier, as shown in

Table 3. MSE and efficiency result comparison of the training and testing stages based on the FFB feature techniques.

FET	Models	Training Stage		Testing Stage
		MSE	Eff	MSE	Eff
Statistical	[40 × 10 × 1]	0.0080	0.9523	0.0190	0.8865
	[40 × 20 × 1]	0.0072	0.9568	0.0197	0.8820
	[40 × 30 × 1]	0.0038 *	0.9775 *	0.0182 *	0.8914 *
Histogram	[25 × 10 × 1]	9.9402 × 10−5 *	0.9994 *	0.0136 *	0.9189 *
	[25 × 15 × 1]	9.8412 × 10−5	0.9994	0.0163	0.9024
	[25 × 20 × 1]	9.5647 × 10−5	0.9994	0.0144	0.9140
GLCM	[40 × 10 × 1]	9.9991 × 10−5	0.9994	0.0291	0.8263
	[40 × 20 × 1]	1.2126 × 10−4	0.9993	0.0330	0.8030
	[40 × 30 × 1]	9.9943 × 10−5 *	0.9994 *	0.0278 *	0.8338 *
BGLAM	[45 × 11 × 1]	8.9770 × 10−5	0.9995	0.0242	0.8556
	[45 × 22 × 1]	9.6629 × 10−5 *	0.9994 *	0.0177 *	0.8942 *
	[45 × 33 × 1]	9.5860 × 10−5	0.9994	0.0215	0.8712
Gabor	[40 × 10 × 1]	9.9573 × 10−5	0.9994	0.0922 *	0.4489 *
	[40 × 20 × 1]	9.9895 × 10−5 *	0.9994 *	0.0966	0.4228
	[40 × 30 × 1]	9.9873 × 10−5	0.9994	0.1330	0.2048

Notes: FET = Feature extraction techniques, MSE = Mean square error, Eff = Efficiency, * = The best result.

.

Table 3 indicates that the MLP and MSE learning stage of the learning procedures did not exceed 0.003. The higher proficiency score observed revealed a selection scale of the architectural MLP model [40 × 30 × 1], [25 × 10 × 1], [40 × 30 × 1], [45 × 22 × 1] and [40 × 20 × 1] over all the tracks with statistical color function, color histogram, GLCM, BGLAM and GW, respectively, for the FFB recording system. After several training sessions, the MLP model was able to learn and perfectly match the target in the training phase with extreme efficiency and with complete FFB palm oil training datasets. During the test phase, Figure 13a–e shows the classification of ROC plots performed by the FFB oil palm maturity classification system, with a higher AUC score observed in the MLP models.

3.2. Classification Based on KNN

The basic principle of the oil palm grading system based on nearest neighbor (NN) approximation is that two FFB images with similar color, texture and thorn features should reveal similar classes and grades. Thus, using the FFB images of similar ripeness is sensible when identifying the new FFB image. All images in the database can be grouped based on their ripeness features. The nearest neighbor technique is defined as dividing a sample set into categories, with each category holding similar samples that share the same features. The testing sample is determined by the known classifications of the training samples.

Based on the samples’ characteristics, five main steps were described to classify FFB images of oil palm into their categories (under-ripe, ripe and over-ripe). Indeed, choosing the best k-values and appropriate distance measurements ensures the accuracy of the results of the KNN classifier, which were usually chosen experimentally by static validation with a set of k-values and distance measurements. Thus, the best k-value that can be used with feature extraction techniques (statistical color feature, color histogram, Gabor wavelet, GLCM and BGLAM) was verified.

Figure 14 shows the ROC area for the best results performed by KNN with different values of k = 1, 3, 5, 7 and 9 and with different distance metrics, Euclidean, city-block, cosine and correlation, for the FFB oil palm maturity grading system with feature extraction techniques. Therefore, the experimental results show that k-value = 1 with the city-block distance technique provides the greatest AUC scores equal to 93.00%, 92.00%, 91.00%, 92.00% and 80% using feature extraction techniques, including statistical, color histogram, GLCM, BGLAM and Gabor wavelet, respectively, based on the KNN algorithm, as shown in Figure 14.

3.3. Classification Based on SVM

The SVM algorithm is implemented in the FFB maturity classification of oil palm, and the input data include three sets of vectors in the n-dimensional SVM space. These data create a discrete hyper-plane in this space, which increases the “margin” between the three datasets and reduces the expected generalization error. In the case of oil palm FFB ripeness grading, three target categories exist, namely under-ripe, ripe and over-ripe. In this case, OAA is used, in which each class is split out and all the other classes are merged in the oil palm FFB grading system to solve multiclass issues with less computation time. An important aspect of the SVM model-building process is parameter tuning.

Three different types of kernel functions, linear, polynomial and radial (RBF), were used to perform the classification task. To control the trade-off between maximizing the headroom and minimizing the training error, the sigma of RBF was set from 1 to 100, while the polynomial distance was also set from 1 to 4. The magnitude of the regularization parameter C was set from 1 to 1000 for polynomial kernels and RBF, as explained in

Table 4. Best results of RBF kernel function based on sigma values and c with FFB ripeness grading.

FET	RBF-Sigma	C	Accuracy %
			ROI1	ROI2	ROI3
Statistical	1	1000	90	90	82
Histogram	50	100	89	90	91
GLCM	1	500	75	78	79
BGLAM	10	500	92	90	92
Gabor	10	500	76	87	89

Note: FET = Feature extraction techniques.

.

As shown in Figure 15, the kernel function provided a significantly higher accuracy rate for the FFB maturity classification of oil palm. The results are based on different values of sigma and c, as examined by other research [111], and a comparison of linear and nonlinear polynomial kernel functions. Therefore, as demonstrated in Table 4, the experimental results show that RBF-sigma = 10 with C = 500 provides the greatest results of 92% using BGLAM with ROI3 based on the SVM algorithm, as shown in Figure 15.

3.4. Experimental Results

Four experiments were carried out. In experiment 1, the texture characteristics of the oil palm were extracted and the classification was performed for the FFB type classification. In experiments 2, 3 and 4, oil palm color, texture and thorn features were extracted. The classification was then conducted for the Nigrescens, Oleifera and Virescens FFB ripeness grading.

The complete picture of the threshold between the sensitivity and 1- specificity is displayed by plotting the ROC curve across a series of threshold points. The AUC is considered to be an effective measurement of the inherent validity of a grading system test. This curve is suitable for (a) assessing the discriminatory ability of a test to pick correctly the under-ripe, ripe and over-ripe classes; (b) finding the optimal threshold point to minimize class misclassification and (c) comparing the efficacy of ROI1, ROI2 and ROI3 for assessing the same sample or class, as illustrated in Figure 16.

3.4.1. FFB Type Grading System Results

The oil palm grading system was able to accurately classify the three different oil palm FFB types based on the external texture features and properties by using feature extraction techniques GLCM and BGLAM and supervised machine learning classifiers ANN, KNN and SVM, as critically explained in

Table 5. Results of test computing FFB type classification based on GLCM and BGLAM using ANN, KNN and SVM.

Image Size	Classifiers	Texture Feature Extraction Techniques
		GLCM		BGLAM
		Testing Accuracy (%)	Time (s)	Testing Accuracy (%)	Time (s)
ROI1	ANN	89.00	4.08	91.00	1.02
	KNN	87.00	4.06	75.00	0.98
	SVM	79.00	4.7	90.00	1.74
ROI2	ANN	86.00	0.553	89.00	0.43
	KNN	84.00	0.50	76.00	0.38
	SVM	67.00	1.13	77.00	1.14
ROI3	ANN	86.00	0.56	93.00 **	0.44 **
	KNN	77.00	0.51	89.00	0.40
	SVM	69.00	1.22	82.00	1.32

Note: ** = The best result.

.

Table 5 indicates that the fastest and most accurate method and technique for the oil palm type grading system is the BGLAM feature extraction technique combined with the ANN supervised machine learning technique applied on pruning a 100 × 100-pixel FFB image with the ROI3. This finding achieved an optimal accuracy of 93.00% and an image processing speed of 0.44 s in the test performance.

3.4.2. FFB Ripeness Grading System Results

The maturity classification task was trained and tested for the three closest classes, over-ripe, ripe and under-ripe, based on the three FFB maturity models of oil palms: color, texture and thorns.

Color Model

The ripeness grading system testing performance based on the color model for different FFB image ROIs was evaluated. The results are clearly illustrated in

Table 6. Results of test computing FFB ripeness classification based on statistical and histogram using ANN, KNN and SVM.

T	C	Image Size	Color Feature Extraction Techniques
			Statistical Color Feature		Color Histogram
			Testing Accuracy (%)	Time (s)	Testing Accuracy (%)	Time (s)
T1	ANN	ROI1	93.00	3	92.00	2.7
		ROI2	92.00	2.5	92.00	1.4
		ROI3	92.00	2.65	94.00 **	1.6 **
	KNN	ROI1	81.00	2.3	81.00	2.5
		ROI2	82.00	1.2	82.00	1.2
		ROI3	82.00	1.02	82.00	1.3
	SVM	ROI1	81.00	5.3	81.00	7
		ROI2	81.00	4.5	82.00	6
		ROI3	80.00	5	80.00	7
T2	ANN	ROI1	93.00	3	93.00	2.7
		ROI2	92.00	2.5	93.00	1.4
		ROI3	91.00	2.65	94.00 **	1.6 **
	KNN	ROI1	92.00	2.3	93.00	2.5
		ROI2	91.00	1.2	90.00	1.2
		ROI3	92.00	1.02	92.00	1.3
	SVM	ROI1	90.00	5.3	90.00	7
		ROI2	90.00	4.5	90.00	6
		ROI3	82.00	5	91.00	7
T3	ANN	ROI1	91.00	3	90.00	2.7
		ROI2	88.00	2.5	93.00 **	1.4 **
		ROI3	90.00	2.65	92.00	1.6
	KNN	ROI1	74.00	2.3	78.00	2.5
		ROI2	74.00	1.2	79.00	1.2
		ROI3	87.00	1.02	79.00	1.3
	SVM	ROI1	69.00	5.3	78.00	7
		ROI2	73.00	4.5	78.00	6
		ROI3	72.00	5	79.00	7

Notes: T = Types, T1 = Nigrescens, T2 = Oleifera, T3 = Virescens, C = Classifier, ** = The best result.

.

Table 6 indicated the optimal methods and techniques that are the fastest and most accurate for the ripeness grading system. The data are based on the color histogram feature extracted combined with the ANN technique applied to the 100 × 100-pixel FFB image size with ROI3. The results achieved 93.00% accuracy and 1.6 s image processing speed in terms of testing performance for Nigrescens and Oleifera and 100%, 93% testing performance accuracy and 1.4 s image processing speed based on the ANN technique applied with ROI2. For Virescens, the statistical color feature accurately obtained 93% testing performance based on ANN for the different oil palm types. However, the results were limited by the slow processing time compared with the color histogram performance and the oil palm system objectives.

Texture Model

The ripeness grading system testing performance based on the texture model for different FFB image ROIs was evaluated. The results are clearly illustrated in

Table 7. Results of test computing FFB ripeness classification based on GLCM, GLAM and Gabor by using ANN, KNN and SVM.

T	C	Image Size	Texture Feature Extraction Techniques
			GLCM		BGLAM		Gabor
			Testing Accuracy (%)	Time (s)	Testing Accuracy (%)	Time (s)	Testing Accuracy (%)	Time (s)
T1	ANN	ROI1	91.00	3.6	89.00	1	86.00	1.03
		ROI2	89.00	1.7	90.00	0.40	81.00	0.43
		ROI3	91.00	2.2	92.00 **	0.43 **	80.00	0.44
	KNN	ROI1	79.00	2.8	80.00	1	74.00	1.04
		ROI2	77.00	1.5	82.00	0.39	77.00	0.39
		ROI3	78.00	1.6	81.00	0.40	77.00	0.41
	SVM	ROI1	80.00	7.7	80.00	1.9	76.00	1.97
		ROI2	76.00	5.5	81.00	1	82.00	1.79
		ROI3	79.00	6	81.00	0.85	74.00	2.00
T2	ANN	ROI1	92.00	3.6	92.00	1	83.00	1.03
		ROI2	91.00	1.7	93.00 **	0.40 **	79.00	0.43
		ROI3	92.00	2.2	93.00	0.43	83.00	0.44
	KNN	ROI1	79.00	2.8	90.00	1	78.00	1.04
		ROI2	80.00	1.5	91.00	0.39	90.00	0.39
		ROI3	91.00	1.6	92.00	0.40	79.00	0.41
	SVM	ROI1	75.00	7.7	92.00	1.9	76.00	1.97
		ROI2	79.00	5.5	90.00	1	88.00	1.79
		ROI3	80.00	6	92.00	0.85	89.00	2.00
T3	ANN	ROI1	87.00	3.6	88.00	1	82.00	1.03
		ROI2	89.00	1.7	93.00 **	0.40 **	81.00	0.43
		ROI3	86.00	2.2	91.00	0.43	77.00	0.44
	KNN	ROI1	76.00	2.8	77.00	1	74.00	1.04
		ROI2	72.00	1.5	79.00	0.39	73.00	0.39
		ROI3	72.00	1.6	90.00	0.40	78.00	0.41
	SVM	ROI1	65.00	7.7	76.00	1.9	72.00	1.97
		ROI2	64.00	5.5	79.00	1	67.00	1.79
		ROI3	67.00	6	80.00	0.85	68.00	2.00

Notes: T = Types, T1 = Nigrescens, T2 = Oleifera, T3 = Virescens, C = Classifier, ** = The best result.

.

As indicated in Table 7, the fast and accurate method and techniques used for the oil palm FFB ripeness grading system based on the texture model were primarily the BGLAM combined with the ANN technique. This technique was applied to the ROI3 with 92.00% testing performance accuracy with a 0.43 s image processing speed for Nigrescens. Moreover, the BGLAM combined with the ANN technique applied to the ROI2 achieved 93.00% testing performance accuracy with a significant image processing speed of 0.40 s for Oleifera and Virescens. Due to the sensitivity of SVM to noise and the weakness of the Gabor wavelet and GLCM techniques with texture features and processing time, the limitations of these methods and techniques are clearly stated in the testing result tables.

Thorn Model

The ripeness grading system performance of the oil palm FFB types for testing based on the thorn model for the different ROIs was evaluated. The results are clearly illustrated in

Table 8. Results of test computing FFB ripeness classification with statistical, histogram, GLCM, BGLAM and Gabor using ANN, KNN and SVM.

T	Technique	Image Size
		ROI1		ROI2		ROI3
		Testing Accuracy (%)	Time (s)	Testing Accuracy (%)	Time (s)	Testing Accuracy (%)	Time (s)
T1	Statistical and KNN	79.00	4.06	78.00	2.3	79.00	2.5
	Histogram and SVM	80.00	9	81.00	8	81.00	8.5
	GLCM and ANN	87.00 **	3.7 **	84.00	2.6	85.00	2.9
	BGLAM and ANN	76.00	1.6	79.00	0.8	78.00	0.83
	Gabor and KNN	78.00	1.6	74.00	0.78	74.00	0.84
T2	Statistical and KNN	87.00	4.06	90.00	2.3	79.00	2.5
	Histogram and SVM	88.00	9	80.00	8	79.00	8.5
	GLCM and ANN	91.00	3.7	89.00	2.6	88.00	2.9
	BGLAM and SVM	91.00	2.5	89.00	1.36	91.00 **	1.20 **
	Gabor and KNN	89.00	1.6	82.00	0.78	86.00	0.84
T3	Statistical and ANN	84.00	4.06	86.00	2.3	87.00	2.5
	Histogram and SVM	73.00	9	77.00	8	79.00	8.5
	GLCM and ANN	88.00	3.7	84.00	2.6	87.00	8.5
	BGLAM and ANN	84.00	1.6	82.00	0.8	87.00 **	0.83 **
	Gabor and KNN	74.00	1.6	73.00	0.78	78.00	0.84

Notes: T = Types, T1 = Nigrescens, T2 = Oleifera, T3 = Virescens, ** = The best result.

.

Due to data noise, the thorn model shows poor results based on performance and processing time, while BGLAM combined with ANN technology with SVM technology applied to ROI3 achieved a test performance of 91.00% and an Oleifera image processing speed of 1.20 s.

4. Conclusions

An FFB fruit palm oil ripeness classification system was designed based on remote sensing sensors (CCD camera) and image processing technologies as computer vision applications for inspection of agricultural crop quality.

The system aims to ensure the maturity class of different types of FFB palm oil based on external characteristics, such as color, texture and thorns. Image processing methods and techniques, including the acquisition and segmentation of images in ROI1, ROI2 and ROI3 and the extraction of image properties as a function of the statistical function of the color, histogram color, GLCM, BGLAM and Gabor wavelet, were implemented.

In addition, decision-making for image classification through training and testing of the system based on the different algorithms, SVM, KNN and ANN, was implemented on a maturity classification system. The training and testing of oil palm FFB species (Nigrescens, Oleifera and Virescens) and maturation (under-ripe, ripe and over-ripe) depending on the color, texture and pattern of the thorns were extracted.

AUC and ROC were used to accurately estimate and evaluate the performance of different classifiers based on system performance, processing time and system cost. The results showed that the texture models were improved with ANN classifiers as the best result of the algorithm classifier, ANN-based BGLAM with ROI3, provides 93.00% accuracy with a shorter image processing time of 0.44 (s) for FFB type recognition. Meanwhile, the BGLAM algorithm that relies on ANN and ROI3 obtained 92.00% accuracy and a short processing time of 0.43 (s) for Nigrescens, plus the algorithm BGLAM based on ANN and ROI2 obtained 93.00% accuracy and a short processing time of 0.40 (s) for Oleifera and Virescens for maturity classification.

In the final analysis, different predictions were used. Maximum accuracy was obtained using an ANN classifier with the highest prediction accuracy observed compared to all the other classifiers. The following more accurate prediction is indicated by the different classifiers: KNN and SVM, respectively. The scope of the existing work is limited to investigation of the relationship between oil palm fruit ripeness level and image processing approach and AI.

As mentioned above, the authors have implemented several experiments based on different methods and techniques for automation of a real-time oil palm FFB ripeness grading system that carried out satisfactory results, but, in the future, the existing work can be extended to include some recommended practical actions and scientific studies of the system’s hardware and software for developing the current system and improving the results.

In terms of hardware development, proper hardware design and development make it easier for the programmer to set his algorithm for a high-accuracy performance result. (1) Since the illumination system is one of the most important hardware parts in the oil palm grading system, in order to control the lighting beam incident on FFB to be reflected to the camera, a linear polarizer (LP) filter should be fixed at the camera and each light source, and (2) it is important to use other types of sensors, such as a thermal camera, to collect valuable information about the oil palm FFB ripeness and build grading system models based on the obtained information.

Regarding software development, the real-time oil palm FFB ripeness grading system was implemented as a solution for effective oil palm FFB ripeness grading. However, in order to improve the oil palm system functionality and performance, different methods and techniques should be proposed based on the system software, such as (1) using the oil palm FFB internal feature lab analysis information, such as oil content and free fatty acid, to correlate with external features of FFB, such as color and texture features, to validate and support the oil palm FFB ripeness result; (2) applying the retrieval methods and techniques as a decision-making system based on the similarity calculation as proposed and illustrated in Figure 5; (3) further research is needed to generalize the system for other agriculture applications by considering the size, weight and shape of FFB during the system design. Hence, that assembles the system to be a multipurpose application system, which can be used in similar applications for different agricultural crops. Although the existing study utilizes image processing, similar results are expected to be obtained using a portable device. The proposed method has the potential to be a rapid on-site assessment tool for ripeness classification in the oil palm industry.