Enhancing Seed Germination Test Classification for Pole Sitao (Vigna unguiculata (L.) Walp.) Using SSD MobileNet and Faster R-CNN Models

Featured Application

This study introduces a machine learning-based system to automate the classification of pole sitao (Vigna unguiculata (L.) Walp.) seeds into normal, abnormal, and ungerminated categories during germination tests. This system could potentially expedite and improve the reliability of seed germination testing for pole sitao.

Abstract

The classification of germinated pole sitao (Vigna unguiculata (L.) Walp.) seeds is important in seed germination tests. The automation of this process has been explored for different grain and legume seeds but is only limited to binary classification. This study aimed to develop a classifier system that can recognize three classes: normal, abnormal, and ungerminated. SSD MobileNet and Faster R-CNN models were trained to perform the classification. Both were trained using 1500 images of germinated seeds at fifth- and eighth-day observations. Each class had 500 images. The trained models were evaluated using 150 images per class. The SSD MobileNet model had an accuracy of 0.79 while the Faster R-CNN model had an accuracy of 0.75. The results showed that the average accuracies for the classes were significantly different from one another based on one-way ANOVA at a 95% confidence level with an F-critical value of 3.0159. The SSD MobileNet model outperformed the Faster R-CNN model in classifying pole sitao seeds, with improved precision in identifying abnormal and ungerminated seeds on the fifth day and normal and ungerminated seeds on the eighth day. The results confirm the potential of the SSD MobileNet model as a more reliable classifier in germination tests.

1. Introduction

Pole sitao (Vigna unguiculata (L.) Walp.), a member of the Leguminosae family, is one of the most cultivated crops in the Philippines. It is also popularly known as asparagus bean, Chinese long bean, garter bean, snake bean, or yard long bean. It can be served steamed, sautéed, buttered, or cooked together with other vegetables [1]. The quality of the pole sitao seed is one of the main factors that determine the harvests’ quality and quantity. In choosing the best seed to plant, it should have a good germination rate to maximize the yield and achieve the desired quality of the harvested vegetable. However, the manual inspection and classification of seeds for germination tests can be a time-consuming, inconsistent, and laborious task for the inspector [2]. Seeds subjected to germination tests can be classified into three categories: normal, abnormal, and ungerminated [3]. Seeds under the normal classification have well-developed and complete germinate parts. Seeds under the abnormal classification have stunted or missing germinate parts. Lastly, seeds under the ungerminated classification show no sign of germination. With the amount of detail required for inspection, manual inspection and classification are time-consuming, especially if there are many seeds to be inspected and classified. Given the prescribed observation days of seeds, the inspection and classification should be finished within the prescribed time frame. Also, the inspector can have an inconsistent judgment, especially for inexperienced inspectors, and the judgment varies from person to person inspecting the seeds. Moreover, it can be laborious and cause eye strain since inspecting and classifying seeds include a series of steps from inspecting them one by one to classifying them based on the set criteria.

A possible tool that can be used to address the difficulty of manual classification is to automate the process and use machine vision and machine learning. Machine vision helps to automate the process of classifying seeds in the germination test. Similarly, this technique combined with machine learning technology can be used in performing the classification for germination tests, which is also becoming a powerful tool for this process. Machine learning is a state-of-the-art technology that is used to mimic the human brain for independent decision-making of the machine. It uses models that act as a brain that can be trained to learn the features and characteristics of the input dataset. In modern machine learning, the technology uses neural network-based models that can process data in a short amount of time [4]. By utilizing these technologies, it is possible to assess and classify the quality of seeds based on the set standards with relatively higher speed and accuracy compared to conventional methods.

Some studies have been conducted to solve this problem by automating the abovementioned process. In the study by Lurstwut and Pornpanomchai [5] on rice seeds, the authors developed MATLAB-based software called Rice Seed Germination Analysis. The software claims to classify germinated rice seeds from ungerminated seeds. The authors trained the software using four algorithms and models: Euclidean Distance (ED), Rule-Based System (RBS), Fuzzy Logic (FL), and Artificial Neural Network (ANN). A total of 600 images of rice seeds were used to train the models, and 120 images were used for testing. The software generated testing accuracies of 87.50%, 100%, 100%, and 100% for ED, RBS, FL, and ANN, respectively.

Another study on automated seed germination tests was conducted by Genze et al. [6] for various crops. They used machine learning models to classify the germinated seeds of Zea mays (maize), Secale cereale (rye), and Pennisetum glaucum (pearl millet) into two classifications: germinated and non-germinated. A total of around 24,000 seeds for the three species were used for the study, 2449 of which were used as testing data, while the remaining were used as a training dataset. The Faster R-CNN model was used, and it generated mean precision values of 97.90%, 94.25%, and 94.21% for maize, rye, and pearl millet, respectively.

Although studies on automating seed classification have produced good accuracy values, there are still gaps that need to be addressed. Gill and Singh [7] applied image processing for seed germination tests. Lurstwut and Pornpanomchai [8] and Nguyen et al. [9] developed classifiers for paddy rice germination, while Khoenkaw [10] and Peng et al. [11] focused on wild rice and rape seeds, respectively. These studies did not examine legume germination, like pole sitao. ElMasry et al. [12] utilized a linear discriminant analysis model for cowpea seeds but did not differentiate between normal and abnormal germination. Mladenov and Dejanov [13] proposed a color and texture model with a Radial Basis Function Network (RBFN) to identify normal, abnormal, and non-germinated seeds. Colmer et al. [2] developed software for the phenotypic analysis of various seeds without specifically classifying seed germination types. Lastly, Ureña, Rodríguez, and Berenguel [14] performed an analysis using an artificial vision system and a fuzzy logic-based classifier for lettuce, cauliflower, and tomato seeds. Their procedures followed the guidelines of the International Seed Testing Association [3], but they did not report the accuracy of the system in classifying abnormal and normal seed germinates.

Recent papers have applied machine learning to seed classification automation. Although several crop species have been studied, research on pole sitao is lacking. Moreover, the relevant studies have only attempted to classify seeds into germinated and non-germinated categories. Few studies have attempted to differentiate between normal germination, abnormal germination, and the absence of germination. To address these gaps, this paper explored the potential of machine learning for classifying pole sitao seeds that were subjected to germination tests. The general objective was to develop a machine learning-based classifier system for pole sitao seeds in seed germination tests. Specifically, this study aimed to train machine learning models and evaluate their performance in classifying pole sitao seeds in a germination test.

2. Materials and Methods

2.1. Collection and Preparation of Germinated Seed Images for the Dataset

Pole sitao seeds were germinated following the guidelines of the International Seed Testing Association [3] for seed germination. As shown in Figure 1, the researchers soaked the seeds for 24 h to break dormancy. The soaked seeds were arranged on a board with a moistened paper towel. This served as germination media for the germination test. Each board contained 50 seeds that were 4 cm apart and arranged in a 10 × 5 matrix, as shown in Figure 2. A total of 1500 seeds were used for the germination tests. Each germination test run had 8 germination media to complete the needed 400 seeds per run. The germination media were kept in a sealed box to maintain a temperature between 20 °C and 30 °C and a relative humidity between 90% and 95%.

Each germination test run lasted for 8 days. Digital images of the seeds in the germination media were captured on the 5th day and the 8th day of the germination test. An 8-megapixel Apple iPad (8th Generation) camera was used to capture images of the seeds. Each seed in the captured images was then manually inspected and labeled using an image labeling application called LabelImg (Figure 3). The labels included the classification of the seed (normal, abnormal, and ungerminated) and the day of observation. The authors then selected 1500 labeled seeds. Each class had 500 representative images. The labeled seeds were then used for the model training. This is comparable to the 490 seed images per classification used by de Medeiros et al. [15] in training their model, which achieved an overall accuracy of 83%. To make sure that this study’s generated accuracy matched their generated accuracy, the number of training samples was set higher than that used by Medeiros et al. [15]. Aside from the training dataset, 150 images for each of the three germination classes were randomly selected from the labeled seed pool and used as an evaluation or a testing dataset.

2.2. Development of the Classifier System

The foundation for the classifier system was established within an Anaconda 3 virtual environment. This preliminary step was essential to isolate the TensorFlow installation alongside other necessary libraries. Following the activation of the virtual environment, path configurations were implemented to enable dynamic file and URL access. The installation of TensorFlow 2.8 and other required libraries was achieved through the Anaconda 3 Command Prompt.

A general process flow of the classifier system is shown in Figure 4. The procedure begins with the system loading an image. This includes a visual representation of a germination tray. The system then evaluates the image to determine the presence of a seedling. If a seedling is detected, the system proceeds to classify the seedling into one of the predefined categories: normal, abnormal, or ungerminated. Upon successful classification, each seedling is labeled accordingly. The output, which consists of the labeled seedlings, is then displayed.

2.3. Model Selection and Training Process

The classifier system for the germination test of pole sitao was developed using TensorFlow as the main machine learning library, Jupyter Notebook as the program environment, and Python as a programming language. The Single Shot Detector (SSD) MobileNet and Faster Region-based Convolutional Neural Network (Faster R-CNN) models were used in this study because they represent one-stage and two-stage models, respectively, which are the main categories of deep learning-based architectures.

The SSD MobileNet model is a machine learning model that is commonly used in mobile phone applications due to its simple, fast, and accurate detection capability. This model belongs to the one-stage model category, which directly analyzes the bounding boxes and classification of the object from the input image. The SSD MobileNet model includes feature extraction, i.e., it extracts the features of the object using six layers of resolution within the image. Each layer uses different feature maps generated from the different aspect ratios. It then generates proposals containing the features of the object. The proposals are then filtered using non-maximum suppression criteria to assign the necessary classification [16].

The Faster Region-based Convolutional Neural Network (Faster R-CNN) model, a two-stage model, first generates a region of interest (ROI) and then analyzes the object’s features. This model was developed to classify multiple objects within an image [6]. This model utilizes a Region Proposal Network (RPN), a feature that improves the performance of typical R-CNN and Fast R-CNN architectures. It reuses the results from the slow search selective algorithm of CNN instead of running additional search selective algorithms. In effect, the system would yield a faster training time and ensure the high accuracy of the model [17]. It includes convolutional layers, which are a set of filters used to learn the features of the image. The convolutional layers are input into the RPN, which generates region proposals according to the features of the image. The model then analyzes the features of the generated proposal from the RPN. This process is referred to as ROI pooling. After extracting the desired features, the generated features will be fed to the classifier for the final decision [18].

The SSD MobileNet model belongs to the one-stage model category and analyzes the features of the object as a whole. It was used in this study since the objects or samples had the same features and were not highly different from one another, especially the normal and abnormal classes. On the other hand, the Faster R-CNN model was used as it was one of the latest CNN models that can analyze data faster and generate more accurate results than other models. It belongs to the two-stage model group, which first generates a region of interest (ROI) before it analyzes the object’s features. These models were trained using the generated dataset. The dataset was fed to the program and learned according to its germination classification, as shown in the models’ training process flow in Figure 5 for the SSD MobileNet and Faster R-CNN models. The characteristics of the fed images were extracted and learned using a set of filters embedded in the models. The SSD MobileNet model generated proposals from the extracted features of the pole sitao seeds. After that, the proposals underwent a series of filtration layers to assign their classification. The Faster R-CNN model generated the region proposal and analyzed it using ROI pooling.

2.4. Training Metrics for the Models

Training a model is evaluated using metrics such as loss and learning rate. Training loss or logarithmic loss is defined as the performance of the model against the training dataset [19], which can be computed as follows:

$$L=\frac{-1}{N} \sum _{i=1}^N\sum _{j=1}^M{y}_{ij}\ast \log \left(p_{ij}\right)$$

(1)

where:

L is the training loss.

N is the number of samples.

M is the class.

y_ij is the sample i, which belongs to class j

p_ij is the probability of sample i belonging to class j.

In this metric, the model is evaluated in terms of how it fits the dataset. The value of this metric ranges from 1 to 0, with 0 being a perfect loss. However, it should be noted that training loss should not be zero to prevent the model from performing poorly in the evaluation dataset. A zero loss means that the training dataset fits perfectly into the model, where it cannot perform well in a new set of data in the evaluation stage. On the other hand, the learning rate is defined as the learning performance of a model in terms of step size in each iteration [20]. The step is used to define the number of iterations for the model to pass through the dataset. The learning rate can be computed using a weight updater as shown in the following equation:

W(n + 1) = W(n) + ŋ[d(n) − Y(n)]X(n)

(2)

where:

ŋ is the learning rate.

d is the desired output.

Y is the expected output.

X is the input.

W(n) is the current weight.

W(n + 1) is the updated weight.

2.5. Validation of the Trained Models

According to Banda et al. [21], a confusion matrix is used to evaluate these metrics. The matrix, as shown in Figure 6, collects the number of samples that fall in each category. True Positive (TP) is when the system classifies a positive object as positive. It also means that the system correctly classifies a positive object. On the other hand, False Positive (FP) means that the system classifies a positive object as negative. This means that the system incorrectly classifies a positive object. Moreover, True Negative (TN) is when the system classifies a negative object as negative, which also means that the system correctly classifies a negative object or an empty image. Lastly, False Negative (FN) means that the system classifies a positive object as negative, which also means that the system fails to detect and classify a positive object.

The trained models were validated using 450 samples or 23.08% of the total samples used in this study. The metrics used to evaluate the performance of the machine learning-based classifier systems were accuracy, precision, sensitivity, and F1. According to Mishra [19], accuracy is defined by the following equation:

$$Accuracy=\frac{TP+TN}{TP+FN+TN+FP}$$

(3)

where:

TP is the true positive sample.

TN is the true negative sample.

FN is the false negative sample.

FP is the false positive sample.

It measures performance by obtaining the ratio of the correct prediction to the total number of samples. On the other hand, precision is defined by the following equation:

$$Precision=\frac{TP}{TP+FP}$$

(4)

where:

TP is the true positive sample.

FP is the false positive sample.

It measures the performance of the system in classifying the detected objects in the image. It is defined as the ratio of correctly classified objects to the total number of detected images. Moreover, sensitivity is defined by the following equation:

$$Sensitivity=\frac{TP}{TP+FN}$$

(5)

where:

TP is the true positive sample.

FN is the false negative sample.

It is the ratio of correctly classified objects to the number of correctly classified objects and undetected objects within the image. Lastly, F1 is defined by the following equation:

$$F1=\frac{2\times Precision\times Sensitivity}{Precision+Sensitivity}$$

(6)

It measures the predictive accuracy of the classifier system by taking the harmonic mean of sensitivity and precision.

Models are evaluated using accuracy, which measures the capability of the model to predict objects correctly. Accuracy is used to evaluate the model’s performance, i.e., to determine whether it performed well or not based on a given dataset. According to Valentin [22] and Barkved [23], a model performs poorly if its accuracy is below 70%. In this case, other models that will perform better should be considered. Moreover, if the model’s accuracy is between 70 and 90%, then the model is good and compliant with the industry standard. However, if the model’s accuracy is greater than 90%, there is most likely an overfitting in the training or evaluation process. Moreover, a confidence score is used, along with the accuracy metric, which measures the confidence of the model to predict the object. It is a system-generated score that numerically evaluates the detected object. The value can range from 0 to 1, where 0.5 is set to be the standard threshold for accepting this score. A value of at least 0.5 will be accepted and anything below it will be rejected [24].

3. Results

3.1. Training Metrics for the SSD MobileNet and Faster R-CNN Models

The training loss metrics for the SSD MobileNet and Faster R-CNN models are shown in Figure 7 and Figure 8, respectively. It can be observed that the training loss had a decreasing trend, which means that the dataset performed well with the model in the training. The final value of loss for the SSD MobileNet model was 0.5408, while for the Faster R-CNN model, it was 0.5387. The final values of the training loss of both models were still high compared to the recommended near-zero values of training loss, as suggested by Mishra [19]. However, this is a normal scenario that happens when the model has already stopped learning.

For the learning rate metrics, it can be observed that both models stopped learning at 25,000 steps, after which the training process was terminated, as shown in Figure 9 and Figure 10 for the SSD MobileNet and Faster R-CNN models, respectively. The generated learning rate metrics have a normal trend, where it learns faster at the beginning of the training process, suddenly slows down, and then stops. This means that the model already learned the features of the fed training dataset.

The classifier system was successfully developed, and it classified seeds through bounding boxes. The image was loaded into the system and processed to classify the seeds on it. Some seeds were correctly and incorrectly classified by the system. Also, some seeds went undetected. Figure 11a shows the raw image fed to the classifier system, while Figure 11b shows the image after passing through the classifier system.

3.2. Confusion Matrix

The confusion matrix provided in

Table 1. Confusion matrix class data of the classified seeds on the 5th and 8th observation days.

Confusion Matrix	SSD MobileNet		Faster R-CNN
	5th Day	8th Day	5th Day	8th Day
TP	346	361	337	336
FP	72	51	75	68
TN	-	-	-	-
FN	32	38	38	46
Total	450	450	450	450

details the performance of two models, i.e., SSD MobileNet and Faster R-CNN, in classifying seed germination across two observation periods, i.e., the fifth and eighth days. On the fifth-day observations for the SSD MobileNet model, there were 346 TPs and 72 FPs. The eighth-day observations showed a slight improvement to 361 TPs and fewer FPs at 51. The Faster R-CNN model displayed 337 TPs and 75 FPs on the fifth-day observations. A marginal decrease to 336 TPs and 68 FPs was observed on the fifth-day observations. TNs were not applicable in this study; thus, no observations were recorded for this category. Overall, the models were tested with a total of 450 seeds each. The SSD MobileNet model showed an increase in correct classifications from the fifth to the eighth day, while the Faster R-CNN model showed consistent performance with a slight reduction in FPs.

4. Discussion

4.1. Class Data Shares

The models were evaluated based on the germination class. The summary of the data is shown in Figure 12 and Figure 13 for the SSD MobileNet model and in Figure 14 and Figure 15 for the Faster R-CNN model. It can be observed that TPs generated the highest percentage class for both the models and among the germination classes. On the other hand, FNs generated the least share of data for both models and germination classes, except for the ungerminated class, which generated higher shares than FPs.

4.2. Comparison of SSD MobileNet and Faster R-CNN Model Performances over Time

It was observed that the SSD MobileNet model performed well compared to the Faster R-CNN model for all the evaluation metrics based on average values, as shown in Figure 16. The performance of the models during the fifth- and eighth-day observations is shown in

Table 2. Performance of the models in the classifier system.

Metric	SSD MobileNet		Faster R-CNN
	5th Day	8th Day	5th Day	8th Day
Sensitivity	0.92	0.90	0.90	0.88
Precision	0.83	0.88	0.82	0.83
F1	0.87	0.89	0.86	0.85
Accuracy	0.77	0.80	0.75	0.75

. The SSD MobileNet model generated values of 0.92, 0.83, 0.87, and 0.77 for sensitivity, precision, F1, and accuracy, respectively, during the fifth day of observations. Moreover, during the eighth-day observations, it generated values of 0.90, 0.88, 0.89, and 0.80 for sensitivity, precision, F1, and accuracy, respectively. On the other hand, the Faster R-CNN model generated values of 0.90, 0.82, 0.86, and 0.75 for sensitivity, precision, F1, and accuracy, respectively, during the fifth-day observations. During the eighth-day observations, it generated values of 0.88, 0.83, 0.85, and 0.75 for sensitivity, precision, F1, and accuracy, respectively.

4.3. Statistical Validation of Model Performance across Germination Classes

Statistical analysis was performed to determine if the class accuracy means are significantly different from one another. This was performed to verify if the performance of the model in classifying the germination class was the same as other germination classes or not.

For the SSD MobileNet model, based on

Table 3. Statistical analysis for the SSD MobileNet model’s accuracy on the 5th day.

Source	Sum of Squares	df	Mean Squares	F
Class	7123.87	2	3561.94	7.68	Significant
Error	207,251.70	447	463.65
Total	214,375.57	449

and

Table 4. Statistical analysis for the SSD MobileNet model’s accuracy on the 8th day.

Source	Sum of Squares	df	Mean Squares	F
Class	10,874.41	2	5437.21	14.61	Significant
Error	166,304.33	447	372.05
Total	177,178.74	449

, the analysis gave F-test values of 7.68 and 14.61 for the fifth-day and eighth-day observations, respectively. On the other hand, for the Faster R-CNN model, based on

Table 5. Statistical analysis for the Faster R-CNN model’s accuracy on the 5th day.

Source	Sum of Squares	df	Mean Squares	F
Class	13,411.28	2	6705.64	13.00	Significant
Error	230,523.23	447	515.71
Total	243,934.51	449

and

Table 6. Statistical analysis for the Faster R-CNN model’s accuracy on the 8th day.

Source	Sum of Squares	df	Mean Squares	F
Class	21,570.32	2	10,785.16	20.26	Significant
Error	237,987.69	447	532.41
Total	259,558.02	449

, the statistical analysis generated F-test values of 13.00 and 20.26 for the fifth-day and eighth-day observations, respectively.

4.4. Performance Evaluation and Model Comparison in Seed Germination Classification

The performance of the models using the training dataset was evaluated based on the machine-classified testing dataset in terms of TPs, FNs, and FPs. It was observed that TPs generated the highest percentage class for both the models and among the germination classes. The higher share of TPs was an indication that the image segmentation and the models performed well in the classifier system. On the other hand, FNs generated the smallest share of data for both models and germination classes, except for the ungerminated class. This observation suggests that both models have difficulty in detecting ungerminated seeds. This implies that both models will most likely have a higher possibility of not detecting an ungerminated seed than classifying it incorrectly.

Overall, it can be observed that the SSD MobileNet model performed better than the Faster R-CNN model. For sensitivity, the better performance of the SSD MobileNet model can be attributed to the generation of many feature maps or proposals that have been enclosed that filtered characteristics of the seed. In this case, it was able to detect and classify a seed. For the Faster R-CNN model, proposals were generated; however, only one ROI was derived from these proposals and considered in the classification. That ROI mostly missed the characteristics of the seeds, resulting in undetected seeds. On the other hand, the better performance of the SSD MobileNet model in terms of the precision metric was due to its feature extraction learning method, in which the generated proposals were analyzed compared to one region of interest (ROI) analyzed by the Faster R-CNN model. In this case, it was able to filter the characteristics of the seed well, resulting in correctly classified seeds. Since the SSD MobileNet model performed better than the Faster R-CNN model in sensitivity and precision metrics, it resulted in better performance using the F1 metric. This is because this metric is a combination of precision and sensitivity metrics, and it is used to compute the harmonic mean to evaluate the predictive power of the classifier system. In this case, the SSD MobileNet model performed better for both precision and sensitivity compared to the Faster R-CNN model. Hence, it was expected to generate a higher F1 value.

In terms of accuracy, the SSD MobileNet model generated a higher value of 0.79 compared to the Faster R-CNN model, which had a value of 0.75. With this, it can be said that the SSD MobileNet model performed well compared to the Faster R-CNN model in classifying the seeds. However, the models’ performance obtained in this study was different from the performance obtained in the study by Zeren et al. [25]. In their study, the Faster R-CNN model performed better than the SSD MobileNet model in detecting an airport. The difference can be attributed to the type of the dataset, the number of training and testing samples, and the complexity of the samples used.

Lastly, after performing the statistical analysis, both models generated F-test values higher than their respective F-critical values. The SSD MobileNet model generated 7.68 for the fifth day and 14.61 for the eighth day. On the other hand, the Faster R-CNN model generated F-test values of 13.00 and 20.26 for the fifth-day and eighth-day observations, respectively.

5. Conclusions

This research concludes that the seed class significantly influences the accuracy of seed classification by the machine learning models. It was found that the SSD MobileNet model outperformed the Faster R-CNN model in classifying pole sitao seeds, with improved precision in segregating abnormal and ungerminated seeds on the fifth day and normal and ungerminated seeds on the eighth day. The analysis, corroborated by the statistical validation, underscores the SSD MobileNet model’s higher average accuracy of 0.79 compared to the Faster R-CNN’s accuracy of 0.75. These results confirm the potential of the SSD MobileNet model as a more reliable classifier in germination tests.