Machine Learning-Based Classification of Mushrooms Using a Smartphone Application

Abstract

Worldwide, a large number of cases of harmful mushroom exposure and consumption result in hallucinations, sickness, and death. One contributing factor is that certain poisonous mushrooms closely resemble their edible counterparts, making it difficult for general public collectors to distinguish one from the other. We propose a method to classify mushroom types from field-collection images using a smartphone application based on a convolutional neural network. The application helps people without proper mycology background or training to distinguish poisonous mushrooms from edible ones with which they may be confused. We showed three case studies to classify two-, three-, and five-class models by optimizing their training steps by cross-validation. An android app was developed by transferring the server-based trained model and allowing users to obtain probability scores for the correct genus classification. Our experiments showed that this method could provide sensitivity and specificity of two-, three-, and five-class mushroom models ranging from 89% to 100% using an image from the field with diverse backgrounds and objects.

1. Introduction

Mushrooms are a highly sought delicacy by both chefs and amateur mushroom cooks. However, morphological similarities between edible and poisonous species often result in the unintended consumption of harmful fungi with severe or even fatal consequences. For instance, Amanita is a genus of poisonous mushroom species. According to reports, a person consuming only five fried slices of Amanita can experience symptoms including central nervous system depression, ataxia, waxing, waning obtundation, religious hallucinations, and hyperkinetic behavior [1]. Other mushrooms can be also highly toxic. For example, in a reported case, a 53-year-old consumed Gyromitra and experienced vomiting and diarrhea, subsequently expressing hypotension, anuria, jaundice, hemiplegia, and coma, followed by death on the 3rd day [2]. In total, at least 133,700 cases (7428/year) of mushroom toxicity, with 52 fatalities, were reported in the U.S. from 1999 to 2016 [3].

Smartphones have been available at virtually every geographical location in both developed and developing countries. Recent versions of smartphones boast a multicore central processing unit (CPU), large random-access memory (RAM), cellular and WiFi connectivity, and high-resolution cameras utilizing complementary metal-oxide semiconductor (CMOS) sensors, which render them a hand-held research computer. Several groups have proposed transforming smartphones from conventional communication devices into portable instruments via various attachments that transduce a target analyte into an imaging format. Some applications include the colorimetric analysis [4,5,6,7], luminescence detection [8,9,10,11,12], spectrometry [13,14,15], and surface plasmon resonance analysis [16,17,18]. In addition, the computing power of the modern smartphone allows easier integration of machine learning algorithms to assist decision-making when visual observations cannot be used for accurate discrimination of the samples [16,17,18,19].

Here, we propose utilizing a conventional smartphone as a mushroom-image classifier that can provide reliable categorization of mushroom images and provide necessary safety information for both public and professional mushroom pickers. Multiple approaches to analyzing mushrooms using machine learning employing support vector machines (SVMs) [20,21], decision trees, naive Bayes, and ensemble algorithms based on mushroom data from the UCI Machine Learning Repository [22], which contains 22 morphological features, have been described. However, the result generated from this mushroom database assumed that morphological characteristics such as gill shape and cap shape had already been extracted objectively and that all the data were already in numerical formats as opposed to the raw images. If an application were developed utilizing the methods, users would still be required to manually extract the mushroom’s morphological attributes, which may be based on a subjective interpretation of the morphological traits. In addition, the ecosystem in which particular mushrooms are found is another critical feature mycologists utilize for identification. Regrettably, the morphological feature-based classifications might not incorporate this information into the classifiers. This paper proposes a method to classify mushrooms belonging to several easily confused genera. In order to achieve this task, we trained and tuned deep convolutional neural network models [23]. Subsequently, the models were integrated into an Android mobile application so that anyone could determine the edibility of mushrooms by photographing them in situ.

2. Materials and Methods

2.1. Mushroom Images

As the first crucial step in training a machine learning classifier, a vast number of photos were downloaded from the internet using the “down-load-all-images” Chrome extension [24]. Then, the photos with additional elements that could not be deleted, such as titles, text overlays, watermarks, or authorship annotations, were eliminated from the database. Images of mushrooms outside of a natural context were also removed from the database. Finally, an expert mycologist examined all the collected photographs to confirm their website-assigned labels and verify their genuine identities. The total number of validated photos for each model and class within each model is summarized in

Table 1. Total number of images used for machine learning training and testing.

		Total	Training	Testing
2-class model	Gyromitra	238	190	48
	Morchella	238	190	48
3-class model	Clavulina	497	369	128
	Inocybe	497	369	128
	Marasmius	497	369	128
5-class model	Agaricus	257	215	42
	Amanita	328	288	40
	Cantharellus	453	409	44
	Pleurotus	399	349	50
	Tricholoma	456	412	44

.

2.2. Machine Learning Algorithm

While recognizing and classifying all dangerous mushroom species is the objective, we began by constructing databases for three distinct scenarios. The first model was able to classify species of the morphologically similar genera of Gyromitra (poisonous species) and Morchella (highly sought edible species). The second model was able to classify genera of Clavulina, Inocybe, and Marasmius, and the third model was able to classify genera of Agaricus (some edible species), Amanita (includes deadly species), Cantharellus (prized edibles), Pleurotus (prized edibles), and Tricholoma (some toxic species). The first two models relied on the same neural network architecture (Figure 1), but the third model employed a different one. A convolutional neural network and a feed-forward neural network comprised the architecture of the first two models. For the convolutional neural network, input variables were 224 × 224-pixel images in three-channel RGB color and six convolutional layer blocks. Max-pooling was applied after 1st, 3rd, 4th, and 5th convolutional layers. A rectified linear unit (ReLU) activation function was used after the six convolutional layers [25]. The feed-forward network consisted of three fully connected layers; the number of features in the last layer was equivalent to the number of classes. After each fully connected layer, exponential linear unit (ELU) [26] activation was used with 70% dropout rates. This neural network architecture was unable to obtain satisfactory classification results for the 5-class model. Therefore, we decided to utilize the fine-tuned ResNet system [23] with 152 layers model and added an output layer with a number of neurons equivalent to the number of mushroom species classes. The hardware specification of the server was Intel^® Xeon^® CPU @ 2.30GHz, 34GB RAM, and Tesla P100-PCIE-16GB.

2.3. Classifier Training and Testing

RGB channels were normalized with means of 0.485, 0.456, 0.406, and standard deviations of 0.229, 0.224, and 0.225, respectively, as suggested in the literature [27]. Optimal hyperparameters such as the number of layers, the number of neurons, and the learning rate were calculated using this model. In addition, batch size was set to 32, and a cross-entropy loss [28] was used as a cost function for all models with an optimizer set to Adam [29] with a learning rate of 0.0001, and L2 penalty to 0.0001. Moreover, the AMSGrad variant was utilized as suggested by Reddi et al. [30] while the first beta parameters were modified from 0.9 to 0.7 with 20 epochs. Using our training data, stratified k-fold cross-validation (k = 10) was performed to determine how well our systems were prepared to classify designated mushroom types.

To test the performance of our models, 96, 384, and 220 images were used for each model. Based on correct and incorrect predictions, a cross-validation table was generated and expressed as true positive (TP), true negative (TN), false positive (FP), and false negative (FN) results; these values were used to report the statistical parameters and receiver operating curves (ROC) as follows.

$$\mathrm{Sensitivity}=\frac{TP}{TP+FN}$$

(1)

$$\mathrm{Specificity}=\frac{TN}{TN+FP}$$

(2)

$$\mathrm{Accuracy}=\frac{TP+TN}{TP+FP+TN+FN}$$

(3)

2.4. Mobile App Development and Testing

Android Studio version 4.0 with SDK version 28 was used for the app development platform. To use trained PyTorch models in Android, we used PyTorch Android version 1.5.0. Three of the server-based pre-trained models were ported into the smartphone app called Purdue University Mushroom App (PUMA); the flow chart of the app is shown in Figure 2. The output is shown with classification results and associated probability. For this classification task, a sigmoid function with α = 5 was used to compute the estimated probability of mushroom scores predicted by our trained model:

$$\mathrm{p}\left(x\right)=\frac{\alpha }{1+e^{-x}}\times 100\%$$

(4)

where x is a feature from the penultimate feed-forward layer.

Pre-trained models from servers were deployed to various Android devices, such as Samsung Galaxy S9 and Note 20, Google Nexus 5, and Pixel 2, with the same images used in the testing on our server-based model. Using these Android smartphones, test results were recorded as confusion matrices, and the performance results were calculated using Equations (1)–(3). To explore the effect of image size on the statistical output, we tested the classification results for intentionally scaled-down images. For this experiment, original images were scaled from 90% to 10% of the initial size, and the respective statistical accuracy was calculated.

3. Results

3.1. Model Training

For the two-class model, Figure S1A displays the server-based cross-validation training results for the genera Gyromitra and Morchella. Both cases showed around 98%, 98%, and 99% for sensitivity, specificity, and accuracy, respectively. Figure S1B shows the receiver operating curve (ROC) for the training results. For the three-class model, Figure S2A displays the server-based training result expressed as statistical outputs for the genera Clavulina, Inocybe, and Marasmius. All cases showed around 96%, 98%, and 97% for sensitivity, specificity, and accuracy, respectively. Figure S2B shows the ROC for the training results. For the five-class model, Figure 3A displays the server-based training result expressed as statistical outputs for the genera Agaricus, Amanita, Cantharellus, Pleurotus, and Tricholoma. Except for Tricholoma, all cases showed around 95%, 98%, and 97% for sensitivity, specificity, and accuracy, respectively. Tricholoma recorded 90% sensitivity, the lowest value among all the cases. Figure 3B shows the ROC for the training results. For the significance test, Table S1 shows the average accuracy and confidence interval, while Table S2 shows the significance test for two-, three-, and five-class models. Detailed statistical comparisons of the performance of trained models are provided in the Supplemental Materials (Tables S1 and S2, Figure S5).

3.2. Model Testing

After the training session had generated classification models, a batch of set-aside non-training images was used for testing. For the server-based model and server-based classification, and for the server-based model transferred to smartphones, the results were expressed in two distinct ways. Figure S3A,B shows the test results for the two-class model for the server and for a smartphone. Classification accuracy was more than 90% in both cases. Figure S4A,B shows the test results for the three-class model. In server-based testing, the specificity for Clavulina was the lowest among the three classes but still higher than 90%. In phone-based testing, the specificity for Marasmius was the lowest result among the three classes but higher than 90% as well. Both server-and phone-based models showed classification accuracy well above 90%. Figure 4A,B shows the test results for the five-class model. In server-based testing, Agaricus had the lowest sensitivity value (~82%) among all the classes, whereas Tricholoma recorded the lowest sensitivity (90%) in phone-based testing. All the cases except Agaricus from the server-based training showed above 90% classification accuracy.

3.3. Mobile App

Once launched, the app starts with the PUMA logo, as shown in Figure 5A. Users are then able to interact with the app using four buttons: Camera, Gallery, Database, and Mode. The Camera button allows the user to take a picture by launching a default camera application on the mobile device system. With this default camera app, the user can take as many pictures as desired and then return to the main menu (Figure 5B). The Gallery button enables the user to select a photo from a default gallery app to make a prediction. However, when PUMA is launched for the first time, the Gallery button is disabled and remains so before the users choose a type of classification model by clicking Mode. The app provides three classification models, as described in the Materials and Methods section, which can be selected via drop-down menus. After the user chooses the type of classification, the label and the accompanying class probability of the identified species are displayed. On the same screen, the user can put their own collection identification number (collection ID) on the result (Figure 5C). Lastly, the database allows the users to see the classification history. The users can check this history with a representative thumbnail of collected images, including a classification label from pre-trained models, a probability of being a classified mushroom species, a collection ID of each mushroom, and a timestamp when the user performs the prediction by clicking Gallery (Figure 5D). Moreover, we provide geographical information using Google Maps (Figure 5E). To accomplish this, the app collects the location’s latitude and longitude when photos are taken.

3.4. Inter-Platform Comparison

One of the questions that arise when using models trained on different platforms is the transferability of the result. Most wild mushroom collection occurs in the remote field, where cellular signal strength can be weak or nonexistent. Therefore, we compared the classification accuracy of server-based testing and smartphone-based testing results to determine the degree of consistency. Figure 6 shows the comparison of three reported performance measures (sensitivity, specificity, and accuracy). Variation was defined as a ratio of the server-based performance measure to the smartphone-based measures. Figure 6A shows a ±5% discrepancy between the two methods for the three-class model, while Figure 6B demonstrates a wider gap (maximum of +15% and a minimum of −10%) between the two platforms. For the three-class classification task, Clavulina showed the most stable testing results from the inter-platform comparison with the lowest variation rates of sensitivity, specificity, and accuracy, whereas Marasmius recorded the highest variation rates of the performance measures. Overall, inter-platform testing in the three-class classification showed stable variation results. The five-class classification showed less stability. Agaricus recorded the highest variation rates in sensitivity, about 118%. In addition, Tricholoma species recorded about 90% variation in specificity. Except for those two types, the other three mushrooms recorded stable variation rates. In particular, Pleurotus recorded the lowest variation rate among the five classes. Furthermore, while accuracy showed minimal variations, other performance measures revealed larger variations between the two platforms. For example, smartphone-based sensitivity results reported higher rates than server-based results and vice-versa for specificity.

3.5. Inter-Phone Comparison

In addition to the inter-platform comparison, an inter-phone comparison was conducted to determine whether there was a brand and/or operating system version dependency. While all of the tested smartphone brands utilized the Android operating system, some were more recent than others, including Google’s Nexus 5, Pixel 2, Samsung’s Galaxy S8, and Samsung’s Galaxy Note 20.

Table 2. Inter-phone testing results—2 classes.

		Pixel 2	Nexus 5	Galaxy s8	Galaxy Note 20
Gyromitra	sensitivity	0.9362	0.9167	0.9167	1
	specificity	0.9184	0.9167	0.9167	0.9231
	accuracy	0.9271	0.9167	0.9167	0.9583
Morchella	sensitivity	0.9184	0.9167	0.9167	0.9231
	specificity	0.9362	0.9167	0.9167	1
	accuracy	0.9271	0.9167	0.9167	0.9583

,

Table 3. Inter-phone testing results—3 classes.

		Pixel 2	Nexus 5	Galaxy s8	Galaxy Note 20
Clavulina	sensitivity	0.9921	0.9453	1	1
	specificity	0.9922	0.9922	0.9922	0.9922
	accuracy	0.9922	0.9766	0.9948	0.9948
Inocybe	sensitivity	0.9466	0.9609	0.9545	0.9618
	specificity	0.9843	0.9377	0.9921	0.9921
	accuracy	0.9714	0.9455	0.9792	0.9818
Marasmius	sensitivity	0.9606	0.8992	0.9762	0.9764
	specificity	0.9729	0.9727	0.9806	0.9844
	accuracy	0.9688	0.9481	0.9792	0.9818

and

Table 4. Inter-phone testing results—5 classes.

		Pixel 2	Nexus 5	Galaxy s8	Galaxy Note 20
Agaricus	sensitivity	0.9189	0.9231	0.9231	0.9231
	specificity	0.9669	0.9777	0.9777	0.9831
	accuracy	0.9587	0.9679	0.9679	0.9724
Amanita	sensitivity	0.925	0.925	0.925	0.9487
	specificity	0.9831	0.9831	0.9831	0.9831
	accuracy	0.9725	0.9725	0.9725	0.977
Cantharellus	sensitivity	0.875	0.8936	0.8936	0.8936
	specificity	0.9882	0.9883	0.9883	0.9882
	accuracy	0.9633	0.9679	0.9679	0.9677
Pleurotus	sensitivity	0.9792	0.9792	0.9792	0.9792
	specificity	0.9824	0.9824	0.9824	0.9822
	accuracy	0.9817	0.9817	0.9817	0.9816
Tricholoma	sensitivity	0.8444	0.8636	0.8636	0.8636
	specificity	0.9653	0.9655	0.9655	0.9653
	accuracy	0.9404	0.945	0.945	0.9447

show the results of Equations (1)–(3). While the two-class model showed above 90% accuracy for all brands, there were some performance variations between different smartphone brands. For example, sensitivity for Gyromitra ranged from 100% (Note 20) to 91.6% (S8 and Nexus 5), while specificity was in the range of 92.3%-91.8% on all four models. For the three-class model, sensitivity for Clavulina varied from 100% (S8 and Note 20) to 94.5% (Nexus 5), Inocybe resulted in 99.2% (Note 20) to 93.7% (Nexus 5), and Marasmius reported 97.6% (Note 20) to 89.9% (Nexus 5). For the five-class model, Agaricus showed a large variation in PPV, from 92.3% (Note 20) to 85% (Pixel 2). Amanita reported variations of specificity from 94.8% (Note 20) to 92.5% (the other three models). The rest of the three classes showed variations of about a 2% difference depending on the smartphone model.

3.6. Image-Scaling Effect

Another component of the inter-phone comparison was a test of the effect of image resolution on classification accuracy. Since many phones have their own CMOS sensors with different native pixel resolutions, all the training data were challenged against lower-resolution images from 100% (full scale) to 10% scale of input images. In addition, users can take many mushroom images at varying distances from the object, which results in different pixel resolutions for each mushroom image. Results were summarized in Figure 7A for two- and three-class models and Figure 7B for the five-class model. As the image was downsized from the original image, classification performance gradually deteriorated, which was reflected in the accuracy values (Equation (3)). Despite a 50–70 percent reduction in size, the accuracy of the reconstructed image was close to that of the original. However, the five-class model’s sensitivity yielded significantly lower results than specificity. For example, the sensitivity for Tricholoma genera was overall 10–20% lower than that of other counterparts.

4. Discussion

A large number of established machine learning algorithms were previously tested against the standard datasets from the UCI machine learning repository [22]. Among the available datasets exists one for mushrooms; this dataset could be used to train and test newly developed applications. One of the limitations of utilizing this database is that the image features used by machine learning have already been extracted. While this works fine in some experimental settings, it is not trivial to consistently extract those morphological features of mushrooms in the real-life mushroom foraging situation. Additionally, relying on the phenotypic mushroom characteristics alone would ignore informative descriptors indicative of the mushrooms’ growth environments, such as surrounding nature.

Additionally, various mushroom orientations (top view, front view, under the cap view), and different sizes (zooming) effects, into a single machine learning platform. Therefore, users could simply take images with their smartphones, and a pre-trained library (for remote locations with unreliable network access) or a server-based library could be used to provide accurate classification of the mushroom species.

Our training data accurately reflect the reality of mushroom foraging. The data includes various image perspectives (photographs from various angles, photographs under the cap, images displaying the front, top, and side, etc.), and the surrounding environment. In addition, as shown in Figure 2, the collection of mushroom images may involve taking a large number of consecutive photographs and sorting them by collection identification numbers. At the earlier stage of the app development, we added to the data input a collection identification feature so that all the images on the phone were classified with their unique collection numbers (Figure 5C). Since the offline classification was conducted on the spot, all the longitude and latitude information was also saved for easy tracking of the geographical location of each collected sample. We have previously discussed the advantages and disadvantages of server-based and mobile testing [4]. While the server-based method is more flexible and constantly refreshes the up-to-date models for better accuracy, smartphone-based testing conducts the classification without any network service present. Most wild-mushroom foraging happens in remote areas of the natural habitat lacking reliable network service. Therefore, offline, on-phone training was a critical feature to implement in this study.

However, training deep neural networks on a smartphone is so difficult that implementing this feature would be impractical. Consequently, transferring the pre-trained model from the server was the most logical course of action. The results were dependent on the classifier and the number of classes. While high levels of overall accuracy were relatively easy to reach, more granular metrics, such as sensitivity and specificity, showed some variation between the two platforms. Interestingly, the smartphone-based method demonstrated greater sensitivity, resulting in a lower false-negative rate. If our sample was a toxic mushroom, the smartphone would be less likely to misidentify it as an edible one.

Based on the results of inter-phone testing, our trained models generated consistent performance metrics regardless of the employed platform. We automatically resize captured images to the optimal image resolution for each of the devices to ensure that our mobile application and trained models produce the best results. The sheer diversity of mushroom genera and species is one limitation of our classification model. The current model provides proof-of-concept for classifying mushrooms at the genus level, whereas our future work will concentrate on refining classification models to discriminate among species within specific genera. The current supervised-learning method provides the pre-trained class with the closest matching label. While the probability calculation provided a measure of image similarity to the classes defined in database, the conventional discriminative classification is by definition unable to correctly process images from new and merging classes. Future work will involve the implementation of non-exhaustive learning techniques capable of automatically detecting the presence of new, never-before-seen classes in order to enable the detection and labeling of species not defined in the initial training database [31,32].