On Using Deep Artificial Intelligence to Automatically Detect Apple Diseases from Leaf Images

Abstract

Plant diseases, if misidentified or ignored, can drastically reduce production levels and harvest quality. Technology in the form of artificial intelligence applications has the potential to facilitate and improve the disease identification process, which in turn will empower prompt control. More specifically, the work in this paper addressed the identification of three common apple leaf diseases—rust, scab, and black rot. Twelve deep transfer learning artificial intelligence models were customized, trained, and tested with the goal of categorizing leaf images into one of the aforementioned three diseases or a healthy state. A dataset of 3171 leaf images (621 black rot, 275 rust, 630 scab, and 1645 healthy) was used. Extensive performance evaluation revealed the excellent ability of the transfer learning models to achieve high values (i.e., >99%) for F1 score, precision, recall, specificity, and accuracy. Hence, it is possible to design smartphone applications that enable farmers with poor knowledge or limited access to professional care to easily identify suspected infected plants.

1. Introduction

Global circumstances in terms of climate change (e.g., drought, extreme weather conditions, wildfires, and unseasonable temperatures), ramifications of political unrest (e.g., reduction of fertilizer exports), and supply chain disruptions are straining food production. It is estimated that by the year 2050, the demand for food will increase by 70% as the global population reaches nearly 9.1 billion [1]. These conditions necessitate the optimal usage and conservation of existing resources. To this end, protecting plants from diseases is of paramount importance. It is one of the major factors that help reduce waste in agricultural production [2]. Although popular and easy, traditional methods of detecting diseases by naked eye observations are insufficient to determine the disease [3]. Generally, it takes time, effort, experience, and knowledge of plant diseases to diagnose these ailments. In addition, a large number of workers and material costs may be required in large farms [4,5].

Expert identification of plant diseases goes through several methods that relate to the shape of the leaf, the trunk of the tree, and the shape of the fruit [6]. In addition, possible diseases could manifest themselves through changes in the color of the leaf or fruit, and/or the appearance of spots [7]. Thus, leaf images typically carry significant clues into the sickness state of the tree. Hence, it is possible to use these images to determine the type of disease and treat the plant before the onset of irreversible damage or yield loss [8]. Artificial intelligence (AI) has come to play an increasingly diverse role in the development of smart practical applications of great benefit. More specifically, recent advances in deep learning have empowered complex image classification and analysis models [9,10]. Convolutional neural networks (CNNs) are a subclass of deep learning algorithms that is suitable to discover features in images without the need for explicit feature extraction or complex image processing algorithms [11].

Apple is one of the most productive fruits in the world [12], but it is greatly affected by fungi that appear clearly on its leaves. Several diseases affect apples, reduce their yield, or destroy entire fields [13]. Some of these include: black rot, apple scab, and rice apple rust [14]. Cedar-Apple rust is one of the common apple diseases. It is transmitted to apples from the surrounding cedar trees, and it can weaken the apple tree and reduce the production of fruits over time [15]. Hence, a large-scale infection can lead to adverse effects. One of the most used treatment methods is to cut the affected branch or leaves or to remove cedar trees that are within a radius of 1 to 3 miles. Apple scab disease is a seasonal infection that affects apples during the later stages of winter and the beginning of spring. It is caused by a fungus, which makes apparent changes to the fruits and leaves. In this case, crop loss may reach 70% [16]. Experts use pesticides or biological control treatments to stop its spread. Thus, early detection and control are highly beneficial. Black rot disease is another fungal infection that affects the stems, leaves, and fruits [17]. It weakens the infected tree and can spread to other trees, which causes a significant crop loss.

Several AI-based studies were conducted to detect and classify apple diseases, see

Table 1. A summary of the latest literature on the classification of apple diseases.

Study and Year	Objective	Dataset	Approach
Yu et al., 2022 [21]	Classification of five diseases + one healthy	11,397 leaf images	Modified res50 model.
Chakraborty et al., 2021 [23]	Classification of infected (two classes) and non-infected apple leaves	500 leaf images	Image segmentation and SVM.
Alqethami et al., 2022 [1]	Binary classification (disease vs. healthy)	240 leaf images	Segmentation, feature extraction, and machine learning.
Li et al., 2021 [19]	Classification of apple images into premium, middle, and poor grade	3600 fruit images	Custom CNN, Inception-v3, and SVM.
Fang et al., 2019 [13]	Seven-way classification of apple diseases	7056 leaf images	Fine tuned VGG16.
Geetharamani and Pandian [20]	39-way classification (38 diseases + 1 background)	54,305 leaf images (61,486 after augmentation)	Nine-layer CNN.
Dubey and Jalal, 2016 [24]	Four-way classification of apple diseases	320 fruit images	Defect segmentation using K-means, color and shape feature extraction, and classification using SVM.
Dubey and Jalal, 2012 [18]	Four-way classification of apple diseases	431 fruit images	Segmentation, feature extraction, and multi-class SVM.
This work	Four-way classification of apple diseases	3171 leaf images	Deep transfer learning (12 models).

. Some of these studies utilized fruit images while others rely upon leaf images as input. Dubey and Jalal [18] combined image processing and AI techniques to classify fruit images into four possible classes (i.e., blotch, rot, scab, and healthy) with a 93% accuracy. K-means clustering was used for image segmentation and four features (i.e., global color histogram, color coherence vector, local binary pattern, and complete local binary pattern) were extracted from the images after segmentation. The classification was performed using multiclass support vector machine (SVM). In a later study by the same authors [18], Zernike moments were combined with the set of previous features, which resulted in a 95.94% accuracy. Li et al. [19] developed a model based on a CNN with the purpose of identifying the quality of apples (i.e., premium, middle, or poor) using fruit images as input. They reported a 95.33% accuracy.

Geetharamani and Pandian [20] proposed a nine-layer CNN to classify leaf diseases into one of 38 disease categories that can infect 13 plant types. An accuracy of 96.46% was achieved. However, image augmentation was used in a fashion that may have led to data leaking as the total number of images was artificially increased. A more concrete approach would have discarded the originals and kept the augmented images. Fang et al. [13] developed an improved CNN model based on VGG16. Using leaf images of healthy apples as well as seven diseases (i.e., leaf blight, cedar rust, gray spot, leaf rust, black rot, and black spot), they achieved an accuracy range of 95.0% to 99.7%. However, their approach is based mainly on tuning model hyperparameters rather than major design modifications. Yu et al. [21] proposed a ResNet50-based model [22], which they called multiple-step optimization ResNet (MSO-ResNet). They reported an average precision, recall, and F1 score of 95.7%,95.8%, and 95.7%, respectively, in identifying five leaf disease states and one healthy. Chakraborty et al. [23] differentiated between infected (black rot and cedar apple rust) and non-infected apple leaves. Using leaf images as input, they reduced the noise using a Gaussian filter, and segmented infected regions of the leaves using histogram equalization and the Otsu thresholding algorithm. Ten features were extracted from the segmented images, which fed a multiclass support vector machine (SVM) classifier. Similarly, Alqethami et al. [1] proposed a new methodology with 3 prediction models (SVM, K-nearest neighbors (KNN) and CNN), The researchers categorized the leaves into four labels according to the type of disease as Black Rot, Cedar Apple Rust, Apple scab and Healthy, using features that pertain to color, shape, and texture.

In this work, the application of deep transfer learning in identifying apple diseases was investigated. Such an approach renders explicit feature extraction and complex image processing unnecessary. In addition, the robustness and power of well-known deep CNN models were harnessed toward the specific goal. The contributions of this paper are as follows:

• Implement a deep transfer learning-based diagnosis system using leaf images for three apple diseases—scab, black rot, and rust. In addition, a fourth, healthy class was part of the classification;
• Implement transfer learning of 12 deep convolutional neural networks models for the classification of leaf images into four classes. Such a system can aid farmers in quick disease identification and control;
• Compare the performance using multiple indices that reflect the true behavior of the algorithms. In addition, the training behavior and time requirements were also included.

The remainder of this paper is organized as follows: the dataset, deep learning models, and performance evaluation metrics and setup are explained in detail in Section 2, Section 3 presents the performance evaluation results along with comparison to the related literature and discussion of the models, and we conclude in Section 4.

2. Materials and Methods

Figure 1 shows an architecture diagram of the diagnosis system. The use of CNNs renders any feature extraction or segmentation (i.e., separation of relevant image parts) needless. This is because such steps are performed inherently by the intricacies of the models’ layers and mathematical operations. The next few subsections discuss each part of the system in detail.

2.1. Dataset

The dataset used in this study consists of 3171 images divided into four categories: 621 black rot, 275 rust, 630 scab, and 1645 healthy [20,25]. Each image is a photo of an individual leaf from one of the four classes, which was taken under unified background and light conditions. The images are publicly available in JPEG format and a resolution of 256 × 256 pixels. They were later resized to match the requirements of the specific deep learning model. No cropping or other preprocessing operations were conducted on the images. Samples of the images of the three diseases and healthy leaves are shown in Figure 2.

2.2. Deep Transfer Learning

Transfer learning, as the name suggests, allows the customization of a generically trained model to be used for a specific application. Such an approach has the advantage of utilizing well-established and well-designed models without repeating the model-building efforts [26]. Lower-numbered layers (i.e., closer to the input) typically learn common features (e.g., colors). On the other hand, layers closer to the output need to be refitted and retrained to serve the targeted output based on the application. This approach has been successfully used for many deep learning applications in the literature [9,10].

In this paper, 12 pre-trained convolutional neural network models were used separately to classify apple leaf images. These models were: DarkNet-53 (53 layers) [27], DenseNet-201 (201 layers) [28], EfficientNet-b0 (237 layers) [29], GoogLeNet (22 layers) [30], Inceptionv3 (48 layers) [31], Inception-ResNet (164 layers), MobileNetv2 (53 layers), ResNet-18 (18 layers), ResNet-101 (101 layers) [22], ShuffleNet (50 layers) [32], SqueezeNet (18 layers) [33], and Xception (71 layers) [34]. The models have common features in the type of operations being conducted on the input (e.g., convolutions and pooling). However, they differ in their depth (i.e., number of layers), width (i.e., input size and intermediate fields), and network structure. Moreover, their computational requirements may be different based on the type of innovation introduced in the model to reduce the number of required mathematical operations (e.g., residual networks). All of the models were used based on the same hyperparameters (e.g., number of epochs) and were pre-trained using the ImageNet database of one million images [35].

The CNN design literature has witnessed great efforts in improving the performance and efficiency of these networks. The modifications range from optimizing certain parameters to reforming the network architecture with new processing elements, connection types, block designs, and depths [36]. GoogLeNet is a type of small CNN that considers the spatial locality of image pixels at various granularity levels. Moreover, it was the first to introduce the inception block concept. ShuffleNet reduced the number of required operations and improved the efficiency of GoogLeNet by introducing the shuffle process. The ResNet, Incpetionv3, and Inception-ResNet models are depth-based CNNs, which rely on a larger number of network layers in an effort to efficiently capture the input features and characteristics. The ResNet family of models introduced residual learning and used bypass pathways (i.e., skip connections) in a data-independent and parameter-free manner. The Inceptionv3 and Inception-ResNet models used small asymmetric filers to improve the computational cost. The DenseNet-201 is a multipath design that aims to solve the vanishing gradient problem. This was accomplished using dense connectivity (i.e., feed-forward cross-layer connections). The Xception model is a width-based multi connection CNN that emphasized the importance of the width of the architecture through parallel processing within the network layers. SqueezeNet uses feature maps and suppresses the less important features using the SE-block, which performs two operations: squeeze (i.e., suppress) and excitation. The EfficientNet-b0 is the baseline EfficientNet model with further versions reaching level b7. The MobileNetv2 model, as the name suggests, was intended for resource-limited mobile devices. This was accomplished by reducing the number of trainable parameters using the depth-wise separable convolution operations.

2.3. Experimental Setup

The training parameters were unified for all models and set as follows: The minimum batch size, which is the number of input instances to be processed at the same time, was set to 16. This allows faster training, but at the expense of more memory requirements. The maximum number of epochs was set to 10, which was enough for the learning process to reach maximum accuracy and the loss curve to flatten without unnecessary extra training. The learning rate was set to $3\times {10}^{-4}$. The solver optimization algorithm for network training was the stochastic gradient descent with momentum (SGDM), which is commonly used for training due to its fast convergence [37].

In order to check the models’ ability to learn with more data and their overfitting/underfitting behavior, the data were split for training/testing using two methods 50/50 and 70/30. After that, input images were augmented to improve the training process by introducing variety to the dataset [38]. This was performed by randomly shifting the images along the x-axis and y-axis using the range [−30, 30], and random scaling in the range [0.9, 1.1]. Moreover, the images were randomly reflected along the x-axis.

The models were implemented and evaluated using MATLAB R2021a software running on an HP OMEN 30L desktop GT13 with 64 GB RAM, NVIDIA^® GeForce RTX^TM 3080 GPU, Intel^® Core^TM i7-10700K CPU @ 3.80 GHz, and 1 TB SSD.

2.4. Performance Evaluation

Several metrics were used to evaluate the performance of the models, see Equations (1)–(5), where $T_P$ is the true positive, $F_N$ is the false negative, $F_P$ is the false positive, and $T_N$ is the true negative. The ability to identify a diseased leaf image in the positive correct class is measured by the true positive rate (i.e., $Recall$ or sensitivity). Moreover, the true negative rate (i.e., $Specificity$) measures the number of correctly classified normal leaf images as such. A highly sensitive model will recognize a large proportion of positive cases, but this may have been accomplished at the cost of a large number of false positives. $Precision$ provides more insight into the results by measuring the percentage of true positive images (i.e., image of leaves with a disease) as a percentage of all reported positive images, including false ones. $Accuracy$ measures the ratio of correctly identified positive and negative cases to the total number of images. In the case of imbalanced datasets with big differences in the number of images per class, the F1 score provides a good indicator of accuracy [39]. In addition, the training and validation time was reported.

$$Accuracy=\frac{T_P+T_N}{P+N}$$

(1)

$$Precision=\frac{T_P}{T_P+F_P}$$

(2)

$$Recall=\frac{T_P}{T_P+F_N}$$

(3)

$$Specificity=\frac{T_N}{T_N+F_P}$$

(4)

$$F1=2\times \frac{T_P}{2\times T_P+F_P+F_N}$$

(5)

where $T_P$: The number of correctly classified images. $F_P$: The number of wrongly classified images. $F_N$: The number of images missed by the classifier. $T_N$: The number of correctly classified images as the negative class. P: The number of all images considered as the positive class. N: The number of all images other than the positive class.

3. Results and Discussion

The experiments aimed at evaluating the effectiveness of the CNN models in identifying the correct health category of apple leaves. The performance evaluation metrics were recorded for all models along with the required training and validation times.

Starting with a data split of 50/50, the mean overall F1 score, precision, recall, specificity, and accuracy for each deep learning model are shown in

Table 2. The mean overall F1 score, Precision, Recall, Specificity, and Accuracy for each deep learning model and 50/50 data split.

Model	F1 Score	Precision	Recall	Specificity	Accuracy
SqueezeNet	98.2%	99.0%	97.6%	99.0%	98.1%
GoogLeNet	98.2%	98.3%	98.1%	99.4%	98.4%
Inceptionv3	99.9%	99.9%	99.9%	99.9%	99.9%
DenseNet-201	99.9%	100.0%	99.9%	100.0%	99.9%
MobileNetv2	99.6%	99.7%	99.6%	99.8%	99.6%
Resnet101	99.1%	99.2%	99.0%	99.7%	99.2%
Resnet18	99.4%	99.6%	99.3%	99.7%	99.4%
Xception	98.8%	99.0%	98.6%	99.6%	99.0%
Inception-ResNet-v2	98.1%	98.9%	97.4%	99.1%	98.2%
ShuffleNet	99.7%	99.8%	99.7%	99.8%	99.7%
DarkNet-53	99.8%	99.9%	99.8%	99.9%	99.8%
EfficientNet-b0	99.3%	99.5%	99.2%	99.8%	99.5%

. All models achieved exceptionally high F1 score in addition to the other metrics. The Incpetionv3 achieved the highest value (F1 score = 99.9%). The lowest-performing model was InceptinResNet-v2 with an F1 score of 98.1% and a recall value of 97.4%. However, such performance is extremely excellent. This performance is confirmed and further details are revealed by the confusion matrices in Figure 3. They show that in most cases, different classes were classified perfectly. However, some images from the scab class caused the most error by being misclassified as healthy in several models.

Although the reported performance is very high, deep learning models require a large number of images in order to realize their full potential. The ramifications of an increased size of the training dataset were evaluated by setting the training/validation data split as 70/30. The mean overall F1 score, precision, recall, specificity, and accuracy for all models and 70/30 data split are shown in

Table 3. The mean overall F1 score, Precision, Recall, Specificity, and Accuracy for each deep learning model and 70/30 data split.

Model	F1 Score	Precision	Recall	Specificity	Accuracy
SqueezeNet	98.4%	98.8%	98.1%	99.3%	98.5%
GoogLeNet	98.1%	98.8%	97.6%	99.1%	98.1%
Inceptionv3	100.0%	100.0%	100.0%	100.0%	100.0%
DenseNet-201	99.7%	99.8%	99.6%	99.9%	99.8%
MobileNetv2	100.0%	100.0%	100.0%	100.0%	100.0%
Resnet101	99.5%	99.5%	99.5%	99.8%	99.6%
Resnet18	99.8%	99.8%	99.8%	99.9%	99.8%
Xception	99.4%	99.7%	99.2%	99.7%	99.5%
Inception-ResNet-v2	96.5%	97.9%	95.3%	98.4%	96.7%
ShuffleNet	99.9%	99.9%	99.9%	99.9%	99.9%
DarkNet-53	99.7%	99.8%	99.6%	99.8%	99.7%
EfficientNet-b0	99.3%	99.6%	99.0%	99.7%	99.4%

. Nearly all models achieved near-perfect specificity (i.e., true negative rate). Moreover, most models improved on their smaller training dataset performance (i.e., 50/50 split) with Inceptionv3 and MobileNetv2 achieving perfect results (i.e., all images were correctly classified). However, the Incpetion-ResNet-v2 model seems to have suffered from overfitting of the training data and actually produced a slightly worse performance compared with that of the 50/50 data split. The corresponding confusion matrices are shown in Figure 4. The leaf images showing the scab disease caused the most errors in all non-perfect models.

Machine learning models are typically susceptible to overfitting or underfitting. Further insight into the training and validation behavior of the deep learning models is necessary to expose any deficiencies. A sample of the training and validation progress and the cross entropy loss during training for the Darknet53 model using a 50/50 data split is shown in Figure 5. The figure shows a stable learning process and a steady loss decline for both training and validation. Moreover, Figure 6 and Figure 7 show the training and validation progress curve using 70/30 data split for the SqueezeNet and MobileNetv2 models, respectively. Similarly, both figures show no overfitting or underfitting visible. In addition, the learning process seems stable learning process with decreasing loss.

The mean training and validation times for all models and data split strategies are shown in

Table 4. The mean training and validation times for all algorithms and data split strategies. All times are in seconds.

Data Split	50/50	70/30
Model
SqueezeNet	118.9	130.6
GoogLeNet	260.1	293.7
Inceptionv3	571.4	703.1
DenseNet-201	1891.1	2298.2
MobileNetv2	824	1091.2
Resnet101	587	866
Resnet18	164.6	238.12
Xception	2249.4	4240
Inception-ResNet-v2	1784.9	3814.0
ShuffleNet	637.2	1657.0
DarkNet-53	439.6	1543.6
EfficientNet-b0	1347.2	3846.8

. Two observations can be drawn from the table. First, although the classification performance of the models is comparably close, there is a big speed gap between the fastest and slowest models (i.e., 118.9 s vs. 2249.4 s using 50/50 split, and 130.6 s vs. 4240 s using 70/30 split). This makes the faster models more favorable for smartphone deployment. Second, some models scaled better than others with more training data (e.g., SqueezeNet vs. ShuffleNet).

The application of artificial intelligence in the agricultural sector is receiving increasing attention in the literature.

Table 5. A comparison to the latest results in classification of apple diseases from leaf images.

Study and Year	Performance
Yu et al., 2022 [21]	Average precision, recall, and F1 score of 95.7%, 95.8%, and 95.7%, respectively.
Chakraborty et al., 2021 [23]	Accuracy 96%, Recall 96%, Precision 96%, F1 score 96%.
Alqethami et al., 2022 [1]	Accuracy 82.25% (SVM), 70.3% (KNN), and 98.5% (GoogleNet).
Li et al., 2021 [19]	Best accuracy 95.33% (custom CNN).
Fang et al., 2019 [13]	Accuracy 95–97%.
Geetharamani and Pandian [20]	Accuracy 96.46%.
Dubey and Jalal, 2016 [24]	Accuracy 95.94%.
Dubey and Jalal, 2012 [18]	Accuracy 93%.
This work	(>99%) for all measures.

shows a comparison of the latest results in classification of apple diseases. Li et al. [19] and the two studies by Dubey and Jalal [18,24] relied upon fruit images as input. Such an approach hinders large-scale automated measurements as it may require 3D images from all sides of the fruit to truly capture the disease state of the fruit. A single 2D image may capture one side of the fruit but miss the infected side. Such a problem is not present when dealing with leaf images. The other studies included in the table use leaf images as input. Chakraborty et al. [23] used image segmentation to extract features of the disease, which were fed to a support vector machine classifier to achieve a 96% F1 score. A similar approach was used by Alqethami et al. [1], but failed to achieve better performance. The other studies either designed their own CNN or modified existing CNN models without transfer learning. Yu et al. [21] modified a res50 model to achieve an F1 score of 95.7%. Whereas, Fang et al. [13] fine-tuned a VGG16 model to achieve an accuracy range of 95–97%. Geetharamani and Pandian [20] built their own nine-layer model and achieved an accuracy of 96.46%. The work in this paper is different in that it does not rely upon explicit image processing algorithms (e.g., segmentation), nor on the quality of a proposed feature. In addition, new CNN designs may need extensive evaluation to establish their quality. Moreover, some of their results used augmented data that replicated, not replaced, the original data, which may skew their results. Using deep transfer learning has the credibility of established models while avoiding any overhead in terms of image processing and feature extraction. Moreover, superior performance in comparison to the literature was achieved over all metrics.

The present study has some limitations. First, the dataset may need to include other possible apple diseases. Moreover, other plant diseases may be combined in the dataset. However, we do not see great value in terms of field deployment, by complicating the classification problem. For example, although it is necessary to include other plant diseases (e.g., corn rust), jumbling all the diseases in one model will hurt the classification performance and speed of the model. A user (i.e., farmer) should be able to choose the correct plant type when checking for a disease. Second, more diverse leaf images in terms of the picture background may be desirable to simulate actual deployment scenarios. Third, it is highly worthwhile to embed the model and other future models in smartphone applications that can be made available on smartphone application stores. This will allow the system to grow, improve, and identify any shortcomings.

4. Conclusions

Climate change due to global warming from green gasses and the resulting conditions (e.g., drought, wildfire, and extreme weather conditions, and unseasonable temperatures) necessitates optimal use of available agricultural resources and the implementation of conservation strategies. Moreover, geopolitical unrest and its ramifications on food prices, availability of fertilizers, and supply chain problems exacerbate the shortages in available food. Thus, protecting plants from disease is essential to mitigate existing circumstances and reduce waste in agricultural production. To this end, the work in this paper investigated the application of deep learning artificial intelligence in the identification of apple diseases from leaf images.

Recent advances in deep convolutional neural networks and transfer learning were applied to the problem of apple disease identification. Such an approach has the potential to achieve high classification accuracy with little overhead to the specialists and great benefit to the farmers. Moreover, it does not require specialist knowledge, elaborate preprocessing of images, nor does it incur errors from image processing techniques. Future work will focus on improving the system by applying incremental learning approaches to evolve the application during deployment. Moreover, we will consider 3D images for disease identification from fruit images. In addition, the development of ensemble deep learning models for plant disease classification is another avenue for research. Regarding deployment, the extra mile of wrapping the deep learning models in smartphone applications should be a worthwhile effort.