Application of a Latent Diffusion Model to Plant Disease Detection by Generating Unseen Class Images

Abstract

Deep learning-based methods have proven to be effective for various purposes in the agricultural sector. However, these methods require large amounts of labelled data, which are difficult to prepare and preprocess. To overcome this problem, we propose the use of a latent diffusion model for plant disease detection by generating unseen class images. In this study, we used images of healthy and diseased grape leaves as training datasets and utilized the latent diffusion model, known for its superior performance in image generation, to generate images of diseased apple leaves that were not included in this dataset. Image-to-image generation was utilized to preserve the original healthy leaf features, which enabled the appropriate image generation of diseased apple leaves. To ascertain whether the generated diseased apple leaf images could be used to detect leaf diseases, a deep learning-based classification model was trained to discriminate between diseased and healthy apple leaves from a dataset with a mixture of actual and generated images. Results showed that leaves were accurately classified, indicating that diseased apple leaves not included in the training data could be used to identify the actual diseased apple leaves. Our approach opens up new avenues for improving plant disease detection methods.

1. Introduction

Since AlexNet [1], a convolutional neural network (CNN) architecture, achieved the highest score in image classification, deep learning methods have been developed and used in a variety of fields. IoT sensors are now being installed in agricultural fields, enabling the acquisition of large amounts of important data related to agriculture. Deep learning is being used to record such data and improve the quantity and quality of crops using image-based classification [2]. Numerous researchers have used deep learning techniques and methods to enhance and automate tasks [3]. For example, applications related to fruit detection and yield estimation have been reported [4,5,6,7,8]. In addition, crop disease outbreaks are a serious problem that threatens food supply. Identifying a disease correctly when it first appears is a crucial step for efficient disease management [9]. Therefore, attempts to apply deep learning to crop disease images to detect the diseases have been reported [10,11,12,13,14]. These applications have reportedly achieved high accuracy. However, deep-learning methods require a large amount of training data that are difficult to prepare and preprocess. For example, Mohanty et al. [9] used 41,112 from 54,306 images as training data and achieved 99.35% accuracy for 14 crops and 26 diseases using a convolutional neural network. Thus, approximately 1,000 images of training data per class were required to construct a deep learning system. Data augmentation to train the deep learning model with less data without overfitting the training data and zero-shot learning [15,16], to enable the model to recognize classes not included in the training data, are examples of crucial methods for mitigating this problem. In the application of deep learning approaches to agriculture-related studies, data augmentation methods have been studied using an image generation model called the Generative Adversarial Network (GAN) [17,18]. However, GANs are expensive to train, and the adversarial nature of the training generators and classifiers makes GAN learning unstable. Nevertheless, the use of image-generation models to prepare vast volumes of training data is considered to be promising because it may solve this difficulty.

In recent years, large-scale language models [19] have attracted attention in natural language processing based on the discovery of scaling laws. Latent Diffusion Models [20], which are multimodal models used for natural language processing, have garnered attention as image generation models. Latent diffusion models have been reported to outperform GANs in image synthesis and many other aspects. The use of generative models, which are an advanced form of deep learning approaches, in the field of agricultural informatics is promising. Studies on latent diffusion models have reported the generation of images with combinatorial features that are not available in the training data, such as signs labeled “Latent Diffusion” and “Picasso’s painting of zombies”. The use of latent diffusion models offers the prospect of building deep-learning systems that can process unseen class data. In disease detection using images in the agricultural field, it is necessary to prepare a sufficient number of images related to various diseases of various crops as needed, but it is sometimes difficult to prepare images quickly based on actual disease-infested individuals. If it is possible to generate images by combining images of plants and diseases that are not in the training data by extracting and integrating features of known plants and known diseases, as well as images generated by integrating features of two different training data sets, “Picasso painting style” and “zombie painting”, then when preparing images, it will be possible to construct a disease detection model that can respond quickly to various diseases, even for disease classes for which it is difficult to prepare images. The application of extracting and integrating features from existing data to generate new combinations of data is necessary to fulfill the challenges of using deep learning in agricultural applications. Training latent diffusion models is computationally expensive. However, a technique called Low-Rank Adaptation (LoRA) [21] additionally trains the output layer of a pre-trained model based on general data. The application of this method is expected to reduce the computational cost.

This study examines the feasibility of using latent diffusion models to build disease detection models for disease classes for which no training data exists. A latent diffusion model was used to generate diseased leaf images of an unseen class that was not included in the training data. Then, to ascertain whether the generated diseased leaf images can be used to detect leaf diseases, the generated and actual leaf images were inputted into a deep learning-based classification model that discriminates between diseased and healthy leaves. The classification performance results were used to determine the effectiveness of the generative model. It is important to understand the methods of application of latent diffusion models under optimum experimental conditions.

2. Materials and Methods

2.1. Materials

2.1.1. Image Generation Model

In this study, the Latent Diffusion Model (LDM) [20] is used for image generation. LDM is one of the approaches for high-quality image generation and is computationally efficient because it performs diffusion in the latent space. The basic structure of LDM consists of an encoder, diffusion model, and decoder. The encoder compresses high-dimensional input image data into a low-dimensional latent space. The diffusion model uses U-Net, which learns an inverse process of adding noise to the input image step by step to generate a desirable image from a noise image. Finally, the decoder restores the image to its original high-dimensional source. The diffusion process described above accepts input from text prompts that dictate the image generation conditions. The input text is converted to a latent vector through a pre-trained text encoder such as contrastive language–image pretraining (CLIP) [22], which is used to adjust the output probability so that the generated image is the intended condition. In this study, we used Stable Diffusion [23] and v1-5-pruned-emaonly.ckpt, one of the implemented versions of LDM available in open source. The ckpt file is the latest version of the published weights. The weights were trained using the open-source dataset LAION 5 B [24], consisting of 5.85 billion image-prompt pairs. Image-prompt pairs were associated using CLIP.

2.1.2. Dataset

We used the PlantVillage dataset in this study [25]. The PlantVillage dataset is an open-source dataset that contains 54,306 images of 56 classes of healthy and diseased leaves from 14 plant species. Each image in the dataset was annotated with labels indicating plant species and specific diseases. Each class consisted of approximately 1000 images, which was useful to create a baseline for our analyses.

2.2. Methods

2.2.1. Generation of Diseased Leaf Images

To verify that the generative model is capable of learning plant characteristics such as leaf shape and changes due to diseases, LoRAs were created for several plant types to verify the learning curve and output of the model at each stage of the process. These results were then used to examine the qualitative performance of the model in synthesizing images of the disease and plant species included in the model, and to set the conditions for subsequent experiments. Original prompts were used to provide output instructions using simple languages, and the model was thus trained to provide an output of the target image using the prompt as the instruction. Prompts were assigned to the corresponding plant type (apple or grape) and health status (healthy or black-rotted). The original prompts were presented as random strings to avoid conflicts with existing prompts that were pre-reserved to represent other features. Along with these prompts, weights were assigned to them. The weight allows the emphasis placed on a particular feature to be adjusted. Negative weights were also provided, which made it possible to weaken the significance of undesirable features. Prompts to express the Latent Diffusion image generation conditions are provided as vectors; therefore, multiple conditions can be provided by addition, and the output can be adjusted by weighting the condition vectors.

In this experiment, a black-rotted apple leaf was used as the unseen class. The black-rotted apple leaf images were generated based on features extracted from 30 images of healthy apple, healthy grape, and black-rotted grape leaf categories randomly selected from the PlantVillage Dataset. The model was optimized with 500 iterations using Adam. The learning rate for U-net was 0.0001, and the learning rate for the text encoder was 0.005. Values for the sampling steps and the classifier-free guidance (CFG) scale were 20 and 7, respectively, the image size was 256 × 256, and the denoising strength was 0.7. In the first attempt, the text-to-image translation was used for image generation. Latent diffusion for text-to-image translation involves translating textual descriptions into images by traversing a latent space where semantic content is represented compactly. Next, image-to-image translation was used to generate images by transferring the disease features to the images of healthy leaves. Image-to-image latent diffusion remodels existing images based on new textual inputs, preserving underlying structures while altering superficial details. During experiments, we observed the quality of the output images and determined the conditions for image generation.

2.2.2. Training a Classification Model for Unseen Classes

A classification model was used to determine if the generated black-rotted apple leaf image was of equal quality to that of the actual black-rotted apple leaf, and if the generated image could be a substitute for the actual image and used for detecting actually diseased leaves. An overview of the proposed method is shown in Figure 1. Although the latent diffusion model was trained using only real data, the classification model was trained using real healthy leaf images and black-rotted generated leaf images.

The classification model ResNet18 was implemented using PyTorch. The final output layer was replaced with a fully connected layer with two-dimensional output, and used as a binary classification model that discriminated between healthy and diseased leaves by comparing the values. First, the classification model was trained using 10 images of actual healthy apple leaves and ten generated images of black-rotted apple leaves. The parameters of the latent diffusion model used in the experiment were those of 10 iterations. We then added 10 generated healthy apple leaf images to the healthy leaf training class of the training data and repeated the experiment.

3. Results and Discussion

3.1. Generation of Diseased Leaf Images

Image generation was performed using independently trained LoRAs. An example of images generated using weights for case 1, specified in

Table 1. The weights of conditions added for each case.

Generate Condition	Weight
	Case 1	Case 2	Case 3
Apple	1	1	1
Grape	−1	1	−0.4
Black-rotted	1	1	1
Healthy	−1	1	−0.4

, is shown in Figure 2a. The entire image was darkened and unclear and did not form a leaf shape. Latent diffusion image generation conditions are given as vectors; thus, the output can be adjusted by weighting the condition vectors. We hypothesized that the negative weights did not function as intended, hence we ceased the negative weighting and generated images under different conditions. The weights for images generated for case 2, specified in Table 1, are shown in Figure 2b. While leaf-like shapes could be generated, the overall brightness and tint of the images were brighter than average for the training data available, and the fine details of the veins were also lost. We found it challenging to match the intended features with prompts on a one-to-one basis; therefore, we used image-to-image translation and added instructions by means of images to avoid excess deviation from the shape and color of the actual leaf. After providing image instructions, images were generated with weights for case 3, shown in Table 1. Negative weights were used again because the method of image generation was altered to image-to-image. Examples of images used for providing instructions and the corresponding generated images are shown in Figure 3. In order to preserve the characteristics of original images, only the denoising strength was altered to a smaller value of 0.47, while the other parameters were kept the same. This method generates an image with black-rotted disease symptoms while retaining the features of the leaf shape. Other examples of generated black-rotted leaf images are shown in Figure 4 and actual black-rotted leaf images are shown in Figure 5. We decided to use this method and parameters in the experiment.

3.2. Training a Classification Model for an Unseen Class

As described in the previous chapter, although the conditions for generating black-rotted apple images were determined, the image generation itself was not stable, and it was difficult to generate a large number of images that were qualitatively satisfactory. Therefore, 10 qualitatively satisfactory generated images were selected from the generated images obtained and evaluated using a classification model. The accuracy of the classification model trained using the test data for ten healthy apple leaves and ten black-rotted apple leaves was 0.463. An accuracy value below 0.5 is not considered a proper classification; however, it is considered simply as a random assignment of classes. We then plotted the precision recall to investigate the cause. This is illustrated in Figure 6a. Precision refers to the proportion of instances that the model predicts as positive (i.e., true positives), and recall indicates the proportion of true positives to the total number of positives. Comparison of precision and recall is useful for examining the characteristics of data asymmetries. Because precision and recall have a tradeoff relationship, the curve should be convex in shape to the upper right. From Figure 6a, in which both precision and recall are decreased, we consider that the classification model learned the features that unintentionally entered the generated images and was trained as a classifier to detect the generated images instead of those with diseases. Because the recall value was 0.1000, and 90% of the black-rotted apple leaf images were classified as healthy, we assumed that the disease data, which consisted of actual images, were classified into a class of healthy ones that were trained as actual image data.

To prevent images from being classified according to whether they were generated or actual images, the experiment was conducted again with ten generated healthy apple leaf images added to the training data of the healthy apple leaf class, which consisted of actual images, keeping other conditions the same. The precision–recall curve for the case in which a class with actual images was trained with the addition of generated images is shown in Figure 6b. The curve has a convex shape in the upper-right corner, indicating that it was modified to learn originally intended instructions, and the tendency of the classification model to learn the differences between actual and generated images was mitigated. The classification model was trained ten times under the same conditions, and the accuracy of the model at each epoch was verified. The mean and standard errors of accuracy are shown in Figure 7. The mean accuracy values increased over 10 epochs, and the standard errors were not large. In this trial, epoch 10 had the highest mean accuracy of 0.819 and standard error of 0.006.

In this study, the performance of the latent diffusion model in generating images of plants could not be directly evaluated quantitatively but was evaluated qualitatively by observing the generated images. There was also a problem in stably generating images that satisfied quality requirements by the latent diffusion model. More efficient and stable operation of the latent diffusion model and corresponding function models, as well as quantitative evaluation and optimization, are future issues. Moreover, the data mapped in the latent space were restored once in the real space and used again as an input for the classification model with a neural network. If an end-to-end model that performs this process in latent space without restoration is constructed, a significant reduction in computational time can be expected, which will facilitate detailed verification. Thus, it is desirable to develop a method to mechanically optimize the latent diffusion model for a defined purpose, and to optimize it simultaneously with the classification model.

4. Conclusions

Using healthy apples and black-rotted healthy grapes as training data, we varied the learning conditions and created a LoRA for the latent diffusion model. Although the image generation conditions were first provided as prompts on a one-to-one basis, text-to-image generation failed to produce sufficiently high-quality images. Thus, we used image-to-image generation and added instructions using images to avoid excess deviation from the shape and color of the actual leaf. The conditions for image generation were determined after observing the output. In the classification experiment, we trained the classification model using images of healthy apple leaves and generated those for black-rotted leaves. However, the learning curve indicated that the training did not function properly, and the classification model was trained to detect the generated image rather than that of the disease. We then used the generated black-rotted apple leaf images and added the generated healthy apple leaf images to the healthy apple leaf class to avoid learning of the differences between the actual and generated images. Consequently, a classification accuracy of 0.819 was achieved. The accuracy suggested the possibility of constructing a classification model for plant leaves of disease classes not included in the training data by training the classification model using images generated by the latent diffusion model. In practice, a moderate mixture of generated diseased leaf images not included in the training data and generated healthy leaf images included in the training data was found to be effective in detecting diseased leaves. The present study used black-rotted apple leaves as an example, but it is expected that further validation by increasing the number of diseases and plant types, as well as the number of generated images, will enable a classification model that can respond to more disease cases in the future.