Data Augmentation Method for Plant Leaf Disease Recognition

Abstract

Recently, several plant pathogens have become more active due to temperature increases arising from climate change, which has caused damage to various crops. If climate change continues, it will likely be very difficult to maintain current crop production, and the problem of a shortage of expert manpower is also deepening. Fortunately, research on various early diagnosis systems based on deep learning is actively underway to solve these problems, but the problem of lack of diversity in some hard-to-collect disease samples remains. This imbalanced data increases the bias of machine learning models, causing overfitting problems. In this paper, we propose a data augmentation method based on an image-to-image translation model to solve the bias problem by supplementing these insufficient diseased leaf images. The proposed augmentation method performs translation between healthy and diseased leaf images and utilizes attention mechanisms to create images that reflect more evident disease textures. Through these improvements, we generated a more plausible diseased leaf image compared to existing methods and conducted an experiment to verify whether this data augmentation method could further improve the performance of a classification model for early diagnosis of plants. In the experiment, the PlantVillage dataset was used, and the extended dataset was built using the generated images and original images, and the performance of the classification models was evaluated through the test set.

1. Introduction

Recently, rapid climate change has had a great impact on the natural ecosystem and human life, and specifically on crop productivity [1]. In particular, an increase in temperature generally shortens the life cycles of plant pathogens while increasing their density, which also increases the possibility of generating mutants with strong pathogenicity [2]. In addition, it causes problems such as crop diseases and pests, which were rare in the past, or the expansion of the areas vulnerable to plant diseases that occur only in a limited region. Currently, the decline in crop yields due to plant diseases is estimated at around 16% worldwide [3], but if climate change continues, it may become very difficult to maintain current crop productivity. Therefore, the problem of crop productivity reduction due to climate change may intensify in the future, and there is also a growing lack of professional manpower to help solve it.

Deep learning has made a breakthrough in the field of computer vision in recent years, and there has been much research on how to use it for the early diagnosis of plant diseases [4,5,6,7,8,9]. Deep learning is suitable for diagnosing visual disease symptoms in the leaves and stems of plants because it is possible to extract and learn high-level features from images. Using this deep learning method, Mohanty et al. [10] conducted a plant image disease study based on a deep convolutional neural network model. The authors conducted an experiment to classify 26 disease images from 14 crop species using Google LeNet and AlexNet models and reported an average accuracy of 99.3%. Ferentinos et al. [11] also pointed out that models taken on datasets generated in controlled experimental environments achieved 99.5% accuracy when tested with models AlexNet, AlexNetOWTBn, GoogLeNet, OverFeat, and VGG, but did not perform the same in real-world environments. The accuracy dropped to 33% when using the learned model in the real world. Hassan et al. [12] conducted a study on image-based plant disease recognition using the Shallow VGG network. The Shallow VGG network is constructed using only a few layers from a pre-trained VGG network and serves to perform feature extraction from input data. The authors proposed to classify the features extracted through this through the Random Forest and Xgboost models. As a result of the experiment, it was reported that the two combined models proposed have better classification performance than the Vanilla VGG Network model. Lee et al. [13] performed image-based plant leaf disease research based on Recurrent Neural Network. The authors point out that CNN-based classification models do not clearly capture disease areas, and to solve this problem, they proposed a network combining Gated Recurrent Units (GRU) and Attention modules. They verified through experiments that the proposed method can capture clear disease regions unlike CNN and reported that classification performance was improved.

Although many vision-based plant disease recognition advanced studies using deep learning are in progress, some diseases are not common in nature, making it difficult to collect the same amount of data from healthy plant samples. Therefore, datasets for plant disease diagnosis collected from natural environments often lack samples of these rare disease classes, a problem known as imbalanced data in machine learning. When trained from such imbalanced data, most supervised learning models suffer from the overfitting problem, in which decision boundaries are biased toward major classes [14,15]. To solve this problem, previous work has adopted a manual augmentation method using professional manpower, but this is very inefficient and has high costs.

In this paper, we propose a data augmentation method based on image-to-image translation that can increase the sample diversity of diseased leaf datasets with insufficient numbers. The proposed augmentation method performs translation between healthy and diseased leaf images through the cycle-consistent generative adversarial networks (Cycle GAN) [16]. However, vanilla CycleGAN shows poor plant leaf image translation results, and we point out and improve on the following two problems: (1) poor reflection of the evident texture of the target disease, and (2) poor preservation of the shape of the input leaf image due to indiscriminate transformation in the background region. To solve this problem, we further utilize an Attention mechanism and a binary mask that explicitly indicates the location of the leaves. The attention mechanism is a method of selectively focusing on key information within given information and can greatly improve the expressive ability of the proposed plant leaf translation model. In deep learning, attention mechanisms originated in human perceptual systems as a method of selecting key information to focus on [17,18]. Many studies combining attention mechanisms have been conducted, and it has been demonstrated that the model can significantly improve the feature expression ability [17,18,19,20,21]. We used the Convolutional Block Attention Module (CBAM) [22] to construct a network that allows the generator to decide which features should be preferentially selected to deceive the discriminator. CBAM is a lightweight attention module that can learn attention maps corresponding to spaces and channels from a given feature map. We use CBAM to configure the generator network as a Pre-Activation Residual Attention Network and perform adaptive feature refinement during the training process. Also, we train a background loss function that regulates the generator to not perform unnecessary transformations on background regions. The background loss suppresses unnecessary transformations between the input data and the transformed image through a binary mask explicitly representing the background region of the input image. This assists the attention module in quickly focusing on the leaf itself, excluding the background where it cannot perform a transformation. Through these improvements, we generated a more plausible diseased leaf image compared to existing methods and conducted an experiment to verify whether this data augmentation method could further improve the performance of a classification model for early diagnosis of plants. We conducted experiments using apple, potato, and grape leaf data provided by the Plant Village [23] dataset and verified whether the proposed method can solve the problems of images produced by vanilla CycleGAN. We then compared the experimental results of various classification models to verify whether the extended dataset from the images generated by the proposed method can improve performance. Experimental results prove that the proposed augmentation method can solve the problem of lack of diversity in some disease samples that are difficult to collect by generating usefully translated images and contributing to improving classification performance.

The main contributions of this study are as follows:

• In this study, we focus on the overfitting problem caused by imbalanced data problems created by some disease images that are rarely observed in natural environments and propose a novel plant image augmentation method based on CycleGAN to solve this problem.
• We point out two issues that vanilla CycleGAN needs to address for image augmentation: (1) poor reflection of the evident texture of the target disease, and (2) poor preservation of the shape of the input leaf image due to indiscriminate transformation in the background region.
• To improve the poor image conversion results of CycleGAN, we utilized the attention mechanism and binary mask, and solved the bias problem of classification models by constructing an extended dataset.
• We have conducted extensive experiments to verify that the proposed method can improve the overfitting problem due to imbalanced dataset.

2. Related Studies

Recently, various data augmentation studies on the early diagnosis of plants using Generative Adversarial Networks (GAN) are being conducted. Most of them aim to correct problems with existing models to verify that more realistic images can be generated or to solve overfitting problems by replacing insufficient data.

Wu et al. [24] worked on recovering tomato disease data via DCGAN (Deep Convolutional GAN) [25]. Their contribution demonstrates that samples generated via DCGAN can improve the efficiency of classification models. However, it was confirmed that the generated samples are not clear and are not likely images in reality. Also, there is a problem of not being able to control the class of the image to be created.

Deng et al. [26] proposed an RHAC_GAN model that improves ACGAN (Auxiliary Classifier GAN) to solve the problem of tomato disease data augmentation. RHAC_GAN aims to solve the problem of lack of diversity among generated images by using hidden variables obtained from the discriminator along with inputs from the generator. Regarding the lack of diversity in the generated images, the authors explain that traditional ACGAN generators have difficulty learning different information within a class. In addition, a network was constructed by combining the Attention module and the Residual Block so that the generated images reflected obvious disease features. Plant leaf images generated by RHAC_GAN are sharper and more realistic compared to DCGAN and ACGAN.

Cap et al. [27] proposed a LeafGAN model that solved CycleGAN’s problem of changing unwanted content such as background when transforming plant leaf images. Their contribution is to propose an LFLSeg module that can segment leaf regions relatively simply and use it to control the transformation of background regions. The author proposed a background loss function to control the transformation of the background area by the generator and used only the input from which the background was removed as the input of the discriminator so that the discriminator for the background area did not react. The experiment was conducted based on actual data collected in the natural environment, and it was reported that classification performance was partially improved when learning was performed with a dataset augmented by the proposed model.

Xu et al. [28] proposed a Style-Consistent Image Translation (SCIT) model that aims at converting to the target domain while preserving information about the parts related to styles independent of labels in the translation process of plant images. To this end, we proposed a style loss calculated through a pre-trained vgg19 network. The authors verified through experiments that the amount of change to the target domain can be adjusted according to the change in the coefficient value of style loss.

3. Materials and Methods

3.1. Data Augmentation Method for Plant Leaf Disease Recognition

The plant data augmentation method proposed in this study was studied by borrowing the structure of CycleGAN, as shown in Figure 1, and the main suggestion is to improve the expression ability of the model using the Pre-Activation Residual Attention Block (PaRAB) and Binary Mask. These improvements allow the proposed model to generate visually plausible plant leaf data. The proposed method solves the bias problem of early diagnosis models of plants by contributing to increasing the diversity of samples by making diseased samples with insufficient numbers from normal plant leaves.

The entire training process of the proposed method and each loss function can be confirmed in Figure 1. The red line represents the elements needed to calculate each loss function, and the black line represents the elements used as inputs for each model. Basically, two generators $G:X\to Y$ and $F:Y\to X$ are utilized, where $X$ and $Y$ are the domains to be transformed, $X$ means a healthy leaf class, and $Y$ means a disease-affected leaf class to be transformed. Consequently, the two generators are converted to domains that are opposite to each other. In addition, the discriminators competing with the generators $D_Y$ and $D_X$ are required for training, where $D_Y$ distinguishes the fake leaf image $G\left(x\right)$ from the real leaf image $y\in Y$, and $D_X$ distinguishes the fake leaf image $F\left(y\right)$ from the real leaf image $x\in X$. $G$, $D_Y$, and $F$, $D_X$, which correspond to each other as a pair, proceed with adversarial learning and perform transformation between both domains.

In addition, we utilize the binary mask $M_x$ and $M_y$ required to regulate the generator from converting to unnecessary areas. The binary mask explicitly denotes leaf areas, $M_x$ points to healthy leaf areas, and $M_y$ points to disease-infected leaf areas. The leaf mask is obtained using a pre-trained Mask R-CNN [29] model, and for time efficiency, all mask data used for learning were extracted before training. the binary mask data is used by the generator to learn the background loss function and is also passed along as input to the discriminator as additional information about the leaf region. Since this method is used only as an input of a discriminator that is not actually used during the test process, it is reasonable because there is no need to have mask data when making a new sample by the generator during the test process. Additionally, giving additional information to discriminators is already well known to further enhance the model’s expressiveness in previous studies such as CGAN and ACGAN. This allows the generator to create new samples that preserve the shape of the input leaves well and assists the attention module within PaRAB to quickly focus on the leaf region, excluding backgrounds where transformation cannot be performed.

3.2. Pre-Activation Residual Attention Block

In the initial training stage, the generator of CycleGAN attempts to transform the entire image region evenly and indiscriminately and learns to deceive the discriminator. This also includes background areas that are unnecessary for transformation. Moreover, CycleGAN lacks an attention mechanism, so the generator cannot discern the features that should be considered more important in the translation between the two domains. Due to these problems, the generator requires an exceptionally long learning time to accurately reflect the texture of the target domain.

To solve this problem, we adopt CBAM and propose a Pre-activation Residual Attention Block (PaRAB), which allows adaptive feature refinement for the given feature responses by explicitly modeling the interdependence including the channel and spatial. As shown in Figure 2, PaRAB is configured by deploying an attention module before combining the output of the pre-activation residual and the low-level features delivered via skip connection. We construct a generator network through PaRAB and train the learnable parameters of the attention module using an adversarial loss function. This allows the generator to selectively emphasize features that are advantageous for deceiving the discriminator while suppressing less important ones. As a result, the generator can create an image that clearly shows the texture of the target domain.

As shown in Figure 3, the two sub-modules within the PaRAB are calculated sequentially, and the attention process takes the feature map $F\in {ℝ}^{C\times H\times W}$ as an input to generate a channel attention map $M_c\in {ℝ}^{C\times 1\times 1}$ and a spatial attention map $M_s\in {ℝ}^{1\times H\times W}$, which are used by multiplying them sequentially with each given input. The attention module is placed after the second instance normalization layer. These attention processes are as follows:

$$\begin{gathered} F^{\prime }=M_C\left(F\right)\otimes F \\ {F}^{″}=M_S\left(F^{\prime }\right)\otimes F^{\prime } \end{gathered}$$

(1)

The channel attention module integrates the information of a given input feature $F$ to generate an attention map $M_c$ that models the interdependence between channels. These calculations are performed through average pooling and max pooling operations, and they generate channel descriptions $F_{avg}^c\in {ℝ}^{C\times 1\times 1}$ and $F_{max}^c\in {ℝ}^{C\times 1\times 1}$. The calculated two-channel descriptions pass through the shared network with a bottleneck compressed by a reduction ratio $\mathrm{r}$, and then add up to finally obtain the channel attention map $M_c$ through the sigmoid function. These channel attention processes are as follows:

$$M_c\left(F\right)=\sigma \left(MLP\left(P_{avg}\left(F\right)\right)+MLP\left(P_{max}\left(F\right)\right)\right)$$

(2)

The spatial attention module integrates the information of a given input feature $F$ to generate an attention map $M_s$ that models the interdependence between spatial. These calculations are performed through average pooling and max pooling operations, and they generate spatial descriptions $F_{avg}^s\in {ℝ}^{1\times H\times W}$ and $F_{max}^s\in {ℝ}^{1\times H\times W}$. Then, by integrating the two spatial descriptions, the model finally obtains the spatial attention map $M_s$ using the sigmoid activation function after a 7 × 7 convolution. These spatial attention processes are as follows:

$$M_s\left(F\right)=\sigma \left(f^{7\times 7}\left([P_{avg}\left(F\right);P_{max}\left(F\right)\right]\right))$$

(3)

Since the two sub-modules proceed sequentially, the execution order can be changed arbitrarily. The authors report that, in general, using the channel attention module first showed better performance, and in this case, the input of the spatial attention module, used later, corresponds to $F^{\prime }\in {ℝ}^{C\times H\times W}$.

3.3. Network Archtecture

The generator network receives a real image as a single input, consisting of three down-sampling layers, an up-sampling layer, and six bottleneck layers. The down-sampling layer and the up-sampling layer used 3 × 3 convolutional filters, the stride value was 2 and the padding value was 1. The bottleneck layer was composed of the Pre-Activation Residual Attention Block introduced in Section 3.2. The entire network referred to the structure of Pre-activation ResNet [30], and Instance normalization (IN) [31] and Leaky ReLU were applied to all hidden layers.

Table 1. Related parameter and Output shape of Generator.

Layer	Activation	Normalization	Output Shape
Image	-	-	(256 × 256 × 3)
Conv 7 × 7	-	-	(256 × 256 × 64)
Down-sample	Leaky ReLU	Instance Normalization	(128 × 128 × 128)
Down-sample	Leaky ReLU	Instance Normalization	(64 × 64 × 256)
Down-sample	Leaky ReLU	Instance Normalization	(32 × 32 × 512)
PaRAB × 6	Leaky ReLU	Instance Normalization	(32 × 32 × 512)
Up-sample	Leaky ReLU	Instance Normalization	(64 × 64 × 256)
Up-sample	Leaky ReLU	Instance Normalization	(128 × 128 × 128)
Up-sample	Leaky ReLU	Instance Normalization	(256 × 256 × 64)
Conv 1 × 1	Linear	-	(256 × 256 × 3)

shows the shape of the input image and the detailed generator network configuration. Also, unlike using tanh activation in the generator’s output layer in general, using linear activation helped to obtain clearer results.

Since the discriminator receives the results of the channel-wise concatenation operation from the input image and the corresponding mask, unlike the generator, the input channel is 6. We utilized Spectral Normalization techniques to stabilize discriminator training. This method, proposed in SNGAN [32], is a simple and efficient weight normalization method, which prevents the gradient of the discriminator from exploding by making the Lipschitz constant bounded. This is very important because generator network learning relies entirely on feedback from discriminators. Therefore, spectral normalization (SN) was applied to all layers, and the activation function was Leaky ReLU. The down-sampling layer used a 4 × 4 filter, the stride value was 2 and the padding value was 1. For other layers, use the stride value of 1. The overall structure of the model follows the configuration of the discriminator in PatchGAN [33]. We have confirmed that the best results are not to use feature normalization techniques such as Instance Normalization and Batch Normalization.

Table 2. Related parameter and Output shape of Discriminator.

Layer	Activation	Normalization	Output Shape
Image $\oplus$ Mask	-	-	(256 × 256 × 6)
Down-sample	Leaky ReLU	Spectral Normalization	(128 × 128 × 64)
Down-sample	Leaky ReLU	Spectral Normalization	(64 × 64 × 128)
Down-sample	Leaky ReLU	Spectral Normalization	(32 × 32 × 256)
Conv 4 × 4	Leaky ReLU	Spectral Normalization	(31 × 31 × 512)
Conv 1 × 1	Linear	Spectral Normalization	(30 × 30 × 1)

shows the shape of the input image used and the detailed discriminator network configuration.

3.4. Loss Function

Image mapping functions $G$ and $F$ perform adversarial learning with discriminators $D_X$ and $D_Y$. The adversarial loss function used is the least squares loss function, and the objective function is as follows:

$$\begin{gathered} {\mathcal{L}}_{adv}\left(G,D_Y\right)={\mathbb{E}}_{y~p_{data\left(y\right)}}\left[{\left(D_Y\left(y\oplus m_y\right)-1\right)}^2\right]+{\mathbb{E}}_{x~p_{data\left(x\right)}}\left[(D_Y{\left(G\left(x\right)\oplus m_x\right)}^2\right] \\ {\mathcal{L}}_{adv}\left(F,D_X\right)={\mathbb{E}}_{y~p_{data\left(y\right)}}\left[{\left(D_X\left(x\oplus m_x\right)-1\right)}^2\right]+{\mathbb{E}}_{x~p_{data\left(x\right)}}\left[(D_X{\left(F\left(y\right)\oplus m_y\right)}^2\right] \end{gathered}$$

(4)

where ⊕ denotes a concatenation channel-wise operation that is calculated using a given real image and corresponding mask data. Since the test phase does not use the discriminator, the mask can serve as an additional input. We confirmed that the discriminator learns more stably when provided with such additional information. In addition, the cycle-consistency loss and identity loss used for training are as follows:

$${\mathcal{L}}_{cycle}={\mathbb{E}}_{x~p_{data\left(x\right)}}\left[{\left|F\left(G\left(x\right)\right)-x\right|}_1\right]+{\mathbb{E}}_{y~p_{data\left(y\right)}}\left[{\left|G\left(F\left(y\right)\right)-y\right|}_1\right]$$

(5)

$${\mathcal{L}}_{idt}={\mathbb{E}}_{y~p_{data\left(y\right)}}\left[{\left|G\left(y\right)-y\right|}_1\right]+{\mathbb{E}}_{x~p_{data\left(x\right)}}\left[{\left|F\left(x\right)-x\right|}_1\right]$$

(6)

The purpose of the cycle-consistency loss is to limit transformations that do not reflect the characteristics of the input data. Therefore, it minimizes the difference when restoring fake images $G\left(x\right)$ and $F\left(y\right)$, generated from input images to the original domain. The identity loss helps preserve the color and tint of each domain. The background loss helps to prevent the transformation of the background regions between the mapped image and the input image, and is as follows:

$${\mathcal{L}}_{bg}={\mathbb{E}}_{x~p_{data\left(x\right)}}\left[{\left|m_x^b\otimes \left(G\left(x\right)-x\right)\right|}_1\right]+{\mathbb{E}}_{y~p_{data\left(y\right)}}\left[{\left|m_y^b\otimes \left(F\left(y\right)-y\right)\right|}_1\right]$$

(7)

where $\otimes$ denotes an element-wise product operation calculated using a given real image and corresponding background mask data. Basically, the mask $m_x$ represents the leaf region, so $m_x^b=1-m_x$ is calculated and used for the background loss function.

Finally, the loss function for training the proposed model consists of a combination of the four loss functions described above, where λ is a coefficient for adjusting the balance of each loss, and the final loss function is as follows:

$$\mathcal{L}={\mathcal{L}}_{adv}+{\mathsf{\lambda }}_{cycle}{\mathcal{L}}_{cycle}+{\mathsf{\lambda }}_{idt}{\mathcal{L}}_{idt}+{\mathsf{\lambda }}_{bg}{\mathcal{L}}_{bg}$$

(8)

4. Results and Discussion

In this chapter, we conduct performance evaluation experiments to verify the validity of the proposed model. We verify the improved image quality with the proposed method, and then validate the classification performance with a dataset containing augmented plant leaf images.

4.1. Plant Village Dataset

The Plant Village dataset provides healthy and unhealthy leaf images for 14 crops, and there are various studies using it. An experiment was conducted using apple, potato, and grape leaf data among 14 crops, and the sample images are shown in Figure 4, Figure 5 and Figure 6. As shown in

Table 3. Configuring the Apple dataset used to train the proposed model.

Class	Train	Support	Ratio
Healthy	500	1645	30.4%
Black rot	500	621	80.5%
Scab	500	630	79.4%
Rust	275	275	100%
Total	1775	3171	55.9%

,

Table 4. Configuring the Grape dataset used to train the proposed model.

Class	Train	Support	Ratio
Healthy	423	423	100%
Black rot	500	1180	42.4%
Esca	500	1383	36.2%
Blight	500	1076	46.5%
Total	1923	4062	47.3%

and

Table 5. Configuring the Potato dataset used to train the proposed model.

Class	Train	Support	Ratio
Healthy	152	152	100%
Early Blight	500	1000	50%
Late Blight	500	1000	50%
Total	1152	2152	53.5%

, the dataset configuration for the experiments of the proposed generative models basically extracted and used 500 data for each class. The minor class, which lacked the number of data, was randomly extracted whenever the learning data pair was configured. Since the proposed model performs only conversions for two domains, an experiment was conducted by constructing a pair of health and disease classes individually to learn a single model.

As shown in

Table 6. Configuring the Apple dataset used to train classification models.

Class	Train/Aug	Test	Support
Healthy	300/200	200	1645
Black rot	300/200	200	621
Scab	300/200	200	630
Rust	75/425	200	275
Total	2000	800	3171

,

Table 7. Configuring the Grape dataset used to train classification models.

Class	Train/Aug	Test	Support
Healthy	223/227	200	423
Black rot	300/200	200	1180
Esca	300/200	200	1383
Blight	300/200	200	1076
Total	2000	800	4062

and

Table 8. Configuring the Potato dataset used to train classification models.

Class	Train/Aug	Test	Support
Healthy	52/348	100	152
Early Blight	300/100	100	1000
Late Blight	300/100	100	1000
Total	1200	300	2152

, a fixed number of samples were used for testing from the original data for the dataset composition for the experiment of classification models. This is to construct a correct test set because there are very few classes. Therefore, in the case of the apple and grape dataset, 200 samples were configured as a test set, and in the case of the potato dataset, 100 samples were configured as a test set. Afterward, the dataset composition for learning combines data not used for testing and augmented data and uses them for model learning. In the case of the apple and grape datasets, 500 samples were basically used for learning, and 400 samples were used for learning for the potato datasets.

4.2. Experiment Setup

The training parameters for learning the proposed model are shown in

Table 9. Parameters used in training.

Parameter	Value
Total Iteration	100,000
${\beta }_1$, ${\beta }_2$ (Adam)	0.5/0.999
Learning Rate for $G$, $D$	0.0002
${\mathsf{\lambda }}_{cycle}$	10
${\mathsf{\lambda }}_{idt}$	5
${\mathsf{\lambda }}_{bg}$	10
$r$ (Reduction ratio)	16

, and the results of the proposed model and the CycleGAN model are evaluated through FID (Frechet Inception Distance) [34]. In addition, through the image, it is verified whether the conversion problem for the background area and the clear disease texture are reflected.

An experiment is conducted to verify whether the plant leaf images generated from the proposed method can further improve the performance of a classification model for early diagnosis of plants. The models used to evaluate classification performance are ResNet-18 [35], DenseNet-161 [36], MobileNet-v2 [37], and EfficientNet-b0 [38], and the experiment was conducted through fine-tuning from the pre-learned weights of the ImageNet [39] dataset. Precision, recall, and F1 Score were used to evaluate the performance of classification models.

4.3. Results

In the result of the apple leaf in Figure 7, the black rot disease image generated through CycleGAN does not preserve the shape of the original leaf image well by performing the translation on some background areas. On the other hand, the results generated through the proposed method reflect more distinct disease characteristics for both Rust and Black rot classes. In addition, the generated results well preserve the shape of the original real leaf image, so the validity of the proposed method can be confirmed. Even in the grape leaf results in Figure 7, the diseased leaf images generated through CycleGAN do not preserve the shape of the input image in common and reflect the poor texture. The cause of these problems is that the generator performs training to deceive the discriminator by evenly and indiscriminately trying to transform the entire training image area. In contrast, the translation results of the proposed method show that explicitly regulating transformation only in the leaf region requiring transformation can create visually valid plant leaf data. The result of potato leaf transformation in Figure 7 also shows that the transformation through the proposed method produces much more stable and realistic results. The result of potato leaf transformation in Figure 7 also shows that the transformation through the proposed method produces much more stable and realistic results. In addition, the generator can select the features necessary for conversion and unnecessary features well by partially performing conversion only on the necessary areas in the original image and preserving the color of the original image in some areas.

Quantitative evaluation of images generated by GANs is a very difficult problem. Initially, to evaluate the quality of the generated image, it was verified whether it follows the distribution of the training image well through Average Log-likelihood, but this is an indicator that is far from visual validity. Moreover, the generated images are high-dimensional data, making direct comparison difficult. To solve this problem, the proposed Frechet Inception Distance (FID) score calculates the Wasserstein-2 distance between two distributions through the average vector and covariance matrix of the feature vectors of the real image and the generated image. Therefore, the lower the better, and 0 corresponds to the highest score.

Table 10. Comparison of generated Apple leaf image FID scores.

Class	CycleGAN	Our	Diff
Black rot	48.78	43.63	5.15
Scab	61.54	58.39	3.15
Rust	68.53	62.18	6.35

,

Table 11. Comparison of generated Grape leaf image FID scores.

Class	CycleGAN	Our	Diff
Black rot	39.36	25.83	13.53
Esca	44.59	38.63	5.960
Blight	59.29	52.98	6.310

and

Table 12. Comparison of generated Potato leaf image FID scores.

Class	CycleGAN	Our	Diff
Early Blight	68.10	58.70	9.4
Late Blight	70.37	63.30	7.07

are the results of evaluating the samples generated through these FID scores, and the samples generated through the proposed method as a whole are low. In addition, the Black rot class in grape leaves showed the largest difference at 13.53 compared to the previous results.

Table 13. Comparison of test results of Apple leaf classification models.

Model	Augmentation	Precision	Recall	F1 Score
ResNet 18	Imbalance	0.9768	0.9743	0.9755
	CycleGAN	0.9791	0.9762	0.9774
	Proposed	0.9793	0.9787	0.9790
DenseNet 161	Imbalance	0.9879	0.9859	0.9867
	CycleGAN	0.9912	0.9899	0.9905
	Proposed	0.9903	0.9926	0.9914
	Imbalance	0.9855	0.9852	0.9853
MobileNet v2	CycleGAN	0.9868	0.9845	0.9856
	Proposed	0.9882	0.9888	0.9885
	Imbalance	0.9969	0.9968	0.9968
EfficientNet b0	CycleGAN	0.9996	0.9985	0.9991
	Proposed	0.9998	0.9992	0.9995

shows the performance of the classification models according to each augmented technique for the apple leaf data. Overall, models trained with augmented data through the proposed method and CycleGAN show better results than models trained with imbalanced data. This indicates that augmented disease images can improve actual generalization performance. In the overall experiment, each model generally showed excellent F1 scores when trained over augmented datasets via the proposed method, and the best results using EfficientNet and our method achieved 0.9995 F1 scores.

Table 14. Comparison of test results of Grape leaf classification models.

Model	Augmentation	Precision	Recall	F1 Score
ResNet 18	Imbalance	0.9768	0.9743	0.9755
	CycleGAN	0.9791	0.9762	0.9774
	Proposed	0.9793	0.9787	0.9790
DenseNet 161	Imbalance	0.9879	0.9859	0.9867
	CycleGAN	0.9912	0.9899	0.9905
	Proposed	0.9903	0.9926	0.9914
	Imbalance	0.9855	0.9852	0.9853
MobileNet v2	CycleGAN	0.9868	0.9845	0.9856
	Proposed	0.9882	0.9888	0.9885
	Imbalance	0.9969	0.9968	0.9968
EfficientNet b0	CycleGAN	0.9996	0.9985	0.9991
	Proposed	0.9998	0.9992	0.9995

and

Table 15. Comparison of test results of Potato leaf classification models.

Model	Augmentation	Precision	Recall	F1 Score
ResNet 18	Imbalance	0.9768	0.9743	0.9755
	CycleGAN	0.9791	0.9762	0.9774
	Proposed	0.9793	0.9787	0.9790
DenseNet 161	Imbalance	0.9879	0.9859	0.9867
	CycleGAN	0.9912	0.9899	0.9905
	Proposed	0.9903	0.9926	0.9914
	Imbalance	0.9855	0.9852	0.9853
MobileNet v2	CycleGAN	0.9868	0.9845	0.9856
	Proposed	0.9882	0.9888	0.9885
	Imbalance	0.9969	0.9968	0.9968
EfficientNet b0	CycleGAN	0.9996	0.9985	0.9991
	Proposed	0.9998	0.9992	0.9995

show the performance of the classification models according to each augmentation technique for the grape and potato leaf data. Unlike the experiments in Table 13, the two datasets used for the experiments in Table 14 and Table 15 lack samples from health classes. Although the results of learning augmented datasets via CycleGAN in some classification models were excellent, the best were those utilizing EfficientNet and our method, which achieved F1 scores of 0.9994 and 0.999, respectively. We have experimentally confirmed that training a model on an augmented dataset usually solves the overfitting problem and improves overall performance. In addition, we confirmed that EfficientNet showed excellent performance even on imbalanced datasets and achieved the best results in all experiments when used with the proposed method in this paper.

Figure 8 shows the result of the two-dimensional visualization of feature vectors extracted from EfficientNet, which learned an augmented apple leaf dataset through the proposed augmentation method, through the t-distributed static neighbor embedding (t-SNE) technique. We find that the distribution of feature vectors extracted from the augmented 425 Rust samples and the real 75 Rust samples overlap, meaning that the generated image reflects the clear disease features of the real image class. Accordingly, the learned classification model may well extract features distinguished from other classes from the generated image.

5. Discussion

Data generation research through GAN is actively being conducted in various domains. However, it is very difficult to train GAN models well, and usually requires large amounts of data for successful results. In this study, we focused on improving the expressiveness of the model through a concentration mechanism, not a method to increase the amount of data. Through this method, by fixing the number of data required for learning and partially modifying the model, it was possible to generate a clearer image than the result of the existing CycleGAN. The advantage of the proposed method is that it preserves the shape of the input image very well. These results show very clear transformation results compared to the results of GAN-based plant leaf augmentation techniques proposed by Wu et al. [24] and the Conditional GAN-based plant leaf augmentation techniques proposed by Abbas et al. [40]. However, our proposed method has a limitation in that it can only be augmented by the number of healthy leaf images as a 1:1 mapping function. To address these issues, there is room for improvement by conducting research through additional models such as StyleGAN [41] and StarGANv2 [42]. Also, some diseases can significantly damage the shape of healthy input leaves. Since the proposed method regulates maintaining the original input shape, it does not reflect these disease shapes. Therefore, it is necessary to proceed with future research on the generative model that considers the morphological transformation of diseased leaves. In addition to the disease texture, future research can be conducted through the UGATIT [43] model when considering additional transformations related to morphology. The UGATIT model is a model that can perform translation considering the shape of the target domain.

6. Conclusions

Recently, weather changes caused by climate change have had a great impact on crop productivity. If climate change continues as it is now, it is expected to be very difficult to maintain current crop production. To address this, deep learning-based plant early diagnosis studies are actively underway, but some rare disease samples have the problem of increasing the bias of learning models because they are difficult to collect.

Therefore, in this study, we study a data augmentation technique based on the Cycle-consistent Generative Adversarial Network to increase the diversity of rare disease samples by solving the above problem. The main content of the study is to address the problem of imbalances in plant disease data by generating diseased leaf images to be converted from healthy plant leaf images that are commonly collectible. We also propose a Pre-activation Residual Attention Block to improve the representation ability of the model, which allows the generator to select features that should be considered more important during domain transformation. Furthermore, by learning the background loss function, the unnecessary transformation between the input leaf image and the fake leaf image generated from it was suppressed. Experiments were conducted to verify whether the plant leaf image generated from the proposed method could further improve the performance of the classification model for early diagnosis of plants, and Precision, Recall, and F1 Score were evaluated by indicators. The models used in the experiment were ResNet-18, DenseNet-161, MobileNet-v2, and EfficientNet-b0, all of which were fine-tuned using pre-trained weights. Experimental results confirmed that for all three plant leaves, the highest F1 score was achieved when the EfficientNet and the proposed data augmentation method were used together. In addition, improved FID scores and classification performance were achieved compared to CycleGAN as a whole. Through this, it was confirmed that the data augmentation method proposed in this study improves the generalization performance of various classification models.