An Improved Approach to Detection of Rice Leaf Disease with GAN-Based Data Augmentation Pipeline

Abstract

The lack of large balanced datasets in the agricultural field is a glaring problem for researchers and developers to design and train optimal deep learning models. This paper shows that using synthetic data augmentation outperforms the standard methods on object detection models and can be crucially important when datasets are few or imbalanced. The purpose of this study was to synthesize rice leaf disease data using a Style-Generative Adversarial Network Adaptive Discriminator Augmentation (SG2-ADA) and the variance of the Laplacian filter to improve the performance of Faster-Region-Based Convolutional Neural Network (faster-RCNN) and Single Shot Detector (SSD) in detecting the major diseases affecting rice. We collected a few unbalanced raw samples of rice leaf diseases images grouped into four diseases namely; bacterial blight (BB), tungro (TG), brown-spot (BS), and rice-blast (RB) with 1584, 1308, 1440, and 1600 images, respectively. We then train StyleGAN2-ADA for 250 epochs whilst using the variance of the Laplacian filter to discard blurry and poorly generated images. The synthesized images were used for augmenting faster-RCNN and SSD models in detecting rice leaf diseases. The StyleGAN2-ADA model achieved a Fréchet Inception Distance (FID) score of 26.67, Kernel Inception Distance (KID) score of 0.08, Precision of 0.49, and Recall of 0.14. In addition, we attained a mean average precision (mAP) of 0.93 and 0.91 for faster-RCNN and SSD, respectively. The learning curves of loss over 250 epochs are 0.03 and 0.04 for Faster-RCNN and SSD, respectively. In comparison to the standard data augmentation, we achieved a t-test p-value of $9.1\times {10}^{-4}$ and $8.3\times {10}^{-5}$. Hence, the proposed data augmentation pipeline to improve faster-RCNN and SSD models in detecting rice leaf diseases is significant. Our data augmentation approach is helpful to researchers and developers that are faced with the problem of fewer imbalanced datasets and can also be adopted by other fields faced with the same problems.

1. Introduction

Rice is an important cereal grain that has largely contributed to global food security over the last half-century [1,2]. The recent climate change, rapid population growth, rampant degradation of the ecosystem, pests, and rice diseases are a threat to global food security [3]. Nonetheless, rice leaf diseases have one of the most devastating effects on rice production; this is so because 37% of global rice production is lost through diseases during growth and harvest periods [4]. However, the devastating impact of the disease can be reduced or prevented if correctly detected; this is true because rice leaf disease detection provides a visual indication of the need for precise treatment before they spread further. The absence of a precise disease detection system poses a serious threat to agriculture. As such, accurate detection of rice leaf disease is crucially requisite for high yield and food security [5,6]. Whilst the existing methods for rice leaf disease detection require agricultural experts to visually examine and diagnose the disease which is prone to human error, time-consuming, labor-intensive, low supply, costly, and could be more challenging in large fields where visual detection affects the spatial estimation of disease spread for control. The recent advancement in computer vision (CV) has made it possible to detect plant leaf disease using machine learning (ML) models.

Over the decade, there have been numerous studies using various CV algorithms to detect rice leaf diseases, such as image processing, support vector machine (SVM), image classification, object detection, and pattern recognition with various degrees of success [7,8,9,10,11,12,13,14]. Joshi & Jadhav [15] proposed a system for detecting rice leaf diseases using a Neighbor and Minimum Distance classifier (k-Nearest) while Jiang et al. [10] used an SVM and deep learning (DL) to detect rice leaf disease. A more advanced approach is the use of object recognition and detection algorithms. Kiratiratanapruk et al. [8] used Faster-Region-Based Convolutional Neural Network (faster-RCNN), RetinaNet, You Only Look Once (YOLOv3), and Mask-Region-Based Convolutional Neural Network (Mask-RCNN) detection models to detect rice leaf diseases using 6330 training images with an accuracy of 75.92%, 70.96%, and 36.11%, respectively. In addition, Chen et al. [14] used a transfer learning (TL) approach to detect and classify various rice leaf diseases using DenseNet and Inception models with 500 images. However, these existing methods could be optimized to achieve better efficacy for detecting crop disease. It is quite evident that most researchers use quite a few datasets for training deep neural networks, which as a result affects the model generalization and detects poorly in real scenarios.

Moreover, DL models require considerable data to perform at an optimum rate, otherwise, they run into an overfitting problem and generalize poorly [16]. In addition, labeling a considerably large amount of plant leaf disease data is tedious and time-consuming. The absence of a diverse large dataset is a significant setback for developing a model that is fit to detect rice leaf disease. Previous studies have used standard augmentation methods to create random new samples to increase the size of the image dataset [17], while others used an adaptive method in generating new samples [18]. Moreover, these methods artificially expand the same training datasets by modifying their positions and colors without introducing new “unique” data to the model. Hence, the model has already seen these data but in a different state of position and color which does not impact much on the model generalization [19]. To enhance the diversity of the dataset and also to solve the problem of a small dataset, new data can be created using a Generative Adversarial Network (GANs). GAN is a generative ML framework that uses two neural networks, a generator, and a discriminator [20]. The generator generates a high-dimensional perceptual object from a latent space while the discriminator solves a classification task by distinguishing real objects from the input dataset from fake ones generated by the generator. Intuitively, the training sets an adversarial game between two players, then proceeds in three steps of each round and iterates as many rounds as required. Recently, the StyleGAN2 Adaptive Discriminator Augmentation (SG2-ADA) architecture performs better than most of GAN’s methods and can generate high-quality objects with fewer training datasets and low computational cost by implementing ADA and TL, respectively [21]. Furthermore, the class imbalance issue is prevalent in plant leaf disease data [22] i.e., the sample data of one pathology may be far more widespread than other sample data which often results in biased models and there is a lack of studies that utilize SG2-ADA to generate a quality new set of data for rice leaf disease detection system.

In this study, we propose a pipeline for synthetic augmentation of rice leaf disease dataset using SG2-ADA to increase the efficacy of deep neural networks in detecting rice leaf disease, we adopted the standard faster-RCNN and the Single Short Detector (SSD) as our deep convolutional neural networks. While GAN augmentation takes more time and resources compared to the standard augmentation techniques, our work shows that the GAN technique can improve the efficacy of the rice leaf disease detection model better than the standard approach, which is central in high-stake agricultural decision-making. Furthermore, there have been several studies conducted to examine the efficacy of GAN-based data augmentation, albeit these studies are few in contrast to the effectiveness of the GAN approach over the standard augmentation approach. We aim to fill the research gap by exploring the problem of small datasets and class disease imbalances by evaluating the performance of standard and GAN-based data augmentation methods. Hence, to the best of our knowledge, this is the first work that uses SG2-ADA to synthetically augment the rice leaf disease dataset in order to enhance the performance of rice leaf disease detection models.

2. Relative Works and Challenges

The objective of this section is to understand the research progress and developmental trends of generative models and some key issues while also illustrating the various DL models that have adopted the GANs to generate augmented images and the significance of the augmentation process.

Since the introduction of GAN architecture by Goodfellow et al. [23], many related architectures have subsequently been proposed with some impressive results. While Deep Convolutional Generative Adversarial Network (DCGAN) and Conditional Generative Adversarial Network (cGAN) could be referred to as extensions [24]. The advanced GANs are Wasserstein Generative Adversarial Network (WGAN), Big Generative Adversarial Network (BigGAN), Cycle-Consistent Generative Adversarial Network (CycleGAN), Style-Based Generative Adversarial Network (SG), and SG2-ADA [25]. The DCGAN extends the foundational GAN by implementing Convolutional Neural Network (CNN) settings which provides stability to the training process but bring about an issue of mode collapse whereby the model produces a single type or small set of output [26]. In addition, the DCGAN’s have a vanishing gradient, i.e., when the generator fails to learn due to information starvation which results in a poor generator and a robust discriminator [27]. WGAN resolved the issues of mode collapse, vanishing gradient, and stability during training but prolonged the training session and, at times, produced poor output [28]. The BigGAN offers more stability during training and has better results than WGAN but requires more extensive data samples and time [29,30]. SG architecture provides an advanced solution by improving the traditional GAN architectural model and training process by redesigning the generator normalization, modifying progressive growth, and regularizing the generator [31]. They provided an advanced method for high-resolution image synthesis, but sometimes they generate uneven parts inside an image and require a considerable amount of quality datasets [32]. SG2 traces the unnatural blob-like artifact caused by SG by stabilizing the high-resolution training; moreover, using small datasets leads to discriminator overfitting [33]. The SG2-ADA produces excellent results with a small dataset while achieving a better result [34].

In addition, SG2-ADA has been adopted in augmenting training datasets in various fields of DL applications, such as generating images for facial recognition, medical image synthesis for a brain tumor, generating synthetic datasets for liver lesion classification, generating images for manufacturing components, and generating datasets for credit card fraudulent transaction predictions [35,36,37,38,39]. Recently, studies have been aimed at automating plant leaf disease detection based on DL methods with various degrees of success using different model architectures and pre-trained models, such Fast-RCNN, Faster-RCNN, Region-based Convolutional Neural Networks (R-CNN), Histogram of Oriented Gradients (HOG), YOLO, Region-based Fully Convolutional Network (R-FCN), Single Shot Detector (SSD) and Spatial Pyramid Pooling (SPP-net) [40]. However, a common problem in the field of plant leaf disease detection is the lack of sufficient and diverse training datasets, which often leads to an undesirable effect on the model’s performance while making the model run into an overfitting problem because of the way DL models learn, i.e., DL model requires a considerable amount of data to perform well [41].

The standard augmentation methods artificially expand the same training datasets by modifying their positions and colors to create more data while the GANs create a new set of diverse data. Hence, the former method does not impact much on the model generalization because it is not introducing new datasets to the object detection model but instead modifies their positions and colors [19,42], this point proves to be significant in improving the accuracy and generalization of DL models. Lastly,

Table 1. Limitations and advantages of the standard method.

Limitations of Standard Data Augmentations	Advantages of Standard Augmentation
Standard data augmentation still inherits biases from the original datasets.	It reduces the cost of collecting or generating and labeling new datasets.
Finding an effective optimal standard data augmentation approach can be challenging.	It reduces data scarcity and overfitting whilst improving model accuracy.
Quality assurance for standard data augmentation is costly.	Helps with resolving class imbalance and creates variability in data models.

[43] shows the major limitations and advantages of the standard augmentation method.

3. Methodology and Tools

This section discusses the research scheme, methods, GAN, SG2-ADA, advantages, and its performances.

3.1. Research Scheme

The approach proposes a semi-supervised data augmentation pipeline that leverages generative adversarial networks’ ability to generate high-quality synthetic images by utilizing the SG2-ADA model to generate synthetic images. Sequentially, the best-quality synthetic images are filtered using some image processing techniques and combined with the original raw data. This in turn amplifies the dataset and is used to solve the issue with the class imbalance and limited dataset. Subsequently, the combined annotated dataset is fed into a CNN network for training an object detection model and the results are evaluated using mean average precision metrics. Lastly, we compared the resulting analysis of the GAN-based augmentation with the standard augmentation technique.

3.2. Research Method

The proposed GAN-based augmentation pipeline trains an SG2-ADA model using a limited dataset to generate high-quality synthetic images. The goal here is to reproduce a statistical pattern and property of the original dataset through SG2-ADA by modeling its probability distribution over the data space. After training the generative model until it reaches convergence. The GAN model generates new data with the same characteristics as the original data using a truncation factor of ($\psi$ = 0.25,$\psi$ = 0.50, $\psi =0.75$, and ψ = 1.0); noting that the increase in truncation helps to further diversify the generated images by shrinking the sampling region of the latent space. Part of the generated images come with some defects, such as blurriness and noise distortion which could significantly degrade the performance of the CNN models when applied [44]. As such, we applied an image processing technique to detect the amount of blurriness in the generated images, then filter and discard some of the blurry and other poorly generated images by using the variance of the Laplacian method (see Equations (1)–(5)). The method is implemented by passing a single channel of an image and convolving it with a $3\times 3$ kernel while taking the variance of the response. If the variance is $\le 100$ then the image is considered blurry; otherwise, the image is not blurry. (See Figure 1 and Figure 2).

Laplacian can be given as:

$${\nabla }^{2}f=\frac{{\partial }^{2}f}{{\partial }^{2}x}+\frac{{\partial }^{2}f}{{\partial }^{2}f}$$

(1)

Partial derivative with respect to x:

$$\frac{{\partial }^{2}f}{{\partial }^{2}x}=f\left(x+1,y\right)+f\left(x-1,y\right)-2f\left(x,y\right)$$

(2)

Partial derivative with respect to y:

$$\frac{{\partial }^{2}f}{{\partial }^{2}y}=f\left(x,y+1\right)+f\left(x,y-1\right)-2f\left(x,y\right)$$

(3)

Combined the partial derivatives of x, y.

$${\nabla }^{2}f=\left[f\left(x+1,y\right)+f\left(x-1,y\right)+f\left(x,y+1\right)+f\left(x,y-1\right)\right]-4f\left(x,y\right)$$

(4)

Then forms a 3 × 3 filters for $\left(x,y\right)$ coordinates.

$$\begin{array}{ccc}x-1,y-1& x,y-1& x+1,y-1\\ x-1,y& x,y& x,y+1\\ x-1,y+1& x,y+1& x+1,y+1\end{array}\to \begin{array}{ccc}0& 1& 0\\ 1& -4& 1\\ 0& 1& 0\end{array}$$

(5)

Then, the filtered generated images are combined with the existing raw images. This can be represented mathematically. (See Equation (6)) shows a union of existing and generated synthetic data.

$$A\cup B=\left\{x:x\in Aorx\in B\right\}$$

(6)

where

• $A=\left\{x,existingrawdata\right\}$
• $B=\{x,generatedsyntheticdata\}$

Lastly, the combined balanced data is passed forward to Faster-RCNN or SSD model for training using the proposed augmentation pipeline, while also applying the standard augmentation technique on faster-RCNN and SSD model, (see Figure 3). Hence, the results of the two methods are compared based on the average precision and loss metrics.

The following Algorithm 1 shows a step-by-step implementation of the proposed augmentation pipeline; our method follows a simple optimization framework that involves generating synthetic images to solve the problem of a small and imbalanced dataset. We start off by declaring the inputs; $A_i\in {ℝ}^{n\times m\times 3}$ is original images, $B_i\in {ℝ}^{n\times m\times 3}$ is synthetic images generated by a pre-trained SG2-ADA generator $G\left(A_i\in {ℝ}^{n\times m\times 3}\right)$, $lf$ is the Laplacian filter, $D_{total}$, and $C_{aug}$ is a set of an empty array. The next step follows $functionAUG\left(B_i\right)$, which detects and discards blur images by measuring the variance $\sigma {\left(B_{i+1}\right)}^2$. Lastly, the highest quality synthetic images are annotated and labeled accordingly and used to augment the dataset to solve the problem of small, imbalanced datasets whilst improving the efficacy for detecting rice leaf disease in deep learning models.

Algorithm 1 Step-by-step implementation of the proposed augmentation pipeline

Start   $A_i\in {ℝ}^{n\times m\times 3}rawinputimages$ $B_i{ℝ}^{n\times m\times 3}\mathrm{synthetic}\mathrm{images}\mathrm{from}\mathrm{pretrained}\mathrm{generator}G\left(A_i\right)withSG2-ADA$ $lfisLaplacianfilter\leftarrow \begin{array}{ccc}0& 1& 0\\ 1& -4& 1\\ 0& 1& 0\end{array}$ $D_{total}\leftarrow \varnothing initializeemptyset$ $\mathbf{input}\mathbf{:}{A}_i=\left\{a_{1,}{a}_2{a}_{3,}\dots ,a_n\right\},B_i=\left\{b_{1,}{b}_2{b}_{3,}\dots ,b_n\right\},G(.),lf$ function$AUG\left(B_i\right)$   $C_{aug}\leftarrow \varnothing initializeemptyset$    for $B_i\in B\mathit{d}\mathit{o}\mathbf{=}\mathbf{>}\mathrm{iterate}\mathrm{over}\mathrm{all}\mathrm{the}\mathrm{synthetic}\mathrm{images}$     $B_{i+1}=convolution\left(B_i,lf\right)$     $\mathit{i}\mathit{f}\sigma {\left(B_{i+1}\right)}^2<100\mathit{t}\mathit{h}\mathit{e}\mathit{n}$     $discard\leftarrow \mathit{T}\mathit{r}\mathit{u}\mathit{e}\mathbf{=}\mathbf{>}\mathrm{Blurry}\mathrm{images}$    if not discarded then     $C_{aug}\leftarrow C_{aug}\cup B_i\mathbf{=}\mathbf{>}\mathrm{Good}\mathrm{quality}\mathrm{synthetic}\mathrm{images}$    end for    $\mathit{r}\mathit{e}\mathit{t}\mathit{u}\mathit{r}\mathit{n}{C}_{aug}$    $D_{total}\leftarrow A_i\cup AUG\left(B_i\right)$    $D_{train},D_{test},D_{eval}\leftarrow splitandlabel{D}_{total}$    $CNNModel\left(D_{train},D_{test},D_{eval}\right)$    $score\leftarrow evaluateCNNModel$    End

3.3. GAN

GAN is a generative model capable of creating high dimensional perceptual plausible objects similar in characteristics to its training data [45]. It employs two neural networks, a generator and a discriminator pitting them against each other to create new objects. The generator generates a high-dimensional perceptual object, while the discriminator distinguishes real images from the datasets or fake items created by the generator. As part of an adversarial network, training is set up as an adversarial game between two players, and training proceeds in three steps for each round and iterates as many rounds as possible [46]. The phases are explained below while Figure 4 shows the graphical illustration of the whole process.

First-phase: the generator pulls some relatively small number of dimensions from a simple distribution, which is a latent noise vector, such as a multi-dimensional uniform distribution from $0\mathrm{to}1\mathrm{or}1\mathrm{to}-1,$ which is used as an input to the generator that produces a sample; then, the sample is taken which is passed to the discriminator. And so the network back propagates from the loss through the output of the discriminator, but because this is a generator training step, the discriminator weight values are held and fixed (no updates). Then, the network computes the derivatives with respect to the inputs. Weights are applied to the generator such that it improves image generation. Figure 4 below illustrates the process.

Intuitively, noise $\left(z\right)$ is passed to a Generator neural network $\left(G\right)$ that provides an output of $G\left(z\right)$ with a cost function (see Equation (7)).

$$\frac{1}{m}{\displaystyle \sum }_{i=1}^m\log \left(1-D\left(G\left(z^i\right)\right)\right)$$

(7)

Second-phase: the generator creates some number of noise vectors from the latent space, then they are fed into the generator to produce some number of synthetic samples. The samples are then passed to the discriminator which distinguishes real from fake with the same notion of classification loss except that instead of trying to fool the discriminator the loss is expected to be high i.e., the model is doing a bad job if it cannot tell real from fake. The stochastic gradient descent is applied and the neural network is back-propagated to the discriminator with the weights being updated in the direction of better able to distinguish real from fake and then it stops without updating the generator Figure 5 below illustrates the process.

Intuitively, the discriminator is a classifier that trains to classify fake input when $\left(P=0\right)$ with a denotation of output as $D\left(G\left(z\right)\right)$ and real input when $\left(P=1\right)$ with a denotation of output as $D\left(x\right)$, with a cost function (see Equation (8)).

$$\frac{1}{m}{\displaystyle \sum }_{i=1}^m\left[\log D\left(x^{\left(i\right)}\right)+\log \left(1-D\left(G\left(z^{\left(i\right)}\right)\right)\right)\right]$$

(8)

Third-phase: samples are pulled from real images and then passed to the discriminator to make sure the discriminator is able to tell that real images are real, rather than just fake images are fake, this phase marks an end to a training round process. Hence, the whole phases are repeated over and over until eventually, the generator gets to a point where the discriminator really cannot tell real from fake at which point the training session stops. That means the generator can produce high-dimensional perceptual plausible objects. Therefore, the final cost function is (see Equation (9)). See (Figure 6 and Figure 7) for the GAN and the intuitive GAN process, respectively.

$$\underset{G}{\min }\underset{D}{\min }V\left(D,G\right)={\mathbb{E}}_{x~Pdata\left(x\right)}\left[\log D\left(x\right)\right]+{\mathbb{E}}_{z~Pz\left(z\right)}\left[\log \left(1-D\left(G\left(z\right)\right)\right)\right]$$

(9)

The following algorithm shows a summary implementation of the generative adversarial nets proposed in 2014 by Goodfellow et al. [23]. Algorithm 2 Mini-batch stochastic gradient descent training GAN. The number of steps to apply to the discriminator, k, is a hyper-parameter and k = 1.

Algorithm 2 A summary implementation of the generative adversarial nets proposed in 2014 by Goodfellow et al. [23]

for training iterations do     for k steps do          $\mathrm{mini}-\mathrm{batch}m\mathrm{noise}\mathrm{samples}\left\{z^1,\dots ,z^{\left(1\right)}\right\}\mathrm{from}\mathrm{noise}\mathrm{prior}{P}_g\left(z\right)$           $\mathrm{mini}-\mathrm{batch}m\mathrm{noise}\left\{x^1,\dots ,x^{\left(m\right)}\right\}\mathrm{from}\mathrm{data}\mathrm{generating}\mathrm{distribution}{P}_{data}\left(x\right)$       ${\nabla }_{{\theta }_d}\frac{1}{m}\sum _{i=1}^m\left[\log D\left(x^{\left(i\right)}\right)+\log \left(1-D\left(G\left(z^{\left(i\right)}\right)\right)\right)\right]$ end for $\mathrm{mini}-\mathrm{batch}m\mathrm{noise}\mathrm{samples}\left\{z^1,\dots ,z^{\left(m\right)}\right\}\mathrm{from}\mathrm{noise}\mathrm{prior}{P}_g\left(z\right).$ update the generator by descending it is stochastic gradient:                                                    $\frac{1}{m}{\displaystyle \sum _{i=1}^m}\log \left(1-D\left(G\left(z^i\right)\right)\right)$ 9. end for

3.4. Standard Data Augmentation

In computer vision, standard data augmentation method is used as a tool to artificially expand the training dataset by modifying the existing training sets through several transformations. The reason for this is when existing datasets are small or imbalanced, preventing model overfitting or improving model performance. We carried out the following transformation on the datasets. The following techniques form our standard augmentation datasets.

• Geometric transformation on the existing datasets: random crop, flip, rotate;
• Color transformations: blur, sharpen and noise;
• Random erasing [47];
• Mixing images [48].

3.5. SG2-ADA and Its Performances

The SG2-ADA is a state-of-the-art generative model that could generate high-quality synthetic images with fewer training datasets in contrast to other types of GAN that requires a large number of data to perform well, though a large dataset could significantly improve its performances [21]. The model was designed to work with a limited available dataset using an innovative data augmentation technique called ADA, see Figure 5, which is similar to the standard augmentation method but the augmentation process is not fixed but adaptive. The standard augmentation technique causes the generated images to acquire some negative properties of the augmentation parameter, for instance, cropping an image will result in cropped generated images [21]. The ADA works based on an improved balanced consistency regularization (bCR) that applies augmentation shown to the discriminator in a controlled and adaptive fashion based on the level of overfitting [21]. This makes the SG2-ADA perform well in the absence of unlimited data and faster during training without compromising performance. Figure 8 visually illustrate the SG2-ADA architecture.

4. Detection Algorithm and Its Applicability

Object detection algorithms allow a computer system to “see” its environs by detecting instances of objects belonging to certain classes in digital visual images or videos. Moreover, these algorithms are classified into two main types: one-stage and two-stage detection algorithms. The former combines all the processes in a single go by predicting bounding boxes around an object making them faster and structurally simple, such as SSD, YOLO, and RetinaNet, while the latter divides the task into two stages, by firstly making a region proposal of objects using deep features and lastly classifying objects with their respective bounding box regression for each object, such as Faster-RCNN, Mask-RCNN, and granulated-RCNN. This makes the two-staged object detection algorithm slower, and structurally complex with better accuracy. However, object detection algorithms have been applied across various spectrums with success, such as autonomous driving, medical feature detection in healthcare, pedestrian detection, theft detection, performance assessment in sports, farm automation, and plant disease detection [49,50,51,52,53,54,55].

4.1. Implementing Algorithm and Its Procedures

Our study utilized both one-staged and two-staged object detection algorithms, namely, SSD and faster-RCNN, respectively.

4.1.1. SSD Model

We used an input image of size $300\times 300$ pixel for implementing the SDD model. Then, we used a VGG16 net as its backbone network without its fully connected (FC) layers. Six auxiliary convolutional layers were added with distinct kernel sizes that aid in detecting our target objects at multiple scales. However, the convolutional layers decreased the feature map sizes, while increasing the depth progressively (see Figure 9).

Our feature maps are of sizes $38\times 38,19\times 19,10\times 10,5\times 5,3\times 3$, and $1\times 1$38 × 38, 19 × 19, 10 × 10, 5 × 5, 3 × 3, and 1 × 1 while computing both the location and the class scores using a $3\times 3$ convolution filter. For the loss function, regression (localization) loss, and classification (confidence scores) loss (see Equations (10), (11) and (12), respectively).

$$L\left(x,c,l,g\right)=\frac{1}{N}\left(L_{conf}\left(x,c\right)+\propto L_{loc}\left(x,l,g\right)\right)$$

(10)

$$L_{loc}\left(x,l,g\right)={\displaystyle \sum }_{i\in Pos}^N{\displaystyle \sum }_{m\in \left\{cx,cy,w,h\right\}}{x}_{ij}^{k}smoot{h}_{L1}\left(l_i^m-{\widehat{g}}_j^m\right)$$

(11)

$$L_{conf}\left(x,c\right)=-{\displaystyle \sum }_{i\in Pos}^N{x}_{ij}^{p}log\left({\widehat{c}}_i^p\right)-{\displaystyle \sum }_{i\in Neg}\log \left({\widehat{c}}_i^p\right)$$

(12)

where

• ${\widehat{c}}_i^p=\frac{exp\left(c_i^p\right)}{{{\displaystyle \sum }}_{p}exp\left(c_i^p\right)}$
• $N=numberofpositvematches$
• $\propto =weightforthelocalizationloss$
• $l=predictedboundingbox$
• $g=groundtruthboundingbox$
• ${\widehat{c}}_i^p=softmaxactivatedclassscorefordefaultboxiandcategoryp$
• $x_{ij}^p=matchingindicatorbetweendefaultboxiandgroundtruthboxj$
• $ofcategoryp$
• $x_{ij}^k=matchingindicatorbetweendefaultboxi$
• $andgroundtruthboxjofcategoryk$

4.1.2. Faster RCNN

Faster-RCNN is a state-of-the-art object detection algorithm that consists of a Region Proposal Network (RPN) and fast-RCNN. The RPN generates region proposals that are fed to Fast-RCNN to detect classes of objects, (see Figure 10). Our study used an input image of size 256 × 256 pixels which generates an $~2000$ anchors per image by $\left(\frac{256}{16}\right)\ast \left(\frac{256}{16}\right)\ast 9$ with 3 aspect ratios of (1:1, 1:2 and 2:1). We then implemented the Non-Maximum Suppression (NMS) based on the image classification and $IoU\mathrm{of}7$. Positive labels are assigned to an anchor with the highest $IoU$ with a ground truth box or $IoU$ that overlaps higher than a 0.7 score with any ground truth. The loss function for the RPN is:

$$L\left(\left\{p_i\right\},\left\{t_i\right\}\right)=1/N_{cls}\left({{\displaystyle \sum }}_i{L}_{cls}\left(p_i,p_i^{\ast }\right)\right)+\lambda /N_{reg}\left({{\displaystyle \sum }}_i{p}_i^{\ast }\times L_{reg}\left(t_i,t_i^{\ast }\right)\right)$$

(13)

where

• $p_i=predictedpropabilityofanchorscontainingaclassofobjectornot$
• $p_i^{\ast }=anchorsgoundtruthvaluescontainingaclassofobjectornot$
• $t_i=coordinatesofpredictedanchors$
• $t_i^{\ast }=groundtruthcoordinatesrelatedwiththeboundingboxes$
• $L_{cl}=classifierloss$
• $L_{reg}=regressionloss$
• $N_{cls}=normalizationparamofmini-batchsize$
• $N_{reg}=normalizationparamofregression$
• $\lambda =10,ordertomaken=bothlossparameterequallyweightedright.$

Figure 10. Faster-RCNN model (Cha et al. [57]).

Figure 10. Faster-RCNN model (Cha et al. [57]).

We then implemented the second stage of faster-RCNN, which is the object detection network. Zeiler and Fergus (ZF) Net [58] was used as the backbone network, which uses the $RoI$ pooling layer for making region proposals generated from RPN (first stage) into a fixed-size feature map with an output of size ($7\times 7\times \mathrm{D})\left(\mathrm{where}\mathrm{D}=256\mathrm{for}\mathrm{ZF}\right)$. In addition, the generated fixed-size feature maps were sent to two fully connected layers, where they were flattened and then sent as outputs with two distinct tasks assigned to them. The first layer predicts the region proposal using a softmax of $\mathrm{N}+1\mathrm{output}\mathrm{params}$ while the second layer determines the bounding box location of the object in the given image using a bounding box regression with $4\times \mathrm{N}\mathrm{output}\mathrm{params}$.

4.2. Performance Measurement

We adopted de jure standard metrics for evaluating the performances of object detection and generative models, such as mean average precisions, loss values, Fréchet Inception Distance (FID), Precision and Recall (P&R), and Kernel Inception Distance (KID). The following explains these metrics in details.

4.2.1. Mean Average

To evaluate the performances of the proposed augmentation pipeline and the standard method on the objection detection models, we used the mean average precision (mAP) metric (see Equation (14)). The evaluation was carried out for the faster-RCNN and SSD object detection models. The metric was conducted in two-level groups; the proposed GAN-based augmentation datasets, and the standard augmentation datasets. We set the intersection over union (IoU) to 0.7, which implies that only values greater than or equal to the set values are considered positive detection. In addition, we also evaluated the loss classification of our model. A higher mAP indicates better detection model.

$$mAP@\propto =\frac{1}{n}{\displaystyle \sum }_{i=1}^{n}A{P}_{i}fornclasses$$

(14)

4.2.2. Fréchet Inception Distance (FID)

The generated synthetic images are close to real images and could be difficult to distinguish them by human visual perception. FID is one of the most recommended and widely used metrics for calculating the similarities between real and synthetic images. It does so by calculating the Wasserstein-2 distance between the two images, real and synthetic, using the Inception-v3 neural network feature space. A lower FID value means higher quality and diverse image.

$$FID={\left|\left|{\mu }_r-{\mu }_g\right|\right|}^2+Tr\left({{\displaystyle \sum }}_r+{{\displaystyle \sum }}_g-2\left({\left({{\displaystyle \sum }}_r{{\displaystyle \sum }}_g\right)}^{\frac{1}{2}}\right)\right)$$

(15)

where ${\mu }_r$ and ${\mu }_g$ represent the mean of the real and synthetic images, respectively. ${{\displaystyle \sum }}_r\mathrm{a}\mathrm{n}\mathrm{d}{{\displaystyle \sum }}_g$ represent the covariance matrices of the real and synthetic images, respectively.

4.2.3. Kernel Inception Distance (KID)

This metric was proposed by Binkowski et al. [59] to replace FID because the former has no unbiased estimator, is computationally heavy, and performs poorly on small datasets. The metric measures image generation quality by determining the difference in the generated image and the training distributions in the representation space of an InceptionV3 net based on a pre-trained ImageNet.

4.2.4. Precision & Recall (P&R)

Meanwhile, Sajjadi et al. [60] used a different approach and proposed a quality evaluation method that utilizes two different factors: Precision and Recall. While Precision shows how close the generated image is to the real image, Recall measures the differences in the distribution of the generated and real images.

5. Experimental Results and Comparative Analysis

This section describes the dataset in detail, the preprocessing of the data, the experimental scheme, and the comparative analysis of our results.

5.1. Experimental Datasets Description

Our study used Rice Leaf Disease Image Samples from Mendeley Data [61]; it is an open-source dataset with 5932 images of rice leaf disease. The images are grouped into four rice leaf diseases as Bacterial blight (BB), Tungro (TG), Brown Spot (BS), and Rice Blast (RB). The category of BB, TG, BS, and RB have 1584, 1308, 1440, and 1600 images, respectively. The raw images come in different sizes ranging from 150 pixels to 300 pixels.

5.2. Experimental Scheme and Process

The proposed approach was implemented on a Linux-based system with Intel Core i7 8700k, 2 NVIDIA Titan XP 12GB, and 32GB of RAM. Pytorch 1.10 framework was adopted for implementing the StyleGAN2-ADA, while Tensorflow 2.7 was adopted for implementing Faster-RCNN and SSD models. We trained faster-RCNN and SSD models from scratch separately using two instances of the datasets to detect four different pathologies of rice leaf diseases to examine the model performances. We implemented two types of data augmentation methods (GAN-based augmentation and standard augmentation). We adopted the binary cross-entropy loss for 200 epochs for better performance. Lastly, we updated the hyper-parameters for every 50 epochs except for the batch size.

• Learning rate: {1$\times {10}^{-2},3\times {10}^{-3}$}
• Batch size: {1}
• Optimizer: {Adagrad, Adam}

5.3. Image Data Preprocessing

This study carried out two images pre-processing for training the GAN and object detection models. For training, the GAN model complied with the requirement of SG2-ADA model by converting the sizes of the raw data into a uniform scale of $256\times 256$ pixels. We used the Python Imaging Library (PIL) to implement the resizing tasks [62]. While for the object detection models, we labeled the datasets and annotated them based on bounding boxes using LabelImg [63]. Then, we converted the dataset to tf-record formats which were divided into 70% for training, 15% for evaluation, and 15% for testing the performances of the model.

5.4. Synthetic Data Generation

Our experiment with the proposed GAN-based augmentation pipeline used an SG2-ADA model, which shows that we can generate high-quality refined rice leaf disease images with a few available training datasets which can be used directly for data augmentation to improve Faster-RCNN and SSD models for rice disease detection. See Figure 11 for real images and Figure 12 for synthetic images.

The visual quality of the synthetic images generated by the proposed pipeline is good and would be difficult to distinguish between real and fake images by a non-expert.

The model generated 50,001 images of 4 classes of rice leaf disease, which include BL, TG, BS, and RB though 46% of the synthetic images were blurry and distorted, which was detected and filtered out using the variance of the Laplacian method. Afterimage preprocessing, we produced 26,694 synthetic images.

Table 2. Results of the SG2-ADA.

	FID	KID	P & R
			Precision	Recall
SG2-ADA	26.67	0.08	0.49	0.14

, shows the evaluation results of the network using the FID, KID, and P&R metrics. The network achieved an FID score of 26.67, KID score of 0.08, Precision of 0.49, and Recall of 0.14 after training for 144 h. It was also observed that the FID value decreased during the course of training.

5.5. Comparison of Mean Average Precisions (mAP)

Table 3. Results of the proposed GAN-based augmentation pipeline generative model.

CNN Models	Rice Leaf Diseases				Standard Augmentation (mAP)	GAN-Based Augmentation (mAP)
	BB	TG	BS	RB
Faster-RCNN	0.78|0.90	0.89|0.93	0.79|0.92	0.88|0.95	0.84	0.93
SSD	0.77|0.89	0.85|0.90	0.76|0.91	0.84|0.93	0.81	0.91

, shows the comparisons of mAP scores between the standard augmentation method and the GAN-based augmentation method. On Faster-RCNN, we observed that the GAN-based augmentation method improves the model performance with a difference of 0.09 mAP score on the validation datasets. On the SSD model, we also see that the GAN-based augmentation method improves the model performance with a difference of 0.1 mAP score.

Furthermore, the result showed that RB achieved the best average precision (AP) score of 0.95 for Faster-RCNN and 0.93 for SSD model for the GAN-based augmentation pipeline method, while TG achieved the best result with an AP score of 0.89 for Faster-RCNN and a 0.85 AP score for the SSD model for the standard augmentation method, while the result showed that BB achieved the lowest average precision (AP) score of 0.90 for Faster-RCNN and 0.89 for the SSD model for the GAN-based method. BB also achieved the lowest AP score of 0.78 for Faster-RCNN and 0.77 AP score for the SSD model for the standard augmentation method.

5.6. Comparison of Learning Curves

In Figure 13, we compared training and validation learning curves alongside a standard augmentation and GAN-based augmentation on the Faster-RCNN model. We observed that GAN-based augmentation improves the model efficacy. In the final steps, we observed a 0.18 difference between training losses of standard augmentation and the GAN-based augmentation model. We also see a 0.204 difference between the validation losses of standard augmentation and the GAN-based augmentation model. Each curve shows the averages across five runs under an augmentation type. This gives a clear sense of variability in these curves and the significance of the differences between the two types of augmentation methods.

In Figure 14, we compared training and validation learning curves alongside a standard augmentation and GAN-based augmentation on the SSD model. We observed that GAN-based augmentation improves the model efficacy. In the final steps, we observed a 0.25 difference between training losses of standard augmentation and the GAN-based augmentation model. We also see a 0.22 difference between the validation losses of standard augmentation and the GAN-based augmentation model. Each curve shows the averages across five runs under an augmentation type. This gives a clear sense of variability in these curves and the significance of the differences between the two types of augmentation methods.

5.7. K-Fold Validation

We carried out a k-fold validation, splitting the entire datasets into k-folds where k = 5, and at a specific iteration, one block is held for validation while the other block is to train the network. The final score is computed by taking the average scores across k-folds. Figure 15 below shows the k-fold dataset iterations representation, groups, and classes.

In addition, we compared the object classification accuracies of faster-RCNN and SSD based on the GAN and the standard method on five datasets (k = 5). The last row is the total average accuracy of the four methods on k = 5. In the majority of the datasets, the accuracy of the proposed augmentation pipeline is higher than the standard augmentation on all k-folds. The classifier with our proposed method is found to result in the best prediction average accuracy, which was 91.83%. The other results are found to be 83.78%, 78.92%, 84.47%, 88.71%, 77.93%, and 83.37% (See

Table 4. K-fold cross validation classification accuracy (%) of the four model methods.

Dataset (k = 5)	Gan- Faster-rcnn		Standard Faster-rcnn		Gan-ssd		Standard-ssd
	Validation	Train	Validation	Train	Validation	Train	Validation	Train
k-fold 0	85.25	91.78	81.56	84.21	84.32	92.81	79.21	82.95
k-fold 1	76.11	89.33	71.28	77.37	77.51	84.58	71.13	76.27
k-fold 2	87.21	94.08	82.93	86.18	85.98	86.47	80.91	82.78
k-fold 3	91.38	95.10	87.39	90.08	90.38	92.98	85.72	89.97
k-fold 4	78.94	88.85	71.43	83.59	79.17	86.71	72.67	84.88
Average	83.78	91.83	78.92	84.29	83.47	88.71	77.93	83.37

)

5.8. Comparison of Losses with Boxplots

The boxplot in Figure 16 visualizes the difference in the minimum validation and training losses across all our samples. We achieved further comparative insight regarding standard augmentation and GAN-based augmentation methods across the training and validation losses using boxplots.

The study hypothesis is to show that the minimum validation and training loss differs significantly between the proposed augmentation pipeline and the standard augmentation method in classifying rice leaf diseases. Therefore, at a significance level of <= 0.05 (see

Table 5. Pairwise comparison between the minimum training and validation losses from the proposed augmentation pipeline and the standard augmentation method.

Augmentation	Model	Data Type	t-Test (p-Value)
Proposed vs. Standard	Faster-RCNN	Training	$4.8\times {10}^{-3}$
Proposed vs. Standard	Faster RCNN	Validation	$9.1\times {10}^{-4}$
Proposed vs. Standard	SSD	Training	$3.6\times {10}^{-6}$
Proposed vs. Standard	SSD	Validation	$8.3\times {10}^{-5}$

), our analysis confirms our hypothesis that the minimum validation and training loss is significantly higher in the standard augmentation when compared to the proposed augmentation pipeline across the two object detection models. Hence, our result implies that the proposed augmentation pipeline for improving the performances of an object-detection model when faced with a small or imbalanced dataset is effective and therefore significantly improves the models’ efficacy.

6. Concluding Remarks

The recent advancement in DL methods has made it possible to efficiently detect rice leaf disease automatically, albeit a large, diverse, and balanced dataset is required to achieve an optimum result. Currently, there are few balanced rice leaf disease datasets available in open source when compared to millions of other datasets freely available. Previous studies have used the standard augmentation methods to create new samples to increase the size of the image dataset by modifying the positions (rotation, etc.) and colors (brightness, etc.) without introducing “new” diverse data to the model which does not impact much on the model efficacy and generalization. We proposed a data augmentation pipeline using an SG2-ADA state-of-the-art generative model and variance of the Laplacian method to generate and filter out high-quality “new” synthetic rice leaf disease images to improve the performance of the CNN-based object-detection models. We compared the performances between the proposed augmentation pipeline and the standard augmentation method on one-staged and two-staged object detection models, SSD and Faster-RCNN, respectively. Based on our results we conclude that: (1) the proposed augmentation pipeline is efficient in producing high-quality synthetic images of rice leaf disease with few training datasets. (2) The visual quality of the synthetic images was very close to real images and could be difficult to visually distinguish. (3) We show that the proposed augmentation pipeline could be an effective tool for amplifying, diversifying, and correcting the imbalances in plant-based disease datasets. (4) We observed that the proposed augmentation pipeline yielded better efficacy when compared to the standard augmentation method. (5) Faster-RCNN and SSD model proves to be effective in detecting various classes of rice leaf disease. (6) This work skillfully combines several concepts, approaches, techniques, and components, such as Data Augmentation, Object Detection Models, Rice Leaf Disease Detection, SG2-ADA, Laplacian filter, Faster-RCNN, and SSD models. Hence, generative model-based augmentation is a promising area of study to improve model performance and to solve the issue of generalization when data is limited or imbalanced, though it comes with time and resource costs. This is true, especially in the area of agriculture, medicine, etc.