A Residual LSTM and Seq2Seq Neural Network Based on GPT for Chinese Rice-Related Question and Answer System

Abstract

Rice has a wide planting area as one of the essential food crops in China. The problem of diseases and pests in rice production has always been one of the main factors affecting its quality and yield. It is essential to provide treatment methods and means for rice diseases and pests quickly and accurately in the production process. Therefore, we used the rice question-and-answer (Q&A) community as an example. This paper aimed at the critical technical problems faced by the agricultural Q&A community: the accuracy of the existing agricultural Q&A model is low, which is challenging to meet users’ requirements to obtain answers in real-time in the production process. A network based on Attention-ResLSTM-seq2seq was used to realize the construction of the rice question and answer model. Firstly, the text presentation of rice question-and-answer pairs was obtained using the GPT pre-training model based on a 12-layer transformer. Then, ResLSTM(Residual Long Short-Term Memory) was used to extract text features in the encoder and decoder, and the output project matrix and output gate of LSTM were used to control the spatial information flow. When the network contacts the optimal state, the network only retains the constant mapping value of the input vector, which effectually reduces the network parameters and increases the network performance. Next, the attention mechanism was connected between the encoder and the decoder, which can effectually strengthen the weight of the keyword feature information of the question. The results showed that the BLEU and ROUGE of the Attention-ResLSTM-Seq2seq model reached the highest scores, 35.3% and 37.8%, compared with the other six rice-related generative question answering models.

1. Introduction

With the advent of big data, people are facing problems such as information explosion and information overload. Search engines [1] as the primary sources of information provide users with a large amount of useful information and bring great inconvenience and trouble to users. It is difficult to meet people’s complex search needs through keyword matching or shallow analysis of the problem. The search results are not concise and accurate enough, so it is difficult for users to locate the information they need quickly. As an alternative to search engines, the question-answering model is mainly limited to the size of the question-answering data set, which cannot fully cover all questions in a particular field. It is easy to cause wrong answers, multiple answers, and no retrieved answers, and it is also difficult to meet the question-answering needs of professional fields. In agriculture, the problem of diseases and pests in rice production has always been one of the main factors affecting its quality and yield [2]. Therefore, it is vital to provide rapid and accurate control methods and means for rice diseases and pests in the production process. At the same time, farmers often encounter a series of agricultural problems such as breeding and planting, diseases and pests, and water and fertilizer management; most of the solutions are network queries and expert telephone consultations [3]. However, there are some problems with both methods, such as redundancy of network query answer information, lags, and limited expert resources in expert answers. There are question data with different expressions but the same semantics. Repeated answers to such questions by experts will consume a lot of resources. With the penetration of big data technology into agricultural production, agricultural data has shown explosive growth [4]. Still, most agricultural data are messy and unclassified, making it challenging to recognize and use by people directly. The agricultural science and technology information service platform comprises the China Agricultural Technology Extension app and web agricultural technology extension platform. It is a comprehensive service platform including an agricultural technology question-and-answer (Q&A) [5] community, agricultural science and technology service, guidance, market price services, and other functions.

In the agricultural science and technology information service platform, the agricultural question-and-answer (Q&A) community plays a vital role in the technical exchange between farmers and agricultural technicians and users’ access to solutions to problems encountered in agricultural production. Users ask more than 1000 questions in the rice Q&A module every day. Agricultural data are mainly managed by manual screening features and shallow learning models. However, due to the high dimensionality, sparsity, and poor standardization and professionalism of agricultural data, it is not easy to find high-quality Q&A pairs. At present, the critical technical problems faced by the agricultural Q&A community are the low accuracy of the existing agricultural Q&A model and the challenge of meeting users’ requirements to obtain solutions in real-time in the production process. The question answering system generally includes semantic analysis of user questions, answer extraction, and answer generation [6], and the question answering model is a vital research hotspot for answer generation. Therefore, using deep learning and natural language processing technology to automatically and quickly mine the text features of rice knowledge for the construction of a question-answering model is a significant problem to be solved [7]. At the same time, rice is one of the essential food sources and has a wide planting area in China. Therefore, we used the rice Q&A community as an example. This paper constructs the semantic model of the rice Q&A community by using deep learning and natural language processing technology to deeply excavate the semantic information of rice-related text, which is of great significance to improving the performance of the Q&A community and provides ideas for the development of an intelligent rice-related Q&A system.

The early question answering system can only accept the question sentences of specific templates, and there are less question answering data. With the development of the Internet, the acquisition of data has become simple, and the retrieval question-answering technology has developed rapidly. Methods based on logical reasoning, template matching, machine learning, and data redundancy have been proposed to retrieve the answers according to the shallow semantics of the question sentences. However, retrieval question answering has the limitation that answers and questions must have common keywords. With the rise of encyclopedia websites, it is more convenient to obtain high-quality structured data. A large number of knowledge bases have been established. The rise of machine learning technology has promoted the research of question answering systems based on the knowledge base. The purpose of Q&A is to answer a given question in natural language accurately. Q&A does not need to format the query and return the list of relevant documents but enables users to interact with the machine more naturally. It is a classical research problem in natural language processing involving problem analysis, answer retrieval, and answer ranking technology. Question answering has been studied for many years and implemented in various systems. Baseball is considered the earliest question answering system [8], while recent chat robots such as Siri, Alexa, and Cortana are also popular. However, there are still many challenges in the research, from early Q&A templates to recent Q&A systems. Common paradigms of classical machine learning include feature engineering and machine learning algorithms, which require a lot of effort to build handmade functions [9]. As a machine learning method involving deep architecture, deep learning has attracted extensive attention in recent years and has been verified in various fields. Generally speaking, intelligent Q&A can be divided into retrieval-based intelligent agricultural Q&A and generative intelligent agricultural Q&A.

For the questions users raise, the retrieval-based intelligent question answering system usually retrieves the most matching answer from the knowledge base. It has evolved from the traditional handwriting rule matching to the retrieval question answering scheme through the deep learning semantic processing method. In the knowledge base, through semantic processing, the system identifies the user’s knowledge acquisition intention, accurately and concisely answers natural language questions, accurately matches the knowledge, and gives feedback to the user. However, the semantics of agricultural problems are complex and changeable, which brings significant challenges to intelligent question answering. More and more scholars have researched retrieval-based intelligent question answering methods. The traditional retrieval agricultural question answering system is often a simple matching with the knowledge base, which cannot find the deep semantic relationship between problems and has some limitations. At this stage, researchers used the method of deep learning to mine the semantics of question sentences to improve the accuracy and reliability of matching in the question answering system. Medelyan et al. [10] proposed a new algorithm to automatically extract index words from documents related to the agricultural field for document indexing based on a thesaurus. Through machine learning technology and text semantic information processing, controlled vocabulary eliminates the emergence of meaningless or incorrect phrases and effectively improves the system performance. To improve the efficiency of literature retrieval in the agricultural field, Wang et al. [11] constructed the agricultural domain ontology by segmenting and cleaning the literature and using association analysis and improved hierarchical clustering to find the relationship between domain concepts. Experiments show that this method can improve the clustering effect of relationships between concepts in the agricultural field and the construction effect of domain ontology. Tao et al. [12] proposed a deep retrieval model for context retrieval to solve the low utilization of dialogue interaction information in the retrieval question and answer model. As an essential part of the model, the interaction block comprises a self-attention module, an interaction module, and a compression module of compression results. The depth of the model comes from stacking multiple interaction blocks, which are represented interactively in an iterative manner. The evaluation results with the benchmark model show that the depth retrieval model has higher accuracy. No new text information will be generated based on the retrieval question answering model. The reply type and content can only retrieve more appropriate answers from the predefined knowledge base. Once the user’s questions exceed the scope of the knowledge base, the answering question system will not be able to answer the user [13] accurately. Compared with retrieval-based question answering, the generation model of generative question answering does not depend on the constructed knowledge base. Still, the model learns from the knowledge base to generate new responses. The advantage of this method is that it is more intelligent and closer to human dialogue. Suktarachan et al. [14] developed a question-and-answer service system for farmers, focusing on similar technologies previously applied to the original system of language generation, subject roles, and conceptual vocabulary structure through a Thai short message service.

There has been significant research in the field of Q&A systems. Still, there is no specific Q&A system in the agricultural field to give practical answers by analyzing the questions raised by farmers. To solve this limitation, Gaikwad et al. [15] developed a question and answer system in agriculture. The system’s input is the corpus of documents related to agriculture and a set of predefined question templates. The question-and-answer task is divided into sub-tasks such as information database preparation, question processing, answer retrieval, and evaluation. The system provides convenience for farmers to obtain information. Muller et al. [16] proposed an open retrieval generative question answering method for the multilingual environment to solve the problems of multilingualism, multi-source code, and open domain in natural language processing. Firstly, the pre-trained multilingual T5 model is used to model the task. The questions corresponding to the candidate samples are input into the model for optimization and adjustment, and finally, the training model generates an appropriate answer. It can provide a reference for the research of Q&A systems in agriculture. The above research has the problems of small vocabulary, feature coefficient, and single text feature extraction, which cannot integrate rich features. In this paper, the traditional end-to-end question answering model is optimized, and the LSTM neural network is optimized in the encoder and decoder. The residual connected LSTM extracts the semantic features of deep-seated rice question answering pairs. Then, the attention mechanism is added to the encoder and decoder to strengthen the weight of key information in question sentences to mine the semantic information of user question sentences. The main work of this paper is as follows:

(1)

A method based on Residual LSTM and Seq2Seq is proposed.

(2)

The pre-training model GPT is cleverly transferred to the context embedding layer, which provides a new idea of sentence pair representation.

(3)

The deep neural network is used to extract text features from the encoder and decoder, and the attention mechanism is connected between the encoder and decoder to strengthen the weight information of question keywords in the sentence.

2. Materials and Methods

2.1. Corpus Preparation

We derived the rice-related Q&A data from the “China Agricultural Technology Promotion” Q&A community [17]. A total of 15,000 commonly used Q&A data pairs of rice were identified by agricultural experts, including five categories (diseases and insect pests, weeds and pesticides, storage and preservation, cultivation management, and others); the amount of data in each category is 6557, 1425, 1128, 3509 and 2381. In

Table 1. Sample of rice-related Q&A pair data.

Quesetion	Answer
How to determine the water shortage of rice seedbed in the management of rice seedbed.	In the morning, there are water drops on the tips of leaves, so you don’t need to water them when you spit water. If you don’t have water drops, you have to water them thoroughly.
When can rice use herbicides correctly?	Weeding should be carried out reasonably before and after rice transplanting or at the 4–6 leaf stage of rice seedlings.
Rice seedlings usually emerge several days after sowing.	Rice seedlings generally emerge from 7 to 10 days after sowing.
Rice damping blight mostly occurs at several leaf stages of rice seedlings.	Rice damping blight mostly occurs in the stage of two leaves and one heart.
Why should rice be dried before sowing?	Sun-drying can promote the post-ripening of seeds and improve the activity of enzymes.
What are the fungicides commonly used in rice seedbeds?	There are mainly 30% metalaxyl, 85% oxidil, dixone, ethylallicin and so on.
What are the advantages of rice floppy disk seedling raising?	Rice seedling tray is beneficial to centralized management, convenient for throwing seedlings, with many roots and fast growth after transplanting.
What are the symptoms of rice sheath blight?	Brown striped patches mainly appear on the basal stems.
What are the main diseases of rice at seedling stage?	The main diseases of rice seedling stage are damping wilt, bakanae disease and seedling blast.
How to choose excellent rice varieties.	Choose varieties suitable for local high yield, high quality, strong stress resistance and high efficiency
When can rice use herbicides correctly?	Weeding should be carried out reasonably before and after rice transplanting or at the 4–6 leaf stage of rice seedlings.
When will barnyard grass in paddy fields be controlled?	Barnyard grass in paddy fields is best removed when the seedlings are ploughed and weeded after turning green.
What problems should be paid attention to when using rice germinator?	Set germination acceleration time and heating upper limit, keep reasonable water level, and check the pipes frequently.
Which soaking agent is better for rice seed soaking?	Soaking rice seeds with strong chlorine essence or potassium permanganate.
What chemical fertilizer should be applied in rice seedling stage?	On the basis of basal tiller fertilizer, recovery fertilizer should be properly applied according to seedling conditions, mainly compound fertilizer containing phosphorus and potassium
What are the precautions for rice seed coating?	Selection of seed coating agent, coating temperature, film solidification, safe storage and strict prevention of seed drying.
What causes the dead stalk of rice in the later stage?	Sheath blight is the main cause of rice stalk death, and rice planthopper can also cause rice stalk death.
What is the cause of rice bakanae disease?	Seeds carry bacteria, and spores of pathogens sneak into seeds and spread in the first year.

, a sample of rice related Q&A text data is shown:

The above rice-related text Q&A data sets are different from the text data sets in general fields and have the following characteristics:

(1)

High domain specificity. All the text data are question-and-answer pairs related to rice, and the semantic boundaries of sentences are fuzzy. Therefore, it is difficult to transfer the general domain model to the semantic domain of rice knowledge text for training.

(2)

The sequence of question-and-answer pairs is short. It is challenging to extract semantic information from features generated during short text training, which increases the difficulty of model recognition.

(3)

All kinds of problems are unevenly distributed. In the corpus, the data volume of diseases and insect pests, cultivation management and other three categories is large, while the data volume of weeds and pesticides, storage and preservation is small. The specific data volume category distribution is shown in Figure 1.

2.2. Methods

This paper used Sequence to Sequence (Seq2seq) [18] based on the attention mechanism and ResLSTM to construct a rice-related question answering model. The model consists of four parts: the embedding layer, the encoder, the attention mechanism, and the decoder. The pre-trained GPT model represents the intelligent rice-related Q&A text. The attention mechanism pays attention to the correlation between the coded information and the current output state, as shown in Figure 2.

2.2.1. Embedding Layer

The 12-layer transformer based GPT pre-training model was used to vectorize the built rice answer selection text data. The BERT pre-training model was built using the transformer decoder, while the GPT pre-training model was built using the transformer encoder. The Bert model is a two-way language model, which can better obtain the information of word context and solve the phenomenon of polysemy. The GPT model is a one-way language model; it is more suitable for text generation, text translation, logistic regression and other tasks. Therefore, this paper used GPT as the vectorization method of rice question and answer text data. The word vector dimension was set to 300.

2.2.2. Residual Long-Term and Short-Term Memory Networks

Researchers put forward the LSTM (Long Short-Term Memory) [19] neural network by improving and optimizing the RNN (Recurrent Neural Network). In RNN [20] processing text sequences, gradient explosion or disappearance is easy to occur, leading to the loss of text features and the inability to obtain long-distance text data when extracting text features. The researchers optimized and improved the RNN model and introduced the gate mechanism to solve the above problems. It effectively solves the problem of artificially expanding tasks that are difficult to solve by RNN [21], and RNN solves the problem that gradients tend to disappear. LSTM mainly includes an input gate, output gate, forgetting gate, and memory unit. Combining these three gate structures and memory units dramatically improves the ability of the cyclic neural network to process long-range data [22] and dramatically solves the problems of long-range dependence and gradient loss. The calculation formula is:

$$f_t=\sigma (w_f+U_f{h}_{t-1}+b_f)$$

(1)

$$i_t=\sigma (w_i{x}_t+U_i{h}_{t-1}+b_i)$$

(2)

$$o_t=\sigma (w_o{x}_t+U_c{h}_{t-1}+b_c)$$

(3)

$${\widehat{c}}_t=\tanh (W_c{x}_t+U_c{h}_{t-1}+b_c)$$

(4)

$$c_t=f_t\circ c_{t-1}+i_t\circ {\widehat{c}}_t$$

(5)

$$h_t=o_t\circ \tanh (c_t)$$

(6)

In the formula, x_t represents the current input information, h_t−1 represents the historical hidden layer information, $\sigma$ represents the sigmoid function, tanh represents the hyperbolic tangent function, $\circ$ represents the vector corresponding to the bit-by-bit multiplication operation, and W and b represent the trainable parameters in the model. The LSTM network structure diagram is shown in Figure 3:

When people use RNN to process sequential tasks, they often lose gradients and undergo gradient explosion. To solve this problem, researchers put forward LSTM based on RNN, a deep neural network optimized and improved based on RNN. It can solve the problem of gradient disappearance in RNN. It has advantages over RNN in the long-term complex data processing. Compared with RNN, LSTM avoids the gradient disappearance problem caused by increasing training time. However, with the increase in data dimension and data volume, the number of LSTM layers and the amount of training data will overfit the model. Therefore, in this paper, residual LSTM is used as a feature extractor to overcome the over-fitting problem. When training multi-layer LSTM, ResLSTM provides an additional low-space shortcut by using the output layer to separate spatial and temporal fast paths. ResLSTM uses the output projection matrix and output gate of LSTM to control the spatial information flow. When the training network reaches the optimal effect, the network only keeps the identical mapping vector value of the input features, effectively reducing the network parameters and improving feature extraction. The structure diagram of ResLSTM is shown in Figure 4.

The residual network’s output consists of two parts, which receive the input of the previous layer and the output of the mapping of this layer. The two features are spliced, and the calculation formula is as follows:

$$y=F(x;W)+x$$

(7)

where y is the output result, x is the input of the previous layer, $F(x;W)$ is the feature map, and W is the internal weight coefficient of the LSTM network. $F(x;W)$ means that y is obtained from input x mapped by the LSTM neural network. When there is a constant mapping of input x, $F(x;W)$ needs to learn the residual mapping y-x. When the loss function of the model tends to be stable, the LSTM network transmits it directly through constant mapping, thus simplifying the parameters of the training model and improving the model effect. The residual LSTM model includes three main gate structures: forget gate, input gate, and output gate. The calculation formula for each state is as follows:

$$f_t=\sigma (W_f{h}_{t-1}+V_f{x}_t+b_f)$$

(8)

$$i_t=\sigma (W_i{h}_{t-1}+v_i{x}_t+b_i)$$

(9)

$${\tilde{c}}_t=\tanh (W_c{h}_{t-1}+V_c{x}_t+b_c)$$

(10)

$$c_t=c_{t-1}{f}_t+i_t{\tilde{c}}_t$$

(11)

$$o_t=\sigma (W_o[h_{t-1},x_t]+b_o)$$

(12)

$$r_t=\tanh (c_t)$$

(13)

$$m_t=W_p{r}_t$$

(14)

$$h_t=o_t(m_t+W_h{x}_t)$$

(15)

Here, x_t denotes input data, h_t denotes a hidden state, c_t is defined as a cell state, f_t denotes an output state of an updated forgetting gate, i_t and c are an updated input gate and an output state, respectively, and o_t is an output state of an updated output gate. W and V correspond to the weight matrix, b is the deviation coefficient, and σ and tanh are activation functions.

2.2.3. Encoding-Decoding Structure

Seq2seq is the primary neural network model for processing sequence tasks at present. It was initially applied in machine translation [23] and is suitable for sequence-to-sequence applications. Seq2seq is composed of an encoder and decoder, which can extract the features of input data effectively. The Seq2seq model has been widely used in machine translation [24]. The input sequence collects all the words that make up the question. The encoder converts the input sequence into feature vectors of fixed size. In the encoding process, there are three steps: First, the string is marked as a list. Each tag is then associated with a vector, which is a subset of the input vocabulary. The hiding state will be updated according to the input for each time step. Finally, it is embedded by the encoder network to generate an intermediate state. During training and testing, the decoder works differently from the encoder part in the model. The goal of the decoder is to generate an answer according to the hidden state of the encoder. The decoder calculates the previous layer’s hidden state by using the previous layer’s hidden vector, the hidden output vector of the previous layer, and the original hidden vector. Finally, the loss is calculated according to the predicted output of each time step of the model. The structure of Seq2seq is shown in Figure 5. The codec is usually a multi-layer RNN or LSTM structure, where the intermediate vector C contains x₁, x₂, …, x_m encoding information. For time t, the output y_t−1 at the previous time, the hidden layer states s_t−1 and c at the previous time are input to the decoder as inputs. Finally, the hidden layer state s_t of the decoder is obtained, thereby predicting the output value.

This paper used the Seq2Seq structure as the rice question answering model. For the Seq2Seq model, many different neural network structures can be used in the encoder and decoder, and different neural network models are often selected according to different model characteristics. In this paper, the encoder and decoder use a residual LSTM neural network model. The encoder outputs the variable-length sequence as a fixed-length sequence. In the rice question and answer text data set, the question-and-answer pair can be expressed as <H, Y>, H represents the question raised by the user, and Y represents the corresponding answer, so the question-and-answer pair can be expressed as the following formula:

$$H=\langle h_1,h_2,h_3,\dots ,h_t\rangle$$

(16)

$$Y=\langle y_1,y_2,y_3,\dots ,y_k\rangle$$

(17)

In the formula, T and K represent the maximum lengths of question H and answer Y.

The encoder transforms an indefinite length input sequence H into a constant dimension context vector C through a series of nonlinear transformations. In the decoder, the sequence and the context vector are outputCUsed to predict the next output word $y_k$. The calculation formula is as follows:

$$y_k=\arg \max P(y_k)={\displaystyle \prod _{k=1}^{T}P(y_k|\{y_1,y_2,\dots ,y_{k-1}\},C)}$$

(18)

According to the maximum likelihood estimation, the conditional probability of the output sequence in a given input sequence, the loss function is as follows:

$$P(Y|X)={\displaystyle \prod _{k=1}^{N}P(y_k|C,y_1,y_2,\dots ,y_{k-1})}$$

(19)

$$L=-\frac{1}{n}\log P(y_1,\dots ,y_m|h_1,\dots ,h_t)=-\frac{1}{n}{\displaystyle {\sum }_{t-1}^n\log P(y_m|y_1,\dots ,y_{m-1},C)}$$

(20)

To determine the specific output at this point, it needs to map the output of the decoder to the dimensions of the vocabulary and use the Softmax function to calculate the final output to calculate the probability distribution of each word. Usually, the word with the highest frequency is selected as output at this time. However, this Seq2Seq model still has some defects. Firstly, the encoder converts the input sequence into a context vector, which leads to the semantic loss of high compression ratio information. Then the word information at the back end of the input sequence will be concealed by the information at the front end. The transmission of information will deviate, making the information in the sequence more critical. To solve these problems, we introduce an attention mechanism into the model.

2.2.4. Attention Mechanism

The sequence-to-sequence model has two parts: encoder and decoder. In the traditional Seq2Seq model, the encoder and decoder exchange data through an intermediate semantic vector, and the length of the semantic vector is fixed. When a long sequence of text is input into the encoder, with the encoding and decoding process, the feature information behind the text sequence will cover the feature information before it. We used an attention mechanism between the encoder and the decoder; the feature information of keywords can be retained in the input feature information of the decoder to the input sequence of the decoder, such as c2 in Figure 6. The training model selectively learns these outputs and finally associates them with the corresponding output sequence [25]. The structure diagram of the attention mechanism based on Seq2Seq is as follows:

The calculation formula of attention mechanism is as follows:

$$s_i=f(s_{i-1},y_{i-1},c_i)$$

(21)

$$c_i={\displaystyle \sum _{j=i}^{T_x}{a}_{ij}{h}_i}$$

(22)

$${\alpha }_{ij}=\frac{\exp (a_{ij})}{{\displaystyle {\sum }_{k=1}^{T_x}\exp (e_{ik})}}$$

(23)

$$e_{ij}=a(s_{i-1},h_j)$$

(24)

$$e_{ij}=v^T\tanh (W_s{s}_{i-1}+W_h{h}_t)$$

(25)

In the formula, $s_i$ is the hidden layer state of RNN at time i, c_i represents the weight value of each output h_i. v, W_s, and W_h denote the weights, offset terms and embedding layer parameters in the encoder and decoder recurrent neural networks.

3. Results

3.1. Hardware, Software Environment and Evaluation Indicators

The server’s hardware environment was NVIDIA Corporation (Santa Clara, CA, USA) device 1e04 (Rev A1), the GPU was NVIDIA GeForce RTX 2080ti, and both the research and control experiments were done in the Ubuntu 18.04 environment(Canonical, London, UK). For training, the deep learning framework Pytorch was utilized in conjunction with Cuda10.1.

We used the BLEU [26] scoring method to evaluate the model’s performance. The BLEU score is used to calculate the similarity between the model output and the actual sentence. It is usually a number from 0 to 1. If the value is close to 1, it can be considered a better model, and 0 does not match. BLEU1, BLEU2, BLEU3, and BLEU4 represent the accuracy of 1-g, 2-g, 3-g, and 4-g, respectively. BLEU is the average value of BLEU1, BLEU2, BLEU3, and BLEU4, and its formula is as follows:

$$BLEU=\frac{{\displaystyle {\sum }_{c\in candidates}{\displaystyle {\sum }_{n-gram\in c}Coun{t}_{clip}(n-gram)}}}{{\displaystyle {\sum }_{c’\in candidates}{\displaystyle {\sum }_{n-gram’\in c’}Count(n-gram’)}}}$$

(26)

In the formula, the count clip (n-gram) represents the maximum number of n-grams in the output, and in the actual sentence, count (n-gram) represents the number of n-grams in the output and n represents the length of the context word.

The ROUGE scoring method [27] is used to evaluate the output of the answer by the model, and the proportion of N tuples in the answer sequence output by rice question and answer in the standard answer sequence is calculated. The word overlap between the output answer sequence and the standard answer sequence is calculated as the standard to measure the similarity. The calculation formula is as follows:

$$ROUG{E}_N(c)=\frac{{\displaystyle {\sum }_{s\in S}{\displaystyle {\sum }_{gra{m}_n\in s}match(gra{m}_n)}}}{{\displaystyle {\sum }_{s\in S}{\displaystyle {\sum }_{gra{m}_n\in s}conut(gra{m}_n)}}}$$

(27)

In the formula, C represents the generated answer, S represents the standard answer, match (gram) represents the frequency of N original phrases in the output answer, and count (gram) represents the frequency of N original phrases in the standard answer.

3.2. Model Training and Parameter Setting

We used the ResLSTM neural network for encoding and decoding in this paper. The initial learning rate of the model was set to 0.01, the number of hidden layer nodes was set to 100, and 50 rounds of training were set. We used the Pytorch deep learning framework to build a neural network. In the experiment, we divided 15,000 question-and-answer pairs into training sets and test sets according to the ratio of 9:1, and a random gradient descent algorithm updated the model weights. There were 13,500 pairs of training sets and 1500 pairs of test sets. We trained the ResLSTM-Attention-Seq2seq model with the data set constructed in this chapter. When the model began to converge, we saved the model and related parameters. The specific steps are as follows: first, 13,500 pairs of standard rice question-and-answer pairs are input as training sets, and the neural network encodes the question text; then, the loss between the generated answer and the correct answer of the rice question answering model is calculated, and the model parameters are updated; finally, when the model loss function converges, the weight of the model network parameters is saved. The model training process and testing process are shown in Figure 7.

3.3. Model Hyper-Parameter Setting

Firstly, we set batchsize 32, 64, and 128 to train the model. Batchsize is a super essential parameter of the deep learning model. Too much batchsize leads to poor generalization ability. The BLEU score of the model was shown in

Table 2. Bleu value of ResLSTM-Attention-Seq2seq under different batch sizes.

Batchsize	BLEU1 (%)	BLEU2 (%)	BLEU3 (%)	BLEU4 (%)
32	36.7	35.6	34.3	35.1
64	34.6	33.1	32.5	33.9
128	33.7	31.5	29.8	32.7

. When the batchsize was set to 32, the ResLSTM-Attention-Seq2seq model had the highest BLEU value and obtained the best answer generation effect, with BLEU1 and BLEU2, BLEU3, and BLEU4 reaching 36.7%, 35.6%, 34.3%, and 35.1% respectively. Therefore, this article set batchsize to 32.

Then, we set dropout to 0.1, 0.3, and 0.05, leaving the other parameters unchanged. The BLEU score of the ResLSTM-Attention-Seq2seq model is shown in

Table 3. Bleu value of ResLSTM-Attention-Seq2seq under different drops.

Dropout	BLEU1 (%)	BLEU2 (%)	BLEU3 (%)	BLEU4 (%)
0.1	36.9	35.8	34.9	33.7
0.3	34.8	33.9	34.1	30.5
0.05	33.6	32.8	31.8	29.6

. When we set dropout to 0.1, the model had the highest BLEU value and achieved the best answer generation effect, so this paper set dropout to 0.1.

3.4. Comparative Analysis of Text Vectorization Test

A 12-layer Chinese GPT model was used to vectorize the text data of rice question-and-answer pairs. At the same time, we compared it with Glove [28], BERT [29], and ELMo [30] pre-training models. The text features trained by four models are input into the Seq2Seq-LSTM neural network. From

Table 4. Effects of seq2seq LSTM Q&A model under different pre-training models.

Pre-Training Model	BLEU1 (%)	BLEU2 (%)	BLEU3 (%)	BLEU4 (%)	ROUGE (%)
GPT	32.9	31.8	30.7	27.6	32.7
Glove	29.7	27.7	25.4	23.1	28.7
BERT	31.8	30.1	27.6	27.1	31.6
ELMo	28.6	25.1	23.6	22.7	27.7

, it can be seen that four different text pre-training models were used in the embedding layer, and the GPT pre-training model obtained the highest BLEU and ROUGE, with BLEU1, BLEU2, BLEU3, BLEU4, and ROUGE reaching 32.9%, 31.8%, 32.7%, 27.6%, and 32.7% respectively. Compared with Glove’s BLEU1, BLEU2, BLEU3, BLEU4, and ROUGE, the GPT pre-trained model method was improved by 3.2, 4.1, 5.3, 4.5, and 4 percentage points. Glove cannot solve the problem that words have different meanings in different contexts and cannot consider the correlation of all words in the whole sentence. After the GPT model was pre-trained on rice-related question corpus to obtain text features, it achieved context and word order information simultaneously, which improved the BLEU and ROUGE of the neural network. It showed that GPT could solve the problem that words have different meanings in different contexts. Compared with BLEU1, BLEU2, BLEU3, BLEU4, and ROUGE of BERT, the pre-trained model method with GPT was improved by 1.1, 1.7, 3.1, 0.5, and 1.1 percentage points, respectively. Although BERT and GPT are all based on Transformer architecture, BERT is a bidirectional language model [31]. At the same time, GPT is a unidirectional language model, and GPT is more suitable for logistic regression and text generation tasks [32]. Therefore, this paper used the GPT pre-training model and transformed rice-related question-and-answer pairs into word vectors for input into the neural network model.

3.5. Comparative Analysis of the Results of Rice Generative Question and Answer Model

We compared the ResLSTM-Seq2seq model with four other question answering models (LSTM-Seq2seq, BiLSTM-Seq2seq, GRU-Seq2seq, and BiGRU-Seq2seq) in rice-related Q&A data. The above five basic models all used an encoding-decoding structure. However, different neural networks were used in the encoder and decoder to extract text features. The GPT text pre-training model was used in the embedding layer to represent the text of the question-and-answer pairs.

Table 5. BLEU and ROUGE comparison of different deep learning Q&A models.

Question Answering Model	BLEU1 (%)	BLEU2 (%)	BLEU3 (%)	BLEU4 (%)	ROUGE (%)
ResLSTM-Seq2seq	35.1	33.2	32.1	29.8	36.1
LSTM-Seq2seq	32.9	31.8	28.7	27.6	32.7
BiLSTM-Seq2seq	33.7	32.1	29.6	28.1	34.6
GRU-Seq2seq	29.7	28.6	25.3	24.1	30.5
BiGRU-Seq2seq	30.6	28.7	27.6	25.1	31.7

compared five different deep learning question and answer models in BLEU and ROUGE. It can be seen from Table 5 that the model of BiLSTM-Seq2seq was improved by 1.2, 1.7, 0.9, 0.5, and 1.9 percentage points in BLEU1, BLEU2, BLEU3, BLEU4, and ROUGE, respectively, compared with LSTM-Seq2seq. Compared with GRU-Seq2seq, the model of BiGRU-Seq2seq improved by 0.9, 0.1, 1.7, 1, and 1.2 percentage points in BLEU1, BLEU2, BLEU3, BLEU4, and ROUGE. It showed that the answers generated by BiLSTM-Seq2seq and BiGRU-Seq2seq were more similar to the standard answers. The answers generated were more comprehensive due to the use of bidirectional recurrent neural networks in coding and decoding units to extract more accurate rice-related question features. Compared with BiLSTM-Seq2seq and BiGRU-Seq2seq, ResLSTM-Seq2seq had higher BLEUs and ROUGE of 35.1%, 33.2%, 32.1%, 29.8%, and 36.1%. Compared with the bidirectional recurrent neural network, ResLSTM can better extract text features, express text information, and reduce feature loss through residual connection, thus improving the accuracy of rice-related answer generation.

We added the attention mechanism [33] in BiLSTM-Seq2seq, BiGRU-Seq2seq, and ResLSTM-Seq2seq. Attention-BiLSTM-Seq2seq, Attention-BiGRU-Seq2seq, and Attention-ResLSTM-Seq2seq were obtained. We kept other parameters unchanged, the GPT pre-training model was used to obtain the text representation of rice-related question-and-answer pairs, and training was carried out on the data set of rice-related question-and-answer pairs. Figure 8 shows the loss rate trend chart of six rice-related knowledge question and answer models in the rice-related question-and-answer training set. Compared with the other five rice-related generative question and answer models, the Attention-ResLSTM-Seq2seq model converged faster after 30 rounds of training. It can be seen from

Table 6. BlEU and ROUGE comparison of different deep learning Q&A models.

Question Answering Model	BLEU1 (%)	BLEU2 (%)	BLEU3 (%)	BLEU4 (%)	ROUGE (%)
ResLSTM-Seq2seq	35.1	33.2	32.1	29.8	36.1
Attention-ResLSTM-Seq2seq	37.7	36.6	34.9	32.1	37.8
BiLSTM-Seq2seq	33.7	32.1	29.6	28.1	34.6
Attention-BiLSTM-Seq2seq	35.6	33.2	30.9	28.6	35.7
BiGRU-Seq2seq	30.6	28.7	27.6	25.1	31.7
Attention-BiGRU-Seq2seq	32.9	30.8	27.9	25.9	33.8

that the BLEUs and ROUGE of Attention-BiLSTM-Seq2seq have increased by 1.9, 1.1, 1.3, 0.5, and 1.1 percentage points, respectively, compared with BiLSTM-Seq2seq. Compared with BiGRU-Seq2seq, the BLEUs and ROUGE of Attention-BiGRU-Seq2seq are increased by 2.3, 2.1, 0.3, 0.8, and 2.1 percentage points. Compared with ResLSTM-Seq2seq, the BLEUs and ROUGE of Attention-ResLSTM-Seq2seq increased by 2.3, 2.1, 0.3, 0.8, and 1.9 percentage points, respectively. It showed that the attention mechanism can effectively improve the accuracy and comprehensiveness of answers generated by the question-answering model. The attention mechanism can strengthen the weight information of keywords in rice questions between encoder and decoder and make the neural network better understand the semantics of questions and analyze the characteristics of questions. Attention-ResLSTM-Seq2seq achieved the best effect among the six models, with BLEU1, BLEU1, BLEU1, BLEU1 and ROUGE reaching 37.7%, 36.6%, 34.9%, 32.1%, and 37.8%.

4. Discussion

It can be seen from

Table 7. BLEU and ROUGE of different models and different types of rice-related Q&A pairs.

Question Answering Model	Diseases and Insect Pests (%)	Weeds and Pesticides (%)	Storage and Preservation (%)	Cultivation Management (%)	Other (%)
ResLSTM-Seq2seq	35.1	32.7	36.1	29.8	30.2
Attention-ResLSTM-Seq2seq	38.7	34.1	37.1	33.5	31.6
BiLSTM-Seq2seq	34.1	28.7	35.9	27.8	25.1
Attention-BiLSTM-Seq2seq	36.6	28.8	35.9	29.8	26.9
BiGRU-Seq2seq	32.7	25.8	30.1	25.6	25.8
Attention-BiGRU-Seq2seq	32.4	27.7	31.5	26.5	26.7

that compared with five models (BiLSTM-Seq2seq, BiGRU-Seq2seq, ResLSTM-Seq2seq, Attention-BiLSTM-Seq2seq, Attention-BiGRU-Seq2seq), Attention-ResLSTM-Seq2seq had the highest BLEU on the five categories of data (diseases and insect pests, weeds and pesticides, storage and preservation, cultivation management, and others). BLEU was more significant than 31.6%, and the answer generation effect was better than in other models. In the data set with sufficient experimental data on diseases, insect pests, and cultivation management, the average BLEU of this model reached 38.7% and 37.1%. It was significantly higher than the other five answer generation models. It can be seen from Table 7 that the effect of each model in two categories of data sets (diseases and insect pests, cultivation management) was mostly higher than that of other categories, which was because the deep learning answer generation model was in the process of continuous iterative training. The larger the data set, the better the training effect of the model; the BLEUs of this model were 34.1%, 33.5%, and 31.6%, respectively, in the data sets (weeds and pesticides, storage and preservation, and others), which were higher than those of other models. This showed that the Attention-ResLSTM-Seq2seq model can effectively extract the features of short texts for text similarity calculation even if the data are insufficient, and also showed that this model had good robustness.

Table 8. BLEU and ROUGE of six models under different text vector presentation methods.

Model	GPT		BERT		ELMo		Glove
	BLEU	ROUGE	BLEU	ROUGE	BLEU	ROUGE	BLEU	ROUGE
ResLSTM-Seq2seq	32.5	36.1	31.2	33.6	30.9	32.5	28.7	32.1
Attention-ResLSTM-Seq2seq	35.3	37.8	34.1	35.7	32.7	34.8	30.6	31.6
BiLSTM-Seq2seq	30.9	34.6	27.6	31.7	26.6	30.1	25.9	29.6
Attention-BiLSTM-Seq2seq	32.1	35.7	30.5	32.9	30.8	31.6	28.6	30.7
BiGRU-Seq2seq	28.1	31.7	25.7	30.6	23.7	29.7	23.1	28.7
Attention-BiGRU-Seq2seq	29.3	33.8	28.5	32.1	27.8	30.6	26.9	28.5

shows the BLEU and ROUGE of six sequence-to-sequence rice-related question and answer models under the four text representations of GPT, BERT, Glove, and ELMo. The BLEU and ROUGE of GPT text representation were higher than that of the other three text representation methods in six neural network models. Attention-ResLSTM-Seq2seq model achieved the best results in four text representation methods: BLEU and ROUGE, reaching 35.3%, 37.8%, 34.1%, 35.7%, 32.7%, 34.8%, 30.6%, and 31.6%, respectively, and the answer generation effect was better than in the other six neural network models. It can be seen from Table 8 that the average BLEU and ROUGE of BERT and GPT text representation methods were higher than that of Glove and ELMO. Glove text representation ignores polysemous words and long-distance semantic related information in different contexts. It is a static word vector representation method. However, ELMO solves the problem of polysemy dynamically, because ELMO uses LSTM for feature extraction; compared with BERT and GPT, which are based on Transformer as the underlying structure, LSTM’s feature extraction ability was far weaker than Transformer. At the same time, GPT is a one-way language model, while BERT is a two-way language model, so GPT is more suitable for text generation, text translation, and answering question systems. Therefore, GPT as the text representation method can improve the accuracy and comprehensiveness of the answers generated by the rice-related question and answer model.

Table 9. Comparison of response time and effect of three network models.

Model	BLEU (%)	ROUGE (%)	Response Time s
Attention-ResLSTM-Seq2seq	29.8	27.7	21
Attention-BiLSTM-Seq2seq	26.5	25.1	23
Attention-BiGRU-Seq2seq	23.7	23.7	19

shows the response time, BLEU, and ROUGE of three sequence-to-sequence rice-related question answering models based on an attention mechanism in 1500 test sets. The model Attention-ResLSTM-Seq2seq proposed in this paper meets the requirements of quick answers to rice questions, and Attention-BiGRU-Seq2seq had the fastest response time, which is due to the simple structure of the GRU model and few training layers and model parameters. The model Attention-ResLSTM-Seq2seq proposed in this paper can generate 1500 answers in 21 s, its BLEU and ROUGE reach the highest scores of 29.8% and 27.7%, and it had a better effect under the condition of little difference in response time.

5. Conclusions

We used a network based on Attention-ResLSMT-Seq2seq to construct the rice generative question answering model in this paper. Firstly, the GPT pre-training model based on a 12-layer transformer was used to obtain the text representation of rice question-and-answer pairs. Then, ResLSTM was used to extract text features in the encoder and decoder. The attention mechanism was connected between the encoder and decoder to strengthen the weight of keyword feature information in questions. The results showed that the rice question and answer model based on Attention-ResLSMT-Seq2seq had the highest scores of 35.3% and 37.8% in BLEU and ROUGE compared with the other six question and answer models. Based on the work of this study, the following aspects can be further studied:

(1)

The representation of rice knowledge text dataset is still based on manual annotation, which requires a lot of supervision. Semi-supervised and unsupervised models will become one of the main development directions. Reduce the complexity of semantic understanding model processing, improve the efficiency of model processing, and provide edge computing, so as to improve the universality of the model.

(2)

For the fusion of multi-modal agricultural semantic analysis and image analysis, image text features can be used to assist text semantic feature recognition, and complex semantics can be processed through unified dimension mapping and model construction.

(3)

We will transfer the rice knowledge question answering text semantic model to the construction of other crop text semantic models, and we will provide key technical support for the construction of other crop intelligent question answering models in the future.