A Dynamic Attention and Multi-Strategy-Matching Neural Network Based on Bert for Chinese Rice-Related Answer Selection

Abstract

To allow the intelligent detection of correct answers in the rice-related question-and-answer (Q&A) communities of the “China Agricultural Technology Extension Information Platform”, we propose an answer selection model with dynamic attention and multi-strategy matching (DAMM). According to the characteristics of the rice-related dataset, the twelve-layer Chinese Bert pre-training model was employed to vectorize the text data and was compared with Word2vec, GloVe, and TF-IDF (Term Frequency–Inverse Document Frequency) methods. It was concluded that Bert could effectively solve the agricultural text’s high dimensionality and sparsity problems. As well as the problem of polysemy having different meanings in different contexts, dynamic attention with two different filtering strategies was used in the attention layer to effectively remove the sentence’s noise. The sentence representation of question-and-answer sentences was obtained. Secondly, two matching strategies (Full matching and Attentive matching) were introduced in the matching layer to complete the interaction between sentence vectors. Thirdly, a bi-directional gated recurrent unit (BiGRU) network spliced the sentence vectors obtained from the matching layer. Finally, a classifier was employed to calculate the similarity of splicing vectors, and the semantic correlation between question-and-answer sentences was acquired. The experimental results showed that DAMM had the best performance in the rice-related answer selection dataset compared with the other six answer selection models, of which MAP (Mean Average Precision) and MRR (Mean Reciprocal Rank) of DAMM gained 85.7% and 88.9%, respectively. Compared with the other six kinds of answer selection models, we present a new state-of-the-art method with the rice-related answer selection dataset.

1. Introduction

Rice is one of the essential food crops in China, with a wide planting area. Diseases and insect pests are critical factors affecting rice yield and quality in rice production; thus, it is essential to obtain the treatment methods of rice-related problems quickly and accurately in the planting process. With the rapid development of the internet era, questions asking, answering, and discussion in the online question-and-answer (Q&A) community [1], it has become an essential way to seek answers and meet their own information needs. The “China agricultural technology promotion information platform” is a comprehensive service platform specialized in providing an agricultural technology Q&A community, expert guidance, online learning, achievement delivery, and knowledge exchange, among others, which plays a vital role in helping farmers find solutions to problems. The Rice-related Q&A community has accumulated a large number of users and content, and it has produced a large number of low-quality texts, which has dramatically affected users’ efficiency in retrieving satisfactory answers. Therefore, improving the performance of answer quality prediction has become particularly important. The challenge of redundancy, sparsity [2], and the poor standardization of agricultural texts leads to inaccurate text feature extraction and the difficulty in determining the relationship between features. The critical technical link to realize an intelligent agricultural Q&A community is to detect the correct answer from the candidate answer text dataset and return it to the user quickly, automatically, and accurately. Traditional answer selection [3] relies on manual screening and it is challenging to process text data efficiently. Due to human feature selection, it does not automatically and accurately judge the correct answer from a large amount of agricultural text data. Therefore, deep learning [4] and natural language processing technology [5] to realize the intelligent selection of rice-related question answers is a significant problem to be solved by the “China agricultural technology promotion information platform”.

In recent years, deep neural networks have made remarkable achievements in many natural language processing tasks, including answer selection. The core problem is to obtain distinctive semantic features from question-and-answer sentences with deep neural networks. Lei et al. [6] proposed an answer selection model based on a CNN (Convolutional Neural Networks) [7], to extract semantic information from question-and-answer sentences, map them into low-dimensional distributed representation vectors, and learn a semantic matching function, semantic matching of question-and-answer pairs. Severyn et al. [8] also proposed an answer selection model based on a CNN, which simultaneously learns the intermediate representation and final representation of question-and-answer pairs to generate a more refined semantic representation of question-and-answer pairs for semantic similarity matching. Kalchbrenner et al. [9] proposed a DCNN (Dynamic Convolutional Neural Network) for answer selection. To accurately present the semantic information in question-and-answer sentences, the DCNN adopts a multi-layer wide convolution and dynamic K-MAX pooling operation, which retains the word order information in the sentence and the relative position between words and can dynamically deal with question-and-answer pairs of different lengths. Compared with the traditional statistical learning model, the performance based on a CNN has been significantly improved. However, a single CNN cannot effectively extract the contextual semantic correlation information in question-and-answer pairs, which is vital for selecting the most suitable answer from the answer sequence. Some models use RNNs (Recurrent Neural Networks) for answer selection tasks. Wang et al. [10] obtained semantic representations of different granularities in the text through an RNN. Then, they extracted their semantic matching information from the semantic interaction information of different granularities in the question-and-answer text to calculate the semantic matching degree of the question-and-answer pairs. Wang et al. [11] applied the stacked bi-directional LSTM (Long Short-Term Memory) [12] network to capture the forward and backward context information from the question-and-answer sentence. Cai et al. [13] used the bi-directional LSTM network to extract question-and-answer features at various scales. It used three different similarity matrix learning models to obtain the overall similarity of question-and-answer pairs from the local feature similarity.

With the good performance of the attention mechanism [14] in the sentence representation task, researchers applied it to the answer selection task. Tan et al. [15] proposed QA-LSTM-CNN with attention, where the sentence vector representation obtained through the LSTM-CNN network is taken into the attention network. The attention distribution of different units in the sentence vector, and sentence correlation information were obtained. Secondly, there are also different methods and strategies in applying attention mechanisms. Santos et al. [16] proposed the attention pooling networks and the concept of attention pooling. In the case of pairwise sorting or classification with the neural network, the attention pooling can perceive the current input pair, so the information from two input items can directly affect the calculation represented by each other. The target task is regarded as a three-tuple similarity solution in the answer selection task. After the pre-training model [17] was proposed, researchers fine-tuned it to solve the downstream target tasks, and its efficiency has been verified to a great extent. Thirdly, the researchers also optimized the pre-training model under a large corpus, only used as the underlying language representation structure to obtain sentence representation, and combined it with other network structures to model the target task [18].

However, mainstream neural networks have recently focused more on the representation of sentence interaction, which have their shortcomings in the expression of model performance. This paper proposed an effective method to solve the problem of incomplete sentence representation and sentence interaction. To solve the problem of incomplete sentence representation, this paper embedded the word vector into the sentence representation by using a twelve-layer Chinese Bert pre-training model [19]. The word vector was generated by the pre-training language model containing some contextual semantic information. Secondly, this paper also introduced a dynamic attention mechanism to filter the irrelevant information in the sentence vector to understand the sentence vector better. To solve the problem of insufficient information interaction between sentences, this paper introduced a multi-Strategy matching strategy in the matching layer. The obtained sentence vectors interact with each other through two different matching strategies to better capture the semantic correlation information between question and candidate answer. The main work of this paper is as follows:

• A method based on dynamic attention mechanisms and multi-strategy matching was proposed.
• The pre-training model was cleverly transferred to the context embedding layer, which provides a new idea of sentence pair representation.
• The dynamic attention mechanism can effectively screen the irrelevant information in sentence representation and improve sentence expression.
• The model introduced two matching strategies to capture the interaction between the question and candidate answer fully.

2. Materials and Methods

2.1. Corpus Preparation

The rice-related dataset was derived from the Q&A community of the China Agricultural Technology Extension Information Platform [20] in this paper. First, 5000 common rice-related questions were selected, classified into five categories, including 1519, 475, 1503, 376, and 1127 pairs, respectively, regarding diseases and pests, weeds, pesticides, cultivation management, storage and transportation, and OTHERS. There were multiple answers for each question in the Q&A community, and five candidate answers for each question were chosen. Only one of them was positive, marked as 1, and the other four were negative, marked as 0. In addition, 25,000 pairs of the rice-related Q&A dataset were obtained, of which the positive example was 5000, and the negative example was 20,000. Examples of rice-related answer selection samples are shown in

Table 1. Sample of rice-related answer selection dataset.

Question	Answer	Label
What is the reason for the stiff seedling of rice?	Rice seedling stiffness is caused by climate, water quality, temperature and humidity, lack of zinc and so on.	1
What is the reason for the stiff seedling of rice?	Mainly pay attention to the treatment time and solution concentration.	0
What is the reason for the stiff seedling of rice?	The dead seedlings of rice are mainly in the cold soaked field with high groundwater level.	0
What is the reason for the stiff seedling of rice?	This phenomenon is very common and does no harm to rice	0
What is the reason for the stiff seedling of rice?	Rice seedling needs, disinfection, seed drying, seed soaking, Northern Greenhouse breeding	0
What are the transmission routes of Rice False Smut?	The main transmission routes of Rice False Smut are seed, air flow and soil.	1
What are the transmission routes of Rice False Smut?	Rice false smut usually occurs in rice.	0
What are the transmission routes of Rice False Smut?	Rice false smut mainly occurs in the panicle and is mainly infected by bacteria.	0
What are the transmission routes of Rice False Smut?	The control time of Rice False Smut is mainly chemical seed dressing during seedling raising.	0
What are the transmission routes of Rice False Smut?	According to the characteristics of rice false smut, the control effect can reach more than 85%.	0
What are the symptoms of bud rot in rice fungal bacterial wilt?	The tooth root is withered and yellow, with brown mold layer, rotten and easy to break.	1
What are the symptoms of bud rot in rice fungal bacterial wilt?	Strengthen water and fertilizer management and timely prevent and control diseases and pests.	0
What are the symptoms of bud rot in rice fungal bacterial wilt?	Fungal bacterial wilt in rice is a dead seedling of bacterial wilt.	0
What are the symptoms of bud rot in rice fungal bacterial wilt?	Bacterial base rot of rice is mainly transmitted by overwintering pathogens on rice straw, rice pile and weeds.	0
What are the symptoms of bud rot in rice fungal bacterial wilt?	Bacteria are solitary, short rod-shaped and blunt at both ends.	0

.

The rice-related answer selection dataset was constructed, which was different from the answer selection dataset in the general field. It had the following characteristics:

• Highly field-specific. All stationary was in Q&A pairs in rice-related fields, and the boundary of sentence semantics was fuzzy. Therefore, it was difficult to transfer the general domain model to rice diseases and pests for training.
• The distribution of all types of question pairs was not uniform, in which diseases and pests, cultivation management, and OTHERS question-and-answer pairs were most distributed, while weeds and pesticides, and storage and transportation question pairs were relatively limited.
• The sequence of question-and-answer pair was not long. The statistical analysis found that the maximum length of the sample in the rice-related question-and-answer pair set was 150 characters, the average length was 79 characters, and the maximum length and average length of the sample in the Sogou news dataset were 38,872 characters and 1250 characters, respectively [21]. It was challenging to extract semantic information from the features generated in the training process of our short text, which increased the difficulty of model recognition.

2.2. Methods

This paper utilized the Dynamic Attention and Multi-Strategy Matching model, as shown in Figure 1. The model consists of four parts: the embedding layer, multi-strategy interaction layer, polymerization layer, and output layer. We utilized the twelve-layer Chinese Bert model to expand the text feature words and calculated the weighted word vector according to its importance. A dynamic attention with two different filtering strategies was used in the dynamic layer to remove the sentence’s noise effectively, and a multi-strategy interaction layer and polymerization layer were used to extract local features of different granularities of text. Finally, the Softmax function was employed on the extracted feature vectors for output.

2.2.1. Embedding Layer

Text semantic representation has developed from the one hot representation to the current mainstream neural network, including Word2vec [22] and GloVe [23]. Although the problem of word context has been solved to a certain extent with those models, the problem of polysemy where words have different meanings in different contexts still exists. This paper used a twelve-layer Chinese Bert as a language feature extraction and representation, which can obtain the rich grammatical and semantic features of agricultural texts and solve the polysemy problem.

Devlin of the Google team proposed Bert [24] in 2018, and they applied it to various natural language processing tasks. Bert adopts a transformer language model [25] and an encoder and a decoder structure, and it abandons the recursive structure and utilizes the attention mechanism to determine the relationship between the input and output. The transformer model structure is shown in Figure 2.

Each base layer of the transformer model encoder contains two sub-layers: the attention mechanism layer adopts multi-head attention, and the other is a fully connected feedforward neural network. In addition to these two layers, another attention mechanism layer is added to the decoder. Residual connection and layer normalization (LN) are introduced into each model’s sublayer; that is, there is a residual connection around each sublayer in each encoder and decoder, followed by an LN operation. The transformer is the first model built entirely on the self-attention mechanism. Instead of the traditional encoder–decoder architecture, it must combine the inherent mode of CNN or RNN [26]. Compared with the recurrent neural network, the transformer captures a more extended range of information and realizes parallelization, which improves computing speed.

Bert’s pre-training objective function adopts a masked language model (MLM); some words are masked randomly and then predicted in the pre-training process. In this way, the representation of texts in two different directions can be learned. The input of the Bert model is the word embedding, generated by the addition of token embedding, segment embedding, and positional embedding, as shown in Figure 3.

The first mark of each input sentence is [CLS], which corresponds to the transformer’s output. It represents the whole sentence and can be used for downstream classification tasks. Mark [SEP] is used to separate two sentences. For the sentence classification task, only one sentence is the input, which means only one segment vector is used for a single sentence. The language technology platform (LTP) tool developed by Harbin University is used as a word segmentation tool, which blocks all the Chinese characters that make up the same word and then trains to obtain segmented words. This paper used the word vector obtained in the Bert pre-training model as the context embedding vector, spliced the obtained context embedding word vector with the word vector in the word representation layer, and normalized to obtain a new sentence vector representing Q and A.

2.2.2. Dynamic Attention Mechanism

After the sentence vectorization of Q and A, a dynamic attention mechanism was introduced to remove the redundant information in the sentence representation. When processing sequence information with the traditional static attention mechanism, the position of each time-point in the question needs to be compared with positions of all the time-points in the answer, which is repetitive and time-consuming. Therefore, this paper applied two different filtering mechanisms (K-threshold filtering and K-max filtering) based on dynamic attention to remove noise. By applying the filtering strategy in the attention layer, the weight of irrelevant information was set to zero. Then, the final sentence representation was obtained according to the new weight. The weight of important information occupied a more significant proportion in the new sentence representation to improve the computational efficiency in the matching layer.

The initial attention weight matrix was obtained with the traditional attention mechanism method. The semantic similarity between the question unit ${\alpha }_i$ to be matched and the candidate answer unit ${\beta }_j$ was calculated. For the calculation of the similarity scoring function ${\omega }_{ij}$ (also known as attention scoring function), this paper adopted point multiplication operation, and the expression is as follows:

$${\omega }_{ij}={\alpha }_i·{\beta }_j$$

(1)

After obtaining the similarity, the Softmax function was used for normalization, and the normalized result was taken as the attention weight of the question unit in the candidate answer sequence.

Similarly, the attention weight of candidate answer unit ${\beta }_j$ in the question sequence was calculated. Assuming the weight coefficients of ${\beta }_j$ and ${\alpha }_i$ after normalization are ${\phi }_{ij}^{\alpha }$ and ${ \phi }_{ij}^{\beta }$, the expression is as follows:

$${\phi }_{ij}^{\beta }=\frac{\exp ({\omega }_{ij})}{{\displaystyle \sum _{k=1}^{l_{\alpha }}\exp ({\omega }_{kj})}}$$

(2)

$${\phi }_{ij}^{\alpha }=\frac{\exp ({\omega }_{ij})}{{\displaystyle \sum _{k=1}^{l_{\beta }}\exp ({\omega }_{kj})}}$$

(3)

After obtaining the weight coefficient of each word in the sentence, the filtering strategy is obtained with the proposed way above. For question Q, it is assumed that the weight matrix obtained can be expressed as $\phi =[{\phi }_{1j},\cdots ,{\phi }_{ij},\cdots ,{\phi }_{l_{\alpha }j}{]}^T$, where ${\phi }_{ij}$ is the abbreviation of ${\phi }_{ij}^{\beta }$. In the K-threshold filtering, assuming a threshold K, representing the correlation strength, retains a weight greater than the threshold K, the weight less than the threshold K is set to zero, and then the weight coefficient of each word is calculated according to the new weight. The expression is as follows:

$$\left\{\begin{array}{l}{\phi }_{ij}={\phi }_{ij}\hspace{1em}{\phi }_{ij}\ge K, \\ {\phi }_{ij}=0\hspace{1em} {\phi }_{ij}<K.\end{array}\right.$$

(4)

$${\phi }_{ij}=\frac{{\phi }_{ij}}{{\displaystyle \sum _{k=1}^{l_{\alpha }}{\phi }_{ij}}}$$

(5)

The filtering of irrelevant information is realized in sentences by setting different threshold values of K; moreover, the size of K can be dynamically adjusted accordingly.

In the K-MAX filtering method, the weight coefficients were arranged in the weight matrix into descending order to obtain the weight sequence table T. Secondly, the weight value with a significant weight coefficient was retained in the first K bits and the weight after K bits was set to zero. Then, the weight coefficient of each word was calculated according to the new weight. The expression is as follows:

$$\left\{\begin{array}{l}{\phi }_{ij}=\frac{{\phi }_{ij}}{{\displaystyle \sum _i^T{\phi }_{ij}}}\hspace{1em}i\in T, \\ {\phi }_{ij}=0\hspace{1em}\hspace{1em} j\notin T.\end{array}\right.$$

(6)

For answer A, the same method was used to obtain the new weight coefficient of each word in the answer A. According to the new weight coefficient, the soft alignment vectors ${\sigma }_j^{\beta }$, ${\sigma }_j^{\alpha }$ of each word in Q and A was calculated, respectively, that is, the weighted sum was calculated. The calculation formula is as follows:

$${\sigma }_j^{\beta }={\displaystyle \sum _{i=1}^{l_q}{\phi }_{ij}^{\beta }}{\alpha }_i,\hspace{1em}{\sigma }_i^{\alpha }={\displaystyle \sum _{j=1}^{l_q}{\phi }_{ij}^{\alpha }}{\beta }_j$$

(7)

where ${\sigma }_j^{\beta }$ is the soft alignment vector corresponding to ${\beta }_j$ in $\alpha$, and the expression of ${\sigma }_i^{\alpha }$ is similar. Finally, the output of this layer is expressed as $Q:[{\sigma }_1^{\beta },{\sigma }_2^{\beta },\cdots ,{\sigma }_j^{\beta },\cdots ,{\sigma }_{l_{\beta }}^{\beta }]$, $A:[{\sigma }_1^{\alpha },{\sigma }_2^{\alpha },\cdots ,{\sigma }_i^{\alpha },\cdots ,{\sigma }_{l_{\alpha }}^{\alpha }]$.

2.2.3. Multi-Strategy Interaction Layer

Most previous sentence matching methods use a single operation method to interact between sentence units, where it is challenging to capture the deep-seated semantic association information between sentences. Therefore, this paper used the multi-Strategy matching to exchange information between question and answer. Applying various matching strategies to calculate the information interaction between sentence units will produce better matching results. Two different matching strategies for comparison in different directions were adopted here: Full matching and Attentive matching.

First, we defined a multi-dimensional cosine matching function $f_x$, compared the matching degree between the two vectors through the matching function, and returned the result to $x$:

$$x=f_x(a_1,a_2;V)$$

(8)

where $a_1$ and $a_2$ are d-dimensional vectors, $V\in {\Re }^{l\times d}$ is a trainable parameter matrix, $l$ represents the number of matching angles, the matching result $x=[x_1,\cdots ,x_k,\cdots ,x_l]$ is an $l$-dimensional vector, and the element $x_k\in x$ represents the matching result of the k-th matching angle. It is obtained by calculating the cosine similarity between the two weighted vectors. The expression is as follows:

$$x_k=\mathrm{cosine}(V_k·a_1,V_k·a_2)$$

(9)

where $V_k$ represents the k-th row of parameter matrix $V={[V_1,\cdots ,V_k,\cdots ,V_l]}^T$, controls the k-th matching angle, and assigns weights to different dimensions of d-dimensional space.

To thoroughly compare each sentence unit vector of the question sentence with all unit vectors of the candidate answer, this layer introduces two different matching strategies based on the matching function $f_x$. In the Full matching strategy, each sentence unit vector of the question is compared with the last sentence unit vector in the candidate answer.

In the whole matching layer, a bi-directional gated recurrent unit network [27] is used to realize parameter transfer, so forward and backward transfer is distinguished during comparison. Assuming that the return result is $x_j^{\beta }$ after comparing the sentence unit ${\sigma }_j^{\beta }$ in question Q with the last sentence unit in answer A, then:

$${(\stackrel{\rightharpoonup }{x_j^{\beta }})}^{full}=f_x(\stackrel{\rightharpoonup }{{\sigma }_j^{\beta }},\stackrel{\rightharpoonup }{{\sigma }_{l_{\alpha }}^{\alpha }};V^1)$$

(10)

$${(\overleftarrow{x_j^{\beta }})}^{full}=f_x(\overleftarrow{{\sigma }_j^{\beta }},\overleftarrow{{\sigma }_1^{\alpha }};V^1)$$

(11)

In the Attentive matching strategy, the cosine similarity of each sentence unit in question ${\sigma }_j^{\beta }$ and answer A to be matched is first calculated. The similarity can be taken as the attention weight of the question unit in answer sentence A, then the obtained weight ${\beta }_{ij}$ is further normalized, and, finally, all sentence units in the answer are weighted and summed. The attention vector representation ${\sigma }_j^{mean}$ corresponding to the question unit is obtained.

$$\stackrel{\rightharpoonup }{{\beta }_{ij}}=\cos \mathrm{ine}(\stackrel{\rightharpoonup }{{\sigma }_j^{\beta }},\stackrel{\rightharpoonup }{{\sigma }_i^{\alpha }})\hspace{1em}i=1,\cdots ,l_{\alpha }$$

(12)

$$\overleftarrow{{\beta }_{ij}}=\mathrm{cosine}(\overleftarrow{{\sigma }_j^{\beta }},\overleftarrow{{\sigma }_i^{\alpha }})\hspace{1em}i=1,\cdots ,l_{\alpha }$$

(13)

$$\overrightarrow{{\sigma }_j^{mean}}={\displaystyle \sum _{i=1}^{l_{\alpha }}\frac{\overrightarrow{{\beta }_{i,j}}}{{\displaystyle \sum _{i=1}^{l_{\alpha }}\overrightarrow{{\beta }_{i,j}}}}}·\overrightarrow{{\sigma }_i^{\alpha }}$$

(14)

$$\overleftarrow{{\sigma }_j^{mean}}={\displaystyle \sum _{i=1}^{l_{\alpha }}\frac{\overleftarrow{{\beta }_{i,j}}}{{\displaystyle \sum _{i=1}^{l_{\alpha }}\overleftarrow{{\beta }_{i,j}}}}}·\overleftarrow{{\sigma }_i^{\alpha }}$$

(15)

The question unit ${\sigma }_j^{\beta }$ to be matched is matched and compared with the obtained attention vector representation ${\sigma }_j^{mean}$, and the expression is as follows:

$$(\overrightarrow{x_j^{\beta }}{)}^{att}=f_x(\overrightarrow{{\sigma }_j^{\beta }},\overrightarrow{{\sigma }_j^{mean}};V^3)$$

(16)

$$(\overleftarrow{x_j^{\beta }}{)}^{att}=f_x(\overleftarrow{{\sigma }_j^{\beta }},\overleftarrow{{\sigma }_j^{mean}};V^4)$$

(17)

The above two matching strategies are applied to process the sentence unit in question Q, and each sentence unit contains four vectors. Further, all the vectors are spliced in each sentence unit to obtain the final representation $x_j^{\beta }$ of the sentence unit.

$$x_j^{\beta }=\mathrm{concatenate}[(\overrightarrow{x_j^{\beta }}{)}^{full},(\overleftarrow{x_j^{\beta }}{)}^{full},(\overrightarrow{x_j^{\beta }}{)}^{att},(\overleftarrow{x_j^{\beta }}{)}^{att}]$$

(18)

Similarly, the result $x_i^{\alpha }$ is calculated. Finally, the final representation Q: $[x_1^{\beta },x_2^{\beta },\cdots ,x_j^{\beta },\cdots ,x_{l_{\beta }}^{\beta }]$ and A: $[x_1^{\alpha },x_2^{\alpha },\cdots ,x_i^{\alpha },\cdots ,x_{l_{\alpha }}^{\alpha }]$ of this layer are presented.

2.2.4. Polymerization Layer

The drive of this layer was to integrate the forward and backward information obtained by the matching layer, splice the question representation and candidate answer representation, and generate the sentence vector containing the question and candidate answer information representation. In this paper, a bi-directional gated recurrent unit network was used to fuse the sequence information of forward sentence unit vectors $\overrightarrow{x_j^{\beta }}$ and $\overrightarrow{x_i^{\alpha }}$, and backward sentence unit vectors $\overleftarrow{x_j^{\beta }}$ and $\overleftarrow{x_i^{\alpha }}$ transmitted by the matching layer. Each sentence unit is passed through the BiGRU network to obtain a new sentence unit vector $x_j^{\beta }$, and then the last unit vector in the forward transmission and backward transmission is taken as the connection vector. After, the connected vectors $r_{\alpha }$ and $r_{\beta }$ are further spliced to obtain the output representation of this layer, the scoring vector score. In the output layer, this paper used the scoring vector to sort the candidate answers, where M is the trainable parameter variable of the output layer.

$$\overrightarrow{x_j^{\beta }}=GRU(\overrightarrow{x_1^{\beta }},\cdots ,\overrightarrow{x_j^{\beta }})\hspace{1em}j=1,\cdots ,l_{\beta }$$

(19)

$$\overleftarrow{x_j^{\beta }}=GRU(\overleftarrow{x_1^{\beta }},\cdots ,\overleftarrow{x_j^{\beta }})\hspace{1em}j=1,\cdots ,l_{\beta }$$

(20)

$$r_{\alpha }=GRU(\overleftarrow{{\mu }_1^{\beta }},\overrightarrow{x_{l_{\beta }}^{\beta }})$$

(21)

$$r_{\beta }=GRU(\overleftarrow{{\mu }_1^{\alpha }},\overrightarrow{x_{l_{\alpha }}^{\alpha }})$$

(22)

$$Score={[r_{\alpha },r_{\beta }]}^{T}M$$

(23)

2.2.5. Output Layer

To consider the correlation between matching questions and candidate answers, the list wise method is used. The corresponding candidate answer set is represented as $A\{A_1,A_2,\cdots ,A_N\}$ and the label set is $Y\{Y_1,Y_2,\cdots ,Y_N\}$, and the score vector is calculated by the question Q and each candidate answer. Finally, the Softmax classifier is applied to obtain the final output vector S:

$$S=\mathrm{Softmax}([Scor{e}_1,\cdots ,Scor{e}_j,\cdots ,Scor{e}_N])$$

(24)

$$Scor{e}_j=\mathrm{model}[Q,A_j]$$

(25)

Finally, the Kullback–Leibler (KL) divergence loss function was used, and the expression is as follows:

$$Y=\frac{Y}{{\displaystyle \sum _{i=1}^N{y}_i}}$$

(26)

$$Loss=\frac{1}{n}{\displaystyle \sum _1^{n}KL(S||Y)}$$

(27)

3. Results

3.1. Hardware, Software Environment, and Evaluation Indicators

The server’s hardware environment was NVIDIA Corporation device 1e04 (Rev A1), the GPU was an NVIDIA GeForce RTX 2080TI, and both the research and control experiments were performed in the Ubuntu 18.04 environment. For training, the deep learning framework Pytorch was utilized in conjunction with Cuda10.1. The training and validation sets’ network batch sizes were set to 16 and 32, respectively, during the experiment design and control phase. All network models were given a total of 50 iterations. A total of 25,000 pairs of the rice-related Q&A dataset were divided into the training set and test set according to the ratio 9:1. There were 22,500 training sets and 2500 test sets.

In the experiment, Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR) were used as the evaluation indexes of model performance.

MRR: take the position reciprocal of the standard answer in the ranking results given as the evaluation index, and then average all the questions. The calculation formula is as follows:

$$MRR=\frac{1}{\left|Q\right|}{\displaystyle \sum _{i=1}^{\left|Q\right|}\frac{1}{ran{k}_i}}$$

(28)

where $\left|Q\right|$ represents the number of questions, and $ran{k}_i$ represents the position where the correct answer first appears in the predicted answers returned by the model for a certain question. If all the correct answers do not appear, set $\frac{1}{ran{k}_i}$ equal to 0.

Map: first, calculate the average precision of standard answers in each question, and then average the average precision of standard answers calculated for all questions. MAP is calculated as follows:

$$MAP=\frac{1}{\left|Q\right|}{\displaystyle \sum _{i=1}^{\left|Q\right|}AveP(C_i,A_i)}$$

(29)

$$AveP(C,A)=\frac{{\displaystyle {\sum }_{k=1}^n(P(k)·rel(k))}}{\min (m,n)}$$

(30)

$$rel(k)=1,\hspace{1em}{ \mathrm{C}}_k\in \mathrm{StandardAnswer}$$

(31)

$$rel(k)=0,\hspace{1em}{ \mathrm{C}}_k\notin \mathrm{StandardAnswer}$$

(32)

where $AveP$ represents the average precision; $k$ represents the ranking of $n$ prediction answers returned; $m$ is the number of standard answers; $n$ is the number of predicted answers; if $\min (m,n)=0$, set $AveP(C,A)$ to 0; $P(k)$ is the prediction precision up to $k$; $rel(k)$ indicates whether the k-th prediction answer is the correct answer. If yes, it is recorded as 1; otherwise, it is recorded as 0.

3.2. Text Vectorization Processing and Analysis

We used the twelve-layer Chinese Bert model to vectorize the rice-related question-and-answer dataset in this paper. At first, it was compared with the GloVe, TF-IDF [28], and Word2vec vectorization model. The text features obtained from the training of the four models were directly put into the Softmax classifier with a full connection layer. It can be seen from

Table 2. Model matching effect under different embedding layers.

Model	MAP (%)	MRR (%)
TF-IDF	63.7	65.9
Glove	71.8	73.9
Word2Vec	72.6	74.7
BERT	75.3	78.2

that four different word vector conversion tools were used in the embedding layer, and the twelve-layer Chinese Bert pre-training model achieved the highest MAP and MRR, reaching 75.3% and 78.2%, respectively. The effect of the TF-IDF method was worst, which indicates that TF-IDF mainly considers the importance of word frequency and the position information of words, and TF-IDF ignores the relationship between words. After Bert pre-training, the method improved the MAP and MPP compared with word2vec by 2.7% and 3.5%, respectively, which indicates that word2vec considers the surrounding information of words but ignores the word order problem. Due to window size limitations, it cannot consider the correlation of all words in the whole sentence. After the text representation obtained by Bert, it can consider the context and word order information simultaneously, which improves the MAP and MPP of the neural network. It shows that Bert can solve the problem where words have different meanings in different contexts. Therefore, the Bert pre-training model was used to transform the rice-related answer selection dataset in this paper; then, word vectors were used to input into the neural network model.

3.3. Experiment

The DAMM model proposed in this paper was compared with six other answer selection models (LSTM [12], Coattention-BiLSTM [13], CNN [29], Attention-CNN [30], RNN [10], and Attention_BiRNN [31]) on the rice-related answer selection dataset, and all used the twelve-layer Chinese Bert model to vectorize the text.

Table 3. Effects of different models on rice-related answer selection dataset.

Model	MAP (%)	MRR (%)
LSTM	76.6	78.5
Coattention-BiLSTM	82.7	83.7
CNN	78.6	79.9
Attention-CNN	79.5	81.3
RNN	81.7	82.1
Attention_BiRNN	83.6	85.7
DAMM	85.7	88.9

shows the comparison of seven different deep learning models in MAP and MRR. Compared with LSTM, the MAR and MPP of BiLSTM based on the self-attention mechanism increased by 4.7 and 4.4 percentage points, respectively; the MAP and MRR of the convolutional neural network model based on the attention mechanism (Attention-CNN) increased by 3.2 and 3.9 percentage points compared with CNN model, respectively; and the MAP and MRR of BiRNN based on the attention mechanism increased by 1.9% and 3.6% compared with RNN model, respectively. The model DAMM proposed in this paper obtained the highest MAP and MRR, reaching 85.7% and 88.9%, respectively; the MAP and MRR were significantly better than the other six deep learning models. Figure 4 shows the comparison of the MAP with seven answer selection models during the training process.

It can be seen from

Table 4. Effect of different models on rice-related answer selection dataset.

Model	MAP (%)					MRR (%)
	1	2	3	4	5	1	2	3	4	5
LSTM	84.2	73.7	75.8	79.7	79.7	83.7	77.8	79.7	75.1	81.1
Coattention-BiLSTM	83.1	81.9	81.6	78.5	78.7	85.7	81.1	84.1	81.1	86.7
CNN	74.1	78.4	79.1	79.7	76.7	82.1	74.1	77.7	79.6	84.1
Attention-CNN	86.6	81.2	81.9	72.6	75.3	84.7	78.6	83.9	78.7	85.6
RNN	82.3	77.7	81.5	79.3	77.7	83.5	79.3	82.3	78.1	82.3
Attention_BiRNN	83.3	79.9	82	83.3	79.9	85.1	78.9	83.4	82	83.9
DAMM	89.1	82.8	87.1	83.6	86.1	93.1	83.7	92.9	86.7	92.9

Note: 1, 2, 3, 4, and 5 represent the data of diseases and pests, weeds and pesticides, cultivation management, storage and transportation, and OTHER five categories, respectively.

that, compared with LSTM, Coattention-BiLSTM, CNN, Attention-CNN, RNN, and Attention_BiRNN, DAMM had the highest matching performance in the dataset of five categories (diseases and insect pests, weeds and pesticides, cultivation management, storage, and transportation, and OTHER). The MAP and MRR of DAMM were more significant than 82.8% and 83.7%, respectively, and the overall classification effect was better than those of the other models. The MAP of the DAMM model was slightly higher than those of the other models with sufficient data of diseases and pests, cultivation management, and OTHER. This is because, in the process of iterative training, the more significant the dataset, the better the training effect of the model. In the two categories (weeds and pesticides, storage and transportation) with fewer data, the MAP and MRR of DSMM models were significantly higher than those of the other six models, which indicates that the DAMM model can still effectively extract the features with short text under the condition of insufficient data. It also shows that the model has good robustness.

Table 5. Response time and precision of four network models.

Model	Response Time (s)	MAP (%)	MRR (%)
Coattention-BiLSTM	14	80.7	82.6
Attention-CNN	10	79.6	81.5
Attention_BiRNN	13	81.5	82.7
DAMM	11	83.7	86.9

shows the response time, MAP, and MRR of four neural network models based on the attention mechanism on 2500 test sets, which meets the requirements for quick answer selection of the rice-related Q&A dataset. Attention-CNN is fastest in response time due to the simple structure of the Attention-CNN model, fewer training layers, and fewer model parameters. The model DAMM proposed in this paper was able to accurately judge rice-related question and answer pairs in the test set of 2500 question and answer pairs in 11 s; the MAP and MRR reached 83.7% and 86.9%, respectively. It had the best performance compared with the other three models.

4. Discussions

4.1. Effectiveness of Embedding layer

Table 6. Effect of different models on rice-related answer selection dataset.

Model	BERT		Word2vec
	MAP (%)	MRR (%)	MAP (%)	MRR (%)
LSTM	76.6	78.5	74.7	75.7
Coattention-BiLSTM	82.7	83.7	77.6	79.2
CNN	78.6	79.9	75.3	76.2
Attention-CNN	79.5	81.3	76.5	77.2
RNN	81.7	82.1	78.1	79.8
Attention_BiRNN	83.6	85.7	79.7	81.2
DAMM	85.7	88.9	82.5	83.7

shows the MAP and MRR of seven neural network models (LSTM, Coattention-BiLSTM, CNN, Attention-CNN, RNN, and Attention_BiRNN) with the BERT text representation and Word2vec text representation in the rice-related question-and-answer dataset. This paper proposed the Bert text representation method, which had a higher MAP and MRR than the Word2vec text representation method under seven neural network models. The DAMM model achieved the highest MAP and MRR with Bert text representation and word2vec text representation, reaching 85.7%, 88.9%, 82.5%, and 83.7%, respectively. The answer selection effect was significantly better than those of the other six neural network models. It can be seen from Table 5 that the Bert text representation method improved MAP and MRR in each group of comparative experiments. This is because the word2vec text representation method ignores the problems of polysemy and long-distance semantic association information in different contexts. The Bert text representation method can solve the above problems to improve the effectiveness of the answer selection in the rice-related question-and-answer dataset.

4.2. Effectiveness of Dynamic Attention Mechanism

Table 7. Effect of different models on rice-related answer selection dataset.

Label	Model	MAP (%)	MRR (%)
1	DAMM without K-threshold and K-max	76.6	78.5
2	(DAMM without K-threshold and K-max)+attention	79.5	80.5
3	DAMM without K-threshold	81.1	83.2
4	DAMM without K-max	82.5	84.7
5	DAMM	85.7	88.9

shows a set of experiments undertaken to prove the effectiveness of the dynamic attention mechanism in the DAMM model proposed in this paper. After removing the k-threshold and K-MAX attention mechanism from the DAMM, model 1 (DAMM without k-threshold and K-MAX) was obtained. It can be seen from Table 7 that, compared with the DAMM model, the MAP and MRR of model 1 on the rice-related answer selection dataset decreased by 9.1% and 10.4%, respectively. After adding the attention mechanism to model 1, model 2 ((DAMM without k-threshold and K-max) + attention) was obtained; compared with model 1, the MAP and MRR of model 2 increased by 2.9% and 2%, respectively, proving that adding an attention mechanism can improve the model’s effectiveness because the attention mechanism can strengthen the weight of keywords in the process of answer selection. Finally, after removing the K-threshold and K-max attention mechanisms in model DAMM, model 3 and model 4 were obtained; compared with model 2, the MAP and MRR increased by 1.6%, 2%, 2.7%, and 4.2%, respectively. This is because the application of the filtering strategy of irrelevant information can reset to zero; then, according to the new weight, to obtain the final sentence representation, compared with the attention mechanism, the dynamic mechanism can make the weight of important information in the new sentence occupy a more significant proportion of allocation, raising the computation efficiency of the matching layer.

4.3. Effectiveness of Multi-Strategy Interaction Layer

In order to explore the impact of the matching layer, two strategies were removed in the matching layer to obtain model 1. Then, each strategy was removed alternatively to obtain model 2 and model 3 while ensuring that other factors remained unchanged. The impact of different matching strategies was evaluated according to the difference between the obtained MAP and MRR indicators. It can be seen from

Table 8. Effect of different models on rice-related answer selection dataset.

Label	Model	MAP (%)	MRR (%)
1	DAMM without Full-Matching and Attentive-Matching	79.9	81.1
2	DAMM without Full-Matching	82.6	84.5
3	DAMM without Attentive-Matching	82.7	85.7
4	DAMM	85.7	88.9

that, compared with the DAMM model, the MAP and MRR of model 1 on the rice-related answer selection dataset decreased by 5.8% and 7.8%. Compared with model 1, the MAP and MRR of model 2 and model 3 on the rice-related answer selection dataset increased by 2.7%, 3.4%, 2.8%, and 4.6%, respectively. This indicated that adding one of the two strategies can improve the effectiveness. Full-matching and Attentive Matching strategies strengthen the interaction between knowledge representations of question-and-answer pairs. The model can accurately obtain the critical information related to the semantic information of question-and-answer pairs in the knowledge base. Compared with the DAMM model, the MAP and MRR of model 2 and model 3 on the rice-related answer selection dataset decreased by 3.1%, 4.4%, 3%, and 3.2%, respectively. This shows that adding two matching strategies can improve the model’s effectiveness.

5. Conclusions

In this paper, a dynamic attention mechanism and multi-Strategy matching model was proposed to complete the rice-related answer selection task. The model combined the advantages of the pre-training model in language representation. At the same time, a dynamic attention mechanism was introduced to remove some irrelevant and redundant information to obtain sentence representation efficiently. Secondly, multi-Strategy matching was introduced to compare different units between sentences, and the semantic association between target sentences was fully obtained. The experimental results showed that the overall performance of the proposed model was better than that of the baseline model. In addition, the model proposed in this paper can be applied not only to answer selection tasks but also to matching and sorting tasks—for example, automatic question answering, machine reading comprehension, and dialogue system. Currently, this paper migrated the trained Bert pre-training model because there is an out-of-vocabulary (OOV) problem in the text field that cannot be ignored. In future work, we will first train the Elmo model on our dataset and embed the new word vector representation obtained from the training into the context embedding layer; we will explore the influence of the corpus knowledge field on word vector representation. In addition, we will further explore the impact of other existing pre-training models on downstream processing tasks, such as general pre-training (GPT) and universal language model fine-tuning (ULMFIT), which are currently well used; and then analyze and study the representation mechanism, applicable scenarios, and migration strategies of the language model. Finally, we will also explore the matching strategy of deep interaction between sentences.