A New Chinese Named Entity Recognition Method for Pig Disease Domain Based on Lexicon-Enhanced BERT and Contrastive Learning

Featured Application

Our work provides reliable technical support for the information extraction of pig diseases in Chinese. It can be applied toother domain-specific fields, thereby facilitating seamless adaptation for named entity identification across diverse contexts.

Abstract

Named Entity Recognition (NER) is a fundamental and pivotal stage in the development of various knowledge-based support systems, including knowledge retrieval and question-answering systems. In the domain of pig diseases, Chinese NER models encounter several challenges, such as the scarcity of annotated data, domain-specific vocabulary, diverse entity categories, and ambiguous entity boundaries. To address these challenges, we propose PDCNER, a Pig Disease Chinese Named Entity Recognition method leveraging lexicon-enhanced BERT and contrastive learning. Firstly, we construct a domain-specific lexicon and pre-train word embeddings in the pig disease domain. Secondly, we integrate lexicon information of pig diseases into the lower layers of BERT using a Lexicon Adapter layer, which employs char–word pair sequences. Thirdly, to enhance feature representation, we propose a lexicon-enhanced contrastive loss layer on top of BERT. Finally, a Conditional Random Field (CRF) layer is employed as the model’s decoder. Experimental results show that our proposed model demonstrates superior performance over several mainstream models, achieving a precision of 87.76%, a recall of 86.97%, and an F1-score of 87.36%. The proposed model outperforms BERT-BiLSTM-CRF and LEBERT by 14.05% and 6.8%, respectively, with only 10% of the samples available, showcasing its robustness in data scarcity scenarios. Furthermore, the model exhibits generalizability across publicly available datasets. Our work provides reliable technical support for the information extraction of pig diseases in Chinese and can be easily extended to other domains, thereby facilitating seamless adaptation for named entity identification across diverse contexts.

1. Introduction

Pig diseases, such as African swine fever or highly pathogenic porcine reproductive and respiratory syndrome (PRRS), have become major issues threatening the healthy development of the pig industry in China. These diseases have not only caused significant economic losses to society but have also impacted food safety. In the context of large-scale and intensive pig breeding practices, it is of great significance to establish intelligent diagnostic and preventive measures for pig diseases. Early prevention and timely diagnosis are pivotal for maintaining swine health and mitigating potential losses.

Named Entity Recognition (NER) assumes a critical role in this endeavor by identifying specific entities within textual corpora, serving as the cornerstone for numerous downstream tasks in natural language processing. These tasks include but are not limited to information retrieval, intelligent question answering, and knowledge graph construction. The extraction of entity information from the pig disease domain, including diseases, disease sites, symptoms, drugs, etc., can rapidly extract key information from vast quantities of unstructured data. This process provides reliable core data for intelligent diagnosis, dialogue consultation, drug recommendation among veterinarians and farmers, and other application scenarios related to pig diseases. Additionally, it offers veterinarians more comprehensive and accurate diagnostic information, thereby improving the efficiency and accuracy of diagnoses.

However, the existing entity recognition methods mostly focus on the recognition of a person, location, organization, etc. Given the pressing need to bolster disease surveillance and management in swine, there arises an urgent imperative to develop specialized NER methodologies tailored to the specific lexicon of pig disease terminology in Chinese.

The early NER methods include rule-based recognition methods and statistics-based machine learning recognition methods. In recent years, with the rapid development of neural networks, methods of deep learning are more suitable for the task of NER and have become the mainstream method [1,2,3,4,5].

The deep learning-based NER method can learn more complex features and achieve good results. In contrast to the rule-based and statistics-based approaches, deep learning-based NER methods do not necessitate an abundance of artificial features. Therefore, deep learning-based methods have attracted wide interest from researchers. Common deep learning models include convolutional neural network (CNN), recurrent neural network (RNN), graph neural network (GNN), deep neural network (DNN), generative adversarial network (GAN), long short-term memory network (LSTM), Transformer and BERT (bi-directional encode representation from transformers), and so on [1,6]. Deep learning models have become the mainstream method and achieve state-of-the-art results in NER. However, the scalability of deep learning models applied in a specific domain remains a significant challenge. The primary challenges are the presence of specialized vocabulary and domain-specific knowledge, coupled with the scarcity of labeled datasets. Additionally, Chinese named entity recognition faces difficulties such as ambiguous boundaries and entity ambiguity [1,2].

The lexicon-based NER method can effectively avoid segmentation errors and improve the accuracy of entity boundary recognition by integrating potential word information into feature vectors. Currently, a large number of lexicon-enhanced Chinese entity extraction methods have been proposed, with better performance than methods based on character embedding or word embedding [7,8]. Lattice-LSTM [9] has achieved new benchmark results on several public Chinese NER datasets. However, the Lattice-LSTM model architecture is complex, which limits its application in many industrial areas requiring real-time NER responses. A convolutional neural network-based method that incorporates lexicons using a rethinking mechanism was proposed, which can model all the characters and potential words that match the sentence in parallel [10]. A lexicon-based graph neural network with global semantics was proposed to tackle word ambiguities. In this model, the lexicon knowledge is used to connect characters to capture the local composition, while a global relay node can capture global sentence semantics and long-range dependency [11]. A Lexicon-Enhanced BERT (LEBERT) for Chinese sequence labeling was put forward [12]. The model integrates external lexicon knowledge into BERT layers directly with a Lexicon Adapter layer and achieves better performance than both lexicon-enhanced models and the BERT baseline in Chinese datasets. More character–word association models have been proposed, such as SoftLexicon [13], FLAT [14], and PLTE [15].

The pre-trained NER method effectively leverages deep bidirectional contextual information. It demonstrates superior performance with shorter training times, reduced labeling data requirements, and improved results compared to traditional models. Currently, BERT [16] is widely used, followed by ELMo [17], RoBERTA [18], ERNIE [19], ALBERT [20], and others. At present, the pre-trained models and lexicon are integrated by utilizing their respective strengths. Li proposed Flat-Lattice Transformer for Chinese NER, which converts the lattice structure into a flat structure consisting of spans [14]. Li proposed the LEBERT-BiLSTM-CRF model for elementary mathematics text NER, which integrates external lexicon knowledge into BERT layers directly with a lexicon adapter layer and performs better than other NER models [21].

Contrastive learning acquires feature representations of samples by comparing positive and negative samples in feature space. This approach has garnered significant attention in the fields of computer vision (CV) and natural language processing (NLP). The ConSERT (Contrastive Framework for Self-supervised Sentiment Representation Transfer) and SimCSE (Simple Contrastive Learning of Sentiment Embedding) models, which use different data enhancement methods and comparative learning loss function to learn the representation of sentences, obtain SOTA results on the task of text semantic similarity [22,23]. Contrastive learning with prompt guiding for few-shot NER (COPNER) was proposed and outperforms state-of-the-art models with a significant margin in most cases. This method introduces category-specific words that COPNER composed out of prompts as supervised signals for contrastive learning to optimize entity token representation [24]. Moreover, Named Entity Recognition in low-resource scenarios based on contrastive learning has also received considerable attention [25,26,27]. He proposed a novel prompt-based contrastive learning method for few-shot NER without template construction and label word mappings [25]. Li proposed a multi-task learning framework CLINER for few-shot NER [26].

In the field of livestock husbandry, text mining, Named Entity Recognition (NER), intelligent question-and-answer systems, and artificial intelligence (AI) technologies have been gradually applied. However, this field faces numerous challenges, including the prevalence of technical terms, complex knowledge structures, fine knowledge granularity, and a lack of labeled datasets [28]. Seok created a BERT-DIS-NER model that adds a CRF layer to BERT for disease named entity recognition and used syllable unit-based named entity recognition that can reflect the characteristics of disease names. The F1-score is 0.81, trained with human data and fine-tuned with animal data [29]. Kung designed and implemented an intelligent knowledge question-and-answer system for pig farming based on bi-GRU and SNN methods, combined with the LTSM deep learning method [30].

NER methods have found extensive applications in the agricultural domain and other specific fields [31,32,33,34,35,36]. Nonetheless, there remains an apparent gap in current research concerning the accurate recognition of named entities within the domain of pig diseases in Chinese. Pig disease data are characterized by complex entities, fuzzy boundaries, and domain-specific vocabulary, which encompasses specialized terminologies drawn from the domains of animal husbandry and veterinary science.

Furthermore, the resources in the field of pig diseases are confined and dispersed, exacerbating the scarcity of publicly available benchmark corpora and labeled datasets specific to this domain in Chinese. While considerable research has been devoted to NER systems in human medicine [37,38,39,40], it remains impractical to directly transfer such models to the domain of pig diseases due to the domain-specific rules and vocabulary governing this domain. Hence, named entity recognition in the field of pig diseases needs to be further explored. A model of Pig Disease Chinese Named Entity Recognition (PDCNER) is proposed in this paper. The main contributions of the paper are as follows:

(1)

We propose a simple yet effective NER model that integrates enhanced lexicon and contrastive learning for the complex pig disease domain, making the model more sensitive to texts in this domain and improving predictions for entities. The lexicon-enhanced BERT facilitates the direct integration of external lexicon knowledge of pig diseases into BERT layers via a Lexicon Adapter layer.

(2)

To enrich the semantic feature representation and improve performance under data scarcity conditions, we propose a lexicon-enhanced contrastive loss layer on top of the BERT encoder. Experimental results on small sample scenarios and common public datasets demonstrate that our model outperforms other models.

(3)

Given the lack of an annotated corpus for the pig disease domain, we collected and annotated a new Chinese corpus and annotated datasets consisting of 7518 entities. To address the insensitivity of word segmentation caused by the specialization of the pig disease domain, we constructed a lexicon for identifying specific terms in pig diseases using frequency statistics methods under the guidance of veterinarians.

The remainder of the paper is organized as follows. Section 2 introduces the dataset and method proposed in this paper. Section 3 provides a detailed description of our experiments and analyzes the results. Finally, the conclusions are presented in Section 4.

2. Materials and Methods

2.1. Materials

2.1.1. Corpus Collection and Pre-Processing

Due to the lack of NER public benchmark datasets in the pig disease domain, a new Chinese pig disease corpus was constructed and annotated under the guidance of animal disease experts and veterinarians. To ensure the quality of the data, we collected information on pig diseases from professional books, published standards, Baidu Encyclopedia, and official websites. The data source details are mentioned in Appendix A.

After the data acquisition, we performed basic data pre-processing steps. Firstly, optical character recognition (OCR) technology was used to convert the books and standards into text format. Secondly, useless data, such as garbled characters or special symbols, were manually deleted and wrong words were modified in raw text. Thirdly, the duplicate data and invalid data were removed. Ultimately, a comprehensive and effective text corpus containing 1.45 million characters was obtained (Corpus I of pig disease).

2.1.2. Analysis of Corpus

First, we analyze the characteristics of the corpus. The corpus of pig diseases contains many specialized terms, such as disease names, medicine names, and body parts. Examples include “腹膜炎 (peritonitis)”, “盐酸头孢塞呋 (ceftazidime hydrochloride)”, and “膀胱黏膜 (bladder mucosa)”. Additionally, boundary ambiguity and nested entities are common problems in named entity recognition (NER). For instance, the phrase “非洲猪瘟 (African swine fever)” encompasses two distinct entities: the location entity “非洲 (Africa)” and the disease entity “猪瘟 (swine fever)”. This ambiguity results in inaccuracies during the word segmentation process, thereby increasing the challenges associated with recognizing these entities.

2.1.3. Corpus Annotation

According to the characteristics of the corpus and referring to existing labeling standards in fields such as human medicine, we developed labeling criteria in collaboration with veterinary experts. These criteria include non-overlapping labeling, non-nested labeling, and covering as many types as possible. Finally, a multi-round iteration approach was used to label and refine the data, ensuring the accuracy and consistency of the labeling.

On the basis of the above work, 152,596 characters were selected from Corpus I to form Corpus II for entity labeling. The Label Studio tool and BIEO labeling method were used to label entities. B represents the start position of the entity, I represents the inside of the entity, E represents the end position of the entity, and O represents the other. Five pig disease attributes, including pig type, disease name, body parts, symptom, and medicine, were labeled in the corpus text. Finally, the annotated pig disease corpus containing 7518 entities was obtained under the guidance of pig disease experts. The statistical information of labeled entities is presented in

Table 1. Statistics of annotated entities.

Category	Category Definition	Examples	Numbers	Proportion of the Total
Type	Name of different types of pig	妊娠母猪, 仔猪 (Pregnant sows, piglets)	735	9.78%
Disease	Name of pig disease	猪丹毒, 胸膜炎 (Porcine erysipelas, pleurisy)	958	12.74%
Body parts	Body position, organs, and system of pigs	心脏, 巨噬细胞 (Heart, Macrophages)	2063	27.44%
Symptom	External performance	气喘, 咳嗽, 水肿 (Asthma, cough, swollen)	2973	39.55%
	caused by diseases
Medicine	Medications for treating diseases	替米考星, 克林霉素 (Timicosin, clindamycin)	789	10.49%
Total			7518	100%

.

2.1.4. Construction of Lexicon and Pre-Training Word Embedding

We constructed a lexicon in the pig disease domain based on Corpus II and professional books. Firstly, the most commonly used professional terms were extracted from Corpus II by word segmentation and frequency statistics. Then, some professional words in the glossary of professional books were manually added to the dictionary under the guidance of veterinarians, such as “猪副嗜血杆菌病 (Haemophilus parasuis)”, “噻苯达唑 (Thiabendazole)”, and “内阿米巴原虫 (Entamoeba spp.)”. Finally, the pig disease lexicon comprising 2391 professional terms was obtained for understanding the specific words and technical terms. Subsequently, this lexicon was incorporated into the built-in dictionary of Jieba to avoid the incorrect segmentation of words.

To obtain a high-quality embedded representation of pig diseases, we trained Corpus I and the lexicon. The Gensim tool (v.3.8.3) was used to train the Word2Vec model with a word vector dimension of 200. The construction process of the lexicon and the pre-training of word embeddings are illustrated in Figure 1.

2.2. Methods

The structure of the PDCNER model is shown in Figure 2. Firstly, the Chinese sentences in the pig disease corpus are converted into a character–word pair sequence, and both Chinese character features and lexicon features are used as inputs. Secondly, a lexicon adapter is added between Transformer layers, which is used to dynamically extract the most relevant matching items. The word of each character uses the bi-linear attention mechanism from character to word, and the lexicon adapter is applied between adjacent Transformers in BERT. The lexicon features and BERT representations are fully interacted through multi-layer encoders in BERT, so that lexicon knowledge can be effectively integrated into BERT. The contrastive loss layer is above the Lexicon-Enhanced BERT encoder, ensuring that similar samples are as close as possible, while dissimilar samples are as far apart as possible. Embeddings of the same type of entity are treated as positive samples, whereas embeddings of different types of entities are treated as negative samples. Considering the correlation between consecutive labels, a Conditional Random Field (CRF) layer is employed to label the sequence.

2.2.1. Char–Word Pair Sequence

According to the Lexicon-Enhanced BERT in [12], we firstly expand the character sequence into a sequence of character–word pairs for applying lexical information of pig disease.

A Chinese sentence with n characters, ${\mathrm{S}}_{\mathrm{C}}=\left\{{\mathrm{c}}_1,{\mathrm{c}}_2,\cdots ,{\mathrm{c}}_{\mathrm{n}}\right\}$. We identify all potential words within the sentence by comparing the character sequence with the lexicon of pig disease. To achieve this, we first create a Trie data structure based on the lexicon. Then, we examine all character subsequences within the sentence, matching them with the Trie to identify all potential words. For instance, in the truncated sentence “非洲猪瘟 (African swine fever)”, we can identify five distinct words: “非洲 (Africa)”, “非洲猪 (African pig)”, “猪瘟 (swine fever)”, “瘟 (epidemic disease)”, and “非洲猪瘟 (African swine fever)”. In the field of pig disease, African swine fever is a complete disease name and should not be separated. Following this, for each matched word, we associate it with the characters that compose it. In conclusion, we pair each character with its associated words and transform the Chinese sentence into a sequence of character–word pairs, represented as:

$${\mathrm{S}}_{\mathrm{Cw}}=\left\{\left({\mathrm{c}}_1,{\mathrm{ws}}_1),({\mathrm{c}}_2,{\mathrm{ws}}_2\right),\cdots ,\left({\mathrm{c}}_{\mathrm{n}},{\mathrm{ws}}_{\mathrm{n}}\right)\right\}$$

(1)

where ${\mathrm{c}}_{\mathrm{i}}$ denotes the i-th character in the sentence, and ${\mathrm{ws}}_{\mathrm{i}}$ signifies the words matched and assigned to ${\mathrm{c}}_{\mathrm{i}}$.

2.2.2. Lexicon Adapter

Using the lexicon adapter proposed in LEBERT [12], the pig disease lexicon information is directly injected into BERT for integrating lexical features.

For the i-th character in a character–word sequence, the input is$\left(h_{\mathrm{i}}^{\mathrm{c}}, x_i^{ws}\right)$, and $h_{\mathrm{i}}^{\mathrm{c}}$ represents the character vector, the output of a transformation layer in BERT. $x_i^{ws}=\left\{x_{i1}^w,x_{i2}^w,\cdots ,x_{im}^w\right\}$ represents a group of word embeddings. The j-th word in $x_i^{ws}$ is represented as follows:

$$x_{ij}^w={\mathrm{e}}^{\mathrm{w}}\left(w_{ij}\right)$$

(2)

where $e^w$ is the pre-trained word embedding list and $w_{ij}$ represents the j-th word in$w{s}_i$.

To align these two different representations, a nonlinear transformation is used for each word vector:

$$v_{ij}^w={\mathrm{W}}_2\left(\tanh \left({\mathrm{W}}_1{x}_{ij}^w+{\mathrm{b}}_1\right)\right)+{\mathrm{b}}_2$$

(3)

where ${\mathrm{W}}_1\in {\mathrm{R}}^{d_c\times d_w}$, ${\mathrm{W}}_2\in {\mathrm{R}}^{d_c\times d_c}$, and $d_w \mathrm{and} d_c$represent the dimension of word embedding and BERT’s hidden size, respectively. ${\mathrm{b}}_1{\mathrm{and}\mathrm{b}}_2$ are scaler bias.

Each character is associated with a variety of words, but the degree of contribution from each word differs. For instance, in the field of pig diseases, the words “非洲 (Africa)” and “猪瘟 (swine fever)” are more important than “非洲猪 (African pigs)” and “瘟 (epidemic disease)”. In order to find the most relevant words, the character–word attention mechanism is used. The correlation of each word is calculated as follows:

$$a_i=\mathrm{softmax}\left(h_i^c{\mathrm{W}}_{\mathrm{attn}}{\mathrm{V}}_{\mathrm{i}}^{\mathrm{T}}\right)$$

(4)

where ${\mathrm{W}}_{\mathrm{attn}}\in {\mathrm{R}}^{d_c\times d_c}$ is the bi-linear attention mechanism. All $v_{ij}^w$ are assigned to the i-th character $V_i=\left(v_{i1}^w,\cdots ,v_{im}^w\right)$, where m is the total number of assigned words.

Lastly, the lexicon information is integrated into the vector representation of the character.

$${\tilde{h}}_i=h_i^c+{\displaystyle \sum }_{j=1}^m{a}_{ij}{v}_{ij}^w$$

(5)

2.2.3. Lexicon-Enhanced BERT

The lexicon adapter is attached between transformer layers in BERT so that the knowledge of the pig disease lexicon can be injected into BERT. A sequence of characters$\left\{{\mathrm{c}}_1,{\mathrm{c}}_2,\cdots ,{\mathrm{c}}_{\mathrm{n}}\right\}$ is input into the input embedding of BERT and then $\mathrm{E}=\left\{{\mathrm{e}}_1,{\mathrm{e}}_2,\cdots ,{\mathrm{e}}_{\mathrm{n}}\right\}$ is obtained by adding token, segmentation, and position embedding. After that, E is input into the Transformer encoders. Each layer is as follows.

$$\begin{array}{c}G=\mathrm{LayerNormalization}\left(H^{l-1}+Multiheadattention\left(H^{l-1}\right)\right)\\ H^l=\mathrm{LayerNormalization}\left(G+FFN\left(G\right)\right)\end{array}$$

(6)

where $H^l=\left\{h_1^l,h_2^l,\cdots ,h_n^l\right\}$ represents the output of the l-th layer and $H^0$ = E. FFN represents the two-layer feed-forward network with RELU as the hidden activation function.

The lexicon information was input between the k-th and (k+1)-th layer Transformer. $H^k=\left\{h_1^k,h_2^k,\cdots ,h_n^k\right\}$ were first obtained after k consecutive Transformer layers. Subsequently, each character–word pair $\left(h_i^k,x_i^{ws}\right)$ was processed through the lexicon adapter to obtain a new hidden layer representation and the $i_{th}$ pair was converted into ${\tilde{h}}_i^k$ accordingly.

$${\tilde{h}}_i^k=\mathrm{Lexicon}\mathrm{Adapter}\left(h_i^k,x_i^{ws}\right)$$

(7)

$H^k=\left\{h_1^k,h_2^k,\cdots ,h_n^k\right\}$ are input into the remaining (L-K) Transformer, as there are 12 layers of Transformers in BERT. Finally, the output of the L-th Transformer $H^L$ used for the name entity recognition task is obtained.

2.2.4. Lexicon-Enhanced Contrastive Learning

The normalized temperature-scaled cross-entropy loss, denoted as NT-Xent, was used as our contrastive loss function [41]. For each training iteration, we randomly select N texts from the datasets to form a mini-batch, which yields 2N feature representations. The model is then trained to identify each data point’s corresponding pair among the 2(N − 1) negative samples present within the batch.

$${\mathcal{L}}_{i,j}=-\log {\displaystyle \frac{\exp \left(\mathrm{s}\left(r_i,r_j\right)/T\right)}{{{\displaystyle \sum }}_k^{2N}{\mathrm{I}}_{\left[k\ne i\right]}\exp \left(\mathrm{s}\left(r_i,r_k\right)/T\right)}}$$

(8)

where r_i, r_j represent the embedding for entities of the same type, whereas r_i, r_k denote the embedding for entities of different types. s() refers to the cosine similarity function, where $T$ acts as the temperature parameter, and I serves as an indicator function. Ultimately, we calculate the final contrastive loss by averaging the classification losses of all 2N instances within the batch.

2.2.5. Construction of Positive and Negative Pairs

Construction of positive and negative samples is very important for contrastive learning, which aims to learn better feature representations of data. We construct them according to the following steps.

• Positive sample pairs based on the same label token

We take tokens of the same entity type as positive sample pairs.

Direct Matching: Within a sequence or document, directly identify all tokens labeled with the same entity type and use them as positive sample pairs. For example, if both “fattening pigs (育肥猪)” and “breastfeed sows (哺乳母猪)” are labeled as “type” entities in a sentence, they form a positive sample pair.

Cross-Document Matching: Identify all tokens labeled with the same entity type across different paragraphs or sentences. Although these tokens may appear in different contexts, they can still be considered positive sample pairs due to their shared entity type.

2.

Negative sample pairs based on entities of different types

We take tokens of different entity types as negative sample pairs. Specifically, we select tags labeled with different entity types as negative sample pairs. For example, in different sentences, “Porcine blue ear disease (猪蓝耳病)” is labeled as a “disease” entity, and “盐酸头孢塞呋 (ceftazidime hydrochloride)” is labeled as a “medicine” entity, thus constituting a negative sample pair. Additionally, entities labeled as ‘O’ can also form negative sample pairs with entities labeled with the five specified types.

3. Results

3.1. Evaluation

To identify a named entity for pig diseases, it is necessary to correctly identify both the boundaries of the entity and its corresponding categories. The proposed PDCNER model is evaluated using standard measures, Precision (P), Recall (R), and F1, which are computed using Equations (9)–(11).

$$P={\displaystyle \frac{T_P}{T_P+F_P}}\times 100\%$$

(9)

$$R={\displaystyle \frac{T_P}{T_P+F_n}}\times 100\%$$

(10)

$$F1={\displaystyle \frac{2PR}{P+R}}\times 100\%$$

(11)

where T_p represents the number of positive samples that are accurately predicted, while F_p denotes the count of positive samples that are inaccurately predicted. Additionally, F_n stands for the number of negative samples that are incorrectly predicted.

3.2. Experimental Settings

Experiments. The hardware environment that the experimental research relied on was Intel(R) Xeon(R) Silver4116 [email protected] GHz (Intel, Santa Clara, CA, USA), GPU@NVIDIA Tesla P100 (NVIDIA, Santa Clara, CA, USA). The software environment was Python3.8 and tensorflow 2.0.0. The model parameters were set as follows: based on BERT_BASE [16] version, with 12 transformer layers, 768 hidden layers, and 12 multi-head attention mechanisms. The lexicon corresponds to the vocabulary of pre-trained word embeddings in field of pig disease. During the training process, we incorporate the Lexicon Adapter between the first and second Transformer layers of BERT [12]. Meanwhile, the parameters of BERT and the pre-trained word embedding were fine-tuned. The hyperparameters are shown in

Table 2. Hyperparameters of the proposed model.

Hyper Param	Value	Hyper Param	Value
batch_size	16	dropout	0.5
learning rate	0.00001	optimizer	Adam
epoch	20	maximum sentence length	256

. This study randomly divided the datasets into training, validation, and test sets according to a ratio of 7:2:1.

3.3. Results

3.3.1. Comparison with Baseline Models

We evaluate the effectiveness of PDCNER against widely used NER models on the pig disease corpus. The overall findings of our experiments are presented in

Table 3. Comparison of experimental results for different NER models.

Model Category	Model	P (%)	R (%)	F1 (%)
baseline model without pre-training	BILSTM_CRF	71.58	67.51	69.49
pre-trained model	BERT-BiLSTM-CRF	80.29	84.73	82.45
	BERT-CRF	81.62	84.39	82.98
	BERT-CNN-CRF	82.44	80.55	81.48
	BERT-WWM-ext-BiLSTM-CRF	81.73	85.47	83.56
	RoBERTa-BiLSTM-CRF	81.64	85.31	83.43
pre-trained model with lexicon	BERT-BILSTM-CRF-SoftLexicon	82.99	84.73	83.85
	LEBERT	87.18	86.45	86.81
	PDCNER (ours)	87.76	86.97	87.36

The bold in the last row denotes the results we obtained in our model.

.

According to Table 3, several inferences can be drawn. Firstly, the F1-scores of pre-trained models are higher than those of models without pre-training by more than 10%, such as the BILSTM_CRF model. This indicates that pre-trained models, with their deeper network structures, have learned more language features and enhanced their ability to recognize entities. Secondly, the pre-trained model incorporating lexicon information demonstrates better performance than models without lexicon integration. This improvement is primarily due to the integration of dictionary information specific to the pig disease domain, which effectively captures entity boundaries and word information. Thirdly, the recognition performance of our model significantly outperforms other models, achieving a micro-average F1-score of 87.36% in recognizing five major entities. This experimental result indicates that our model can effectively improve entity recognition performance in the domain of pig disease.

Comparative analysis with the results of BERT-BiLSTM-CRF reveals notable improvement in the precision, recall, and F1-score of PDCNER, with improvements of 7.47 percentage points, 2.24 percentage points, and 4.91 percentage points, respectively.

Through comparative evaluation utilizing the same datasets and downstream model, PDCNER demonstrates superior accuracy in identifying pig disease entities compared to LEBERT, exhibiting improvements in precision, recall, and F1-score by 0.58 percentage points, 0.52 percentage points, and 0.55 percentage points, respectively.

3.3.2. The Recognition Effect on Different Entities

For better understanding of the proposed approach, we evaluate the PDCNER model separately on the five entities type, disease, body parts, symptom, and medicine, which are presented in Figure 3. We found that the F1-scores for type, disease, and medicine all exceeded 90%, with the F1-score for type being the highest at 95.41%. Conversely, the lowest F1-score was for the entity of symptom, at 80.92%.

On the other hand, the F1-scores of disease entities and medicine entities were 92.96% and 90.05%, respectively. Both disease and medicine entities include a large number of technical terms, yet the method proposed in this paper achieves a good recognition effect on these two entities. The results demonstrate that PDCNER fully utilizes both Chinese character features and lexicon knowledge in the pig disease domain at the input level, and the lexicon adapter can effectively leverage pig disease knowledge.

3.3.3. The Recognition Effect on Small Sample

In order to verify the reliability and robustness of PDCNER in the condition of scarce data for entity recognition, we used 1%, 10%, 30%, and 50% of labeled samples for experimentation. The results can be found in Figure 4 and

Table 4. Results of the small-sample experiments.

Model	1%			10%			30%
	P	R	F1	P	R	F1	P	R	F1
BERT-BiLSTM-CRF	31.79	1.80	3.41	67.92	75.84	71.66	74.59	81.44	77.87
LEBERT	17.39	11.43	13.79	74.81	83.47	78.91	74.05	80.14	76.97
PDCNER (ours)	18.18	11.43	14.04	85.00	86.44	85.71	87.22	86.17	86.69
Model	50%			100%
	P	R	F1	P	R	F1
BERT-BiLSTM-CRF	79.18	83.45	81.26	80.29	84.73	82.45
LEBERT	81.09	83.4	82.23	87.18	86.45	86.81
PDCNER (ours)	87.68	86.71	87.19	87.76	86.97	87.36

The bold in the last row denotes the results we obtained in our model.

. The result shows that the PDCNER model has obvious improvement compared to BERT-BILSTM-CRF and LEBERT.

The F1-score of PDCNER reaches 85.71% when the sample size is 10%, which is only 1.65% lower than that of the full sample. As the sample size increases to 30%, the F1-score of the PDCNER model further improves to 86.69%, showing a marginal decrease of only 0.67% compared to the full sample. Moreover, it outperforms the BERT-BiLSTM-CRF and LEBERT models by 14.05% and 6.8%, respectively, with only 10% samples available. These results demonstrate the PDCNER model’s capability to achieve higher recognition accuracy even under data scarcity scenarios.

3.3.4. The Recognition Effect on Public Datasets

To assess the generalization capability of PDCNER, we conducted evaluations across three public benchmark datasets: Weibo, Ontonotes, and Resume. As illustrated in

Table 5. F1-score for each model on public datasets.

Model	Weibo			Ontonotes			Resume
	P	R	F1	P	R	F1	P	R	F1
BERT-BiLSTM-CRF	71.29	67.1	69.13	84.11	80.2	82.11	96.56	97.23	95.89
LEBERT	78.2	74.64	76.38	89.76	82.68	86.07	95.89	97.42	96.65
PDCNER (ours)	81.42	76.56	78.91	89.68	83.43	86.44	96.07	97.36	96.71

The bold in the last row denotes the results we obtained in our model.

, the PDCNER model achieved the highest F1-scores across all three datasets. Specifically, the F1 values of PDCNER were 9.78%, 4.33%, and 0.82% higher than those of the BERT-BiLSTM-CRF model for the Weibo, Ontonotes, and Resume datasets, respectively. Additionally, compared to the LEBERT model, the PDCNER method showed improvements of 2.53%, 0.37%, and 0.06% for the same datasets. These results indicate that PDCNER not only exhibits superior performance on the pig disease corpus but also demonstrates remarkable generalization capability across different domains.

4. Discussion

Our experimental results demonstrate that the proposed PDCNER method effectively utilizes pig disease feature information by seamlessly integrating it into the BERT architecture. Specifically, the Lexicon Adapter is inserted between the first and second transformer layers within BERT, facilitating the infusion of pig disease lexicon knowledge into the model’s representations.

In addition to the lexicon-enhanced BERT’s contribution to improvement, contrastive learning has also played a significant role. The lexicon-enhanced contrastive learning method improves the model’s capacity for semantic representation of text based on the normalized temperature-scaled cross-entropy loss function. This loss function enhances the model’s ability to identify similar entities by minimizing the distance between positive samples (i.e., the same type of entities) while simultaneously maximizing the distance from negative samples (i.e., different types of entities). Consequently, this approach reduces the risk of the model incorrectly classifying different entities. Furthermore, it enables the model to adjust the weighting of distance measurements, allowing it to focus more on distinguishing subtle differences between different entities during the training process.

Judging from the recognition results of different entities, despite the sample sizes of type, disease, and medicine entities being only about 10%, their F1 values are all above 90%. In contrast, although symptoms have the largest sample size, accounting for 39.55%, their F1 value is the lowest. This disparity arises primarily because the boundaries of pig type and disease entities are very clear, whereas the boundaries of symptom entities are more ambiguous. For instance, type entities typically end with terms such as “pigs (猪)” (e.g., sick pigs (患病猪), conservation pigs (保育猪), fattening pigs (育肥猪)), and disease entities usually end with terms like “disease (病),” “inflammation (炎),” and “plague (瘟)” (e.g., porcine blue ear disease (猪蓝耳病), necrotic enteritis (坏死性肠炎), African swine fever (非洲猪瘟)). In contrast, symptom entities are often composed of complex verbs, conjunctions, and modifiers, such as “feed intake continuing to decrease (采食量持续下降)” and “continuous spasmodic cough (连续痉挛性咳嗽)”. Moreover, the average length of complex symptom entities is approximately eight Chinese characters, which contributes to a lower overall recognition rate.

5. Conclusions

High-quality extraction of entities related to pig diseases is critical for intelligent consultation, question answering, technical recommendations, and other application scenarios.

In this study, we constructed a corpus, labeled datasets, and lexicon for Chinese named entity recognition specific to pig diseases, encompassing 152,596 characters, 7518 entities, and 2391 professional terms. To tackle the challenges of entity identification in the pig disease domain, such as the scarcity of annotated data, numerous technical terms, and fuzzy boundaries, we propose the PDCNER model. This model integrates lexicon information from the pig disease domain into BERT’s Transformer layers at the lower level and employs contrastive learning to enhance representation quality and generalization capability. The results indicate that the PDCNER model surpasses the performance of BERT-BiLSTM-CRF and other mainstream models, achieving precision, recall, and F1-score of 87.76%, 86.97%, and 87.36%, respectively. This demonstrates high-quality entity recognition in the field of pig diseases. Moreover, small-sample experiments confirm that our model is more suitable than other models for completing the named entity recognition task in data-scarce scenarios. Experiments on public datasets also verify its generalization ability. Our approach provides a reference for improving NER performance in domain-specific applications. It can be easily applied to other fields, facilitating seamless adaptation for named entity identification across diverse contexts.

In future work, we plan to focus on the following aspects:

(1)

We aim to enhance the identification of more fine-grained entity types in the pig disease domain, including appearance symptoms and anatomical symptoms.

(2)

We will further optimize the pig disease corpus and construct a more extensive professional vocabulary to enhance NER performance.

(3)

We will explore the application of our model in other animal diseases, such as chicken or cow diseases, and investigate methods to handle nested entities and discontinuous entities based on our approach.