KCB-FLAT: Enhancing Chinese Named Entity Recognition with Syntactic Information and Boundary Smoothing Techniques
Abstract
:1. Introduction
- This article proposes a method KCB-FLAT that achieves good results on four Chinese datasets: Weibo, MSRA, Resume, and a self-built dataset ZJ.
- We combine the smoothing process with syntactic information extraction, which discretizes the entity probabilities to the entity boundaries, effectively alleviating the problem of over-confidence of the model in entity labelling.
- We control the smoothing factor on the Boundary Smoothing method to control the degree to which the model differentiates the probability of entities and analyze how these adjustments impact overall model performance.
- We conducted experiments on our self-built dataset, ZJ, by integrating it, and the results show that our model is well-suited for Chinese named entity recognition tasks within the judicial domain.
2. Related Work
2.1. Current Status of Named Entity Identification
2.2. Current Status of Syntactic Information Extraction in NLP
2.3. Application of Smoothing Techniques in Model Performance Enhancement
3. Methodology
3.1. The Overview of the KCB-FLAT
3.2. KVMN for Syntactic Embedding
3.3. Cross Transformer for Semantic Fusion
3.4. Boundary Smoothing Module
4. Experimental Design
4.1. Datasets and Evaluation Indicators
4.2. Parameter Settings
4.3. Results and Discussion
4.4. Analysis and Discussion
4.5. In-Depth Analyses
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Yin, D.; Cheng, S.; Pan, B.; Qiao, Y.; Zhao, W.; Wang, D. Chinese Named Entity Recognition Based on Knowledge Based Question Answering System. Appl. Sci. 2022, 12, 5373. [Google Scholar] [CrossRef]
- Bose, P.; Srinivasan, S.; Sleeman, W.C., IV; Palta, J.; Kapoor, R.; Ghosh, P. A Survey on Recent Named Entity Recognition and Relationship Extraction Techniques on Clinical Texts. Appl. Sci. 2021, 11, 8319. [Google Scholar] [CrossRef]
- Chen, S.; Pei, Y.; Ke, Z.; Silamu, W. Low-Resource Named Entity Recognition via the Pre-Training Model. Symmetry 2021, 13, 786. [Google Scholar] [CrossRef]
- Ahmad, P.N.; Shah, A.M.; Lee, K. A Review on Electronic Health Record Text-Mining for Biomedical Name Entity Recognition in Healthcare Domain. Healthcare 2023, 11, 1268. [Google Scholar] [CrossRef] [PubMed]
- Huang, C.; Wang, Y.; Yu, Y.; Hao, Y.; Liu, Y.; Zhao, X. Chinese Named Entity Recognition of Geological News Based on BERT Model. Appl. Sci. 2022, 12, 7708. [Google Scholar] [CrossRef]
- Szczepanek, R. A Deep Learning Model of Spatial Distance and Named Entity Recognition (SD-NER) for Flood Mark Text Classification. Water 2023, 15, 1197. [Google Scholar] [CrossRef]
- Yang, J.; Teng, Z.; Zhang, M.; Zhang, Y. Combining discrete and neural features for sequence labeling. In Proceedings of the Computational Linguistics and Intelligent Text Processing: 17th International Conference, CICLing 2016, Konya, Turkey, 3–9 April 2016; pp. 140–154. [Google Scholar] [CrossRef]
- He, H.; Sun, X. F-score driven max margin neural network for named entity recognition in Chinese social media. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Part 3: 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2015), Valencia, Spain, 3–7 April 2017; pp. 713–718. [Google Scholar] [CrossRef]
- Yao, L.; Huang, H.; Wang, K.-W.; Chen, S.-H.; Xiong, Q. Fine-Grained Mechanical Chinese Named Entity Recognition Based on ALBERT-AttBiLSTM-CRF and Transfer Learning. Symmetry 2020, 12, 1986. [Google Scholar] [CrossRef]
- Chiu, J.P.; Nichols, E. Named entity recognition with bidirectional LSTM-CNNs. Trans. Assoc. Comput. Linguist. 2016, 4, 357–370. [Google Scholar] [CrossRef]
- Guo, C.; Pleiss, G.; Sun, Y.; Weinberger, K.Q. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, Australia, 6–11 August 2017; Volume 3. [Google Scholar] [CrossRef]
- Wang, Z.; Shang, J.; Liu, L.; Lu, L.; Liu, J.; Han, J. Crossweigh: Training named entity tagger from imperfect annotations. arXiv 2019, arXiv:1909.01441. [Google Scholar]
- Enwei, Z.; Jinpeng, L. Boundary Smoothing for Named Entity Recognition. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland, 22–27 May 2022; Muresan, S., Nakov, P., Villavicencio, A., Eds.; Association for Computational Linguistics: Stroudsburg, PA, USA, 2022; Volume 1, pp. 7096–7108. [Google Scholar] [CrossRef]
- Deng, Z.; Tao, Y.; Lan, R.; Yang, R.; Wang, X. Kcr-FLAT: A Chinese-Named Entity Recognition Model with Enhanced Semantic Information. Sensors 2023, 23, 1771. [Google Scholar] [CrossRef]
- Li, X.; Yan, H.; Qiu, X.; Huang, X. FLAT: Chinese NER Using Flat-Lattice Transformer. arXiv 2020, arXiv:2004.11795. [Google Scholar]
- Tourani, A.; Bavle, H.; Avşar, D.I.; Sanchez-Lopez, J.L.; Munoz-Salinas, R.; Voos, H. Vision-Based Situational Graphs Exploiting Fiducial Markers for the Integration of Semantic Entities. Robotics 2024, 13, 106. [Google Scholar] [CrossRef]
- Wang, N.; Ge, R.F.; Yuan, C.F.; Wong, K.F.; Li, W.J. Company name identification in Chinese financial domain. J. Chin. Inf. Pro. 2002, 16, 1–6. [Google Scholar]
- Sohrab, M.G.; Miwa, M. Deep exhaustive model for nested named entity recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 31 October–4 November 2018; pp. 2843–2849. [Google Scholar] [CrossRef]
- Markus, E.; Kevin, P.; Adrian, U. ManyEnt—A Dataset for Few-shot Entity Typing. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain, Online, 8–13 December 2020; pp. 5553–5557. [Google Scholar] [CrossRef]
- Shen, Y.; Ma, X.; Tan, Z.; Zhang, S.; Lu, W. Locate and label: A two-stage identifier for nested named entity recognition. arXiv 2021, arXiv:2105.06804. [Google Scholar]
- Li, J.; Fei, H.; Liu, J.; Wu, S.; Zhang, M.; Teng, C.; Ji, D.; Li, F. Unified named entity recognition as word-word relation classification. In Proceedings of the AAAI conference on artificial intelligence, Seattle, WA, USA, 28 June 2022; pp. 10965–10973. [Google Scholar] [CrossRef]
- Tan, Z.; Shen, Y.; Zhang, S.; Lu, W.; Zhuang, Y. A sequence-to-set network for nested named entity recognition. arXiv 2021, arXiv:2105.08901. [Google Scholar]
- Ma, X.; Hovy, E. End-to-end sequence labeling via bi-directional lstm-cnns-crf. arXiv 2016, arXiv:1603.01354. [Google Scholar]
- Li, X.; Feng, J.; Meng, Y.; Han, Q.; Wu, F.; Li, J. A unified MRC framework for named entity recognition. arXiv 2019, arXiv:1910.11476. [Google Scholar]
- Yu, J.; Bohnet, B.; Poesio, M. Named Entity Recognition as Dependency Parsing. arXiv 2020, arXiv:2005.07150. [Google Scholar]
- Yu, B.; Hang, Z.; Shu, X.; Liu, T.; Wang, Y.; Wang, B.; Li, S. Joint extraction of entities and relations based on a novel decomposition strategy. arXiv 2019, arXiv:1909.04273. [Google Scholar]
- Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R.R.; Le, Q.V. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv 2019, arXiv:1906.08237. [Google Scholar]
- Athiwaratkun, B.; Santos, C.N.D.; Krone, J.; Xiang, B. Augmented natural language for generative sequence labeling. arXiv 2020, arXiv:2009.13272. [Google Scholar]
- Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; Zettlemoyer, L. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv 2019, arXiv:1906.08237. [Google Scholar]
- Ni, J.; Abrego, G.H.; Constant, N.; Ma, J.; Hall, K.B.; Cer, D.; Yang, Y. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv 2021, arXiv:2108.08877. [Google Scholar]
- Daneshfar, F.; Aghajani, M.J. Enhanced text classification through an improved discrete laying chicken algorithm. Expert Syst. 2024, 41, e13553. [Google Scholar] [CrossRef]
- Liang, L.-X.; Lin, L.; Lin, E.; Wen, W.-S.; Huang, G.-Y. A Joint Learning Model to Extract Entities and Relations for Chinese Literature Based on Self-Attention. Mathematics 2022, 10, 2216. [Google Scholar] [CrossRef]
- Lou, C.; Yang, S.; Tu, K. Nested named entity recognition as latent lexicalized constituency parsing. arXiv 2022, arXiv:2203.04665. [Google Scholar]
- Ma, J.; Ballesteros, M.; Doss, S.; Anubhai, R.; Mallya, S.; Al-Onaizan, Y.; Roth, D. Label Semantics for Few Shot Named Entity Recognition. In Proceedings of the Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, 22–27 May 2022; pp. 1956–1971. [Google Scholar]
- Nie, Y.; Tian, Y.; Song, Y.; Ao, X.; Wan, X. Improving named entity recognition with attentive ensemble of syntactic information. arXiv 2020, arXiv:2010.15466. [Google Scholar]
- Petrovska, B.; Atanasova-Pacemska, T.; Corizzo, R.; Mignone, P.; Lameski, P.; Zdravevski, E. Aerial Scene Classification through Fine-Tuning with Adaptive Learning Rates and Label Smoothing. Appl. Sci. 2020, 10, 5792. [Google Scholar] [CrossRef]
- Wu, P.; Cui, Z.; Gan, Z.; Liu, F. Three-Dimensional ResNeXt Network Using Feature Fusion and Label Smoothing for Hyperspectral Image Classification. Sensors 2020, 20, 1652. [Google Scholar] [CrossRef]
- Mahayossanunt, Y.; Nupairoj, N.; Hemrungrojn, S.; Vateekul, P. Explainable Depression Detection Based on Facial Expression Using LSTM on Attentional Intermediate Feature Fusion with Label Smoothing. Sensors 2023, 23, 9402. [Google Scholar] [CrossRef]
- Ashish, V.; Noam, S.; Niki, P.; Jakob, U.; Llion, J.; Aidan, N.G.; Lukasz, K.; Illia, P. Attention is all you need. Adv. Neural Inf. Process. Syst. 2017, 30, 5998–6008. [Google Scholar] [CrossRef]
- Rafael, M.; Simon, K.; Geoffrey, H. When Does Label Smoothing Help? Adv. Neural Inf. Process. Syst. 2019, 32, 4696–4705. [Google Scholar] [CrossRef]
- Lukasik, M.; Bhojanapalli, S.; Menon, A.; Kumar, S. Does label smoothing mitigate label noise? In Proceedings of the International Conference on Machine Learning, Virtual, 13–18 July 2020; pp. 6448–6458. [Google Scholar] [CrossRef]
- Collobert, R.; Weston, J.; Bottou, L.; Karlen, M.; Kavukcuoglu, K.; Kuksa, P. Natural language processing (almost) from scratch. JML 2011, 12, 2493–2537. [Google Scholar] [CrossRef]
- Gu, Y.; Qu, X.; Wang, Z.; Zheng, Y.; Huai, B.; Yuan, N.J. Delving deep into regularity: A simple but effective method for Chinese named entity recognition. arXiv 2022, arXiv:2204.05544. [Google Scholar]
- Wu, S.; Song, X.; Feng, Z.; Wu, X. NFLAT: Non-Flat-Lattice Transformer for Chinese Named Entity Recognition. arXiv 2022, arXiv:2205.05832. [Google Scholar]
- Xiong, L.; Zhou, J.; Zhu, Q.; Wang, X.; Wu, Y.; Zhang, Q.; Gui, T.; Huang, X.; Ma, J.; Shan, Y. A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition. arXiv 2023, arXiv:2305.12485. [Google Scholar]
- Shen, Y.; Song, K.; Tan, X.; Li, D.; Lu, W.; Zhuang, Y. Diffusionner: Boundary diffusion for named entity recognition. arXiv 2023, arXiv:2305.13298. [Google Scholar]
- Wu, S.; Song, X.; Feng, Z. MECT: Multi-Metadata Embedding based Cross-Transformer for Chinese Named Entity Recognition. arXiv 2021, arXiv:2107.05418. [Google Scholar]
- Ma, R.; Peng, M.; Zhang, Q.; Huang, X. Simplify the usage of lexicon in Chinese NER. arXiv 2019, arXiv:1908.05969. [Google Scholar]
- Mao, Q.; Li, J.; Meng, K. Improving Chinese Named Entity Recognition by Search Engine Augmentation. arXiv 2022, arXiv:2210.12662. [Google Scholar]
- Hu, D.; Wei, L. SLK-NER: Exploiting second-order lexicon knowledge for chinese NER. arXiv 2020, arXiv:2007.08416. [Google Scholar]
Entity Type | Amount of Entity | Number of Tags |
---|---|---|
Crime Location | 3356 | 28,976 |
Place Name | 3360 | 23,076 |
Person Name | 5047 | 13,487 |
Organization Name | 1667 | 6353 |
Accusation | 1846 | 5837 |
Laws and regulations | 1517 | 6203 |
Judgement | 2086 | 7456 |
Total | 19,079 | 91,348 |
Model | KVMN | CT | RA | BS | MSRA | ZJ | Resume | |
---|---|---|---|---|---|---|---|---|
Baseline (FLAT) | 93.45 | 94.27 | 94.93 | 63.42 | ||||
+KVMN | √ | 95.98 | 96.04 | 96.23 | 69.27 | |||
+KVMN&CT | √ | √ | 96.24 | 95.64 | 95.98 | 69.78 | ||
+RA | √ | 96.14 | 95.66 | 96.02 | 66.98 | |||
+BS | √ | 96.02 | 95.32 | 95.72 | 66.83 | |||
+BS&RA | √ | √ | 96.30 | 95.97 | 96.09 | 67.15 | ||
= 0.2) | √ | √ | √ | √ | 96.59 | 96.87 | 97.02 | 71.37 |
Models | MSRA | ZJ | Resume | |
---|---|---|---|---|
NFLAT [44] | 94.55 | 95.40 | 95.58 | 61.94 |
Li et al. (2022a) [21] | 95.97 | 96.08 | 96.32 | 72.18 |
Xiong et al. (2023) [45] | 95.40 | 95.51 | 95.97 | 68.14 |
DiffusionNER (2023) [46] | / | 94.88 | 94.53 | / |
RICON [47] | 96.12 | 96.01 | 95.98 | 66.82 |
MECT [47] | 96.21 | 96.13 | 95.91 | 70.40 |
LSTM + Lexicon augment [48] | 93.47 | 96.02 | 95.58 | 61.22 |
AESINER [35] | 96.53 | 96.42 | 96.40 | 70.54 |
Mao et al. (2022) [49] | / | 95.98 | 96.29 | 70.81 |
SLK-NER [50] | / | 96.01 | 95.78 | 64.00 |
Ours (KCB-FLAT) | 96.59 | 96.87 | 97.02 | 71.37 |
奥林匹克公园 (The Olympic Park) | ||||||
---|---|---|---|---|---|---|
Entity | AO | LIN | PI | KE | GONG | YUAN |
Gold label | B-LOC.NAM | I-LOC.NAM | I-LOC.NAM | I-LOC.NAM | I-LOC.NAM | E-LOC.NAM |
Baseline (FLAT) | B-LOC.NAM | I-LOC.NAM | I-LOC.NAM | E-LOC.NAM | B-LOC.NOM | E-LOC.NOM |
KCB-FLAT(Ours) | B-LOC.NAM | I-LOC.NAM | I-LOC.NAM | I-LOC.NAM | I-LOC.NAM | E-LOC.NAM |
ZJ | 邯郸市公安局丛台分局 (Congtai Branch of Handan Public Security Bureau) | |||||||||
---|---|---|---|---|---|---|---|---|---|---|
Entity | HAN | DAN | SHI | GONG | AN | JU | CONG | TAI | FEN | JU |
Gold label | B-ORG | I-ORG | I-ORG | I-ORG | I-ORG | I-ORG | I-ORG | I-ORG | I-ORG | E-ORG |
Baseline (FLAT) | B-ORG | I-ORG | I-ORG | I-ORG | I-ORG | E-ORG | O | O | O | O |
KCB-FLAT(Ours) | B-ORG | I-ORG | I-ORG | I-ORG | I-ORG | I-ORG | I-ORG | I-ORG | I-ORG | E-ORG |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Deng, Z.; Huang, Z.; Wei, S.; Zhang, J. KCB-FLAT: Enhancing Chinese Named Entity Recognition with Syntactic Information and Boundary Smoothing Techniques. Mathematics 2024, 12, 2714. https://doi.org/10.3390/math12172714
Deng Z, Huang Z, Wei S, Zhang J. KCB-FLAT: Enhancing Chinese Named Entity Recognition with Syntactic Information and Boundary Smoothing Techniques. Mathematics. 2024; 12(17):2714. https://doi.org/10.3390/math12172714
Chicago/Turabian StyleDeng, Zhenrong, Zheng Huang, Shiwei Wei, and Jinglin Zhang. 2024. "KCB-FLAT: Enhancing Chinese Named Entity Recognition with Syntactic Information and Boundary Smoothing Techniques" Mathematics 12, no. 17: 2714. https://doi.org/10.3390/math12172714