CSMNER: A Toponym Entity Recognition Model for Chinese Social Media

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This paper proposes a place-named entity recognition model for Chinese social media. By integrating the BERT pre-trained model with the improved IDCNN-BiLSTM-CRF architecture and introducing the boundary extension tension module, the accuracy of place name recognition has been enhanced to some extent. The performance of CSMNER in handling place name NER tasks in social media texts was verified through experiments on WeiboNER, MSRA, and a self-constructed dataset CSNER, providing new methods and tools for extracting geographical information from Chinese social media texts. The manuscript is structured logically and written in an academic style. However, several issues need to be clarified.

1. Are the experimental results in Table 6 only for place names? The comparison results in the table seem to use the experimental results from the original paper. Since the original paper includes results for both place names only and for all entities, these two types of results may differ significantly, making the comparison less rigorous.

2. This paper combines the BERT pre-trained model with the IDCNN-BiLSTM-CRF architecture to complete the NER task. As far as I know, similar NER work has been published recently in the same field. It is recommended to compare and analyze with the recent models. For example, 1) Zhao Y, Zhang D, Jiang L, et al (2024) EIBC: a deep learning framework for Chinese toponym recognition with multiple layers. J Geogr Syst 1–19. https://doi.org/10.1007/s10109-024-00441-4. 2) Ma K, Tan Y, Xie Z, Qiu Q, Chen S (2022) Chinese toponym recognition with variant neural structures from social media messages based on BERT methods. J Geogr Syst 24(2):143-169. https://doi.org/10.1007/s10109-022-00375-9

3. In the Boundary Extension Module (BE), the features obtained by IDCNN are used as the Value, and the features obtained by BiLSTM are used as the Query and Key. Why are the features constructed this way? Additionally, it is recommended to add an experiment to compare the efficiency of the model in the manuscript with and without the BE module.

4. The paper does not demonstrate the advantages of the self-constructed dataset CSNER over other databases. And how do other algorithms perform on it? Can the dataset be made publicly available?

5. The BERT pre-training model is adopted in this paper. What is the impact of the pre-training model on the experimental results? Why not use a more robust pre-trained model like RoBERTa?

6. Are there two X_BILSTM in the input of Figure 5?  

7. Please add a reference to Figure 1 in the text.

8. In the 'Example' column of Table 1, it is recommended to add the original Chinese text to facilitate reading. It seems there are some errors for 'central bank' in the Chinese version.

9. There are issues with the naming (do not match the content) and format of Figure 6.

10. The reference formatting is inconsistent, and some references are incomplete, such as [28].

Author Response

I appreciate your reviewing this manuscript on your busy schedule and your critical questions and detailed suggestions for revision. I will respond to each of your questions below.

 

Comments 1: Are the experimental results in Table 6 only for place names? The comparison results in the table seem to use the experimental results from the original paper. Since the original paper includes results for both place names only and for all entities, these two types of results may differ significantly, making the comparison less rigorous.

Response 1: Thank you for pointing out this issue, I fully agree with you and I have made two tables in the paper to differentiate them in order to ensure the rigor and scientific validity of the study. Table 6 shows the F1 scores of the CSMNER model when recognizing only toponym entities compared to other studies with the same dataset. The F1 scores shown in Table 7 are the sum of the F1 scores obtained when recognizing the three entities PER, LOC and ORG. The reason for showing the combined performance of the CSMNER model for the three types of entities at the same time is not only to show the advancement of the model in toponym recognition in the face of Chinese social media, but also to illustrate that the model has better robustness. This part of the modification can be found in Section 4.2 of the article.

 

 

Comments 2: This paper combines the BERT pre-trained model with the IDCNN-BiLSTM-CRF architecture to complete the NER task. As far as I know, similar NER work has been published recently in the same field. It is recommended to compare and analyze with the recent models.

Response 2: Thank you for your suggestions, I totally agree with you. To prove the effectiveness of the model, I searched the latest papers related to toponym recognition by searching on the Web of Science platform and found that there are very few relevant papers that can be searched in the past three years. At the same time, to ensure the rigor and reliability of the model comparison, so we can only choose the relevant studies from 2022 to the present, using the same dataset for experiments for comparison, and the relevant content of the comparative analysis is shown in chapter 4.2 of the paper.

 

 

Comments 3: In the Boundary Extension Module (BE), the features obtained by IDCNN are used as the Value, and the features obtained by BiLSTM are used as the Query and Key. Why are the features constructed this way? Additionally, it is recommended to add an experiment to compare the efficiency of the model in the manuscript with and without the BE module.

Response 3: Thank you for your suggestions. I totally agree with you and I will explain below.

① The BE module, as the innovative structure proposed in this paper, adopts the multi-head attention mechanism in its core part, and the reason why  is used as Value and  as Query and Key respectively is as follows:

1. When is used as Value, IDCNN can capture local contextual information through IDCNN via the convolutional layer, and toponyms often depend on their surrounding lexical environment. For example, "Beijing" as a toponym can provide important clues from its preceding and following words. In the attention mechanism, Value represents the actual information stored in the model.
2. The features generated by IDCNN as Value means that the model will utilize these local features to build the final output representation. These features contain the key local contextual information required for toponym recognition.
3. BiLSTM can integrate information in both forward and backward directions, which means that it is able to capture the complete contextual information of each word in the input sequence. The LSTM unit can handle long-range dependencies efficiently, which is crucial for recognizing those complex toponyms that span across multiple words.
4. In the attention mechanism, Query and Key are used to determine which information in the Value is relevant. The features generated by BiLSTM as Query and Key means that the model will decide which local information (i.e., the Value of the IDCNN) is most relevant at the present time based on these features, which guides the focusing of the attention mechanism.
5. The features of BiLSTM contain dynamic contextual information, which makes it possible to flexibly match the most relevant local features according to the different positions of the input sequence when used as Query and Key, thus improving the accuracy of toponym recognition.

 

② In order to compare the efficiency of the model with and without the BE module, I show the training process of the model after adding each module on top of Baseline in Figure 8 and Figure 9 in Section 4.4 of the article. It can be found that after adding the BE module, the F1 score of the model improves rapidly and the Loss value decreases rapidly, which indicates that the model is able to capture effective features faster during the training process. Meanwhile, the F1 score of the model improves more stably, indicating that the addition of BE module is effective.。

 

 

Comments 4: The paper does not demonstrate the advantages of the self-constructed dataset CSNER over other databases. And how do other algorithms perform on it? Can the dataset be made publicly available?

Response 4: Thank you for pointing this out and I agree with you.

① In order to highlight the advantages of the self-constructed dataset CSNER, I have made additional improvements on the original basis of the paper, which are mainly elaborated in section 3.1 of the paper. Its main advantages can be summarized in the following points:

1. Strong relevance: the CSNER dataset is all derived from the weibo platform, which is one of the largest social platforms in China, and can more accurately reflect the language style, expression habits and toponym usage on the microblogging platform.
2. Strong domain: Weibo, as a very active social media, contains many informal terms, idioms and Internet buzzwords, which can be well covered by the CSNER dataset to improve the comprehension ability of the model.
3. Timeliness: the corpus collected by CSNER dataset is the data from 2015 to present, so it can ensure that the dataset covers the latest online phrases, hot topics and emerging toponyms.
4. Controllable quality: the self-constructed dataset can ensure the consistency and accuracy of the annotation rules and reduce the annotation errors.

 

② The effect of other algorithms on CSNER dataset can be referred to the effect of Baseline model and Baseline + IDCNN(ReLU) model in the paper. Because the BiLSTM + CRF model architecture in Baseline and BiLSTM + IDCNN(ReLU) + CRF model architecture are more common in the field of named entity recognition, they can be used as references. For how the other complex models perform on the CSNER dataset, we can't test it. However, the CSNER dataset has been comprehensively evaluated in the paper, and fully meets the criteria for use. Compared with the public dataset WeiboNER, the CSNER dataset is larger in size, more complex in corpus information, and involves a wider range of domains.

 

③ In order to promote the development of Chinese named entity recognition models and mutual academic exchanges, the self-constructed dataset CSNER can be directly accessed from https://figshare.com/s/77cb3523143208578619 and can be accessed and obtained directly from.

 

 

Comments 5: The BERT pre-training model is adopted in this paper. What is the impact of the pre-training model on the experimental results? Why not use a more robust pre-trained model like RoBERTa?

Response 5: Thank you for pointing this out.

① The BERT model is useful for the Chinese social media named entity recognition task in this paper as follows:

1. BERT's bi-directional encoder is better at capturing short social texts with variable contexts and containing abbreviations or slang than manually designed feature engineering. These features contain rich contextual information and language comprehension.
2. The WordPiece disambiguation strategy adopted by BERT to decompose the input text subwords (Tokens) makes the BERT model itself powerful in capturing the semantic information of fragmented text, which is suitable for complex and trivial social short text data. This ability to capture fine-grained features of words is important for this task.
3. The next sentence prediction (NSP) training strategy of BERT plays an important role in understanding the contextual relationships between phrases and can capture the corresponding features in the inter-sentence dimension to enhance the performance of the recognition task.

 

② Although in reference [35], its authors stated that the RoBERTa pre-training model is more robust than the BERT pre-training model, the reason for adopting the BERT pre-training model is that after multiple rounds of comparative analyses using the two pre-training models separately, it was found that the scheme using RoBERTa was always 1%-2% lower in F1 scores than the scheme using BERT.

The various reasons are analysed as follows:

1. For short social text data such as Weibo, the data of RoBERTa pre-training model does not match perfectly, resulting in its performance not being fully utilised. Moreover, the experiments conducted by the authors of the literature [35] were mainly on long texts and some complex contexts, which does not mean that it will perform well in short social texts.
2. RoBERTa removes the next sentence prediction (NSP) task compared to the BERT pre-training model, which causes more difficulties for short social texts that are already lacking in valid information.
3. RoBERTa has more complex pre-training strategies compared to BERT, such as longer sequences and higher number of training steps, which can lead to model overfitting on the named entity recognition task, especially on smaller datasets like WeiboNER.

 

 

Comments 6: Are there two X_BILSTM in the input of Figure 5?

Response 6: Thanks for pointing this out, the reason there are two X_BILSTM vectors is because at first it was considered to multiply each of the two X_BILSTM vectors with the  and  vectors. The result shows that the picture does not express enough completeness, so I have highlighted the details of the BE module on top of the original to show the structure of the BE module in its entirety. In fact, the process of calculation is that X_BILSTM is multiplied by  and  respectively to obtain two Key and Value query matrices.

 

 

Comments 7: Please add a reference to Figure 1 in the text.

Response 7: Thank you for your careful reading of this issue noted, and I have added a reference to Figure 1 in my paper.

 

 

Comments 8: In the 'Example' column of Table 1, it is recommended to add the original Chinese text to facilitate reading. It seems there are some errors for 'central bank' in the Chinese version.

Response 8: Thank you for pointing this out, and to make it easier to understand, I have added a column of Chinese translations to Table 1 as you suggested. At the same time, after checking in the Oxford English Dictionary and the Wade-Giles University English Dictionary, the translation of the term "central bank" should be fine, but I have replaced it with the term "European Union", which occurs more frequently in the dataset, as an example.

 

 

Comments 9: There are issues with the naming (do not match the content) and format of Figure 6.

Response 9: I totally agree with you that the title of figure 6 had a problem with the meaning it conveyed when translated into English, and I have modified it to follow the content of the picture.

 

 

Comments 10: The reference formatting is inconsistent, and some references are incomplete, such as [28].

Response 10: Thank you for pointing this out, I have fixed the error and checked all other references for additional information.

Reviewer 2 Report

Comments and Suggestions for Authors

Please find attached.

Comments for author File: Comments.pdf

Author Response

I appreciate your reviewing this manuscript on your busy schedule and your critical questions and detailed suggestions for revision. I will respond to each of your questions below.

 

Comments 1: Clarity and context：

All the presented research is described with attention to detail and reliably. The authors focus mainly on the Chinese language, which they explain in a convincing way. However, it would be worth delving deeper into the latest studies on the identification of toponyms using Named Entity Recognition for other languages. Examples of such studies are provided below.

Isn't the term toponym more appropriate instead of "place name"? This also applies to the title of the manuscript.

The authors use acronyms inconsistently and do not always explain them. IDCNN acronym is not explained at the first occurrence. What does the acronym CSNER mean? There are various translations of it in the text.

Response 1: Thank you for giving your opinion, I have scrutinized it and found these important issues.

① Thank you for pointing this out, I totally agree that the latest research on toponym recognition is very much in need of in-depth study, and I have read these papers you recommended before. These research results are mainly adopted to some exogenous information to assist in toponym recognition, and all of them have achieved very good results. My research results are also inspired by the in-depth study of the Transformer model in the following article.

• Cillian Berragan ; Alex Singleton; Alessia Calafiore ; Jeremy Morley Transformer Based Named Entity Recognition for Place Name Extraction from Unstructured Text. International Journal of Geographical Information Science 2023 , 37 , 747-766 ,doi: 10.1080/13658816.2022.2133125.

In the subsequent research, I will consider improving the performance of the current model according to some latest methods.

 

② After considering the exact explanation of the academic terminology, I agree with you and choose to use the term "toponym" instead of "place name".

 

③ Meanwhile, thanks so much for the tip, the abbreviations of IDCNN, BiLSTM, CRF, CSMNER and CSNER have been supplemented, which are explained in the abstract and the main body of the paper respectively.

 

 

Comments 2: Data and methodology：

The data used is well described, although the key Table 3 requires clarification regarding Entity type.

The research methodology is well described, although the graphics illustrating it are of average quality (Fig.1, 2 and 5). Their better quality would significantly improve the overall perception of the manuscript.

Response 2: Thank you very much for the problems you pointed out. To explain the structure and data flow of the research model as clearly as possible, I have drawn and supplemented some more details on the original basis to improve the quality of the pictures. And the content of the paper has been supplemented and modified according to the content of the pictures.

 

 

Comments 3: Statistics and uncertainties:

The Evaluation metrics section requires careful review as it contains unnecessary repetitions.

The analysis is statistically correct and includes an ablation experiment. The only doubts are raised by the metrics in Table 6 given on two different scales.

Response 3: Thank you for pointing this out. I agree with you completely. The original Table 6 (now Table 7) contained data from several different evaluation scales, which biased the statistical analysis of the table. Therefore, I chose to display all data results on a scale of 0-100 with a maximum precision of two decimal places. Regarding your reference to the duplication of evaluation metrics, I did not fully understand it because I am showing the performance of the most recent research results in the last two years on two publicly available datasets, WeiboNER and MSRA, which have chosen the three commonly used model evaluation metrics of Precision, Recall, and F1.

 

Comments 4: Conclusions:

The presented conclusions result directly from the conducted analyses. They are robust, valid and reliable. Unfortunately, without access to data and models, this is impossible to verify.

It's a pity that the authors did not make the results of their work public.

Response 4: Thank you for raising this issue. To ensure the reliability and validity of the research work, I have uploaded the code and data of the work product, which can be accessed by a wide range of researchers via https://figshare.com/s/77cb3523143208578619 for access and retrieval.

 

 

Comments 5: References:

The references are well chosen, although they require minor editorial adjustments. Journal names are used inconsistently ([5] vs [6]).

One of the most important items cited by the authors [33] is not correctly described in the references. This problem unfortunately affects a larger number of references.

Considering the profile of the journal, it would be worth mentioning recent works using spatial information in ML models.

Response 5: Thank you for pointing this out. I completely agree with your point of view. Therefore, in addition to the citations [5], [6] and [33] you mentioned, I carefully checked the other cited documents and made supplements and improvements.

 

 

Comments 6: Detailed comments:

• Table 1：Why is I-LOC given twice for location?

• <line 255>：Section should not start with the figure.

• Table 3：It is not clear what the second column in this table means - Entity type equal 3 (LOC?).

• Section 4.1.3 Experimental settings：What is this numbering inside the section for?

• Table 6：The data in this table must be standardized because it may be misleading. Either range 0-1 or 0-100.

Response 6: Thanks for pointing this out, I totally agree with you.

① In Table 1, I-LOC is given twice because "Tian an men" is three characters in Chinese, which form a complete geographical name entity. Therefore, "Tian" is labeled as B-LOC, and "an" and "men" are labeled as I-LOC. That is, "an" and "men" are the middle and ending parts of the geographical name entity. I have also added the Chinese translation of the named entity in the example for easier understanding.

 

② According to your reminder, I have adjusted the content of the chapter where Figure 2 is located, and no longer use the picture as the beginning of the chapter.

 

③ The meaning of the entity type of 3 mentioned in Table 3 is that the dataset contains three annotated entity types, namely person (PER), location (LOC) and organization (ORG). To make the thesis presentation clearer, I have added comments below the table.

 

④ Because chapter 4.1.3 is the description of the experimental setup, the experimental setup is divided into two parts to expand the description, 1. is the setup of hyper-parameters; 2. is the description of the training strategy. Therefore, we chose to add two numbers to separate the descriptions.

 

⑤ In the original Table 6 (now Table 7), I fully agree with the data standardization issue you pointed out, and I have standardized the evaluation criteria to 0-100.

Reviewer 3 Report

Comments and Suggestions for Authors

In this manuscript, CSMNER  (Chinese Social Media Named Entity Recognition) is proposed to solve the problem of recognizing place names in Chinese social media texts. The proposed model extracts local boundary features and contextual semantic features of place names using the improved IDCNN and BiLSTM module for feature extraction over two channels. Moreover, a BE module is introduced in this manuscript to evaluate the influence of contextual information on the local feature information and improve the recognition of place name entity boundaries. Extensive experiments were conducted on various datasets (WeiboNER, MSRA, CSNER) to verify the effectiveness of the proposed approach. The performance of the approach is evaluated and compared with other relevant place name recognition models. The obtained results are presented, analyzed and discussed. Furthermore, the future directions of work are listed. The authors have presented their work well. The work is technically sound, and the references provided by the authors are applicable and relevant, there are 44 citations

 Please consider the following corrections and comments:

1.      In the Conclusions section, the authors should address the influence of complex noise and mention some other tasks related to geographic information (since they mentioned this as future research).

2.      In addition to the fact that the effectiveness of the proposed model is demonstrated and emphasized, the limitations of the presented approach should also be discussed in the concluding part of the manuscript.  

3.      Please elaborate how difficult it would be to adapt the proposed CMNER model for other languages, e.g. English?

Author Response

I appreciate your reviewing this manuscript on your busy schedule and your critical questions and detailed suggestions for revision. I will respond to each of your questions below.

 

Comments 1: In the Conclusions section, the authors should address the influence of complex noise and mention some other tasks related to geographic information (since they mentioned this as future research).

Response 1: Thank you for pointing this out and I totally agree with you. Therefore, I have followed your suggestion and discussed the possible effects of complex noise on the identification of toponymic entities in the conclusion section. At the same time, the downstream tasks related to geographic information that can be applied to the task of toponym recognition are described to clearly indicate future research efforts.

 

 

Comments 2: In addition to the fact that the effectiveness of the proposed model is demonstrated and emphasized, the limitations of the presented approach should also be discussed in the concluding part of the manuscript.

Response 2: Thank you for pointing this out and I fully agree with you, therefore I have analyzed and explained in detail the current limitations and possible problems of the CSMNER model in sections 4.2, 4.3, 4.4 and 5 of the article, as well as a comparative analysis with the results of other recent research. I provide a brief overview below:

① The performance improvement of toponym recognition on the standardized MSRA news dataset is not significant enough.

 

② Due to the limitation of the size of the training corpus, accurate recognition may not be possible when facing different domains, which greatly limits the recognition accuracy of the CSMNER model in different scenarios and affects the robustness of the model.

 

③ Complex noise generates a large amount of meaningless contextual information, causing recognition bias.

 

 

Comments 3: Please elaborate how difficult it would be to adapt the proposed CMNER model for other languages, e.g. English?

Response 3: Regarding this issue you mentioned because it is not the focus of what the thesis wants to elaborate, I have not reflected much relevant content in the thesis, and I will answer your question from the following aspects:

Your question involves the problem of transfer learning.

 

① Language structure difference: due to the different grammar, vocabulary and punctuation usage between Chinese and English, Chinese usually does not have obvious word boundary separations, while English has spaces to separate words. This directly affects the processing of word division and lexical labelling, which in turn affects the recognition of toponymic entities.

 

② Different naming rules: the naming rules of Chinese and English toponyms differ in that Chinese toponyms may contain elements such as Chinese characters, numbers, pinyin, etc., whereas English toponyms may contain letters, hyphens, spaces, etc.

 

③ Cultural and geographical differences: Chinese toponyms may contain cultural elements such as historical figures, myths and legends, natural landscapes, etc., while English toponyms may reflect more of the country's history, religious beliefs or geographical features.

 

④ Lack of training data: when migrating the model from one language to another, it may face the problem of insufficient training data in the target language. If the target language has limited annotation data, the model may not be able to fully learn the features of toponyms in the target language.

 

⑤ Differences in labelling systems: Chinese and English toponymic entities may have different labelling criteria, such as the hierarchy of the toponym (country, province, city, street, etc.) and the definition of boundaries. Inconsistency in labelling will increase the difficulty of model training.

 

⑥ Model architecture and parameter adjustment: the original model architecture may need to be adjusted for the characteristics of the target language.

 

In summary, applying the trained CSMNER model for Chinese social media directly to other languages usually fails to achieve ideal recognition results. This is because language differences, differences in cultural backgrounds, as well as differences in model architecture parameters and training data require adaptation to the target language. Nevertheless, it is possible to perform transfer learning based on the CSMNER model because there are still some similar features between other languages and Chinese, which are stored as parameters in the model and can be directly utilised. Re-training the model is necessary to achieve better results.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The authors have revised their manuscript and addressed my previous comments. However, it is recommended to address the following issues before publishing.

1.This article focuses on the recognition of Chinese toponym entity. It is advisable to present the text of the examples in Table 1 in Chinese, accompanied by their respective English translations. For example, {“天安门”, [B-LOC, I-LOC, I-LOC]}.

2.In the fourth example of Table 11, the recognition result — 'No. 1 bus' — is not reflected in the English translation, potentially leading to reader confusion.

3.Line 936: 'No. 11 bus' should be corrected to 'No. 1 bus'.

4.The figures in this PDF version are not very clear.

Author Response

Comments 1: This article focuses on the recognition of Chinese toponym entity. It is advisable to present the text of the examples in Table 1 in Chinese, accompanied by their respective English translations. For example, {“天安门”, [B-LOC, I-LOC, I-LOC]}.

Response 1: Thank you for pointing this out, I couldn't agree with you more. Considering that this paper is about identifying toponymic entities in Chinese, it is indeed more accurate to label them in the Chinese presentation state.

 

 

Comments 2: In the fourth example of Table 11, the recognition result — 'No. 1 bus' — is not reflected in the English translation, potentially leading to reader confusion.

Response 2: Thank you for pointing this out, I completely agree with you, and I have corrected this misunderstanding caused by this cultural context in the paper by labeling it as a semantic error in the error types in Table 11. Also to explain this semantic difference clearly, I have added a comment at the bottom of Table 11 again to show the meaning of the paper.

 

 

Comments 3: Line 936: 'No. 11 bus' should be corrected to 'No. 1 bus'

Response 3: Thank you for correcting this error, I have made the change.

 

Comments 4: The figures in this PDF version are not very clear.

Response 4: Thank you for pointing this out. I have enlarged or bolded the parts of the image that are not displayed clearly to improve clarity, while ensuring that the font is consistent with the body of the text.