Methods of Annotating and Identifying Metaphors in the Field of Natural Language Processing
Round 1
Reviewer 1 Report
This is a survey paper that covers the annotation and identification of metaphors in the field of NLP. The paper is well-written and the introduction to the conceptual metaphor theory and linguistic metaphor is thorough and proper in depth.
The description of the work on using more recent neural-network based methods is up to the published work in 2021. Apparently more work has been published since then especially on recognition of metaphors. A few more survey papers on the same topic have been published in the past two years. Thus this work could be extended to cover the work during this period.
Author Response
Dear reviewer,
Thank you for your time and effort invested in helping us improving the manuscript.
As per your suggestion, our paper is extended by covering additional work published in the past two years (from 2021 to 2023), namely, Song et al. (2021), Choi et al. (2021), Zhang et al. (2022), Maudslay and Teufel (2022), Li et al. (2023), Ge et al. (2022).
Extending the manuscript with papers published in the past two years has given the better overview of more contemporary approaches, given that field is progressing fast.
Thank you!
Reviewer 2 Report
This paper provides an overview of methods of annotating and
identifying metaphors and publicly available datasets.
The paper has an unfortunate tendency to state ideas and theories as definitive (e.g., “all people”, “always”…). Many statements in the text need to be more nuanced.
A more serious drawback of the paper is its lack of comparison among the many methods described. While it does make some effort to relate different methods, the paper is largely a long list of methods, organized by sector, but with no summaries at all. Each of the cited papers is described with a paragraph, using lots of technical terms that remain often unexplained.
Thus, the paper has value, but is unsatisfying.
Specific points:
..prove their theory by citing a series .. - I would rather say “support their theory..”; examples do not constitute a proof
..distinguish between conventional metaphors and novel metaphors. - is this a binary choice? Or rather a continuous scale between the two?
..is further proof for the .. ->
..is further evidence for the ..
..containing 23 primary metaphor .. ->
..containing 23 primary metaphors ..
..i.e. concepts. ->
..i.e., concepts.
..none of these lists is final. ->
..none of these lists is definitive.
..all people have the same patterns of perception .. - surely not all
..warmth is always something good .. - I might surmise that warmth may not be so associated in very hot places
..frequently used procedure of .. ->
..frequently used procedures of ..
..whether a word is a metaphor or not) .. - “is” or “is part of”? Metaphors usually (always?) consist of more than one word
..”Find metaphor-related words (MRWs) by examining the text on a word-by-word basis.” - what criteria is used to find?
..When a word is a new-formation coined, .. - how is this determined?
..which must be viewed .. - again, use of definitive terms; why “must”? Why not “are best viewed”?
..whether they are one lexical unit .. ->
..whether each is one lexical unit ..
..overview of previous researchs .. ->
..overview of previous research ..
..Founds poured into .. - founds?
..hyponymic connection .. -what is this? hypernyms too
..i.e. by identifying .. ->
..i.e., by identifying ..
(i.e., add a comma after each “i.e.”)
..extracted from Reuters .. ->
..extracted from the Reuters ..
..precision of 0.72 and a response of 0.80. - do you mean “recall”?
..not LDA stop words) .. - this paper is clearly addressed to experts in the field, as it repeatedly uses technical terms in the field, without further explanation; the average journal reader will not understand what are ”LDA stop words”
..threshold was set for each language (-10.0 for English and -13.0 for Spanish), .. - of what relevance here are these numbers? The text is giving too much information about very specific papers.
Other technical terms used here without citation or explanation: binary maximum entropy, Weka tool, decision tree, support vector machine, random forest, and combined classifier
..word2vec, fastText, GloVe) and contextual (e.g. ELMo, BERT, RoBERTa, GPT) .. -no citations are given here for all these items
..CBOW architecture is similar to the feed forward network and predicts the
current word based on the context, while the skip-gram model is similar
in its architecture to the CBOW model, but instead of predicting the
current word based on the context, this model uses a log-linear classifier, .. - this passage is typical of this article: i.e., a clear, lucid explanation, but lacking in suitable contextual explanations. The text does not explain what CBOW is, nor what a feedforward network is, nor a skip-gram model, nor a log-linear classifier.
..GloVe combines are the global matrix factorization and the local context window.. - again, a valid statement, but no explanation about either global matrix factorization or local context window.
..biLM consists of LSTM (Long Short Term Memory) [65] layers and a CNN (Convolutional Neural Network) [66] layer in which character convolution .. - again, a valid statement, but no explanations about: LSTM, CNN; of what relevance is a “character convolution”? It is insufficient to simply list a series of facts when one is doing an overview. One needs to explain and put ideas into context.
..difference in the number parameters .. ->
..difference in the number of parameters ..
..cosine similarity method is used for the calculation. - another example of a correct and relevant factual statement, but with no context or further explanation; e.g., why use cosine?
..adding a gating function .. - and what is the significance of this?
..has a GloVe word embeddings ..->
..has GloVe word embeddings ..
..each of these additional linguistic information affects .. ->
..each of these additional linguistic information aspects affects ..
..Lemmatization .. - paper uses this term a lot, but gives no explanation
Author Response
Dear reviewer,
Thank you for your time and effort invested in helping us improving the manuscript.
Below (in italic) please find our responses to your suggestions:
Specific points:
..prove their theory by citing a series .. - I would rather say “support their theory..”; examples do not constitute a proof
Corrected.
..distinguish between conventional metaphors and novel metaphors. - is this a binary choice? Or rather a continuous scale between the two?
This is something we can argue, but we have decided not to go into this discussion in this paper as this would require (at least) a section for itself. Footnote with explanation was added.
..is further proof for the .. ->
..is further evidence for the ..
Corrected.
..containing 23 primary metaphor .. ->
..containing 23 primary metaphors ..
Corrected.
..i.e. concepts. ->
..i.e., concepts.
Corrected.
..none of these lists is final. ->
..none of these lists is definitive.
Corrected.
..all people have the same patterns of perception .. - surely not all
..warmth is always something good .. - I might surmise that warmth may not be so associated in very hot places
Changed, added citation from Grady (2010) which gives better insight.
..frequently used procedure of .. ->
..frequently used procedures of ..
Corrected.
..whether a word is a metaphor or not) .. - “is” or “is part of”? Metaphors usually (always?) consist of more than one word
MIP procedure identifies metaphor on word level, defining for each word if it is used metaphorically. Of course that metaphors often consist of more than one word – in this case each word should be marked as metaphor.
..”Find metaphor-related words (MRWs) by examining the text on a word-by-word basis.” - what criteria is used to find?
..When a word is a new-formation coined, .. - how is this determined?
..which must be viewed .. - again, use of definitive terms; why “must”? Why not “are best viewed”?
This is MIPVU procedure and is presented as is, without the intervention in it. Scope of the manuscript does not allow for going into details but gives the brief overview and by referencing the procedure instructs the reader to read the MIPVU book to get more familiar with the procedure.
..whether they are one lexical unit .. ->
..whether each is one lexical unit ..
Corrected.
..overview of previous researchs .. ->
..overview of previous research ..
Corrected.
..Founds poured into .. - founds?
Typo, fixed.
..hyponymic connection .. -what is this? hypernyms too
Mistake in translation, as paper was originally written in Croatian and then translated to English. Corrected to hyponym relationship.
..i.e. by identifying .. ->
..i.e., by identifying ..
(i.e., add a comma after each “i.e.”)
Corrected.
..extracted from Reuters .. ->
..extracted from the Reuters ..
Corrected.
..precision of 0.72 and a response of 0.80. - do you mean “recall”?
Yes, thank you. Fixed.
..not LDA stop words) .. - this paper is clearly addressed to experts in the field, as it repeatedly uses technical terms in the field, without further explanation; the average journal reader will not understand what are ”LDA stop words”
..threshold was set for each language (-10.0 for English and -13.0 for Spanish), .. - of what relevance here are these numbers? The text is giving too much information about very specific papers.
Other technical terms used here without citation or explanation: binary maximum entropy, Weka tool, decision tree, support vector machine, random forest, and combined classifier
Simplified and removed redundant information.
..word2vec, fastText, GloVe) and contextual (e.g. ELMo, BERT, RoBERTa, GPT) .. -no citations are given here for all these items
Corrected, referenced.
..CBOW architecture is similar to the feed forward network and predicts the
current word based on the context, while the skip-gram model is similar
in its architecture to the CBOW model, but instead of predicting the
current word based on the context, this model uses a log-linear classifier, .. - this passage is typical of this article: i.e., a clear, lucid explanation, but lacking in suitable contextual explanations. The text does not explain what CBOW is, nor what a feedforward network is, nor a skip-gram model, nor a log-linear classifier.
Simplified and removed redundant information.
..GloVe combines are the global matrix factorization and the local context window.. - again, a valid statement, but no explanation about either global matrix factorization or local context window.
..biLM consists of LSTM (Long Short Term Memory) [65] layers and a CNN (Convolutional Neural Network) [66] layer in which character convolution .. - again, a valid statement, but no explanations about: LSTM, CNN; of what relevance is a “character convolution”? It is insufficient to simply list a series of facts when one is doing an overview. One needs to explain and put ideas into context.
..cosine similarity method is used for the calculation. - another example of a correct and relevant factual statement, but with no context or further explanation; e.g., why use cosine?
..adding a gating function .. - and what is the significance of this?
Manuscript is intended for readers who are familiar with the basic neural network and NLP concepts or are willing to conduct a research to understand it. This manuscript cannot go into the explanation of this concepts as this would change the scope and remove the focus of this survey paper from the metaphor.
..difference in the number parameters .. ->
..difference in the number of parameters ..
Corrected.
..has a GloVe word embeddings ..->
..has GloVe word embeddings ..
Corrected.
..each of these additional linguistic information affects .. ->
..each of these additional linguistic information aspects affects ..
Corrected.
..Lemmatization .. - paper uses this term a lot, but gives no explanation
Simple explanation given in footnote.
Thank you!
Reviewer 3 Report
The manuscript entitled -"Methods of annotating and identifying metaphors in the field of Natural Language Processing." presents a brief overview of the cognitive-linguistic theory of the metaphor annotation methods, which are used for research in linguistics in the field of natural language processing. The paper also provides an overview of relevant data sets used in metaphor identification over the last decade. The manuscript is written well and was fun to read. However, some minor issues need to be fixed.
- In most cases, context is presented by the author wise. However, I suggest rearranging everything in context and topic-wise. For example- X is proposed by y,z author.
- Although a review paper has no specific page limit, it needs to be more concise. Include some graphs and tables to summarize some of the text. Adding charts to the manuscript is always helpful for the readers to understand the whole concept at a glance.
Author Response
Dear reviewer,
Thank you for your time and effort invested in helping us to improve our manuscript.
We have accepted both of your suggestions:
- Paper is rearranged as you suggested – topic/approach to metaphor identification is at the forefront, followed by the authors.
- Tables that summarizes the overview of metaphor identification methods are added at the end of each section, as well as the table that shows all publicly available datasets. Furthermore, flowchart showing the MIP and MIPVU flow as well as overlap between these two procedures is presented in the section describing annotation methods and procedures.
Making these changes have improved our paper and will allow readers to have better overview of methods and procedures.
Thank you.
Reviewer 4 Report
The paper is extremely verbose to very limited academic significance. It is just a pure summary of annotating and identifying metaphors using NLP.
Without delving into the comparison of algorithmic complexities, performance metrics, ease of use and other factors, this paper currently demonstrate almost no contribution to the body of knowledge.
Moreover, the organization, and presentation of the need significant improvement to make this study attractive to the readers. Because, at presents the paper is missing logical flow of concepts and extremely monotonous (as if generated by AI). The authors should introduce conceptual diagrams, figures, and more tables to articulate genuine contribution.
Author Response
Dear reviewer,
Thank you for your time and effort invested in helping us to improve our manuscript.
Following changes are introduced in revised manuscript:
- Explained goal of the paper.
- Flowchart showing the MIP and MIPVU flow as well as overlap between these two procedures.
- Table 1 that shows all publicly available datasets.
- Tables that summarize the overview of metaphor identification methods (traditional machine learning methods in table 2, and neural network and word embedding methods in table 3)
- Paper is rearranged to show the content topic and approach wise, instead of author wise.
- Better overview of approaches to metaphor identification with the use of word embeddings.
- Paper is extended by covering additional work published in the past two years (from 2021 to 2023), namely, Song et al. (2021), Choi et al. (2021), Zhang et al. (2022), Maudslay and Teufel (2022), Li et al. (2023), Ge et al. (2022).
Thank you!
Round 2
Reviewer 2 Report
The paper is clearly better. There still remain lots of small grammatical errors, of which I list many below (but clearly not all).
Specific points:
..paper consists in systematic, .. ->
..paper consists of systematic, ..
..between common, conventional metaphors and novel metaphors. ->
..among common, conventional, and novel metaphors.
..statistical overview of corpus that .. ->
..statistical overview of corpora that ..
..and at it is shown in Figure 1, .. ->
..and as it is shown in Figure 1, ..
.. – once a metaphore is detected, MIPVU guide us on determining the metaphore type. ->
.. – once a metaphor is detected, MIPVU guide us on determining the metaphor type.
..CorMet system [57], proposed by .. ->
..The CorMet system [57], proposed by ..
..Concrete Category Overlap (CCO) algorithm was .. ->
..The Concrete Category Overlap (CCO) algorithm was ..
..work these authors [62], present ..
..work, these authors [62] present ..
..Approach to metaphor identification using ..
..An approach to metaphor identification using ..
..vector machine classifiers gives the ..
..vector machine classifier gives the ..
..4.4. Metaphore identificiation ..
..4.4. Metaphor identification ..
..better resulst in identifying ..
..better results in identifying ..
..When creating a word embeddings using ..
..When creating word embeddings using ..
..approach metaphore identification as a ..
..approach metaphor identification as a ..
..on VUA dataset of 0.583 ..
..on the VUA dataset of 0.583 ..
..a GloVe word embeddings in ..
..GloVe word embeddings in ..
..approaches the metaphore identificiation ..
..approaches the metaphor identification ..
..figurative langage is approach taken by ..
..figurative language form the approach taken by ..
..Contrastive Learning approach in the CATE ..
..The Contrastive Learning approach in the CATE ..
..are feed to transformer encoder, ..
..are fed to a transformer encoder, ..
.. Finally, cros-entropy loss function ..
.. Finally, the cross-entropy loss function ..
..and conventional metaphor are ..
..and conventional metaphors are ..
..allows to evaluate if model can judge ..
..allows evaluation whether a model can judge ..
..more metaphorical than other.
..more metaphorical than others.
..Architecture of the FrameBERT incorporate ..
..The architecture of the FrameBERT incorporates ..
..continue the reserch in this field ..
..continue the research in this field ..
..confirmation of its succes ..
..confirmation of its success ..
..verification and chekup on ..
..verification on ..
Author Response
Dear reviewer,
Thank you once more for your time and effort!
All grammatical errors you have listed are fixed, as well as few others found while proofreading.
Thank you!
Reviewer 4 Report
The updated manuscript is in a much better shape than before. However, the authors need to address the following:
1) Advantages and disadvantages of the reviewed methods should be clearly highlighted. Without the clear comparisons of the reviewed methods, the contribution of the paper seems insignificant.
2) Quality of Figure 1 should be improved.
3) Authors should ideally present a graph representing the number of papers reviewed per year. Without, the use of graphs and charts, the paper seems to have less analytical value.
Author Response
Dear reviewer,
Thank you once more for your time and effort!
Regarding your suggestions, please find our answers below (in italic):
1) Advantages and disadvantages of the reviewed methods should be clearly highlighted. Without the clear comparisons of the reviewed methods, the contribution of the paper seems insignificant.
We agree that comparison adds additional analytical value in most cases. However, the ambition of the paper was to present the maximum number of methods so that the reader can get a clear and simple overview of the field and the existing research. The approach taken is more descriptive than analytical, which maybe was not evident in the initial version, but is now more clearly stated in the paper.
To avoid confusion, introduction is extended with additional explanation on what is the goal of the paper.
2) Quality of Figure 1 should be improved.
Figure 1 is improved.
3) Authors should ideally present a graph representing the number of papers reviewed per year. Without, the use of graphs and charts, the paper seems to have less analytical value.
As there is rarely more than one research published per year, we find that presenting them in graph would not be of much benefit.
However, following your suggestion, we added a new table to the paper in Appendix A, where we grouped all the research according to the year in which they were published. This gives better chronological overview and adds in analytical value.
Thank you!
