Auto-Scoring Feature Based on Sentence Transformer Similarity Check with Korean Sentences Spoken by Foreigners

Round 1

Reviewer 1 Report

I agreed to review this paper since the topic and abstract sound very interesting for me. However, in its current form, I found the paper raw, inconsistent, and incomplete, so I couldn't recommend it to publication.

Though, after reading, I couldn't deduce well the focus of the paper, I rely on what a reader can guess from the title, the abstract and the keywords suggested by the authors. Thus, the main topics that must be addressed in the paper include speaking proficiency training models and evaluation, speech recognition quality scoring, and sentence similarity. Unfortunately, the paper covers these particular aspects at a very surface level.

In Introduction and Related Works section, the authors spend much time in discussing historical, social and demography issues along with common approaches to STT, which are not so relevant to the particular goals of their research. As a result of these scattered structure, the authors arrive to the contradictory conclusions: on the one hand, "STT could provide a cheap, reliable (...) to automatically transcribe speech in many languages" (line 96); on the other hand, "... can not be used for Korean" (line 115). There is no any concrete evidence provided by the authors, what makes Korean language specific, and what are specific challenges.

Only by page 3, the authors finally arrive to citing a (very-very small) number of relevant references (namely, [8]-[12]).

Around 30% of the main text is dedicated to the extensive presentation of Rest API, which is definitely not the major topic that the readers expect to find in a paper written for advancing in the domain of CALL. There is no discussion on pedagogical issues (exercise selection and configuration), interface issues (fro example, how to upload the new exercises to the system, how to make the learning system multimodal), approaches to sentence similarity evaluation, approaches to assessment, etc. Rest API, again Rest API, and performance evaluation, the latter seems to be too early, with respect to a proof-of-concept phase of the work. Could you please go back to the title, and focus on the aspects announced in the title?

The reasons of choosing the Android platform are clear for many, no need to hide them by presenting an absolutely irrelevant reference [14] on customer-based study of smartphone usage in Poland (especially if you address the benefits of foreigners living in Korea, as you insisted in the introduction).

In Conclusion, the authors present a very speculative thesis that higher scores obtained because of STT produced similarity mean that the user pronounces the sentences properly. In fact, the higher scores may be indicators of good STT algorithms, not a reliable measure of speaking proficiency. Indeed, one of evident goals of developing STT systems is to teach them to recognize well even the imperfect speech. It means that the idea in lines 411-412 could not be posted "just in parentheses", but needs careful analysis, since it is one of core points of the paper itself.

 

Author Response

Thank you for your kind input and comments. We have answered the reviews in a separate file, therefore, please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Dear authors,

the idea is very interesting!

Unfortunately, the theoretical justification is inadequate. The problem that is solved in this paper is not clearly formulated (56). The number of contributions is not high (105). The paper contains information about the country's geographical location that is off-topic. (28-31)

It would be necessary to extend the theoretical part and justify the necessity of this study.

Author Response

Thank you for your kind input and comments. We have answered the reviews in a separate file, therefore, please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Line 39 - please correct the word albeit

Table 1 - lack of period paragraph in study number 10 explanation

Table 2 - Title should be «Mobile Application» an first line Requirements and Description. 

Table 3 - Title should be «Rest API» an first line Requirements and Descrition

Author Response

Thank you for your kind input and comments. We have answered the reviews in a separate file, therefore, please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

This paper is very interesting. It examined the development of training service for foreigners to help them in- 11 crease their ability in speaking Korean. Overall, I find the paper to be quite well-written. As the paper deals with issues including language proficiencies,  I suggest the following references for the authors to be included in the paper. 

 

Sung, C. C. M. (2020). Exploring language identities in English as a lingua franca communication: experiences of bilingual university students in Hong Kong. International Journal of Bilingual Education and Bilingualism, 23(2), 184-197.

Sung, C. C. M. (2014). Hong Kong university students’ perceptions of their identities in English as a lingua franca contexts: An exploratory study. Journal of Asian Pacific Communication, 24(1), 94-112.

Sung, C. C. M. (2016). Does accent matter? Investigating the relationship between accent and identity in English as a lingua franca communication. System, 60, 55-65.

Sung, C. C. M. (2019). Investments and identities across contexts: A case study of a Hong Kong undergraduate student’s L2 learning experiences. Journal of Language, Identity & Education, 18(3), 190-203.

Sung, C. C. M. (2018). Out-of-class communication and awareness of English as a Lingua Franca. ELT Journal, 72(1), 15-25.

Author Response

Thank you for your kind input and comments. We have answered the reviews in a separate file, therefore, please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The authors made an effort to improve their contribution. I accept their comments on the points 1, 2 (partially), and 4. However, I still believe that that paper is not well balanced as I mentioned in Point 2. This lack of balance makes difficult to catch the major focus of this work.

In the concluding remarks, the authors are still far from providing a good non-speculative rationale for judgements on adequacy of user's speech evaluation. They didn't address the bias between the quality of speech and the quality of STT systems. Additions in lines 534-545 are interesting but aren't completely relevant to the goals of this specific work.

Pedagogical aspect of using the suggested approach for improving language proficiency have not been addressed.

Since other reviewers seem to be happy with this manuscript, and the editors can make the decision even without my acceptance, let me keep my opinion just as a sort of encouragement for further improvements of this work.

Author Response

Thank you for your kind input and comments. We have answered the reviews in a separate file, therefore, please see the attachment.

Author Response File: Author Response.pdf