Next Article in Journal
Cluster-Based Routing Protocol with Static Hub (CRPSH) for WSN-Assisted IoT Networks
Previous Article in Journal
Regional Ecology Supporting Sustainable Development
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sustainability
Volume 14
Issue 12
10.3390/su14127303
sustainability-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Effectiveness Study on Online or Blended Language Learning Based on Student Achievement: A Systematic Review of Empirical Studies

by
Tong Zhou
and
Wei Zhang
*
School of English and International Studies, Beijing Foreign Studies University, Beijing 100089, China
*
Author to whom correspondence should be addressed.
Sustainability 2022, 14(12), 7303; https://doi.org/10.3390/su14127303
Submission received: 28 April 2022 / Revised: 25 May 2022 / Accepted: 7 June 2022 / Published: 15 June 2022
(This article belongs to the Section Sustainable Education and Approaches)
Browse Figures
Versions Notes

Abstract

:
The ubiquitous impacts resulting from the COVID-19 pandemic have profoundly changed the education sector and marked research interest in online or blended learning can be witnessed. As a pervasive learning activity of paramount significance, online language learning has aroused widespread attention. Nonetheless, few systematic reviews concerning the effectiveness of online language learning have been published. With the help of CiteSpace, this study systematically investigated 103 included articles from the SSCI of empirical studies from 44 journals for the purpose of filling the research gap in this field, providing a better understanding of the research trends, exploring effective ways to implement online language courses, and testifying to the ability of CiteSpace to track research hotspots. The findings show that effectiveness studies on online language learning principally focus on assisted tools (42.72%), instructional approaches (36.89%), and specific courses (20.39%). Lack of adequate cooperation among research institutions and the dominant position of online English learning (82.52%) can be witnessed. Despite the small sample size of 103 included articles, the validation of CiteSpace in terms of tracking the research trends or hotspots is confirmed. However, the proportion of each research focus is not compatible with the results of a comprehensive full-text analysis. This literature review also probes into various methods to measure effectiveness more scientifically and effective ways to implement online language courses. Theoretical as well as practical implications and future research directions are clarified.

1. Introduction

The COVID-19 outbreak was declared by the World Health Organization in January 2020 [1], and it has spread across the globe and created a public health emergency [2,3]. However, as the indispensable sector of society, the education sector has properly addressed the challenges and problems by the replacement of traditional learning in the classroom with online learning and distance learning [4]. With the advancement of artificial intelligence, robotics, virtual reality, and the surge in digital information, an increasing number of courses are being implemented online, including some lectures of the university during the pandemic, which has given rise to substantial changes regarding the teaching, learning, and evaluation process. Teachers are required to learn more course-related technologies, while students have to be accustomed to the new learning mode [5]. These days, online learning has appealed to many researchers because of COVID-19 and quarantine measures, and a sharp interest in online and distance learning can be witnessed [6,7,8].
It is since 1998 that the Internet has been applied to online learning and teaching for students and teachers in the United States of America [9]. Despite the fact that online learning is often used interchangeably with distance learning, e-learning, and Internet-based learning, it was clearly defined as a student’s “access to learning experiences via the use of some technology” by Moore in 2011 [10] (p. 30). Likewise, online language learning refers to students’ experience of learning a language online supported by some technology. Blended learning, also known as hybrid learning, refers to “any combination of learning delivery methods, including most often face-to-face instruction with asynchronous and/or synchronous computer technologies” [11] (p. 231), which makes the best of traditional learning in classrooms and advanced technologies. This systematic review will include research that explores effectiveness based on students’ perceptions or achievements in both online learning and blended learning. Effectiveness was intended to be measured from the perspective of student achievement. If the language knowledge or skills of students are improved or enhanced, then it is said that effectiveness is confirmed. Language knowledge or skills may vary according to different learning content.
Although significant strides have been made to enhance the educational quality of online courses across the whole world, Viola argued that the effectiveness of online psychology courses is negative due to an obvious lack of engagement [12]. Is such a proposition consistent with the situation of those students learning a language online? The question calls for further attention; however, existing literature reviews concerning the effectiveness of language learning in online courses are rather scarce. Few systematic reviews about the effectiveness of relevant dimensions of online language learning can be found through database searching on the Web of Science. Acquah examined the effectiveness of digital games on high school students who were learning a second language, and the research method, gaming platform, game genre, design purpose of games, and key game features were analyzed. It was found that 70% of the report outcomes from the included studies were completely positive, and digital learning games were used as effective learning tools. Digital game-based language learning (DGBLL) was considered to be an interesting and engaging way to learn a second language [13]. Apart from this review, others have barely focused on the online learning effectiveness of courses in other disciplines such as health professions, and studies examining physiotherapy in particular [14,15,16,17,18] critically reviewed the effectiveness of online physiotherapy learning and users’ perceptions of it. Cook summarized the effectiveness of Internet-based health profession learning compared with no intervention and a noninternet intervention [19].
The objectives of this study are to fill in the gap that almost no studies concerning the effectiveness of online language learning can be found, to explore the effectiveness of online language learning from the perspective of student achievement, to provide effective methods to measure the effectiveness of online language learning as well as effective ways to implement online courses, and to testify to the ability of CiteSpace to track research trends or hotspots, which demonstrates both theoretical and practical contributions. This systematic review attempts to address the problems by reviewing empirical studies on learning outcomes or student achievement. As defined by the American Educational Research Association and the American Psychological Association, empirical studies are performed by direct or indirect means of observations or experiments, and research methods are not limited to what is commonly called qualitative and quantitative methods [20,21]. In the context of this study, effectiveness should be measured by direct or indirect means of observations or experiments, such as a combination of qualitative and quantitative approaches and mixed methods to collect data, including quasi-experimental design, interviews, surveys, questionnaires, self-reflection, observations, feedback, and so on. As for the significance of this study, it lies in the ability to provide a better understanding of the status quo about online language learning and shed light on how online learning influences language learners and its effectiveness based on student achievement, which refers to the overall performance of students after learning a language online, especially the enhancement of language proficiency, either language knowledge or skills. This study was carried out to answer the following questions:
(1) What are the status quos, research trends, or hotspots concerning the effectiveness of online or blended language learning?
(2) Which methods are appropriate for measuring the effectiveness of online language learning?
(3) How can teachers implement online language courses more effectively?
(4) Will the employment of CiteSpace be conducive to examining the research trends or focus in this field?
In this study, specific courses in online language learning refer to a series of language lessons or lectures given by teachers in order to help students to acquire new language knowledge or learn new language skills. As for assisted tools, they are defined as the thing, stuff, or instrument that students or teachers use in the online or blended courses to help them learn new language skills or achieve teaching goals. The main purpose of using assisted tools is to enhance the performance of students in learning a language. The instructional approaches in this study refer to the ways or methods that teachers use in language courses to facilitate students’ language learning in a more effective way so as to improve their language skills or knowledge. For the purpose of avoiding redundancy, online or blended learning hereinafter specifically refers to language learning in the context of this study.
This literature review is organized through five sections. In Section 1, the research background, significance, questions, and gap are presented. In Section 2, the research methods of this systematic review are clarified, including the process of database searching, inclusion and exclusion criteria, article screening, and data extraction. In Section 3, an overview of 103 included articles, a visualization analysis based on CiteSpace, and a comprehensive full-text analysis are made. Section 4 describes the interpretations of major findings, theoretical and practical implications, and future research directions. Concluding remarks are presented in Section 5.

2. Research Method

In this study, a systematic literature review method was adopted to demonstrate a review concerning online language learning effectiveness and grasp the state of the fast-growing and complex online learning domain. However, in many research fields, the complexity of science mapping is commonly shared [22]. In this research, CiteSpace, a widely used and continuously evolving software, was used to help to implement visual analytic functions. The version of the software used in this study was CiteSpace.6.1.R2 (developed by Chaomei Chen, http://cluster.cis.drexel.edu/~cchen/citespace/, accessed on 25 March 2022). With the help of CiteSpace, such functions as time distribution analysis, keyword co-occurrence analysis, cluster analysis, burstiness analysis, and co-citation analysis can be executed, and the research trends and relationships among multitudes of research topics can be understood [23,24]. The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) is a reporting guideline with a checklist of 27 items designed to avoid research bias and potential problems, which is conducive to obtaining scientific and objective reporting of systematic reviews [25]. This systematic literature review was conducted with the help of the PRISMA 2020 statement, which includes the checklists, explanation, and elaboration, as well as the flow diagram. In accordance with the PRISMA guidelines, the information sources, eligibility criteria, search strategy, selection process, and data collection process are described in detail. Therefore, the general research framework of this study included data collection through database searching, inclusion and exclusion criteria for the review, literature selection, data extraction, holistic analysis via CiteSpace, and in-depth analysis. The data collection and extraction under the guidance of PRISMA lay a solid foundation for the following comprehensive full-text analysis.

2.1. Data Collection

When it comes to the input data of this systematic review, the data were generated by the collection of literature from multiple search queries to the Web of Science database, including both the Social Science Citation Index (SSCI) and Arts & Humanities Citation Index (A&HCI), which ensured the quality and credibility of the search results. This study aimed to investigate the effectiveness concerning any topics in online or blended language learning based on student achievement. Due to the limited number of articles, it was attempted to acquire much more results at the beginning of the data collection process. Firstly, “online” or “distance” learning is more general than blended learning, which enabled to acquire more results regarding online learning. Moreover, the term “learning” or “course” is conducive to obtaining the effectiveness results from students’ perceptions or based on student achievement. Finally, the terms “language” and “effectiveness” were essential in this study, so they were mentioned in all four search terms of database searching. Hence, the terms for article searching included “online language learning effectiveness,” “distance language learning effectiveness,” “online language course effectiveness,” and “distance language course effectiveness.” As for the time span, it was chosen from 2006 to 2022 because the year 2005 saw increasing accessibility of information and communication technology (ICT), and the technology of online courses has demonstrated considerable advances since 2006 [26]. A review of technology-supported courses should take this factor into account, so the year 2006 was selected as the start year. The language of the literature was selected as English, and the document type was limited to articles only. The search queries and refinement procedures are shown in Table 1. Based on all the refinement procedures above, the query generated 222 results.

2.2. Specifying Inclusion and Exclusion Criteria

In line with the purpose of this systematic review, the following criteria were specified to remove irrelevant and redundant articles. The inclusion and exclusion criteria were as follows: (1) Empirical studies—the research methods should conform to the norms of empirical research, such as direct or indirect means of observations or experiments, which are not limited to qualitative and quantitative methods; thus, reviews or meta-analyses were excluded. (2) Language—the research should focus on online language learning instead of online courses in other disciplines, such as courses related to environment or disease. To be more specific, language refers to natural language in this context, such as Chinese, English, French, and so on. Thus, artificial language, such as programming language, was excluded. (3) Online—“online learning” refers to students’ “access to learning experiences via the use of some technology” [10] (p. 130). On the basis of this definition, the learning activities of the research should take place, whether purely online or in technology-supported classrooms. Both online learning and blended learning were taken into account. (4) Effectiveness—the study should focus on the effectiveness of courses and new technologies of online learning based on student achievement or students’ perceptions of online or blended learning.

2.3. Article Screening

Through rounds of article screening based on a thorough analysis of the titles, keywords, and abstracts of 222 articles that were found after the refinement procedures, 103 articles were finally identified to be relevant to and appropriate for this systematic review and were included in the research. Detailed screening steps are shown in Figure 1. Overall, article screening was carried out based on the aforementioned criteria. Twelve articles, whose types belong to review or meta-analysis, were excluded because they were not empirical studies, and their research methods were not direct or indirect means of observations or experiments. The learning content of 53 articles was not a language but some content from other disciplines. The students were acquiring knowledge about environment, anatomy, tobacco, disease, nursing leadership, dementia, agriculture, and so on, rather than language. Hence, these 53 articles were excluded. It should be noted that a large proportion of articles focused on programming languages, including Java, Python, and natural language processing. A total of 17 articles investigated traditional learning activities offline in classrooms. A small number of articles did not explore effectiveness from the perspective of student achievement. In total, 103 articles were selected for inclusion, while 84 articles were excluded.

2.4. Data Extraction

Inductive thematic analysis refers to “a process of coding the data without fitting into a preexisting coding frame, or the researcher’s analytics preconceptions” [27] (p. 83). With the help of this technique, all 105 articles were coded in order to keep track of the relevant literature, draw key information, and identify salient themes. At first, the articles were generally coded by researcher, methodology, journal, and publication date. Next, another round of coding was guided by language type, specific learning content, student type, major themes, key findings, reference number, and so on. Finally, all the findings of the selected articles were summarized. The specific learning content was coded in order to guarantee whether the course was a language course because this study only considered online language learning. Major themes were coded for classification of the research focus, and key findings were coded for clarification of the effectiveness of each research focus. Student type was coded for a general understanding of the participants in the study, especially for the convenience of scholars who aspire to conduct similar future research.

2.5. Data Analysis

A holistic analysis based on CiteSpace and a full-text in-depth analysis was conducted. Visualization analysis of the annual publications, journals, keyword co-occurrence, keyword burstiness, keyword cluster, leading countries or regions and research institutions, authors, and co-cited authors comprehensively revealed the pivotal content and research trends with reference to the effectiveness of online or blended language learning. The full-text in-depth analysis demonstrates as much detailed information as possible concerning the effectiveness of online or blended language learning.

3. Results

3.1. Overview of the Included Articles

According to the information of the queries, although the initial records were searched in both the SSCI and the A&HCI of the Web of Science, it was found that all of the 103 included articles belonged to the SSCI, including some articles belonging to both the SSCI and A&HCI. Table 2 shows the bibliographic statistics of all the included articles. An average of 13.95 citations per article and 1437 times cited can be observed, which further guarantees the quality and credibility of the included articles.
In accordance with the statistics in Figure 2, the annual publications concerning the effectiveness of online language learning remained stable from 2006 to 2012, with only two articles on average each year, showing that scholars paid little attention to this topic at that time. However, the number of publications has risen significantly since 2012, which illustrates that much progress has been made in online or blended language learning domains. Another significant rise in annual publications was observed in 2020, when COVID-19 began to affect activities in all walks of life across the whole world, especially in the education sector. Students and teachers have had to take advantage of online learning due to the spread of the pandemic and quarantine measures. During this period, researchers focused on the effectiveness of online language learning and aimed to enhance students’ online learning effectiveness. A total of 18 publications concerning this topic were found in 2021, which is the highest number from 2006 to 2022. Although the number decreased to 8 in 2022, it does not mean scholars are not interested in this topic anymore. Instead, it is primarily due to the incomplete calculation until April 2022. According to a rough estimate, it is believed that the number will continue to increase provided that all the publications in 2022 are calculated.
In Table 3, 15 journals are listed based on the number of included articles, and the journal Computer Assisted Language Learning has the highest proportion, accounting for 13.59%, with 14 articles published. The number of included publications in this journal is almost two times the number of articles published in the other journals. The quantity of articles published in the other journals is less than 10, whose proportion is relatively smaller. The journal Sustainability published four relevant articles, which account for 3.88%.

3.2. Visualization Analysis Based on CiteSpace

3.2.1. Keyword Co-Occurrence Analysis

The most prominent and important keywords and their co-occurrence relationship in the effectiveness of online or blended language learning are presented in Figure 3, which provides general knowledge of this topic. Based on the information from this figure, the investigators will master the key research content in this field. Based on the statistics provided by CiteSpace and Figure 3, the top 20 keywords are: “student, language, English, online, education, design, classroom, acquisition, learner, foreign language, communication, blended learning, distance learning, computer-assisted language learning, skill, impact, comprehension, framework, technology, environment,” through which the research trends can be witnessed. Figure 4 presents the keywords that co-occur with the keyword “language,” and the specific research content of language study can be analyzed. What stands out among all the keywords is the language of English, showing that English received much more attention than other languages. Hence, the effectiveness of learning English online, rather than other minor languages, received extensive attention, and it is still the research hotspot. It is evident that scholars have focused on the impact that online or blended learning has made on English language learning, especially the comprehension of various English skills. It can be observed from Figure 3 and Figure 4 that many keywords related to technological approaches are presented, and their node sizes are relatively larger. Hence, one potential research focus is related to these technological approaches, which refer to instructional approaches in this study, and the effectiveness of these instructional approaches in online or blended language learning might be one of the research hotspots. As for the centrality, the top five keywords with a higher centrality are “student (0.5), acquisition (0. 24), language (0.23), English (0.22), education (0.14)”. The higher the centrality is, the more important the keywords are.

3.2.2. Keyword Burstiness Analysis

Keywords with the strongest citation burst are listed in Figure 5. Some keywords describe the instructional approaches in online learning, such as collaborative learning, feedback, computer-assisted language learning, and computer-mediated communication, which suggests that scholars concentrated on the instructional approaches that are employed in online learning. For example, research into collaborative learning began in 2009 and ended in 2013. During that period, collaborative learning was a research hotspot, and such research trends lasted longer than other hotspots. Most of the keywords listed in Figure 5 lasted for 2 or 3 years, such as environment, computer, acquisition, technology, listening comprehension, and so on. The keywords with the strongest citation burst that lasted for at least 4 years include instruction, collaborative learning, comprehension, computer-assisted language learning, and framework.

3.2.3. Keyword Cluster Analysis

The clusters in the keyword co-occurrence map refer to the irregular areas where the keyword co-occurrence network has gathered together, and each cluster is provided with a label based on closely related words in the areas. Eleven clusters are presented in Figure 6, and the occurrence time of the keywords is also reflected in the figure. The smaller the number of the cluster is, the more keywords in the cluster are contained. In Figure 6, some clusters are associated with instructional approaches, such as flipped instruction, interactive learning, and collaborative learning, while some clusters are related to assisted tools in online or blended learning, such as dictionary and audio news trainer. Judging from the clusters, it can be implied that the studies concerning the effectiveness of online or blended learning are principally linked with instructional approaches and assisted tools.

3.2.4. Visualization Analysis of Leading Countries or Regions and Research Institutions

Figure 7 shows the collaborating countries or regions in this field of research, and Figure 8 shows the collaborating institutions. The larger the nodes are, the larger the number of publications that can be found from this country or region. Among all the research institutions from 30 countries or regions, institutions from China published the most articles regarding the effectiveness of online or blended learning. A total of 53 articles were found from China, indicating that China is one of the most contributing and active countries in this field. The second and third contributing countries are the USA and Iran, with a count of 20 and 16, respectively. Meanwhile, the collaborating countries with China are the USA, Japan, Malaysia, Iran, Saudi Arabia, Singapore, and so on. As for the institutions, National Cheng Kung University, Lunghwa University of Science and Technology, and The University of Hong Kong in China made the most significant contributions to the development of this field, with a total of 13 publications. It is fair to say that China is one of the leading countries in this field, and institutions from China are the major contributors to the studies on the effectiveness of online language learning. However, the results of the statistical analysis show that more collaborations between institutions and countries are required in order to make more progress in the studies related to this topic.

3.2.5. Visualization Analysis of Authors and Co-Cited Authors

On the basis of Figure 9, the authors who published the most articles about this topic can be found. Ya-Ting C. Yang, a professor at the Institute of Education and Centre for Teacher Education, National Cheng Kung University in China, published three articles about this topic. All the authors, including Y. Chen, W. Hsu, Y. Huang, C. Lin, G. Liu, S. Marandi, R. Shadiev, S. Tseng, W. Wu, and Y. Yang, published two articles. As is shown in Figure 10, Yu-Fen Yang, from the Graduate School of Applied Foreign Languages, National Yunlin University of Science and Technology, is the author who has been cited the most by the other authors, including W. Hwang, X. Sun, M. Asoodar, H. Huang, W. Hsu, Z. Ge, and so on. The total frequency of Yu-Fen Yang being cited is 13. D. Garrison ranks second with 12 citations, and R. Ellis third, with 11 citations. The other authors who have been cited frequently include R. Ellis, F. Davis, C. Chen, H. Chen, W. Wu, R. Mayer, and so on. It can be judged that the authors mentioned above are some of the most influential and significant co-cited authors in the study of online language learning effectiveness.

3.3. Comprehensive Full-Text Analysis of 103 Included Articles

Based on the co-occurrence keywords in Figure 3 and Figure 4, the top 25 keywords with the strongest citation burst in Figure 5, and the top 11 clusters in Figure 6, it was found that assisted tools and instructional approaches in online or blended learning were frequently presented. Although it was rather challenging to categorize all 103 included articles and summarize their themes on account of the fact that some articles focused on multiple themes, this study attempted to divide the effectiveness studies on online or blended learning into three types: the effectiveness of specific courses, assisted tools, and instructional approaches. The research hotspots of research into the effectiveness of online or blended language learning are assisted tools, which accounted for 42.72% of the 103 articles, and the proportion of articles exploring the effectiveness of instructional approaches was 36.89%. There were 22 articles investigating the effectiveness of specific courses, accounting for 20.39%.
Visualization network and in-depth full-text analysis revealed that significant improvement, qualitative progress, substantial gains, or positive enhancement were described as effective based on student achievement in 86.41% of all the included articles. As one of the predominant languages, online English learning received the most attention (82.52%), especially its effectiveness in English writing. Some useful assisted tools included LANGA, SW-PAL, DWright, VACLS, ANT, and VoIm, while the effective instructional approaches in online language learning included digital-game-based learning, online collaborative learning, online flipped writing, eTandem learning, and so forth.

3.3.1. Effectiveness Studies on Specific Courses in Online or Blended Language Learning

A total of 21 out of 103 articles were identified as effectiveness studies on specific courses in online or blended language learning, and 90.48% of studies focused on online English learning with English as the target language, including English writing, speaking, reading, translation, interpreting, grammar, English for Specific Purpose, and overall English learning. Except for American K-12 students, all of the students who were dedicated to learning a foreign language were from China, South Korea, Russia, Saudi Arabia, Iran, Turkey, and so forth. The majority of the studies reported an improvement in students’ language proficiency and confirmed the effectiveness of online language learning, and three researchers had mixed attitudes toward the effectiveness of online learning, and only one study demonstrated the negative effects of learning German pronunciation online. Massive open online courses, known as MOOCs, were first developed by the University of Manibota by George Siemens and Stephen Downes [28], and it is widely advocated in the whole world now, which was also discussed. Detailed information about the 21 articles can be observed in Table 4.

3.3.2. Effectiveness Studies on Assisted Tools in Online or Blended Language Learning

A total of 44 articles were identified as effectiveness studies on assisted tools in online or blended learning, among which 40 articles showed positive perceptions toward the assisted tools that were investigated in the experiment, accounting for 90.91%. Two articles showed negative perceptions, and two studies reported mixed attitudes toward the effectiveness of online tools, which accounted for 4.55%. Generally speaking, the target learning language was English, and the online tools aimed to assist students in learning English writing, vocabulary, grammar, collocations, listening, speaking, pronunciation, reading, interpreting, and overall skills. Table 5 shows other detailed information about the effectiveness studies on assisted tools.

3.3.3. Effectiveness Studies on Instructional Approaches in Online or Blended Language Learning

Thirty-eight articles were identified as effectiveness studies on instructional approaches among the 103 included articles, and English learning still received the most attention as the target language. Other languages included Chinese, German, Spanish, French, Italian, and Welsh. Positive outcomes accounted for 84.21%, and negative 2.6%. Researchers held mixed attitudes toward five instructional approaches, and three approaches were associated with online feedback. Among all the effectiveness studies on instructional approaches in online or blended learning, almost 30% of them paid attention to English writing, and the instructional approach included online academic writing tutors, automated online form of strategy instruction, online flipped writing, courseware-implemented task-based instruction, online indirect data-driven learning, computer-mediated collaborative writing, online film clip watching and writing, integration of online learner-centered blogging approach, and online corrective feedback. Qualitative progression or positive enhancement was confirmed with regard to the effectiveness of instructional approaches by nine scholars [94,95,96,97,98,99,100,101,102]. The other two studies showed mixed outcomes due to students’ learning proficiency and the types of online corrective feedback [103,104]. Table 6 shows a detailed description of all 38 articles.

4. Discussion

4.1. Interpretations of Major Findings

4.1.1. Extensive Research Interest in Assisted Tools and Instructional Approaches

As for the status quo and research trends concerning the effectiveness of online or blended language learning, the current studies on this topic mainly focus on the effectiveness of assisted tools (42.72%), instructional approaches (36.89%), and specific courses (20.39%) in online or blended language learning. Ya-Ting C. Yang and Yu-Fen Yang are considered to be the most influential researchers. In terms of the major journals, Computer Assisted Language Learning is the major research publication for those studies concerning the effectiveness of online language learning. In addition, the top five journals that publish most of the included articles are Computer Assisted Language Learning, Recall, Interactive Learning Environments, Computers Education, and Educational Technology Society. Sustainability ranks eighth with four articles. According to the visualization analysis based on CiteSpace, the research hotspot is the effectiveness of instructional approaches, such as collaborative learning, feedback, and assisted tools, such as blogs, which is consistent with the in-depth full-text analysis of all the articles. To be more accurate, the number of studies on assisted tools is more than that on instructional approaches with six articles.
The results of this study indicate that the number of articles with regard to the effectiveness of online or blended language learning has increased drastically since 2020, the year of the outbreak of COVID-19. The number of articles related to this topic is also expected to increase in 2022, confirming that the pandemic has influenced the education mode significantly, and effective ways to implement online learning against the background of COVID-19 are continuously being explored so as to enhance teaching quality and students’ learning effectiveness. Students’ online learning experience and academic achievements were indeed influenced during the COVID-19 pandemic, which is in agreement with the findings of the research carried out by Omar [2].

4.1.2. Lack of Adequate Cooperation among Research Institutions

One unanticipated finding is that there is inadequate cooperation among various research institutions in this field. The top three contributing research institutions are all from China—National Cheng Kung University, Lunghwa University of Science and Technology, and The University of Hong Kong—indicating that China is also one of the leading countries. Nonetheless, all the nodes in Figure 8 locate randomly without showing obvious connections, and only the nodes between Alzahra University and Iran University of Science and Technology can be traced. Even in China, where most of the studies in this field were conducted, fewer links can be seen between research institutions. National Central University seems to cooperate with other institutions more frequently, but the number of collaborations is also rather limited. Therefore, it is fair to say that there is an obvious lack of collaboration between research institutions when investigating the effectiveness of online language learning.

4.1.3. Dominant Position of Online English Learning

Another pivotal finding is the dominant position of English as the target learning language that is investigated. Among all the keywords presented in Figure 3 based on CiteSpace, English is the only language that appears in the figure, which is in line with the in-depth analysis. As one of the dominant languages in the world, English attracts the most attention from scholars who are dedicated to studying effectiveness in online or blended language learning, with a proportion of 82.52%. Almost all aspects of English learning were investigated, such as English writing, listening, speaking, reading, grammar, vocabulary, translation, and interpreting. English writing was the most popular learning content that was investigated with regard to effectiveness studies on online or blended learning. Varieties of assisted tools and instructional approaches were applied to English writing online courses so as to enhance the students’ writing performance. Not only did students learn English the most online, but also the researchers focused on the effectiveness of learning English the most online. Despite the fact that other minor languages were also investigated, such as Dutch, Turkish, Croatian, Welsh, and so on, effectiveness studies on learning those languages online still account for a small proportion.

4.1.4. Various Methods Combined to Measure Effectiveness

A thorough analysis of 103 included articles revealed that the research methodology of effectiveness studies on online or blended learning usually combined qualitative and quantitative approaches and employed mixed methods to collect data, including quasi-experimental design, interviews between teachers and students, semi-structured interviews between educators and language program providers, cross-sectional learning satisfaction surveys, questionnaires, self-reflection, transcripts of interactions, observation, peer and instructor feedback, and so forth, which guarantees the quality and credibility of the data in the experiment. The employment of questionnaires and semi-structured interviews to collect data was frequently mentioned when the researchers aspired to learn about the true perceptions of both teachers and students toward specific courses, assisted tools, or instructional approaches. However, it is worth mentioning that the reliability and validity of various items in the questionnaires must be tested for the purpose of guaranteeing the accuracy of measuring effectiveness and further analysis [47]. Moreover, the questionnaires were usually administered at least two times at the beginning of the online language learning and the end of learning, depending on the research subjects as well as the data collected.

4.2. Theoretical and Practical Implications

4.2.1. Theoretical Implications

One of the most significant contributions of this study is to confirm the validation of CiteSpace in terms of tracking the research trends or hotspots in certain fields. In this study, one of the purposes of using CiteSpace was to judge the possible research focus regarding studies on the effectiveness of online or blended language learning. Besides the utilization of CiteSpace, a comprehensive, in-depth analysis of the research focus of all the 103 articles was made, which contributed to dividing the 103 articles into three types, namely, the effectiveness of assisted tools (42.72%), instructional approaches (36.89%), and specific courses (20.39%). After comparing the major findings of CiteSpace and those of the full-text analysis, it was found that they correspond to each other. For one thing, the results of the in-depth analysis are consistent with the research focus on the effectiveness of instructional approaches based on Figure 3, Figure 4, Figure 5 and Figure 6. In Figure 3, one of the most notable nodes is “computer-mediated communication.” In Figure 4, the co-occurrent keywords with “language” are as follows: “feedback, data-driven learning, computer-interactive reading, collaborative learning, computer-assisted language learning,” and so on. In Figure 5, “instruction, collaborative learning, computer-assisted language learning, and computer-mediated communication” are listed. In Figure 6, the second cluster is “flipped instruction,” the third cluster is “interactive learning,” and the sixth cluster is “collaborative learning.” Hence, it is believed that the major findings of the comprehensive full-text analysis are compatible with the analysis of the research focus based on all of the nodes, keywords, and clusters aforementioned in Figure 3, Figure 4, Figure 5 and Figure 6. For another, the research focus on the effectiveness of assisted tools is consistent with the major findings of the top 11 clusters in Figure 6 based on CiteSpace, which include the fifth cluster of “dictionary” and the eighth cluster of “audio news trainer.” The investigation into assisted tools is also compatible with the top 25 keywords with the strongest citation burst in Figure 5, such as “computer, blog, technology,” and so forth. Likewise, the same is also true for the research focus on the effectiveness of specific courses. Therefore, it is fair to say that this study contributes to the validation of the use of CiteSpace to track research trends and hotspots.
Another significant theoretical implication is that the real proportion of each research focus by comprehensive full-text analysis is not in line with the proportion of the research focus judged by the nodes, keywords with the citation burst, and clusters provided by CiteSpace. For instance, in accordance with Figure 3, Figure 4, Figure 5 and Figure 6, most of the research is connected with instructional approaches because they account for a larger proportion in terms of keywords or clusters. However, the studies concerning the effectiveness of instructional approaches account for 36.89% based on the findings of the full-text analysis, with a gap of 5.89%. The inconsistency may be attributed to the limited quantity of the articles collected in this study or the principle of categorization. Provided that more articles are included, the proportion of each research focus based on CiteSpace might be more accurate.
A comparison of the findings with those of other studies confirms the general effectiveness of online language learning, which is also consistent with that of Zou [47], who concluded that only 0.47% of participants considered online language learning ineffective or extremely ineffective, which means 99.53% of the participants recognized the effectiveness of online learning. However, the major findings of this study are different from those of the research conducted by Martin [29], who held that the pronunciation skills of German learners did not improve significantly after having courses the whole semester. In contrast to this study, the effect sizes for those distance German pronunciation learners are negligible primarily due to the lack of classroom interaction, less production of spoken German, and less feedback about their communications through German. As for the reasons for the discrepancy between the major findings of the two studies, the language type should be taken into account. The majority of the target languages that were analyzed were English, which is slightly different from German. Online English pronunciation learning might be suitable for distance learners, but those distance German learners may find it ineffective to learn it online, German pronunciation in particular. Another possible reason is whether the learners accepted target pronunciation training, which was not clarified by the researchers in most of the included articles.

4.2.2. Practical Implications

This study provides new insight into the most effective ways to implement language courses online. When it comes to the effectiveness based on students’ achievement or learners’ perception, 89 out of 103 articles reported positive outcome, with descriptions of robust improvement, significant enhancement, positive gains, confirmed effectiveness, high satisfaction, better achievement, substantial outperformance, and so on. Only four articles demonstrated insignificant achievement or negative perceptions of the effectiveness. In addition, 9.71% of the studies revealed mixed perceptions toward the effectiveness investigated due to learners’ language proficiency, identity, or other factors. Some assisted tools were deemed rather effective: LANGA for learning Spanish vocabulary, ANT for learning English listening, SW-PAL for learning English writing, Memrise for learning English vocabulary, VACLS for learning English collocations, Twitter for learning English pronunciation, and ORCIT for learning English interpreting. Online game-based interactive learning, online corrective feedback, online flipped writing, mobile-assisted pronunciation training, eTandem learning, and online collaborative learning were all effective instructional approaches to online language learning.
The advantages of learning a language online were acknowledged. The affordance, flexibility, convenient retrieval, and interaction observations of MOOCs were highly praised by hospitality English learners because information about the subject can be retrieved anytime, anywhere, without disappearing [42]. Spatial and temporal convenience, self-paced learning, and one-to-one feedback were considered the major advantages of online learning when trainees learn English translation and interpreting online. They were generally satisfied with the blended mode and more satisfied with online translation learning than interpreting. According to over half of trainees’ evaluations, online translation classes are as effective as traditional learning in classrooms [39]. Generally speaking, online language learning has exerted a significant positive influence on students’ competence in various kinds of language skills. In all the 103 included articles, participants included undergraduates, postgraduates, adult learners, elementary school students, junior high school students, deaf students, or students with disabilities. They came from a large number of countries, including China, the United States, the United Kingdom, Russia, France, Saudi Arabia, Malaysia, Japan, Turkey, Spain, Australia, Croatia, the Czech Republic, and so on.
In addition, this study probed into the factors that may negatively affect the effectiveness of online language learning; firstly, from the perspective of online language learners, their identity, and their proficiency level matter. Among all the 17 researchers who investigated the effectiveness of specific courses, 3 of them argued that the effectiveness of online language learning depends on students’ identity. The online courses with project-based assignments and higher-level knowledge activities were not helpful to credit-recovery students, but they tended to improve noncredit-recovery students’ learning outcomes, who logged in more frequently and stayed logged in for longer [41]. There were some negative beliefs about online Dutch learning, and some participants in the interview held that online Dutch learning was only suitable for those students with high proficiency levels. For those with low literacy, they were unable to learn online. The constraints on the effectiveness of online language learning include resources as well as technical and pedagogical support [30]. Secondly, when it comes to teaching methods, target training is of paramount significance. Classroom interaction and feedback are important for online pronunciation learners, without which their skills are slow to improve, which can be observed from the comparison of purely online learning and its combination with innovative Cued Pronunciation Reading (iCPRs) [29]. Even though students obtained substantial gains through online language learning, those who followed the standard curriculum without targeted pronunciation training still do not show obvious improvement, which further testifies to the significance of the target training when students learn a language online. Thirdly, the specific learning content also makes a difference in the effectiveness of learning a language online. The effectiveness of learning online English grammar for Saudi undergraduates was mixed, depending on grammar structures. Online English grammar learning “seems to have no or little effect” on the achievement of learners of limited language levels when they learned simple grammar structures, such as a simple sentence. Instead, when they learned complex sentences and compound sentences, online learning was more functional because they could deduce English sentence patterns from samples of the language they had mastered [40] (p. 334).
Despite mounting evidence of the positive influence of learning a language online, there are still some problems and challenges that teachers and students have to confront. One of the key challenges for teachers is to keep up with increasingly evolving technologies. The workload of online language teaching is much more than that of traditional learning in classrooms, which puts a heavier burden on teachers and makes them reluctant to incorporate new technologies into their classrooms. Moreover, the expensive infrastructures are also a prerequisite for course designers. It is reported that teachers lack sufficient formal training about the advantages of technology, and some teachers even have no relevant prior experience, especially senior teachers. Therefore, technology literacy for teachers is a crucial skill that should be taken into serious account before designing online language courses. As for students, lack of interaction and engagement in online language learning contributes to most of the dissatisfaction. It is a widely held view that the atmosphere of sitting before a computer is completely different from the atmosphere in the classroom, and it is easy for those students with less self-control to be distracted by other things. Limited access to technologies and Internet connections also poses greater challenges to students who need to learn online due to COVID-19 or other constraints. Adequate technical and pedagogical help must be provided by institutional policymakers and administrators.

4.3. Future Research Directions

After analyzing the interpretations of the major findings and theoretical as well as practical implications, it is believed that despite these promising results, several questions remain unanswered at present.
To begin with, future studies should focus more on online minor language learning due to the currently excessive interest in online English learning. Although the effectiveness of learning English online has been thoroughly investigated, the effectiveness of learning a minor language online is also of great significance. The most targeted language of studies on online language learning is English, and a large multitude of researchers have devoted themselves to learning activities in English writing, reading, speaking, vocabulary, grammar, translation, and so on. Figure 4 further testifies to this proposition, and it was found that the keyword “language” is associated with English more frequently than other languages. Although Arab is presented in Figure 4, the node size is far smaller than English, which indicates that English is still the research hotspot among all the languages. As a matter of fact, the effectiveness of learning other minor languages online also deserves attention; therefore, for future studies, the effectiveness of learning other minor languages, such as French, German, Spanish, Turkish, and so on, which have seldom been paid attention to, should be carried out. Future research is also required to establish whether the type of language is a factor in the effectiveness of online language learning. Provided that the assisted tools and instructional approaches are the same, will the effectiveness then be different when students learn different languages online?
Furthermore, studies concerning the most effective assisted tools for learning English writing online should be conducted. Online English writing has received much more attention from scholars, and the effectiveness of numerous assisted tools has been analyzed. However, there is no comparison of the effectiveness among varieties of tools. The most effective tools in this field are still unknown. The CSCL system, SW-PAL, DWright, online automated essay evaluation system, online forums, blogs, and wikis, as well as online corpora, are all considered to be effective assisted tools in online or blended English writing courses. The factors affecting the effectiveness of assisted tools for learning English writing online should be explored. Only one tool showed no significant improvement, that is, EJP-Write, an online writing tutorial system. The results of 25 graduate students who aspired to enhance their English academic writing skills indicated that the improvements were not significant, and the tools could not completely meet their anticipations primarily due to the Chinese-interfaced system. The students could not read Mandarin Chinese, and they could not use the tool directly and easily because of language barriers [60]. Compared with EJP-Write, another online writing tutorial system, DWright, was incorporated into blended courses successfully, and the system served as an effective mediating tool to provide English writing learners, especially EFL undergraduates, with sufficient practice, helping the students to enhance their English writing. In further studies, researchers should carry out studies on identifying the most effective courses, assisted tools, or instructional approaches for each language skill in online learning under the guidance of mathematical statistics so as to provide a reference for students and teachers to implement online language learning more efficiently.
In addition, more cooperation should be established among different research institutions. At present, the majority of studies were carried out in China, where abundant research results have been made. Nonetheless, inadequate cooperation leads to the simplicity of participants who learn a language online. The results might not be applied to those students who learn a language online in other parts of the world, and the practical implications may be limited to only a small number of participants. Due to the dispersion of online or distance learners’ locations, it is feasible to include participants from other regions of the world. More cooperation will also be strengthened among research institutions in this way.
Finally, in this study, the proportion of each research focus based on CiteSpace was found to be inconsistent with the proportion of that through comprehensive full-text analysis, which might result from the limited number of included articles. As for further studies, more included articles, such as at least 500, should be included in order to certify the proportion of research focus between the findings of CiteSpace and in-depth analysis.

5. Conclusions

In this study, 103 empirical studies from the SSCI were systematically summarized for the purpose of mapping existing studies concerning the effectiveness of online or blended language learning.
The extensive research interest in assisted tools (42.72%), as well as instructional approaches (36.89%), and the lack of adequate cooperation among research institutions in this field were confirmed. English was deemed the dominant language in this field, and various methods to measure the effectiveness of online language learning more scientifically were analyzed. As for theoretical contributions, this study affirmed the validation of the utilization of CiteSpace in terms of tracking research trends or hotspots. However, the proportion of each research focus judged by CiteSpace was not consistent with that of comprehensive full-text analysis. The comparison of the findings also certified the major findings and interpretation of them in other studies. When it comes to practical contributions, the effective ways to implement online language courses, the advantages of learning a language online, the factors that negatively affect the effectiveness of online language learning, and the problems and challenges that teachers and students have to confront were discussed in detail.
This study leaves several avenues for future research. Online minor language learning should be paid adequate attention to, and its effectiveness also matters. Studies concerning the comparison of the effectiveness of all the assisted tools in one specific class, such as English writing, should be carried out in order to obtain the most effective assisted tools. Research institutions can cooperate more with those in other parts of the world to conduct cross-region research so as to obtain statistics of some other participants across the world, especially those learners of online or distance learning. More articles should be included to testify the proportion of each research focus based on CiteSpace.
The generalizability of the results is limited by the sample size because only a limited number of language learning was investigated, whereas there are large multitudes of languages in this world. Although the effectiveness of learning some languages online is confirmed, such as English, Chinese, Russian, and Spanish, there are still many languages left to be explored. The major limitations of this study also lie in the categorization of all the included studies. Some studies combined both assisted tools and instructional approaches, and it was rather challenging to distinguish one major theme from another. Some researchers assessed the effectiveness of specific courses based on one particular assisted tool and one particular instructional method, which also made it difficult to categorize their studies.

Author Contributions

Conceptualization, T.Z. and W.Z.; methodology, W.Z.; software, T.Z.; formal analysis, T.Z.; investigation, T.Z. and W.Z.; resources, W.Z.; data curation, T.Z.; writing—original draft preparation, T.Z.; writing—review and editing, W.Z.; visualization, T.Z.; supervision, W.Z.; project administration, W.Z.; funding acquisition, W.Z. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the National Social Science Fund of China, grant number 19ZDA338; the Fundamental Research Funds for Central Universities, grant number 2022JS004; and the Excellent Researcher Program at Beijing Foreign Studies University.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Acknowledgments

Genuine acknowledgments go to the editors and anonymous reviewers for their enlightening and insightful suggestions.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Bender, L. Key Messages and Actions for COVID-19 Prevention and Control in Schools; UNICEF: New York, NY, USA, 2020; Available online: https://tinyurl.com/eyrn44a (accessed on 10 April 2022).
  2. Omar, H.A.; Ali, E.M.; Belbase, S. Graduate Students’ Experience and Academic Achievements with Online Learning during COVID-19 Pandemic. Sustainability 2021, 13, 13055. [Google Scholar] [CrossRef]
  3. Mahase, E. China coronavirus: WHO declares international emergency as death toll exceeds 200. BMJ 2020, 368, m408. Available online: https://www.bmj.com/content/368/bmj.m408 (accessed on 8 April 2022). [CrossRef] [PubMed] [Green Version]
  4. Ahmad, S.F.; Rahmat, M.K.; Mubarik, M.S.; Alam, M.M.; Hyder, S.I. Artificial Intelligence and Its Role in Education. Sustainability 2021, 13, 12902. [Google Scholar] [CrossRef]
  5. Krishnan, S.D.; Norman, H.; Yunus, M.M. Online Gamified Learning to Enhance Teachers’ Competencies Using Classcraft. Sustainability 2021, 13, 10817. [Google Scholar] [CrossRef]
  6. Zhu, Y.; Tan, J.C.; Cao, Y.; Liu, Y.L.; Liu, Y.Z.; Zhang, Q.; Liu, Q. Application of Fuzzy Analytic Hierarchy Process in Environmental Economics Education: Under the Online and Offline Blended Teaching Mode. Sustainability 2022, 14, 2414. [Google Scholar] [CrossRef]
  7. Saleh, S.S.; Nat, M.; Aqel, M. Sustainable Adoption of E-Learning from the TAM Perspective. Sustainability 2022, 14, 3690. [Google Scholar] [CrossRef]
  8. Stoyanova, T.; Stoyanov, P.; Remnova, A.; Kushniruk, S.; Rakityanska, L.; Drobyazko, S. System-Cluster Technology of e-Learning Improvement under the Conditions of COVID-19. Sustainability 2021, 13, 14024. [Google Scholar] [CrossRef]
  9. Wallis, J.W.; Parker, J.A. Use of the Internet for Teaching in Nuclear Medicine. Semin. Nucl. Med. 1998, 28, 165–176. [Google Scholar] [CrossRef]
  10. Moore, J.L.; Dickson-Deane, C.; Galyen, K. e-learning, online learning, and distance learning environments: Are they the same? Internet High. Educ. 2011, 14, 129–135. [Google Scholar] [CrossRef]
  11. So, H.J.; Brush, T.A. Student perceptions of collaborative learning, social presence and satisfaction in a blended learning environment: Relationships and critical factors. Comput. Educ. 2008, 51, 318–336. [Google Scholar] [CrossRef]
  12. Viola, S.; Saeki, E.; Hendricker, E. Distance education in graduate training programs: Lessons learned from school psychology students. J. Educ. Online 2019, 16, n2. Available online: http://eric.ed.gov/?id=EJ1223939 (accessed on 8 April 2022).
  13. Acquah, E.O.; Katz, H.T. Digital game-based L2 learning outcomes for primary through high-school students: A systematic literature review. Comput. Educ. 2020, 143, 19. [Google Scholar] [CrossRef]
  14. Vaona, A.; Banzi, R.; Kwag, K.H.; Rigon, G.; Cereda, D.; Pecoraro, V.; Tramacere, I.; Moja, L. E-learning for health professionals. Cochrane Database Syst. Rev. 2018, 1–84. [Google Scholar] [CrossRef] [PubMed]
  15. Car, L.T.; Soong, A.; Kyaw, B.M.; Chua, K.L.; Low-Beer, N.; Majeed, A. Health professions digital education on clinical practice guidelines: A systematic review by Digital Health Education collaboration. BMC Med. 2019, 17, 139. [Google Scholar] [CrossRef] [Green Version]
  16. Kononowicz, A.A.; Woodham, L.A.; Edelbring, S.; Stathakarou, N.; Davies, D.; Saxena, N.; Car, L.T.; Carlstedt-Duke, J.; Car, J.; Zary, N. Virtual Patient Simulations in Health Professions Education: Systematic Review and Meta-Analysis by the Digital Health Education Collaboration. J. Med. Internet Res. 2019, 21, 20. [Google Scholar] [CrossRef] [PubMed]
  17. Kyaw, B.M.; Saxena, N.; Posadzki, P.; Vseteckova, J.; Nikolaou, C.K.; George, P.P.; Divakar, U.; Masiello, I.; Kononowicz, A.A.; Zary, N.; et al. Virtual Reality for Health Professions Education: Systematic Review and Meta-Analysis by the Digital Health Education Collaboration. J. Med. Internet Res. 2019, 21, 13. [Google Scholar] [CrossRef] [Green Version]
  18. Macznik, A.K.; Ribeiro, D.C.; Baxter, G.D. Online technology use in physiotherapy teaching and learning: A systematic review of effectiveness and users’ perceptions. BMC Med. Educ. 2015, 15, 160. [Google Scholar] [CrossRef] [Green Version]
  19. Cook, D.A.; Levinson, A.J.; Garside, S.; Dupras, D.M.; Erwin, P.J.; Montori, V.M. Internet-based learning in the health professions—A meta-analysis. JAMA-J. Am. Med. Assoc. 2008, 300, 1181–1196. [Google Scholar] [CrossRef]
  20. Duran, R.P.; Eisenhart, M.A.; Erickson, F.D.; Grant, C.A.; Green, J.L.; Hedges, L.V.; Levine, F.J.; Moss, P.A.; Pellegrino, J.W.; Schneider, B.L. Standards for Reporting on Empirical Social Science Research in AERA Publications. Educ. Res. 2006, 35, 33–40. Available online: https://www.academia.edu/32050123/Standards_for_Reporting_on_Empirical_Social_Science_Research_in_AERA_Publications?from=cover_page (accessed on 10 April 2022).
  21. Hughes, S.; Pennington, J.L.; Makris, S. Translating Autoethnography Across the AERA Standards: Toward Understanding Autoethnographic Scholarship as Empirical Research. Educ. Res. 2012, 41, 209–219. [Google Scholar] [CrossRef]
  22. Chen, C. Science Mapping: A Systematic Review of the Literature. J. Data Inf. Sci. 2017, 2, 1–40. [Google Scholar] [CrossRef] [Green Version]
  23. Lu, Y.; Hong, X.; Xiao, L. Toward High-Quality Adult Online Learning: A Systematic Review of Empirical Studies. Sustainability 2022, 14, 2257. [Google Scholar] [CrossRef]
  24. Chen, C. A Practical Guide for Mapping Scientific Literature; Nova Science Publishers: Hauppauge, NY, USA, 2016. [Google Scholar]
  25. Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. Int. J. Surg. 2021, 88, 105906. [Google Scholar] [CrossRef] [PubMed]
  26. Jin, B.; Kim, J.; Baumgartner, L.M. Informal Learning of Older Adults in Using Mobile Devices: A Review of the Literature. Adult Educ. Q. 2019, 69, 120–141. [Google Scholar] [CrossRef]
  27. Braun, V.; Clark, V. Using thematic analysis in psychology. Qual. Res. Psychol. 2006, 3, 77–101. [Google Scholar] [CrossRef] [Green Version]
  28. Mackness, J.; Waite, M.; Roberts, G.; Lovegrove, E. Learning in a Small, Task-Oriented, Connectivist MOOC: Pedagogical Issues and Implications for Higher Education. Int. Rev. Res. Open Distrib. Learn. 2013, 14, 140–159. [Google Scholar] [CrossRef] [Green Version]
  29. Martin, I.A. Pronunciation development and instruction in distance language learning. Lang. Learn. Technol. 2020, 24, 86–106. Available online: http://hdl.handle.net/10125/44711 (accessed on 6 April 2022).
  30. De Paepe, L.; Zhu, C.; Depryck, K. Development and implementation of online Dutch L2 courses in adult education: Educators’ and providers—Perceptions of constraints and critical success factors. Innov. Lang. Learn. Teach. 2018, 13, 277–291. [Google Scholar] [CrossRef]
  31. Saeed, M.A.; Alharbi, M.A.; Yassin, A.A. Sustaining Synchronous Interaction Effectiveness in Distance Writing Courses: A Mixed Method Study in a KSA University. Sustainability 2021, 13, 13675. [Google Scholar] [CrossRef]
  32. Zhou, C.Y. Empirical Study on the Effectiveness of Teaching Model of College English Writing within Blended Learning Mode. Educ. Sci. Theory Pract. 2018, 18, 1060–1076. [Google Scholar] [CrossRef] [Green Version]
  33. Pan, H.; Xia, F.; Kumar, T.; Li, X.; Shamsy, A. Massive Open Online Course Versus Flipped Instruction: Impacts on Foreign Language Speaking Anxiety, Foreign Language Learning Motivation, and Learning Attitude. Front. Psychol. 2022, 13, 833616. [Google Scholar] [CrossRef] [PubMed]
  34. Aydin, B. An e-class application in a Distance English Language Teacher Training program (DELTT): Turkish learners’ perceptions. Interact. Learn. Environ. 2008, 16, 157–168. [Google Scholar] [CrossRef]
  35. Huang, H.C. Online reading strategies at work: What teachers think and what students do. ReCALL 2013, 25, 340–358. [Google Scholar] [CrossRef]
  36. Bailey, D.; Almusharraf, N.; Hatcher, R. Finding satisfaction: Intrinsic motivation for synchronous and asynchronous communication in the online language learning context. Educ. Inf. Technol. 2021, 26, 2563–2583. [Google Scholar] [CrossRef] [PubMed]
  37. Yang, Y.T.C.; Chuang, Y.C.; Li, L.Y.; Tseng, S.S. A blended learning environment for individualized English listening and speaking integrating critical thinking. Comput. Educ. 2013, 63, 285–305. [Google Scholar] [CrossRef]
  38. Yang, Y.T.C.; Gamble, J.H.; Hung, Y.W.; Lin, T.Y. An online adaptive learning environment for critical-thinking-infused English literacy instruction. Br. J. Educ. Technol. 2013, 45, 723–747. [Google Scholar] [CrossRef]
  39. Lee, J.; Huh, J. Why not go online? A case study of blended mode business interpreting and translation certificate program. Interpret. Transl. Train. 2018, 12, 444–466. [Google Scholar] [CrossRef]
  40. Abuseileek, A.F. The effect of using an online-based course on the learning of grammar inductively and deductively. ReCALL 2009, 21, 319–336. [Google Scholar] [CrossRef]
  41. Zheng, B.B.; Lin, C.H.; Kwon, J.B. The impact of learner-, instructor-, and course-level factors on online learning. Comput. Educ. 2020, 150, 11. [Google Scholar] [CrossRef]
  42. Hsu, R.L.W. A Grounded Theory Exploration of Language Massive Open Online Courses (LMOOCs): Understanding Students’ Viewpoints. Sustainability 2021, 13, 2577. [Google Scholar] [CrossRef]
  43. Zhang, R. Blended Course Evaluation in the Context of English for Specific Purposes: Accountability and Development. SAGE Open 2021, 11, 21582440211054502. [Google Scholar] [CrossRef]
  44. Shi, J.Y.; Fan, L.D. Investigating Teachers’ and Students’ Perceptions of Online English Learning in a Maritime Context in China. SAGE Open 2021, 11, 21582440211040800. [Google Scholar] [CrossRef]
  45. Kamal, M.I.; Zubanova, S.; Isaeva, A.; Movchun, V. Distance learning impact on the English language teaching during COVID-19. Educ. Inf. Technol. 2021, 26, 7307–7319. [Google Scholar] [CrossRef] [PubMed]
  46. Mellati, M.; Khademi, M. MOOC-based educational program and interaction in distance education: Long life mode of teaching. Interact. Learn. Environ. 2018, 28, 1022–1035. [Google Scholar] [CrossRef]
  47. Zou, B.; Huang, L.L.; Ma, W.L.; Qiu, Y.Q. Evaluation of the effectiveness of EFL online teaching during the COVID-19 pandemic. SAGE Open 2021, 11, 21582440211054491. [Google Scholar] [CrossRef]
  48. Lin, S.L.; Wen, T.H.; Ching, G.; Huang, Y.C. Experiences and Challenges of an English as a Medium of Instruction Course in Taiwan during COVID-19. Int. J. Environ. Res. Public Health 2021, 18, 12920. [Google Scholar] [CrossRef] [PubMed]
  49. Lai, C.; Shum, M.; Tian, Y. Enhancing learners’ self-directed use of technology for language learning: The effectiveness of an online training platform. Comput. Assist. Lang. Learn. 2016, 29, 40–60. [Google Scholar] [CrossRef]
  50. Arslanyilmaz, A. Computer-assisted foreign language instruction: Task based vs. form focused. J. Comput. Assist. Learn. 2013, 29, 303–318. [Google Scholar] [CrossRef]
  51. Shadiev, R.; Huang, Y.M. Facilitating cross-cultural understanding with learning activities supported by speech-to-text recognition and computer-aided translation. Comput. Educ. 2016, 98, 130–141. [Google Scholar] [CrossRef]
  52. Levak, N.; Son, J.B. Facilitating second language learners’ listening comprehension with Second Life and Skype. ReCALL 2016, 29, 200–218. [Google Scholar] [CrossRef]
  53. Usai, F.; O’Neil, K.G.R.; Newman, A.J. Design and Empirical Validation of Effectiveness of LANGA, an Online Game-Based Platform for Second Language Learning. IEEE Trans. Learn. Technol. 2018, 11, 107–114. [Google Scholar] [CrossRef]
  54. Vazquez-Cano, E.; Mengual-Andres, S.; Lopez-Meneses, E. Chatbot to improve learning punctuation in Spanish and to enhance open and flexible learning environments. Int. J. Educ. Technol. High. Educ. 2021, 18, 20. [Google Scholar] [CrossRef]
  55. Elbourhamy, D.M.; Mohammdi, H.M. An intelligent system to help deaf students learn Arabic Sign Language. Interact. Learn. Environ. 2021, 1–16. [Google Scholar] [CrossRef]
  56. Debevc, M.; Stjepanovic, Z.; Holzinger, A. Development and evaluation of an e-learning course for deaf and hard of hearing based on the advanced Adapted Pedagogical Index method. Interact. Learn. Environ. 2014, 22, 35–50. [Google Scholar] [CrossRef]
  57. Yang, Y.F. Transforming and constructing academic knowledge through online peer feedback in summary writing. Comput. Assist. Lang. Learn. 2015, 29, 683–702. [Google Scholar] [CrossRef]
  58. Chew, C.S.; Wu, W.C.V.; Idris, N.; Loh, E.F.; Chua, Y.P. Enhancing Summary Writing of ESL Learners via a Theory-Based Online Tool: System Development and Evaluation. J. Educ. Comput. Res. 2019, 58, 398–432. [Google Scholar] [CrossRef]
  59. Liu, G.Z.; Lu, H.C.; Lin, V.; Hsu, W.C. Cultivating undergraduates’ plagiarism avoidance knowledge and skills with an online tutorial system. J. Comput. Assist. Learn. 2018, 34, 150–161. [Google Scholar] [CrossRef]
  60. Hsu, W.C.; Liu, G.Z. Genre-based writing instruction blended with an online writing tutorial system for the development of academic writing. Digit. Scholarsh. Humanit. 2018, 34, 100–123. [Google Scholar] [CrossRef]
  61. Wang, Z.J. Computer-assisted EFL writing and evaluations based on artificial intelligence: A case from a college reading and writing course. Libr. Hi Tech 2020, 40, 80–97. [Google Scholar] [CrossRef]
  62. Miyazoe, T.; Anderson, T. Learning outcomes and students’ perceptions of online writing Simultaneous implementation of a forum, blog, and wiki in an EFL blended learning setting. System 2010, 38, 185–199. [Google Scholar] [CrossRef]
  63. Asoodar, M.; Atai, M.R.; Vaezi, S.; Marandi, S.S. Examining effectiveness of communities of practice in online English for academic purposes (EAP) assessment in virtual classes. Comput. Educ. 2014, 70, 291–300. [Google Scholar] [CrossRef]
  64. Crosthwaite, P.; Storch, N.; Schweinberger, M. Less is more? The impact of written corrective feedback on corpus-assisted L2 error resolution. J. Second. Lang. Writ. 2020, 49, 16. [Google Scholar] [CrossRef]
  65. Wang, Y.H. Developing and Evaluating an Adaptive Business English Self-Learning System for EFL Vocabulary Learning. Math. Probl. Eng. 2014, 2014, 972184. [Google Scholar] [CrossRef] [Green Version]
  66. Ma, Q.; Tang, J.L.; Lin, S.R. The development of corpus-based language pedagogy for TESOL teachers: A two-step training approach facilitated by online collaboration. Comput. Assist. Lang. Learn. 2021, 1–30. [Google Scholar] [CrossRef]
  67. Wu, Q. Designing a smartphone app to teach English (L2) vocabulary. Comput. Educ. 2015, 85, 170–179. [Google Scholar] [CrossRef]
  68. Hu, S.M.; Vongpumivitch, V.; Chang, J.S.; Liou, H.C. The effects of L1 and L2 e-glosses on incidental vocabulary learning of junior high-school English students. ReCALL 2014, 26, 80–99. [Google Scholar] [CrossRef]
  69. Gay, F. Investigating the effects of technology-enhanced vocabulary learning strategy instruction on supporting mixed-ability EMI learners studying a journalism and communication major: An action research project at a university in China. J. Engl. Acad. Purp. 2022, 55, 14. [Google Scholar] [CrossRef]
  70. Basal, A. Learning collocations: Effects of online tools on teaching English adjective-noun collocations. Br. J. Educ. Technol. 2019, 50, 342–356. [Google Scholar] [CrossRef] [Green Version]
  71. Shen, W.W.; Lin, J.M.; Cheng, W.K.; Hong, Z.W. Developing and evaluating an online video-assisted collocation learning system for EFL students. Interact. Learn. Environ. 2021, 1–15. [Google Scholar] [CrossRef]
  72. Comelles, E.; Laso, N.J.; Forcadell, M.; Castano, E.; Feijoo, S.; Verdaguer, I. Using online databases in the linguistics classroom: Dealing with clause patterns. Comput. Assist. Lang. Learn. 2013, 26, 282–294. [Google Scholar] [CrossRef]
  73. Pinto-Llorente, A.M.; Sanchez-Gomez, M.C.; Garcia-Penalvo, F.J.; Casillas-Martin, S. Students’ perceptions and attitudes towards asynchronous technological tools in blended-learning training to improve grammatical competence in English as a second language. Comput. Hum. Behav. 2016, 72, 632–643. [Google Scholar] [CrossRef]
  74. Read, T.; Kukulska-Hulme, A. The Role of a Mobile App for Listening Comprehension Training in Distance Learning to Sustain Student Motivation. J. Univers. Comput. Sci. 2015, 21, 1327–1338. [Google Scholar] [CrossRef]
  75. Chen, H.J.H. Developing and evaluating SynctoLearn, a fully automatic video and transcript synchronization tool for EFL learners. Comput. Assist. Lang. Learn. 2011, 24, 117–130. [Google Scholar] [CrossRef]
  76. Yang, Y.T.C.; Gamble, J.; Tang, S.Y.S. Voice over instant messaging as a tool for enhancing the oral proficiency and motivation of English-as-a-foreign-language learners. Br. J. Educ. Technol. 2011, 43, 448–464. [Google Scholar] [CrossRef]
  77. Hwang, W.Y.; Shadiev, R.; Hsu, J.L.; Huang, Y.M.; Hsu, G.L.; Lin, Y.C. Effects of storytelling to facilitate EFL speaking using Web-based multimedia system. Comput. Assist. Lang. Learn. 2014, 29, 215–241. [Google Scholar] [CrossRef]
  78. Shih, R.C. Blended learning using video-based blogs: Public speaking for English as a second language students. Australas. J. Educ. Technol. 2010, 26, 883–897. [Google Scholar] [CrossRef] [Green Version]
  79. Chien, C.W. Taiwanese EFL undergraduates’ self-regulated learning with and without technology. Innov. Lang. Learn. Teach. 2016, 13, 1–16. [Google Scholar] [CrossRef]
  80. Mompean, J.A.; Fouz-Gonzalez, J. Twitter-Based Efl Pronunciation Instruction. Lang. Learn. Technol. 2016, 20, 166–190. Available online: https://www.learntechlib.org/p/176114/ (accessed on 6 April 2022).
  81. Hamilton, H. The efficacy of dictionary use while reading for learning new words. Am. Ann. Deaf. 2012, 157, 358–372. Available online: https://www.jstor.org/stable/26234850 (accessed on 6 April 2022). [CrossRef]
  82. Chen, C.M.; Wang, J.Y.; Chen, Y.C. Facilitating English-Language Reading Performance by a Digital Reading Annotation System with Self-Regulated Learning Mechanisms. Educ. Technol. Soc. 2014, 17, 102–114. Available online: https://www.jstor.org/stable/jeductechsoci.17.1.102 (accessed on 6 April 2022).
  83. Chan, C.H.Y. Building an online library for interpretation training: Explorations into an effective blended-learning mode. Comput. Assist. Lang. Learn. 2014, 27, 454–479. [Google Scholar] [CrossRef]
  84. Carsten, S.; Ciobanu, D.; Mankauskiene, D. The challenge of evaluating open interpreter training resources: Case study of ORCIT. Interpret. Transl. Train. 2021, 15, 490–505. [Google Scholar] [CrossRef]
  85. Tan, S.; O’Halloran, K.L.; Wignell, P. Multimodal research: Addressing the complexity of multimodal environments and the challenges for CALL. ReCALL 2016, 28, 253–273. [Google Scholar] [CrossRef] [Green Version]
  86. Vymetalkova, D.; Milkova, E. Experimental Verification of Effectiveness of English Language Teaching Using MyEnglishLab. Sustainability 2019, 11, 1357. [Google Scholar] [CrossRef] [Green Version]
  87. Yang, M.T.; Liao, W.C. Computer-Assisted Culture Learning in an Online Augmented Reality Environment Based on Free-Hand Gesture Interaction. IEEE Trans. Learn. Technol. 2014, 7, 107–117. [Google Scholar] [CrossRef]
  88. Taghizadeh, M.; Ejtehadi, A. Investigating pre-service EFL teachers’ and teacher educators’ experience and attitudes towards online interaction tools. Comput. Assist. Lang. Learn. 2021, 1–35. [Google Scholar] [CrossRef]
  89. Suh, S.; Kim, S.W.; Kim, N.J. Effectiveness of MMORPG-based instruction in elementary English education in Korea. J. Comput. Assist. Learn. 2010, 26, 370–378. [Google Scholar] [CrossRef]
  90. Chen, Y.L. The Effects of Virtual Reality Learning Environment on Student Cognitive and Linguistic Development. Asia-Pac. Educ. Res. 2016, 25, 637–646. [Google Scholar] [CrossRef]
  91. Hsu, C.C.; Chen, Y.L.; Lin, C.Y.; Lien, W.C. Cognitive development, self-efficacy, and wearable technology use in a virtual reality language learning environment: A structural equation modeling analysis. Curr. Psychol. 2022, 41, 1618–1632. [Google Scholar] [CrossRef]
  92. Mok, J. A case study of developing student-teachers’ language awareness through online discussion forums. Lang. Aware. 2013, 22, 161–175. [Google Scholar] [CrossRef]
  93. Lim, H.A.; Ellis, E.M.; Sonnenschein, D. Effect of Sing and Speak 4 Kids: An Online Music-Based Speech and Language Learning Game for Children in Early Intervention. Child Lang. Teach. Ther. 2022, 38, 02656590221080308. [Google Scholar] [CrossRef]
  94. Dugartsyrenova, V.A. Supporting genre instruction with an online academic writing tutor: Insights from novice L2 writers. J. Engl. Acad. Purp. 2019, 44, 14. [Google Scholar] [CrossRef]
  95. Sarre, C.; Grosbois, M.; Brudermann, C. Fostering accuracy in L2 writing: Impact of different types of corrective feedback in an experimental blended learning EFL course. Comput. Assist. Lang. Learn. 2019, 34, 707–729. [Google Scholar] [CrossRef]
  96. Ranalli, J. Online Strategy Instruction For Integrating Dictionary Skills And Language Awareness. Lang. Learn. Technol. 2013, 17, 75–99. Available online: http://llt.msu.edu/issues/june2013/ranalli.pdf (accessed on 6 April 2022).
  97. Wu, W.C.V.; Yang, J.C.; Hsieh, J.S.C.; Yamamoto, T. Free from demotivation in EFL writing: The use of online flipped writing instruction. Comput. Assist. Lang. Learn. 2019, 33, 353–387. [Google Scholar] [CrossRef]
  98. Tsai, S.C. Implementing interactive courseware into EFL business writing: Computational assessment and learning satisfaction. Interact. Learn. Environ. 2018, 27, 46–61. [Google Scholar] [CrossRef]
  99. Sun, X.Y.; Hu, G.W. Direct and indirect data-driven learning: An experimental study of hedging in an EFL writing class. Lang. Teach Res. 2020, 29, 1362168820954459. [Google Scholar] [CrossRef]
  100. Jiang, W.; Eslami, Z.R. Effects of computer-mediated collaborative writing on individual EFL writing performance. Comput. Assist. Lang. Learn. 2021, 1–30. [Google Scholar] [CrossRef]
  101. Chen, R.H. Effects of Deliberate Practice on Blended Learning Sustainability: A Community of Inquiry Perspective. Sustainability 2022, 14, 1785. [Google Scholar] [CrossRef]
  102. Lin, M.H. Learner-Centered Blogging: A Preliminary Investigation of EFL Student Writers’ Experience. Educ. Technol. Soc. 2015, 18, 446–458. Available online: https://www.jstor.org/stable/jeductechsoci.18.4.446 (accessed on 6 April 2022).
  103. Rienties, B.; Lewis, T.; McFarlane, R.; Nguyen, Q.; Toetenel, L. Analytics in online and offline language learning environments: The role of learning design to understand student online engagement. Comput. Assist. Lang. Learn. 2018, 31, 273–293. [Google Scholar] [CrossRef] [Green Version]
  104. Lephalala, M.; Pienaar, C. An evaluation of markers’ commentary on ESL students’ argumentative essays in an ODL context. Lang. Matters 2008, 39, 66–87. [Google Scholar] [CrossRef]
  105. Tang, X.F.; Taguchi, N. Digital Game-Based Learning of Formulaic Expressions in Second Language Chinese. Mod. Lang. J. 2021, 105, 740–759. [Google Scholar] [CrossRef]
  106. Godfroid, A.; Lin, C.H.; Ryu, C. Hearing and Seeing Tone Through Color: An Efficacy Study of Web-Based, Multimodal Chinese Tone Perception Training. Lang. Learn. 2017, 67, 819–857. [Google Scholar] [CrossRef]
  107. Yu, Y.T.; Tsuei, M.P. The effects of digital game-based learning on children’s Chinese language learning, attention and self-efficacy. Interact. Learn. Environ. 2022, 1–20. [Google Scholar] [CrossRef]
  108. Henderson, C. The effect of feedback timing on L2 Spanish vocabulary acquisition in synchronous computer-mediated communication. Lang. Teach. Res. 2019, 25, 185–208. [Google Scholar] [CrossRef]
  109. Fernandez-Toro, M.; Furnborough, C. Evaluating alignment of student and tutor perspectives on feedback on language learning assignments. Distance Educ. 2018, 39, 548–567. [Google Scholar] [CrossRef]
  110. Yang, Y.F.; Meng, W.T. The Effects Of Online Feedback Training On Students’ Text Revision. Lang. Learn. Technol. 2013, 17, 220–238. Available online: http://llt.msu.edu/issues/june2013/yangmeng.pdf (accessed on 6 April 2022).
  111. Chang, C.; Lin, H.C.K. Effects of a mobile-based peer-assessment approach on enhancing language-learners’ oral proficiency. Innov. Educ. Teach. Int. 2019, 57, 668–679. [Google Scholar] [CrossRef]
  112. Lin, C.J.; Hwang, G.J. A Learning Analytics Approach to Investigating Factors Affecting EFL Students’ Oral Performance in a Flipped Classroom. Educ. Technol. Soc. 2018, 21, 205–219. Available online: https://www.jstor.org/stable/26388398 (accessed on 6 April 2022).
  113. Lan, E.M. A comparative study of computer and mobile-assisted pronunciation training: The case of university students in Taiwan. Educ. Inf. Technol. 2021, 27, 1559–1583. [Google Scholar] [CrossRef]
  114. Cui, G.Y. An experimental research on blended learning in the development of listening and speaking skills in China. S. Afr. Linguist. Appl. Lang. Stud. 2014, 32, 447–460. [Google Scholar] [CrossRef]
  115. Ge, Z.G. Exploring the effect of video feedback from unknown peers on e-learners’ English-Chinese translation performance. Comput. Assist. Lang. Learn. 2019, 35, 169–189. [Google Scholar] [CrossRef]
  116. Karras, J.N. The effects of data-driven learning upon vocabulary acquisition for secondary international school students in Vietnam. ReCALL 2016, 28, 166–186. [Google Scholar] [CrossRef]
  117. Hirschel, R.; Fritz, E. Learning vocabulary: CALL program versus vocabulary notebook. System 2013, 41, 639–653. [Google Scholar] [CrossRef]
  118. Daskalovska, N. Corpus-based versus traditional learning of collocations. Comput. Assist. Lang. Learn. 2015, 28, 130–144. [Google Scholar] [CrossRef]
  119. Yeh, H.C.; Tseng, S.S. Using the ADDIE Model to Nurture the Development of Teachers’ CALL Professional Knowledge. Educ. Technol. Soc. 2019, 22, 88–100. Available online: https://www.jstor.org/stable/10.2307/26896712 (accessed on 6 April 2022).
  120. Bahari, A.; Zhang, X.; Ardasheva, Y. Establishing a computer-assisted interactive reading model. Comput. Educ. 2021, 172, 15. [Google Scholar] [CrossRef]
  121. Dashkina, A.I.; Khalyapina, L.P.; Kobicheva, A.M.; Odinokaya, M.A.; Tarkhov, D.A. Developing a Model of Increasing the Learners’ Bilingual Professional Capacity in the Virtual Laboratory Environment. Appl. Sci. 2020, 10, 7022. [Google Scholar] [CrossRef]
  122. Hu, P.J.H.; Hui, W.; Clark, T.H.K.; Tam, K.Y. Technology-assisted learning and learning style: A longitudinal field experiment. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 2007, 37, 1099–1112. [Google Scholar] [CrossRef]
  123. Wang, X.; Zhang, W. Improvement of Students’ Autonomous Learning Behavior by Optimizing Foreign Language Blended Learning Mode. SAGE Open 2022, 12, 21582440211071108. [Google Scholar] [CrossRef]
  124. Kuo, Y.C.; Chu, H.C.; Huang, C.H. A Learning Style-based Grouping Collaborative Learning Approach to Improve EFL Students’ Performance in English Courses. Educ. Technol. Soc. 2015, 18, 284–298. Available online: https://www.jstor.org/stable/jeductechsoci.18.2.284 (accessed on 6 April 2022).
  125. Jin, H.; Karatay, Y.; Bordbarjavidi, F.; Yang, J.; Kochem, T.; Muhammad, A.A.; Hegelheimer, V. Exploring global online course participants’ interactions: Value of high-level engagement. ReCALL 2022, 1–18. [Google Scholar] [CrossRef]
  126. Jung, Y.; Kim, Y.; Lee, H.; Cathey, R.; Carver, J.; Skalicky, S. Learner perception of multimodal synchronous computer-mediated communication in foreign language classrooms. Lang. Teach. Res. 2017, 23, 287–309. [Google Scholar] [CrossRef]
  127. Shekary, M.; Tahririan, M.H. Negotiation of meaning and noticing in text-based online chat. Mod. Lang. J. 2006, 90, 557–573. [Google Scholar] [CrossRef]
  128. Yang, S.J. Language learners’ perceptions of having two interactional contexts in eTandem. Lang. Learn. Technol. 2018, 22, 42–51. Available online: http://hdl.handle.net/10125/44577 (accessed on 6 April 2022).
  129. Hedayati, H.; Marandi, S.S. Iranian EFL teachers’ perceptions of the difficulties of implementing CALL. ReCALL 2014, 26, 298–314. [Google Scholar] [CrossRef]
  130. Yu, C.M.; Chang, H.T.; Chen, K.S. Developing a performance evaluation matrix to enhance the learner satisfaction of an e-learning system. Total Qual. Manag. Bus. Excell. 2016, 29, 727–745. [Google Scholar] [CrossRef]
  131. Lee, L. Scaffolding Collaborative Exchanges Between Expert and Novice Language Teachers in Threaded Discussions. Foreign Lang. Ann. 2009, 42, 212–228. [Google Scholar] [CrossRef]
Sustainability 14 07303 g001 550
Figure 1. Flow diagram of detailed steps of article screening.
Figure 1. Flow diagram of detailed steps of article screening.
Sustainability 14 07303 g001
Sustainability 14 07303 g002 550
Figure 2. Annual publications from 2006 to 2022.
Figure 2. Annual publications from 2006 to 2022.
Sustainability 14 07303 g002
Sustainability 14 07303 g003 550
Figure 3. Keyword co-occurrence visualization network analysis.
Figure 3. Keyword co-occurrence visualization network analysis.
Sustainability 14 07303 g003
Sustainability 14 07303 g004 550
Figure 4. Co-occurrent keywords with “language”.
Figure 4. Co-occurrent keywords with “language”.
Sustainability 14 07303 g004
Sustainability 14 07303 g005 550
Figure 5. Top 25 keywords with the strongest citation burst.
Figure 5. Top 25 keywords with the strongest citation burst.
Sustainability 14 07303 g005
Sustainability 14 07303 g006 550
Figure 6. Timeline visualization of the top 11 clusters.
Figure 6. Timeline visualization of the top 11 clusters.
Sustainability 14 07303 g006
Sustainability 14 07303 g007 550
Figure 7. Visualization analysis of collaborating countries or regions.
Figure 7. Visualization analysis of collaborating countries or regions.
Sustainability 14 07303 g007
Sustainability 14 07303 g008 550
Figure 8. Visualization analysis of collaborating research institutions.
Figure 8. Visualization analysis of collaborating research institutions.
Sustainability 14 07303 g008
Sustainability 14 07303 g009 550
Figure 9. Author collaboration visualization network analysis.
Figure 9. Author collaboration visualization network analysis.
Sustainability 14 07303 g009
Sustainability 14 07303 g010 550
Figure 10. Co-cited author visualization network analysis.
Figure 10. Co-cited author visualization network analysis.
Sustainability 14 07303 g010
Table
Table 1. Search queries and refinement procedures.
Table 1. Search queries and refinement procedures.
SetResultsRefinement
1243Topic: (“online language learning effectiveness” OR “distance language learning effectiveness” OR “online language course effectiveness” OR “distance language course effectiveness”)
Indexes = SSCI, AH&CI
Time Span = 2006–2022
2239Refined by LANGUAGES: (English)
3222Refined by DOCUMENT TYPES: (Article)
Table
Table 2. Bibliographic statistics of 103 publications extracted from Web of Science.
Table 2. Bibliographic statistics of 103 publications extracted from Web of Science.
Total PublicationsH-IndexSum of Citing ArticlesSum of Times CitedAverage Citations per Item
103231323143713.95
Table
Table 3. Top 15 journals that published most of the included articles.
Table 3. Top 15 journals that published most of the included articles.
JournalNumber of PublicationsProportion
(%)
Computer Assisted Language Learning1413.59
Recall87.77
Interactive Learning Environments76.80
Computers Education65.83
Educational Technology Society54.85
Language Learning Technology54.85
Sage Open43.88
Sustainability43.88
British Journal of Educational Technology32.91
Education and Information Technologies32.91
Journal of Computer Assisted Learning32.91
Language Teaching Research32.91
IEEE Transactions on Learning Technologies21.94
Innovation in Language Learning and Teaching21.94
Interpreter and Translator Trainer21.94
Table
Table 4. Overview of effectiveness studies on specific courses in online or blended language learning.
Table 4. Overview of effectiveness studies on specific courses in online or blended language learning.
LanguageLearning ContentParticipantsGeneral OutcomeReferences
GermanPronunciationUndergraduates in the USNo significant improvement,
negligible		[29]
DutchOverall Dutch learningAdult immigrants in
Flanders		Mixed[30]
EnglishWritingUndergraduates in
Saudi Arabia		Effective, positive[31]
Undergraduates in ChinaSignificant improvement[32]
SpeakingUndergraduates in IranImprovement and betterment[33]
ReadingAdults in TurkeyPositive mainly[34]
Undergraduates in ChinaPositive, enhancement[35]
Writing and speakingUndergraduates in
South Korea		Improvement[36]
Listening and speakingCollege students in ChinaSignificant improvement[37]
Reading and writingUndergraduates in ChinaPositive, significant[38]
Translation and
interpreting		Adult learners in
South Korea		General satisfaction,
positive evaluation		[39]
GrammarUndergraduates in
Saudi Arabia		Mixed[40]
Language and literatureK-12 students in the USMixed[41]
English for
Specific
Purpose		Hospitality EnglishLearners in China Positive, helpful[42]
Agriculture and forestry EnglishGraduates in China Effective, meaningful[43]
Maritime EnglishCollege students in ChinaPositive[44]
Overall English skillsCollege students in Russia and the United Arab
Emirates		Effectiveness confirmed,
significant difference		[45]
MOOC learners in IranEfficient[46]
Undergraduates and
postgraduates in China		Effectiveness recognized[47]
College students in ChinaModerately high satisfaction[48]
Undergraduates in ChinaPositive enhancement[49]
Table
Table 5. Overview of effectiveness studies on assisted tools in online or blended language learning.
Table 5. Overview of effectiveness studies on assisted tools in online or blended language learning.
Learning ContentParticipantsAssisted ToolsGeneral Outcome References
TurkishHigh school students in the USComputer-assisted task-based language instruction (CATBI) tool Mixed[50]
Russian and ChineseJunior high school students in China and UzbekistanSpeech-to-text recognition
system;
computer-aided translation
system		Positive[51]
Croatian and EnglishCollege students in Australia, Croatia, Bosnia, and HercegovinaSecond Life, SkypePositive gains[52]
Spanish vocabularyNative English
learners 		LANGA, an online game-based platformSignificant gains[53]
Spanish punctuationLearners in National University of Distance EducationChatbotSubstantial improvement[54]
Arabic sign knowledgeArab deaf studentsA new system consists of two subsystems: Speech to ArSL translation subsystem and ArSL to speech translation subsystemSignificant improvement[55]
Sign languageLearners, including the deaf or hard of hearingSign language streaming videos and subtitlesSatisfaction[56]
English writingGraduate students in China Computer-supported collaborative learning (CSCL) systemImprovement[57]
Undergraduates in Malaysia Summary Writing-Pal
(SW-PAL)		Significant improvement[58]
Undergraduates in
China		DWright, an online writing
tutorial system		Positive improvement[59]
Graduate students in China EJP-Write, an online writing
tutorial system 		No significant
improvements		[60]
Undergraduates in ChinaOnline automated essay
evaluation system 		Significant improvement[61]
Undergraduates in
Japan		Online forums, blogs, and wikis.Positive[62]
Undergraduates in IranOnline blogsEffective, satisfaction[63]
Postgraduates in
Australia		Online corporaMixed[64]
English vocabularyUndergraduates in
China		Adaptive business English
self-learning system		Better performance,
positive, effective		[65]
College students in ChinaOnline corpusEffective[66]
Undergraduates in ChinaWord Learning-CET6Significant
outperformance		[67]
Junior high school
students in China 		Chinese and English
e-gloss		Enhancement, useful[68]
Undergraduates in ChinaMemrise, technology-enhanced support with a focus on online resourcesBeneficial impact[69]
English collocationUndergraduates in TurkeyFour online tools: Concordance website,
Oxford Online Collocation
Dictionary, World Wide Web corpus, Google Docs		Significantly better
performance		[70]
Undergraduates in
China		Online video-assisted
collocation learning system (VACLS)		Significant improvement, helpful, useful[71]
English grammarUndergraduates in Spain Online corpus databaseEffectiveness, satisfaction[72]
358 recruited students Online tools: podcast, videocast, online tests, online
glossary, and forums		Rather positive[73]
English listeningCollege students in SpainAudio News Trainer (ANT)Effective[74]
Undergraduates in
China		SynctoLearn, a fully automatic video and transcript
synchronization tool		Positive[75]
English speakingUndergraduates in AsiaVoice over instant
messaging (VoIM)		Effective enhancement[76]
Elementary school
students in China		Web-based multimedia systemSignificant
outperformance		[77]
English public
speaking		College students in ChinaVideo-based blogsEfficient, productive[78]
English pronunciationUndergraduates in
China		23 online resources
and 18 Apps		Less effective[79]
Language learners in SpainTwitterBeneficial effect,
active engagement		[80]
English readingDeaf high school
students		Online bilingual multimedia English-ASL dictionaryEffective[81]
Grade 7 students at a junior high school in ChinaDigital reading annotation
system		Significant improvement[82]
English interpretingUndergraduates in ChinaWeb-based library of
interpreting practice resources		An effective role[83]
Interpreting students in different locationsOnline Resources for
Conference Interpreter Training (ORCIT)		Enhancement, useful[84]
Overall English skillsEnglish learnersSecond Life, 3D virtual worldsRelatively effective,
more engagement		[85]
Undergraduates in The Czech RepublicMy EnglishLabBetter results[86]
Graduate students in ChinaVirtual English Classroom, VECARImprovement, promotion[87]
College students and teacher educators in Iran37 online interaction toolsEffectiveness[88]
Elementary students in KoreaMassive multiplayer online role-playing game (MMORPG)Useful, improvement[89]
Undergraduates in
China		Online 3D VR English language learning
platform		Positive affection[90]
College students in ChinaWearable virtual reality
language learning platform		Improvement[91]
English language usersOnline discussion forumSignificant enhancement[92]
Word production in multiple languages2–6-year-old childrenSing and Speak 4 Kids (SS4Kids)Significant improvement[93]
Table
Table 6. Overview of effectiveness studies on instructional approach in online or blended language learning.
Table 6. Overview of effectiveness studies on instructional approach in online or blended language learning.
Learning ContentParticipantsInstructional ApproachGeneral OutcomeReferences
Chinese expressionsCollege students in the USGame-based interactive
learning online		Robust improvement[105]
Chinese toneOnline learnersVisualization: five multimodal methodsSubstantial benefits[106]
Overall Chinese skillsElementary students in ChinaDigital game-based learning (DGBL)Significant enhancement[107]
Spanish vocabularyCollege students in the USSynchronous Computer-Mediated Communication (SCMC):
online corrective feedback		Effective[108]
Chinese, French, German, Italian, Spanish, Welsh writingDistance undergraduate learnersOnline feedback alignmentMixed[109]
French and SpanishUndergraduates in the UKComputer-assisted language learning (CALL)Mixed[103]
English writingEnglish learners in
Russia		Online academic writing tutorEnhancement, value[94]
College students in
China		Online feedbackOverall positive, but mixed[110]
College students in FranceOnline corrective feedbackQualitative progress[95]
Undergraduates in South AfricaComputer-mediated
feedback in the ESL-ODL
context		Mixed[104]
College students in the USAutomated online form of
strategy instruction		Significant gains[96]
Undergraduates in
China		Online flipped writingPositive enhancement[97]
College students in
China		Courseware-implemented task-based instructionSignificantly better, great satisfaction[98]
Undergraduates in ChinaOnline indirect data-driven learningGreater effectiveness,
practically meaningful		[99]
College students in ChinaComputer-mediated collaborative writing (CMCW)Higher gains,
more benefits		[100]
Undergraduates in
China		Online film clip watching and writingClear improvement[101]
Undergraduates in
China		Integration of online learner-centered blogging approachPositive development[102]
English speakingUndergraduates in
China		Mobile-supported peer
assessment (M-PA)		Effective promotion[111]
Undergraduates in
China		Online community-based flipped learningPositive enhancement[112]
English pronunciationUndergraduates in
China		Mobile-assisted pronunciation training (MAPT)Significant improvement[113]
English listening and speakingUndergraduates in ChinaBlended learning modeObvious improvement[114]
English translationLearners at an e-learning college in ChinaOnline peer video feedbackEffective improvement[115]
English vocabularySecondary students in VietnamOnline data-driven learning (DDL)Significant increase[116]
Undergraduates in
Japan		Computer-assisted language learning with spaced repetitionSignificant gains[117]
English collocationUndergraduates in the Republic of MacedoniaOnline corpus-based learningBetter results[118]
English readingElementary students in ChinaComputer-assisted language learning (CALL)Effective, promotion[119]
English learners in IranCALL: computer-assisted interactive reading model (CAIRM)Significant improvement,
positive perceptions		[120]
Overall English skillsCollege studentsContent and language integrated learning approach (CLIL) in the virtual laboratory environmentMore positive attitude,
better outcomes		[121]
Undergraduates in ChinaTechnology-assisted learningSignificantly greater
effectiveness		[122]
Undergraduates in ChinaOptimized blended learning model based on SPOCImprovement[123]
Postgraduates in ChinaLearning style-based
collaborative learning		Outperformance[124]
Graduate students in the USAsynchronous computer-mediated communication (ACMC):
online discussion forums		Effective[125]
College students from Korea, Japan, and
China		Synchronous computer-mediated communication (SCMC):
online chatting or discussion		Improvement, ceiling effect[126]
Persian English learnersSynchronous computer-mediated communication
SCMC: online chatting		Effective, helpful[127]
Korean and English
language learners		Pair work and group discussion in eTandem learningMixed[128]
English learners in IranComputer-assisted language learning (CALL)Negative, insufficient[129]
Overall foreign
language skills		Language learnersComputer-assisted language learning (CALL)Positive, satisfaction[130]
Undergraduates from France, Germany, Russia, Spain, and so onAsynchronous computer-mediated communication (ACMC):
weekly online threaded
discussions		Positive, very beneficial[131]
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Zhou, T.; Zhang, W. Effectiveness Study on Online or Blended Language Learning Based on Student Achievement: A Systematic Review of Empirical Studies. Sustainability 2022, 14, 7303. https://doi.org/10.3390/su14127303

AMA Style

Zhou T, Zhang W. Effectiveness Study on Online or Blended Language Learning Based on Student Achievement: A Systematic Review of Empirical Studies. Sustainability. 2022; 14(12):7303. https://doi.org/10.3390/su14127303

Chicago/Turabian Style

Zhou, Tong, and Wei Zhang. 2022. "Effectiveness Study on Online or Blended Language Learning Based on Student Achievement: A Systematic Review of Empirical Studies" Sustainability 14, no. 12: 7303. https://doi.org/10.3390/su14127303

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop