Analyzing Log Data of Students Who Have Achieved Scores Adjacent to the Minimum Passing Grade for a K-MOOC Completion in the Context of Learning Analytics
Round 1
Reviewer 1 Report
Manuscript has an interesting and relevant approach to Mooc and engagement of students attending a Mooc course.
Nonetheless there are some topics to clarify:
- When authors define the selectin criteria for the 60 K-MOOCs, it is not clear if the three aspects must be (or not) overlapped.
- If there are K-MOOC (Korean Massive Open Online Course) supported by the Korean Ministry of Education and supervised by the National Institute for Lifelong Education, since 2015, with more than 2,000 courses from 140 institutions, it is not clear why to focus the analysis only in 60 courses from March 2019 and February 2020. Specially when pointing for future research the need to examine how motivation and engagement of MOOC students change over time…
- Discussion would benefit of an in-deep theoretical background.
Author Response
We appreciate you for your precious time in reviewing our paper and providing valuable comments. It was your valuable and insightful comments that led to possible improvements in the current version. The authors have carefully considered the comments and tried our best to address every one of them. We hope the manuscript after careful revisions meet your high standards. The authors welcome further constructive comments if any.
Please find enclosed the revised manuscript in Word format. All modifications in the manuscript have been highlighted in red.
----------------------------------------------------------------------------------------
Response to Reviewer 1 Comments
Point 1:
When authors define the selectin criteria for the 60 K-MOOCs, it is not clear if the three aspects must be (or not) overlapped. If there are K-MOOC (Korean Massive Open Online Course) supported by the Korean Ministry of Education and supervised by the National Institute for Lifelong Education, since 2015, with more than 2,000 courses from 140 institutions, it is not clear why to focus the analysis only in 60 courses from March 2019 and February 2020.
Response 1:
- We really appreciate this significant comment, which helped us refine the manuscript. In order to make the remark clearer, we revised as to read as follows.
“The 60 K-MOOCs were selected as the target courses to conduct this research by meeting all of the following three criteria, 1) courses that have been selected as excellent courses, Blue Ribbon Courses, or have more than 500,000 initial log data during the log data collection period, 2) courses operated twice or more, and 3) courses with the same operating period.” (p.5)
Point 2:
Specially when pointing for future research the need to examine how motivation and engagement of MOOC students change over time…
Response 2:
- To put additional suggestions for future research relevant to the change of student motivation and engagement in MOOCs, the following sentence has been included.
“Additionally, this study compared only the degree of participation in learning activities of students who have achieved scores adjacent to the minimum passing grade of K-MOOCs operated from March 2019 to February 2020 as of an academic year. We propose that future studies track log data of different student population such as at-risk students enrolled in MOOCs and examine how their motivation and engagement change over time to guide timely intervention efforts.” (p.16)
- Point 3:
- Discussion would benefit of an in-deep theoretical background.
Response 3:
- We appreciate your suggestions that allowed us to greatly improve the quality of the manuscript. Several sentences have been included in the part of “Discussion and conclusion” as follows.
“Lastly, in order to improve the learning persistence in MOOCs, it is necessary to apply learning analytics to not only explore motivation and engagement from the students’ perspective, but also pay attention to the interventions from the perspective of MOOC instructors, developers and operators. From the instructor side, it is critical to provide individualized instructional interventions according to student motivation and engagement by analyzing behavioral data during the course. From the developer’s point of view, the MOOC system should improve the User Experience, and MOOC operators need to seek effective course operation strategies based on learning analytics. As emphasized by Ahn and Lee [34], rather than focusing only on the quantitative growth of MOOC content, it is required to emphasize the improvement of high-quality service by analyzing students' learning behavior.
The results of this study suggest that it is meaningful to perform analysis on the entire log data rather than specific log data, and it is necessary to consider it when (re)designing dashboards and analysis tools in the future. Based on these results, by collecting, analyzing, and measuring MOOC students’ log data, that is learning analytics, their engagement and interactions should be investigated to improve MOOC completion rates. Moreover, MOOC instructors and developers should explore more specific design guidelines on how to provide individualized hints and feedback with effective digital textbooks or reference materials for the large number of students. Unlike the existent university courses in which only learners who have been selected according to certain criteria after paying tuition for the purpose of acquiring credits participate, MOOCs should be designed and developed to individualized learning environments for large-scale learners who are expected to have diverse spectrums in various aspects such as knowledge level and learning motivation.” (p.15-16)
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The manuscript provided is well-structured and well-written and brings interesting information regarding the behavior of MOOC students. The authors could conduct some changes to improve the work. Following, I will suggest some of these improvement opportunities.
First, the authors could highlight the motivation and problem research in the abstract. Second, the authors could provide a short section concerning the foundations of learning analytics. This section could bring the main concepts, challenges, and current applications in the area.
Third, the manuscript could present related works to highlight their work's novelty in contrast to previous studies. Finally, the study's authors could describe the challenges in collecting and analyzing MOOC data.
Author Response
We appreciate you for your precious time in reviewing our paper and providing valuable comments. It was your valuable and insightful comments that led to possible improvements in the current version. The authors have carefully considered the comments and tried our best to address every one of them. We hope the manuscript after careful revisions meet your high standards. The authors welcome further constructive comments if any.
Please find enclosed the revised manuscript in Word format. All modifications in the manuscript have been highlighted in red.
----------------------------------------------------------------------------------------
Response to Reviewer 2 Comments
Point 1:
First, the authors could highlight the motivation and problem research in the abstract.
Response 1:
We agree with your comments. We highlighted the motivation and problem research in the abstract by restructuring the whole paragraph.
Point 2:
Second, the authors could provide a short section concerning the foundations of learning analytics. This section could bring the main concepts, challenges, and current applications in the area.
Response 2:
In response to reviewer’s comment, we clearly explained “learning analytics” by adding paragraphs with literature review to highlight relevant theories including learning analytics in the part of “Related work”. While explaining the theory of learning analytics as a theoretical background, the title has been modified by adding "~in the context of learning analytics".
Point 3:
Third, the manuscript could present related works to highlight their work's novelty in contrast to previous studies. Finally, the study's authors could describe the challenges in collecting and analyzing MOOC data.
Response 3:
We appreciate your suggestions that allowed us to greatly improve the quality of the manuscript. In the part of “Discussion and conclusion”, we added the sentences to compare our findings with those of previous studies and suggest future research of MOOC log data based on learning analytics.
Author Response File: Author Response.pdf
Reviewer 3 Report
Overall interesting article on analyzing students' behaviors by categorizing them into completion and non-completion groups. A few suggestions/questions:
1. The introduction can be better structured to highlight your contributions and please also consider separating out the related work. I was already lost after reading the intro, unclear about what I should expect in the following text.
2. I wonder if it is ethical to analyze students' log data and, more importantly, propose personalized learning based on their demographics. Did students consent to be recorded and compared by their demographics? This would likely lead to discrimination in teaching practice.
3. Please move table 4 before table 3. The definitions should precede their usage.
Author Response
We appreciate you for your precious time in reviewing our paper and providing valuable comments. It was your valuable and insightful comments that led to possible improvements in the current version. The authors have carefully considered the comments and tried our best to address every one of them. We hope the manuscript after careful revisions meet your high standards. The authors welcome further constructive comments if any.
Please find enclosed the revised manuscript in Word format. All modifications in the manuscript have been highlighted in red.
----------------------------------------------------------------------------------------
Response to Reviewer 3 Comments
Point 1:
- The introduction can be better structured to highlight your contributions and please also consider separating out the related work. I was already lost after reading the intro, unclear about what I should expect in the following text.
Response 1:
We went through the whole sentences and paragraphs in the part of “Introduction” and restructured it. So “Related work” section is separated from the “Introduction”.
Point 2:
I wonder if it is ethical to analyze students' log data and, more importantly, propose personalized learning based on their demographics. Did students consent to be recorded and compared by their demographics? This would likely lead to discrimination in teaching practice.
Response 2:
- We sincerely appreciate valuable comments, which helped us add the following sentences on personal information protection act.
“Among the courses taken by learners who agreed to the use of learning log data for research, in 2021, when the K-MOOC log data was collected for this study, the courses with the same operating period were 60 courses operated from March 2019 to February 2020 as a Korean academic year.” (p.5)
Point 3:
Please move table 4 before table 3. The definitions should precede their usage.
Response 3:
To accommodate your kind remark, Table 4 is rearranged in front of table 3.
Author Response File: Author Response.pdf
Round 2
Reviewer 3 Report
The authors addressed all my concerns in the revision.
