MOOC Learners’ Engagement with Two Variants of Virtual Patients: A Randomised Trial

Abstract

Introduction: Massive Open Online Courses (MOOCs) are an increasingly popular form of education in health professional education. VPs have been introduced in MOOCs to increase interactivity. There is a lack of research in understanding the reasons behind high dropout rates in MOOCs. We explored how learners interact with VPs and compared the significance of different VP designs on dropout rates. Methods: RCT of 378 participants split into two groups to interact with two VPs using different design: branching and linear. Data on node progression and VP attempts was analysed using descriptive and quantitative analysis. Results: Eight groups of learner interaction patterns were identified. The majority of learners completed the VP in a linear path in one attempt. A significant number either completed the case in a loop path in one attempt, completed in a linear path in multiple attempts or dropped out without attempting the case. VP design has a significant effect on dropout rates of learners. There is a higher dropout rate from a branched VP compared to linear VP. Discussion: Prior research showed that branched VPs are more engaging and promote greater learning compared to linear VPs. However, our results indicate that branched VPs had greater dropout compared to VPs that require less time to be solved. Conclusions: We conclude that branching had a negative effect on completion of the VP activity in the MOOC. Moreover, we believe that more complex VPs require more effort on task and this might not be a design that facilitates the interaction in a MOOC audience, where the participants might wish to acquire the basic medical knowledge offered by the course.

1. Introduction

Massive Open Online Courses (MOOCs) are an increasingly popular form of education in health professional education. In 2013, 98 MOOCs were offered related to health-care and medical education [1] and their numbers continue to increase. Still, there are challenges reported in relation to MOOCs, where lack of student interaction and engagement is a repeated concern [2,3,4] as well as low completion rates [5]. Possible causes of the high dropouts include the voluntary nature of MOOCs, but also the limited interactivity that MOOCs offer to participants. Interactivity is seen, not only as an important element of learning process in elearning contexts [6], but also as a way to increase student motivation and completion in particular in MOOCs [7].

Virtual Patients (VPs) are “interactive computer simulations of real-life clinical scenarios for the purpose of training, education or assessment” [8]. VPs have been previously introduced in MOOCs to increase levels of interactivity of learners with the MOOC content [9,10]. In one study, the results indicated that the VPs were perceived as a helpful exercise in the course by 83.1% of the responding participants (n = 442) [11]. Other positive outcomes of MOOCs using VPs was seen in a Urology MOOC which had a higher overall student completion rate 10.4% compared to the reported 5% average completion in MOOCs [12].

In particular in medical education contexts, VPs are known to facilitate the training of clinical reasoning skills [13] but may also promote critical thinking and decision making, enable active and self-directed learning. As such, VPs provide students with interactive and dynamic problem-solving situations. Different VP designs can offer a variety of learning opportunities. For instance, branched VPs can provide learners alternative routes to reaching the desired outcome and potentially offer several different outcomes depending on the options [14].

Branching in virtual patients permits adding intrinsic (i.e., consequential) feedback to the scenarios. Feedback is known to be one of the most powerful instructional methods in simulation [15] and it forms a crucial feature of virtual patient design that determines user satisfaction and engagement [16]. The wealth of different feedback forms [17] makes the choice of the right method difficult and as a scoping review has shown [18], there is not sufficient high-quality, evidence-based recommendations for feedback in medical education. By focusing on the influence of branching elements of virtual patients we would like contribute to the discussion on the right balance of instructional versus intrinsic feedback [19] in virtual patients. Intrinsic feedback allows an experiential approach to learning [20] but this may lead to confusion in novice learners and be contrasted with the wish of learners to get specific feedback [21] which is easier provided by corrective instructional feedback given in response to multiple-choice questions. The openness of MOOCs to different groups of learners makes it interesting to observe how branching will influence learners’ engagement in such a digital learning environments. Within a MOOC environment the right choice of feedback, may lead to positive effects on the dropout rates [11,12]. There is, so far, no firm evidence how this design feature may impact student completion of the VP activities and therefore impact the MOOC completion. Increased understanding of how the participants interact with different VP designs can inform the implementation process of the VPs and consequently make it more efficient.

The aim of this study is therefore to explore the influence of branching points on the learners’ engagement. In particular, we aimed to investigate: (i) How learners interact with VPs in MOOCs in terms of navigation patterns; and (ii) how branching points in a VP case may influence the dropout rate of learners within the VPs.

2. Methods

2.1. Study Context

Karolinska Institutet (KI) launched a Urology MOOC on the edX platform called an “Introduction to Urology”. This course was launched for the first time in autumn of 2015 and it was offered over a five week period [12].

The course is aligned to KI’s medical programme’s surgical course as part of the medical curriculum offered at one of the four main teaching hospitals in Stockholm, Sweden. The main learning outcome for this course was that students should be able to describe, argue and reason on the most common symptoms related to the urinary tract. The course consisted of video components, multiple choice questions, 3D models, glossary, discussion forums, a webinar, two VP cases and a final exam.

A VP focusing on bladder cancer was created in two different variants: one that offered many branching points but one end-state as the final outcome (VP1) and a second variant that replaced the branching points with multiple choice questions (VP2) that limited the navigation options of the participants to the main path. In particular, VP2 was created based on a main path of the VP1, including the same information about the patient diagnosis and treatment. Part of VP1 and VP2 cases are illustrated in Figure 1 where the nodes are represented by a number; in VP2 the multiple choice questions that replaced the branching nodes of VP1 are depicted. Nodes of VP1 and VP2 labeled with the same number imply that the nodes of the two cases consist of the same content.

The VPs were integrated in week 2 of the MOOC which had a focus on “Hematuria”. We implemented the VPs in the system OpenLabyrinth 3 [22] which was hosted in a cloud infrastructure and embedded within the edX platform using the IMS LTI interface, a process we had tested previously [10]. The users were automatically authenticated using credentials from the host system and had to declare the completion of the VP activity by inserting into the edX platform a four digits code that could be found by reaching the last node of the VP. By that, the students were acquiring course credits for the activity completion that was however not mandatory.

We modified the layout of the VP system to follow the graphical schema of the hosting MOOC platform and displayed it as a task in the weeks’ activity list, in order to create a smooth and seamless user integration. While solving the case, the learner interacted with Violet, a patient complaining for hematuria. Learners were required to take the patient history, order lab examinations and proceed to diagnostic and therapeutic decisions. The decisions were made either selecting one of a few options and exploring the consequential development of the scenario depending on the choice made (VP1) or selecting one of a few opinions in a MCQ and receiving instant corrective feedback (VP2). Additionally, the students were able to review the history of choice and return to one of the previous steps (looped behaviour).

2.2. Study Design

The study had a randomized design where the MOOC participants were randomly allocated to two different groups, facilitated by the A/B testing mechanism in the edX platform. Participants were allocated into either Group A where they received VP1 or Group B where they interacted with VP2. The participants were blinded to the intervention and for the purpose of avoiding contamination the discussion forums were disabled in the respective section of the VPs.

2.3. Participants

During autumn 2015, from 15 September to 27 October, 4925 participants enrolled to participate in the MOOC. The participants represented 172 countries; the three most represented ones were USA, Sweden and India. The median learner age was 30 years; 40.4% of the learners were females and 59% males. The remaining 0.6% did not report gender. In total, 514 participants completed the course. During week 2 of the course, 1349 participants logged in, and out of them 378 interacted with the VPs as either in Group A or Group B. In this study, we included the data for the 378 participants.

2.4. Data Collection

Data for the users’ VP pathway was collected from the cloud infrastructure where the OpenLabyrinth VP system was deployed. The data were collected and mapped to a user identification number to allow anonymization. The information collected from the OpenLabyrinth system included information regarding the time spent on VPs, node progression and number of attempts per VP case.

2.5. Data Analysis

To address the first research question, we conducted descriptive data analysis. We first classified the participants retrospectively depending on whether they had completed the case or dropped out. Within each group, we further classified them on the number of attempts; whether they attempted to solve the case more than one time. Subsequently, we classified the drop-out group based on whether they started the case, and the completed group based on their pattern of completion. To address the second research question, we analysed participant data for VP1 and VP2 separately, following the learner interaction patterns identified. We proposed a null hypothesis that a linear versus branched VP design has no influence on the dropout rate of learner. We examined the null hypothesis using the Pearson Chi-Square Test from IBM SPSS 64-bit at the 0.05 significance level for identifying the statistical significance between the two groups.

2.6. Ethics

The research here falls outside of the vetting process of research in accordance with the Swedish Ethical Review Act (2003:460) as it does not handle sensitive personal data as defined in the Swedish Personal Data Act. Further, according to edX praxis and in compliance with ethical guidelines, at registration, all course participants gave their informed consent that their engagement and performance data, in unidentified form, could be used in coming research publications.

3. Results

3.1. Learners’ Interaction Patterns with VPs in MOOCs

Figure 2 depicts results of our analysis on how learners interact with the VPs in the MOOC. The participants were first classified whether they dropped out or completed the VPs. There is a total of 71 dropout participants and 307 participants who completed the VPs.

Next, the participants were classified whether they attempted to solve the VPs once or more than once. Dropout participants were further classified based on whether they started or did not start the VP case. The participants that completed were further classified based on whether they completed the VP case in a linear or loop behaviour. A linear behaviour meant that participants did not go back to review previously visited nodes or completed questions. Loop behaviour refers to participants that went back to review previously visited nodes and completed questions. We mapped out each participant’s node progression and categorized them into eight different subcategories of interaction patterns.

In Figure 2 we label as:

• Distance between the black dots: length of VP case;
• Red dots: introduction page of VP case;
• Initiated line: user has loaded VP scenario;
• Straight line: linear behaviour in VP scenario;
• Loop line: loop behaviour in VP scenario;
• Multiple straight and loop lines: multiple linear and loop behaviours in VP scenario respectively;
• Incomplete line: dropout from VP scenario;
• Line reaching end dots (Straight or loop): completed VP scenario.

In

Table 1. Learner Interaction Patterns of VP1 and VP2.

Learner Interaction Patterns			Total (n = 378)
VP dropout	1 Attempt	Does not Start	51
		Start	14
	>1 Attempt	Does not Start	3
		Start	3
Completed	1 Attempt	Linear behaviour	209
		Loop behaviour	46
	>1 Attempt	Linear behaviour	36
		Loop behaviour	16

we label:

• Attempt: the act of loading the VP introduction page but not starting the VP scenario.
• Start: act of loading the VP scenario after the VP introduction page.

The majority of participants (n = 209) completed their VP case in 1 attempt following a linear path. A large group (n = 46) completed their VP case in 1 attempt in a loop path, meaning the participants returned to previous elements of the VP. Many participants (n = 36) also completed their VP case with more than 1 attempt in a linear path. It is also important to note that a big group (n = 51) dropped out in their first attempt without starting their VP case.

3.2. Influence of Branching Points in a VP Case on the Drop-Out Rate of Learners

To identify whether there is any influence of the branching points in VPs on the learners’ dropout rate from the assignment we analysed participant data for VP1 and VP2 separately, following the learner interaction patterns identified above. We excluded drop-out participants who did not initiate the VP case as their reasons for dropout was not influenced by the different VP designs or branching points in the VPs. We included participants who completed the VPs and participants who started the VP and dropped out (

Table 2. Observed total dropout, (expected total dropout).

VP Design	Dropout
	Yes	No
VP1	16 (9)	149 (156)
VP2	1 (8)	158 (151)

).

The calculated p-value is .000253 and the result is statistically significant at p < 0.05. The null hypothesis that VP branching points has no influence on the dropout rate of learner is therefore rejected and the results indicate that the VP1 versus the VP2 design has an influence on the dropout rate of learners. As the VP cases only differ in the added branching points of the VP1, it may be concluded that the VP1 design correlates with a higher dropout rate compared to the VP2.

4. Discussion

This study was conducted in a MOOC setting, aiming to understand how the MOOC audience may behave when interacting with modalities such as VPs for the purpose of informing the learning design process. In particular, we aimed to understand how the learners interact with VPs in MOOCs and how a VP design with branching points influences the dropout rate of learners compared with a VP design that includes MCQs instead. To test these questions two VP forms were designed and the MOOC participants were allocated to engage with the two forms in a randomized trial setting.

The results demonstrate that the majority (81% of total) of participants completed the VP case of bladder cancer. This is in line with prior research results, which demonstrate high participants’ satisfaction when engaging with VPs in MOOCs [11]. Nineteen percent (19%) of the total participants dropped out from the exercise, of which the bulk of these participants dropped out without starting the VP (76% of dropouts).

Prior research showed that branched VPs are more engaging and promote greater learning compared to linear VPs [23]. However, our results indicate that branched VPs had greater dropout compared to VPs that require less effort to be solved. This could be due the fact that branched VPs have a higher level of complexity, and place a higher cognitive load on novice learners, but also, there could be an impact due to the difference in target audience. In the previous study [23] the linear cases were paper-based and the activity was part of a PBL setting for medical students when group processes could compensate the complexity of decision making.

We conclude that more complex VPs require more effort on task and this might not be a design that facilitates interaction in a MOOC audience, where the participants might wish to acquire the basic medical knowledge offered by the course.

One of the strengths of this study is the proposed classification of learner interaction patterns while progressing through a VP including factors as multiple attempts and looped progress. The study delivers quantitative data on how popular these behavioural models are in a MOOC audience. It is yet to be investigated whether belonging to any of those groups is sign of a particular learner characteristic (e.g., experienced vs novice learners) or results in superior knowledge. Further research is also required in order to identify any potential correlations between the satisfaction outcomes of the participants having different learning behaviours that could inform design of more adaptive VPs for the MOOCs in future. We plan to expand the current study by investigating differences in engagement in VPs having different characteristics than just the branching points.

Limitations

One of the study’s limitations is that the results are based on the analysis of the learners’ interaction in only one case and therefore might not be representative. More research on the learning patterns in different VP designs is required. Baseline differences in the motivation or engagement of the participants as well as their professional background were not investigated.

The starting point of our research was to look into ways how to lower MOOC dropout rates. In the experimental setting at hand we have investigated VP drop-outs which are not identical with MOOC dropout (as the learner might switch to a different learning modality in the MOOC). Yet, we believe that loss of interest in one form of learning is likely to be correlated with disengagement in the whole course.

Further research on the topic may investigate the potential reasons of the higher dropout rates identified in the branched case in more depth by conducting a survey and examining the background correlation to the learning patterns and dropout behaviours. We plan also to look into the impact that different VP designs and features may have on the knowledge, skills and satisfaction outcomes.

5. Conclusions

This study’s results show differences in completion rates of the participants that interacted with different form of VPs. This could have an impact on the design of future MOOCs in health care educations settings. The study suggests that the integration of interactive elements in a MOOC setting has the potential to play a part in reducing the high drop-out rates previously observed in the MOOCs. From a design perspective, our results suggest that novices may benefit by first interacting with linear VPs before progressing to more challenging branched VPs.