Effect of Peer Influence and Self-Reflection on Scaffolded Out-of-Class Activity Administered Using a Mobile Application

Abstract

Student engagement with out-of-class activities is becoming more difficult as students spend fewer hours outside the classroom studying the content. This research developed a mobile educational platform, Dysgu, to provide students with an optimal learning experience outside of the classroom. Dysgu includes social networking and gamification features to increase student engagement. The platform offers interactive auto-graded assessments to help students practice concepts and take tests. Students can see their scores and a summary of the performance of the rest of the class. We used Dysgu for multiple out-of-class activities at two universities with different student demographics for two semesters. The data shows that students obtain better grades when using Dysgu. We also saw more on-time or ahead-of-time submissions with Dysgu. Survey responses indicated several Dysgu features which students found helpful. We conclude that digital educational platforms should consider features to support scaffolding to master the concept, peer influence to keep students engaged, self-reflection to foster critical thinking, and easy adaption of the platform to reduce faculty workload and improve students’ acceptance of the system.

1. Introduction

Classroom instructions and interactions are important for student learning. However, a big part of student learning involves practicing the concept outside of the class. In fact, college guidelines suggest that students should spend 2–3 h outside the classroom (per credit hour of class per week) reviewing and practicing the concepts [1,2]. However, students are struggling to keep up with these out-of-class activities due to several reasons, including job commitments and the medium of these out-of-class activities [3,4]. Thus, it is essential to develop methods that will allow the effective usage of the students’ available time for studying outside the classroom and provide students with interactive out-of-class activities that will involve them with the help of engaging in the features of such activities and the medium through which such activities are provided. The objective of this study is to evaluate the role of peer influence, self-reflection, and scaffolding in a mobile platform in order to keep students engaged with out-of-class activities.

In this paper, we present a mobile application that allows students to complete their out-of-activities at their preferred schedule. The application allows students to see their scores in these activities and compare their standing with their peers in the class. Additionally, the application supports the scaffolding of the activities along with contextual notifications for them to be kept involved with those activities. The data suggests that students find it easier to use an app rather than a pen-paper version of out-of-class activities. We also saw better student performance in the activities (e.g., lower D/F/W rates). Our findings suggest that peer influence and gamification can engage students in out-of-class activities while improving their grades and reducing the rate of procrastination.

2. Materials and Methods

This section contains a review of the literature related to some of the major features of Dysgu—the mobile application we developed for engaging students with out-of-class activities. We later provide a description of Dysgu’s significant features and details of the studies carried out at two different universities.

2.1. Related Work

The role of peer influence in non-academic and academic settings has been studied in depth [5,6]. There are many forms of direct peer influence in education, including peer learning, peer-teaching, peer feedback, and peer assessment [7,8]. Multiple research has repeatedly found positive impacts due to peer instructions on student performance and retention [9,10]. Indirect peer influence also comes in various forms, including gamification [11]. One study revealed that one of the non-academic factors influencing university students’ self-regulation of study is peer influence [12]. Multiple research has identified connections between peer influence and improved student engagement, as well as academic performance [13,14].

Self-reflection is linked with improved learning and academic performance [15,16]. Many researchers have studied the role of intervention techniques in self-reflection and academic performance [17,18]. Additionally, researchers showed that being aware of student activities supports self-reflection [19,20].

Researchers have noted that student engagement can be increased through instructional scaffolding [21,22]. Such scaffolding can be achieved through dynamic assessments and providing various forms of support, including executive control (e.g., support for time management) [23].

With the increasing access to mobile (i.e., cell) phones, recommendations are made to include mobile platforms in teaching and learning to support student learning outcomes and increase student engagement [24]. Mobile platforms are frequently used to complement teaching and learning in formal and informal educational settings [25,26]. Mobile games also leverage the mobile platform for education [27]. Others use mobile apps to train or teach concepts [28,29]. At the same time, some have used mobile app development within a course to motivate students and improve student learning [30,31]. Numerous studies have shown that mobile learning is effective in engaging students [32,33] and increasing student achievements [34,35].

2.2. Dysgu and Its Features

Dysgu [36,37] provides features to achieve student learning and engagement outside of the classroom. Although the authors have visualized this system, design decisions for Dysgu were mostly directed by several mobile learning requirement catalogs [38,39] to make the software more responsive and engaging. Dysgu has two different applications: the faculty side and the student side. Both sides communicate with a cloud-based repository which stores all the data. Some of Dysgu’s major features are described next.

(A).

The system’s background operations are transparent to the stakeholder so that there is no need for administration or the day-to-day management of the software. It is achieved through a cloud-based repository system designed explicitly for Dysgu. Neither the faculty nor the students have to worry about managing or paying for such cloud infrastructure and services, as Dysgu only uses the cloud vendors’ free options.

(B).

Dysgu allows activities to be put in learning paths with different degrees of difficulty to support instructional scaffolding. As shown in Figure 1a, a module can have different learning paths with different activities. Some of the modules’ paths can be designated as practice paths to help students practice the concept. Other paths are designated as student learning outcome (SLO) paths, where students are assessed on their learning skills. Additionally, some paths can be assigned as extra credit, which creates problems as this requires additional effort from the students. Students see the same module in the app (Figure 1b) with a different representation where exercises in the SLO path are revealed one at a time, and other paths are shown fully at the beginning.

(C).

Students can earn a score in the SLO path, which is used to calculate their course grades; however, students can only earn points by solving problems in other paths. Points are an internal currency in the software which students can use to extend the deadlines of the modules.

(D).

Each exercise in a path is modeled as an interactive activity. While these activities must be finished within a timeframe, they have interactive elements, such as multiple screens, multiple user interface components to interact with, cause-effect scenarios, and ways to traverse the problem before submitting it for grading. Figure 2a shows a sample interactive activity where answers can have right or wrong hints, or students can select multiple answers or modify their answers by looking at the question’s components. Figure 2b shows another example where students classify entries by dragging them to the corresponding bucket.

(E).

To motivate and engage students, Dysgu provides lightweight gamification and social networking aspects within student privacy regulations. Students can compare their progress (Figure 3a) with their classmates. Dysgu shows the student score, placement (module, path, exercise-specific) compared to the class, and timing information regarding problem-solving activities. Additionally, students are awarded badges depending on different conditions (the fastest in class to answer, highest score, etc.), and they can see what badges they have won and compare them with the rest of the class (Figure 3b). Additionally, it also shows how many other students have checked their progress, which might encourage the students to engage more in the activities.

(F).

Dysgu is developed to support several adaptations for different class situations and personalization to address student needs. For instance, the client app allows students to set (Figure 4a) the number of notifications they will receive from the app each day or if there are any blackout days. This enables students to change and update their study time to match their class, work, and other schedules and instruct the app when to remind them of pending tasks and how many times to remind them during the day.

Table 1. Dysgu notifications.

Notification Type	Sample Notification
Encouraging	Congratulations! You have the highest score in the module so far. Congratulations! You have the most points in the class. Congratulations! You have a higher score than the class average on a module. Congratulations! On average, you answered questions the fastest. Congratulations! On average, you answered questions with fewer tries.
Cautioning	Careful! You have the lowest score in the module so far. Careful! You have the lowest point in the module so far. Careful! You have a lower score than the class average on a module. Careful! On average, you answered questions the slowest. Careful! On average, you answered questions with the most tries. You did not work on any single problem today!
Neutral	You have module(s) due in less than 12 h! Do not forget to finish your activities. You did not solve any problem in the last 24 h!

shows a list of notifications that Dysgu uses to remind students of their progress, and Figure 4b shows how students can visualize such notifications. Although a student can control the number and time of their notifications, some (neutral ones) cannot be controlled by students and will appear if corresponding conditions are met.

2.3. Study Details

Dysgu was deployed in the CS1301 course at the University of Texas-El Paso (UTEP) and in the CSC/CIT 1311 course at Winston-Salem State University (WSSU). Both of these courses are freshmen-level courses and expose students to Java programming language.

Table 2. Details of Dysgu deployment.

	UTEP		WSSU
	Without Intervention	With Intervention	Without Intervention	With Intervention
Semesters	Spring 2021, Spring 2022 (Section B)	Fall 2021, Spring 2022 (Section A)	Spring 2019, Fall 2019	Spring 2020, Spring 2022
Number of students	66	40	31	17

shows the data that was collected before using the software in the class and during its usage in the class. Note that, at UTEP, in Spring 2022, Author 2 taught two sections of the CS1 course. Dysgu was deployed in Section A, whereas Section B students used Blackboard to submit their responses to the same questions.

UTEP is a majority-minority Hispanic-serving Institution that enrolls over 27,000 undergraduate students, of which over 85% are Hispanic-American [40]. WSSU is a predominately black university where 86% of the students are African American or from other minority groups [41].

Table 3. Student population information.

Gender (in %)			Course Load (in %)			Out-of-Class Work Hours (in %)
	UTEP	WSSU		UTEP	WSSU		UTEP	WSSU
Male	77.27	58.82	Less than full time (<12 credit hours)	29.55	5.88	Less than 10 h	50	52.94
Female	18.18	41.18	Full-time (12–16 credit hours)	61.36	70.59	10 to 20 h	18.18	17.65
Prefer not to say	4.55	0	Above 16 credit hours	9.09	23.53	21 to 30 h	15.91	17.65
						30 to 40 h	11.36	11.76
						Above 40 h	4.55	0

shows student information on both campuses during the intervention semesters. It is nice to see a gender balance in one of the campuses (WSSU), whereas the other (UTEP) showed regular patterns found in similar studies (i.e., 77% male). UTEP has 70% of students enrolled in 12 or more credit hours, whereas WSSU has more than 93% of students with such a course load. It is eye-opening to see that roughly half of the students work extensively (e.g., more than 10 h) outside the class for financial reasons. Having software such as Dysgu allows us to address such situations that students are in. Out-of-class activities through Dysgu allow students to work on them anytime and anywhere, allowing them to best use their time.

The same assignment was used before and after deploying Dysgu in the class. Each assignment asked students a sequence of questions that students needed to answer by the due date. Before Dysgu, questions from different topics were given to the student through the learning management system (LMS) with a deadline of a week. Students were expected to submit their answers using the LMS. During Dysgu, we used the same set of questions to assess the students’ learning. However, to comply with Dysgu’s scaffolded assignment approach, each assignment was structured as follows:

• Questions from the same topic were put on a module with the topic’s name.

  ▪

  Therefore, each assignment had multiple modules in them (having the same questions as before for assessment and scoring).

• Each module:

  ▪

  Has one SLO path with the questions enabled one at a time. Once the student answers a question, the next question in that path is turned on. Each question can only be tried once.

  ▪

  Has one practice path with questions to practice the specific topic.

  ▪

  Has an optional extra credit path. Questions in the extra credit path only gave points (not scores).

Table 4. Module comparison for Dysgu and LMS.

How They are Same

How They are Different

At WSSU, each assignment was given on a Monday, and students were asked to submit it by midnight on Friday. At UTEP, each assignment was given a week to complete. At both universities, each assignment asked the same questions on the same topics.

At WSSU, students were asked to submit an MS Word document containing answers in Canvas LMS. At UTEP, students were given auto-graded questions in Blackboard LMS. Students had time until the due date to answer the questions, whereas, in Dysgu, once students started to answer a question within a module, they had a set amount of time (around 5–10 min) to answer it. The module was open until the final due date.

shows the properties and comparisons of those assignments.

The same instructors (both authors) taught the courses before Dysgu was used and during the deployment of Dysgu. Similar course settings and teaching practices were consistently maintained before and during the Dysgu interventions. To diminish the impact of confounding variables, such as the size, display, and appearance of mobile devices, each student was given an identical Samsung tablet for the study duration, which the students took home. The software was used two times during the semester at WSSU and three times at UTEP.

Table 5. Modules used in this study.

University	Module	Content
UTEP	First	Variable
	Second	Conditional/Selection
	Third	Method
WSSU	First	Programming Ethics
		Java Basics
		Classes and Objects
	Second	Decision Structures
		Loops
		Arrays
		Text processing

shows the content of each module.

At UTEP, the course was an introductory CS course (CS1), and the Dysgu-based out-of-class activities were deployed throughout the semester. At WSSU, the course was a freshman programming-I course, and the out-of-class activities administered by Dysgu were to assess their knowledge of the content of that course. Therefore, those modules were administered at the beginning of the semester. The study was performed following all Institutional Review Board (IRB) protocols involving human subjects, where student participation was entirely voluntary, and students were not provided with any extra benefits or penalized for non-participation. In addition, all students signed a consent agreement before the start of the study, where they were informed about data collection and other related factors of the study.

3. Results

It is hypothesized that the students’ usage of Dysgu, with its real-time status updates, allows students to self-reflect on their progress and compare it with their peers. Such influence will allow the students to engage with the content more and eventually learn the subject material better. Dysgu’s impact on the different student demographics and students’ open-ended feedback regarding the strengths and limitations of the features of Dysgu are also explored. This research was directed by the following research questions:

• Is there any improvement in student performance after using this software?
• What is the effect of the new intervention on student procrastination and participation?
• Are there any impacts on students with self-reflection and peer influence?
• What feature of the mobile platform do students find most engaging and helpful?
• What type of student behavior is noticed when using this software for out-of-class activities?
• Is there any notable difference between how the intervention impacts two distinct student demographics?

To measure the students’ learning improvement in the topics, we used the assignment grades as evaluation metrics. The graded Dysgu-based out-of-class exercises contributed toward 12% of the total course grade at UTEP and 10% of the total course grade at WSSU. However, the students’ long-term knowledge acquisition was further tested on the same topics during the midterm and final exams, and each of these exams contributed between 15% and 20% toward the total course grade. In addition, assignment and course failure rates are also considered a measure of knowledge acquisition and retention. A pre- and post-survey was conducted before and after deploying Dysgu in the classroom to observe the students’ perceptions about the different facets of out-of-class activities. In addition, to assess student engagement and perception of the software and its effectiveness, an experience survey was offered to the learners only after they completed using Dysgu. The students gave their opinion about the statements in all surveys using a Likert scale of four values (Strongly Agree, Agree, Disagree, and Strongly Disagree), with an agreement scale ranging from strongly agree (4) to strongly disagree (1). To ensure that the students were actually looking at the survey content, not just “clicking it through,” each survey was designed to contain both positive and negative questions. Additionally, the experience survey asked students to select the most helpful features from a list of features provided by the software.

3.1. Student Grade

Research question A seeks to investigate the role of Dysgu on student performance. Figure 5 shows grade comparisons of the student’s work before and after using Dysgu. The results indicate that fewer students were failing, and the average student’s grade was improving. Although for UTEP, the improvements in the highest grade were more pronounced, for WSSU, improvement in the failure rate was more noticeable. It was also observed that, along with reducing course failure and withdrawal rates at WSSU, this intervention was also helpful for average or above-average students, who would otherwise receive a lower grade; however, with this intervention, they were able to engage more and achieve higher grades. On average, across both universities, the D/F/W rate was reduced by 27%.

3.2. Timing Information

Our second research question seeks to understand the effect of the new intervention on student procrastination and participation. A critical aspect of the new intervention (i.e., Dysgu) was to start students working on the out-of-class activity as early as possible and reduce their chance of procrastinating and submitting subpar work. This also provided one way to track when students started their work on the modules. Figure 6 shows when students started working before the deadline. It is obvious that most of the students started working more than 24 h before the deadline. This not only indicates an improvement in the students’ procrastination behavior but also shows that students wanted to start early so that they could see their progress in the activities and compare them to the rest of the class. Such an outcome also sheds light on research question C as it correlates to the student survey responses, as presented in Section 3.3. Without Dysgu, we would never have such timing data from the LMS and could not relate it to student performance and behavior. Additionally, the student’s final submission time is also compared, and it was assumed that if students submitted the activity just before it was due, they might have started working on it late. Since LMS provides us with the student’s submission time, it is possible to compare the two times. As shown in Figure 7, with Dysgu, more than half of the students submitted their work more than a day before the deadline. This improvement was clearly visible in both universities. In both cases, with Dysgu, more students submitted before a day, and fewer submitted just before the deadline compared to when they did not use Dysgu. The benefits of early submission also represented an early start on the assignment and an improved grade, as depicted earlier.

3.3. Student Experience

The fourth research question focuses on finding the most engaging and helpful components of the mobile educational platform. To gather student impressions of the software and to learn through out-of-class exercises, an experience survey was given on both campuses during the intervening semesters. The survey contained 10 questions with a Likert scale of four values (Strongly Agree, Agree, Disagree, and Strongly Disagree) and two open-ended questions. Figure 8 shows the results of some of the questions of this survey. For simplifications, “Strongly Agree” and “Agree” were combined as “Agreeing,” and the two disagreement answer choices were combined as “Disagreeing” in these charts. Figure 8 shows that students, on average, reported very high levels of agreement on the survey questions. On both campuses, the students found it helpful to be able to compare their grades with the rest of the class (Figure 8a) and noted that the ability to compare their progress to the rest of the class encouraged their involvement with the out-of-class activities (Figure 8e). In general, students felt more confident in completing Dysgu out-of-class activities than pen-and-paper-based out-of-class activities (Figure 8b) and found it enjoyable to complete these activities as per their own schedule (Figure 8c). Students also noted that badges and points encouraged them to stay involved in out-of-class activities (Figure 8d). These results show strong support for Dysgu-based out-of-class activities and a preference towards the Dysgu features, which were included to improve student engagement (e.g., badge, grade comparison).

In general, students on both campuses show similar impressions of the intervention using Dysgu. Although students at WSSU agreed less (compared to UTEP students) on feeling more confident completing Dysgu activities, their response to question iii of the pre- and post-survey contradicts this. The WSSU students’ agreement on question (iii) (“I prefer out-of-class activities that can be completed using mobile devices”) increased by 26% after using Dysgu.

Table 6. Descriptive statistics for experience survey response.

	(a)		(b)		(c)		(d)		(e)
	µ	σ	µ	σ	µ	σ	µ	σ	µ	σ
UTEP (N = 24 *)	3.38	0.71	3.42	0.72	3.5	0.72	3.33	0.86	3.42	0.78
WSSU (N = 17)	3.27	0.70	2.87	0.99	3.2	0.94	2.8	0.86	3.0	0.85

* At UTEP, every participant did not complete the experience survey.

shows the descriptive statistics (µ = mean, σ = standard deviation) for those same questions. The results indicate that the average response represents the students’ uniform attitude towards the intervention, as reflected by a lower standard deviation value.

In the same experience survey, students were asked to pick the app’s most useful features. Figure 9 shows a few of the most chosen features (across both campuses) that students thought were useful in the learning process and for engagement with out-of-class activities. The ability to compare their grades with the rest of the class was chosen as the top feature at both campuses. The second most liked feature was the ability for self-assessment. It is evident from the chart that students liked the features that helped them to succeed in the rest of the course. It also showcases that scaffolded out-of-class activities that provide interactions are mostly preferred. Additionally, a mobile device’s notification features are preferred—most likely for helping students manage time and keep up with their peers.

3.4. Pre and Post-Survey

Pre- and post-surveys were conducted to assess the students’ change in perception of the new intervention. The pre-survey was given before the students were introduced to the app, and the post-survey was conducted after the end of the intervention. As earlier, we combined the two “agrees” into an Agreement and the other as a Disagreement. We then aggregated all the responses for each statement and calculated their percentages. For each statement, we then compared the change in percentage for each of the two (Agreement and Disagreement) choices, which are listed in

Table 7. Pre- and post-survey results.

Statements	UTEP	WSSU
i. Faster feedback on assignments helps me to learn better.	Agreement increases	Disagreement increases
	2.27%	12.5%
ii. Learning to use technologies for coursework is simply additional work beyond the normal coursework.	Disagreement increases	Disagreement increases
	8.97%	15.07%
iii. I prefer out-of-class activities that can be completed using mobile devices.	Agreement increases	Agreement increases
	3.59%	28.68%
iv. Using interactive out-of-class activities enhances my learning.	Agreement increases	Agreement increases
	1.2%	11.03%
v. I like classes that allow me to practice concepts taught in-class in a hands-on-approach, such as take-home activities.	Agreement increases	Disagreement increases
	1.2%	0.37%
vi. Performing out-of-class hands-on-exercises are NOT helpful in preparing for midterm and final exams.	Agreement increases	Disagreement increases
	2.15%	16.54%
vii. When I see how my peers have performed in the out-of-class activities, it increases my course engagement.	Agreement increases	Agreement increases
	3.47%	4.8%

.

For each statement in the survey, we listed whether the students’ agreement or disagreement increased and the amount of the increase. For most of the statements, we assumed a response pattern that we anticipated seeing because of the intervention. However, in a few of the statements, the changes were not as expected and are highlighted in orange in the table. For statement I, we anticipated that faster feedback would be helpful for learning; hence, we expected to see an increase in agreement with this statement. However, we can see a bit of an anomaly in the student response in one of the campuses (WSSU), although their grade and submission timeframe improved significantly, as shown in Figure 5 and Figure 7. This could be due to the students not seeing any connection between receiving faster feedback and better learning. The value of different forms of feedback (e.g., formative, immediate) has been studied in depth and shows that different types of feedback affect the students’ performance, motivation, and engagement [42]. This disagreement shows that further investigation is needed to understand student perception of learning and what is the impact of different types on student learning.

We observed a similar anomaly for statement v. This statement asked students if they liked classes with hands-on activities such as out-of-class activities. We expected students to agree with this statement. However, one of the campuses showed a very small disagreement (<0.5%) and needed further investigation. Similarly, with statement vi, we expected an increase in disagreement as any form of practice could be helpful for tests. One of the campuses (i.e., UTEP) showed an agreement with this statement, indicating that the students did not find the out-of-class activities helpful for the exams.

Other than these anomalies, we can see that students are largely in agreement with the statements related to the usefulness of interactive out-of-class activities and the ability to see how their peers are doing in those activities. Overall, the pre- and post-survey data reconfirm the novelty of this research and the usefulness of the presented approach.

3.5. Usage Analytics

To investigate research question E, Dysgu collected student usage analytics while the students were using the app for out-of-class activities. As shown in Figure 10a, the students at both universities checked their status, which provides a comparison of their scores with their peers in the class. This was the most checked feature of the app in both universities. The second most used feature at UTEP was the badge students received to encourage their participation and efforts. At WSSU, the second most used feature was the notifications students received. The third most used feature at both universities was the points students received for the extra-credit pathway. We also noticed that students from both campuses were using the app mostly in the morning (Figure 10b).

3.6. Student Comments

The last question on the experience survey was open-ended. We asked for participants’ suggestions on improving Dysgu or their experience with Dysgu. We received a total of 39 text responses for this question across two campuses over the two semesters. We analyzed the responses to identify emerging themes in the responses. There were three major themes repeated throughout the response (e.g., GUI, more challenging questions). We coded those themes and counted their occurrences in the responses. The three themes in the responses were related to: the Dysgu app (e.g., graphical user interface (GUI), the features of Dysgu, that they liked Dysgu and had no suggestions for its improvement, and suggestions related to the content (e.g., challenging questions). Figure 11 shows the distribution of these themes in the responses. Following is a brief discussion of these themes.

Dysgu-related suggestions: These responses were related to various aspects of the Dysgu app. One of those aspects was the GUI—the user interface of Dysgu. Most responses in this theme suggested making the interface more user-friendly and intuitive to use. One such response was, “I understand that the app is in an early stage, but I feel like having more modern-looking menus, tabs, and that kind of material inside the app would have made the experience even more enjoyable and fun”.

Within Dysgu codes, aside from the GUI sub-theme, another theme emerged that was related to the features of Dysgu. Students suggested additional features in Dysgu to make it more effective or engaging. Most suggestions in this area were related to the time limit of the questions and the ability to pause the timer if the app was turned off. One suggestion was related to peer communication, “another is being able to communicate/text with classmates within the app for help.”

Some responses were related to the performance of the app or the device. Some suggestions were to make the response of the app faster. Dysgu is available for android platforms. One suggestion was to make the app suitable for all phones, “If it was available on my typical device (phone), I would use it much more …”

Content: This theme is the least related to Dysgu. The responses related to this theme discussed how the questions could be more challenging or better articulated. One such response was, “The dysgu should include more challenging questions too”. Responses to this theme also made the suggestion of having more assignments, more questions, or having an adaptive system that would set the difficulty of the questions based on the student’s level. A student suggested, “The app was great. One thing I would suggest is to put a challenge section where the student gets questions one at a time based on the material learned, the next question being harder than the last. The moment they get a question wrong, the challenge stops…”

Liked it/nothing to add: 27% of the participants liked the Dysgu app. They did not have any suggestions on improving Dysgu or the experience with Dysgu.

3.7. Self-Reflection, Peer-Influence, and Different Student Demographics

Going back to our research questions, to answer research question C (to study any impacts of Dysgu on students regarding self-reflection and peer influence), we looked at the data related to student grades, submission time, and their feedback on the useful features of Dysgu. Results from the pre-post survey and the experience surveys showed that the students found the ability to see their progress in the app helpful. This shows that students across both campuses value features that enable them to reflect on their standing in the class—thus, self-reflection is essential. Similarly, students noted that knowing their standing in the class compared to the rest of the class was also helpful in keeping them involved with the out-of-class activities. We saw this in both student demographics, indicating that peer influence acts as a strong motivator for students to stay engaged with class activities outside the classroom. These statements can be complemented with the grade and timing data, as we see improved grades and better submission times when Dysgu was deployed. Altogether the data indicates that self-reflection and peer influence positively impact student engagement and performance.

The last research question (question F) seeks to identify any notable difference between how the intervention impacts the two distinct student demographics. In terms of grades, we saw a significant drop in D/F/W rates at WSSU, with an increase in the number of students getting either grades B or C. At UTEP, the D/F/W rate slightly increased with Dysgu; the number of students receiving grade A also increased. Though the grade distribution is different on these campuses (with fewer B and C grades at UTEP compared to WSSU), the number of students with better grades increased on both campuses. In terms of the timeline, students from both campuses started work on the activity more than 24 h before the deadline. Most students also submitted the activity more than 24 h before the deadline at both campuses. Both student groups worked mainly in the morning or in the evening. The afternoon hours had the least activity. Without Dysgu, the students at WSSU were more likely to submit their work within the last five hours of the deadline. We also saw a similar preference for Dysgu features across the campus. One notable difference here was that UTEP students viewed their badges a lot; WSSU students, on the other hand, were checking the app’s notifications. Both student groups used the app in the morning, but the usage percentage was higher at WSSU. Overall, we can see that there are differences in the usage behavior of the app, but the results (in terms of grades and submission time) are somewhat the same across both demographics (i.e., Hispanic and African American).

4. Discussion

We investigated the impact of a mobile platform in engaging students with out-of-class activities. Dysgu allowed us to study the impact of scaffolded out-of-class activities administered through a mobile platform in order to keep students engaged outside the class. Dysgu also includes features that support self-reflection and peer influence. The results show that students perform better when they remain engaged with out-of-class activities and with their peers. We also saw increased on-time or ahead-of-time submissions using Dysgu. The student responses showed that students preferred mobile platforms for out-of-class activities. They also favored the notification feature to remind them of the deadline. The grading and timing data and experience surveys show that peer influence and self-reflection in the mobile platform are good motivators and have a positive impact on performance and on-time submission. The results of this study show the promise of peer influence, self-reflection, and scaffolding in keeping students engaged with out-of-class activities. This result should pave the path for further longitudinal studies to identify the long-term effects of peer influence and self-reflection on student learning and engagement.

We conducted multiple studies across two campuses with a majority-minority population. Our study revealed several Dysgu features which students preferred across both campuses. One of the key findings of our study was that the design of mobile educational platforms should consider four key factors to achieve higher student engagement, student learning, and improved self-efficacy in students, as described next (Figure 12).

• Scaffolding—digital platforms should support scaffolding in various forms. Content or assessments should be dynamic and adaptive to support the students with different learning paces. For example, an assessment can start with easier questions and build the difficulty levels of subsequent questions based on the student’s responses. Auto-graded practice and assessment questions would make it easier for instructors to provide instant feedback—which also supports student learning and engagement.
• Peer influence—Peer influence acts as a motivator for many students. Digital platforms should consider the effective integration of peer influence in supporting student learning and increasing student engagement. Chapin [43] identified three factors that impact the self-efficacy of early Computer Science students: engagement with the problem, engagement with others, and engagement with the environment. Scaffolding, when used right, can be leveraged to create engagement with the problem. Similarly, knowing how peers are doing in the class can help students stay engaged with others in the class and the class environment; eventually, this can help to improve self-efficacy. However, attention should be given while designing a system that supports peer influence to limit any potential negative consequences of peer influence.

• Self-reflection—students should be aware of their learning status. Any educational platform should have support for some form of self-reflection. Self-reflection can be supported through automatic grading in tests or reflective journaling. Self-reflection not only helps with acquiring content knowledge but also with metacognition and developing critical thinking skills.
• Usability—Digital tools should be at least available on mobile devices to support easier access and ubiquitous learning. These tools should also be available for different platforms (e.g., cell phones, tablets, laptops). Tools should be user-friendly and intuitive to use. The tool’s design should consider usability to keep students engaged with the platform. Gamification (e.g., badge, points) can provide an engaging virtual learning environment.

Although Dysgu utilizes the transformative power of mobile technology for out-of-class activities, student engagement in active learning outside the classroom, and the possibility of providing early intervention, has some limitations. Most significantly, the app is only supported in the Android ecosystem. Although Dysgu is created in a way that supports any interactive activity, the current version of the app only supports two different kinds of activities, and other types of activities need to be developed for broader acceptance. Finally, data from the long-time usage of the system should be collected to see any trends developing that might provide better insights into this approach.

The use of any technology requires an understanding of the potential impacts and unintended consequences. Any new tool or technique thus requires extreme caution and monitoring when deployed. A platform that supports peer influence and promotes self-awareness is no different. In designing such systems, we should strive to create an inclusive environment, keeping many factors, such as collaboration, reflection, scaffolding, and communication, in mind. Systems should ensure that peer influence is positive and that self-reflections support developing self-efficacy. With careful design and deployment, such systems can create an inclusive and equitable environment that is conducive to student engagement and growth.