The Impact of Dashboard Feedback Type on Learning Effectiveness, Focusing on Learner Differences
Abstract
:1. Introduction
2. Literature Review
2.1. Feedback on the Dashboards
2.2. The Impact of Dashboards on Student Learning
3. Methods
3.1. Participants and Methods
3.2. Research Tools
3.2.1. Descriptive Dashboard
3.2.2. Prescriptive Dashboard
3.3. Evaluation Procedure
4. Results
4.1. H1. Learning Performance of the Descriptive Dashboard and the Prescriptive Dashboard
4.1.1. Comparison between Two Experimental Groups and the Control Group
4.1.2. Comparison between the Two Experimental Groups
4.2. H2. Learning Strategy of the Descriptive Dashboard and the Prescriptive Dashboard
4.3. H3. Learning Attitude of the Descriptive Dashboard and the Prescriptive Dashboard
4.4. H4. Influence of Individual Factors
4.4.1. Prior Knowledge Level
4.4.2. Learning Attitude Level
4.4.3. Learning Strategy Level
5. Findings
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Conflicts of Interest
References
- Bodily, R.; Verbert, K. Review of Research on Student-Facing Learning Analytics Dashboards and Educational Recommender Systems. IEEE Trans. Learn. Technol. 2017, 10, 405–418. [Google Scholar] [CrossRef]
- Yoo, Y.; Lee, H.; Jo, I.-H.; Park, Y. Educational Dashboards for Smart Learning: Review of Case Studies. In Emerging Issues in Smart Learning; Chen, G., Kumar, V., Kinshuk, Huang, R., Kong, S.C., Eds.; Lecture Notes in Educational Technology; Springer: Berlin/Heidelberg, Germany, 2015; pp. 145–155. [Google Scholar] [CrossRef]
- Winne, P.H. A metacognitive view of individual differences in self-regulated learning. Learn. Individ. Differ. 1996, 8, 327–353. [Google Scholar] [CrossRef]
- Zimmerman, B.J.; Schunk, D.H. Self-Regulated Learning and Academic Achievement: Theoretical Perspectives; Routledge: Uniondale, NY, USA, 2001. [Google Scholar]
- Sedrakyan, G.; Malmberg, J.; Verbert, K.; Järvelä, S.; Kirschner, P.A. Linking learning behavior analytics and learning science concepts: Designing a learning analytics dashboard for feedback to support learning regulation. Comput. Hum. Behav. 2018, 107, 105512. [Google Scholar] [CrossRef]
- Jovanović, J.; Saqr, M.; Joksimović, S.; Gašević, D. Students matter the most in learning analytics: The effects of internal and instructional conditions in predicting academic success. Comput. Educ. 2021, 172, 104251. [Google Scholar] [CrossRef]
- Verbert, K.; Govaerts, S.; Duval, E.; Santos, J.L.; Van Assche, F.; Parra, G.; Klerkx, J. Learning dashboards: An overview and future research opportunities. Pers. Ubiquitous Comput. 2013, 18, 1499–1514. [Google Scholar] [CrossRef] [Green Version]
- Kokoç, M.; Altun, A. Effects of learner interaction with learning dashboards on academic performance in an e-learning environment. Behav. Inf. Technol. 2019, 40, 161–175. [Google Scholar] [CrossRef]
- Schwendimann, B.A.; Rodriguez-Triana, M.J.; Vozniuk, A.; Prieto, L.P.; Boroujeni, M.S.; Holzer, A.; Gillet, D.; Dillenbourg, P. Perceiving Learning at a Glance: A Systematic Literature Review of Learning Dashboard Research. IEEE Trans. Learn. Technol. 2016, 10, 30–41. [Google Scholar] [CrossRef]
- Susnjak, T.; Ramaswami, G.S.; Mathrani, A. Learning analytics dashboard: A tool for providing actionable insights to learners. Int. J. Educ. Technol. High. Educ. 2022, 19, 1–23. [Google Scholar] [CrossRef]
- Siemens, G.; Long, P. Penetrating the Fog: Analytics in Learning and Education. Educ. Rev. 2011, 46, 30. [Google Scholar]
- Sønderlund, A.L.; Hughes, E.; Smith, J. The efficacy of learning analytics interventions in higher education: A systematic review. Br. J. Educ. Technol. 2018, 50, 2594–2618. [Google Scholar] [CrossRef]
- Leitner, P.; Ebner, M.; Ebner, M. Learning Analytics Challenges to Overcome in Higher Education Institutions. In Utilizing Learning Analytics to Support Study Success; Ifenthaler, D., Mah, D.-K., Yau, J.Y.-K., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 91–104. [Google Scholar] [CrossRef]
- Umer, R.; Susnjak, T.; Mathrani, A.; Suriadi, L. Current stance on predictive analytics in higher education: Opportunities, challenges and future directions. Interact. Learn. Environ. 2021, 1–26. [Google Scholar] [CrossRef]
- Daniel, B. Big Data and analytics in higher education: Opportunities and challenges. Br. J. Educ. Technol. 2014, 46, 904–920. [Google Scholar] [CrossRef]
- Yoo, M.; Jin, S.-H. Development and Evaluation of Learning Analytics Dashboards to Support Online Discus-sion Activities. Educ. Technol. Soc. 2020, 23, 1–18. [Google Scholar]
- Bao, H.; Li, Y.; Su, Y.; Xing, S.; Chen, N.-S.; Rosé, C.P. The effects of a learning analytics dashboard on teachers’ diagnosis and intervention in computer-supported collaborative learning. Technol. Pedagog. Educ. 2021, 30, 287–303. [Google Scholar] [CrossRef]
- Zamecnik, A.; Kovanović, V.; Grossmann, G.; Joksimović, S.; Jolliffe, G.; Gibson, D.; Pardo, A. Team interactions with learning analytics dashboards. Comput. Educ. 2022, 185, 104514. [Google Scholar] [CrossRef]
- Gutiérrez, F.; Seipp, K.; Ochoa, X.; Chiluiza, K.; De Laet, T.; Verbert, K. LADA: A learning analytics dashboard for academic advising. Comput. Hum. Behav. 2018, 107, 105826. [Google Scholar] [CrossRef]
- Hu, Y.-H.; Lo, C.-L.; Shih, S.-P. Developing early warning systems to predict students’ online learning performance. Comput. Hum. Behav. 2014, 36, 469–478. [Google Scholar] [CrossRef]
- Joseph-Richard, P.; Uhomoibhi, J.; Jaffrey, A. Predictive learning analytics and the creation of emotionally adaptive learning environments in higher education institutions: A study of students’ affect responses. Int. J. Inf. Learn. Technol. 2021, 38, 243–257. [Google Scholar] [CrossRef]
- Zheng, L.; Zhong, L.; Niu, J. Effects of personalised feedback approach on knowledge building, emotions, co-regulated behavioural patterns and cognitive load in online collaborative learning. Assess. Eval. High. Educ. 2021, 47, 109–125. [Google Scholar] [CrossRef]
- Sadallah, M.; Encelle, B.; Maredj, A.-E.; Prié, Y. Towards fine-grained reading dashboards for online course revision. Educ. Technol. Res. Dev. 2020, 68, 3165–3186. [Google Scholar] [CrossRef]
- Fleur, D.S.; van den Bos, W.; Bredeweg, B. Learning Analytics Dashboard for Motivation and Performance. In Intelligent Tutoring Systems; Kumar, V., Troussas, C., Eds.; Lecture Notes in Computer Science; Springer International Publishing: Cham, Switzerland, 2020; pp. 411–419. [Google Scholar] [CrossRef]
- Few, S. Information Dashboard Design; O’Reilly Media, Inc.: Sebastopol, CA, USA, 2006. [Google Scholar]
- Iandoli, L.; Quinto, I.; De Liddo, A.; Shum, S.B. Socially augmented argumentation tools: Rationale, design and evaluation of a debate dashboard. Int. J. Hum.-Comput. Stud. 2014, 72, 298–319. [Google Scholar] [CrossRef] [Green Version]
- Feild, J. Improving Student Performance Using Nudge Analytics; International Educational Data Mining Society: Paris, France, 2015. [Google Scholar]
- Sansom, R.L.; Bodily, R.; Bates, C.O.; Leary, H. Increasing Student Use of a Learner Dashboard. J. Sci. Educ. Technol. 2020, 29, 386–398. [Google Scholar] [CrossRef]
- Ott, C.; Robins, A.; Haden, P.; Shephard, K. Illustrating performance indicators and course characteristics to support students’ self-regulated learning in CS1. Comput. Sci. Educ. 2015, 25, 174–198. [Google Scholar] [CrossRef]
- Valle, N.; Antonenko, P.; Valle, D.; Sommer, M.; Huggins-Manley, A.C.; Dawson, K.; Kim, D.; Baiser, B. Predict or describe? How learning analytics dashboard design influences motivation and statistics anxiety in an online statistics course. Educ. Technol. Res. Dev. 2021, 69, 1405–1431. [Google Scholar] [CrossRef] [PubMed]
- Brown, M. Seeing students at scale: How faculty in large lecture courses act upon learning analytics dashboard data. Teach. High. Educ. 2020, 25, 384–400. [Google Scholar] [CrossRef]
- Guerra, J.; Ortiz-Rojas, M.; Zúñiga-Prieto, M.A.; Scheihing, E.; Jiménez, A.; Broos, T.; De Laet, T.; Verbert, K. Adaptation and evaluation of a learning analytics dashboard to improve academic support at three Latin American universities. Br. J. Educ. Technol. 2020, 51, 973–1001. [Google Scholar] [CrossRef]
- Ahn, J.; Campos, F.; Hays, M.; Digiacomo, D. Designing in Context: Reaching Beyond Usability in Learning Analytics Dashboard Design. J. Learn. Anal. 2019, 6, 70–85. [Google Scholar] [CrossRef] [Green Version]
- Widjaja, H.A.E.; Santoso, S.W. University Dashboard: An Implementation of Executive Dashboard to University. In Proceedings of the 2014 2nd International Conference on Information and Communication Technology (ICoICT), Bandung, Indonesia, 28–30 May 2014; IEEE: Bandung, Indonesia, 2014; pp. 282–287. [Google Scholar] [CrossRef]
- Hilliger, I.; De Laet, T.; Henríquez, V.; Guerra, J.; Ortiz-Rojas, M.; Zuñiga, M.Á.; Baier, J.; Pérez-Sanagustín, M. For Learners, with Learners: Identifying Indicators for an Academic Advising Dashboard for Students. In Addressing Global Challenges and Quality Education; Alario-Hoyos, C., Rodríguez-Triana, M.J., Scheffel, M., Arnedillo-Sánchez, I., Dennerlein, S.M., Eds.; Lecture Notes in Computer Science; Springer: Cham, Switzerland, 2020; pp. 117–130. [Google Scholar] [CrossRef]
- Chen, L.; Lu, M.; Goda, Y.; Yamada, M. Design of Learning Analytics Dashboard Supporting Metacognition; International Association for the Development of the Information Society: Lisbon, Portugal, 2019. [Google Scholar]
- Yilmaz, F.G.K.; Yilmaz, R. Learning analytics as a metacognitive tool to influence learner transactional distance and motivation in online learning environments. Innov. Educ. Teach. Int. 2020, 58, 575–585. [Google Scholar] [CrossRef]
- Han, J.; Kim, K.H.; Rhee, W.; Cho, Y.H. Learning analytics dashboards for adaptive support in face-to-face collaborative argumentation. Comput. Educ. 2020, 163, 104041. [Google Scholar] [CrossRef]
- Hellings, J.; Haelermans, C. The effect of providing learning analytics on student behaviour and performance in programming: A randomised controlled experiment. High. Educ. 2020, 83, 1–18. [Google Scholar] [CrossRef]
- Aguilar, S.J.; Karabenick, S.A.; Teasley, S.D.; Baek, C. Associations between learning analytics dashboard exposure and motivation and self-regulated learning. Comput. Educ. 2020, 162, 104085. [Google Scholar] [CrossRef]
- Wang, D.; Han, H. Applying learning analytics dashboards based on process-oriented feedback to improve students’ learning effectiveness. J. Comput. Assist. Learn. 2020, 37, 487–499. [Google Scholar] [CrossRef]
- Blau, I.; Shamir-Inbal, T. Re-designed flipped learning model in an academic course: The role of co-creation and co-regulation. Comput. Educ. 2017, 115, 69–81. [Google Scholar] [CrossRef]
- Jung, H.; Park, S.W.; Kim, H.S.; Park, J. The effects of the regulated learning-supported flipped classroom on student performance. J. Comput. High. Educ. 2021, 34, 132–153. [Google Scholar] [CrossRef]
- Lai, C.-L.; Hwang, G.-J. A self-regulated flipped classroom approach to improving students’ learning performance in a mathematics course. Comput. Educ. 2016, 100, 126–140. [Google Scholar] [CrossRef]
- Jivet, I.; Scheffel, M.; Schmitz, M.; Robbers, S.; Specht, M.; Drachsler, H. From students with love: An empirical study on learner goals, self-regulated learning and sense-making of learning analytics in higher education. Internet High. Educ. 2020, 47, 100758. [Google Scholar] [CrossRef]
- Kokoc, M.; Kara, M. A Multiple Study Investigation of the Evaluation Framework for Learning Analytics: Instrument Validation and the Impact on Learner Performance. Educ. Technol. Soc. 2021, 24, 16–28. [Google Scholar]
- Sanders, E.B.-N.; Stappers, P.J. Co-creation and the new landscapes of design. Codesign 2008, 4, 5–18. [Google Scholar] [CrossRef] [Green Version]
- Liu, W.; Lee, K.-P.; Gray, C.M.; Toombs, A.L.; Chen, K.-H.; Leifer, L. Transdisciplinary Teaching and Learning in UX Design: A Program Review and AR Case Studies. Appl. Sci. 2021, 11, 10648. [Google Scholar] [CrossRef]
- Xu, Z.; Makos, A. Investigating the Impact of a Notification System on Student Behaviors in a Dis-course-Intensive Hybrid Course: A Case Study. In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge; LAK ’15, Poughkeepsie, NY, USA, 16–20 March 2015; Association for Computing Machinery: New York, NY, USA, 2015; pp. 402–403. [Google Scholar]
- Holanda, O.; Ferreira, R.; Costa, E.; Bittencourt, I.I.; Melo, J.; Peixoto, M.; Tiengo, W. Educational resources recommendation system based on agents and semantic web for helping students in a virtual learning environment. Int. J. Web Based Communities 2012, 8, 333–353. [Google Scholar] [CrossRef]
- Muldner, K.; Wixon, M.; Rai, D.; Burleson, W.; Woolf, B.; Arroyo, I. Exploring the Impact of a Learning Dashboard on Student Affect. In Artificial Intelligence in Education; Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.F., Eds.; Lecture Notes in Computer Science; Springer International Publishing: Cham, Switzerland, 2015; pp. 307–317. [Google Scholar] [CrossRef]
- Kearsley, G.; Shneiderman, B. Engagement Theory: A Framework for Technology-Based Teaching and Learning. Educ. Technol. 1998, 38, 20–23. [Google Scholar]
- Abelson, R.P.; Rosenberg, M.J. Symbolic Psycho-Logic: A Model of Attitudinal Cognition. In Attitude Change; Routledge: Uniondale, NY, USA, 1968. [Google Scholar]
- Wang, Y. Study on Learning Strategies of High-Efficiency Mathematics Learning in Grades 5–6 of Primary School; Tianjin Normal University: Tianjin, China, 2019. [Google Scholar]
- Theobald, M.; Bellhäuser, H.; Imhof, M. Deadlines don’t prevent cramming: Course instruction and individual differences predict learning strategy use and exam performance. Learn. Individ. Differ. 2021, 87, 101994. [Google Scholar] [CrossRef]
- Kellen, K.; Antonenko, P. The role of scaffold interactivity in supporting self-regulated learning in a community college online composition course. J. Comput. High. Educ. 2017, 30, 187–210. [Google Scholar] [CrossRef]
Variable | Group | N | M | SD | SE | F | Post hoc |
---|---|---|---|---|---|---|---|
Pre-test scores | Experimental group A | 48 | 74.5521 | 15.63726 | 2.25704 | 1.306 | |
Experimental group B | 49 | 75.8776 | 16.36578 | 2.33797 | |||
Control group C | 48 | 71.0729 | 13.05226 | 1.88393 | |||
Post-test results | Experimental group A | 48 | 92.5729 | 6.51103 | 0.93979 | 4.128 * | C < A * C < B * |
Experimental group B | 49 | 93.1939 | 6.79983 | 0.9714 | |||
Control group C | 48 | 89.0521 | 9.30582 | 1.34318 |
Experimental Group A (Descriptive Analytics) | Experimental Group B (Prescriptive Analytics) | t | p | |||
---|---|---|---|---|---|---|
Mean | SD | Mean | SD | |||
Strategy Score | 4.3755 | 0.58293 | 4.1101 | 0.43942 | 2.535 | 0.013 |
Cognitive strategies | 4.3923 | 0.57917 | 4.0773 | 0.46685 | 2.952 | 0.004 |
Metacognitive strategies | 4.381 | 0.6403 | 4.0821 | 0.50656 | 2.552 | 0.012 |
Resource Management Strategy | 4.3588 | 0.59735 | 4.1558 | 0.41812 | 1.943 | 0.055 |
Variable | Experimental Group A | Experimental Group B |
---|---|---|
Prior knowledge | −0.910 ** | −0.911 ** |
Learning strategy | 0.288 * | 0.196 |
Attitudes | 0.200 | 0.120 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wang, H.; Huang, T.; Zhao, Y.; Hu, S. The Impact of Dashboard Feedback Type on Learning Effectiveness, Focusing on Learner Differences. Sustainability 2023, 15, 4474. https://doi.org/10.3390/su15054474
Wang H, Huang T, Zhao Y, Hu S. The Impact of Dashboard Feedback Type on Learning Effectiveness, Focusing on Learner Differences. Sustainability. 2023; 15(5):4474. https://doi.org/10.3390/su15054474
Chicago/Turabian StyleWang, Han, Tao Huang, Yuan Zhao, and Shengze Hu. 2023. "The Impact of Dashboard Feedback Type on Learning Effectiveness, Focusing on Learner Differences" Sustainability 15, no. 5: 4474. https://doi.org/10.3390/su15054474