Next Article in Journal
Effects of Plant Communities on Human Physiological Recovery and Emotional Reactions: A Comparative Onsite Survey and Photo Elicitation Study
Next Article in Special Issue
Psychometric Properties for Multidimensional Cognitive Load Scale in an E-Learning Environment
Previous Article in Journal
Access to COVID-19 Vaccination during the Pandemic in the Informal Settlements of Rome
Previous Article in Special Issue
Influence of COVID-19 Pandemic Uncertainty in Negative Emotional States and Resilience as Mediators against Suicide Ideation, Drug Addiction and Alcoholism
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Online Problem-Based Learning Intervention on Self-Directed Learning and Problem-Solving through Group Work: A Waitlist Controlled Trial

by
Florence M. F. Wong
1,* and
Crystal W. Y. Kan
2
1
School of Nursing, Tung Wah College, Hong Kong, China
2
School of Health Sciences, Caritas Institute of Higher Education, Hong Kong, China
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2022, 19(2), 720; https://doi.org/10.3390/ijerph19020720
Submission received: 6 November 2021 / Revised: 4 January 2022 / Accepted: 5 January 2022 / Published: 10 January 2022
(This article belongs to the Special Issue Psychology of Learning in Higher Education)

Abstract

:
Background: Small group work embraces independent study and interactive learning, which enhance knowledge acquisition and skills. Self-directed learning (SDL) and problem-solving (PS) are essential skills in the development of the nursing profession. During the coronavirus pandemic, virtual learning was indispensable. However, little is known about how students develop SDL and PS abilities through online learning through group work. Objective: To evaluate the effects of the online intervention on SDL and PS abilities through interactive group work. Methods: A randomised waitlist-control trial was carried out. A structured intervention using problem-based learning (PBL) as a guideline was used to direct student learning in small group work. Assessments were scheduled at Time 0 (baseline), Time 1 (8th week), Time 2 (16th week), and Time 3 (28th week). Results: The mean student age was 21.45 (SD = 0.86). About 78% of students were female. There was no significant difference in demographic characteristics and analysis at the baseline. Students in the intervention group reported greater improvement in the SDLRS and PSI at the 8th week, whereas those in the waitlist control group reported greater improvement in the SDLRS and PSI at the 16th week. Sustained effects in the SDLRS and PSI were observed in both the intervention and waitlist control groups at the 16th and 28th weeks, respectively. A repeated-measure analysis was performed to compare the SDLRS and PSI in different periods and revealed statistically significant results (p < 0.001) in all subscales of SDLRS and PSI in the four study periods. Conclusions: The guidelines appear to be an effective treatment for SDL and PS ability enhancement with sustainable effects through interactive group work. The guidelines with explicit instructions and learning objectives provide directions and guidance to students to learn more effectively. The educator plays a vital role in facilitating the students’ SDL and PS ability improvement.

1. Introduction

1.1. Background

Small group work is a well-known teaching and learning pedagogy in nursing education, as it encourages a student-centred learning approach and broadens student learning [1,2,3,4]. It embraces learning and development through independent learning and group collaboration [3,4,5,6]. Effective small group learning based on this premise should include students’ active engagement in learning and a sense of collaboration [3,4,5,6]. Since nursing practice emphasises evidence-based knowledge and skills to ensure optimal patient care, self-directed learning (SDL) has been encouraged to be developed for lifelong learning. In the advanced changes in this complex and sophisticated healthcare service, nurses are required to make immediate and appropriate clinical decisions to promote patient care recovery. During nursing education, students can achieve knowledge acquisition and skill development in the classroom and laboratories. The problem-based learning (PBL) approach has been entrenched to integrate students’ individual learned knowledge and skills during their learning process for a higher quality of education and more competent care for patients [7]. To maintain the quality of clinical care and to reach optimal patient outcomes, PBL is used to develop students’ problem-solving (PS) ability. With the growing utilization of interactive small group learning and PBL methods, it is imperative to better understand how students develop their SDL and PS abilities through preparing themselves to be immersed in a collaborative working environment [8,9]. The results of this study suggest that the guideline can help to increase nurse educators’ awareness to facilitate students’ SDL and PS skill learning through an effective interactive small group learning approach.

1.2. Background Literature

Small group work is a well-known teaching-learning pedagogy in nursing education to enhance students’ learning through active involvement in individual study and group interactions [1,2,3,4,6]. This teaching-learning approach embraces advantages in developing both in-depth theoretical knowledge and essential skills [10,11] and provides a good platform for students to assess, formulate, and analyse the problem, generating and consolidating the knowledge and solution together [4,12]. Regarding group interactions in small group learning, students share and broaden their learning through discussion, feedback, and clarification to achieve common goals [1,5,6]. Students also develop self and group values and gain peer support to articulate their thoughts and formulate their viewpoints to cultivate independent learning and problem-solving (PS) abilities and skills [9,13]. Moreover, this approach helps students to create and engage in a learning environment to enhance individual and group developments, such as self-directed learning, team interactions, and skill development [11,14,15].
Self-directed learning (SDL) is a learning process employed to allow learners to obtain self-mastery and ownership of learning to direct, regulate, and be accountable for their learning [16]. Moreover, SDL motivates students to understand their values, interests, weaknesses, and strengths [9,17] and encourages students to promote self-discipline with more accountability and learning motivation [18,19]. They can use their acquired knowledge and skills for better decision-making, enhancing competence and confidence [17,20]. With the advantages of the current advanced digital technology, student SDL attitudes can be promoted to foster personal and professional development [6,21,22].
Not limited to knowledge enhancement, evidence documented that core higher-intellectual skills, such as communication, critical thinking, leadership, problem-solving, management, and collaborative skills, can be developed through interactive small group learning [5,10]. In the last few decades, technology has rapidly improved in healthcare settings. Such advanced and sophisticated technology has promoted patient recovery and healthcare services on the one hand, and on the other, nurses acquire a great deal of accountability to make immediate and appropriate clinical decisions. PS ability is an essential skill that includes the process of identifying a problem, analysing its existence and impacts, and critically considering possible solutions to the problem. Inadequate PS ability can be the cause of students’ failure in performance [7,22,23,24,25]. Today, PS skill training is emphasised, but its assessment is usually neglected in the current nursing education.
A recent systematic review reported that interactive small group work with an appropriate teaching-learning strategy develops students’ competence in both SDL and PS abilities through independent study and group learning [26]. PBL is known as a favourable teaching-learning method. This learning method allows students to recall what they have learned, identify what they want to know, and discover the way they can learn to solve the problems [7,23,27]. PBL provides important benefits in knowledge enhancement and skill development during nursing education [7,9,28,29]. Online teaching has been adopted due to the coronavirus pandemic. This study developed a structured online PBL intervention to enhance SDL and PS abilities in undergraduate nursing students through interactive small group work. The results of this study provide knowledge about the use of the structured online PBL guidelines to enhance students’ SDL and PS abilities for personal and professional development through an interactive small group approach.

1.3. Aims

This aim of this study was to examine the effects of the structured PBL intervention via online on SDL and PS abilities in undergraduate nursing students through interactive small group work.

2. Materials and Methods

2.1. Design

A waitlist-control trial with a randomised design was adopted with a total of 299 students. They were divided into two groups and randomly assigned to take the 7-week clinical practicum in two different periods. The first group of students were assigned to the intervention group. They received the intervention first, and then they had 7 weeks of clinical practice after the completion of the group work. After seven weeks, the second group of students in the waitlist control group received the intervention after their 7 weeks of clinical practice.

2.2. Conceptual Framework

The input-process-output (IPO) model developed by Albion and Gibson [30] acted as the conceptual framework in this study. It consists of three dimensions: input, process, and output. The input is a requirement from the environment, including the facilitator and the online structured PBL guideline. The scenario development for PBL tasks consists of concepts, context, artefacts, storyline, and scenario. The context is related to the environmental elements and the problems are identified. The artefacts are the sub-problems that can be produced by a main problem and can be gathered and solved through a step-by-step solution (problem-solving process). The storyline seeks to understand and explain the problem and then describe the problem-solving progress and possible resolutions. The process includes operations and activities that mediate between the inputs and the outcomes. The outputs are the consequences of the process actions. Figure 1 illustrates the conceptual framework of the PBL approach.

2.3. Online Structured PBL Guideline

The online structured PBL guideline was designed for small group work based on the IPO conceptual framework and the work by Cadorin et al. [19] and Meo [31]. The guidelines were designed to enhance student SDL and PS abilities through briefing, peer support, free discussion, Socratic questioning, guided reflection, tutor facilitation, and group member and tutor feedback. The educator first formulated a PBL case scenario with an authentic problem. To allow students to learn little by little and step by step, the educator set learning objectives and guiding questions of the fragmented case scenario in each lesson. Supplementary Materials S1 shows an example of case scenario using PBL approach to guide students to think and achieve the respective learning objectives. They were encouraged to establish relevant learning objectives to broaden their learning through self-study and group discussions. Peer–peer interaction through group discussion provided a good platform for students to reflect, give feedback, and evaluate their learning. The educator played important facilitating roles to guide students to think, learn, and work independently and in a group through a variety of resources. The educator encouraged students to perform self-evaluation with adequate support and feedback for individual and collaborative learning. Table 1 illustrates the online structured PBL guidelines for small group learning in detail.

2.4. Setting and Participants

Participants who were undergraduate nursing students and had been involved in interactive small group work were included in this study. Eligible students were recruited from the bachelor’s in nursing programme at a professional training institution. They were randomly assigned to either the intervention or the waitlist control group by the computerised system. The priori power analysis was performed to reach the desired power of 0.80 and a Type I error of 0.05 with an effect size of 0.4 using G-Power. The sample size was calculated, and a minimum of 200 students (100 students in each group) were needed.

2.5. Tools for Outcome Measures

One set of questionnaires was used to collect demographic characteristics (age, gender, and average hours of small group work), the Self-Directed Learning Readiness Scale (SDLRS), and the Problem-Solving Inventory (PSI).
The SDLRS was developed by Fisher et al. [32] and used to assess student perception of skills and attitudes related to their self-directedness in learning. This tool consists of 40 items with a five-point Likert scale from 1 (strongly disagree) to 5 (strongly agree). The SDLRS includes three subscales: self-management (SM; 13 items), desire for learning (DL; 12 items), and self-control (SC; 15 items). The SM subscale reflects the characteristics of an individual being able to manage his or her learning. The DL subscale is associated with an individual’s desire for learning. The SC subscale is related to the features of self-control and being in control of one’s learning. The ranges of the overall, SM, DL and SC scores were from 40 to 200, 13 to 65, 12 to 60, and 15 to 75, respectively. Higher scores on the SDLRS indicate stronger SDL ability. Mean scores greater than 150 indicate a high level of SDL ability. The internal reliability values for Cronbach’s alpha for the overall SDLRS, SM, DL, and SC in this study were reported as 0.899, 0.872, 0.871, and 0.903, respectively, indicating very good reliability.
Heppner and Petersen [33] developed the PSI that consists of 32 items with a six-point Likert scale ranging from 1 (strongly agree) to 6 (strongly disagree). It is used to measure an individual’s perceptions regarding his or her PS abilities and PS style in daily life. The PSI includes three subscales: approach-avoidance style (AAS; 16 items; score range from 16 to 96), problem-solving confidence (PSC; 11 items; score ranging from 11 to 66), and personal control (PC; 5 items; score ranging from 5 to 30). The PSC effectively assesses self-perceived confidence, belief and self-assurance in solving problems. The AAS measures an individual’s tendency to approach or avoid problems. Finally, the PC assesses elements of self-control of emotions and behaviour. A higher AAS reflects more avoidance than approaching problems. A higher PSC indicates lower levels of PS confidence. Lastly, a higher PC signifies a more negative perception of personal control of problems. The total score of PSI ranges from 32 to 192. Lower scores on each subscale and the total PSI indicate higher functional PS abilities. The internal reliability values for Cronbach’s alpha of the overall PSI, AAS, PSC, and PC in this study were reported as 0.751, 0.799, 0.772, and 0.649, respectively, indicating good reliability.
Due to the smoothness of the study, English versions of these two instruments with the authors’ permission were adopted. To increase their applicability in this study, two experts in nurse education reviewed and verified them. A pilot study for tool validity was also conducted in 20 undergraduate students who were not included in this study. The results of the internal consistency, with Cronbach’s alphas, of the SLDR and PSI are reported in Table 2.

2.6. Data Collection

All eligible students were invited and asked to complete one set of questionnaires before the course started (T0). The first group of students were assigned to the intervention group and were required to form a group of five to seven members. Only one tutor (the principal investigator) acted as the facilitator to provide case information, supervision, guidance and support and monitor individual and group learning. The communications between students and the tutor included regular meetings via Blackboard Collaborate Ultra and other channels, such as Zoom meetings and WhatsApp.
In the first lesson, students received clear instructions with rules and regulations, and their roles and responsibilities were explained concerning working on a small group project. Each group would have an individual case scenario with learning objectives. In each lesson, students were expected to take on individual tasks for self-study and share their learning in the group. Before the next lesson, students were required to submit their learning records to the tutor. During the 8th week, all students in the intervention and waitlist control groups were required to complete the questionnaires (SDLRS and PSI) (T1) again. Students in the waitlist control group started the course and received the same instructions and intervention as the first group on the 8th week. All students were required to complete the questionnaires (SDLRS and PSI) during the 16th week and 28th week. Figure 2 shows the flow diagram of student recruitment and allocation.

2.7. Ethical Considerations

Ethics approval was obtained from the study professional training institution. All students were assured that they could withdraw from the study without any accountability at any time. All data related to personal information were kept confidential. Student numbers were recorded to link to the same students in the code list for data collection in the four study periods. The records with the student numbers were destroyed after all data entry was completed.

2.8. Data Analysis

Statistical analysis was performed using the IBM SPSS software (v.26) (SPSS Inc., Chicago, IL, USA). Differences in subjects’ characteristics (age, gender, and hours spent in small group work per week and levels of SDLRS and PSI in the four study periods: baseline, 8th, 16th and 28th weeks after intervention) between the intervention and control groups were examined using t-tests (continuous data) and the chi-square test (categorical data).
The intervention effect was evaluated by comparing the intervention and control groups. A repeated-measure analysis of variance (ANOVA) was performed to detect the effects of the outcome measures (SDLRS and PSI) with time (pre-intervention, post-intervention and follow up) as the within-subject variable and the main effects of time and group (intervention and control groups) as the between-subject variable. For all models, the normality of the regression residuals was examined. All statistical tests involved were two-sided, and a p-value < 0.05 was considered statistically significant.

3. Results

3.1. Participant Characteristics

A total of 199 students participated in the study, including 101 (50.8%) from the intervention group and 98 (49.2%) from the waitlist control group. There was no attrition. The mean age was 21.45 ± 0.86 years old (ranging from 20 to 25 years). No significant differences in age and gender were found, but the hours spent on small group work were different between the two groups. Table 3 illustrates the student characteristics in detail.

3.2. Analysis of Outcomes

3.2.1. SDLRS and PSI between Two Groups in Four Stages

Table 4 illustrates the descriptive analytical results of the four study periods. The results of this study demonstrate that students in the intervention group performed better on the SDLRS and PSI from the baseline to the 8th week. The scores of all SDLRS subscales were maintained, and the performance on the PSI subscales was continuously improved from the 8th to the 16th week and even in the 28th week. Students in the waitlist control group exhibited poorer performance on the SDLRS and PSI at the baseline and 8th week. However, the scores for all SDLRS subscales increased, and the performance on the PSI subscales improved from the 8th to the 16th week and even in the 28th week.
The independent sample t-test results to compare the intervention and waitlist control groups indicated that all subscales of the SDLRS and PSI were statistically significant at the 8th week. At the 16th week, only the DL and PC subscales exhibited statistically significantly negative differences between the intervention and waitlist control groups. At the 28th week, no significant difference existed between the two groups. Table 5 lists the differences between the SDLRS and PSI of the two groups.

3.2.2. Study Period Changes between Intervention and Waitlist Control Groups

The repeated-measure ANOVA results demonstrate that significant changes occurred in all subscales of the SDLRS and PSI from the baseline to the 8th week (p < 0.001). Significant results were also noted in most SDLRS and PSI subscales from the 8th to the 16th week. The sustained effects on SDLRS and PSI were noted, and no significant difference was found from the 16th to the 28th week. Table 4 also illustrates the changes in the subscales of the SDLRS and PSI.

4. Discussion

The online structured PBL intervention acting as guidelines appeared to be an effective approach to enhance SDL and PS abilities through interactive small group work in professional education. The conceptual framework was useful in guiding the educator more effectively to organise the tasks and materials for the online PBL [34]. The results of this study suggest that the guidelines made significant improvements in SDL and PS abilities at the 8th and 16th weeks in both the intervention and waitlist control groups. The sustained effects were also observed in the 28th week. The improvement of student SDL and PS abilities was determined by the well-planned intervention with clear guidelines for student learning, the involvement of students and educators, and regular group discussions via various channels. The online structured guidelines with PBL intervention directed and motivated students’ learning initiatives and their self-mastery of learning to deal with problems. The results of this study indicate that the students’ desire for learning and personal control were significantly improved after the intervention. The results of this study also demonstrate the effects of the PBL intervention on SDL and PS abilities, even via online methods. These guidelines provided a comprehensive plan to allow educators to prepare their teaching well to lead and monitor student learning, contributing to students’ knowledge enhancement and skill development.
SDL is an essential component of initiative learning and lifelong learning to motivate students to acquire new information and critically evaluate and apply the information in their studies and future careers [19,22,25,34]. The PS ability enables students to make decisions through a series of critical thinking, logical assessment, and analytical processes [3,29]. As SDL and PS abilities are closely related [23,27,29], the nature of small group work with independent learning and group learning greatly promotes student SDL and PS abilities to achieve learning objectives. This study revealed the short- and long-term benefits of SDL and PS skill enhancement are already reported in this study. The development of two skills is a challenging learning process that requires the involvement of both educators and students. Students with better SDL ability can be highly motivated [23,29,35]. Students enhance their knowledge in a specific area and learn more comprehensively through sharing, discussions, and criticisms, while small group learning embeds self-motivated learning and interactive learning [14,25,36]. Students with SDL and PS skills are more likely to apply knowledge appropriately in different situations to enhance their competence, particularly when dealing with new or complicated clinical situations [7,20,21]. The improvements in student SDL and PS abilities are crucial for increasing their confidence, independence, sense of accountability, and assertiveness for their professional development [15,37]. Educators should evaluate and determine the appropriateness and development of SDL and PS.
PBL is an effective teaching-learning pedagogy for knowledge acquisition and multi-skill development, such as PS, SDL, critical thinking, and clinical reasoning [7,18,23,28,29]. Thus, PBL is crucial to encouraging students to become SDL learners and work more collaboratively [20,23,29]. This study uses online PBL intervention through small group work, confirming improved SDL and PS abilities. A possible explanation for this result is using case segmentation (five parts) to direct students to achieve specific learning outcomes based on the available case information. Students received their focuses of learning for each lesson and searched for new information to achieve their learning objectives. Student interests grew with the learning motivation, promoting better SDL and PS attitudes and behaviours to achieve optimal outcomes [22,26,37]. More importantly, this study also revealed significantly sustained positive effects of the intervention on SDL and PS abilities, which continuously occurred in the two groups after intervention at the 16th and 28th weeks. This study supports the assertion that the utility of the PBL intervention can enhance SDL and PS abilities, which foster lifelong student learning and professional competence. The online PBL intervention through small group work anticipates more focused learning areas, higher learning initiative, and better learning proficiency to achieve more efficient learning outcomes. This learning format facilitates student learning [6,18,36] and provides a platform for students to interact with others so as to develop personal and professional skills through interactive collaboration [6,23,29,38].
Small group work represents a shift from educator-directed teaching to student-directed learning [1,2,3]. Educators play the facilitating role to support and guide students to learn more deeply and effectively through their individual study and peer–peer interaction in a group. As small group teaching and learning pedagogy is commonly adopted in nursing education [24], it facilitates the teaching team and learning environment when students value the interactive activities and learning features that increase their learning satisfaction and knowledge retention [39]. The guidelines adopted in our study include good planning of active learning strategies to accommodate a variety of learning methods using online and PBL modalities. Online learning has been reported for over a decade in previous studies, including in nursing. Maintaining effective interaction among students and with the tutor can be a challenge in small group learning online. This specific teaching-learning method has been examined and has reported significant enhancement in a variety of aspects worldwide, such as self-regulatory abilities [40,41,42]. Online education moves professional education forward to promote student learning and staff development in more effective and quality teaching and learning modalities in different countries [43,44,45]. As today’s technology becomes more advanced, online teaching with student interactions which facilitate student learning with positive outcomes in knowledge acquisition and skill development has become a trend in nursing education [44,45,46].
Regarding the significant positive effects of the PBL intervention during the study periods after the intervention, the educators played an essential role in facilitating student learning following the guidelines. The guidelines employed in this study can give clear directions to the educator(s) and students to enable more interactive teaching and learning engagement. The educators may understand more about individual student learning needs, PS abilities, and group interactive learning and allow autonomous learning to encourage students to engage in self-directed learning activities [19,23]. As SDL is connected to the student’s motivation to learn, the ability to search for resources, and the ability to implement strategies to achieve learning outcomes, educators embrace their vital role in effectively facilitating students to understand their accountability and self-evaluate their learning [16,19]. The educator can provide timely feedback and guidance for students to learn more effectively, progressively improving student SDL and PS abilities [12]. The outcomes of this study encourage educators and programme developers to design structured interventions for quality learning using interactive small groups in nursing education curricula [44,45]. Support is necessary for educators to implement strategies for more positive online learning experiences of students through interactive small groups.

Limitations

This study is limited to one cohort of Year 4 nursing students from one educational institution. Therefore, the results of this study are difficult to generalise to students in other years or disciplines. Due to the coronavirus pandemic, virtual teaching was adopted, and the online survey was the most significant challenge. Multiple mass and individual emails or other contact methods were used to remind students to ensure responses during the study periods. However, the accuracy of the data may not be guaranteed. As with other studies, although this study reported effective results in improving students’ SDL and PS abilities by small group learning online, knowledge about its effects on SDL and PS in students who resist is inadequate. Further studies to understand the learning of this specific group may be needed.

5. Conclusions

Interactive learning through small group activities is favourable for personal and professional development in professional training. This study provides preliminary support for a structured intervention with PBL via an online approach to improve student SDL and PS abilities for both short and long periods. Therefore, the structured guidelines to improve students’ SDL and PS abilities through the individual and group learning format in interactive group work are promising.

Supplementary Materials

The following supporting information can be downloaded at https://www.mdpi.com/article/10.3390/ijerph19020720/s1. Supplementary Materials S1: An example of case scenario using PBL approach.

Author Contributions

Conceptualization, F.M.F.W. and C.W.Y.K.; methodology, F.M.F.W.; software, F.M.F.W.; validation, F.M.F.W. and C.W.Y.K.; formal analysis, F.M.F.W.; investigation, F.M.F.W. and C.W.Y.K.; resources, F.M.F.W.; data curation, F.M.F.W. and C.W.Y.K.; writing—original draft preparation, F.M.F.W.; writing—review and editing, F.M.F.W.; visualization, F.M.F.W.; supervision, F.M.F.W. and C.W.Y.K.; project administration, F.M.F.W. and C.W.Y.K..; funding acquisition, F.M.F.W. All authors have read and agreed to the published version of the manuscript.

Funding

This study was supported by College Research Grant of Tung Wah College (CRG2018/03).

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the College Research Ethics Committee of Tung Wah College (protocol code REC2019044 and approved on 9 October 2019).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy reason.

Acknowledgments

We would like to specially thank Lawrence Lam and David Wong, Tung Wah College, for their valuable advice on data analysis for this study. We would also like to acknowledge Heppner and Fisher for their kind generosity of their permission to use their instruments, Problem-Solving Inventory (PSI) and Self-Directed Learning Readiness Scale (SDLRS), respectively.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Mennenga, H.; Smyer, T. A model for easily incorporating team-based learning into nursing education. Int. J. Nurs. Educ. Scholarsh. 2010, 7, 4. [Google Scholar] [CrossRef]
  2. Mills, D.; Alexander, P. Small Group Teaching: A Toolkit for Learning; Higher Education Academy: York, UK, 2013; Available online: http://books.koondis.com/sgl-lit/Small_group_teaching_1.pdf (accessed on 20 March 2021).
  3. Sanasuttipun, W.; Tungjairob, V.; Musiksukont, S.; Lerthamatewe, W.; Chanwatana, B. Effectiveness of small group discussion on critical thinking, self-directed learning, and learning satisfaction of Thai nursing students. J. Nurs. Sci. 2009, 27, 8–16. [Google Scholar]
  4. Shaw, J.; Mitchell, C.; Del Fabbro, L. Group work: Facilitating the learning of international and domestic undergraduate nursing students. Educ. Health 2015, 28, 124–129. [Google Scholar] [CrossRef] [Green Version]
  5. Edmunds, S.; Brown, G. Effective small group learning: AMEE guide No. 48. Med. Teach. 2010, 32, 715–726. [Google Scholar] [CrossRef]
  6. Wong, F.M.F. A phenomenological research study: Perspectives of student learning through small group work between undergraduate nursing students and educators. Nurse Educ. Today 2018, 68, 153–158. [Google Scholar] [CrossRef] [PubMed]
  7. Bruce, J.; Lack, M.; Bomvana, N.M.; Qamate-Mtshali, N. Problem-based Learning: Nursing students’ attitude, self-reported competence, tutorial performance and self-directed learning readiness. J. Nurs. Educ. Pract. 2018, 8, 11. [Google Scholar] [CrossRef] [Green Version]
  8. Gijbels, D.; Dochy, F.; van den Bossche, P.; Segers, M. Effects of problem-based learning: A meta-anlaysis from the angle of assessment. Rev. Educ. Res. 2005, 75, 27–61. [Google Scholar] [CrossRef] [Green Version]
  9. Zhang, X.; Meng, L.; Liu, H.; Luo, R.; Zhang, C.; Zhang, P.; Liu, Y. Role of academic self-efficacy in the relationship between self-directed learning readiness and problem-solving ability among nursing students. Front. Nurs. 2018, 5, 75–81. [Google Scholar] [CrossRef] [Green Version]
  10. Chiriac, E. Group work as an incentive for learning—students’ experiences of group work. Front. Psychol. 2014, 5, 558. [Google Scholar] [CrossRef] [Green Version]
  11. Noohi, E.; Abaszadeh, A.; Maddah, S.S. University engagement and collaborative learning in nursing students of Kerman University of Medical Sciences. Iran. J. Nurs. Midwifery Res. 2013, 18, 505–510. [Google Scholar]
  12. Samson, P.L. Fostering student engagement: Creative problem-solving in small group facilitations. Collect. Essays Learn. Teach. 2015, 8, 153–164. [Google Scholar] [CrossRef]
  13. Malta, S.; Dimeo, S.B.; Carey, P.D. Self-direction in learning: Does it change over time? J. Allied Health 2010, 39, e37–e41. [Google Scholar] [PubMed]
  14. Wong, M.F.F. Development of higher-level intellectual skills through interactive group work: Perspectives between students and educators. Med. Clin. Res. 2020, 5, 27. [Google Scholar]
  15. Wong, F.M.F. A cross-sectional study: Collaborative learning approach enhances learning attitudes of undergraduate nursing students. GSTF J. Nurs. Health Care (JNHC) 2018, 5. [Google Scholar] [CrossRef]
  16. Cheng, S.F.; Kuo, C.L.; Lin, K.C.; Lee-Hsieh, J. Development and preliminary testing of a self-rating instrument to measure self-directed learning ability of nursing students. Int. J. Nurs. Stud. 2010, 47, 1152–1158. [Google Scholar] [CrossRef] [PubMed]
  17. Malekian, M.; Ghiyasvandian, S.; Cheraghi, M.A.; Hassanzadeh, A. Iranian clinical nurses’ readiness for self-directed learning. Glob. J. Health Sci. 2015, 8, 157–164. [Google Scholar] [CrossRef] [Green Version]
  18. Alharbi, H.A. Readiness for self-directed learning: How bridging and traditional nursing students differs. Nurse Educ. Today 2018, 61, 231–234. [Google Scholar] [CrossRef]
  19. Cadorin, L.; Dante, A.; Bulfone, T.; Viera, G.; Palese, A. Enhancing self-directed learning among Italian nursing students: A pre-and post-intervention study. Nurse Educ. Today 2015, 35, 746–753. [Google Scholar] [CrossRef] [PubMed]
  20. Applin, H.; Williams, B.; Day, R.; Buro, K. A comparison of competencies between problem-based learning and non-problem-based graduate nurses. Nurse Educ. Today 2011, 31, 129–134. [Google Scholar] [CrossRef]
  21. Cadorin, L.; Suter, N.; Dante, A.; Williamson, S.N.; Devetti, A.; Palese, A. Self-directed learning competence assessment within different healthcare professionals and amongst students in Italy. Nurse Educ. Pract. 2012, 12, 153–158. [Google Scholar] [CrossRef]
  22. Tekkol, İ.A.; Demirel, M. An investigation of self-directed learning skills of undergraduate students. Front. Psychol. 2018, 9, 23–24. [Google Scholar] [CrossRef]
  23. Roh, Y.S.; Kim, S.S. Integrating problem-based learning and simulation: Effects on student motivation and life skills. CIN: Comput. Inform. Nurs. 2015, 33, 278–284. [Google Scholar] [CrossRef]
  24. Cheng, C.Y.; Liou, S.R.; Hsu, T.H.; Pan, M.Y.; Liu, H.C.; Chang, C.H. Preparing nursing students to be competent for future professional practice: Applying the team-based learning-teaching strategy. J. Prof. Nurs. 2014, 30, 347–356. [Google Scholar] [CrossRef] [PubMed]
  25. Khiat, H. Academic performance and the practice of self-directed learning: The adult student perspective. J. Furth. High. Educ. 2017, 41, 44–59. [Google Scholar] [CrossRef]
  26. Wong, F.M.F.; Tang, A.C.Y.; Cheng, W.L.S. Factors associated with self-directed learning among undergraduate nursing students: A systematic review. Nurse Educ. Today 2021, 104, 104998. [Google Scholar] [CrossRef] [PubMed]
  27. Luo, R.Z.; Zhang, X.H.; Zhang, C.M.; Liu, Y.H. Impact of self-directed learning readiness and learning attitude on problem-solving ability among Chinese undergraduate nursing students. Front. Nurs. 2019, 6, 143–150. [Google Scholar] [CrossRef] [Green Version]
  28. Avdal, E.Ü. The effect of self-directed learning abilities of student nurses on success in Turkey. Nurse Educ. Today 2013, 33, 838–841. [Google Scholar] [CrossRef]
  29. Choi, E.; Lindquist, R.; Song, Y. Effects of problem-based learning vs. traditional lecture on Korean nursing students’ critical thinking, problem-solving, and self-directed learning. Nurse Educ. Today 2014, 34, 52–56. [Google Scholar] [CrossRef] [PubMed]
  30. Albion, P.; Gibson, I.W. Interactive Multimedia and Problem-Based Learning: Challenges for Instructional Design; US Department of Education: Educational Resources Information Center (ERIC): Washington, DC, USA, 1998. [Google Scholar]
  31. Meo, S.A. Basic steps in establishing effective small group teaching sessions in medical schools. Pak. J. Med. Sci. 2013, 29, 1071–1076. [Google Scholar] [CrossRef] [PubMed]
  32. Fisher, M.; King, J.; Tague, G. Development of a self-directed learning readiness scale for nursing education. Nurse Educ. Today 2001, 21, 516–525. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  33. Heppner, P.P.; Petersen, C.H. The development and implications of a personal problem solving inventory. J. Couns. Psychol. 1982, 29, 66–75. [Google Scholar] [CrossRef]
  34. Alshaye, I.; Tasir, Z.; Jumaat, N.F. The Conceptual Framework of Online Problem-Based Learning towards Problem-Solving Ability and Programming Skills. In Proceedings of the IEEE Conference on e-Learning, e-Management, and eService (IC3e) 2019, Pulau Pinang, Malaysia, 19–21 November 2019. [Google Scholar]
  35. Soliman, M.; Al-Shaikh, G. Readiness for self-directed learning among first year Saudi medical students: A descriptive study. Pak. J. Med. Sci. 2015, 31, 799–802. [Google Scholar] [CrossRef] [PubMed]
  36. Rezaee, R.; Mosalanejad, L. The effects of case-based team learning on students’ learning, self-regulation and self-direction. Glob. J. Health Sci. 2015, 7, 296–305. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  37. El-Gilany, A.H.; Abusaad, F.E.S. Self-directed learning readiness and learning styles among Saudi undergraduate nursing students. Nurse Educ. Today 2013, 33, 1040–1044. [Google Scholar] [CrossRef]
  38. İlçin, N.; Tomruk, M.; Yeşilyaprak, S.S.; Karadibak, D.; Savci, S. The relationship between learning styles and academic performance in TURKISH physiotherapy students. BMC Med. Educ. 2018, 18, 291. [Google Scholar] [CrossRef]
  39. Leslie, A.; Ewens, B.; Maslin-Prothero, S. Enhancing the Online Learning Experience Using Virtual Interactive Classrooms. Aust. J. Adv. Nurs. 2015, 32, 22–31. [Google Scholar]
  40. Chen, S.W.; Stocker, J.; Wang, R.H.; Chung, Y.C.; Chen, M.F. Evaluation of self-regulatory online learning in a blended course for post-registration nursing students in Taiwan. Nurse Educ. Today 2009, 29, 704–709. [Google Scholar] [CrossRef]
  41. McCutcheon, K.; O’Halloran, P.; Lohan, M. Online learning versus blended learning of clinical supervisee skills with pre-registration nursing students: A randomized controlled trial. Int. J. Nurs. Stud. 2008, 82, 30–39. [Google Scholar] [CrossRef] [PubMed]
  42. Whittaker, A. Effects of Team-Based Learning on Self-Regulated Online Learning. Int. J. Nurs. Educ. Scholarsh. 2015, 2, 45–54. [Google Scholar] [CrossRef]
  43. Phillips, J.M. Strategies for active learning in online continuing education. J. Contin. Educ. Nurs. 2005, 36, 77–83. [Google Scholar] [CrossRef] [Green Version]
  44. Perry, B.; Edwards, M. Interactive teaching technologies that facilitate the development of online learning communities in nursing and health studies graduate courses. Teach. Educ. Q. 2010, 37, 147–172. [Google Scholar]
  45. Johnson, A.E. A nursing faculty’s transition to teaching online. Nurs. Educ. Persp. 2008, 29, 17–22. [Google Scholar]
  46. Hampton, D.; El-Mallakh, P. Opinions of Online Nursing Students Related to Working in Groups. J. Nurs. Educ. 2017, 56, 611–616. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Conceptual framework of input-process-output (IPO) (Albion and Gibson 1998).
Figure 1. Conceptual framework of input-process-output (IPO) (Albion and Gibson 1998).
Ijerph 19 00720 g001
Figure 2. Flow diagram of student recruitment and allocation.
Figure 2. Flow diagram of student recruitment and allocation.
Ijerph 19 00720 g002
Table 1. A structured problem-based learning guideline for small group learning.
Table 1. A structured problem-based learning guideline for small group learning.
Aims
This Guideline Developed Based on Cadorin et al. (2015) and Meo (2013) Provides Clear Instructions for the Tutor and Students to Participate in Small Group Learning.
Rules and regulations for studentsInstructionExpected outcomes
  • attain 80% attendance.
  • actively participate in group activities, including group formation, objectives setting, sharing of individual work, discussion, and giving feedback.
  • show and respond with respect and acceptance to others.
  • be accountable for individual tasks and share learning with groupmates.
  • approach tutor anytime when problems encountered by the individual and/or group.
  • submit work on time, including individual tasks to share in the next lesson and the learning reports regarding individual learning and group learning.
  • bear consequences, such as grade deduction, if violating any rules and regulations.
(a)
The tutor has to upload students’ rules and regulations to Blackboard before or on the 1st week of the semester.
(b)
The tutor has to introduce each point of the rules and regulations to students in the 1st lesson.
(c)
Students have to follow all rules and regulations.
Students are able to
understand their own involvement and accountability of learning.
develop higher intellectual skills, including communication and collaborative skills.
enhance their learning with the support from tutor.
Roles and responsibilities of students
  • show a positive attitude and be friendly all the time.
  • show respect and acceptance and be considerate to one another.
  • use positive body language and eye contact.
  • cooperate with other groupmates.
  • be active in setting individual and group learning objectives; individual study and group discussions via various communication channels; and group sharing and discussions to enhance group learning capacity in each meeting.
  • record and submit an individual learning report before each lesson.
  • take turns when giving ideas.
  • approach your tutor when you have problems.
  • The tutor has to upload students’ roles and responsibilities to Blackboard before or on the 1st week of the semester.
  • The tutor has to introduce each point of the roles and responsibilities to students in the 1st lesson.
  • Students have to follow all roles and responsibilities.
Students are able to
have a clear direction and picture what and how they are involved.
understand their own accountabilities.
develop higher intellectual skills, such as interpersonal communication, problem-solving, and collaborative skills.
Roles and responsibilities of the tutorInstructionExpected outcomes
  • be a facilitator to guide students to meet the learning outcomes.
  • encourage groups to select a leader and assign tasks.
  • monitor and facilitate group discussion.
  • observe and evaluate the involvement of each student in the group.
  • allow students to ask questions, correct misunderstandings and give appropriate advice.
  • reply in simple wording and give easy and appropriate examples.
  • allow groups to showcase their work.
  • evaluate and give feedback on students’ individual and group learning.
  • give groups autonomy and time to prepare high-quality work.
  • always appreciate good attitudes and behaviours as well as goal achievement.
  • conclude the session with concise take-home message.
  • The course coordinator should introduce the roles and responsibilities to all tutors involved in small group teaching.
  • The course coordinator should explain the logistics and teaching to the involved tutors.
  • All tutors have to meet their students via an online platform at least once a week and provide guidance and support accordingly.
  • All tutors should facilitate and monitor students’ individual learning and in a group.
  • Tutors should support one another if needed.
All tutors
  • have clear direction and understanding to guide their students.
  • may provide their teaching consistently.
  • may enhance their competence in teaching small groups.
Students are able to
4.
learn more deeply and effectively.
5.
have more support and guidance.
6.
enhance their learning both individually and in a group.
Roles and responsibilities of a group
  • always maintain group dynamics and collaboration in an acceptable manner.
  • set achievable goals and objectives.
  • hold regular meetings and discussions through various channels for group communication.
  • share workload according to individual groupmates’ talents and/or interests.
  • explore the potential of each member to maximise that of the group.
  • make appropriate decisions as a group.
  • strengthen group cohesiveness through effective communication, goal achievement, mutual empowerment, higher job satisfaction, etc.
(a)
The tutor has to upload roles and responsibilities of a group to Blackboard before or on the 1st week of the semester.
(b)
The tutor has to introduce each point of the roles and responsibilities to student groups in the 1st lesson.
(c)
Student groups have to follow all roles and responsibilities.
All student groups are able to
have a clear direction and picture of what and how they should be involved.
understand their accountabilities.
  • enhance the collaborative dynamics of the group.
  • promote constructive and effective communication.
  • complete their group project more smoothly and efficiently.
Table 2. The results of the internal consistence of the SLDR and PSI.
Table 2. The results of the internal consistence of the SLDR and PSI.
Internal Consistency
(Cronbach’s alphas)
SMSCDLOverall SDLAASPSCPCOverall PS
0.890.910.820.950.900.850.870.95
Table 3. Student characteristics (n = 199).
Table 3. Student characteristics (n = 199).
Overall
(n = 199)
Waitlist-Control
(n = 98)
Intervention
(n = 101)
p
n%n%n%
Gender 0.275
Male4321.51818.42524.8
Female15678.08081.67675.2
Age 0.990
Mean (SD)21.45 (SD 0.86)21.44 (SD 0.80)21.47 (SD 0.92)
20–2113567.56667.36968.3
22–235527.52828.62726.7
24–2594.544.155.0
Hours spent on small group work (hour) 0.015 *
1115.522.098.9
23618.11414.32221.8
37437.23636.73837.6
44020.12626.51413.9
≥53819.12020.41817.8
* p < 0.05.
Table 4. Self-directed learning and problem-solving abilities between the two groups at baseline, pre- and post-intervention through small group work.
Table 4. Self-directed learning and problem-solving abilities between the two groups at baseline, pre- and post-intervention through small group work.
OverallControlIntervention
MeanSDMeanSDMeanSD
Self-directed learning ability
SM
Baseline45.627.1544.747.2546.477.00
8-week46.887.6244.137.5749.556.76
16-week51.306.6052.007.5150.626.34
28-week51.957.0952.217.8051.706.35
SC
Baseline58.277.9357.847.8258.688.07
8-week59.528.2256.217.5562.737.58
16-week63.317.2863.997.8162.646.70
28-week64.317.3664.138.4264.486.19
DL
Baseline47.568.8048.0410.9647.106.03
8-week48.446.2545.956.0350.865.50
16-week50.015.7851.866.0450.185.42
28-week51.505.6151.386.0051.615.23
Overall SDLR
Baseline151.4519.89150.6220.73152.2519.10
8-week154.8520.29146.3019.06163.1517.94
16-week165.6118.58167.8519.99163.4516.91
28-week167.7618.61167.7221.12167.7915.92
Problem-solving ability
AAS
Baseline39.646.4539.326.2039.946.69
8-week39.446.8942.285.8136.696.77
16-week35.926.5736.635.8935.247.13
28-week33.567.9633.818.6033.337.32
PSC
Baseline25.635.1025.605.4125.654.80
8-week25.255.4527.315.2623.254.89
16-week22.484.7922.684.9122.294.68
28-week21.195.7521.376.2921.015.20
PC
Baseline12.672.8412.442.6712.903.00
8-week12.803.3013.392.9612.233.53
16-week12.283.3512.793.3111.783.32
28-week11.344.0111.334.0711.363.98
Overall PS
Baseline77.9412.3077.3711.9578.5012.67
8-week77.4913.2882.9711.5072.1712.77
16-week70.6812.2772.1011.3569.3113.02
28-week66.0915.5066.5016.7165.6914.30
SM: self-management, DL: desire for learning, SC: self-control, SDL: self-directed learning, AAS: approach-avoidance style, PSC: problem-solving confidence, PC: personal control, PS: problem-solving, SDLR: self-directed learning readiness.
Table 5. Changes of self-directed learning and problem-solving abilities through small group work between two groups at four study periods (n = 199).
Table 5. Changes of self-directed learning and problem-solving abilities through small group work between two groups at four study periods (n = 199).
Independent Samples t-TestRepeated Measures
Periodstp95% CIPeriodsMauchly’s TestTests of Within-Subject Effects
SM SM
Baseline1.700.090−0.27 to 3.71Baseline to 8th week F(1,197) = 15.87, p < 0.001 ***
8th week5.35<0.001 ***3.42 to 7.428th week to 16th week F(1,197) = 67.65, p < 0.001 ***
16th week−1.400.164−3.32 to 0.5716th week to 28th week F(1,197) = 1.21, p = 0.273
28th week−0.510.165−2.50 to 1.484 periodsꭓ2 (5) = 22.54,
p < 0.001 ***
F(3,591) = 22.14, p < 0.001 ***
SC SC
Baseline0.750.453−1.38 to 3.07Baseline to 8th week F(1,197) = 27.53, p < 0.001 ***
8th week6.08<0.001 ***4.40 to 8.638th week to 16th week F(1,197) = 70.89, p < 0.001 ***
16th week−1.300.194−3.38 to 0.6916th week to 28th week F(1,197) = 3.51, p = 0.063
28th week0.330.745−1.73 to 2.424 periodsꭓ2 (5) = 36.94,
p < 0.001 ***
F(3,591) = 20.45, p < 0.001 ***
DL DL
Baseline−0.750.456−3.40 to 1.52Baseline to 8th week F(1,197) = 20.09, p < 0.001 ***
8th week6.00<0.001 ***3.30 to 6.528th week to 16th week F(1,197) = 99.65, p < 0.001 ***
16th week−2.060.041 *−3.28 to −0.0816th week to 28th week F(1,197) = 9.08, p = 0.003 *
28th week0.300.768−1.34 to 1.814 periodsꭓ2 (5) = 192.40,
p < 0.001 ***
F(3,591) = 15.69, p < 0.001 ***
Overall SDL Overall SDL
Baseline0.650.519−3.95 to 7.20Baseline to 8th week F(1,197) = 31.93, p < 0.001 ***
8th week−6.42<0.001 ***11.68 to 22.038th week to 16th week F(1,197) = 100.49, p < 0.001 ***
16th week−1.620.108−9.57 to 0.7716th week to 28th week F(1,197) = 4.87, p = 0.028 *
28th week0.030.980−5.15 to 5.29 4 periodsꭓ2 (5) = 53.03,
p < 0.001 ***
F(3,591) = 26.01, p < 0.001 ***
AAS AAS
Baseline0.670.503−1.19 to 2.42Baseline to 8th week F(1,197) = 35.96, p < 0.001 ***
8th week−6.25<0.001 ***−7.34 to −3.828th week to 16th week F(1,197) = 24.31, p < 0.001 ***
16th week−1.510.134−3.23 to 0.4416th week to 28th week F(1,197) = 0.69, p = 0.406
28th week−0.420.673−2.72 to 1.764 periodsꭓ2 (5) = 41.26,
p < 0.001 ***
F(3,591) = 12.01, p < 0.001 ***
PSC PSC
Baseline0.710.944−1.38 to 1.48Baseline to 8th week F(1,197) = 26.15, p < 0.001 ***
8th week−5.64<0.001 ***−5.48 to −2.648th week to 16th week F(1,197) = 33.93, p < 0.001 ***
16th week−0.580.561−1.74 to 0.9416th week to 28th week F(1,197) = 0.002, p = 0.960
28th week−0.440.663−1.97 to 1.264 periodsꭓ2 (5) = 29.16,
p < 0.001 ***
F(3,591) = 11.76, p < 0.001 ***
PC PC
Baseline1.150.252−0.33 to 1.26Baseline to 8th week F(1,197) = 12.30, p = 0.001 ***
8th week−2.510.013 *−2.07 to −0.258th week to 16th week F(1,197) = 0.136, p = 0.713
16th week−2.140.034 *−1.93 to −0.0816th week to 28th week F(1,197) = 4.21, p = 0.042 *
28th week0.050.958−1.10 to 1.164 periodsꭓ2 (5) = 34.01,
p < 0.001 ***
F(3,591) = 4.76, p = 0.003 *
Overall PS Overall PS
Baseline−0.650.519−2.32 to 4.57Baseline to 8th week F(1,197) = 37.84, p < 0.001 ***
8th week6.28<0.001 ***−14.20 to −7.408th week to 16th week F(1,197) = 26.34, p < 0.001 ***
16th week1.620.108−6.21 to 0.6216th week to 28th week F(1,197) = 0.89, p = 0.346
28th week−0.370.715−2.16 to 3.554 periodsꭓ2 (5) = 45.11,
p < 0.001 ***
F(3,591) = 12.73, p < 0.001 ***
SM: self-management, DL: desire for learning, SC: self-control, SDL: self-directed learning, AAS: approach-avoidance style, PSC: problem-solving confidence, PC: personal control, PS: problem-solving, SDLR: self-directed learning readiness. * p < 0.05; *** p < 0.001.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Wong, F.M.F.; Kan, C.W.Y. Online Problem-Based Learning Intervention on Self-Directed Learning and Problem-Solving through Group Work: A Waitlist Controlled Trial. Int. J. Environ. Res. Public Health 2022, 19, 720. https://doi.org/10.3390/ijerph19020720

AMA Style

Wong FMF, Kan CWY. Online Problem-Based Learning Intervention on Self-Directed Learning and Problem-Solving through Group Work: A Waitlist Controlled Trial. International Journal of Environmental Research and Public Health. 2022; 19(2):720. https://doi.org/10.3390/ijerph19020720

Chicago/Turabian Style

Wong, Florence M. F., and Crystal W. Y. Kan. 2022. "Online Problem-Based Learning Intervention on Self-Directed Learning and Problem-Solving through Group Work: A Waitlist Controlled Trial" International Journal of Environmental Research and Public Health 19, no. 2: 720. https://doi.org/10.3390/ijerph19020720

APA Style

Wong, F. M. F., & Kan, C. W. Y. (2022). Online Problem-Based Learning Intervention on Self-Directed Learning and Problem-Solving through Group Work: A Waitlist Controlled Trial. International Journal of Environmental Research and Public Health, 19(2), 720. https://doi.org/10.3390/ijerph19020720

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop