**An Interprofessional E-Learning Resource to Prepare Students for Clinical Practice in the Operating Room—A Mixed Method Study from the Students' Perspective**

**Ann-Mari Fagerdahl 1,2,\*, Eva Torbjörnsson 1,3 and Anders Sondén 1,3**


**Abstract:** The operating room is a challenging learning environment for many students. Preparedness for practice is important as perceived stress and the fear of making mistakes are known to hamper learning. The aim was to evaluate students' perspectives of an e-learning resource for achieving preparedness. A mixed methods design was used. Students (*n* = 52) from three educational nursing and medical programs were included. A questionnaire was used to explore demographics, student use of the e-learning resource, and how the learning activities had helped them prepare for their clinical placement. Five focus group interviews were conducted as a complement. Most students (79%) stated that the resource prepared them for their clinical placement and helped them to feel more relaxed when attending to the operating room. In total, 93% of the students recommended other students to use the e-learning resource prior to a clinical placement in the operating room. Activities containing films focusing on practical procedures were rated as the most useful. We conclude that an e-learning resource seems to increase students' perceived preparedness for their clinical practice in the operating room. The development of e-learning resources has its challenges, and we recommend student involvement to evaluate the content.

**Keywords:** clinical learning environment; e-learning; operating room; student preparedness

#### **1. Introduction**

The operating room (OR) environment is challenging for students in relation to achieving their learning objectives. Feelings of anxiety, humiliation, and other emotional obstacles for effective learning have been described by both medical and nursing students [1,2]. Some of these emotional barriers can be reduced if the students are well-prepared before their clinical practice [3,4]. Preparedness can be divided into a general part and a specific part. The general part should consist of information about the OR setting, etiquette, and the professional roles of the staff, in combination with workshops on practical skills. The specific part is the information needed on day-to-day basis, i.e., which supervisor the OR student should follow [1,2].

Methods for delivering general introductory sessions to students have been described by several authors, but there is weak evidence as to which arrangement is the most effective [2,5]. It has been concluded, however, that the introductory sessions should have an interprofessional perspective, as interprofessional teamwork is essential for creating a safe surgical environment for patients [6]. However, interprofessional learning (IPL) activities pose logistical and scheduling challenges [7]. One way to overcome these timetabling and geographic barriers is e-learning [8]. Another advantage of e-learning is that it is wellsuited for learning practical skills within the perioperative setting, due to the possibility to incorporate multimedia [8,9].

**Citation:** Fagerdahl, A.-M.; Torbjörnsson, E.; Sondén, A. An Interprofessional E-Learning Resource to Prepare Students for Clinical Practice in the Operating Room—A Mixed Method Study from the Students' Perspective. *Healthcare* **2021**, *9*, 1028. https://doi.org/ 10.3390/healthcare9081028

Academic Editors: Luís Proença, José João Mendes, João Botelho and Vanessa Machado

Received: 16 July 2021 Accepted: 4 August 2021 Published: 11 August 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

For many years, the medical students and the OR nurses in our OR department at Södersjukhuset, Karolinska Institutet have attended a pre-theatre workshop on surgical hand preparation and sterile gloving technique before entering the OR. The workshop contains a lecture, followed by practical training. The general nurses and anesthetic nurses have only a 15 min lecture about guidelines for clothes and aseptic techniques. A survey aimed at the medical students in 2016 showed that the students perceived that the general introduction was too sparse; moreover, the practical workshop was too short, and it lacked an interprofessional approach. Therefore, an interprofessional faculty at our institution created a complementary e-learning resource, defined as a software-based resource distributed online with the aim to enhance knowledge and performance [9] for all students attending the OR [10].

The aim of the e-learning resource was to better prepare the different student categories (nurses, OR nurses, anesthetic nurses, and medical students) and to reduce emotional barriers, hence creating a better foundation for learning. The focus of the learning outcomes in the e-learning resource was set on skills and interprofessional collaboration.

In 2018, we performed a pilot study evaluating the e-learning resource. It was concluded that it was valuable to the students, but it was difficult to draw conclusions on why and how it was valuable due to lack of qualitative data. Moreover, only medical students participated in the evaluation, so no conclusions could be drawn for the other student categories. There was also a lack of knowledge regarding ideas for improvement of the e-learning resource.

The aim of this study was thus to explore the perspectives of all student categories using the new e-learning resource, with a focus on preparedness for practice.

#### **2. Materials and Methods**

#### *2.1. Design*

An explanatory sequential mixed methods design was used, i.e., data was collected in two consecutive phases: first the quantitative data and then the qualitative data. Thereafter, the data was merged to achieve methodological integration [11,12]. Questionnaires were used to gather qualitative data. Focus group (FG) interviews were used to deepen the knowledge from the questionnaires and to obtain suggestions and ideas for improvement of the e-learning resource. This specific qualitative data collection method was chosen for its ability to help participants to explore and explain their perceptions further in interaction with others [13].

#### *2.2. The E-Learning Resource*

The e-learning resource used in this project was a package of online learning materials using Articulate Storyline® (Articulate Global, New York, NY, USA) and consisted of prerecorded lectures and video demonstrations of skills which could be accessed on different digital devices such as computers or mobile phones. The software used to produce the learning material were PowerPoint® (Microsoft Corporation, Redmond, WA, USA) and Screencast-omatic ® (UserVoice, San Francisco, CA, USA), while the films were recorded using a regular camcorder with a microphone.

The resource was based on seven interprofessional learning outcomes, each one forming the base for a learning activity in the online program. The majority of the learning outcomes were considered generic, except "surgical hand preparation" and "gowning procedure" that were directed to the OR nurses and medical students exclusively (Table 1).

Four of the learning activities were followed by a formative assessment in order to give immediate feedback to students.


**Table 1.** Learning outcome and learning activities of the e-learning resource (Torbjornsson et al., 2018).

#### *2.3. Participants*

Students from three educational programs were included in the study: 4th year medical students (*n* = 24), 3rd year nursing students (*n* = 12), and 1st year perioperative specialist nursing students, specializing in either OR nursing or Anesthesiology nursing (*n* = 16). The medical and nursing students all had their clinical placement at the OR ward in the same hospital, while the perioperative nursing students did their clinical placement in two different hospitals in Stockholm connected to the university. The students received written information regarding the study in their ordinary online learning management system (Ping-Pong AB, Stockholm, Sweden) and verbal information in their course introduction at campus. An email to all eligible students was sent with information on the e-learning resource and the study, together with a link to the e-learning resource on their study platform. All nursing students had a link to the evaluation questionnaire on their study platform. The medical students were given the questionnaire on paper during their examination week at the end of the semester. All students received information regarding the focus groups (FG), and the students who were willing to participate were invited to contact the researcher by mail. In total, 52 students (33 women and 19 men) out of 117 enrolled in the studied programs answered the questionnaires, giving a response rate of 44%. Out of them, 65% had used the e-learning resource prior to their clinical placement (Table 2).

#### **Table 2.** Demographics of the study population.


#### *2.4. Data Collection*

#### 2.4.1. Questionnaire

The questionnaire was developed by the research group and was based on the questionnaire used in the pilot study by Torbjornsson et al. [10]. To address face and content validity, the questionnaire was discussed within the expert group and modified by adding further questions and using another scale for the answers (a 5-level Likert scale instead of a 4-level) [14]. None of the students asked questions about the questionnaire that suggested that they had difficulties to understand it.

The questionnaire consisted of 16 questions: 4 were demographic, 3 contained information regarding the use of the e-learning resource, and 9 were questions where the students rated how well the different learning activities had helped them prepare for their clinical placement in the OR (on a 5-level Likert scale: very little; little; some; large; very large). There was also one open-ended question where the students could give improvement suggestions on the resource (suppl Document S1).

#### 2.4.2. Focus Group Interviews

The FG interviews focused on evaluating the e-learning resource and the students' perceptions regarding if and how it helped them to prepare for their clinical practice. The students were divided into groups based on their profession. The aim was to create homogeneity in the groups and avoid any form of hierarchy that may inhibit an open atmosphere enabling everyone to feel confident to speak out [13].

The FG were attended by a moderator and conducted by the first and second author. The FG interviews lasted 22–45 min and were documented by note-taking from the moderator. A semi-structured interview guide was used, and probing questions were used to further enable the participants to elaborate. Five FG interviews were conducted with a total of 17 students (2–5 students/FG): medical students (*n* = 9), nursing students (*n* = 2), and perioperative specialist nursing students (*n* = 6). All of the focus groups contained participants of the same educational program.

#### *2.5. Data Analysis*

#### 2.5.1. Questionnaire—Quantitative Data

The quantitative data analysis was performed on the questionnaires from the students with descriptive statistics [15]. Continuous variables are presented with mean and standard deviation (SD) and categorical variables as *n* (percent). No comparative analyses between the different student categories were made. The quantitative analyses were performed with IBM SPSS statistics version 23.0 (IBM Inc., Chicago, IL, USA).

#### 2.5.2. Focus Group Interviews—Qualitative Data

The notes from the FG were read and reread to identify patterns and tendencies. The text units were condensed into meaning units, labelled with a code, and sorted into different categories based on the focus areas of the questionnaire. The focus of the qualitative analysis was to extend and to deepen the knowledge from the questionnaires.

The analysis was performed by two members (AMF and ET) of the research group, and the result was discussed until consensus was reached. Directed content analysis using a deductive approach, based on the different areas of the questionnaire, was used [16].

#### *2.6. Ethical Considerations*

The study was performed in accordance with good clinical practice and research as per the Helsinki Declaration [17]. Before any data collection began, the students were informed that the participation in this study was voluntary with the purpose of a scientific analysis and publication. They were also informed that their participation in no way would affect their grades, that they could cease participation at any time, and that the collected data would be completely discarded if they were to withdraw from the study. Completing and returning the questionnaire implied their consent to participate. To ensure confidentiality,

one of the three authors did the initial analysis of the questionnaires and FG interviews and matched the participants' data using numeric codes. *Healthcare* **2021**, *9*, x 5 of 12 **3. Results**

The demographics of the study population is shown in table 1. The majority of the

*Healthcare* **2021**, *9*, x 5 of 12

#### **3. Results**

The demographics of the study population is shown in Table 1. The majority of the students (79%) stated that the e-learning resource had prepared them for their clinical placement in the OR, and the medical students rated the e-learning resource as the least useful. However, three quarters of the medical students still rated the e-learning resource as useful to some extent and none rated it as not useful at all (Figure 1). **3. Results** The demographics of the study population is shown in table 1. The majority of the students (79%) stated that the e‐learning resource had prepared them for their clinical placement in the OR, and the medical students rated the e‐learning resource as the least useful. However, three quarters of the medical students still rated the e‐learning resource as useful to some extent and none rated it as not useful at all (Figure 1). students (79%) stated that the e‐learning resource had prepared them for their clinical placement in the OR, and the medical students rated the e‐learning resource as the least useful. However, three quarters of the medical students still rated the e‐learning resource as useful to some extent and none rated it as not useful at all (Figure 1).

**Figure 1.** Feelings of preparedness for clinical placement at the OR. **Figure 1.** Feelings of preparedness for clinical placement at the OR.

In total, 93% of the students recommended other students to use the e‐learning re‐ source prior to a clinical placement at the OR ward. The differences between student groups are shown in Figure 2. In total, 93% of the students recommended other students to use the e-learning resource prior to a clinical placement at the OR ward. The differences between student groups are shown in Figure 2. In total, 93% of the students recommended other students to use the e‐learning re‐ source prior to a clinical placement at the OR ward. The differences between student groups are shown in Figure 2.

**Figure 1.** Feelings of preparedness for clinical placement at the OR.

**Figure 2.** Students' recommendation of the resource to other students. **Figure 2.** Students' recommendation of the resource to other students.

#### **Figure 2.** Students' recommendation of the resource to other students. *3.1. The E-Learning Resource in Preparation for Clinical Placement at the OR*

The students were asked to what extent the different learning activities of the elearning resource had helped them prepare for their placement at the OR. Eighty percent of the students perceived that all the different learning activities, at least to some extent, had helped them prepare for their clinical placement (Figure 3).

helped them prepare for their clinical placement (Figure 3).

*3.1. The E‐Learning Resource in Preparation for Clinical Placement at the OR*

The students were asked to what extent the different learning activities of the e‐learn‐ ing resource had helped them prepare for their placement at the OR. Eighty percent of the students perceived that all the different learning activities, at least to some extent, had

The FG interviews revealed that many of the students felt different levels of anxiety prior to their clinical placement at the OR. The students explained that the main cause of this anxiety was the feeling of being in an unfamiliar environment and a sense of being out of place. These feelings gave rise to insecurity and stress and made them more sensi‐ tive and vulnerable to what people said or to events that occurred. Students from all pro‐ grams thought that the e‐learning resource was a way to reduce the perceived stress and had prepared them for the OR placement: The FG interviews revealed that many of the students felt different levels of anxiety prior to their clinical placement at the OR. The students explained that the main cause of this anxiety was the feeling of being in an unfamiliar environment and a sense of being out of place. These feelings gave rise to insecurity and stress and made them more sensitive and vulnerable to what people said or to events that occurred. Students from all programs thought that the e-learning resource was a way to reduce the perceived stress and had prepared them for the OR placement:

*"You want to be as well prepared as possible when you arrive. Although you may not be able to practice so much practically wearing gloves and such things, you want to be able to see… because then it is good because then you can see it over and over again… and you become a little more confident when you come out if you have seen it…" (Perioper‐ "You want to be as well prepared as possible when you arrive. Although you may not be able to practice so much practically wearing gloves and such things, you want to be able to see* . . . *because then it is good because then you can see it over and over again* . . . *and you become a little more confident when you come out if you have seen it* . . . *"*

#### *(Perioperative nurse)*

In the FG interviews, all three student categories commented that the e‐learning re‐ source should be a mandatory learning activity for all student groups prior to clinical placement at the OR. In the FG interviews, all three student categories commented that the e-learning resource should be a mandatory learning activity for all student groups prior to clinical placement at the OR.

The nursing students stated that the e‐learning resource saved time for them: by be‐ ing better prepared, they could use theirtime in the OR more efficiently. Medical students, having a two‐hour OR preparation workshop prior to attending their clinical placement, thought the workshop per se prepared them well, but that the e‐learning resource was a good complement. The nursing students stated that the e-learning resource saved time for them: by being better prepared, they could use their time in the OR more efficiently. Medical students, having a two-hour OR preparation workshop prior to attending their clinical placement, thought the workshop per se prepared them well, but that the e-learning resource was a good complement.

#### *3.2. The Students' Perception of the Content in the Learning Resource*

*ative nurse)*

*3.2. The Students' Perception of the Content in the Learning Resource* In total, 52% of the students perceived, to a large or very large extent, that the e‐ learning resource contained all the elements needed to prepare them for their clinical placement. The perioperative nursing students rated the content highest (67%) while the In total, 52% of the students perceived, to a large or very large extent, that the elearning resource contained all the elements needed to prepare them for their clinical placement. The perioperative nursing students rated the content highest (67%) while the medical students expressed a need for additional information.

medical students expressed a need for additional information. The three learning activities containing films and focusing on practical procedures ('gloving technique', 'surgical hand preparation', and 'gowning procedure') were rated most useful (Figure 3). All of the medical students and a majority of the other student categories perceived that the latter two activities, to a very large extent, had prepared them for the OR.

The least valuable activities according to the students were 'OR-design' and 'radiation safety'. 'Radiation safety' got the lowest ratings of all activities, particularly by the medical and perioperative nursing students (Figure 4). The least valuable activities according to the students were 'OR‐design' and 'radia‐ tion safety'. 'Radiation safety' got the lowest ratings of all activities, particularly by the medical and perioperative nursing students (Figure 4).

them for the OR.

*Healthcare* **2021**, *9*, x 7 of 12

The three learning activities containing films and focusing on practical procedures ('gloving technique', 'surgical hand preparation', and 'gowning procedure') were rated most useful (Figure 3). All of the medical students and a majority of the other student categories perceived that the latter two activities, to a very large extent, had prepared

**Figure 4.** Preparedness for clinical placement at the OR regarding the different learning activities from an all‐professions perspective. **Figure 4.** Preparedness for clinical placement at the OR regarding the different learning activities from an all-professions perspective.

The FG interviews revealed that the medical students and perioperative nursing stu‐ dents wanted more information directed to their special needs in their specific profession. The medical students expressed a wish for films that could give them a general overview of the workflow and patient process in the OR, together with instructions on what hap‐ pens when something goes wrong—for example, what to do if they are unsterile during surgery. They thought this would ease the stress and fear of doing something wrong. *"Another suggestion is to also inform about what happens when something goes wrong,* The FG interviews revealed that the medical students and perioperative nursing students wanted more information directed to their special needs in their specific profession. The medical students expressed a wish for films that could give them a general overview of the workflow and patient process in the OR, together with instructions on what happens when something goes wrong—for example, what to do if they are unsterile during surgery. They thought this would ease the stress and fear of doing something wrong.

*for example when we get unsterile or similar. So that when and if it happens it will not be so huge but you know what to do if it happens and how to handle this. Takes a little bit of the stress boost…" (Medical student)* The medical students also requested films containing specific surgical specialties de‐ *"Another suggestion is to also inform about what happens when something goes wrong, for example when we get unsterile or similar. So that when and if it happens it will not be so huge but you know what to do if it happens and how to handle this. Takes a little bit of the stress boost* . . . *"*

#### pending on what kind of surgery they were doing in their clinical placement. The periop‐ *(Medical student)*

erative nurses mentioned elements like positioning on the OR bed and instrument knowledge. The nursing students proposed that the e‐learning resource should be divided into two parts: part 1 with basic interprofessional content, which would be mandatory for all students, and part 2 with optional, more profession‐oriented content. *3.3. Design and Layout of the E‐Learning Resource* The majority of the students stated that they had used the e‐learning resource on their The medical students also requested films containing specific surgical specialties depending on what kind of surgery they were doing in their clinical placement. The perioperative nurses mentioned elements like positioning on the OR bed and instrument knowledge. The nursing students proposed that the e-learning resource should be divided into two parts: part 1 with basic interprofessional content, which would be mandatory for all students, and part 2 with optional, more profession-oriented content.

computer and not on their cell phone. The reason was that the resource was not ade‐

#### quately adapted to the cell phone format. The students thought that they would have used *3.3. Design and Layout of the E-Learning Resource*

The majority of the students stated that they had used the e-learning resource on their computer and not on their cell phone. The reason was that the resource was not adequately adapted to the cell phone format. The students thought that they would have used the e-learning resource more often if it was better adapted to the cell phone. This was particularly important for the activities containing films.

The perioperative nursing students and the nursing students experienced the resource as being too messy and lacking a well-defined flow. This was a major obstacle when conducting the activity.

*"It is confusing the whole arrangement I think... you do not know which (learning activity) one is inside and so you click back, and you end up somewhere else... it is difficult to remember..."*

#### *(Perioperative nursing student)*

The medical students agreed that there could be a more structured arrangement; however, they did not consider this to be a major issue.

All student groups commented that the technical form of the learning activity had some drawbacks. The most important part identified by all the focus groups was that the students wanted to be able to see which activities they had performed, which they had left, and finally when they had succeeded with the entire learning activity. Moreover, they wanted to see how long the films were, how much time had passed, and how much was left. One suggestion from the medical students was that the activities could change color when they had been performed. Students from all categories also wished for the possibility to pause and to rewind if they needed to repeat something, without the need to restart a module in the e-leaning resource.

The students all agreed that the maximum amount of time for this kind of learning resource should not be more than 30 min and that the films should not exceed five minutes each.

#### *3.4. Interprofessional Perspective*

The learning activity 'Professions' was rated differently by the student groups. Common for all student groups was that they considered the interprofessional knowledge in the e-learning resource an important feature, that it was important to learn about each other's responsibilities at the OR, and that this was essential for a successful interprofessional collaboration in the future. However, the FG interviews identified several requests for improvement regarding the interprofessional approach. Students wanted deeper knowledge regarding the task of the different professions working at the OR ward and not only, as earlier described, a film focusing on the patient process or journey throughout the surgical procedure at the OR. They also proposed an additional film following the different professions in their daily work.

*"We are lacking an overall picture of what is happening at the operating room, a description of the flow. To be able to prepare even better. A kind of "patient journey" through the flow to the surgery department and also a "staff journey" to gain an increased understanding of other professions in the surgery. That is generally lacking in teaching in general."*

*(Medical student)*

#### **4. Discussion**

Every semester, the OR receives students from different education healthcare programs. Many of them, regardless of student category, perceive the learning environment at the OR as extremely stressful [1,2]—something that is known to hamper their learning [18]. This study shows that an e-learning resource based on seven interprofessional learning outcomes enhanced the different student categories' perceived preparedness for their clinical placement. It did so by making the environment less unfamiliar by explaining the expected role of the student as well as the role of the other professions at the OR and how they interact. The e-learning resource also gave students the possibility to learn specific skills that are known to induce stress when performed in the real environment [19]. These learning activities could be seen "over and over again" and be repeated just before practice.

The students rated the activities that contained film and focused on practical skills as most valuable. This is consistent with previous findings that skills training such as sterility and operating room etiquettes have been seen as particularly important [2]. Radiation safety got the lowest rating. The rationale for that part could be discussed. It might have been better to name the activity 'safety in the OR' and include, for example, laser safety and how to manage surgical smoke. The lack of student involvement is probably one of the reasons that we missed that in the design. One can also argue that knowledge regarding radiation may be quite abstract for the students and that it does not seem as important as the practical skills [18]. Overall, the findings from the FG interviews conclude that it is important to have student involvement in the design phase of an e-learning resource. For example, there was a request from both the medical students and the perioperative nurses to have more films that contained information regarding specific skills, such as instrument knowledge and surgical skills.

The format is of importance when developing online learning material. The students in this study demanded easy access, preferably in cell phones, with short films and an easy way to find the different contents of the learning activities. Since digital use in society has exploded in recent years and the generation of today's students have been using digital devices their entire life, they would be a useful resource when creating different online activities. Haraldseid, Friberg, and Aase [20] concluded that active student involvement in the development of technological learning material for clinical skills training could enhance the knowledge of the most important learning needs of the students. It could also make the learning activities more effective and attractive [20].

Interprofessional collaboration is known as a major stressor for students attending the OR. Students fear to be despised by the surgeon or nurse when doing something wrong or for simply being in the room [2]. The students expressed the need for knowledge regarding actions when doing something wrong, to be better prepared for such situations. In the FG interviews, the students expressed that they had identified the importance of interprofessional collaboration by using the e-learning resource, and they requested a deeper knowledge of the functions of the different professions, despite the resource not being an IPL resource by definition [21]. The students expressed the IPL ground of learning with and particularly about each other as an area that should be expanded, since they experienced this as important for their psychological preparation for clinical placement at the OR.

Just-in-time teaching (JiTT) is a learning model shown in research to enhance student motivation and to give the students a sense of control. JiTT is defined as a method where the students prepare just before the lesson and lesson time, focusing on specific questions that they experience as difficult and demanding [22]. This e-learning resource may be seen as a way of using the JiTT method, since the students can go through the activities just prior to the clinical task and be better prepared. The main advantage is to be able to repeat the specific element as many times as needed for the student in an easy-access way (on their cell phone, for instance). This creates the possibility for students to tailor their learning to meet their own specific individual learning needs [9,23], which is particularly important given that we address such a broad spectrum of different students. Furthermore, in a stressful environment like the OR, the stress and anxiety of students may inhibit learning and prolong the learning curve [2]. To prepare students by using e-learning in the practical procedures, they may feel less stress when arriving at the OR, and the threshold for learning can be lowered.

The development of an e-learning resource, such as the one that we have described, could be a useful learning method for student groups in other contexts where practical skills and interprofessional collaboration is important.

#### *Limitations*

It may be argued that a limitation of this study is that we did not have a control group and did not perform a comparative study to assess the effectiveness of the e-learning resource. The aim of the study was, however, not to measure student preparedness or specific knowledge or skills, but to evaluate and describe the students´ perceptions and to explore the value of the resource from the students´ perspective.

It is recommended that the ideal size of a focus group is six to ten people [24]. In our study, we had a convenience sample which was based on the voluntariness and interest of the students, and we unfortunately did not manage to receive further participants.

However, Cote-Arsenault and Morrison-Beedy [25] emphasize that the aim of the study together with developmental levels of the participants is more important than a set number. Since we included students with several years of university studies and we chose to have the FGs separated for the different categories, we believe that the low number of participants in the groups did not inhibit the creativity and data received, nor did we feel that the participants felt pressured to speak, which is described as being a risk in low-numbered FGs [25]

To only rely on notetaking during the FG interviews and not audiotapes is a limitation. However, as the moderators had high knowledge regarding the setting and the appearance of the e-learning resource, it was not perceived as a problem. The low response rate in the quantitative part of the study can seem to be a problem for the validity (44%). There is a risk of nonresponse bias; however, it is tempting to believe that it does not have the same impact on the result as it may have when it comes to sensitive data such as aspects on quality of life [26].

It also needs to be mentioned that the questionnaire was not evaluated with a psychometric test. However, we believe the fact that the questionnaire was evaluated in the expert group, as well as in the pilot study, increases the validity. Further, the use of FG gave a deeper knowledge regarding the e-learning resource.

#### **5. Conclusions**

We conclude that an e-learning resource seems to increase students' perceived preparedness for their clinical practice at the OR. The students stated that they felt more relaxed when attending the OR, which may, according to the literature, lead to a better learning environment and improved learning. The development of e-learning resources has its challenges, and we recommend student involvement to evaluate the content of the learning activities as well as to prevent technical issues.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/article/10.3 390/healthcare9081028/s1, Document S1: translated version of the questionnaire.

**Author Contributions:** Conceptualization, methodology, formal analysis, investigation, resources, data curation, writing—original draft preparation, and writing—review and editing, A.-M.F., E.T. and A.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** The authors have received an unrestricted pedagogical grant from Karolinska Institutet, Department of Clinical Science and Education, Södersjukhuset.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


### *Article* **Perception of the Online Learning Environment of Nursing Students in Slovenia: Validation of the DREEM Questionnaire**

**Lucija Gosak 1,\*, Nino Fijaˇcko <sup>1</sup> , Carolina Chabrera <sup>2</sup> , Esther Cabrera <sup>2</sup> and Gregor Štiglic 1,3,4**

	- <sup>4</sup> Usher Institute, University of Edinburgh, Edinburgh EH8 9YL, UK
	- **\*** Correspondence: lucija.gosak2@um.si; Tel.: +386-2-300-47-35

**Abstract:** At the time of the outbreak of the coronavirus pandemic, several measures were in place to limit the spread of the virus, such as lockdown and restriction of social contacts. Many colleges thus had to shift their education from personal to online form overnight. The educational environment itself has a significant influence on students' learning outcomes, knowledge, and satisfaction. This study aims to validate the tool for assessing the educational environment in the Slovenian nursing student population. To assess the educational environment, we used the DREEM tool distributed among nursing students using an online platform. First, we translated the survey questionnaire from English into Slovenian using the reverse translation technique. We also validated the DREEM survey questionnaire. We performed psychometric testing and content validation. I-CVI and S-CVI are at an acceptable level. A high degree of internal consistency was present, as Cronbach's alpha was 0.951. The questionnaire was completed by 174 participants, of whom 30 were men and 143 were women. One person did not define gender. The mean age of students was 21.1 years (SD = 3.96). The mean DREEM score was 122.2. The mean grade of student perception of learning was 58.54%, student perception of teachers was 65.68%, student academic self-perception was 61.88%, student perception of the atmosphere was 60.63%, and social self-perception of students was 58.93%. Although coronavirus has affected the educational process, students still perceive the educational environment as positive. Nevertheless, there is still room for improvement in all assessed areas.

**Keywords:** education; learning environment; nursing student; transcultural adaptation; psychometric properties; health care

#### **1. Introduction**

Due to the coronavirus pandemic (COVID-19), which was reported in Wuhan, China [1–4] and soon after, the first major outbreak in Europe spread rapidly to Slovenia [5,6]. Governments issued directives on social isolation and living at home, so colleges and universities around the world were closed [7]. COVID-19 has forced education systems around the world to find alternatives to personal teaching [8]. Online distance learning platforms are the only available way of learning and teaching during unprecedented events such as the outbreak of COVID-19 [9–11]. However, it is important to distinguish between online distance education and distance learning in an emergency as a temporary solution. Online education provides students with flexibility and choice [12]. This involves implementing education using information and communication technology [13] and represents an easily accessible teaching method [14].

Online learning promotes student-centered learning, in which case courses are easy to manage [15], resulting in better knowledge and self-efficacy for some students [16]. It increases performance, encourages critical thinking, and improves writing skills for

**Citation:** Gosak, L.; Fijaˇcko, N.; Chabrera, C.; Cabrera, E.; Štiglic, G. Perception of the Online Learning Environment of Nursing Students in Slovenia: Validation of the DREEM Questionnaire. *Healthcare* **2021**, *9*, 998. https://doi.org/10.3390/healthcare 9080998

Academic Editors: José João Mendes, Vanessa Machado, João Botelho and Luís Proença

Received: 20 July 2021 Accepted: 1 August 2021 Published: 5 August 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

most students [17]. Through the accelerated use of online learning, educators and carers need to consider the pedagogical and practical challenges posed by the integration of online learning [18]. Negative aspects highlighted are a lack of appropriate infrastructure for some students, less effective communication and interaction, inability to implement practical applications, lack of socialization, lack of motivation, less objective exams, and the possibility of deteriorating health [19].

Despite growing evidence that online learning is just as effective as traditional learning tools, there is very little evidence of what works, when, and how online learning improves teaching and learning [20]. Therefore, in this study, we decided to evaluate the online learning environment of students using the Dundee Ready Education Environment Measure (DREEM) tool [21–23]. Any learning environment that meets students' internal and external needs is likely to lead to better and more promising learning outcomes [24]. Achieving an optimal educational environment must meet the expectations of students regarding the school atmosphere, teaching, teachers, students, school staff, educational equipment, and the physical environment [25]. A good learning environment for students in clinical practice depends on the structure of student admission, the pedagogical atmosphere, and the participation of those involved [26]. The educational environment has an impact on students' learning outcomes, preparation for practice, and student satisfaction [27]. Also, the perception of the learning environment is related to well-being and stress in students [28].

The main goal of the research is a validation of the questionnaire focusing on the assessment and perception of nursing students about the online learning environment. The goal is also to test psychometrically the DREEM tool [22,23]. The validation of the DREEM tool is performed within the Erasmus+ project Digital Toolbox for Innovation and Nursing Education (I-BOX), which aims to develop material for teaching nursing students and nurses. Based on the obtained results, we will also assess where the greatest deviations occur in the assessment of the learning environment and thus encourage the improvement of the learning environment for students.

#### **2. Materials and Methods**

#### *2.1. Study Design*

We used quantitative research methodology [29–31]. Data for assessing the educational environment by undergraduate and postgraduate nursing students were collected using an online questionnaire between November 2020 and January 2021. The survey questionnaire was previously translated into Slovenian language and validated in the Slovenian environment for the first time.

#### *2.2. Assessment Tool*

To assess the online educational environment, we used the DREEM tool [22,23]. DREEM is a validated tool for assessing the educational environment in health care professions worldwide [32]. In addition to being used to diagnose deficiencies in the current educational environment, DREEM is also used to compare different groups, monitor the same group over time, and assess factors influencing the educational environment [33,34]. The DREEM tool includes five subscales: students' perception of learning (SPL); students' perception of teachers (SPT); students' academic self-perception (SAP); students' perception of the atmosphere (SPA) and students' social self-perception (SSP). The maximum score is 200 [35]. The use of the questionnaire was previously authorized by the authors [22,23]. The survey questionnaire was translated from English into Slovenian and then back to the original language [36]: Independently by two researchers, the survey questionnaire was translated from English into Slovenian. Both researchers had the necessary knowledge of English, andragogy, and nursing. Thus, we obtained two versions of the translation, which we merged into one in the next step, based on consultation between experts. If disagreement was present, a third researcher was involved. In the last step, two experts with the necessary knowledge of English translated a joint version of the Slovenian questionnaire

into English. Thus, we obtained two forms of reverse translation and subsequently merged them into a common form [29,30].

Questionnaires were distributed using an online survey platform ENKA from which the results were then downloaded and analysed using IBM SPSS Statistics 27.

#### 2.2.1. Validation of Assessment Tool

We assessed the validity of the content and the validity of the construct in the survey questionnaire and performed confirmatory factor analysis [37,38]. To determine the content validity, we included experts who have the necessary knowledge in the field [29,30,37,39]. Based on the recommendations where six to ten experts are required [40], we included six experts who work as nursing teachers. The questions in the questionnaire were rated on a four-point scale from 1 to 4, where 1 represents statements that are not relevant; 2, deficient/poorly understood statements; 3, partially understandable/partially relevant statements; and 4, entirely understandable/completely relevant claims [41]. To assess the content validity of the questionnaire, we calculated the content validity of individual claims (I-CVI) and content validity of the whole questionnaire (S-CVI) [41–46]. For the internal reliability analysis, we calculated Cronbach's α, which presents us with a measure of internal reliability between several items [47]. Cronbach's alpha coefficients and interpreted the values as follows: ≥0.90, excellent; 0.80–0.89, good; 0.70–0.79, acceptable; 0.60–0.69, questionable; 0.50–0.59, poor; and <0.50, unacceptable [48]. Correlations between items are an essential element in the analysis of the items representing a specific concept. Correlations between items examine the extent to which ratings of one item are related to ratings of all other scale items [49–51].

I-CVI represents the quotient between the number of experts who rated each question with a grade of 3 or 4 and between the number of all experts, which in our case was six [42,44–46,52]. The probability of agreement was calculated using the formula Pc = [N!/A! (N-A)!] 0,5N where N represents the number of evaluators, and A represents the number of consents [42,44–46,52,53]. We used the following formula to calculate the kappa determination of the compliance agreement: k = (I-CVI − Pc)/(1 − Pc). I-CVI represents item content validity index, and Pc represents the probability of chance agreement [42,44–46,52]. The S-CVI represents the proportion of questions rated by two experts with a score of 3 or 4 [39,42,52].

#### 2.2.2. Perception of the Learning Environment

The DREEM tool includes 50 items, 41 positive and nine negatives, related to learning perception (12 items), teacher perception (11 items), academic self-perception (eight items), atmospheric perception (12 items), and social self-perception (seven items). Each item is rated on a five-point Likert scale (from 1—strongly disagree to 5—strongly agree), where reverse-coding is used for nine statements [22,32]. Questions 4, 8, 9, 17, 25, 35, 39, 48, and 50 are reverse-coded [22,32,54]. The highest score indicates an ideal educational environment [22,32]. The categorization of the sub-scale for all items is as follows: lower than 50 represents a very poor level, range 51–75 is defined as a "plenty of problems" category, range 76–150 represents more positive than negative category, and higher than 150 represents an excellent score [35]. When analysing an individual item, it is necessary to pay attention to those with a mean score lower than 2. There are also possible improvements in the measured assumptions with a mean score between 2 and 3 [55–57].

#### *2.3. Ethics of Research*

Before the research, we obtained ethical permission from the institutional ethical commission (No. 038/2020/2176-02/504). The authors of the questionnaire were asked for permission to use and translate it. Individuals who submitted responses to the online questionnaire also agreed to participate in the survey [22,23]. As part of the research, we sent students an invitation to participate in the research by e-mail. The online questionnaire also informed the participants about the purpose and goals of the research. Participants

had the opportunity to refuse to participate in the anonymous survey. The survey was conducted from November 2020 until January 2021. We also informed them that we would use the results exclusively for research. In doing so, we will not disclose information from which the individuals involved could be identified. The risks and burdens of research are minimal.

#### **3. Results**

Of the 298 invited participants, 174 participants completed the questionnaire (response rate: 58.4%). Of these, 17% (*n* = 30) were men and 83% (*n* = 143) were women (one person did not specify their gender). The average age of the participants was 21.1 years (SD = 3.96). The youngest person was 18 years old, and the oldest was 46 years old. Other basic characteristics of the students involved are shown in Table 1.


**Table 1.** Sample characteristics.

N = sample size; % = percent.

#### *3.1. DREEM Tool Validation Results*

The DREEM questionnaire was backtranslated from English into Slovenian by two experts. The content validity and reliability of the DREEM tool questionnaire in the Slovenian environment to assess the perception of the learning space in nursing students are presented below.

#### 3.1.1. Content Validity of the Questionnaire

Table 2 presents the I-CVI, Pc, and k coefficient calculations for all questions in the DREEM tool. I-CVI for all questions in the Slovenian version of the questionnaire is acceptable. The I-CVI for all questions except question 20 was 1.000. The I-CVI for question twenty, "The teaching is well focused," was 0.833. The probability of agreement on all questions is 0.016, and on the twentieth question, 0.094. Kappa on the determination of the agreement on adequacy for all questions is 1. For the twentieth question, it is 0.816.


#### **Table 2.** Content validity of the DREEM tool.


#### **Table 2.** *Cont*.

No. = Number of question; N = sample size; A = number of agreements; I-CVI = item content validity index; Pc = probability of chance agreement; k = kappa designating agreement on relevance.

> The evaluation of two experts was included in the S-CVI assessment. None of them rated the question with a score of 1 or a score of 2 with a final S-CVI of 1.000 and is acceptable for the Slovenian environment (Table 3).

**Table 3.** Scale content validity of the DREEM tool.


S-CVI = scale content validity.

#### 3.1.2. Reliability of the Questionnaire

Supplementary Materials presents the correlations between the items in each scale in the DREEM tool questionnaire. Item correlations ranged between −0.038 and 0.620.

Cronbach's alpha was 0.951, which indicates a high level of internal consistency. Table 4 represents the values of Cronbach's alpha with specific items deleted. Removing any question other than question 17, "Cheating is a problem in this school," and question 25, "The teaching over-emphasizes factual learning," would reduce the value of Cronbach's alpha. Corrected item-total correlation for question 17 was 0.186, and 0.192 for question 25.





Figure 1 presents a graph for screen analysis. The graph shows the eigenvalue scree plot for 50 instrument elements and points at one factor. *Healthcare* **2021**, *9*, x 8 of 16

**Figure 1.** Analysis Scree Plot. **Figure 1.** Analysis Scree Plot.

#### *3.2. Results of Perception of the Learning Environment 3.2. Results of Perception of the Learning Environment*

Online teaching was perceived more positively than negatively. The mean assessment of student perception of learning is 28.1/48, student perception of teachers is 28.9/44, student academic self-perception is 19.8/32, student perception of the atmosphere is 29.1/48, and social self-perception of students is 16.5/28 (Table 5). All individual subscales are statistically related (*p* < 0.001). Online teaching was perceived more positively than negatively. The mean assessment of student perception of learning is 28.1/48, student perception of teachers is 28.9/44, student academic self-perception is 19.8/32, student perception of the atmosphere is 29.1/48, and social self-perception of students is 16.5/28 (Table 5). All individual subscales are statistically related (*p* < 0.001).


**Table 5.** Mean score of DREEM tool. **Table 5.** Mean score of DREEM tool.

SPL = Students perception of learning; SPT = Students perception of teachers; SAP = Students academic self-perception; SPL = Students perception of learning; SPT = Students perception of teachers; SAP = Students academic self-perception; SPA = Students perceptions of atmosphere; SSP = Students social self-perceptions; SD = standard deviation.

SPA = Students perceptions of atmosphere; SSP = Students social self-perceptions; SD = standard deviation.

Based on the Shapiro–Wilk test for women and the Kolmogorov–Smirnov test for men, we found that the individual values of the scales in students were unevenly distributed according to gender. Based on the Mann–Whitney U test, we identified a statistically significant relationship between the assessment of student perception of learning by gender (U = 1346,500; *p* = 0.024). The mean SPL score for men was 24.9/48 (SD = 8.82). For women, this mean score was 28.9/48 (SD = 7.27). There is no statistically significant difference by gender between the other subscales. Nevertheless, in all subscales, the scores were higher for women than for men: subscale SPT (29.3 vs. 28.1), subscale SAP (20.0 vs. 19.0), subscale SPA (29.4 vs. 28.6), and subscale SSP (16.4 vs. 16.6) (Figure 2). Based on the Shapiro–Wilk test for women and the Kolmogorov–Smirnov test for men, we found that the individual values of the scales in students were unevenly distributed according to gender. Based on the Mann–Whitney U test, we identified a statistically significant relationship between the assessment of student perception of learning by gender (U = 1346,500; *p* = 0.024). The mean SPL score for men was 24.9/48 (SD = 8.82). For women, this mean score was 28.9/48 (SD = 7.27). There is no statistically significant difference by gender between the other subscales. Nevertheless, in all subscales, the scores were higher for women than for men: subscale SPT (29.3 vs. 28.1), subscale SAP (20.0 vs. 19.0), subscale SPA (29.4 vs. 28.6), and subscale SSP (16.4 vs. 16.6) (Figure 2).

To show the relationship between age and individual subscales, we performed a Pearson correlation test. The age of students is statistically significantly related to the SAP subscale score (r = 0.212; *p* = 0.007) and the SPA subscale score (r = 9.213; 0.007).

Based on the Kruskal–Wallis test, we found that the study program attended by students affects the SAP score. The mean grade of SAP students attending the undergraduate first degree study program nursing care is 19.7/32 (SD = 5.05), the score of students attending the postgraduate second degree study program nursing care is 25.67/32 (1.53), and the score of students who attend a postgraduate third degree study program nursing care is 26/32 (SD = 8.49). *Healthcare* **2021**, *9*, x 9 of 16

**Figure 2.** Gender comparison in subscales.

**Figure 2.** Gender comparison in subscales.

To show the relationship between age and individual subscales, we performed a Pearson correlation test. The age of students is statistically significantly related to the SAP subscale score (r = 0.212; *p* = 0.007) and the SPA subscale score (r = 9.213; 0.007). Based on the Kruskal–Wallis test, we found that the study program attended by students affects the SAP score. The mean grade of SAP students attending the undergraduate first degree study program nursing care is 19.7/32 (SD = 5.05), the score of students attending the postgraduate second degree study program nursing care is 25.67/32 (1.53), and the score of The mean assessment of student perception of learning is 28.1/48, which means a more positive perception. Problematic assumptions with a mean grade of ≤2 in the SPL subscale are "I am encouraged to participate in class," which has an average grade of 1.8 (SD = 0.83), and "The teaching over-emphasizes factual learning," which has a mean grade of 1.3 (SD = 0.68) (Table 6); 69.2% of men (*n* = 18) and 64% of women (*n* = 80) agree that teachers being encouraging to participate. Table S1 in Supplementary Materials present the links between SPL items.


students who attend a postgraduate third degree study program nursing care is 26/32 (SD = 8.49). **Table 6.** Subscale SPL.

22 The teaching is sufficiently concerned to develop my confidence. 150 2.6 (0.89) No. = Number of question; N = sample size; M = mean; SD = standard deviation.

24 The teaching time is put to good use. 152 2.9 (0.61) 25 The teaching over-emphasizes factual learning. 151 1.3 (0.68) 38 I am clear about the learning objectives of the course. 150 2.7 (0.86)

47 Long-term learning is emphasized over short-term. 147 2.7 (0.95)

The mean score of student perception of teachers is 28.9/44, which means that it is moving in the right direction. The item "The teachers are authoritarian" received the lowest mean value of 1.9 (SD = 0.98) (Table 7); 39.4% of respondents (*n* = 62) do not agree with this statement, and 25.5% (*n* = 48) neither agree nor disagree with this statement. Table S2 in Supplementary Materials present the links between SPT items.



No. = Number of question; N = sample size; M = mean; SD = standard deviation.

The mean score of students' academic self-perception is 19.8/32, representing that feelings are more on the positive side. None of the items in the SAP subscale received a lower mean score than 2 (Table 8). With the highest mean score, the item "I have learned a lot about empathy in my profession" stands out, with a mean score of 3.1 (SD = 0.65). A total of 89.9% of respondents (*n* = 134) agree that they learned a lot about empathy in the profession during their studies in the current year. Table S3 in Supplementary Materials present the links between SAP items.

**Table 8.** Subscale SAP.


A score of students' perceptions of the atmosphere is 29.1/48, meaning that the atmosphere is more positive than negative. The lowest score was given to the statement "This school is well timetabled" and was 1.5 (SD = 1.10) (Table 9); 51.3% of respondents (*n* = 81) disagree that the schedule is well planned, 25.9% (*n* = 41) neither agree nor disagree with the statement. Table S4 in Supplementary Materials present the links between SPA items.


**Table 9.** Subscale SPA.

No. = Number of question; N = sample size; M = mean; SD = standard deviation.

The mean score of students' social self-perception is 16.5/28, meaning that social perception is not too bad (Table 10). The item "There is a good support system for students who get stressed" and the item "I am too tired to enjoy this course" get a lower score of 2, more specifically 1.8 (SD = 1.06) and 1.7 (SD = 0.97). 43.1% of the surveyed (*n* = 69) students are too tired to participate in the lectures. Table S4 in Supplementary Materials present the links between SPL items.



No. = Number of question; N = sample size; M = mean; SD = standard deviation.

Supplementary Materials represents the inter-item correlations of the subscale.

#### **4. Discussion**

To the best of the authors 'knowledge, this is the first study to assess students' perceptions of the educational environment in Slovenia. We wanted to obtain information to assess the learning environment of nursing students. Our study was conducted during the COVID-19 pandemic, when colleges were forced to move their education online. Thus, despite the challenges of social distancing, isolation, and quarantine measures [58], they continued to provide education for nurses [59].

The assessment of the learning environment in the nursing student participants of this study is more positive than negative, as in many studies where this tool was used [15,35,54,60–68]. So far, only one study has been conducted that provides researchers with insight into the differences between personal and online teaching. In the United Kingdom, researchers conducted a national cross-sectional study to assess the learning environment during online teaching. They found that the assessment of the learning environment was lower than in live teaching [21].

We wanted to assess if there are differences between individual scales according to gender. In our study, differences were detected only in the assessment of learning perception (SPL), where women had a higher score than men (28.9 for women vs. 24.9 for men, *p* = 0.024). No statistically significant differences were detected in other subscales. The overall score is also higher for women (124.3; SD = 29.04) compared with men (116.1; SD = 32.1). Similar results were also obtained in another study where researchers found higher scores in women than in men [62]. This means that women have a better perception of the educational environment. Studies detect gender differences in study habits, which in turn affect student outcomes [69]. Also, female students are more willing to participate and work in a team than male students [70]. There are also differences in the acceptance of e-learning between men and women [71]. In contrast to our study, however, Fooladi found that perceptions of the learning environment are lower in women among vulnerable groups [72].

There is no statistically significant difference between years of enrolment in our study. The highest DREEM score is detected in the first year of study, where the mean grade is 124.15 (SD = 31.89). Other research finds that perception of the learning environment differs according to student performance, and also a difference between individual years of study [73]. Shrestha, et al., also note that the learning environment assessment is highest among students in the first year of study [74].

Of particular concern is that most students disagree with the claim that the schedule is well planned. Only 20.7% of respondents (*n* = 36) rate schedules as well-planned. This can also be related to the observation that 40.2% of students (*n* = 70) are often too tired to participate in lectures. Students are primarily concerned with time management in distance learning [75,76]. It is important to reduce the academic burden on students and help students develop time management skills, which significantly contributes to their success [77,78]. Stress and overload in nursing students can lead to burnout, anxiety, and depression [79].

Nebhinani, et al. point out that there is a great need to plan and implement various stress management programs [80]. Only 23.5% of respondents (*n* = 41) in our study agree that a good support system is in place in the presence of stress. Like our study, students in eastern Nepal perceived that they do not have a good support system during times of stress [74]. Numerous studies have found increased stress in students due to an outbreak of coronavirus disease [10,81–83], so support in this area is particularly important at this time. Stress connected with distance learning for students mainly leads to a lack of concentration, motivation, and technical difficulties [84].

56.3% of students (*n* = 98) believe that teachers focus too much on teaching based on data memorization, and 36.2% of students (*n* = 63) believe that teachers are too authoritative in their work. Nevertheless, most students (*n* = 117; 67.2%) believe that teaching is sufficiently focused on developing competencies related to their profession.

Health science students will receive such a good education, but its effectiveness must be rigorously and regularly evaluated [85]. Therefore, it is of the utmost importance that such research is continued, and the rate of improvement is assessed. Only in this way can we achieve the best possible learning environment for students.

#### *Limitations*

There is a possibility of bias due to low response to the survey questionnaire. The reason for this might be in the fact that questionnaires were sent to the students in an online form, which usually results in low response rates. The study also took place within one faculty and cannot be generalized on a wider scale. Also, the limitation is that the assessment of the educational environment was carried out only during online teaching and cannot be compared with the evaluation of the learning environment during the traditional implementation of the learning process. Another limitation is that the online survey was conducted only from November 2020 to January 2021 and not in other study periods.

#### **5. Conclusions**

Nursing students generally rate their learning environment more positively than negatively, but there is still room for improvement in all categories. Greater emphasis is needed on the organization and timing of lessons to achieve better concentration of students in classes and reduce their level of stress. Educational organizations are also recommended to set up a good support system for students. The need to change the approach by teachers and their role was also perceived. With an authoritative approach and too much emphasis on factual learning, we negatively affect the student's motivation and willingness to work. Teachers can improve this through appropriate pedagogical and andragogic education.

It is important that learning organizations and teachers also focus on providing a suitable and appropriate learning environment for students during distance learning. This is the only way they can contribute to positive learning outcomes and gain student experience. However, this presents a unique challenge, as the teacher has no contact with students when teaching online.

In the future, we plan to conduct a longitudinal study to observe the impact and variation of different factors in assessment of the learning environment over time.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/article/10 .3390/healthcare9080998/s1, Inter-item correlations of the subscale.

**Author Contributions:** Conceptualization, L.G., C.C., E.C., and G.Š.; Data curation, L.G., N.F., C.C., E.C., and G.Š.; Formal analysis, L.G. and G.Š.; Methodology, L.G., N.F., and G.Š.; Supervision, L.G. and G.Š.; Validation, L.G., N.F., and G.Š.; Visualization, L.G. and G.Š.; Writing—original draft, L.G. and G.Š.; Writing—review & editing, L.G., N.F., C.C., E.C., and G.Š. All authors have read and agreed to the published version of the manuscript.

**Funding:** The project "Digital Toolbox for Innovation in Nursing Education (I-BOX)" has been funded with support from the European Commission (2019-1-ES01-KA203-065836) under the Erasmus+ program. This publication reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein. This study was also supported by the "knowledge through creative pathways 2016–2020" scheme cofunded by the European Union from the European Social Fund and the Republic of Slovenia and the Slovenian Research Agency (grant numbers N2-0101 and P2-0057).

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Ethics Committee of Faculty of Health Sciences, University of Maribor (038/2020/2176-02/504, 10.06.2020).

**Informed Consent Statement:** Participants agreed to participate in the research by completing and submitting a questionnaire.

**Data Availability Statement:** Data is currently not available for sharing, due to the further data collection process. Contact the first author for more information.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

