*Article* **Lapnurse—A Blended Learning Course for Nursing Education in Minimally Invasive Surgery: Design and Experts' Preliminary Validation of Its Online Theoretical Module**

**Juan Francisco Ortega-Morán 1,\*, Blas Pagador <sup>1</sup> , Juan Maestre-Antequera <sup>1</sup> , Javier Sánchez-Fernández <sup>1</sup> , Antonio Arco <sup>2</sup> , Francisco Monteiro <sup>2</sup> and Francisco M. Sánchez-Margallo <sup>1</sup>**


**Citation:** Ortega-Morán, J.F.; Pagador, B.; Maestre-Antequera, J.; Sánchez-Fernández, J.; Arco, A.; Monteiro, F.; Sánchez-Margallo, F.M. Lapnurse—A Blended Learning Course for Nursing Education in Minimally Invasive Surgery: Design and Experts' Preliminary Validation of Its Online Theoretical Module. *Healthcare* **2021**, *9*, 951. https:// doi.org/10.3390/healthcare9080951

Academic Editors: Luís Proença, José João Mendes, João Botelho and Vanessa Machado

Received: 22 June 2021 Accepted: 26 July 2021 Published: 28 July 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

**Abstract:** Background: The implantation of Minimally Invasive Surgery (MIS) leads to the specialization of nurses in this surgical field. However, there is no standard curriculum of MIS Nursing in Europe. Spanish and Portuguese nurses are inexperienced and have poor training in MIS. For that, a blended learning course for nursing education in MIS (Lapnurse) has been developed. This work aims to detail the course design and to preliminary validate by experts its online theoretical module. Methods: Lapnurse consists of an online module with nine theoretical lessons and a face-to-face module with three practical lessons. The e-learning environment created to provide the online module, with didactic contents based on surgical videos and innovative 3D designs, has been validated by two technicians (functionality) and four nurses with teaching experience in MIS (usability and content). Results: The E-learning platform meets all technical requirements, provides whole and updated multimedia contents correctly applied for educational purposes, incorporates interactivity with 3D designs, and has an attractive, easy-to-use and intuitive design. Conclusions: The lack of knowledge in MIS of Spanish and Portuguese nurses could be addressed by the blended learning course created, Lapnurse, where the e-learning environment that provides theoretical training has obtained a positive validation.

**Keywords:** blended learning; design; education; e-learning; laparoscopy; minimally invasive surgery; nursing; training course; validation

#### **1. Introduction**

Health professionals' mobility between countries has been a common practice for many years. As an example of this, the migration of nurses between Spain and Portugal is easier because of their geographical contiguity and cultural affinity, and it has always existed in both directions [1]. That cross-border migration of nurses is enriching for both destination and origin countries, as long as these health professionals are adequately trained to practice in the destination country. Minimally Invasive Surgery (MIS) techniques have been implanted in recent years as a routine surgical practice, making it necessary to meet the training needs of professionals in this surgical field [2], including nurses. However, as far as we know, there is no standard curriculum of MIS Nursing in Europe. Thus, appropriate means should be available for nurses to acquire the knowledge that allows them to develop new attitudes and skills in the workplace [3], as required in MIS.

E-learning may be a good tool to improve the quality of education [4], allowing students flexibility in time, place, and access to content. Furthermore, it improves their learning ability by allowing them to progress at the most appropriate speed [5]. However, those education programs carried out entirely online can raise inequalities for users

with computer illiteracy [6]. Moreover, e-learning has a series of interrelation drawbacks between teachers and students, such as the inability of students to receive immediate feedback from the teacher [5]. For that, blended learning, which combines both online and face-to-face learning, is a method that allows overcoming these barriers of e-learning, combining the flexibility of timing and the convenience of online delivery with the spontaneity and interpersonal interaction of face-to-face learning [7].

Although the benefits of e-learning and blended learning have been demonstrated in several studies, not enough is known about what nurses think about continuing education through e-learning [8,9] or blended learning [10]. For that, a study has been performed to firstly identify and compare training needs and level of experience in MIS of Spanish and Portuguese nurses, in order to check if they are qualified to work in this field, and to also check whether e-learning technologies would be useful for a successful MIS training. A total of 81 nurses from Extremadura (Spain) and 113 nurses from Alentejo (Portugal) participated by filling in a questionnaire that showed a lack of experience and training of nurses in MIS. In order to meet their training needs, they consider essential the implementation of e-learning environments that offer courses with useful contents for their work and a focus on clinical practice, to incorporate interactivity into these environments, and to use surgical videos in online training processes. Such study indicated that nurses are technologically competent to be trained by e-learning, and the blended learning is their preferred method to learn MIS.

Therefore, we have designed and developed a laparoscopic training course for nursing, named Lapnurse, through blended learning method with interactive didactic contents. The main objectives of this study were to:


#### **2. Materials and Methods**

#### *2.1. Course Design*

Blended learning method, as preferred by nurses, has been selected to implement the Lapnurse course. This course consists of (1) a theoretical module with nine lessons, to be taught through an e-learning web environment, and (2) a practical module with three lessons, to be carried out in person in a specialized reference centre (Figure 1).

The contents of the course have been designed based on the role of nurses in MIS. The tasks of circulating nurses are mainly focused on maintaining a safe and comfortable environment in the surgical room during the surgery. They do not participate in intervention and work outside of the sterile field, and have been responsible for the inspection of surgical equipment and instruments, and also the preparation of the documentation related to the surgery and the patient. However, the scrub nurses are directly involved in the surgical intervention working within the sterile field. They prepare the operating room for the patient and the instruments. They assist the surgical team by donning sterile masks, gloves, and gowns, as well as by passing the instruments during surgery. For this, surgical nurses need to acquire the necessary knowledge about instruments, equipment, and techniques related to laparoscopic interventions in order to perform their work in a safe way. Different sections of the developed course have been designed to meet all these training needs in MIS.

As the e-learning environment used to provide the theoretical module of Lapnurse, the Moodle web application [11] has been selected (Figure 2). This free distribution course management system is very useful in the educational field, since it allows for incremental performance and satisfaction when compared to traditional didactic lectures [12].

**Figure 1.** Blended learning course. **Figure 1.** Blended learning course.

**Figure 2.** The Spanish version of the e-learning environment used to provide the theoretical module of Lapnurse. **Figure 2.** The Spanish version of the e-learning environment used to provide the theoretical module of Lapnurse.

Lessons consist of sequential pages providing all resources of the course (text, images, videos, and 3D designs), in Spanish and Portuguese languages. On the left side of the environment, the menu of each lesson is displayed, so that the user can view the tabs Lessons consist of sequential pages providing all resources of the course (text, images, videos, and 3D designs), in Spanish and Portuguese languages. On the left side of the environment, the menu of each lesson is displayed, so that the user can view the tabs that the

that the lesson contains and can navigate through them. At the end of each lesson, a questionnaire evaluates the knowledge acquired by the students through multiple choice

attempt is automatically scored and the results are saved in the gradebook, although they

• Student: They will be the main receiver of didactic contents in an active knowledge search. For aspects of formal and non-formal learning, the student acts as a consumer of content, e.g., accessing didactic content and courses, but without permission to create new content or edit existing ones. For aspects of informal learning, the student is considered an active participant within the professional network of the environment. They will participate in forums and discussions, and will interact with

• Teacher: A MIS surgeon with broad experience in teaching. As such, the main role of the teacher will be as a content provider for formal learning, creating and editing didactic contents, and loading them into the environment. For informal learning,

To encourage collaborative learning, synchronous and asynchronous means for communication between users have been included (chat and two forums, respectively). The chat allows participants to have a discussion in text format in a synchronous way in real time. Chats are especially useful when a group does not have the possibility to meet physically to talk face-to-face. The forum allows participants to have asynchronous discussions that take place over an extended period of time. The automatically created forum "News and announcement" indicates what is happening in the course, and it is only posted by teachers and an administrator. Additionally, the "Discussion forum" has been

created, in which all users can add or reply to posts.

other users in the community.

are displayed to the user, with feedback comments and correct answers.

The main roles interacting with the system are the following:

lesson contains and can navigate through them. At the end of each lesson, a questionnaire evaluates the knowledge acquired by the students through multiple choice and true/false questions, with two attempts to answer them within a limited time. Each attempt is automatically scored and the results are saved in the gradebook, although they are displayed to the user, with feedback comments and correct answers.

To encourage collaborative learning, synchronous and asynchronous means for communication between users have been included (chat and two forums, respectively). The chat allows participants to have a discussion in text format in a synchronous way in real time. Chats are especially useful when a group does not have the possibility to meet physically to talk face-to-face. The forum allows participants to have asynchronous discussions that take place over an extended period of time. The automatically created forum "News and announcement" indicates what is happening in the course, and it is only posted by teachers and an administrator. Additionally, the "Discussion forum" has been created, in which all users can add or reply to posts.

The main roles interacting with the system are the following:


#### *2.2. Didactic Surgical Videos*

Surgical videos are the basis of the didactic contents offered by Lapnurse, since they are very important and useful for nurses in online training processes and have increasingly been used for continuing medical education in e-learning technologies [13,14].

These are endoscopic videos corresponding to digestive, gynaecological, thoracic, and urologic surgical interventions. Video sections of interest to users have been selected from raw videos, avoiding long duration because the time away from work is the main drawback of health professionals who want training [15] and short contents are ideal for them [16]. A wide sample of up-to-date videos of each surgical specialty has been included to provide innovative information on advances in applied technology in line with the current surgical trend.

#### *2.3. Interactive 3D Designs*

New interactive 3D designs of laparoscopic instrumental and equipment have been developed to incorporate the interactivity demanded by nurses. They are PDF files with the 3D model (Figure 3), which allow the user to perform some interactive actions over the objects, such as rotate and move the model, zoom in or out, generate defined views, add comments, make 3D measurements, make section views, and select visualization of each component of the design, among others.

they will be considered an active participant within the professional network of the environment. They will be able to upload multimedia content, participate as a tutor solving doubts in forums and debates, and interact with other users in the commu-

Surgical videos are the basis of the didactic contents offered by Lapnurse, since they are very important and useful for nurses in online training processes and have increasingly been used for continuing medical education in e-learning technologies [13,14].

These are endoscopic videos corresponding to digestive, gynaecological, thoracic, and urologic surgical interventions. Video sections of interest to users have been selected from raw videos, avoiding long duration because the time away from work is the main drawback of health professionals who want training [15] and short contents are ideal for them [16]. A wide sample of up-to-date videos of each surgical specialty has been included to provide innovative information on advances in applied technology in line with

New interactive 3D designs of laparoscopic instrumental and equipment have been developed to incorporate the interactivity demanded by nurses. They are PDF files with the 3D model (Figure 3), which allow the user to perform some interactive actions over the objects, such as rotate and move the model, zoom in or out, generate defined views, add comments, make 3D measurements, make section views, and select visualization of

• Administrator: The administrator will normally be an expert in computer systems, responsible for all management and technical aspects of the environment. Among their tasks are user management, system monitoring, and technical assistance.

**Figure 3.** Interactive 3D design. **Figure 3.** Interactive 3D design.

nity.

*2.2. Didactic Surgical Videos* 

the current surgical trend.

*2.3. Interactive 3D Designs* 

each component of the design, among others.

All interactive 3D designs have been included in a Glossary within lesson 3. When any term of the glossary appears in the text of the course, that word is highlighted so that users can click it to visualize and interact with interactive 3D designs without having to visit the glossary. All interactive 3D designs have been included in a Glossary within lesson 3. When any term of the glossary appears in the text of the course, that word is highlighted so that users can click it to visualize and interact with interactive 3D designs without having to visit the glossary. *Healthcare* **2021**, *9*, x 6 of 14

#### *2.4. Preliminary Validation by Experts of the Online Theoretical Module 2.4. Preliminary Validation by Experts of the Online Theoretical Module*

A pilot study for a preliminary validation by experts of the online theoretical module of Lapnurse has been performed according to the scheme shown in Figure 4. A pilot study for a preliminary validation by experts of the online theoretical module of Lapnurse has been performed according to the scheme shown in Figure 4.

**Figure 4.** Scheme of the validation methodology. **Figure 4.** Scheme of the validation methodology.

**3. Results** 

Functionality, contents, and usability tests have been performed in order to get an e-learning platform of high quality [17]. Functionality validation checks the correct functioning of the system, as well as compliance with the functional requirements, through a checklist (see Appendix A). Content validation evaluates the suitability of the lessons and contents offered to meet the learning requirements. For that, a five-point Likert scale survey (1—completely disagree, to 5—completely agree) has been used to subjectively Functionality, contents, and usability tests have been performed in order to get an e-learning platform of high quality [17]. Functionality validation checks the correct functioning of the system, as well as compliance with the functional requirements, through a checklist (see Appendix A). Content validation evaluates the suitability of the lessons and contents offered to meet the learning requirements. For that, a five-point Likert scale survey (1—completely disagree, to 5—completely agree) has been used to subjectively evaluate

evaluate the specific content of each lesson and the global content of the course (see Appendix B). The minimum threshold for considering a validation as positive has been set

checklist with a list of requirements regarding the usability of the e-learning platform has been used to assess the design and layout of the web environment (see Appendix C).

centages with the software SPSS 15.0 for Windows (IBM, Armonk, NY, USA).

Two technicians with extensive experience in the development of distance training tools (functionality validation) and four nurses (two from Spain and two from Portugal) with teaching experience and advanced knowledge in MIS (content and usability validation) carried out such tests. Participants had no relationship with the course or its creators, and the non-probability convenience sampling technique has been used to conduct sampling of participants once the course was developed, where emails were sent to contacts whose data are stored in databases of partners involved in the study. Four to five participants are enough for detecting usability problems in a preliminary validation of a web environment [19], and a questionnaire is the most usual method [20–22]. Results were not statistically analysed looking for significant differences between Spanish and Portuguese participants because it is not appropriate in groups of two users each. The analysis of the data was carried out using mean values, standard deviations, and per-

The two technical evaluators verified that the e-learning platform meets all technical requirements related to access management, user profile and interactions, environment

maintenance, and formal and informal learning.

the specific content of each lesson and the global content of the course (see Appendix B). The minimum threshold for considering a validation as positive has been set to 3.5 out of 5 points in the Likert scale, according to 70% of agreement [18]. A "yes/no" checklist with a list of requirements regarding the usability of the e-learning platform has been used to assess the design and layout of the web environment (see Appendix C).

Two technicians with extensive experience in the development of distance training tools (functionality validation) and four nurses (two from Spain and two from Portugal) with teaching experience and advanced knowledge in MIS (content and usability validation) carried out such tests. Participants had no relationship with the course or its creators, and the non-probability convenience sampling technique has been used to conduct sampling of participants once the course was developed, where emails were sent to contacts whose data are stored in databases of partners involved in the study. Four to five participants are enough for detecting usability problems in a preliminary validation of a web environment [19], and a questionnaire is the most usual method [20–22]. Results were not statistically analysed looking for significant differences between Spanish and Portuguese participants because it is not appropriate in groups of two users each. The analysis of the data was carried out using mean values, standard deviations, and percentages with the software SPSS 15.0 for Windows (IBM, Armonk, NY, USA).

#### **3. Results**

The two technical evaluators verified that the e-learning platform meets all technical requirements related to access management, user profile and interactions, environment maintenance, and formal and informal learning.

Averaged results from the validation of specific contents of each lesson are shown in Tables 1 and 2, and of the global contents in Figure 5. In both cases, all issues far exceed the validation threshold, most of them reaching almost the highest score.


**Table 1.** Results from the validation of specific contents of lessons 1–5 (Mean values ± Standard deviation).



Lessons most important for experts are instrumental and equipment, ergonomics, and laparoscopy for nursing, and the least interesting are those related to diagnostic and approach techniques. Resources most important are lessons and didactic contents, while the least interesting are the means of communications between users (chat and forum).

Subjective usability results related to the design of the e-learning environment show a positive answer from all experts to all questions, as shown in Figure 6.

**Figure 5.** The blue continuous line represents the results of the validation of the global content of the course (1—Completely disagree, to 5—completely agree). The red dashed line indicates the minimum threshold necessary for positive validation. **Figure 5.** The blue continuous line represents the results of the validation of the global content of the course (1—Completely disagree, to 5—completely agree). The red dashed line indicates the minimum threshold necessary for positive validation. *Healthcare* **2021**, *9*, x 8 of 14

Averaged results from the validation of specific contents of each lesson are shown in Tables 1 and 2, and of the global contents in Figure 5. In both cases, all issues far exceed

**Table 1.** Results from the validation of specific contents of lessons 1–5 (Mean values ± Standard

**Question Lesson 1 Lesson 2 Lesson 3 Lesson 4 Lesson 5** 

**Q1** 4.75 ± 0.50 4.75 ± 0.50 5.00 ± 0.00 5.00 ± 0.00 5.00 ± 0.00 **Q2** 5.00 ± 0.00 5.00 ± 0.00 5.00 ± 0.00 4.75 ± 0.50 4.75 ± 0.50 **Q3** 5.00 ± 0.00 4.75 ± 0.50 4.50 ± 0.58 5.00 ± 0.00 4.50 ± 0.58 **Q4** 4.75 ± 0.50 5.00 ± 0.00 5.00 ± 0.00 5.00 ± 0.00 5.00 ± 0.00 **Q5** 4.25 ± 0.50 4.75 ± 0.50 4.00 ± 1.41 4.75 ± 0.50 4.75 ± 0.50

**Table 2.** Results from the validation of specific contents of lessons 6–9 (Mean values ± Standard

**Question Lesson 6 Lesson 7 Lesson 8 Lesson 9** 

**Q1** 4.75 ± 0.50 5.00 ± 0.00 5.00 ± 0.00 5.00 ± 0.00 **Q2** 4.75 ± 0.50 4.75 ± 0.50 4.50 ± 0.58 5.00 ± 0.00 **Q3** 4.25 ± 0.50 5.00 ± 0.00 5.00 ± 0.00 4.75 ± 0.50 **Q4** 5.00 ± 0.00 5.00 ± 0.00 5.00 ± 0.00 5.00 ± 0.00 **Q5** 4.75 ± 0.50 4.75 ± 0.50 4.75 ± 0.50 4.75 ± 0.50

**Mean ± SD Mean ± SD Mean ± SD Mean ± SD Mean ± SD** 

**Mean ± SD Mean ± SD Mean ± SD Mean ± SD** 

the validation threshold, most of them reaching almost the highest score.

deviation).

deviation).

#### **4. Discussion 4. Discussion**

MIS.

The traditional used method for MIS training consisting of the attendance of nurses to in-person courses requires incorporating the technological multimedia innovation according to the growth and evolution of this emerging surgical technique. For that, the laparoscopic training course for nursing Lapnurse through the blended learning method has been developed, with a face-to-face practical module and an online theoretical module. Findings obtained in this study regarding the successful design and preliminary validation by experts of the online module of the course suggest that Lapnurse could contribute to meet the training needs in MIS of Spanish and Portuguese nurses. In this way, they could be better able to work in the MIS field in their country or in another. Since the 1990s, changes in education policy, international relations, and the labor market The traditional used method for MIS training consisting of the attendance of nurses to in-person courses requires incorporating the technological multimedia innovation according to the growth and evolution of this emerging surgical technique. For that, the laparoscopic training course for nursing Lapnurse through the blended learning method has been developed, with a face-to-face practical module and an online theoretical module. Findings obtained in this study regarding the successful design and preliminary validation by experts of the online module of the course suggest that Lapnurse could contribute to meet the training needs in MIS of Spanish and Portuguese nurses. In this way, they could be better able to work in the MIS field in their country or in another. Since the 1990s, changes in education policy, international relations, and the labor market facilitated the migration of

facilitated the migration of nurses to work in other European countries. The recognition of qualifications under the OECD has facilitated such mobility of health workers [23]. In

Despite this high mobility rate, as far as we know, a standard curriculum of MIS nursing in Europe is still missing. Therefore, a new EU framework for nurses in MIS techniques should be developed. The content of courses such as Lapnurse should be applicable to European regulations so that such nurses trained in MIS could increase their employability in Europe and also contribute to the implantation of MIS at the European level.

Most studies about the validation of e-learning platforms are oriented to be performed by end users [17,25], but only few studies have been focused on instructors' satisfaction [26,27]. The instructor's point of view is often omitted but is also important [28], since the instructor is one of the main players of the learning process [29]. For that, in our work the validation tests have been performed by nurses with teaching experience in

One rather neglected issue in the validation tests is the reliability assessment of the e-learning environment. Due to its flexibility and ease of use, Moodle has been selected as the e-learning platform to provide the theoretical module of Lapnurse. It offers sufficient guarantees because it has been validated in several fields [30–33]. In this sense, system quality of an e-learning environment, in terms of navigability, accessibility, and stability [34,35], is critical for a good user experience of e-learning [36] and has a positive impact on use and satisfaction [37]. Therefore, we have successfully performed a functionality nurses to work in other European countries. The recognition of qualifications under the OECD has facilitated such mobility of health workers [23]. In 2016, 145,487 nurses lived in a member state other than their country of citizenship [24]. Despite this high mobility rate, as far as we know, a standard curriculum of MIS nursing in Europe is still missing. Therefore, a new EU framework for nurses in MIS techniques should be developed. The content of courses such as Lapnurse should be applicable to European regulations so that such nurses trained in MIS could increase their employability in Europe and also contribute to the implantation of MIS at the European level.

Most studies about the validation of e-learning platforms are oriented to be performed by end users [17,25], but only few studies have been focused on instructors' satisfaction [26,27]. The instructor's point of view is often omitted but is also important [28], since the instructor is one of the main players of the learning process [29]. For that, in our work the validation tests have been performed by nurses with teaching experience in MIS.

One rather neglected issue in the validation tests is the reliability assessment of the e-learning environment. Due to its flexibility and ease of use, Moodle has been selected as the e-learning platform to provide the theoretical module of Lapnurse. It offers sufficient guarantees because it has been validated in several fields [30–33]. In this sense, system quality of an e-learning environment, in terms of navigability, accessibility, and stability [34,35], is critical for a good user experience of e-learning [36] and has a positive impact on use and satisfaction [37]. Therefore, we have successfully performed a functionality evaluation from a technical point of view, which takes into account the possible interaction between user, content, and technology.

The positive validation of contents highlights that they are whole and complete, updated, and correctly applied for educational purposes. This agrees with Back [38], who indicates that medical e-learning environments should be designed with availability of all relevant contents, always up-to-date, and with relevance for practical clinical work. Participants value contents as unique and innovative, but without reaching a score as high as the other questions. This feature should be improved in future editions of the course because one of the key webpage factors to obtain good SERP (Search Engine Ranking Positions) rating score from a search engine is the quality of contents, in the sense of being original and unique [39].

Some e-learning scenarios only provide unidirectional information for online use or downloads [38]. However, the learning outcomes and the engagement of users in their learning process can be favorably influenced by integrating interactivity [40], thus avoiding user dropouts in the e-learning environment. In Lapnurse, interactivity has been included through 3D designs of the glossary, which have been valued by participants as useful and complete, with interactive, realistic, and good quality 3D objects. The innovative 3D tools offered by this course provide originality and suppose a novelty compared with other elearning scenarios. They would allow nurses to better know the instruments and equipment they have to manage during surgical interventions, since they could interact with them and not only see them in a simple picture or video as before in other online platforms.

Surgical videos and multimedia contents are important resources for the experts, agreeing thus with Bloomfield and Jones [41], which promotes video clips with clinical demonstrations as the most useful feature of e-learning due to being visual aids that can help students develop mental representations in their learning process. However, the incorporation of interactive components in surgical videos would be welcome, since interactive videos are being increasingly applied to skills-based training to overtake traditional passive watching of video-based training material [42].

User engagement with the e-learning platform can be achieved with usability features such as user-friendliness and ease of use [43]. Positive results from our usability validation show that the developed e-learning environment is attractive, organized, easy to use, and has a consistent design and layout with intuitive navigation. Despite this, ease of use has no influence on instructor satisfaction, but perceived usefulness and services quality on using the e-learning environment [29]. Participants have indicated that contents have

good quality, provide added value and useful information, and are correctly applied for educational purposes. Moreover, they consider that the e-learning platform includes useful tools for users, such as the glossary of 3D designs, chat, and forums.

The Lapnurse course could be used as a basis for creating similar courses for training other surgical staff involved in laparoscopic interventions, such as surgeons or anesthetists. There are contents and sections related to laparoscopy in general that are common to any staff, such as ergonomics, instruments and equipment (including the glossary of interactive 3D designs), and diagnostic and approach techniques. It would only be necessary to adapt to the target staff some specific sections, such as laparoscopy for nursing. Moreover, this course could be generalized to other surgical disciplines for which the surgical staff need training, such as endoscopy, microsurgery, cardiovascular, etc. For that, taking into account the successful validation of usability and functionality of the e-learning platform, the structure and layout could be the same, and only the content of the course should be adapted to the surgical discipline. However, nowadays mobile internet and smartphones are increasingly popular in nursing education and practice [44], so structure and layout of the course can change from a PC to a mobile phone. According to Back [38], medical elearning environments should have compatibility for all operating systems and unrestricted mobile access via tablet or smartphone.

There is a growing trend in the use of Massive Open Online Courses (MOOCs) in traditionally taught courses through this blended method [45]. But nevertheless, according to the definition of MOOC [46], maybe we cannot categorize our course as blended MOOC. First, a MOOC is massive, allowing the registration of thousands of participants. However, if we allow the registration of such a large number of participants in the online theoretical module of the course, it would be practically unfeasible to manage the practical module of the course, since this part is usually performed in specialized centres with capacity of 10–15 people each time. And second, a MOOC is open, in which any type of participants could register without any kind of restriction in prerequisite terms. However, Lapnurse is a specialization course of surgical staff, for which a minimum requirement of being a nurse is needed.

#### *4.1. Study Limitations*

More studies are needed to provide additional evidence from students at a national and international level performing a user-level validation, since only technicians and expert teachers have been involved in this preliminary validation.

#### *4.2. Comparison with Prior Work*

As far as we know, there are no similar blended learning laparoscopic courses specially addressed to nurses, so this original course with innovative 3D designs could improve their training in MIS as members of surgical teams involved in the interventions.

#### **5. Conclusions**

Although surgery is commonly present in training programs for nursing, Portuguese and Spanish nurses are inexperienced in MIS. Therefore, a laparoscopic training course for nursing has been developed through the blended learning method. The e-learning environment developed for the online theoretical module facilitates the accessibility to high quality didactic material to nurses in a remote and asynchronous way, allowing a greater implementation of the new MIS techniques in health systems.

Positive preliminary validation of Lapnurse could motivate developers to design similar courses. First, this kind of e-learning platform should preliminarily be validated to guarantee instructors' satisfaction so that contents offered to final users can meet learning requirements. Second, contents should be complete, updated, and relevant for clinical practice and educational purposes, but also unique and innovative to attract users and thus stand out from the contents of similar e-learning platforms. Additionally, interactivity should be included in the platform, and even in surgical videos, to favor learning outcomes and to get users' engagement. And, finally, reliability and quality of the system should be assessed for a good user experience. Such e-learning environments should be easy to use, attractive, and with intuitive navigation to favor users' engagement, but with useful and quality services to meet the instructors' satisfaction. Moreover, this kind of course should be developed for PC and mobile devices so that users can perform them anywhere, anytime, and on any device.

Successful preliminary validation of functionality, contents, and usability suggests that Lapnurse can solve the gap of training needs in MIS of both Spanish and Portuguese nurses. So, they could be properly qualified to work in this field in their own country, but also in the neighboring one. Since the obtained results were largely positive for valuing the e-learning environment and the blended learning method, it is a good starting point for considering in future studies the created course as a tool that could improve learning outcomes.

**Author Contributions:** Conceptualization, J.F.O.-M. and B.P.; methodology, J.F.O.-M., B.P., J.M.-A., J.S.-F., A.A. and F.M.; software, J.F.O.-M., B.P., J.M.-A., J.S.-F., A.A. and F.M.; validation, J.F.O.-M. and A.A.; formal analysis, J.F.O.-M. and A.A.; investigation, J.F.O.-M., B.P., A.A. and F.M.; writing original draft preparation, J.F.O.-M.; writing—review and editing, J.F.O.-M., B.P., J.M.-A., J.S.-F., A.A., F.M. and F.M.S.-M.; supervision, F.M.S.-M.; project administration, J.F.O.-M. and A.A.; funding acquisition, J.F.O.-M., B.P., A.A. and F.M.S.-M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This study was carried out under the MITTIC Project (0606\_MITTIC\_4\_E), Technological Modernisation and Innovation based on ICT in strategic and traditional sectors, financed jointly by the European Regional Development Fund (ERDF), through the Operational Program of Crossborder Cooperation Spain—Portugal (POCTEP) 2007–2013. The APC was funded by "Consejería de Economía, Ciencia y Agenda Digital, Junta de Extremadura" and co-funded by European Union (ERDF "A way to make Europe"). Grant number GR18199.

**Institutional Review Board Statement:** Ethical review and approval were waived for this study by the Ethical Committee of the Jesús Usón Minimally Invasive Surgery Centre because it was not within the scope of Law 14/2007 of 3rd July on Biomedical Research. This national regulation indicates that human health-related research involving invasive procedures needs approval by the ethical committee, but in this study the participants were not involved in invasive procedures.

**Informed Consent Statement:** Patient consent was waived for this study because it was not within the scope of Law 14/2007 of 3rd July on Biomedical Research. This national regulation indicates that human health-related research involving invasive procedures needs patient consent, but in this study the participants were not involved in invasive procedures. For this reason, it was not considered necessary for participants to sign a written informed consent. Only verbal informed consent was considered sufficient. All participants were informed about the objectives, length, and procedure of the study, the institutions responsible for the research, the contact people, the confidentiality and anonymity of the research data, and that their participation was voluntary, and they could withdraw from the study at any time if they wished to.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author.

**Acknowledgments:** The authors would like to gratefully acknowledge all technical and nursing experts from Spain and Portugal who willingly participated in the study.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **Appendix A**

This appendix includes the functional requirements evaluated:

• Access management requirements: user registration, login, logout, password recovery, and password change.


#### **Appendix B**

This appendix includes the questionnaire for content validation.

Note: This questionnaire has been translated into English for the readers of this article. The original one that was used during the validation process was in Spanish and Portuguese.

#### *Questionnaire for Content Validation*

Please read the different statements listed below concerning the specific content of each lesson of the theoretical module of the course and select the answer that best indicates the extent to which you agree with each of them, where 1 stands for "completely disagree" and 5 for "completely agree".


Please read the different statements listed below concerning the global content of the theoretical module of the course and select the answer that best indicates the extent to which you agree with each of them, where 1 stands for "completely disagree" and 5 for "completely agree".


Please answer the next two open questions:


#### **Appendix C**

This appendix includes the questionnaire for usability validation.

Note: This questionnaire has been translated into English for the readers of this article. The original one that was used during the validation process was in Spanish and Portuguese.

#### *Questionnaire for Usability Validation*

Please read the different statements concerning the usability of the e-learning environment listed below and indicate whether or not you agree with each of them


#### **References**


### *Article* **Analysis of the Content and Comprehensiveness of Dermatology Residency Training Websites in Taiwan**

**Po-Yu Chen 1,2, Ying-Xiu Dai 2,3, Ya-Chuan Hsu <sup>4</sup> and Tzeng-Ji Chen 1,2,\***


**Abstract:** With a growing trend in the popularity of web-based resources, it is important to evaluate residency program websites for providing accurate information for dermatology residency applicants. Little is known about the quality of dermatology residency websites in Taiwan. The aim of the study is to assesses the quality of official websites of dermatology training programs in Taiwan. A literature search for all related studies from inception to 31 July 2020 was performed using PubMed without restriction on language. We used criteria that had 6 domains and 25 items to evaluate 23 official websites of the dermatology training programs in Taiwan from August to September 2020. Of the 23 training programs, only 6 (26%) of the websites met more than half of the criteria. Notably, the items "features of the department" and "comprehensive faculty listing" were included in all websites. The criteria for interview process, board pass rates, social activities and information on the surrounding area were not met by all websites. Evidently, there is much room for improvement for the dermatology training program websites in Taiwan.

**Keywords:** dermatology; residency website; residency training

#### **1. Introduction**

#### *1.1. Health Information on the Internet*

In modern society, the Internet has gradually become more and more important for humans. According to statistics from the International Telecommunication Union, the percentage of Internet users worldwide increased from 16% in 2005 to 53.6% in 2019 [1]. In Taiwan in 2019, 92.78% of people used the Internet. It has become a convenient and generalized tool for acquiring information in Taiwan. A previous study reported that 80% of Internet users use the Internet to acquire healthcare information [2], and medical staff search it for information they need. Online resources for medical knowledge have become available to medical students and patients through computers and smart phones [3]. For physicians, clinicians and students, online resources represent an extensive source of continuing education in different specializations [4–6]. Patients can find several educational materials on the Internet, especially professional websites. In fact, a previous study showed that online patient education materials from the websites of major dermatology associations, such as the American Academy of Dermatology, was written well and appropriate for the general population [7]. Indeed, Internet has become a rather significant tool for physicians, medical students and patients.

#### *1.2. Taiwanese Residency Training Programs: Dermatology*

Since 2019, in Taiwan, medical students have been required to obtain a medical license after graduation and to finish postgraduate year training (PGY) within 2 years if

**Citation:** Chen, P.-Y.; Dai, Y.-X.; Hsu, Y.-C.; Chen, T.-J. Analysis of the Content and Comprehensiveness of Dermatology Residency Training Websites in Taiwan. *Healthcare* **2021**, *9*, 773. https://doi.org/10.3390/ healthcare9060773

Academic Editors: Luís Proença, José João Mendes, João Botelho and Vanessa Machado

Received: 26 May 2021 Accepted: 17 June 2021 Published: 21 June 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

they decide to receive advanced residency training [8]. Thereafter, they need to choose a specialization and complete residency training; for this purpose, they apply to several programs all over Taiwan, but many young doctors in the PGY-2 level are unfamiliar with the processes. Of all 23 specializations, dermatology is one of the most popular in Taiwan. The training period for dermatology residency is 4 years. Due to the advance of aesthetic medicine and the national health insurance system, there is tough competition to be accepted into a dermatology training program. Approximately 1400 medical students are able to obtain their medical license every year, but only about 28 positions per year are available for dermatology residency training. Even though this capacity can be adjusted by an agreement between the Taiwanese Dermatological Association and the Ministry of Health and Welfare, it is still much less than other specializations [9].

In the past, information on the training programs was difficult to obtain. Applicants often have to ask their senior residents or teachers for information or search the public information on the program websites. Applicants are less competitive without knowing people in dermatology departments. With widespread use, the Internet has become a common source of information for medical students applying for residency. Previous studies have indicated that the website content influenced an applicant's decision-making, especially for those who had never been engaged in the department [10]. Prospective applicants rely heavily on the websites of residency training programs to obtain significant information regarding the application process, as well as the unique aspects of the different programs [11]. Therefore, the importance of maintaining an informative and accessible website continues to grow. Many studies abroad have evaluated the website content of different specialties, including dermatology [12–18]. However, there had been no research that studied the contents of websites on residency training programs in Taiwan. In this study, standardized criteria were used to evaluate the websites of dermatology training programs in Taiwan. It is expected to provide dermatology programs information about how to enrich the content and display the strengths of their program to applicants. By enhancing their websites, the training programs also can attract more excellent applicants.

#### **2. Materials and Methods**

#### *2.1. Establishing the Criteria*

We searched for previous research that evaluated the websites of different residency or fellowship programs through the online literature database PubMed, using the keywords "Internet standards," "internship and residency," "fellowship and scholarship," and "information dissemination" from April to May 2020. We found 6 studies that performed questionnaire surveys to understand what content is important for applicants. After reevaluation, we developed a set of evaluation criteria with 25 content items categorized into 6 domains: recruitment information, department information, education and research, clinical work, incentives and frequently asked questions and answers (Table 1) [10,11,19–23]. The process of establishing the criteria is shown in Figure 1. The evaluation criteria were published previously in the *Taiwan Journal of Family Medicine* [23]. This is the first study to use these criteria to evaluate residency program websites in Taiwan.

#### *2.2. Internet Search for Taiwanese Dermatology Training Program Websites*

Thereafter, using the keywords "dermatology" and "training capacity," we performed Google search for the online official website of the Taiwanese Dermatological Association to check for the list of qualified dermatology training hospitals. We linked our search to the website of the Taiwan Ministry of Health and Welfare to review the publication list of qualified dermatology training hospitals and the training capacity after revision in July 2020 [9]. The number of items that met the criteria was calculated for each website. We conducted the evaluations twice with different reviewers. The first review was conducted by the first author around August 2020. A second review was conducted by the second author around September 2020, blinded to the results of the first review. Since this study did not involve human subjects, it did not need institutional review board approval.

**Figure 1.** Flowchart of establishing the criteria and evaluating the websites. **Figure 1.** Flowchart of establishing the criteria and evaluating the websites.

*2.2. Internet Search for Taiwanese Dermatology Training Program Websites*  **Table 1.** Criteria for evaluating the websites of residency and fellowship programs.


**Table 1.** *Cont.*


\* How to proceed with the recruitment (e.g., written tests, face to face interview, etc.). † The location of the department/hospital. § The number of inpatient cases or weekly outpatient cases per resident. || Independent office, independent rest room when on-call, etc.

#### **3. Results**

#### *Website Evaluation*

Overall, 23 dermatology residency training programs were approved by the Taiwan Ministry of Health and Welfare, and their official websites were evaluated in this study. The name of the training programs/departments and their website links are listed in Supplement Table S1. Based on the common administrative regional classification in Taiwan, the distribution of the training programs among the 23 hospitals was as follows: 56.6% (13 programs) in the northern area, 17.4% (4 programs) in the central area, 21.7% (5 programs) in the southern area and 4.3% (1 program) in the eastern area. Of the 23 hospitals, 39.1% (9 programs) were public and 60.9% (14 programs) were private.

In comparison with the results of 2 surveys, there was no difference. The final results are listed in Table 2. The websites met a mean and median of 13 of 25 items (52%) of the criteria, with a maximum of 18 items (72%) and a minimum of 4 items (16%). Over half of the criteria were met by 26% (6 of 23 programs) of the dermatology departments, of which 3 were at public hospitals and 3 were at private hospitals. However, the dermatology websites of Cathay General Hospital, Taipei City Hospital, Renai Branch and Heping Fuyou Branch met only four items (16%) of the criteria.

The distribution of the items on the websites is listed in Table 3 and Figure 2. Most of the websites (over 85%, 20 of 23 programs) contained information on the following 4 items: features of the department (100%, 23 programs); comprehensive faculty listing (100%, 23 programs); description of the environment and equipment (91.3%, 21 programs); and history of the department (87%, 20 programs). Notably, the items on interview process, board pass rates, social activities and information on the surrounding area were not posted on any dermatology website.

We further analyzed and compared the proportion of public and private programs/ departments that contained each item in our criteria. Public and private programs had the tendency to show similar information on most items (*p* > 0.05) in the 6 domains. These results are listed in detail in Table 3.


**Table 2.** Conditions of training program websites.

\* The hospitals were divided into four areas (i.e., northern, central, southern and eastern), according to location, Council for Economic Planning and Development, Executive Yuan. † The Taipei Municipal Wangfang Hospital; Shuang Ho Hospital, Ministry of Health and Welfare were regarded as public hospitals even if these were managed by the Taipei Medical University.

**Table 3.** Percentage of dermatology residency websites that contain publicly accessible information on the 25 list items.



**Figure 2.** Item distribution over all programs. **Figure 2.** Item distribution over all programs.

#### We further analyzed and compared the proportion of public and private pro-**4. Discussion**

sites [13,18].

grams/departments that contained each item in our criteria. Public and private programs *4.1. Analysis of Website Contents*

had the tendency to show similar information on most items (*p* > 0.05) in the 6 domains. These results are listed in detail in Table 3. **4. Discussion**  *4.1. Analysis of Website Contents*  Previous related studies, most of which were from the United States of America Previous related studies, most of which were from the United States of America (USA) and Canada, have been conducted to evaluate the websites of residency and fellowship specializations [14–20]. To our best knowledge, this study was the first to evaluate the content of dermatology residency program websites in Taiwan. Our study found highly variable content and quality among these websites. Most programs need improvements in the functioning of their webpages.

(USA) and Canada, have been conducted to evaluate the websites of residency and fellowship specializations [14–20]. To our best knowledge, this study was the first to evaluate the content of dermatology residency program websites in Taiwan. Our study found highly variable content and quality among these websites. Most programs need improvements in the functioning of their webpages. Among the criteria items, the most common information addressed were "features Among the criteria items, the most common information addressed were "features of the department" and "comprehensive faculty listing." Through this, people can learn more about the development of the department and choose which doctor they prefer. Applicants can also learn of senior role models for their medical careers. Over half of the websites included "list of current residents/fellows," but only 21.7% mentioned "alumni information and outcomes." This information could show close relationships between the

of the department" and "comprehensive faculty listing." Through this, people can learn

websites included "list of current residents/fellows," but only 21.7% mentioned "alumni information and outcomes." This information could show close relationships between the department and its staff, but it was not valued by most programs. The items of "program goals," "research activities and accomplishments" and "educational resources available to residents" were included in more than half of websites. Information on the research achievements and education resources allow applicants to compare different activities and resources. Patients and the general public can also improve their medical knowledge. Among the least common and poorly represented information were program-related specifics, such as "recruitment criteria," "message from the chairman/program director," "salary," "interview process," "board pass rates," "social activities," and "information on the surrounding area." In most websites, announcements on the "recruitment criteria" and "interview process" appeared only during the period of application; at other times, we could not find the relevant information. Simple statements on recruitment would encourage more excellent applicants to apply and provide them with an understanding of the process. Notably, "message from the chairman/program director" was often neglected among the websites evaluated in this study but had been more frequent in foreign webdepartment and its staff, but it was not valued by most programs. The items of "program goals," "research activities and accomplishments" and "educational resources available to residents" were included in more than half of websites. Information on the research achievements and education resources allow applicants to compare different activities and resources. Patients and the general public can also improve their medical knowledge.

Among the least common and poorly represented information were program-related specifics, such as "recruitment criteria," "message from the chairman/program director," "salary," "interview process," "board pass rates," "social activities," and "information on the surrounding area." In most websites, announcements on the "recruitment criteria" and "interview process" appeared only during the period of application; at other times, we could not find the relevant information. Simple statements on recruitment would encourage more excellent applicants to apply and provide them with an understanding of the process. Notably, "message from the chairman/program director" was often neglected among the websites evaluated in this study but had been more frequent in foreign websites [13,18].

Salary is an item that most people and organizations do not disclose to the public in Taiwan. Most applicants might presume that the directors would think that these issues are not directly related with clinical training: "if you ask about these, it means you are only interested in the money and benefits, and do not have a passion for the medical training." This observation had been reported in the USA and Taiwan [16].

Information about clinical work for residents was less clear on the internet. Information on "work hours," "expected caseload," and "on-call expectation" was provided in fewer than 10% of websites. Recently, the case loading and work hours of residency have been hot topics due to cases of death from overwork. Since residency doctors were included in the regulations of the 2019 Taiwanese Labor Standards Act, programs have been required to follow the Guideline of Residents Labor Rights Protection and Working Hours. Disclosure of this information is important for applicants [24].

#### *4.2. Comparison with Research from the USA*

Prior research includes one 2016 study about dermatology residency program websites in the USA [15]. We compared this with our study to understand the differences and found the following similarities and differences. Similarities with the USA study: program contact information, 83.5%; program description, 95.7%; research opportunities, 86.1%; current resident listing and 64.3%; current faculty listing, 89.6%. Our results were also all >50%. Other percentages >50% in the USA study were rotations and electives, 87.8%; applicant information, 88.7%; interview process, 64.3% (all >50%). In our study, these items were <40%. Most dermatology residency websites in the USA and Taiwan included content that described programs and listed faculty, residents and research accomplishments. However, Taiwanese websites rarely had information about recruitment, interviews and rotation. The disparity suggested much room for improvement in the website content in Taiwan.

#### *4.3. Limitations*

There were several limitations in our study. First, the criteria were according to foreign studies and may not be applicable to programs in Taiwan. In addition, the reliability and validity need to be verified further in the future. Second, our results represented a single snapshot in time. Owing to differences in navigability among the websites based on individual subjectivity, some information may have been overlooked during data collection. However, the likelihood that the differences in collection would change the nature of the presented data was low, even is this study was primarily descriptive in nature. Finally, our study focused only on the content of the websites and did not assess website design and quality. Each website has a unique design; therefore, quality can be difficult to compare. This aspect may be further analyzed in future.

#### **5. Conclusions**

Most of the dermatology websites from Taiwan did not contain sufficient information, especially recruitment, board pass rate, alumni, social activities, surrounding area, clinical work, salary, ancillary benefits and frequently asked questions and answers were especially neglected.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/article/10.339 0/healthcare9060773/s1, Table S1: Dermatology residency websites included in this study.

**Author Contributions:** Conceptualization, T.-J.C.; methodology, P.-Y.C. and Y.-C.H.; formal analysis, P.-Y.C. and Y.-X.D.; investigation, P.-Y.C. and Y.-X.D.; writing—original draft preparation, P.-Y.C. and Y.-X.D.; writing—review and editing, Y.-C.H. and T.-J.C.; supervision, T.-J.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** The study was supported by a grant (V109E-002-1) from Taipei Veterans General Hospital.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Data is contained within the article.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

