# **The Impact of Mobile Technology in the Battle against COVID-19 Successes and Failures**

Edited by Daniele Giansanti Printed Edition of the Special Issue Published in *Healthcare*

www.mdpi.com/journal/healthcare

## **The Impact of Mobile Technology in the Battle against COVID-19: Successes and Failures**

## **The Impact of Mobile Technology in the Battle against COVID-19: Successes and Failures**

Editor

**Daniele Giansanti**

MDPI Basel Beijing Wuhan Barcelona Belgrade Manchester Tokyo Cluj Tianjin

*Editor* Daniele Giansanti Centro Nazionale per le Tecnologie Innovative in Sanita Pubblica ` Istituto Superiore di Sanita` Rome Italy

*Editorial Office* MDPI St. Alban-Anlage 66 4052 Basel, Switzerland

This is a reprint of articles from the Special Issue published online in the open access journal *Healthcare* (ISSN 2227-9032) (available at: www.mdpi.com/journal/healthcare/special issues/ COVID Mobile).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. *Journal Name* **Year**, *Volume Number*, Page Range.

**ISBN 978-3-0365-4794-7 (Hbk) ISBN 978-3-0365-4793-0 (PDF)**

© 2022 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.

## **Contents**


Reprinted from: *Healthcare* **2022**, *10*, 235, doi:10.3390/healthcare10020235 . . . . . . . . . . . . . . **93**


Reprinted from: *Healthcare* **2021**, *9*, 371, doi:10.3390/healthcare9040371 . . . . . . . . . . . . . . . **165**

## **About the Editor**

#### **Daniele Giansanti**

Dr. Giansanti received an MD in Electronic Engineering at Sapienza University, 1991, Rome; a PHD in Telecommunications and Microelectronics Engineering at Tor Vergata University, 1997, Rome; and an Academic Specialization in Cognitive Psychology and Neural Networks at Sapienza University, Rome, 1997. His Academic Specialization was in Medical Physics, Sapienza University, Rome, 2005. Dr. Giansanti was in charge of the Design of VLSI Asics for DSP in the Civil Field (1991–1997) during his MD and PHD, and he served as a CAE-CAD-CAM system manager and Design Engineer in the project of electronic systems (Boards and VLSI) for the Warfare at Elettronica spa (1992–2000), one of the leaders in the military field. More importantly, he also conducts varied research at ISS (the Italian NIH) (2000–today) in the following fields:

1) Biomedical engineering and medical physics with the design and construction of wearable and portable devices (three national patents).

2) Telemedicine and e-Health: technology assessment and the integration of new systems in the field of the tele-rehabilitation, domiciliary monitoring, digital pathology, and digital radiology.

3) Mhealth: recent interest in the field of the integration of smartphones and tablet technology in health care with particular attention to the opportunities and the relevant problems of risks, abuse and regulation.

4) Acceptance of and consensus in the use of robots for assistance and rehabilitation.

5) Challenges and acceptance of the use of Artificial Intelligence in Digital Radiology and Digital Pathology.

6) Cybersecurity in the health domain.

7) Technologies for frail and disabled people.

Dr. Giansanti is a Professor at Sapienza and Catholic University in Rome and a tutor of theses. He is a Board Editor and reviewer for several journals. He has 129 indexed publications on Scopus and more than 200 contributions such as monographies and chapters to congresses.

## **Preface to "The Impact of Mobile Technology in the Battle against COVID-19: Successes and Failures"**

Mobile technology has undergone rapid development in the last decade and immediately found fertile ground for use in digital healthcare applications.

The advantages are many and interconnected:











The additional benefits for citizens and health systems in the period of the COVID-19 pandemic are:




There is a particular need for scholars to focus both on the innovations in this field during the pandemic and on the problems hampering the use of mobile technology to facilitate the correct and effective introduction of this technology into routine clinical programs in stable health care models. All professionals working in this sector were encouraged to contribute with their experiences. This reprint contains contributions from various experts and different fields. Aspects relating to the success and failures of employment, the medical experience, and acceptance are addressed. Particular space was also given to the role of social media, the use of apps (also presenting critical issues), and innovative apps for contact tracing. The digital divide and the infodemic were also investigated along with their impacts on citizens during the pandemic, for example, in following government directives relating to prevention and vaccination. We dedicate this reprint to all those involved with different roles in digital health.

> **Daniele Giansanti** *Editor*

## *Editorial* **The Role of the mHealth in the Fight against the Covid-19: Successes and Failures**

#### **Daniele Giansanti**

Centre Tisp, Istituto Superiore di Sanità, Via Regina Elena 299, 00161 Roma, Italy; daniele.giansanti@iss.it; Tel.: +39-06-4990-2701

#### **1. The Covid-19: A Pandemic Exploded during the Mobile Technology Era**

The spread of the SARS-CoV-2 coronavirus has pushed all affected countries to analyze all of the opportunities offered by current technology, which generated both a high number of solutions and a great debate on their actual ability to face the challenges promoted by pandemic spread. Previously, an epidemic caused by a coronavirus (SARS-CoV), the SARS (Severe Acute Respiratory Syndrome) of 2003, had been addressed. The epidemic lasted over a period of time from November 2002 to July 2003. During the previous pandemic, current mobile technologies were not available and in particular the smartphone as we know it today did not exist.

During each phase of the evolution of the pandemic, mobile technology (*mTech*) has played and is still playing an *important triple role*.

The *first role* is the traditional one played in the field of *digital health* [1–5] by connecting citizens to the health system and providing them with highly innovative technological solutions.

The *second role* is to support teaching, work and relational activities in an exceptional way, allowing social distancing between subjects, such as through messaging and/or video conferencing and/or social network tools [6,7].

The *third role* is specific to this pandemic and consists in providing *mHealth* solutions for controlling and monitoring the spread of the pandemic, such as through App-based solutions for the *digital contact tracing* [8,9].

The Covid-19 pandemic has hit every corner of the planet; however, the access to these technological solutions has not always been and still is not uniform due to the phenomenon called *digital divide* which depends on multifaceted aspects ranging from the lack of access to instrumental and network resources, to cultural and social barriers [10,11] and also to possible forms of communication disability.

#### **2. The Traditional Role of the** *mHealth* **in the Covid-19 Era**

During the pandemic, *mHealth*, which rests its foundations on *mTech*, continued to play its traditional role, in a more incisive and impressive way to increase the action of social distancing. In many realities we have gone from an *mHealth* used only in pilot experiences, and/or merely linked to research experiences, to an *mHealth* used in routine clinical applications regulated from every point of view [1] (including reimbursement). This last point has been reached in some cases also thanks to a rapid regulatory update, which often has provided for specific exceptions [2].

The advantages provided by *mHealth* [4,5] for the patient in the pandemic era are the following ones [3]:


**Citation:** Giansanti, D. The Role of the mHealth in the Fight against the Covid-19: Successes and Failures. *Healthcare* **2021**, *9*, 58. https://doi.org/10.3390/ healthcare9010058

Received: 25 December 2020 Accepted: 5 January 2021 Published: 8 January 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).


#### **3. The Generalized Support of the Mobile Technology**

During the pandemic there has been and currently there is an unprecedented use of Apps dedicated to communication for messaging/video. These Apps are mainly used [6,7] in the following sectors:


Often thanks to these technologies we have witnessed the development and proliferation of spontaneous networks which in fact have provided, where applied, real psychological support to avoid isolation and loneliness [7].

#### **4. The New Boundaries Explored by the** *mHealth*

During the pandemic, it was possible to explore a new potential of *mHealth* thanks to the availability of technological resources not accessible in the previous SARS pandemic. The key resource is definitely the *smartphone*. In fact, only in 2008 did we gradually witness the development of mobile technologies as we know them today, thanks to the *smartphone*, which has, compared to previous mobile technologies, some peculiar characteristics. The availability of this device was useful for creating *contact tracing* in *digital* form (*DCT*), widely used in epidemiology to allow monitoring and control of the spread of the pandemic. In general, the *smartphone* differs from the basic mobile phone due to the presence of the following features [4,5]:


Among the above listed *smartphone features* useful in the context of the *DCT* [8,9], we find in particular:


Virtually almost all governments on the planet have invested energy to build Appbased solutions for the *DCT*; some governments creating national Apps for *DCT*s (like Italy), other ones creating local regional Apps (like the USA); some nations carrying out a precise monitoring of the population, based on GPS, even on a mandatory basis; other nations using Bluetooth based protocols, with a purely voluntary membership, to ensure greater respect for privacy [8,9].

#### **5. The Obstacles Caused by the** *Digital Divide*

An important obstacle to a full and complete use of technologies in the ways described above has in some cases been represented by the *digital divide* [10,11] which is still mainly caused by the following problems:


#### **6. Conclusions**

The Covid-19 pandemic has created an unprecedented impetus for the development of *mHealth* [1–5]. This development involved both the enhancement and standardization of already consolidated solutions in *digital health* and the exploration of new potentials such as those of the *DCT* [8,9]. In many cases, the simple *mTech* itself has represented a *real lifebuoy* [6,7] both for the continuation of normal activities (working and teaching) and for providing a safety net. However, this has not always happened in a uniform way. The *digital divide* was a cause of this [10,11].

Making a map of these aspects, by means of scientific article contributions is important both to consolidate experiences and to ensure that they are not lost for the future and in particular for the post-pandemic era.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Behavioral and Emotional Changes One Year after the First Lockdown Induced by COVID-19 in a French Adult Population**

**Sylvie Rousset 1,\*, Aurélie Level <sup>2</sup> , Florine François <sup>1</sup> and Laurent Muller <sup>2</sup>**


**\*** Correspondence: sylvie.rousset@inrae.fr

**Abstract:** (1) Background: The lockdown had various consequences on physical activity and food consumption behaviors. The post-lockdown has been much less studied. The aim of this study is to compare behaviors one year after the first lockdown in a group of normal-weight (NW) or overweight French adults (OW). (2) Methods: Over a period of 4 days, both at the beginning of May 2020 (lockdown) and in June 2021 (free living post-lockdown), the same French adults used the WellBeNet smartphone application to record their sedentary behavior, physical activity (PA), food consumption and emotions. (3) Results: One year post first lockdown, the weight and body mass index increased (+1.1 kg; +0.4 kg.m−<sup>2</sup> , *p* < 0.01), and sedentary behaviors increased (+5.5%, *p* < 0.01) to the detriment of light-intensity activities (−3.3%, *p* = 0.10) in the whole group. Some food categories, such as alcohol, tended to be consumed more (+0.15 portion/day, *p* = 0.09), while fatty, salty and sugary products decreased (−0.25 portion/d, *p* = 0.02) but without a change in the food balance score. A higher number of both positive and negative emotions were scored per day (+9.5, *p* < 0.0001; +2.9, *p* = 0.03), and the positive ones were perceived stronger (+0.23, *p* = 0.09). Simultaneously, the desire to eat was lower (−11.6/100, *p* < 0.0001), and the desire to move remained constant. Sedentary/active behaviors and the desire to eat changed differently in NW and OW adults after the lockdown. (4) Conclusions: In general, the post-lockdown period was less favorable for physical activity practice and resulted in a similar food balance score but was more conducive to mental wellbeing.

**Keywords:** post-lockdown; sedentary behavior; physical activity; food choice; positive emotions; desire to eat; adult; COVID-19; smartphone

**1. Introduction**

The COVID-19 epidemic has forced many governments in the world to confine the population. In France, the first lockdown began in March 2020 and ended mid-May, with a travel ban and the closure of nonessential stores in order to limit human contact and, thus, the spread of the virus [1]. Outdoor activity was limited to a maximum of one hour and to within a radius of one kilometer around the home. In a CREDOC (Centre de Recherche pour l'Étude et l'Observation des Conditions de Vie (https://www.credoc.fr, accessed on 1 July 2021)) survey, 72% of respondents said they were worried about the risks of serious illness, and 63% said they were worried about COVID-19. This atmosphere of threat to health instills a feeling of insecurity on a daily basis [2]. The lockdown had an impact on lifestyle habits such as the physical activity (PA) level and eating behavior. Disease outbreaks often influence and change health-related behaviors.

Many studies in 2020 compared PA behaviors before and during the lockdown and showed contrasting results. A qualitative Canadian study showed that during the COVID-19 pandemic, people reacted differently: some of them maintained PA, others reduced PA

**Citation:** Rousset, S.; Level, A.; François, F.; Muller, L. Behavioral and Emotional Changes One Year after the First Lockdown Induced by COVID-19 in a French Adult Population. *Healthcare* **2022**, *10*, 1042. https://doi.org/10.3390/ healthcare10061042

Academic Editor: Daniele Giansanti

Received: 26 April 2022 Accepted: 31 May 2022 Published: 3 June 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

due to a lack of time or motivation and others reported they had more time to exercise [3]. In a New Zealand study, moderately pre-lockdown active individuals were significantly more active during and after the lockdown [4]. The reasons given to increase PA were to get outside during the lockdown by engaging in simple activities, such as running or cycling and to maintain physical and psychological wellbeing post-lockdown. In contrast, the pre-lockdown very active participants were less active during and after the lockdown. Their level of PA decreased because they were unable to practice their preferred PA under restrictions imposed by the lockdown, and they may lose their routine habits or develop other habits following changes in their life situation post-lockdown [4]. In Spain, a great reduction in physical activity was observed during the lockdown. It was the European country with the highest rate of inactivity because there was no space at home to carry out exercises [5]. However, post-lockdown, most of the subjects exercised two or three days a week [6].

It has been shown that quarantine and isolation can have effects not only on the level of PA but also on the consumption of unhealthy food and anxiety [7]. As regards diet, some people chose to eat more snacks and fatty, salty and sugary products to cope with depression and anxiety brought on by the epidemic. Other people chose to cook themselves food and consumed more fruit and vegetables during the lockdown than before. More recent studies investigated the post-lockdown period to determine whether behaviors modified by the lockdown living conditions were maintained. In an Australian study, the authors observed a lower quantity of physical activity, poorer mental wellbeing during than after the lockdown; moreover, junk food, soft drinks and alcohol were consumed more during the lockdown [8]. In a Chinese study, the authors compared dietary diversity during and after the lockdown [9,10]. To assess a diversity score, the authors measured the food consumption in 12 categories during both periods and assigned one point for each consumed food category. The mean dietary diversity score was high and similar during and after the lockdown (9.7 ± 2.1 vs. 9.2 ± 2.0). People were more likely to adopt a healthier diet, especially among those who were worried about contracting the virus [11]. People who consumed more raw food, seafood and imported frozen food had higher diversity scores than those whose consumption stayed the same or decreased [9,10]. However, some people adopted irrational behaviors, such as drinking alcohol or vinegar, to prevent COVID-19 during the lockdown, and this practice substantially decreased post-lockdown without disappearing completely. Although the Chinese government had already dismissed this rumor, more than 10% of the studied population was still purposely drinking more alcohol [9,10]. This behavior has also been reported in many other countries. In Italy, the participants reported an increase in healthy food, a decrease in junk food consumption and more time to cook during the lockdown [12]. These changes were stronger for young and restrained eaters. However, these new habits were partially discontinued post-lockdown; the participants consumed less healthy food and cooked less, but the reduction in junk food was maintained. In Spain, cooking at home is a usual habit, and most of the subjects did not order food at home in the post-confinement period. The consumptions of fruit and vegetables increased by 27% and 21%, respectively, in Spanish consumers during the lockdown compared to the pre-COVID-19 period [13] and stayed in an appropriate quantity after the lockdown. Depending on the country, the lockdown drastically changed life conditions, behavioral components and psychological states.

Our goal is to contribute to the literature that examines the impact of lockdowns on health. In particular, we examined whether exiting a period of lockdown improves an individual's eating habits, physical activity level and psychological states. Accordingly, we assessed dietary intake, physical activities and emotions of a group of French adults during and one year after the first lockdown in the spring of 2020.

In this context of craze for health applications, we chose to use a research mobile app to record behaviors and emotions. This innovative method to collect data is easy, accurate and immediate. First, the intensity of physical activities were recorded automatically by the native accelerometers of the mobile allowing non subjective and precise data. Second, the data were recorded in vivo. Respondents recorded their food consumption and emotions as they experienced them. This avoided the possible information loss that happens when questionnaires are filled out after the event.

#### **2. Materials and Methods**

#### *2.1. Experimental Design*

The volunteers downloaded the WellBeNet app at the Play Store (WellBeNet is not applicable to the iOS system. Even if Android represents approximately 80% of the market share, not taking iPhones into account could generate a sampling bias) on their own smartphone and followed the instructions directly from the app for three to five consecutive days, from 11 April to 7 May 2020, and after the lockdown, from 1 June to 3 July 2021. This observational study was conducted according to the guidelines of the Declaration of Helsinki. The protocol was approved by the French Committee for the Protection of Human Subjects (Sud-Est VI). It was registered under the references 2020/CE 19. All participants provided informed consent prior to participating in the study (https://activcollector.clermont.inra.fr/home/publications/InformationConsentGael, accessed on 1 April 2020).

#### *2.2. Measures*

After entering age, gender, height and weight into the app, volunteers were then asked to use the three WellBeNet features: eMouve, NutriQuantic and EmoSens.

eMouve provides an accurate estimation of time spent in four levels of activity: immobility, light, moderate, and vigorous intensity in the normal weight and overweight volunteers (NW and OW). These activity thresholds were determined in several previous publications [14,15]. The average absolute error of PA intensity estimation was approximately 3.25% compared to the reference methods [14,15]. Volunteers were asked to wear the smartphone in their pant pocket to collect accelerometry data during the waking period (8:00 AM to 10:00 PM). Time spent in each activity was expressed as a percentage of the whole recording time.

NutriQuantic collects the daily number of meals and portions consumed in each of the 11 food (the number of hot drinks was counted but was not associated with a nutritional score) groups divided in four meta-categories: plant products (fruit, vegetables, legumes and nuts), animal products (meat, fish, eggs and dairy products), junk food (fatty, salty and sugary products; snacks and alcohol) and starchy food (refined starchy and whole-grain starchy products). The content of each food category and the size of the portions are presented in the WellBeNet app and in a guide sent to each participant. A food balance score was calculated for each of the 11 food groups according to the number of portions, the French and international nutritional guidelines [16] and international recommendations [17]. The score in each food group varied among 0 (unsatisfactory), 0.5 (intermediate) and 1 (satisfactory). The nutritional balance score of the diet resulted from the sum of the scores obtained by the 11 food groups [18].

With EmoSens, the volunteers assessed their body image, physiological state (desire to eat and to move) and emotions in the morning, at midday and in the evening [19]. To do this, they selected one of the nine silhouettes defined by Stunkard et al. [20] and chose to color it in one of the following hues: orange, yellow, red, gray or white. The orange color is associated with fear and anxiety; red with desire, hate and passion; yellow with happiness, joy and love; gray with depression and sadness; and white with neutrality [21,22]. They then rated on 10-point scales either none, one or more of the 20 emotional terms from the Geneva Wheel [23,24]. Finally, volunteers scored their desire to eat and move on unstructured scales. The scores varied between 0 and 100. To evaluate the relative part of the desire to eat, the score of this one was divided by the sum of the two desires.

#### *2.3. Sample*

This observational study was conducted with 91 adult volunteers living in Grenoble and the surrounding area. A total of 72% were women, aged 38 years (±8 y), with no difference in age or weight status between the two sexes (Table 1). They were recruited by e-mail by the Grenoble Applied Economics Laboratory (GAEL), which complied with the French General Data Protection Regulation.

**Table 1.** Anthropometric characteristics by gender and weight status (Mean (SD)) at the beginning of the study.


During the lockdown, 25% of them were alone; the others were in a family or in a couple. Anthropometric characteristics are shown in Table 1. Height and weight are close to the average for French adults. A total of 63% of the volunteers had a normal body mass index (BMI), between 18.5 and 24.9 kg/m−<sup>2</sup> . A total of 7% were underweight (<18.5 kg·m−<sup>2</sup> ), and 30% were overweight or obese (BMI > 25 kg·m−<sup>2</sup> ). There were as many overweight people in our sample as in the French population aged 18–39 years (34%, [25]).

#### *2.4. Statistical Analyses*

The data analysis plan was determined prior to data collection to serve our purpose of comparing behavioral variables during and after the lockdown of normal-weight and overweight volunteers and of volunteers living alone and with family. All variables were tested for normality (Shapiro–Wilk test). For normally distributed variables (immobility and light- and moderate-intensity activities; daily number of meals and portions; whole starch, fruit, nut and dairy products; fatty, salty and sugary products; meat, fish and eggs; desire to eat and to move; and the number and mean of positive emotions and the mean of negative emotions), the differences between periods (lockdown/post-lockdown) were evaluated by a Student's paired t-tests for all volunteers and for each weight status (normalweight, NW and overweight, OW). Variations in other non-normally distributed variables were examined using Wilcoxon signed-rank test for paired responses. A MANOVA was also performed on the four variables of immobility, light-, moderate- and vigorous-intensity activity to determine the overall effect of body mass index group (BMI) on the variation in physical activity profile. Statistical analyses were carried out with SAS version 9.4 statistical analysis software.

#### **3. Results**

Table 2 presents the average changes in all variables considered by weight status. The table includes statistical tests for differences evaluated in the full sample and in NW and OW between the first lockdown period and the following year.

**Table 2.** Variations after the lockdown in anthropometry, physical activity, food eating and feelings in the full sample and by weight status. *S* is the value of the Wilcoxon signed-rank test; *t* is value of the paired Student's *t*-test, and *p* is the *p*-value. Means in bold are statistically significant at the 10% level.



**Table 2.** *Cont.*

#### *3.1. Evolution of Anthropometry*

We observed a generalized increase in anthropometric variables. The increase is statistically significant for the full sample of volunteers with an average increase of 1.1 kg in body weight and of 0.49 kg/m<sup>2</sup> in BMI. This trend is reflected in the NW population (+0.96 kg and 0.32 kg/m<sup>2</sup> ) and in the OW population (+1.50 kg and +0.53 kg/m<sup>2</sup> ), but it is not statistically significant for the latter.

Regarding family status, the volunteers with families gained weight (+1.2 kg, *p* = 0.04), whereas the increase was nonsignificant for those living alone (+1.0 kg, *p* = 0.13).

#### *3.2. Evolution of Physical Activities*

Inactivity times increased significantly after the lockdown (+5.5% for the full sample). This increase was mainly at the expense of light activity times, which decreased significantly

in the whole sample (−3.3%). However, the pattern of substitutions between activity types differed across the two BMI groups (Manova, F = 2.59, *p* = 0.04). Changes were rather more extreme in OW. First, the shift toward more immobility was more significant in OW (+6.7%) than in NW (+4.9%, Figure 1). Second, immobility times in OW replaced vigorous activity times to a greater extent than light activity times (−4.8% and −2.1%). On the contrary in NW, vigorous and light activity times decreased by −2.1% and −3.3%, respectively, post-lockdown. Finally, the moderate intensity activities remained unchanged for both BMI groups (−0.1%). types differed across the two BMI groups (Manova, F = 2.59, *p* = 0.04). Changes were rather more extreme in OW. First, the shift toward more immobility was more significant in OW (+6.7%) than in NW (+4.9%, Figure 1). Second, immobility times in OW replaced vigorous activity times to a greater extent than light activity times (−4.8% and −2.1%). On the contrary in NW, vigorous and light activity times decreased by −2.1% and −3.3%, respectively, post-lockdown. Finally, the moderate intensity activities remained unchanged for both BMI groups (−0.1%). Sedentary behaviors increased in volunteers living with families after the lockdown (+6.0%, *p* = 0.001). This was not the case in volunteers living alone (+3.9%, *p* = 0.40).

We observed a generalized increase in anthropometric variables. The increase is statistically significant for the full sample of volunteers with an average increase of 1.1

Regarding family status, the volunteers with families gained weight (+1.2 kg, p = 0.04), whereas the increase was nonsignificant for those living alone (+1.0 kg, p = 0.13).

Inactivity times increased significantly after the lockdown (+5.5% for the full sample). This increase was mainly at the expense of light activity times, which decreased significantly in the whole sample (−3.3%). However, the pattern of substitutions between activity

kg in body weight and of 0.49 kg/m² in BMI. This trend is reflected in the NW population (+0.96 kg and 0.32 kg/m2) and in the OW population (+1.50 kg and +0.53

*Healthcare* **2022**, *10*, x FOR PEER REVIEW 6 of 11

kg/m2), but it is not statistically significant for the latter.

*3.1. Evolution of Anthropometry* 

*3.2. Evolution of Physical Activities* 

**Figure 1***.* Variation in immobility and PA after lockdown in normal weight and overweight French adults. **Figure 1.** Variation in immobility and PA after lockdown in normal weight and overweight French adults.

*3.3. Evolution of Eating Behaviors*  Overall, eating behaviors remained remarkably stable between the two periods. That Sedentary behaviors increased in volunteers living with families after the lockdown (+6.0%, *p* = 0.001). This was not the case in volunteers living alone (+3.9%, *p* = 0.40).

#### is, the total number of meals, of servings and the food balance score all remained unchanged (Table 2). Only a few changes in the consumption per food categories were no-*3.3. Evolution of Eating Behaviors*

ticeable. One year after the lockdown period, OW consumed more plant products in general (+0.7 servings per day) and fruits in particular (+0.6). NW consumed fewer fatty, salty and sugary products (−2.2) and fewer starchy products (−1.7) but also more alcohol (+0.2). Vegetable, meat, fish and egg consumption after the lockdown was lower than during the lockdown in volunteers living alone (−0.4 serving/d, *p* = 0.07 and −0.2 serving/day, Overall, eating behaviors remained remarkably stable between the two periods. That is, the total number of meals, of servings and the food balance score all remained unchanged (Table 2). Only a few changes in the consumption per food categories were noticeable. One year after the lockdown period, OW consumed more plant products in general (+0.7 servings per day) and fruits in particular (+0.6). NW consumed fewer fatty, salty and sugary products (−2.2) and fewer starchy products (−1.7) but also more alcohol (+0.2).

Vegetable, meat, fish and egg consumption after the lockdown was lower than during the lockdown in volunteers living alone (−0.4 serving/d, *p* = 0.07 and −0.2 serving/day, *p* = 0.07), resulting in a lower food balance score (−1.9, *p* = 0.07). Volunteers with families consumed fewer fatty, salty and sugary products after the lockdown period (−0.3 serving/day, *p* = 0.01).

#### *3.4. Evolution of Emotions*

First, the desire to eat decreased dramatically, especially in NW. On the other hand, the desire to move remained stable, resulting in an improved balance between the desires to eat and to move (Table 2). Second, participants were happier one year after the lockdown: (i) the average ratings of positive emotions increased significantly; (ii) silhouettes were more frequently colored in yellow, which is associated with happiness, to the detriment of white (neutral) silhouettes; and (iii) participants reported on average nine more positive emotions than during the lockdown. Nevertheless, they also reported three more negative emotions on average and increased the use of red, usually associated with passion but also

hate. Third, the corpulence of the chosen silhouettes remained unchanged, except in NW who perceived themselves to be thinner one year after the lockdown.

Emotions perceived by the volunteers living alone were more positive after the lockdown as indicated by the increasing use of the color yellow (+18%, *p* = 0.06), the higher number and mean of positive emotions (+9, *p* = 0.006; +0.7, *p* = 0.04) and the lower mean of negative emotions (−0.6, *p* = 0.04). In volunteers living with families, both the positive and negative emotions were scored at a higher number (+10, *p* < 0.0001; +4, *p* = 0.01, respectively) but with no difference in the mean. The red color was also more used (+4%, *p* = 0.001). In both subgroups, the desire to eat was lower (−15.1, *p* = 0.03; −10.4, *p* = 0.002) after the lockdown period.

#### **4. Discussion**

Our results contradict those obtained in Spain and Australia that show less sedentary and more active behavior after the lockdown [6,8]. Travel and movement restrictions differed from one country to the next. In France, people could engage in outdoor physical activities for one hour a day, walk the dog and go shopping to the grocery store/supermarket. In Spain, on the other hand, only essential shopping was authorized during the lockdown. Outdoor physical activities were therefore made impossible, and the lack of suitable space at home led to sedentary behaviors [5]. The Spaniards seized the lifting of the lockdown as an opportunity to resume exercising [6]. Furthermore, the end of the first lockdown in France coincided with the return to work where telecommuting was strongly favored. Thus, in August 2021, 59% of French people stayed at home and telecommuted for two to four days a week, but with less free time to move and more time sitting in front of a screen [26].

How physical activity is measured may also explain the differences among studies. While the current literature almost exclusively uses questionnaires, physical activity here was derived from accelerometry data that accurately discriminate four activity categories to rank intensity level [14,15]. For instance, questionnaires cannot evaluate activities of light-intensity or short duration; they collect data that are subjective and approximate. This enabled us to distinguish between NW participants who primarily decreased lowintensity activities and overweight participants who instead rather decreased periods of vigorous activity. Other studies reporting objective measures of physical activities are scarce. Examples include obese adolescents in Maltoni et al. [27] and patients undergoing bariatric surgery in Andreu et al. [28].

Regarding food intake, the literature provides mixed results. While some studies report healthier eating habits (more fruit and vegetables and less junk food consumption) due in part to greater involvement in meal preparation during the lockdown [29,30], others report increased consumption of unhealthy food such as frozen pizza, cheese, sausage and potato chips in Deschasaux-Tanguy et al. and Nilsen [31,32] and snacks, cereals and sweets in Pellegrini et al. [33]. Our results are more in line with the latter observation, with greater consumption of fatty, sugary, salty and starchy food for NW and fewer plant products for OW during than after the lockdown. Nevertheless, the extent of these changes is not sufficient to generate a significant improvement in the nutritional balance score after the lockdown. A significant portion of our sample continued to telecommute part-time (49.3%) or full-time (29.6%) even after the lockdown. By staying home, they may have retained their lockdown eating habits.

In contrast to results from other countries [9,10,34], greater alcohol intake was observed post-lockdown. Alcohol consumption is an essential part of French culture and is associated with social interactions. In Guignard et al., one in five French drinkers reported lower consumption, and only one in ten reported higher consumption during the lockdown [35]. Restrictions placed on social interactions during the lockdown reduced opportunities to drink with acquaintances. These restrictions ceased with the reopening of bars and restaurants, and, thus, there were chances to share convivial moments with others and with alcohol.

In line with the literature, participants reported more of a desire to eat during the lockdown period than after. Gao et al. (2021) associated the increased desire of Chinese respondents to eat high calorie food during the lockdown with social media exposure [36]. Sanchez et al. (2021) identified anxiety as the top factor explaining stronger and more frequent hunger sensations during than before the lockdown for 74% of their Spanish respondents [37]. Previous studies showed that emotional eating may be associated with stressful life events [38] and anxiety [39]. Our French respondents instead appeared more emotional after the lockdown; they reported significantly more emotions, both positive and negative, and they reported significantly fewer neutral feelings. Moreover, they did not appear significantly less anxious after the lockdown. Therefore, as in the Andreu et al. (2022) study on obese patients [28], we found no evidence of more emotional eating during than after the lockdown.

Finally, we found that our participants gained weight between the two periods. Our anthropometric measurements were consistent with those showed by Zeigler [40]. Our data on emotions, nutrition and physical activity only partially explained this weight gain. First, anxiety, loneliness and boredom could lead to the consumption of palatable food and finally to weight gain [41]. As we have just seen, we did not find a decrease in these feelings after the lockdown. Nevertheless, this nondecrease did not imply a nonexistence. It could simply mean that the end of the lockdown did not make these emotions disappear, thus reviving the hypothesis of emotional eating, both during and after the lockdown. Second, respondents showed more desire to eat and consume larger quantities of starchy, fatty, salty and sugary products during the lockdown. However, these changes were not large enough to modify the food balance score. Finally, respondents were more active during the lockdown than after the lockdown, which might at first glance be at odds with the post-lockdown weight gain. Our second weight measurement occurred one year after the first lockdown. In the meantime, participants returned to work primarily through telecommuting, and two more additional lockdowns took place. Thus, the French continued to spend most of their time at home, preserving the emotional eating of the lockdown but with less free time to exercise.

Some differences in behaviors and emotions were noticeable between volunteers living alone and those living with families. The change in eating behavior during the post-lockdown period was detrimental to the health of those living alone (lower food balance score). The increase in post-lockdown sedentary behavior was detrimental in people with families. With regard to emotions, the post-lockdown period improved the condition of people living alone. Deprived of social relations during the lockdown, they may have suffered from a lack of contact. Social relationships can alleviate distress and anxiety [42]. They were associated with lower stress levels, lower worry about COVID-19 and less fatigue [43]. These findings show that social connections play a significant role in resilience by mitigating negative physical and mental health outcomes. This change in positive emotion between the two periods was less pronounced for those living with family. There was a higher number of negative emotions and a larger use of the color red. The post-lockdown period and the return-to-work could be a source of stress and worsening of the evening mood with more constrained schedules, such as an earlier and shorter sleep [44].

This work has several limitations. First, the results are contingent on a specific sample. Only volunteers who owned an Android smartphone were eligible to participate. Those who had an iPhone were not selected. Fortunately, they are few in number: only 20% of people had an iPhone. Another bias relates to the location of our sample and the unbalanced gender size. Our sample included more women exclusively from the Grenoble area. Second, the results may be subjected to a reporting bias. Whereas physical activities are automatically recorded by the WellBeNet application, diets and emotions were selfreported. This could lead to oversights, misperceptions and, more generally to a reporting bias. To our knowledge, there are currently no measures based on actual observation of

diets and emotions under such ecological conditions. Moreover, examining differences rather than absolute levels mitigates such reporting problems.

The strength of the study was the use of a smartphone app designed for and developed by the Human Nutrition Research team of INRAE. Time spent in sedentary behavior and physical activity was accurately measured, and the food servings could be recorded immediately after meals. The emotions could also be recorded in real time. All of this data were recorded into the same app. As the volunteers used their own smartphone, there was no risk of contagion between the volunteers and researchers.

#### **5. Conclusions**

This study investigated the effects of leaving the lockdown. The WellBeNet smartphone app provided data on anthropometric measurements, food intake, physical activities and perceived emotions of 91 adults during and one year after the French first lockdown. Body weight and BMI increased between the two periods. Participants were more sedentary and engaged in less light-intensity activity after the lockdown. The composition of diets changed little with fewer fatty, salty and sugary products and less starchy food but more alcohol in the post-lockdown period. Overweight participants also consumed more fruit. The desire to eat was less intense than during the lockdown. Finally, participants perceived a higher number of emotions, both positive (to a larger extent) and negative (to a lesser extent).

This work contributes to anticipating the negative aspects of a lockdown through preventive actions during a new or upcoming COVID-19 epidemic. Prevention action should address overeating as a reaction to stress or boredom. The population should be educated about the dangers of increased consumption of fatty, salty and sugary food and a decreased consumption of fruit. These unhealthy variations during the lockdowns seem to be unconscious or estimated to be without health damage. Finally, people living alone during a lockdown should receive extra support as they appear to be at greater emotional risk.

**Author Contributions:** Conceptualization, S.R. and L.M.; Formal analysis, L.M.; Funding acquisition, A.L. and L.M.; Investigation, A.L. and F.F.; Methodology, S.R., A.L. and F.F.; Project administration, S.R. and L.M.; Resources, S.R. and L.M.; Supervision, F.F. and L.M.; Validation, S.R. and F.F.; Visualization, A.L. and F.F.; Writing—original draft, S.R. and F.F.; Writing—review and editing, S.R., A.L., F.F. and L.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the French Research and Innovation Ministry. This funding agency had no role in the design, analysis, or writing of this article.

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Helsinki and approved by the Ethics Committee Sud-Est VI (protocol code 2020/CE 19, 15 April 2020).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author.

**Acknowledgments:** The authors thank the volunteers who participated in this study.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **The Impact of Using mHealth Apps on Improving Public Health Satisfaction during the COVID-19 Pandemic: A Digital Content Value Chain Perspective**

**Junwei Cao <sup>1</sup> , Guihua Zhang 2,\* and Dong Liu <sup>3</sup>**


**Abstract:** The use of mobile technology and equipment has been found to be successful in the governance of public health. In the context of the coronavirus disease 2019 (COVID-19) pandemic, mobile health (mhealth) apps are expected to play an important role in the governance of public health. This study establishes a structural equation model based on the digital content value chain framework, identifies the main values created by mhealth apps in the prevention and control of COVID-19, and surveys 500 citizens of China. The data were analyzed using an independent *t*-test and partial least squares structural equations (PLS-SEM). The results showed that people who use mhealth apps are more satisfied with public health governance than those who do not; the healthcare assurance value of mhealth apps and healthcare confidence positively influence the interaction between users and mhealth app functions, the interaction with information, and the interaction with doctors to improve users' satisfaction with public health governance; and the parasocial relationships between doctors and users of mhealth apps positively affect the interactions between users and doctors to improve users' satisfaction with public health governance. This study confirms the potential of mhealth apps toward improving public health governance during the COVID-19 pandemic from a new perspective and provides a new theoretical basis whereby mobile technology can contribute toward improving public health governance.

**Keywords:** mobile health app; public health; digital content value chain; COVID-19

#### **1. Introduction**

Public health governance aims to prevent diseases, extend life, and promote health through organized social efforts, and it focuses on the impact of social conditions on health, such as the health system, social conditions, and the link between inequality and poor health [1]. The emergence of coronavirus disease 2019 (COVID-19) has compelled the world to invest a substantial amount of anti-epidemic materials, which has exacerbated the pressure on public health and medical resources. It even affects daily medical care [2]. Meanwhile, to reduce the risk of cross-infection, people are reluctant to visit medical institutions, even when they have health problems. This also complicates public health governance [3].

The use of mobile technology and mobile devices in public health governance has been proven successful [4,5]. Medical and public health services that are provided to the public through mobile phones, patient testing equipment, personal digital assistants, and other wireless devices are referred to as mobile health (mhealth) [6]. Currently, mhealth mainly provides services based on the form of a smartphone app [7,8]. The emergence of mhealth apps has changed the supply mode of health services and brought about benefits for both healthcare providers and recipients [9]. On the one hand, doctors use mhealth

**Citation:** Cao, J.; Zhang, G.; Liu, D. The Impact of Using mHealth Apps on Improving Public Health Satisfaction during the COVID-19 Pandemic: A Digital Content Value Chain Perspective. *Healthcare* **2022**, *10*, 479. https://doi.org/10.3390/ healthcare10030479

Academic Editor: Daniele Giansanti

Received: 18 January 2022 Accepted: 1 March 2022 Published: 4 March 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

apps to process patient information and monitor patient health [10]. On the other hand, individuals use mhealth apps to obtain health information for immediate diagnosis [11].

Mobile health apps digitize traditional healthcare services and provide users with healthcare services via the Internet. The role of mhealth apps in public health management during the COVID-19 pandemic reflects the impact of their digital content (DC) value on society. Mobile health apps are expected to play an important role in the COVID-19 pandemic [12]. Therefore, many studies have indicated the important functional value of using mhealth apps for COVID-19 [2,3,12]. Presently, the COVID-19 epidemic recurs, thus overwhelming public health services, and thus, the pressure on public health governance remains at momentarily high levels [1], and the use of the mhealth apps allows patients to easily obtain health information and receive medical care, thus reducing the frequency of patient visits to the hospital and minimizing population mobility in areas of high risk [2–4]. Mobile health apps effectively promote information exchange, storage, and delivery, and they improve the ability of patients to monitor and respond to diseases [12–15]. They can also be used for training [16,17], information sharing [18–20], risk assessment [18], symptom self-management [16], contact tracking [18], family monitoring [21], and decisionmaking [3] during the COVID-19 pandemic.

Mhealth is a digital platform that deserves to be valued not only for its functional value but also for its emotional and social value [22]. In terms of emotional value, mHealth apps have proven to be an effective way to deliver mental health services [23–26]. A mobile medical app can significantly reduce stress and significantly improve people's well-being by identifying emotional states and reducing loneliness [25]. In particular, the use of mHealth apps strengthens users' resilience and makes it easier for them to get out of a difficult situation and maintain a positive mental state [26]. In terms of social value, the development of mHealth can alleviate the shortage of medical resources to a certain extent [27], improve the quality of medical services in remote areas and for vulnerable groups, and helps to maintain social harmony [28].

A review of the relevant literature revealed that (1) the current research has made many arguments for the functional value of the mhealth app in resisting COVID-19. However, there is a lack of research on its emotional and social values, especially for COVID-19, and the specific connotations manifested in these value dimensions of mhealth are even less clear. (2) The mechanisms of the impact of these values of the mhealth app on public health governance are also unclear. Therefore, the research questions of this study are presented as follows.

RQ1: What values do mhealth apps mainly create during the COVID-19 pandemic?

RQ2: How do the values created by mhealth apps improve public health governance during the COVID-19 pandemic?

To answer these questions, this study adopts the DC value chain framework combined with the relevant literature to construct a research model to analyze the value transfer of mhealth apps during the COVID-19 pandemic.

#### **2. Theoretical Background and Hypothesis Development**

*2.1. Digital Content Value Chain Framework*

With the rapid development of information technology, an increasing amount of traditional content is converted into DC for delivery [22]. Traditional content delivers physical value through physical value streams, while DC delivers digital value through DC value streams [29].

Digital content value streams refer to the entire DC process, from generation to use in a computer-mediated network. Digital content value streams encompass three stages: DC value creation, DC value interaction, and DC use [22,29,30]. Digital content promotes interactions between users and DC by creating new values. Ultimately, users create new values through DC use [22].

To clarify the relationship between the various parts of the DC value stream, Kim and Kim [22] proposed a DC value chain framework based on the DC value stream. As shown in Figure 1, in the DC value creation stage, the DC system, DC, and DC users dominate, thereby creating the functional, emotional, and social DC values. In the DC value interaction stage, the user interacts with the system, content, and members because of the DC value and enters the user stage through the intermediary effect of interaction. Users create a new value based on the value given by the DC (such as performance improvement and satisfaction with the product). The DC value chain framework analyzes the DC platform from the user's perspective, can better understand the value transfer process in a specific DC platform, and more intuitively analyzes the user's perception of the DC value. dominate, thereby creating the functional, emotional, and social DC values. In the DC value interaction stage, the user interacts with the system, content, and members because of the DC value and enters the user stage through the intermediary effect of interaction. Users create a new value based on the value given by the DC (such as performance improvement and satisfaction with the product). The DC value chain framework analyzes the DC platform from the user's perspective, can better understand the value transfer process in a specific DC platform, and more intuitively analyzes the user's perception of the DC value.

To clarify the relationship between the various parts of the DC value stream, Kim and Kim [22] proposed a DC value chain framework based on the DC value stream. As shown in Figure 1, in the DC value creation stage, the DC system, DC, and DC users

*Healthcare* **2022**, *10*, x FOR PEER REVIEW 3 of 19

**Figure 1.** Digital content value chain framework. **Figure 1.** Digital content value chain framework.

Mobile health apps digitize traditional healthcare services and provide users with healthcare services and medical knowledge through the Internet. They have developed from the earliest apps that can only provide a single service to DC-providing platforms for multiple services, such as appointment registration, online diagnosis, drug purchase, and health knowledge search. Mobile health apps can monitor the user's physical data and provide users with medical advice and instant online medical services, thereby alleviating the pressure on medical resources. These apps can also relieve users' anxiety about diseases by providing corresponding health management information [31,32], as well as functional, emotional, and social values. Users gain value and create new value through interactions with mhealth app system functions, health information interactions, and member interactions. Therefore, it is essential to analyze the value delivery of mhealth apps during the COVID-19 pandemic based on the DC value chain framework Mobile health apps digitize traditional healthcare services and provide users with healthcare services and medical knowledge through the Internet. They have developed from the earliest apps that can only provide a single service to DC-providing platforms for multiple services, such as appointment registration, online diagnosis, drug purchase, and health knowledge search. Mobile health apps can monitor the user's physical data and provide users with medical advice and instant online medical services, thereby alleviating the pressure on medical resources. These apps can also relieve users' anxiety about diseases by providing corresponding health management information [31,32], as well as functional, emotional, and social values. Users gain value and create new value through interactions with mhealth app system functions, health information interactions, and member interactions. Therefore, it is essential to analyze the value delivery of mhealth apps during the COVID-19 pandemic based on the DC value chain framework perspective.

#### perspective. *2.2. Main Values of mHealth App during the COVID-19 Pandemic*

*2.2. Main Values of mHealth App during the COVID-19 Pandemic*  The most important functional value created by mhealth apps during the COVID-19 pandemic is the provision of effective medical protection. Mobile health apps were rather unpopular before the emergence of COVID-19 [33]. However, with the increasing prevalence of COVID-19, the public health department has been encouraging patients to avoid using face-to-face medical services as much as possible to prevent cross-infection and effectively use limited public health resources [3,34]. Therefore, many countries have begun to use mhealth apps on a large scale to provide consultation, monitoring, and care services for patients [3]. Mobile health apps allow for the exchange of two-way data between patients and healthcare personnel to realize remote medical consultation, psychological consultation, health education, and obtain medical protection. It meets The most important functional value created by mhealth apps during the COVID-19 pandemic is the provision of effective medical protection. Mobile health apps were rather unpopular before the emergence of COVID-19 [33]. However, with the increasing prevalence of COVID-19, the public health department has been encouraging patients to avoid using face-to-face medical services as much as possible to prevent cross-infection and effectively use limited public health resources [3,34]. Therefore, many countries have begun to use mhealth apps on a large scale to provide consultation, monitoring, and care services for patients [3]. Mobile health apps allow for the exchange of two-way data between patients and healthcare personnel to realize remote medical consultation, psychological consultation, health education, and obtain medical protection. It meets users' utilitarian medical needs [9,35]. Satisfaction with utilitarian needs can positively affect user intentions [36,37]. Users must meet utilitarian medical needs through frequent interactions with mhealth apps; therefore, we propose the following hypotheses:

**H1a.** *The healthcare assurance capabilities of mhealth apps have positively affected the interactions between users and the health functions of mhealth apps during the COVID-19 pandemic.*

**H1b.** *The healthcare assurance capabilities of mhealth apps have positively affected the interactions between users and the health information of mhealth apps during the COVID-19 pandemic.*

**H1c.** *The healthcare assurance capabilities of mhealth apps have positively affected the interactions between users and mhealth app doctors during the COVID-19 pandemic.*

The most important emotional value created by mhealth apps is confidence. It has been confirmed that the ability of mhealth apps to give users confidence is an important dimension in evaluating its quality, and it positively affects users' satisfaction with mhealth apps and their continued use intentions during the COVID-19 pandemic [38,39]. The COVID-19 pandemic can cause psychological problems. In particular, there has been no specific medicine for the treatment of new coronary pneumonia, which is more likely to cause depression, anxiety, insomnia, and other negative emotions [40]. As a health information platform, mhealth apps can deliver positive health information and provide users with psychological intervention [41,42], for example, enlightening people about the pathogenesis of COVID-19 and elucidating the epidemic prevention dynamics of the government and related organizations to reduce users' doubts and give users confidence.

Confidence has also been proven to be an important indicator for evaluating the quality of mhealth apps and user interactions [39], and it significantly predicts user behavior by triggering positive emotions [43]. During the COVID-19 pandemic, the information in mhealth apps can boost the user's confidence, the user's evaluation of the interactive quality of mhealth apps may be improved, and users tend to interact with mhealth apps more. Therefore, this study proposes the following hypotheses:

**H2a.** *During the COVID-19 pandemic, the healthcare confidence-giving value of mhealth apps has positively affected the interactions between users and the health functions of mhealth apps.*

**H2b.** *During the COVID-19 pandemic, the healthcare confidence-giving value of mhealth apps has positively affected the interactions between users and mhealth apps' health information.*

**H2c.** *During the COVID-19 pandemic, the healthcare confidence-giving value of mhealth apps has positively affected the interactions between users and mhealth app doctors.*

During the COVID-19 pandemic, the social value of mhealth apps has manifested at the level of a positive doctor–patient relationship [44]. In face-to-face diagnosis and treatment, patients usually feel pressure, because they are passive. In telemedicine, patients feel that they have the initiative, which reduces the pressure to visit a doctor [45]. Several studies have proven that medical services in an information network environment are more patient-centric, which allows patients and doctors to collaborate better and improve their mutual satisfaction [45].

The active doctor–patient relationship in the mhealth app platform is that of the society criterion. Parasocial relationships refer to the emotional bonds formed between the audience and the media characters. It is a one-way relationship. Parasocial relationships improve the user's recognition of the media and increase the user's participation [46,47]. In the mhealth app, doctors are the "media people" on the platform. The platform displays their personal information to the users. Users can choose their favorite doctors according to their preferences and establish one-way connections (user–function interaction and user–information interaction). After the diagnosis is complete, the doctor will be unable to actively communicate with the user. This process is completely dominated by the user. The user participates in the interaction with his or her own positive imagination of the selected doctor, which can easily form parasocial relationships [48]. Second, the health data monitoring function of the mhealth app can provide more accurate health information

(user–function interaction and user–information interaction) when users communicate with doctors [49,50]. A study confirmed that 80% of doctors are satisfied when patients show digital health information [51]. Finally, accurate health information provided by patients can improve a doctor's diagnosis and treatment performance [44]. Consequently, patients' satisfaction with doctors improves, and they are more willing to use the various interactive mechanisms of the mhealth app platform (user–function interaction, user– information interaction, and user–doctor interaction). Therefore, this study proposes the following hypotheses:

**H3a.** *During the COVID-19 pandemic, the parasocial relationship between doctors and patients in mhealth apps has positively affected the interactions between users and the health functions of mhealth apps.*

**H3b.** *During the COVID-19 pandemic, the parasocial relationship between doctors and patients in mhealth apps has positively affected the interactions between users and the health information of mhealth apps.*

**H3c.** *During the COVID-19 pandemic, the parasocial relationship between doctors and patients in mhealth apps has positively affected the interactions between users and mhealth app doctors.*

The interaction between mhealth apps and users transfers the value to the users' satisfaction with public health governance. The information systems success theory points out that the use of information systems and satisfaction will interact, which will eventually affect individuals or organizations and generate net benefits [52,53]. The digital value chain framework also confirms that increasing the interaction between the system and the user, between the content and the user, and between the user and the user can play a key role in improving users' satisfaction, process efficiency, product quality, and use efficiency [22,29,54].

During the COVID-19 pandemic, users have solved their health concerns by continuously using the health service function of mhealth apps and improved their confidence in health management, thereby enhancing the interactions between doctors and patients, as well as forming a good doctor–patient relationship. The value created by mhealth apps through interactions may alleviate the medical pressure of patients during the epidemic and improve patients' satisfaction with public health governance. Therefore, this study proposes the following hypotheses:

**H4a.** *During the COVID-19 pandemic, the interaction between users and the health functions of mhealth apps positively affected users' satisfaction with public health governance.*

**H4b.** *During the COVID-19 pandemic, the interaction between users and the health information of mhealth apps has positively affected users' satisfaction with public health governance.*

**H4c.** *During the COVID-19 pandemic, the interaction between users and mhealth app doctors has positively affected users' satisfaction with public health governance.*

#### **3. Research Model and Questionnaire Survey**

*3.1. Digital Content-Value Chain Framework*

This study proposes a research model of the mhealth app value chain during the COVID-19 pandemic (Figure 2). The model shows that an mhealth app transfers the value created by itself to public health governance through interactions with users. Mobile health apps do not directly affect public health governance during the COVID-19 pandemic as an auxiliary medical mobile phone app, but they can provide users with functional (healthcare assurance), emotional (patient healthcare confidence), and social (patient–patient relationship) values that promote the interactions between users and mhealth apps, meet

medical needs, increase confidence in health management, and establish a harmonious doctor–patient relationship, thereby increasing users' satisfaction with the COVID-19 public health governance. mhealth apps, meet medical needs, increase confidence in health management, and establish a harmonious doctor–patient relationship, thereby increasing users' satisfaction with the COVID-19 public health governance.

created by itself to public health governance through interactions with users. Mobile health apps do not directly affect public health governance during the COVID-19 pandemic as an auxiliary medical mobile phone app, but they can provide users with functional (healthcare assurance), emotional (patient healthcare confidence), and social (patient–patient relationship) values that promote the interactions between users and

*Healthcare* **2022**, *10*, x FOR PEER REVIEW 6 of 19

**Figure 2.** Proposed research model. **Figure 2.** Proposed research model.

#### *3.2. Questionnaire Survey 3.2. Questionnaire Survey*

The questionnaire is designed according to the current research and the COVID-19 background. The questionnaire uses a 5-point Likert scale based on existing research and combined with the actual situations of the current research. All the questions are compulsory. If a question is unanswered, the questionnaire cannot be submitted. After the questionnaire is completed, we invite experts in the field of management information systems to investigate it, and 60 undergraduates are also invited to test it. The final questionnaire is presented in Appendix A. The questionnaire is designed according to the current research and the COVID-19 background. The questionnaire uses a 5-point Likert scale based on existing research and combined with the actual situations of the current research. All the questions are compulsory. If a question is unanswered, the questionnaire cannot be submitted. After the questionnaire is completed, we invite experts in the field of management information systems to investigate it, and 60 undergraduates are also invited to test it. The final questionnaire is presented in Appendix A.

Before we give out the questionnaires, we consult the school ethics committee to ensure that there are no ethical issues in the questionnaire. All the participants are informed of the following information: (1) The questionnaire is innominate. (2) The content and purpose of the questionnaire. (3) You have the right to answer or not answer. Before we give out the questionnaires, we consult the school ethics committee to ensure that there are no ethical issues in the questionnaire. All the participants are informed of the following information: (1) The questionnaire is innominate. (2) The content and purpose of the questionnaire. (3) You have the right to answer or not answer. (4) No private information involved. (5) After completing the questionnaire, you will receive a gift.

(4) No private information involved. (5) After completing the questionnaire, you will receive a gift. A serious epidemic broke out in Yangzhou, China from 28 July to 30 August 2021. The epidemic prevention measures implemented by the government made people stay at home, which provided a good analysis environment for this study. This study employed an online questionnaire in Yangzhou from 20 August 2021 to 30 August 2021. By using the snowballing survey method, we randomly recruited 100 users who used mhealth apps during this period to conduct a survey and asked them to send the questionnaire to their friends. If the people who were investigated choose "mhealth app was not used in the epidemic", they skipped the questions about mhealth and answered the questions about public health satisfaction. Finally, we received a total of 581 questionnaires after we eliminated 93 unqualified and invalid answers (e.g., answer time less than two minutes; more than 70% of the answers were the same), while the effective questionnaires are sorted out. Among them, 316 people used mhealth app health services, information search, online diagnosis, and other functions during this period and answered all the A serious epidemic broke out in Yangzhou, China from 28 July to 30 August 2021. The epidemic prevention measures implemented by the government made people stay at home, which provided a good analysis environment for this study. This study employed an online questionnaire in Yangzhou from 20 August 2021 to 30 August 2021. By using the snowballing survey method, we randomly recruited 100 users who used mhealth apps during this period to conduct a survey and asked them to send the questionnaire to their friends. If the people who were investigated choose "mhealth app was not used in the epidemic", they skipped the questions about mhealth and answered the questions about public health satisfaction. Finally, we received a total of 581 questionnaires after we eliminated 93 unqualified and invalid answers (e.g., answer time less than two minutes; more than 70% of the answers were the same), while the effective questionnaires are sorted out. Among them, 316 people used mhealth app health services, information search, online diagnosis, and other functions during this period and answered all the questions. A total of 172 people said they had not used mhealth apps during this period and only answered the questions on public health satisfaction.

questions. A total of 172 people said they had not used mhealth apps during this period and only answered the questions on public health satisfaction. This study conducted a necessary demographic survey of people who have used mhealth apps (Table 1). Among the 316 people who had used these apps, 128 were male This study conducted a necessary demographic survey of people who have used mhealth apps (Table 1). Among the 316 people who had used these apps, 128 were male (40.5%) and 188 were female (59.5%). The proportion of people aged 31–40 was the highest (N = 87, 27.5%), followed by people aged 21–30 (N = 84, 26.6%). Among all the respondents, 140 (44.3%) had no higher education, 154 (48.7%) had a bachelor's degree, and 22 (7%) had a master's or doctoral degree. The monthly income of most respondents ranged from 2001 to 3000 yuan (USD 295–440) (N = 80, 25.3%), followed by those with a monthly income of 3001–4000 yuan (USD 441–558) (N = 71, 22.5%). In terms of app brands, 74 people used

Age

Income (Per month)

Education

mHealth app Brand

"Pingan Health", 70 people used "Chunyu Doctor", 66 people used "Dinxiang Doctor", and remaining 106 people reported using other mhealth apps (e.g., Hao Doctors). These apps have some commonalities, as shown in Figure 3. They are all integrated apps with many functions, and one app can meet multiple healthcare needs. These functions include ecommerce, health knowledge search, service reservation, health consultation, task-specific processing, medical service review, etc.


**Table 1.** Demographic details of the survey respondents.

**Figure 3.** The main functions of respondents' commonly used mhealth apps. **Figure 3.** The main functions of respondents' commonly used mhealth apps.

**Table 1.** Demographic details of the survey respondents. **Items Options Frequency (Total = 316) Percentage (%)**  To avoid a nonresponse bias, a paired samples *t*-test was conducted for the top 20 and bottom 20 respondents who submitted the questionnaire. The results showed no significant differences between the two groups.

> 18–20 55 17.4 21–30 84 26.6 31–40 87 27.5 41–50 44 13.9 51 years or above 46 14.6

RMB 1000–2000 47 14.9 RMB 2001–3000 80 25.3 RMB 3001–4000 71 22.5 RMB 4001–5000 68 21.5 More than RMB 5000 50 15.8

High School 140 44.3 Bachelor's Degree 154 48.7 Master or PhD Degree 22 7

Pingan Health App 74 23.4 Chunyu Doctor App 70 22.2 Dinxiang Doctor App 66 20.9

Other 106 33.5

#### **4. Methods**

There are two types of structural equation models (SEMs)—one is based on covariance (CB-SEM), and the other is based on variance (VB-SEM). In this study, the VB-SEM partial least squares SEM (PLS-SEM) and the corresponding software package (Smartpls3.0) were used. PLS-SEM is a second-generation multivariate data analysis method that is mainly used to carry out exploratory theoretical research. This method can ensure the integrity of all relationships between independent and dependent variables [55]. Compared with CB-SEM, (1) PLS-SEM is more suitable for models with more than six variables [56]; (2) PLS-SEM is good for processing small sample data [56]; and (3) PLS-SEM can process non-normally distributed data [56]. In summary, the PLS-SEM method is more suitable than the CB-SEM in the theoretical development stage, and it has been shown that PLS-SEM can replace the CB-SEM in most social science research cases [56] and is widely used in social, economic, and business research [57,58].

A multivariate normality analysis was performed on the data using a network calculator (http://www.biosoft.hacettepe.edu.tr/MVN/, accessed on 3 January 2022) [59]. The results of the multivariate normality analysis showed that Mardia's multivariate skewness β = 75.184, *p* < 0.01 and multivariate kurtosis β = 788.253, *p* > 0.05, which suggest multivariate non-normality [60]. In addition, there are seven variables in this study. Therefore, it is suitable to use PLS-SEM for data analysis in this study.

#### **5. Results**

#### *5.1. Pretest Results*

This study measures the impact of the value created by mhealth apps on public health governance during the COVID-19 pandemic through the user's satisfaction with public health governance after using mhealth apps. Therefore, it is necessary to survey people who have or have not used mhealth apps to determine whether there is a difference in the satisfaction of the population to public health governance. If there is no difference, it means that the value transfer created by mhealth apps does not exist, and there is no need for further analysis.

This study compares the satisfaction with public health governance of 316 people who have used mhealth apps and 172 people who have not through an independent sample *t*-test. The results are presented in Table 2. The average satisfaction of users who have used mhealth apps regarding public health governance during the COVID-19 pandemic is higher than that of those who have not used it, and the difference is significant.

**Table 2.** Independent *t*-test results.


#### *5.2. Common Method Bias Test Results*

Common method bias is a problem that can easily appear in the questionnaire. Harman's single-factor analysis is widely used to detect common method deviations in social science research [61]. This method indicates that a single factor can be extracted. If the variance is less than 40%, it means that the survey data are less affected by the deviation of the commonly used methods [62]. The Harman data analysis conducted in this research shows that the ratio of extracted variables is 30.49% (less than 40%).

We further used the full variance inflation factor (VIF) test method to conduct a common method bias test on the data. Some studies have pointed out that all variables and dump variables in the model are subjected to the full VIF test. If the VIF value is greater than 3.3, the model may be affected by common method bias. If all the VIFs obtained by the full VIF test are equal to or lower than 3.3, it can be considered that the model has no common method bias [57,63]. This has been widely applied in various studies [60]. The full VIF test results of this study show that all the VIF values are less than 3.3. We consider the test results of the two common method bias methods. Therefore, the authors believe that common method bias is not a serious issue in this study.

#### *5.3. Measurement Model Results*

First, the composite reliability is used to evaluate internal consistency reliability. As Table 3 shows, the composite reliability (CR) value of each structure is greater than 0.7, and Cronbach's α is also greater than 0.7, indicating that the questionnaire items have high reliability [64]. This study confirms convergent validity by evaluating the average variance extracted (AVE). When the AVE value is higher than 0.5, it is considered that the standard for convergence validity is met. In our model, the AVEs are all higher than 0.5, with the lowest value being 0.648, indicating that the scale has good convergence validity [64].


**Table 3.** Measurement model results.

Abbreviations: HAC (healthcare assurance capacity); ACO (healthcare confidence); PSR (parasocial relationships); UFI (user–function interaction); UII (user–information interaction); UDI (user–doctor interaction); SPH (satisfaction with public health).

Second, this study tests the discriminating validity using the heterotrait–monotrait ratio (HTMT) test. The results are shown in Table 4, where the value between the variables meets the requirement of less than 0.85 [64]. This study continues testing discriminating validity through the Fornell–Larcker criterion test (comparing the square root of AVE with the correlation coefficient), and the square root of each variable is greater than the correlation coefficient with other variables, which meets the test requirements in this study (see Table 5). In addition, as shown in Table 3, the factor loading of all items in this study is higher than 0.7, with the lowest value being 0.750, which meets the requirements of the threshold standard [64]. These results indicate that the discriminative validity of the scale in this study meets these requirements [64].

**Table 4.** Heterotrait–monotrait ratio (HTMT) test results.


Abbreviations: HAC (healthcare assurance capacity); ACO (healthcare confidence); PSR (parasocial relationships); UFI (user–function interaction); UII (user–information interaction); UDI (user–doctor interaction); SPH (satisfaction with public health).

**Table 5.** Fornell–Larcker criterion test results.


Abbreviations: HAC (healthcare assurance capacity); ACO (healthcare confidence); PSR (parasocial relationships); UFI (user–function interaction); UII (user–information interaction); UDI (user–doctor interaction); SPH (satisfaction with public health).

Third, we tested the goodness of fit of the model. The fit degree is calculated by the square root of the product of the R<sup>2</sup> mean of the communality, while the result of the goodness of fit must be higher than 0.1. If it is higher than 0.36, it indicates a high fitness (the medium and low fitness ranges are 0.25 to 0.36 and 0.1 to 0.25, respectively). The goodness of fit of the model is 0.383 in this study, according to the measurements and calculations, and this indicates that the goodness of fit of the model is very high [65]. Standardized root mean square residuals (SRMRs) are also a standard for model fitting. When the SRMR value is 0, it indicates that it is perfect. However, it is recommended that an SRMR value less than 0.08 is taken as a suitable fitting threshold for PLS-SEM [60]. The SRMR value of the model in this study is 0.069, which meets the threshold value. The goodness of fit of the model is suitable according to the two tests.

Finally, collinearity problems are tested, and the VIFs between all the variables are lower than 5. Therefore, this signifies that there are no collinearity problems in this study.

#### *5.4. Structural Model Results Healthcare* **2022**, *10*, x FOR PEER REVIEW 11 of 19

We consider the overall explanatory power, R<sup>2</sup> , and path coefficient of the structural model to test the research model.

As shown in Figure 4 and Table 6, healthcare assurance capacity has a significant positive effect on user–function interaction (ß = 0.323, *p* < 0.001), user–information interaction (ß = 0.248, *p* < 0.001), and user–doctor interaction (ß = 0.207, *p* < 0.001) impacts; thus, H1a, H1b, and H1c are supported. As shown in Figure 4 and Table 6, healthcare assurance capacity has a significant positive effect on user–function interaction (ß = 0.323, *p* < 0.001), user–information interaction (ß = 0.248, *p* < 0.001), and user–doctor interaction (ß = 0.207, *p* < 0.001) impacts; thus, H1a, H1b, and H1c are supported.

**Figure 4.** Test results of the structural model test. Note: \*\*\* *p* < 0.001 and \*\* *p* < 0.05. **Figure 4.** Test results of the structural model test. Note: \*\*\* *p* < 0.001 and \*\* *p* < 0.05.


Healthcare confidence has a significant positive effect on user–function interactions **Table 6.** Hypothesis testing results.

H1c: HAC →UDI 0.207 0.054 3.817 0.000 Support H2a: ACO → UFI 0.428 0.043 9.933 0.000 Support Abbreviations: HAC (healthcare assurance capacity); ACO (healthcare confidence); PSR (parasocial relationships); UFI (user–function interaction); UII (user–information interaction); UDI (user–doctor interaction); SPH (satisfaction with public health).

H2b: ACO → UII 0.373 0.05 7.513 0.000 Support H2c: ACO → UDI 0.211 0.057 3.705 0.000 Support H3a: PSR → UFI 0.041 0.05 0.819 0.413 Reject Healthcare confidence has a significant positive effect on user–function interactions (ß = 0.428, *p* < 0.001), user–information interactions (ß = 0.373, *p* < 0.001), and user–doctor interactions (ß = 0.211, *p* < 0.001); thus, H2a, H2b, and H2c are supported.

H3b: PSR → UII 0.051 0.055 0.931 0.352 Reject H3c: PSR → UDI 0.270 0.056 4.814 0.000 Support H4a: UFI → SPH 0.359 0.052 6.874 0.000 Support Parasocial relationships have no significant positive effects on user–function interactions (ß = 0.041, *p* > 0.05) and user–information interactions (ß = 0.051, *p* > 0.05); thus, H3a and H3b are not supported. Parasocial relationships have a significant positive impact on user–doctor interactions (ß = 0.270, *p* < 0.001); thus, H3c is supported.

H4b: UII →SPH 0.120 0.049 2.456 0.014 Support H4c: UDI → SPH 0.316 0.052 6.051 0.000 Support Abbreviations: HAC (healthcare assurance capacity); ACO (healthcare confidence); PSR (parasocial User–function interactions have a significant positive impact on satisfaction with public health; thus, H4a is supported. User–information interactions have a significant positive impact on satisfaction with public health (ß = 0.120, *p* < 0.05); thus, H4b is supported.

relationships); UFI (user–function interaction); UII (user–information interaction); UDI (user–doctor

User–doctor interactions have a significant positive impact on satisfaction with public health (ß = 0.316, *p* < 0.001); thus, H4c is supported.

#### *5.5. Mediation Effect Results*

The interactions among the user, content, and system links values are created by DC with useful values. The analysis of the value flow of DC must analyze the flow process from the value created by DC to DC use [22]. Therefore, it is necessary to perform an additional analysis to test whether the interaction between users, content, and the system has a mediation effect during the flow of a value created by mhealth apps to use value.

This study analyzes the mediating role of the model using SmartPls 3.0. As shown in Table 7, the interactions between users and doctors, information, and features in mhealth apps mediates the interactions between the healthcare coverage created by mhealth apps and users' satisfaction with public health governance. The interactions between users and doctors, information, and features in mhealth apps mediate between the confidence in health created by mhealth apps and users' satisfaction with public health governance. Interactions between users and doctors in mhealth apps mediate between both the value of parasocial relationships created by mhealth apps and users' satisfaction with public health governance. However, the interactions between users and information and features in mhealth apps do not mediate between the value of the parasocial relationship created by mhealth apps and users' satisfaction with public health governance.


**Table 7.** Mediation effect results.

#### **6. Discussion and Implications**

#### *6.1. Discussion of Key Findings*

The healthcare assurance value created by mhealth apps had a positive impact on user–function interactions, user–information interactions, and user–doctor interactions in this study, which verifies that the functional value of the DC proposed by Kim and Kim [22] promotes and expands user–system interactions. This means that the healthcare assurance value created by mhealth apps for users during the COVID-19 pandemic also actively promotes the interactions between users and information and doctors in mhealth apps. This is because mhealth apps are complicated and comprehensive health management apps under the current circumstances. Moreover, it is included in several services, such as health monitoring, appointment registration, online diagnosis, drug purchase, health knowledge search, and so on [18–20]. To better complete healthcare assurance, each function interacts with the users. Especially in the case of insufficient medical resources and limited travel during the COVID-19 pandemic, it needs to be completed, from health monitoring (user–function interactions) to online diagnosis (user–doctor interactions), self-health management (user–information interactions) through mhealth apps. It also confirms that the healthcare assurance value created by mhealth apps enables users to improve their satisfaction with public health governance under the intermediary role of

user–function interactions, user–information interactions, and user–doctor interactions in this study. This also shows that users apply mhealth apps (DC–value interactions) to convert the DC creation value obtained from mhealth apps to their satisfaction with public health governance during the COVID-19 pandemic. It also promotes the interactions between mhealth apps and users and contributes to mhealth apps playing a greater role in healthcare assurance in public health governance during the COVID-19 pandemic.

Healthcare confidence value has a positive impact on user–function interactions, user–information interactions, and user–doctor interactions. The results prove [18–20] the emotional value of DC to facilitate user–content interactions and enlarge the relationships in the study. This refers to the value of mhealth apps in giving users confidence in health management during the COVID-19 pandemic while actively promoting the functions of users and mhealth apps and the interactions between doctors. Confidence improves the quality of interactions between users and mhealth apps and promotes user participation [33]. Users are confident in managing their health by using mhealth apps during the COVID-19 pandemic, so they will try their best to interact with mhealth apps. It also confirms that, under the mediation effect of these interactions, the confidence value created by mhealth apps enables users to improve their satisfaction with public health governance in the study. The results show that users convert healthcare confidence gained from mhealth apps to their satisfaction with public health governance during the COVID-19 pandemic (DC use values). Improving the various interactive experiences between mhealth apps and users will help mhealth apps play a greater role in public health governance during the COVID-19 pandemic.

In this study, the parasocial relationship values created by mhealth apps and the user–doctor interaction relationship are significant. This result verified that the social value of DC promotes user–user interactions in the Kim and Kim [22] study. This signified that the parasocial relationship can be conducive to reduce the inherent prejudice of users and improve the emotional attachment for doctors [46,47]. With the help of the parasocial relationship, users have a positive impression of doctors and tend to be satisfied with the doctor's treatment. Finally, it can facilitate interactions between doctors and users. This study demonstrated that, under the mediation effect of user–doctor interactions, the parasocial relationship created by mhealth apps improves users' satisfaction with public health governance. This means that users will convert the parasocial relationship attained by mhealth apps into satisfaction with public health governance during the COVID-19 pandemic. Improving the interactive experience between users and mhealth apps will help mhealth apps play a greater social role in improving doctor–patient relationships in public health governance during the COVID-19 pandemic.

However, the impact of the parasocial relationship values created by mhealth apps on user–function interactions and user–information interactions has not been verified in this study. User–function interactions and user–information interactions have no mediating role between the parasocial relationship value and public health governess satisfaction. A possible explanation is that the parasocial relationship will lead to the emotional attachment of users. After the parasocial relationship between the user and the doctor is established, the user is more likely to rely on the doctor, and he or she is unwilling to use the monitoring and information management functions in mhealth apps when he or she has health problems.

#### *6.2. Theoretical Contribution*

Previous studies have pointed out various functional values of mhealth apps for public health governance during the COVID-19 pandemic [16,18–20]. However, this study not only verified the mhealth apps' functional value from an empirical perspective but also demonstrated the emotional and societal values for public health governance during the COVID-19 pandemic, and it clarified that, after the functional, emotional, and social values are created in mhealth apps through user–function interactions and user– information interactions, user–doctor interactions flow until satisfaction with public health

governance. This provides a new view for further research on the effect of mhealth on public health governance.

In the original DC value chain framework, it confirms the relationships of functional values for user–system interactions, emotional values for user–content interactions, and social values for user–user interactions. These values flow to the value that users have used in a single way [22]. Through an empirical study of value creation via mhealth apps, this study confirmed that, in the DC value chain, the value created by DC is related to various interactions of users, and the value created can flow into the post-use value through different forms of interactions. In addition, this study further refines the functional, emotional, and social values in the DC value chain framework in specific contexts, thus expanding the dimensions of value creation in DC and enriching the antecedents that influence the interactions between users and DC. The research results of this study enrich the DC value chain framework and expand the application scope of the DC value chain framework.

#### *6.3. Practical Contribution*

The results of this study can provide some useful practical suggestions for the management of public health during the COVID-19 pandemic.

First, it enriches mhealth apps and expands the scope of application to guarantee the basic medical needs of users. In COVID-19, recommendation algorithms can be cleverly used to give the most appropriate results and analyze suggestions based on the questions submitted by users and suggest and associate with related functions, such as online prescriptions, face-to-face consultation appointments, etc. This can contain the users' basic medical needs by constant interactions, which can improve their satisfaction with public health governance during COVID-19, and it is beneficial to stabilize public health order.

Second, it is necessary to provide users with healthcare confidence during the COVID-19 pandemic. This requires the public health governance department to formulate effective epidemic prevention measures in line with public opinion and pass it onto society through the efficient information distribution function of mhealth apps. Mhealth app-operating companies need to push more information about professional hospitals, public welfare organizations, and emergency hotlines to convince users that they can still get effective help when they require it urgently, even though the epidemic has affected public health order. Mobile health apps can also be embedded with interactive games (such as psychometric tests, luck predictions, etc.) to monitor changes in users' psychological states and allow users to share positive results with friends and family to convey positive emotions.

Third, it is necessary to establish a harmonious doctor–patient relationship in the governance of public health during the COVID-19 pandemic. We suggest that the mobile app in COVID-19 needs to establish a public relations governance department to take up as much social responsibility as possible to instantly reconcile the disputes that arise in the platform. It needs to be humane when dealing with problems, taking care of the feelings of any party as much as possible, weakening conflicts, emphasizing mutual understanding, and creating an atmosphere in the platform where patients trust doctors and doctors take care of patients. Therefore, the user will be satisfied with the doctor, so the doctors also will gain self-professional identity. In this way, a benign doctor–patient relationship cycle is formed, and the governance of public health during the COVID-19 pandemic is improved.

#### *6.4. Limitations and Future Research*

This study has some limitations. First, this study is conducted in a city that implemented strict epidemic prevention measures (restricted travel) during the epidemic. Such results may be unsuitable for cities that implement general epidemic prevention measures. Meanwhile, many countries have begun to implement policies that coexist with COVID-19. There are no strict epidemic prevention measures; therefore, the results of this study may be unsuitable for these countries. It is necessary to compare studies with different epidemic prevention measures in future studies. Second, PLS-SEM is effective for dealing with small samples [64], but there will inevitably be representative problems due to the small sample

size. Future studies should consider other methods (such as big data analysis) to analyze the value transfer process of mhealth apps during the COVID-19 pandemic. Third, the three most important specific values are listed in the study, and some value dimensions may be ignored. Therefore, further studies are necessary. Fourth, this study did not measure the effect of the length of time using the mhealth app on the value chain transmission in the COVID-19 epidemic, and future studies are necessary to analyze the effect of time of use. Fifth, since most of the mhealth apps on the Chinese market are integrated with multiple functions, the findings of this study cannot explain the value transfer of disease-specific and function-specific mhealth in COVID-19, and further research is needed. Finally, this study refers to the constant user–mhealth interaction that may not be beneficial in some studies, especially because it has a lot to do with age [66], but these problems in the effect of the mhealth app value chain are ignored, and it is suggested for future research that multigroup analyses are based on these situations.

**Author Contributions:** Conceptualization, J.C. and G.Z.; methodology, J.C. and G.Z.; formal analysis, J.C.; investigation, D.L.; writing—original draft preparation, J.C.; writing—review and editing, J.C. and G.Z.; supervision, D.L.; and project administration, D.L. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available upon request from the corresponding author. The data are not publicly available for ethical reasons.

**Conflicts of Interest:** The authors declare no conflict of interest.


#### **Appendix A**


#### **References**


#### *Comment*

## **The** *Accessibility* **and the** *Digital Divide* **in the Apps during the COVID-19. Comment on Cao et al. The Impact of Using mHealth Apps on Improving Public Health Satisfaction during the COVID-19 Pandemic: A Digital Content Value Chain Perspective.** *Healthcare* **2022,** *10***, 479**

**Daniele Giansanti \*, Antonia Pirrera and Paola Meli**

Centre Tisp, The Italian National Institute of Health, 00161 Rome, Italy; antonia.pirrera@iss.it (A.P.); paola.meli@iss.it (P.M.)

**\*** Correspondence: daniele.giansanti@iss.it; Tel.: +39-06-4990-2701

We are writing to you as the corresponding author of the interesting study "The Impact of Using mHealth Apps on Improving Public Health Satisfaction during the COVID-19 Pandemic: A Digital Content Value Chain Perspective" [1].

With this correspondence, through two *points of view*, we intend to, *first* recall, through a short analysis, the merits and added value of your study set in the Special Issue "The Impact of Mobile Technology in the Battle against COVID-19: Successes and Failures" https://www.mdpi.com/journal/healthcare/special\_issues/COVID\_Mobile (accessed on 5 May 2022) [2,3]. *Second*, reflect with you on some important aspects, such as the accessibility and the *digital divide*, which have an impact on the topic you are addressing and which we believe can be of scientific interest for future insights and research.

We found that this is a work particularly stimulating and that gives a great added value in the field and in particular in the Special Issue.

Specifically, we believe that this study has the great merit of, at the same time: (a) focusing on important key aspects of the support of the mHealth during the COVID-19 pandemic; (b) highlighting the opportunities and the potentialities; (c) proposing a useful model to assess the impact of this technology in mHealth; and (d) giving a quantitative assessment, which is very useful both for the current period and in perspective.

Specifically, we agree with the authors that this study has both theoretical and practical implications.

*From a theoretical point of view* this study: (I) not only verified the mHealth Apps' functional value from an empirical perspective but also demonstrated the emotional and societal values for public health governance during the COVID- pandemic and the positive impact between user-doctor. satisfaction. (II) Further refines the functional, emotional, and social values in the Digital Content (DC) value chain framework in specific contexts, thus expanding the dimensions of value creation in DC and enriching the antecedents that influence the interactions between users and DC.

*From a practical point of view* this study has three useful suggestions: (I) It enriches mHealth Apps and expands the scope of application to guarantee the basic medical needs of users. In COVID-19, recommendation algorithms can be cleverly used to give the most appropriate results and analyze suggestions based on the questions submitted by users and suggest and associate with related functions. (II) It is necessary to provide users with healthcare confidence during the COVID-19 pandemic. This requires the governance bodies to formulate effective epidemic prevention measures in line with public opinion and pass it onto society through the efficient information distribution function of mHealth apps. Also, the self-assessment Apps could be useful to monitoring these perceptions, trends, and

**Citation:** Giansanti, D.; Pirrera, A.; Meli, P. The *Accessibility* and the *Digital Divide* in the Apps during the COVID-19. Comment on Cao et al. The Impact of Using mHealth Apps on Improving Public Health Satisfaction during the COVID-19 Pandemic: A Digital Content Value Chain Perspective. *Healthcare* 2022, *10*, 479. *Healthcare* **2022**, *10*, 1252. https://doi.org/10.3390/ healthcare10071252

Academic Editor: Lorenzo Mantovani

Received: 6 May 2022 Accepted: 27 June 2022 Published: 5 July 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

opinions. In this case the gamification could be useful. (III) It is also necessary to establish a harmonious doctor–patient relationship in the. These mHealth Apps must be designed to allowing and facilitating this.

The whole study is both interesting and comprehensive. There is no criticism. There are many insights into the future based on what emerged in the study, during the pandemic, specifically in the Special Issue [2,3] and on how mHealth and other apps should consider.

We have noted that two problems in the Apps use, that can be also interconnected with each other, which were already present before the pandemic were often referred to in this period.

A first problem is the 'digital divide'. The digital divide regards the gap between those who have effective access to information technologies and those who are partially or totally excluded from it. The digital divide has three sides. The first is the difficulty in the access to the infrastructures [4]. The second level is represented by the literacy of people with the technologies [5]. The third level is represented by the potential benefit level in terms of economic, cultural, social, and personal types of using the technologies [6]. During the COVID-19 the Digital Divide was evident in all of the three levels and has been exacerbated [7–10].

A second problem is the accessibility of disabled people to these technologies [11]. Disabilities are of various nature, forms, and complexity (auditory, cognitive, neurological, physical, speech, visual). The new software development policies are increasingly, rightly, pushing towards a design of citizen-oriented systems that are more accessible to disabled people [12]. Designers and institutions must increasingly take this into account.

This also applies to mHealth and Apps. Digital Divide and accessibility are linked. For example, also benefits of accessibility [12]:


Accessibility and the Digital Divide are two important and interrelated aspects that governments and designers must increasingly consider when it comes to the use of technologies and the accessibility of citizens for the purposes of monitoring and disseminating information.

This is particularly true when we talk about healthcare where Apps and mHealth can give an important added value as you have fully illustrated in your study [1].

Considering these factors and your study we would like to discuss this with you and have a reflection as a reply. We believe that this would be of great added value for the Special Issue and would further enrich it.

**Author Contributions:** Conceptualization, D.G.; methodology, D.G., A.P. and P.M.; software, D.G.; validation, D.G., A.P. and P.M.; formal analysis, D.G., A.P. and P.M.; investigation, D.G., A.P. and P.M.; resources, D.G., A.P. and P.M.; data curation, D.G.; writing—original draft preparation, D.G.; writing—review and editing, D.G., A.P. and P.M.; visualization, A.P. and P.M.; supervision, D.G.; project administration, D.G., A.P. and P.M.; funding acquisition, D.G., A.P. and P.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This study was also supported and funded by the Italian Ministry of Health—Ricerca Corrente.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


#### *Reply*

## **Reply to Giansanti et al. The** *Accessibility* **and the** *Digital Divide* **in the Apps during the COVID-19. Comment on "Cao et al. The Impact of Using mHealth Apps on Improving Public Health Satisfaction during the COVID-19 Pandemic: A Digital Content Value Chain Perspective.** *Healthcare* **2022,** *10***, 479"**

**Junwei Cao <sup>1</sup> , Guihua Zhang 2,\* and Dong Liu <sup>3</sup>**


**Citation:** Cao, J.; Zhang, G.; Liu, D. Reply to Giansanti et al. The *Accessibility* and the *Digital Divide* in the Apps during the COVID-19. Comment on "Cao et al. The Impact of Using mHealth Apps on Improving Public Health Satisfaction during the COVID-19 Pandemic: A

Digital Content Value Chain Perspective. *Healthcare* 2022, *10*, 479". *Healthcare* **2022**, *10*, 1259. https:// doi.org/10.3390/healthcare10071259

Academic Editor: Lorenzo Mantovani

Received: 27 June 2022 Accepted: 27 June 2022 Published: 6 July 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Thank you for your suggestions for our article [1]. I think it would be very interesting to analyze the role of mHealth apps in COVID-19 from the perspective of the "digital divide" as you suggested, which is something we have ignored in our study.

We saw in a WHO report the need to prevent the harmful misuse of mobile phone data when countries use such data for epidemic surveillance, and the need to work towards equitable access to cell phones and the Internet for poor people and low-income countries, by investing in infrastructure and designing statistical systems for data collection [2]. We strongly agree with the research suggesting that the digital divide is highly dependent on age [3]. For example, in China's epidemic prevention and control, the government tracks people's trips and test results are released through an app on a smartphone, resulting in many older people not being able to travel properly and making it difficult to keep statistics on this demographic. This increasingly highlights the problem of the digital divide.

In terms of mHealth apps, we believe that addressing app accessibility can help alleviate the digital divide, but it is not the main issue. Users of mHealth apps have high demands on aesthetics and ease of use. As such, mHealth apps need to have efficient, intuitive, and easy-to-use application layouts [4,5]. However, there is a lack of research to analyze the design of mHealth apps around the usage characteristics of elderly and disabled people, and there are very few such mHealth apps, which also leads to neglecting the needs of elderly and disabled people when using mHealth apps on a large scale in epidemics. Fortunately, however, electronic device suppliers have made a lot of effort in helping people with disabilities to use their devices. For example, current smartphones, whether IOS or Android, have built-in intelligent assistants that can help users operate most of the functions of the app on the device by voice. This to some extent alleviates the problem of the digital divide between the elderly and the disabled.

The accessibility of the electronic device that hosts the mHealth app is more important than the accessibility of the mHealth app. The study notes that citizens in this epidemic are often unable to use mHealth due to lack of access to tools, cultural barriers, communication barriers, and social barriers [3]. Therefore, we should address the social issues that hinder the accessibility of electronic devices, by reducing the price of electronic devices and increasing education on the use of electronic devices for the elderly and people with disabilities.

The core of this response focuses on another aspect of the secondary digital divide created by too much information and overly complex system provisions. That is, with the attention of society, the elderly and other people with disabilities are already able to use

digital systems easily, but as the market demand increases, the continuous complexity of the functions of these digital systems makes it more and more difficult for people with disabilities to use them (creating a second digital divide) and eventually they stop using them. Therefore, I think that with the developments of the times, the digital divide for the elderly and the disabled may not necessarily appear in the accessibility of software and hardware, but instead will appear after the use of the app. With the development of information technology, apps give more and more information to users and information becomes more and more accessible, which leads to the phenomenon of information overload. Information overload is closely related to age and knowledge reserve [6,7], information overload can lead to user fatigue, fear, and other psychological problems, eventually leading them to stop using the app [8]. Specifically, an elderly person could have used the mobile medical app normally, but the large amount of information about the epidemic in a short period of time could have overwhelmed the elderly person, which caused information overload. At the same time, with the development of the epidemic, the mobile health app was updated with a large number of new features in a short period of time, and the elderly could not understand these features in a short period of time, which caused system feature overload, and eventually, the elderly stopped using it or even resisted using it. Information overload may also trigger an information cocoon effect, keeping seniors forever stuck in homogenized information and unable to receive new information. However, the relationship between information overload and the digital divide is not yet fully confirmed, so the digital divide caused by inappropriate use is also well worth studying. In COVID-19, such a phenomenon is obvious, the Chinese government promotes several QR codes in epidemic prevention, people need to show these QR codes to epidemic prevention officials frequently when they travel, these QR code generation functions are integrated in some apps that are already very popular, many elderly people will use these apps but cannot find the location of these QR codes in the apps, and this creates a digital divide due to too much information (i.e. too many new features affecting the use, too much information becomes difficult to search). There are already companies in China that have made improvements for such problems, for example, some companies have developed a button that can be easily attached to the back of smartphones, and when the elderly need to display these QR codes, they can reach the function by pressing the button, which is convenient for the elderly population, but the intention of using this tool and the market prospect are still not clear enough and need further research.

We have several ideas for the follow-up study. First, we can design a model based on the updated information system success theory to introduce some variables related to app accessibility and the accessibility of electronic devices hosting apps, and improve the current mHealth app operation problem and electronic device penetration problem to reduce the occurrence of digital divide in elderly years by demonstrating the effects of these variables on the satisfaction of elderly people's usage and the degree of usage. Secondly, I think an experiment can be designed to give different amounts of information stimuli to compare the characteristics of users using mHealth apps under different information overloads, and based on the results, suggestions can be made to design the operation of mHealth apps in the epidemic to reduce the digital divide caused by inappropriate use. Thirdly, policy factors and family factors are also variables worth including in the analysis to help address the impact of the digital divide on the use of mHealth apps from a social perspective. Finally, a model can be designed to analyze the usage intentions and market prospects of "one-touch" devices that currently help older adults address the second digital divide.

In conclusion, thank you again for your comment on our research. We will learn from your research in the area of digital divide and follow your suggestions for our next research study.

**Author Contributions:** Conceptualization, J.C. and G.Z.; project administration, D.L. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Review* **COVID-19 and the** *Infodemic***: An Overview of the Role and Impact of Social Media, the Evolution of Medical Knowledge, and Emerging Problems**

**Francesca Corinti <sup>1</sup> , Daniela Pontillo <sup>1</sup> and Daniele Giansanti 2,\***


**Abstract:** The *infodemic* is an important component of the *cyber-risk* in regard to the poor and uncontrolled dissemination of information related to the COVID-19 pandemic. The purpose of this study was to perform a narrative review based on three points of view to allow for an overall picture of this issue. The points of view focused on: (a) the volume of use of social media (a key element of the infodemic) and the position of international health domain bodies; (b) the evolution of scientific production in the life sciences; (c) emerging issues. The research methodology was based on Google and PubMed searches and a qualification process based on a standard checklist and an evaluation of eligibility based on parameters with five score levels applied by two experts (plus one in case of discrepancy). The three points of view stressed the key role of social media as a dissemination tool of the *infodemic* among citizens. The impact on citizens depends on various social factors and involves indirect (e.g., vaccine avoidance) and direct risks such as mental problems and the risk of suicide. The widespread diffusion of social media, conveyed by mobile technologies, also suggests their use as countermeasures, calibrated based on citizens' level of both technological and health literacy. Effective and promising countermeasures in this direction are based both on initiatives of contact by apps or SMS and the collection of data based on surveys and finalized to the particular intervention. The review also suggests as further areas of in-depth research: (a) to combat *high-level infodemic* produced by scientific publications that are not yet official (preprint) or that have undergone peer review with bias/distortion; (b) focusing on the impact of the *infodemic* considering its spread in different languages.

**Keywords:** infodemic; pandemic; social media; COVID-19; infodemiology; infoveillance

#### **1. Introduction**

Unlike other pandemics, COVID-19 exploded in an era in which new communication technologies based on social media have spread. Social media have peculiarities compared to other paper, television, and radio information dissemination systems. Their potential to disseminate information (true or false) places them at the center of attention of scholars and stakeholders in this delicate period due to the pandemic. Aspects such as the mechanism of dissemination of distorted information, the impact on the population, and the direct and indirect risks to public health are of continuous scientific interest today, and the pandemic represents, in a certain sense, an environment of growth regarding medical knowledge in this field.

#### *1.1. New Communication Media Based on Social Media in the Pandemic Era*

Social media are different from industrial media such as newspapers, television, and cinema. While social media are relatively low-cost tools that allow to publish and access information, traditional media require substantial financial investments to publish

**Citation:** Corinti, F.; Pontillo, D.; Giansanti, D. COVID-19 and the *Infodemic*: An Overview of the Role and Impact of Social Media, the Evolution of Medical Knowledge, and Emerging Problems. *Healthcare* **2022**, *10*, 732. https://doi.org/ 10.3390/healthcare10040732

Academic Editor: Yogesan Kanagasingam

Received: 18 February 2022 Accepted: 12 April 2022 Published: 14 April 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

information [1,2]. Industrial media are commonly referred to as traditional, broadcasting, or mass media. A common feature of both social media and industrial media is the ability to obtain a large audience. Both a blog post and a television broadcast can reach millions of individuals [3]. There are different parameters [4,5] that are useful for both describing the difference between the two media and what comprises their different roles in spreading the information:


The current pandemic is characterized by the presence of social media that can instantly transfer information to anyone with a smartphone. Therefore, it is essential to ensure that the information traveling through these systems is not distorted or misleading. Furthermore, the pandemic has been a driving force in the growth of mobile technologies, internet technologies in general, and social media. The number of users of mobile technologies and apps related to social media has grown considerably. In Italy, the data are reported in CENSIS (an Italian socioeconomic research institute) reports.

The latest Italian CENSIS report [6] outlined, in brief, the following: between 2019 and 2021, there was a strong increase in the use of the Internet by Italians (83.5% of users, with a positive difference of 4.2 percentage points), while those who used smartphones rose to 83.3% (a record growth compared to 2019: +7.6%) as well as a total of 76.6% who used social networks (+6.7%).

For print media, on the other hand, the now historic crisis is accentuated, starting with newspapers sold on newsstands, which in 2007 were read by 67.0% of Italians, reduced to 29.1% in 2021 (−8.2% compared to 2019). The same is true for weeklies (−6.5% in the twoyear period) and monthlies (−7.8%), which were hard hit by the effects of the pandemic.

Among young people (14–29 years old), there was a further step forward in the use of media, in general, and of online platforms, in particular: 92.3% used WhatsApp; 82.7% YouTube; 76.5% Instagram; 65.7% Facebook; 53.5% Amazon; 41.8% video conferencing platforms (compared to 23.4% of the total population), 36.8% Spotify; 34.5% TikTok; 32.9% Telegram; 24.2% Twitter.

Even among the elderly (65 years and over), something is moving, given that internet use increased significantly from 42.0% to 51.4% and social media users increased from 36.5% to 47.7%. The need to maintain contact, at least virtually, with loved ones during the period of the most rigid isolation must have played a significant role in the confidence acquired in using the network by those over 65 years old.

These data, even if related to a country (Italy), suggest that attention to the correct dissemination of information must be directed particularly towards social media. These systems can rapidly disseminate both correct and distorted information. In the case of the spread of distorted information, they are sources of the *infodemic*.

#### *1.2. Basis and Purpose the Review*

Social media, using mobile technology, have been and still are an important component regarding the poor and uncontrolled dissemination of information related to the COVID-19 pandemic. It is important to understand: (a) how the *infodemic* evolved over time, also due to the evolution of social media and understanding how the latter occurred; (b) what are the positions of national and supranational bodies on this issue and the volume of the problem; (c) which themes are most dealt with; (d) what directions comprise future prospects.

In line with the above, we had the idea to propose a *narrative review* focused on the on the *infodemic* with the aim:


This study was arranged according to following structure (also including Section 1 (Introduction), Section 4 (Discussion), and Section 5 (Conclusion)):


#### **2. Methods**

According to the purpose this narrative review, three points of view were focused on to allow for an overall picture of the issue. The methods have both a general and a specific framework. The general framework is that we followed a narrative checklist [7] as a supporting tool in the data synthesis. The specific framework for each goal was as follows:

First goal: Google searches were applied to (a) find international documents on the application of social media; (b) find documents on the positions of government agencies sorted by priority of importance. Two experts (plus an expert adjudicator in cases of discordance) performed these searches.

Second goal: Analyses were conducted on both the volume of PubMed's scientific production on this topic and any relevant articles on (a) important passages related to the foundation of *infodemic* (before the pandemic) as a subject of study and (b) on events/current initiatives to integrate consent on actions to be carried out. The selection processes were carried out by the three experts (i.e., two experts plus an adjudicator) based on a qualification procedure as illustrated in the next point. Box 1 lists the search keys.

**Box 1.** Search keys used (also with plurals).


*Third goal:* The search was performed using PubMed. A specific qualification process was based on a scoring system (with different parameters and a score with 5 levels) applied by two experts (plus one adjudicator in cases of discordance) to include each reference (Table 1). It was possible to assign a score to these parameters ranging from a minimum score of one up to a maximum of five. As far as the "added contributions to the field" parameter was concerned, we used a weighting procedure. In consideration of the criticality of the first moments of the pandemic and relativizing the importance of the first period of the pandemic, the vote assigned was multiplied by:


**Table 1.** Parameters used for the qualification (\* 1.3 was used for studies published in the first three months, and 1.15 was used for studies published in the period ranging from three to six months).


A study was excluded if, regardless of the score, there were critical issues of conflict of interest (for example, it was conducted without guarantees of objectivity by the system manufacturer).

A study was included in the review if all parameters, after the weighting procedure, showed a score ≥ 3.

#### **3. The** *Infodemic***: Volume of the Problem and Position of the International Bodies, Scientific Production, and Key Points**

*3.1. The Infodemic: The Volume of the Problem and the Positions of Key International Bodies* 3.1.1. The Volume of the Problem

Various documents and sources were analyzed with reference to the volume of use of social media. Based on the list of priorities drawn up, the Statista source was proposed (https://www.statista.com, accessed on 11 April 2022) [8]. This source draws insights into consumers from industries and markets worldwide. Covering the offline and online world, the Statista Global Consumer Survey offers a global perspective on consumption and media usage.

The results of a search (data refer to October 2021) are shown in Table 2. This table shows the most used social media together with the type of interaction/service provided, manufacturer, and country of the manufacturer.


**Table 2.** Volume of use of social media by application.

The results of another search (data refer to January 2022) are shown in Table 3.

**Table 3.** Volume of use of social media with reference to the Internet and mobile technology.


This table highpoints the global diffusion of social media while also considering the technology used. Most users accessed social media through mobile technology, which is easy, wearable, and low cost, and *96.1%* of those with mobile internet accessed social media; *98.81%* of those who accessed social media did so with mobile internet. The outcome of both of the searches highlights a very large volume of social media use, confirming, at an international level, what has been pointed out in Italy at the national level [6] and, thus, corroborating the disruptive impact on the population of the dissemination of information (correct or distorted).

#### 3.1.2. The Position of the Key International Bodies

National and international bodies have dealt with the *infodemic* with reference to the role of social media. Here, there are some important positions of the "abovementioned government" entities in line with the *first objective of this study.* In the following are the positions selected by experts from key international bodies (also as joint positions) and from the US.

The WHO, whose primary role is to direct international health within the United Nations' system and to lead partners in global health responses defines the *infodemic* as follows (literal quote) [9]:

"*An infodemic is too much information including false or misleading information in digital and physical environments during a disease outbreak. It causes confusion and risk-taking behaviours that can harm health. It also leads to mistrust in health authorities and undermines the public health response. An infodemic can intensify or lengthen outbreaks when people are unsure about what they need to do to protect their health and the health of people around them. With growing digitization—an expansion of social media and internet use—information can spread more rapidly. This can help to more quickly fill information voids but can also amplify harmful messages. Infodemic management is the systematic use of risk- and evidence-based analysis and approaches to manage the infodemic and reduce its impact on health behaviours during health emergencies. Infodemic management aims to enable good health practices through 4 types of activities:*


According to the definition of *infodemic* by the WHO, the importance of the roles of digital technology and *social media* in this field appear.

In relation to the term *infodemic*, it is essential also to recall the important joint statement by nine international bodies: WHO, UN, UNICEF, UNDP, UNESCO, UNAIDS, ITU, UN Global Pulse, and IFRC [10]. We report the following passage (literal quote), particularly centered on *social media*:

"*We further call on all other stakeholders—including the media and social media platforms through which mis- and disinformation are disseminated, researchers and technologists who can design and build effective strategies and tools to respond to the infodemic, civil society leaders and influencers—to collaborate with the UN system, with Member States and with each other, and to further strengthen their actions to disseminate accurate information and prevent the spread of* . . . "

Always in relation to the term *infodemic*, it is essential to recall the European position, highlighted in a *speech by Vice President Vˇera Jourová at the EU Commission on 4 June 2020.* We report a brief section (literal quote) that particularly focuses on *social media* [11]:

"*The Coronavirus pandemic has been accompanied by an unprecedented 'infodemic', according to the World Health Organisation. . . . . .*

*While this argument is happening in the US, Twitter, Facebook and other platforms are global and relevant for politicians and users in Europe as much as they are in the US. I have been saying for a long time that I want platforms to become more responsible, therefore I support Twitter's action to implement transparent and consistent moderation policy. This is not about censorship. Everyone can still see the tweets. But it is about having some limits and taking some responsibility of what is happening in the digital world.*"

*In the US, the* Centers for Disease Control and Prevention also face this issue [12]. Among the recommendations related to deployment processes correlated to COVID-19 we found (literal quote): "*Prevent "infodemics"(an excessive amount of information about an issue that makes it difficult to identify a solution)—this builds trust, increases probability that health advice will be followed, and manages rumors/misunderstandings*."

#### *3.2. The Infodemic: The Evolution of Scientific Production*

In line with the second objective of the study, this section deals with the evolution of scientific production in this area starting from the definition of the disciplines around the *infodemic* up to international consensus initiatives. We carried out incremental searches on *PubMed*.

These searches refer to the date 1 February 2022.

A first search was performed with the key *Infodemic (Title/Abstract)* [13], and 405 studies (34 reviews) were found, all of which were concentrated between 2020 and the reference date and were associated with the COVID-19 pandemic.

A free search with the key *Infodemic* [14] led to 407 papers, with 1 paper published in the year 2009 [15], which highlights how this term did not firsts appear scientifically with the COVID-19 pandemic but in an earlier era.

The first appearance of the term *infodemic* appeared in a study [15] proposed in an editorial by Eysenbach et al.

The authors also focused on the role of the *infodemiology* [16], the science that deals with *infodemic*. The term *infodemiology* was defined here [16] as a portmanteau of information and epidemiology. Although, in 2009, the spread of smartphones and *social media* had only just begun, the authors identified some basic concepts that can be extrapolated to date. These concepts concern, in particular, the definition and positioning of *infodemiology* and *infoveillance*. *Infodemiology* was defined (literal quote) [16] "*as the science of distribution and determinants of information in an electronic medium, specifically the Internet, or in a population, with the ultimate aim to inform public health and public policy*." *Infoveillance* has been defined as the science that deals with surveillance in *infodemiology* [16].

The same authors of the study, the first to frame the science of *infodemiology*, applied the tools described above in the case of flu in 2006 [17] and highlighted how *infodemiology* is important for dealing with *cyber-behavior* [18]. The great explosion in this area, as we have seen, has occurred over the past two years associated with the pandemic. It covered both the pandemic in general [13] and (more recently) vaccines as well. The *infodemic* associated with the pandemic, in general, can lead to extreme behavior of underestimating the pandemic or terrorism. *Infodemic* associated with vaccines can cause avoidance [19].

A refinement of the search with the key *Infodemic* (*Title/Abstract) AND (Vaccine)* reports 64 papers [20] (15.8% of the total).

When we focused on the role of *social media* and searched with the key *Infodemic (Title/Abstract) AND (Social Media),* we found 223 papers [21] (56% of the total).

Further refining the research to highlight works dealing with vaccines, with the key *(Title/Abstract) AND (Social Media) AND (Vaccine),* we found 35 papers [22] (8.6% of the total).

There were 18 reviews in relation to the role of *social media* in the *infodemic* [23]; only 1 of these review also concerned vaccines [24].

A focus has also emerged among this research [25]. This study is a summary document produced by international experts after a meeting at the *First WHO Infodemiology Conference* [25].

The document is, in fact, an act of global orientation to address the *infodemic* with particular emphasis on the role of *social media*. The document, available since 15 September 2021, was developed through an analysis of the evidence, a review, and consensus of actions by the experts. In brief, it recommends:


#### *3.3. The Infodemic: The Key Emerging Issues*

In line with *the third objective* of this study, this section analyzes the issues on the *infodemic* by theme, based on the selected papers using both the proposed keys and the qualification process [26–49]. The papers were, as expected, mainly reviews; however, full scientific articles were also selected dealing with very peculiar aspects. The structure for this section starts with considering the methods of dissemination (a); then, it analyzes the social impact (b), the direct and indirect health risks (c), and the countermeasures (d).

#### 3.3.1. Methods of the Dissemination of the Infodemic

An *infodemic* has a source. This source may be a *misinformation* or a *disinformation*. It is important to clarify the differences between these two terms. Indeed, they have different roles in the *infodemic*. The authors of [26] make a differentiation based on what they call "the cognitive domain". Thus, it may be said that "disinformation" refers to the deliberate creation or sharing of false information, whereas "misinformation" is not intended to mislead the receiver.

As for *disinformation*, in the Cambridge dictionary we find [27]: "*false information spread in order to deceive people. e.g., they claimed there was an official disinformation campaign by the government*."

As for *misinformation*, in the Cambridge dictionary we find [28]: "*wrong information, or the fact that people are misinformed. e.g., there's a lot of misinformation about the disease that needs."*

There are several methods, now recognized, for the dissemination of the *infodemic*, whether it is *misinformation* or *disinformation*. A recent systematic review [29] conducted a literature search covering 12 scholarly databases to retrieve various types of peer-reviewed articles that reported the causes [29].

Social media usage, low level of health/eHealth literacy, and fast publication process and preprint service are identified as the major causes of the Infodemic.

In addition, the vicious circle of human rumor-spreading behavior and the psychological issues from the public (e.g., anxiety, distress, and fear) emerge as characteristics of the *infodemic*.

*Three important concepts* emerged, corroborated also by other reviews.

The *first concept*relates to the importance of social media as a key factor for the *infodemic*, both as a vehicle of communication and as regards the mastery of correct use (literacy). This was also confirmed in another review [30] that stressed the magnitude of the problem of COVID-19 misinformation on social media, its devastating effects, and its intricate relation to *digital health literacy*. As it was highlighted in [44], digital health literacy can help improve prevention and adherence to a healthy lifestyle, improve capacity building, and enable users to take the best advantage of the options available, thus strengthening the patient's involvement in health decisions and empowerment and, finally, improving health outcomes.

The *second concept* is that there is also a *high-level infodemic* (HLI) generated, for example, from the scientific literature of the preprints—scientific works published before the peer review process. The presence of this HLI was also highlighted in [31] and extended to official academic and institutional publications of any type. The authors highlighted this among the causes of the exponential increase in COVID-19-related publications that often included biases in the peer-review and editorial process.

The *third concept* is the important role of word of mouth with the *infodemic,* which follows a peer-to-peer mechanism.

#### 3.3.2. The Social Impact

A recent scoping review [32] that focused on the social impacts highlighted that particular socio-environmental conditions (e.g., low educational level and younger age), psychological processes and attitudes (such as low levels of epistemic trust, the avoidance of uncertainty, extraversion, collective narcissism, and a conspiracy-prone mindset), and contextual factors (e.g., high levels of self-perceived risk and anxiety) seemed to underpin the adherence to beliefs that are not solely the domain of paranoids and extremists but a widespread phenomenon that has caused important health, social, and political consequences during the pandemic. All of this, in the early stages of the pandemic, led to incorrect adherence to virus defense measures (such as wearing masks and social distancing), and now during the vaccination process, it is also (in addition to this) leading to avoidance by part of the population [33,34].

#### 3.3.3. The Health Risks

The health risks related to the infodemic are indirect and direct. Among the indirect risks, there are those related to the non-adherence to defense measures against the contagion and those related to the avoidance of receiving vaccines [45,46]. These lead to the consequent possibility of contracting and spreading the virus. Among the direct consequences, there is adherence to incorrect medical indications. The spread of misinformation is particularly alarming when spread by medical professionals, and existing data on YouTube suggesting vitamin D has immune-boosting abilities can add to viewer confusion or mistrust regarding health information [47]. Further, the suggestions made in the videos may increase the risks of other poor health outcomes such as skin cancer from solar UV radiation [47]. It has been seen that the infodemic has a strong impact on well-being in general [48]. In particular, the scientific literature showed a growth in the direct consequences that had an impact on mental problems. These consequences had an important impact on the psychological sphere, they can even lead to suicide and affect healthcare professionals, patients [35], and common people [36]. In [36], a recent scoping review, articles were analyzed according to average age, gender, and education level; place and period of the studies; exposure time to COVID-19 information; main signs and symptoms related to mental health; main sources of information; suggestions for mitigating the effects of the *infodemic*; knowledge gaps. As a result, it was shown that the most present repercussions of the *infodemic* on adult and elderly mental health were anxiety, depression, and stress, and the most affected groups were young adults and females. A systematic review [35] was performed based on a search from 1 January 2020, to 11 May 2021. Studies that addressed the impact of fake news on patients and healthcare professionals around the world were included. By analyzing the phenomenon of fake news in health, it was possible to observe that *infodemic* knowledge could cause psychological disorders and panic, fear, depression, and fatigue. The extreme consequences of these issues can be represented by suicide as shown in the narrative review in [37]. This narrative review summarized the sociocultural risk and predisposing factors for suicidal behavior in developing countries during the COVID-19 pandemic. The findings revealed fear of being infected, growing economic pressure, and lack of resources due to the lockdown were the factors most responsible in the four countries studied for the current increase in suicides. There were a few cultural differences that were specified in the narrative

#### 3.3.4. Countermeasures

Countermeasures are a very important aspect and must consider the connection with mobile technologies, any involvement in the *health domain*, and whether it is dealing with HLI.

In cases of HLI, the *infodemic* starts directly from scientific works both published after the peer-reviewed process and earlier when they are in the preprint state.

Indeed, the COVID-19 pandemic catalyzed the rapid dissemination of papers and preprints investigating the disease and its associated virus, SARS-CoV-2. We report two initiatives in the case of HLI. In [38], the authors applied a massive online open-publishing approach for improving the publication process using an automatic approach for referenced preprints, journal publications, websites, and clinical trials. Continuous integration workflows retrieved up-to-date data from online sources nightly, regenerating some of the manuscript's figures and statistics. The proposed architecture improved the interaction of the scientists and minimized the *misinformation*. In [39], the activity by a workgroup (WG) at the Istituto Superiore di Sanità (ISS), the Italian National Institute of Health, was reported. The WG proposed a *workflow* allowing for two cultural mediation activities. A first cultural mediation activity was designed for actors in the *health domain* during the lockdown. During the Italian lockdown, experts in the WG selected scientific works that were constantly reviewed by experts in the field both in the WG and externally. The reviews were made available online in appropriate periodical publications [39]. A second cultural mediation activity concerned stakeholders during the Italian lockdown, to whom daily

reports on the critical analysis of the newly published preprints were addressed. The Dear Pandemic project is also aimed [49] in this direction. It is a multidisciplinary, social-mediabased science communication project, the mission of which is to educate and empower individuals to successfully navigate the overwhelming amount of information circulating during the pandemic with the two aims: (1) to disseminate trustworthy, comprehensive, and timely scientific content about the pandemic to lay audiences via social media; (2) to promote media literacy and information on hygiene practices, equipping readers to better manage the COVID-19 infodemic within their own networks.

Of course, it is not possible to carry out a complete examination of the possible countermeasures, which are far-reaching and can range from an SMS that arrives on your mobile device, a wall poster, or to a radio message of a testimonial. In line with the purpose of this study and considering the role of mobile technology and social media, we report several relevant actions that emerged from the literature search. The same study [29] that investigated the dissemination of the *infodemic* also proposed a comprehensive list of countermeasures, summarized from different perspectives. The authors emphasized, among the most important countermeasures: risk communication and consumer health information; the use of the mobile technology and social media as a means to reach citizens. The key issue is to offer content that is easy to access via mobile technology and that is displayed in a calibrated and understandable way for common people in a *client–server* manner. Alternatively, it is important to send the same types of information in a distributed way through mobile technology as well as via *ICT pushes* such as through dedicated app messages (e.g., WhatsApp), SMS, tweets, and emails.

In [40], the authors reviewed and provided insight regarding methodologies and the construction of content on health information-seeking behaviors (HISBs). A total of 13 survey tools from eight countries were identified after a review. This review [40], in line with the previous review, highlights how tools (surveys) shared by means of mobile technology, in a *client–server* way, could be useful sensors for the decision makers in healthcare.

The *ICT pushes* particularly need to be tuned to the level of technological literacy. An SMS is more suitable for people with low literacy, who do not use smartphones (but simple mobile phones) such as, for example, the elderly. Other systems are more suitable for people with a higher level of literacy. Subtleties in the composition and the language used of messages sent using *ICT pushes* to reach people have been shown to be very important in affecting behavior and minimizing misinformation [41]. In [41], text messages designed to make vaccination salient and easy to schedule boosted appointments and vaccination rates and minimized avoidance.

The study reported in [42] is very important for the relevance of the *ICT push* method. The study explored how the WHO uses its Twitter profile to inform the population on vaccines against the coronavirus, thus preventing or mitigating misleading or false information both in the media and on social networks. The analysis showed that the WHO is decidedly committed to the use of these tools to disseminate messages that provide the population with accurate and scientific information as well as to combat mis- and disinformation about the SARS-CoV-2 vaccination process. Even the use of artificial intelligence (AI) can be useful in this activity. In [43], it was highlighted how AI-based approaches are useful to improving e-health literacy, including AI-augmented lifelong learning, AI-assisted translation, simplification, summarization, and AI-based content filtering. Furthermore, it exhibited usefulness in combating the *infodemic*, presenting the general advantage of matching the right online health information to the right people.

#### **4. Discussion**

#### *4.1. Considerations Emerging from the Study*

The term *infodemic* has particularly resonated over the last two years due to the fact of its association with the COVID-19 pandemic. However, it was not coined recently, but was inherited from previous experiences associated with *infodemiology* and *infoveillance* [15–18]. The scientific production around this theme exploded over the last two years and was strongly linked both to the COVID-19 pandemic and to mobile technologies and social media, the new vehicles for the transmission of information [4]. The study started from the significant increase in the use of these devices during the pandemic as evidenced by national data [6] and confirmed by international data (Tables 2 and 3 [8]).

There are *three points of view* reported in this review.

The *first point of view* illustrates how: (a) recent statistical reports [8] highlight the widespread use of social media through mobile technology; (b) the positions of international and relevant bodies on this issue (i.e., WHO, UN, UNICEF, UNDP, UNESCO, UN-AIDS, ITU, UN Global Pulse, IFRC, EUC, and CDC) [9–12] which, in some cases, emerged through joint initiatives that emphasized the role of social media in the dissemination of the infodemic.

The *second point of view* examined the evolution of scientific production in this field. This started from the first important experiences in this sector [15–18], well before the pandemic and up to the first important recent initiative on an international consensus [25]. It emphasized the role of the first studies [15–18], the high production over the last two years [13,14], and the strategic importance of the initiative of international consensus that produced a document with recommendations at the international level [25].

The *third point of view* detailed in-depth the emerging issues related to the infodemic in the recent scientific production arranged into four themes: (a) *dissemination methods*; (b) *social factors*; (c) *health risks*; (d) *countermeasures*.

The role of social media in *dissemination methods* has been reiterated by several studies [29–31]. *Social factors* [32–34] are important regarding the taking root of the phenomenon that can lead to *health risks*, with not only indirect (for example, due to the avoidance of vaccines [34]) but also direct impacts on mental problems [35,36] and predisposition to suicide [37]. Among the most important *countermeasures,* we found those that combated the infodemic using the same tools that caused its spread such as social media and mobile technology. Studies show the usefulness of calibrated ICT pushes [41] from relevant bodies [42] and client–server solutions at the government level, for example, based on surveys [40].

#### *4.2. Boundaries of the Considered Studies and Suggestion for Further Research*

The studies reported in the review did not take into consideration scientific productions in languages other than English and how research in this area is evolving also using the preprint databases dedicated to COVID-19. In general, the topic is broad, and the reviews (even systematic) focus on specific aspects while not being able to address them all at the same time. Even our study could not analyze all the implications and problems, which, however, with a three-point approach, tried to provide an integrative review by correlating social media developments at the local and global level [6,8], positions of international bodies [9–12], consensus initiatives [25], and recent scientific productions by selecting wide-ranging reviews and focused article reviews [26,29–43], aiming to provide an overview of the problem. We must also consider that, although with caution in regard to the results, useful indications concerning emerging directions of research in this area also come from the preprints accessible on the websites dedicated to the pandemic. While the reliability of the preprint results is postponed to the end of the peer-review process (approximately 1–3 months), looking at these websites immediately allowed us to gain a perspective on the studies in progress (beyond the final results).

Among the most visited websites of preprints dedicated to the pandemic, we found the website COVID-19 SARS-CoV-2 preprints from medRxiv and bioRxiv [50] and the website Researche Square-SARS-CoV-2 and COVID-19 Preprints [51]. The integration with studies also coming from these preprint sites allowed us to close the study with an even broader perspective by integrating the study conducted so far with previews of the directions of research in this field. Hot topics [52,53] for research development emerged from these databases. A prime example is the role of editors in peer-reviewed journals in controlling the infodemic by avoiding excessive volumes of rapid COVID-19 publications with sometimes hasty peer reviews [52]. Another example [53] relates to how the infodemic has a different behavior based on the languages used. A recent study [53] conducted a comparative study of anti-vaxxers' aggressive behaviors by analyzing tweets in English and Japanese. They found two common features across these languages. First, anti-vaxxers most actively transmit targeted messages or replies to users with different beliefs. Second, the influential users are more likely to receive the most toxic replies from the anti-vaxxers. However, pro-vaccine sites with a low number of followers are subjected to higher hard replies in English, which ask for special support that differs from the Japanese case. This suggests the need for both language-dependent and -independent countermeasures against infodemic. Other studies, from these databases, confirmed the right direction of the countermeasures identified by the studies taken into consideration based on mobile technologies using, for example, surveys at the national level [54] or the support of categorization processes to protect citizens against fake news [55].

#### *4.3. Limitations*

This review took into consideration a limited number of sources (Google for the first point of view; PubMed for the other two) and only sources in English. However, it should be noted that (a) an important part of scientific production and direct or related initiatives in this area are in national languages other than English (as we highlighted in the case of the Italian CENSIS report [6]); (b) other databases, such as those of the preprints, could provide in real time interesting indications of the development directions on the research in progress [47,48,52,53].

#### *4.4. Prospects*

As a narrative review, the study focused on various topics with three points of view to try to provide an overall and unitary vision to the phenomenon comprehending (a) the volume of use of social media (a key element of the infodemic) and the position of international health domain bodies; (b) the rapidity of the evolution of scientific production in this field of life sciences; (c) the key emerging issues.

Future studies could be dedicated to deepening each of the issues addressed. Taking into account the rapid evolution of the topic, it could also be useful to transform this study into a living narrative review with periodic updates. To do this, however, it will be necessary to verify the feasibility of this transformation, which must include periodic updates with appropriate editorial tools and methodologies.

#### **5. Conclusions**

This review focused on the *infodemic* in the COVID-19 era and addressed *three points of view* to allow for an overall picture of the issue.

The *points of view* focused on:


The study emphasized the key role of social media as a dissemination tool of the *infodemic* among citizens, documented both by the position of international bodies and by a growing scientific production. The impact on citizens, involved and not involved in the *health domain*, depends on various social factors and involves indirect (e.g., vaccine avoidance) and direct risks, e.g., mental problems and the risk of suicide. The widespread diffusion of social media conveyed by mobile technologies also suggests their use as a countermeasure, calibrated based on the citizen's level of both technological and health literacy. The sending of correctly informative *ICT pushes* to mobile devices and the provision or collection of information aimed at interventions (through *client–server architectures* accessible from mobile devices, based, for example, on electronic surveys) are proving to be effective and promising countermeasures. This review also suggests as future research directions: (a) the face solutions for minimizing the high-level *infodemic* produced, for

example, by scientific publications yet at the stage of the peer review or published with a peer review with bias/distortion; (b) focusing on the impact of the *infodemic* considering its spread in different languages.

**Author Contributions:** Conceptualization, D.G.; methodology, D.G.; formal analysis, D.G., D.P. and F.C..; investigation, D.G., D.P. and F.C.; data curation, D.G.; writing—original draft preparation, D.G.; writing—review and editing, D.G., D.P. and F.C.; visualization, D.G., D.P. and F.C.; supervision, D.G.; project administration, D.G. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Abbreviations**

List of acronyms of the cited institutional bodies.


#### **References**


## *Article* **Evaluation of the Patient Experience with the Mawid App during the COVID-19 Pandemic in Al Hassa, Saudi Arabia**

**Eman AlAli <sup>1</sup> , Reem AL-Dossary <sup>2</sup> , Saja Al-Rayes <sup>1</sup> , Naof Al-Ansary <sup>1</sup> , Deema Alshawan <sup>1</sup> , Salma Almulla <sup>1</sup> , Fahad Alanezi <sup>3</sup> , Zahraa Alakrawi <sup>1</sup> , Norah Alnaim <sup>4</sup> , Linah Saraireh <sup>3</sup> , Razaz Waheeb Attar <sup>5</sup> , Nouf Alaenzi <sup>6</sup> , Hayathem bin Hasher <sup>7</sup> , Bashair AlThani <sup>3</sup> , Lojain Alsulaiman <sup>8</sup> , Naif Alenazi <sup>9</sup> , Beyan Hariri <sup>1</sup> and Turki Alanzi 1,\***


**Abstract:** (1) Introduction: The objective of this study was to evaluate the patient experience with the Mawid application during the COVID-19 pandemic in Al Hassa, Saudi Arabia. (2) Methodology: A quantitative cross-sectional survey was designed to evaluate the patient experience with the Mawid app during the COVID-19 pandemic in Al Hassa, Saudi Arabia. A total of 146 respondents completed the questionnaire. (3) Results: More than half of the participants (65.8%) opined that application was easy to use. Furthermore, 65.1% of the participants considered it to be very easy and easy to search for the required information; and 63.7% of the respondents reflected that it was easy to book an appointment. There was a statistically significant difference between the ease of searching for the required information (*p*-value = 0.006); the ease of undoing an unwanted move and gender (*p*-value = 0.049); the ease of searching for the required information and educational level (*p*-value = 0.048); the ease of booking an appointment and educational level (*p* = 0.049); and the ease of searching for the required information and the labor sector of the respondents (*p* value = 0.049) among the genders. No significant differences were identified among the age groups. (4) Conclusions: Overall, most participants suggested that the Mawid app was easy to use and had a potentially useful set of features to help mitigate and manage the COVID-19 pandemic in Al Hassa, Saudi Arabia.

**Keywords:** Mawid app; COVID-19 pandemic; mobile application; primary healthcare centers; Al Hassa; Saudi Arabia

#### **1. Introduction**

The COVID-19 pandemic that emerged at the end of 2019 in the city of Wuhan, China, has caused an impact on all aspects of the daily life of human beings on a global scale [1].

**Citation:** AlAli, E.; AL-Dossary, R.; Al-Rayes, S.; Al-Ansary, N.; Alshawan, D.; Almulla, S.; Alanezi, F.; Alakrawi, Z.; Alnaim, N.; Saraireh, L.; et al. Evaluation of the Patient Experience with the Mawid App during the COVID-19 Pandemic in Al Hassa, Saudi Arabia. *Healthcare* **2022**, *10*, 1008. https://doi.org/10.3390/ healthcare10061008

Academic Editors: Daniele Giansanti and Norbert Hosten

Received: 3 March 2022 Accepted: 28 April 2022 Published: 30 May 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

According to the World Health Organization, as of 25 February 2022, there have been 430,257,564 confirmed cases of COVID-19, including 5,922,047 deaths [2].

Faced with this situation, all of the countries of the world have made multiple efforts to mitigate the disease and stop the spread of the virus [3]. Several efforts have been directed at producing medicines and vaccines, and developing technological systems, such as telemedicine, to provide healthcare services based on advances in information and communication technologies [4].

In this regard, some countries of the world have developed healthcare informatics applications to notify people about the levels of the spread of the COVID-19 virus, report cases of COVID-19, raise awareness about COVID-19, book appointments, disseminate preventive measures on a massive scale, carry out tests to detect the COVID-19 virus, give information to people on available healthcare sites, self-manage symptoms of the disease, perform remote consultations, etc. [3–5].

Several applications have been implemented in various countries of the world, such as Italy, Greece, Singapore, Switzerland, Malaysia, Vietnam, the United Kingdom, Germany, New Zealand, Spain, the USA, Norway, Canada, Brazil, Czech Republic, Austria, France, Bangladesh, Germany, the United Arab Emirates, Bahrain, Oman, Saudi Arabia, Kuwait, Hungary, Israel, China, India, Ghana, Japan, Iceland, and Australia, among others [3,6–10]. Many of these applications are based on GPS and Bluetooth technologies and focus on the use of contact tracing apps, digital consultation, appointments booking, etc. [11]. These applications have incorporated functionalities capable of consultation support, self-evaluation, and other features. None of the analyzed applications have incorporated social media platforms [11].

It is pertinent to mention that there are many concerns related to patient privacy, data security and protection, and technical barriers to using these applications [12–16]. These privacy concerns decrease people's willingness to use these applications [13]. Similarly, a recent study [17] has identified 39 factors from 15 studies for evaluating the eHealth application from multiple stakeholders' perspective. Few factors that can be related to eHealth users include computer literacy levels, knowledge about eHealth, cost-effectiveness of application, cultural constraints, willingness to use, resistance to change, privacy and security, etc. Moreover, studies [17–20] have adopted different factors and models, such as perceived usefulness, technology adoption models, etc., for evaluating the acceptance of eHealth applications. These factors were derived from the studies focused on different applications, reflecting that every application can have few general acceptance factors, and there can be few factors in specific to each application that can be analyzed based on its use/purpose of development. Furthermore, some studies have suggested that for an application to be effective in controlling the COVID-19 pandemic, it must have broad public support and must be used by more than 50% of the population [12,21]. This situation suggests that efforts are needed to encourage people to use these apps to slow down the spread of the COVID-19 virus and mitigate the impact of this disease [13].

Regarding Saudi Arabia, the healthcare system has been adapted and changed to handle the evolution of the COVID-19 pandemic. In this regard, during the pandemic, the digital healthcare areas have been strengthened; technologies associated with telemedicine, artificial intelligence, machine learning, and computer networks were developed to increase awareness and knowledge about the COVID-19 pandemic [21].

In addition, the Saudi Arabian health authorities developed three new applications: Tetamman, Tawakkalna, and Tabaud [21]. Tetamman was designed to provide COVID-19 test results, to ask for help, to check up on COVID-19 symptoms, to contact positive cases, and provide alerts through messages. Tawakkalna was developed to request permits to travel during the curfew and report suspected cases of COVID-19 infected people. In addition, Tabaud was created for people to notify if they had previous contact with people infected with COVID-19, and request medical support (50) [3]. Moreover, several applications developed before the pandemic, such as HESN, Mawid, SEHA, and Sehaty, were modified and adapted to handle certain aspects of the COVID-19 pandemic.

Studies have evaluated other mHealth applications in Saudi Arabia. For instance, Tawakkalna application was identified to be average in terms of users experience and efficiency [22]. Another study evaluated Seha mHealth application [23] and identified that it was effective in delivering healthcare services by improving access through streamlined appointment bookings. Similarly, Mawid application was also identified to be easy to use and the users were highly satisfied with the services provided by the application [24]. Furthermore, another study reviewed 12 mHealth applications in Saudi Arabia [9] and identified that different applications were developed for different purposes, such as contact tracing, awareness building, appointment booking, online consultation, etc.; therefore, analyzing the experiences using common scales may not deliver effective outcomes.

For example, Sehhaty was adapted to book appointments for COVID-19. SEHA was reformed to provide information about COVID-19 awareness. HESH was modified to include records for COVID-19 patients. The Mawid application, related to the objective of the present investigation, was improved during the pandemic to allow people to book, cancel, and change or schedule new appointments at 2400 healthcare primary centers located in Saudi Arabia. It can also provide information on COVID-19 awareness and can be used to trace COVID-19 patients [3,25]. In addition, the application can provide patients with information on the precautions that should be taken to avoid becoming infected with the COVID-19 virus in case of travel and other events [25].

Mawid is a mobile application provided by the Ministry of Health to enable patients to book, cancel and/or reschedule their appointments at primary healthcare centers, as well as manage their referral appointments [26]. It helps the users to assess the risk of COVID-19 contamination. Users are advised to enter symptoms and their travel details into the application for the risk assessment test. It also helps users in increasing awareness about COVID-19, and the precautionary measures to be taken, in addition to booking, cancelling and/or rescheduling appointments [25,27]. The assessment process using the application is presented in Figure 1. *Healthcare* **2022**, *10*, x 4 of 16

Users can book appointments at 2400 healthcare centers across Saudi Arabia, free of

Although several studies have been carried out in Saudi Arabia on the applications implemented during the COVID-19 pandemic by the health authorities of this country, it is necessary to know the opinion that patients have about the ease of use of these applications to adapt them as best as possible to the demands of the users [3,11,21,25]. In this sense, the objective of this study was to evaluate the patient experience with the MAWID

A quantitative cross-sectional survey was designed to evaluate the patient experience with the MAWID application during the COVID-19 pandemic in Al Hassa, Saudi Arabia. The questionnaire was developed using Google surveys, for which a survey link was generated, which is distributed to the participants through social media platforms (Facebook, Instagram, Twitter (San Francisco, CA, USA)). Community pages and groups related to Smart wearables were identified, and the survey link was posted in the pages and groups

Thus, the application can be an effective tool for not only delivering healthcare services, but also for tracking and monitoring epidemics. Considering its growing use, the application is evaluated in terms of its usability, user satisfaction, user experience, and usefulness.

application during the COVID-19 pandemic in Al Hassa, Saudi Arabia.

**Figure 1.** COVID-19 self-assessment test in Mawid application. requesting the members to participate in the survey Participants were asked to complete **Figure 1.** COVID-19 self-assessment test in Mawid application.

*2.1. Study Settings and Participants* 

**2. Methodology** 

Users can book appointments at 2400 healthcare centers across Saudi Arabia, free of cost, using the application. In addition, the application has provided >500,000 consultations regarding COVID-19 and >250,000 self-assessment tests have been taken by the users [27]. Thus, the application can be an effective tool for not only delivering healthcare services, but also for tracking and monitoring epidemics. Considering its growing use, the application is evaluated in terms of its usability, user satisfaction, user experience, and usefulness.

Although several studies have been carried out in Saudi Arabia on the applications implemented during the COVID-19 pandemic by the health authorities of this country, it is necessary to know the opinion that patients have about the ease of use of these applications to adapt them as best as possible to the demands of the users [3,11,21,25]. In this sense, the objective of this study was to evaluate the patient experience with the MAWID application during the COVID-19 pandemic in Al Hassa, Saudi Arabia.

#### **2. Methodology**

#### *2.1. Study Settings and Participants*

A quantitative cross-sectional survey was designed to evaluate the patient experience with the MAWID application during the COVID-19 pandemic in Al Hassa, Saudi Arabia. The questionnaire was developed using Google surveys, for which a survey link was generated, which is distributed to the participants through social media platforms (Facebook, Instagram, Twitter (San Francisco, CA, USA)). Community pages and groups related to Smart wearables were identified, and the survey link was posted in the pages and groups requesting the members to participate in the survey Participants were asked to complete the questionnaires and submit them electronically using the survey link. A total of 146 respondents completed the questionnaire. Ethical approval was granted by the Ethical Review Committees at the Imam Abdulrahman bin Faisal University.

#### *2.2. Inclusion and Exclusion Criteria*

This study included adult patients who had accessed the Mawid application at least once and were citizens of Al Hassa, Saudi Arabia. The rest of the people from Saudi Arabia were excluded.

#### *2.3. Questionnaire Design*

The questionnaire designed by the research team, and it consisted of two sections. The first section contained five questions related to the demographic information of participants (age, gender, marital status, education level, and profession). The second section included five questions about the ease of use of the MAWID app by patients during the COVID-19 pandemic in Al Hassa, Saudi Arabia (ease of logging into the program, clarity of the data and its arrangement within the program, ease of searching for the required information, ease to reserve an appointment, ease to undo an unwanted move, and ease of changing an appointment). These questions were evaluated using a 5-point Likert scale: very difficult (1), difficult (2), neutral (3), easy (4), and very easy (5). The questionnaire is shown in Appendix A (Table A1), with screenshot of electronic survey (Figure A1).

#### *2.4. Sampling Method*

A snowball random sampling technique was used to recruit the participants. The sample size included the total number of patients who completed and submitted the survey electronically. Furthermore, the participants were requested to forward the survey link to their friends and colleagues so that larger number of responses can be received.

#### *2.5. Data Collection*

The data were collected during 1 October 2021 to 31 October 2021. The purpose and significance of the study were explained to the respondents. They were informed that their participation was voluntary, and their responses will be confidential. A summary of the project and the researchers' contact details were provided to the participants. Participants were informed that answering and submitting the survey indicated that they consented to participate in the survey.

#### *2.6. Validity and Reliability of the Questionnaire*

The questionnaire was validated by three expert professors from the Imam Abdulrahman bin Faisal University. Furthermore, to evaluate the reliability of the questionnaire, a pilot study was carried out with five participants. Analysis of pilot study results revealed Cronbach alpha (α) value to be greater than 0.70 for all the relevant items, indicating good reliability and internal consistency [28].

#### *2.7. Data Analysis*

Data were extracted, revised, coded, and fed into IBM SPSS version 22 (Armank, NY, USA). The statistical analysis was performed using two-tailed tests. *p*-values less than 0.05 were considered statistically significant. For all the variables, a descriptive analysis was performed based on the distribution of frequencies expressed in percentages. The relationships between the categorical variables were tested using the Pearson χ 2 test.

#### **3. Results**

The demographic information of the respondents is shown in Table 1. The data indicated that more than half of the participants (80.2%) were below 40 years of age. Furthermore, most of them were female (97.5%) and married (78.1%). In addition, 71.2% of the participants were bachelor's degree holders, and 42.5% of the respondents worked in the government sector.


**Table 1.** Demographic information (*n =* 146).

Figures 2–7 illustrates the opinion of the respondents (very easy, easy, neutral, difficult, or very difficult) related to the ease of use of the Mawid application (ease of logging into the application, clarity of the data and its arrangement within the program, ease of searching for the required information, ease to reserve an appointment, ease to undo an unwanted move, and ease to change an appointment).

24.70%

40.40%

unwanted move, and ease to change an appointment).

participants were bachelor's degree holders, and 42.5% of the respondents worked in the

18–30 63 43.2% 31–40 54 37.0% >40 29 19.9%

Male 11 7.5% Female 135 92.5%

Single 23 15.8% Married 114 78.1% Divorced/widow 9 6.2%

Student 50 34.2%

High school student 29 19.9% Diploma 13 8.9% Bachelor 104 71.2%

Governmental employee 62 42.5% Privet sector employee 34 23.3%

Figures 2–7 illustrates the opinion of the respondents (very easy, easy, neutral, difficult, or very difficult) related to the ease of use of the Mawid application (ease of logging into the application, clarity of the data and its arrangement within the program, ease of searching for the required information, ease to reserve an appointment, ease to undo an

*n* **%** 

government sector.

**Table 1.** Demographic information (*n =* 146).

Age (years)

Gender

Marital status

Education level

Work sector

**Figure 2.** Ease of logging into the program of the Mawid application (*n =* 146). **Figure 2.** Ease of logging into the program of the Mawid application (*n =* 146).

*Healthcare* **2022**, *10*, x 7 of 16

**Figure 3.** Clarity of the data and its arrangement within the program (*n =* 146). **Figure 3.** Clarity of the data and its arrangement within the program (*n =* 146).

4.10%

2.10%

28.80%

**Figure 4.** Ease of searching for the required information (*n =* 146).

Very easy Easy Neutral Difficult Very difficult

25.30%

22.60%

43.80%

**Figure 3.** Clarity of the data and its arrangement within the program (*n =* 146).

4.10%

0.70%

26.00%

**Figure 4.** Ease of searching for the required information (*n =* 146). **Figure 4.** Ease of searching for the required information (*n =* 146). *Healthcare* **2022**, *10*, x 8 of 16

**Figure 5.** Ease to book an appointment (*n =* 146). **Figure 5.** Ease to book an appointment (*n =* 146).

31.50%

36.30%

Very easy Easy Neutral Difficult Very difficult

6.20%

3.40%

**Figure 6.** Ease to change an appointment (*n =* 146).

28.80%

34.90%

*Healthcare* **2022**, *10*, x 9 of 16

**Figure 6.** Ease to change an appointment (*n =* 146). **Figure 6.** Ease to change an appointment (*n =* 146).

**Figure 5.** Ease to book an appointment (*n =* 146).

24.00%

9.60%

2.70%

**Figure 7.** Ease to undo an unwanted movement (*n =* 146). **Figure 7.** Ease to undo an unwanted movement (*n =* 146).

Figure 2 shows that more than half of the participants (65.8%) thought that it was easy and very easy to log into the application program. Figure 2 shows that more than half of the participants (65.8%) thought that it was easy and very easy to log into the application program.

Figure 3 indicates that 69.1% of the respondents believed that it was easy and very easy to handle the application and believed that the data was adequately arranged in the application program. Figure 3 indicates that 69.1% of the respondents believed that it was easy and very easy to handle the application and believed that the data was adequately arranged in the application program.

Figure 4 displays that 65.1% of the participants considered that it was very easy and easy to search for the required information. Figure 4 displays that 65.1% of the participants considered that it was very easy and easy to search for the required information.

Figure 5 indicates that 63.7% of the respondents reflected that it was easy and very

Figure 6 shows that 58.9% of the participants expressed that it was easy and very easy

Finally, Figure 7 suggests that more than half of the participants (59.6%) expressed

On the other hand, Tables 2–5 shows the statistical relationship between the ease of use of the Mawid app (ease of logging into the application, clarity of the data and its arrangement within the program, ease of searching for the required information, ease to reserve an appointment, ease to undo an unwanted move, and ease to change an appoint-

**Age** 

*n* **%** *n* **%** *n* **%** 

**18–30 31–40 >40** *p-***Value** 

ment) and age, gender, educational level, and work location of the participants.

**Table 2.** Statistical relationship between the ease of use of the Mawid app and age (*n =* 146).

0.422 Easy 42 66.7% 33 61.1% 21 72.4%

Easy 44 69.8% 37 68.5% 20 69.0% 0.614

Neutral 17 27.0% 20 37.0% 6 20.7% Difficult 4 6.3% 1 1.9% 2 6.9%

Neutral 18 28.6% 13 24.1% 7 24.1%

that it was easy and very easy to undo unwanted movements.

easy to book an appointment.

to change an appointment.

Ease of logging into the program

Clarity of the data and its arrangement within the program

Figure 5 indicates that 63.7% of the respondents reflected that it was easy and very easy to book an appointment.

Figure 6 shows that 58.9% of the participants expressed that it was easy and very easy to change an appointment.

Finally, Figure 7 suggests that more than half of the participants (59.6%) expressed that it was easy and very easy to undo unwanted movements.

On the other hand, Tables 2–5 shows the statistical relationship between the ease of use of the Mawid app (ease of logging into the application, clarity of the data and its arrangement within the program, ease of searching for the required information, ease to reserve an appointment, ease to undo an unwanted move, and ease to change an appointment) and age, gender, educational level, and work location of the participants.

**Table 2.** Statistical relationship between the ease of use of the Mawid app and age (*n =* 146).



**Table 3.** Statistical relationship between the ease of use of the Mawid app and gender (*n =* 146).

\* Statistically significant difference.

**Table 4.** Statistical relationship between the ease of use of the Mawid app and educational level (*n =* 146).



\* Statistically significant difference.

**Table 5.** Statistical relationship between the ease of use of the Mawid app and working sector (*n =* 146).


\* Statistically significant difference.

Table 2 suggests that there was not a significant statistical relationship between the ease of use of the Mawid app and age.

Table 3 indicates that there was a significant statistical difference between the ease of searching for the required information and gender (*p*-value = 0.006), and between the ease to undo an unwanted move and gender (*p*-value = 0.49).

Table 4 describes that there was a significant statistical difference between the ease of searching for the required information and educational level (*p*-value = 0.048), and between the ease to reserve an appointment and educational level (*p* = 0.049).

Table 5 suggests that there was a significant statistical difference between the ease of searching for the required information and the working sector of respondents (*p* value = 0.049).

#### **4. Discussion**

The findings of this study on the evaluation of the patient experience with the Mawid application during the COVID-19 pandemic in Al Hassa, Saudi Arabia, suggested that more than half of the respondents thought that it was easy and very easy to log into the application and that the data was adequately arranged in the app program. Moreover, these participants believed that it was easy and very easy to search for the required information, to reserve an appointment, undo an unwanted move, and change an appointment. Less than a third of the participants had a neutral opinion regarding the mentioned attributes of the application. The rest of the participants, about a tenth of them, believed that it was difficult and very difficult to manage the features of the application. It is important to mention that since the COVID-19 outbreak in Saudi Arabia, the application has delivered consultation services for more than half a million people [27,29].

Similarly, a previous study conducted in Saudi Arabia showed that more than 50% of the participants who had accessed the Mawid app at least once considered it had a good interface to find the information on COVID-19 available in Saudi Arabian healthcare centers [24]. This interface facilitated the search for medical appointments, remote communication between patients and doctors, and knowledge of the symptoms and treatment of COVID-19 [24]. Furthermore, according to the participants, most of the instructions of the Mawid app were easy to follow. In general, people and Saudi Arabian government health authorities have found that the Mawid app was easy to use [27,29,30].

In a different context, a study conducted in the Netherlands revealed that participants positively valued the information available in an app designed to offer education, selfassessment, and monitoring of the COVID-19 [31]. Another study conducted in the United States found that the MyCOVIDKey app was a useful tool for COVID-19 contact tracing, but it needed simple modifications to improve usability [32]. Similarly, in a study carried out in England, participants viewed the NHS COVID-19 app positively [33]. However, the respondents considered that the interface was challenging, difficult, and complex. They thought that these factors would limit its use by the UK population (79) [33].

On the other hand, there was a statistically significant difference between the ease of searching for the required information and gender, the ease of undoing an unwanted move and gender, the ease of searching for the required information and educational level, the ease of booking an appointment and educational level, and the ease of searching for the required information and the labor sector of the respondents. However, there was no statistically significant relationship between the ease of use of the Mawid app and the rest of the variables. A preceding study also revealed that there were significant statistical differences between the ease of use and age, and between the ease of use and gender [24].

The main limitation of this study was the small sample size, which limits the generalizability of the results on the ease of use of the Mawid app by COVID-19 patients in Al Hassa, Saudi Arabia. In addition, this study only considered Mawid application in specific, while there are also other applications being introduced by MoH, Saudi Arabia. Moreover, this study was conducted in single setting (Al Hassa city), even though Mawid application is used across Saudi Arabia. Future studies should aim to increase the sample size of participants, including other respondents from different cities in Saudi Arabia. It would also be interesting to assess the level of acceptance of the applications used during the COVID-19 pandemic in Saudi Arabia.

Despite limitations, this study has both theoretical and practical implications. Firstly, the study contributes to the literature relating to mHealth applications in the context of

Middle East. Secondly, the findings from this study can aid decision-makers such as Ministry of Health in improving the Mawid or other similar applications according to the attitudes of the users (findings from this study). Furthermore, the findings can also be generalized in the Middle east context where similar applications are developed and implemented.

#### **5. Conclusions**

Overall, most participants suggested that the Mawid app was easy to use and had a potentially useful set of features to help mitigate and manage the COVID-19 pandemic in Al Hassa, Saudi Arabia. In addition, there was a statistically significant difference between the ease of searching for the required information and gender, the ease of undoing an unwanted move and gender, the ease of searching for the required information and educational level, the ease of booking an appointment and educational level, and the ease of searching for the required information and the labor sector of the respondents.

**Author Contributions:** Conceptualization, E.A., R.A.-D., S.A. (Salma Almulla), F.A., Z.A., L.S., N.A. (Nouf Alaenzi); H.A., N.A. (Naif Alenazi), B.H. and T.A.; methodology, R.A.-D., S.A.-R., S.A., F.A., L.S., R.W.A., H.A., B.H. and T.A.; validation, E.A., Z.A., L.S. and L.A.; formal analysis, E.A., R.A.- D., N.A.-A., D.A., F.A., N.A. (Norah Alnaim), R.W.A. and N.A. (Naif Alenazi); investigation, E.A., R.A.-D., D.A., S.A., Z.A., N.A. (Norah Alnaim), R.W.A., N.A. (Nouf Alaenzi), H.A., B.A. and T.A.; resources, N.A. (Norah Alnaim), writing—original draft preparation, E.A., S.A.-R., N.A.-A., L.S., N.A. (Nouf Alaenzi), H.A., B.A., L.A., N.A. (Naif Alenazi), B.H. and T.A.; writing—review and editing, D.A.; visualization, E.A. and N.A. (Naif Alenazi); project administration, F.A., R.W.A., N.A. (Nouf Alaenzi), B.A. and N.A. (Naif Alenazi); funding acquisition, N.A.-A., L.S., B.H. and T.A. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Imam Abdulrahman Bin Faisal University, and approved by the Institutional Review Board IRB-2021-03-242.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy clause.

**Conflicts of Interest:** No conflict of interest were identified.

#### **Appendix A.**

*Appendix A.1. Survey Questionnaire*

Measuring the Satisfaction Level among Patients Regarding the Mawid App in PHC in Al Hassa, KSA

Greetings,

Your participation and response to this survey will help us achieve our goal as postgraduate students in healthcare quality and patient safety at Imam Abdulrahman Bin Faisal University and contribute to

the study of the customer's satisfaction about the use of the Mawid app. Targeted group:


The purpose of this study:

Through this study, we seek to measure the satisfaction level of the customers about the ease of use of the Mawid app in Al Hassa region in Saudi Arabia. The main objective is to evaluate the patient experience with the MAWID application during the COVID-19 pandemic in Al Hassa, Saudi Arabia.

Note:


I hope you fill out the following Questionnaire:

Please answer the following questions:

1. Measuring the level of satisfaction with the Mawid program

**Table A1.** Measuring the level of satisfaction with the Mawid program.


	- ∗ Age: (18–30/31–40/>41)
	- ∗ Gender: (male/female)
	- ∗ Social status (single/married)
	- ∗ Academic level (secondary/diploma/bachelor's/postgraduate)
	- ∗ Profession: (student/government employee/private sector employee)

Thank you *Healthcare* **2022**, *10*, x 15 of 16

**References** 

(accessed on 28 February 2022).

**2020**, *27*, 208–211.

*Multidiscip. Healthc.* **2021**, *14*, 45–57.

public perceptions. *Crit. Public Health* **2021**, *32*, 31–43.

approach. *JMIR Public Health Surveill.* **2020**, *6*, e20572.

they-ve-fa (accessed on 27 February 2022).

concerns are high. *Comput. Hum. Behav.* **2021**, *119*, 106718–106718.


1. Kumar, A.; Singh, R.; Kaur, J.; Pandey, S.; Sharma, V.; Thakur, L.; Sati, S.; Mani, S.; Asthana, S.; Sharma, T.K.; et al. Wuhan to

4. Keshvardoost, S.; Bahaadinbeigy, K.; Fatehi, F. Role of telehealth in the management of COVID-19: Lessons learned from pre-

6. Times of India. Govt Highlights Advantage of Aarogya Setu, but Stops Short of Making it Compulsory in New Guidelines. Available online: https://timesofindia.indiatimes.com/india/govt-highlightsadvantage-of-aarogya-setu-but-stops-short-of-

7. Kyodo. Japan's Coronavirus Contact-Tracing App Launched Amid Privacy Concerns. Available online: https://www.japan-

8. Davidson, H. Chinese City Plans to Turn Coronavirus App into Permanent Health Tracker. Available online: https://www.theguardian.com/world/2020/may/26/chinese-city-plans-to-turn-coronavirus-app-into-permanent-health-tracker

9. Turki, A. A review of mobile applications available in the app and google play stores used during the COVID-19 outbreak. *J.* 

10. Zimmermann, B.M.; Fiske, A.; Prainsack, B.; Hangel, N.; McLennan, S.; Buyx, A. Early perceptions of COVID-19 contact tracing apps in German-speaking countries: Comparative mixed methods study. *J. Med. Internet Res.* **2021**, *23*, e25525. 11. Alghamdi, S.; Alqahtani, J.; Aldhahir, A. Current status of telehealth in Saudi Arabia during COVID-19. *J. Fam. Community Med.*

12. Samuel, G.; Roberts, S.L.; Fiske, A.; Lucivero, F.; McLennan, S.; Phillips, A.; Johnson, S.B. COVID-19 contact tracing apps: UK

13. Chan, E.Y.; Saqib, N.U. Privacy concerns can explain unwillingness to download and use contact tracing apps when COVID-19

14. Privacy Fears Stop Us Using COVID Contact Tracing Apps. It's Not the Only Reason They've Failed. Available online: https://www.euronews.com/next/2021/08/05/privacy-fears-stop-us-using-covid-contact-tracing-apps-it-s-not-the-only-reason-

15. Elkhodr, M.; Mubin, O.; Iftikhar, Z.; Masood, M.; Alsinglawi, B.; Shahid, S.; Alnajjar, F. Technology, privacy, and user opinions of COVID-19 mobile apps for contact tracing. Systematic search and content analysis. *J. Med. Internet Res.* **2021**, *23*, e23467. 16. Walrave, M.; Waeterloos, C.; Ponnet, K. Adoption of a contact tracing app for containing COVID-19: A health belief model

2. WHO Coronavirus (COVID-19) Dashboard. Available online: https://covid19.who.int/ (accessed on 28 February 2022). 3. Alassaf, N.; Bah, S.; Almulhim, F.; AlDossary, N.; Alqahtani, M. Evaluation of official healthcare informatics applications in

Saudi Arabia and their role in addressing COVID-19 pandemic. *Health Inform. Res.* **2021**, *27*, 255–263.

COVID-19 mobile apps: A systematic review of the literature. *J. Med. Internet Res.* **2020**, *22*, e23170.

making-itcompulsory-in-new-guidelines/articleshow/75793837.cms (accessed on 28 February 2022).

times.co.jp/news/2020/06/19/national/japan-contact-tracing-app-launched/ (accessed on 28 February 2022).

**Figure A1.** Screenshot of the survey questionnaire. **Figure A1.** Screenshot of the survey questionnaire.

World: The COVID-19 pandemic. *Front. Cell. Infect. Microbiol.* **2021**, *11*, 596201.

#### **References**


## *Review* **A Narrative Review of the Launch and the Deployment of Telemedicine in Italy during the COVID-19 Pandemic**

**Daniele Giansanti 1,\*, Giovanni Morone <sup>2</sup> , Alice Loreti <sup>3</sup> , Marco Germanotta <sup>4</sup> and Irene Aprile <sup>4</sup>**


**Abstract:** Telemedicine is making an important contribution to the fight against the COVID-19 pandemic and to supporting the health domain. Its use has registered initial problems with oftenpatchy practise. The objective of this study was to analyze the launch and deployment of telemedicine in Italy through a narrative review. The narrative review faced two points of view: (a) the first point of view revised the institutional initiatives of the Italian government developed to promote the use of telemedicine; (b) the second point of view reviewed the evolution of scientific literature in the sector, with reference to the Italian situation. In the second point of view, we applied both a standard narrative checklist and an eligibility approach. The first point of view reported an analysis of national documents aimed at promoting, through indications and recommendations, the use of telemedicine. The second point of view analyzed 39 qualified references. The analysis highlighted: (a) that initially, there was a disorientation, followed by reflections that emerged immediately after; (b) a telemedicine application not only in the traditional sectors (e.g., diabetology, cardiology, oncology, neurology) but also in new and fields never explored before; and (c) a high level of acceptance and a desire to continue in the after-pandemic future (which emerged in some studies through dedicated questionnaires). The study offers stimuli for both stakeholders and scholars to improve the use of telemedicine during the pandemic and in the future.

**Keywords:** COVID-19; telemedicine; digital health; eHealth; mHealth; telehealth; telemonitoring; telerehabilitation

#### **1. Introduction**

Telemedicine as a diagnostic, monitoring, and rehabilitation treatment tool is showing great potential during the pandemic, as highlighted by Bahsnur et al. [1]. We believe that the current spread of telemedicine, compared with previous pandemic situations, can be explained by the simultaneous occurrence of unprecedented conditions of technological availability and exceptional medical circumstances. In fact, one aspect that drove this boom was the vastness of the pandemic, the most terrible of the past century; however, the real engine of the boom of telemedicine during the epidemic for the SARS-CoV-2 virus of 2019, has been mobile technology based on smartphones [2]; these are capable of hosting telemedicine applications, for example, based on wearable sensors, which in the past needed proper technological solutions [3]. During each wave of the pandemic's evolution, telemedicine has shown exponential development [2]. In reference to the Italian situation, despite the available innovative solutions, cultural barriers and organization limits did not allow an easy introduction of these solutions to the health domain. The introduction happened with lights and shadows [4]. We can identify two periods of maturation of the use of telemedicine during the pandemic. During the first period, around the first lockdown

**Citation:** Giansanti, D.; Morone, G.; Loreti, A.; Germanotta, M.; Aprile, I. A Narrative Review of the Launch and the Deployment of Telemedicine in Italy during the COVID-19 Pandemic. *Healthcare* **2022**, *10*, 415. https://doi.org/10.3390/ healthcare10030415

Academic Editors: Tao-Hsin Tung and Andrea Tittarelli

Received: 3 December 2021 Accepted: 14 February 2022 Published: 23 February 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

(March–July 2020), two critical issues were found. The first issue can be summarized as a lost opportunity to widely spread telemedicine for long-term patients, as highlighted by Omboni [5]. The author believes that Italy was unprepared to use telemedicine in the first phase of the emergency. The second issue can be summarized as a lost opportunity to provide adequate telemedicine services to subjects renouncing the emergency recovery. Vigano et al. [6], highlighted that (a) in the first phase of the pandemic there was a significant decline in the number of patients who accessed the emergency room, and hospitalized patients; (b) this would have caused a presumable increase in health needs in the immediate future, to possibly be addressed with remote techniques, which were not completely usable at the time. After the first lockdown, a period of maturation can be identified, in which responses to some critical issues began to appear; these were based exceptional needs related to the pandemic that emerged, and several institutional indications on the use of telemedicine and in the area of interventions were produced. In particular, numerous reports by the Istituto Superiore di Sanità (the Italian National Institute of Health), focused both on telemedicine and on sectors needing telemedicine, have been published (such as, for example, the reports in [7–10]). Other stakeholders, such as the Regional Government and the Italian Ministry of Health, have also activated initiatives to promote the use of telemedicine. We can report as political initiatives:


#### **2. Purpose of the Study**

The purpose of the study was to address the launch and deployment of telemedicine in Italy through a narrative review, addressing both the promotion initiatives at national level and the experiences of application also at the level of acceptance. All of this facilitates both answering the important question of "how telemedicine started in Italy during the pandemic and how it is going", and making a point about the use of telemedicine in Italy.

#### **3. Methods**

This narrative review faced two points of view: (a) the first was aimed at analysing institutional government initiatives designed to promote the use of telemedicine and to raise awareness among stakeholders; (b) the second was aimed at analysing the evolution of scientific literature in the sector, through a Pubmed overview, with reference to the Italian situation. We followed both a narrative checklist and an eligibility approach, based on a scoring system (with different parameters and a score with five levels) applied by two qualified experts, to include each reference found in the second point of view.

We followed the narrative checklist reported in [13].

The manuscript was developed in accordance with this checklist, which requires compliance with some qualifying points in the development of the document, starting from the title and ending with the conclusions.

Table 1 shows the parameters used for the qualification process applied, before inclusion, on the references found in the second point of view.

We assigned a score to these parameters, ranging from a minimum score of one (very poor) up to a maximum of five (outstanding). As far as the "added contribution to the field" parameter is concerned, we used weighing.

Both to consider the criticality of the first phases of the pandemic and to relativize the reference to the importance of the first period, the assigned vote was multiplied by:

• A factor × 1.3 (for studies published in the first three months of the pandemic).

• A factor × 1.15 (for studies published in the period ranging from 1 to 6 months of the pandemic).

**Table 1.** Parameters used for the eligibility.


\* 1.3 was used for the studies published in the first three months of the pandemic, and 1.15 was used for the studies published in the period ranging from 1 to 6 months of the pandemic.

The study was excluded if, regardless of the score, there were critical issues of conflict of interest (for example, if it was conducted without guarantees of objectivity by the system manufacturer). The reference was included in the review if all parameters after weighing showed a score higher than three in AND logic.

The keys reported in Table 2 were applied in the second point of view.

**Table 2.** The keys applied in the search (COVID-19 was also changed with SARS-Cov-2 during the searches).


#### **4. Results**

*4.1. An Overview of Italian National Recommendations and Indications*

In Italy, an important role in fighting the pandemic was played by the Istituto Superiore di Sanità (ISS). The ISS defined various working groups [14] with ISS researchers and experts on the various strategic issues related to the fight against the pandemic. The groups also worked in synergy with each other. Important products from the working groups were the ISS COVID-19 Reports containing guidelines and recommendations for all the insiders in the health domain. The COVID-19 Reports provide essential and urgent information for emergency management and are subject to updates. These reports were produced in the national language] and translated into English or other languages [15] to share/export the knowledge. We accessed the online archive and focused on content dedicated to telemedicine. Table 3 shows the COVID-19 reports that dealt directly or indirectly with telemedicine.


**Table 3.** ISS reports dealing with the telemedicine.

The first document [7] provided support for the realization of services in Telemedicine during a COVID-19 emergency, offering indications, identifying operational problems, and proposing solutions supported by evidence, but that are also easily dispensable in practice. The indications aimed to be used in various combinations to provide health services and psychological support; they also aimed to proactively monitor the health conditions of people in quarantine, in isolation, after discharge from the hospital, or of those who were isolated at home due to the rules of social distancing but were in need of continuity of care, even if they were not COVID-19 infected. The second document [8] provided the scientific indications to support the implementation of telemedicine health services for pediatric patients, both in early childhood and in developmental age, and during the different phases of the COVID-19 pandemic. It described how telemedicine can solve operational problems in managing the doctor–patient–family relationship in the pediatric field. It also provided concrete elements for the definition of specific characteristics, and the eligibility and exclusion criteria of the pediatric patient, also affected by rare or common chronic diseases.

The third document [9] illustrated the results of the development and submission of a survey (in September 2020) proposed by the National Centre for Innovative Technologies in Public Health and the National Centre for Rare Diseases of the ISS; its aim was to investigate the state of use of technologies (also based on telemedicine) by people with frailty, disabilities, and rare diseases. The document was intended to report evidence to stakeholders through the survey tool which played a sensor role. The fourth document [10] reported guidelines for the appropriate support of children with adrenal insufficiency during the current SARSCoV-2 pandemic emergency. Contact with reference centers to ensure advice from specialists was highly recommended, also using telemedicine systems.

All four of the documents [7–10] also highlighted particular attention to rare diseases and the frailty towards which telemedicine can play an important supporting role.

#### *4.2. An Overview of Italian National Scientific Literature Production*

The eligibility process applied to the selected references, after the elimination of the duplicated ones, returned 39 works [5,16–53], including a review focused on the relationship between telemedicine and radiotherapy [39].

A total of 17 studies were published in 2021, and the remaining 22 in 2020.

The eligibility process also showed that the selected papers did not show critical issues regarding conflict of interest.

Figure 1 reports the average scores assigned by the two experts after the weighing process, both for each parameter and averaged for all the parameters.

14/21

24/20

[9] Giansanti D. et al. Technologies to support frailty, disability and rare diseases: development and submission of a survey during the pandemic emergency COVID-19. Version from 18 June 2021.

[10] ISS COVID-19 Rare Diseases Working Group Interim guidelines for the appropriate support of children with adrenal insufficiency during the current SARSCoV-2 pandemic emergency. Version from 10 May 2020.

*4.2. An Overview of Italian National Scientific Literature Production* 

process, both for each parameter and averaged for all the parameters.

relationship between telemedicine and radiotherapy [39].

issues regarding conflict of interest

The eligibility process applied to the selected references, after the elimination of the duplicated ones, returned 39 works [5,16–53], including a review focused on the

The eligibility process also showed that the selected papers did not show critical

Figure 1 reports the average scores assigned by the two experts after the weighing

A total of 17 studies were published in 2021, and the remaining 22 in 2020.

Outcome from a survey on the use of technologies (also telemedicine) during the pandemic

Interim guidelines for the appropriate support of children with adrenal insufficiency also using telemedicine

**Figure 1.** Output from the qualification process. **Figure 1.** Output from the qualification process.

Table 4 reports the references of the studies selected in the narrative review using the eligibility process with a summary of their focus. The studies show: Table 4 reports the references of the studies selected in the narrative review using the eligibility process with a summary of their focus.







The studies show:


#### 4.2.1. Disorientation in Telemedicine Applications Emerging in Some Studies

The study reported in [5] expressed dissatisfaction with the lost opportunity to widely spread telemedicine during the lockdown, wherein it is reported, and quoted verbatim: "Italy was found unprepared to manage lockdown patients with chronic diseases, due to limited availability and diffusion of large-scale telemedicine solutions." Among the specific causes hindering the implementation of effective telemedicine solutions, the author indicates, specifically for long-term patients management: (a) the scattered distribution and heterogeneity of available tools; (b) the lack of integration with the electronic health record of the national health system; (c) the poor interconnection between telemedicine services operating at different levels; (d) the lack of a real multidisciplinary approach to the patient management; and (e) the heavy privacy regulations and lack of clear guidelines, together with the lack of reimbursement.

In addition, the study reported in [51] emphasized that Italy did not include telemedicine in the essential levels of care granted to all citizens within the National Health Service, while other nations authorized, reimbursed, and actively promoted the use of telemedicine. The study stimulated the stakeholders to take action in the direction of telemedicine. The study in [33] discussed the legal problems in telemedicine delivery, ranging from profiles on the subject of authorization and accreditation to those concerning the protection of patient confidentiality.

#### 4.2.2. Reflections Emerging after the Italian National Lockdown

The study in [46] highlighted that: (a) the Italian lockdown model (in March–May 2020) has been imitated by many other states; (b) Italy was probably not a model in the use of telemedicine. However, there was an opportunity to reflect on this and inspire models that could be useful after the first lock down period. The following sectors on which to focus during the pandemic were detected [46]:


#### 4.2.3. Collected Evidence of Telemedicine Use

By analyzing the publications found in Pubmed to date, we can trace a picture of telemedicine use in the health domain regarding monitoring, surveillance, and continuity of care.

We find various applications of telemedicine in diabetology [17,22,23,27,41] and also in children [31], where we see the use of ICT integration tools with self-assessment devices. Cardiology has also recorded the use of telemedicine. The study in [48] described a telemedicine experience in heart failure management during COVID-19. The study in [36] analyzed data from three telemedicine centers focusing on heart care. The study in [40] reported the evolution of a multidisciplinary center for myocarditis towards a telemedicine system. Important applications are recorded in oncology, such as in breast cancer [21], lung cancer [25], connections to the stoma centers [29], the management of patients with differentiated thyroid cancer [39], and related radiotherapy applications [35]. The neurology sector has also recorded an important use of applications in Parkinson's disease [44]; in the use of telemedicine for the multidisciplinary assessment of patients with Amyotrophic Lateral Sclerosis [47]; for multiple sclerosis [28]; for a headache specialist center; and in applications of telephysiotherapy [47]. The overview also reported the application of telemedicine in other sectors less common in telemedicine solutions, such as: colorectal surgery [24]; wound care [20]; urological benign diseases [32]; endocrinology [34]; inflammatory bowel disease [37]; teleconsulting with a bariatric center; and chronic liver disease [42]. There has been the use of systems, in some simple cases, such as telephony [32,50], and in other more complex cases, such as specialized servers [53], which have allowed the application of telemedicine with success.

#### 4.2.4. Example of the Expansion of the Boundaries

The boundaries of the use of telemedicine have been expanded during the pandemic. Three examples are show in transgender mental health monitoring [26], in the field of the abortion [18,19], and in the field of the animal-assisted therapy. Transgender people are a vulnerable group with a higher incidence of mental health issues and, during the COVID-19 outbreak, they may have faced psychological, physical, and social obstacles. The study in [26] evaluated the impact of the pandemic and access to health care services during the COVID-19 pandemic on the mental health of transgender people living in Italy. An anonymous web-based survey was conducted among transgender people living in Italy. It highlighted how telemedicine services may serve to mitigate negative psychological effects. The studies in [18,19] focused on the application of telemedicine in the field of abortion. Induced abortion is legal in Italy, but with restrictions. The online abortion provider Women on Web serves as an alternative way to access abortion. The study highlighted an increase in requests during the COVID-19 pandemic compared with the previous years without a pandemic (when its use was not sensibly appreciable). The most common reasons for requesting a telemedicine abortion through the system were privacy-related; however, this shifted to COVID-19-specific reasons during the pandemic. Another example of the expansion of the boundaries of telemedicine in complementary and alternative medicine is reported in [54]. The latter is a survey that was administered remotely to quantify the impact of animal-assisted therapy during the lock down. Through the survey, which also reported as a self-assessment test for anxiety, it was shown that pet owners had lower levels of anxiety.

#### 4.2.5. Examples on the Acceptance of Use

Several studies have accompanied the use of telemedicine with the application of questionnaires (in some cases even standardized) to investigate acceptance and satisfaction [21,22,25,29,38,43,47], from which the desire to continue with telemedicine even in post-pandemic periods have also clearly emerged, directly or indirectly. Telemedicine received a high degree of acceptance, for example, in oncology, where both a study on patients with breast cancer [21] and in the output of a project on lung cancer monitoring [25]

displayed this. The study in [22] investigated the individual and contextual determinants of the perceived quality of telemedicine and teleassistance services, and willingness to continue with them among patients with diabetes. The study showed both a high level of acceptance and several determinants. These socio–demographic and correlated factors should be considered in the implementation of care pathways integrating in-person visits with the telemedicine. In addition, applications in neurology showed a high acceptance, as in the case of a study of the multidisciplinary assessment of patients with amyotrophic lateral sclerosis using telemedicine [47], and in a study embedding telephysiotherapy services [43]. Notably, the study in [38] also assessed the positive impact of a teleconsulting technology in a single bariatric center. Both interesting and innovative for oncology is the study in [29], which reported a high level of acceptance from patients involved in the experience of telemedicine consultations at a stoma center.

#### **5. Discussion**

The COVID-19 pandemic, as highlighted by Negrini et al. [4], represented an important engine for the development of telemedicine in Italy. Here, we have seen two important phases in the launch of the telemedicine. Many critical issues in the second phase have been addressed, and efforts have been made to improve the usability of telemedicine services, as well as standardization aspects, through institutional and political actions.

In this study we resumed these changes and reported an overview based on two points of view.

The first point of view reported the initiatives for issuing recommendations and indications for the use of telemedicine by the ISS through public reports [7–10]; these were in the national language and translated, in many cases, into English, and in other cases, into other languages. These reports have been a particular stimulus on the national scene for the use of broad-spectrum telemedicine, particularly in the case of various types of frailties, and also due to rare diseases.

The second point of view reported an analysis of the literature from Pubmed to examine the spread of telemedicine. This analysis highlights:


If we compare the development of telemedicine during the pandemic in Italy with the USA, a nation that showed one of the best telemedical preparedness [1], we can highlight some important considerations. As we highlighted in [54] in a comment to [1], the telemedicine boom during the COVID-19 has not been identical across the world, for example it was different between the USA, Italy and Europe. Different regulations, and a less enlightened and more conservative political approach have, in many cases, hampered the spread of telemedicine in the first months of the pandemic. To cite a first example, whereas in the United States, the system based on medical insurance has clearly defined the reimbursement procedures, in Italy and in Europe this has not happened so explicitly. In USA, there were immediately derogations to the law for the use of messaging and video communication systems to be applied to telemedicine. Europe has not clearly made explicit derogations to current regulations in the first phases of the pandemic. However, after an initial disorientation, and some phasing initiatives, as shown in [46], telemedicine

began to be used and stimulated, supported by a public- and equity-based healthcare approach. Then, in the USA—where the health system itself, based on private insurance, had allowed a rapid response to the use of telemedicine—scholars began to question, after a few months, the disparities and inequalities of telemedical treatment based on a private health system [55].

#### *Limitations*

The study, based on a review of Pubmed and of the ISS online database, has limitations. As regards the publications, it analyzed scientific productions in the English language. It did not analyze publications in other languages (Spanish, French or Italian). It analyzed only the online database Pubmed, which, however, is strategic in the health domain, where the overview is focused. It is not a systematic review, given that the topic (the launch of a technology) required a type of investigation based on polyhedral sources (some nonscientific publications) and specific filtering more suitable for other types of reviews, such as narrative reviews, realistic reviews, hermeneutical reviews, rapid reviews, or simple overview reviews (all reviews admitted in the journal).

#### **6. Conclusions**

In conclusion, the study highlights that: (a) in Italy, after the first moment of disorientation, telemedicine was used broadly and effectively; (b) new fields of telemedicine application were also explored; (c) dedicated questionnaires showed a high level of acceptance of telemedicine, and a desire to continue using this technology; (d) important suggestions emerged to invest in the use of telemedicine during the pandemic, and in the future after the pandemic.

By comparing the results of this study with other studies focused on other realities based on a different approach to the health system (for example, a private approach), we can highlight how the COVID-19 pandemic has been a stimulus for the development and use of telemedicine, and for the reviewing of regulations and policies in order to improve the use of this service, which can represent an instrument of equity and protection (thanks to social distancing) in this period, and an opportunity for the future.

**Author Contributions:** Conceptualization, D.G.; methodology, D.G., A.L., M.G., I.A. and G.M.; software, all; validation, all; formal analysis, D.G., A.L., M.G., I.A. and G.M.; investigation, all; resources, all; data curation, all; writing—original draft preparation, D.G.; writing—review and editing, D.G., A.L., M.G., I.A. and G.M.; visualization, all; supervision, D.G.; project administration, D.G. and A.L. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Project Report* **Technologies to Support Frailty, Disability, and Rare Diseases: Towards a Monitoring Experience during the COVID-19 Pandemic Emergency**

**Daniele Giansanti 1,\*, Antonia Pirrera <sup>1</sup> , Paola Meli <sup>1</sup> , Mauro Grigioni <sup>1</sup> , Marta De Santis <sup>2</sup> and Domenica Taruscio <sup>2</sup>**


**Abstract:** This report illustrates the design and results of an activity of surveillance proposed by the National Centre for Innovative Technologies in Public Health and the National Centre for Rare Diseases of the *Istituto Superiore di Sanità* with the aim of monitoring the state-of-use of technologies by people with frailty, disabilities, and rare diseases. The results of the surveillance activity reported in this report are as follows: *(a) An international Webinar*; *(b) A Full report published by the Istituto Superiore di Sanità* (ISS); *(c)* an electronic survey tool, for periodic monitoring; *(d)* an initial summary of the survey (15 September–30 November 2020), giving an overall picture relating to the state-of-use of technologies by the interviewed; *(e)* an understanding of the needs that emerged, causing reflection on the current state-of-the-art and offering important stimuli for all the stakeholders involved.

**Keywords:** COVID-19; SARS-CoV-2; frail people; rare diseases; remote assistance; remote rehabilitation; survey; technology

#### **1. Introduction**

The World Health Organization estimates that over one billion people live with some form of disability [1]. This corresponds to approximately 15% of the world population, with up to 190 million (3.8%) people aged 15 and over. The number of people with disabilities is also increasing, due to the progressive aging of the population and the increase in chronic health conditions. Disability is extremely varied, and some associated clinical situations can result in pathological conditions that require extensive healthcare needs. However, in general, all people with disabilities, as well as all other citizens, have the right to access traditional health services. Although Article 25 of the United Nations Convention on the Rights of Persons with Disabilities reinforces the rights of people with disabilities to achieve the highest standards of healthcare without discrimination, in reality, there are still few countries that provide adequate and quality services.

Furthermore, very few countries collect disaggregated data by disability in the health sector, and this has become much more evident and burdensome during the emergency caused by COVID-19; there has been no consistent inclusion in the responses put in place to control the pandemic. People with disabilities do not always receive adequate support. On the contrary, they are often exposed to risks with serious consequences of contracting COVID-19, develop severe COVID-19 symptoms, and have the potential of worsening health, both during and after the pandemic [2].

Focusing attention on the national territory, Istat (The Italian National Institute of Statistics) estimates that 3.1 million disabled people, in Italy, constitute 5.2% of the resident population [3]. Of these, almost 1.5 million are represented by the elderly over 75 (i.e., more

**Citation:** Giansanti, D.; Pirrera, A.; Meli, P.; Grigioni, M.; De Santis, M.; Taruscio, D. Technologies to Support Frailty, Disability, and Rare Diseases: Towards a Monitoring Experience during the COVID-19 Pandemic Emergency. *Healthcare* **2022**, *10*, 235. https://doi.org/10.3390/ healthcare10020235

Academic Editor: Alessandra Gorini

Received: 12 December 2021 Accepted: 19 January 2022 Published: 26 January 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

than 20% of the population in that age group). If we also add to this number the people who declare they have minor limitations, the total number of people with disabilities in Italy rises to 12.8 million. There is talk of different types of disabilities, ranging from the highest degree of difficulty in the essential functions of daily life, to much milder limitations, including chronic diseases such as diabetes, heart disease, chronic bronchitis, liver cirrhosis or malignancy, senile dementias, behavioral disorders, and rare diseases [3,4].

It is evident that, for such a large group of citizens with specific needs and fragility, technological resources represent an indispensable tool for the continuity of care/therapy, and, in the COVID-19 era, these are transformed into an essential lifeline. We should consider that there have been several new technological proposals because of COVID-19, such as [5,6], but not with a special focus on frail people. Accessibility and the use of technologies are not only a current issue; they are also vital for persons living with a disability, because they can make a significant difference to life quality. This depends a great deal on both the offer of continuity of remote care and on how it is possible to cope with the problem of the *Digital Divide* that, where available, hinders access. The *Digital Divide* concerns the gap between those who have effective access to information technologies and those who are partially or totally excluded from it. The *Digital Divide* has three polarities/levels of intervention. The *first level* of the *Digital Divide* is represented by the difficulty in access to the infrastructures; this remains a problem, even in the richest and most technologically advanced countries in the world [7]. The *second level* is represented by literacy, characterized by the skills that enable individuals to seek, understand, and use information in ways that promote and maintain health based on Digital Health [8]. The *third level* is represented by the potential benefit level [9]. This concerns the extent to which economic, cultural, social, and personal types of engagement with the Internet result in a variety of economic, cultural, social, and personal outcomes. The three levels of the digital divide are also evident during the pandemic [10–15], where digital resources are fundamental.

#### **2. The Idea of the Surveillance Project**

The COVID-19 pandemic and the consequent obligation of social distancing has offered a great stimulus for the development of digital technologies for the continuity of treatments and cures; however, the limits of effective access to these digital technologies have often exacerbated the disparity [15], accentuating the difficulties that "frail people", their families, and caregivers face daily.

The *National Center for Innovative Technologies in Public Health* (Centro Nazionale Tecnologie Innovative in Sanità Pubblica, TIPS), together with the *National Center for Rare Diseases* (Centro Nazionale Malattie Rare, CNMR), with the collaboration of the Press Office of the Istituto Superiore di Sanità (ISS) and internal and external experts of the ISS, has developed an online survey entitled "Technologies to support frailty, disability and rare diseases: the COVID-19 experience".

The study had several objectives.

The *first objective* was to design the survey electronically in order to easily administer it, by submitting it and collecting data easily using mobile technology.

The *second objective* was to identify which technologies were used during home isolation and physical and social distancing, to carry out, where possible, daily activities (work, school, etc.) and health and social-health treatments in a period in which all facilities and services have been closed or suspended.

The *third objective* was to monitor and identify the real accessibility and usability of the technologies currently available by "frail people", their families, and caregivers.

The *fourth objective* was to disseminate the results orally by means of a Webinar to an international audience in order to compare and discuss solutions.

The *fifth objective* was to disseminate nationally and internationally the results in a way that is useful for both stakeholders and citizens.

#### **3. Methods**

The online electronic survey, *The Central Tool*, reaches its target subjects through the most common web communication tools (e-mail, social media, etc.) by simply sending a link that allows direct access to the survey and provides preliminary results in real time. Furthermore, in the specific case of the COVID-19 emergency context, the online survey was also able to overcome the restrictions of social distancing.

In this study, Microsoft Forms was chosen, which is available in the Office 365 suite provided to the staff of the Istituto Superiore di Sanità and which, for this reason, respects the IT security aspects required by current regulations from a systems point of view. The following modules were used:


The dissemination took place through the web pages of the ISS site, the thematic site of the Ministry of Health (www.malattierare.gov.it, accessed on 23 January 2021) [17], and Uniamo—the Federation of rare diseases (www.uniamo.org, accessed on 23 January 2021) [18]. Furthermore, news was provided via the ISS Rare Diseases Toll-Free Telephone line, the sites of reference associations such as the Interregional Working Group for Electronic and IT aids for the disabled (GLIC), and the Scientific Association for Digital Health, as well as by social media such as the Facebook, Linkedin, Twitter, and Instagram accounts of various entities and institutions.

To minimize the potential bias caused by the *Digital Divide*, we invited (during submission) those more familiar with technology to support the less familiar.

As regards the questions of type (c) with 6-level evaluation and the Likert questions in (d) (e.g., Question 23) with sub-questions at 6 levels, it was possible to assign a minimum score of 1 and a maximum of 6; therefore, the value theoretical mean (TM) is 3.5. This value can be referred to by comparison in the analysis of the answers. An average response value below TM indicates a more negative than positive response. An average value above TM indicates a more positive than negative response.

#### **4. Results and Discussion**

The study, in line with the objectives, produced several important results. The *first result* is the survey [19], accessible via:

	- Quick Response Code (QR Code) (Figure 1); for those with only the paper version of the document, there is a Quick Response Reader available on most smartphones.

**Figure 1.** The QR Code. **Figure 1.** The QR Code.

The *second and the third results* are the two disseminative products of the study: The *second and the third results* are the two disseminative products of the study:

The international oral dissemination was carried out through a Webinar, using the

 The outcome dissemination was carried out by means of the publication of Rapporti ISS COVID‐19 (COVID‐19 Reports). The COVID‐19 Reports are aimed at healthcare professionals to address the different aspects of the pandemic. They provide essential and urgent information for emergency management and are subject to updates. They are produced by the COVID ISS working groups, made up of researchers from the *Istituto Superiore di Sanità*, who can also work in collaboration with other institutions. We published it in English [21] to allow the scientific community who had followed

the Webinar to directly read the outcomes in an extensive manner.

emerged from the 313 frail people interviewed in the pandemic period:

cial and health facilities due to the pandemic, the following is apparent:

rehabilitation and/or therapeutic support technologies;

lems and difficulties in using the tool effectively.

should be acknowledged by bodies and institutions.

esis. We are therefore faced with a normal distribution.

17th online webinar meeting organized by the National Centre for Rare Diseases. It was titled "17° Scientific Meeting Online COVID‐19 and Rare Diseases, 28 January 2021 (h 15.00–16.30 CET), Istituto Superiore di Sanità (Rome), Italy Aula Pocchiari,

The *fourth result* is the summary of the current state that the responses received could take, highlighting the main problems encountered by fragile citizens and their families. It

The *fifth result* is that the data collected has revealed important critical issues that

The sample of 313 frail people had a normal distribution in relation to age. We tested the normal distribution of age by the Smirnov–Kolmogorov test of normality*,* which is suitable for samples such as ours [22]. The null hypothesis was that our data follows a normal distribution. We achieved *p* = 0.51. Because *p* > 0.05, we accepted the null hypoth‐

Considering the enormous global upheaval, with drastic and sudden closures of so‐

Only *9.2*% of respondents (29 persons out of 313 people) had benefited from remote

Of these, 31% (of this subsample of 9.2%, i.e., 9 out of 29 people) encountered prob‐

 There has been an increase in the use of generic *eHealth* and *mHealth* technologies and communication and messaging tools, which all represent an *essential lifeline* [10,11]. There was a general difficulty in using and/or accessing specialized technologies for treatment or rehabilitation, with insufficient remote support for continuity of care. There was a strong desire to be able to access and use technologies appropriately, also through specific training that allows them to exploit their full potential.

and it involved several groups in the discussion [20].


The *fourth result* is the summary of the current state that the responses received could take, highlighting the main problems encountered by fragile citizens and their families. It emerged from the 313 frail people interviewed in the pandemic period:


The *fifth result* is that the data collected has revealed important critical issues that should be acknowledged by bodies and institutions.

The sample of 313 frail people had a normal distribution in relation to age. We tested the normal distribution of age by the Smirnov–Kolmogorov test of normality, which is suitable for samples such as ours [22]. The null hypothesis was that our data follows a normal distribution. We achieved *p* = 0.51. Because *p* > 0.05, we accepted the null hypothesis. We are therefore faced with a normal distribution.

Considering the enormous global upheaval, with drastic and sudden closures of social and health facilities due to the pandemic, the following is apparent:


We applied a frequency test to estimate significance (χ 2 ) [23].

The first *χ* 2 test was applied to the first group of 9.2% (29 out of 313 people) who had benefited from technology. The tested hypothesis was the significance of the difference in frequency between the group of 29 people who accessed the services and the group of 284 people who did not access it.

The χ 2 test returned a highly significant outcome, as Equation (1) shows:

$$\chi2 = \frac{\left(284 - 156.5\right)^2}{156.5} + \frac{\left(29 - 156.5\right)^2}{156.5} = 207.7 \text{ } p << 0.005\tag{1}$$

The second χ 2 test was applied to 31% of the above subgroup of 29 people who accessed the technologies (9 out 29 people) but encountered problems in the use of the technology. The tested hypothesis was the significance of the difference in frequency between the group of 9 people who accessed the services and encountered problems and the group of 20 people who accessed the services without problems. The χ 2 test returned a significant outcome, as Equation (2) shows:

$$\chi2 = \frac{\left(9 - 14.5\right)^2}{14.5} + \frac{\left(20 - 14.5\right)^2}{14.5} = 4.2, \ p < 0.05 \tag{2}$$

This result is very important, especially if we consider the fact that only 20 people (29 people accessing the technology minus 9 people with difficulties during the use of the technology) among the total of 313 accessed the technology in a satisfactory way.

The third χ 2 test applied to over 90% of people (282 out of 313) who believed that "the technology could be useful during the pandemic and in the future". The tested hypothesis was the significance of the difference in frequency between the group of 282 people who believe that "the technology could be useful during the pandemic and in the future" and the group of 31 people who had the opposite opinion. The χ 2 test returned a highly significant outcome, as Equation (3) shows:

$$\chi2 = \frac{(282 - 156.5)^2}{156.5} + \frac{(31 - 156.5)^2}{156.5} = 201.3, \ p << 0.005 \tag{3}$$

These significant results, in synthesis, made it possible to verify that: (a) very few people accessed, in a satisfactory manner, the technology; (b) a concrete and accessible solution was not found for the clear demand/expectation of technology, and (c) in consideration of this, it would be important to investigate the causes in order to propose effective interventions that also consider the tools suggested by the interviewees. These results also highlight the urgent need to implement innovative technological platforms and tools, but also to provide training courses for professionals, frail people, and their family members/caregivers and ultimately support services that offer constant assistance and also, where needed, psychological support for families.

#### **5. Conclusions and Future Work**

This report illustrates the design and results of an activity of surveillance proposed by the National Centre for Innovative Technologies in Public Health and the National Centre for Rare Diseases of the Istituto Superiore di Sanità, with the aim of investigating the state-of-use of technologies by people with frailty, disabilities, and rare diseases. The core element was an online questionnaire that was developed with simple and effective electronic tools based on Microsoft Forms, made available to ISS users, which was used during the COVID-19 pandemic, but would also be useful with simple upgrades in other periods. The project allowed the provision of a first overall summary relating to the stateof-use of technologies by citizens with fragility, disabilities, and rare diseases. From the summary emerged both the needs of the interviewees and the potential of the technology, causing reflection on the current state-of-the-art and offering important stimuli for the stakeholders involved. In fact, the demand for the supply of technology for continuity of care was not matched by an adequate supply response. The project also highlighted the usefulness of the disseminative tools set up by the Istituto Superiore di Sanità. Based on the evidence reported, some actions are planned for the future. In particular, it is deemed necessary to promote specific initiatives, some of which may be proposed directly by the TISP Centre and the CNMR Centre, jointly with all the other stakeholders involved in these issues. Among these could be, for example, awareness campaigns and training courses, or the elaboration of recommendations and documents with good practices useful to other competent bodies at national and international levels. Furthermore, it is considered to be of primary importance that this survey is used for periodic monitoring of these issues, both during and after the pandemic period, in order to plan appropriate strategies tailored to frail people. The authors and the two Centers, after this positive collaboration experience, are continuing the collaboration on surveillance activities, in a larger group in a project in collaboration with other national and international bodies, such as the WHO and the

CENSIS (an Italian national socio-economic research institute for surveys to the Italian people), dealing with the assistive technologies on the population.

**Author Contributions:** Conceptualization, All; methodology, All; software, D.G.; validation, D.G., A.P., P.M. and M.D.S.; formal analysis, D.G.; investigation, D.G., A.P., P.M., M.D.S., M.G. and D.T.; resources, All; data curation, D.G., A.P., P.M. and M.D.S.; writing—original draft preparation, D.G.; writing—review and editing, D.G., A.P., P.M., M.D.S., M.G. and D.T.; visualization, All; supervision, M.G. and D.T.; project administration, M.G. and D.T. All authors have read and agreed to the published version of the manuscript.

**Funding:** The authors report no involvement in the research by the sponsor that could have influenced the outcome of this work.

**Acknowledgments:** The following have contributed to this work, within the scope of their specific skills: Group: "Rare Diseases COVID-19"; *Chair:* Domenica Taruscio; Members: Domenica Taruscio (ISS), Luigi Bertinato (ISS), Marta De Santis (ISS), Maurizio Pocchiari (ISS), Patrizia Popoli (ISS), Patrizia Caprari (ISS), Mirella Taranto (ISS), Anna Ladogana (ISS), Massimo Aquili (Ministero della Salute), Annalisa Scopinaro (UNIAMO), Paola Meli (ISS), Francesca Masiello (ISS), Maria Elena Congiu (Ministero della Salute), Francesco Gabbrielli (ISS), Nicola Vanacore (ISS), Marco Silano (ISS), Paolo Salerno (ISS), Claudio Carta (ISS), Amalia Egle Gentile (ISS), Giancarlo Liumbruno (ISS), Rosa Anna Cauteruccio (Ministero della Salute), Marco Salvatore (ISS), Daniele Giansanti (ISS), Giovanni Rezza (ISS).

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Usability of Telemedicine Mobile Applications during COVID-19 in Saudi Arabia: A Heuristic Evaluation of Patient User Interfaces**

**Raniah N. Aldekhyyel \*,†, Jwaher A. Almulhem † and Samar Binkheder**

Medical Informatics and e-Learning Unit, Medical Education Department, College of Medicine, King Saud University, Riyadh 11451, Saudi Arabia; jalmulhem@ksu.edu.sa (J.A.A.); sbinkheder@ksu.edu.sa (S.B.)

**\*** Correspondence: raldekhyyel@ksu.edu.sa

† Authors have equal contribution to this manuscript.

**Abstract:** The coronavirus disease 2019 (COVID-19) pandemic has impacted the use of telemedicine application (apps), which has seen an uprise. This study evaluated the usability of the user interface design of telemedicine apps deployed during the COVID-19 pandemic in Saudi Arabia. It also explored changes to the apps' usability based on the pandemic timeline. Methods: We screened ten mHealth apps published by the National Digital Transformation Unit and selected three telemedicine apps: (1) governmental "Seha"® app, (2) stand-alone "Cura"® app, and (3) private "Dr. Sulaiman Alhabib"®app. We conducted the evaluations in April 2020 and in June 2021 by identifying positive app features, using Nielsen's ten usability heuristics with a five-point severity rating scale, and documenting redesign recommendations. Results: We identified 54 user interface usability issues during both evaluation periods: 18 issues in "Seha" 14 issues in "Cura", and 22 issues in "Dr. Sulaiman Alhabib". The two most heuristic items violated in "Seha", were "user control and freedom" and "recognition rather than recall". In "Cura", the three most heuristic items violated were "consistency and adherence to standards", "esthetic and minimalist design", and "help and documentation" In "Dr. Sulaiman Alhabib" the most heuristic item violated was "error prevention". Ten out of the thirty usability issues identified from our first evaluation were no longer identified during our second evaluation. Conclusions: our findings indicate that all three apps have a room for improving their user interface designs to improve the overall user experience and to ensure the continuity of these services beyond the pandemic.

**Keywords:** telemedicine; mHealth; heuristic evaluation; usability; Saudi Arabia; COVID-19

#### **1. Introduction**

The coronavirus disease 2019 (COVID-19) pandemic has negatively impacted the world on different dimensions. The virus has spread rapidly with more than 196 million confirmed cases as of the 1st of August 2021 [1]. The threat of an imminent surge of COVID-19 patients drove healthcare organizations to act quickly to develop and deploy mobile health technologies [2,3], with telemedicine solutions in particular seeing an uprise [2,4,5]. Like other countries, healthcare organizations in Saudi Arabia responded to the pandemic by creating strategies to control the spread of disease, including the use of mHealth apps to provide telemedicine care for their patients [6,7]. While many studies have shown the benefits of telemedicine apps on patients and providers [2], the usability of these apps needs to be addressed more fully [8].

Ensuring excellent usability is at the core of patient engagement [9]. Given the rapid increase in telemedicine apps during the pandemic and insufficient usability assessments, the potential impacts on user engagement and experience are not clear but are substantial [10]. Performing standardized usability assessments designed to capture the user's experience with telemedicine apps is critical in ensuring a positive user experience. Usability is defined by the International Organization for Standardization (ISO) as "the extent to which

**Citation:** Aldekhyyel, R.N.; Almulhem, J.A.; Binkheder, S. Usability of Telemedicine Mobile Applications during COVID-19 in Saudi Arabia: A Heuristic Evaluation of Patient User Interfaces. *Healthcare* **2021**, *9*, 1574. https://doi.org/ 10.3390/healthcare9111574

Academic Editor: Daniele Giansanti

Received: 14 October 2021 Accepted: 16 November 2021 Published: 18 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

the product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use." [11]. Usability is considered as a vital measure that captures users' experience and helps inform the design of mHealth apps [12]. Researchers can use several methods to evaluate usability, including the heuristic evaluation method, involving several experts examining the system's interface design [13]. Heuristic evaluation has been used extensively by different researchers [14–18] due to its low cost, ease of use, and the involvement of a small number of experts [19,20].

Saudi Arabia has many advances in digital healthcare, with specific strategic plans put in place for the advancement of healthcare using information technology [21]. Changes in insurance policies announced by the Saudi Council of Cooperative Health Insurance [22] during the pandemic indicating that telemedicine services would be covered by insurance companies influenced the rapid deployment of telemedicine services. While the effectiveness of telemedicine care has been published in the literature [23], with specific studies focusing on telemedicine user satisfaction during the pandemic [24–27], little is known about the ease and usability of telemedicine apps [8].

In this study our goal was to complete a heuristic evaluation to assess the usability of telemedicine apps, deployed in Saudi Arabia during the pandemic. We conducted the usability evaluation using Jakob Nielsen's 10 usability heuristics for interface design [13]. We also explored changes to the usability of apps based on the pandemic timeline through conducting the evaluation during two different time periods.

#### **2. Materials and Methods**

We followed three phases in our study: Phase I was selecting telemedicine apps, Phase II was conducting the heuristic evaluation during two different time periods, and Phase III was data analysis. We conducted the first evaluation one month after announcing the first COVID-19 case in Saudi Arabia [28], and the country's lockdown during April 2020, while the second evaluation was 14 months after our first evaluation (June 2021). We followed the same heuristic evaluation process during both evaluation periods.

#### *2.1. Phase I. Telemedicine Apps Selection*

In line with the government lockdown measures, the Saudi National Digital Transformation (NDT) Unit [29] during the time of our study published a document outlining a total of 10 mHealth apps (Appendix A, Table A1). On the 11th of April 2020, we independently reviewed the document and selected apps that met the criteria of a telemedicine mobile app, based on the definition of "telemedicine" as outlined in the National Saudi Telemedicine Policy: "mobile applications that provide remote interaction between a patient and a healthcare provider delivered through video, and/or audio, and/or picture, and/or text, and/or data" [30]. Any mHealth app, which did not include a telemedicine feature, such as apps developed for medication delivery, medical encyclopedias, or patient portals were excluded from our evaluation The three apps we selected covered three main types of telemedicine services; (1) the governmental app "Seha"® [7], (2) the stand-alone private app "Cura"® [31], and (3) a private app called "Dr. Sulaiman Alhabib"®, which is a paid telemedicine service provided by a private hospital [32].

#### Apps Description

"Seha" app provides free telemedicine consultation services for all citizens and residents. Users are required to register in the app using their mobile number. Once registration is confirmed through a text message sent to the user's mobile, users can request for a consultation with a Ministry of Health's physician up to three times per month. The app is not linked to a certain hospital/clinic nor to a specific unified patient medical record number. The app also includes an artificial intelligence technology feature in the form of an automatic health assessment tool.

"Cura" app is a stand-alone telemedicine app providing a paid consultation service to its users. The app offers on-demand consultations with general practitioners, specialists, and consultants. Users can choose a consultation with a specific physician from viewing a list of available physicians. The app also offers different wellness program packages. Like "Seha", users register once using their mobile number and receive a confirmation through a text message. Consultations are offered with a fee that users are required to pay in advance. The app is not linked to a certain hospital/clinic or a specific unified patient medical record number.

"Dr. Sulaiman Alhabib" app is developed by Dr. Sulaiman Al Habib Medical Group; a private hospital with over 10 branches in Saudi Arabia. The app provides a variety of services for the hospitals' patients and is integrated with their medical record system. The app provides a wide range of services for patients. Telemedicine consultation feature was added during the early months COVID-19 pandemic in 2020. The number of consultations offered to its users is based on their specific insurance coverage.

#### *2.2. Phase II. Evaluation Procedure*

To conduct the usability evaluation, we used Jakob Nielsen's 10 usability heuristics for interface design [13,33] due to their widespread use [14–18]. After we performed an unstructured qualitative overview of the three apps, we designed an online form using google forms [34], which contained two sections: (1) features of the apps using a yes/no nominal scale, and (2) Nielsen's ten usability heuristics with a 5-point severity rating scale [35] (Appendix B).

Given that we have no affiliation with the organizations, which developed the apps included in our study, and our background in health informatics and experience in usability testing and evaluation methodologies, we conducted the evaluation ourselves. Before each evaluation, we briefly discussed the heuristics and the severity classification to ensure that we followed a standardized evaluation process. Each of us then installed the three apps on our personal mobile phones (iPhone 11) and registered to access the apps. Using the standardized online form, we independently reviewed the apps and completed a real time teleconsultation with a physician to identify compliance with the heuristics. We completed separate forms to identify the apps' features, record issues related to the heuristics, provide descriptions, and assign the severity ratings, and record the location of the issues.

#### *2.3. Phase III. Data Analysis*

Following the evaluation, we compiled the forms into a single form, and together we discussed our findings, generated consensus ratings, and provided redesign recommendations. We calculated frequencies and percentages for the usability issues and assigned the location of the issues to one of the following categories: (1) registration, (2) log in, (3) orientation on how to use the app, (4) initiating a consultation, (5) waiting for physician (6) during consultation, and (7) end of consultation. We identified the categories based on the steps users would follow to complete a consultation with a physician through the apps.

After completing both evaluations, we further analyzed our findings by examining the usability issues resulting from the first evaluation to check if they were still an issue in our second evaluation or were they resolved.

To avoid bias, we followed the recommendations outlined by McDonagh et. al in selecting studies for review [36], specifically: (1) defining an inclusion and exclusion criteria, and (2) applying dual review during the selection and evaluation phases—having two evaluators independently assess mhealth apps for inclusion and evaluate the apps using Nielsen's ten usability heuristics. The same evaluators conducted both evaluations and none were affiliated with the organizations responsible for developing the apps.

#### **3. Results**

Table 1 shows an overview of the features of the three apps.


#### **Table 1.** Apps Features.

+ Depends on each user's insurance coverage plan.

A summary of the usability issues identified in "Seha", "Cura", and "Dr. Sulaiman Alhabib" apps during the two evaluation periods, with the location of issues, severity rating, and redesign recommendations are presented in Tables A2–A4 respectively (Appendix C). In total, we identified 54 user interface usability issues during both evaluation periods: 18 issues in "Seha" app (9 from the first and 9 from the second), 14 issues in "Cura" app (9 from the first and 5 from the second), and 22 issues in "Dr. Sulaiman Alhabib" app (12 from the first and 10 from the second). In "Seha" app, the two most heuristic items violated were "user control and freedom" and "recognition rather than recall", with three unique usability issues identified in each. We found no issues under the "recognition diagnosis, and recovery from errors" heuristic. In "Cura" app the three most heuristic items violated were "consistency and adherence to standards", "esthetic and minimalist design", and "help and documentation", with three unique usability issues identified in each. We found no issues under the two heuristics: "visibility of system status" and "recognition diagnosis, and recovery from errors". In "Dr. Sulaiman Alhabib" app the most heuristic item violated was "error prevention", with four unique usability issues identified, followed by "user control and freedom", and help and documentation", with three unique usability issues identified in each. The "flexibility and efficiency of use" heuristic item among all apps did not include accelerators or an ability to tailor frequent actions based on inexperienced and experienced users, therefore we considered this item not applicable in our evaluation.

Based on the location of issues among the three apps, we found the most usability issues were during the "consultation initiation" (*n* = 21), followed by "orientation" (*n* = 9),

C).

"during consultation" (*n* = 7), "registration" (*n* = 5), and "login" (*n* = 5). The least number of issues were categorized as "waiting for physician" (*n* = 4), and "end of consultation (*n* = 3). Notably, results of our first evaluation showed two location categories: "orientation" and "consultation initiation" related to the nine usability issues identified in "Cura" app. The only five usability issues categorized as "registration" were found in "Dr. Sulaiman Alhabib" app, and the only four usability issues categorized as "waiting for physician" were identified in "Seha" app. of issues were categorized as "waiting for physician" (*n* = 4), and "end of consultation (*n*  = 3). Notably, results of our first evaluation showed two location categories: "orientation" and "consultation initiation" related to the nine usability issues identified in "Cura" app. The only five usability issues categorized as "registration" were found in "Dr. Sulaiman Alhabib" app, and the only four usability issues categorized as "waiting for physician" were identified in "Seha" app. When we compared between the two evaluation periods, the numbers of usability

When we compared between the two evaluation periods, the numbers of usability issues in "Seha" app were similar in both evaluations, however the average severity rating was slightly higher in the second evaluation. In "Cura" app, the number of usability issues in the second evaluation was lower while the average severity rating was considerably higher in the second evaluation compared to the first evaluation. Average severity ratings for the "Dr. Sulaiman Alhabib" app was slightly changed between both evaluations while the number of usability issues was higher in the first evaluation in contrast to the second evaluation (Figure 1). issues in "Seha" app were similar in both evaluations, however the average severity rating was slightly higher in the second evaluation. In "Cura" app, the number of usability issues in the second evaluation was lower while the average severity rating was considerably higher in the second evaluation compared to the first evaluation. Average severity ratings for the "Dr. Sulaiman Alhabib" app was slightly changed between both evaluations while the number of usability issues was higher in the first evaluation in contrast to the second evaluation (Figure 1).

"during consultation" (*n* = 7), "registration" (*n* = 5), and "login" (*n* = 5). The least number

*Healthcare* **2021**, *9*, x 5 of 18

**Figure 1.** Frequency and average severity ratings by evaluation period among the three apps. **Figure 1.** Frequency and average severity ratings by evaluation period among the three apps.

The average severity ratings based on the heuristics in "Seha" app, showed two catastrophic issues: "error prevention" (identified from the first evaluation), and "help and documentation" (identified from the second evaluation). In the "Cura" app, issues related to both "consistency and standards" and "error prevention" items were rated as major issues in the first evaluation. Notably, five out of ten heuristic items did not involve any usability issues in the second evaluation. In "Dr. Sulaiman Alhabib" app, issues related to "recognition rather than recall" and "help and documentation" were rated as catastrophic in the first evaluation and issues related to "error prevention" were rated as catastrophic The average severity ratings based on the heuristics in "Seha" app, showed two catastrophic issues: "error prevention" (identified from the first evaluation), and "help and documentation" (identified from the second evaluation). In the "Cura" app, issues related to both "consistency and standards" and "error prevention" items were rated as major issues in the first evaluation. Notably, five out of ten heuristic items did not involve any usability issues in the second evaluation. In "Dr. Sulaiman Alhabib" app, issues related to "recognition rather than recall" and "help and documentation" were rated as catastrophic in the first evaluation and issues related to "error prevention" were rated as catastrophic in the second evaluation (Figure 2).

in the second evaluation (Figure 2). Our first evaluation resulted in the discovery of 30 user interface usability issues among the three apps, with 10 of these issues no longer identified from our second evaluation. Three out of nine issues in both "Seha" and "Cura" apps were resolved, while four Our first evaluation resulted in the discovery of 30 user interface usability issues among the three apps, with 10 of these issues no longer identified from our second evaluation. Three out of nine issues in both "Seha" and "Cura" apps were resolved, while four out of 12 issues were resolved in "Dr. Sulaiman Alhabib" app (Tables A2–A4 Appendix C).

out of 12 issues were resolved in "Dr. Sulaiman Alhabib" app (Tables A2–A4 Appendix


**Figure 2.** Average severity ratings by heuristic items among the three apps. H7: Flexibility and efficiency of use is not displayed in the chart. **Figure 2.** Average severity ratings by heuristic items among the three apps. H7: Flexibility and efficiency of use is not displayed in the chart.

#### **4. Discussion 4. Discussion**

Several telemedicine apps have been developed in Saudi Arabia ranging from free to paid services in response to the pandemic. With the increased availability of these apps, it is essential to measure the apps' usability from a user's perspective, to ensure continuity of these services beyond the pandemic. Our study was conducted to explore the usability issues related to three telemedicine apps used in Saudi Arabia during the pandemic, using Nielsen's 10 heuristics. We performed two evaluations during two time periods to explore any changes to the usability of apps based on the pandemic timeline. We found that following a standardized approach in identifying the features of the telemedicine apps along with conducting the heuristic evaluation was a feasible and efficient method to evaluate the apps' user interfaces. This method helped highlight positive features as well as classify usability issues, which may potentially assist the apps' developers in resolving issues in future updates. We also used a standardized severity rating score for each issue we identified based on the 10 heuristics items. The rating helped highlight the significant usability issues and prioritize them to allocate possible resources in overcoming these issues [13,14]. Our evaluation also suggested possible redesign solutions, which if implemented can po-Several telemedicine apps have been developed in Saudi Arabia ranging from free to paid services in response to the pandemic. With the increased availability of these apps, it is essential to measure the apps' usability from a user's perspective, to ensure continuity of these services beyond the pandemic. Our study was conducted to explore the usability issues related to three telemedicine apps used in Saudi Arabia during the pandemic, using Nielsen's 10 heuristics. We performed two evaluations during two time periods to explore any changes to the usability of apps based on the pandemic timeline. We found that following a standardized approach in identifying the features of the telemedicine apps along with conducting the heuristic evaluation was a feasible and efficient method to evaluate the apps' user interfaces. This method helped highlight positive features as well as classify usability issues, which may potentially assist the apps' developers in resolving issues in future updates. We also used a standardized severity rating score for each issue we identified based on the 10 heuristics items. The rating helped highlight the significant usability issues and prioritize them to allocate possible resources in overcoming these issues [13,14]. Our evaluation also suggested possible redesign solutions, which if implemented can potentially enhance the overall user experience.

tentially enhance the overall user experience. When developing telemedicine apps, healthcare organizations providing telemedicine services in Saudi Arabia must be aware of the current governing regulations [37–40], and accreditation bodies [41]. During the pandemic many efforts have been made by these organizations to develop and update their regulations to serve as a guide for healthcare organizations and developers. A national online training course for healthcare providers has also recently been launched by the Saudi Commission for Health Specialties to ensure a standardized approach in providing telemedicine care [42]. Utilizing these resources When developing telemedicine apps, healthcare organizations providing telemedicine services in Saudi Arabia must be aware of the current governing regulations [37–40], and accreditation bodies [41]. During the pandemic many efforts have been made by these organizations to develop and update their regulations to serve as a guide for healthcare organizations and developers. A national online training course for healthcare providers has also recently been launched by the Saudi Commission for Health Specialties to ensure a standardized approach in providing telemedicine care [42]. Utilizing these resources would ensure a high standard of telemedicine care and an overall positive user experience.

would ensure a high standard of telemedicine care and an overall positive user experience. Beyond the results of our usability evaluation, our study demonstrated four key findings. First, the "Dr. Sulaiman Alhabib" app was the only app in our study linked to a hospital medical record system. "Seha" and "Cura" apps, which lacked integration

with a medical record system may potentially affect the overall patient care experience since the medical record represents the main method for documenting the patient's health encounter. The importance of documentation in a patient's record has clearly been outlined in one of the provisions of the Saudi telemedicine regulations [30]. The regulation states that health care providers need to have access to the patient's relevant health information and that all patient's data and activities conducted during a telemedicine encounter be documented in the patient's medical record [43]. A possible solution for this significant concern is incorporating the Shared E-Health File; a unified national electronic system that enables information exchange among different hospitals [44]. Incorporating an access to the Shared E-Health File within telemedicine apps [30] may potentially improve the level of care provided to patients and data interoperability. Specific measures would need to be put in place to overcome the challenges that the unified medical record system and the EHR cloud systems may bring. Challenges such as data protection and security issues are critical challenges for its acceptance among patients and healthcare providers [45,46].

Second, there was a slight difference between the usability issues identified during both evaluations based on the pandemic timeline. Although the number of usability issues were higher during our first evaluation, the average severity ratings for all apps were higher during our second evaluation. This may indicate the developers' efforts in continuously working towards enhancing the users' experience. In both evaluations, there were issues with "help and documentation". Adding a separate accessible page outlining user instruction on how to use the app and access the telemedicine service is vital in enhancing the overall user experience [47]. Without having adequate user instructions, users may find difficulty in using the app, particularly with lack of technical support contact and the different types of users. When developing these apps, several age-related issues should be considered including cognition, perception, and behavior issues [8]. Providing help and support also is needed to overcome some technological barriers such as low technology literacy related to using telemedicine apps [48]. Although the apps we reviewed in our study were overall user friendly, special consideration should be provided to consider experienced and non-experienced users since we found "lack of flexibility" common within all three apps. Enabling users to customize user interfaces and create shortcuts might add a more personalized approach and a positive user experience [16].

Third, the rapid deployment of telemedicine apps in anticipation of a surge in COVID-19 cases may explain why we found most of the identified issues categorized as major problems and four out of seven catastrophic usability issues in "Dr. Sulaiman Alhabib" app's user interfaces. "Dr. Sulaiman Alhabib" app's telemedicine service was the only service developed in response to the pandemic and to changes to the country's insurance policies [22]. It remains to be seen whether this service will last beyond the resolution of the pandemic and what role this will have on the use of telemedicine, particularly for their hospital's patients

Lastly, the evaluation process itself resulted in identifying shared user tasks among the three apps. These tasks outlined the steps the user needed to perform to complete a specific telemedicine encounter. The identification of tasks helped us categorize usability issues into structured locations, which could potentially be used for future studies focusing on performing a cognitive walkthrough as a usability evaluation method [49].

Our study has several limitations. First, our app selection process was based on a publication issued by the NDT during the early months of the pandemic. These apps may not have represented the most used apps by the public during the time of our study. Relying on a different source, such as top downloads in App Store or Google Play, could have resulted in other apps included in our evaluation. Second, we conducted a heuristic evaluation, which depends on experts' expertise. While this type of evaluation has proven useful in identifying usability issues, it may not be comprehensive in identifying all difficulties, which may be captured in usability tests with human participants [33]. Conducting a usability user test, which includes both types of users (healthcare providers and patients), considering different age groups may overall enhance the user experience. Lastly, because

we used a heuristic evaluation method to assess the usability of the user interface, which is considered a method with limited generalizability [50], our study findings may be limited. Utilizing a combination of evaluation methods, such as cognitive walkthroughs and simulated interaction may provide a more comprehensive picture.

#### **5. Conclusions**

Heuristic evaluation studies have the potential to assist software designers and developers to discover severe usability issues that may have an effect on user acceptance of these apps. We evaluated three telemedicine apps used in Saudi Arabia using a heuristic evaluation method with a focus on understanding the usability issues in the apps user interface during COVID-19. We identified 54 user interface usability issues that may have an effect on the overall usability. Overall, our findings indicate that the three apps have a room for improvement by enhancing their user interfaces to improve the overall user experience.

**Author Contributions:** Conceptualization, R.N.A. and J.A.A.; formal analysis, R.N.A., J.A.A. and S.B.; methodology, R.N.A.; project administration, R.N.A.; visualization, S.B.; writing—original draft, R.N.A. and J.A.A.; writing—review and editing, R.N.A., J.A.A. and S.B. All authors have read and agreed to the published version of the manuscript.

**Funding:** No funding.

**Institutional Review Board Statement:** This study was approved by the Institutional Review Board at King Saud University.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** The authors thank King Saud University for their scientific support.

**Conflicts of Interest:** The authors declare no financial interest and no conflict of interest with respect to the research, authorship, and/or publication of this article.

#### **Appendix A**

**Table A1.** List of mHealth apps published by NDT [29] in 2020 during the time of our study \*.


\* The list of apps included in the NTD document may have since been updated to include more apps.

#### **Appendix B**

Form used for evaluation, which was filled out independently by each reviewer using Google forms.

*App Features* 1—Name of app:

	- 2—Type of patients the application serves
	- 2—What clinical specialties are provided to the patient?

3—Is there a limit to the number of consultations with the healthcare provider per month?


4—Indicate the availability of the below features:


### *Usability Heuristics Evaluation Based on Nielsen's Heuristics*

Please rate each usability heuristics item based on your inspection.


#### **Appendix C**

**Table A2.** "Seha" app (versions 1.0.35 and 1.0.36): usability issues identified based on Nielsen's heuristics.



#### **Table A2.** *Cont.*


#### **Table A2.** *Cont.*

Total Issues identified from the two evaluations: 18

† The issue was resolved—no longer identified in our second evaluation.

**Table A3.** "Cura" app (versions 1.8.9 and 2.0.0): usability issues identified based on Nielsen's heuristics.



#### **Table A3.** *Cont.*


#### **Table A3.** *Cont.*

† The issue was resolved—no longer identified in our second evaluation.

**Table A4.** "Dr. Sulaiman Alhabib" app (versions 4.2.3 and 4.4.4): usability issues identified based on Nielsen's heuristics.




† The issue was resolved—no longer identified in our second evaluation.

#### **References**


## *Opinion* **The Vaccination Process against the COVID-19: Opportunities, Problems and** *mHealth* **Support**

**Rossella Simeoni <sup>1</sup> , Giovanni Maccioni <sup>2</sup> and Daniele Giansanti 2,\***


**Abstract:** The vaccination against the COVID-19, finally available, has the potential to represent an important defence against the pandemic. The identification of both obstacles and tools to combat them are, at this moment, of strategic importance. Previous experiences on vaccinations have shown solutions and paths to take, also based on the behavioural sciences. The objective of the *opinion* is to face how mobile technology can help us both to fight these problems and to optimize the vaccination process. The *opinion* has four polarities. The *first polarity* consists in having detected the problems hampering an effective vaccination process. These problems have been grouped into the following four: *Electronic and Informatic divide*, *Escape*, *Exposure risk*, and *Equity*. The *second polarity* consists in having verified how the mobile technology can be useful to face the identified problems. The *third polarity* highlights the usefulness and importance of using electronic surveys. These tools are based on mobile technology. They are useful problem sensors for the stakeholders. The *fourth polarity* faces how mobile technology and *mHealth* can be of aid to optimize the flow of the vaccination process, from the first call up to the certification. This polarity is supported by an example based on the Italian national App IO. The study highlights: (a) on one side, the potential of mobile technology; on the other side, the need for interventions to reduce the *digital divide* with the purpose to increase its use. (b) How the role of mobile technology can be complementary to other intervention methods.

**Keywords:** COVID-19; SARS-CoV2; *mHealth*; mobile technology; *digital divide*

#### **1. Introduction**

*1.1. Background*

COVID-19 is still prevalent. It is still a terrible threat all over the world. As of 4 August 2021, COVID-19 is the cause of 199,466,211 confirmed cases with 4,244,541 confirmed deaths [1]. Until now, the COVID-19 vaccine is the most promising measure to placate contagion with the hope to reach herd immunity [2].

Unfortunately, public vaccination intention does not appear to be uniform [3]. There are many issues hampering a harmonious and optimized process of vaccination. This was shown through large-scale studies, in which probing tools were also used [4–6].

A clear division, for example, exists between medical professionals and laypeople. While the insiders to the healthcare eagerly promote the vaccination campaign, some laypeople show doubt, hesitancy, opposition, and hostility toward COVID-19 vaccines [7].

The contrast of these issues is important. It is currently one of the main objectives of politicians and stakeholders of the health domain. It is particularly important to find adequate tools to combat these problems. It is also important to find adequate solutions to optimize the vaccination process itself.

Scholars have known, even before the pandemic, that adherence to vaccination processes can be improved through: (I) the knowledge of the problems; (II) the use of special tools and precautions to prevent and/or minimize them [8,9].

**Citation:** Simeoni, R.; Maccioni, G.; Giansanti, D. The Vaccination Process against the COVID-19: Opportunities, Problems and *mHealth* Support. *Healthcare* **2021**, *9*, 1165. https:// doi.org/10.3390/healthcare9091165

Academic Editors: Jitendra Singh and Tin-Chih Toly Chen

Received: 12 July 2021 Accepted: 31 August 2021 Published: 6 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Jarret et al. [8] highlighted the following useful tools: the use of social mobilization, mass media, communication tool-based training for health-care workers, non-financial incentives, and reminder/recall-based interventions.

Troung et al. recently found in their study [9] that seven major factors promote vaccine hesitancy or acceptance: demographic factors (ethnicity, age, sex, pregnancy, education, and employment), accessibility and cost, personal responsibility and risk perceptions, precautionary measures taken based on the decision to vaccinate, trust in health authorities and vaccines, the safety and efficacy of a new vaccine, and lack of information or vaccine misinformation. They concluded that this approach was useful for strategies to address the present situation with the COVID-19 pandemic. These two studies [8,9] highlight that adherence to vaccination processes depend on many wide-ranging factors also related to the behavioural sciences, such as psychology, sociology, communication sciences.

In particular, communication plays a key role for both the studies [8,9]. Kaufman et al. [10] focused on communication aspects related to vaccines. They reported on the importance of the *face-to-face* interaction in this field. It is an activity playing a basic role. However, now it cannot be played in an adequate way due to the restrictions.

There are many tools dedicated to the communication that can help us to fight these problems. Furthermore, these tools can also improve the vaccination process. Through radio, television, and journalistic communications, in fact, we are all witnessing the dissemination of messages with the aim of raising awareness among citizens. Certainly, the use of mobile technologies, based essentially on the smartphone [11] as a communication tool is an important novelty, compared to previous pandemics (for example the Sars-Cov-1). It is, therefore, of prime importance to investigate its role in this context.

In this study, we dedicated ourselves to an *opinion* on the possible positive role of *mobile technology* (*mTech*) and the *mHealth* both to fight the problems, hampering the vaccination process, and to optimize the vaccination process itself.

*mTech* and *mHealth* have helped [11] us so much during this pandemic. They are still helping us so much in the following three distinct activities: (a) remote monitoring and therapy; (b) continuity of daily life activities through chat, video conferencing, and electronic connection tools; and (c) new epidemiological monitoring opportunities, such as the digital contact tracing. It is therefore important to face how these technologies can help us now, during the vaccination process.

#### *1.2. Purpose of the Study*

This *opinion piece* has the general objective of highlighting the potential support of *mTech* and *mHealth* in the vaccination process: to combat the various obstacles to an effective and rapid vaccination process; and to optimize the vaccination process itself.

In detail, the work has as its objective to: (a) Directly highlight the main categorized problems related to the vaccination process. (b) Directly highlight what can be the usefulness of *mTech* and *mHealth* both to combat the main problems and to optimize the vaccination process. (c) Indirectly reaffirm the importance for the citizens to connect to the *Health Domain.*

#### *1.3. Organization of the Study*

The work is organized in four sections (plus the introduction and the conclusions).

Section 2 reports the main problems hampering the vaccination process.

Section 3, through the evidence from the literature and field evaluations, addresses the objective to highlight how *mHealth* and *mTech* can be of aid to fight these problems. In line with the article type, this section highlights the authors' strong position regarding the importance of polls in this difficult battle.

Section 4, through an Italian example, highlights how *mHealth* and *mTech* can be useful in optimizing the vaccination process.

#### **2. The** *Digital Divide***, the** *Escape***, the** *Exposure Risk***, the** *Equity***, and COVID-19 2. The Digital Divide, the Escape, the Exposure Risk, the Equity, and COVID-19**  *2.1. General Considerations*

*Healthcare* **2021**, *9*, x 3 of 15

*2.1. General Considerations*

importance of polls in this difficult battle.

useful in optimizing the vaccination process.

The effectiveness of a vaccination process depends on the ability to reach all citizens according to principles of fairness, respecting the priorities. The effectiveness of a vaccination process depends on the ability to reach all citizens according to principles of fairness, respecting the priorities.

line with the article type, this section highlights the authors' strong position regarding the

*Section 4*, through an Italian example, highlights how *mHealth* and *mTech* can be

It is therefore of primary importance to meticulously achieve connecting the citizens to the vaccination process. It is also of primary importance to avoid the *escape*, the *hesitancy*, and the *loss* of citizens. It is therefore of primary importance to meticulously achieve connecting the citizens to the vaccination process. It is also of primary importance to avoid the *escape*, the *hesitancy*, and the *loss* of citizens.

We must also bear in mind that in the era of digitalization the vaccination processes are particularly based on digital technologies. Therefore, the ability and possibility to connect and/or to join these technologies is strategic. We considered the above, the problems identified by Troung et al. [9], and the role of the priority based on the risk. We identified in this opinion piece a categorization arranged into four issues. They all start with E (Figure 1). We must also bear in mind that in the era of digitalization the vaccination processes are particularly based on digital technologies. Therefore, the ability and possibility to connect and/or to join these technologies is strategic. We considered the above, the problems identified by Troung et al. [9], and the role of the priority based on the risk. We identified in this opinion piece a categorization arranged into four issues. They all start with E (Figure 1).

**Figure 1.** The problems hampering the COVID-19 vaccination process, as identified in our opinion piece. **Figure 1.** The problems hampering the COVID-19 vaccination process, as identified in our opinion piece.

#### *2.2. Electronic & Informatic Divide 2.2. Electronic & Informatic Divide*

The *digital divide* in the two components *Electronic and Informatic Divide* regards the gap between those who have effective access to information technologies and those who are partially or totally excluded from it. The *digital divide* has three polarities/levels of intervention. The *first level* of the *digital divide* is represented by the difficulty in the access to the *infrastructures*; this today remains a problem also in the richest and most technologically advanced countries in the world [12]. The *second level* is represented by *the literacy*, characterized by the skills that enable individuals to seek, understand, and use information in ways which promote and maintain health based on digital health [13]. The *third level* is represented by the *potential benefit level* [14]. This regards the extent to which economic, cultural, social, and personal types of engagement with the Internet result in a The *digital divide* in the two components *Electronic and Informatic Divide* regards the gap between those who have effective access to information technologies and those who are partially or totally excluded from it. The *digital divide* has three polarities/levels of intervention. The *first level* of the *digital divide* is represented by the difficulty in the access to the *infrastructures*; this today remains a problem also in the richest and most technologically advanced countries in the world [12]. The *second level* is represented by *the literacy*, characterized by the skills that enable individuals to seek, understand, and use information in ways which promote and maintain health based on digital health [13]. The *third level* is represented by the *potential benefit level* [14]. This regards the extent to which economic, cultural, social, and personal types of engagement with the Internet result in a variety of economic, cultural, social, and personal outcomes. During the COVID-19 pandemic, the problem of the *digital divide* has been exacerbated [15].

The three levels of the *digital divide* are evident also during the pandemic [15–17], where digital resources were fundamental [11,15–18]. The difficulties of the citizens in accessing the digital technologies are multifaceted [15]. They may depend on cultural, ethnic, social, national, and political factors [19,20]; furthermore, they could exacerbate

the disparities [20]. The vaccination process is also based (or will be based) on the use of both electronic and computer technologies, with the purpose to achieve effectively the citizens. The *literacy* with the electronic devices, such as tablets, smartphones, and available software—as, for example, with the Italian national App (App IO) (https://io.italia.it/ accessed on 4 September 2021) [21]—could make the difference in the access to vaccines.

#### *2.3. Escape*

Vaccine *escape* certainly represents an important problem. The lack of confidence in technologies can lead to being forgotten or can lead to not being sufficiently able to join the digitized vaccination process [22]. There is certainly a strong correlation with the problem described above. However, the *escape* from vaccines is also caused by many other factors, such as: (a) the *infodemic* [23], i.e., the pandemic of bad and/or distorted information campaigns; (b) the membership in groups with special beliefs (for example, against vaccines); (c) other diversified multifaceted motifs to be carefully investigated. These problems have a strong impact on the person's behaviour; furthermore, they must be addressed through suitable tools from the sciences of communication, psychology, and sociology. With the *escape*, we also have the risk of losing those with the greatest risk from exposure, such as frails, who are at risk of becoming seriously ill, as described in the following.

#### *2.4. Exposure Risk*

The risk of exposure includes two categories: (a) The risk of those who are particularly exposed to viral contamination events, due to work activity, such as healthcare workers or other categories in contact with the public; and (b) the risk of those who, due to their conditions of frailty towards COVID-19, have the potential to contract the disease in a severe form. Prioritization policies must take these aspects into account. It is easy to verify how studies are addressing the issues related to the vaccine assignment priority, based on different hypotheses [24–26]. They consider the risk of exposure and other accompanying factors, such as the maintenance of anti-contagion measures, also using simulation models [27].

#### *2.5. Equity*

*Equity* is a term that, when it is presented into the health domain [27], assumes an important meaning. This meaning is strongly reaffirmed in the pandemic period [28]. *Equity* in the health domain is when everyone can be as healthy as possible. This is achieved by decreasing social disparities and investing more in those who have less, to give everyone the same opportunities. When we talk about the COVID-19, the impact of inequality is immediate because we immediately see that it is synonymous with the continuity of the spread of the SARS-CoV-2 virus, with the difficulty of caring for all citizens, with the difficulty of preventing the disease through vaccination. It is for this reason why many nations are placing the emphasis on this problem through appropriate strategies [29,30].

Therefore, the vaccination process cannot ignore the concept of *equity* as it is interpreted in the health domain. Research in PubMed with the query "(equity [Title]) AND (COVID-19)" showed that the problem of equity is strongly perceived from different points of view, starting from the ethics up to the social vision [31–43].

#### **3. The Role of Mobile Technology and** *mHealth* **to Fight the Problems Hampering the Process**

#### *3.1. General Considerations*

In this section, through the evidence from the literature and field evaluations, we address the objective to highlight how *mHealth* and *mTech* can be of aid to fight the obstacles. In general, *mTech* tries to reach citizens both to bring them closer to the vaccination process and to avoid the *escape, the hesitancy, and the loss*. There can be dedicated apps (which can also use artificial intelligence algorithms) or web sites, with accessibility through *mTech,*

designed to fight disinformation (e.g., the infodemic). They may: (I) provide answers to FAQs; (II) inform adaptatively about the priorities in the process (e.g., the frail people); (III) fight the *digital divide*; or (IV) simply try to reach a large population equally. We report below some examples developed around the world in this direction. However, *mTech* also allows us to reach the citizens with tools, such as surveys, helping the stakeholders in the health domain to investigate, in a targeted manner, the position of citizens towards these issues. Our opinion is that their use is important and strategic, since they can allow the creation of large virtual focus groups on important issues. In line with the article type, this paragraph also highlights the authors' strong position regarding the importance of polls in this period.

#### *3.2. Example of mTech Applications*

The contribution in the vaccination process of *mHealth* is undisputed by now. Each technologically advanced nation is organizing itself through solutions based on *mTech* and *mHealth* to meticulously achieve the vaccination of citizens and to avoid the *escape,* the *hesitancy,* and the *loss* of people.

Apps can be useful to affect COVID-19 vaccine *hesitancy, escape, and other* related problems. In [44], a national experience performed in Japan is reported. The authors assessed an application (App) based on *mTech*. They successfully assessed a COVID-19 vaccine information *chatbot* inserted in a popular Japanese messenger app (*LINE*). This free app (*Corowa-kun*) aimed to automatically answer FAQs related to the vaccines.

Additionally, artificial intelligence integrated into *mTech* can be useful to fight the vaccine hesitancy, escape, and other problems. The deluge of unverified information, which spreads faster than the epidemic itself, is an unprecedented phenomenon that has put millions of lives in danger. In fact, this has the potential to alter people's behaviours by making them lose their lucidity.

In ref. [45], the authors describe their created app named *WashKaro*, allowing a multipronged intervention for mitigating misinformation through conversational interactions based on artificial intelligence (AI).

The app *WashKaro* offers correct information in line with the WHO guidelines (including vaccines). It uses AI and delivers information in the right format in local languages. The authors successfully tested it on a wide sample of citizens.

As we have highlighted, the *digital divide* and the *escape* have connection points.

It is quite evident that the lack of literacy with IT generates an *escape*, but also the inability for those with economic problems to access the IT infrastructures, due to the inability to guarantee *equity*, which also is related to an *escape*.

The Italian government provided, during the pandemic, economic financial aids [46] aimed at equipping the less wealthy people with IT (essentially with the mobile technology, with digitalization kits and smartphones).

This had the clear intention of minimizing the *first level* of the *digital divide*. As it is well known, in rich nations, this has shifted from the difficulty to the physical access to the difficulty in the material access [12].

The *escape*, as we have anticipated, also depends a lot on disinformation (e.g., the infodemic).

The Italian NIH has provided a repository of medical information on its web site with the access also in non-desktop/mobile mode to dispel false news about the vaccine [47].

The *escape* also depends on how the institutions reach the citizens through *mTech*. Subtleties in the composition of messages and the language used to reach them are therefore also important to affect their behaviour [48]. In [48], the authors presented two sequential randomized controlled trials (RCTs) that tackle this challenge with behavioural science insights. In the first RCT, text messages designed to make vaccination salient and easy to schedule boost appointment and vaccination rates by 86% and 26%, respectively. Nudges that make patients feel endowed with the vaccine heighten these effects, but addressing vaccine hesitancy via a video-based information intervention does not yield benefits beyond

simple text. In the second RCT, they further find that receiving a second reminder boosts appointment and vaccination rates by 52% and 16%, respectively. Their findings suggest that text-based nudges can substantially affect the behaviour and therefore increase and accelerate COVID-19 vaccinations at almost zero marginal cost, highlighting the promising role of behavioural science in addressing a critical component of the COVID-19 pandemic response.

#### *3.3. The Usefulness of Dedicated Electronic Surveys Based on mTtech in the COVID-19 Era*

The surveys, which today can travel electronically through *mTech,* represent an important tool in the hands of stakeholders to monitor problems. We discussed in [15] that, using accurate diffusion solutions, we can minimize the *digital divide* bias and reach a large population. The surveys have proved to also be highly effective in relation to the pandemic. It clearly emerges that the survey tools proved to be useful for investigating: the impact of bandwidth limitations [49]; the attitude, knowledge, and practice towards COVID-19 [50]; the learning methods [51]; the *equity* and the *digital divide* in the racial and ethnic differences in the comparisons of posts shared on COVID-19 [52]; the *equity* and the *digital divide* in the racial and ethnic differences in the areas of remote assistance during the COVID-19 pandemic [53]; and the impact of the electronic and informatic divide based on age [15].

A recent study in the UK proposed a survey to investigate the *escape* from vaccines [22]. Our opinion is that electronic surveys can be useful to combat the four problems described in Section 2.

The US experience with the electronic surveys [54] conducted through *the U.S. COVID-19 Trends and Impact Survey, 2020–2021* is also very important. This survey is a tool that allows a continuous real-time measurement and monitoring of problems related to the COVID-19. It has operated since 6 April 2020. It is an internet based electronic tool. It operates by inviting a random sample of Facebook active users each day. Over 20 million responses have been received in the first year of operation. The survey has been repeatedly revised to respond to emerging public health priorities. The last revisions are also facing the problems hampering the vaccination process. The study [54] highlighted that large online surveys can provide continuous, real-time indicators of important outcomes for the health domain that are not subject to reporting delays and backlogs.

#### 3.3.1. Experiences and Considerations in the Vaccination Process

As demonstrated in the study in the UK [22] and in the US [54], the surveys represent a powerful tool to investigate the reasons that lead citizens to escape from vaccines.

As the English and US study show: (I) the COVID-19 pandemic is an experience never tried previously; (II) considering this, it is difficult to find ready-made intervention models to inspire; (III) surveys must also be constructed considering the novelty of the experience we are living in; furthermore, the available mobile technologies allow a widespread dissemination.

In line with the objectives of the opinion piece, we wanted to highlight: (IV) the usefulness of these tools in the widespread data-collection; and (V) the potential usefulness for politicians and decision makers in general.

Always in line with our objectives, we also wanted to start a scientific discussion in this area. We believe that: (VI) these surveys must be wide-ranging and embedded in the national realities where they are applied, where ethnic and cultural factors can have a significant impact; (VII) they are tools subject to *bias* due to difficulties to access to technology (e.g., *digital divide*). However, it has also been verified in previous studies [15], how some insights in the submission phase allow us to limit the impact of this *bias*. For example, a *peer-to-peer* diffusion with multimedia tools with a clear message to support the less accustomed to IT is useful [15].

#### 3.3.2. The Proposal and Test of an Electronic Survey

We designed and validated a wide-ranging electronic survey (eS). We have calibrated it on the Italian national reality. In the introduction, as in [15], we have clearly suggested to support those less familiar with the IT.

As for the questions related to the *escape*, we were inspired by the survey in [22].

Our survey was accompanied with other questions (for example, open questions and/or free questions) to obtain feedback that gives an idea of the critical issues with regards also to the *Equity*, *digital divide* and the *exposure risk*.

The survey was submitted to a first sample to get an initial feedback on its effectiveness and the ability to bring out any critical issues through mobile technology. This survey was tested in the period 1–25 March 2021 on a first sample of citizens, and we are currently evaluating how to transfer it to stakeholders in the health domain, for a structured and broader use. For the test, we focused to a *mission critical* sample. The first test sample selected was represented by the elderly, who represent [15] the persons with the greatest criticality towards the technology: *invited*: *155*; *participated*: *153*; *males/females*: *77/76*; *min age/max age*: *66/77*; *averaged*: *73.3*.

We then subsequently subjected this survey to an acceptance validation/assessment on two different samples, using some parameters related to the theme of this study (Table 1). Among the *mission critical* persons, in the case of the continuation of a high state of infection with Sars-Cov-2 due to a failure of the vaccination process, we find health workers. We addressed these persons for the validation/acceptance of the survey. Therefore, the two samples were selected considering their sensitivity, experience, and exposition risk in relation to COVID-19. The first group is represented by technical operators of medical radiology: *invited*: *103*; *participated*: *103*; *males/females*: *50/53*; *min age/max age*: *28/57*; *averaged*: *41.3*.


**Table 1.** Opinions of the 103 insiders on the survey (1 = minimum grade; 6 = maximum grade).

The second group is represented by biomedical laboratory technicians: invited: 89; participated: 89; males/females: 46/43; min age/max age: 28/56; averaged: 42.3.

Both groups have had and have, albeit operationally different, an important exposure to COVID-19. The former, for example, has exposure during confirmation chest radiology. The second has an exposure from sampling to analysis of the samples.

A comparative opinion of them is, of course, important, even if it is not exhaustive (it would be necessary to hear politicians, decision makers, economists, and other figures), in fact, it gives us an important idea of who struggles every day with this problem, in line with a piece of opinion. No technical problems were reported.

In [55], the electronic survey is available. As for the questions (*Q21, Q22, Q24*) on the intention or non-intention to receive the vaccine, we were inspired by the survey in [22]. Furthermore, different types of questions were used (*choice, multiple choice*, *Likert*, *open*, *graded evaluation*) to face the themes related the *escape*, the *exposure risk*, the *equity*, and the *digital divide* [55]. In relation to all those who participated in the survey (153 out of 155) the number of 129 (84.31%) showed their intention to be vaccinated, while the number of 24 (15.69%) no, a percentage that if confirmed in wide-ranging investigations would

be decidedly worrying. The two main reasons for no are in order: the concern for future effects (24 votes), the concern for side effects (4 votes). The two groups involved in the eS validation process were free to try the survey after receiving it. Table 1 highlights the highly positive opinion on the survey on the first group of assessors. The Shapiro–Wilk test, which is preferable for small samples, successfully tested the normality on the two groups.

The answers of this second sample of assessors did not differ significantly from those of the first in average (T-student, *p* > 0.5 in all the comparisons among the averaged values).

#### **4. The** *mTech* **and the Usefulness to Optimize the Process: The Italian Example**

#### *4.1. General Considerations*

The vaccination process is benefiting from *Digital Health* for the optimization of the path, from the first call, based on predefined priorities, to the issuance of the certificate. The two key components of *Digital Health* are the electronic health (*eHealth*) and *mHealth* (which uses *mTech*).

The *eHealth* is fundamental for many activities, such as, for example, in the connection between the Hospital Information System and national networks (with medical records and data).

The *mHealth* is essential to connect the citizen to the process through *mTech.* In many countries with a fair degree of technological evolution, there are (national) apps connecting citizens to the public administration.

These Apps allow access through the so-called digital identity. They have been equipped, in some cases, with useful functions for Digital Health applications. These functions also allow the management of the vaccination process in an optimized way.

#### *4.2. An Example of the Italian Approach*

In Italy, the booking processes are also based on the national *App IO* [21]. The app of the public services uses an access based on the *digital identity*. Aware of the problem of the *digital divide* (Section 2.2) from a literacy point of view [13], the Italian state, during last autumn, activated familiarization initiatives towards this app by inserting a reimbursement program (Cashback) [56].

The program managed by the app allows a reimbursement for most of payments made by electronic money. This has made it possible to decrease the *digital divide* from a literacy point of view and to arrive more ready for the vaccination process also managed by this app.

Basically, in Italy, *Digital Health* also uses this App in the vaccination process.

It is also through this App that priority processes related to vaccination can be managed. There is the possibility to access various sections of a message box (Figure 2). Figures 2–4 reports for the message box of an author (Giansanti D).

Figure 2, A reports the message box of the vaccination process. Figure 2, B shows the message box of the reimbursement program.

Figure 3 reports the message box highlighting an example of the priority changes during the last spring, following the legislative changes. The active priorities are shown in the section A. They consider the exposure risk, dynamically updated based on national models and regulations. The screenshot shows a change in priority as reported in section B. Figure 4 shows the end of the process manged by the app with the production of the EU certificate (*Green Pass*), subtitled with an English translation available in the message box.


Figure 3 reports the message box highlighting an example of the priority changes

during the last spring, following the legislative changes. The active priorities are shown in the section A. They consider the exposure risk, dynamically updated based on national models and regulations. The screenshot shows a change in priority as reported in section B. Figure 4 shows the end of the process manged by the app with the production of the

**Figure 2.** The message box dedicated: ( **Figure 2.** The message box dedicated: ( **A**) to the vaccination program ( **A B**) the Cashback. ) to the vaccination program (**B**) the Cashback.

**Figure 3.** The message box highlights an example of the priority changes during the last spring following the legislative changes (**A**) (citizens over 80; frail people; personnel in the health domain). (**B**) Information on the changed priority during the submission (teachers, police and army personnel, care-givers and relatives of frails are excluded). **Figure 3.** The message box highlights an example of the priority changes during the last spring following the legislative changes (**A**) (citizens over 80; frail people; personnel in the health domain). (**B**) Information on the changed priority during the submission (teachers, police and army personnel, care-givers and relatives of frails are excluded).


**Figure 4.** The message box related to the EU COVID-19 vaccine certificate (**left**) and the final EU **Figure 4.** The message box related to the EU COVID-19 vaccine certificate (**left**) and the final EU certificate, with the English translation (**right**) after the download.

#### certificate, with the English translation (**right**) after the download. **5. Conclusions**

*5.1. Highlights*

**5. Conclusions**  *5.1. Highlights* 

ing applications [60].

reach the citizen and not to lose him.

The COVID-19 pandemic shows numerous peculiarities when it is compared with previous pandemics. We can highlight, among these ones, those relating to the availability of the *mTech*, based on smartphones. While it was once necessary to develop dedicated wearable technological solutions [57] for medical applications, today, the smartphone allows the citizen to interact with the health domain through *mTech* and *mHealth* applications [58]. The important role of this technology has been seen and is continuing to be seen in telemedical applications, such as in the activities related to the continuity of care in the The COVID-19 pandemic shows numerous peculiarities when it is compared with previous pandemics. We can highlight, among these ones, those relating to the availability of the *mTech*, based on smartphones. While it was once necessary to develop dedicated wearable technological solutions [57] for medical applications, today, the smartphone allows the citizen to interact with the health domain through *mTech* and *mHealth* applications [58]. The important role of this technology has been seen and is continuing to be seen in telemedical applications, such as in the activities related to the continuity of care in the various forms of teleconsultation, television, and tele-diagnosis [59]. We have also seen, during the pandemic, the leading role of these technologies in the most common applications of daily life, from teaching to e-banking [11], or in the innovative digital contact tracing applications [60].

The *opinion* has addressed the role of *mTech* in this period, in which mass vaccination is being carried out, and in which incredible efforts are being made both to meticulously

The experience related to vaccination suggests problems to be faced and solutions. The communication is certainly playing an important role in this moment [8,9]. It has a

various forms of teleconsultation, television, and tele-diagnosis [59]. We have also seen,

The *opinion* has addressed the role of *mTech* in this period, in which mass vaccination is being carried out, and in which incredible efforts are being made both to meticulously reach the citizen and not to lose him.

The experience related to vaccination suggests problems to be faced and solutions. The communication is certainly playing an important role in this moment [8,9]. It has a key role, as it can strongly affect the behaviour of the citizen. However, in this moment, the communication has lost the important component of *face-to-face* interaction [10] due to social distancing. Experts and scholars are undoubtedly facing optimal strategies to positively influence the behaviour, through solutions based on traditional means. We have seen the important role of these tools, for example, in convincing elderly people to get vaccinated [61]. It was also seen how the role of the incentives, offered through these messages, can have a positive impact on the intention to vaccinate, as illustrated for example in [62]. It was highlighted in this study as a message that offers cash money, compared to lottery, is more captivating.

In our study, it has been highlighted how mobile technology can be particularly helpful in this moment on this issue. It was highlighted how specific Apps can play an important role as a communication tool capable of allowing the dissemination of correct information, fighting the *infodemic* [44,45,47], and recalling citizens with calibrated messages [48].

The importance of electronic surveys has also been elucidated in [54], traveling through *mTech,* as sensors for politicians and stakeholders in the health domain. This is the case of the US experience with the electronic surveys [54], through the *U.S. COVID-19 Trends and Impact Survey, 2020–2021*, which had 2,000,000 hits. This national survey has undergone continuous updates, and the latest revisions:


focused precisely on this theme.

The large-scale usefulness of electronic surveys was also reaffirmed through our brief analysis reported in this study. *mTech* is also today essential in optimizing the vaccination process, following the citizen in all the phases ranging from the first call up to the issuance of the certificate as illustrated through an example based on the Italian IO App [21].

#### *5.2. Final Reflections*

Both the importance and the potential of the role of *mTech* clearly emerges in the study. However, the importance of not being *cut off* from access to these technologies also clearly emerges indirectly. These difficulties fall within the problem of the *digital divide*. They range from the difficulties in accessing infrastructures up to the degree of literacy [12–14].

While in poor countries there is still a physical difficulty in accessing the infrastructure, in rich countries this difficulty is becoming material [12].

Initiatives that improve access to infrastructure are welcome, for example through the provision of economic bonuses for the purchase of IT [46]. Initiatives that improve *literacy* are also welcome, as the initiative that we have reported and described in the study [56].

Through our study, the importance of electronic survey tools also emerges, as in the case of US surveys [54]. It is true that these surveys, as highlighted in [15], using appropriate precautions can allow us to minimize the *bias* due to the *digital divide* (for example, by inviting those most familiar with technology to support the less familiar). It is also true that these biases in general cannot be eliminated. We must therefore not forget other solutions. We must not forget other methods, such as the qualitative methods, which, as it is well known, provide a methodology for the description of phenomena, such as understanding and behaviour [63–65]. They are based on inductive reasoning, according to

which hypotheses derive from observations. We must not forget other emerging techniques with qualitative output based on the so-called sentiment analysis. The sentiment analysis is a methodology evaluating the movements of opinion on social media, also analysing the activity on Facebook, Twitter, Instagram, Youtube, Twitch, news and blog platforms [66]. This methodology takes into consideration only citizens who interact with social media (not conditioned by the *digital divide*), however, in this way, large movements of opinion are kept under control.

#### *5.3. Final Thought*

The final thought for the stakeholders in the health domain is that *mTech* can play an important role in this phase. Nevertheless, it remains an important slice of the population with difficulty in accessing the IT, both in rich and in disadvantaged countries. It is therefore necessary to invest in initiatives that minimize the *digital divide*. It is also necessary not to forget that *mTech* will never be able to supplant other methodologies, such as the irreplaceable one of *face-to-face* communication. It will have a useful, but certainly complementary, role.

**Author Contributions:** Conceptualization, D.G.; methodology, D.G.; software, G.M. and D.G.; validation, D.G. and R.S.; formal analysis, All; investigation, All; resources, All; writing—original draft preparation, D.G.; writing—review and editing, D.G., R.S. and G.M.; visualization, All; supervision, D.G.; project administration, D.G. and R.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Healthcare Professionals' Role in Social Media Public Health Campaigns: Analysis of Spanish Pro Vaccination Campaign on Twitter**

**Ivan Herrera-Peco 1,\* ,† , Beatriz Jiménez-Gómez 1,†, Juan José Peña Deudero <sup>2</sup> , Elvira Benitez De Gracia <sup>2</sup> and Carlos Ruiz-Núñez <sup>3</sup>**


**Abstract:** The COVID-19 pandemic has generated a great impact worldwide both on the population health but also on an economic and social level. In this health emergency, a key element has been and still is the need for information, which has become a daily concern for many people. Social media represent powerful tools for searching and gathering health-related information, thus becoming a new place where health authorities need to be present to disseminate information of preventive measures like vaccines against COVID-19, as well as try to block information against these public health measures. The main goal of this study was to analyze the role that healthcare professionals have in Twitter to support the campaign of public institutions on vaccination against COVID-19. To address this study, an analysis of the messages sent on Twitter containing the hashtag #yomevacuno, between 12 December 2020 was developed using the NodeXL software (Social Media Research Foundation, Redwood, CA, USA), focusing on content analysis of tweets and users' accounts to identify healthcare professionals. The results show that healthcare professionals represent only 11.38% of users, being responsible for 6.35% of impressions generated by the network #yomevacuno. We can observe that traffic information generated by healthcare professionals is not significant in comparison with institutions (*p* = 0.633), but it is compared to common users (*p* = 0.0014). The most active healthcare professionals were pharmacists (40.17%), nurses (27.17%), and physicians (12.14%). Their activity (90.43% of messages) was mainly focused on sharing messages generated by other users' accounts. From original content generated by healthcare professionals, only 78.95% had a favorable storytelling on the vaccine, but without sharing information about vaccines or vaccination. As a conclusion for this study, the participation of healthcare professionals in the dissemination and generation of information within the #yomevacuno communication strategy, led by the Spanish Ministry of Health, has been scarce. We emphasize the need to enhance communication skills in social networks to support public health campaigns through these increasingly important social media.

**Keywords:** COVID-19; healthcare professionals; public health; vaccines; social media

#### **1. Introduction**

COVID-19 disease, which started in Wuhan, China, with the first case reported on December 2019 and continuing today, was defined by World Health Organization (WHO) as an international outbreak of a public health emergency and declared as a pandemic on 11 March 2020 [1]. This disease has a number of characteristics that have facilitated its rapid spread, such as the long incubation period [2] and the high number of asymptomatic carriers [3]. The impact of this pandemic worldwide, regarding the level of deaths and

**Citation:** Herrera-Peco, I.; Jiménez-Gómez, B.; Peña Deudero, J.J.; Benitez De Gracia, E.; Ruiz-Núñez, C. Healthcare Professionals' Role in Social Media Public Health Campaigns: Analysis of Spanish Pro Vaccination Campaign on Twitter. *Healthcare* **2021**, *9*, 662. https:// doi.org/10.3390/healthcare9060662

Academic Editor: Daniele Giansanti

Received: 25 April 2021 Accepted: 28 May 2021 Published: 2 June 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

infected individuals is indisputable [4], but one of the aspects that should also be taken into account is the impact on mental health caused not only by the disease, but also by all the measures implemented to curb and/or prevent its transmission, including the generation of information and its dissemination [5,6].

As has been described in other health emergencies, individuals may generate negative emotions resulting in stress, anxiety, fear, or uncertainty [7,8], not to mention irritability, anger, frustration, phobia, fear, and intolerance [9,10].

One of the elements that affects individuals' mental health and, in addition, may lead to a decrease in adherence to recommendations by health authorities to address the struggle against COVID-19, is the information consumed by the population [10,11]. Importantly, in health emergencies such as that caused by the COVID-19 pandemic, the need for information has become a daily concern for many people [12].

The term 'infodemic' is a mixture of "information" and "epidemic" and refers to an abundance of information which can be either accurate or inaccurate. According to the World Health Organization (WHO), COVID-19-related 'infodemic' is as dangerous as the virus itself. One of the fastest ways to obtain health-related information is from the internet and, in particular, within social media [13]; not to mention the information spread through messaging apps such as WhatsApp, Telegram, etc. [14], elements that are essential to understand the problem of current 'infodemic'. Social media represent powerful tools for searching for health-related information or for gathering such information. It is in this aspect in which it can evidently be seen that the population in general, and especially patients, share information, or opinions, about subjects related to health [15].

An example of the importance of this type of media for the spread of information is the social network Twitter, which has approximately 271 million users being responsible for over 500 million tweets every day [16]. While other social networks, such as Facebook, Instagram, Tiktok, etc., do provide health information, Twitter is the one playing the most important role in disseminating information during the COVID-19 pandemic [17–20].

However, both the easy and quick access to these platforms and the lack of control over the veracity of the content posted, mean that they can be considered as rapid means of dispersing unverified health information [20,21], constituting a potential threat to public health [12,21] since disinformation, misinformation, and conspiracy theories hinder mitigation, transmit misleading messages about the disease, and promote ineffective precautionary measures [22,23].

'Infodemic' cannot be eliminated, yet it can be managed [24]. The rapid detection of health misinformation is essential for such management and it involves appropriate training in evidence-based practice, together with a good strategy for dissemination of information [25], which helps the population to be well informed and able to effectively react to a pandemic [26].

In an emergency situation where public health is at risk, the role of healthcare professionals as key elements for communication strategies, based upon reliable and verified information, proves essential [27,28]. In this sense, healthcare professionals would become active agents in spreading information and controlling false information, either misinformation or disinformation, in order to protect the public from wrong contents. Thus, increasing empowerment and health promotion, and playing a crucial role in supporting individuals and communities into the understanding of public healthcare messages [29,30].

The main objective of this work is to analyze the role of healthcare professionals during the start of the campaign launched in the social network Twitter, by the Ministry of Health of the Government of Spain, in favor of vaccination against COVID-19. More specifically, what type of professionals participated, their role in the impact instilled by the campaign, and the type of messages they conveyed through this social network.

#### **2. Materials and Methods**

#### *2.1. Study Design and Ethics*

An observational, retrospective, time-limited study was proposed, in which activity on the social network Twitter was analyzed.

This study was considered exempt from ethical review because it was performed upon a social network and the study did not interfere with any patient or human data beyond measuring internet activity among Twitter users. Also considering that this study only compiled data from users who consented on Twitter to disclose their data openly (i.e., no privacy settings were selected by users) being completely public.

Furthermore, users' accounts have been anonymized in order to develop good research practices in social networks [31].

#### *2.2. Data Collection*

The information from the tweets was extracted through an API (application programming interface) search tool, using the professional version of the software NodeXL (Social Media Research Foundation, Redwood, CA, USA).

To achieve the objectives proposed in this study, the keyword "yomevacuno" ('igetvaccinated') and the hashtag #yomevacuno were selected. The main reason for this selection was that this is the very hashtag used by Spanish health authorities to start a support campaign for anti-COVID-19 vaccines and the vaccination itself, as the best way to stop the spread of COVID-19 and raise awareness on the utility of vaccines. Although COVID-19 is a pandemic, we believe that it is necessary to analyze the situation across countries, in a specific manner, because the social situation in each country has its own particularities. For this reason, this study focuses on an observational analysis of the network 'yomevacuno' in Spain, excluding users that could communicate in Spanish on Twitter but are located in a different place from Spain.

The Twitter users included in the analysis of the data were those who had sent tweets with the above-mentioned characteristics during a predefined period. Unverified users were also included, to analyze the dissemination of messages.

The tweet selection criteria for this study were: (i) tweets published in Spanish language; (ii) tweets containing the hashtag #yomevacuno or the keyword "yomevacuno" or the phrase "yo me vacuno"; (iii) users located in Spain; (iv) tweets that were published between 14 December (00:00 a.m. CET) and 28 December 2020 (23:59 p.m. CET).

With the data collected from the hashtag #yomevacuno, it was observed that a total of 3038 Twitter users participated, amounting for 915,736 impressions (visualization and interaction with tweets). In addition, it was found that there were a total of 4918 interactions, including 421 (8.56%) tweets (considered as original content), 2377 retweets (48.33%), 126 replies (2.56%), and 1994 mentions (40.54%)

#### *2.3. Data Analysis*

The analysis of the data obtained was performed in several steps. The first step was to analyze the most influential Twitter users who posted under the aforementioned hashtag, as well as their characteristics. We have used a traditional social network analysis technique like the betweenness centrality score (BCS). This centrality measure in social network terms, is associated with the user's power in the network, understanding it like the importance of connecting and transmitting information across the entire network [32]. The BCS measures the influence of a vertex over the flow of information to other vertices, always assuming that information will travel through the shortest vertex path. The BSC value reflects how a user can control the information, choosing whether to share it or not, disclosing it to his/her network [33,34]. In our study, the BCS is the value used to define the influential users in the network #yonomevacuno. The Twitter users are compiled and grouped by nodes using the Clauset–Newman–Moore cluster algorithm.

In relation to the hypothetical activity in the network 'yomevacuno', the BCS allows us to identify the content, activities, and/or influential users that would be strongly associated with overall Twitter activity measured by the metrics of total tweets, impressions, retweets, and replies [35]. It is important to define that tweets are associated with the creation of original content by another user, meanwhile, retweets are an indicator that shows the transmission of a tweet sent by another user (it is not original content). Finally, the impression is an indicator of information propagation obtained when the number of tweets is multiplied by the number of followers [35].

Finally, an analysis of both the users' account description and the contents of the tweets was performed. With regard to users' accounts, we analyzed the description of users identifying as healthcare professionals (HCP from now). Furthermore, original tweets analysis was taken into account, since these are deemed to be the ones generating the original content disseminated throughout the user network. Prior to the content analysis of original tweets, the coding variables were defined. The first variable, 'media', captured the presence of media in the tweet and the type of media employed (i.e., video, image, or document), if applicable. The variable 'message function' was coded using three coding variables: 'information', 'action', or 'community'; where 'information' means tweets which main purpose was to inform, educate, or update the reader on COVID-19 transmission, symptoms, or how vaccines work. 'Action' tweets were intended to prompt changes in the behavior of other Twitter users. 'Community' tweets were associated with stories from members of the community about community-building, vaccines, or COVID-19 disease experience.

Finally, content credibility of tweets was performed, where researchers analyzed the existence of external links that allowed independent corroboration, and analysis of the structure of tweets' searching for clues about possible failures in credibility like inappropriate wording, spelling, and/or grammar [20].

The content coding was performed independently by two researchers and corroborated by a third person, so that any approach and focus differences were always discussed and resolved with full agreement.

#### *2.4. Statistical Analysis*

This study is quantitative and observational. For data statistical analysis, descriptive and inferential statistics, we used the Statistical Package for the Social Sciences software (SPSS) version 23.0. (IBM, Armonk, NY, USA) Kolmogorov–Smirnoff non-parametric analysis was performed for comparison of means. The statistical level of significance was set at *p* < 0.05.

#### **3. Results**

#### *3.1. Users Analysis*

Within the #yomevacuno network, 3038 users were found, of which 346 (11.38%) identified themselves as healthcare professionals in their user description. Within these users, identified as healthcare professionals, it was observed that the four professional groups with the highest activity were pharmacists, 139 users (40.17%); nurses, 94 users (27.17%); physicians, 42 users (12.14%); and psychologists, 25 users (7.23%) (see Figure 1).

In relation to the messages sent through the 'yomevacuno' network by healthcare professionals, 397 messages were generated, 38 of which were tweets (9.58%), 181 retweets (45.59%), 10 replies (2.52%), and 168 mentions (42.31%) (Table 1), with a creation of original content.

**Figure 1.** Healthcare professionals in 'yomevacuno' network. Where HCP, means healthcare professionals, and non-HCP means users than do not define themselves as healthcare professionals (including public and private institutions and organizations). **Figure 1.** Healthcare professionals in 'yomevacuno' network. Where HCP, means healthcare professionals, and non-HCP means users than do not define themselves as healthcare professionals (including public and private institutions and organizations).


**Table 1.** Messages from healthcare professionals in #yomevacuno network.

Note: %, is percentage.

were mainly accounts of official organizations, six out of the 10. The remaining user accounts were individual profiles, three of which belonging to healthcare professionals (Table 2). **Table 1.** Messages from healthcare professionals in #yomevacuno network. **'Yomevacuno' Network Healthcare Professionals %** The hashtag #yomevacuno was analyzed and the users that participated in this network were ranked by the betweenness centrality score, finding that the 10 most influential users were mainly accounts of official organizations, six out of the 10. The remaining user accounts were individual profiles, three of which belonging to healthcare professionals (Table 2).

Tweets 421 38 9.03 Retweets 2377 181 7.61 Replies 126 10 7.94 **Table 2.** Top ten influential users ranked in #yomevacuno network by their betweenness centrality score (which measure their influence over the flow of information in the network).


Pos5 Healthcare professional (pharmacist) 433,595.884 *3.2. Influence of Healthcare Professionals in Potential Impressions in 'Yomevacuno' Network*

Pos6 Healthcare professional (virologist) 177,106.144 Pos7 Healthcare professional (physician) 160,118.699 Pos8 Official account of European Commission 152,567.392 With regards to the influence of users labeled as healthcare professionals within the hashtag studied, #yomevacuno, it is found that they generated an amount of 58,177 impressions on the network, which represented 6.35% of the total impressions (Table 3). While the

Pos9 Citizen (journalist) 82,384.904

rest of users in this network were identified as non-healthcare professional and generated 93.65% of total impressions in the network 'yomevacuno'. From this group, users identified as institutions generate 54.45% of impressions with an average of 356.2 interactions per user (Table 3). We observed that the impressions from HCPs compared with non-HCPs is not significant (*p* = 0.129), likewise when comparing HCPs with institutions (*p* = 0.99), meanwhile the impressions generated by HCPs against common users present a significant difference (*p* = 0.0014).

**Table 3.** Impressions generated by users identified in #yomevacuno network.


Note: HCP means healthcare professionals. S.E. means standard error. % means percentage. N.s. means no significance. \*\*\* means statistical significance (*p* < 0.001).

> Within the healthcare professionals, it was observed that those generating the highest number of impressions were pharmacists with 22,808 (39.2%), followed by physicians 14,819 (25.47%), nurses 11,041 (18.98%), psychologists 3332 (5.73%), and others (biologists, biochemists, etc.) with 2490 impressions (10.62%).

#### *3.3. Content Analysis*

We proceeded to analyze all the tweets generated by healthcare professionals on the twitter network under the hashtag #yomevacuno, finding that 78.95% (30) had a favorable storytelling on the vaccine, 15.79% (six) did not generate an opinion nor provide information as they only sent the hashtag #yomevacuno, and 5.26% (two) were tweets not associated with healthcare information of any kind. From 30 original tweets we observed that 12 (40%) were associated with the category 'inform', 11 (36.7%) were messages included in 'actions', and seven (23.7%) were messages defined as 'community'.

The data show us that the main activity of healthcare professionals in the network 'yomevacuno' was associated with retweets (Table 1), 181 messages that represent 45.6% of total messages from healthcare professionals. It was noted that the most widespread messages were related to the announcement by the Spanish Ministry of Health on the approval of the Pfizer vaccine by the EMA (repeated 63 times) (Figure 2), and the statement by the official account of the Spanish Government on the start of vaccination in Spain (repeated 44 times). With regards to the provision of truthful information and reliable sources about the safety of vaccines, the message came from the Spanish Vaccinology Association and was repeated on 11 occasions (Table 4).

In relation to the analysis of the credibility perceived in healthcare professionals' tweets, we found that only three tweets (5.26%) offered external links that allowed independent corroboration; in addition, we observed presentation problems (inappropriate wording, spelling, and/or grammar) in 27 tweets (71.05%).

*Healthcare* **2021**, *9*, 662


141

**Figure 2.** Image of the most shared tweet by healthcare professionals in #yomevacuno network. **Figure 2.** Image of the most shared tweet by healthcare professionals in #yomevacuno network.

#### In relation to the analysis of the credibility perceived in healthcare professionals' **4. Discussion**

tweets, we found that only three tweets (5.26%) offered external links that allowed independent corroboration; in addition, we observed presentation problems (inappropriate wording, spelling, and/or grammar) in 27 tweets (71.05%). In the present study, the role of HCPs in the dissemination and generation of content on Twitter was evaluated by analyzing #yomevacuno, focused on boosting the vaccination awareness campaign against COVID-19 in Spain.

**4. Discussion** In the present study, the role of HCPs in the dissemination and generation of content on Twitter was evaluated by analyzing #yomevacuno, focused on boosting the vaccination awareness campaign against COVID-19 in Spain. It is important to highlight that the participation within social media, and Twitter particularly, have a voluntary nature. This situation means that HCPs participate at dif-It is important to highlight that the participation within social media, and Twitter particularly, have a voluntary nature. This situation means that HCPs participate at different level adopting specific roles based in their experience in social media. We could have a mixture of lurkers, observers, passive users, and of course, active contributors. Non-participant users [36] continue to belong to the network 'yomevacuno' and they have potential access to important information related to vaccines and pro-vaccine news. Situation that could explain the low participation of HCPs in traffic information in the network analyzed.

ferent level adopting specific roles based in their experience in social media. We could have a mixture of lurkers, observers, passive users, and of course, active contributors. Non-participant users [36] continue to belong to the network 'yomevacuno' and they have potential access to important information related to vaccines and pro-vaccine news. Situation that could explain the low participation of HCPs in traffic information in the network analyzed. As can be seen, individual users, HCPs and non-HCPs, had a lower weight than institutions within all the traffic, impression generated in the analyzed network, something that contrasts with analyses performed in other vaccination campaigns, such as the one carried out in 2018 for influenza vaccination in Spain [22] or international campaigns on awareness of public health issues, where individual users were the main generators of information and traffic in the networks versus institutions [26,33]. However, it is remark-As can be seen, individual users, HCPs and non-HCPs, had a lower weight than institutions within all the traffic, impression generated in the analyzed network, something that contrasts with analyses performed in other vaccination campaigns, such as the one carried out in 2018 for influenza vaccination in Spain [22] or international campaigns on awareness of public health issues, where individual users were the main generators of information and traffic in the networks versus institutions [26,33]. However, it is remarkable that HCPs have a more important role in traffic generated in the network 'yomevacuno' than common users, since our findings according to the previous bibliography, suggest that health authorities should appeal to HCPs' social responsibility to attract them as followers for the messages (tweets) that can be generated in public health campaigns, so that the messages can have more dissemination and more credibility, as well as the campaign itself [37].

able that HCPs have a more important role in traffic generated in the network 'yomevacuno' than common users, since our findings according to the previous bibliography, suggest that health authorities should appeal to HCPs' social responsibility to attract them as followers for the messages (tweets) that can be generated in public health campaigns, so that the messages can have more dissemination and more credibility, as well as the campaign itself [37]. The common role of HCPs in social media is focused on activities of a personal nature rather than professional nature [30,35]. This can be linked with the low participation of users identified as healthcare professionals in the dissemination of information, that can be considered as a work activity [30] for vaccination awareness, and it could be explaining why HCPs have more weight in traffic information than common users in the network 'yomevacuno'.

The common role of HCPs in social media is focused on activities of a personal nature rather than professional nature [30,35]. This can be linked with the low participation of users identified as healthcare professionals in the dissemination of information, that can In relation with HCP's messages in the network 'yomevacuno', there exist a low number of tweets (original content). Situation that could be associated with the low level

of followers that could be observed in HCPs Twitter accounts, being a common strategy trying to retweet health messages from other sources, like public health institutions, rather than tweeting themselves [36].

Likewise, it was observed that the messages generated by healthcare professionals did not provide relevant information on either vaccines or the situation at that point regarding COVID-19. This situation contrasts with other studies in which it was observed that healthcare professionals used the network to send reassuring messages, or to provide understandable information to the public on specific measures and situations related to the COVID-19 pandemic [38,39].

Although there are numerous messages on the network in favor of the vaccination campaign, it can be noted that a high percentage of them have a political and communicative nature, with a low number of messages with scientific content and providing information on the vaccines themselves and their usefulness. When analyzing the tweets, as messages with original content, it can be found they do not meet many of the elements that Zubiaga et al. [20] defined as important elements that a tweet must have to be considered reliable. That is, tweets, in order to be considered reliable information, must present the characteristics of authority, support, independent corroboration, and presentation (appropriate writing, spelling, and grammar) [20], and text plausibility [20,40,41]. After analyzing the data collected, it was found that the original tweets from healthcare professionals did not offer independent corroboration and even the presentation failed, triggering poor credibility perceptions by users.

This situation, coupled with the increased stress and anxiety levels described during the pandemic [17,42] and the users' lack of trust towards information coming up from official institutions in times of great social confusion [19] as occurs in the COVID-19 pandemic [7,8,43], raise the need for reference figures, not associated with national or international healthcare institutions.

It is important to point out that healthcare professionals are considered by the population as an essential element for the understanding of health-related messages [29] and their absence in social networks as reference elements may generate distrust and even disaffection towards truthful healthcare information [42], not preventing the spread of antivaccine messages [19]. This situation can make it easy for network users to be redirected to irrelevant information about the importance of vaccination or, worse, to be redirected to inaccurate or false information about vaccines in general, and vaccination against COVID-19 in particular [18].

This study has some limitations. Firstly, the social network Twitter has been assessed, which limits the analysis of the campaign to its users. Secondly, we analyzed messages sent in Spanish from users geolocated in Spain and, moreover, the categorization of healthcare professionals was based on themselves presenting as such, which means the number of healthcare professionals may be underestimated, as there are many who do not wish to state their profession in their profiles.

However, we want to highlight an important strength of this study, because to the knowledge of the authors, the present study is the first to address an analysis of the role of healthcare professionals how key elements of a pro-vaccination campaign, against the COVID-19, in the Spanish speaking community in Twitter. This study is the first and will be able to allow initiate further developments focused to improve the efficacy of future public health campaigns.

#### **5. Conclusions**

It is of paramount importance that healthcare professionals understand the need for being present on social networks from a professional point of view, so that they can become central elements in the dissemination and creation of reliable information from a scientific point of view, aimed at health care.

Although it is extremely important for public institutions to be present and lead these campaigns, we believe it is very important to bear in mind that these institutions

should not prompt rejection in a certain part of the population, and this is where healthcare professionals can perform as reference figures to which users of social networks can turn to, in order to obtain healthcare information.

Another important element observed in this study is the low generation of original content and, in addition, the generated tweets had the problem of not providing links to external sources, with 71% presenting issues of inappropriate writing, spelling, and/or grammar, making an impression of low reliability.

We believe that this situation suggests the need to implement training actions for healthcare professionals on the use of social networks to enhance their participation and improve the effectiveness of communication. It is important to focus these actions on showing how to prepare reliable tweets, focusing on the plausibility of content, attaching external and reliable sources, and taking care of tweets writing.

We consider it necessary that healthcare professionals, as individual users, can actively collaborate in the dissemination of public health prevention policies through social networks.

**Author Contributions:** I.H.-P., contributed to the concept, design, analysis, and interpretation of data, and writing—original draft preparation; C.R.-N., contributed to design, analysis, and interpretation of data; B.J.-G., contributed to and writing—original draft preparation; J.J.P.D., contributed to collecting data and revising critically the manuscript; E.B.D.G., contributed to revising critically the manuscript. All authors are accepting and agreeing that the work is original; any methods and data presented are described accurately and honestly; any relevant interests have been disclosed. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** This study was considered exempt from ethical review because it was performed upon a social network and the study did not interfere with any patient or human data beyond measuring internet activity among Twitter users. Also considering that this study only compiled data from users who consented on Twitter to disclose their data openly (i.e., no privacy settings were selected by users) being completely public. Furthermore, users' accounts have been anonymized in order to develop good research practices in social networks.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Data available on request from the corresponding author.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Communication* **Patient Experiences with the Transition to Telephone Counseling during the COVID-19 Pandemic**

**Augustine W. Kang 1,\* , Mary Walton <sup>2</sup> , Ariel Hoadley <sup>3</sup> , Courtney DelaCuesta <sup>1</sup> , Linda Hurley <sup>2</sup> and Rosemarie Martin <sup>1</sup>**


**Abstract:** *Background:* To identify and document the treatment experiences among patients with opioid use disorder (OUD) in the context of the rapid move from in-person to telephone counseling due to the COVID-19 pandemic. *Methods:* Participants (*n* = 237) completed a survey with open-ended questions that included the following domains: (1) satisfaction with telephone counseling, (2) perceived convenience, (3) changes to the therapeutic relationship, (4) perceived impact on substance use recovery, and (5) general feedback. Responses were coded using thematic analysis. Codes were subsequently organized into themes and subthemes (covering 98% of responses). Interrater reliability for coding of participants' responses ranged from 0.89 to 0.95. *Results:* Overall, patients reported that telephone counseling improved the therapeutic experience. Specifically, 74% of respondents were coded as providing responses consistently indicating "positive valency". "Positive valency" responses include: (1) feeling supported, (2) greater comfort and privacy, (3) increased access to counselors, and (4) resolved transportation barriers. Conversely, "negative valency" responses include: (1) impersonal experience and (2) reduced privacy. *Conclusions*: Telephone counseling presents its own set of challenges that should be investigated further to improve the quality of care and long-term patient outcomes.

**Keywords:** opioid use disorder treatment; telehealth services; qualitative; needs assessment

#### **1. Introduction**

The ongoing opioid epidemic and the COVID-19 pandemic constitute a syndemic [1]. More than 40 states in the United States have reported increases in opioid-related mortality in the first six months of the pandemic, which has become more complicated and deadly as the pandemic persists [2].

Medications for opioid use disorder (MOUD) are among the most systematically governed treatment approaches in the United States [3]. Although MOUD is the evidencebased standard of care for OUD, access is limited primarily due to the strict federal and state regulations mandating in-person medical and clinical encounters to initiate and maintain MOUD. However, the COVID-19 public health emergency led to an immediate cascade in relaxing laws, regulations, and policies to enable ongoing treatment by reducing financial and administrative obstacles and expanding the role of telemedicine, to name a few [4]. These changes resulted in shifts in the access and delivery of MOUD, providing an opportunity to improve treatment and thus reduce opioid-related morbidity and mortality in a time of national crisis.

Rates of telehealth services provisions for substance use disorder (SUD) had been generally low before COVID-19 even though telehealth services increase patient access,

**Citation:** Kang, A.W.; Walton, M.; Hoadley, A.; DelaCuesta, C.; Hurley, L.; Martin, R. Patient Experiences with the Transition to Telephone Counseling during the COVID-19 Pandemic. *Healthcare* **2021**, *9*, 663. https://doi.org/10.3390/ healthcare9060663

Academic Editor: Daniele Giansanti

Received: 28 April 2021 Accepted: 31 May 2021 Published: 2 June 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

adherence, and retention in treatment while yielding equivalent outcomes to in-person care [5,6]. For OUD treatment specifically, some evidence indicates similar rates of counseling attendance, drug-positive urinalysis results, and retention in treatment between telehealth versus in-person-based provision of services [7,8].

As CODAC Behavioral Healthcare, Inc., the largest outpatient opioid treatment organization in the state of Rhode Island (USA), transitioned from in-person encounters to a virtual telephonic platform, it was unclear how the therapeutic relationship between patients and counselors would change as a result. Positive therapeutic relationships (or alliances) are important for treatment engagement as it indicates (1) high-quality interactions between patients and their counselors, (2) personal bonds between patients and their counselors, and (3) a collaborative relationship of task and goal development for the patients' substance use recovery journey [9]. Hence, CODAC and Brown University partnered to conduct patient satisfaction surveys to explore patients' perspectives on telephone counseling during the pandemic. The aim of this paper is to qualitatively examine patient responses to open-ended survey questions to gain insight into how telephone counseling may have impacted the patients' treatment experience.

#### **2. Materials and Methods**

*Study Design*. The present study examines data from a larger quality improvement project (at CODAC) to assess patient and counselor experiences with telephone counseling in the context of COVID-19 risk mitigation. Cross-sectional survey data was used to understand the experiences of patients, counselors, and prescribers who had participated in telephone counseling sessions and/or provided services to patients across seven opioid treatment program (OTP) clinics (under the ambit of CODAC) located across Rhode Island during the COVID-19 pandemic. Data from administrative records included insurance status, age, gender, and race/ethnicity. Counseling services were required at least once per month and could receive one of the three FDA-approved medications for opioid use disorder: methadone, buprenorphine, or extended-release naltrexone. All data were deidentified. The CODAC research oversight committee reviewed and approved the project.

*Participants.* From 3 July to 8 November 2020, prospective participants were invited to complete the survey via their CODAC-based counselors during routine telephone counseling sessions or via OTP staff in-person at the clinic. Patients who provided verbal consent to counselors received an invitation via text message to participate in a web-based, patient satisfaction survey. Patients approached in-person at the clinics completed paper surveys. All patients who completed the survey were entered into raffles at each treatment site for a \$25 gift card.

Approximately 16% of all CODAC patients who had at least one telephone counseling session at a clinic from 16 March to 8 November 2020 and who received in-person counseling prior to the COVID-19 mitigations completed the survey. The survey included five open-ended questions that queried the following domains respectively: (1) satisfaction, (2) convenience, (3) relationship with their counselor, (4) substance use and recovery, and (5) general feedback (Table 1).

*Qualitative Analysis Approach.* Open-ended responses to the five questions described above were coded by two independent raters following the principles of inductive thematic analysis [10]. Specifically, both raters read all responses, assigned preliminary codes to the texts, and then discussed emergent codes and themes collectively with the study authors. A codebook containing two major themes and four subthemes was developed via an iterative coding process (i.e., assigning and re-assigning the names of codes if necessary, taking into account the context of emerging evidence as the qualitative coding process proceeded) covering all five open-ended questions. Two other raters then subsequently re-applied the codebook to the open-ended responses into codes and themes. The final codebook covered 98% of patients' responses. Interrater reliability for coding of patient responses ranged from 0.89 to 0.95.


Participant characteristics are presented in Table 2.

Gender

covery, please do so.

If you would like to explain how counseling has or has not helped in your substance use or re-

Is there anything else you would like to tell us about your counseling experiences during the

pandemic?

*Healthcare* **2021**, *9*, x 3 of 8

*Healthcare* **2021**, *9*, x 3 of 8

If you would like to explain how counseling has or has not helped in your substance use or re-

Is there anything else you would like to tell us about your counseling experiences during the

Male 105 (52) 83 (52) 22 (50) 0.798 b

*Qualitative Analysis Approach.* Open-ended responses to the five questions described above were coded by two independent raters following the principles of inductive thematic analysis [10]. Specifically, both raters read all responses, assigned preliminary codes to the texts, and then discussed emergent codes and themes collectively with the study authors. A codebook containing two major themes and four subthemes was developed via an iterative coding process (i.e.**,** assigning and re-assigning the names of codes if necessary, taking into account the context of emerging evidence as the qualitative coding process proceeded) covering all five open-ended questions. Two other raters then subse-

**Table 1.** Open-ended questions. feedback pandemic?

covery, please do so.

#### **3. Results**

Substance use and recovery

General

Participant characteristics are presented in Table 2. Female 98 (48) 76 (48) 22 (50) Race/Ethnicity **Table 2.** Demographics and valency characteristics (*n* = 237).

Substance use and recovery

> General feedback


**Table 2.** Demographics and valency characteristics (*n* = 237). White, non-Hispanic 133 (81) 100 (79) 33 (90) 0.562 Black, non-Hispanic 6 (4) 5 (4) 1 (3) b **Variable** *M* **±** *SD***/***n***(%) Valency** *<sup>p</sup>* **Positive Mixed/Negative <sup>ƚ</sup>**

Notes: Missing data for gender = 34 and for race/ethnicity = 73. Notes: Missing data for gender = 34 and for race/ethnicity = 73. ƚ Negative-only participants were *n* = 29 (12% of total responses). \* Insurance status was PHI and hence was not matched to participant IDs in this survey. Figures presented are aggregated data for the broader participant population. a Independent samples *t*-test. b Chi-square analysis. Negative-only participants were *n* = 29 (12% of total responses). \* Insurance status was PHI and hence was not matched to participant IDs in this survey. Figures presented are aggregated data for the broader participant population. <sup>a</sup> Independent samples *t*-test. <sup>b</sup> Chi-square analysis.

Figures 1 and 2 summarize the results of the analysis. The codes/subthemes were grouped into two overall themes: "positive valency" and "negative valency". Participants were grouped into "positive valency" and "negative valency" if their responses across the five open-ended questions were consistently coded as positive or negative, respectively. To create a more parsimonious narrative, we grouped "mixed" valency responses (i.e., there were both positive and negative valence responses across the five questions at the participant level) with "negative" valency. "Neutral" participants (*n* = 10) were defined as providing answers that reflected indifference (e.g., "the same") and were excluded from further discussion. (Table 2). Figures 1 and 2 summarize the results of the analysis. The codes/subthemes were grouped into two overall themes: "positive valency" and "negative valency". Participants were grouped into "positive valency" and "negative valency" if their responses across the five open-ended questions were consistently coded as positive or negative, respectively. To create a more parsimonious narrative, we grouped "mixed" valency responses (i.e., there were both positive and negative valence responses across the five questions at the participant level) with "negative" valency. "Neutral" participants (*n* = 10) were defined as providing answers that reflected indifference (e.g., "the same") and were excluded from further discussion. (Table 2).

**Figure 1.** Overview of subthemes and codes—positive valency.

**Figure 2.** Overview of subthemes and codes—negative valency.

Within each valency theme, two similar subthemes were observed: (1) therapeutic relationship factors (defined as factors impacting the relationship and/or process of interaction between counselor and patient) and (2) person-level factors (defined as factors operating at the individual level that impact the counseling experience).

#### *3.1. Positive Valency—Therapeutic Relationship Factors*

Participants reported that they felt supported by their counselors and were appreciative of their efforts during the transition into telephone counseling. One participant wrote, "*My counselor goes above and beyond to make sure I have everything I need during this troubling time"* (P03, or Participant 03). Participants also described how despite the transition to telephone counseling and the lack of in-person contact they remained satisfied with service. For example, one participant wrote, "*I love my counselor. I can be completely honest with her on the phone or in-person*" (P02). Some participants also explained that telephone counseling provided more comfort and privacy relative to in-person settings. For example, "*Feel more comfortable because for me I am shy so talking on the phone feels more comfortable"* (P01). Increased access to one's counselor (i.e., a greater sense of connectedness) was also reported, such as, "*We talk more on the phone than we do in person"* (P10) and " . . . *but as*

*I said before I think it's easier to talk to her more over the phone if need be"* (P05). In addition, some participants also report increased accountability to their counselor (and perhaps, by extension, to their recovery journey) due to frequency of counseling, "*We talk more. It used to be once a month* . . . *now I get to talk to [Redacted, name of counselor] once a week. This way nothing gets missed and nothing get[s] unmentioned"* (P06) and feeling empowered over their substance use management (*"I feel now in control of my recovery by not having the feeling that I need to be somewhere at a certain time"* (P17)).

#### *3.2. Positive Valency—Person-Level Factors*

Many participants described that telephone counseling made the experience of receiving treatment more convenient compared to in-person counseling. For example, one participant explained that telephone counseling was " . . . *quick and easy, no lines to wait in*" (P18), indicating that telephone counseling may be more time-effective for some. Furthermore, participants also reported that telephone counseling made it easier to manage one's work schedule. For example, one participant mentioned, " . . . *it's the easiest way instead of having to take time out of work I can just step away for a phone call"* (P09). Participants who likely do not have reliable personal transportation methods also said that telephone counseling resolved previously experienced transportation barriers, such as one who explained that it was, " . . . *more convenient because I don't need to drive or get a ride"* (P04). Telephone counseling also resolved time-related family considerations (e.g., "*Don't have to drag my kids out"* (P11)). In addition, participants also mentioned that telephone counseling allowed for more flexibility compared to in-person counseling (e.g., "*It fits my schedule better and doesn't make counseling and dosing related"* (P12)). Lastly, participants recognized that, amidst the pandemic, switching to telephone counseling provided a sense of safety (e.g., " . . . *keeps me from getting COVID-19*" (P13)). Figure 3 summarizes the frequencies of codes for the theme of positive valency.

#### *3.3. Negative Valency—Therapeutic Relationship Factors*

Some participants were generally dissatisfied with telephone counseling (e.g., "*Every time I ask my counselor for help they took a long time or forgot"* (P16). Many participants explained that telephone counseling felt more impersonal compared to in-person counseling. For example, one participant described that "*Most issues can be handled by telephone, but obviously sometimes physical presence is required* . . . *there is something lost between the counselor and client. Certainly [even more so] for new clients who have not yet built a rapport with their coun-* *selors*" (P15). Some participants also reported reduced counselor contact after switching to telephone counseling (e.g., "*Less contact"* (P14) and *"Don't get many calls"* (P07)).

#### *3.4. Negative Valency—Person-Level Factors*

Participants also explained that they may not receive adequate privacy at home for counseling. In addition, some participants also mentioned that an adjustment to telephone counseling was necessary. For example, "*Initially, I was a bit hesitant because I wasn't home alone. However, once I worked out at-home privacy issues, I felt more confident talking and working things out*" (P08). Figure 4 summarizes the frequencies of codes for the theme of negative valency.

**Figure 4.** Frequencies of codes—negative valency.

#### **4. Discussion**

Results suggest that telephone counseling for MOUD may facilitate the therapeutic experience and treatment engagement among patients. However, our analysis also identified that telephone counseling presents its own set of challenges that may undermine treatment experiences and should be investigated further to improve the quality of care and long-term patient outcomes among the MOUD patient population.

Our findings suggest telephone counseling fostered a sense of convenience, support, and comfort (in terms of discussing one's substance use recovery), which is consistent with previous research examining telephone counseling approaches for SUDs [11]. These factors may contribute to an improved therapeutic alliance and increase the likelihood of long-term treatment engagement [12]. In the context of MOUD provision, these factors may be beneficial in improving treatment across patient populations [13,14]. Considering that individuals with SUDs experience a gap between treatment need and utilization [15], telephone counseling for MOUD could be a viable way to increase treatment access and engagement to help bridge this gap.

Future efforts to integrate various telehealth approaches in MOUD treatment provision should further explain and proactively mitigate negative patient experiences and potential barriers to virtual treatment engagement. Our results identified perceptions of an impersonal experience for some; impersonal experiences have been reported to predict reduced treatment engagement and a weaker therapeutic alliance between patient and provider [16]. While an impersonal experience has consistently been reported in the telemedicine experience, to our knowledge no papers have reported and/or explored how

perceptions of an impersonal experience with telehealth counseling for MOUD treatment may impact treatment engagement and outcomes.

Some limitations of our study should be noted. The cross-sectional study design restricted our ability to examine changes in the patients' perspectives toward telephone counseling. For example, it is unclear if fatigue with the pandemic and the extended engagement of telephone counseling will adversely affect patients' perspectives of telephone counseling, or if perspectives toward telephone counseling will change post-pandemic. In addition, the survey required patients to recall their counseling experiences pre-pandemic and contrast them with their current telephone counseling experience, which may have introduced some recall bias. Patients who did not complete any telephone counseling sessions were not eligible to participate in the study, which may have limited the scope of our data. Furthermore, our study population was primarily White and was limited to the geographical region of Rhode Island (USA), limiting the generalizability of our findings. Finally, we did not include measures of addiction severity and how it may impact therapeutic alliance in the telephone counseling context. Regardless, our findings fill a key gap in the literature in illustrating the perspectives among patients about the transition to telephone counseling.

#### **5. Conclusions**

Most patients in our study reported a positive experience with using telephone counseling for OUD treatment. In the post-pandemic setting, adopting a hybrid inperson/telehealth approach may be one way to assuage concerns regarding an impersonal experience. The expansion of the traditional system of in-person care delivery models into telephone counseling due to the pandemic holds significant promise for improving accessibility to and management of MOUD treatment among the patient population. Future research should adopt current in-person MOUD provision models and tailor evidencebased approaches to the unique nuances of the telehealth (or a hybrid telehealth/in-person) service approach.

**Author Contributions:** Conceptualization, A.W.K., M.W. and R.M.; methodology, A.W.K.; software, A.H.; validation, R.M.; formal analysis, A.W.K.; investigation, A.W.K., M.W., A.H., C.D., and R.M.; resources, L.H. and R.M.; data curation, M.W., A.H., and C.D.; writing—original draft preparation and revisions, A.W.K., M.W., A.H., C.D., L.H., and R.M.; supervision, L.H. and R.M.; project administration, M.W., L.H., and R.M.; funding acquisition, R.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by NIH U01 DA050442-01.

**Institutional Review Board Statement:** The CODAC research oversight committee reviewed and approved the project.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy and ethical considerations.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


## *Article* **Dramatic Increases in Telehealth-Related Tweets during the Early COVID-19 Pandemic: A Sentiment Analysis**

**Tiffany Champagne-Langabeer \* , Michael W. Swank, Shruthi Manas, Yuqi Si and Kirk Roberts**

School of Biomedical Informatics, The University of Texas Health Science Center, 7000 Fannin Suite 600, Houston, TX 77030, USA; Michael.W.Swank@uth.tmc.edu (M.W.S.); shruthi.manas@uth.tmc.edu (S.M.); yuqi.si@uth.tmc.edu (Y.S.); Kirk.Roberts@uth.tmc.edu (K.R.)

**\*** Correspondence: Tiffany.Champagne@uth.tmc.edu; Tel.: +1-713-500-3790

**Abstract:** The COVID-19 pandemic resulted in a large expansion of telehealth, but little is known about user sentiment. Tweets containing the terms "telehealth" and "telemedicine" were extracted (n = 192,430) from the official Twitter API between November 2019 and April 2020. A random subset of 2000 tweets was annotated by trained readers to classify tweets according to their content, including telehealth, sentiment, user type, and relation to COVID-19. A state-of-the-art NLP model (Bidirectional Encoder Representations from Transformers, *BERT*) was used to categorize the remaining tweets. Following a low and fairly stable level of activity, telehealth tweets rose dramatically beginning the first week of March 2020. The sentiment was overwhelmingly positive or neutral, with only a small percentage of negative tweets. Users included patients, clinicians, vendors (entities that promote the use of telehealth technology or services), and others, which represented the largest category. No significant differences were seen in sentiment across user groups. The COVID-19 pandemic produced a large increase in user tweets related to telehealth and COVID-19, and user sentiment suggests that most people feel positive or neutral about telehealth

**Keywords:** telehealth; telemedicine; Twitter; NLP; COVID-19

#### **1. Introduction**

The novel Coronavirus (COVID-19) was first identified in Wuhan, China, and rapidly spread across the globe, creating a public health crisis of unprecedented proportions [1]. The COVID-19 pandemic has catalyzed dramatic increases in telehealth utilization, and in the United States (US), many regulatory constraints have been relaxed to facilitate safer, contact-free access to healthcare [2]. Telehealth capabilities are vast, ranging from simple telephone consultations while consulting a patient's medical documentation to more complex live-video conferencing, remote monitoring, and diagnostic assessment. In this study, we use the broadest definition of telehealth to include telemedicine. Although several effective vaccines are now in distribution and herd immunity is a current target, some measure of social distancing will remain in effect in 2021. Telehealth will thus continue to be an effective tool for the delivery of healthcare while reducing the risk of infection that may come from in-person contact between patient and provider [3].

During the current global pandemic, people have become more active on social media as an outlet for communication, and mobile technology has increasingly become more important as a tool for government and health organizations to disseminate information [4,5]. Twitter is the most popular microblogging platform in the US and provides a rich source of data for determining the insights and thoughts of its users. Twitter was used successfully by researchers to detect emerging public health issues [6]. An important tool in the analysis of social media is Natural Language Processing (NLP), which is broadly defined as the use of computer algorithms to analyze large amounts of human language (predominantly text data) in order to extract meaning [7]. NLP is widely used to examine unstructured data and to determine how patterns manifest through the

**Citation:** Champagne-Langabeer, T.; Swank, M.W.; Manas, S.; Si, Y.; Roberts, K. Dramatic Increases in Telehealth-Related Tweets during the Early COVID-19 Pandemic: A Sentiment Analysis. *Healthcare* **2021**, *9*, 634. https://doi.org/10.3390/ healthcare9060634

Academic Editors: Daniele Giansanti and Tin-Chih Toly Chen

Received: 26 April 2021 Accepted: 25 May 2021 Published: 27 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

evaluation of language and key words, especially in social science research [8]. Although NLP has great potential for monitoring public discourse, the reliability and veracity of user-generated tweets remain a concern, as they cannot be validated as either factual or scientific [9]. However, leveraging a methodology from the field of artificial intelligence developed by a team from Google and presently considered the state-of-the-art model in NLP pre-training and language representation, Bidirectional Encoder Representations from Transformers (BERT) allows a significant number of posts to be analyzed in a short period of time with high accuracy [10,11]. As a barometer of public sentiment, Twitter is an ideal platform for analysis, as the focus is not on factual information, but rather on how people feel about trending topics or events. Social networks such as Twitter have the capability to amass a significant number of posts from users across the world, thus generating polarizing opinions, dangerous rumors, or real-time notifications about disaster threats or responses. At its most extreme, this source of data may prove a benefit and possible predictor of cyber-attacks as users express their thoughts and feelings in a public forum [12]. Sentiment, being an expression of an emotional state, can be classified as positive, negative, or neutral. Further analyzing the lexicon within specific tweets can uncover motivational factors for behavior [13]. The global proliferation of social media platforms such as Twitter allows for an endless stream of people to express their opinions and feelings about emerging public health events and has created new opportunities for informatics research to probe the COVID-19 pandemic era zeitgeist [14]. The purpose of this paper is to analyze a large body of tweets both pre-COVID-19 and early-COVID-19 to determine both interest and sentiment regarding telehealth and the influence of COVID-19.

#### **2. Materials and Methods**

#### *2.1. Raw Data*

Tweets published between 1 November 2019 and 9 April 2020 containing the terms "telehealth" and "telemedicine" were extracted using the official Twitter application programming interface (API). Our data preparation and analysis workflow consisted of the following steps: (1) Data collection; (2) Exclusions and de-duplications; (3) Preprocessing/qualitative review; (4) Unsupervised machine learning. There were a total of 117,242 telehealth and 88,321 telemedicine tweets collected. After excluding retweets and duplications (i.e., 13,133 tweets contained both terms), the total was further analyzed (n = 192,430). Preprocessing included a qualitative review of a random subset of 2000 tweets for categorical analyses.

#### *2.2. Manual Data Annotation*

Prior to NLP-based classification, the raw data had to be "labeled" for supervised machine learning. In total, 2000 tweets were manually double-annotated by a group of three individuals for the following: (1) The relation to telehealth (yes, no); (2) The sentiment (positive, neutral, negative); (3) The user category (clinician, consumer, policymaker, vendor, other); (4) The relationship to COVID-19 (yes, no). A tweet received a positive sentiment if it contained optimistic, encouraging, or validating language, e.g., "Telehealth is a valuable tool to provide care; protect people in this COVID19 pandemic". A tweet received a negative sentiment code if the tweet contained emotional words that conveyed pessimistic, debasing, or discouraging feelings, e.g. "Do you want people to keep dying and you aren't doing anything about it?" Finally, a tweet received a neutral sentiment if it included neither negative nor positive words; these tweets frequently expressed educational or objective informational phrases, e.g., "Effects of a telehealth educational intervention on medication adherence". When annotating telehealth-related tweets for sentiment, there was a possibility that a tweet could mention both telehealth and a sentiment—but have a sentiment unrelated to telehealth. Annotators were trained to evaluate the sentiment only as it related to telehealth, and these data were used to train the machine learning model.

A user was regarded as a clinician if the tweet contained key phrases which alerted to clinical events or activities, such as "Excited to speak to residents about ethics and

telemedicine in medical careers". A user was regarded as a consumer if the tweet contained phrases signaling they had used the technology as a patient or an obvious third party, such as "Telemedicine is being offered. Have a video session next week." A user was regarded as a vendor if the tweet included phrases which suggested the user had an economic stake in promoting a product or service, such as "Dermatology Telemedicine Physician seeking Dermatologists to join". The user was considered a policymaker if a policy, governmental entity, or institutional course of action was discussed in the tweet. The user was considered "other" if the tweet was unable to be easily placed into any one category. Any case-insensitive use of the terms "covid", "covid-19", or "coronavirus" indicated a relationship to COVID-19.

The first 200 tweets were manually annotated by all three annotators, then reconciled as a group that included an expert in NLP (KR) to calibrate the annotations. The remaining 1800 tweets were double-annotated by two of the three annotators. Afterward, all disagreements regarding the classification of the tweets were further reconciled to ensure a consistent set of manual annotations. The annotator agreement with the reconciled standard was 0.78 for telehealth (Cohen's Kappa), 0.78 for COVID-19-related (Cohen's Kappa), 0.77 for user (Fleiss's Kappa), and 0.67 for sentiment (Fleiss's Kappa). All 2000 manually annotated tweets were used to train/evaluate the NLP model, as described in the next section.

#### *2.3. Automatic NLP-Based Annotation*

In order to categorize the remaining tweets, two NLP models were evaluated. Both were based on BERT, considered the most innovative model in NLP pre-training and language representation [10,11]. The first model was the standard BERT-base model without any domain-specific pre-training. The second model was BERT-base pre-trained (an unsupervised process) on the raw text of the 192,430 tweets (referred to below as BERTtelehealth). These two BERT models were fine-tuned (i.e., supervised training) on all four tasks (telehealth, sentiment, user, COVID-19) using the standard BERT TensorFlow code, resulting in a total of eight fine-tuned models. The models were evaluated by splitting the dataset in the ratio 80:20 (i.e., 4:1 train/test split); thus, 80% of the subset (1600 tweets) was used for training, and 20% of the subset (400 tweets) was used for testing. The purpose of evaluating both BERT-base and BERT-telehealth was to assess whether the additional unsupervised pre-training on this dataset improved the ability of the supervised model to make correct predictions on the four tasks.

Finally, the best-performing model for each of the four tasks (which all happened to be BERT-telehealth models, as described in the Results) was run across the full dataset of 192,430 tweets in order to analyze the overall reaction, on Twitter, to the impact of COVID-19 on telehealth.

#### **3. Results**

From a sample of tweets obtained between November 2019 and April 2020, there was a total of 192,430 tweets related to telehealth and telemedicine. Prior to the COVID-19 outbreak in November 2019, there was a relatively small and stable baseline of telehealth or telemedicine-related tweets. In the initial 4-month period sampled here, there were on average approximately 2700 telehealth-related tweets per week, with slight decreases to 2000 per week in activity during the Thanksgiving and Christmas holidays in 2019. This was followed by a rapid escalation of activity, reaching a peak of 35,625 tweets during the week of 15 March 2020—a nearly 13-fold increase (Figure 1). Given the stable, low baseline activity during the first 4 months, we defined a cutoff point as the week in which total tweets were greater than 1.5 times during the previous 4-week average, which was the week of 1 March 2020; therefore, as shown in Figure 1, *pre* is 3 November 2019 through 28 February 2020, and *post* is from 1 March through 5 April 2020. Throughout the entire period, there were no significant differences in sentiment of positive to neutral to negative from week to week.

from week to week.

activity during the first 4 months, we defined a cutoff point as the week in which total tweets were greater than 1.5 times during the previous 4-week average, which was the week of 1 March 2020; therefore, as shown in Figure 1, *pre* is 3 November 2019 through 28 February 2020, and *post* is from 1 March through 5 April 2020. Throughout the entire period, there were no significant differences in sentiment of positive to neutral to negative

**Figure 1.** Total weekly tweets and sentiment from November 2019 to April 2020. **Figure 1.** Total weekly tweets and sentiment from November 2019 to April 2020.

The NLP results for the eight models (two BERT models; four categories) are shown in Table 1. More details, including per-class performance and confusion matrices, can be found in the supplemental material (Tables S1 and S2). For all four categories, the BERT model with additional telehealth-related tweets (BERT-telehealth) outperforms the opendomain BERT model (BERT-base) by a small margin. As a result, all further analyses below were conducted using the output of the BERT-telehealth model on the corpus of 192k tweets. The NLP results for the eight models (two BERT models; four categories) are shown in Table 1. More details, including per-class performance and confusion matrices, can be found in the supplemental material (Tables S1 and S2). For all four categories, the BERT model with additional telehealth-related tweets (BERT-telehealth) outperforms the open-domain BERT model (BERT-base) by a small margin. As a result, all further analyses below were conducted using the output of the BERT-telehealth model on the corpus of 192k tweets.

**Table 1.** Evaluation of the eight fine-tuned BERT models on the test set. Note: sentiment and user are not binary, so precision/recall/F1 are macro metrics and AUROC is not applicable.  **BERT Model Accuracy Precision Recall F1 AUROC \***  Telehealth *BERT-base* 98.3% 98.5% 99.7% 99.1 0.982 *BERT-telehealth* 98.5% 98.8% 99.5% 99.2 0.989 Sentiment *BERT-base* 67.8% 63.6% 56.3% 58.8 N/A *BERT-telehealth* 70.4% 70.0% 61.7% 64.5 N/A User *BERT-base* 67.5% 53.8% 53.7% 53.7 N/A *BERT-telehealth* 69.0% 57.6% 54.7% 56.0 N/A As can also be seen from Table 1, while BERT-telehealth is very accurate at assessing whether a tweet with the telehealth keywords is in fact relevant to telehealth (98.5%), and is still quite accurate at assessing whether a tweet is related to COVID-19 (94.9%), accuracy is lower on both sentiment (70.4%) and user (69.0). Part of the reason for the lower accuracies is that, unlike telehealth and COVID-19 classifications, these are not binary classifications (sentiment has three classes, user has five) which makes the task more challenging. The larger issue, however, is the ambiguity in these tweets for these categories due to the short length of the text, which is an inherent characteristic of tweets. Since the BERT-telehealth model outperforms BERT-base in almost every case, we used this model for all subsequent analyses.

COVID-19 *BERT-base* 93.6% 91.3% 83.2% 87.1 0.940 *BERT-telehealth* 94.9% 94.5% 85.2% 89.6 0.952 \* AUROC, area under the receiver operating characteristic: evaluation metric utilized to determine the model's performance. Table 2 shows the distribution of tweets by user type, with three-fourths of the coded tweets (75.1%) classified as "other", followed by 12.9% tweets as "vendor". Clinicians represented 7.9%, tweets from consumers were 3.3%, and tweets from policymakers represented the lowest category of the users with just 0.8% of tweets (Table 2).


**Table 1.** Evaluation of the eight fine-tuned BERT models on the test set. Note: sentiment and user are not binary, so precision/recall/F1 are macro metrics and AUROC is not applicable.

\* AUROC, area under the receiver operating characteristic: evaluation metric utilized to determine the model's performance.

**Table 2.** Distribution of tweets by user type.


Table 3 shows the distribution of all user tweets by sentiment. Overall, the majority of the tweets presented either a positive (58.6%) or neutral (37.6%) sentiment, with a small remaining portion (3.8%) conveying a negative sentiment (Table 3). Tweets were redacted to protect user anonymity.

Prior to the pandemic entering Europe or the United States regions, few of the telehealth or telemedicine-related tweets referred to COVID-19 (Figure 2). Once COVID-19 became more salient due to increased awareness, COVID-19 related tweets began to increase dramatically, especially after 1 March 2020. Even as overall telehealth and telemedicine-related tweets increased, the division between COVID-19 and non-COVID-19 related tweets was roughly symmetrical, with no obvious differences in sentiment, which remained overwhelmingly positive or neutral for both categories.

**Table 3.** Distribution of tweets by sentiment.


**Figure 2.** Telehealth tweets related to COVID-19 and unrelated to COVID-19. **Figure 2.** Telehealth tweets related to COVID-19 and unrelated to COVID-19.

Given that consumers as the end-users are the ones most likely to benefit from telehealth and telemedicine, we examined user sentiment from consumers only. Although overall sentiment in telehealth-related tweets across all users was overwhelmingly positive, it would seem unlikely that, for example, vendors would express a negative sentiment on a social media platform [15]. As Figure 3 shows, consumers showed mostly positive (60.0%) or neutral (38.2%) sentiment before 1 March 2020 (pre-), and this was essentially the same post-, with 59.9% positive and 35.5% neutral. There was a slight shift from neutral to negative, with negative sentiment in telehealth-related tweets among consumers increasing from 1.8% pre- to 4.6% post-. Given that consumers as the end-users are the ones most likely to benefit from telehealth and telemedicine, we examined user sentiment from consumers only. Although overall sentiment in telehealth-related tweets across all users was overwhelmingly positive, it would seem unlikely that, for example, vendors would express a negative sentiment on a social media platform [15]. As Figure 3 shows, consumers showed mostly positive (60.0%) or neutral (38.2%) sentiment before 1 March 2020 (pre-), and this was essentially the same post-, with 59.9% positive and 35.5% neutral. There was a slight shift from neutral to negative, with negative sentiment in telehealth-related tweets among consumers increasing from 1.8% pre- to 4.6% post-.

We further analyzed the text of the 192,430 tweets for the most used bigrams and unigrams (single words) and displayed them in a word cloud visualization (Figure 4). Bigrams are pairs of adjacent words that may or may not be grammatically correct or have semantic meaning [16]. The top five unigrams by frequency from 1 to 5 were telemedicine, telehealth, coronavirus, healthcare, and patients; while the top five bigrams were telemedicine technology, telehealth services, behavioral health, healthcare provider, and remote monitoring.

*Healthcare* **2021**, *9*, x FOR PEER REVIEW 7 of 10

**Figure 3.** Consumer sentiment of tweets analyzed pre- and post- 1 March 2020. **Figure 3.** Consumer sentiment of tweets analyzed pre- and post- 1 March 2020. monitoring.

**Figure 4.** Word cloud visualization of most used telehealth-related bigrams and unigrams found in tweets between November 2019 and April 2020. **Figure 4.** Word cloud visualization of most used telehealth-related bigrams and unigrams found in tweets between November 2019 and April 2020.

#### **4. Discussion**

**Figure 4.** Word cloud visualization of most used telehealth-related bigrams and unigrams found in tweets between November 2019 and April 2020. In this study, we analyzed 192,430 publicly posted messages related to telehealth and telemedicine during a time of heightened awareness and demand for this healthcare platform. The first finding from this work is that telehealth tweets were low and stable during the 4 months (November 2019–February 2020) preceding the announcement of social distancing guidelines, followed by a dramatic increase in activity during the week of 1 March 2020. This increase correlated with news coverage of the pandemic, and in particular the announcement from the World Health Organization that COVID-19 was determined to be a pandemic [17].

Although telehealth and telemedicine were utilized for a number of years prior to the COVID-19 pandemic, interest dramatically increased as evidenced by telehealth-related tweets shortly after the onset of the pandemic. Surveys of individuals conducted during this same period also showed favorable views by healthcare providers; however, there were concerns expressed regarding patients with public versus private insurance [18]. Although it is likely that COVID-19 was the impetus for increased telehealth tweet activity, this generated two different types of interest in telehealth. The first was that people were tweeting about telehealth and COVID-19 because they were concerned about the pandemic, seeking testing, wanting to see their doctor, considering they might have COVID-19, or other information-seeking requests related to COVID-19. The other reason people may have used Twitter stemmed from the need for social distancing, implemented either voluntarily or through specific policies—including the healthcare delivery system and access to physicians, regardless of the symptoms or disease. These tweets were not directly related to COVID-19, but rather indirectly related due to the pandemic's impact on the wider healthcare system.

The next finding was that sentiment remained mostly positive, followed by neutral, with low negative ratings. The distributions of sentiment did not change significantly despite a substantial increase in volume. Thus, the pandemic increased overall interest in telehealth without changing sentiment. This has implications for those in the field of telemedicine, as patients may share openly about their experiences using social media in an unstructured way, as opposed to using structured surveys [19]. Further, a vendor seeking to gain traction in a new market or policymaker looking to draw attention to new initiatives could follow a similar approach to other social media outlets, aligning themselves with reputable, influential users with large numbers of followers or a significant volume of tweets [20].

There are limitations to using NLP as opposed to manual annotations. While using a machine-learning model is a feasible way to categorize a large number of tweets and allows for a democratization of language, there are limits to the veracity of classification. Notably, as shown in Table 1, the NLP results are far from perfect, which could effectively introduce a sample bias into the downstream analysis. The largest user category defined here is *Other*, with 75% of total tweets, which suggests that there was not enough information in the tweets to make a definitive user categorization without defaulting to more identifiable meta-data. In February–March 2020, as healthcare became more critical and less accessible during the height of the pandemic, it is possible more users who were less familiar with the platform were driven to Twitter to seek information. The inability to strategically place users into groups may suggest the classification algorithm is unable to identify relevant new users, thus increasing the classification category of "other" by the NLP pre-training model. Studies have concurred that healthcare has proven to be a challenging field for social media and sentiment analysis due to the high usage of nouns, low self-identification, and more objective speech [21,22]. These results were also seen in our bigram and unigram analysis. This means the model did not always make an accurate prediction of user type, despite the relative precision at which it was able to categorize the topic of the tweet. For example, many users may post information about a particular product that they like; however, the model may categorize these users as vendors, when in fact they are consumers discussing a product. Although consumers represented one of the smallest categories in terms of total tweets, they frequently used phrases such as "my doctor" or "my appointment" which allowed the model to make a more accurate classification as user. Lastly, it is possible that a user posted a tweet about telehealth then express an unrelated sentiment. Although the annotators were trained to catch this abnormality, the NLP model may have missed this. Therefore, there is a possibility that some of the telehealth-related tweets in this study had sentiments unrelated to telehealth.

Other research has assessed the sentiment specifically from healthcare providers and showed similar results. Tweets from providers were mostly positive but had themes specific to access to telemedicine and the safety of telemedicine as a delivery mechanism [23]. Future research could attempt to qualitatively analyze the tweets according to themes, thus providing additional data on the topics discussed. This additional category might allow for inter-category relationships to be established, such as consumers seeking specific information regarding telemedicine.

#### **5. Conclusions**

Social media platforms such as Twitter have provided a conduit for public sentiment during the COVID-19 pandemic. In this research, we analyzed a large body of tweets related to telemedicine and telehealth during the initial quarantine period before and after March 2020. Using unsupervised NLP methodologies, we were able to show that sentiment towards this delivery modality remained positive or neutral despite a significant increase in volume.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/article/10 .3390/healthcare9060634/s1, Table S1: BERT-base, Table S2: BERT-telehealth.

**Author Contributions:** Conceptualization, T.C.-L. and K.R.; methodology, K.R., Y.S., and S.M.; validation, K.R.; formal analysis, Y.S. and S.M.; writing—original draft preparation, T.C.-L. and M.W.S.; writing—review and editing, T.C.-L. and M.W.S.; visualization, T.C.-L. and M.W.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Data sharing is not applicable.

**Acknowledgments:** The authors would like to thank Jordan Godfrey-Stovall and Meghana Gudala for their assistance with the manual annotation of the telehealth tweets.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **The** *Digital Divide* **in the Era of COVID-19: An Investigation into an Important Obstacle to the Access to the** *mHealth* **by the Citizen**

**Daniele Giansanti 1,\* and Giulia Veltro <sup>2</sup>**


**Abstract:** In general, during the COVID-19 pandemic there has been a growth in the use of digital technological solutions in many sectors, from that of consumption, to *Digital Health* and in particular to *mobile health* (*mHealth*) where an important role has been played by mobile technology (*mTech*). However, this has not always happened in a uniform way. In fact, in many cases, citizens found themselves unable to take advantage of these opportunities due to the phenomenon of the *Digital Divide* (DD). It depends on multifaceted aspects ranging from the *lack of access to instrumental and network resources*, *to cultural and social barriers* and also *to possible forms of communication disability*. In the study we set ourselves the articulated goal of developing a probing methodology that addresses the problems connected to DD in a broad sense, capable of minimizing the bias of a purely electronic submission and evaluating its effectiveness and outcome. At the moment, we have submitted the survey both electronically (with an embedded solution to spread it inside the families/acquaintances) and using the wire phone. The results highlighted three polarities (a) the coherence of the two methods; (b) the outcome of the entire submission in relation to key issues (e.g., familiarity on contact tracing Apps, medical Apps, social Apps, messaging Apps, Digital-health, non-medical Apps); (c) a *Digital Divide* strongly dependent on age and in particular for the elderly is mainly evident in the use of *mTech* in general and in particular in *mHealth* applications. Future developments of the study foresee, after adequate data-mining, an in-depth study of all the aspects proposed in the survey, from those relating to access to resources, training, disability and other cultural factors.

**Keywords:** COVID-19; medical devices; *mHealth*; electronic surveys; digital health; digital divide

#### **1. Introduction**

The COVID-19 pandemic was characterized by unprecedented development and use of digital technologies. These in many cases have proved to be an important resource for accessing services while maintaining social distancing. In general, there has been a growth in the use of digital technological solutions in many sectors, from that of consumption, where the use of e-banking, digital transactions and online orders has become increasingly widespread, to *Digital Health* (DH) [1–3] and in particular to mobile health (*mHealth*) where an important role has been played by *mobile technology* (*mTech*) [4]. Citizens, in particular, found themselves receiving various offers of technological resources based on *mTech*, which in addition to the world closely linked to consumption, were concentrated in three sectors:

#### *1.1. Work, School, Social Communication*

Here, *mTech* has been useful and is currently useful to support teaching, work and relational activities in an exceptional way, allowing social distancing between subjects, such as through messaging and/or video conferencing and/or social network tools [5,6].

**Citation:** Giansanti, D.; Veltro, G. The *Digital Divide* in the Era of COVID-19: An Investigation into an Important Obstacle to the Access to the *mHealth* by the Citizen. *Healthcare* **2021**, *9*, 371. https:// doi.org/10.3390/healthcare9040371

Academic Editor: Tao-Hsin Tung

Received: 25 February 2021 Accepted: 22 March 2021 Published: 26 March 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

#### *1.2. Connection to Health Services*

Here, *mTech* has carried out and is carrying out the traditional role of *mHealth* in the field of *digital health* [1–4] by connecting citizens to the health system and providing them with highly innovative technological solutions.

#### *1.3. New Services for Epidemiological Monitoring*

Here, *mTech* has carried out and is carrying out a specific role in this pandemic and consists of providing *mHealth* solutions for controlling and monitoring the spread of the pandemic, such as through App-based solutions for the digital contact tracing [7,8].

In many cases, the simple *mTech* itself has represented a real *lifebuoy* [5,6] both for the continuation of normal activities (working and teaching) [4] and for providing a safety net.

However, this has not always happened in a uniform way. In fact, in many cases, citizens found themselves unable to take advantage of these opportunities due to the phenomenon of the *Digital Divide* (DD) [9]. It depends on multifaceted aspects ranging from the lack of access to instrumental and network resources, to cultural and social barriers [10,11] and also to possible forms of communication disability. In details the DD is mainly caused by the following problems [4].

#### *1.4. Access to Resources*

Access to the data network limited or by the availability of resources in the region or in some cases by political reasons, such as for example due to tensions between ethnic groups and/or groups belonging to different government positions within the same state.

#### *1.5. Social Factors*

Due, for example, to access difficulties in disadvantaged social categories who, even for economic reasons, cannot access these technologies.

#### *1.6. Cultural Factors*

Even within regions with full access to technologies, uneven access to technologies was found due to cultural and training barriers. Certainly, the mobile-born, for example, have experienced a better ability to adapt than even elderly teachers and elderly doctors.

#### *1.7. Disabilities*

Disabilities, such as communication disabilities, which generally represent an obstacle in a non-pandemic period to access to technologies, continued to represent an obstacle even during the COVID-19 pandemic.

With effective vaccines now available, it is appropriate at this time to have some reflections about COVID-19 in relation to the *quality of life* issues [11]. DD was included [11], along with 11 others (health inequality, gender inequality, economic disadvantage, family well-being, impact on holistic well-being, economic development versus saving lives, consumption versus environmental protection, individual rights versus collective rights, international collaboration versus conflict, prevention of negative well-being, and promotion of positive well-being), among the factors affecting the *quality of life* in the COVID-19 era. While the COVID-190 s *digital health* expansion could improve the *quality of life*, the DD could exacerbate disparities [12].

#### **2. Purpose of the Study**

At the date of writing this article, a search on Pubmed with the two keys "COVID-19" and "*Digital Divide*" has returned 47 works, ranging on the problems listed above and well identified (this is the self-upgradable link: https://pubmed.ncbi.nlm.nih.gov/?term= digital+divide+COVID-19&sort=date).

From a look at the contributions, it also emerged that the survey tool is important and useful.

At the time of writing this study, it emerged from a research on pubmed *((COVID-19) AND (Digital Divide) AND (Survey))* that the survey tool proved to be useful for investigating:


From the analysis of the previous works as a whole, it is evident that DD depends in an articulated way on various factors.

Each of these works:


In light of these considerations, we have set ourselves the following objectives:


#### **3. Materials and Methods**

At the time of the pandemic, two polling methods proved appropriate for maintaining social distance.

A first method is based on the administration of electronic surveys using websites, social networks and other multimedia tools such as electronic messaging tools with *peer to peer* dissemination techniques well-established during the pandemic.

A second method is based on the administration of surveys by telephone.

In consideration of our topic, the *digital divide*, that is the difficulty of accessing digital resources, we prepared two administration anonymous solutions:


This approach was aimed at minimizing bias.

We decided to use *Microsoft Forms (Microsof corporationt, Redmond, Washington, USA)* as an eS, as we had used it with success in many other applications [2].

Figure 1A shows the Quick Response code related to the eS with the following link. Very importantly, in the introduction (Figure 1B) of the questionnaire there is also a request to help others who are not confident in the technology to fill in the questionnaire. https://forms.office.com/Pages/ResponsePage.aspx?id=\_ccwzxZmYkutg7V0sn1 ZEvPNtNci4kVMpoVUounzQ3tUN0lXRkExQTVVUTdUOVdETURCNU9UN0czUy4u.

We decided to use *Microsoft Forms (Microsof corporationt, Redmond, Washington, USA)* 

Figure 1A shows the Quick Response code related to the eS with the following link. Very importantly, in the introduction (Figure 1B) of the questionnaire there is also a re-

**Figure 1.** (**A**) The Quick Response code of the electronic survey (eS). (**B**) Request to support people who are not confident with the technology (e.g., Grandparents, parents). **Figure 1.** (**A**) The Quick Response code of the electronic survey (eS). (**B**) Request to support people who are not confident with the technology (e.g., Grandparents, parents).

#### *3.1. Electronic Submission (ES) 3.1. Electronic Submission (ES)*

This approach was aimed at minimizing bias.

as an eS, as we had used it with success in many other applications [2].

Nci4kVMpoVUounzQ3tUN0lXRkExQTVVUTdUOVdETURCNU9UN0czUy4u.

At the moment we have submitted the eS, using social networks, Web sources and messengers, to a wide sample of 4555 citizens; among them 4512 (Table 1) agreed to participate. The participants could use their smartphone to access to the survey on the Web, on social networks and on messengers. The minimum age was 12 years; the maximum age was 85; the mean age was 49.3. The division by sex was: 2311 males 2201 females. At the moment we have submitted the eS, using social networks, Web sources and messengers, to a wide sample of 4555 citizens; among them 4512 (Table 1) agreed to participate. The participants could use their smartphone to access to the survey on the Web, on social networks and on messengers. The minimum age was 12 years; the maximum age was 85; the mean age was 49.3. The division by sex was: 2311 males 2201 females.

**Table 1.** Characteristics of the participants in the two submissions, electronic based on smartphone (*ES*) and telephonic (phone submission (*PS*)) based on a wire call. **Table 1.** Characteristics of the participants in the two submissions, electronic based on smartphone (*ES*) and telephonic (phone submission (*PS*)) based on a wire call.


#### At the moment, we have interviewed 1337 citizens; among them 1312 agreed (Table *3.2. Phone Submission (PS) Using a Wire Call*

1) to participate. The minimum age was 12 years; the maximum age was 84; the mean age was 48.9. The division by sex was: 642 males and 670 females. Both the *ES* and *PS* were conducted in Italy during the second wave of the COVID-19 pandemic from 30 November 2020 until 14 February 2021. At the moment, we have interviewed 1337 citizens; among them 1312 agreed (Table 1) to participate. The minimum age was 12 years; the maximum age was 84; the mean age was 48.9. The division by sex was: 642 males and 670 females.

*3.3. Methodological Flow*  Both the *ES* and *PS* were conducted in Italy during the second wave of the COVID-19 pandemic from 30 November 2020 until 14 February 2021.

#### The methodological approach primarily involves submitting both *ES* and *PS* surveys. *3.3. Methodological Flow*

The methodological approach primarily involves submitting both *ES* and *PS* surveys. After submission, a first important step will be based on the analysis of the two surveys on both samples to investigate any coherence or inconsistency through a robust statistical approach. This analysis will be based on some key elements, addressed in the survey, of the interaction with mobile technology in general (e.g., ability to use WhatsApp, App for social networks, generic Apps), of *mHealth* in general (medical Apps), of new *mHealth* tools (App for digital contact tracing) and digital health tools to interact with the health care processes.

The second step, after verifying the statistical coherence between the *ES* and the *PS*, will consist in focusing on the *ES* and investigating the same key elements on two subsamples of different ages (young and elderly subjects) to verify if the approach in the behavior towards technology depends on the age, however reporting a further validation of statistical significance with the corresponding subsamples of the *PS*.

We will follow two steps:


Application of the T-Student for the assessment of the significance of the difference with a *p* value lower than 0.01, when comparing the two groups different in age.

Regarding the statistical confidence interval of the investigated parameters, we set the goal of 95%.

Among the most used tests to verify if a distribution is approximate to a normal one are: The Shapiro–Wilk test, which is preferable for small samples.

The Kolmogorov–Smirnov test, which instead is used for more numerous samples. In consideration of the large amount of data, we opted for Kolmogorov–Smirnov.

#### **4. Results and Discussion**

#### *4.1. General Outcome*

The amount of data is large and further datamining will be required. Here, with aim of the article we present the outcome of the submission, comparing the two methods, *ES* and *PS*. The two methods, as shown below, report coherent and very similar results. We used the Student-*t* test to assess the statistics and fixed the lower limit to the acceptance of the H0 hypothesis (equality between averages) to *p* = 0.1.

To question 15 (Q15) (see Supplementary Materials for the questions) "Do you have one or more smartphones?" 95.545% of the participants to the ES answered yes, while 94.981% of the participants to the PS answered yes (*no significance in the differences*; *p* = 0.198; Student-*t* test). Among the owners of smartphones, we then deepened the investigation relating to subsequent questions, classified as evaluation questions with a six-level psychometric scale; it was possible therefore to assign a minimum score of one and a maximum of six with, therefore, a theoretical mean value (TMV) of 3.5. The TMV can be referred to for comparison in the analysis of the answers. An average value of the answers below TMV indicates a more negative than positive response. An average value above TMV indicates a more positive than negative response.

Figure 2 shows the degree of familiarity with social network and messaging Apps (Q16–Q17) for both *ES* and *PS*. The results are very similar (*no significance in the differences*; *p* = 0.217, *p* = 0.237; Student-*t* test). For Apps which proved to be a lifebuoy during the pandemic, for maintaining special connections and as a support, including psychological support [5,6], the TMV >3.5 indicates a *more positive than negative* response showing a familiarity degree.

Figure 3 shows the familiarity with *Immuni*, the Italian App for the digital contact tracing (DCT) (Q18). Additionally, in this case the results were very similar for the two methods (*no significance in the differences*; *p* = 0.284; Student-*t* test) for a type of App which, in a pandemic period, is of vital importance for surveillance and monitoring. However, the TMV <3.5 indicates a *more negative than positive* response showing a low familiarity degree for a very strategic App.

*Healthcare* **2021**, *9*, 371 6 of 16

**Figure 2.** Coherence in the degree of familiarity with social networks, Q16 (*p* = 0.217) and messaging Apps, Q17 (*p* = 0.237) both for the ES and the PS. The graphic indicates a high coherence between the two groups. **Figure 2.** Coherence in the degree of familiarity with social networks, Q16 (*p* = 0.217) and messaging Apps, Q17 (*p* = 0.237) both for the ES and the PS. The graphic indicates a high coherence between the two groups. in a pandemic period, is of vital importance for surveillance and monitoring. However, the TMV <3.5 indicates a *more negative than positive* response showing a low familiarity degree for a very strategic App.

**Figure 3.** Coherence in the degree of familiarity with the App Immuni for digital contact tracing (DCT) both for the *ES* and the PS (*p* = 0.284*).* The graphic indicates a high coherence between the two groups. **Figure 3.** Coherence in the degree of familiarity with the App Immuni for digital contact tracing (DCT) both for the *ES* and the PS (*p* = 0.284). The graphic indicates a high coherence between the two groups.

For question Q19 on the familiarity of the other types of App there was a coherence For question Q19 on the familiarity of the other types of App there was a coherence in the results (*no significance in the differences*; *p* = 0.301; Student-*t* test), indicating a familiarity just above the threshold TMV (Figure 4).

**Figure 3.** Coherence in the degree of familiarity with the App Immuni for digital contact tracing (DCT) both for the *ES* and the PS (*p* = 0.284*).* The graphic indicates a high coherence between the two groups. in the results (*no significance in the differences*; *p* = 0.301; Student-*t* test), indicating a familiarity just above the threshold TMV (Figure 4). Figure 5 reports the specific outcome with respect to aspects of digital health and with reference to *mHealth* (*no significance in the differences*; *p* = 0.333, *p* = 0.291; Student-*t* test) for both the ES and PS. The answers to the two questions indicate a value around the threshold for both the questions, with a value just below for the first question mainly related to the *mHealth* and just above for the other.

For question Q19 on the familiarity of the other types of App there was a coherence

in the results (*no significance in the differences*; *p* = 0.301; Student-*t* test), indicating a famili-

*Healthcare* **2021**, *9*, 371 7 of 16

**Figure 4.** Coherence in the degree of familiarity with other Apps (non-medical Apps) both for the ES and the PS. (*p* = 0.301). The graphic indicates a high coherence between the two groups. **Figure 4.** Coherence in the degree of familiarity with other Apps (non-medical Apps) both for the ES and the PS. (*p* = 0.301). The graphic indicates a high coherence between the two groups. threshold for both the questions, with a value just below for the first question mainly related to the *mHealth* and just above for the other.

**Figure 5.** Coherence in the degree of familiarity assessment with *mHealth* (*p* = 0.333) processes and the Digital Health (*p* = 0.291) both for the *ES* and *PS. The graphic indicates a high coherence between the two groups.*  **Figure 5.** Coherence in the degree of familiarity assessment with *mHealth* (*p* = 0.333) processes and the Digital Health (*p* = 0.291) both for the *ES* and *PS. The graphic indicates a high coherence between the two groups.*

Surely what emerges from the analysis conducted through the graphs and the statis-Surely what emerges from the analysis conducted through the graphs and the statistics referred to from time to time is important and can be highlighted with the following.

**Figure 5.** Coherence in the degree of familiarity assessment with *mHealth* (*p* = 0.333) processes and the Digital Health (*p* = 0.291) both for the *ES* and *PS. The graphic indicates a high coherence between the two groups.*  tics referred to from time to time is important and can be highlighted with the following. Firstly, from a general point of view, the two methodologies *ES* and *PS* had comparable performances as evidenced by the significance statistics. This was not taken for granted given that it is notorious that the methods of administration based on digital tech-Firstly, from a general point of view, the two methodologies *ES* and *PS* had comparable performances as evidenced by the significance statistics. This was not taken for granted given that it is notorious that the methods of administration based on digital technologies present bias towards subjects affected by *Digital Divide*.

Surely what emerges from the analysis conducted through the graphs and the statisnologies present bias towards subjects affected by *Digital Divide*. The result certainly depends on the invitation present in the electronic survey for The result certainly depends on the invitation present in the electronic survey for widespread dissemination and support in the compilation of those less accustomed to digital technology.

tics referred to from time to time is important and can be highlighted with the following. Firstly, from a general point of view, the two methodologies *ES* and *PS* had comparable performances as evidenced by the significance statistics. This was not taken for widespread dissemination and support in the compilation of those less accustomed to digital technology. This certainly leads us to highlight, in the case of *ES*, the presence of a sort of solidarity between subjects with a greater degree of ability in the digital towards those with a lower degree.

granted given that it is notorious that the methods of administration based on digital tech-

widespread dissemination and support in the compilation of those less accustomed to

digital technology.

Specifically, if we consider the trend across the sample, we see how the following emerges in detail:


On the one hand, the great role of mobile technology must be recognized without a doubt, as confirmed in the survey; on the other, it should be highlighted that, despite the strategic importance of the Apps in healthcare in this period, whether it is an App for digital contact tracing [7,8] for epidemiological monitoring, or an App for *mHealth* in general [4], the familiarity remains low.

This is certainly an important point on which to act strongly to reduce the Digital Divide in citizens of all the ages.

#### *4.2. In-Depth Study in Two Sub-Samples: Young People and the Elderly against the Digital Divide*

An important question to answer in this pandemic period is that of the generational relationship with *digital technology*. In other words, it is important to analyze how the DD acts between different generations. In this period, we have been bombarded by electronic surveys on various aspects related to the pandemic arriving through social networks, all surveys certainly affected by *bias*, as they did not adequately take into account the DD which also depends on familiarity with *digital technology* that the older ones do not have. With our two-way approach we tried to minimize this *bias*.

To obtain a first idea of DD based on age, we compared two samples in particular in two age groups in the *ES*:


Many questions were asked using a graded psychometric scale (1 = minimum grade; 6 = maximum grade).

#### *4.3. Intergroup Comparison*

We report here the results of the *ES* for two very different age groups to analyze this question. We used the Student-*t* test to assess the statistics and fixed the higher limit to the acceptance of the H1 hypothesis (difference between averages) to *p* = 0.01. The comparison relative to the degree of familiarity of each parameter was made by comparing the corresponding parameter for young and elderly, and this was done parameter by parameter.

To question 15 (Q15) (see Supplementary Materials for the questions) "Do you have one or more smartphones?"100% of the *young* subjects answered yes, while only 64.3% of the elderly answered yes (*high significance in the differences*, *p* = 0.009, Student-*t* test).

Among the owners of smartphones, we then deepened the investigation with the subsequent questions.

Figure 6 shows the degree of familiarity with social network and messaging apps (Q16–Q17) for young people and the elderly, from which there emerges a decidedly lower use by the elderly (*high significance in the differences*; *p* = 0.008, *p* = 0.007; Student-*t* test) of these Apps which proved to be a *lifebuoy* during the pandemic, for maintaining special connections and as a support, including as a psychological support [5,6]. Figure 6 shows the degree of familiarity with social network and messaging apps (Q16–Q17) for young people and the elderly, from which there emerges a decidedly lower use by the elderly (*high significance in the differences*; *p* = 0.008, *p* = 0.007; Student-*t* test) of these Apps which proved to be a *lifebuoy* during the pandemic, for maintaining special connections and as a support, including as a psychological support [5,6].

To question 15 (Q15) (see Supplementary Materials for the questions) "Do you have one or more smartphones?"100% of the *young* subjects answered yes, while only 64.3% of the elderly answered yes (*high significance in the differences*, *p* = 0.009, Student-*t* test).

Among the owners of smartphones, we then deepened the investigation with the

*Healthcare* **2021**, *9*, 371 9 of 16

subsequent questions.

**Figure 6.** Difference in the degree of familiarity with social networks, Q16 and messaging apps, Q17 for young people and the elderly; the graphic indicates a high difference (*p* = 0.008, *p* = 0.007) between the two groups of young \* and elderly. **Figure 6.** Difference in the degree of familiarity with social networks, Q16 and messaging apps, Q17 for young people and the elderly; the graphic indicates a high difference (*p* = 0.008, *p* = 0.007) between the two groups of young \* and elderly.

Figure 7 shows familiarity with Immuni, an App for digital contact tracing (DCT) (Q18). Additionally, in this case the *young* people showed greater familiarity (*high significance in the differences*; *p* = 0.008, Student-*t* test) with a type of App which in a pandemic period is of vital importance for surveillance and monitoring. Additionally, to question Q19 on the familiarity of the other types of App there was a greater familiarity (*high significance in the differences*, *p* = 0.008, Student-*t* test) for *young* people than for the *elderly* Figure 7 shows familiarity with Immuni, an App for digital contact tracing (DCT) (Q18). Additionally, in this case the *young* people showed greater familiarity (*high significance in the differences*; *p* = 0.008, Student-*t* test) with a type of App which in a pandemic period is of vital importance for surveillance and monitoring. Additionally, to question Q19 on the familiarity of the other types of App there was a greater familiarity (*high significance in the differences*, *p* = 0.008, Student-*t* test) for *young* people than for the *elderly* (Figure 8).

(Figure 8). Figure 9 reports the specific outcome with respect to aspects of digital health and to *mHealth*. From the answers relating to this battery of questions, it is evident that the DD of the elderly specifically related to the *Digital Health* appears very clearly (*high significance*  Figure 9 reports the specific outcome with respect to aspects of digital health and to *mHealth*. From the answers relating to this battery of questions, it is evident that the DD of the elderly specifically related to the *Digital Health* appears very clearly (*high significance in the difference*; *p* = 0.008, *p* = 0.009; Student-*t* test).

*in the difference*; *p* = 0.008, *p* = 0.009; Student-*t* test). The confidence interval of each parameter was always ≥95%.

The significance of the results was also validated through a further comparison of statistical significance between *ES* and *PS* with reference to these two subsamples. The answer to each question between *ES* and *PS* showed no significance in the differences (*no significance in the differences*, *p* lower limit of acceptance of H0, fixed to 0.1; Student-*t* test).

Table 2 reports the two trends of the answers for both the groups. It shows a clear trend that, even for young people, there is a significant decrease in using or familiarity with Immuni, *mHealth*, and Digital Health. This trend is the same for elderly people. However, the elderly from what emerges in the table have a different tendency in dealing with these Apps and technologies. It is therefore important to analyze the differences in detail.

*Healthcare* **2021**, *9*, 371 10 of 16

**Figure 7.** Difference in the degree of familiarity with the App Immuni for DCT for young people **Figure 7.** Difference in the degree of familiarity with the App Immuni for DCT for young people and the elderly. The graphic indicates a high difference (*p* = 0.008) between the two groups of young \* and elderly people. young \* and elderly people.

**Figure 8.** Differences in the degree of familiarity with other Apps (non-medical Apps) for young **Figure 8.** Differences in the degree of familiarity with other Apps (non-medical Apps) for young people and the elderly. The graphic indicates a high difference (*p* = 0.008) between the two groups of young \* and elderly people.

of young \* and elderly people.

of young \* and elderly people.

**Figure 8.** Differences in the degree of familiarity with other Apps (non-medical Apps) for young people and the elderly. The graphic indicates a high difference (*p* = 0.008) between the two groups

people and the elderly. The graphic indicates a high difference (*p* = 0.008) between the two groups

. **Figure 9.** Differences in the degree of familiarity with *mHealth* (*p* = 0.008) processes and the *Digital*  **Figure 9.** Differences in the degree of familiarity with *mHealth* (*p* = 0.008) processes and the *Digital Health* (*p* = 0.009) for young \* people and the elderly.


**Table 2.** Trends in the two groups.

*Health* (*p* = 0.009) for young \* people and the elderly.

in the rating (equal to the 9.166%).

with this essential App in the pandemic period.

*significance in the differences*, *p* lower limit of acceptance of H0, fixed to 0.1; Student-*t* test). Table 2 reports the two trends of the answers for both the groups. It shows a clear trend that, even for young people, there is a significant decrease in using or familiarity with Immuni, *mHealth*, and Digital Health. This trend is the same for elderly people. However, the elderly from what emerges in the table have a different tendency in dealing with these Apps and technologies. It is therefore important to analyze the differences in detail. If we compare in detail the two groups, it can be observed that with regard to the use of mobile technology in general, while (a) in the use of the App for messaging and social media the average difference in the score is two (*p* = 0.007, Student *t* test), this difference (b) drops to 1.2 in the case of generic Apps (*p* = 0.008 Student *t* test), both due to a greater use of the latter by the elderly, with an average increase of 0.25 in the rating (equal to the 4.166%) and to a lower use of the latter by young people with an average decrease of 0.55 in the rating (equal to the 9.166%).

**Table 2.** Trends in the two groups.  **Social Media Messaging Immuni Non-Medical Apps** *mHealth* **Digital Health Young** 5.9 5.8 4.8 5.3 4.4 4.2 The difference (c) between the two groups in the use of the App for digital contact tracing is 1.9 as an average value and similar to the difference regarding the use of social and messaging Apps (*p* = 0.008 Student *t* test). However, in both groups there is an average decrease in the score, which in the elderly is equal to 0.95 (equal to he 15.83%) and in the young is equal to 1.05 (equal to 17.5%) clearly indicating a sharp decrease in familiarity with this essential App in the pandemic period.

**Elderly** 3.9 3.8 2,9 4.1 3.2 3.5 If we compare in detail the two groups, it can be observed that with regard to the use of mobile technology in general, while (a) in the use of the App for messaging and social media the average difference in the score is two (*p* = 0.007, Student *t* test), this difference (b) drops to 1.2 in the case of generic Apps (*p* = 0.008 Student *t* test), both due to a greater If we look at the differences in behavior between the two groups in the use of apps for *mHealth* (d), we realize that between the two groups here the difference is narrower than the first comparison and is equal to 1.2 on average (*p* = 0.008; Studenti *t* test). This depends on a different way of interacting; in fact, young people with respect to the use of social and messaging apps show a very strong and a marked decrease in the degree of familiarity equal to 1.45 on average (equal to 24.1%); the elderly also show a decrease in familiarity, but this decrease is less and equal to 0.65 (equal to 10.83%).

use of the latter by the elderly, with an average increase of 0.25 in the rating (equal to the 4.166%) and to a lower use of the latter by young people with an average decrease of 0.55 The difference between the two groups narrows further if we make the same comparison with the response relating to Digital Health.

and messaging Apps (*p* = 0.008 Student *t* test). However, in both groups there is an average decrease in the score, which in the elderly is equal to 0.95 (equal to he 15.83%) and in the young is equal to 1.05 (equal to 17.5%) clearly indicating a sharp decrease in familiarity

The difference (c) between the two groups in the use of the App for digital contact

If we look at the differences in behavior between the two groups in the use of apps for *mHealth* (d), we realize that between the two groups here the difference is narrower

Here, (e) the difference is only 0.7 as a mean value (*p* = 0.008; Student T test). The familiarity of young people with digital health processes drops by 1.65 on average (equal to 27.5%) if compared to that relating to social apps and WhatsApp. The elderly, on the other hand, making the same comparison lose less; in fact, the decrease recorded by them is 0.35 (equal to 5.83%).

What emerges from this analysis conducted through the comparison between the groups carried out with the aid of the previous graphs, Table 2 and the statistics applied from time to time is the following:


#### **5. Conclusions**

#### *5.1. Why the Need of a Study on the Digital Divide*

The COVID-19 pandemic has created an unprecedented impetus for the development of *mHealth* [1–4]. This development involved both the enhancement and standardization of already consolidated solutions in *digital health* and the exploration of new potentials [2].

Numerous initiatives have been seen aimed at strengthening familiarity with digital technologies.

Some initiatives have also been based on the development of surveys to thoroughly analyze the causes of the *Digital Divide* [13–17].

Other initiatives have focused on improvement initiatives on populations and/or ethnic groups that were disadvantaged from the start [18,19]. One thing is certain, the citizen during the pandemic and even subsequently will be increasingly called to interact with these technologies and above all with *mHealth*.

Self-care has proven to be a robust tool in times of the COVID-19 pandemic.

Important diseases such as diabetes [20] or cardiac diseases [21] benefit greatly from *mHealth* initiatives that rely on self-care and remote monitoring.

Even psychology and psychiatry, spurred on by new needs, have adapted during the pandemic to remote methodologies [22–24].

Further examples can be seen, without resulting in a review, in relation to new directions of digital health that are significantly increased at the moment, such as teleophthalmology [25] or totally new ones such as digital contact tracing [8,9].

There is no doubt that in the use of non-pharmaceutical technologies [26] will also significantly depend on overcoming cultural barriers.

It will be necessary to invest a lot of energy, also taking into account that the vaccination processes themselves, the most important resource of the moment, rely heavily on digital solutions, and that those who are already familiar with such technologies starting from multimedia tools [27] will find themselves able to cope better at this time.

Following this reasoning, there is no doubt that, in many cases, the simple *mTech* itself has represented a *real lifebuoy* [5,6] both for the continuation of normal activities (working and teaching) and for providing a safety net.

However, this has not always happened in a uniform way. The *digital divide* was a cause of this [9–17]. Making a map and an investigation of these aspects, is important both to consolidate experiences and to improve the diffusion of the medical technologies and their fruition. This depends on many factors ranging from difficulties in accessing instrumental resources to cultural and pathological problems (such as disabilities) [4]. Some studies have faced this using the surveys [13–17] focusing on particular factors and/or particular problems. An investigation of how these factors and problems act together is particularly important, and it is even more so in a pandemic period and was aimed in this study.

#### *5.2. What Has Been Proposed in the Study to Investigate the Digital Divide*

A study on this issue conducted with purely multimedia technologies would pre-sent many limitations and above all it would risk excluding an important part of subjects affected by DD through bias.

To specifically address such an important problem, we prepared an articulated study and designed a survey that reached the largest number of subjects with an approach dedicated to the purpose. In particular we designed a survey that was submitted through two channels.

The first channel was the electronic one and included a strong invitation to intra-family and intra-relationship sharing with the support of those with less familiarity with digital technologies.

The second channel was based on interview contacts via the public wire telephone network.

#### *5.3. What Are the Highlights from the Study?*

The outcome of the study has five polarities.

A first polarity consists in having designed a methodology that allows the investigation of different aspects connected with the *Digital Divide* and that at the same time allows the estimation the possible impact of bias in wide-ranging surveys.

The second polarity consists of having verified, through an appropriate statistical approach, the consistency of the results of the two submission procedures (*ES* and *PS*). This indirectly showed us that the solutions made in the *ES* relating to an encouragement to share the survey with those less accustomed to digital technology have had a positive effect and that there was a sort of *intra-digital-cultural* solidarity.

A third polarity relative to the two *ES* and *PS* samples is a clear trend found in the sample. In particular, both in *ES* and *ES* with coherence it has been highlighted:


The fourth polarity consists of a comparison between two subsamples of different ages to investigate the possible different degree of impact of the *Digital Divide* and therefore to verify if this characteristic depends on age. At the Istituto Superiore di Sanità, before the pandemic, we focused a lot on the problems of young people's interaction with smartphones in general and therefore with apps and the Internet [29,30] and we have seen how this also leads to neuromuscular problems such as text neck [31,32] and psychological ones such as addiction [33]. However, we must note that this study seems to show us that in the face of this the young person with his ability to interact with these tools has a low

degree of *Digital Divide* and an ability to interact with many Apps that during the pandemic have allowed and are allowing minimization of the sense of loneliness and the consequent related psychological problems.

Both an *intragroup* and an *intergroup* analysis were conducted on the two subsamples of different ages that highlighted:


The fifth polarity that indirectly emerges from the study are the obvious suggestions for stake holders in relation to actions to:


Certainly, information, training and inter-generational sharing of knowledge will be fundamental.

#### *5.4. What Are the Added Values of the Study?*

From a general point of view the study presents four added values.

The first added value is the product (Supplementary 1) represented by the electronic survey tool that can be easily submitted through the *mTech* on the net during the pandemic.

The second added value is represented by the survey which addresses in addition to the aspects of resources, such as those of the network [13], also other types of broadspectrum problems.

The third added value is represented by the possibility of using this product, after minimal changes even in non-pandemic/post-pandemic periods.

The fourth added value is represented by the outcome with reference to: (a) the two methods *ES*, *PS* in the case of all the samples; (b) the two sub-samples in two age groups in the case of the *ES*, the first sample related to *young people* (aged 15–25 years), the second sample related to *elderly people* (aged 65–75 years). This outcome highlighted a DD strongly dependent on age and in particular a DD for the elderly particularly evident in the use of *mTech* in general and, in particular, in *mHealth* applications.

#### *5.5. What the Study Supports and the Further Initiatives*

From a general point of view, this article supports the initiatives that aim to reduce the *digital divide*, identifying the causes and bringing out potential solutions in a structured way, so that they can be submitted to stakeholders for a targeted and articulated approach.

Future developments of the study foresee, after adequate data-mining, an in-depth study of all the aspects proposed in the survey (Supplementary 1), from those relating to access to resources (e.g., the connection) up to those related to the training, disability and other cultural factors.

**Supplementary Materials:** The electronic survey is available online at https://www.mdpi.com/ article/10.3390/healthcare9040371/s1.

**Author Contributions:** Author Contributions: Conceptualization, D.G.; methodology, D.G.; software, D.G. and G.V.; validation, D.G. and G.V.; formal analysis, D.G. and G.V.; investigation, D.G. and G.V.; resources, D.G. and G.V.; data curation, D.G. and G.V.; writing—original draft preparation, D.G.; writ-ing—review and editing, D.G.; visualization, D.G. and G.V.; supervision, D.G.; project admin-istration, D.G. Both authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The survey is available as a Supplementary Materials.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Healthcare* Editorial Office E-mail: healthcare@mdpi.com www.mdpi.com/journal/healthcare

MDPI St. Alban-Anlage 66 4052 Basel Switzerland

Tel: +41 61 683 77 34

www.mdpi.com

ISBN 978-3-0365-4793-0