**eHealth in Chronic Diseases**

Editors

**Irene Torres-Sanchez Marie Carmen Valenza**

MDPI • Basel • Beijing • Wuhan • Barcelona • Belgrade • Manchester • Tokyo • Cluj • Tianjin

*Editors* Irene Torres-Sanchez University of Granada Spain

Marie Carmen Valenza University of Granada Spain

*Editorial Office* MDPI St. Alban-Anlage 66 4052 Basel, Switzerland

This is a reprint of articles from the Special Issue published online in the open access journal *International Journal of Environmental Research and Public Health* (ISSN 1660-4601) (available at: https: //www.mdpi.com/journal/ijerph/special issues/ehealth diseases).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. *Journal Name* **Year**, *Volume Number*, Page Range.

**ISBN 978-3-0365-2902-8 (Hbk) ISBN 978-3-0365-2903-5 (PDF)**

© 2022 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.

## **Contents**

**Felix Muehlensiepen, ¨ Sandra Kurkowski, Martin Krusche, Johanna Mucke, Robert Prill, Martin Heinze, Martin Welcker, Hendrik Schulze-Koops, Nicolas Vuillerme, (FPSH4DIFUU and Johannes Knitza** Digital Health Transition in Rheumatology: A Qualitative Study Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 2636, doi:10.3390/ijerph18052636 . . **1**

### **Jennifer M. Bingham, Jennifer Stanislaw, Terri Warholak, Nicole Scovis, David R. Axon, Jacques Turgeon and Srujitha Marupuru**

Assessment of Glycosylated Hemoglobin Outcomes Following an Enhanced Medication Therapy Management Service via Telehealth

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 6560, doi:10.3390/ijerph18126560 . . **13**

**Honoria Ocagli, Giulia Lorenzoni, Corrado Lanera, Alessandro Schiavo, Livio D'Angelo, Alessandro Di Liberti, Laura Besola, Giorgia Cibin, Matteo Martinato, Danila Azzolina, Augusto D'Onofrio, Giuseppe Tarantini, Gino Gerosa, Ester Cabianca and Dario Gregori** Monitoring Patients Reported Outcomes after Valve Replacement Using Wearable Devices: Insights on Feasibility and Capability Study: Feasibility Results Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 7171, doi:10.3390/ijerph18137171 . . **21**

#### **Rose Calixte, Sumaiya Islam, Zainab Toteh Osakwe, Argelis Rivera and Marlene Camacho-Rivera**

Pattern of Use of Electronic Health Record (EHR) among the Chronically Ill: A Health Information National Trend Survey (HINTS) Analysis Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 7254, doi:10.3390/ijerph18147254 . . **35**

#### **Andr´ea Gomes Martins Gaspar, Pedro Escada and Lu´ıs Velez Lap˜ao**

How Can We Develop an Efficient eHealth Service for Provision of Care for Elderly People with Balance Disorders and Risk of Falling? A Mixed Methods Study Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 7410, doi:10.3390/ijerph18147410 . . **49**

#### **Abdullah H. Alsharif**

Cross Sectional E-Health Evaluation Study for Telemedicine and M-Health Approaches in Monitoring COVID-19 Patients with Chronic Obstructive Pulmonary Disease (COPD) Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 8513, doi:10.3390/ijerph18168513 . . **67**

#### **Daniela Laranja Gomes Rodrigues, Gisele Silvestre Belber, Igor da Costa Borysow, Marcos Aurelio Maeyama and Ana Paula Neves Marques de Pinho**

Description of e-Health Initiatives to Reduce Chronic Non-Communicable Disease Burden on Brazilian Health System

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 10218, doi:10.3390/ijerph181910218 . **83**

#### **Cynthia Lamper, Ivan Huijnen, Maria de Mooij, Alb`ere K ¨oke, Jeanine Verbunt and Mari ¨elle Kroese**

An eCoach-Pain for Patients with Chronic Musculoskeletal Pain in Interdisciplinary Primary Care: A Feasibility Study

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 11661, doi:10.3390/ijerph182111661 . **95**

#### **Aria Zand, Audrey Nguyen, Courtney Reynolds, Ariela Khandadash, Eric Esrailian and Daniel Hommes**

Patient Experience and Satisfaction with an e-Health Care Management Application for Inflammatory Bowel Diseases

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 11747, doi:10.3390/ijerph182211747 . **115**

#### **Kieren J. Egan, Kathryn A. McMillan, Marilyn Lennon, Lisa McCann and Roma Maguire**

Building a Research Roadmap for Caregiver Innovation: Findings from a Multi-Stakeholder Consultation and Evaluation

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 12291, doi:10.3390/ijerph182312291 . **127**

#### **Brian M. Green, Casey A. Hribar, Sara Hayes, Amrita Bhowmick and Leslie Beth Herbert**

Come for Information, Stay for Support: Harnessing the Power of Online Health Communities for Social Connectedness during the COVID-19 Pandemic

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 12743, doi:10.3390/ijerph182312743 . **145**

#### **Qinliang Liu and Xiaojing Li**

The Interactions of Media Use, Obesity, and Suboptimal Health Status: A Nationwide Time-Trend Study in China

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 13214, doi:10.3390/ijerph182413214 . **157**

### **Saeed M. Alghamdi, Ahmed M. Al Rajah, Yousef S. Aldabayan, Abdulelah M. Aldhahir, Jaber S. Alqahtani and Abdulaziz A. Alzahrani**

Chronic Obstructive Pulmonary Disease Patients' Acceptance in E-Health Clinical Trials Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 5230, doi:10.3390/ijerph18105230 . . **171**

#### **Woo-Seok Choi, Nam-Suk Kim, Ah-Young Kim and Hyung-Soo Woo**

Nurse-Coordinated Blood Pressure Telemonitoring for Urban Hypertensive Patients: A Systematic Review and Meta-Analysis

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 6892, doi:10.3390/ijerph18136892 . . **183**

#### **Katherine Silang, Hangsel Sanguino, Pooja R. Sohal, Charlie Rioux, Hyoun S. Kim and Lianne M. Tomfohr-Madsen**

eHealth Interventions to Treat Substance Use in Pregnancy: A Systematic Review and Meta-Analysis

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 9952, doi:10.3390/ijerph18199952 . . **207**

#### **Jim Wiegel, Bart Seppen, Marike van der Leeden, Martin van der Esch, Ralph de Vries and Wouter Bos**

Adherence to Telemonitoring by Electronic Patient-Reported Outcome Measures in Patients with Chronic Diseases: A Systematic Review

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 10161, doi:10.3390/ijerph181910161 . **225**

#### **Chiara Mack, Yannik Terhorst, Mirjam Stephan, Harald Baumeister, Michael Stach, Eva-Maria Messner, J ¨urgen Bengel and Lasse B. Sander**

"Help in a Heartbeat?": A Systematic Evaluation of Mobile Health Applications (Apps) for Coronary Heart Disease

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 10323, doi:10.3390/ijerph181910323 . **239**

#### **Sang-Hyun Park, Jong-Ho Shin, Joowoong Park and Woo-Seok Choi**

An Updated Meta-Analysis of Remote Blood Pressure Monitoring in Urban-Dwelling Patients with Hypertension

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 10583, doi:10.3390/ijerph182010583 . **255**

#### **Beatriz Brea-G ´omez, Irene Torres-S ´anchez, Araceli Ortiz-Rubio, Andr ´es Calvache-Mateo, Irene Cabrera-Martos, Laura L ´opez-L ´opez and Marie Carmen Valenza**

Virtual Reality in the Treatment of Adults with Chronic Low Back Pain: A Systematic Review and Meta-Analysis of Randomized Clinical Trials

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 11806, doi:10.3390/ijerph182211806 . **293**

#### **Felix Holl, Jennifer Kircher, Walter J. Swoboda and Johannes Schobel**

Methods Used to Evaluate mHealth Applications for Cardiovascular Disease: A Quasi-Systematic Scoping Review

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 12315, doi:10.3390/ijerph182312315 . **325**

### **Marco Cascella, Franco Marinangeli, Alessandro Vittori, Cristina Scala, Massimo Piccinini, Alessandro Braga, Luca Miceli and Renato Vellucci**

Open Issues and Practical Suggestions for Telemedicine in Chronic Pain Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 12416, doi:10.3390/ijerph182312416 . **341**

### **Andr ´es Calvache-Mateo, Laura L ´opez-L ´opez, Alejandro Heredia-Ciur ´o,Javier Mart´ın-Nu´ nez, ˜ Janet Rodr´ıguez-Torres, Araceli Ortiz-Rubio and Marie Carmen Valenza**

Efficacy of Web-Based Supportive Interventions in Quality of Life in COPD Patients, a Systematic Review and Meta-Analysis

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 12692, doi:10.3390/ijerph182312692 . **351**

#### **Samuel Bonet Olivencia, Arjun H. Rao, Alec Smith and Farzan Sasangohar**

Eliciting Requirements for a Diabetes Self-Management Application for Underserved Populations: A Multi-Stakeholder Analysis Reprinted from: *Int. J. Environ. Res. Public Health* **2022**, *19*, 127, doi:10.3390/ijerph19010127 . . . **371**

### *Article* **Digital Health Transition in Rheumatology: A Qualitative Study**

**Felix Mühlensiepen 1,2,\*, Sandra Kurkowski 3, Martin Krusche 4, Johanna Mucke 5, Robert Prill 2,6, Martin Heinze 1,7, Martin Welcker 8, Hendrik Schulze-Koops 9, Nicolas Vuillerme 10,11,12, Georg Schett 13,14 and Johannes Knitza 10,13,14**


**Abstract:** The global COVID-19 pandemic has led to drastic changes in the management of patients with rheumatic diseases. Due to the imminent risk of infection, monitoring intervals of rheumatic patients have prolonged. The aim of this study is to present insights from patients, rheumatologists, and digital product developers on the ongoing digital health transition in rheumatology. A qualitative and participatory semi-structured fishbowl approach was conducted to gain detailed insights from a total of 476 participants. The main findings show that digital health and remote care are generally welcomed by the participants. Five key themes emerged from the qualitative content analysis: (1) digital rheumatology use cases, (2) user descriptions, (3) adaptation to different environments of rheumatology care, and (4) potentials of and (5) barriers to digital rheumatology implementation. Codes were scaled by positive and negative ratings as well as on micro, meso, and macro levels. A main recommendation resulting from the insights is that both patients and rheumatologists need more information and education to successfully implement digital health tools into clinical routine.

**Keywords:** rheumatology; chronic disease; digital health; eHealth; telemedicine; remote care; patient perspective; qualitative research; fishbowl discussion; content analysis

**Citation:** Mühlensiepen, F.;

Kurkowski, S.; Krusche, M.; Mucke, J.; Prill, R.; Heinze, M.; Welcker, M.; Schulze-Koops, H.; Vuillerme, N.; Schett, G.; et al. Digital Health Transition in Rheumatology: A Qualitative Study. *Int. J. Environ. Res. Public Health* **2021**, *18*, 2636. https:// doi.org/10.3390/ijerph18052636

Academic Editor: Irene Torres-Sanchez

Received: 20 January 2021 Accepted: 2 March 2021 Published: 5 March 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

#### **1. Introduction**

COVID-19 has led to drastic changes in the management of patients with rheumatic diseases. Due to the imminent risk of infection, monitoring intervals of rheumatic patients have prolonged [1,2]. Some were cancelled or postponed, whereas others were replaced by video consultations [1,3], thereby catalyzing digital disruption in healthcare. Besides the COVID-19-induced uptake of digital health [4], the Digital Health Act, passed in December 2019, reshapes the German healthcare system [5]. Measures include the introduction of digital health applications into the statutory health insurance scheme, the implementation of digital patient records, and the promotion of video consultation hours [3,5]. Due to aggravating challenges, such as the declining number of rheumatologists [6], the aging population, and the need for early diagnosis [7] and continuous monitoring [8], rheumatology is considered to have a great potential to benefit from a digital health transition [2,9].

In a response to the current challenges and uptake of digitization, a virtual fishbowl discussion was organized at the annual scientific conference of the German Society of Rheumatology (DGRh Congress), which took place in September 2020. Patients, rheumatologists, and industry stakeholders had the opportunity to exchange information and experiences on digital health in rheumatology. The aim of this study is to present insights from patients, rheumatologists, and digital product developers on the ongoing digital health transition and innovation potentials in rheumatology in Germany.

#### **2. Materials and Methods**

The authors conducted a virtual fishbowl discussion [10] on the question "How does the internet affect the doctor–patient relationship?" at the first virtual annual conference of the German Society for Rheumatology 2020 (9–12 September 2020). The event was scheduled for 90 min. It consisted of two introductory key notes on digital health services in rheumatology and the actual fishbowl discussion. The discussion was recorded, transcribed verbatim, and later examined using qualitative content analysis [11].

Recently, the fishbowl technique has been successfully implemented in national rheumatology conferences [12]. It is a validated method, fostering open group discussions and engagement of all audience members [13]. By including an expert panel (inner circle) and one empty chair for an alternating audience member, the technique promotes a dynamic and direct exchange with the audience. The authors intended to hold a faceto-face fishbowl discussion at DGRh Congress 2020. Due to the COVID-19 pandemic, the entire conference was held virtually, using Zoom™-based software (Zoom Video Communications Inc, San Jose, CA, USA) [14]. Hence, the authors successfully tested the fishbowl discussion as an online format (Figure 1).

**Figure 1.** Virtual fishbowl discussion (in accordance with Muehlensiepen et al. [10]).

The inner circle of the fishbowl discussion consisted of one patient representative, two rheumatologists, and two digital product developers. To contribute to the discussion, they used the "hand raise" function to communicate with the moderator (F.M.). Audience members (the outer circle) who wanted to take part in the discussion and take the empty seat in the inner circle had to report this interest in the chat and were registered in the speaker list. When it was their turn, the participants' cameras and microphones were activated by tech support. After their statement, the participants left the inner circle of the discussion again.

As the fishbowl was part of the official congress program, delegates could also participate in the discussion without taking part in the study: either by not contributing verbally or by indicating that their contribution would be deleted from the recording. However, delegates did not use this option.

Analysis was based on qualitative content analysis by Philip Mayring [11]. The fishbowl discussion was audio-recorded and subsequently anonymized and transcribed verbatim. The transcript was imported into MAXQDA 2020 Software (VERBI GmbH, Berlin, Germany) [15]. The data analysis involved four steps (Figure 2).

**Figure 2.** Qualitative content analysis: iterative coding process.

Step 1: Two scientists with a background in healthcare science (F.M., male, PhD) and social science (S.K., female, M.Sc.) independently examined the material line by line, coded inductively with regard to the content (thematic coding) and formulated as closely to the notes as possible. Additionally, codes were scaled in accordance with Kurkowski et al. [16]. Formal and scaled codes were assigned simultaneously and deductively to matching text sections. Positive and negative connotations were differentiated (scaling). Codes were scaled to the category "positive rating" if they contained clearly positive expressions (e.g., "ideal," "great opportunity"), whereas the codes were scaled as "negative" if they expressed a clearly negative connotation (e.g., "we do not wish," "a negative example"). If codes could not be assigned to a positive or negative scaling, they were not considered in the further analysis. Step 2, after approximately 70% of the material was coded, the two scientists (F.M. and S.K.) exchanged their codes and compared their codes and assignments. After this, they defined the main categories and subsumed relating codes. The main categories represent the central themes and aspects of the discussion: (1) digital rheumatology use cases, (2) user descriptions, (3) adaptation to different environments of rheumatology care, and (4) potentials of and (5) barriers to digital rheumatology implementation (Figure 3). The categories, potentials (4), and barriers (5) of digital rheumatology implementation were grouped into micro, meso, and macro levels. Codes were assigned to the macro level if they related to the German health system. The meso level referred to rheumatology care as a whole. The micro level concerned the individual patient–practitioner level. Step 3: After reworking the code system, one scientist (F.M.) coded the remaining notes. Step 4: S.K. performed an additional consistency check, and inconsistencies were resolved. To ensure validity of the data analysis, the results of the analysis were reviewed and confirmed by participants of the fishbowl discussion (M.W., H.S.K., and J.K.) before drafting the

manuscript. For the presentation of the results, representative quotes of the discussion transcript were selected, translated, and included in the text. The manuscript has been compiled in accordance with the Consolidated Criteria for Reporting Qualitative Research (COREQ) [17] (please refer to Supplementary Materials 1).


**Figure 3.** Qualitative content analysis: main categories (according to Kurkowski et al. 2020 [16]).

Ethical approval was not sought for the present study as this manuscript reports on the results of a discussion held during a virtual medical conference. Despite this, all authors declared to adhere to the Declaration of Helsinki in its current form in all steps of this project [18]. Discussion participants were informed of the audio recording, transcription, and subsequent analysis of the discussion. In the course of transcription, all data and personal references were removed. There was no risk involved in participating in the discussion. The manuscript does not allow identification of individuals.

#### **3. Results**

A total of 476 delegates attended the 90-min session. The actual fishbowl discussion lasted 56 min. In total, there were 19 content contributions by 12 delegates: 9 rheumatologists, 2 digital product developers, and 1 patient representative. Seven of the 19 contributions came from audience members from the outer circle (37%).

#### *3.1. Digital Rheumatology Use Cases*

Delegates discussed various examples of how digital health could complement rheumatology care. Implementation of electronic health records in Germany was considered to have great potential for boosting doctor–patient relationships as well as tracking previous medical care, resulting in more efficient use of scarce time:

*"Electronic patient records—the next big thing. I see this as an extremely great opportunity to strengthen the doctor–patient relationship and this precious therapy time of five minutes and so on." (Patient representative I)*

Participants pointed out the importance of implementing a digital national health portal where patients could access validated and evidence-based information about their medical conditions:

*"I consider the concept as a chance to provide sensible content to patients, to people, in a centralized way." (Rheumatologist I)*

The introduction of healthcare apps into German Social Health Insurance system is expected to have great potential and a positive impact on rheumatology care as well:

*"I think this extension and supplementation of care through digital health apps, DiGAs, will be a nice, good, viable addition." (Digital product developer I)*

*"On the one hand sensitivity and specificity [of diagnosis apps] is very poor at 50 percent. But on the other hand it's still 20 percent higher than the referral of the non-experienced general practitioner, with a specificity of about 30 percent. That means there's a clear potential for improvement without any physical examination, as you said, with the methods of [name of a certain digital health application] or [name of another application] or other diagnostic apps. But the power has already been demonstrated." (Rheumatologist II)*

Delegates also discussed negative examples of digital healthcare in chronic diseases. Approaches were seen as negative when they are disconnected from the "regular" health system and digital healthcare is not controlled by medical practitioners:

*"But that's what you read now in magazines again and again in a striking way: A patient has foam in her urine, feels somehow bad, weary, and so on, goes to the convenience store, somehow hands in her sample and then receives her results digitally, oh, she knows it's lupus? That is, I think, not exactly the good start to a chronic patient's career and good care that we actually want." (Patient representative I)*

Additionally, delegates identified insecure sources and misinformation on diseases as a danger of digital health. Furthermore, they expressed their opposition to invalidated digital health applications without proof of benefit. Interestingly, video consultations were not a main discussion topic.

#### *3.2. User Descriptions*

Discussants identified patients with a stable course of their chronic rheumatic disease as potential users of digital health tools:

*"And, for a well-cared for or well-adjusted patient like me, a doctor's contact is really necessary only once a year for therapy monitoring reasons, and if there is something beyond that, like maybe checking whether my blood pressure needs to be adjusted, talking to the doctor once in a while, that would of course be ideal to cover the perhaps necessary two to three doctor's contacts a year once briefly via a secure telemedicine platform." (Patient representative I)*

Digital health can also significantly help considering rare diseases:

*"Particularly in the area of rare diseases or unusual symptoms, patients also do research on the internet if they do their research well, find good sources or do research at organizations or patient self-help organizations. Patients who use swarm intelligence can also arrive more quickly at more or less target-oriented diagnostic suggestions. That is a knowledge gain overall." (Patient representative I)*

Among physicians, early adopters were mentioned as potential users of digital health. Less technophile rheumatologists are not likely to use digital health very soon.

*"My rheumatologist is like that; she tells me that she won't invest in any digital infrastructure here and I know five, six, seven rheumatologists who are of the same age." (Patient representative I)*

According to the patient representative, monetary incentives or even sanctions for practitioners who decline to connect to digital infrastructure would be strategies to spread digitization in rheumatology care. According to the delegates, knowledge and digital health literacy are low among both practitioners and patients.

#### *3.3. Adaptation to Different Environments of Rheumatology Care*

Throughout the fishbowl discussion, participants agreed that digital health can complement current rheumatology care, but not replace face-to-face appointments. Digital health tools should be directed by medical practitioners. Diagnosis as well as important therapy decisions should be carried out by rheumatologists after a thorough clinical examination. Diagnosis, exclusively based on digital health, was strongly rejected by the discussants:

*"Initial diagnosis is only possible by the rheumatologist in direct physical contact and after detailed clinical examination. And then later on, you can talk about how the patient can be followed up by the internet or video consultation or anything else. That's my opinion at least." (Rheumatologist III)*

*"Hey, a qualified initial diagnosis should please, please, please be made by a qualified rheumatologist. I think that is the consensus here in the panel." (Patient representative I)*

However, on the other side, participants mentioned different environments of rheumatology care, where digital healthcare could contribute. These are medical history, follow-up appointments, and disease monitoring:

*"Many patients have no idea what kind of medication they are taking, what kind of therapy they actually want, and I often spend most of my time gathering all this information. Electronic patient records could be a huge relief and I think that it is the key to speed up digitization." (Rheumatologist II)*

*"Yes, I am well-adjusted and my CRP is fine. But with new digital tools, for example the app that [name of a digital health application] is going to launch in the area of disease monitoring or something like [name of another application]* ... *new solutions for disease monitoring are entering the market, which will contribute to maintaining the ability to work." (Patient representative I)*

Participants of the fishbowl discussion also highlighted the importance of digital approaches to ensure adequate and safe rheumatology care during the COVID-19 pandemic. They believe that secure and validated information on the internet can help people access adequate rheumatology care. Most importantly, this applies to the therapy of rare diseases.

#### *3.4. Potentials of Digital Rheumatology Implementation*

In regard to the healthcare system (macro level), the discussants highlighted the German Digital Health Act, which is anticipated to contribute to a digital transition of healthcare in the coming years:

*"It has been passed and eight months later the first products are about to be approved. I think that is the speed of light for the health system and shows again how the legislator wants to support this transition." (Digital product developer I)*

To increase the knowledge of medical practitioners and patients with regard to digital health services, the discussants call for education and information campaigns:

*"It can only be done with education and information campaigns. How should patients and practitioners receive the information, that there is the possibility of prescribing apps* ... *and which ones are available* ... *and how good they are?" (Digital product developer II)*

Despite this momentum, the benefits of digital health applications still have to be proven:

*"Digitization is not positive in itself, but it might be positive in its effect and this effect still has to be proven, in other words, evaluated. And in the digitization law and specifically also for the introduction of apps, there is still a necessity to prove the benefit." (Rheumatologist I)*

According to the participants, a broader spectrum of rheumatology care services (meso level) will be available through digital transition. This provides the opportunity to effectively allocate the scarce human resources available in rheumatology care. The effective use of digital health in rheumatology can be supported by the innovative medical professional associations and active patient organizations:

*"How do the individual specialist areas position themselves in regard to digitalization and apps? I think that the rheumatologists—to praise them once again—are pretty much at the forefront here, as is the group of Young Rheumatologists and the Digital Commission. Ultimately, I consider it very state of the art to deal with the topic in this way." (Digital product developer II)*

In addition, digital health was also seen as a way to enhance the attractiveness of rheumatology as a profession in general:

*"I think this is precisely where digital aids could offer a very exciting aspect and expansion: To the extent that the job description of a rheumatologist becomes much more interesting and attractive when one knows that one can deal with digital, innovative tools in the treatment of patients." (Digital product developer I)*

On the micro level, respectively, individual level, patients and physicians could benefit from an improved accessibility to rheumatology care and the reduction of "unnecessary" appointments:

*"Because I also think it might be quite pleasant for patients to do without one or two unneeded doctor's appointments." (Rheumatologist III)*

*"As long as things are fine, I don't need to travel through the city to a non-accessible doctor's office, anyways." (Patient representative I)*

Since much of the relevant information can be provided digitally in advance, the digital transition could increase consultation time:

*"Digitization allows us to have more time to talk. That is the quality of care, which is also increasing, because if I have already answered all the questions asked beforehand, then I have maybe five more minutes in the conversation with the patient, and that is an essential five minutes of conversation to improve the perceived care." (Rheumatologist I)*

#### *3.5. Barriers to Digital Rheumatology Implementation*

At the macro level, the discussants identified limited digital infrastructure and equipment as barriers to digital health transformation in Germany:

*"In many parts of society, that not only include special circumstances, but also students and the general population, from patients to doctors, who simply do not have the technical equipment. And I don't even want to mention the 5G network, which is also not available in Germany." (Rheumatologist IV)*

On the meso level, the discussants pointed out that the current remuneration systems in rheumatology care are not designed for digital approaches and does not incentivize them:

*"* ... *also the remuneration of digital services. And I think this is the challenge right now, especially in rheumatology* ... *set it up digitally and get away from 'well, I only want to see the patients who also bring me the money, but I try to care for the patients according to their needs with digital support'." (Digital product developer I)*

This results in a barrier on the micro level at the same time. Practitioners could suffer financial losses through the use of digital health. Furthermore, digital health could lead to a wealth gap on the part of patients, which results in health opportunities being distributed unequally, a digital divide in healthcare:

*"We also have to consider that digitalization creates a wealth gap: people who cannot afford large contracts, good mobile phones, good tablets, do not have good access. And this is also evident in telemedicine and applications, where these systems are not being used. Thus, I believe that we also have to consider the social aspect." (Rheumatologist I)*

Finally, the confusing numbers of available digital tools, limited individual knowledge and interest concerning digital health represent a barrier to digital transition in daily rheumatology practice:

*"These apps that we are supposed to recommend—to pick up on this—I personally have a very, very hard time with them in my day-to-day practice. Surprisingly, not a single patient has asked me for an app so far. So, all this information is not yet there and then I once again ask you: Why has no patient asked for it yet?" (Rheumatologist V)*

#### **4. Discussion**

We performed a qualitative study using a virtual fishbowl discussion on digital transition in rheumatology with 476 participants at the first virtual conference of German Society for Rheumatology 2020. The qualitative approach provides a unique insight on the status quo of digital rheumatology from the perspectives of patient representatives, rheumatologists, and digital product developers.

The participants revealed a positive view toward digital transition. They identified digital health as a valuable addition to current rheumatology care, mentioning various use cases that could support existing services: electronic patient records, digital health applications (diagnosis assistance/monitoring), and implementation of a health information portal. The participants of the fishbowl discussion rejected digital approaches if these were decoupled from the traditional healthcare system. Final diagnosis and therapeutic decisions should exclusively be made by rheumatologists. Digital health should be integrated into rheumatology care routines but must not entirely eradicate personal patient–doctor interaction [2]. Digital rheumatology is considered as supportive regarding anamnesis, follow-up appointments, monitoring, and (validated) information on services and diseases. The discussants highlighted patients with stable disease courses as potential telemedicine users. This is in line with the research by Kulcsar et al. [19]. The importance of digitally provided information for patients with rare diseases was stressed. The accelerating and connecting effect of digitization—"digital crowdsourcing"—to inform patients is consistent with previous results by Krusche et al. [20] and Ruffer et al. [21]. On the other hand, with regard to the medical practitioners, less technophile doctors would not be likely to adopt digital health in their clinical routines, according to the discussants. Similarly, low eHealth literacy and skepticism was observed among older rheumatic patients [22]. Overall, knowledge on digital health is still very limited among practitioners and patients according to the discussants. Previous survey data [23,24] confirm this lack of knowledge as a main barrier. It also appears that in particular highly affected patient groups are significantly less interested in eHealth [25]. On the macro level, the participants in the fishbowl discussion pointed out that German Digital Health Care Act as well as education and information campaigns could foster digital transition. The weak digital infrastructure and equipment in Germany is a barrier from the discussants' perspectives. Regarding rheumatology care as a whole (meso level), digital transition could support effective allocation of scarce rheumatology workforce and enhance the attractiveness of rheumatology as a profession. It has been demonstrated that telemedicine-based monitoring is not clinically inferior to standard follow-up [26]. Furthermore, it has been demonstrated that digitally complemented lectures increase students' interest in rheumatology [27]. The participants identified inadequate remuneration systems and a lack of incentives as a potential barrier to digital health on the meso level. This underlines the high relevance of innovative and sustainable funding systems for large-scale implementation of eHealth [28]. On the individual patient–practitioner level (micro level), digital health could improve accessibility of rheumatology care services, increase individual consultation time, and at the same time enable the care of more patients overall by time savings in clinical practice. This is in line with recent qualitative findings that point to the potential of digital health contributing to patient-centered care in rheumatology [29]. The opportunities of digital health in rheumatology care are contrasted by possible financial losses among practitioners and limited knowledge about digital health as barriers. Finally, the influence of the digital transition

on social disparities and the allocation of health opportunities is controversially discussed and needs to be intensively observed [30–32].

Study findings could have direct implications for informing guidelines on remote care in rheumatology and can help to successfully implement digital health tools in the management of rheumatic diseases. A main strength of this study is the inclusion of potential users of digital health and in particular patient representatives. To the best of our knowledge, this is the first qualitative study in rheumatology that included the three stakeholders: patients, rheumatologists, and digital product developers. We also highly recommend the virtual fishbowl format [10], as it is an outstanding option for multi-perspective, interactive discussions and, during COVID-19, a solid alternative to physical face-to-face debates. Compared to other discussion formats, i.e., panel discussions or large debates, fishbowl discussions encourage active participation of non-experts and enable the inclusion of multiple perspectives [12].

The study design has several limitations. The study reports on the content of one 54 min discussion session. Because conference delegates chose to participate in the discussion based on their interest in the topic, a self-selection bias and a rather positive attitude toward digital health is likely to exist. The two introductory key notes on digital services in rheumatology might have positively biased the subsequent fishbowl and our results. Although fishbowl is an inclusive, participatory discussion technique, not all attendees participated equally in the discussion. The discussion was held at a virtual medical conference including mainly rheumatologists. The patient perspective is underrepresented, as mostly physicians expressed their views and only one patient representative actively took part in the discussion.

#### **5. Conclusions**

German patients, rheumatologists, and digital health developers expressed a generally positive attitude toward the digital health transition in rheumatology. It could contribute to effective allocation of scarce resources, improve accessibility, and enhance the attractiveness of rheumatology as a profession. Digital health may certainly complement rheumatology care, but (final) diagnosis and key decisions are to remain in the hands of rheumatologists, according to the participants of the fishbowl discussion. Patients and rheumatologists need more information and education to successfully implement digital health tools into clinical routine.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/1660-460 1/18/5/2636/s1: Table S1: COREQ Checklist.

**Author Contributions:** Study conception and design, F.M., J.M., M.W., H.S.-K., and J.K.; acquisition of data, F.M., M.K., J.M., M.W., H.S.-K., and J.K.; analysis and interpretation of data, F.M., S.K., M.K., R.P., M.H., M.W., N.V., H.S.-K., G.S., and J.K. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Ethical approval was not sought for the present study because the manuscript describes findings of a discussion at a virtual medical conferences. There was no risk involved in participating in the discussion. The study was conducted according to the guidelines of the Declaration of Helsinki.

**Informed Consent Statement:** Discussion participants were informed of the audio recording, transcription, and subsequent analysis of the discussion. In the course of transcription, all data and personal references were removed. The manuscript does not allow identification of individuals.

**Data Availability Statement:** All data relevant to the study are included in the article. For further questions regarding the reuse of data, please contact the corresponding author (felix.muehlensiepen@ mhb-fontane.de).

**Acknowledgments:** The authors would like to thank all participants for their valuable contributions, as well as the organizers of the DGRh Congress 2020 and the technical support. The present work is part of the PhD thesis of the last author, J.K. (AGEIS, Université Grenoble Alpes, Grenoble, France).

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Assessment of Glycosylated Hemoglobin Outcomes Following an Enhanced Medication Therapy Management Service via Telehealth**

**Jennifer M. Bingham 1,2,\*, Jennifer Stanislaw 2, Terri Warholak 2, Nicole Scovis 1, David R. Axon 2, Jacques Turgeon 3,4 and Srujitha Marupuru <sup>2</sup>**


**Citation:** Bingham, J.M.; Stanislaw, J.; Warholak, T.; Scovis, N.; Axon, D.R.; Turgeon, J.; Marupuru, S. Assessment of Glycosylated Hemoglobin Outcomes Following an Enhanced Medication Therapy Management Service via Telehealth. *Int. J. Environ. Res. Public Health* **2021**, *18*, 6560. https://doi.org/10.3390/ ijerph18126560

Academic Editors: Irene Torres-Sanchez and Marie Carmen Valenza

Received: 19 May 2021 Accepted: 17 June 2021 Published: 18 June 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

**Abstract:** (1) Background: Regular contact with a medication therapy management (MTM) pharmacist is shown to improve patients' understanding of their condition; however, continued demonstration of the value of a pharmacist delivered comprehensive medication review (CMR) using enhanced MTM services via telehealth is needed. The study aimed to describe a pilot program designed to improve type 2 diabetes mellitus (T2DM) management through enhanced condition specific MTM services. (2) Methods: This retrospective study included patients with T2DM aged 40–75 years who received a pharmacist-delivered CMR between January and December 2018. An evaluation of glycosylated hemoglobin (HbA1c) values 3 months pre- and post-CMR was performed. Wilcoxon signed-rank and chi-square tests were used. (3) Results: Of 444 eligible patients, a majority were female (58%) with a median age of 70 years. Median HbA1c values post-CMR were lower than pre-CMR (median 7.1% range 4.5–13.6; median 7.4% range 4.5–13.9, respectively; *p* = 0.009). There were fewer participants with HbA1c >9% post-CMR (*n* = 66) than pre-CMR (*n* = 80; *p* < 0.001) and more with HbA1C <6.5% post-CMR (*n* = 151) than pre-CMR (*n* = 130; *p* < 0.001). (4) Conclusion: This program evaluation highlighted the value of an enhanced condition specific MTM service via telehealth. Patients had improved HbA1c values three months after receiving a single pharmacist delivered CMR.

**Keywords:** type 2 diabetes; pharmacist; glycosylated hemoglobin; medication therapy management; comprehensive medication review; telehealth; T2DM

#### **1. Introduction**

Medication therapy management (MTM) services aim to optimize medication use, reduce medication-related problems, and reduce overall healthcare costs. One component of MTM services is an annual comprehensive medication review (CMR), which may be conducted by pharmacists or other qualified providers in a variety of ways, including in-person at provider offices and community pharmacies, or via the telephone [1,2]. It is known that MTM pharmacists play an integral role in assessing the patient's understanding of his/her conditions and therapeutic regimens through regular contact and accessibility [3]. Yet, patients do not typically receive regular follow-up contact from MTM pharmacists beyond the annual CMR encounter.

Evidence linking pharmacist involvement to improved clinical outcomes and patient empowerment has led to the expansion of the pharmacist's role in chronic disease management for several chronic conditions [4–9]. An example of this is in the management

of type 2 diabetes [10]. For example, one study found that, for patients with type 2 diabetes, pharmacist delivered care significantly improved their ability to self-manage their condition, as seen by decreased glycosylated hemoglobin (HbA1c) clinical values 6 months post intervention [11]. Research also shows that through MTM counseling on lifestyle modifications (e.g., physical activity, nutrition), disease state management, and medication adherence, pharmacists make an important contribution to the diabetes care team [12]. Hence, MTM services are a crucial component of ensuring optimal outcomes in diabetes care given their ability to improve medication safety and effectiveness.

Although previous work has demonstrated the economic, clinical, and humanistic benefits of pharmacist delivered MTM services [8–11], an enhanced, condition specific MTM program for patients with type 2 diabetes mellitus (T2DM) has not yet been evaluated in the literature. A pilot, condition specific, pharmacist delivered MTM program was therefore implemented by a national MTM provider to empower self-management of tT2DM, optimize medication regimens, address gaps in care continuity, improve glycemic control, reduce risks of diabetes-related complications, and improve chronic condition management for patients with T2DM. Patients were identified by their respective health plan based on eligibility criteria set forth by the Centers for Medicare and Medicaid Services and then contacted by the national MTM provider. The purpose of this paper is to describe, within the context of MTM, a pilot program designed to improve T2DM management through enhanced condition specific MTM services integrated into an annual pharmacist delivered CMR service.

#### **2. Materials and Methods**

#### *2.1. Description of Program*

The CMR included an interactive, systematic assessment of patient-specific health information and medications to identify and resolve drug therapy problems and to improve health outcomes. The CMR was conducted using an audio-only telehealth (i.e., telephone call) application. Quality checks were performed on CMRs conducted by the pharmacists to ensure consistency and standardization.

First, the medication list was reconciled within the software program (RxCompanion™, Tucson, AZ, USA) and medication names, strengths, doses, routes, frequencies, and indications were recorded and assessed for drug-drug interactions, medication safety concerns, and appropriateness using telehealth. Next, patient allergies and reactions were recorded and assessed for contraindications. Pharmacists at the call centers were assisted by proprietary software that raised alerts for medication non-adherence, therapeutic duplications, and missing guideline-directed therapy.

Next, the pharmacist conducted a condition specific review via telephone. If the patient was diagnosed with T2DM, pharmacists were expected to address each of the following through teach-back education: (1) fasting blood sugars; (2) patient understanding of how to identify and manage hypoglycemia and hyperglycemia; (3) patient knowledge of how their most recent HbA1c compared to the previous value; (4) whether their HbA1C was uncontrolled based on provider recommendations; and (5) a review of results with selfmonitoring of blood glucose. The pharmacist used these findings to have meaningful conversations about lifestyle factors to improve blood glucose management and address specific times of the day that were problematic for the patient.

Based on the patient response, the pharmacist provided teach-back education and an individualized written summary to empower self-management of T2DM. The patient was mailed a personalized medication list in the Centers for Medicare & Medicaid Services standardized CMR format upon completion of the CMR. The pharmacist directly contacted the provider via telephone or facsimile if they identified patient issues related to access to medication, therapeutic recommendations warranting a dosage change. The pharmacist referred the patient to their provider for further diabetes management and/or suggested a diabetes educator or dietician as needed.

#### *2.2. Description of Sites*

The MTM service provider offered an annual standardized CMR to qualified patients based on benefits eligibility. The MTM service provider started providing services via telehealth in 2006. Five national MTM call centers employed by the MTM service provider were included in the pilot program and presented a suite of MTM services to meet the performance needs of health plans and patients, mainly through pharmacist-delivered telehealth medication reviews. The centers consisted of a team dedicated to improving health, wellness, and chronic disease management through MTM services adopting an interprofessional team model that included: pharmacists, pharmacy technicians, student pharmacists, pharmacy residents, nursing students, and registered nurses.

#### *2.3. Study Population*

Patients were included in this program evaluation if they received an annual CMR in 2018 from a pharmacist employed by the MTM service provider. Additional patient criteria included: age 40 to 75 years; diagnosis of T2DM denoted by international classification of disease (ICD)-10 coding (e.g., E11.0–E11.9); and, had a presence of one or more oral antihyperglycemic prescriptions based on claims data. The Centers for Medicare & Medicaid Services (CMS) assess certain oral antihyperglycemics in performance metrics to analyze medication adherence for diabetes medications, which include: biguanides, sulfonylureas, thiazolidinediones, dipeptidyl peptidase-4 (DPP) inhibitors, incretin mimetics, meglitinides, or sodium-glucose co-transporter-2 (SGLT2) inhibitors. Medicare beneficiaries who only use insulin are not included in this CMS metric, and were subsequently excluded from the study. Additionally, patients who were 40 years of age or younger and older than 75 years were excluded from the study.

#### *2.4. Data Collection and Analysis*

This program evaluation used retrospective patient data from one MTM service provider between 1 January 2018 and 31 December 2018. The variables of interest included: (1) age; (2) gender; (3) race/ethnicity; (4) number of medications; (5) HbA1c three months pre-CMR; and (6) HbA1c three months post-CMR. Health marker data was provided by health plan to the national MTM provider. A pre-post study design was used. For nominal data, a chi-square test was used. For internal level data with a non-normal distribution, a Wilcoxon signed-rank test was used. Descriptive data were also calculated. All tests used an a-priori α level of 0.05. Data analysis was conducted using IBM SPSS Statistics v2015 (IBM Corp, Armonk, NY, USA). This program evaluation was approved by the institutional review board (No. 1911128095).

#### **3. Results**

A total of 444 patients were included in the program evaluation, of which 264 were female and 180 male. The median age of patients was 70 years of age (range 40–75). The median number of medications per patient was 21 (range 9–60). Additional baseline patient demographic information is outlined in Table 1.

Median HbA1c values post-CMR were lower than pre-CMR scores (median 7.1% range 4.5–13.6 and median 7.4% range 4.5–13.9, respectively; Z = −2.60, *p* = 0.009). An HbA1c threshold of greater than 9% was used to identify patients at highest risk for microvascular and macrovascular complications, as well as mortality. There were fewer participants with HbA1c >9% post-CMR (*n* = 66) compared to pre-CMR (*n* = 80; *p* < 0.001), and there were more participants with HbA1C <6.5% post-CMR (*n* = 151) compared to pre-CMR (*n* = 130; *p* < 0.001). A subgroup analysis was conducted to assess the impact in adults and the elderly. Of 444 patients, 104 (23%) were adults and 340 (75%) were older adults. Additional data concerning the number and percent of patients in each HbA1c strata preand post-CMR are in featured in Table 2.


**Table 1.** Baseline characteristics for participants (*n* = 444).

**Table 2.** Comparison of HbA1c values pre-comprehensive medication review (CMR) intervention and post-CMR intervention.


HbA1c: glycosylated hemoglobin; \* HbA1c data had a skewed distribution, hence median and range were reported. Differences between the pre- and post-groups were calculated using the Wilcoxon signed-rank test given that the data were dependent (i.e., the same people were included in the pre- and post-groups).

#### **4. Discussion**

Our study highlights the effect of an enhanced condition specific MTM service designed to improve HbA1C control in patients with T2DM via telehealth. The most important finding was the significant difference between median HbA1c values pre- and post-CMR. This provides preliminary evidence to support the value of an enhanced diabetes specific MTM service in preventing the risk of microvascular and macrovascular complications associated with an HbA1c >9% [13]. The evidence also supports the value in assisting patients to achieve a desired HbA1c <6.5%, especially those who have a longer remaining life expectancy, fewer comorbidities, and a lower risk of hypoglycemia. Hence, we demonstrated in this study that the telehealth approach, by itself, had a significant impact on HbA1C control and represents another complimentary option to the use of medical devices, biosensors, or smart phone applications to support patients in their own homes.

MTM services are a crucial component of ensuring optimal outcomes in diabetes care [14]. The MTM pharmacist plays an integral role in assessing the patient's understanding of his/her condition and therapeutic regimens through regular contact and accessibility [3]. One study found that, for patients with T2DM, pharmacist-delivered information significantly decreased HbA1c values and improved patient knowledge [11]. Another study found improved indicators following pharmacist provided MTM services [15]. Anderson et al. also found improved T2DM health markers following telehealth MTM pharmacist services [8]. In addition to managing T2DM, MTM service provider pharmacists also help reduce cardiovascular related complications and morbidity. Research shows that through MTM counseling on lifestyle modifications, disease state management, and medication adherence, pharmacists make an important contribution to the diabetes care team [12].

It is well known that preventative care is one of the most efficient ways to maintain patient health and to reduce healthcare costs [16]. Moczygemba et al. found that telephonic MTM programs reduced both medication- and health-related problems [10]. Thus, CMRs via telehealth can be thought of as preventative healthcare. The results of this study suggest that pharmacists can provide preventative care that may help to address unmet patient needs stemming from primary care provider shortages in the United States.

Telephonic CMRs are an effective method to enhance patient care and to reduce healthcare expenditures because they can optimize medication therapy and empower patients [16]. Another benefit of telephonic CMRs is that they can provide access to pharmacists who speak the patient's preferred language, thus increasing culturally responsive patient care [17]. This is applicable because the MTM service provider in our study provides CMRs via telehealth to patients in over 30 languages. An additional advantage of telephonic CMRs is improved ease for patients to reconcile medication lists while remaining in their residences, rather than transporting pill bottles to another site and unintentionally forgetting one at home [1]. Future research should evaluate the relative association of each of these components on patient outcomes.

The results of our pilot program further suggest a role for MTM pharmacists in providing preventative care that may help to address unmet patient needs stemming from primary care provider shortages in the United States. One novel aspect of this program was the use of a telephonic condition specific review designed to assist patients with poorly controlled T2DM. The condition review allowed the pharmacist to discover patient specific challenges in glucose management and offered customized solutions based on their HbA1c and fasting blood glucose goals. In addition, the condition reviews were an effective method to enhance patient care by optimizing medication therapy and empowering patients. These results support the need for pharmacists to fill gaps in care subsequent to national provider shortages in the United States of America [16].

The successes of the pilot program provides lessons learned to other MTM providers aiming to improve quality measures for patients with T2DM. These findings parallel other studies that have shown pharmacist-delivered medication counseling and healthcare education to be valuable in improving clinical values [17].

#### *Limitations*

This pilot evaluation study design prohibited the establishment of a causal relationship between the CMR and diabetes outcomes, as it did not control for length of the CMR, class of antihyperglycemic medication, specific changes to the medication list, patientspecific responses to the condition review, other comorbidities, or whether the patient

sought services from their provider during the three-month time post-CMR. Hence, the study was not able to capture the clinical significance, nor the cost-effectiveness of the outcomes observed as there were limitations in data available to the research team. The study also did not capture longitudinal effects on clinical outcomes beyond three months post-CMR. In addition, the large proportion of patients with unknown race and ethnicity limits generalizability. Thus, these pilot program findings are not generalizable to all Medicare beneficiaries. In future studies, it would be prudent to collate data from the telehealth application regarding the length of the consultation. It is also suggested that researchers collect specifics from the software program on the type of medication and compare differences in the list pre- and post-intervention. One final suggestion is to integrate a complimentary follow-up consultation with the patient to assess for diabetes management from other healthcare providers.

#### **5. Conclusions**

This program evaluation highlights the value of pharmacist delivered CMR telehealth services aimed to decrease HbA1C values in patients with T2DM. Future research should evaluate the impact in a more diverse population in other countries over a longer time to determine the cost effectiveness and clinical significance of the intervention.

**Author Contributions:** Methodology, J.S., J.M.B., T.W. and N.S.; validation, J.S., D.R.A. and S.M.; formal analysis, J.S. and S.M.; investigation, J.S.; resources, J.S.; writing—original draft preparation, J.S., J.M.B. and S.M.; writing—review and editing, J.S., J.M.B., D.R.A., T.W., N.S., S.M. and J.T.; supervision, J.M.B.; project administration, J.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Ethical review and approval were waived for this study, due to human research exemption.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data are not publicly available due to proprietary concerns.

**Acknowledgments:** The authors would like to acknowledge Orsula V. Knowlton, Dana M. Filippoli, and Ann M. Taylor for their contributions to the project.

**Conflicts of Interest:** Terri Warholak received funding from Arizona Department of Health Services, Merck and Co., Pharmacy Quality Alliance and Tabula Rasa HealthCare Group. Jennifer M. Bingham and Nicole Scovis have disclosed an outside interest in Tabula Rasa HealthCare Group. Conflicts of interest resulting from this interest are being managed by The University of Arizona in accordance with its policies. David R. Axon received funding from the American Association of Colleges of Pharmacy, Arizona Department of Health Services, Merck and Co., Pharmacy Quality Alliance, and Tabula Rasa HealthCare Group. Srujitha Marupuru received funding from Tabula Rasa HealthCare Group and Merck and Co. The other authors did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors for this study.

#### **References**


## *Article* **Monitoring Patients Reported Outcomes after Valve Replacement Using Wearable Devices: Insights on Feasibility and Capability Study: Feasibility Results**

**Honoria Ocagli 1, Giulia Lorenzoni 1, Corrado Lanera 1, Alessandro Schiavo 2, Livio D'Angelo 2, Alessandro Di Liberti 2, Laura Besola 3, Giorgia Cibin 4, Matteo Martinato 1, Danila Azzolina 1,5, Augusto D'Onofrio 4, Giuseppe Tarantini 2, Gino Gerosa 4, Ester Cabianca <sup>6</sup> and Dario Gregori 1,\***


**Abstract:** Wearable devices (WDs) can objectively assess patient-reported outcomes (PROMs) in clinical trials. In this study, the feasibility and acceptability of using commercial WDs in elderly patients undergoing transcatheter aortic valve replacement (TAVR) or surgical aortic valve replacement (SAVR) will be explored. This is a prospective observational study. Participants were trained to use a WD and a smartphone to collect data on their physical activity, rest heart rate and number of hours of sleep. Validated questionnaires were also used to evaluate these outcomes. A technology acceptance questionnaire was used at the end of the follow up. In our participants an overall good compliance in wearing the device (75.1% vs. 79.8%, SAVR vs. TAVR) was assessed. Half of the patients were willing to continue using the device. Perceived ease of use is one of the domains that scored higher in the technology acceptance questionnaire. In this study we observed that the use of a WD is accepted in our frail population for an extended period. Even though commercial WDs are not tailored for clinical research, they can produce useful information on patient behavior, especially when coordinated with intervention tailored to the single patient.

**Keywords:** surgical aortic valve replacement; transcatheter aortic valve replacement; physical function; wearable devices; feasibility

### **1. Introduction**

Use of consumer wearable devices (WDs) is increasing both in daily usage and in clinical trials [1]. Fuller et al. [1], in a recent systematic review, showed that various brands of WDs were used in clinical studies, offering accurate measures for steps and heart rate.

The International Data Corporation (IDC) reported a Year-Over-Year Growth of 35.1% in 2020 in the global market of WDs [2]. In clinical setting, the introduction of WDs has brought new challenges to face. At first, the European Medicines Agency did not release any specific guidance addressing the use of WDs. While still recognizing the importance

**Citation:** Ocagli, H.; Lorenzoni, G.; Lanera, C.; Schiavo, A.; D'Angelo, L.; Liberti, A.D.; Besola, L.; Cibin, G.; Martinato, M.; Azzolina, D.; et al. Monitoring Patients Reported Outcomes after Valve Replacement Using Wearable Devices: Insights on Feasibility and Capability Study: Feasibility Results. *Int. J. Environ. Res. Public Health* **2021**, *18*, 7171. https:// doi.org/10.3390/ijerph18137171

Academic Editors: Irene Torres-Sanchez and Marie Carmen Valenza

Received: 25 April 2021 Accepted: 29 June 2021 Published: 4 July 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

of their use for drugs development, the appropriateness for a specific population, the validity of collected data and the management of large amounts of data collected through the device, issues concerning the choice of the suitable device depending on safety have emerged [3].

A recent US national survey on the use of wearable healthcare devices showed that 82.38% of the people involved in the study are willing to share their health data recorded by their WDs with their healthcare professionals [4]. The use of these technologies tends to decline with advancing age, although the elderly, especially those with a chronic disease, are one of the populations that could benefit most from continuous monitoring in their daily setting. The elderly, however, have poor knowledge on the use of WDs [5]. Various studies have investigated the impact of commercial WDs on the elderly. For example, WDs appear to be accurate in measuring step counts and activity duration in community-dwelling adults [6] and have a positive impact on health as their use had increased physical activity in obese patients [7]. Since WDs are often not tailored for the elderly [8], it is important to evaluate their acceptability in their daily life. A study evaluating the acceptance of WDs among the elderly showed that they seem to accept them and understand the importance of their use in healthcare setting [9].

Patients-reported outcome measures (PROMs) have been suggested in literature to be integrated in clinical trials [10], since clinical outcomes do not measure the patients' perception of their health status or functional being. The assessment of PROMs (i.e., physical function and quality of life), should be encouraged in elderly patients [11]. Physical function assessment can be useful for evaluating outcomes not directly related to the disease, but still relevant to maintaining personal dependence [12]. WDs can objectively assess physical function in daily clinical practice [13,14].

High levels of physical function are essential for the success of cardiac procedures, so much so that ad hoc cardiac rehabilitation programs have been established to improve patients' functional recovery through exercise therapy [15]. In the available literature some trials have shown that, for example, after both transcatheter aortic valve replacement (TAVR) and surgical aortic valve replacement (SAVR), physical function is improved. SAVR and TAVR are both highly valuable options for patients with heart failure (HF), a condition that affect mostly elderly patients as showed in a study on time trends in first hospitalization for HF in a community-based population [16]. Patients who had a higher level of physical function before the procedure showed more favorable trajectories [12]. Various studies have compared the two procedures in both high [17,18] and low surgical risk patients [19]. In these studies, New York Heart Association (NYHA) score or, more often self-reporting questionnaire [12,20,21] were used to evaluate physical functions of these patients. However, these instruments have some limitations related to self-reporting, for example in recall and desirability biases.

The present work aims to describe the feasibility of using WDs to monitor PROMs in patients who undergoing TAVR/SAVR enrolled in the run-in phase of the Capability study [22]. In this study, the feasibility and acceptability of using a commercial wearable device in elderly patients undergoing TAVR or SAVR will be explored.

#### **2. Materials and Methods**

#### *2.1. Study Design and Inclusion Criteria*

The study design characteristics and inclusion criteria are described elsewhere [22]. This is a multicenter prospective observational study, that enrolled patients undergoing SAVR and TAVR according to the evaluation of the local heart team since March 2018 at the University hospital of Padova and the hospital of Vicenza.

#### *2.2. Data Collection and Procedures*

Patients were enrolled at least one week prior to the procedure. Questions related to socio-demographic characteristics, risk factors, physical activity and clinical characteristics were the information collected at baseline assessment. WDs along with the smartphone were delivered at baseline assessment. Participants completed the same assessments, except for demographic characteristics, at one month, three months, six months and twelve months after the procedure. From March 2020 to the end of the follow up period, assessments were carried out only by telephone given the restrictions related to the COVID-19 pandemic.

#### *2.3. Ethical Considerations*

The study is registered in Clinicaltrials.gov (NCT03843320) and approved by the hospital ethics committee with the protocol No 943 (4 January 2019). Written informed consent was obtained from all patients for study participation and for data collection through their Garmin© device account after proper explanation of the study outcomes by a physician.

#### *2.4. Device*

A Garmin© Vívoactive® 3 smartwatch activity tracker device and a smartphone with the Garmin Connect© application (GARMIN, Olathe, KS, U.S.) installed in the smartphone for data transfer were provided to each patient at baseline assessment. The device used in the study was commercially available and was chosen for its ability to estimate steps count and sleep duration in free living environments both in the general and elderly population [23]. Both the patient and his/her caregiver were provided with information and trained on the use of the WD and the smartphone.

#### *2.5. Device Setup and Usage*

A personal account of the Garmin Connect© application has been created for each patient. The app contains only patient demographics (i.e., gender, age, weight, height, wrist of usage of the WD). Both the participants and their caregivers were not granted access to the account. All notifications were disabled in the device and the smartphone was cleaned by all the unrelated applications, to avoid affecting behaviors of the participants. Participants were asked to wear the device on their wrist 24 h a day, including while showering and sleeping, except while charging. They were asked to charge the device daily and sync data weekly. A member of the study assisted the patients at each of follow up for connectivity issues and for collecting data, while also remaining available by phone and possibly in person to troubleshoot the device at any time.

Patients were also informed that they would be asked to return the device at the end of the study. However, at the last follow up, participants were asked to choose whether to continue using the device with the smartphone for private use. After the end of the study, we collected no further data from the device.

#### *2.6. Measurements*

Physical function was assessed through a series of standard and validated tests: Duke Activity Status Index (DASI) [24], Activity of Daily Living (Barthel Index) and Instrumental Activity of Daily Living (IADL)). The DASI is a measure of functional capacity that can be obtained by self-administered questionnaire and already used in patients that underwent TAVR [25]. The Barthel Index (BI) evaluates activity of daily living [26] and has been used in aortic valve replacement [27]. The DASI score ranges from 0 to 58.2, Barthel Index ranges from 0 to 100, for both the instrument, the higher the score, the higher the functional status. IADLs were evaluated with the scale of Lawton and Brody [28]. The higher the score, the greater the person's abilities. The Epworth sleepiness scale (ESS) was used to evaluate the "subject's general level of daytime sleepiness" [29], a higher score means higher sleepiness and could be interpreted as follows: 0–5 lower normal daytime sleepiness, 6–10 higher normal daytime sleepiness, 11–12 mild excessive daytime sleepiness, 13–15 moderate excessive daytime sleepiness, 16–24 severe excessive daytime sleepiness [30].

#### *2.7. Acceptance of the Technology*

Compliance with the use of the WD (i.e., the time the device was worn) was determined by calculating the proportion of days the device was active and the total number of days before and after the procedure. At the end of the follow up, patients were given a questionnaire on technology acceptance, based on the work of Puri et al. [9]. The instrument investigated six key dimensions for WD acceptance: perceived usefulness, perceived ease of use, privacy concerns, perceived risks, facilitating conditions and equipment characteristics. Device acceptance was assessed with the question L33 as suggested in the validation study [9], "Would you use the device you used during the last year to continue to monitor or track your physical activity or health?"

#### *2.8. Statistical Analysis*

Continuous variables were reported as I, II (median) and III quartiles, categorical variables were presented as absolute numbers and percentages. Wilcoxon's test and Chisquared test were used to evaluate differences between TAVR and SAVR, respectively, for continuous and categorical variables. The compliance on the use of the WDs was summarized computing the number of days before the procedure divided by the number of days that data was synchronized in that period, the same was carried out for the follow up period. Significance was evaluated for *p*-value lower than 0.05. Data analysis was performed with R software (version 4.0.3) [31].

#### **3. Results**

#### *3.1. Baseline Characteristics*

The patients considered eligible for the study were 17, 12 TAVR and 5 SAVR, 4 from the center of Vicenza and the remaining from the University hospital of Padova. Eight patients completed the entire follow up period (Flowchart Figure 1) with an enrolment rate of 47%. Four patients were found to be ineligible after completion of the baseline assessment.

**Figure 1.** Flowchart of eligible patients in the two centers, Padova and Vicenza.

Table 1 reports patients baseline demographic characteristics by type of procedure, TAVR or SAVR. The overall sample had a median age of 79 years, 78 in the SAVR group and 82 in the TAVR group (*p*-value = 0.046).


**Table 1.** Baseline characteristics of the sample according to the type of intervention, surgical aortic valve replacement (SAVR) or transcatheter aortic valve replacement (TAVR).

Abbreviations: NYHA = New York heart association; COPD = Chronic obstructive pulmonary disease.

Except for age, there were no statistically differences between the SAVR and the TAVR groups. Patients were mainly female (10, 59%), married or cohabiting (12, 71%). Hypertension was the main risk factors in both groups, (14, 82%). Only two patients underwent TAVR procedure in urgent status. Patients, according to clinical frailty scale were more than vulnerable in the TAVR group (4, 33%; vulnerable, 3, 25% mildly frail and, 2, 17% moderate frail). Ejection fraction was similar in the two groups, median of 60 and 58, respectively, in SAVR and TAVR (*p*-value 0.49).

#### *3.2. Score Trend*

Table 2 reports the trend of the Barthel Index, DASI score, IADL index and ESS score at each follow up according to the type of procedure. SAVR patients reported higher physical function levels than TAVR especially at 12-month follow up (BI 100 vs. 85, DASI 19.9 vs. 12.8, respectively, for SAVR and TAVR). TAVR recorded lower levels of physical function at baseline according to the DASI score (30.4 vs. 14.4). Physical function level was similar at 12 months of follow up compared to the baseline assessment in both groups according to BI. The DASI score instead showed a decrease for both groups: 30.4 vs. 19.9 and 14.4 vs. 12.8, respectively, for SAVR and TAVR. As for the IADL, there was a decrease at 12 months from baseline both for TAVR and SAVR (6 vs. 4 for SAVR, 6 vs. 5 for TAVR). The ESS score was highest at 12 months follow up for both SAVR and TAVR, both groups had lower normal daytime sleepiness at each follow up.


**Table 2.** Baseline characteristics according to the type of intervention. Data are reported for that underwent SAVR (3 patients) and TAVR (5 patients).

Abbreviations: SAVR = Surgical aortic valve replacement; TAVR = transcatheter aortic valve replacement.

#### *3.3. Device's Data*

Figures 2–4 show the daily trend for each patient of rest heart rate, the number of steps and the number of hours of sleep, respectively, recorded by the device divided by baseline and follow up period and according to the procedure. Patients that underwent TAVR had a median rest heart rate higher than SAVR in the follow up period (median 64 vs. 57) (Figure 2).

**Figure 2.** Rest heart rate trend recorded by the device according to procedure and baseline vs. follow-up period. Data in the box are the I, median and III quartile of rest heart rate.

**Figure 3.** Number of steps trend recorded by the device according to procedure and baseline vs. follow-up period. Data in the box are the I, median and III quartile of number of steps.

**Figure 4.** Number of hours slept trend recorded by the device according to procedure and baseline vs. follow-up period. Data in the box are the I, median and III quartile of hours of sleep.

As for the number of steps, the SAVR group seems more active, median of 2701 steps/day vs. 1735 of TAVR (Figure 3).

Regarding the number of hours of sleep, both groups after the procedure had a similar median number of hours of sleep (7.3 SAVR vs. 7.0 TAVR) (Figure 4).

#### *3.4. Compliance and Acceptance of the Technology*

Data were mainly synchronized by the caregivers in both groups, only one TAVR patient synchronized by himself the device. The WDs and the smartphones were charged mainly by the caregivers, two TAVR patients charged their devices themselves. None of the patients or their caregivers had used a wearable device prior to the study. The compliance, evaluated as the percentage of days device use divided by the total number of days, was similar in both groups and reached a median of 75.1% in SAVR and 79.8% in TAVR (Table 3).

**Table 3.** Compliance on the use of the device according to the procedure, SAVR vs. TAVR. Data are reported as I quartile, median and III quartile, median and standard deviations.


Abbreviations: SAVR = Surgical aortic valve replacement; TAVR = transcatheter aortic valve replacement.

Figure 5 shows an overview of WDs wear time for the entire study period. Valid and missing data days are showed for each participant. The percentages of missing data vary from 0, no data, up to 70.15%. Noticeably, the patients with only 0.55% and 0% of missing data were those who could rely on the daily assistance of the caregiver. Only one patient collected less than 30% of data: in this case the caregiver was not often in contact with the patient and the patient was not able to use the smartphone autonomously.

**Figure 5.** Wearable device wear time for the entire follow-up period for each participant. A line that represents the period of enrolment is reported for each participant, in blue are reported the days without missing data, in red the days with missing data.

One patient, not reported in the analysis, stopped using the activity tracker before the surgery due to the tightness of the strap and the caregiver unwillingness to synchronize data. Missing data varied between 11.5% and 32.88%: in these cases, caregivers were available on a weekly basis. The patient who collected data by himself eventually reduced his compliance due to the long waiting-time for the surgery. In the preoperative period, SAVR patients synchronized data more often than TAVR ones, however this result could be affected by the difference in the length of the pre-operative period, which was longer for SAVR patients. In the follow up period, the compliance was similar on both groups, 75.1% vs. 75.9% for SAVR and TAVR, respectively.

#### *3.5. Technology Acceptance Questionnaire*

Table 4 reports the technology acceptance questionnaire scores for each of the seven dimensions based on the type of procedure. Except for perceived risk dimension (median 10 vs. 5.5 SAVR vs. TAVR, *p*-value = 0.016) there were no differences in all the dimensions among the two groups. The perceived risk dimensions score was higher in patients that underwent SAVR. Four patients reported that they would wish to continue to use the device (3 TAVR, 1 SAVR). The device characteristics satisfied the patients, the equipment characteristics reached for both group the maximum score, the same for perceived ease of use, also perceived usefulness showed good levels. Conversely, privacy concern (median 8 vs. 9, SAVR vs. TAVR), perceived risk (10 vs. 5 SAVR vs. TAVR) and subjective norm (10 vs. 9, SAVR vs. TAVR) showed lower level of satisfaction.

**Table 4.** Differences in the technology acceptance questionnaire scores per dimension according to the procedure for the overall sample. For each dimension is reported the maximum score. Data are reported as I, II and III quartiles. In parenthesis is reported the highest score for each dimension.


Abbreviations: SAVR = Surgical aortic valve replacement; TAVR = transcatheter aortic valve replacement.

#### **4. Discussion**

This study explores the feasibility and acceptability of using a commercial wearable device in elderly patients undergoing TAVR or SAVR.

The main findings of our study showed that there was an overall good compliance in wearing the device (75.1% vs. 79.8%, SAVR vs. TAVR) with half of the patients willing to continue using the device. Physical function decreased after both SAVR and TAVR, more in TAVR patients. The ESS score increased after the procedure but remained as "normal daytime sleepiness" according to the score, the WDs showed an increase in the number of hours slept up to 7 h per night for both groups.

Our results showed a good overall acceptance of wearing the device during the follow up period. These results are in line with what was reported in a recent literature review which showed a high-level adherence in long term daily use [32]. The acceptance rate in our study was high compared to other studies using the same device [33]. Our results considered a longer follow up period. Studies in the literature, in order to evaluate the compliance and acceptance of a device, considered shorter periods ranging from a few days [34] to weeks [5]. The number of valid days in the follow up period was lower than in the study of Henriksen et al. [35] (265 vs. 292). This result is promising in our population as data synchronization was usually performed by a caregiver. However, this high compliance can be explained by the desire of the patients to use of the WD to contribute to the study.

The main discomfort reported by our patients were related to the need to use another smartphone to synchronize data with the website and connectivity issues, as also reported in a review of activity trackers for senior citizens [36]. This could easily be avoided by allowing the patients or the caregiver to download the app directly to their own smartphone. Other technical problems were related to changing accidentally the setup of the WD or of the smartphone or forgetting to charge them. In one case, the main problem was related to the unavailability of a WI-FI connection at home. To solve these problems the presence of a technically skilled staff member available on request has proven to be a key point. Other studies showed that technical problems could reduce compliance in the use of WDs [37].

There is a growing interest in evaluating the acceptability of wearable devices both in the general and elderly population. A greater acceptance of these devices can improve the quality of real time data collection [38]. Various studies have shown that older adults accept the use of wearable devices, especially after facing acceptance barriers [9,35,39]. Perceived ease of use is one of the domains that has higher level in our results. This is in line with the findings of a recent study [40] and related to the fact that, while technology use has been increasing also in elderly people, they still need additional information and support to adopt it [41]. A recent study showed that commercial wearable devices are reliable for measuring physical activity level in elderly patients in real-life setting [42]. Despite this increase and the fact that elderly population is the one that can benefit the most from the use of these devices, especially when chronic illnesses are present [34], very few elderlies currently use daily WDs [5] or consider them in health monitoring [43]. However, the perceived ease of use recorded with the questionnaire was high for both groups, respectively, median of 30 for SAVR and 32 points for TAVR (maximum 35).

Even though the compliance was high, the perceived usefulness was not as high in both groups, median of 16 and 17, respectively, for SAVR and TAVR (64% and 68% of the overall score). This is likely sue to the fact that most of the device-related procedures were performed by caregivers and not by patients. During the follow up encounters, the researcher reinforced the importance of collecting data from the activity tracker. Caregivers, on the other hand, reported that having the possibility to see heart rate, number of steps and number of hours of sleep was useful for obtaining information on the health of the patient.

Participants when asked How much would you be willing to pay for the device you wore during the last year? (question L35) answered mainly 0 euro (4, 50%), 2 from 1 to 50 euros and 1 from 51 to 100 euros. This contrasts with what Kekade [5] reported. This may be related to the fact that in our study the population was extremely elderly (median 78 and 82, respectively, for SAVR and TAVR), while the Kekade study considered elderly over 65 years of age.

Cardiac rehabilitation after both TAVR and SAVR have shown improved functional capacity in a recent review [27]. A reduction in physical function level both according to BI and DASI score from baseline to one year follow up was showed in our sample. This result is in contradiction of what reported in other metanalysis for TAVR [21] and both TAVR and SAVR [12]. Cardiac rehabilitation was required only for two participants, both underwent SAVR. So, the sole adoption of the wearable devices is not enough to improve physical activity in these patients. In the future it would be useful to help patients in recognizing the long-term benefits of the device, along with social support as suggested by Kononova [44] in his study on tracker perceptions among older adults. Moreover, it would be helpful to use the commercial wearable activity tracker in a broader physical activity intervention as shown by the review of Brickwood [45].

Our data showed that patients that underwent SAVR were more active than TAVR patients. A functional decline or lack of improvement after the procedure was found, as already found by Kim in his study evaluating changes in functional status in the year after aortic valve replacement [12]. These differences between SAVR and TAVR and no change in functional status for TAVR may be related to the fact that TAVR group had a higher median age and none of them had rehabilitation after the procedure.

#### *Limitations*

The main limitations of this study derived from the difficulties to recruit patients with our strictly inclusion criteria. Moreover, the study design was adapted to the limitations derived from the spread of the pandemic of COVID-19.

#### **5. Conclusions**

Given the importance of developing a proper observational study to evaluate the use of wearable devices in elderly patients that underwent TAVR or SAVR, our feasibility study provided useful insights on how to implement further our project. In this study we have observed that the use of a WD is accepted in our frail population for an extended period. The use of a WD for collecting data allows the collection of data on daily basis, directly at home, with improved quality since data does not have to be manually entered and checked. However, the use of WDs in clinical trials requires an additional effort on behalf of the research team. A researcher must be available to set up the device at the beginning of the study and to solve problems related to the device. The collection of data through a WD requires always to have an application for collecting data: this could cause problems related to connectivity and device communication. Moreover, the transmission of data requires a minimum ability in the use of technologies in participants. The help of a caregiver is required especially when participants are elderly and sometimes this is not possible.

However, the presence of the device alone is not enough to encourage healthy behaviors. Therefore, it would be useful to create a coordinated intervention with a physiatrist to implement a physical activity program tailored to the single patient. As also reported in the literature, elderly people need more training session than young people, this is particularly true in population such as ours, who are not used to any type of electronic device. Considering that older people, especially frail ones, often rely on the help of a caregiver, they also need to be trained in the potential of these tools. Another suggestion could be the creation of custom reports to allow patients to view their progress in terms of physical activity. In our experience it would be faster to download the app directly on the smartphone of patients or assistants to avoid connectivity problems.

Even though commercial wearable devices were not tailored for clinical research, they could produce useful information on the behavior of the patient. Furthermore, the implementation of the use of these devices, especially in elderly, will minimize the need to attend the clinical study center, with the potential of reducing the time and costs related to person-by-person visits.

**Author Contributions:** Conceptualization, D.G., G.T., D.A. and G.L.; methodology, D.G. and G.L.; software, C.L.; formal analysis, D.A. and H.O.; investigation, H.O., A.S., L.D., A.D.L., L.B. and G.C.; resources, A.D., G.T., G.G. and E.C.; data curation, H.O.; writing—original draft preparation, H.O.; writing—review and editing, G.L., D.A. and C.L.; visualization, C.L.; supervision, D.G.; project administration, M.M.; funding acquisition, G.T. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by an unrestricted grant provided by Edwards Lifesciences. The funding body was not involved in the design of the study and collection, analysis and interpretation of data and in writing the manuscript.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Ethics Committee of the hospital of Vicenza (protocol code 943, 4 January 2019).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy restrictions.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

1. Fuller, D.; Colwell, E.; Low, J.; Orychock, K.; Tobin, M.A.; Simango, B.; Buote, R.; Van Heerden, D.; Luan, H.; Cullen, K.; et al. Reliability and Validity of Commercially Available Wearable Devices for Measuring Steps, Energy Expenditure, and Heart Rate: Systematic Review. *JMIR mHealth uHealth* **2020**, *8*, e18694. [CrossRef] [PubMed]


## *Article* **Pattern of Use of Electronic Health Record (EHR) among the Chronically Ill: A Health Information National Trend Survey (HINTS) Analysis**

**Rose Calixte 1,\*, Sumaiya Islam 2, Zainab Toteh Osakwe 3, Argelis Rivera <sup>4</sup> and Marlene Camacho-Rivera <sup>5</sup>**


**Abstract:** Effective patient–provider communication is a cornerstone of patient-centered care. Patient portals provide an effective method for secure communication between patients or their proxies and their health care providers. With greater acceptability of patient portals in private practices, patients have a unique opportunity to manage their health care needs. However, studies have shown that less than 50% of patients reported accessing the electronic health record (EHR) in a 12-month period. We used HINTS 5 cycle 1 and cycle 2 to assess disparities among US residents 18 and older with any chronic condition regarding the use of EHR for secure direct messaging with providers, to request refills, to make clinical decisions, or to share medical records with another provider. The results indicate that respondents with multimorbidity are more likely to share their medical records with other providers. However, respondents who are 75 and older are less likely to share their medical records with another provider. Additionally, respondents who are 65 and older are less likely to use the EHR for secure direct messaging with their provider. Additional health care strategies and provider communication should be developed to encourage older patients with chronic conditions to leverage the use of patient portals for effective disease management.

**Keywords:** chronic conditions; patient portal; health communication

#### **1. Introduction**

Patient–physician communication, particularly patients' satisfaction with physicians' communication approaches, is important for better outcomes in patient-centered health care organizations [1,2]. Finding effective ways of maintaining communication between health care providers and patients outside of the health care organization is important for disease management and care coordination [1,2]. With legislation mandating the meaningful use of electronic health records (EHR), by 2015, 98% of hospitals and 78% of private practices in the United States offered patient portals [3,4]. Patient portals also provide effective communication tools with the potential to increase patient engagement in self-care management. Patients can also use patient portals for prescription renewal, appointment management, checking lab results, and messaging their providers [5]. Therefore, the use of patient portals is an increasingly common approach in patient-centered care practices [1,2].

With the aging of the US population, certain diseases are becoming more prevalent [6]. Additionally, with people living longer, the prevalence of multimorbidity is also increasing [7–12]. Patients with multiple chronic conditions usually require a team of specialists

**Citation:** Calixte, R.; Islam, S.; Osakwe, Z.T.; Rivera, A.; Camacho-Rivera, M. Pattern of Use of Electronic Health Record (EHR) among the Chronically Ill: A Health Information National Trend Survey (HINTS) Analysis. *Int. J. Environ. Res. Public Health* **2021**, *18*, 7254. https:// doi.org/10.3390/ijerph18147254

Academic Editors: Irene Torres-Sanchez and Marie Carmen Valenza

Received: 4 June 2021 Accepted: 4 July 2021 Published: 7 July 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

for managing their conditions [13,14]. Maintenance of their treatment plans requires effective communication with patients and coordination of care among their team of providers to achieve better outcomes [15]. Prior research has shown that access to patient portals resulted in improved diabetes-related outcomes and adherence to hypertensive medication [16,17]. Additionally, a recent study of patients with chronic kidney disease (CKD) found that the use of EHR improves patient-centered outcomes such as CKD-specific knowledge, while reducing CKD-related stress [5]. The ongoing COVID-19 pandemic has accelerated the use of technologies such as patient portals for health care delivery [18,19]. Despite the increasing reliance on the use of technology for both preventive and follow-up care, some patients lack resources to engage effectively in telehealth [18].

Despite the proliferation of health-related internet use (HRIU) and the widespread use of the internet by the general population, disparities still exist in terms of access and use of internet for disease management [20–25]. Recent studies suggest that despite the increasing trend in the use of patient portals, the acceptability of patient portals in the general population remains unusually low [18]. Additionally, sociodemographic disparities exist in assessing patient portals as a tool for disease management and communication with providers [26–28]. A recent study on disparities in health-related internet content that focused on the noninstitutionalized population of the US assessed health information seeking behavior in three domains relevant to health communication (health care, health information-seeking, and user-generated content/sharing). The study indicated age, gender, race/ethnicity, education, and income related disparities across multiple domains of health communication [29].

Several studies have examined potential explanations for low EHR engagement, and HRIU more broadly, within racial and ethnic minority populations. A recent study which examined disparities in trust in sources of cancer-related health information among Hispanics in the US observed that older Hispanics had higher odds of trusting cancer information from a religious organization compared to younger Hispanics [30]. Another study of men with chronic conditions developed an eHealth usage score using seven domains of eHealth communication, which included EHR use [31]. The eHealth questions used to create the score asked whether respondents had done the following: used a computer, smartphone or other electronic means to (1) look for health information or medical information for yourself, (2) look for health or medical information for someone else, (3) buy medicine or vitamins online, (4) look for assistance for the care you provided someone else, (5) use email or the internet to communicate with a doctor or doctor's office, (6) track health care costs/changes, or (7) look up medical tests. This study identified disparities in eHealth usage across social and demographic characteristics. Particularly, education and income were positively correlated with eHealth score, with participants with higher levels of education and those with higher incomes having increased scores for eHealth usage. However, the same study observed that age and Hispanic ethnicity were negatively correlated with eHealth score, such that older patients had lower scores of eHealth usage, and Hispanics also had lower scores of eHealth usage [31]. Low eHealth usage among individuals of Hispanic ethnicity may be due to low English proficiency and lower levels of health literacy [32,33]. Among older individuals, lower levels of health literacy and technological skill have been found to be associated with lower eHealth usage [34,35]. A previous study examining associations between health literacy and health information seeking found that participants with chronic conditions were more likely to be engaged in health information seeking and higher instances of seeking care based on information found on the Web. Additionally, participants with chronic conditions had higher eHealth literacy scores compared to participants without chronic conditions [36]. When compared to patients without chronic health conditions, patients with a history of chronic conditions reported frequent use of patient portals for different aspects of health care delivery such as checking lab reports, messaging their doctors, and setting up appointments [36]. However, this study failed to adjust for social and demographic factors that are associated with both chronic disease status and health information seeking

behaviors. While many studies have looked at disparities in the usage of patient portals in patients with specific chronic conditions, no studies thus far have looked at disparities in the usage of patient portals in only people with chronic conditions in a nationallyrepresentative sample of the noninstitutionalized US population. As EHR usage has been associated with significant improvements in patient self-management of chronic diseases, as well as improved quality of care given by providers, understanding disparities in EHR use may provide important insights into health care disparities among adults living with chronic health conditions in the US [37].

#### **2. Materials and Methods**

The National Cancer Institute's Health Information National Trends Survey (HINTS) is a publicly available national representative survey of the noninstitutionalized adult population that collects data about Americans' use of cancer related information. Data for this study came from the HINTS 5 cycle 1 (*N* = 3335), collected from January 2017 to May 2017, and HINTS 5 cycle 2 (*N* = 3504), collected from January 2018 to May 2018. The sampling design for the HINTS survey has been described extensively [22,38]. The response rate was 32.4% for HINTS 5 cycle 1 and 32.985% for HINTS 5 cycle 2.

The goal of this study was to assess differences in EHR usage among respondents with chronic diseases conditions. Using self-reported data, the study was restricted to respondents with any of the following conditions: diabetes, hypertension, lung disease, heart conditions, depression, cancer, and arthritis. The final analytic sample was further restricted to respondents who reported accessing their online medical record at least once in the past 12 months for various reasons (*N* = 736 and 816 respectively) for a total sample size of 1552. The outcomes of EHR usage were assessed using 4 HINTS questions relating to the purposes of accessing the online medical record to (1) securely message their health care provider, (2) request a refill of medications, (3) make a decision on how to treat illness or condition, and (4) securely share it with another provider.

Primary predictors of interest were gender, race/ethnicity, age, education, income, multimorbidity, and nativity status. The covariates of interest were smoking status, employment status, regular access to a health care provider, insurance status, general health status, and family history of cancer. To measure the change in EHR use across the two HINTS releases, we used a dummy-coded variable to represent the survey year.

We used multivariable regression models to find patterns of associations of sociodemographic characteristics with domains of eHealth usage in the population of US patients with chronic conditions who have access to their online records. The use of online records to securely message health care providers in the past 12 months and the use of online records to request prescription refills in the past 12 months were analyzed using Poisson regression with a log link and robust estimates of standard errors [39]. The use of online records to make decisions on how to treat an illness or condition in the past 12 months and the use of online records to securely share health records with another provider in the past 12 months were analyzed using logistic regression. To account for the complex survey design used to collect the data, we used jackknife replicate weights to compute accurate standard errors, with all analyses weighted to provide nationally representative estimates. We conducted all statistical analyses using SAS 9.4® and Stata 16®. The threshold for the significance of the *p*-value was set to ≤0.05.

We summarized the data using appropriate descriptive statistics such as frequency (percent) and weighted percent (standard error). We presented the multivariable regression models using incident rate ratios (IRR) and odds ratio (OR) with a 95% confidence interval (CI).

#### **3. Results**

#### *3.1. Descriptive Results*

The final analytic sample comprising respondents with at least one chronic condition that accessed their online medical record at least once in the past 12 months resulted in 1552 participants from the two HINTS cycles. We have presented the summary of the sociodemographic characteristics of the respondents in Table 1. The analytic sample was 56% female, 42% 18–49 years old, 70% non-Hispanic White, 78.2% with some college education, 58% employed, and 47.2% with an income of \$75,000 or more. We combined Asian and other races for the purpose of the multivariable analysis. The sample analyzed was 88.8% US born, and 93% reported speaking English very well. Additionally, 96.4% had access to health insurance, 83% had a regular health care provider, and over 73% of the participants reported having a family history of cancer.


**Table 1.** Sociodemographic characteristics of HINTS respondents with at least one chronic condition who accessed their patient portal at least once in the past 12 months (*N* = 1552).


SE: Standard Error.

Table 2 presents the clinical characteristics of the respondents. About 59% of the analytic sample were never smokers. Overall estimates suggest that about 56% of respondents reported having more than one chronic condition. Within the analytic sample, the most prevalent chronic condition was high blood pressure, with 55% of the respondents reporting having high blood pressure. However, about 45% reported being in very good or excellent health, and less than 20% reported being in fair or poor health.

**Table 2.** The Clinical characteristics of HINTS respondents with at least one chronic condition who accessed their patient portal at least once in the past 12 months (*N* = 1552).




SE: Standard Error.

We have presented the summary for the outcome variables in Table 3. Of all the respondents who accessed their medical records in the past 12 months, 47.3% of the participants used it 1–2 times, while less than 10% accessed their medical record 10 or more times. In the EHR communication domains, 48% of the respondents used the online medical record system to securely message their health care provider, 43.8% of the participants used the online medical record system to request a refill of medications, 21.1% of the respondents used their online medical record system to make a decision on how to treat an illness or condition, and 12% securely shared their medical record with another provider.

**Table 3.** Summary statistics of patient portal-related communication (*N* = 1552).



SE: Standard Error.

#### *3.2. Multivariable Model*

We have presented the results of the multivariable model in Table 4. Among HINTS respondents with chronic conditions who accessed their online medical record at least once in the past 12 months, respondents 65 to 74 years and those 75 years or older were significantly less likely to use the system to securely message their health care providers compared to respondents 18 to 34 years (IRR = 0.73, 95% CI = 0.57, 0.94; and IRR = 0.54, 95% CI = 0.35, 0.83) respectively.




**Table 4.** *Cont.*

The multivariable regression model (logistic and Poisson) was adjusted for insurance status, employment status, having a regular provider, general health status, smoking status, and family history of cancer. We present the results of the Poisson regression models as incident rate ratios (IRR) and the results of the logistic regression models as odds ratio (OR) with 95% confidence interval (CI). \* *p*-value ≤ 0.05, \*\* *p*-value ≤ 0.01.

> The results of the multivariable model indicate that race, age, and survey years are associated with respondents who reported the use of the online medical record to make a decision about treating a condition or an illness. Respondents in the 65 to 74 age range had reduced odds of using the medical record to make a decision regarding treating a condition or illness (OR = 0.30, 95% CI = 0.11, 0.84). There was a significant increase in the use of the online medical record to make a decision on how to treat a condition or illness from HINTS 5 cycle 1 to HINTS 5 cycle 2 (OR = 1.99, 95% CI = 1.23, 3.20). Additionally, respondents in the other racial category had increased odds of using the online medical record to make decisions on how to treat a condition or illness in the past 12 months (OR = 2.64, 95% CI = 1.12, 6.24).

> Those in the 75 years and older age range had reduced odds of securely sharing their online record with another provider compared to respondents aged 18 to 34 years (OR = 0.17, 95% CI = 0.03, 0.99). Respondents with multimorbidity (2 or more diagnoses) had significantly increased odds of securely sharing their online record with another provider in the past 12 months (OR = 2.04, 95% CI = 1.16, 3.59). Lastly, respondents in the other racial category had increased odds of securely sharing their online medical record

with another provider in the past 12 months compared to Non-Hispanic Whites (OR = 3.61, 95% CI = 1.25, 10.42).

The sociodemographic characteristics of the respondents were not associated with their likelihood of requesting prescription refills using the online medical record. No other predictors of interest were significantly associated with any of the outcomes after adjustment for covariates.

#### **4. Discussion**

To our knowledge, this is the first study to examine disparities in accessing patient portals for disease management among chronically ill noninstitutionalized adults using nationally representative data. Our study examined a broad range of reasons for the use patient portals for disease management, including secure messaging with providers, requesting medications refills, and sharing medical records with other providers.

Our study is consistent with previous studies examining eHealth communication in US adults, revealing that age disparities exist in the use of eHealth communication methods, with older participants having a lower rate of eHealth communication across multiple domains [26,27,29,40]. However, our study yielded specific insights into the use of the patient portals, while the other studies were focused on the more general use of the eHealth domain of communication, such as personal email, searching for health information, buying medications online, and sharing health content though social networking sites [26,27,29,40]. Other studies of seniors 65–79 years old revealed that patients aged 70 years and older were less likely to register to use web portals [24,25]. Some potential explanations as to why older respondents were less likely to be engaged in use of the patient portals include the accessibility of the patient portals, safety concerns, and the usefulness of the portal for communication as opposed to face-to-face meetings with providers. This result, however, is concerning, as the likelihood of using the patient portal may be reflective of the ability to use telehealth services, and because the COVID-19 pandemic has accelerated the expansion of telehealth services [41]. Since a recent study of older adults' readiness to engage in eHealth and mHealth indicated that over 80% of the respondents reported having access to the internet at home, and 44% of those using the internet reported doing so on a mobile device [42], further studies should be done to understand the digital divide for older patients and how to engage them in eHealth.

In a prior HINTS study of HRIU use among cancer survivors, there has been an increasing trend in the use of HRIU [21,43]. In our study, we similarly found an increase in the use of EHR to make clinical decisions about how to treat an illness or condition. However, no increasing trend was noted in any of the other domains analyzed. This result underscores the importance of encouraging and promoting the use of the portal for other aspects of chronic disease management, particularly communicating with providers and sharing medical records with other providers. Despite the increasing trend in HRIU among cancer survivors, in a prior HINTS study of cancer survivors, there were age, race, education, and geographic disparities in the use of HRIU [27,44]. This study is in line with our result of lower use of online medical records for secure communication, to treat medical conditions, and to share with other providers among older participants. With the recent COVID-19 pandemic, health care providers have shifted to the use of telehealth for primary and specialty care for disease management [45]. The shift to telehealth during the pandemic could further widen the gap of health outcomes for patients with low and limited access to technology and those who are not ready to adopt new or emerging technologies.

Our study indicates that respondents with two or more chronic conditions were 2.04 times more likely to share the EHR with another provider. This result may be attributed to the need for different specialists to be involved in the care of patients with multimorbidity. This current finding is somewhat in line with other study that indicated that patients with chronic conditions were more engaged in care-seeking behavior based on health-related internet searches [36]. Patients with multimorbidity should be encouraged to participate in other forms of eHealth communication through the patient portal for disease management.

Prior studies have found sociodemographic disparities in the use of HRIU [23,28,29,46]. In our study, those identified as other race (Asian and others) were significantly more likely than those identified as White to use the portal for clinical decision making and transferring medical records to another provider. However, only 3.63% of the participants self-identified as Asian, and only 2.95% identified as other, limiting any form of generalization of the results. Although sociodemographic disparities exist in who is being offered access to EHR [47], in our study, which focused on only respondents who reported accessing their EHR at least once in the preceding 12 months, characteristics such as income and education level were not associated with the different domain of using EHR. We suspect that the result is due to our restriction of the analytic sample to respondents who reported having accessed the web portal at least once in the past 12 months. This subset of respondents is important because those respondents were not only offered access to their EHR, but have indicated using it. This restriction on the inclusion criteria for the final analytic sample could potentially control for some of the disparities that exist in accessing EHR, such as access to and use of the internet [26], and being offered its use [47]. For that reason, we did not adjust for the access/use of internet because using the web portal is an indication of internet access/usage.

#### **5. Study Strengths and Limitations**

There are several strengths to this study. First, the HINTS survey is a national survey of adults in the United States 18 and older. Thus, the subsample used in this study represents adults in the United States living with chronic conditions who have accessed their online health portal at least once in the past 12 months. We also used two HINTS cycles, allowing us to see changes in the use of EHR over time. Additionally, the study included an exploration of different reasons for using the EHR in disease management.

Despites the strengths, our study has some limitations. Given that HINTS is a crosssectional study, we cannot make conclusions about causality. Furthermore, the survey has participants with a limited number of chronic conditions. Therefore, we can assess the association of respondents' characteristics with reasons for using EHR only in a subset of patients with chronic conditions. Future iterations of the survey should ask participants about other prevalent chronic conditions. In addition, future iterations of HINTS should include questions that address barriers to using the online medical records.

#### **6. Conclusions**

A recent HINTS brief indicated that the proportion of US adults accessing their online medical records increased from 27% in 2014 to 40% in 2018 [48]. Despite the increase in the access and use of EHR, our study reveals age as a factor of disparity in assessing the EHR for health care management. However, past studies have shown that over 80% of older adults are already using the internet, and 44% have access to smartphones. Therefore, health care providers should develop strategies to inform older patients and their proxies about the accessibility of the EHR as a secure means of communication to providers. Some strategies can include the development of user-friendly patient portals for multiple platforms that encourage greater use. Other strategies should include eHealth literacy programs that address patients' concerns regarding safeguarding their protected health information. Additional technological improvements may include the use of integrated displays to decrease user cognitive load [49–51], integration of linked EHR records at the household level to facilitate delivery of services, and embedding of health literacy tools (e.g., embedded medical search engines, integrated AI voice chatbots for on-demand self-care advice) to facilitate meaningful patient engagement.

The use of electronic patient portals raises challenges and ethical issues regarding older adults. Notably, disparities in internet access, a key factor for the use of e-health, persist among underserved populations such as older adults and individuals of lower socioeconomic status [52,53]. Older adults may encounter more challenges in the use of EHR and electronic patient portals because older generations must learn and acquire the

necessary skills needed to navigate the internet and are less comfortable using technology compared to their counterparts. Furthermore, some older adults with certain illnesses may require support to navigate the complexities of eHealth portals [54]. Our findings are consistent with prior work using HINTS data, which showed that older US adults were less likely to engage in eHealth, and point to a need for additional support to ensure equitable access to e-health for older adults [22]. Other ethical aspects that are of importance are autonomy, privacy, confidentiality, consent, and beneficence [55]. Those ethical issues can be extended to the adaptability and accessibility of patient portals for personal use by patients. With older participants being less likely to engage in eHealth communication through the patient portal, adopters of EHR should consider the issue of autonomy, privacy, confidentiality, and equality of access as they encourage older patients and their proxies to make full use of the system for disease management.

**Author Contributions:** Conceptualization, R.C. and M.C.-R.; methodology, R.C and M.C.-R.; software, R.C.; formal analysis, R.C.; investigation, R.C., S.I., Z.T.O., A.R., and M.C.-R.; resources, R.C., S.I., and M.C.-R.; data curation, R.C.; writing—original draft preparation, R.C., S.I., Z.T.O., A.R., and M.C.-R.; writing—review and editing, R.C., S.I., Z.T.O., A.R., and M.C.-R.; visualization, R.C. and S.I.; supervision, R.C. and M.C.-R.; project administration, R.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data that support the findings of this study are available from https://hints.cancer.gov/data/download-data.aspx.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **How Can We Develop an Efficient eHealth Service for Provision of Care for Elderly People with Balance Disorders and Risk of Falling? A Mixed Methods Study**

**Andréa Gomes Martins Gaspar 1,2,\*,†, Pedro Escada <sup>3</sup> and Luís Velez Lapão 1,†**


**Abstract:** This study aimed to identify relevant topics for the development of an efficient eHealth service for elderly people with balance disorders and risk of falling, based on input from physicians providing healthcare to this patient group. In the quantitative part of the study, an open multiple-choice questionnaire was made available on the website of the Portuguese General Medical Council to assess the satisfaction with electronic medical records regarding clinical data available, the time needed to retrieve data and the usefulness of the data. Of the 118 participants, 55% were dissatisfied/very dissatisfied with data availability and 61% with the time spent to access and update data related to the focused patient group. Despite this negative experience, 76% considered future e-Health solutions as pertinent/very pertinent. Subsequently, these findings were further explored with eight semi-structured interviews. The physicians confirmed the reported dissatisfactions and pointed out the lack of comprehensive data and system interoperability as serious problems, causing inefficient health services with an overlap of emergency visits and uncoordinated diagnostics and treatment. In addition, they discussed the importance of camera and audio monitoring to add significant value. Our results indicate considerable potential for e-Health solutions, but substantial improvements are crucial to achieving such future solutions.

**Keywords:** balance disorders; falls; elderly care; eHealth; mixed methods

#### **1. Introduction**

*1.1. The Burden of an Ageing Population: Portugal and the World*

As is observed in other health systems [1–7], Portugal's increasing life expectancy in recent decades has not been followed by an increase in healthy life years [4,6]. The prevalence of chronic diseases, comorbidities, disabilities and falls have increased with aging [1–7]. In fact, there are many causes for elderly people's falls to happen, including age, environmental factors, inappropriate clothing and shoes, risky behavior, medications, and balance disorders [1,7–9].

Elderly falls represent an important public health problem, being the main cause of accidental death in the population over 65 years of age [1,8–10]. Although Portugal has one of the lowest rates of fall-related mortality in the elderly population of the Western European region, this issue has received attention from the Portuguese government [11].

The burden of aging with balance disorders and falls, and the insufficient access to healthcare data by health professionals, have led to additional medical visits, overdiagnosis, repeated diagnostic tests and multiple prescriptions [12–15]. This misuse of healthcare provision is costlier and unsustainable for the current healthcare provision model and considered unsuitable for responding to elderly population demand [4,12–15]. In order to

**Citation:** Gaspar, A.G.M.; Escada, P.; Lapão, L.V. How Can We Develop an Efficient eHealth Service for Provision of Care for Elderly People with Balance Disorders and Risk of Falling? A Mixed Methods Study. *Int. J. Environ. Res. Public Health* **2021**, *18*, 7410. https://doi.org/10.3390/ ijerph18147410

Academic Editors: Irene Torres-Sanchez and Marie Carmen Valenza

Received: 23 May 2021 Accepted: 8 July 2021 Published: 11 July 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

relieve this pressure, new strategies have been recommended, including person-centered health systems and the utilization of devices and systems supported by Information Systems and Technologies (IST) [2,5,16] (Figure 1).

**Figure 1.** eHealth Framework for the elderly with balance disorders care provision. (Authors own elaboration).

These healthcare solutions have revealed the potential to provide quality health services with complete, interoperable data in near real-time [17–20], and eHealth services with the engagement of patients and families for self and remote management of chronic conditions and prevention of risky behaviors [2,20–22]. Indeed, many researchers have studied the potential of digital sensors to identify early balance deficit and identify fallers among elderly people, improving the data quality of clinical tests and functional scales as the Timed Up and Go Test (TUGT) and the Berg Balance Scale [23–29]. Other researchers have evaluated the benefits of eHealth devices in balance training, reducing the risk of falling [29–31]. The eHealth services seem to have the potential to be a complementary method for preventive monitoring of falls, telerehabilitation, and monitoring of effective rehabilitation for elderly with balance disorder and risk of falling [23–31], aligned with the eHealth definition: "*an emerging field in the intersection of medical informatics, public health and business, referring to health services and information delivered or enhanced through the Internet and related technologies. In a broader sense, the term characterizes not only a technical development, but also a state-of-mind, a way of thinking, an attitude, and a commitment for networked, global thinking, to improve health care locally, regionally, and worldwide by using information and communication technology*" [32]. However, there are still constraints to be overcome: technological obsolescence; unsuitable technological devices; regulation, standardization, auditing, inspection and quality control; lack of interoperability; health professional resistance; low organizational capability for new ways of working and organization; shortage of digital training [33–38]. In Europe, both the skill development in old age and the aging of younger generations of technology users have led to a growing number of elderly people able to use health and care services online, although to a lesser extent in Portugal [39].

#### *1.2. Portugal and the Strategies Supported by IST*

To build up a shared ecosystem of health information, the Portuguese Ministry of Health created an Electronic Health Record (EHR) called "Sclínico," which is unfortunately not yet available in all health units of the Portuguese National Health Service (NHS) [40]. More recently, another digital service was made available on the NHS' digital platform to allow the sharing of clinical information between all levels of health care and to promote the interaction between the citizen and the family health unit [40]. In addition, the Portuguese elderly people can use the current telephone and digital service of the NHS Call

Center, known as "SNS 24," which is responsible for the triage of first-level emergencies and guiding the population about health problems [40]. Another NHS telephone service, known as Senior Proximity Project, was implemented to identify the risks and needs of elderly people to reduce morbidity and promote more autonomy and health literacy [41]. Several public health units have provided retinal examination by teleradiology, teleconsultation, telediagnostic-telepathology, telemonitoring of cardiac and pulmonary diseases and telerehabilitation of osteoarticular disease of shoulder and knee [40]. Additionally, the electronic prescription system and the treatment guide for the user have allowed patients, including elderly people with chronic diseases, to obtain their medication without ever going to a health care unit [40].

However, in recent reviews [23–31], it was pointed out that the clinical applicability of eHealth devices and services in screening, assessing and treating elderly people with balance disorders and the risk of falling in Portugal is still unknown. Therefore, we aim at studying how to obtain an efficient eHealth service for the provision of care for elderly people with balance disorders and the risk of falling.

The purposes of this explanatory sequential mixed methods approach [42] were: (a) to identify and understand how to overcome the medical difficulties about availability of clinical data in the electronic medical record (EMR) relatively to the context of healthcare provision for elderly with balance disorders and risk of falling; (b) to know and understand the medical relevance about eHealth services to support health care for elderly people with balance disorders and risk of falls; (c) to understand how to develop an efficient eHealth service to support health care for elderly people with balance disorders and risk of falls.

The increasing interest of elderly people in medical digital devices and the eHealth potential to enhance health promotion and physician–patient interaction to mitigate care access inequities and to allow remote management of balance disorders and risk of falling are opportunities that should be further explored for active and healthy aging [29]. This could be viewed as an opportunity to mitigate the aging pressure in the health systems.

#### **2. Materials and Methods**

*2.1. Study Design*

From June to August 2019, the authors performed a quantitative observational descriptive study [42] to identify the difficulties about clinical data and the relevance of eHealth (Figure 2).

**Figure 2.** Design of the mixed methods study performed.

This first phase aimed at responding to the following specific research questions: "Do physicians have difficulties accessing current clinical data in EMR relatively to the context of healthcare provision for elderly with balance disorders and risk of falling?"; "In this context, what is the medical satisfaction level with the use (e.g., time spent to access and fill in clinical data) and quality (e.g., availability of sufficient and understandable clinical data) of the current clinical data in the electronic medical record (EMR)?", and "Could eHealth services be relevant to improve healthcare?".

From December 2019 to April 2020, a qualitative exploratory and descriptive phenomenological study [42] was performed to understand the quantitative results and how to obtain an efficient eHealth service to support health care for elderly people with balance disorders and risk of falls. This second phase explored the following research questions: "What are the medical difficulties related with current clinical data in the context of health care provision for elderly with balance disorders and risk of falling?"; "What strategies can be implemented to improve clinical data?"; "What do you think about the contribution of eHealth?"; "How can an eHealth service be suitable? What are the necessary strategies? What difficulties must be overcome?".

#### *2.2. Materials*

The questionnaire, entitled "Health contribution to the provision of health care for the elderly at risk of falling due to balance disorders" (in the original: "A contribuição do eHealth na prestação de cuidados de saúde ao idoso com risco de queda por distúrbios do equilíbrio") was developed with 18 multiple choice questions [42]. It included sociodemographic data of the participants, availability of data in the EMR and relevance of eHealth in the context of health care for the elderly with balance disorder and risk of falling. Except for the demographic questions, alternative responses on quantity, frequency and evaluation were used, and a non-response ("Do not know/Do not answer") was provided [42] (see Table A1). The usability, technical functionality and time to complete the questionnaire were tested. The access link was available through the website of the Portuguese General Medical Council (Ordem dos Médicos de Portugal), the entity that regulates medical practice in Portugal (https://ordemdosmedicos.pt/inquerito-acontribuicao-do-ehealth-na-prestacao-de-cuidados-de-saude-ao-idoso/ (accessed on 25 June 2019)). The information regarding this open survey was provided online. The eligible participants were specialist physicians who provide healthcare in Portugal for the elderly with balance disorders and risk of falling, including family physicians, internal medicine physicians, physical medicine and rehabilitation (PMR) physicians, neurologists, otolaryngologists and physicians with competence in Geriatrics. For the advertisement of the study and disclosure of the access link of the questionnaire, the authors requested, via email, the collaboration of the Portuguese Society of Physical Medicine and Rehabilitation, Portuguese Association of General and Family Medicine, Portuguese Society of Internal Medicine, Center for Geriatric Studies of the Portuguese Society of Internal Medicine, Portuguese Society of Otorhinolaryngology and Portuguese Otoneurology Association. The Portuguese Society of Neurology was also contacted through this institution's website. The questionnaire was distributed online using the survey software SurveyMonkey® [43]. Each question was made available in turn, with the possibility of returning to the previous questions. All questions had one mandatory answer [44]. During the study time, the IP address of the participants was used to eliminate potential duplicate responses from the same user [44].

Regarding the qualitative study, the same interviewer (one of the authors) conducted individual semi-structured interviews [42]. Four primary thematic categories were discussed: current clinical data in the context of health provision for elderly with balance disorders and risk of falling (i.e., understanding of quantitative results), interventions to improve the clinical data, understanding of eHealth relevance pointed out by physicians in the quantitative research, and strategies to improve the use of eHealth services (see Table A2). The sampling was intentional [42], with a purposeful search for physicians

with: (a) healthcare provision for elderly with balance disorders and risk of falling; (b) coordination function in health units and; (c) easy access by the interviewer. The potential interviewees were invited by the interviewer, in person or via telephone, to participate in the study. The participant number was defined after saturation or redundancy of responses; that is, the sampling process was completed when no new information emerged from the new interviews [42]. Respecting the anonymous participation of the quantitative study, the interviewer did not ask if the interviewee had participated in the previous study.

#### *2.3. Data Analysis*

Firstly, to determine the quantitative frequency tables [42], a descriptive and exploratory statistics of the data from the questionnaires were performed. The software IBM SPSS Statistics version 26 (IBM Corporation, Armonk, NY, USA) was used [45]. Secondly, a descriptive analysis of demographic data [42] of interviewed was performed. All the interviews were manually coded and transcribed by the interviewer, allowing content analysis of interviews [42]. For a better comprehension of the quotes, the authors entered words in round brackets.

#### *2.4. Ethical Considerations*

The survey's aim was clearly identified in both the website of the Portuguese General Medical Council and on the SurveyMonkey® link [SURVEY PREVIEW MODE] A contribuição do eHealth na prestação de cuidados de saúde ao idoso com risco de queda por distúrbios do equilíbrio Survey (surveymonkey.com, accessed on 25 June 2019). The physicians could voluntarily participate and leave the study until the submission of the questionnaire. The information of the quantitative study was treated confidentially and anonymously by using respondent e-mails confidentiality and anonymous responses features of the software SurveyMonkey® [43].

Regarding the interviews, the participants signed a consent form and received a copy of this and information about the study. They could leave the study until one month after the interview's date. The audio recording was authorized by the participants. To guarantee confidentiality, all interviews were manually coded. The transcriptions omitted information to avoid identifying respondents. All data were kept anonymous [42]. The information from the questionnaires and interviews and the audio records were kept in a safe place (external disk with access code) within the period provided by the Portuguese law [46], always safeguarding the confidentiality of the information obtained.

#### **3. Results**

#### *3.1. First Phase: Quantitative Research*

The online questionnaire had a total of 118 responses. This represents 1% of the total universe of 12,214 [47] family physicians, internal medicine physicians, PMR physicians, neurologists and otolaryngologists registered in Portugal (Table 1).


**Table 1.** Study quantitative: Demographic data of the participants. PMR: physical medicine and rehabilitation.


**Table 1.** *Cont*.

<sup>a</sup> According to Portuguese General Medical Council ("Ordem dos Médicos de Portugal")—year 2019 [47].

There was no duplicate response found with the same IP address. A relevant proportion of the respondent activity was directed to provide care to elderly people in the context studied. About the elderly people observed by physicians, 19% of the participants said that their monthly appointment time was more than 50% occupied with elderly patients with balance disorders, while 9% of physicians had their monthly appointment time more than 50% occupied with elderly patients with complaints related to consequent falls. A total of 86% of the physicians recognized the relevance of data about the previous provision of health care to the elderly with balance disorders and risk of falling. However, A total of 43% of all physicians responded that they need to access data from previous care consultations for elderly patients with balance disorder and risk of falling in more than half of cases. The majority of the participants (84%) had access to this information through the hospital or health center electronic medical record. Most respondents (60%) reported that more than half of the medical consultation time had been spent on IST-related activities. Moreover, 50% of participants were dissatisfied or very dissatisfied with the use of IST (e.g., usefulness, quality) in the context of balance disorders and the risk of falling in the elderly.

#### 3.1.1. Socio-Demographic Participant Data

Most of the participants were female, accounting for 72 responses (61%). Younger physicians adhered more to the study: most participants (74%) had 50 years old or less. Although 64% of the eligible physicians were over 50 years old (7787 out of 12,214), the participants over 50 years old represented only 26% (31 out of 118) of the responses.

About 39% of the participants were specialists in Internal Medicine, 38% in Otolaryngology, 15% were family physicians, 4% PMR physicians, and 3% were neurologists. Only 4% were enrolled in the College of Competence in Geriatrics. Comparing the numbers, the family physicians had weak participation (18 out of 7451 family physicians), although being the specialty most represented among the eligible physicians.

Most physicians (82%) had the main job in public healthcare units, and 54% were from the larger Portuguese health region, the Lisbon and Tejo Valey (LTV) Regional Health Administration. (Table 1)

3.1.2. Difficulties and Medical Satisfaction Level Related to Current Clinical Data in the EMR (Electronic Medical Registration)

61% of all respondents were dissatisfied or very dissatisfied with the time spent accessing clinical data in the EMR, rising to 65% when considering only professionals who have the main job in a public healthcare institution. Regarding the availability of sufficient and understandable clinical data in the EMR, 55% of the physicians revealed dissatisfaction or a lot of dissatisfaction, with values of 59% for public health professionals as their main job. Again, 61% of all participants also expressed dissatisfaction or great dissatisfaction with time spent to fill in new data in the EMR, reaching 64% among professionals with the main job in a public health institution (Table 2).

**Table 2.** Questionnaire: Satisfaction degree with clinical data in the EMR—Context of health care provision for elderly with balance disorders and risk of falling.



**Table 2.** *Cont*.

S: Satisfied or very satisfied. D: Dissatisfied or very dissatisfied. Others: Did not use or Did not answer or Did not know.

#### 3.1.3. Relevance of the Use of eHealth

The possibility of using eHealth for elderly patients with balance disorder and risk of falling was considered pertinent or very pertinent by 76% of all physicians and also by professionals with public healthcare as the main job. Regarding the medical specialties with more than 30 responses, 72% (33 out of 46) and 82% (37 out of 45) of internal medicine physicians and otolaryngologists, respectively, considered remote services as pertinent or very pertinent. If we consider only the participants of Internal Medicine and Otolaryngology working in the public sector as their main job, the percentages remain at 72% (31 out of 43) and rise to 86% (25 out of 29), respectively (Table 3).

**Table 3.** Questionnaire: Relevance degree about the use of eHealth in the context of care provision for the elderly with balance disorders and the risk of falling.


Others: Did not answer or Did not know.

#### *3.2. Second phase: Qualitative Research*

The same interviewer conducted a total of seven face-to-face semi-structured interviews and one semi-structured interview by mobile phone due to coronavirus pandemic limitations. This interview phase was limited to senior physicians who provided healthcare to the elderly, with different training in technology and medical experience.

#### 3.2.1. Socio-Demographic Participant Data

Five male and three female physicians, aged 47–66 years old, participated in the study. Two were family physicians, two internal medicine physicians, one neurologist and three otolaryngologists. All of them were either graduated or senior consultants. Six physicians were coordinators in their public health units, and two were coordinators of private otoneurology units. One physician was from the Regional Health Administration of the center (Center) of Portugal, and the others were from Lisbon and Tejo Valey (LTV) (Table 4).


**Table 4.** Study qualitative: Socio-demographic data of the participants and interview features.

M: Male; F: Female; LTV: Lisbon and Tejo Valey.

#### 3.2.2. Content Analysis

As mentioned previously, four primary thematic categories were discussed. Twelve subthemes emerged from data analysis (Table 5).

**Table 5.** Thematic categories of the qualitative research.

#### **Thematic Categories**

1. Current clinical data in the context of health provision for elderly with balance disorders and risk of falling: understanding of the medical dissatisfaction identified in the quantitative research

1.1. Availability

1.2. Barriers

2. Interventions to improve the clinical data

2.1. Interoperability of computer health systems

2.2. New work organization

3. eHealth contribution in this context: understanding of the relevance observed in the quantitative research

3.1. eHealth benefits

4. Strategies to improve eHealth services for a more effective healthcare provision

4.1. Clinical and interactive data


The interviewees pointed out some misuse of healthcare provision by the elderly people in Portugal, meaning using above what is necessary of emergency visits, drug therapies and complementary diagnostic tests: *The elderly Portuguese population has no specific education on how to access healthcare services properly* (Participant 5). They also agreed on the need to access complete data: ... *the elderly people often represent complex patients* ... *the intervention* ... *requires multiple specialties* . . . (Participant 1).

Relative to the medical dissatisfaction with available clinical data identified in the quantitative research, all interviewees highlighted the lack of a comprehensive data set and the lack of interoperability of computer systems: *We* (Physicians) *get to know more or less the drugs that are prescribed* ... *We don't know more* ... (Participant 4); *I have asked them* (family physicians) *to send me information. So, I can get a sense of what is going on with the patient.* (Participant 8); *... the data records are, sometimes, incomplete, they are not very explicit* (Participant 1); *The computer systems* ... *have great incompatibilities with each other.* ... *because the operating systems are different, or because the internet browser is different.* (Participant 5).

Some interventions to improve clinical data were pointed out: more investments in the interoperability of health information systems and in the organization of work with time for remote interaction and consultation: *Medicine will have to be a Medicine of shared information*. (Participant 1); *We should have some time allocated for this* (remote consultations). *So that we can keep our head on it and we will be really effective*. (Participant 4). Regarding the use of eHealth services for elderly people with balance disorders and the risk of falling, only Interviewee 2 questioned its proper applicability, justifying the doubts due to his lack of experience: ... *I don't know how this* (remote health care provision) *is done at a distance* ... *I don't even see myself doing a thing like that* ... *I think the physician-patient relationship is something that is impossible to be computerized.* (Participant 2). The other physicians, similar to most of the participants of the quantitative research, considered the eHealth contribution to be beneficial: ... *it* (eHealth) *could be a great help because vertigo has many decompensations* ... (and) *they* (patients) *are afraid of being* ... *without connection to the physician.* (Participant 7).

The interviewees pointed out the potential benefits of eHealth as a complementary channel to healthcare: rational use of resources with lower pressure on hospital resources, more healthcare access, better communication between medical specialties, closer physician– patient relation and more participation of patient and caregiver at home. However, they only agreed with the eHealth use for a follow-up consultation. The age was not considered a limitation for eHealth use. About the improvement of the use of eHealth services, the interviewees mentioned the need for more discussions to address the essential parameters for remote interaction and the need for involving eHealth system managers and programmers: ... *it is crucial the collaboration between the technology and those responsible for the technology* ... (Participant 6). In addition, the need for availability of clinical and interactive data and the motivation for human involvement were mentioned.

As essential strategies, the participants considered the inclusion of medications in use, the analyzes and imaging tests, and the registration of activities of daily living [48]. The use of questionnaires, calendars and graphics of trends on the occurrence of balance disorders and falls, supported by physician and patient's records, was also considered as a closer way of managing the disease: ... *simple questions like "Have you had a fall last year?", "Was there any injury? Yes, are you afraid of falling due to this injury?* (Participant 5); ... *interactive questionnaires* ... *, for example, in the recurrent vertigo* ... *to have documented how many episodes* ... *what kind of triggers* ... (Participant 7). Warning messages for adverse effects of medications or falls were classified as beneficial. The availability of individual balance exercises with a checklist and the possibility of uploading patient videos for clinical follow-up were other issues discussed: ... *the patient could record what they are feeling, for example, eye movements and then they uploaded the recording* ... *We* (physicians) *could include some exercises on the platform* ...

The interviewees mentioned the relevance of camera and audio for monitoring of balance rehabilitation and closer interaction, especially in cases of gait assessment and depression: ... *a phone call is one thing. If there is a camera it may even allow you* (physician) *to see, for example, the patient's gait* ... (Participant 3); ... *to monitor through videos, through cameras, as long as the patient gives his consent, of course* . . . (Participant 7).

The eHealth service management by a physician was considered essential: *When there are changes* (in the health)*, the physician can also be consulted.* (Participant 3); ... *always a physician*. (Participant 7).

In addition, the security of using eHealth was discussed: ... what type of password one (physician) should use ... encrypted ... addressed to the clinical team with security code (Participant 5).

The participants also referred to the relevance of active motivation and involvement of patient and caregiver in disease prevention and control: ... *Patients cannot continue to think that the responsibility of their health belongs to the physician* ... *the patients have to be involved and responsible for their health* ... (Participant 7); ... *we often think that our elderly people do not have the ability to manage new technologies* ... *but we can have a caregiver who can contact us remotely* ... (Participant 5) Finally, the investment in medical awareness and eHealth training was highlighted, allowing better physician involvement: ... *this is a work to be done in medical education* ... *after some time of implementation* (of medical education)*, I am convinced that it* (eHealth) *will be the future of medicine* . . . (Participant 1).

#### **4. Discussion**

A mixed-methods study was performed to know how to develop an efficient eHealth service for the provision of care for elderly people with balance disorders and the risk of falling, i.e., the research problem. Our findings revealed negative experiences with EMR, contributing to a misuse of the health care system. Despite this, the highlighted relevance of eHealth in this matter is an incentive for the development of future solutions.

Unfortunately, we had a low response number in the quantitative research, as described in other studies [49,50]. This limited the comparison between the specialties and between public or private health provision groups. Despite this, we could confirm the presence of constraints regarding the data availability in Portugal. We observed medical dissatisfaction with the information systems in general. In total, 50% of the participants of our study were dissatisfied or very dissatisfied with the current usefulness and quality of IST in the context of balance disorders and the risk of falling in the elderly. The physicians were dissatisfied or very dissatisfied with available data in EMR and time spent to access and update clinical data. According to the interviews, the incomplete or not understandable information about medical consultations and the lack of integration of clinical data between the health units have contributed to the misuse of healthcare provisions by elderly people, with multiple consultations, repeated prescriptions, polypharmacy and increasing costs that could be minimized with an appropriate digital service. These limitations and consequent costs have been reported by other authors [4,12–15]. The participants pointed out the need for investments in the interoperability of health information systems and in the organization of work to overcome this situation, as previously proposed in other studies [19–21]. In fact, data are essential for healthcare provision, monitoring of population health status and decision making. To reach real-time universal data in healthcare, interoperability issues should be addressed.

As in previous studies [2,5,16], our findings also confirmed the relevance of the use of eHealth services. However, the current way of working and interacting with patients should be restructured, including dedicated time to interact digitally with patients. eHealth can be leveraged as a complementary method to provide healthcare services, including preventive monitoring of falls and telerehabilitation with evaluation and monitoring of balance diseases and falls. For the interviewed participants, the remote consultation or management should be only for follow-up consultations, and it cannot fully replace the face-to-face clinical evaluations. As mentioned by Catan et al. [34], face-to-face consultations reduce anxiety whenever people need a physician. Only one interviewed revealed skepticism about digital solutions due to the lack of eHealth clinical experience. Several studies have already revealed the influence of limited knowledge about telemedicine on the perception of the potential of eHealth [19,35].

Finally, we confirmed the need for improvement of eHealth services for a more effective healthcare provision. The qualitative research allowed exploring interventions to achieve an efficient eHealth service to support healthcare for elderly people with balance disorders and risk of falling. Several suggestions were pointed out: the inclusion of complete clinical data, the possibility for interactive communication, message alerts and remote availability of balance exercises. Camera and audio were considered essential elements for closer interaction, allowing remote viewing of the gait, as well as the balance exercises performed. All the parameters should be aligned between technology experts and physicians to design suitable technological services. eHealth services and devices should be user-friendly and suitable for both the health professional and for the patient.

For the participants, the physician emerged as the main manager of eHealth service, but not necessarily the only one [51]. As in other studies [35,37], physicians also highlighted the need for investment for confidentiality and security of data. This should always be ensured. Relatively to human resources, strategies to motivate, educate and train the elderly patient and caregiver were also discussed. Self-care and self-management of health and disease (e.g., promotion of health and prevention of disease) should be further encouraged [2,5,16]. The need for health professional awareness and training to use all of the potential of digital solutions were mentioned, including the investment in professional health education. Thus, the potential of digital health could be widely used with motivated and trained human resources [2,20]. The Portuguese health system should be adjusted to tackle aging demand, overcoming the constraints of the EHR and the lack of interoperability of the information systems. The implementation of a universal digital health coverage system supported by comprehensive digital tools with camera and audio resources can better contribute to active and healthy aging [52] with more efficient management of health care for elderly with balance disorder and risk of falling. The design and the development of a balance disorder-related remote service, with the recommended functionalities, is an opportunity worth to be explored. The strategies identified and discussed in this study will be fed into a Design Science [50] process to design and implement a future eHealth service for a more effective provision of healthcare for elderly with balance disorders at a distance.

#### *Limitations*

Regarding the participants of the web-based questionnaire, we should acknowledge possible selection biases [53] of the quantitative research. Despite the intention to recruit physicians of different specialties, a small participant size was observed as in other online studies. This made it difficult to compare the results between specialties and to know if there is a difference between the public or private health provision groups. Another consideration, already highlighted in other web-based surveys, is the age of the respondents. Most participants of this research were younger physicians who seem to have more technological resources and online interests and to be more receptive to web-based questionnaires. In addition, the higher percentage of responses from otolaryngologists can be explained by their focus of interest in inner ear diseases that can promote balance disorders. The family physicians had weak participation (only 18 out of 7451), although this specialty represented most of the eligible participants. Due to the reduced or null number of responses from geographically more remote areas (e.g., Azores), we could not analyze and, in the second phase of the mixed methods study, explore more deeply the data of these participants that could most benefit from the potential of eHealth.

Additionally, the multiple-choice questions of the questionnaire allowed an easier analysis, but this approach did not allow the inclusion and discussion of supplementary opinions of the respondents.

Relatively to the qualitative research, the sampling was intentional. The last interview was conducted via mobile phone due to the limitation of the coronavirus pandemic.

The population targeted in both studies was limited to physicians with the provision of outpatient health care to the elderly in Portugal.

All these facts limited the generalizability of the findings.

#### **5. Conclusions**

Despite significant obstacles in existing digital solutions, 76% of the Portuguese physicians included in this study considered future e-Health services as highly relevant for complementary healthcare for elderly people with balance disorders and the risk of falling. The use of eHealth services comprised of digital technologies such as cameras, sensors and audio monitoring may reinforce such solutions. Additionally, these services may represent considerable potential for reducing the excess of emergency visits, and the overlap of drug therapies and diagnostic procedures and improved treatment. This may increase both health care efficiency and quality and contribute to relieving pressure in the escalating health care costs. However, significant constraints regarding the current availability of clinical data in EHR care systems were described. The insufficient quality of both available data in EMR and in the time needed to access such data and to register new clinical data was stated. We would like to highlight that our study group's description of incomplete, or not understandable, information about medical consultations and the lack of integration of clinical data indicate serious challenges to overcome. More research about this topic is also required to further enhance the knowledge about the use of digital tools in this field of health care.

**Author Contributions:** Conceptualization, A.G.M.G. and L.V.L.; methodology, A.G.M.G. and L.V.L.; formal analysis, A.G.M.G. and L.V.L.; investigation, A.G.M.G. and L.V.L.; data curation, A.G.M.G.; writing—original draft preparation, A.G.M.G., P.E. and L.V.L.; writing—review and editing, A.G.M.G., P.E. and L.V.L.; funding acquisition, L.V.L. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was partially supported by Fundação para a Ciência e a Tecnologia (FCT) for funds to Global Health and Tropical Medicine (GHTM) (UID/04413/2020 to LVL).

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Scientific Council and the Ethics Council of Instituto de Higiene e Medicina Tropical of Universidade NOVA de Lisboa, Portugal (date of approval: 21 September 2018).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the qualitative study.

**Data Availability Statement:** The data that support the findings of this study are available on request from the corresponding author, AGMG.

**Acknowledgments:** Portuguese General Medical Council (Ordem dos Médicos), for the approval and availability of the link on the website of the Institution; Portuguese Societies and Associations who contributed to the dissemination of the link with the partners.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **Appendix A**


**Table A1.** Quantitative study: answer choices of the questionnaire.


#### **Appendix B**

**Table A2.** Qualitative study: Interview guide.


#### **References**


## *Article* **Cross Sectional E-Health Evaluation Study for Telemedicine and M-Health Approaches in Monitoring COVID-19 Patients with Chronic Obstructive Pulmonary Disease (COPD)**

**Abdullah H. Alsharif**

Department of Management Information Systems, College of Business Administration-Yanbu, Taibah University, Yanbu, Saudi Arabia; alsharifa@taibahu.edu.sa

**Abstract:** Monitoring COVID-19 patients with COPD has become one of the major tasks in preventing transmission and delivering emergency healthcare services after vaccination in case of any issues. Most COVID-19-affected patients are suggested to self-quarantine at home or in institutionalized quarantine centers. In such cases, it is essential to provide remote healthcare services. For remote healthcare monitoring, two approaches are being considered in this study, which include mHealth and Telehealth. A mixed-methods approach is adopted, where survey questionnaires are used for collecting information from 108 patients and semi-structured interviews are used with seven physicians regarding mHealth and Telehealth approaches. Survey results indicated that mHealth is rated to be slightly more effective than Telehealth, and interview results indicated that Telehealth is identified to be slightly more effective than mHealth in relation to parameters including usefulness, ease of use and learnability, interface and interaction quality, reliability, and satisfaction. However, both physicians and patients opined that both mHealth and Telehealth have a promising future with increasing adoption. Based on the findings, it can be concluded that both mHealth and Telehealth are considered to be effective in delivering remote care for COPD patients infected with COVID-19 at home. Implications of the study findings are discussed.

**Keywords:** mHealth; Telehealth; COPD; usefulness; evaluation; satisfaction

#### **1. Introduction**

COVID-19, since its identification in December 2019, has been affecting many people across the world, resulting in different waves of rising infections and deaths. As of 9 May 2021, there are 1.5 billion confirmed COVID-19 cases, including 3 million deaths reported globally [1]. The vast majority COVID-19 cases are mainly found in the West, including the Americas (63 million), followed by Europe (52 million). However, a steep rise in the number of cases was identified in the past 2 months in Southeast Asia (25 million), especially India (22 million) [1]. The number of deaths related to COVID-19 has been on the surge, with more than 570,000 reported deaths in the USA [2], 419,114 deaths in Brazil [3], and 242,362 deaths in India [4]. The rise in the number of COVID-19-related deaths were attributed to older age of patients and previous health complications such as diabetes, blood pressure, asthma and other critical health conditions which increase the risk of death among patients [5–9]. One such health condition wherein COVID-19 can have a serious impact on people is chronic obstructive pulmonary disease (COPD). It is identified that COVID-19 patients with COPD have a high risk of admission to an intensive care unit, mechanical ventilation, or death [10]. In addition, levels of angiotensin-converting enzyme 2 (ACE2), the reported host receptor of the virus responsible for COVID-19 (SARS-CoV-2), have been observed to be increased in patients with COPD [11]. Given the severity associated with COVID-19-infected COPD patients, there is increasing pressure on the health system to provide additional care. There is a need to maintain and reinforce followup and close management for these patients, with the aim of limiting collateral effects that

**Citation:** Alsharif, A.H. Cross Sectional E-Health Evaluation Study for Telemedicine and M-Health Approaches in Monitoring COVID-19 Patients with Chronic Obstructive Pulmonary Disease (COPD). *Int. J. Environ. Res. Public Health* **2021**, *18*, 8513. https://doi.org/10.3390/ ijerph18168513

Academic Editors: Marie Carmen Valenza and Irene Torres-Sanchez

Received: 25 May 2021 Accepted: 14 July 2021 Published: 12 August 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

could be induced by non-optimal management of COPD during and after the pandemic [12]. Strategies such as increased vaccination can help in preventing the mortality rate of COVID-19 infected COPD patients. However, in spite of administering over 11 billion vaccine doses globally [1], there is still a considerable global population that needs access to vaccines, especially in low- and middle-income group countries [13,14]. However, despite vaccination and other implementation of preventive and mitigation strategies such as lockdowns, increasing vaccination, etc. the new cases continue to rise globally. The risk of drug shortage due to COVID-19 restrictions is another major factor that can affect the treatment of COVID-19-infected patients with COPD and other critical respiratory conditions such as asthma, diabetes, etc. [15]. Moreover, the complications after vaccination were identified to be mild to moderate, which include pain, fever, headache, diarrhea, etc. [16,17]. Recently, black fungal infection (mucormycosis) is on the rise in Southeast Asia, resulting in a rising number of deaths and blindness among the vaccinated [18,19]. Therefore, there is an increasing need to monitor COVID-19 patients after treatment and vaccinated people, which can significantly increase the burden on healthcare systems.

Considering these factors, the best available options for managing the increased burden of managing COVID-19 patients in home quarantine and vaccinated people with additional health conditions such as COPD is to adopt effective remote monitoring of the patients' conditions and adopt effective health information management techniques [20]. Effective information management techniques including real-time monitoring, data storing, transfer, retrieval, and update, are essential for improved clinical decision-making. Timely response to chronic conditions can help in preventing mortalities and provide quality care to the patients [21,22]. It has been identified that medically necessary, time-sensitive procedures can efficiently manage resources and improve clinical decision-making for treating COVID-19 patients [23]. Advances in technology have helped in developing various monitoring systems such as smart watches and diabetes monitors that can be integrated with mobile applications and transfer real-time data to hospital servers, improving the effectiveness of remote monitoring by efficiently managing the health information of the patients. eHealth, for instance, is an emerging field at the intersection of medical informatics, public health, and business, referring to the health services and information delivered or enhanced through the internet and related technologies [24]. mHealth and Telehealth are two major eHealth approaches that are being extensively used during the COVID-19 pandemic to deliver remote healthcare services. mHealth is a medical and public health practice supported by mobile devices, such as mobile phones, patient monitoring devices, personal digital assistants (PDAs), and other wireless devices [25], whereas telemedicine is the use of medical information exchanged from one site to another via electronic communications to improve a patient's clinical health status [26]. While Telehealth is mainly used for information exchange, mHealth can be used for both information exchange and remote monitoring and diagnosis. The difference between both approaches lies in the modes of information transmission. In mHealth, the information is transferred through mobile applications, while in Telehealth, information has to be communicated by the patients to the medical representative.

In relation to chronic diseases, such as COPD, associated with COVID-19, parameters such as oxygen levels, electrocardiograms (ECGs), carbon dioxide partial pressure (PaCO2), etc. can be monitored remotely (mHealth) using sensors and devices linked to the patients mobile, which transmits information to hospitals through the mobile applications [27]. The cost of care assistance for chronic diseases such as COPD are dramatically increasing, as a result of which remote healthcare management systems such as Telehealth can be used to reduce costs and increase healthcare efficiency [28]. Therefore, remote monitoring solutions such as mHealth and Telehealth can be considered as an alternative to traditional healthcare operations, which not only decreases the burden on the already strained healthcare sector due to COVID-19, but also improves healthcare information management and delivers quality care for patients. In this context, it was observed that mHealth technologies could help in mitigating the effects of the COVID-19 pandemic [29] by increasing the reach of

local guidance for healthcare professionals in managing COVID-19 outbreak and treatment procedures [30], and in coordinating mHealth infrastructure for managing the COVID-19 pandemic [31]. Studies [32–35] have evaluated mHealth applications in relation to various parameters, including usefulness, ease of use, learnability, interface and interaction quality, reliability, satisfaction, and future use, etc., which reflected positive outcomes. Similarly, Telehealth approaches were also identified to be effective in relation to the above-mentioned parameters [36].

Both mHealth and Telehealth approaches have some advantages and drawbacks. For instance, patients at home may not regularly monitor their health condition and may not enter the health data into the mobile applications, which may lead to ineffective remote monitoring. However, the Telehealth approach may address this issue, as healthcare practitioners may call the patients and regularly collect the data and provide instant feedback. However, repeated calls from the healthcare professionals may create discomfort for the patients. However, for effective healthcare management, the need for accurate and daily health data is essential, and both mHealth and Telehealth can serve this purpose. However, there is a lack of research on comparing both approaches in remote monitoring of COPD patients infected with COVID-19 in home quarantine or vaccinated patients at home. Therefore, the purpose of this study is aimed to evaluate Telehealth and mHealth approaches in monitoring (remote monitoring of exercise tolerance, comorbidity, and smoking habits, oxygen levels, blood pressure, sugar levels, and other factors, as prescribed by the patients' respective hospitals) COVID-19 patients with COPD after the treatment and vaccination.

#### **2. Materials and Methods**

The purpose of this study is to compare and evaluate the mHealth and Telehealth approaches in monitoring COVID-19 patients with COPD after treatment and vaccination while at home. A mixed-methods approach employing both qualitative (semi-structured interviews) and quantitative (questionnaire-based survey) were adopted for collecting the data regarding mHealth and Telehealth approaches from physicians and patients, respectively, in Saudi Arabia.

#### *2.1. Questionnaire Design*

As discussed in the introduction section, there are various parameters used for evaluating both mHealth and Telehealth applications [32–36]. To cover different contexts of evaluation, Telehealth Usability Questionnaire (TUQ) is adopted in this study, including items related to various parameters (usefulness, ease of use, interface quality, interaction quality, reliability, satisfaction, and future use) from [36]. TUQ is preferred in this study because it covers wide range of parameters that are used individually in various studies [29–32], making a comprehensive list of items to be used in evaluating Telehealth applications. Furthermore, items from (MAUQ) [37], including ease of use, satisfaction, usefulness, are considered along with TUQ for developing questionnaires for mHealth and Telehealth evaluation, respectively, as shown in Appendix A. Both questionnaires (mHealth and Telehealth) have same set of questions with parameters including usefulness (three items), ease of use and learnability (three items), interface quality (three items), interaction quality (three items), reliability (three items), and satisfaction and future use (four items). Multiple-choice answers and five-point Likert scale ratings [38] were used for answering the questions by the participants. The questionnaire was initially designed in English (a copy of survey questionnaire is presented in Appendix A), which was then translated to Arabic using two professional Arabic translators. The Arabic version questionnaire was designed using QuestionPro application, and a survey link was generated for accessing the survey. A pilot study was conducted with 12 randomly selected patients from Saudi Arabia for evaluating the questionnaire. Based on the feedback from pilot study participants, few changes were made in relation to the questions' formulation and grammatical errors in Arabic. In addition, Cronbach's alpha for all items in in the questionnaire was

identified to be greater than 0.81, revealing good consistency and reliability. In addition, the interview questionnaire included eight questions reflecting the interviewees' experiences with Telehealth and mHealth, ease of use, satisfaction and future use, usefulness, learnability, interface quality, interaction quality, and reliability (as shown in Appendix B). Thus, semi-structured interviews were adopted in order to evaluate both mHealth and Telehealth from the perspectives of healthcare practitioners, and a survey instrument was adopted to evaluate both approaches from perspectives of patients, reflecting the two main actors in remote monitoring in healthcare system.

#### *2.2. Recruitment*

COPD patients diagnosed with COVID-19 and vaccinated COPD patients were recruited for the survey using the survey link generated using QuestionPro application. The survey link was forwarded to the patients through emails and other social media platforms such as Facebook and WhatsApp. The survey was conducted for a period of 5 weeks from 15 April to 20 May 2021. Physicians were contacted through emails and over phone, requesting them to participate in the interviews. Interviews were scheduled from 16 April to 5 May 2021, and were conducted online. On average, each interview lasted for 35 min.

#### *2.3. Sampling*

Considering the purpose and objective of the study, which is to collect the data from a specific group of population (COPD patients diagnosed with COVID-19 and vaccinated COPD patients at home involved in remote monitoring), a purposive sampling approach was adopted [39]. Accordingly, a nonprobability sample was obtained based on the objective of the study, which mainly focused on analyzing the patients' and physicians' perceptions of mHealth and Telehealth. The survey link was initially forwarded 142 patients using various online channels. A total of 108 students participated in the survey, reflecting a response rate of 76.05%. While 19 physicians were contacted for semi-structured interviews, 7 accepted invitations and took part in the interviewees. Low response rate in interviews is due to the busy schedules of the physicians owing to the COVID-19 outbreak.

#### *2.4. Data Analysis*

The survey was developed using Google forms and conducted for a period of 5 weeks. Both survey and interview data are analyzed and discussed using nine themes, which included experiences, usefulness, ease of use, learnability, interface quality, interaction quality, reliability, satisfaction, and future use. In relation to survey data, relative frequencies for each item under these themes and statistical tests (*t*-tests) are used for analyzing the data, which are presented in the following section.

#### **3. Results and Discussion**

#### *3.1. Survey Results*

The final sample achieved for the study was 108. The demographic information of the participants is presented in Table 1. Among the total participants, 58.3% were male (63/108), and 41.7% were female (45/108). Considering the age groups, 43.5% were aged between 35 and 44 years (47/108), followed by 30.6% between 25 and 34 years (114/479), 12% between 18 and 24 years (13/108), 12% between 45 and 54 years (13/108), and 1.9% participants aged 55 or more than 55 years (2/108). Focusing on the education of the participants, 38.9% were bachelor's degree graduates (42/108), followed by 30.6% master's graduates (33/108), 15.7% high school graduates or diploma graduates (17/108), and 14.8% doctorates (16/108). Demographics of the participants reflected good participation levels by both genders. Moreover, the majority of the participants were aged between 25 and 44 years, reflecting the population who is better equipped with the skills of using health information technologies [40–42], and has good education levels.


**Table 1.** Frequency distribution of demographic variables.

All the participants were having an experience of using both Telehealth and mHealth approaches. In relation to the experience of using Telehealth (via telephone/mobile), 65.7% of the participants had 2 or less than 2 years of experience (71/108), followed by 17.6% having 2 to 5 years of experience (19/108), 9.3% having 10 or more years of experience (10/108), and 7.4% having 5 to 10 years of experience (8/108). In relation to the experience of using mHealth (via mobile application/smart sensors), 63.9% of the participants had 2 or less than 2 years of experience (69/108), followed by 25.9% having 2 to 5 years of experience (28/108), 6.5% having 10 or more years of experience (7/108), and 3.7% having 5 to 10 years of experience (4/108). The experience levels of the participants in relation to mHealth and Telehealth reflected almost similar statistics, with the majority of them having 2 years or less and 2 to 5 years of experience.

Participants' opinions on the usefulness of both mHealth and Telehealth are presented in Table 2. Improved access to healthcare was identified to be a highly rated factor related to usefulness, followed by the time-saving factor, and approaches meeting the healthcare needs of the users. Both approaches were identified to be similarly rated by the participants in relation to the usefulness parameter.


**Table 2.** Usefulness of mHealth and Telehealth.

Furthermore, to identify the differences of opinions in relation to mHealth and Telehealth, a *t*-test was conducted, as shown in Table 3. The mean scores of mHealth (Mean = 3.64, SD = 1.12) and Telehealth (Mean = 3.61, SD = 1.08), identified in the analysis, reflected that participants found that both mHealth and Telehealth approaches to be effective in terms of usefulness. *t*-value, as shown in Table 3, was found to be (*t* = 0.2004) at 0.05 confidence interval, and was identified as not statistically significant (*p* > 0.05). Therefore, no significant differences of opinions in relation to the usefulness of mHealth and Telehealth can be observed. The findings are similar to [32], reflecting the usefulness of both approaches. As both approaches are aimed at improving access to healthcare and

save time in accessing healthcare needs, it is possible that both approaches are identified to be useful by the participants.


**Table 3.** Difference in usefulness of mHealth and Telehealth.

Participants' opinions on the ease of use and learnability parameters are presented in Table 4. It can be observed that simple to use and easy to learn factors of mHealth are slightly greater than that of Telehealth; no major differences were identified in relation to the ability of the approach for enhancing productivity. While the mHealth application is mobile-based, and easy to use, Telehealth completely relies on calls in providing care and monitoring health information. Therefore, slight differences in terms of ease of use and learnability factors can be expected.

**Table 4.** Ease of use and learnability of mHealth and Telehealth.


Furthermore, to identify the differences of opinions in relation to mHealth and Telehealth, a *t*-test was conducted, as shown in Table 5. The mean scores of mHealth (Mean = 3.72, SD = 1.12) and Telehealth (Mean = 3.69, SD = 1.1), identified in the analysis, reflected that participants found that both mHealth and Telehealth approaches to be effective in terms of ease of use and learnability. *t*-value, as shown in Table 5, was found to be (*t* = 0.1986) at 0.05 confidence interval, and was identified as not statistically significant (*p* > 0.05). Therefore, no significant differences of opinions in relation to ease of use and learnability parameters of mHealth and Telehealth can be observed. These findings can be related to [33], in which ease of use and learnability were rated to be effective by the majority of the participants.



While mHealth applications have a mobile interface, Telehealth does not have any physical interface, but the quality of the interface can be identified from the communication between patients and healthcare practitioners over telephone or mobiles. In relation to the interface quality, both mHealth and Telehealth approaches were rated slightly above average (Mean = 2.5), as shown in Table 6. It is interesting to note that only 50.9% of the participants either strongly agreed or agreed that the Telehealth approach meets their healthcare needs, and only 52.7% of the participants either strongly agreed or agreed that the mHealth approach meets their healthcare needs, reflecting that there is a considerable number of participants who are not happy with interface quality in both approaches.


**Table 6.** Interface quality of mHealth and Telehealth.

Furthermore, to identify the differences of opinions in relation to mHealth and Telehealth, a *t*-test was conducted, as shown in Table 7. The mean scores of mHealth (Mean = 3.42, SD = 1.05) and Telehealth (Mean = 3.43, SD = 1.06), identified in the analysis, reflected that participants found that both mHealth and Telehealth approaches to be effective in terms of interface quality. *t*-value, as shown in Table 7, was found to be (*t* = 0.0697) at 0.05 confidence interval, and was identified as not statistically significant (*p* > 0.05). Therefore, no significant differences of opinions in relation to interface quality parameters of mHealth and Telehealth can be observed. These findings related to mHealth are similar to [34], indicating good interface quality of mHealth applications.

**Table 7.** Difference in interface quality of mHealth and Telehealth.


In relation to interaction quality, no significant differences were identified between mHealth and Telehealth, as identified from Table 8. In relation to the ability of the approaches reflecting in similar to personal interactions, less than 50% of the participants reflected the opinion that these approaches are similar to personal interactions. Moreover, more than 50% of the participants stated that they are not able to express their opinions effectively on mHealth and Telehealth applications.

**Table 8.** Interaction quality of mHealth and Telehealth.


Furthermore, to identify the differences of opinions in relation to mHealth and Telehealth, a *t*-test was conducted, as shown in Table 9. The mean scores of mHealth (Mean = 3.41, SD = 1.05) and Telehealth (Mean = 3.40, SD = 0.99), identified in the analysis, reflected that participants found that both mHealth and Telehealth approaches to be effective in terms of interaction quality. *t*-value, as shown in Table 9, was found to be (*t* = 0.0720) at 0.05 confidence interval, and was identified as not statistically significant (*p* > 0.05). Therefore, no significant differences of opinions in relation to interface quality parameters of mHealth and Telehealth can be observed. These findings related to mHealth are similar to [33,34], indicating average interaction quality of mHealth and Telehealth approaches.


**Table 9.** Difference in interaction quality of mHealth and Telehealth.

In relation to reliability (Table 10), the mHealth approach was rated as slightly better than the Telehealth approach, stating that the approach was similar to hospital visits in delivering the care, and also in the ability to fix issues by receiving messages through the application (in comparison to messages received through calls). The majority of the participants (>50%) were identified to be neutral (neither agree nor disagree) in relation to all the factors listed in the reliability parameter.

**Table 10.** Reliability of mHealth and Telehealth.


Furthermore, to identify the differences of opinions in relation to mHealth and Telehealth, a *t*-test was conducted, as shown in Table 11. The mean scores of mHealth (Mean = 3.27, SD = 1.03) and Telehealth (Mean = 3.08, SD = 1.05), identified in the analysis, reflected that participants found that both mHealth and Telehealth approaches to be effective in terms of reliability. *t*-value, as shown in Table 11, was found to be (*t* = 1.3424) at 0.05 confidence interval, and was identified as not statistically significant (*p* > 0.05). Therefore, no significant differences of opinions in relation to the reliability parameter of mHealth and Telehealth can be observed. Considering the less acceptance of eHealth in Saudi Arabia, due to various factors of influence, it may be possible that low responses were identified in relation to reliability factor, which can be compared to the findings in [43].

**Table 11.** Difference in Reliability of mHealth and Telehealth.


In relation to satisfaction (Table 12), it can be identified that participants were slightly more satisfied with mHealth compared to Telehealth across all the factors related to satisfaction and future use. Considering the overall satisfaction, there is no difference of opinions expressed in relation to both approaches. However, while 60% of the participants agreed that they would use mHealth in the future, 64% of the participants stated they would use Telehealth in the future, indicating a slightly more preference towards Telehealth over mHealth.

**Table 12.** Satisfaction and future use of mHealth and Telehealth.


Furthermore, to identify the differences of opinions in relation to mHealth and Telehealth, a *t*-test was conducted, as shown in Table 11. The mean scores of mHealth (Mean = 3.63, SD = 1.06) and Telehealth (Mean = 3.53, SD = 1.04), identified in the analysis, reflected that participants found that both mHealth and Telehealth approaches to be effective in terms of satisfaction and future use. *t*-value, as shown in Table 13, was found to be (*t* = 0.6998) at 0.05 confidence interval, and was identified as not statistically significant (*p* > 0.05). Therefore, no significant differences of opinions in relation to the reliability parameter of mHealth and Telehealth can be observed.


**Table 13.** Difference in satisfaction and future use of mHealth and Telehealth.

These findings may be compared to [31,32,40] in relation to satisfaction levels. Moreover, preference over these approaches may be influenced by the recent COVID-19 outbreak, which has led to increased adoption of eHealth approaches due to the surge in COVID-19 cases and preventive measures such as lockdowns and curfews. However, lack of reliability as identified in [43] can be one of the reasons for leaning more towards Telehealth rather than mHealth.

#### *3.2. Interview Results*

A total of seven healthcare practitioners were interviewed, and all of them were males. Among them, three participants belonged to the age group of 35–44 years; another three in 45–54 years; and one participant in 25–34 years. Four participants were general physicians, one participant was a dentist, one was a surgery specialist, and another was a medical specialist. Three participants had experience of 2 or less years in using mHealth and Telehealth approaches, two had an experience of 2 to 5 years, and another two had an experience of 5 to 10 years. The participants' experience levels and roles reflect a good sample for collecting the information about mHealth and Telehealth.

Focusing on the opinions expressed in relation to Telehealth, all the participants identified good usefulness levels for Telehealth. One of the interviewees identified it to be an easier approach to reach patients without any difficulty or requirement to learn new technologies, reflecting the edge over mHealth. Another interviewee identified Telehealth to be an effective approach in providing distant care. In relation to mHealth, only one participant identified its usefulness to be poor, while the rest of them indicated good levels of usefulness. One of the interviewees identified it to be very useful, providing real-time information at any time. The findings in relation to usefulness indicated that the majority of the participants identified both approaches to be of good usefulness. However, Telehealth was identified to be slightly more useful compared to mHealth.

In relation to ease of use, all the interviewees mentioned Telehealth to be very easy to use and learn. However, one of the interviewees mentioned that additional training relating to compliances and standards is required in using Telehealth. One of the interviewees mentioned that Telehealth may not be easy to learn in the beginning, but as handling of it improves, one can learn effectively. Similarly, all the interviewees identified mHealth to be easy to use, with little knowledge of computers and technology. However, one of the interviewees mentioned that experience is required in using mHealth, and another interviewee mentioned that mHealth is hard to use. Another interviewee mentioned that mHealth may be effective for learning for only those who use mobiles more frequently, and have experience of using applications. The overall analysis of responses indicated that Telehealth was rated slightly more than mHealth in terms of ease of use and learnability.

In relation to interface and interaction quality, all the interviewees reflected Telehealth to be good. Similarly, in relation to mHealth, only one interviewee identified it with poor

interface and interaction quality. Using Telehealth was identified to be effective because interaction using Telehealth takes comparatively less time than visits, which can save time; while using mHealth, it was indicated that a lot of time could be saved as it is one-way messaging. Both approaches indicated time-saving as an outcome of effective interface and interaction quality. However, Telehealth was slightly rated more than mHealth.

Focusing on the reliability parameter, all the interviewees indicated Telehealth to be reliable but extended their statement that reliability may depend on many factors, including the physician and patient and the technology used to connect them. While Telehealth mainly relies on calls, it is sometimes possible that internet technologies such as voice over internet protocol or applications such as skype or zoom may be used, which may raise concerns over security and privacy. However, in relation to mHealth, five interviewees stated it to be reliable. One of the interviewees stated that more research is needed to assess its reliability, while others raised privacy and security concerns over mHealth.

Findings relating to usefulness, ease of use, learnability, interface and interaction quality, and reliability from interviewees' perspectives reflected a slightly greater preference towards Telehealth compared to mHealth in contrast to patients' perspectives. The differences of opinions among the participants may be related to their experience and understanding of these approaches, and also the features and design of the applications they have been using, which can influence their perspectives [44–46].

In relation to the satisfaction parameter, all the interviewees reflected good satisfaction levels about the Telehealth approach. However, focusing on the mHealth approach, one interviewee stated moderate satisfaction and another interviewee stated poor satisfaction. Findings reflected that the interviewees are slightly more satisfied with Telehealth compared to mHealth. These results regarding satisfaction contrasted with survey results, where participants identified with being slightly more satisfied with mHealth compared to Telehealth, supporting findings from [47–50]. Furthermore, in relation to future use prospects, Telehealth was identified to be a promising approach in reducing clinical visits, improving quality healthcare. Similarly, mHealth was also identified to be having a promising future where electronic health records can be integrated with daily monitoring systems, providing 24 × 7 remote healthcare services which can significantly improve effectiveness and efficiency of care. The results have indicated that both mHealth and Telehealth would be increasingly adopted in the future, similar to survey results.

#### **4. Conclusions**

This study has compared and evaluated mHealth approaches in remote monitoring of COPD patients diagnosed with COVID-19. The importance of this study arises from the rising complications and effects of COVID-19 after the recovery and during the home quarantine, associated with the additional complications of COPD condition. To evaluate the approaches, both patients' and physicians' perspectives are considered. The findings have indicated that patients' views were in contrast to physicians' views. While patients leaned towards mHealth, physicians leaned towards Telehealth. However, both approaches were identified to be effective in terms of their usefulness, ease of use and learnability, interface and interaction quality, reliability, satisfaction, and future use, with minor differences. This study has few limitations. As different patients use different mHealth applications and adopt different practices in Telehealth, there could be a certain bias in the results. In addition, the number of participants in both survey and interviews was lower due to the impact of the COVID-19 outbreak. Therefore, generalizations should be made with care. This study has both theoretical and practical implications. First, this study addresses the gaps in the literature in evaluating the remote monitoring approaches in the context of COVID-19 pandemic. Second, the findings can be used to improve the mHealth and Telehealth approaches in relation to the needs of the patients and physicians. Moreover, as the study has been conducted in Saudi Arabia, the findings can only be compared to the population with similar demographics. Therefore, future research may focus on

the evaluation of different remote monitoring approaches in different regions reflecting varying demographics.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The author declares no conflict of interest.

#### **Appendix A**

#### **Evaluating Telehealth and mHealth approaches in monitoring the COVID-19 patients with chronic obstructive pulmonary disease (COPD): Survey Questionnaire (Telehealth)**




#### **Appendix B**

**Evaluating Telehealth and mHealth approaches in monitoring COVID-19 patients with chronic obstructive pulmonary disease (COPD): Survey Questionnaire (mHealth)**





#### **Appendix C**

#### **Interview Questionnaire Telehealth**


#### **mHealth**


#### **References**


## *Article* **Description of e-Health Initiatives to Reduce Chronic Non-Communicable Disease Burden on Brazilian Health System**

**Daniela Laranja Gomes Rodrigues 1,\*, Gisele Silvestre Belber 1, Igor da Costa Borysow 1, Marcos Aurelio Maeyama <sup>2</sup> and Ana Paula Neves Marques de Pinho <sup>1</sup>**

> <sup>1</sup> Social Responsibility Department, Hospital Alemão Oswaldo Cruz, São Paulo 01323-020, Brazil; gsbelber@haoc.com.br (G.S.B.); iborysow@haoc.com.br (I.d.C.B.); ampinho@haoc.com.br (A.P.N.M.d.P.)

<sup>2</sup> Telehealth Center, Florianópolis 88040-900, Brazil; marcos.aurelio@univali.br

**\*** Correspondence: dlrodrigues@haoc.com.br; Tel.: +55-11-985421342

**Abstract:** Chronic non-communicable diseases (NCD) account for 72% of the causes of death in Brazil. In 2013, 54 million Brazilians reported having at least one NCD. The implementation of e-Health in the Unified Health System (SUS) could fill gaps in access to health in primary health care (PHC). Objective: to demonstrate telehealth strategies carried out within the scope of the Institutional Development Support Program of the Unified Health System (PROADI-SUS) and developed by Hospital Alemão Oswaldo Cruz, between 2018 and 2021, on evaluation, supply, and problemsolving capacity for patients with NCDs. Methodology: a prospective and descriptive study of three projects in the telehealth areas, using document analysis. The Brasil Redes project used availability, implementation, and cost-effectiveness analysis, TELEconsulta Diabetes is a randomized clinical trial, and Regula Mais Brasil is focused on the waiting list for regulation of specialties. All those strategies were developed within the scope of the SUS. Results: 161 patients were attended by endocrinology teleconsultation in one project and another two research projects, one evaluating Brazil's Telehealth Network Program, and another evaluating effectiveness and safety of teleconsultation in patients with diabetes mellitus referred from primary care to specialized care in SUS. Despite the discrepancy in the provision of telehealth services in the country, there was an increase in access to specialized care on the three projects and especially on the Regula Mais Brasil Collaborative project; we observed a reduction on waiting time and favored distance education processes. Conclusion: the three projects offered subsidies for decision-making by the Ministry of Health in e-Health and two developed technologies that could be incorporated into SUS.

**Keywords:** chronic non-communicable diseases; unified health system; primary health care; Brazil

#### **1. Introduction**

With the change in the epidemiological and demographic profile of the population, in addition to the growing increase in chronic pathologies and people in vulnerable situations, the need for an integrated approach to the individual concerning health care emerges [1–3]. Chronic non-communicable diseases (CNCD) are described as diseases that involve the circulatory system, diabetes, cancer, and chronic respiratory diseases. This group of diseases usually occurs more commonly in developing countries, accounting for 63% of the causes of death in the world [4,5] and corresponding to one-third of deaths in people under 60 years of age [6]. The last national health survey conducted by the Brazilian government in 2013 [7] showed that 72% of the causes of death were related to CNCD, that is, by diseases that could have their impact reduced if health strategies focused mainly on primary care were implemented.

Digital health, or e-Health, can be defined as "the safe and cost-effective use of information and communication technologies in support of health and health-related fields",

**Citation:** Rodrigues, D.L.G.; Belber, G.S.; Borysow, I.d.C.; Maeyama, M.A.; Pinho, A.P.N.M.d. Description of e-Health Initiatives to Reduce Chronic Non-Communicable Disease Burden on Brazilian Health System. *Int. J. Environ. Res. Public Health* **2021**, *18*, 10218. https://doi.org/10.3390/ ijerph181910218

Academic Editors: Irene Torres-Sanchez and Marie Carmen Valenza

Received: 31 August 2021 Accepted: 15 September 2021 Published: 28 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

with the World Health Organization considering making e-Health a global priority for health systems development [8]. The implementation of e-Health can substantially contribute to filling some gaps in the performance of primary health care (PHC), especially in developing countries such as Brazil [9]. One of these strategies is to expand the use of telemedicine to provide diagnostic and consulting support to professionals in distant locations, supporting the physician in such locations and improving the skills and knowledge of remote healthcare providers [10].

The distribution of income and access of the Brazilian population to technological resources is characterized by geographical and social inequality. Even so, the report "Digital 2020", published by *We Are Social* in January 2020, identified 150.4 million internet users in Brazil [11], accounting for an increase of 6.0% (8.5 million) when compared to the same month of the previous year, and by June 2020 there were identified 234 million active smartphones, a number that has been larger than the Brazilian population since 2017, according to the 31st edition of the Getúlio Vargas Foundation (FGV) Annual Research on the IT Usage [12]. Considering the public health area, there is a partnership between the Ministry of Health and the Ministry of Science, Technology, and Communications to provide internet access in 100% of health care units in Brazil [13]. That way, we can recognize that there is an increasingly favorable context for the wide deployment of telemedicine in the country. Despite this, there are still challenges and important aspects to be considered, for example, a Brazilian resolution that allows greater use of telehealth strategies throughout the country [14,15].

This article aims to describe telehealth strategies carried out in the years 2018 to 2021 by the Hospital Alemão Oswaldo Cruz, in partnership with the Brazilian Ministry of Health through the Support Program for Institutional Development of the Unified Health System (PROADI-SUS) [16,17].

#### **2. Materials and Methods**

Between 2018 and 2021, Hospital Alemão Oswaldo Cruz, in partnership with the Ministry of Health, through PROADI-SUS, developed projects and actions in Telehealth aimed at improving management in various spheres of the SUS. Of these, we review three strategies aimed at reducing the impact of CNCD on health using different strategies in e-Health for this purpose. As these are three different projects, we will describe the methodology of each one separately.

#### *2.1. Brasil Redes Project*

The Brasil Redes Project aimed to carry out a diagnostic evaluation of the National Telehealth Brazil Networks Program (PNTBR–in Portuguese), which completed 10 years of existence in 2017 [18]. This program aims to promote strategies to support primary health care through distance communication actions that are carried out by telehealth centers. As of 2011, the main activities to be carried out by the centers were as defined: teleconsulting, telediagnosis, tele-education, and training second opinion [19].

Five studies were carried out for this evaluation: evaluability study, implementation evaluation, and cost-effectiveness evaluation of two services offered in the list of activities foreseen by the program and program cost analysis [20]. The evaluability study was the first stage, aiming to identify the resources, the stakeholders, the processes carried out, the evaluative questions, and the expectations regarding the results of the referred program. This study included data collection work between the months of August and December 2018. The second study was the implementation evaluation, which had two major stages: socio-historical analysis of the genesis of the program and the implementation evaluation itself. The economic evaluations addressed the teledermatology service of the Telehealth Center of Santa Catarina [21] and the telecardiology service of the Telehealth Center of the Hospital das Clínicas of the Federal University of Minas Gerais [22] (Figure 1). The last study evaluated the unit value of the electrocardiogram and Holter report to be implemented as a procedure made available to the population by the Unified Health System.

**Figure 1.** Brazil's states with telehealth initiatives included in the current article.

Table 1 outlines a systematization of the five studies carried out.

#### *2.2. Regula Mais Brasil Colaborativo*

Regula Mais Brasil Collaborative [17] is a project developed to qualify the outpatient care regulation process (evaluation of referrals to specialized care) using telehealth technologies, with an evaluation of cases in regulation, use of access protocols to specialties, and teleconsulting support for case resolution. Furthermore, this project monitors the waiting period process, where the patient is registered in the online system where he awaits the emergence of a vacancy for the requested specialty. Usually, this waiting time is 1 to 2 years.

Motivated by the COVID-19 pandemic [15], the Ministry of Health of Brazil authorized teleconsultations to face the pandemic, which had been prohibited in Brazil until then, and invited the Hospitals to participate in the Regula Mais Brasil Collaborative to offer teleconsultations, thus expanding the scope of the project. The purpose of this offer was to reduce the risk of transmission of the virus, both in displacement and in the environment of health services, to avoid contact with possible contaminants, and to maintain the care of acute and chronic diseases in an off-site manner. In addition, the importance of this offer was enhanced by the closing of several elective outpatient care services at the beginning of the pandemic, which could lead to the worsening of cases due to a lack of assistance.

#### *2.3. Teleconsulta Diabetes*

This project's main objective, until in recruitment phase, is to conduct clinical research to test the hypothesis that teleconsultation is non-inferior to the face-to-face care of patients with diabetes mellitus (DM) type II referred from primary health care (PHC) to specialized care in SUS [23].

This is a randomized, pragmatic, phase-2, single-center, open, non-inferiority clinical trial, with central randomization, allocation confidentiality, and data analyst blinding, to assess the efficacy and safety of specialized care by teleconsultation compared to face-toface care. A total of 250 participants of both sexes over 18 years old, with type 2 diabetes mellitus, will be included. This sample size will allow evaluating the non-inferiority of up to 0.5% between groups, assuming a standard deviation of 1.30.


The outcomes that will be analyzed in this study are: the mean difference in the percentage of glycated hemoglobin (HbA1c) post- and pre-intervention in patients diagnosed with type 2 DM in 3 and 6 months, fasting blood glucose measurements and blood count, measurements of serum urea and creatinine, lipid profile measurements, systolic pressure measurements, in-office measurements, in-office diastolic pressure measurements, measurements of body weight and body mass index (BMI), the incidence of any adverse events, and quality measurements of patients' lives using the diabetes quality of life measurement questionnaire (DQOL) [24].

In the micro-costing analysis [25], this study performs the definition of the consumption time equations of each resource per product/service and extrapolation of the findings to define the productive capacities of the basic health units (UBS–in Portuguese) through real care data by the flow of regulation in the city of Joinville.

#### **3. Results**

The main target audience of the Telessaúde Brasil Redes National Program is primary health care teams, and with that many of the actions offered by the centers were focused on caring for people with CNCD. In this sense, the project results highlighted opportunities for improvement of the telehealth program for CNCD at the federal level. In Brazil's public health system, there is a necessity to homogenize the financing values considering the production potential of the centers, mainly for telediagnosis, and monitoring and evaluation methods of results. For the spheres of state centers aiming to value their actions, there is a need to reflect on challenges in the use of these services by health workers and on regional partnerships. The set of methods offered a broad vision of the program and articulated more detailed analyses of certain services offered by the telehealth centers. The purpose was to offer subsidies to the Ministry of Health in improving the telehealth program and using the expertise of certain centers as inspiring the formulation of procedures used to be offered by SUS.

Since the creation of the Brazil Redes program in 2007, 33 telehealth centers have been identified in various regions of the country (Figure 1), with very diversified health access strategies, as described in Table 2.


**Table 2.** Contributions of the evaluative studies of the National Telehealth Brazil Networks Program to MS decision-making, with a focus on CNCD.


In this sense, the centers sought to articulate different fronts of activities to contemplate permanent education and the improvement of primary care professionals, whether through courses, virtual lectures, teleconsulting, production of training second opinions, and telediagnosis. Regarding the processes evaluated in the economic component, the two studies carried out with the Telehealth Center at Clinic's Hospital from the Federal University of Minas Gerais clarify the cost and budget impact forecast for the implementation of telecardiology services in the public service network, which can greatly contribute to the improvement of care for people with CNCD.

The Hospital Alemão Oswaldo Cruz carried out consolations in the Regula Mais Brasil Collaborative Project via teleconsultation of users, in the specialties of neurology, endocrinology, orthopedics, and mental health, totaling 1097 teleconsultations in this period. Specific actions for CNCDs are concentrated in the specialty of endocrinology, in which 161 teleconsultations were carried out, all with SUS users in the city of Recife, city located in the state of Pernambuco, in northeastern Brazil (Figure 1). The results obtained in this project are displayed in Table 3.

Depending on the prevalence of endocrinological diseases, diabetes mellitus appears as the most prevalent reason for teleconsultations performed. Regarding the technology used for contact between the doctor and the patient, video calling (audio and video) was mostly used, since it represents the best possibility of remote interaction. The exclusive audio resource was only used when it was not possible to use video calling.


**Table 3.** Results of Regula Mais Brazil Project.

The outcomes of the last teleconsultation were performed to allow an assessment of the resolvability potential of this assistive technology. The follow-up outcomes in PHC (40.3%) represent cases in which the specialist, based on the assessment in teleconsultation, considers that, based on the complexity of the case, the patient can be followed up in the PHC, close to their residence, avoiding unnecessary referral and displacement of the patient to the specialized center. In addition, he receives the first guidance and conduct by the specialist via teleconsultation. Likewise, the reassessment in teleconsultations (23.4%) also represents the cases in which digital technology allows for adequate follow-up with a specialist, without the need to travel. On the other hand, cases in which the outcomes indicate the need for a face-to-face consultation with the specialist, whether immediate or elective, represent the limitation of teleconsultation in the assessment or follow-up of cases. It is worth mentioning that all cases evaluated did not have a first face-to-face consultation which, in some cases, limits a better diagnosis.

What corroborates the potential of teleconsultations is the result of the Net Promoter Score [26], which assesses user satisfaction concerning the service, and which, on a scale from 0 to 100, presented an index of 93 points in teleconsultations in endocrinology carried out and evaluated by the project patients.

Regarding the results of the teleconsultation diabetes research project, it was not yet possible to demonstrate aspects of the research results related to the results of complementary exams, questionnaires measuring the quality of life in diabetes (DQOL), and satisfaction assessment with telemedicine, due to the small number of participants included so far. Of the patients included in the study, the mean age was 59.4 years, with a predominance of 78% of white people, followed by black (12%) and mixed race (10%), and a slight predominance of males (54%) of the included participants.

In addition, within the scope of the teleconsultation project, the time-driven activitybased costing (TDABC) micro-costing methodology, or costs based on time and activity, was adopted because it allows the identification of the unit cost of the service within the expected efficiency conditions, also acting as a comparison metric. In this scenario, the first hypothesis considered the duration of the teleconsultation and the face-to-face consultation to be identical, as well as the duration of the pre-consultation activities. This is a preliminary conclusion, considering that recruitment, until this period, was minimal and with low statistical power to detect differences in times. The second hypothesis considered that the costs with materials and structure of the UBS's and the Polyclinic are also identical since the Health Department of Joinville is in the middle of the process of changing the

methodology for this type of cost and must provide these data in a manner more accurate by mid-2021. In this hypothetical scenario, the cost of teleconsultation was 5% higher than that of a face-to-face consultation. As in both cases, the consultation is responsible for more than 90% of the total cost, the difference in duration between the two types of consultation will be the determining factor for the difference in costs between them. More details that relate the methodologic and statistic description are outlined in the design paper published before [23].

#### **4. Discussion**

Teleconsultation is a medical act and must abide by the Brazilian Medical Code of Ethics (MCE) [27–29]. As provided in Article 37 of the MCE, the physician is prohibited from "prescribing treatment or other procedures without direct examination of the patient, except in cases of urgency or emergency and proven impossibility of performing it, and, in such circumstances, must do so immediately after ceasing the impediment" [30]. The State of Emergency in Public Health of National Importance (ESPIN) [31] triggered during the SARS-CoV-2 pandemic fits into this prerogative, thus supporting teleconsultations during this exceptional period where the Brazilian society had the opportunity to experience remote medical care more extensively.

Critics of telemedicine approval warn that the possible overuse and careless use of telemedicine could turn doctors into "telemarketing operators", which could lead to poor quality in appointments, medical errors, and unemployment by reducing the number of face-to-face doctors [32]. Although the movements of criticism to teleconsultation caused a fanfare in the mainstream media, they do not seem to reproduce what the Brazilian medical professional wants. A survey carried out in February 2020 by the São Paulo Physicians Association (APM) with over 2200 physicians from 55 different specialties revealed that 64.39% of doctors wanted a regulation that would allow the expansion of services and assistance to the population, including direct doctor-to-patient teleconsultation [33].

This whole context evidences the lack of alignment between the federal government, the Brazilian Medical Federal Council, health plan operators, medical associations, and medical professionals, especially concerning teleconsultations. The challenge is how to expand access to medical services mainly to specialists for populations in remote regions, reduce healthcare costs and the displacement of patients, and on the other hand, minimize the fear of damage to the medical profession [27]. Other challenges now faced by specialists are data insecurity, trivialization of teleconsultations, and the production of misdiagnoses and prescriptions, in addition to avoiding possible unemployment of doctors. The country should take advantage of the current situation and promote a wider discussion on the benefits and limitations of permanent and full permission to use telemedicine, bringing to the agenda socioeconomic, cultural, and technological issues.

The Brazil Redes program showed the presence of decentralized and disseminated strategies throughout the national territory, with actions in telehealth, focused not only on CNCDs but also other health promotion strategies. The actions of teleophthalmology with identification of diabetic retinopathy based in Mato Grosso [34,35] confirm the efficiency and potential of supporting the health of the population using the Unified Health System if the use of similar strategies were more widespread in the country. They also serve as a model as strategies to be implemented in other developing countries.

The Regula Mais Brasil project, through the implementation of teleconsultation in the Unified Health System, demonstrated, in an unprecedented way, how strategies that allow the population's access to different medical specialties can be feasible, in addition to allowing the capillarization of care, through the strategy to take the specialist wherever the patient is and not the other way around. Especially in the period of the SARS-CoV-2 pandemic, this project can maintain access to health care, especially for users with diabetes and sequelae after a stroke, ensuring the maintenance of care [36].

Regarding the teleconsultation diabetes research project, if the efficacy and safety for patients assisted by teleconsultation are confirmed, once the normative issues are resolved by the Federal Council of Medicine (CFM), this will be able to effectively contribute to the promotion access of patients to the public health system, including specialist physicians. Likewise, there may be an increase in the resolvability of the population's health needs, breaking the geographical barrier that a country with continental dimensions, such as Brazil, imposes on the provision and standardization of health, in addition to the potential savings for health systems, which may be used safely and with quality.

This paper has several limitations: one of then is the fact we make a compilation of three different strategies implemented at SUS in a tentative way to delivery medical assistance to the general population. As the initial projects were not designed to analyze the impact on outcomes, for example, the incidence of myocardial infarct or stroke, the authors only could initiate a discussion, showing that e-Health strategies can be performed in a public health system.

#### **5. Conclusions**

Although there is evidence in favor of the use of telehealth strategies to deal with CNCDs, in Brazil, due to difficulties in accessing technology in addition to important care gaps and legal impediments, we observe a lack of strategies in this area for health promotion within the scope of the SUS.

Therefore, given the above, there is a need to generate reliable scientific evidence of the effectiveness and safety of applicability within the context of regulation and access to the SUS. In this article, the authors demonstrate how specific initiatives in telehealth, through partnerships between the Ministry of Health and Hospitals of Excellence, can foster the development of new research and assistance strategies within the scope of e-Health in Brazil.

**Author Contributions:** Conceptualization, D.L.G.R.; methodology, D.L.G.R., G.S.B., I.d.C.B. and M.A.M.; validation, D.L.G.R., G.S.B., I.d.C.B., M.A.M. and A.P.N.M.d.P.; formal analysis, D.L.G.R., G.S.B., I.d.C.B. and M.A.M.; investigation, D.L.G.R., G.S.B., I.d.C.B. and M.A.M.; resources, A.P.N.M.d.P.; data curation, D.L.G.R., G.S.B., I.d.C.B. and M.A.M.; writing—original draft preparation, D.L.G.R., G.S.B., I.d.C.B. and M.A.M.; writing—review and editing, D.L.G.R., G.S.B., I.d.C.B., M.A.M. and A.P.N.M.d.P.; supervision, D.L.G.R.; funding acquisition, A.P.N.M.d.P. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Brazil's Ministry of Health, by the Institutional Development Support Program of the Unified Health System (PROADI-SUS).

**Institutional Review Board Statement:** The study TELEconsulta Diabetes was conducted according to the guidelines of the ResNo466 of the National Health Council, is the basis for the definition of ethical precepts in research involving humans in Brazil, since we are not signatories to the Declaration of Helsinki and approved by the Ethics Committee of (CAAE: 03434218.1.0000.0070, protocol code 3.623.207 and approval on 5 October 2019).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Conflicts of Interest:** The authors declare no conflict of interest and the funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


### *Article* **An eCoach-Pain for Patients with Chronic Musculoskeletal Pain in Interdisciplinary Primary Care: A Feasibility Study**

**Cynthia Lamper 1,\*, Ivan Huijnen 1,2, Maria de Mooij 1, Albère Köke 1,2, Jeanine Verbunt 1,2 and Mariëlle Kroese <sup>3</sup>**


**Abstract:** eHealth could support cost-effective interdisciplinary primary care for patients with chronic musculoskeletal pain. This study aims to explore the feasibility of the eCoach-Pain, comprising a tool measuring pain complexity, diaries, pain education sessions, monitoring options, and chat function. Feasibility was evaluated (June–December 2020) by assessing learnability, usability, desirability, adherence to the application, and experiences from patients and general practitioners, practice nurses mental health, and physiotherapists. Six primary healthcare professionals (PHCPs) from two settings participated in the study and recruited 29 patients (72% female, median age 50.0 years (IQR = 24.0)). PHCPs participated in a focus group. Patient data was collected by evaluation questionnaires, individual interviews, and eCoach-Pain-use registration. Patients used the eCoach during the entire treatment phase (on average 107.0 days (IQR = 46.0); 23 patients completed the pain complexity tool and used the educational sessions, and 12 patients the chat function. Patients were satisfied with the eCoach-Pain (median grade 7.0 (IQR = 2.8) on a 0–10 scale) and made some recommendations for better fit with patient-specific complaints. According to PHCPs, the eCoach-Pain is of added value to their treatment, and patients also see treatment benefits. However, the implementation strategy is important for successful use of the eCoach-Pain. It is recommended to improve this strategy and involve a case-manager per patient.

**Keywords:** chronic musculoskeletal pain; primary care; eHealth; blended care; interdisciplinary care; feasibility; mixed-methods design

#### **1. Introduction**

Chronic musculoskeletal pain (CMP) is a significant public health problem occurring in 19–28% of the European population [1,2]. It is expected that this number will increase in the next years, in line with an aging population [3]. The current health system for patients with CMP is fragmented, leading to high societal and healthcare costs [4–6]. Therefore, the World Health Organization (WHO) calls for a change in health systems focusing on interdisciplinary rehabilitation care and the improvement of self-management skills of patients on long term [7]. However, in order to reach this, there is a need for changes in knowledge, skills, and attitudes of healthcare professionals, as well as changes in the organization of healthcare.

Challenges in this change are accessibility and cost-effectiveness of rehabilitation care, for which eHealth can be a solution [8]. eHealth is defined as the use of information and communication technology for health [9,10]. A wide range of eHealth tools (such as mobile

**Citation:** Lamper, C.; Huijnen, I.; de Mooij, M.; Köke, A.; Verbunt, J.; Kroese, M. An eCoach-Pain for Patients with Chronic Musculoskeletal Pain in Interdisciplinary Primary Care: A Feasibility Study. *Int. J. Environ. Res. Public Health* **2021**, *18*, 11661. https://doi.org/10.3390/ ijerph182111661

Academic Editors: Irene Torres-Sanchez and Marie Carmen Valenza

Received: 8 October 2021 Accepted: 4 November 2021 Published: 6 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

applications and online interventions) have been developed to improve self-management for acute and chronic pain, with promising results regarding their effectiveness [11–13]. Several reasons for the additional value of eHealth in the treatment for patients with CMP can be mentioned.

First, current care for chronic pain is fragmented and continuity of care for the individual patient is often lacking. eHealth can improve healthcare organization as it can facilitate communication and collaboration between healthcare professionals of different disciplines [14]. Accordingly, the WHO advises integrating rehabilitation care within and between primary (general practice), secondary (general hospital), and tertiary care (specialized care centers) [7]. They advise to implement eHealth to facilitate continuity of care in integrated health systems by stimulating daily activities and participation of patients, which are rehabilitation goals [15].

Second, currently, healthcare professionals receive training on diagnosis and treatment, primarily focused on knowledge within their own discipline [16]. This ranges from biomedical oriented care focusing on attempts to solve the pain, toward biopsychosocial oriented care which focuses on optimizing functioning despite pain [17,18]. However, the recommended approach by the WHO requires an integral biopsychosocial vision applied by all healthcare professionals. Currently, patients receive various treatment approaches causing confusion, resulting in unsuccessful organization of integrated care. An eHealth application can facilitate an integral vision on pain and a common language, which are components of integrated care [19]. In this way, it supports the treatment program of all participating healthcare professionals.

Third, earlier studies indicated that eHealth improves self-care support and improves daily activities for people with chronic illnesses [20–22]. eHealth, consisting of a combination of tools, might be of added value and useful as part of a blended care intervention. This is studied previously with separate tools for online pain education or keeping track of daily activities and participation in combination with face-to-face consultations [23,24]. The combination of tools is not studied previously and might lead to better informed and more actively involved patients with increased autonomy, as well as a shift of the role of the healthcare professional into adviser or coach [25]. Moreover, it is assumed that this blended care can stimulate integrated care in the long term and decrease healthcare costs [26,27].

To study the additional value of eHealth in an interdisciplinary network of healthcare professionals for patients with CMP, we implemented an electronic Coach (eCoach-Pain) to facilitate pain rehabilitation within the South East of the Netherlands [28,29]. Based on feedback in this earlier performed implementation, the eCoach-Pain is further improved into its current version. The eCoach-Pain aims to support the provision of integrated rehabilitation care with a shared biopsychosocial vision on health within the Network Pain Rehabilitation Limburg. Within this network, patients and Primary Health Care Professionals (PHCPs), existing of general practitioners (GPs), physiotherapists (PTs), and practice nurses mental health (PNMHs) use the eCoach-Pain. It comprises a measurement tool for assessing complexity of the pain problem, diaries, pain education sessions, monitoring options, and a chat function. Whether it is feasible to use in clinical practice is currently unknown. Therefore, this study aims to explore the feasibility of the eCoach-Pain for patients and PHCPs.

#### **2. Materials and Methods**

This study (June 2020 and December 2020) had a mixed-methods design. Feasibility was evaluated with a focus on learnability, usability, desirability, adherence to the application, and experiences from patients and PHCPS. These were measured by use of patient questionnaires, data about eCoach-Pain-use, a focus group with PHCPs, and interviews with patients. Ethical approval was obtained from the Medical Ethics Committee Z, the Netherlands (METCZ20190037). Patients did not have to pay for participation in Network Pain Rehabilitation Limburg or the eCoach-Pain. During a patient's first login in the eCoach-Pain, an electronic informed consent for the use of the eCoach-Pain and consent

for transferability of their contact details to the researcher were registered. Additionally, for the telephonic interview, patients were asked for informed consent and for recording the interview. PHCPs were asked for informed consent at the start of the focus group.

#### *2.1. Sample and Setting*

PHCPs (GPs, PNMHs, and PTs) of two interdisciplinary primary care practices were recruited to participate in this feasibility study (*n* = 6). They all participated in the Network Pain Rehabilitation Limburg (situated in the South East region of the province Limburg, The Netherlands). This network within and between primary, secondary, and tertiary care aims to support a shared biopsychosocial vision regarding CMP, early recognition of patients with subacute complaints, and a person-centered referral and treatment.

For the current project, patients were recruited by the participating PHCPs. They were eligible if they were ≥18 years at the start of the study, had CMP or musculoskeletal pain at increased risk of becoming chronic (based among criteria on the STarT MSK tool [30,31]), were willing to improve their functioning despite the pain, and had adequate Dutch literacy to use the eCoach-Pain. Exclusion criteria were pregnancy or any medical (orthopedic, rheumatic, or neurological) or psychiatric disease which could be treated by a more appropriate therapy, according to the expert opinion of the GP.

Once a new patient with CMP, or with an increased risk of developing chronic pain, consulted a PHCP, the patient was asked to use the eCoach-Pain. The PHCP gave the main instructions and sent a manual by email.

#### *2.2. The eCoach-Pain*

The eCoach-Pain has been designed by Sananet Care B.V., based on earlier developed eCoaches, such as for Inflammatory Bowel Disease and heart failure [29,32,33]. It contains different goals or opportunities for both patients with CMP and their PHCPs. For patients, the goal is to improve and maintain self-management in coping with pain. For PHCPs the goal is to facilitate biopsychosocial assessment for treatment planning and to monitor the treatment progress of patients with CMP. The eCoach-Pain has been developed in an iterative co-creative development process with the collaboration of researchers, technical experts, patients, and PHCPs. The results of this study will be published elsewhere [28]. The eCoach-Pain can be used on mobile phones, tablets, laptops, and PCs with internet connections.

#### 2.2.1. Application for the Patient

Each patient has an own account for the eCoach-Pain (Figure 1), which could be created in two different ways: First, the PHCP could create a patient's account by filling in the patient's contact details after which the patient receives two emails, one with an account name and one with a password. Consecutively, the patients' account is automatically linked to that of his/her treating PHCP. Second, patients could do a self-subscription throughout a webpage. In this way, a patient is invited to complete contact details, to create a password, and to connect him/herself to his/her own PHCP. Subsequently, the patient's username is sent to the patient by email.

**Figure 1.** Content of the eCoach-Pain. At the top is the application for the patient, with the pain complexity tool, diaries, educational sessions, and chat function. At the bottom is the application for the primary healthcare professional (PHCP) with an intervention list, an overview of the diaries, and an overview of educational sessions.

After login by the patient, a home screen is presented, which contains four different elements (Figure 1).

1. The pain complexity tool:

The pain complexity tool supports the PHCPs in their decision-making for problemmapping and treatment selection. It consists of two parts:


The eCoach-Pain also contains the possibility to use diaries. The PHCP decides, together with the patient, if and how often diaries will be sent to the patient. Diaries can automatically be sent every week, every two weeks, or once a month. However, the diary option can also be neglected. The automatic setting of these diaries is once a week. The diaries exist of the pain complexity tool with additional questions. This extension exists of additional questions based on the questions in the STarT MSK tool scored with "Yes". The answers to these additional questions could be discussed with the PHCP during consultation and used to adjust the treatment or to provide additional educational material to the patient.

3. Education sessions:

The educational sessions provide patients background information about topics related to pain and pain-related disability, such as the difference between acute and chronic pain, treatment of pain, biopsychosocial influences on their pain, information about work and pain, and treatment options. The educational sessions are interactive (YouTube videos and quiz questions with feedback on answers), and they are integrated to stimulate learning and improving knowledge about (chronic) pain. The educational materials are presented in 13 themes and per theme subdivided over several sessions (Figure 1).

4. Chat function:

The chat function is used to send bidirectional messages containing questions or treatment material between patient and PHCP. All communication between patient and PHCP remains accessible in the eCoach-Pain to enable patients to reread answers, advice, or treatment exercise at later moments and times.

#### 2.2.2. Application for the PHCPs

PHCPs could access the eCoach-Pain via a secured webpage on their own device. The PHCPs were instructed to monitor and analyze the patient's situation within a few working days after the patient had completed the pain complexity tool or diary, and to respond as quickly as possible to messages from the patients. To facilitate interpretation of the pain complexity tool and diaries and to save PHCPs' time, information within the application was supportively presented using overviews, graphs, and colored risk flags. Based on the results of the pain complexity tool, different flags appeared on the intervention list: a red flag for a high risk, an orange flag for a medium risk, and a green flag for a low risk for developing CMP (Figure 1).

PHCPs had only access to data of patients treated by themselves. It was possible that more PHCPs, for example, a PT and GP, have access to the data of the same patient in case it was a joint patient. When the PHCP sent a message to a patient or another PHCP, respectively, the other PHCPs and patient were able to read this message in the chat function.

An instruction meeting of one hour to become familiar with the possibilities of the eCoach-Pain was provided to all PHCPs before the start of the study. The software developers and research team facilitated this meeting. Afterward, a paper-copy instruction manual was provided. Moreover, during the pilot, the PHCPS could contact the service desk of the software developers when help was needed or technical issues occurred.

#### *2.3. Data Collection and Analysis*

#### 2.3.1. Learnability, Usability, and Desirability

In September 2020, when the PHCPs had used the eCoach-Pain already for approximately three months, the researcher sent a questionnaire to participating patients. The questionnaire assessed learnability (5-items), usability (5-items), and desirability (6-items) and was an adjusted version of a questionnaire used in a study by Hochstenbach et al. (2016) [34]. Usability was defined as 'the extent to which the application could be used by patients with CMP to monitor their pain, physical activity, and participation level effectively, efficiently, and satisfactorily in everyday practice'. Learnability was defined as 'the time and effort required for these patients to use the application'. Desirability was defined as 'the extent to which the application was fun and engaging to use for these patients. Patients rated each item on a 1–5 Likert scale (completely disagree–completely agree); higher scores indicated better learnability, usability, and desirability. A separate item about the recommendation of the eCoach-Pain to family and friends on a 5-point scale, and a separate item about the overall acceptance of the eCoach-Pain for treatment purposes on a 10-point scale, were added.

Before data analysis, negatively-keyed items were reversed-scored using Microsoft Excel, version Professional Plus 2016, the Microsoft Corporation, Santa Rosa, CA, USA. Median scores with interquartile ranges per item and category were calculated. To identify differences between the PHCPs disciplines, a sub-analysis with discipline as dependent variable was performed.

#### 2.3.2. Adherence to the Application

To assess the patients' adherence, process data from the pain complexity tool, diaries (filled out or not, time of fill out, answers), and from the educational sessions (opened or not, time of opening, how often opened) were logged on the server. The data collected between June 2020 and September 2020 were exported in October 2020.

Median scores and interquartile ranges were calculated using Microsoft Excel, for the number of days patients were active in the eCoach-Pain, number of completed pain complexity tools, diaries, educational sessions, and chat messages used.

Moreover, data about the PHCPs was collected. The median and interquartile ranges of the number of log-ins of the PHCPs was registered overall and per PHCP discipline.

#### 2.3.3. Experiences of Patients

Based on stratified probability sampling on sex, age, and PHCP, patients were contacted for a telephonic interview by the researcher to gain more insight into the experiences with the eCoach-Pain in September 2020. It was intended to ask approximately 16 patients, of which eight agreed, until data-saturation would be reached. However, as data saturation was not reached after this number of interviews, five additional interviews were performed

in December 2020. Topics discussed in the semi-structured interviews included: the use and acceptance of the pain complexity tool, diaries, educational sessions, and chat function, the supportiveness of the application regarding self-management, and technological functioning of the application. Interviews were audio-recorded.

The audio recordings of the interviews were transcribed verbatim. These written interviews were independently analyzed with inductive and deductive thematic analysis by two researchers (C.L. and M.d.M.) using QSR International Pty Ltd. (Melbourne, Australia) (2018) NVivo (Version 12), https://www.qsrinternational.com/nvivo-qualitative-dataanalysis-software/home (accessed on 4 November 2021) [35–37]. Data were sorted based on pre-defined themes of the semi-structured interview-guide. Within these themes, subcategories were created based on the data. After the first two interviews, main themes and codes were discussed and finalized. Thereafter, all other interviews were analyzed and discussed by adding additional codes under the predefined themes.

#### 2.3.4. Experiences of PHCPs

In September 2020, the researcher (C.L.) and observer (A.K.) held an online focus group interview with all participating PHCPs via Zoom [38]. Technical experts of Sananet B.V. (manufacturers of the eCoach-Pain) were available to take notes for future improvements. Before the start of the focus group, participating PHCPs completed questions about the topics on the agenda for the focus group. This encouraged them to formulate an individual opinion before the focus group started and to share this during the meeting. Topics discussed included: use and acceptance of the application, supportiveness of the application in monitoring, advising and treating patients, fit with daily care, and technical functioning of the application. The focus group was audio-recorded.

During the focus group with the PHCPs, the observer (A.K.) made notes and gave a summary per topic discussed. These summaries were asked to be confirmed by the PHCPs during the focus group. Before analysis, the audio recording was used to add additional notes to the summaries by the researchers (C.L. and M.d.M.). These summaries were independently analyzed with thematic analysis on the topics discussed by two researchers (C.L. and M.d.M.) using QSR International Pty Ltd. (2018) NVivo (Version 12), https://www.qsrinternational.com/nvivo-qualitative-data-analysis-software/home (accessed on 4 November 2021) [35].

#### **3. Results**

Data was collected from 29 patients in total; see Table 1. They were eight male and 21 female participations aged between 24–71 years (Median = 50, IQR = 24). The GPs were the primary contact person for 14 patients, the PTs for 13 patients, and the PNMH for two patients. Sixteen patients used the self-registration webpage, while 13 patients were registered by their PHCP. The GP and PNMH of primary care practice one recruited, together, eight patients. The PT of primary care practice one did not recruit any patients. In primary care practice two, 21 patients were recruited by the GP, PNMH, and PT.


**Table 1.** Patient characteristics.

F: female, M: male, GP: general practitioner, PT: physiotherapist, PNMH: practice nurse mental health. n.a.: not applicable. IQR: interquartile range, *n*: total number, %: percentage of total.

#### *3.1. Learnability, Usability, and Desirability*

Twenty-three patients received the invitation for the evaluation questionnaire (September 2020), of whom 11 patients responded (48%). The responders were older (65 (IQR = 23) years old) than the non-responders (48.0 (IQR = 24) years old), and there were less females than males (female responders: 55% (6 out of 11); female non-responders: 83% (10 out of 12)) compared to the total sample.

Six patients started using the eCoach-Pain after the questionnaire was sent and were therefore not invited. The patients who filled in the evaluation questionnaire were an average of 65 (IQR = 23) years old, 55% (6 out of 11) were female, and, on average, active in the eCoach-Pain for 109 (IQR = 41) days. Ten patients answered all questions, and one patient answered only the questions regarding learnability and usability.

Table 2 presents the overall median score (GP and PT), as well as the median score of the categories, and items separately per discipline (by a GP or PT). The scores show that patients learned quickly how to manage the application (Median = 5.0, IQR = 1.0) and could easily use the different components of the eCoach-Pain (Median = 5.0, IQR = 1.5). The desirability was scored with a median score of 4.0, IQR = 2.0. The overall acceptance, rated by the question "I would like to recommend the application to other patients" was scored with a median of 4.0 (IQR = 2.0). Patients gave the eCoach-Pain a total overall score of 7.0 (IQR = 2.8) on a 0–10 Numeric Rating Scale. Patients subscribed by GPs (*n* = 6) scored 5.0 (IQR = 0.0) for learnability, 5.0 (IQR = 1.0) for usability, and 4.5 (IQR = 2.0) for desirability, while patients subscribed by PTs (*n* = 5) scored 5.0 (IQR = 1.0), 5.0 (IQR = 2.0), and 4.0 (IQR = 1.0), respectively.

**Table 2.** Median (IQR) learnability, usability, and desirability scores for the total patient group, patients subscribed by GPs, and patients subscribed by PTs.\* (1–5).


Scores: 0—totally disagree, 5 or 10—totally agree. <sup>a</sup> Negatively-keyed items were reversed-scored before data analyses but the original question is presented in the table with the reversed-score. \* PNMHs have no patients subscribed which completed the questionnaire.

#### *3.2. Adherence to the Application*

At the end of October 2020, for 26 of the 29 patients (median age 53.5 years (IQR = 24.75), 69% female (18 out of 26)), the export data about adherence to the application was available. Three patients were asked to participate in the interviews in December 2020. At that moment, exports were already performed; therefore, no export data of them were available. At the moment of data export, the included 26 patients were, on average, 107.0 (IQR = 46.0) days active in the eCoach-Pain. Ten of them stopped using the eCoach-Pain prematurely

because they finished treatment (*n* = 3) or did not want to use it anymore (*n* = 7). The other 16 patients were still active in the eCoach-Pain at that time.

Twenty-three patients completed the pain complexity tool (Median = 7.0, IQR = 3.0, 1x low risk, 16x medium risk, 7x high risk). For 21 patients, their PHCPs also answered the second part of the pain complexity tool. On average, the diaries were 6.0 (IQR = 3.5) times filled (*n* = 23).

The educational sessions were opened by 23 patients, and they read, on average, 12.0 (IQR = 5.0) educational sessions per person. On average, each separate educational session in the eCoach-Pain was started by 19.5 (IQR = 6.3) individual patients. In total, 224 unique educational sessions were opened by these patients. As there were 13 sessions, this means that some patients read (a part of) the educational sessions several times. Fourteen patients completed all education sessions.

Twelve messages were sent from seven (27%) unique patients to their PHCP, and five messages were sent from the PHCPs to the patients by the chat function. The patients started all conversations. They often elaborated on their diary answers, technical dysfunction of the eCoach-Pain, or they explained why they were not able to fill in the diaries.

The six PHCPs together logged in on average 6 times (IQR = 16.75), the GPs on average 16 times (IQR = 8), the PTs on average 35.5 times (IQR = 32.5), and the PNMHs on average 2.5 times (IQR = 1.5).

#### *3.3. Experiences of Patients and PHCPs*

At the end of September 2020, 16 patients were asked to participate in a telephonic interview, to which eight agreed. To reach data saturation, five additional patients had to be asked, of which three agreed to be interviewed in December 2020. This led to 11 patients participating in the telephonic interviews (mean duration 15 min). The participants had a median age of 60.0 (IQR = 2) years, 73% was female (8 out of 11), and they were active in the eCoach-Pain for 100.5 (IQR = 31.75) days.

Two GPs (one male and one female), two PTs (one male and one female), and one PNMH (one female) participated in a focus group. In addition, one other PNMH (one female) participated in an individual telephonic interview, as she was not able to participate in the focus group.

#### 3.3.1. Overall Opinion and Usage

Patients stated that they were positive about the eCoach-Pain because the functionality worked well for their treatment, it was easy to use, and text was written in clear and understandable language. The content was perceived as informative concerning their pain complaints and knowledge about pain pathophysiology. The interaction between patient and PHCP in the diaries and the quiz questions in the educational sessions of the eCoach-Pain were experienced of added value. Some patients appreciated the reminders for diaries and educational sessions as it gave them structure and control. However, for other patients, these automatic reminders were perceived as somewhat stressful.

Before patients (*n* = 6) started using the eCoach-Pain, they expected the content would be more tailored to their own medical complaints and history. Furthermore, patients expected that the eCoach-Pain would motivate them for treatment compliance to improve their complaints. Although PHCPs are able to change diary frequency, six patients expected less frequent diaries and repetition of information. Besides, some patients indicated that they had preferred to receive more information (by their PHCP or a pamphlet) about the content, frequency of questions, and expected duration of the eCoach-Pain program when they started to use it. Some patients expected more feedback from the eCoach-Pain itself about their answers or an automatic end-session in the eCoach-Pain to close it. Seven patients found it frustrating that, in their opinion, "non-relevant" questions kept returning. The option to indicate a holiday leave and stop sending reminders during this leave was felt to be missing.

*R16: "Basically, I think it is a good app. However, the questions appear too frequent, too standard."*

Among the PHCPs, the eCoach-Pain was most often used by the PTs. One PT used it to structure the content of the treatment sessions and to deliver additional information to the patient.

*PT2: "I like the idea that every week new educational sessions about pain are open for the patient. And, that I can see what the patient answered, which information they have read, and that I can use that during the treatment session. This causes more structure in my treatments."*

Another PT used it for educational purposes for the patient, as well, but did not use the results to guide or adjust treatment as the other PT did. In this feasibility study, the PNMHs hardly used the eCoach-Pain because it was not clear for them how to integrate it in their treatment. PNMHs perceived the eCoach-Pain options offered as specifically PTs treatment options. PTs registered most patients by themselves, which gave them control over the number of patients in the eCoach-Pain. Furthermore, controlling this registration facilitated the ability to inform patients before the start. In addition, the GPs used the eCoach-Pain to score the pain complexity assessment and to support the referral of the patient to the PT. They did not use it to offer treatment purposes or pain education to the patient. The patients that entered the study by a GP most often used the self-registration webpage. GPs indicated that this route was timesaving for them. GPs mentioned that the eCoach-Pain provided them an extra treatment option above the current treatment when they referred a patient to the PT.

*GP2: "The eCoach-Pain is an extra treatment option above the existing options. As a GP, it is important to know the content of the treatment options when referring to a PT, and it is great that we can offer something extra."*

#### 3.3.2. Pain Complexity Tool and Diaries

Patients indicated difficulties in distinguishing the pain complexity tool from the diaries, as the tool and the diaries both were presented as a questionnaire in the eCoach-Pain. Therefore, in this paragraph, the tool and diaries are presented together. Eight of the 11 patients perceived the usability of the pain complexity tool and diaries as good. They indicated that the pain complexity tool and diaries were easy to use, not too timeconsuming, easy to understand, and that the reminders by email were of added value. In addition, patients perceived the content as easy to keep track of their pain complaints and the amount of questions as good. However, most patients (*n* = 8) indicated that the repetition and frequency of the questions were too high. They also missed background information of the questions in the introduction of the eCoach-Pain.

*R02: "I thought that I had to fill in some questions a few times. However, the questions came every day or week for two or three months. And this was not explained to me beforehand."*

Overall, most patients indicated that they perceived the questions in the pain complexity tool and diaries as less applicable in their situation. As several patients had co-morbidities besides CMP, it was difficult for them to know how to interpret the questions. For some questions, it was unclear for them whether the answers should be given with the perspective of having CMP, or from the perspective as a person having pain and other co-morbidities. For example, it was not always possible to indicate exactly their own pain complaints or to adjust answers to questions properly when their situation changed. Sometimes, the eCoach-Pain gave more insight into patients' complexity and impact of their own complaints, which was perceived as heavy to encounter for some patients. Patients without difficulties in daily social participation or psychosomatic problems perceived answer options as less applicable in their specific situation. However, they understood that general questions were formulated for all different kinds of CMP.

The pain complexity tool was the most important tool for GPs in the eCoach-Pain. GP1 indicated that he used it to objectify referral and to get more insight into the complexity of the pain problem. However, GP1 indicated that the digital version directed the referral more than the paper-version. The eCoach-Pain automatically calculates the score and assigns the best-fitting referral option, while, with the paper-version, this can easily be overruled when necessary, according to the opinion of the GP.

*GP1: "When using the paper-version, you have more freedom in the choice of the treatment. As you can overrule the score of the patient easier. In the eCoach-Pain, the treatment options are more limited based on the answers of the patients. Which is a strength of the eCoach-Pain."*

The two PTs used the pain complexity tools in combination with the diaries. For GP1, the graphical display of the results was especially of added value as it gave insight into the effect of the treatment. As improvement, all PHCPs indicated that the graphical displays of the diaries could be upgraded, as it was not always immediately clear for them if the patient's score was positive or negative.

#### 3.3.3. Educational Sessions

The educational sessions were perceived as interesting with clarifying quiz questions and links to YouTube videos. The sessions about 'What is pain' and 'Pain and being active' were perceived as the most useful sessions. The sessions about work and work disability were not appropriate for every patient as some were retired or did not have a job. Two patients indicated that they desired more subjects and educational sessions, for example, about general health.

*R10: "It was a revelation for me, because through the information in the educational sessions, in addition with information on the same topic given by my PHCP, I understand how my brain controls the pain".*

The usability and comprehensibility of the educational sessions were perceived as good as the language used was easy to understand. However, three patients found the language level even too easy and the repetition of subjects in the text as too much. One patient indicated that it was more useful for her when the sessions were not divided over several days, but that all sessions can be followed at once.

Overall, five of the eleven patients indicated that they did not receive new information in the educational sessions in comparison with what they already knew about pain (out of earlier treatments). Some other patients indicated that they perceived recognition and acceptance of their CMP during the sessions due to explanations about the pathophysiology of pain. One patient indicated the sessions as confronting as she/he recognized her/himself for the first time as a patient with chronic pain.

*R13: "I have read all sessions and the total overview was good for me. But at the same time it was also confronting, maybe that was good, as well."*

The educational sessions were most often used by the PTs, and sometimes by the PNMH. PT1 used it to guide the content of his treatment, and PT2 and both PNMHs used it as additional education material for the patient. They indicated that patients were satisfied with the content of the educational sessions and that it gave them more insight into their pain problem. However, they perceived the educational sessions as less applicable for patients with a lower IQ-score or restricted literacy.

#### 3.3.4. Chat Function and Communication with PHCP

Two patients used the chat function, while nine patients indicated they did not. Those two patients were positive about its usability.

Four patients indicated that they had contact with their PT about the diaries and educational sessions they performed in the eCoach-Pain and rated these of added value. For at least one patient, the physiotherapy treatment was adjusted based on the results

in the eCoach-Pain. Moreover, some patients discussed the diary questions about their psychosocial status. Furthermore, patients indicated that the pain education received by their PT fitted well with the information in the educational sessions. The eCoach-Pain resulted in a better patient-PT treatment relationship. Three patients had questions about the eCoach-Pain and needed extra support from their PHCP, for example, about the content of the eCoach-Pain, when to finish using the eCoach-Pain, or extra practical tips regarding their pain complaints. Moreover, some patients mentioned that they had discussed technical issues with their PHCP, such as logging in, bugs in the sessions, or difficulties with data exchange between the PHCP and patient.

The other seven patients mentioned no contact with their PHCP about their activities in the eCoach-Pain. Reasons for this ranged from patients' holidays and sick leave periods, technical issues which limited eCoach-Pain-use, or the fact that the patient had not filled in the pain complexity tools and diaries before the next contact with the PHCP could take place.

Patients did not bother with the fact that their PHCP was able to track their activity in the eCoach-Pain, while some of them did not know this option before the interview. Two patients mentioned that they felt no need to discuss their activity in the eCoach-Pain with their PHCP. Most patients indicated that the possibility to discuss their activity online with the eCoach-Pain was of added value, especially in the situation of COVID-19 they were in during the pilot period, as live contact with PHCPs was only limited to emergency consultations.

PHCPs indicated that they did not use the chat function of the eCoach-Pain often as they preferred other ways to communicate with the patient, such as email, chat functions of other applications, or a real-life contact. Furthermore, the fact that they had to log in again to answer these messages was another reason not to use the chat function. GPs mentioned that they did not always communicate with the patient about the results of the eCoach-Pain themselves, but, instead, they asked the PT or PNMH to respond to the patient.

#### *GP1: "Because of our work-flow, it is the easiest way that the PT communicates with the patient and has a prominent role in the follow-up."*

They checked if a patient scored a red flag, and only then did they contact the patient or the PT. PT1 discussed the results during nearly each treatment session, while PT2 and the PNMH used a less frequent basis or when the patient had questions about it.

#### 3.3.5. Technical Issues

Six patients did not report any technical issues using the eCoach-Pain. Others mentioned problems in finding how to use all functions of the eCoach-Pain, bugs in sessions, or difficulties connecting their eCoach-Pain to the PHCP's profile. Two patients perceived difficulties with logging in into the eCoach-Pain because they had to renew their password more than once or had to log in several times in a row. Two patients had help from family or friends with logging in, use of a computer, or receiving reminders. There were no problems mentioned with the instruction manual, and nobody contacted the Helpdesk of the software developer during the pilot period. Four patients registered themselves with the self-subscription option via a website without any problems. The others were registered by their PHCP; in one case, the connection between the application of the patient and the application of the PHCP failed.

Overall, PHCPs indicated that the eCoach-Pain is easy to use. However, all PHCPs reported having difficulties with the two-way factor identification for logging-in, which is obligated by the General Data Protection Regulation (GDPR). They perceived a delay in receiving the codes by email or the email is marked as spam. The fact that there is an extra step for logging-in hindered them in using the eCoach-Pain more often. They also indicated that it is difficult for them to combine the eCoach-Pain with other existing applications in daily practice, as each application has its own login system, function, and layout.

#### 3.3.6. Future Usage and Recommendations of the eCoach-Pain

Most patients were satisfied with the eCoach-Pain. Some patients indicated that the eCoach-Pain supports to increase insight in how pain impacts daily activities and participation and that it answers questions about their pain complaints. Moreover, they recommend it for the use of the chat function with their PHCP. Some patients would recommend the eCoach-Pain because they were satisfied with it themselves. Most of them would recommend it to patients with other complaints than their own, as they indicated that the content did not fit perfectly based on their own situation. They would especially recommend it to patients who are recently diagnosed, have problems in daily activities and participation, are low literate, or who want to use an eCoach-Pain frequently.

*R16: "I would recommend it to people who get acquainted with pain complaints, or who have not so much knowledge yet, for them it is useful to get to know more about pain. But for people who have complaints for years, like me, I would not recommend it."*

They would not recommend it to patients with pain complaints for years, elderly who are not familiar with eHealth, or patients who do not want to use the eCoach-Pain frequently. However, some patients who are recently diagnosed would recommend it for patients with chronic complaints.

Most PHCPs indicated that they will keep on using the eCoach-Pain in the future as they find it important to offer the patient something extra besides usual care. However, due to time constraints in daily practice, GPs hope that PNMHs can get a more prominent role in the follow-up of patients and contact other PHCPs about the results in the eCoach-Pain. In this case, the PNMH has to contact the GP when expertise or referral of the patient is needed.

*GP1: "Because of the high work-load in primary care, it would be of added value when someone as a PNMH can get a more prominent role in follow-up of patients. They will also be able to keep track of the eCoach-Pain activities. We as GPs have not enough time to do this properly."*

PTs think they will keep on using the eCoach-Pain in the same way as they did during the feasibility study. However, all PHCPs indicated that the costs of the eCoach-Pain concerning their patient volume are important indicators for future usage. During this feasibility study, these costs were covered by the project budget of NPRL.

#### **4. Discussion**

The current study provides insight into the feasibility of an eCoach-Pain for patients with CMP or a high risk of becoming chronic, and for PHCPs in interdisciplinary primary care. In general, patients and PHCPs had positive experiences using the eCoach Pain. The answers to questions/statements about learnability, usability, desirability, and adherence to the application confirm that the eCoach Pain has sufficient quality for further use. However, some further adjustments for successful implementation and use are needed.

Some patients mentioned that the content of the eCoach-Pain does not fit with their situation, such as multi-morbidities and previous experiences with treatment. An explanation why patients do not find the eCoach-Pain suitable could be that patients with CMP often experience multi-morbidities, such as depression, anxiety disorders, obesity, hypertension, and diabetes [39–42]. It has been shown that these patients with multi-morbidities need a personalized treatment [43,44]. The eCoach-Pain has not enough attention for these multi-morbidities. Some patients mentioned that the eCoach-Pain was more suitable for patients with other complaints than they had. Remarkably, the patients with severe complaints for several years mentioned that the eCoach-Pain was better suitable for patients in a subacute phase or those recently diagnosed. Patients with complaints for several years indicated that the information about CMP in the eCoach-Pain was not new and perceived the education sessions as too basic for them. They indicated that the pain education was given in earlier treatments. However, patients with subacute complaints mentioned that the eCoach-Pain is better suited to patients with a clear diagnosis or, in contrast to the comment

of patient with long-term complaints, patients with a longer duration of complaints. A possible explanation for this finding could be that subacute patients are still searching for an explanation and solution for their complaints and are, therefore, more biomedically oriented and not yet focused on a biopsychosocial treatment. [45]. As accepting of CMP is an ongoing process, it could be that the patients with subacute musculoskeletal pain do not see themselves as patients with CMP [46]. Therefore, further research is needed to discover for which patients group(s) the eCoach-Pain can be used in primary care and, accordingly, how the eCoach-Pain can be aligned for personalized treatment.

The eCoach-Pain is well integrated into the treatment of the PTs. All PHCPs perceived advantages of the use of the eCoach-Pain during physiotherapy treatment. Patients indicated that eCoach-Pain connects to the treatment of the PT. Positive thoughts about blended rehabilitation care for other diseases are also seen in several other studies [47,48]. The integration of an eCoach as blended physiotherapy care for patients with temporomandibular disorders lead to an increase in self-efficacy, support of data collection and personalization of the application in the Netherlands [47]. The review of Orlando et al. (2019) showed an overall positive impact on patient and caregivers' satisfaction and it appears to enhance communication and engagement between healthcare professionals for different kinds of telehealth in rural settings [48]. However, questions about the integration of an eCoach in the treatment, such as duration of the treatment, fit in each consultation, and the frequency of the consultations remain [49]. Tilburg et al. recommend to integrate an eCoach into the total treatment and not to implement it as a separate component to the treatment. Further research needs to design and evaluate the integration of the eCoach-Pain into the treatment to deliver blended care.

eCoaches can stimulate and influence interdisciplinary collaboration in primary care [50]. Based on the findings that PHCPs indicated suboptimal collaboration during treatment, it can be concluded that interdisciplinary collaboration between the PHCPs was a point of attention. Accordingly, it seems that the eCoach-Pain did not contribute to interdisciplinary care. GPs indicated that they preferred to refer the patient automatically to a PT as they had not enough time to contact patients and discuss the treatment plan with the PTs, as purposed in interdisciplinary care. The preference of GPs, due to their lack of time, for treatment of these patients by a PT is in line with another study with eCoaches in primary care in the Netherlands [51]. In this study, the PHCPs perceived advantages of the eCoach-Pain in referring a patient to a PT or adding an extra role of a case-manager (for instance, a PNMH or specialized nurses for mental health) in the future. Previous successful implemented eCoaches used a case-manager as first contact for patients [32,34]. In addition, the Standard of Care for Chronic Pain in the Netherlands advices the use of a case-manager for patients with CMP in primary care [14]. However, the eCoaches in these earlier studies were all implemented in secondary care. Therefore, the role for case-manager in primary care needs to be optimized before implementation. Currently, there is no regular financing of a case-manager for patients with CMP in primary care in the Netherlands yet. However, it is crucial to have a case-manager when focusing on integrated and interdisciplinary primary care to stimulate a common vision and treatment plan [19].

Some patients mentioned technical problems that limited the use of diaries or education sessions, even though they received a reminder. Although the developers could not find an explanation for this, it could have influenced the adherence rates for the diaries and education sessions. Other patients indicated that they received too many reminders for diaries. Therefore, in future use of the eCoach-Pain, attention must be given to the communication between the PHCPs and patients. The PHCPs must discuss in advance the number of diaries and reminders offered, based on the preferences of the patients. Research has shown that shared-decision making for chronic illness with treatments containing more than one session leads to treatment agreement [52]. Therefore, shared-decision making in eCoach-Pain adjustments could lead to increased treatment adherence. Connection to the electronic patient file is another technical problem mentioned. PHCPs, and especially

GPs, experienced barriers in the use of the eCoach-Pain in daily practice which was not connected with their electronic patient file. This caused double registration steps, which was a reason to restrict use of the eCoach-Pain. Therefore, it would be favored to find a possibility to integrate the eCoach-Pain in the electronic patient file (Dutch: Huisarts Informatie Systeem) to avoid extra registration steps [53].

A major strength of this study is the use of qualitative and quantitative data (mixedmethods) alongside objective data on use of the eCoach-Pain from both patients' and PHCPs' perspectives. These data gave a broad overview of the usability of the eCoach-Pain, as well as the experiences. Moreover, the content of the eCoach-Pain was developed together with the PHCPs before the start of the study with a user-centered design [28]. Higgins et al. (2018) recommend user-centered designs and implementation science methods to improve the availability of eHealth tools. [54]. However, the GPs and PNMH rarely used the eCoach-Pain despite their influence in the development process. Reasons for this are the login-facility and lack of time during and after consultations, which are also seen as barriers in the study of Daniëls et al. (2019) in primary care [51].

Some limitations of this study need to be acknowledged. First, the small sample of patients that were available for this study and the limited use of the eCoach-Pain could have introduced selection bias. It could be that patients with, for example, low literacy or co-morbidities, were not asked for participation by the PHCPs with a risk for selection bias. However, despite the small sample, patients differed in demographic characteristics, resulting in sufficient confidence to have studied a representative group of users. Second, not all patients performed all measurements, so the completeness of available data per measurement differed. Six patients did not respond on the evaluation questionnaires, and, for three patients export data of eCoach-Pain-use is missing. This could have led to information bias which could have influenced the data. Third, the sample of primary care practices and PHCPs was small, which could have influenced the results. As for primary care practice 1, all patients are subscribed by a GP or PNMH, and, for primary care practice 2, the PT also subscribed patients, besides the GP and PNMH. Most patients were recruited by primary care practice 2 (*n* = 21), and most of the patients participating in the interviews were also recruited by this practice (9 out of 11). Therefore, limited results were available about the recruitment of GPs in the interviews.

#### **5. Conclusions**

In conclusion, the eCoach-Pain seems to be promising in primary care: the patients, as well as the PHCPs, experienced advantages for treatment of patients with CMP. However, adjustments to the content have to be made for better fit with patient-specific CMP complaints. Moreover, the implementation strategy seems to be an important factor for successful use among PHCPs. This should be improved for successful use in interdisciplinary primary care settings. The involvement of a case-manager for CMP should be further explored when implementing the eCoach-Pain. Thereby, it is important to use usercentered designs and implementation science methods to evaluate adjustments resulting in a successful implementation.

**Author Contributions:** C.L. was involved in data curation, formal analysis, investigation, methodology, project administration, supervision, validation, visualization, and writing—original draft. I.H. was involved in supervision, methodology, validation, and writing—review & editing. M.d.M. was involved in formal analysis, investigation, validation, and writing—review & editing. A.K. was involved in investigation, methodology, supervision, and writing—review & editing. M.K. and J.V. were involved in supervision and writing—review & editing. All authors have read and agreed to the published version of the manuscript.

**Funding:** Health Insurance Companies CZ, VGZ, and Zilveren Kruis Achmea funded the study, but they had no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Medical Ethics Committee Z, the Netherlands (METCZ20190037, latest date approved 19/05/2020). Informed consent will be obtained from study participants.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to the small number of healthcare professionals and patients included. This could interfere with the privacy of the participants.

**Conflicts of Interest:** C.L., M.d.M. and M.K. have nothing to disclose. I.H., A.K. and J.V. report grants from Health Insurance Companies CZ, VGZ, and Zilveren Kruis Achmea, during the conduct of the study.

#### **References**


### *Article* **Patient Experience and Satisfaction with an e-Health Care Management Application for Inflammatory Bowel Diseases**

**Aria Zand 1,2,\*, Audrey Nguyen 1, Courtney Reynolds 1, Ariela Khandadash 1, Eric Esrailian <sup>1</sup> and Daniel Hommes 1,2**


**Citation:** Zand, A.; Nguyen, A.; Reynolds, C.; Khandadash, A.; Esrailian, E.; Hommes, D. Patient Experience and Satisfaction with an e-Health Care Management Application for Inflammatory Bowel Diseases. *Int. J. Environ. Res. Public Health* **2021**, *18*, 11747. https:// doi.org/10.3390/ijerph182211747

Academic Editors: Marie Carmen Valenza, Irene Torres-Sanchez and Paul B. Tchounwou

Received: 19 August 2021 Accepted: 2 November 2021 Published: 9 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

**Abstract:** Background: Rising healthcare expenditures have been partially attributed to suboptimal management of inflammatory bowel diseases (IBD). Electronic health interventions may help improve care management for IBD patients, but there is a need to better understand patient perspectives on these emerging technologies. Aims: The primary aim was to evaluate patient satisfaction and experience with the UCLA eIBD mobile application, an integrative care management platform with disease activity monitoring tools and educational modules. The secondary objective was to capture patient feedback on how to improve the mobile application. Methods: We surveyed IBD patients treated at the UCLA Center for Inflammatory Bowel Diseases. The patient experience survey assessed the patients' overall satisfaction with the application, perception of health outcomes after participation in the program, and feedback on educational modules as well as areas for application improvement. Results: 50 patients were included. The responses indicated that the patients were greatly satisfied with the ease of patient–provider communication within the application and appointment scheduling features (68%). A majority of respondents (54%) also reported that program participation resulted in improved perception of disease control and quality of life. Lastly, a majority of participants (79%) would recommend this application to others. Conclusions: Mobile tools such as UCLA eIBD have promising implications for integration into patients' daily lives. This patient satisfaction study suggests the feasibility of using this mobile application by patients and providers. We further showed that UCLA eIBD and its holistic approach led to improved patient experience and satisfaction, which can provide useful recommendations for future electronic health solutions.

**Keywords:** electronic health; mobile health; mobile applications; chronic disease management; patient experience; inflammatory bowel disease

#### **1. Introduction**

Value-based healthcare (VBHC) can be described as the systematic pursuit of the triple aim in healthcare: to improve the individual's experience, improve health outcomes, and reduce costs [1]. The concept of VBHC is particularly ready for application to long-term management of chronic illnesses since rising healthcare expenditures have been partially attributed to suboptimal management of chronic illnesses, including IBD [2]. The estimated annual disease-attributable cost of IBD is \$6.3 billion [3]. Hospitalizations represented over a third of costs, outpatient services—one third. Reducing hospitalization and readmission rates, therefore, continues to be a challenge in chronic disease management. There is clearly an opportunity to reduce costs by increasing the efficiency of outpatient care and preventing hospitalizations.

Electronic health (e-health) interventions are one solution for more effective IBD care management beyond the clinical setting, both in terms of patient outcomes and cost reduction. Smartphone applications are widely available for consumers, and the large population of smartphone users make applications useful tools to manage chronic illnesses like IBD [4]. In fact, smartphone devices with mobile applications and short message reminders have been used effectively by patients with IBD of mild or moderate severity [5].

Furthermore, mobile health technologies have been shown to improve patient outcomes and quality of life [6]. Patient satisfaction with mobile technologies has been observed for many chronic diseases, including asthma [7], HIV [8], diabetes [9], atrial fibrillation [10], and IBD [11]. IBD patients generally have positive views on mobile applications, but there are desirable improvements. A study of Con et al. [11] surveying 86 IBD patients found that 98.8% of the participants were willing to use communication technologies for IBD management, with mobile applications being one of the top two preferred forms. These previous IBD mobile technologies were often created to assess a major single aspect such as quality of life [5], education curriculum [12], or diets [13]. Additional features that patients seek in their chronic disease management applications include easy user interface [14], tracking of disease symptoms [11], and easy access to medical data and services [11].

A systematic assessment of 26 IBD mobile applications found that applications offered a variety of features including diary functionality, pain tracking, bowel movement tracking, and reminders, with application's content playing a major role in driving patient behavior change [4]. The MyIBD Coach telemedicine tool, which monitors adherence, disease activity, quality of life, and mental health among other measures through validated questionnaires, was shown to be successful, with high rates of patient satisfaction and compliance [15]. It involves collaboration among healthcare providers but is not synchronized with electronic medical records and lacks educational application features on alternative medicine, behavioral health, and physical activity.

To enhance VBHC in IBD, we developed UCLA eIBD to integrate various successful features of previous applications (e.g., appointment reminders, medication trackers) in addition to a healthcare provider portal. UCLA eIBD seeks to provide patients more agency in managing their IBD by increasing their access to healthcare professionals and providing self-help educational modules. Access to care providers through a messaging application provides patients with fast feedback on their conditions and streamlines patient care [16]. The application also contains disease activity, quality of life, and work productivity surveys that facilitate interactions between patients and providers. These tools allow healthcare providers to monitor patients' disease activity and give direct feedback. This comprehensive application therefore seeks to enhance patient outcomes by including direct connections to the healthcare team and extensive module options.

We previously conducted a pilot study of UCLA eIBD, which found significantly fewer endoscopies and decreases in healthcare utilization, long-term steroid use and IBD-related costs [17]. While it is important to evaluate the efficacy and outcomes of IBD management platforms, it is just as crucial to understand patients' satisfaction with these platforms to inform their feasibility. Gathering user feedback is necessary to develop the next generation of applications, improve product design, and reduce the gap between application developers and consumers [18–20]. This study therefore aims to provide an evaluation of perceived patient satisfaction and experience with the UCLA eIBD mobile application.

#### **2. Materials and Methods**

#### *2.1. Objectives*

The primary objective was to measure patient satisfaction and experience with the UCLA eIBD mobile application for care management. The secondary objective was to capture patient feedback on how to improve the mobile application.

#### *2.2. Design and Population*

We surveyed IBD patients treated at the UCLA Center for Inflammatory Bowel Diseases from October 2017 to October 2018. Included patients were at least 18 years old; diagnosed with Crohn's disease (CD) or ulcerative colitis (UC) either by means of endoscopy, imaging, or pathology; and had objectively logged into the application in the past year (assessed on the platform). Patients with intestinal cancer, active chemotherapy, or a known intestinal infection were excluded.

All the eligible patients who had logged into the application in the past year were emailed and asked to complete a patient experience survey. Those who did not complete the survey in response to the initial email were followed up and interviewed via phone. No sample size estimation was performed.

#### *2.3. Description of UCLA eIBD*

UCLA eIBD is a mobile application that administers a clinic-centered, care management program to its users (Figure 1). It was designed to be a comprehensive tool for patients' long-term disease management in the IBD outpatient setting. The features of this application include disease activity monitoring, messaging, educational modules, lifestyle modules, and electronic cognitive behavioral therapy (eCBT). The platform is also integrated with UCLA Health's electronic medical records, allowing patients to view their testing and laboratory results within the application.

**Figure 1.** The UCLA eIBD mobile application is an integrative care management platform for patients and providers.

For disease activity monitoring, a previously validated tool called the Mobile Health Index was integrated to assess the patients' disease activity, quality of life, and work productivity [21]. If the surveys indicated poor disease control or a significant change from prior surveys, a message was automatically generated through the application to clinic staff. The enrolled patients could also elect to take these surveys in their own time if they felt they were experiencing a sudden change in their health.

Lastly, the application provided education through several optional interactive modules designed to promote healthy lifestyle habits, including nutrition (My Menu), exercise (My Yoga, My Fitness), relaxation (My Acupressure, My Meditation), and mental health (My Coach). My Menu teaches patients about specific foods to eat and avoid and includes recipes (breakfast, snack, lunch and dinner) designed for IBD patients. My Yoga provides a 6-week program promoting relaxation and flexibility for users. My Acupressure teaches patients about different pressure points for alleviating IBD pain via instructional videos and pictures. My Meditation is a self-guided mindfulness therapy tool that aims to reduce stress-related health issues. My Coach is a personalized mental life coaching program (6-week mental support program) aimed at improving mental well-being and stress management through a cognitive behavioral therapy method.

#### *2.4. Data Collection and Outcomes*

Patient demographic data were acquired via chart review. The data from the patient experience survey were collected via REDCap [22]. The patient experience survey (Table 1) consisted of 24 items aimed at assessing the patients' overall satisfaction with the application and their perception of health outcomes after participation in the program. Responses were provided either via a Likert scale or open text. Questionnaire items addressing the application's features and interface requested feedback on the ease of application use, ability to communicate with staff, and informativeness of modules. Questionnaire items pertaining to the patient's outcomes asked the patients how effective they felt the application was at improving disease control, work productivity, and quality of life. Lastly, the patients could provide optional open-ended feedback via free text input on ways to improve the application.


#### **Table 1.** Patient experience survey.


**Table 1.** *Cont.*

\* Optional question.

#### **3. Results**

#### *3.1. Patient Demographics*

In total, 151 patients had been active on the mobile application in the past year, of whom 50 patients responded and completed the survey and thus were included in this study. Regarding the type of IBD, 44% were diagnosed with CD (*n* = 22), 56%—with UC (*n* = 28). Our inclusion cohort had a mean age of 43 years (SD, 14 years) and an average BMI of 25.3 (SD, 6.6). Of the patients, 44% were female, and the majority were White (42%) and non-Hispanic (90%) (Table 2). Most of the patients were non-smokers (78%), and 28% of the patients reported alcohol use. The patients stated use of the following medications: anti-TNF (34%), ASA (16%), combination therapy (32%), IMM (10%), and steroids (6%). Previous abdominal surgeries were reported in 36% of the participants.


**Table 2.** Patient demographics.

#### *3.2. Patient Satisfaction*

Fifty participants out of the 151 users responded and completed the patient experience survey to provide feedback on the mobile application (Table 1). Responses to the Likert scale questions indicated that the patients were overall satisfied with the patient–provider communication interface of the application. When asked how easy it was to communicate with the program's staff overall, 52% of the participants responded with "very easy" and 16% responded with "somewhat easy". A majority of the participants also found it easy to schedule appointments through the application, with 52% and 16% responding with "very easy" and "somewhat easy", respectively. In addition, a large majority (88%) of the participants reported that the frequency of completing laboratory tests and surveys and scheduling clinic visits was "just right" (Table 1). Regarding the ease of application use, 74% of the participants indicated the application was either "very easy" or "somewhat easy" to navigate.

Additionally, a majority of the participants reported an improved perception of disease control and QoL; 54% of the participants indicated significant or some improvement in their disease control. When asked how program participation affected QoL, 26% indicated significant improvement, 30%—some improvement. Regarding work productivity, 44% indicated significant or some improvement.

When the participants were asked whether they would recommend this application to their friends, family, or other patients on a ten-point scale, with 10 being most likely,

the median score was 8, and 79% indicated a score greater than 5. When asked about how informative the application was, 46% of the patients felt that the application was "somewhat" or "very" informative.

#### *3.3. Patient Usage of Educational Modules*

A majority of the patients completed modules as part of their participation in the program. The most used modules were My Fitness and My Menu (Table 1). Among the patients who participated in the CBT modules (12%), 28% indicated significant or some improvement in their mental health.

When asked about what they liked and disliked about the modules, the patients identified positive aspects to be the modules' informative content, ease of use, and support of overall well-being (Table 3). For example, one patient said, "They're easy and I feel great afterwards." Another patient expressed liking the modules because they "encourage me to take care of my whole self instead of the focus just being on taking my meds".

**Table 3.** The patients' optional feedback on the modules (*n* = 50). The patients provided open-ended feedback about the educational modules. Their responses were grouped into categories based on the common themes identified across the responses.


The most common reason for not liking the modules was being unsure of the purpose or need for them (8%), particularly for the modules where patients already had their own interventions in place. For example, one patient said they "didn't feel [the modules] applied to me" while another expressed that they "thought [the module] was good but [I have my] own routine for working out [with regards to My Fitness]".

#### *3.4. Patient Feedback*

In the patient experience survey, the patients could provide optional suggestions about additional topics and functionalities they would like the application to cover which were not presently included (Table 4). One participant, for instance, suggested adding a subsection about nutritional advice related to veganism within the My Menu module. Other recommendations included adding a "symptoms tracker", allowing patients to indicate what symptoms or lack thereof they were experiencing and generating in-application reminders for blood draws or laboratory orders. The other patient-recommended categories to add were the ability to chart laboratory results, side effects of their consequent medications, and gender-specific health topics (Table 4).

**Table 4. The patients' optional feedback on UCLA eIBD**. The patients provided open-text suggestions to improve the application in general. These suggestions were grouped into categories of comment types, including improvements in application content such as possible additional topics and features, as well as miscellaneous critiques.


The patients' feedback regarding general comments about the application is also shown in Table 4 (miscellaneous improvement suggestions). One patient stated, "I think this is a great idea and will be very helpful to future patients. I really like being able to communicate with the office without always having to call." Most patients who provided comments also highlighted aspects that could be improved, such as the application's interface (e.g., adding a touch ID option to log in; preventing automatic logoff from the application). Other participants reported critical feedback on the application's content. For instance, one patient stated that the application "is good for people new to IBD, but doesn't offer as much for people who have had IBD for a while and want more in depth information".

#### *3.5. Summary of Principal Findings*

The outcomes suggest that the patients strongly favored the ease of patient–provider communication, with 78% being satisfied. Beneficial outcomes were also seen in patientreported measures, with 54% reporting a perceived improvement in disease control and 56% reporting a perceived improvement in QoL, indicating that a majority of patients felt the platform positively impacted their health. Additionally, the participants rated this application with a median score out of 10 (10 being most likely) to recommend this

application to friends, family, or other patients. My Fitness and My Menu were the two most used optional wellness modules, each reaching the 34% completed status.

#### **4. Discussion**

#### *4.1. Strengths and Comparisons*

Our study collected feedback on patient experiences with the UCLA eIBD application after one year of use. Our results could provide guidance for further application development and provide critical feedback for other e-health applications like this one. In fact, mobile tools such as UCLA eIBD have been shown to have promising implications in improving healthcare delivery and integrating into patients' daily lives. Earlier comparison studies of UCLA eIBD found impacts on costs and healthcare utilization and identified its unique features, such as automated messaging to care coordinators [17,23–25]. To complement the previous outcome studies, this study aimed to understand patients' satisfaction and feedback to help elucidate gaps in the current e-health technologies and inform future designs.

For instance, GI Buddy is a mobile application developed by the Crohn's Colitis Foundation which enables patients to self-monitor their disease and receive reminders about clinical appointments; however, users cannot directly interact with their providers [26]. Similarly, while the current applications for IBD may be useful for patient monitoring and self-management, many lack professional medical involvement and adherence to clinical guidelines [4]. UCLA eIBD addressed this gap by allowing users to make appointments and message their providers via the platform, in which a majority of users found it "easy" or "very easy" to communicate with their providers. Another self-management tool, myIBD Coach, showed feasibility among patients and providers [15]. As many as 79% of UCLA eIBD users would recommend this application to others (indicated by a score of greater than 5 on the recommendation score item), compared to the 93% found in the myIBD Coach feasibility study [15].

The findings of this novel patient satisfaction study demonstrate the feasibility of UCLA eIBD as a home monitoring tool and some advantages it can provide for both patients and providers. In addition to the patient–provider communication features, the platform's educational modules are more diverse than the previous tools and provide patients with more alternatives to aid traditional medicine, such as acupuncture, cognitive behavioral therapy, and meditation. These optional modules may improve IBD patients' well-being and productivity beyond the scope of their disease. For providers, tracking the various modules that the patients use can also provide guidance for tailoring treatment and counseling to the patients' interests, including nutrition, exercise, and mental health. Lastly, the holistic nature of the application, including features about alternative medicine and assessments for work productivity via the Mobile Health Index, can more completely address the complex, multidimensional factors of chronic disease management.

While the integration of mobile health in IBD management is rapidly expanding, our study also presents novel data from the patient perspective and emphasizes a patientcentered approach towards mobile application development. For instance, the patients' suggestions to improve the application were centered on specific content interests and the need for additional educational categories (e.g., female health topics), rather than technical problems or lack of need in an application. The fact that the suggestions were less focused on design features could be explained by the overall satisfaction rate of 74% of the participants finding the application easy to navigate. The current and future applications can thus utilize these methods and/or findings to adapt their platforms to address patients' specific needs, improve satisfaction with their product, and better engage patients in their medical care beyond a doctor's office visit.

#### *4.2. Limitations*

Some study limitations should be noted. As the selected patients were individuals who use smartphones, they may be more adept at the usage of applications. Participants were also actively recruited and agreed to participate in this study; thus, a selection bias may have impacted the study results due to the participants being predisposed to wanting to improve their health via e-health solutions. We further acknowledge the sample size was small and relatively homogenous; however, we feel it was adequate for the purpose of directing the future development of this UCLA application and other healthcare applications.

Additionally, the fact that we invited participants to evaluate the application's feasibility rather than making it mandatory during application usage may explain the response rate of 33%. The response rate should further be considered in the context of challenges associated with adopting e-health technologies into the healthcare space. The obstacles to widespread long-term integration of e-health technologies (e.g., loss of interest, data entry burdens) are still being investigated [27,28]. Despite the growing population of individuals who use mobile health applications, many stop using them over time [29]. Our findings help provide insight to consumer perspectives on application usability and possible explanations to circumvent these challenges.

#### *4.3. Future Directions*

In an era where the use of mobile technology has become irreplaceable in daily life, there is undoubted benefit of incorporating e-health applications in the management of chronic conditions. Studies have shown proven effect of mobile applications, but also that patients still desire improvements to the existing solutions. We showed that UCLA eIBD and its holistic approach led to greater patient experience and satisfaction, which can provide useful recommendations for healthcare providers and application developers. However, larger and controlled studies are recommended to assess its efficacy at a larger scale and its impact on costs.

**Author Contributions:** A.Z.: study concept and design; analysis and interpretation of data; drafting of the manuscript; critical revision of the manuscript for important intellectual content; statistical analysis; study supervision; A.N.: study concept and design; acquisition of data; analysis and interpretation of data; drafting of the manuscript; critical revision of the manuscript for important intellectual content; C.R.: study concept and design; drafting of the manuscript; critical revision of the manuscript for important intellectual content; A.K.: acquisition of data; administrative, technical, or material support, drafting of the manuscript; critical revision of the manuscript for important intellectual content; E.E.: revision of the manuscript for important intellectual content; D.H.: study concept and design; analysis and interpretation of data; drafting of the manuscript; critical revision of the manuscript for important intellectual content; statistical analysis; study supervision. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** This study was approved by the Institutional Review Board at UCLA with IRB protocol number 17-001208. All the procedures performed in the studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

**Informed Consent Statement:** Informed consent was obtained from all the individuals included in the study.

**Conflicts of Interest:** The authors declare that they have no conflict of interest.

#### **References**


## *Article* **Building a Research Roadmap for Caregiver Innovation: Findings from a Multi-Stakeholder Consultation and Evaluation**

**Kieren J. Egan 1,\*, Kathryn A. McMillan 1,2, Marilyn Lennon 1, Lisa McCann <sup>1</sup> and Roma Maguire <sup>1</sup>**


**Abstract:** Across the world, informal (unpaid) caregiving has become the predominant model for community care: in the UK alone, there are an estimated 6.5 million caregivers supporting family members and friends on a regular basis, saving health and social care services approximately £132 billion per year. Despite our collective reliance on this group (particularly during the COVID-19 pandemic), quality of life for caregivers is often poor and there is an urgent need for disruptive innovations. The aim of this study was to explore what a future roadmap for innovation could look like through a multi-stakeholder consultation and evaluation. An online survey was developed and distributed through convenience sampling, targeting both the informal caregiver and professionals/innovators interested in the caregiver demographic. Data were analysed using both quantitative (summary statistics) and qualitative (inductive thematic analysis) methods in order to develop recommendations for future multi-stakeholder collaboration and meaningful innovation. The survey collected 174 responses from 112 informal caregivers and 62 professionals/innovators. Responses across these stakeholder groups identified that there is currently a missed opportunity to harness the value of the voice of the caregiver demographic. Although time and accessibility issues are considerable barriers to engagement with this stakeholder group, respondents were clear that regular contributions, ideally no more than 20 to 30 min a month could provide a realistic route for input, particularly through online approaches supported by community-based events. In conclusion, the landscape of digital health and wellness is becoming ever more sophisticated, where both industrial and academic innovators could establish new routes to identify, reach, inform, signpost, intervene and support vital and vulnerable groups such as the caregiver demographic. Here, the findings from a consultation with caregivers and professionals interested in informal caring are presented to help design the first stages of a roadmap through identifying priorities and actions that could help accelerate future research and policy that will lead to meaningful and innovative solutions.

**Keywords:** caregivers; innovation; research; co-design; interdisciplinary; digital health; participatory design; collaboration at distance

#### **1. Introduction**

Informal (unpaid) carers, also termed "caregivers", are family members and friends who support a loved one who needs help due to illness, frailty, disability, mental health problems or addiction. In the UK, it is estimated that there are at least 6.5 million informal caregivers, a workforce substantially larger than the National Health Service (NHS) [1] and the collective saving to the health and social care services is estimated to be £132 billion per year. This situation is similar across Europe, and beyond, where 80% of all care is delivered by informal caregivers [2]. As the tide of an ageing global population continues to advance in tandem with a shrinking health and social care workforce [3], there is a public

**Citation:** Egan, K.J.; McMillan, K.A.; Lennon, M.; McCann, L.; Maguire, R. Building a Research Roadmap for Caregiver Innovation: Findings from A Multi-Stakeholder Consultation and Evaluation. *Int. J. Environ. Res. Public Health* **2021**, *18*, 12291. https://doi.org/10.3390/ ijerph182312291

Academic Editors: Irene Torres-Sanchez and Marie Carmen Valenza

Received: 28 October 2021 Accepted: 14 November 2021 Published: 23 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

health emergency looming, whereby the pressures on caregivers across the world are set to significantly escalate.

While the recent events during the COVID-19 pandemic have demonstrated the societal value of caring, it also highlighted that many will face short, and long-term, health and wellness consequences. Informal caregivers share many of the same challenges of a professional workforce, such as NHS employees, but lack much of the associated infrastructure. For example, there is little to no: support, training, pre-agreed workload such as hours per week or a working role definition, and caregivers often must balance caring responsibilities around work, families and their own health and wellness needs. While some caregivers do enjoy benefits from caring roles, such as a sense of fulfilment from caring [4], there is now compelling evidence that caregiving adversely impacts on health and wellness, both in the short and long term [5–7]. Crisis points, such as hospitalizations, significant worsening of mental and/or physical health and irreversible changes to caring circumstances are commonplace (even in the absence of COVID-19) and frequently cause deterioration in health for caregivers, and those being cared for [8,9].

Given the considerable number of unmet needs described above, there have been many different innovations developed for the caregiver demographic. Although successful evidence has emerged [10–12], there still remains a striking number of caregivers who remain "out of reach" [8] through a combination of factors, including (but not limited to): digital and health literacy levels, socioeconomic status, health, mobility, and level of dependency required from the person cared for. Thus, there remains a continued need to consult, innovate and evaluate for, and with, caregivers and to develop research excellence in this area. Emerging from the evidence in this field, there are several areas of particular interest, which include: methodologies to improve identification of caregivers (including in hard to reach groups such as ethnic minorities [13], the development and adaption of theoretical/conceptual frameworks, techniques for cultural adaptation [14] and addressing implementation and abandonment issues [15].

Digital health and wellness solutions are a core part of the current drive to deliver equity in health to all—including informal caregivers. In 2020, for example, the WHO (World Health Organization) produced a long-term strategy for the use and scale-up of digital health, highlighting the positive impact digital health can have on the access to, and provision of healthcare as well as the health and wellbeing of the population [16]. Such findings are paralleled in recent review work [17]. Much of this work focuses on caregiver health and wellbeing or signposting to sources of support and are not restricted to one or two types of technology (e.g., mobile app, internet-based, integrate platforms, sensors) [17]. Despite such progress, the market reality is that most startups fail and many successful research studies are challenging to implement. Reducing uncertainties and increasing the availability of market knowledge, such as bettering current understanding of ongoing caregiver needs, would be considerably advantageous.

Taken collectively, innovation for informal caregivers is sorely needed but the landscape still falls short of meeting the everyday needs. Identifying both better ways to collaborate across all stakeholders and methodologies to rapidly, but robustly, develop and test innovations could help improve the translational hit rate. Therefore, here we took a first step to improve engagement with caregivers by undertaking a multi-stakeholder consultation to design a future roadmap for innovation in caregiver research.

#### **2. Materials and Methods**

Initially, this study was due to involve face-to-face interviews and focus groups, however, due to the COVID-19 pandemic we had to be agile and to achieve our aim of engaging with as many caregivers and other stakeholders as we could, we changed methodology and conducted the multi-stakeholder consultation via survey. We developed our own survey approach as we did not find a suitable existing one. Although the same content of surveys was delivered to both caregivers and professional groups (e.g., healthcare professionals, innovators), we tailored wording on occasion to represent each stakeholder group.

The surveys were iteratively designed to collate participant information regarding: (i) demographics, (ii) previous research involvement, (iii) feelings towards research and innovation, (iv) barriers to taking part in research, (v) future participation in research and, (vi) focus/outcomes of future research. No questions were mandatory and, thus, the total number of responses for each question varied.

Ethical approval was granted from the Computer and Information Science (CIS) Department Ethics Committee at the University of Strathclyde, Glasgow. The surveys, aimed at caregivers and those with a professional role interested in caregivers, were designed to take no more than 10 min to complete, and to be used across the UK. See Appendix A for example questions.

Inclusion criteria and survey design: Participants 18 years and over were invited to take part. A broad definition of an informal (unpaid) caregiver was shared with participants, "*People that provide unpaid care by looking after an ill, older or disabled family member, friend or partner*": hereafter referred to as caregivers. A minimum number of hours per week that caregivers needed to be caring for was not specified, leaving it up to participants to self- identify with the term. For professional groups, our participant information sheet stated that we were interested to find "*Professionals (not employed by Universities) who have an interest in innovating around carer health and wellbeing (e.g., health and social care professionals or those working in the technology industry/third sector)*". Where we state professionals, hereafter, this implies professionals/innovators interested in the caregiver demographic.

Survey distribution: Consent was implied after participants read, acknowledged, and accepted the initial terms of the anonymised survey. The distribution of the survey involved sharing the online version of our questionnaire through social media channels (e.g., Twitter), alongside email distribution through networks accessible to partners, such as the Digital Health & Care Innovation Centre (DHI) and Carers UK (e.g., Healthy Ageing Innovation Group and the "Digital Health & Care" Mailing list). The survey was open from 15 June 2020 until 30 September 2020.

Data handling: The survey was constructed using Qualtrics Software. Participants were reminded that any data entered must not contain any identifiable information. An integrated mixed-methods approach was taken for the data analysis. All quantitative analyses (frequencies and summary statistics) were completed using R studio (version 1.1.456). Qualitative analyses were undertaken using a content analysis approach by two researchers (KM and KE) [18]. Free text responses to questions were collated and the first question coded by KM. The coding structure cross-checked and agreed by KE as a measure of inter-rater reliability. Once agreed, the rest of the coding was completed by KM, before being cross-checked by KE. Data were frequently referred back to throughout this process to ensure the coding framework developed was appropriate. The flexible nature of content analysis tolerates the use of a combination of deductive and inductive creation of themes and patterns from the data. Deductive analysis was used to create the key themes and inductive analysis was used to create sub-themes. More specifically, the six key themes were identified from the survey structure (deductive analysis), sub-themes were identified through researcher (KM and KE) interpretation and coding of free text responses (inductive analysis). Data gathered from informal caregivers and professionals were analysed, and are presented, separately in the results section of this paper. Major themes were identified where at least half of respondents aligned with a specific finding. Where respondents identified as both a caregiver and a professional/innovator interested in caregiving, we classed these individuals as part of the caregiver demographic.

Data analysis: This work was developed without an a priori hypothesis but instead was conducted as an observational/explorative piece used to generate a future roadmap/hypothesis. Therefore, our statistical analysis is limited to frequencies, presented as stacked bars throughout the manuscript (we interpreted priorities where >50% of the respondents suggested an item was important/very important or equivalent). Qualitative analysis included the identification and discussion of key themes, such as clear priorities outlined by participants, throughout the narrative. For the purposes of readability and figure simplicity, long statements have been abbreviated and a full list of statements presented to participants can be found in Appendix B.

#### **3. Results**

#### *3.1. Description of Our Sample*

#### 3.1.1. Caregiver Demographics

The survey received 112 responses from informal caregivers (see Table 1). Caregiver respondents were aged 25 to up to 75 to 84 years old, with the mode age within the 45 to 54 years category. Over 85% (96/112) of caregiver respondents identified as a woman/female. Caregivers were highly educated, 60.7% were educated to degree level or equivalent, and 111/112 identified as white (Table 1). Just over half (50.9%, 57/112) of caregivers had been caring for 10 or more years and 6% (7/112) had been caring for 1 year or less. Caregivers varied in their ability to undertake work/study alongside their caring role: 35% were able to continue to work/study, 34% had been forced to reduce their work/study hours and 19.6% had been forced to give up work/study. Almost all (96%) of caregiver respondents were using digital technologies, such as smartphones and laptops on a daily basis. The vast majority of caregiver responses were from Scotland (96%, 108/112), with 1 response from Northern Ireland (<1%) and 3% (3/112 responses) from England.


**Table 1.** Participant Demographics. \* n = 61 for professionals' gender question.

#### 3.1.2. Professionals Demographics

The survey received 62 responses from professionals (Table 1). Professional respondents were aged from 25 to up to 55 to 64 years, most identified as women/female (79%) and 94% of respondents were white. The professional respondents were highly educated, 74% had a degree or equivalent level of education. When asked what their professional role was, 61 of the 62 participants responded. Most respondents (36%, 22/61) worked in healthcare, followed by 25% (15/61) working in social care, 23% (14/61) working in the 3rd sector, 11% of respondents put other, 3% worked in caregiver policy development and one respondent worked in a Small to Medium Enterprise (SME). All participants reported everyday use of technology. Similar to caregivers, 98% (60 /61) of professionals were from Scotland, with 1 response from England (2%).

#### *3.2. Views of Informal Caregivers*

Three key themes were established from the structure of our questioning and eight sub-themes were identified from the thematic analysis of the professionals free-text survey data. The three pre-specified (deductive) themes were; (1) Previous research participation, (2) Future research participation and (3) Future research aspirations. Sub-themes (identified through inductive thematic analysis) are presented in Table 2.



#### 3.2.1. Previous Caregiver Research Participation

Most respondents (79%, 89/112) had not previously participated in research being conducted by a university in relation to their role as a caregiver, while 18% (20/112) of participants had previously taken part in research. For those who had previously taken part in research, the majority had completed surveys: there was no reference by any of the caregivers to taking part in other forms of research, such as focus groups or interviews. Most had taken part previously due to a professional interest in research: for example, it being part of their job, or from a personal interest whereby participating in the research enabled access to specific support or services. The main reasons for not participating in research previously was lack of awareness, being unaware of any ongoing research or not being asked to participate in research prior to the current project. The other barrier towards participating in research was the lack of free time caregivers had allowing them to commit, given the other time pressures in their life. Time was, by far (both in Likert and free text responses) the most significant barrier to taking part in research, followed by money and personal health (See Figure 1).

**Figure 1.** Caregiver responses to a range of statements according to whether they "Strongly agree", "Agree", "Neither disagree or agree", "Disagree" and "Strongly Agree". Statements focused on whether: (i) caregivers have something useful to share with researchers; (ii) researchers could make potential impact on caregiver health and wellbeing, (iii) the role of a caregiver is understood, and (iv) whether caregivers feel connected to researchers at university settings. In addition, we explored views on the barriers of (v) time, (vi) money and (vii) health to participation in research.

Closer inspection of our Likert responses around current and previous experiences of caregiver research suggests over 75% of caregivers agree/strongly agree that caregivers have something "useful to share" with researchers, yet far fewer respondents felt that researchers could make a significant impact on their health and wellness as a caregiver (Figure 1). Few caregivers responding to this work felt they were connected to researchers in universities. When asked to rate whether they felt connected to university research, 29% (24/84) of participants strongly disagreed with this statement (Figure 1). It was suggested that university researchers do not have a particularly strong grasp of the realities of a caring situation.

#### 3.2.2. Future Research Participation

Online and community-based events, along with newsletters, were the most common ways in which caregivers felt they could contribute to university-led research in the future. However, when asked what the best way would be for caregivers to work alongside academia in the future, online and in person were the most popular options, and almost equally split among participants. The split opinion seems to be between the gold standard being face-to-face to ensure a rapport is built between caregiver and researcher, among other benefits, and the practicality and perhaps reality that online would be more suitable, given the time pressures faced by caregivers. The majority (53%, 39/73) of respondents would be willing to commit 30 min or more a month to participating in research.

#### 3.2.3. Future Research Aspirations

In terms of future focus for the health and wellbeing of caregivers, from Likert scale responses, there was a need stated for a diverse range of areas including: information-based research, remote monitoring technology, communication technologies to enable connection between caregivers and policy related research to help shape national agendas and policies (Figure 2a). When asked about different ways in which informal caregivers could provide input in research, many placed high value on all the options presented to them, including developing new ideas and new approaches to working together. Other popular options included ensuring that fresh solutions are relevant for other caregivers. Reviewing innovative technology ideas in development was not a clear priority for caregivers compared to other solutions. Thematic analysis of participant quotations highlighted a strong need to improve the support provided to caregivers. This included financial, emotional, psychological and training and educational support. It was very apparent that participants do not currently feel heard or that they have received sufficient levels and/or quality of support in their role as a caregiver.

**Figure 2.** (**a**) Caregivers' responses to a range of statements according to whether they are "Very important", "Important", "A little important", "Not at all important". Statements included whether caregivers were interested in (i) finding solutions, (ii) developing new approaches to work with researchers, (iii) measuring success, (iv) developing training and support, (v) planning ahead, (vi) developing new ideas, and (vii) reviewing new technologies. (**b**) A separate question for caregivers focused on whether future work would be "Very useful", "Useful", "Somewhat useful" or "Not Useful". Statements included research around: (i) information (ii) policy, (iii) communication, (iv) remote monitoring, (v) mobility, (vi) new ways of learning, (vii) voice activated technologies, (viii) finance based and (ix) VR/AR. VR = Virtual Reality, AR= Augmented Reality.

Further feedback for future work was obtained around which specific areas of collaboration would make the biggest impact on caregiver health and wellness (Figure 2b). There was a wide array of different interests in this group, for example, information science was a particular area of interest highlighted where caregivers viewed research that shares key information around: caring policy, rights, and entitlements as a priority. While more than 50% of the participants viewed research around mobility, innovation and learning, finance and voice activated research as useful/very useful, fewer participants were interested in Virtual Reality/Augmented Reality as a research priority.

#### *3.3. Professionals*

Three key themes were established from the structure of our questioning and eight sub-themes were identified from the thematic analysis. The three pre-specific (deductive) themes were; (1) Previous research participation, (2) Future research participation and (3) Future research aspirations. The key themes and sub-themes are presented in Table 3.


**Table 3.** Key themes and sub-themes from Professionals Survey.

#### 3.3.1. Previous Research Participation

Professionals were asked if they had previously been involved with or collaborated with universities and just over 62% (38/61) of respondents said they had not. When asked to comment further, many of those who had previous experience with research had done so in a professional capacity as part of their role. A key reason for having participated in research or being interested in doing so in the future was the importance and want to share their knowledge and experience.

The majority of professionals suggested that caregivers do not feel connected to universities or researchers and, similarly, many did not think that those working in universities have a good understanding of the challenges faced by caregivers (Figure 3). However, the vast majority of respondents felt that caregiver experiences would be extremely valuable to researchers and that research could have a significant impact on the health and wellbeing of caregivers (Figure 3). When asked about barriers to caregivers participating in research, lack of time was the most significant barrier, followed by money and then poor caregiver health.

#### 3.3.2. Future Research Participation

Professionals stated that online group activities and community-based events were the most popular ways which would make it easier or more likely for people to contribute to caregiver research 82% (46/56) and 66% (37/56), respectively. However, when asked to explain further, the importance of engaging those who are harder to reach and not simply using charity organisations as a means of recruiting caregivers to participate in research came through strongly in the free text answers. Perhaps, reflective of the COVID-19 situation people were living in at the time of completing the survey, online methods followed by in person face-to-face meetings were preferred with phone-based and postal methods being less popular. Many respondents, however, highlighted the benefits of face-to-face meetings.

#### 3.3.3. Future Research Aspirations

We asked professional respondents similar questions to caregivers around research priority areas. Although there was some indication that finding available solutions and measuring success were important, training and support, planning ahead, developing new ideas and developing new approaches to work with research were all a priority that over 50% of professionals thought were "important" or "very important" (Figure 4a). In terms of specific types of technology, respondents felt research and development into all areas of technology would be useful (Figure 4b). Those respondents appeared to think that what would be particularly useful were technologies enabling remote monitoring and communication technologies to facilitate connection between caregivers and other caregivers or professionals. Interestingly, virtual reality technology was of interest, but noticeably less so than other forms of technology (Figure 4b).

**Figure 4.** (**a**) Professionals' responses to a range of statements according to whether they are "Very important", "Important", "A little important", "Not at all important". Statements included whether caregivers were interested in (i) finding solutions, (ii) developing new approaches to work with researchers, (iii) measuring success, (iv) developing training and support, (v) planning ahead, (vi) developing new ideas, and (vii) reviewing new technologies. (**b**) A separate question for professionals focused on whether future work would be "Very useful", "Useful", "Somewhat useful" or "Not Useful". Statements included research around: (i) information (ii) policy, (iii) communication, (iv) remote monitoring, (v) mobility, (vi) new ways of learning, (vii) voice activated technologies, (viii) finance based and (ix) VR/AR.

When asked to elaborate on the most important outputs they felt could come from their involvement in caregiver research, the response fits into four key themes: (1) Solutions that are relevant to caregivers and truly reflective of their needs, (2) Long-term, sustainable solutions, (3) For caregivers to feel empowered and supported in their role, (4) Possible impact on policy and practice.

#### **4. Discussion**

The pressures on the caregiver demographic continue to grow. An ageing growing global population [19], the disruption caused by COVID-19 [8] and limited support for the caregivers are a perfect storm for a population already under strain. Recent UK statistics suggest that year on year, that stress outcomes have been worsening for caregivers, underlining the need for disruptive change [20]. Here, we have explored what a future roadmap for innovation could look like through a multi-stakeholder consultation and evaluation: an area of considerable importance for public health. Through incorporating the views of 174 individuals (including 112 informal caregivers and 62 professionals), there are a wide range of unmet needs, and an appetite to work closer with researchers and universities through both in person and online approaches (see Figure 5). While our responses suggest that working together to understand core needs is plausible for a sizable group of caregivers, care must be taken not to become overzealous in our interpretations many caregivers may remain difficult to reach, including those who are delivering care on 24/7 basis. Nevertheless, this work forms some early steps to engaging the wider caregiver demographic and, taken collectively, our findings suggest a need to: (i) identify and work sustainably with caregivers, (ii) listen to, and encompass the wide range of needs and perspectives of caregivers and other stakeholder groups, (iii) improve the quality and quantity of methodologies for caregiver research and (iv) widely share research knowledge (e.g., successful and non-successful innovations for research/implementation).

**Figure 5.** Summary figure summating both informal caregiver and professionals/innovators interested in the caregiver demographic.

While we achieved our main objectives, there are several limitations. Our work represents an online survey where there are biases caused by convenience sampling. For example, representation from ethnic minorities is relatively sparse, however, this was not a specific objective we aimed to address and further work is required to understand whether our findings are paralleled in such groups. We may also be inadvertently missing out on views on those with substantial caring commitments or those who do not engage in digital technologies. Second is

that, while we have attempted to be as objective as possible in our analyses and interpretations, our questions were pre-specified and, therefore, our qualitative analysis should be interpreted as deductive in terms of thematic approach. Third is that there are many important barriers to innovations that we cannot comment on, such as government incentives, cost-benefit models and start up environments. Our responses are based almost exclusively in the Scottish setting: in future, it would be advantageous to better understand other parts of the UK, alongside more nuanced differences such as differences between urban vs. rural settings. Finally, this work took place during the COVID-19 pandemic, which may have skewed representation to more digitally advanced groups of caregivers and missed important viewpoints.

Our main findings highlight a need to identify and work sustainably with caregivers: a task not straightforward given the lack of emotional, physical, and monetary support readily available. Moving forward, there is a need to find better ways to recognize the value that caregivers bring. This may be achievable through developing a more continuous interaction with caregivers, for example, moving beyond individual projects to open ended or problem focused collaborations. While many caregivers would welcome routes to engage with universities, researchers must see that the process needs to be efficient and inclusive: ideally taking no more than 20 to 30 min a month. Relevant to such a trade-off between researchers aims and citizen engagement is the concept of living labs. Living labs are a methodology/approach that can allow citizens to participate in the design, development, and evaluation of innovative solutions to address societal problems [21]. There is considerable interest in using such approaches to explore health and wellness problems, including scope to carefully explore a range of different issues through participatory design, across the life course [22,23]. These approaches are helping to redefine research methodologies, including the integration of citizen generated and nontraditional data sources [24,25]. Although careful consideration will be required of how to best build upon emerging online methodologies (e.g., ethical use of data), our findings here indicate that a hybrid model of both online and in-person approaches could work well for the caregiver demographic. "Citizen science" approaches could facilitate multiple research stages, such as ideation, co-design of the approach, data-gathering and knowledge transfer of findings through "light touch" interactions (e.g., smartphone/computer-based) or through community "pop up" events. Critically, this would allow multiple routes to help different groups of caregivers influence research agendas within the earliest stages [26]. Such integration could help address known issues around translational failure [15] and be supported through a wide variety of tools such as prioritization methodology [27] alongside recognition in academic publications [28].

Another key finding from this work is that caregivers (who were able to respond to this work) would largely welcome being "championed" and provided with a channel to engage with universities. However, the practicalities of delivering this are yet to be realized—political and social landscapes change over time, as do priorities. While both we and many other research groups [29,30] consult caregivers about current unmet needs on a regular basis, there remains a risk that some specific groups (e.g., according to age, geographical location, ethnicity, socioeconomic status, number of hours caring per week) will be excluded from conversations through selection bias [31]. Further work is needed to contextualize how representative current research is of all caregivers and, critically, we must establish community links to help those in need who do not traditionally connect with research agendas or third sector partners. Lastly, while the concept of sustained engagement proposed here may be new to many university settings, there is a need to ensure efforts augment, opposed to replace existing efforts ongoing elsewhere (e.g., health and social care data, charity, and voluntary sector annual reports).

Given that much of the conversation around caregivers, needs and innovations falls out with traditional models of empirical research (i.e., Randomized Controlled Trials), there is a need to keep furthering knowledge of research methodologies—this could be to improve the value of non-traditional data sources and/or to increase capacity for more caregivers to become involved. A source of inspiration that highlights passive involvement from participants includes the "Dreamlab" project from University College London [32], developed so that individuals can engage with researchers through charging their smartphone

to assist with cancer research. As the concept has already reached 1 million downloads, this is a clear indication that citizens are interested to connect and make impacts to research priorities. Further, the recent contributions of the COVID ZOE smartphone app have also underlined the power of citizen science, whereby seemingly small actions of individuals on a larger scale can accelerate current knowledge to help navigate the unchartered waters of an emerging pandemic [33]. Given such examples and our participant feedback, it is feasible to see smartphones being better used to connect caregivers (and other citizens/professionals) to ongoing research opportunities and emerging agendas, and to signpost others to recruitment calls. An immediate utility of such citizen science approaches could be to connect caregivers of rarer/under-researched conditions to highlight such needs to researchers and funding councils alike.

Another key take-home point from this work is the demonstration of the real depth of knowledge that caregivers hold for the development of future innovations and co-design approaches [34]. For example, caregivers demonstrated intimate knowledge of the daily barriers and enablers to existing technologies, and from the findings presented here, many caregivers are still actively seeking solutions around information needs and remote monitoring, problems that researchers may think are already solved. In terms of contributing specifically to ongoing research, findings from our own work and others [17,29] have highlighted key research topics for caring (e.g., access to information online), and such priorities could form the foundation of conversations with caregivers in a variety of settings. An achievable and realistic first step from here could be to pick exemplar caregiver priorities, and to run pilot projects that could facilitate the development of caregiver co-designed "principles" to help scaffold the way by which researchers engage with caregivers moving forward.

#### **5. Conclusions**

To conclude, the aims of this work were to initiate the development of a roadmap for future caregiver innovation. We found that, despite a number of barriers to participation, many caregivers are willing to regularly contribute to the research agenda and have a wide range of unmet needs that could be better addressed by existing and future research outputs. Visibility is still an issue to caregivers—we were only able to hear from a small subsample of the caregiver population here and even these participants did not feel visible and heard in research agendas. As a wider community, this needs to change. Clearly, accessible routes for engagement (e.g., integrated online and face-to-face approaches) and more formal recognition of their expertise would be welcomed alongside the delivery of real-world impacts on those who care. A next logical step would be to determine which approaches (such as living lab methodologies) are capable of delivering for caregivers not just in terms of research outputs/outcomes, but in terms of transparency, engagement, and experience.

**Author Contributions:** Conceptualization, K.J.E. and K.A.M.; methodology, K.J.E. and K.A.M.; writing—original draft preparation, K.J.E., K.A.M.; writing—review and editing, K.J.E., K.A.M., L.M., R.M., L.M. and M.L.; funding acquisition, K.J.E. and K.A.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received funding from the University of Strathclyde "Strathwide" Early Career Research Grant awards.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the University of Strathclyde Computer and Information Science Ethics Committee.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The full data are not publicly available to ensure anonymity of participants who took part in this work.

**Acknowledgments:** We would like to recognize the input of all caregivers and professionals involved in this work. We are also grateful to Patricia Clark at Carers Scotland (UK) and her colleagues to help us recruit carers to this work and to colleagues within the Digital Health & Care Innovation Centre (DHI) for assisting us to recruit professionals/innovators (including Moira MacKenzie, Kara McKenzie). We would also like to acknowledge contributions from the Centre for Lifelong Learning (CLL) at the University of Strathclyde—including Alix McDonald and Gemma Gilliland.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**

Q1 How old are you in years? Please select one.


Q2 How do you describe you gender? Please select one.


Q2.1 If you selected in another way, please share further information here: Q3 What is your highest level of education? Please select one.


Q4 What is your ethnicity? Please select one.


Q5 How long have you been a carer (in years)


Q6 Are you still able to work or study as you care?


#### **Appendix B**



#### **References**


## *Article* **Come for Information, Stay for Support: Harnessing the Power of Online Health Communities for Social Connectedness during the COVID-19 Pandemic**

**Brian M. Green 1,\*, Casey A. Hribar 2, Sara Hayes 1, Amrita Bhowmick 1,3 and Leslie Beth Herbert <sup>1</sup>**


**Abstract:** The COVID-19 pandemic created a globally shared stressor that saw a rise in the emphasis on mental and emotional wellbeing. However, historically, these topics were not openly discussed, leaving those struggling without professional support. One powerful tool to bridge the gap and facilitate connectedness during times of isolation is online health communities (OHCs). This study surveyed Health Union OHC members during the pandemic to determine the degree of COVID-19 concern, social isolation, and mental health distress they are facing, as well as to assess where they are receiving information about COVID-19 and what sources of support they desire. The survey was completed in six independent waves between March 2020 and April 2021, and garnered 10,177 total responses. In the United States, OHCs were utilized significantly more during peak lockdown times, and the desire for emotional and/or mental health support increased over time. Open-ended responses demonstrated a strong desire for connection and validation, which are quintessential characteristics of OHCs. Through active moderation utilizing trained moderators, OHCs can provide a powerful, intermediate and safe space where conversations about mental and emotional wellbeing can be normalized and those in need are encouraged to seek additional assistance from healthcare professionals if warranted.

**Keywords:** eHealth; chronic disease; online community; social support; COVID-19

#### **1. Introduction**

Although social isolation, loneliness, and mental health distress have always been commonplace, societal stigma often means they are not openly discussed [1]. However, the COVID-19 pandemic created a globally shared stressor that saw a rise in the emphasis on mental and emotional wellbeing. From popular icons such as Michelle Obama freely speaking about experiencing "low-grade depression" as a result of the pandemic and societal unrest in America [2], to the World Health Organization's emphasis on increased mental health infrastructure due to an impending and critical increase in demand [3], the COVID-19 pandemic has changed the way we talk about, prioritize, and consume healthcare and services that address mental health needs.

Despite still being in the midst of the pandemic, research so far points toward COVID-19 s ubiquitous negative impact on mental health [3–6]. A Kaiser Family Foundation survey from April to May of 2021 found that 30% of adults in the United States (U.S.) reported symptoms of depression or anxiety, a rise of more than 10% higher than expected based on pre-pandemic trends. In addition, nearly a quarter of those with symptoms of psychological distress who reported needing mental health services were not able to access them [7]. These COVID-19 impacts are far-reaching; however, factors such as a previous

**Citation:** Green, B.M.; Hribar, C.A.; Hayes, S.; Bhowmick, A.; Herbert, L.B. Come for Information, Stay for Support: Harnessing the Power of Online Health Communities for Social Connectedness during the COVID-19 Pandemic. *Int. J. Environ. Res. Public Health* **2021**, *18*, 12743. https://doi.org/10.3390/ ijerph182312743

Academic Editors: Irene Torres-Sanchez and Marie Carmen Valenza

Received: 29 October 2021 Accepted: 1 December 2021 Published: 3 December 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

lack of social support, perceived likelihood of survival, comorbid mental or physical health conditions, and challenging home demands, such as homeschooling young children, are all contributors to an increased risk for mental health distress [4,6].

In research conducted prior to the pandemic, social connectedness and social support were associated with benefits such as decreased risk of cardiovascular disease and depression, improved immune system functioning, and reduced morbidity and mortality, among others [8–11].

A metanalysis of 40 studies confirmed that comorbidities of mental health and chronic physical conditions are a burden for people, not just in the developed world where research often focuses, but also in developing and emerging countries [12]. While psychological interventions of various types can improve a patient's quality of life [13–15], there are mixed results depending upon in-patient or out-patient settings [16].

A positive state of mind as a result of strong social support, is also linked to greater medication adherence, a critical concern when it comes to living with one or more chronic health conditions [17].

Chronic health conditions are of particular interest in examining societal responses to the COVID-19 pandemic, as physical lockdowns and fears of contagion prevented the ease of access to in-person healthcare [18]. In addition, current research still suggests that those with pre-existing chronic conditions (including those with a history of malignancy) have a greater risk of developing severe COVID-19 and accompanying significant morbidity and mortality than those without [19,20]. Further, the population of chronically ill individuals tends to be older, and inherently possesses an increased susceptibility to social isolation and deteriorating mental health [21].

Especially in the context of the COVID-19 pandemic, telemedicine provides a means for patients to seek care for a variety of concerns in a convenient, accessible, and safe manner. While telemedicine for mental health is on the rise, there are still questions regarding patient willingness to embrace these new options when discussing sensitive issues. Additional concerns include payor issues with telemedicine services and the equitable access to technology required in order to undergo sessions [22–24].

One powerful tool to bridge the gap and facilitate social support and connectedness during times of isolation is online health communities (OHCs). OHCs are online platforms in which individuals with similar health conditions or experiences can share information, support, and connections [25,26]. OHCs provide a temporally flexible space that allows people to connect across geographic locations and often with anonymity [26,27]. OHCs allow patients to play as active or passive a role as they would like, while helping with identity development, self-confidence, personal validation, social interconnectedness, navigation of complex emotions, and fostering a sense of purpose [27–30]. Although OHCs are not a platform for medical care nor advice, they can provide an environment to gauge similar experiences and gain confidence to seek professional support.

In order to facilitate safe, online spaces that provide support, social connection, and information, Health Union created an Adaptive Engagement Model for OHCs [30]. At the beginning of the pandemic, Health Union had 28 active OHCs. Each community is a separate URL that matches the name of the condition, for example, Lupus.net, and has social media pages on Facebook, Instagram, and Twitter.

These OHCs are contextual and situational, and feature three core structural elements, including social support, adaptive engagement, and active moderation. Similar to physical communities, they rely on a shared identity (connection to a specific chronic health condition), social norms, and commonly an experience of societal stigma. These features, taken together in the creation and subsequent management and moderation of the OHCs, are used to facilitate the sharing of relevant health information (or content), build relationships, and harness participant interdependence through passive and active engagement opportunities [30]. Given the Adaptive Engagement Model's high emphasis on combating social isolation and creating space for normalizing conversations about life with a chronic illness, including mental health impacts, these OHCs may serve as a

stepping stone for intermediate support as people struggle with considering and initiating professional services for mental health or emotional support.

The present study was conducted primarily to determine the information needs and supportive resources needed by people with chronic health conditions from Health Union's 28 online communities during the pandemic. After seeing how community members continued to engage and support each other throughout the pandemic and become less interested in specific content regarding COVID, the authors decided to conduct a secondary analysis of these data to further study social connectedness.

The aim of this study was to survey current Health Union OHC members to determine the degree of COVID-19 concern, social isolation, and mental health distress they are facing, as well as assessing where they are receiving information about COVID-19 and what sources of support they require. Although other research focused on the mental and emotional impacts of COVID-19 and the use of telemedicine or other technologies to access healthcare services, our efforts aim to investigate the novel space in between, wherein OHCs may play a critical role in fostering wellbeing, especially when access to traditional in-person services may be limited.

#### **2. Materials and Methods**

#### *2.1. Data Collection and Survey Design*

As this was a secondary analysis of a de-identified data set originally collected for quality improvement purposes, it was considered exempt from human subject review. This research was conducted according to the guidelines of the Declaration of Helsinki and other relevant laws in the U.S. Informed consent was obtained from all survey participants via an introductory email, allowing people to voluntarily click through to accept or decline the invitation to complete the Qualtrics survey. Survey responses were anonymous and not linked to individual identifiers. All email or IP addresses were stripped from the data set prior to data cleaning, storage, and access of the final data set for analysis by the authors.

Data were collected in an online survey format hosted through Qualtrics Survey Software (Qualtrics International Inc., Seattle, WA, USA). The survey was administered in six waves between 19 March 2020 and 19 April 2021. The first survey wave was sent to members of 10 separate OHCs hosted, managed, and moderated by Health Union. Subsequent waves were expanded to eventually reach a total of 28 separate OHCs managed and moderated by Health Union. Each platform provides a space for support, engagement, and education around the chronic health condition of focus [30]. In total, the survey was sent to community members via email across the six waves. Banners and other site features advertising the survey were also displayed on each platform during data collection periods. Data collection for each wave lasted between three and eight days, tailored with the aim of securing an adequate sample size, while being inclusive of people with many different chronic health conditions (Table 1).


**Table 1.** Survey wave sample size, number of OHCs included, dates fielded, and completed responses.

In order to participate in the survey, respondents needed to be at least 18 years old, live in the United States, have a chronic health condition from a pre-specified list (full list in Supplementary Materials), and be aware of COVID-19. Survey questions focused on life with a chronic condition during the pandemic. Initial topics of interest included current treatments used to manage chronic health conditions, where information was obtained, and

concern regarding COVID-19. In addition, questions regarding changes in personal health behaviors as a result of the pandemic, changes in established treatment plans, desired support sources, communication with healthcare providers, and financial impacts were also included.

Given the evolving and uncertain nature of the COVID-19 pandemic, it was necessary to adjust topics slightly to most accurately reflect the current status of the pandemic for that time period. For example, waves three through six included questions about telehealth at the same time that telehealth was expanded by executive order through the Centers for Medicare and Medicaid Services during this time period [31]. Waves five and six included questions surrounding pandemic burnout, quality of life, and vaccination status. Due to this required flexibility, each wave ranged from 35 to 41 questions.

Participants who provided complete responses to the survey were entered in a drawing to receive a U.S. e-gift card for each wave. The drawing for the first wave featured USD 50 gift cards, while waves two through six featured USD 25 gift cards.

#### *2.2. Measures*

#### 2.2.1. Perception of COVID-19

Concern for COVID-19 was assessed in each wave through the question, "At this time, how concerned do you feel about the novel coronavirus (COVID-19)?" Concern was measured on a 7-point Likert scale (1 = Not at all concerned to 7 = Very concerned). Wave six (post-vaccine availability) had a slightly modified version of this question by gauging agreement with the statement, "I am still very concerned about my risk of contracting COVID-19" (7-point Likert scale with 1 = Strongly disagree and 7 = Strongly agree). In a similar fashion, participants were asked about their concern with COVID-19 in the context of having a chronic health condition in each survey wave (Supplementary Materials). Significant concern was a response of 6 or 7, while a lack of concern was a response of 1 or 2. In several waves (waves two, three, and four), participants were also asked in an open-ended question to provide one word that described how they were feeling about COVID-19.

#### 2.2.2. Utilized and Desired Sources of COVID-19 Information and Support

In order to determine current COVID-19 information resource use, respondents were prompted in waves one through four with the question, "What sources are you using to learn more about the novel coronavirus (COVID-19)?" Participants were prompted to select as many as applied from a list of resources, including social networking sites, Internet search engines, government websites, online blogs or support communities, and TV news reports (Supplementary Materials). Subsequently, participants were asked, "What types of information and/or support would be most helpful to you right now?" Respondents were permitted to choose up to three desired types of information or support including information from their healthcare provider about COVID-19 in relation to their health condition, emotional and/or mental health support, financial support for medications, and home delivery options (Supplementary Materials).

#### 2.2.3. Quality of Life and Health-Related Behavioral Changes

Later waves, specifically waves five and six, incorporated questions about quality of life, mental health impacts, burnout, and perception of returning to pre-pandemic "normal" life. Changes in more tangible behaviors were asked in wave five, through a 3-option ranking of doing less, doing the same as, or doing more of a specific behavior pre-pandemic versus present. For example, participants were asked to rank their current social media use with pre-pandemic levels on this scale. Less tangible changes, such as stress, impacts on mental health, and overall quality of life were asked in wave six using a 5-point Likerttype scale, including options such as mental health being much worse during COVID-19, somewhat worse, about the same during COVID-19, somewhat better, and much better during COVID-19 (Supplementary Materials). Wave five also featured the open-ended

question, "What is the biggest struggle that you're having at this point in time, as a result of (or related to) the coronavirus (COVID-19) pandemic?"

#### 2.2.4. Demographics

Participants across all waves were asked a series of demographic questions including age, gender, annual household income, primary health insurance form, residence type (suburban, urban, rural), highest level of education attained, and employment status. Age was selected from a dropdown menu, while others allowed participants to choose from a categorical list of item responses. Participants were asked to select chronic health conditions from a list including, but not limited to, COPD, migraine, asthma, HIV, rheumatoid arthritis, hypertension, and several types of cancers (Supplementary Materials).

#### 2.2.5. Analysis

Responses were analyzed using descriptive statistics and z-tests to explore differences across waves. This analysis was conducted in order to identify differences in the need for mental and/or emotional support and changes to quality of life relative to each wave of the survey and the corresponding time period of the pandemic. Responses to open-ended questions were reviewed for common themes and impactful quotes regarding the need for social support.

#### **3. Results**

#### *3.1. Participants*

In total, there were 10,177 responses across all six waves. The number of survey participants for each wave is shown in Table 1. Demographics of participants for each wave, including, but not limited to, mean age, gender, and employment status, were collected (Table 2). The most commonly experienced chronic health condition was hypertension, followed by asthma, migraine, rheumatoid arthritis, COPD, and multiple sclerosis. Nearly seven in ten had never been diagnosed with cancer. Of those diagnosed with cancer, skin cancer (of any form) was the most common.

**Table 2.** Survey participants; select demographics per wave.


The majority of participants were female; however, this is in line with prior research among people who were seeking health information or support online [30]. Unsurprisingly,

given the chronically ill nature of participants and the older average age, nearly half were either on disability benefits or fully retired, and roughly 30% were either employed full-time, part-time, or self-employed (Table 2).

#### *3.2. COVID-19 Concern*

The percentage of respondents reporting a 6 or 7 on the scale of general COVID-19 concern stayed relatively constant throughout waves one through five, peaking in the first wave with 71% of respondents (*n* = 699) and remaining in the mid-to-high 60% range through to wave five. In wave six, the question was adjusted slightly to inquire about the concern of contracting COVID-19 (after over one year of infections and the introduction of several vaccines). As such, significant concern for personally contracting COVID-19 in wave six was only 38% (*n* = 370). The trends in patterns of concern with regard to personal health history were the opposite for those with general chronic health conditions versus those with cancer. Those with a cancer history reported the highest levels of concern for COVID-19 in relation to their personal health at the beginning of the pandemic (wave one, 86%, *n* = 105). However, this significantly decreased to the 40% range for waves two through four (*p* < 0.01). In contrast, those with non-cancerous chronic health conditions saw a steady trend in concern for COVID-19 in relation to their personal medical history, with those reporting a strong concern hovering between 67% and 71% throughout the field of study.

#### *3.3. Self-Directed Research and Desire for Additional Support*

The top sources respondents used for COVID-19-related information throughout waves one through four were TV news reports (64%, *n* = 4591), government websites such as the CDC or NIH (61%, *n* = 4393), news websites (57%, *n* = 4099), and Internet search engines such as Yahoo or Google (47%, *n* = 3353). Social networking sites such as Facebook and Twitter were used by 38% of respondents (*n* = 2721), and online blogs and support communities were used by 17% (*n* = 1196). Internet searches, websites for healthcare professionals such as academic journals, TV news reports, and social media were all utilized more heavily in the beginning of the pandemic relative to later waves (*p* < 0.01 for wave one versus waves two, three, and four, individually). Online blogs and support communities were utilized significantly more in wave two (19%, *n* = 426, *p* < 0.01) than in any other wave, coinciding with peak lockdown times in the United States (April 2020).

When asked which types of information and/or support would be most helpful to respondents, the three most commonly chosen sources were up-to-date and accurate information about COVID-19 (47%, *n* = 2921), emotional and/or mental health support (25%, *n =* 1581), and financial support for other necessities/bills (22%, *n* = 1356). Most notably, the desire for emotional and/or mental support was highest in the final wave wave four (*p* = 0.01 compared to wave two and *p* < 0.01 for wave three).

#### *3.4. Burnout, Isolation, and Mental Health Distress*

The impacts on mental and emotional health were assessed most directly in waves five and six, after roughly a year of the COVID-19 pandemic. Nearly 60% (*n* = 1143) of respondents in wave five said the pandemic had increased the level of stress and/or anxiety in their daily life, compared to 9% (*n* = 183) who did not share this perception (*p* < 0.01), and 62% (*n* = 1240) were currently worried about returning to "normal" activities (*p* < 0.01). Additionally, in wave five, when asked about family and friends, 38% (*n* = 764) reported that they are keeping in touch with loved ones less than before the pandemic, compared to only a quarter (*n* = 500) reporting an increase in connection (*p* < 0.01). Notably, about one quarter of those who sought telehealth for any reason in this group did so for a mental health counseling or therapy session (25%, *n* = 371).

In wave six, 60% (*n* = 588) said quality of life was somewhat or much worse as a result of the pandemic, while only 9% (*n* = 86) reported an improvement (*p* < 0.01). When asked directly about mental health impacts, 56% of this group felt their mental health was worse or much worse during the pandemic compared to pre-pandemic (*n* = 547, *p* < 0.01), and only 6% (*n* = 63) reported improvements in mental health.

#### *3.5. One-Word Perceptions*

Responses to the one-word, open-ended question about current feelings regarding COVID-19 were reviewed and a word cloud figure was generated (Figure 1). In the word cloud format, the most commonly cited words are represented as larger in the figure. As shown in Figure 1, earlier in the pandemic, aligned with lockdowns and CDC calls to practice social distancing, the terms "anxious", "scared", and "concerned' were the most frequently cited. In wave six, more than a year after the pandemic began and with vaccines becoming available, words like "frustrated", and "tired" are still prominent; however, new terms like "hopeful", "optimistic", and "cautious" are entering into the vocabulary again, occupying more prominent positions in the shared consciousness.

**Figure 1.** One-word perceptions of the pandemic in word-cloud format: (**a**) Wave four (July 2020); (**b**) Wave five (April 2021).

#### *3.6. First-Hand Accounts of Isolation and Longing for Connection*

Wave five included an additional open-ended response where respondents were asked about their biggest current struggle in relation to the COVID-19 pandemic. Upon the first review of responses, isolation, stress, mental health concerns, and longing for social connectedness were common themes. Several notable responses included:

*"The biggest struggle for me is the isolation. The last time I was out to eat with friends or shopping in a store was the end of February 2020. It's a more mental/emotional struggle most days."*

*"Boredom due to seclusion. Normally I busy myself with helping others but being secluded in my room with little to do has begun to wear down my normally positive attitude."*

*"I don't have any help at home, and it's hard for me to manage. I feel incredibly isolatedwhich causes increased depression. I cannot even participate in communal worship because of immunosuppressant medications that increase my risk for COVID. I would like to work, as much as I can, but as a piano teacher, it is not possible, and that is more isolating and makes one feel more 'useless.' Being ill, unemployed, and having no one to have physical contact or interactions with is not normal. It is not conducive to mental health, [which is] hard enough for those with chronic pain and health conditions."*

Alongside survey responses, Health Union's OHCs were continuously monitored for trends in member engagement. Notably, concurrent with the start of the survey fielding in late March 2020 (and the initial increasing mandates to stay at home due to the COVID-19 pandemic), the following unprompted response was shared Health Union's OHC dedicated to asthma:

*"Warm greetings to everyone in the asthma world! As I sit here on day 22 of isolation during the COVID-19 pandemic, I am reminded of my reasons for wanting to share my experiences as an asthma patient and a lung cancer survivor, to name a couple of my health issues. So I am in isolation and I can't help but think about how grateful I am to have this forum to turn to. Not just in today's current environment but always. It is so helpful to hear from so many others who are in the same boat. Asthma often makes me feel isolated and alone. In reality, I am alone, but WE are together in our little corners of the planet doing our best to stay safe and healthy and live our lives. The current climate of the world has intensified that for us."*

#### **4. Discussion**

The overall objective of this study was to determine OHC members' changing perceptions of the COVID-19 virus and its impact on mental wellbeing and need for emotional or social support, as well as to determine where people are seeking their information and support needs from. Although government websites, TV news reports, and Internet searches dominated as information sources early in the pandemic, online blogs and support communities (including OHCs) reached their peak reported usage during the time of mass lockdowns and uncertainty in the U.S. (April 2020). This difference in the time of the pandemic suggests that people living with chronic health conditions sought social connectedness, validation, and peer-to-peer information in a time when they were experiencing more distress. This notion was further strengthened by the overall increasing desire for emotional and/or mental support as the pandemic progressed from the time of waves two through four.

While TV news sites, general Internet searches, and need for financial support were always of high priority to respondents no matter the time frame, the steadily increasing interest in OHCs speaks to the desire for human connection and social support from others during an unpredictable time. By providing a safe and always available online space for these connections, OHCs can provide trusted information, validate concerns and emotions, and provide social support in order to enhance wellbeing.

General concern for COVID-19, specifically concern about the risk of being infected, decreased by wave six when a multitude of infections in the U.S. had already occurred (including potentially among respondents or their family members, thereby potentially decreasing their fear of the unknown). "COVID fatigue" was at an all-time high according to public opinion polls and news reports [32], and several vaccines were newly granted emergency use authorization by the FDA and were becoming widely available. The concerns associated with the earlier phases of the pandemic, such as fear of the unknown and concern about becoming infected, were replaced by new concerns. These included burnout, negative quality of life impacts, reduced connection to loved ones, and mental health deterioration. A large majority reported that the pandemic increased the stress and anxiety they felt in their daily lives, and that they were still apprehensive about returning to a pre-pandemic "normal".

During this same time period, although daily COVID-19 concerns may have been subdued, these were replaced with mental and emotional exhaustion. Despite these high levels of distress, survey participants indicated that they did not engage with formal mental health services. In spite of the high overall use of telehealth services in wave five (73%), only a quarter of these visits were for mental health concerns. This suggests that although mental health issues and the need for support rose throughout the pandemic, there are still barriers (whether they are social, financial, practical, or physical) to being able to access mental health services. From a socialization standpoint, OHCs may be able to facilitate

trust among users of those communities and help normalize discussions around how to seek mental or emotional support, as well as provide encouragement and validation to others who may be uncertain about using telehealth for mental health services.

Based on responses to the one-word perceptions of COVID-19 and the first-hand accounts of isolation shared in response to open-ended questions, people with chronic health conditions are willing and ready to discuss these sensitive topics in OHCs. By continuing to utilize the Adaptive Engagement Model, conversations around emotional and mental health can be validated, and encouraged within a safe and supportive online environment.

OHCs are not without their own inherent risks, however, as shown in recent news accounts of the failure of leading social media companies to appropriately respond to negative impacts [33]. Personal attacks and factually incorrect medical information may arise, and depend on individual members to think critically about information presented and separate personal affiliation from safety [25,34,35]. This can be especially dangerous when it comes to sensitive discussions around emotional and mental health topics, and mentions of potential self-harm. In order to combat this, some OHCs, such as those operated by Health Union, utilized trained moderators, wrote community rules that were shared and enforced through moderation practices, and modeled appropriate responses to users of the OHC. Moderators may have a background in health or social services, and include experienced patient advocates and trained employees who monitor for safe discourse and provide conversation encouragement, resources, and validation where appropriate.

As mentioned above, there is also the risk for mentions of self-harm when creating open dialogue around sensitive and emotional issues. One aspect of OHCs is the unpredictability of people sharing comments or concerns that may not be germane to shared content or topics, and this is often the case when people mention mental health or emotional health concerns. As the COVID-19 pandemic shows in stark relief, existing OHCs, organized around health topics, may quickly become the source that people turn to when seeking support from others. Being prepared for such conversations, with both a strategy as well as an experienced and trained group of community moderators, is critical to quickly adapt to the demands of the pandemic and similar public health crises.

#### *Limitations*

There are several limitations to this study. First, in order to adapt to the changing nature of the COVID-19 pandemic while keeping surveys manageable for chronically ill respondents, several questions from the first survey were removed or tailored across subsequent waves. This made it impossible to compare responses to all variables across all six waves; however, the sample sizes for each survey wave were large enough to gain an understanding of data points of interest from smaller wave groups. Additionally, participants were recruited from people who visited one of the 28 OHCs, clicked on a survey advertisement, and proceeded to the survey consent page, or who had previously opted-in to receive email communications. This recruitment method may have led to sampling bias, specifically for those who had already participated in an OHC environment. Further, although the survey was designed to be as user-friendly as possible, it was not completed by all participants which may indicate that those who completed the survey are more comfortable with the online survey technology, and thus, OHCs.

Future research would be best served by investigating need for mental and/or emotional health support at this current juncture in the pandemic, and by utilizing similar or identical questions throughout all iterations that further explore why people choose to come to (or avoid) OHCs, what information they may be looking for, how they find support or give support to others within these spaces, and if they feel empowered to seek further help from a healthcare professional when needed.

#### **5. Conclusions**

Along with all of society, individuals with chronic illnesses fully experience the impacts of the COVID-19 pandemic, and are likely and are likely to seek out information early and avail themselves of risk reduction strategies. Changes in desired resources over time show an increasing interest from basic information about the virus, transmission risk, and ways to minimize risk, to seeking emotional support. The interest and willingness to talk about mental health impacts and the need for support speaks to the desire for human connectedness in times of societal upheaval and the resultant severe isolation.

The literature reviewed provides a backdrop for the current study and illustrates issues relevant to social connectedness for people with chronic health conditions. This includes the potential negative impact of social media on mental health, which has salience for people with chronic health conditions, particularly during the pandemic. While the increase in the availability of telehealth during the pandemic has the potential to increase the use of mental health services, it is less clear that people were able to avail themselves of these opportunities.

The first-hand accounts of people living with chronic health conditions struggling to find support and social contact are illustrated in the responses to open-ended questions and show a range of emotional impacts and coping strategies. Although mental and emotional distress are common, not all who need professional support feel comfortable or know how to access those services.

OHCs can provide an intermediate and safe space where conversations about mental and emotional wellbeing can be normalized and those in need are encouraged to seek additional assistance from healthcare professionals if warranted. However, OHCs cannot do this through passive engagement only. Active moderation of OHCs using trained and experienced moderators can provide a safe space with planned, real-time strategies to address crisis situations, including future pandemics or public health emergencies, as they arise. This research supports the thesis that OHCs, when managed and moderated appropriately, have the power to normalize mental health discussions, thus providing a unique value to those who experience mental health concerns.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/article/10 .3390/ijerph182312743/s1, File S1: Wave 5 Survey Questions.

**Author Contributions:** Conceptualization, B.M.G. and S.H.; survey questions, S.H. and L.B.H.; methodology, L.B.H.; formal analysis, L.B.H.; resources, A.B.; data curation, L.B.H.; writing—original draft preparation, C.A.H.; writing—review and editing, B.M.G., C.A.H., S.H., A.B. and L.B.H.; project administration, L.B.H. and A.B. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** This study was determined as Exempt under 45 CFR 46.101(b)(#2). The study was conducted according to the guidelines of the Declaration of Helsinki and other relevant laws in the U.S.

**Informed Consent Statement:** Informed consent was obtained from all survey participants involved in the study.

**Data Availability Statement:** The data presented in this study are available by request from the corresponding authors. Additional summary reports using Data from the surveys described in this study are available here: https://health-union.com/blog/covid-19-resources-pharma-marketing (accessed on 30 November 2021).

**Conflicts of Interest:** The authors declare no conflict of interest with the reported research results.

#### **References**


### *Article* **The Interactions of Media Use, Obesity, and Suboptimal Health Status: A Nationwide Time-Trend Study in China**

**Qinliang Liu and Xiaojing Li \***

School of Media & Communication, Shanghai Jiao Tong University, Shanghai 200240, China; liuql201807@sjtu.edu.cn

**\*** Correspondence: lixiaojing@sjtu.edu.cn

**Abstract:** Obesity and suboptimal health status (SHS) have been global public health concerns in recent decades. A growing number of works have explored the relationships between media use and obesity, as well as SHS. This study aimed to examine the time trend of the associations between media use (including traditional media and new media) and obesity, as well as SHS. The data were derived from three national random samples of the Chinese General Social Survey (CGSS), which was separately conducted in 2013, 2015, and 2017. In total, 34,468 respondents were included in this study, consisting of 16,624 males and 17,844 females, and the average age was 49.95 years old (*SD* = 16.72). It found that broadcast use and television use were positively associated with obesity and showed an increasing trend over time. Cellphone use emerged as a risk factor for obesity in 2017 and showed an increasing trend. By contrast, newspaper use, television use, and internet use were negatively associated with SHS, and television use showed a decreasing trend in the association with SHS, while internet and newspaper use showed an increasing trend. In conclusion, media use was positively associated with obesity while negatively associated with SHS. It showed a decreasing trend in the associations between traditional media use and obesity, while revealing an increasing trend in the associations between new media use and obesity, as well as SHS. The practical implications of the findings are discussed.

**Keywords:** media use; obesity; suboptimal health status; time trend; China

#### **1. Introduction**

Over the recent decades, communication technology has changed dramatically. A variety of new media, including the internet, computers, cellphones, and social media have rapidly come to coexist with traditional media (e.g., television, newspaper, and broadcast media), which dominate individual's leisure activities and spare time [1]. Media use has also been integrated into daily life to communicate with peers and maintain social relationships [2], search for or share information, and have fun or entertainment [3]. Despite the ease and frequency of media use having changed radically in recent years, these changes did not always bring positive impacts on health status, but did bring negative consequences [4]. In other words, media use is a two-sided coin that may improve one's health status, but also could result in harmful health outcomes. Widespread and prolonged media use has contributed to the increasing and ongoing debate on its impacts on physical and psychological health, such as obesity or overweight, and the symptoms of suboptimal health status (SHS).

#### *1.1. Media Use and Obesity*

Obesity has been a global public health concern which has caused various health problems and social burden [5,6]. Recent work has indicated that obesity has become widespread in adolescents as well as in young and older adults [7,8], with an increasing trend in recent decades [9]. The rapid rise of obesity suggests that environmental factors along with biological disposition might be responsible for people's weight gain [10].

**Citation:** Liu, Q.; Li, X. The Interactions of Media Use, Obesity, and Suboptimal Health Status: A Nationwide Time-Ttrend Study in China. *Int. J. Environ. Res. Public Health* **2021**, *18*, 13214. https:// doi.org/10.3390/ijerph182413214

Academic Editors: Irene Torres-Sanchez and Marie Carmen Valenza

Received: 26 October 2021 Accepted: 13 December 2021 Published: 15 December 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Among the environmental factors, ubiquitous information communication technology and pervasive media use might account for the widespread obesity [11]. One of the possible explanations lies in the fact that electronic media use or screen media use have displaced time for physical activities or outdoor sports. The more or longer one uses such media, the more likely s/he is to become obese [12,13]. A large body of studies have also examined the associations between media use and obesity or overweight [14–16]. The scope of these studies has involved almost all media types, such as television [17], video games [18], internet and computer [19], social media [20], and digital media [21]. The research outcomes have demonstrated that media use has caused a decrease in physical activity and an increase in sedentary behaviors, which has resulted in the risk of obesity [22,23].

#### *1.2. Media Use and Suboptimal Health Status*

SHS is an intermediate state between health and disease conditions, characterized by declines in vitality, physiological function, and the capacity for adaptation [24]. It includes symptoms of fatigue, non-specific pain, dizziness, anxiety, depression, and functional system disorders [25]. In recent years, more and more people are in SHS. SHS has become a significant public health concern worldwide due to its potential risk for chronic diseases across multiple populations, such as type-II diabetes mellitus, cardiovascular and stroke [26,27].

According to the "biological-psychological-social" medical model, an individual's health status is dictated by his/her biological, psychological, and social factors. Thus, SHS is usually measured or assessed by disturbance symptoms in terms of physiological, psychological, and sociological characteristics, namely physical SHS (PS), mental SHS (MS), and social SHS (SS) [28]. Given that SHS is a medically undiagnosed and functional somatic syndrome, it has been attributed to various factors. Existing studies have revealed that lifestyle and environmental factors were the critical predictors of SHS, such as sleep quality, physical exercise, unhealthy diet, smoking, alcohol drinking, adverse life events, and family living status [29]. In addition, the prevalence of SHS varies among different populations. Many demographic variables, such as age, gender, education, occupation, geographical region, and economic status, were associated with SHS [30].

As a critical predictor of SHS, screen-based media (SBM) has occupied a considerable portion of young peoples' discretionary leisure time [3]. The overuse of electronic devices might be one of the underlying causes of SHS [31]. However, the results related to SHS were mixed. According to the social causation hypothesis, unreasonable media use could induce health problems since excessive or problematic media use increased the risk of poor sleep quality, obesity, lower life satisfaction, and anxiety [32,33]. In addition, social media provided a natural platform for social comparison, which led to more mental health disorders [34]. For instance, more social media use was proved to be positively correlated with weight dissatisfaction and poor body image, which resulted in lower self-esteem and higher depressive symptoms [35]. Those active on Facebook tended to score higher in terms of poor health status than those who used other social media platforms [2].

However, others have argued that individuals could significantly benefit from media use, whether in terms of physical health or psychological health. The social compensation hypothesis interprets that media use can compensate for what individuals lack in real life, including social resources and support [36]. Primarily, individuals view online social media as an attractive way to attain social support and social engagement which they lack offline [37]. These online media are easy to access and anonymous, providing users with a sense of connectedness and reducing feelings of isolation [38]. The new media enable individuals and communities to acquire and maintain social capital. All these benefits have a positive effect on mental health [39]. Besides, electronic or digital media have become the most popular platform for accessing health information [40]. They play critical roles in changing one's attitudes toward health behaviors and persuading individuals to participate in health protection [41]. Furthermore, health-related media use was positively linked to health literacy [42], as well as eHealth literacy [43], which in turn influenced health behaviors and contributed to physical health [44].

#### *1.3. The Current Study*

Although prior studies have provided valuable knowledge to understand the relationships between media use, obesity and SHS [32,45–47], several limitations still exist in the body of scholarship. Firstly, most studies used cross-sectional data and demonstrated a concurrent relationship between media use and obesity or SHS. They could not demonstrate any causal effect over time [48]. Secondly, existing studies preferred to focus on a single type of media use to explore its associations with obesity or SHS, such as television, internet or social media, lacking a comparative perspective among different media [49–51]. Besides, most have focused on new media, ignoring traditional media which are still relevant, especially for certain age groups. Thirdly, most studies were executed in developed countries. However, developing countries like China are experiencing severe obesity and SHS problems [52] and encountering dramatic changes in media technologies in the 21st century.

Therefore, to gain a comprehensive and longitudinal view of the influence of media use, it is necessary to test the time trend of the supposed causality between media use and health status, as well as to consider the disparities between different media uses. The current study aims to reveal the time changes in the associations of media use with obesity and SHS from 2013 to 2017. Further, to compare the effects of different media use, we divided the media genres into traditional media (newspaper, magazine, broadcast, and television) and new media (internet and cellphone). Hence, the specific research questions were proposed:


#### **2. Materials and Methods**

#### *2.1. Data Collection*

The data was drawn from the Chinese General Social Survey (CGSS) in 2013, 2015, and 2017. The CGSS is a nationally presentative and continuous social survey project, implemented on the Chinese mainland by the Renmin University of China and Hong Kong University of Science and Technology annually since 2003. It has been broadly considered as the Chinese counterpart of the General Social Survey (GSS) in the United States. Using a stratified multi-stage probability proportionate to size (PPS) sampling method, samples were drawn from households in all 31 provincial units in Mainland China with three levels in each sampling frame: county or district, community, and household. In a selected county (district), four community-level units (neighborhood committees or village committees) were randomly selected. In a selected communitylevel unit, 25 households were selected, and in each selected household one adult was randomly selected for the survey [53,54]. CGSS is a household survey aimed at adults, and adolescents under 18 years were excluded. All the questionnaires were completed by face-to-face interview with the help of interviewers. 12,000, 10,968, and 12,582 sample cases were collected from 480 villages and urban neighborhood communities from 140 residential districts in 2013, 2015, and 2017, respectively. After excluding the missing answers, e.g., one participant who responded to the options "I don't know" or declined to answer, or in the case of an incomplete questionnaire, then the case was removed from the database. Finally, 11,263, 10,383, and 12,367 respondents were retained in the samples for 2013, 2015, and 2017, respectively. All the samples were pooled as the analysis database.

#### *2.2. Measures*

#### 2.2.1. Media Use

Media use was measured by the following question: "In the past year, how often did you use the following media?" Media types consisted of newspaper, magazine, broadcast, television, internet, and cellphone. The answers were independently rated as "never", "seldom", "sometimes", "often", and "always".

#### 2.2.2. Obesity

Obesity was indicated by body mass index (BMI) that was calculated as weight/height2. According to obesity criteria, theWHO defined overweight in adults as a BMI of 25.0–29.9 kg/m2 and obesity as a BMI of 30.0 kg/m<sup>2</sup> or higher. However, accumulated evidence has consistently supported the use of lower BMI cutoffs in Chinese than those in whites [55–57], and the Working Group on Obesity in Asia also recommended BMI cutoffs of 24·0 kg/m2 to define overweight and 28·0 kg/m2 to define obesity [58]. As proposed by Zhou (2002) [59] for Chinese adults, the standard weight was defined as a BMI < 24 kg/m2, overweight as a BMI of 24–27.9 kg/m2, and obesity as a BMI ≥ 28 kg/m2. Consequently, each respondent was characterized as obese or not by the cut-off value of 28. The attribute of obesity was marked as "yes" or "no".

#### 2.2.3. SHS

SHS was measured by three questions with self-defined items. According to the "biological-psychological-social" medical model, SHS was defined as physical SHS (PS), mental SHS (MS), and social SHS (SS) [28]. We selected the three most related questions responding to the three facets. The first two questions were: "In the past four weeks, what is the frequency of health problems affecting your work and other daily lives?" and "In the past four weeks, how often did you feel depressed?" The answers were rated as "never or seldom", "occasionally", "sometimes", "often", and "always". The third question was, "What is your current health status?" The answers were rated from 1 to 5, as "excellent", "good", "fair", "poor", and "very poor". All the answers were inversely coded.

According to prior studies' outcomes for Cronbach [60] and Bland & Altman [61], Cronbach's alpha was used to investigate the internal consistency of the questionnaire, and the alpha should be more than 0.7 to 0.8 for research purposes and at least be 0.90 for clinical purposes. In this study, the Cronbach's alpha of SHS was 0.774 (α = 0.774), indicating a better internal consistency for the constructed measurement items, which enabled SHS to be measured reasonably. Finally, the summed score of the three questions was marked as the measurement of SHS, and a higher score indicated a more severe SHS.

#### 2.2.4. Covariates

Covariates included sociodemographic characteristics, such as gender (male/female), age (years old), educational level (illiterate/primary school/middle school/high school/college or bachelor's degree and above), and physical activity (inversely coded as "never", "several times/year", "several times/month", "several times/week", and "every day"). Hukou (rural/urban) was included to control for the unobserved geographic characteristics of the respondents' residential location. In China, people are mainly divided into two groups (agricultural hukou holders and non-agricultural hukou holders) based on their birthplace and lineage. Generally, people with agricultural hukou live in rural areas with disadvantages in terms of the economy, education, medical, housing, and other social resources. By contrast, non-agricultural hukou holders who live in urban areas with more public and social resources have a high quality of life. So agricultural hukou and non-agricultural hukou are also called rural hukou and urban hukou [62]. Obesity was included in the regression analysis of the association between media use and SHS, coded as "yes" or "no".

#### *2.3. Statistical Analysis*

Descriptive statistics were employed to describe the characteristics of sociodemographic indicators, media use, rate of obesity, and SHS. One-way ANOVA was used to analyze the P-trend of each variable by period. Logistic regression models were applied to estimate the associations between media use and obesity separately in 2013, 2015, and 2017. The analysis results were presented as odds ratios (ORs) with 95% confidence intervals (CIs), with all covariates being controlled. Multiple linear regression models were applied to estimate the associations between media use and SHS separately in 2013, 2015, and 2017. The results were presented as Beta and SE (standard Error) and controlled all covariates. To test whether the associations between media use and obesity as well as SHS varied over time, the cross-sections were pooled pairwise (e.g., 2013–2015, 2015–2017, and 2013–2017), and dummy-coded as an independent variable separately. For example, in the period of 2013–2015, 2013 was coded "0" and 2015 was coded "1". Interactions of media type with the pairwise year were included separately in the adjusted models to determine whether the association changed significantly over the study periods [63,64]. All statistical analyses were conducted via SPSS 25.0. *p*-values were two-sided, and a value of less than 0.05 was considered statistically significant.

#### **3. Results**

#### *3.1. Characteristics of Demographics, Media Use, Rate of Obesity, and SHS*

Table 1 shows the characteristics for demographics, media use, rate of obesity, and SHS. In total, 34,468 respondents were included in this study, comprising 16,624 males and 17,844 females, and the average age was 49.95 years old (*SD* = 16.72). More than half of the respondents had less educational experience (64.08%) and were rural Hukou (58.98%), and all the respondents did physical activities (Mean = 2.34, *SD* = 1.52) at a relatively low level. As for the frequency of media use, television was rated most among the six media types, while it showed a decline and downward trend from 2013 to 2017, as well as newspapers, magazines, and broadcast. However, internet and cellphone use had an increasing tendency during the same time period. In addition, the prevalence of obesity increased from 5.7% to 7.6%, and the score of SHS increased from 6.39 (*SD* = 2.61) to 6.84 (*SD* = 2.65) during 2013 to 2017. All changes between different years were statistically significant.

#### *3.2. Associations between Media Use and Obesity, and Time Changes*

Table 2 shows the outcomes for logistic regression analysis on the associations between media use and obesity after controlling for age, gender, education, and physical activity. The associations varied by each time point. In 2013, television use was significantly and positively associated with higher odds of obesity, while magazine use was associated with decreasing odds of obesity. Broadcast and television were the significant predictors of obesity in 2015, whereas cellphone use emerged as a new predictor of obesity in 2017. Internet use was not significantly associated with obesity at any time point. These results answered RQ1.

As for the time trend of the association of media use with obesity (see Table 3), the significant interaction term indicates a time trend in pairwise time, but whether it is a decreasing or increasing trend depended on the changes of the predictive effect of media use on obesity during that period. Therefore, newspaper use showed a decreasing trend in the association with obesity since the positively predictive effect declined from 2013 to 2017. Magazine use negatively predicted obesity in 2013, but the predictive effect declined significantly during 2013–2015 and 2015–2017, which showed decreasing association trends over those periods. Broadcast use exhibited a decreasing time trend in the association with obesity during 2015–2017, because the positively predictive effect declined in that period. The positively predictive effect of television use on obesity declined from 2013 to 2017, but these changes were not significant. Therefore, there was no time trend in the association between television use and obesity. Cellphone use positively predicted obesity in 2017

and the predictive effect increased significantly during 2015–2017, thus the association between cellphone use and obesity showed an increasing trend in that period. These results answered RQ3.


**Table 1.** Descriptive characteristics of demographics, media use, obesity, and SHS in 2013, 2015, and 2017.

Note. N-Number, M-Mean, SD-Standard Deviation. Time change was tested by one-way analysis of variance (ANOVA) with Tamhane's T2 since the unequal variances. 1–2 = significant difference in mean between 2013 and 2015, 1–3 = significant difference in mean between 2013 and 2015, 2–3 = significant difference in mean between 2013 and 2017. Detailed information is presented in supplementary material, Table S1.

**Table 2.** Logistic regression analysis of associations between media use and obesity in 2013, 2015, and 2017.


Note. Adjusted for gender, age, rural/urban, education, physical activity. \* *p* < 0.05, \*\* *p* < 0.01, \*\*\* *p* < 0.001. N/A—not applicable. OR—odds ratio, 95% CI—95% confidence interval.


**Table 3.** Logistic regression analysis for the change in the associations between media use and obesity in 2013, 2015, and 2017.

Note. Adjusted for gender, age, rural/urban, education, physical activity, \* *p* < 0.05, \*\* *p* < 0.01, \*\*\* *p* < 0.001. OR—odds ratio, 95% CI—95% confidence interval. a: The pairwise time points are dummy-coded as the independent variable "Year", 2013 was coded "0"and 2015 was coded "1". b: The pairwise time points are dummy-coded as the independent variable "Year", 2015 was coded "0"and 2017 was coded "1". c: The pairwise time points are dummy-coded as the independent variable "Year", 2013 was coded "0"and 2017 was coded "1".

#### *3.3. Associations between Media Use and SHS and Time Changes*

Table 4 shows the outcomes of linear regression analysis on the associations between media use and SHS after controlling for age, gender, education, physical activity, and obesity. Compared to obesity, the findings were somewhat different; excluding broadcast, the use of newspaper, television, and internet were negatively and significantly associated with SHS, separately in 2013, 2015, and 2017. Broadcast use was positively and significantly associated with SHS only in 2013. Magazine use was a significant predictor of SHS in 2017. Cellphone use was not a significant predictor of SHS at any time point. Thus, RQ2 was answered.

As for the time trend analysis of the association of media use and SHS (see Table 5), compared with 2013 and 2015 the negatively predictive effect of newspaper use on SHS increased significantly in 2017, thus the association of newspaper use and SHS had increasing trends both in 2015–2017 and 2013–2017. On the contrary, the positively predictive effect of broadcast use on SHS declined significantly during 2013–2015 and 2013–2017, so it presented a decreasing trend in the association and SHS, and the trend was more significant during 2013–2017. In addition, the negatively predictive effect of television use on SHS declined rapidly from 2013 to 2017, and all the changes were significant among these periods, which indicated decreasing trends in the associations between television use and SHS. By contrast, internet use negatively predicted SHS in 2015 and the predictive effect increased significantly in 2013–2015 and 2015–2017, thus the association between internet use and SHS showed an increasing trend in 2015–2017 and 2013–2017. In addition, magazine and cellphone use showed no significant time trend. These outcomes answered RQ4.


**Table 4.** Hierarchical multiple regression analysis of associations between media use and SHS in 2013, 2015, and 2017.

Note. Adjusted for gender, age, rural/urban, education, physical activity, and obesity. \* *p* < 0.05, \*\* *p* < 0.01, \*\*\* *p* < 0.001. N/A not applicable.

**Table 5.** Hierarchical multiple regression analysis for the change in associations between media use and SHS in 2013, 2015, and 2017.


Note. Adjusted for gender, age, rural/urban, education, physical activity, and obesity. \* *p* < 0.05, \*\* *p* < 0.01, \*\*\* *p* < 0.001. N/A—not applicable. a: The pairwise time points are dummy-coded as the independent variable "Year", 2013 was coded "0", and 2015 was coded "1". b: The pairwise time points are dummy-coded as the independent variable "Year", 2015 was coded "0", and 2017 was coded "1". c: The pairwise time points are dummy-coded as the independent variable "Year", 2013 was coded "0", and 2017 was coded "1".

#### **4. Discussion**

Obesity and SHS have been worldwide health issues as living environments and life patterns have changed rapidly in the past decade [29,65]. The relationships between media use and obesity as well as SHS have been well documented in previous work [35,66,67]. This study aimed to better understand the media use profiles of Chinese adults and the changes in relationships between media use and obesity as well as SHS from a time trend perspective, based on a longitudinal national sample conducted among Chinese adults in 2013, 2015, and 2017.

It revealed that the profiles of media use changed significantly in China from 2013 to 2017, and the changes varied depending on different media types. Traditional media, such as newspapers, magazines, broadcast media and television use, presented a decreasing trend, and newspapers and television declined more significantly. On the contrary, the use frequency of new media (i.e., internet and cellphone) maintained a significantly increasing trend from 2013 to 2017, and internet use increased more than cellphone use. These trends were consistent with the dramatic progress in communication technologies and the rapid growth in Chinese internet users. According to the 34–39th Statistical Report on Internet Development in China, the number of Chinese internet users, internet penetration rate, and cellphone users had increased by 22.2%, 10.9%, and 41.7%, respectively, from 2013 to 2017 [68,69]. However, the opposite development trends of traditional media use and new media use did not mean that new media (i.e., internet and cellphone) had totally displaced the role of traditional media (i.e., newspapers, magazines, broadcast, and television) in China. Conversely, as our study revealed, television was still more popular than other media types among Chinese adults during the period to 2017. Internet was ranked as the second primary media use, behind television use. The possible explanation is that most of the participants had an older age, and may be prone to use traditional media (i.e., television) more than new media (i.e., Internet or cellphone) due to their relatively low level of media literacy or digital skills [70]. In addition, more than half of them were from rural areas, and may have a lower level of education and household economy, which reduces their chances of internet use [71].

Consistent with the time trend of new media use, both the prevalence of obesity and the magnitude of SHS increased significantly from 2013 to 2017, indicating that a higher proportion of Chinese adults suffered from overweight and health problems during that period. These findings were in line with the latest research findings that the prevalence of Chinese obesity changed from 3.1% (2.5–3.7) in 2004 to 8.1% (7.6–8.7) in 2018 [72], and more than half of Chinese adults showed SHS symptoms [24]. Obesity and SHS have become serious public health issues in China, but most current studies attribute the cause to economic developments, socio-cultural norms, food systems, and environment [73]. This study provided useful evidence for the assumption that media use was associated with obesity and SHS and found a significant time trend in these associations from 2013 to 2017.

Broadcast, television, and cellphone use were positively associated with obesity at different time points. Magazine use was negatively associated with obesity in 2013 while the association disappeared in 2015 and 2017. These findings suggested that both traditional and new media use might be risk factors for obesity. There was a significant time trend in the associations between media use and obesity in the study periods, and traditional media and new media demonstrated totally opposite tendencies. Traditional media use (i.e., magazine, broadcast, and television) showed a declining tendency in association with obesity from 2013 to 2017, while new media use (i.e., cellphone) showed an increasing tendency during that period. Cellphone use was increasingly important in predicting obesity and this trend was more significant in 2017 (see Table 2). The underlying mechanism of how media use influences obesity perhaps lies in the fact that media use increases sedentary possibilities and decreases physical activities.

Cellphones are usually characterized by multiple functions and are easy to use for all age groups compared with traditional media. Previous studies also found that cellphone did indeed occupy a large amount of leisure time and even led to cellphone addiction (extensive use or overuse), which brought about obesity or other health problems [74,75]. Furthermore, cellphone use was the only significant predictor of obesity in 2017, emphasizing that the cellphone had displaced traditional media and had the leading risk factor of obesity among the six media types in 2017. Nevertheless, it is worth mentioning that television use showed a decreasing tendency in associations with obesity from 2013 to 2017, while the time trend was not significant. In other words, television use was a stable predictor of obesity among the six media types over the study period. Therefore, television may be a potential risk factor for obesity in the future years.

In the existing literature, media use was demonstrated as a beneficial factor for SHS since it provided health information resources, social engagement, and social support [76]. In this study, newspaper use, television use, and internet use were negatively associated with SHS at each time point. Meanwhile, there were significant trends in the associations of media use with SHS, and the trends differed between media types.

Television showed a rapidly decreasing trend in the association with SHS from 2013 to 2017, while internet use showed a significant increasing trend. Further, the associations between internet use and SHS were stronger than the associations between television use and SHS in 2015 and 2017, indicating that internet use has displaced television and become the strongest predictor of SHS since 2015. The possible explanation is that the internet has been the leading platform and carrier of health information and has become more popular for health information access than television in the digital age [40]. Free access to vast online health information sources has created opportunities for empowerment, information exchange, and engagement in health-promoting behaviors [77]. In addition, online activities could promote well-being by reducing loneliness and facilitating social engagement [78], which might contribute to fewer SHS symptoms. Therefore, the internet should attach great importance to public health promotion in daily life. However, we should not overlook the role of traditional media in health communication, especially for television and newspapers, which were significant predictors of SHS in our study. Traditional media are still valuable resources for providing health-related information for people of low socio-economic status [79].

#### **5. Limitations and Future Studies**

The current study is one of the few that looked at time changes in the associations of different media use with obesity and SHS. Although substantial outcomes were seen when interpreting the results, the following limitations should be considered. Firstly, the repeated cross-sectional data did not allow us to draw causal inferences about the relationships between media use and obesity or SHS. Thus, a panel study and follow-up data are recommended to assess the causal effect of media use on obesity or SHS in future. Secondly, based on second-hand data, the measures of SHS were on a self-defined scale with limited items, which failed to measure SHS comprehensively and precisely. Although we made efforts to better match the three facets of SHS (i.e., physical SHS, mental SHS, and social SHS), the measures were single item and participants may have responded to the question more subjectively in terms of self-rated physical or psychological health, which might underestimate or overestimate the accurate SHS level. Thus, our findings for SHS need to be validated in future studies and a more comprehensive measure of SHS, such as the Suboptimal Health Status Questionnaire-25 (SHSQ-25) [80], should be employed. Thirdly, as the outcomes indicated, most of the respondents were middle-aged or older adults, whose media use status differs from that of young adults or adolescents, or conversely. It was unclear whether the findings could be expanded to adolescents or young adults. Therefore, future studies on time trends of media use and health status should cover more age groups and compare adolescents and older adults.

#### **6. Conclusions**

Based on three nationally representative samples of CGSS from 2013 to 2017 we tested the time trend of media use and the associations with obesity and SHS. We considered the differences between traditional media use and new media use. The results confirmed significant time trends between media use and obesity as well as SHS from 2013 to 2017. Furthermore, our study highlighted that the trend differences between traditional media and new media use were significant. Namely, traditional media use showed decreasing trends, while new media displayed an increasing trend. These findings contribute to a better understanding of how changing media use is impacting adults' health status.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/article/10 .3390/ijerph182413214/s1, Table S1. The difference of media use, obesity and SHS among 2013, 2015, and 2017(ANOVA).

**Author Contributions:** X.L. and Q.L. conceptualized and designed the study together. Q.L. formally analyzed and explained the data, and prepared the original draft. X.L. reviewed and revised the manuscript, and provided funding as well as project administration. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was funded by Chinese National Funding of Social Sciences (grant number 18AXW005).

**Institutional Review Board Statement:** Ethical review and approval were waived for this study, due to the data in this article was second-handed, which didn't join in the original national survey.

**Informed Consent Statement:** "Not applicable" for studies not involving humans.

**Data Availability Statement:** Publicly available datasets were analyzed in this study. This data can be found here: http://cnsda.ruc.edu.cn/index.php?r=site/article&id=180.

**Acknowledgments:** We thank CGSS (Chinese General Social Survey) research team for providing access to the survey and every respondent in the study for their contributions. Also, we are grateful to the journal editors and the anonymous reviewers for their helpful comments and suggestions.

**Conflicts of Interest:** All authors declare that there is no conflict of interest.

#### **References**


### *Review* **Chronic Obstructive Pulmonary Disease Patients' Acceptance in E-Health Clinical Trials**

**Saeed M. Alghamdi 1,2,\*, Ahmed M. Al Rajah 3, Yousef S. Aldabayan 3, Abdulelah M. Aldhahir 4, Jaber S. Alqahtani 5,6 and Abdulaziz A. Alzahrani 1,7**


**Citation:** Alghamdi, S.M.; Rajah, A.M.A.; Aldabayan, Y.S.; Aldhahir, A.M.; Alqahtani, J.S.; Alzahrani, A.A. Chronic Obstructive Pulmonary Disease Patients' Acceptance in E-Health Clinical Trials. *Int. J. Environ. Res. Public Health* **2021**, *18*, 5230. https://doi.org/10.3390/ ijerph18105230

Academic Editors: Irene Torres-Sanchez, Marie Carmen Valenza and Paul B. Tchounwou

Received: 29 March 2021 Accepted: 10 May 2021 Published: 14 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

**Abstract:** Introduction: Telehealth (TH) interventions with Chronic Obstructive Pulmonary Disease (COPD) management were introduced in the literature more than 20 years ago with different labeling, but there was no summary for the overall acceptance and dropout rates as well as associated variables. Objective: This review aims to summarize the acceptance and dropout rates used in TH interventions and identify to what extent clinical settings, sociodemographic factors, and intervention factors might impact the overall acceptance and completion rates of TH interventions. Methods: We conducted a systematic search up to April 2021 on CINAHL, PubMed, MEDLINE (Ovid), Cochrane, Web of Sciences, and Embase to retrieve randomized and non-randomized control trials that provide TH interventions alone or accompanied with other interventions to individuals with COPD. Results: Twenty-seven studies met the inclusion criteria. Overall, the unweighted average of acceptance and dropout rates for all included studies were 80% and 19%, respectively. A meta-analysis on the pooled difference between the acceptance rates and dropout rates (weighted by the sample size) revealed a significant difference in acceptance and dropout rates among all TH interventions 51% (95% CI 49% to 52; *p* < 0.001) and 63% (95% CI 60% to 67; *p* < 0.001), respectively. Analysis revealed that acceptance and dropout rates can be impacted by trial-related, sociodemographic, and intervention-related variables. The most common reasons for dropouts were technical difficulties (33%), followed by complicated system (31%). Conclusions: Current TH COPD interventions have a pooled acceptance rate of 51%, but this is accompanied by a high dropout rate of 63%. Acceptance and dropout levels in TH clinical trials can be affected by sociodemographic and intervention-related factors. This knowledge enlightens designs for well-accepted future TH clinical trials. PROSPERO registration number CRD4201707854.

**Keywords:** systematic review; meta-analysis; telehealth; chronic obstructive pulmonary disease; COPD

#### **1. Introduction**

More than 10% of the population worldwide aged 40 years or older are affected by Chronic Obstructive Pulmonary Disease (COPD) [1]. In general, COPD is caused by smoking cigarettes, which may lead to death or disability [2]. The prevalence of COPD has increased dramatically over the past 30 years and is predicted to be the thirdleading cause of death by 2030 [1–3]. According to the Global Initiative for Chronic Obstructive Lung Disease (GOLD), COPD is a common, preventable, and treatable disease

characterized by persistent airflow limitation that is usually progressive and associated with enhanced chronic inflammatory responses to noxious gases in the airways and the lungs [4]. Airflow limitations lead to worsening respiratory symptoms—exacerbation and, often, hospitalization [5,6]. COPD patients require an appropriate management strategy that aims to minimize the frequency of hospitalizations [7].

Recently, research has focused on delivering COPD care via telehealth (TH) to offer prompt access to healthcare and to increase the capacity of COPD care [8]. Published systematic reviews have found that delivering COPD care using TH may provide a mechanism to encourage collaboration between patients and healthcare providers and may enhance patient knowledge and skills in learning how to deal with their conditions [8,9]. Moreover, it facilitates regular monitoring of patients' clinical data, such as vital signs, to allow the healthcare team to detect any disease deterioration at an early stage before it worsens and provide the necessary care to minimize hospital admissions due to respiratory exacerbation [8,9]. Additionally, TH plays an important role in supporting public health precautions and in mitigating the spread of infections such as COVID-19 [10]. Taking the above into account, TH technologies have different applications to provide health care for COPD patients [8,11].

Using TH in COPD management has been found feasible, valuable, and accessible, but recent evidence shows variation regarding the completion of such interventions [11–13]. Previous clinical trials show that individuals with COPD have positive attitudes toward participating in TH interventions [14–19]. However, evidence about the impact of TH on health service outcomes or patient-related outcomes is still inconclusive [8,11]. This lack of knowledge about the effectiveness of TH for COPD management might lead to poor acceptance of TH interventions and/or a high dropout rate and withdrawal of participants from TH studies [20]. While TH use is promising in COPD management, it is unclear which factors are most associated with acceptance and dropout rates and whether these factors are trial-related, sociodemographic, or intervention-related. Therefore, our review aims to (1) assess the overall acceptance and dropout rates in TH clinical trials, (2) summarize the reasons for dropouts from TH interventions, and (3) explore factors that have an impact on overall acceptance and dropout rates.

#### **2. Methods**

The current systematic review was reported according to the Preferred Reporting Items for Systematic Review and Meta-analysis (PRISMA) guidelines [21].

#### *2.1. The Inclusion Criteria*


#### *2.2. The Exclusion Criteria*


#### *2.3. Search Strategy*

An electronic search of the following databases up to April 2021 was undertaken to retrieve relevant articles: CINAHL, PubMed, MEDLINE (Ovid), Cochrane Library, and Embase. Medical subject headings, subject headings, and/or their combinations used in all databases were as follows: telehealth, telecare, telehomecare, telemonitoring, telerehabilitation, telemedicine, home monitoring, digital monitoring, web-based interventions, internet-based monitoring, e-health, chronic obstructive pulmonary disease, chronic obstructive lung disease, and COPD. The search was conducted in collaboration with the health sciences librarian to ensure that our search included the appropriate and necessary keywords in the review. Keywords and subject terms were customized in each database. A full search strategy from all databases is provided in Supplementary materials Files S1–S10.

#### *2.4. Search Procedures*

The search was performed by the main reviewer (SA), after which all articles were imported to EndNote version X9.3 (Clarivate, Philadelphia, PA, USA), and duplicates were removed. All titles and abstracts were screened by two reviewers (SA and AA). A third reviewer (YA) was available to resolve any persisting disagreements. A manual search of the reference lists of relevant studies was undertaken to identify any potentially relevant articles that were missed by the database search but that might be suitable for inclusion in the review. A full-text review of all suitable articles was undertaken, and any study that did not meet the inclusion criteria was excluded.

#### *2.5. Data Extraction*

A standardized Excel (Microsoft Corporation, Redmond, WA, USA) spreadsheet was created for data extraction. The spreadsheet included information on acceptance and dropout rates, as well as trial-related, sociodemographic, and intervention-related factors. Trial-related factors include study place, study design, and recruitment location. The sociodemographic factors include age, status at recruitment, and smoking history. The intervention-related factors include the components of the intervention, methods of delivery, display, frequency, and duration of the intervention. For any missing data, the authors attempted to contact the corresponding publishers of the included studies and completed the data extraction form.

The quality of the studies was defined based on the Cochrane risk-of-bias tool [22]. Data extraction and quality assessment were performed by two independent reviewers (SA and AA). Any disagreement between reviewers (SA and AA) was resolved by consensus.

#### *2.6. Data Analysis*

The synthesized results in this review focused on the key outcomes of interest, including the acceptance and dropout rates, reasons for dropout, and the possible factors that might impact acceptance and dropout rates. The overall acceptance rate in this paper refers to the number of participants who consented to participate divided by the number of participants who were approached to participate in the trial [23,24], and dropout rate refers to the total number of participants in each treatment arm who dropped out from the clinical trial divided by the total number of the participants who consented to participate in the clinical trial [23].

Possible explanatory trial-related factors, sociodemographic factors, and interventionrelated factors that might be associated with acceptance and dropout rates were identified from the literature [25–30]. All factors were converted to binary data. Trial-related factors are categorized as the place of the study (Europe vs. non-Europe), design of the study (Randomized Control Trial vs. non-Randomized Control Trial), and the recruitment location (one recruitment location vs. more than one). The sociodemographic factors are categorized as age (<69 vs. ≥69), the status of COPD at recruitment (stable vs. non-stable), and smoking history (yes vs. no). The intervention-related factors include components of the intervention (one component vs. more than one), methods of delivery (internet-based vs. other), interactive display (interactive vs. not interactive), frequency (daily vs. weekly), and the duration of the intervention (≤20 weeks vs. >20 weeks). Overall, acceptance and dropout rates were computed for each category to compare factors. Continuous variables are expressed as mean and standard deviations. Categorical variables are expressed as frequency and percentages.

We performed an additional meta-analysis to estimate the pooled difference of acceptance and dropout rates between treatment arms. The estimation of rates weighted by the sample size in each clinical trial, and data were pooled using random-effects models. We expressed rates as proportions and 95% confidence intervals (CIs). Heterogeneity among included studies was assessed using the I-square (I2) value. All statistical analyses in this study were performed using STATA software (StataCorp, College Station, TX, USA). More information about the statistical methods can be found in Supplementary File S1.

#### **3. Results**

#### *3.1. Study Selection*

The search identified a total of 1463 relevant articles, 885 after duplicates had been excluded, with a total of 112 articles maintained for full-text review following the title and abstract screening. The remaining articles were read in full, and 27 articles were considered for the review, as shown in the PRISMA flow diagram (Figure 1).

**Figure 1.** PRISMA flow diagram showing studies related to the TH (telehealth) interventions in COPD (with Chronic Obstructive Pulmonary Disease).

#### *3.2. Study Characteristics*

The majority of the included studies were published in Europe (*n* = 18). Most of the studies were RCT (*n* = 24). Twenty-one out of twenty-seven clinical trials recruited patients from different clinical settings.

#### *3.3. Patient Characteristics*

The included trials [18,31–56] collectively comprised a total of 4157 COPD patients. Participation age (mean ± standard deviation) was 65 ± 7.1 years. In the included studies, 18 studies out of 27 (65%) reported smoking history. COPD GOLD III was the most common GOLD grade among included participants: 3118/4157 (75%). More descriptive details about the studies and sociodemographic characteristics can be found in Supplementary File S3.

#### *3.4. TH Intervention Characteristics*

A variety of TH interventions were provided in the included studies, either selfmanagement programs via telemonitoring or self-management programs combined with other interventions (e.g., exercise, pulmonary rehabilitation, home care). Twenty of the studies out of twenty-seven had more than one component. Fourteen of the studies out of twenty-seven delivered interventions using web-based video or telephone calls. Eighteen of the studies out of twenty-seven delivered the intervention with daily frequencies, while nine studies out of twenty-seven delivered the intervention in weekly frequencies. The duration of the intervention ranged from three weeks to forty-eight weeks. Sixteen of the studies out of twenty-seven had an intervention of >20 weeks. The control groups, for example, were provided with the usual care, written self-management, and written exercise guidelines. More descriptive details about TH and control intervention characteristics can be found in Supplementary Files S4 and S5.

#### *3.5. Acceptance and Dropout Rates*

The total number of individuals with COPD approached to participate in the included clinical trial was 8085. Of these, 3928 patients were ineligible and excluded. A total of 4157 consented and enrolled in the clinical trials. Overall, the unweighted average of acceptance and dropout rates for all included studies were 80% and 19%, respectively. The acceptance and dropout rates in all included studies were stratified by factors, as provided in Table 1.


**Table 1.** Overall acceptance and dropout rates across included studies stratified by trial-related, sociodemographic, and intervention-related factors (*n* = 27) a.


**Table 1.** *Cont.*

<sup>a</sup> Data presented as frequency and percentages or means and standard deviations.

#### *3.6. Meta-Analysis*

An additional meta-analysis of acceptance and dropout rates in all included studies (weighted by the sample size) demonstrated significant differences in the acceptance and dropout rates between TH and control groups. As shown in Table 2, the pooled difference of the acceptance and dropout rates of TH interventions with corresponding 95% CIs was 51% (49% to 52%); *p* < 0.001 and 63% (60% to 67%); *p* < 0.001, respectively. More information about the pooled acceptance and dropout rates in each treatment arm can be found in Supplementary Files S6–S9.



<sup>a</sup> S.E: standard error.

#### *3.7. Reasons for Dropout*

After randomization or allocation to treatment, 1152 participants completed their TH interventions, and 946 withdrew before the end of the study. Of these, only 513 participants provided the reasons for study withdrawal. Reasons for dropout classified as TH-related reasons and individual-related reasons are shown in Table 3.


**Table 3.** Most common reasons for dropout from TH interventions (*n* = 513) a.

<sup>a</sup> Data presented as frequency and percentage.

#### *3.8. Quality Assessment*

Quality assessment was performed using the Cochrane risk-of-bias assessment tool [22]. The studies included showed variation in their risk of bias, but most were limited by a lack of blinding. A summary of our judgments on the potential risk of bias can be found in Supplementary File S10.

#### **4. Discussion**

In the context of COPD, exploring the current acceptance and dropout rates of a different form of TH intervention and its associated factors is a crucial objective in terms of whether TH intervention is feasible or not; furthermore, these data have not been previously pooled in forest plots to summarize the overall acceptance and dropout rates. Our findings revealed that the overall average of acceptance and dropout rates was reasonable for patients with COPD, considering the trial-related, sociodemographic, and interventionrelated factors. The findings indicated that 20% of the eligible participants were expected to refuse the TH interventions, and once participants were allocated to TH interventions, around 19% were expected to drop out and not complete the clinical trial. The weighted acceptance rate was 51% in TH groups, suggesting that TH interventions were accepted among people with COPD. Moreover, the dropout rate was higher in the TH intervention groups versus control groups (63% vs. 37%). This finding was expected because most of the dropout reasons from TH studies were TH intervention-related reasons. However, we identified certain factors that have the potential to improve acceptance and reduce the dropout rate of TH interventions with COPD.

In the context of clinical research on COPD, the acceptance rate was higher in multicenter short-term (<20 weeks) trials that were done in European countries. A possible explanation is that advanced countries have good adoption and success rates of TH implementations compared to low- and middle-class countries [57–59], but it could also be due to other financial, organizational, and economic factors since large multicenter clinical trials need considerable research funding, collaboration between clinical settings, qualified researchers, and a large number of patients [57,60].

Moreover, a high acceptance rate in the studies that remotely monitored people with COPD after hospital discharge has been observed. This is due to the benefits of TH in detecting COPD deterioration in early stages and minimizing hospital readmission due to exacerbations. Moreover, some evidence found that offering TH interventions to people with COPD seems to be an opportunity to provide quality healthcare and to minimize unnecessary hospitalization [61,62]. A patient's smoking history was identified as an additional explanatory factor to increase TH acceptability. Similar findings were found by Watson et al. that recruiting smokers in TH interventions for less than 20 weeks significantly increased the acceptance rate [63]. This result could be because these patients have been chronic smokers for years, and smoking cessation interventions are not working well for them. The increased participation of smokers with COPD in TH interventions could indicate that those patients are searching for more accessible smoking cessation interventions [64].

Furthermore, the high acceptance rate of TH that provides several components (e.g., telemonitoring, tele-coaching, and self-management education) has been inspected. A possible explanation is that these TH interventions provide more individual-based interventions that meet the patient's needs. There were similar findings in the literature about using more approaches to address the patient's needs (e.g., uptake pulmonary rehabilitation), resulting in greater patient engagement and satisfaction [65]. Moreover, TH interventions that provide daily monitoring through convenient methods (e.g., telephone calls and text messages) had a high acceptance rate because they used a simple method of communication [32].

Refusal to complete TH interventions is primarily attributed to the interventions themselves [66,67]. It was noted that TH interventions with multiple components were fraught with complexities and technical difficulties that have resulted in decreased treatment sessions or even termination [18,32]. This could lead to participant dissatisfaction and, ultimately, dropping out of the study. Essential steps at the designing and implementation levels are needed to plan and enable more technical support during the delivery of TH interventions [68]. Another possible explanation for dropout rates could be the discrepancy between the patients' expectations and their abilities to operate and use TH [69]. Most of the studies did not provide training sessions for the participants; specifically, there were no measurements of participants' competency in using the technology or cognitive abilities to use TH. Thus, attention must be paid to accommodating these factors in the design of future TH interventions [70].

Telehealth application is important, and COPD management guidelines have supported its implementation [71–73]. Current systematic reviews provide a comprehensive description and summary of the acceptance and dropout rates of different forms of TH interventions for patients with COPD. Generally, TH interventions are acceptable among the COPD population [11,20,23,24,67,68,74,75]. Nevertheless, methodological queries remain regarding the design of a more acceptable and feasible TH intervention, the best strategy to provide TH interventions (individual vs. community), components of TH interventions (solo vs. joint other treatments), technological aspects (classic vs. advanced), and COPD phenotypes (stable vs. non-stable) that will obtain more benefits from TH interventions. This suggests that we need a more in-depth understanding of the acceptability and feasibility of TH interventions developed for COPD patients, as well as an in-depth understanding of the sociodemographic characteristics of the COPD population and their cognitive abilities when we propose TH solutions [75]. Here, the high dropout rates in TH interventions happened due to both intervention-related reasons and patient-related reasons. An additional contribution of this review is to inform future clinical trials regarding the acceptance and dropout rates of existing TH trials in the context of COPD. Moreover, this review will help researchers to identify the reasons that prevent individuals with COPD from completing TH interventions, and it provides evidence to guide future researchers in designing prospective TH clinical trials accordingly.

#### **5. Limitations**

Some limitations must be considered when attempting to interpret the results of the current review. There was a lack of information and incomplete data across all included studies, which impeded the possibility of exploring more variables that might influence or be influenced by the acceptance and dropout rate. Moreover, there was considerable variation in the quality of the included studies, which could limit the ability to generalize the results of this review.

#### **6. Conclusions**

Current TH interventions with COPD have a pooled acceptance rate of 51%, but this is accompanied by a high dropout rate of 63%. Acceptance and dropout levels in TH clinical trials may be affected by sociodemographic and intervention-related factors. Taken together, continued efforts are needed to improve patients' acceptance by understanding and mitigating the issues contributing to the acceptance and dropout rates that would support the preparation and establishment of effective and well-accepted TH interventions.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/article/10 .3390/ijerph18105230/s1, File S1: Search Strategies of Databases, File S2: More Information about Statistical Methods for Acceptance and Dropout Rates, File S3: Study and Sociodemographic-Related Characteristics among Included Studies, File S4: A Detailed Description of TH Interventions and Parallel Groups, File S5: TH Intervention Characteristics of Included Studies, File S6: Forest Plot of Acceptance Rates in TH Interventions, File S7: Forest Plot of Dropout Rates in TH Interventions, File S8: Forest Plot of Acceptance Rates in Controls, File S9: Forest Plot of Dropout Rates in Controls, File S10: Assessment Risk of Bias for Included Studies.

**Author Contributions:** S.M.A., A.M.A.R., Y.S.A., A.M.A., J.S.A. and A.A.A. developed the idea and designed the study protocol. S.M.A., J.S.A. and A.M.A. designed and wrote the search strategy and the first protocol draft. S.M.A., J.S.A. and A.M.A. planned data extraction and statistical analysis. A.M.A.R., Y.S.A. and A.A.A. provided critical insights. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors of this paper declare no conflict of interest.

#### **Abbreviations**


#### **References**


### *Review* **Nurse-Coordinated Blood Pressure Telemonitoring for Urban Hypertensive Patients: A Systematic Review and Meta-Analysis**

**Woo-Seok Choi 1,2,\*, Nam-Suk Kim 3, Ah-Young Kim <sup>1</sup> and Hyung-Soo Woo <sup>1</sup>**


**Abstract:** Coronavirus disease 2019 (COVID-19) has put hypertensive patients in densely populated cities at increased risk. Nurse-coordinated home blood pressure telemonitoring (NC-HBPT) may help address this. We screened studies published in English on three databases, from their inception to 30 November 2020. The effects of NC-HBPT were compared with in-person treatment. Outcomes included changes in blood pressure (BP) following the intervention and rate of BP target achievements before and during COVID-19. Of the 1916 articles identified, 27 comparisons were included in this review. In the intervention group, reductions of 5.731 mmHg (95% confidence interval: 4.120–7.341; *p* < 0.001) in systolic blood pressure (SBP) and 2.342 mmHg (1.482–3.202; *p* < 0.001) in diastolic blood pressure (DBP) were identified. The rate of target BP achievement was significant in the intervention group (risk ratio, RR = 1.261, 1.154–1.378; *p* < 0.001). The effects of intervention over time showed an SBP reduction of 3.000 mmHg (−5.999–11.999) before 2000 and 8.755 mmHg (5.177–12.334) in 2020. DBP reduced by 2.000 mmHg (−2.724–6.724) before 2000 and by 3.529 mmHg (1.221–5.838) in 2020. Analysis of the target BP ratio before 2010 (RR = 1.101, 1.013–1.198) and in 2020 (RR = 1.906, 1.462–2.487) suggested improved BP control during the pandemic. NC-HBPT more significantly improves office blood pressure than UC among urban hypertensive patients.

**Keywords:** blood pressure; hypertension; COVID-19; nurse; coordination; telemonitoring; urban

#### **1. Introduction**

Essential hypertension is the primary modifiable risk factor for cardiovascular disease, which is a leading cause of death according to the World Health Organization (WHO) [1]. Approximately 45% of American adults had elevated blood pressure in the 2017–2018 time period [2,3], but only 34% were being managed to bring the blood pressure down to the recommended treatment level [4]. It has been reported that managing blood pressure to remain within the recommended level reduced the incidence of stroke by 35–40%, heart attack by 20–25%, and heart failure by more than 50% [5]. Moreover, the American Heart Association/American College of Cardiology in 2017 and the European Society of Hypertension/European Society of Cardiology in 2018 issued guidelines recommending that the target blood pressure in hypertensive patients should be regulated more strictly than recommended by previous guidelines [6,7]. Additionally, recent studies have reported that essential hypertension is one of the most common comorbidities that cause lung damage and mortality due to the coronavirus disease 2019 (COVID-19) [8,9]. Thus, strict blood pressure management is crucial to reduce the incidence of cardiovascular disease (CVD) in patients with risk factors.

Despite preventive measures such as hand washing, self-isolation, mask wearing, and social distancing, COVID-19, which first emerged in Wuhan, China, spread rapidly in high-income countries such as the United States, Spain, Germany, Italy, and Korea in the early days of the pandemic [10]. Although the number of newly confirmed patients

**Citation:** Choi, W.-S.; Kim, N.-S.; Kim, A.-Y.; Woo, H.-S. Nurse-Coordinated Blood Pressure Telemonitoring for Urban Hypertensive Patients: A Systematic Review and Meta-Analysis. *Int. J. Environ. Res. Public Health* **2021**, *18*, 6892. https://doi.org/10.3390/ ijerph18136892

Academic Editors: Irene Torres-Sánchez and Marie Carmen Valenza

Received: 29 May 2021 Accepted: 24 June 2021 Published: 27 June 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

has been decreasing with the administration of the vaccine since December 2020 [11], the COVID-19 pandemic is still ongoing. According to the WHO, as of April 2021, there were approximately 1.6 billion cases of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2)—the virus that causes COVID-19—infections worldwide, including 3 million deaths [12]. However, with the exception of some countries, the global vaccination rate is still too low (below 40%) [12]. In particular, in most countries in Africa, the Middle East, and Southeast Asia, as of the end of April 2021, only approximately 10% of the population had received at least one dose of a vaccine [12]. The drastic increase in demand for medical care during the pandemic has exposed the limitations of traditional medical systems and is adversely affecting the existing medical systems for non-communicable diseases (NCDs) such as essential hypertension [13].

During infectious disease pandemics, a high frequency of direct contact between individuals in densely populated cities can increase the infection risk even in well-established in-person medical infrastructures [14]. Thus, the need for treatment through non-face-toface interaction between doctors and patients is being emphasized more than ever before in cities [15]. Home blood pressure telemonitoring (HBPT) is widely used as an alternative measure in the management of hypertensive patients and is known to be effective in overcoming clinical inertia that may occur in face-to-face care settings [3,16–18]. In addition, remote monitoring of home blood pressure is helpful in managing patients by finding white-coat effects and masked hypertension that may be overlooked by doctors in medical institutions [19,20]. Studies on the effectiveness of nurse-coordinated HBPT (NC-HBPT) to prevent CVD attacks in hypertensive patients existed prior to the COVID-19 pandemic [21–25]. However, evidence on whether or not HBPT statistically enhances blood pressure control by improving communication between patients and healthcare providers has been mixed. Additionally, there is a paucity of literature indicating a solid basis for the effectiveness of nurse-led remote monitoring in the avoidance of CVD.

A coordinator plays a vital role in remote monitoring in chronic diseases [21,26], and nurses are core coordinators in the telemonitoring teams of medical institutions, working either independently or as members of a team [27]. Nurses can provide more regular follow-up, high-quality care, favorable health outcomes, and higher patient satisfaction, all equivalent to that achieved by physicians [28]. A 2005 review recommended nurse-led monitoring as an additional measure to support face-to-face therapy [29]. The literature classifies ancillary interventions to help control blood pressure into one of six categories: care led by health professionals, patient monitoring, education of medical professionals, education of patients, organizational interventions, and appointment reminder systems. Particularly, for the step-by-step care of hypertension that is not controlled by drug treatment, nurse-led regular monitoring was emphasized. However, there is uncertainty about the effectiveness of nurse-led remote monitoring in cities with established medical infrastructure and abundant medical resources. Although many studies have reported the nurses' role in HBPT intervention [3,20,28–31], no study has systematically reviewed the effectiveness of NC-HBPT in urban areas and, particularly, none have presented quantitative results according to the temporal progress. Therefore, comprehensive comparative analysis of outcome data of NC-HBPT performed in urban areas before and during the COVID-19 pandemic is important to overcome the challenges of NCD and communicable disease (CD) management and to determine the future directions for remote monitoring policies.

This study hypothesizes that NC-HBPT in urban adult hypertensive patients is not more effective than usual care (UC) in preventing CVD and that its effect over time would be identical to that of UC. To derive robust results, we conducted a systematic review and meta-analysis of randomized controlled trials (RCTs), which are placed at the top in the hierarchy of evidence-based research. Previous meta-analyses investigated the effect of HBPT in hypertensive patients [32,33], but none have integrated the results of NC-HBPT for efficient implementation in cities yet. Thus, this study aimed to examine and compare the following: mean changes in systolic blood pressure (SBP) and diastolic blood pressure (DBP); rates of achieving the target blood pressure after NC-HBPT; and effects of NC-HBPT over time in an urban area.

#### **2. Materials and Methods**

#### *2.1. Literature Search and Identification of Eligible Studies*

We conducted a review in accordance with the guidelines summarized in the Cochrane Handbook for Systematic Reviews of Interventions [34]. A protocol for this study has been published in the PROSPERO [CRD42020222789], which is an international prospective register of systematic reviews operated by the Center for Reviews and Dissemination at the University of York [35]. Studies evaluating HBPT that were published between the date of inception of the utilized databases and 30 November 2020, were identified. The electronic databases we used included PubMed, EBSCOhost, and the Cochrane library (CENTRAL), and the search was limited to studies published in peer-reviewed journals in English. The related search keywords included "urban", "hypertension", "remote monitoring", "telemonitoring", "telemedicine", and "randomized controlled trials." The adopted search formula was constructed by combining free terms of relevant keywords and Medical Subject Headings (MeSH) terms via truncation, Boolean operators, and phrasing. We first searched in PubMed using this structured formula and sequentially performed additional searches according to the syntax of each database (Appendix A).

All retrieved data were exported to EndNote X8.2 (Thomson Reuters, Philadelphia, PA, USA). Titles and abstracts of each study were screened, and the main text was reviewed as required. We searched for all meta-analyses conducted previously on the topic and reviewed all primary studies and relevant references in those meta-analyses. To find grey literature, we referred to related websites in the United States and Europe (e.g., OpenGrey: http://opengrey.eu/ (accessed on 30 November 2020); Grey Literature Report: https:// www.greylit.org/ (accessed on 10 December 2020). To ensure objectivity and transparency of the eligibility assessment, two of the authors (WSC and AYK) independently conducted the literature search and arrived at a mutually-agreed selection of studies.

#### *2.2. Inclusion and Exclusion Criteria*

An intervention group was defined as one in which patients measured their blood pressure on their own at home, reported the measurements to their doctors, and regularly visited the medical institute for follow-up, and in which a coordinator—including a nurse with or without other health care professionals—was involved in the process. As for nurses, only situations where the registered nurse monitored the patient's home blood pressure using an automatic sphygmomanometer without face-to-face contact were included. All the following actions were also included: consultation with the patient using a telephone line, mobile phone, computer, or letter; education on the disease or intervention process; execution of accompanying interventions such as a text message reminder service; and sending information regarding the patient's health status.

A control group was defined as one in which patients received routine in-person examinations at the doctor's office. Since NC-HBPT is not yet a standardized treatment itself, none of the included studies provided an equivalent of NC-HBPT, and so no active control groups were included in our analyses. Their ethnicity, level of income, and severity of hypertension were not considered separately.

We included studies (1) involving patients with essential hypertension (SBP ≥ 130 mmHg, DBP ≥ 80 mmHg) regardless of hypertension onset or history of pharmacotherapy; (2) including patients who received treatment at an urban medical institution; (3) having an intervention that was provided for ≥2 months; (4) utilizing a RCT design; and (5) involving adults aged ≥17 years. The diagnostic criteria for hypertension in Europe are SBP ≥ 140 or DBP ≥ 90, which is higher than that in the US [7]. Thus, the diagnostic guideline for hypertension published by the 2018 ACC/AHA associations [6], which has a wider diagnostic range, was adopted in searching databases.

However, we excluded studies (1) conducted in several regions with unclear study locations; (2) having a research location that was a mix of urban and rural areas; (3) involving patients with acute CVD or stroke with a drastic change in blood pressure; (4) involving women in the peripartum period; (5) having an intervention provided to patients in nursing homes or care facilities; (6) having an intervention provided as part of another intervention program for a different disease; (7) using a different intervention in the UC group; and (8) utilizing a cluster- or cross-over RCT design. In this study, an "urban area" was defined according to the administrative functions and population size.

#### *2.3. Study Selection*

A further search was conducted by reviewing the full-text manuscripts of studies identified during the first round of screening and their reference lists. All articles related to "coordinator" and "nurse" were additionally identified. Among the blinded RCTs that regularly verified the effects of these two groups, those that reported changes in blood pressure measurements before and after interventions were selected for data synthesis.

Two of the authors (NSK and HW) independently classified and excluded duplicate studies and those that did not meet the inclusion criteria. The main body of potentially valid studies was reviewed, and any disagreement between the two authors was resolved by a senior author (WSC).

#### *2.4. Outcomes*

The primary outcome was the weighted mean difference (WMD) of office SBP and DBP between the baseline and follow-up in the NC-HBPT and UC groups. The secondary outcome was the rate of target blood pressure achievement.

#### *2.5. Data Extraction and Coding*

Two researchers (NSK and AYK) independently extracted data from the selected primary studies. The data were coded in an electronic sheet using Comprehensive Meta-Analysis Software Version 2.2.064 (CMA, Biostat, Englewood, NJ, USA). Patients' age and sex, duration of remote monitoring (months), accompanying interventions, intervention pathway, the coordinator's profession, and outcome data were extracted. When a single article had different follow-up periods for the intervention or different sample sizes, or compared two or more interventions [21,22,26,27,36–40], the results of each intervention were classified as independent comparative studies. For missing or inaccurate data in the primary research materials [27], we referred to journal websites or public trial registries (e.g., the US National Institutes of Health ongoing trials register) or directly contacted the authors. In cases where the standard deviation (SD) values for the mean changes were not presented [38], the values were imputed by calculating the mean of individual studies included in the review, and a sensitivity test was conducted to check for bias [40,41]. For the one study that did not provide baseline data [36], the data from the first assessment and the last follow-up were compared. In case of any disagreement between the researchers, a third researcher (WSC) adjudicated.

#### *2.6. Quality Assessment*

Two researchers (NSK and HW) independently assessed the risk of bias using Review Manager (RevMan, Version 5.4, Copenhagen: The Nordic Cochrane Center) by the Cochrane Collaboration [42], evaluating the RCTs in terms of selection, performance, detection, attrition, reporting biases, and other bias domains. A senior researcher (WCS) resolved any disagreement between the two researchers.

#### *2.7. Quantitative Data Synthesis*

Two researchers (WSC and NSK) analyzed the coded data. A random effects model (REM) was adopted because most of the primary studies were performed in different research institutions or by different researchers.

For computational options for data synthesis, WMD was set as a continuous variable and relative risk (RR) as a dichotomous variable; the changes in the mean office SBP and DBP of the NC-HBPT and UC groups before and after the intervention were extracted first. Hedges' g (g) was additionally calculated to determine the appropriateness of the effect size. If the *g*-value converted from the WMD was at least 0.5, the effect size was deemed appropriate for analysis [43]. RR was used to determine the rate of achieving the target blood pressure and was calculated using the number of samples that reached the target blood pressure during each follow-up period. All results are reported with 95% confidence intervals (CIs), and CMA software was used for statistical analysis.

Overall statistics, including weighted values, were analyzed by combining independent data. A χ<sup>2</sup> test was used to assess differences between studies, and a *p*-value < 0.10 was deemed statistically significant. Clinical heterogeneity was assessed using the Q statistic, tau-square (τ2), and I2 statistic. Q statistic and τ<sup>2</sup> interpreted the numerical values, and I<sup>2</sup> was considered significant at ≥50% [43,44]. To determine the bias in each study, examine its effect on the distribution of summary effect size, and detect outliers, a sensitivity analysis using the "one study removed" method was conducted [45]. In addition, a cumulative meta-analysis was performed to identify the chronological patterns of the effect size of each study [41] (Appendix B).

After determining the effect size of each of the primary studies, their temporal changes were analyzed, and the results before and during the COVID-19 pandemic were compared.

#### **3. Results**

#### *3.1. Study Characteristics*

In total, 1916 potentially relevant articles were initially identified from databases, reference lists of each retrieved paper, and other electronic sources (Figure 1). Six datasets were additionally acquired directly from the authors, and 423 duplicates were excluded. The titles and abstracts of 1499 references were screened, and 1101 references were excluded because they were deemed inappropriate for analysis. The full copies of the remaining 398 potentially eligible articles were scrutinized. Of these, 382 studies that did not meet the aforementioned inclusion criteria—that is, studies with participants less than 17 years of age (*n* = 19), no obtainable data (*n* = 183), no nurse-coordinated intervention (*n* = 86), cluster or cross-over RCTs (*n* = 12), patients with life-threatening CVDs or stroke (*n* = 25), studies not performed in an urban setting (*n* = 43), and studies that were part of a research program for another disease (*n* = 14)—were excluded. A total of 27 individual comparisons met the inclusion criteria and were selected as the final material for data synthesis.

In the meta-analysis, the mean length of NC-HBPT was 7.26 months, and the mean age of participants was 61.35 years in the intervention group and 61.62 years in the control group. Of the 27 comparisons, 20 were nurse alone-led cases [21–23,26,27,39–41,46–49], and seven cases additionally involved experts from other fields [24,25,27,36,38]. The characteristics of these studies are summarized in Table 1.

The medium of administering the intervention was mobile phones for 2 cases [39], mobile-web for 5 cases [38,40,47], web-based for 5 cases [21,25,37], telephone for 14 cases [22–24,26,27,36,46,49], and telephone-linked computers for 1 case [48]. A city was considered metropolitan if its population was at least one million; 11 cases were in metropolitan cities [21–23,36,38,39,47,48], and 16 were in smaller cities.

**Figure 1.** PRISMA flow diagram of the selection of studies included in the systematic review. Note. HBPT, home blood pressure telemonitoring; RCT, randomized controlled trial; CVD, cardiovascular disease; CVA, cerebrovascular accident; SBP, systolic blood pressure; DBP, diastolic blood pressure; BP, blood pressure.


**Table 1.**

Characteristics

 of included individual studies.





*Int. J. Environ. Res. Public Health* **2021**, *18*, 6892

hypertension;

 Non-ISMS,

non-interactive

 short message service; BMI, body mass index; HbA1c, glycated hemoglobin.

#### *3.2. Assessment of Risk of Bias*

The selection and performance processes for most trials were appropriate. In the case of attrition or reporting bias domain, there were studies with an unclear or high risk of bias. A primary study at a low risk of bias in at least four domains was deemed to be of high quality [50], and 23 comparisons were identified as having a high risk of bias in fewer than four domains, suggesting that the overall quality of the materials was relatively high.

Publication bias was assessed using the trim-and-fill method [51]. The point estimates of SBP based on a funnel plot were as follows: WMD = 5.327 mmHg and *g* = 0.723 (0.445–1.002, *p* < 0.001); there were no trimmed studies (Figure 2). For DBP, *g* = 0.362 (0.222–0.503, *p* < 0.001), which represented a meaningful effect size (Figure 3). The funnel plots of both SBP and DBP showed good visual symmetry, and there were no imputed studies for SBP and DBP. Thus, it was concluded that the potential publication biases did not affect the primary outcomes of the materials included in this study.

**Figure 2.** Funnel plot of systolic blood pressure by Hedges' g (plot observed and imputed), random effects model. Notes. Summary effect size (-); Summary effect size of imputed studies (Ⴗ); Individual studies ().

Publication bias was also assessed for the rate of reaching the target office blood pressure as a secondary outcome. The funnel plot showed good visual symmetry (Figure 4). One study was imputed, but the difference in point estimates was not significant (observed RR, 1.261, vs. adjusted RR, 1.240). Thus, the publication bias by the potentially unpublished study did not affect the RR effect size.

**Figure 3.** Funnel plot of diastolic blood pressure by Hedges' g (plot observed and imputed), random effects model. Notes. Summary effect size (-); Summary effect size of imputed studies (Ⴗ); Individual studies ().

**Figure 4.** Funnel plot of rate of target blood pressure by log risk ratio (plot observed and imputed), random effects model. Notes. Summary effect size (-); Imputed study (•); Summary effect size of imputed studies (Ⴗ); Individual studies ().

#### *3.3. Primary Outcomes*

3.3.1. SBP Changes by Nurse-Coordinated HBPT

Data were pooled from the 27 comparisons (16 studies) that included 2860 patients in the NC-HBPT group and 2918 patients in the control group, comprising a total of 5778 patients in both groups [21–27,36–40,46–49]. A significant reduction in blood pressure was observed in the nurse-led intervention group compared with the UC group (5.731 mmHg; 4.120–7.341; *p* < 0.001; Figure 5), and heterogeneity was significant among the studies (I<sup>2</sup> = 71.717%; *p* < 0.001). In the sensitivity test [45], the individual studies did not affect the summary point estimates (WMD: 3.822–7.578 mmHg).


**Figure 5.** Difference in means of office systolic blood pressure changes by nurse-coordinated intervention. Notes. Point estimate of individual study (•); Summary effect size (Ⴗ); SBP, systolic blood pressure; UC, usual care; NC-HBPT, nursecoordinated home blood pressure telemonitoring.

Changes in WMD over time were examined across four time frames. The SBP reduction during time frame I (inception to 2000) was 3.000 mmHg (−5.999 to 11.999; *p* = 0.514) [24]. SBP reductions of 5.150 mmHg (2.383–7.898; *p* < 0.001) and 4.990 mmHg (2.565–7.415; *p* < 0.001) were observed in time frames II (2001–2010) [25–27,47,48] and III (2011–2019) [21,23,33–35,43,46], respectively (Figure 6). A significant SBP reduction of 8.755 mmHg (5.177–12.334; *p* < 0.001) was observed in time frame IV (2020) [22,39,40], the year in which the COVID-19 pandemic began.


**Figure 6.** Difference in means of office systolic blood pressure changes over time. Notes. Point estimate of individual study (•); Subgroup (ż); Summary effect size (Ⴗ); DBP, diastolic blood pressure; UC, usual care; NC-HBPT, nurse-coordinated home blood pressure telemonitoring.

#### 3.3.2. DBP Changes by Nurse-Led Coordination

The WMD analysis of DBP was possible using data pooled from 24 comparisons (14 studies) [21–27,37–40,46–48], and data from 4928 patients (2445 from the NC-HBPT group and 2483 from the UC group) were analyzed. There was a significant decrease in blood pressure in the intervention group compared with the control (2.342 mmHg, 1.482–3.202; *p* < 0.001; Figure 7). Significant heterogeneity was observed between the comparisons (I2 = 51.380%; *p* = 0.002). In the sensitivity test, individual studies did not significantly alter the summary effect size (WMD: 1.343–3.359 mmHg).


**Figure 7.** Difference in means of office diastolic blood pressure changes by nurse-coordinated intervention. Notes. Point estimate of individual study (•); Summary effect size (Ⴗ); DBP, diastolic blood pressure; UC, usual care; NC-HBPT, nursecoordinated home blood pressure telemonitoring.

> The WMD of DBP in the NC-HBPT group was 2.000 mmHg (−2.724 to 6.724; *p* = 0.407) in time frame I [24], 1.947 mmHg (0.524–3.370; *p* = 0.007) in time frame II [25–27,47,48], 2.327 mmHg (0.958–3.695; *p* < 0.001) in time frame III [21,23,37,38,46], and 3.529 mmHg (1.221–5.838; *p* = 0.003) in time frame IV [22,39,40], suggesting that the effect of the intervention was statistically greater closer to the COVID-19 outbreak period (Figure 8).


**Figure 8.** Difference in means of office diastolic blood pressure changes over time. Notes. Summary effect size (Ⴗ); Point estimate of individual study (•); Subgroup (ż); UC, usual care; NC-HBPT, nurse-coordinated home blood pressure telemonitoring.

#### *3.4. Secondary Outcomes*

Rate of Reaching the Target Office Blood Pressure

Using 18 available comparisons (6 studies) [21–23,27,37,38], we calculated the rate of reaching the target office blood pressure in the NC-HBPT group. Data from 4078 patients (2021 from the intervention group and 2057 from the UC group) showed that the rate of reaching the target office blood pressure was significantly higher in the intervention group than in the UC group (RR 1.261, 1.154–1.378; *p* < 0.001). Heterogeneity between the studies was substantial (I<sup>2</sup> = 51.6%). In the sensitivity test, study removal did not have a significant effect on the summary effect size (*p* < 0.001; range: 1.133–1.399).

Changes in the rate of reaching the target blood pressure over time were analyzed by combining the data obtained after the year 2000. RR values of 1.101 (1.013–1.198; *p* = 0.024), 1.400 (1.279–1.534; *p* < 0.001), and 1.906 (1.462–2.487; *p* < 0.001) were found for time frames II [27], III [21,23,37,38], and IV [22], respectively, showing a clear improvement rate.

#### *3.5. Subgroup Analysis*

#### 3.5.1. Size of City

Cities were classified based on their population size, where small- to medium-sized cities had fewer than 1 million residents, and large cities had more than 1 million residents. In small to medium cities (n = 16; participants = 3713) [18,21,24–27,37,40,49], the decrease in SBP by NC-HBPT was 4.134 mmHg (2.275–5.992, *p* < 0.001; I2 = 7.934), while SBP

decreased by 8.355 mmHg (5.937–10.773, *p* < 0.001; I<sup>2</sup> = 82.533) in large cities (n = 11; participants = 2069), compared with the UC group [22,23,36,38,39,47,48] (Appendix C).

#### 3.5.2. Setting

When groups were classified according to the setting in which the study was conducted, the reduction in SBP due to NC-HBPT in primary care clinics (n = 5; participants = 1741) was 3.793 mmHg (0.450–7.136, *p* = 0.026; I2 = 0.000) [27,46,48], 4.353 mmHg (1.877–6.829, *p* = 0.001; I2 = 34.921) in community health centers (n = 9; participants = 1704) [21,26,38,40], and 7.781 mmHg of SBP (1.375–5.483, *p* < 0.001; I<sup>2</sup> = 79.627) in hospitals (n = 13; participants = 2337) compared with the UC group [22–25,36,37,39,47,49].

#### 3.5.3. Duration

For the full duration of NC-HBPT, a total of 5278 people (NC-HBPT group = 2612 vs. UC = 2666) were analyzed. The WMD of SBP was consistently decreased throughout the intervention, by 6.694 mmHg after 3 months (3.644–9.744, *p* < 0.001; I2 = 72.511) [21,22,26,38–40,47], 6.608 mmHg after 6 months (3.777–9.444, *p* < 0.001; I<sup>2</sup> = 85.761) [21,22,26,37–39,41,50], and 3.573 mmHg at 12 months of intervention (0.796–6.349, *p* = 0.012; I<sup>2</sup> = 12.877) compared with the UC group [24,25,27,36,37,46,48].

#### 3.5.4. Coordinator's Profession

A total of 4398 patients were included in the analysis to derive the effectiveness of the nurse-only interventions (n = 20). The decrease in WMD of SBP by nurse alone was 6.132 mmHg (4.262–8.002, *p* < 0.001; I<sup>2</sup> = 78.074) [21,22,26,27,37,39,40,46–49]. Cases where nurses collaborated with other professionals (n = 7) included a total of 1384 people and showed a decrease in SBP of 4.465 mmHg (1.122–7.807, *p* = 0.009; I<sup>2</sup> = 75.373) [24,25,27,36,38]. When classified according to the professions nurses collaborated with, SBP decreased by 2.399 mmHg (−3571–8.369, I2 = 0.000) for nutritionists, 3.000 mmHg (−8.682–14.682, I <sup>2</sup> = 0.000) for lifestyle educators, and 3.000 mmHg (−6.300–12.300, I<sup>2</sup> = 34.534) for community health workers compared with UC. When the collaboration was with a doctor, SBP was reduced by 6.626 mmHg (1.599–11.652, I2 = 67.129) compared with the control group.

#### 3.5.5. Medically Underserved Area

The effect in medically underserved areas reported in each primary study (participants = 5278) was analyzed. In underserved areas (n = 10; participants = 2852), the decrease in SBP was 5.100 mmHg (2.484–7.717, *p* < 0.001; I2 = 49.707) [23,24,26,37,40,48], whereas SBP was reduced by 6.150 mmHg (4.041–8.259, *p* < 0.001) in non-marginalized areas (n = 17; participants = 2930) [21,22,25,27,36,38,39,46,47,49]. In the latter areas, the heterogeneity was substantial (I<sup>2</sup> = 78.192).

#### **4. Discussion**

This study assessed the effectiveness of NC-HBPT in patients with hypertension, a common NCD, in an urban setting. If remote medical services are defined as delivering patients' biological information and managing diseases using information and communications technology (ICT) [52], NC-HBPT can be considered a safe, effective, and timely intervention in the current pandemic situation, which has drastically increased the demand for medical resources, in addition to social measures to prevent the spread of COVID-19 [53].

In this meta-analysis, NC-HBPT achieved an SBP reduction of 5.731 mmHg (4.120–7.341, *p* < 0.001) at an average of 7.26 months. A large-scale meta-analysis using individual patient data showed that an SBP reduction of 4 mmHg can reduce the CVD incidence to 10 events/1000 cases in hypertensive patients with a moderate 5-year CVD risk (11–15%) [54]. A previous meta-analysis examining the effects of HBPT in the same setting without consideration of coordinators reported an SBP reduction of 3.482 mmHg

(2.459–4.505, *p* < 0.001) [55]. Thus, the increased effect of NC-HPBT in preventing CVD can be considered clinically significant.

When examining the effect of NC-HBPT over time, SBP was relatively low at the beginning of the intervention but increased by 3.000 mmHg (−5.999–11.999, *p* = 0.514), and the maximum reduction of 8.755 mmHg (5.177–12.334, *p* < 0.001) was achieved in 2020, when the pandemic began. Although the number of studies included in time frame I is insufficient and lacks significance, the data cannot be completely ignored, as the aforementioned trend undeniably exists based on the comparison of time frame II values (5.140 mmHg, 2.383–7.898, *p* < 0.001) with those in time frames III and IV. Greater SBP reduction was achieved by NC-HBTP over time. Considering the preference for non-faceto-face contact in 2020 and 2021 due to the COVID-19 pandemic and the increased demand for remote medical services, it can be extrapolated from our results that an increase in the potential effectiveness of remote medical services with nurse coordination is possible.

Based on evidence from previous literature suggesting that NC-HBPT is effective [3,20], we examined the effect of a nurse-alone intervention through 20 comparisons [21–23,26, 27,37,39,40,46–49]. HBPT coordinated by a nurse alone achieved an SBP reduction of 6.132 mmHg (4.262–8.002, *p* < 0.001), which was not inferior to the mean reduction of interventions by all coordinators (5.731 mmHg). Moreover, compared with NC-HBPT additionally coordinated by an expert from another field, HBPT was even more effective when a physician was involved in the intervention (n = 3; 6.626 mmHg, 1.599–11.652) [27,38]. HBPT showed limited effectiveness when coordinated by community health workers (3.000 mmHg, −6.300–12.300) [24], nutritionists (2.399 mmHg, −3.571–8.369) [36], or lifestyle educators (3.000 mmHg, −8.682–14.682) [25].

We also found that NC-HBPT was associated with a larger reduction in DBP (2.342 mmHg, 1.482–3.201, *p* < 0.001) than HBPT in the same setting but without consideration of coordinators (1.638 mmHg, 1.084–2.192, *p* < 0.001) [55]. Similar patterns were observed in DBP and SBP changes over time. A higher reduction in DBP of 3.529 mmHg (1.221–5.838, *p* = 0.003) was observed in time frame IV [22,39,40] than in time frame II (1.947 mmHg, 0.524–3.370; *p* = 0.007) [25–27,47,48].

HBPT coordinated by a nurse alone achieved a DBP reduction of 2.389 mmHg (1.393–3.384, *p* < 0.001) [21–24,26,27,37,39,40,47–49], and additional coordination by a doctor achieved a DBP reduction of 2.440 mmHg (−0.166–5.047, *p* = 0.066) [26,38], showing a significantly greater DBP reduction when HBPT is additionally coordinated by experts from other specific fields than in NC-HBPT [24–26,38]. Although the reason for the differences in blood pressure reductions according to the profession of the additional coordinator is unclear, it may have to do with social and organizational factors, the coordinator's level of medical knowledge and experience, and similarity in the education received by the coordinators [56,57]. However, since the number of cases in which NC-HBPT was coordinated by experts from other fields was small and the results were not statistically significant, further research is needed to more accurately determine the validity of the effect of the intervention.

Palmas et al. (2006) reported that a lack of awareness of the benefits of remote medical technologies and the burden of using these technologies have contributed to the low expectations for remote medical services in urban areas [58]. However, the high percentage of hypertensive patients who are highly susceptible to COVID-19 and the environmental factors that are found in densely populated cities contribute to excessive medical demands that cannot be handled by traditional medical systems [59]. Therefore, the importance of remote monitoring technology as a means to efficiently provide medical services with limited resources is being increasingly emphasized. To the best of our knowledge, this is the first meta-analysis to examine the effect of NC-HBPT on urban hypertensive patients over time. We have derived meaningful results regarding the benefits of nurse-coordinated interventions.

In this study, the heterogeneity of the summary effect sizes was found to be substantial. Specifically, the I2 for SBP and DBP were 71.717% (*p* < 0.001) and 51.380% (*p* = 0.002), respectively. Thus, the authors applied a random effects model to the analysis, which did not completely remove the heterogeneity. Therefore, subgroup analysis was performed to assess the causes of the heterogeneity, and several moderators were explored that revealed clinical implications, along with evaluation of heterogeneity.

Despite selecting well-founded blinded RCTs through a transparent and systematic process and deriving solid and integrated results for the primary outcomes without publication biases, our study has some limitations. First, an extensive literature search was conducted on reliable and relevant databases using a structured formula, but the number of studies included was low. By building a more precise search formula, the reliability of our results could be improved. Second, since each time frame did not contain an equal number of comparisons, the analysis results for different periods were not based on the same number of studies. Including as many studies as possible, with an equal number of studies per time frame, can overcome this limitation. Third, while the researchers reasonably set the duration of each time frame to 9 and 10 years according to the technological changes and historical events to explore temporal patterns of outcomes of NC-HBPT, the distinction between each time frame may not have been clear. Thus, it may be necessary to set the time frames based on the turning points of ICT development to increase the precision of the findings. Lastly, for studies that compared interventions with different coordinators or lengths of follow-up, each comparison was counted as an independent primary study, but there were cases where multiple comparisons were included in one study. Although there was substantial heterogeneity between each comparison, and no statistical errors were observed, there may be a lack of optimal scientific robustness due to the methodological limitations of a random effects model. This limitation can also be overcome by updating the results based on a larger number of studies.

#### **5. Conclusions**

This study revealed that NC-HBPT for urban hypertensive patients can deliver clinically and statistically good BP reduction in terms of avoiding CVD outbreak when compared with UC. Our findings may have meaningful implications for policymakers in urban areas that are planning to introduce remote monitoring systems or in areas with inefficient telemedicine systems. However, some included studies in this analysis lack quality. Thus, although the evidence for the benefit of NC-HBPT was found in this review, further research is necessary on the nurses' roles as coordinators. Additionally, future work must consider the effect of multiple variables on NC-HBPT for more efficient implementation of the intervention system in urban areas.

**Author Contributions:** Conceptualization, N.-S.K. and W.-S.C.; methodology, N.-S.K., A.-Y.K., H.-S.W. and W.-S.C.; software, A.-Y.K. and W.-S.C.; validation, N.-S.K., A.-Y.K., H.-S.W. and W.-S.C.; formal analysis, N.-S.K., A.-Y.K. and W.-S.C.; investigation, H.-S.W. and W.-S.C.; writing—original draft preparation, N.-S.K. and W.-S.C.; writing—review and editing, H.-S.W. and W.-S.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A. Searching Formula**

((((((("Hypertension"[Mesh]) OR ((hypertensi\*) OR high blood pressure))) AND (((((("Urban Population"[Mesh]) OR "Urban Health"[Mesh]) OR "Urban Health Services" [Mesh]) OR "Cities"[Mesh])) OR ((((urban\*) OR city) OR cities) OR central cit\*)))) AND ((((("Telemedicine"[Mesh]) OR "Telemetry"[Mesh]) OR "Blood Pressure Monitoring, Ambulatory"[Mesh])) OR (((((((((((telemedicine) OR telemetry) OR telenurs\*) OR telemonitor\*) OR eHealth) OR telehealth) OR remote monitor\*) OR technology) OR telephone) OR

smartphone) OR internet)))) AND (((((((((randomised controlled trial) OR randomized controlled) OR controlled clinical trial)) OR ((((((((randomised[Title/Abstract]) OR randomized[Title/Abstract]) OR placebo[Title/Abstract]) OR drug therapy[Title/Abstract]) OR groups[Title/Abstract]) OR clinical trials as topic[Title/Abstract]) OR randomly[Title/ Abstract]) OR trial[Title/Abstract]))) NOT cluster randomized controlled trials)) NOT cross over study)

#### **Appendix B. Cumulative Meta-Analysis Presenting the Summative Effect at Each Time Point**

**Figure A1.** Cumulative meta-analysis presenting the summative effect at each time point. Note. SBP, systolic blood pressure; UC, usual care; NC-HBPT, nurse-coordinated home blood pressure telemonitoring; Summary effect size (Ⴗ); Point estimate of individual study (•).

### **Appendix C. Subgroup Analysis**


**Table A1.** Subgroup analysis.

Note. WMD, weighted mean difference; SBP, systolic blood pressure; CI, confidence interval; FEM, fixed effect model.

#### **References**


### *Systematic Review* **eHealth Interventions to Treat Substance Use in Pregnancy: A Systematic Review and Meta-Analysis**

**Katherine Silang 1, Hangsel Sanguino 1, Pooja R. Sohal 1, Charlie Rioux 2, Hyoun S. Kim <sup>3</sup> and Lianne M. Tomfohr-Madsen 1,4,5,\***


**Abstract:** Substance use during pregnancy is associated with adverse pregnancy and neonatal outcomes; eHealth interventions offer a potential accessible treatment option. The objective of this systematic review and meta-analysis was to evaluate the effectiveness of eHealth interventions for the treatment of substance use during pregnancy. A comprehensive search of PsycINFO, Medline, CINAHL, Cochrane and Embase databases was conducted from May 2020 to April 2021. The protocol for this study was registered with Prospero (CRD42020205186) through the University of York Centre for Reviews and Dissemination. Two independent reviewers completed screening, data extraction, and quality assessment. RCTs were included if they reported: (a) administration of an eHealth intervention for (b) substance use outcomes, among (c) pregnant individuals. Comprehensive Meta-Analysis Software (CMA) was used to calculate pooled effect sizes (Odds Ratio) to determine the effect of eHealth interventions on substance use outcomes. Six studies were identified with substance use outcomes that included: smoking (*n* = 3), alcohol (*n* = 2), and other (*n* = 1). eHealth interventions were delivered through the internet (*n* = 1), computer (*n* = 3), telephone (*n* = 1), and text (*n* = 1). Results suggested that eHealth interventions significantly reduced substance use in pregnant individuals compared to controls (OR = 1.33, *95%* CI = 1.06 to 1.65, *p* = 0.013). eHealth interventions offer a promising and accessible treatment option to reduce substance use during pregnancy. This work was supported by the generous donors of the Alberta Children's Hospital Foundation, the Canadian Child Health Clinician Scientist Program (CCHCSP), the Canadian Institute of Health Research and the Fonds de Recherche du Québec—Santé.

**Keywords:** pregnancy; substance-related disorders; randomized controlled trials; smoking; alcohol; cannabis; drug use; internet intervention; telemedicine; digital intervention

### **1. Introduction**

Heavy substance use is associated with serious physical and psychological consequences [1]. The risk of developing a substance use disorder is heightened during reproductive years [2] and substance use is prevalent during pregnancy. 11–15% of pregnant individuals reporting use of alcohol, tobacco, cannabis and/or illicit substances [3–5]. The actual prevalence may be higher, as stigma and judgement may cause some pregnant people to be hesitant to report substance use [6].

Heavy substance use in pregnancy has serious short and long-term consequences, including elevated risk of miscarriage [7], low birthweight [8], infant mortality [9], and sudden infant death syndrome [10]. Long term outcomes for children exposed to substances in-utero vary [11]. For example, prenatal cannabis use has been linked to reduced

**Citation:** Silang, K.; Sanguino, H.; Sohal, P.R.; Rioux, C.; Kim, H.S.; Tomfohr-Madsen, L.M. eHealth Interventions to Treat Substance Use in Pregnancy: A Systematic Review and Meta-Analysis. *Int. J. Environ. Res. Public Health* **2021**, *18*, 9952. https://doi.org/10.3390/ ijerph18199952

Academic Editors: Irene Torres-Sanchez and Marie Carmen Valenza

Received: 15 August 2021 Accepted: 17 September 2021 Published: 22 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

attention and executive functioning skills, poor academic achievement, and increased behavioural problems [12]. Prenatal drinking has been linked to multiple long-term effects including cognitive and behavioural issues [13], executive functioning deficits [14], and poor psychosocial outcomes [15].

Given the high prevalence of substance use in pregnancy and its serious associated harms, it is imperative that pregnant individuals receive access to evidence-based supports. Despite results which have shown the effectiveness of psychological interventions in treating substance abuse, the literature has consistently found that in the general population, people often do not seek addiction and mental health services [16]. Potential obstacles to treatment include limited resources, time conflicts, and stigma [17–19]. Pregnant individuals, especially individuals belonging to marginalized ethnic and socioeconomic groups, are also more likely to be experience arrest, prosecution, conviction and/or child removal related to substance use disclosure, contributing to increased hesitancy to seek help [20]. Concerns about separation from family, as well as a lack of childcare are also known treatment barriers for pregnant individuals who use substances [21].

eHealth is an emerging field that is attracting attention for a variety of mental health conditions. eHealth focuses on the delivery of health services and information through web-based programs, remote monitoring, teleconsultation, and mobile device-supported care. eHealth is a potential avenue to address substance use treatment barriers in pregnancy [22], particularly during COVID-19, which has disrupted a number of face-to-face psychotherapy services. Beyond COVID-19, eHealth initiatives have the potential for broad scale health promotion for substance use [23]. eHealth is accessible, which may make it more appealing to those in remote locations. Additionally, it is cost-effective, and can be flexibly incorporated into one's schedule [24]. Given the accessible nature of eHealth interventions, some pregnant individuals may prefer to use eHealth interventions as opposed to traditional face to face treatment. Treatment preference is important to consider because matching patients to their treatment preferences has been shown to result in greater reduction of substance use behaviours [25]. Moreover, patient centered care (PCC) is one of the techniques that has been recommended to improve the quality of substance use disorder treatment—and a key aspect of patient centered care is shared decision making [26].

A number of meta-analyses of eHealth interventions for treatment of substance use disorders in the general population have been conducted, with promising results [27,28]; however, the literature in for eHealth interventions treating substance use in pregnancy has yet to be integrated as a review.

Accordingly, the primary objective of this systematic review and meta-analysis was to evaluate the effectiveness of randomized controlled trials (RCTs) on eHealth interventions delivered during pregnancy with the goal of reducing substance use, where substance use was defined broadly to include any kind of reported alcohol, tobacco, or other drugs. This definition, which includes a variety of substances at varying levels of use was justified by guidelines suggesting that all substance use should be avoided during pregnancy [29]. Substance use was measured by self-reported and objective reports of abstinence.

#### **2. Materials and Methods**

Methods outlined by the Cochrane Collaboration's Handbook [30] and the standards set by Preferred Reporting Items for Systematic review and Meta-Analysis (PRISMA) were used [31,32]. The protocol was registered with Prospero (CRD42020205186) through the University of York Centre for Reviews and Dissemination.

#### *2.1. Search Strategy*

A preliminary search found that the majority of papers investigating substance use in pregnancy were published in psychology and nursing journals. Thus, we searched the five databases that were most likely to capture the literature within these disciplines. Articles published between 1 January 2000and 19 April 2021were identified from Medline®, PsychINFO, EMBASE, CINAHL, and Cochrane CENTRAL. The most recent search took

place on 18 April 2021. English-language restrictions were applied. The search terms included database specific controlled vocabulary, field codes, operators, relevant keywords, and subject headings to identify the participant population (pregnant individuals), the exposure (eHealth interventions), and the outcome (substance use) [33,34]. Key terms used to conduct the search were related to telepsychology, randomized control trials, substance use and pregnancy (see Appendix A). Duplicate articles were removed. The remaining articles were divided and were screened independently by two reviewers from a eight member research team. Pairs of reviewers screened the titles and abstracts to determine eligibility for inclusion in the full-text review, and the first author reviewer (KS) supervised and reviewed ~100 records to ensure >85% consistency. Out of 2560 abstracts, 159 conflicts were identified. The types of conflicts included whether the study was targeting the right population (pregnant people), whether the intervention fit our definition for eHealth, or whether the study included extractable outcomes. These conflicts were resolved by the first author (KS).

#### *2.2. Inclusion and Exclusion Criteria*

Studies were eligible for inclusion if they included: (a) a RCT; (b) an empirical journal article; (c) an eHealth intervention (e.g., video therapy sessions, telephone, SMS, recorded therapy sessions); (d) the goal of the intervention was to reduce substance use; (e) the sample consisted of pregnant individuals; (f) extractable outcomes with respect to substance use; and (g) the intervention took place during pregnancy. If more than one article reported results from the same intervention in the same sample, the more recent study was included in the study. Studies were excluded on the basis that they did not meet inclusion criteria or were irretrievable/unavailable in English.

#### *2.3. Data Extraction*

Two team members independently extracted data into a Microsoft Excel file and conflicts were resolved by consensus with the coders and the first and second authors. Extracted data included authors' names, publication years, country, sample demographics, pregnancy characteristics, substance use parameters, intervention characteristics and administration, and mental health assessments for all groups. Sample characteristics that were extracted when provided included sample size, age, gestational age, ethnicity, race, and gender breakdown. Study characteristics that were extracted when provided included the name of intervention, description, method of administration, degree of interactivity (i.e., completely online or some in-person component), degree of guidance, and participant time spent on the intervention. The outcomes extracted were odds ratios (OR) measuring substance use outcomes post-intervention. Corresponding authors of included articles were contacted if studies had missing or incomplete data.

#### *2.4. Data Analysis*

A random effects meta-analysis was conducted using Comprehensive Meta-Analysis Software (Biostat Inc, Englewood, NJ, USA) [35]. Most studies reported ORs, and these were used to calculate meta-estimates of substance use post-intervention in the intervention groups compared to the control groups. Ref. [36] was the only study to report chi-squares, which were transformed to ORs through the CMA software. Some studies had several posttests (e.g., immediately post-intervention, later follow-up) and outcomes (e.g., smoking, alcohol use, general substance use). To meet the assumption of independence, effect sizes from the same study were aggregated in CMA and the single effect size estimate for each study was used to calculate pooled ORs. A forest plot was also created to display the ORs for each individual study as well as the pooled OR from all the studies. To test for publication bias, the Begg and Mazumdar rank correlation test [37] as well as the Egger's regression test [38] were performed to assess bias by regressing standardized effect size to the studies precision. A significant test indicates publication bias, or significant funnel plot asymmetry [36,38,39]. Meta-regression analyses were originally planned to explore

significant moderators and explore secondary outcomes; however, not enough studies were included to complete these analyses. Sensitivity analyses were also completed to assess the robustness of the synthesized results.

#### *2.5. Quality Assessment*

To assess the quality of the RCT studies, the Cochrane Risk of Bias Tool for randomized trials was used [30]. This tool assesses literature based on seven potential sources of bias within the general categories of selection bias (allocation concealment), performance and detection bias (blinding), attrition bias (incomplete data) and selective reporting bias [39]. Bias was judged individually by a team member and then cross-referenced by the judgment of another team member to complete a 100% check. Total scores range from 0 (unlikely to alter the results), to 7 (greatly weakens confidence in the results). Higher scores indicate lower study quality and a higher risk of biased results. The study informally defined scores from 0–2 as low risk, scores from 3–5 as moderate risk, and scores between 6–7 as high risk.

#### *2.6. Primary Outcome*

The current review aimed to determine whether eHealth interventions delivered during pregnancy reduced substance use when compared to a control group. Substance use was measured using self-reports of frequency and quantity of substances taken, as well as self-report measures of abstinence and objective measures of abstinence. Objective forms of abstinence were defined as a biochemical measure of substance use. For example, in certain studies where smoking was the outcome, carbon monoxide (CO) readings and/or saliva samples were tested for a certain amount of cotinine.

#### **3. Results**

#### *3.1. Study Selection*

This search was originally conducted with an associated study [40], which reviewed eHealth interventions in pregnancy for treatment of depression, anxiety, and insomnia. A wider search was conducted to include substance use for the purposes of this paper. The search identified 5505 relevant articles, with 2945 duplicates removed. In total, 2367 of the articles were excluded after title and abstract review and 193 articles were reviewed at the full-text level. A total of 6 articles met inclusion criteria for this review. See Figure 1 for the PRISMA diagram [32].

#### *3.2. Characteristics of Included Studies*

Table 1 provides characteristics of the included studies. Participant baseline age ranged from 18–37 years old. Gestational age ranged from 4–23 weeks. Of the four studies which reported ethnicity, three studies had a total sample where >85% of participants were of European descent [36,41,42].

With respect to the type of eHealth interventions, most of the interventions were created in a way that communication of services took place through the use of technology (i.e., telephone/text), rather than the use of a specific app to reduce substance use behaviours. Four of the eHealth interventions were delivered via computer or the internet [42–45], one was delivered through text message (SMS) [41] and one was delivered via telephone [36]. The types of interventions that were delivered included: motivational interviewing in one study [43], the use of general health advice (presented educational information regarding substance use without a psychological component) in three studies [42,45,46], and psychoeducation in two studies [36,41]. Three studies assigned control participants to receive treatment as usual from their healthcare providers [41,43,46], one study provided control group participants access to a website with standard advice, [42] and two studies used a time-matched placebo condition [43,44].

Interventions varied with respect to whether the eHealth intervention was guided or unguided, which was defined by whether a therapist/healthcare professional facilitated treatment. Most of the included studies were guided (*n* = 4) [36,42–44]. Two studies were

unguided [41,45]. Length of follow-up varied across studies with follow-up occurring at 4 weeks after baseline [41], 8 weeks after baseline [42], 12 weeks after baseline [43], 16 weeks after baseline [44], up to 22 weeks after baseline [36].

**Figure 1.** PRISMA diagram detailing the database searches, the number of abstracts screened, and the full texts retrieved.




With respect to the type of substances studied, three of the studies assessed smoking using time sensitive self-reported abstinence, as well as dose and dependence tests of drugs [36,41,42]. Two studies assessed alcohol use with time sensitive self-reported abstinence [43,45]. One study measured general substance use using time sensitive self-reported abstinence [44]. Five studies used self-reports of either abstinence or daily substance use behaviours as the outcome [41–45]. Two studies used validated tests of dose and dependence [36,41] carbon monoxide (CO) readings and/or saliva samples were tested for a certain amount of cotinine (<30 mg/mL). One study provided both self-reports of substance use and reports of validated tests of dose and dependence [41]. Only one of the included studies showed a statistically significant benefit of eHealth interventions over the control group [45].

#### *3.3. Risk of Bias in Included Studies RCTs*

The results of the quality assessments are shown in Figure 2. Overall, risk of bias was rated as low for 5 of the 6 studies that were included, where low was defined as a risk of bias score between 0–2. The most common risk of bias was due to attrition (missing data). Other common risks of bias within the current review were detection bias as well as selection bias.

**Figure 2.** Visual plot demonstrating results from quality assessment. Recall that the study informally defined scores from 0–2 as low risk, scores from 3–5 as moderate risk, and scores between 6–7 as high risk.

#### *3.4. Efficacy of eHealth Interventions on Substance Use*

Using a random-effects model, the efficacy of the eHealth interventions was tested by calculating a pooled OR across 1176 participants and comparing the intervention group (*n* = 585) to control group (*n* = 591). Results showed that pregnant participants who received an eHealth intervention for the treatment of substance use had 1.3 times greater likelihood of reduced substance use compared to those who were assigned to a control group (OR = 1.325, 95% CI = 1.062–1.654, Z = 2.490, *p* = 0.013; see Figure 3 and Table 2).



**Figure 3.** Forest Plot displaying individual and overall effect sizes examining the effectiveness of eHealth interventions.

Significant heterogeneity was observed across studies (Q = 4.505, *p* = 0.479, I<sup>2</sup> = 0.000). Egger's regression test (*B* = 1.39, *t*(4) = 4.43, *SE* = 0.324, *p* = 0.012) and the Begg and Mazumdar rank correlation test (*Tau* = 0.600, Z = 1.69, *p* = 0.090) showed mixed findings regarding the presence of publication bias, potentially due to the rank correlation test having lower power [46]. The funnel plot showed clear asymmetry on the positive side which suggests that the overall effect size may be smaller than estimated (See Figure 4).

**Figure 4.** Funnel Plot demonstrating the presence of publication bias.

A subgroup analysis was also conducted on the studies which measured abstinence (*n* = 4), which also revealed a small size effect size where the odds of increased abstinence was 1.25 times greater for pregnant individuals in the intervention group than for individuals within the control group (OR = 1.251, 95% CI = 0.993–1.577). Although secondary aims of the current study were to evaluate potential moderators of treatment for substance use, not enough studies were identified to conduct a moderator analysis.

#### *3.5. Sensitivity Analysis*

After systematically removing one study at a time, it was observed that two studies affected the meta-estimate of the effect size of eHealth intervention during pregnancy by more than 5% [36,41]. Specifically, the two studies affected the meta-estimate such that it made the estimate larger [36,41]. When Bullock et al., (2011) was removed, the pooled effect size increased to 1.69 [36]. When Naughton et al., was removed, the pooled effect size increased to 1.41 [41].

#### **4. Discussion**

#### *4.1. Primary Findings*

The current systematic review and meta-analysis synthesized existing evidence from 6 RCTs on the efficacy of eHealth interventions for substance use among a pregnant population by comparing participants using an eHealth intervention (*N* = 585) to participants in a control group (*N* = 591). Participant ages ranged from 18–37 years old and gestational ages ranged from 7.73–14.7 weeks. All the studies took place in an economically advantaged country which speaks to the potential difference in the accessibility for eHealth interventions for developing countries [47]. Most of the included RCTs measured smoking and drinking outcomes, whereas studies on eHealth interventions for harder drug use were more uncommon. The lack of studies measuring harder drug use among this population may be due to the stigma associated with these substances—particularly during pregnancy [48]. The majority of the eHealth interventions were delivered via computer/internet which is consistent with other reviews on eHealth where most eHealth interventions were internet-based. This may be attributed to the rise in technology use in recent years and that most of the included studies recently took place between the years of 2014–2018. With respect to the types of interventions used, most of the included interventions were brief in nature and had minimal clinician guidance, which could have reduced the current effect sizes. There was also variability in the time of follow-up among studies, with some studies having a shorter follow-up ranging from 4 weeks to 6 months. All of the included studies were assessed to have a low risk of bias which provides support for the high quality of the studies in the current review. The most common type of bias noted among studies was attrition bias, though this is common for eHealth interventions [49].

Results of the meta-analysis showed that eHealth interventions delivered in pregnancy reduced substance use when compared to control conditions (OR = 1.33, *p* = 0.013). This effect size was calculated using a predominantly intent-to-treat sample (ITT). With the exception of one study [43], each of the included studies reported an OR which were included based on an ITT analysis. Though it should be noted that the study which did not use an ITT sample completed a sensitivity analysis and found that the completers sample as opposed to the use of an ITT sample did not significantly differ [43]. Results suggest that eHealth interventions are significantly more effective than control conditions in reducing substance use in pregnancy. Findings from the sensitivity analyses found that two studies when removed, made the effect size larger [36,41]. Of note, both of these studies had smoking as an outcome which suggests that smoking may be less amenable to treatment in comparison to other substances. Furthermore, both the [36] study and the [41] study used a telephone to deliver their intervention. This is in comparison to the other studies which largely relied on the computer/internet. This may suggest that telephone and or text interventions may not be as effective in comparison to other modes of eHealth delivery.

#### *4.2. Consistency with Existing Literature*

The finding of a small yet significant effect of eHealth interventions is consistent with current literature of eHealth interventions for substance use within non-pregnant populations [27,50]. For example, in a meta-analysis examining the effectiveness of internet interventions for adults with substance abuse issues, results showed that internet interventions significantly decreased substance use with a small to moderate effect size (*Hedge's g* = 0.36) [27]. Results are also consistent with a meta-analysis examining the effectiveness of internet interventions for adult alcohol misuse, which also found a small but significant effect size in favour of the internet interventions (*Hedge's g* = 0.20) [50]. With respect to the effectiveness of eHealth interventions in a pregnant population, no review to date has synthesized this information. However, one review has observed the effectiveness of technology-based interventions for substance use among participants who were of a child-bearing age (ages 18–45) [51]. Results from this meta-analysis found that technologybased interventions were efficacious in comparison to control groups in preventing and

reducing substance use for individuals at a child-bearing age, though the effect size was small (*d* = 0.19) [51] which is in line with the current review.

The effect sizes of eHealth interventions are generally consistent with those of face-toface interventions for substance use within the general population. For example, a metaanalysis of psychosocial interventions for substance use in the general population found a significant yet small to moderate effect size for treatment (*Cohen's d* = 0.45) [28]. In another meta-analysis evaluating the effectiveness of motivational interviewing on substance use behaviours in adolescence, interventions were effective in improving substance use outcomes, but the effect sizes were again small (*Cohen's d* = 0.17) [52].

#### *4.3. Obstacles to eHealth Interventions*

In the studies included in this review, there were high attrition rates and varied engagement levels [53,54]. The observed attrition rates varied widely from 2% [44] to 33.5% [42] which is consistent with other eHealth interventions which typically range from 19% to 50% [55,56]. Individuals with substance use problems are more likely to terminate treatment than individuals with other psychosocial problems, with substance use treatment programs reporting the highest dropout rates when compared to individuals with other psychosocial concerns [57]. Taken together, these findings suggest that individuals taking part in a substance use eHealth intervention may require additional guidance and human monitoring to decrease levels of dropout. In the current review, only one study involved human monitoring, though this did not appear to increase effect sizes over and above other studies [36].

Furthermore, there were relatively high levels of participant attrition within the reviewed studies. This is a common feature among substance use eHealth studies and may in part be due to the lack of human monitoring. Across all eHealth programs, retaining pregnant participants to enhance positive outcomes continues to be a challenge. This speaks to the need to evaluate different ways to reduce attrition by improving participant engagement in treatment (e.g., gamification) [49,58] and implementing more rigorous designs which could include human monitoring and follow-ups by clinicians to reduce the high participant dropout rates. For example, the one study which did implement human monitoring had lower attrition rates relative to most of the other included studies [36]. In doing this, future eHealth interventions for substance use would maximize its clinical effectiveness [54,59].

#### *4.4. Strengths, Limitations and Future Directions*

Prior systematic reviews and meta-analyses were limited by the population studied, which to date, have only included the effects of eHealth on substance use within a general population. While some reviews exist on the effect of eHealth interventions on participants of "child-bearing age," these participants are not recruited during pregnancy and do not complete the eHealth intervention during pregnancy. The current study is the first, to our knowledge, to have synthesized the findings from the literature within a specifically pregnant population. This is incredibly important because as mentioned previously, many mental health and substance use behaviours in pregnancy persist into the postpartum period. Moreover, substance use has unique negative effects during pregnancy and pregnant populations may require support that is tailored to their needs.

Despite these strengths, the findings from this meta-analysis should be interpreted in the context of several limitations. Notably, the current review included a small number of studies (*n* = 6). Due to the limited number of studies included in the current analysis, there was not enough statistical power to conduct moderator analyses. Future research should investigate if demographic variables and/or study characteristics moderate the relationship between eHealth interventions and substance use during pregnancy. A larger research base is needed to better understand specifically what types of eHealth interventions and methods of delivery are most effective in pregnancy, and for whom. Substance use outcomes also varied—where most of the included studies assessed smoking cessation.

Consequently, future reviews may benefit from sub-group analyses to investigate if the effectiveness of eHealth interventions vary as a function of the type of substance use being treated and/or differences in using self-report and objective measures to assess treatment success. There was also wide variability in the types of eHealth interventions that were used, with delivery occurring through internet, phone, and text messages, which likely have also influenced the heterogeneity of the reported outcomes in this review. It should also be noted that since all reviewed studies took place in economically advantaged countries (i.e., United States, England, Netherlands), the lack of studies from other countries may limit the generalizability of our findings. Additionally, publication bias was observed in the outcomes Egger's Regression test (*p* = 0.012), which may suggest that effect sizes were over-estimated within the current review [38].

Future interventions should consider the high comorbidity that substance use has with various mental health concerns, including anxiety [60], depression [60], bipolar disorder, [61], attention-deficit hyperactivity disorder [62], and suicidality [63,64]. Moreover, individuals in the general population who were treated in programs providing specific treatment to target concurrent disorders had higher rates of using mental health services, which predicted improvements in both mental health and substance use following treatment [65]. Lastly, future studies should also examine the impact of eHealth treatments during pregnancy and in the postpartum period and beyond, with the hope that eHealth interventions will be able to create long-lasting treatment effects which persist beyond the intervention period.

#### *4.5. Clinical Implications*

According to the World Health Organization, all pregnant people should have access to affordable treatment options that respects their autonomy [66]. eHealth interventions may be best used as a first line intervention in stepped care models, as they may increase accessibility for some clients, and be less costly than more intensive in-person approaches [67,68]. The privacy and anonymity afforded by eHealth interventions may increase the likelihood of seeking support for substance use in pregnancy. eHealth interventions are also in line with international guidelines for the treatment of substance use during pregnancy [66,69]. eHealth interventions have the potential to be tailored to track substance use and treatment progress, as well as provide information on where pregnant people can receive more intensive substance treatment, which is in line with the National Institute for Health and Care Excellence guidelines in the treatment of substance use during pregnancy [69].

However, it is important to recognize that not everyone has access to reliable internet. Indeed, according to a report from the United Nations, over half of the world population does not have access to reliable internet, and there are important sociodemographic disparities in internet access within countries [70]. For example, only 24% of First Nation reserves in Canada have access to reliable internet [71]. Moreover, internet use may be limited be lack of devices (i.e., only one computer for one family), and compromised internet speed due to multiple devices. In another study which looked at telemedicine use in Peru, noted that almost 60% of the population in Peru belong to the lowest socioeconomic strata, preventing them from owning devices such as a computer or smartphone with internet access [72]. Furthermore, due to the lack of owning these devices, these individuals may lack the technological skills to know how to access and utilize telemedicine services, even if a device is provided to them [72]. Additionally, in a study of the disparities in digital access among Medicare beneficiaries, results found that individuals who lacked digital access were higher among those with low socioeconomic status, those 85 years or older, and in certain ethnic communities [73].

Accordingly, improved internet coverage and digital access has been highlighted as an important step into making eHealth more accessible. The future of eHealth should include determining how these interventions can be properly incorporated into the current healthcare system to increase patient accessibility to mental health services.

#### **5. Conclusions**

Taken together, the review found that eHealth interventions are effective in treating substance use during pregnancy. Furthermore, eHealth interventions are a promising healthcare intervention that increase access to care. In order to increase effect sizes, adaptations should be considered with development occurring, in conjunction with patient and provider partners. For example, future eHealth interventions could be more integrated such that treatment is completed in conjunction with therapist or peer support and additional guidance could be provided by having increased interactions with clinicians as opposed to pre-set modules which are to be completed at the patient's pace. This suggestion is supported by the finding that guided eHealth interventions were significantly superior to unguided interventions [74].

**Author Contributions:** Conceptualization, L.M.T.-M. and K.S.; methodology, H.S.; software, P.R.S.; validation, K.S., H.S. and P.R.S.; formal analysis, P.R.S.; investigation, K.S.; resources, L.M.T.-M.; data curation, H.S.; writing—original draft preparation, K.S.; writing—review and editing, K.S.; C.R.; H.S.K.; L.M.T.-M.; visualization, K.S.; supervision, L.M.T.-M.; project administration, L.M.T.-M.; funding acquisition, L.M.T.-M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the generous donors of the Alberta Children's Hospital Foundation, the Canadian Child Health Clinician Scientist Program (CCHCSP), the Canadian Institute of Health Research and the Fonds de Recherche du Québec—Santé.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** We thank the team of research assistants who contributed to data collection and extraction, including Roshni Sohail, Jasleen Kaur, Beatrice Valmana, Songyang (Mark) Jin, and Makayla Freeman.

**Conflicts of Interest:** The authors report no conflict of interest.

#### **Appendix A**


**Table A1.** Terms used in Search Strategy.


#### **References**


## *Systematic Review* **Adherence to Telemonitoring by Electronic Patient-Reported Outcome Measures in Patients with Chronic Diseases: A Systematic Review**

**Jim Wiegel 1,2,\*, Bart Seppen 1,2, Marike van der Leeden 1,3, Martin van der Esch 1,4, Ralph de Vries <sup>5</sup> and Wouter Bos <sup>1</sup>**


**Abstract:** *Background:* Effective telemonitoring is possible through repetitive collection of electronic patient-reported outcome measures (ePROMs) in patients with chronic diseases. Low adherence to telemonitoring may have a negative impact on the effectiveness, but it is unknown which factors are associated with adherence to telemonitoring by ePROMs. The objective was to identify factors associated with adherence to telemonitoring by ePROMs in patients with chronic diseases. *Methods:* A systematic literature search was conducted in PubMed, Embase, PsycINFO and the Cochrane Library up to 8 June 2021. Eligibility criteria were: (1) interventional and cohort studies, (2) patients with a chronic disease, (3) repetitive ePROMs being used for telemonitoring, and (4) the study quantitatively investigating factors associated with adherence to telemonitoring by ePROMs. The Cochrane risk of bias tool and the risk of bias in nonrandomized studies of interventions were used to assess the risk of bias. An evidence synthesis was performed assigning to the results a strong, moderate, weak, inconclusive or an inconsistent level of evidence. *Results:* Five studies were included, one randomized controlled trial, two prospective uncontrolled studies and two retrospective cohort studies. A total of 15 factors potentially associated with adherence to telemonitoring by ePROMs were identified in the predominate studies of low quality. We found moderate-level evidence that sex is not associated with adherence. Some studies showed associations of the remaining factors with adherence, but the overall results were inconsistent or inconclusive. *Conclusions:* None of the 15 studied factors had conclusive evidence to be associated with adherence. Sex was, with moderate strength, not associated with adherence. The results were conflicting or indecisive, mainly due to the low number and low quality of studies. To optimize adherence to telemonitoring with ePROMs, mixed-method studies are needed.

**Keywords:** adherence; patient reported outcomes; patient reported outcome measures; telemonitoring

### **1. Introduction**

The increasing incidence of chronic diseases and the proportional increase in health care costs require efficacy in healthcare [1–3]. To improve efficacy, new ways of delivering healthcare were evaluated, such as the use of telemonitoring for chronic diseases. Telemonitoring, or remote patient monitoring, is defined as the use of technology to monitor patients at a distance [4]. One way of telemonitoring is collecting repetitive electronic patient-reported outcome measures (ePROMs) at home, which allows for a quick and easy way of a frequent capture of important disease-specific outcomes and is already

**Citation:** Wiegel, J.; Seppen, B.; van der Leeden, M.; van der Esch, M.; de Vries, R.; Bos, W. Adherence to Telemonitoring by Electronic Patient-Reported Outcome Measures in Patients with Chronic Diseases: A Systematic Review. *Int. J. Environ. Res. Public Health* **2021**, *18*, 10161. https://doi.org/10.3390/ ijerph181910161

Academic Editors: Marie Carmen Valenza and Irene Torres-Sanchez

Received: 9 August 2021 Accepted: 17 September 2021 Published: 27 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

applied in a variety of chronic diseases such as chronic obstructive pulmonary disease, chronic kidney disease, rheumatoid arthritis, inflammatory bowel disease or congestive heart failure [5–10]. The benefits of telemonitoring by repetitive ePROMs are (i) a reduced number of outpatient visits, (ii) a reduced workload, and (iii) lower healthcare costs [11–13]. Patients that use repetitive ePROMs for telemonitoring reported (iv) improved satisfaction, (v) improved communication with the provider, (vi) more insight into their disease activity, and (vii) more control over their disease [14–16]. However, one of the main problems of telemonitoring by ePROMs is a lack of adherence, with adherence rates down to 20% [17,18]. Although adherence toward health technology is increasingly investigated, there is no comprehensive review regarding what quantitative factors are associated with adherence to telemonitoring by ePROMs specifically.

The low rate of adherence may affect the effectiveness of ePROMs. It is important to identify which factors affect adherence to telemonitoring by repetitive ePROMs, since this allows for a targeted approach in order to improve adherence and therefore its effectiveness [19]. A recent systematic review qualitatively investigated patients' experience of telemonitoring (including but not limited to telemonitoring with ePROMs) and suggested that older patients and patients with less experience with technology were concerned about their ability to use telemonitoring [20]. Furthermore, the coexistence of comorbidity, social support, self-discipline and the use of persuasive design principles in the telemonitoring tool (i.e., use of reminders) were all qualitatively identified as possible factors affecting adherence [21–24]. However, it is not yet known if they influence adherence to telemonitoring by ePROMs specifically and if the quantitative evidence supports and extends these observations.

A systematic review was performed with the aim to identify factors quantitatively associated with adherence to telemonitoring by repetitive ePROMs in patients with chronic diseases in all studies.

#### **2. Materials and Methods**

#### *2.1. Search Strategy*

A literature search was performed from inception to 6 June 2020 and was updated on 8 June 2021, based on the preferred reporting items for systematic reviews and metaanalyses (PRISMA) statement (www.prisma-statement.org, accessed on 5 March 2020). At the initiation of this review, the new PRIMSA guidelines were not yet available; therefore, this review was conducted following the 2009 PRISMA guidelines [25]. To identify all relevant publications, we conducted systematic searches in the bibliographic databases PubMed, Embase, PsycINFO (Ebsco) and the Cochrane Library (Wiley), in collaboration with a medical information specialist. The following terms were used (including synonyms and closely related words) as index terms or free-text words: "electronic patient reported outcome", "e-Health", "Telemonitoring", "Remote patient monitoring", "Mobile applications", "Chronic disease", "Adherence", "Usage", "Dropout", Engagement" and "Compliance". The references of the identified articles were searched for relevant publications. Duplicate articles were excluded. The full search strategies for all databases can be found in Appendix A (Table A1).

#### *2.2. Selection Process*

Two reviewers (JW and BS) independently screened all potentially relevant titles and abstracts for eligibility. We selected articles where (P) patients with chronic diseases (I) telemonitored their symptoms with repetitive ePROMs, (C) no comparison groups were necessary, and (O) where factors potentially positively or negatively associated with adherence were investigated, (S) in trial or cohort studies. Differences in judgement were resolved through consensus. Studies were included if they met the following criteria:


Exclusion criteria were:

The described chronic disease is a mental disorder according to the *Diagnostic and Statistical Manual of Mental Disorders-5* [26].

If necessary, the full text article was checked for the presence of all inclusion criteria.

#### *2.3. Data Collection Process and Data Items*

Data were extracted with the aid of a standardized data form by one reviewer (JW) [27]. The second reviewer (BS) sampled 50% of the articles for accuracy. The extracted information of each study consisted of: author, year of publication, study design, region, participant characteristics (sample size, diagnosis, age, sex, education level, work status and clinical characteristics), telemonitoring characteristics (design, frequency of intended usage, definition of adherence and automated reminders) and outcome measures (adherence rates and results for each investigated factor potentially affecting adherence).

#### *2.4. Methodological Quality of Individual Studies*

Both researchers (JW and BS) assessed the risk of bias of the included studies independently. Disagreements were solved by discussion, and if necessary, a third party was consulted (WB). For RCTs the Cochrane risk of bias tool was used and for cohort studies the risk of bias in nonrandomized studies of interventions (ROBINS-1) [28,29]. The Cochrane risk of bias tool aims to disclose relevant information to the risk of bias within a fixed set of domains through signaling questions in randomized trials. The domains of bias investigated were (1) random sequence generation, (2) allocation concealment, (3) selective reporting, (4) other sources of bias, (5) blinding of participants and personnel, (6) blinding of outcome assessment, and (7) incomplete outcome data and can be judged as "high risk", "low risk" or "unclear risk", if the data are insufficient for judgement. The ROBINS-1 tool evaluates the risk of bias in nonrandomized studies of interventions such as cohort studies and is based on the Cochrane risk of bias tool. The domains of bias that were assessed were (1) confounding, (2) selection of participants, (3) classification of intervention, (4) deviation from intended intervention, (5) missing data, (6) measurement of outcomes, and (7) selection of the reported results. Through signaling questions, each domain was classified as "low risk", "moderate risk", "serious risk", "critical risk" or "no information". The overall risk of bias classification for the article was equal to the classification of the domain with the highest risk of bias. Articles describing subgroup analyses of previously performed RCTs were assessed as an RCT, and the original publication was retrieved to assess the methodological quality properly.

#### *2.5. Data Synthesis*

The identified factors were categorized according to the WHO five dimensions associated with adherence, (1) social/economic factors (i.e., age and level of education), (2) health system/health care team-related factors (i.e., quality of consultations), (3) therapy related factors (i.e., treatment duration), (4) condition-related factors (i.e., symptom severity), and (5) patient related factors (i.e., self-efficacy) [30]. Although this categorization was intended for medication adherence or adherence to health therapy, we used the five dimensions for aggregating the results solely since such a classification does not exist for ePROM/technology adherence. If the factors were investigated in the included studies but were not described by the WHO, we categorized the factor in the most suitable dimension. To assess whether a meta-analysis was possible, we explored the heterogeneity of the included studies based on intervention (length of follow-up, app or web-based intervention, intended frequency of ePROMs and used description of adherence), context (diagnosis)

and target participants (demographics, severity of symptoms and comorbidity) following the guidelines of Pigott et al. [31]. An evidence synthesis was performed when quantitative analysis was not possible due to high heterogeneity. In the evidence synthesis, the results were assigned to one of five levels of evidence: strong, moderate, weak, inconclusive or inconsistent following the criteria adapted from Ariëns et al.; see Table 1 [32]. When available, the results of univariable analysis were used for synthesis; otherwise, the results of multivariable analysis were used.

**Table 1.** Strength of evidence criteria [32].


#### **3. Results**

The literature search generated a total of studies: 3943 in PubMed, 4339 in Embase, 714 in PsycINFO and 2485 in the Cochrane Library. After removing the duplicates of references that were selected from more than one database, 7275 studies remained. All the remaining studies were screened based on title and abstract. A total of 51 full-text studies were assessed for eligibility, and finally five studies were included in the synthesis. The flow chart of the search and selection process is shown in Figure 1.

The included studies consisted of one article that analyzed the subgroups of RCTs, two prospective uncontrolled studies and two retrospective cohort studies [33–37]. Study populations comprised of patients with rheumatoid arthritis, heart failure, chronic pain and systemic auto-immune diseases. Publication dates varied between 2013 and 2020; see Table 1. A detailed description of the study characteristics is presented in Table 2. Table 3 shows the tool and intervention characteristics.

The RCT was judged at high risk of bias on one domain, blinding of participants and personnel [33]. Both prospective uncontrolled studies were judged at serious risk due to risk of confounding [34,35]. Both retrospective cohort studies were judged at moderate risk [36,37]. The full quality assessment is presented in Figure 2 for RCTs and Figure 3 for prospective uncontrolled and retrospect cohort studies. The heterogeneity of the studies was considered too high to perform a meta-analysis; therefore, an evidence synthesis was performed.

#### *3.1. Adherence*

Overall adherence to repetitive ePROMs ranged between 61% and 96% in the studies. The definition of adherence varied between studies. In three studies, the percentage of completed questionnaires was used as a proxy for adherence and reported as a continuous variable, and two studies had a predefined number as cut-off point for high/low.

#### *3.2. Factors Affecting Adherence Classified by the Five WHO Dimensions*

A total of five studies investigated 15 unique factors. Of the factors investigated, eight belonged to the social/economic dimension (1), two factors to condition related (2), three to the patient related dimension (3), one to the therapy related dimension (4), and one to the healthcare team related dimension (5); see Table 4.

**Figure 1.** Flowchart of the search and selection process.



\* SLE = Systemic Lupus Erythematosus, IBD = Inflammatory Bowel Disease, COPD = Chronic Obstructive Pulmonary Disease.


**Figure 2.** Risk of bias assessment following the Cochrane risk of bias tool for RCTs.

**Figure 3.** Risk of bias assessment following the ROBINS-1 for nonrandomized interventions.


**Table 4.** Overview of the investigated factors and their association with adherence.

↑ = significant positive association with adherence, ↓ = significant negative association with adherence, ↔ = no significant association with adherence.

#### *3.3. Social/Economic Dimension*

There was moderate-level evidence that sex is not associated with adherence, as none of the five studies found sex was associated with adherence. The evidence for the association of age and marital status with adherence was inconsistent, while education level, annual income, number of children, area of residence and employment status were inconclusive. Colls et al. showed that patients over 65 years old had higher adherence compared to patients <45 years old, 77% vs. 62% *p* = 0.02 (32). By contrast, the four other studies did not find any difference regarding age and adherence [34–37]. Jamilloux et al. found an association between higher response rate and being married/having a partner (OR = 2.89, *p* = 0.02), in contrast to two studies who did not find any difference in adherence rates regarding marital status [34–36]. Higher education, annual income, area of residence (urban or not), employment status and number of children at home were all investigated by a single study, of which only a greater number of children at home was associated with higher adherence (average of 1.1 children at nonadherent group vs. 1.3 children at adherent group *p* = 0.004) [34].

#### *3.4. Condition-Related Dimension*

There was inconsistent and inconclusive evidence that symptom severity and disease duration, respectively, affected adherence. Colls et al. found that lower baseline disease activity, or lower symptom severity, was associated with higher adherence in comparison with higher disease activity (76% vs. 58%, *p* = 0.02), in contrast to Ross et al. who found that a higher symptom severity was associated with more daily ePROM reports [33,37]. The association between symptom duration and adherence was only investigated by Ross et al., who did not find an association.

#### *3.5. Patient-Related Dimension*

The coexistence of comorbidity (including depression) and previous participation in online surveys was investigated in only one study [36]. Guzman et al. showed that a higher score on the Charlson comorbidity index was correlated with lower adherence (beta = −0.13, *p* < *0*.05) (35). Jamilloux et al. used previous participation in online surveys as surrogate for eHealth literacy and found it to be associated with higher adherence (61% in the adherent group vs. 38% in the nonadherent group OR = 2.56 95% CI 1.02–6.72 *p* = 0.04) [34]. According to Rosen et al., depression was not associated with adherence [35].

#### *3.6. Therapy/Intervention-Related Dimension*

Evidence for the association of satisfaction with the app/web-based intervention and adherence was inconsistent. Ross et al. showed that the number of daily pain entries was positively associated with satisfaction of the app, while Colls et al. did not find an association between global treatment satisfaction and adherence [33,37].

#### *3.7. Healthcare-Team-Related Dimension*

Guzman et al. found a positive association between higher adherence and patients treated at primary care, compared to patients treated at specialized care (OR 2.51 95%CI 1.22–5.19 *p* < 0.03) [36]. Since this was investigated by only one study, the result was classified as inconclusive.

#### **4. Discussion**

One of the main problems of telemonitoring by ePROMs is a lack of adherence and the lack of knowledge of the factors that influence adherence to telemonitoring. This review aimed to identify which factors are associated with adherence to telemonitoring by ePROMs in patients with chronic diseases. The definition of adherence varied between studies, with reported adherence ranging from 61% to 96%. Symptom severity, comorbidity, marital status, eHealth literacy and satisfaction with the intervention may be associated with higher adherence, but the evidence was not conclusive for any of the identified factors. Moderate evidence was found that sex is not associated to adherence.

We quantitatively investigated adherence to telemonitoring with ePROMs, in addition to previously performed qualitative research. Recent qualitative research toward telemonitoring (including but not limited to telemonitoring by ePROMs) found that coexistence of comorbidity, older age and eHealth illiteracy may negatively affect adherence [20,22]. In addition, increased social support, higher self-discipline and the usage of persuasive design in the telemonitoring tool were all qualitatively identified as possible factors positively affecting adherence [21,23,24]. Three of these factors were investigated quantitatively in our included studies: older age, eHealth literacy and comorbidity.

Colls et al. reported higher adherence in older patients, which is contrary to the earlier mentioned qualitative research and the widely accepted unified theory of acceptance and use of technology (UTAUT2) framework [38]. The UTAUT2 describes that older people may have more difficulty in adapting new technologies and hypothesized that older people are less frequently eHealth literate. However, a recent study in patients with chronic diseases showed that only eHealth literacy, and not age, was predictive for eHealth adherence [39]. The fact that smartphone usage in 55+ years old increased from 40% in 2014 to 90% in 2017 in the Netherlands suggests that the eHealth illiterate group might be decreasing in patients of older age [40]. Therefore, age seems to be of diminishing importance in eHealth adherence, while eHealth illiteracy remains a potential barrier. Guzman et al. identified the presence of comorbidity as a factor negatively affecting adherence, in accordance with qualitative research [22]. Multimorbidity is frequently present in patients with chronic diseases, for example in RA with an estimated prevalence of two-thirds of the patients [41]. Described rationales for this negative impact on adherence are that (1) comorbidity may physically limit the use of mHealth, and (2) it may shift the patients' priority away from the primary disease [23]. Therefore, patients with multimorbidity are potentially in extra need for attention or help in order to achieve higher adherence. Measures such as the index of coexistent diseases are best-suited to study the impact of comorbidity on adherence, since they include both the disease severity and physical impairment of the comorbidities [42,43].

Symptom severity was not identified by qualitative research as a barrier for telemonitoring adherence but showed remarkable contradictive results in the included studies, acting both as a facilitator and barrier [33,37]. In studies investigating medicine adherence, it has mainly been described as a potential barrier. Patients considered high symptom severity as a barrier (i) when they are becoming skeptical toward the efficacy if previous treatment had failed, (ii) when establishing routines for disease management are difficult when symptoms are severe, (iii) and when patients experience a reduced quality of interaction with their healthcare provider when symptoms are severe [44–46]. However, high symptom severity may act as a facilitator if the patients experience greater benefits from the intervention when the symptoms are severe [37,47]. Therefore, the role of symptom severity on adherence may depend on the patients' perceived benefits of telemonitoring and how easy the tool is to use even when symptom severity is high.

Only one factor was investigated in both the intervention-related and healthcarerelated dimension. However, investigating intervention and healthcare factors may be valuable. For example, a high level of assistance of the healthcare practitioner may have a positive influence on adherence [48,49]. Furthermore, the frequency at which patients need to report ePROMs influences adherence is unknown. A higher frequency may lead to reporting fatigue and therefore lower adherence, but a higher frequency may also facilitate adherence in acquiring the habit more easily. Future studies investigating ePROM adherence should focus more on healthcare- and intervention-related dimensions. Not many factors are investigated both qualitatively and quantitatively, mainly due to the low number of studies investigating adherence of telemonitoring with ePROMs quantitatively. Quantitative research may help to identify which factors lead toward a decrease or increase in adherence on a group level. Furthermore, it gives the possibility of monitoring the effect of adjustments for improving adherence on group level. Therefore, it is of importance to combine qualitative and quantitative studies in prospective mixed-method studies in order to investigate thoroughly the adherence of telemonitoring by ePROMs.

Given the presented results, where the required skills and ease of use of the telemonitoring tool were mentioned as possible reasons why factors such as eHealth illiteracy, higher disease state and comorbidity may negatively impact adherence, it seems logical to design and use telemonitoring tools with a highly perceived ease of use as a possible first step to improve adherence. It is already known that this is an important factor in the successful implementation of eHealth; however, it also seems to be of importance for the sustained usage over time for telemonitoring [50]. This emphasizes the importance of a user-centered design [51].

#### *Limitations*

This review has several limitations which should be considered when interpreting the results. First, comparing results of the studies is difficult because of the large heterogeneity in diagnosis, intervention and outcome measurements between the included studies. This is partly due to the inclusion of all chronic diseases which was necessary since studies investigating ePROM adherence quantitatively are scarce, despite the increasing interest in telemonitoring by ePROMs especially during the COVID-19 pandemic. During the pandemic, patients with chronic diseases were often instantly telemonitored since outpatient clinic visits limited their access to high-priority care only. Therefore, there must be many currently unpublished data, which can help us understand which patients need extra assistance or are better suitable for telemonitoring. We hope that by underlining this knowledge gap, we can motivate others to analyze and publish their adherence data. Secondly, the methodological quality of the studies was low, leading to weak, inconsistent or inconclusive evidence. This underlines the need for large, well-designed prospective cohort studies with the primary focus on investigating adherence. Thirdly, there is a persisting general lack of knowledge about how often ePROMs should be reported in order for telemonitoring to be effective [52]. This makes determining adherence mostly arbitrary.

#### **5. Conclusions**

The factor sex is not associated with adherence to repetitive ePROMs. Although several studies reported various associations between factors and adherence, the results show to be inconsistent or inconclusive. A limited number of telemonitoring studies by ePROMs report in-depth adherence data, making it difficult to draw conclusions from the studied factors. Future ePROM-guided telemonitoring studies should report adherence data in a standardized way. Mixed-method analyses are preferred for studies investigating telemonitoring adherence with ePROMs.

**Author Contributions:** J.W. and R.d.V. performed the search. J.W. and B.S. independently screened all potentially relevant titles and abstracts for eligibility. J.W. performed the data extraction which B.S. sampled. J.W, B.S., W.B., M.v.d.L. and M.v.d.E. had substantial contribution to the methodology and writing of this manuscript. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**


#### **Table A1.** Pubmed search results.


#### **References**


### *Review* **"Help in a Heartbeat?": A Systematic Evaluation of Mobile Health Applications (Apps) for Coronary Heart Disease**

**Chiara Mack 1, Yannik Terhorst 2, Mirjam Stephan 1, Harald Baumeister 2, Michael Stach 3, Eva-Maria Messner 2, Jürgen Bengel <sup>1</sup> and Lasse B. Sander 1,\***


**Abstract:** For patients with coronary heart disease (CHD) lifestyle changes and disease management are key aspects of treatment that could be facilitated by mobile health applications (MHA). However, the quality and functions of MHA for CHD are largely unknown, since reviews are missing. Therefore, this study assessed the general characteristics, quality, and functions of MHA for CHD. Hereby, the Google Play and Apple App stores were systematically searched using a web crawler. The general characteristics and quality of MHA were rated with the Mobile Application Rating Scale (MARS) by two independent raters. From 3078 identified MHA, 38 met the pre-defined criteria and were included in the assessment. Most MHA were affiliated with commercial companies (52.63%) and lacked an evidence-base. An overall average quality of MHA (*M* = 3.38, *SD* = 0.36) was found with deficiencies in information quality and engagement. The most common functions were provision of information and CHD risk score calculators. Further functions included reminders (e.g., for medication or exercises), feedback, and health management support. Most MHA (81.58%) had one or two functions and MHA with more features had mostly higher MARS ratings. In summary, this review demonstrated that a number of potentially helpful MHA for patients with CHD are commercially available. However, there is a lack of scientific evidence documenting their usability and clinical potential. Since it is difficult for patients and healthcare providers to find suitable and high-quality MHA, databases with professionally reviewed MHA are required.

**Keywords:** coronary heart disease (CHD); apps; mobile health; eHealth; systematic evaluation

#### **1. Introduction**

Cardiovascular diseases and especially coronary heart diseases (CHD) are one of the leading causes of death worldwide [1,2]. According to the global burden of disease study 17.8 million people died from cardiovascular diseases in 2017 [1]. According to the heart disease and stroke statistics the prevalence of CHD in the US ranges from 5.3% for female adults to 7.4% for male adults [3].

CHD and common complications like arrhythmia, myocardial infarction, and heart failure have a significant negative impact on the affected person's health, leading to high mortality and healthcare costs [4–6]. In addition, certain mental and physical conditions and factors are associated with CHD, including depression, cigarette smoking, hypertension, and obesity [7–10].

Disease management and behavior change including lifestyle changes are key aspects of CHD care but often not adequately and enduringly considered in care settings [11]. The

**Citation:** Mack, C.; Terhorst, Y.; Stephan, M.; Baumeister, H.; Stach, M.; Messner, E.-M.; Bengel, J.; Sander, L.B. "Help in a Heartbeat?": A Systematic Evaluation of Mobile Health Applications (Apps) for Coronary Heart Disease. *Int. J. Environ. Res. Public Health* **2021**, *18*, 10323. https://doi.org/10.3390/ ijerph181910323

Academic Editors: Irene Torres-Sanchez and Marie Carmen Valenza

Received: 25 August 2021 Accepted: 20 September 2021 Published: 30 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

large number of risk and lifestyle factors render the prevention and self-management of CHD extensive and complex for patients [11,12]. Therefore, means of promoting disease management and lifestyle changes as well as information are necessary to improve prevention and conventional treatment of CHD [11,13,14]. Mobile health applications (MHA) are discussed to contribute in overcoming this gap in treatment by fostering CHD management [13,15]. First, MHA may support daily monitoring of activities and symptoms [16]. Second, adherence to treatment and lifestyle changes can be increased by self-tracking, feedback, and reminder functions of MHA [16,17]. Third, MHA are accessible at all times and at relatively little costs [18] making MHA a scalable solution to provide general information about CHD, symptoms, and specific lifestyle modifications [19,20]. Fourth, MHA can increase patients' perception to play an active role in their own healthcare and hereby foster self-sufficiency, disease management, and patient autonomy [16,18,21].

However, high-quality applications with suitable content are required, while the quality of MHA is largely unknown due to an intransparent MHA market and a lack of methodologically sound quality assessments [16,22]. Previous studies focused on other cardiological conditions examining the quality of MHA for heart failure [23,24], atrial fibrillation [25], and blood pressure [26]. Hereby the quality of MHA was reported as mostly acceptable [24] or mainly poor [23,26]. This is particularly alarming because MHA can also be harmful [19,22]. Risks and constraints regarding MHA concern data security, privacy, and confidentiality, since missing privacy policies and information transfer to third parties have been observed [22,27]. Furthermore, possible misinformation poses potential risks to users and the sheer number of MHA may lead to consumer confusion [11,19,28–30]. No evaluation of MHA specifically for CHD was found [16].

Therefore, in this study we systematically searched for and conducted a standardized evaluation of MHA for CHD which are available in commercial app stores. Hereby we addressed the following research questions:


#### **2. Materials and Methods**

#### *2.1. App Search Strategy*

With an automated search engine (web crawler) of the 'Mobile Health Application Database' (MHAD) [31] the Google Play store and Apple App store were systematically searched for MHA. Search terms to identify CHD applications included 'coronary heart disease', 'coronary artery disease', 'ischemic heart disease', and 'heart disease' in English and German. A list of all search terms is included in Appendix A. The searches were conducted between December 2020 and February 2021. Duplicates were automatically removed. For the assessment, MHA from the Google Play store were installed on an Honor 6X (BLL-L22) and apps from the Apple App store on an iPad Pro A1652.

#### *2.2. Inclusion Criteria and Process*

The identified MHA were examined for eligibility in a two-step procedure. In the first step, the title and app description were screened and the inclusion criteria for the download of MHA were applied. Apps were downloaded if (a) in the app title or description the subject of coronary heart disease was stated, (b) the app was developed for patients with CHD, persons at risk, or otherwise affected individuals, (c) the MHA was available in German or English language, and (d) download was possible.

In a second step, the identified apps were downloaded and the criteria for inclusion in the evaluation were examined within the app. MHA were included if (a) CHD was focused, a CHD-specific section was included, or the app description stated its use for CHD, (b) no other specific information (such as login/ access data) was required for usage of the app, (c) the application was functional, and d) there were no further technical reasons to eliminate the MHA. Technical malfunctions were tested on two devices.

#### *2.3. Data Extraction, Evaluation Criteria, and Instruments*

Two independent raters (master's degree students in psychology C.M. and M.S., under supervision of a licensed psychotherapist L.B.S.) conducted the acquisition and rating of all included MHA by applying the Mobile Application Rating Scale (MARS) in the German version [32–34]. For all sections of the MARS a good to excellent internal consistency (Omega = 0.793 to 0.904), an overall excellent internal consistency (Omega = 0.929) and a good intra-class correlation (ICC = 0.816, 95% CI: 0.810 to 0.822) were shown [34]. Therefore, with the MARS the quality of MHA can be assessed reliably [34]. The MARS contains a section for classification and for quality rating as well as three additional subscales.

To prepare for the app rating with the MARS, a free online tutorial provided by the developers of the German MARS version was viewed. For the rating, each app was tested by trying out all features. To check the agreement between the raters, the interrater reliability (IRR) was calculated. Here, the intra-class correlation (ICC) needs to be ≥ 0.75 to indicate a sufficient agreement [35]. In case of an ICC below 0.75 the supervisor (L.B.S.) was consulted.

#### *2.4. General Characteristics of MHA*

For this study the MARS classification section was adapted to include the following general characteristics: (1) app name, (2) platform (Android, iOS), (3) affiliation, (4) price, (5) embedment in therapy, (6) user star rating, (7) number of user ratings, (8) app store category, (9) methods, (10) technical aspects, and (11) security and privacy.

#### *2.5. Quality Rating*

For the quality rating with the MARS 19 items are rated on a five-point scale ranging from 1 (inadequate) to 5 (excellent). These items constitute the four dimensions: (A) engagement (five items: entertainment, interest, individual adaptability, interactivity, target group), (B) functionality (four items: performance, usability, navigation, gestural design), (C) aesthetics (three items: layout, graphics, visual appeal), and (D) information quality (seven items: accuracy of app description, goals, quality of information, quantity of information, quality of visual information, credibility, evidence base). To assess the evidence-base, for each MHA Google Scholar was searched for published studies.

#### *2.6. Statistical Analysis*

For each of these four dimensions, the mean score (*M*) and standard deviation (*SD*) were computed as well as a total mean quality score across all four objective dimensions [33]. The scores of both raters were averaged. Additionally, the three subjective subscales of the MARS: (E) therapeutic gain, (F) subjective quality, and (G) perceived impact were evaluated without effect on the overall mean score. Correlation analyses between the available user star ratings (one star to five stars) and the MARS total mean score as well as the objective dimensions were conducted if at least three ratings were available.

#### *2.7. Assessment of Functions*

Subsequently, the employed functions of the included MHA were assessed with a classification from the 'Chances and Risks of Mobile Health Apps' (CHARISMHA) study [36]. The classification is divided into six categories with one to five subcategories each. These are: provision of information (news, reference, learning material, player/viewer, broker), data acquisition, processing, and evaluation (decision support, calculator, meter, monitor, surveillance/tracker), administrative use (administration), calendar and appointmentrelated apps (diary, reminder, calendar), support (utility/aid, coach, health manager) and other (actuator, communicator, game, store, other). Hereby, for each MHA it was examined which functions are employed. Additionally, a correlation between the number of functions and the MARS total score was calculated.

#### **3. Results**

#### *3.1. Search*

In Figure 1 the screening and inclusion process is illustrated. A total of 3078 apps were found through the web crawler. From 1217 apps without duplicates, 38 MHA (3.12%) were included in the evaluation. Of those, 30 apps (78.95%) were available on android, seven apps (18.42%) on iOS, and one app (2.63%) for both.

#### *3.2. General Characteristics*

The characteristics of included MHA are depicted in Table 1. The apps were affiliated with commercial companies (*n* = 20, 52.63%), non-governmental organizations (NGO; *n* = 2, 5.26%), universities (*n* = 2, 5.26%), and governments (*n* = 1, 2.63%). For 13 apps (34.21%) the affiliation was unknown. The basic version was free of cost for most apps (*n* = 34, 89.47%) and required payment for four apps (10.53%) with prices ranging from EUR 1.09 to EUR 3.69 (*M* = 2.57, *SD* = 1.08). In three apps (7.89%) an upgraded or extended pro version was available or in-app purchases were possible. No app was embedded in a treatment concept or had a certification to comply for example with the medical device regulation.

For 12 apps (31.58%) a user rating was available in the Google Play store and for one app (2.63%) in the Apple App store. The median user star rating in the Google Play store was 4.4 (*M* = 4.26, *SD* = 0.47) with five to 1276 ratings (*M* = 220.42, *SD* = 403.29) and the user star rating in the Apple App store was 1.0 with one rating (user ratings last updated on 4 April 2021). MHA were classified in eight app store categories: 'Health & Fitness' (*n* = 18, 47.37%), 'Medical' (*n* = 10, 26.32%), 'Education', 'Books & Reference' (*n* = 3, 7.89% each), 'Lifestyle', 'Food & Drink', 'Entertainment', and 'Social Networking' (*n* = 1, 2.63% each). In 19 apps (50.00%) internet was required for some or all functions and one app (2.63%) had an app community. Most common methods were information and education (*n* = 34, 89.47%), tips and advice (*n* = 25, 65.79%), and feedback (*n* = 16, 42.11%). For most apps a privacy policy (*n* = 26, 68.42%) and contact information (*n* = 33, 86.84%) was provided and in seven apps (18.42%) active consent was required. Login was necessary in six apps (15.79%) and a password protection in three apps (7.89%).

For one MHA ('The Heart App') the accuracy to detect acute coronary syndromes was examined in a diagnostic accuracy study [37]. Otherwise, no study or randomized controlled trial (RCT) was found.

#### *3.3. Quality Rating of MHA*

The MARS rating results for each included MHA are presented in Table 2. The total quality of included MHA was average, with *M* = 3.38 (SD = 0.36) and ranged from *M* = 2.50 to *M* = 4.22. Of the four objective dimensions, the highest-rated was functionality (*M* = 4.06, *SD* = 0.31), thereafter aesthetics (*M* = 3.62, *SD* = 0.47), followed by information quality (*M* = 3.18, *SD* = 0.43), and engagement (*M* = 2.64, *SD* = 0.55). For the additional subjective subscales, the means were lower, with *M* = 2.49 (*SD* = 0.35) for therapeutic gain, *M* = 2.45 (*SD* = 0.52) for subjective quality, and *M* = 1.90 (*SD* = 0.35) for perceived impact. The IRR for the rating of all MHA was excellent (2-way mixed ICC = 0.944, 95%-CI 0.935 to 0.952) and for no single app an ICC below 0.75 was evident. No significant correlations between user star ratings and MARS total mean score (*r* (10) = -0.52, *p* = 0.080) or the objective subscales (*r* (10) = 0.001–0.52, *p* > 0.05) were found. In addition, none of the apps that required payment for the basic version were among the ten highest-rated apps.

**Figure 1.** Flowchart of the app screening and inclusion process.

**Table 1.** General characteristics of included MHA for coronary heart disease.



Note. *n* = number of apps; *M* = mean; *SD* = standard deviation.

#### *3.4. Functions of MHA*

The number of functions of all MHA is presented in Appendix B. The most common function was provision of information, specifically reference (information texts about e.g., CHD, risk factors, or treatment) in 37 apps (97.37%) and a player/viewer (e.g., video or audio files) in six apps (15.79%). Hereof 28 MHA (73.68%) were primarily assigned to the category provision of information. Further functions were decision support (e.g., concerning foods or goals; *n* = 4, 10.53%), calculators such as CHD risk score or body mass index (BMI) calculators (*n* = 13, 34.21%) and monitor of activity (*n* = 1, 2.63%). Hereof six MHA (15.79%) were primarily categorized under data acquisition, processing, and evaluation. Other functions were diary (*n* = 1, 2.63%), reminder (e.g., for medication or workouts) and calendar functions (*n* = 3, 7.89% each), with no MHA primarily being a calendar and appointment-related app. Three MHA (7.89%) functioned as health managers, targeting goals regarding exercise, weight, and nutrition, and were categorized as support apps. Two MHA (5.26%) were communicators and of those, one app (2.63%) was primarily a social network.

The functions of the ten highest-rated apps are shown in Table 3 and a full table depicting all employed functions per MHA is included in Appendix C. In general, many MHA had one (*n* = 17, 44.74%) or two functions (*n* = 14, 36.84%) and in seven apps (18.42%) three or more functions were employed. Of those MHA with three or more functions, five apps (71.43%) were among the ten highest-rated apps. A significant positive correlation with a large effect size was found between the MARS total score and the number of employed functions (*r* (36) = 0.66, *p* < 0.001).


**Table 2.** Means of the MARS rating from highest to lowest total score.

Note. \* fee required. GP = Googly Play, AA = Apple App, E = Therapeutic Gain, F = Subjective Quality, G = Perceived Impact.


Note. ✔ primary function of the MHA, ✓ function is employed in the MHA.


#### **4. Discussion**

This study is the first to systematically review MHA for CHD by assessing the general characteristics, quality, and functions of MHA in European app stores. The overall quality of apps for CHD, as assessed with the MARS, was average (*M* = 3.38, *SD* = 0.36). Here, the functionality and aesthetics of included apps were generally high while deficits in information quality and engagement were shown. This is in line with previous studies which reported a varying, but largely acceptable or poor quality of apps for cardiological conditions such as heart failure, atrial fibrillation, or hypertension [23,24,26]. Since many MHA primarily provide information, the deficits in the average information quality are alarming [11,13]. A total of *N* = 3078 apps were identified by the web crawler and only 38 apps (3.12%) were CHD-specific and met the inclusion criteria. Above that, none of the objective subscales nor the MARS total mean scores were significantly correlated with the user star ratings which may increase the challenge of patients with CHD to identify a reliable MHA restricting the clinical use of MHA. Since none of the paid apps were among the ten highest-rated MHA, requiring payment is also not an adequate indicator of app quality. Furthermore, some MHA with very little CHD-specific information, an overwhelming amount, or questionable content were found. This included MHA implying to cure CHD only by certain yoga practices or solely by specific natural remedies. From a clinical perspective, this misleading information can have harmful effects on affected users such as not seeking professional medical advice or treatment.

Considering this, the lack of evidence regarding the usefulness and effectiveness of MHA is concerning. Only one study [37] investigating the diagnostic accuracy of one MHA could be found, which corresponds to previous app reviews, demonstrating little evidence-base for commercially available MHA [38–41]. This is in line with a validation study that included MHA for several health conditions and reported no evidence-base for 94.8% of the 1299 included MHA [34]. As many of the included MHA only consist of information texts or calculators, efficacy studies are rather inadequate, since symptom reduction due to those MHA alone is unlikely. However, studies examining for example the usability, feasibility, or user acceptance could be of importance and increase the scientific discourse on MHA. This necessity is further increased by the fact, that only few MHA (13.16%) were developed by credible sources such as universities, NGOs, or governmental organizations.

Identified functions included information, CHD risk score calculators, reminders for exercise or medication, feedback to data entries and health managers with goal setting regarding exercise, weight, and nutrition. With regard to the overall conceptualization of MHA we found that the majority comprised solely one or two functions while in only seven apps (18.42%) three or more functions were employed. The number of employed functions was positively correlated with the MARS total score, therefore those MHA with several functions were mostly among the highest-rated apps. Most MHA were limited to information and CHD risk score calculators. Thus, the majority of apps fall short of their potential to foster behavior change in patients using reminders, notifications, achievements, and encouragement and regarding important lifestyle changes like quitting smoking, being more active, or eating healthier [42]. Nevertheless, the embedment of MHA in current treatment models could most likely be valuable for patients as well as for health care providers being able to quickly access patients' data. Independent expert rating platforms or databases such as www.mhad.science (accessed on 19 September 2021) or https://mindapps.org (accessed on 19 September 2021) are necessary to support patients and providers in finding and choosing reliable MHA of high quality [43].

This study has some limitations. First, the web crawler is limited to 200 apps per search term and the European market. This might deform the results by omitting some MHA, even though apps specifically developed for the US market were also found. Second, some apps might be locally restricted and published for specific countries only. Additionally, apps that required specific login/ access information were excluded, since they are not instantly available for most users, reducing the number of MHA. Third, only MHA in

German or English language and for the chosen search terms were covered. Therefore, the number of included MHA is potentially not extensive and future studies could examine the search terms that patients use when looking for CHD apps. Fourth, the development of apps is very rapid [44], resulting in one app ('Atherosclerosis') no longer being available for rating by the second reviewer and some apps no longer being detectable between the first and second screening or for inclusion. Fifth, according to the standard procedure of the MARS two reviewers rated the apps, even though more raters would lead to more accurate estimates.

Sixth, with the MARS the quality of MHA is evaluated in regard to engagement, functionality, aesthetics, and information quality. In addition to those dimensions, other aspects might be relevant for app users as well and a high MARS score does not imply a high effectiveness of the app. Hence, in future studies this systematic review of MHA could be replicated with other instruments like ENLIGHT [45] or different suitable scales.

#### **5. Conclusions**

This first systematic evaluation of MHA for CHD demonstrated an average overall quality of MHA (*M* = 3.38, *SD* = 0.36). The most common functions were information texts and risk score calculators. Only few MHA provide a set of multiple functions and incorporate behavior change techniques limiting the potential for lifestyle changes and support in disease management of users. Most MHA were not developed by a credible source and there is a considerable lack of scientific evidence for the usefulness and efficacy of the included MHA. Nevertheless, some potentially helpful MHA were identified. The results of this study will be made publicly available to users and healthcare providers at www.mhad.science (accessed on 19 September 2021).

**Author Contributions:** Conceptualization, C.M., L.B.S., Y.T., and J.B.; methodology, Y.T., L.B.S., and E.-M.M.; web crawler and tooling, M.S. (Mirjam Stephan), H.B., and E.-M.M.; validation, C.M., L.B.S., and M.S. (Mirjam Stephan); formal analysis, C.M.; investigation, C.M..; data curation, C.M.; writing original draft preparation, C.M.; writing—review and editing, L.B.S., Y.T., C.M., J.B., E.-M.M., H.B., and M.S. (Michael Stach); supervision, L.B.S.; project administration, L.B.S.; All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding. The article processing charge was funded by the Baden-Wuerttemberg Ministry of Science, Research and Art and the University of Freiburg in the funding program Open Access Publishing.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The raw data supporting the conclusions of this article will be made available by the corresponding author on request.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **Appendix A**

**Table A1.** List of all search terms for coronary heart disease.


#### **Appendix B**

**Table A2.** Functions of the included MHA.


**Appendix C**

