**TableA3.**Employedfunctions perincludedMHA.



#### **References**


### *Review* **An Updated Meta-Analysis of Remote Blood Pressure Monitoring in Urban-Dwelling Patients with Hypertension**

**Sang-Hyun Park 1,†, Jong-Ho Shin 1,†, Joowoong Park <sup>2</sup> and Woo-Seok Choi 3,4,\***


**Abstract:** Following the coronavirus disease-2019 pandemic, this study aimed to evaluate the overall effects of remote blood pressure monitoring (RBPM) for urban-dwelling patients with hypertension and high accessibility to healthcare and provide updated quantitative summary data. Of 2721 database-searched articles from RBPM's inception to November 2020, 32 high-quality studies (48 comparisons) were selected as primary data for synthesis. A meta-analysis was undertaken using a random effects model. Primary outcomes were changes in office systolic blood pressure (SBP) and diastolic blood pressure (DBP) following RBPM. The secondary outcome was the BP control rate. Compared with a usual care group, there was a decrease in SBP and DBP in the RBPM group (standardized mean difference 0.507 (95% confidence interval [CI] 0.339–0.675, *p <* 0.001; weighted mean difference [WMD] 4.464 mmHg, *p <* 0.001) and 0.315 (CI 0.209–0.422, *p <* 0.001; WMD 2.075 mmHg, *p <* 0.001), respectively). The RBPM group had a higher BP control rate based on a relative ratio (RR) of 1.226 (1.107–1.358, *p <* 0.001). RBPM effects increased with increases in city size and frequent monitoring, with decreases in intervention duration, and in cities without medically underserved areas. RBPM is effective in reducing BP and in achieving target BP levels for urban-dwelling patients with hypertension.

**Keywords:** blood pressure; remote monitoring; hypertension; telemedicine; urban

#### **1. Introduction**

Hypertension is widely recognized as the most important risk factor for cardiovascular disease (CVD), which is a major cause of total mortality [1]. A 2 mmHg fall in systolic blood pressure (SBP) has been reported to reduce the incidence of ischemic CVD and stroke by 7% [2]. However, even in advanced countries, target blood pressure (BP) is achieved in <50% of patients with hypertension [3,4]. The 2017 American College of Cardiology/American Heart Association (ACC/AHA) and 2018 European Society of Cardiology/European Society of Hypertension (ESC/ESH) treatment recommendations state that BP must be controlled to stricter levels [5,6].

Remote BP monitoring (RBPM) has been recommended for hypertension diagnosis and treatment [5,6], as it has been reported to predict CVD morbidity and mortality with higher accuracy than office BP monitoring [7]. As a method of telemedicine, RBPM is known to be an effective tool to enhance drug adherence and BP control in patients with hypertension [8–12]. RBPM has been suggested as a potential solution to overcome the geographical limitations of healthcare services [13], with significant effects shown in randomized controlled trials (RCTs) and meta-analysis studies [10,14–16]. The 2017

**Citation:** Park, S.-H.; Shin, J.-H.; Park, J.; Choi, W.-S. An Updated Meta-Analysis of Remote Blood Pressure Monitoring in Urban-Dwelling Patients with Hypertension. *Int. J. Environ. Res. Public Health* **2021**, *18*, 10583. https:// doi.org/10.3390/ijerph182010583

Academic Editors: Irene Torres-Sanchez and Marie Carmen Valenza

Received: 23 September 2021 Accepted: 5 October 2021 Published: 9 October 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

ACC/AHA guidelines also recommended RBPM for hypertension diagnosis and control, and for enhancing patients' drug adherence [6].

According to the United Nations, approximately 68% of the human population is predicted to dwell in urban settings by 2050 [17]. Urbanization is a rapidly growing 21st century trend, with significant effects on human health. However, despite increased interest in new health technologies, several studies have reported that remote monitoring has limited application in urban settings where high-quality face-to-face care is possible and healthcare accessibility is high [18,19]. Moreover, there is no comprehensive evidence concerning the effect of RBPM in improving clinical outcomes of urban-dwelling patients with hypertension or whether RBPM can become a standard treatment for hypertension management.

In a previous meta-analysis of RCTs using the Jovell/Navarro-Rubio classification system to determine the strength of evidence, RBPM showed statistically significant reductions in SBP (3.48 mmHg) and diastolic BP (DBP, 1.64 mmHg) compared with usual care (UC) after an average of 7.6 months for patients dwelling in an urban setting. In terms of CVD prevention, however, RBPM induced <0.5% of CVD prevention in low-risk patients with hypertension. Therefore, some studies have concluded that RBPM is of little practical significance to policy-makers [20,21]. The coronavirus disease-2019 (COVID-19) pandemic resulted in a steeply increased demand for telemedicine, even in urban settings, for those otherwise having adequate availability and accessibility to healthcare services. More generally, characteristically dense populations in cities have resulted in the rapid spread of infectious diseases, leading to the expansion of infrastructure for non-face-to-face care in line with a rapid increase in the use of the internet and mobile devices.

Considering the global rate of BP control, according to 2017 ACC/AHA guidelines for hypertension diagnosis and control, which is the latest strict guideline for hypertension diagnosis and control, the proportion of patients achieving the target BP is predicted to decrease further. The use of remote medical care services suddenly increased during the COVID-19 pandemic [14,22,23], and its use needs to be verified based on the integration of previous findings, given that hypertension is a chronic disease requiring long-term management for CVD prevention and for efficient healthcare policies to be implemented in urban settings. Therefore, relevant studies need to be extended through an updated compilation of BP data. The objective of our study is to evaluate whether RBPM could be utilized as an alternative to standard treatment for urban-dwelling patients with hypertension during the COVID-19 pandemic. Thus, this study aimed to determine the relative effects of RBPM compared with UC based on outcomes including SBP, DBP, and BP control rates. Intervention duration, city size, setting, frequency of remote transmission of BP data, and the presence of medically underserved areas (MUAs) in the city were analyzed as secondary factors to evaluate the effects of RBPM. We hypothesized that the effects of RBPM were equivalent to those of UC. To test this hypothesis, relevant, up-to-date RCTs were systematically reviewed and transparent and reliable quantitative data synthesis was performed.

#### **2. Materials and Methods**

#### *2.1. Searching for Eligible Studies*

This study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines of the Cochrane Collaboration and a checklist was provided [Supplementary Materials] [24]. To identify eligible studies, two investigators (SHP and JHS) independently searched the following electronic databases: PubMed, EBSCOhost, Embase, and the Cochrane Library, from RBPM's inception to November 30, 2020. Free terms were used, including and related to *urban*, *hypertension*, and *remote monitoring*, along with medical subject heading (MeSH) terms. Truncation and phrasing methods were applied to derive a structured search formula [20] (Appendix A). The formula was first applied to the Cochrane Library and then converted to suit each database for the subsequent search. Articles written in English were retrieved. To include as many relevant articles

as possible, all systematic reviews and meta-analyses related to the search themes were collected from each database and Google Scholar, and their reference lists were reviewed. To identify gray literature, relevant websites were used, and all studies including those in which the city area was not clearly defined were identified through a manual search.

#### *2.2. Inclusion and Exclusion Criteria*

All included studies were blinded RCT studies with random and uniform allocation of patients with hypertension into an RBPM group and a traditional face-to-face UC group. Articles reporting pre- and post-intervention data were targeted, with participants satisfying the following criteria: (1) patients with hypertension under management through regular visits to an urban medical institution; (2) patients able to measure their own BP at home; (3) patients able to transmit their BP data to the physician via post, phone, Bluetooth device, mobile phone, web, or computer (wired or wireless); (4) adults aged ≥18 years; (5) BP measurement through ambulatory monitoring; and (6) various transmission methods from real-time or a stored and forward method to an automatic or manual method. Exclusion criteria comprised the following: (1) sudden BP changes due to an acute CVD or cerebrovascular accident (CVA); (2) patients undergoing hemodialysis due to acute or chronic renal disease; (3) female patients before and after pregnancy; (4) cases not reported for urban areas or cases for urban and rural areas reported together; (5) cases from unclear target areas; (6) cases where monitoring was aided by medical staff at a nursing management unit or care facility; and (7) cluster trials or cross-over studies.

#### *2.3. Study Selection*

The citations retrieved from each database were exported to EndNote X8.2, and two investigators (SHP and JHS) independently eliminated those not satisfying the criteria to confirm the reliability of identification. First, the title and abstract were screened, and for studies satisfying the criteria, full texts were obtained and scrutinized. Primary studies were selected independently, and their reference lists were reviewed. Final articles for data synthesis were determined after discussion with the senior author (WSC).

#### *2.4. Data Extraction and Coding*

For the selected studies, data extraction was performed independently by two investigators (JHS and WSC), and relevant values were coded in an electronic sheet. The extracted data included demographic and pre- and post-intervention SBP and DBP data. BP data were mostly obtained using an automated device and, in the case of ambulatory BP monitoring (ABPM), the mean of each group was calculated and coded. If an article did not report BP values or standard deviations (SDs), preventing calculations with a 95% confidence interval, the values were first checked on the trial registries website and, in cases where the required information could not be obtained, an attempt was made to contact the author of the article [25,26]. Articles that satisfied the inclusion criteria but did not report the main BP data were excluded from the final data synthesis. For some studies with missing SDs, data imputation was performed using a simple method [27,28]. The mean of all other studies, excluding those with missing data, was obtained. Regarding the rate of BP control, the number of patients satisfying the level of normal BP, determined during the final follow-up period of comparison in each study, was calculated and compared between the two groups. If a single primary study included several different follow-up periods for comparison [26–35]; applied a different, additional intervention [25,34]; or had multiple varying sample sizes and thus reported varying results, each result was included in the analysis as an independent study. Disagreements between investigators were resolved through consultation with the senior author (WSC).

#### *2.5. Quality Assessment and Publication Bias*

The quality assessment of the primary studies included evaluating the risk of bias (RoB) and was performed independently by two investigators (SHP and JP). Using the

Review Manager program (RevMan, version 5.3.5, Copenhagen, Denmark) software from the Cochrane Collaboration, the evaluation was performed according to the Cochrane Handbook for Systematic Reviews of Interventions guidelines [24,36]. Disagreements were resolved through discussion among investigators. To identify publication bias, Egger's regression, classic fail-safe N, Duval and Tweedie's trim-and-fill method, and funnel plots were used.

#### *2.6. Statistical Analysis*

To ensure the reliability of the analysis, coded data were analyzed by two investigators (SHP & JHS) using Comprehensive Meta-Analysis version 2 (CMA, Biostat, Englewood, NJ, USA) software. For primary outcomes, continuous variables comprised the weighted mean difference (WMD) and the standardized mean difference (SMD) obtained from the mean SBP and DBP values measured at baseline and during follow-up in the office. Despite divided opinions regarding the use of continuous variables, SMD has shown a trend of higher statistically significant generalizability and percentage agreement than the WMD in a random effects model (REM) and a fixed effects model (FEM) [37,38]. Therefore, SMD was used in this study to report the results of the data synthesis for continuous variables. Considering the generalizability of each result, the WMD was additionally estimated for comparing the subgroup results [38]. Based on Cohen's general rule of thumb, the effect size was set as follows: SMD 0.2 (small effect); SMD 0.5 (medium effect), and SMD 0.8 (large effect) [39]. Accordingly, when the SMD was ≥0.5, we considered the effect size to be significant in this study. The rate of BP control was a dichotomous variable, for which BP normalization data were extracted from each study, and effect size based on relative risk (RR) was used. A 95% confidence interval (CI) was used for all data. To analyze the inter-rater difference, a χ<sup>2</sup> test was used and the level of significance was set to *p <* 0.10. The model of analysis was applied after assessing the enrolled population of each study and the heterogeneity among research centers. Between-study heterogeneity was presented using Tau-squared (τ2) and I-squared (I2) indices, and the adequacy of results was determined based on Cohen's general rule of thumb [40]. Therefore, in this study, 30 ≤ I <sup>2</sup> ≤ 60 indicated moderate heterogeneity and 50 ≤ I <sup>2</sup> ≤ 90 indicated substantial heterogeneity [39]. To assess the quality of each trial and the consequent impact on the overall effect size, sensitivity was tested using the "one study removed" method (Appendix B, Figure A1). A cumulative analysis was run for a total of 48 comparisons, and the range of summary effect sizes at each step according to temporal progression was determined. *p*-values and the presence of outliers affecting the overall effect size were also determined (Appendix C, Figure A2). An additional sensitivity test was performed to determine differences between the data before and after imputing the missing values.

#### **3. Results**

#### *3.1. Study Characteristics*

Through an initial search of available databases, reference to trial registries, and a manual search of reference lists, a total of 2721 citations were retrieved (Figure 1). Of these, 992 duplicates were removed, leaving 1729 citations to be identified. Next, titles and abstracts for each identified citation were screened, and 1217 irrelevant citations were excluded. For the remaining 512 articles, the full text was obtained and scrutinized, and studies without available data (*n* = 206), studies not performed in an urban area, studies either reporting combined results of urban and rural areas or not reporting the area (*n* = 192), studies conducted on patients with CVD or CVA that may induce a sudden change in BP, studies conducted on patients undergoing hemodialysis or including patients with chronic renal disease, and studies involving female patients before or after pregnancy (*n* = 46) or patients aged <18 years (*n* = 21) were excluded. In total, 32 independent studies (48 comparisons) satisfying the inclusion criteria were used in the final data synthesis (Table 1).

**Figure 1.** PRISMA flow of study. Abbreviations: BP, blood pressure; CVD, cardiovascular disease; CVA, cerebro-vascular accident; DBP, diastolic blood pressure; SBP, systolic blood pressure.

For the primary studies included in the meta-analysis in this study, the duration of RBPM was 2–18 months (mean, 7.37 months), and the number of participants in the UC and RBPM groups was 5666 and 5729, respectively. The mean age of participants in the UC and RBPM groups was 52.63 and 52.17 years, respectively. No significant intergroup differences were found in terms of sex and baseline BP. No differences in ethnicity were observed. Fourteen studies were conducted in primary medical institutions, 12 in community healthcare centers, and 22 in hospitals or higher-level institutions. The completion dates were in or prior to the year 2000 for two studies [41,42], between 2001 and 2010 for 14 studies [25,29,43–50], and between 2011 and 2020 for 32 studies. Seven studies had used mean values for ABPM [47,48,50–54].


**Table 1.**

Characteristics

 of individual primary studies.

tension












system



*Int. J. Environ. Res. Public Health* **2021**, *18*, 10583




274

**Standard Error**

#### *3.2. Risk Assessment*

To check for bias in RCT studies, the Cochrane Group's RoB tool of the Cochrane group was used for domain analysis based on a checklist. Across seven domains, a low risk of selection bias related to sequence generation or allocation concealment was shown. Similarly, the risk of detection bias related to blinding of personnel and patients was appropriately reported. Concerning attrition bias (incomplete outcome data), an unclear or sufficiently high risk was shown that raised concern in a number of studies; however, as most studies showed a low risk (≥4) across the seven domains, the overall RoB was deemed to be low [62].

Egger's regression intercept was 4.516 (1.363–7.669; *p =* 0.005) in two-tailed 95% CIs [37]. The number of studies needed to attain *p >* 0.05 for a classic fail-safe N was 5085. The point estimate of SBP in Duval and Tweedie's trim-and-fill analysis (SMD, 0.507 mmHg (0.339–0.645, *p <* 0.001); WMD, 4.464 mmHg (*p <* 0.001)) coincided with the summary effect size, while no imputed study was found in the funnel plot (Figure 2) [63]. The SMD of DBP was 0.253 (0.215–0.292), and no study was trimmed (Figure 3). In the analysis of the rate of target BP achievement, RR was 1.237 (1.107–1.381), three studies were imputed, and the adjusted value was 1.161 (1.032–1.306, Figure 4). Although RoB assessment detected a certain level of publication bias, the overall data were statistically significant and the analysis results were not rejected.

#### **Funnel Plot of Standard Error by Std diff in means**

**Figure 2.** A funnel plot of the standardized mean difference in systolic blood pressure. Note: summary effect size (-), summary effect size of imputed studies (◆), individual study ().

**Std diff in means**

275

**Funnel Plot of Standard Error by Std diff in means**

**Figure 3.** A funnel plot of standardized mean difference in diastolic blood pressure. Note: summary effect size (-), summary effect size of imputed studies (◆), individual study ().

**Figure 4.** A funnel plot of relative risk for the target blood pressure rate. Note: summary effect size (-), imputed study (•), summary effect size of imputed studies (◆), individual study ().

**Log risk ratio**

A sensitivity test was performed for studies that had been included to prevent smallstudy effects, excluding those with a sample size of ≤50 for the RBPM group [64]. The test results showed an SMD of 0.501 mmHg (0.313–0.689, *p <* 0.001) and a WMD of 4.238 mmHg (*p <* 0.001), indicating that the difference from the overall summary effect size was not clinically significant and that the potential small-study effect was not significant in this study.

#### *3.3. Primary Outcomes*

#### 3.3.1. Systolic Blood Pressure

Across 32 independent studies (48 comparisons), 11,395 patients (UC group, *n* = 5666; RBPM group, *n* = 5,729) were analyzed for SBP [25–35,41–61]. The summary SMD was 0.507 (0.339–0.675, *p <* 0.001), showing an above moderate effect size, and the WMD after conversion was 4.464 mmHg (3.371–5.556, *p <* 0.001; Figure 5). The between-group heterogeneity was significant (I<sup>2</sup> = 70.908%, *p <* 0.001). To determine the effect of individual studies on the total summary effect size, a sensitivity test was performed using the "one study removed" method, whereby each study was sequentially omitted (Appendix B). Here, the point estimate of the summary effect size showed no significant difference and no outliers were detected.


**Figure 5.** A forest plot of standardized mean difference in systolic blood pressure. Note: point estimate of individual study (•), summary effect size (◆); SBP, systolic blood pressure; UC, usual care; RBPM, remote blood pressure monitoring.

When the average effect of RBPM was chronologically divided into three timeframes and compared with the UC group (Phase I, inception of RBPM to 2000; phase II, 2001–2010; phase III, 2011–2020), the WMD was 1.515 mmHg (*n* = 2, −4.031–7.061, *p =* 0.592; I <sup>2</sup> = 0.000%, *p =* 0.478) in phase I [41,42], 4.333 mmHg (*n* = 14, 2.338–6.328, *p <* 0.001; I <sup>2</sup> = 38.554, *p <* 0.001) in phase II [25,29,43–50], and 4.719 mmHg (*n* = 32, 3.343–6.094, *p <* 0.001; I2 = 77.361%, *p <* 0.001) in phase III [26–28,30–35,51–61].

#### 3.3.2. Diastolic Blood Pressure

To determine the effect of RBPM on DBP, data concerning 10,482 patients (UC group, *n* = 5192; RBPM group, *n* = 5290) were analyzed across 29 studies (44 comparisons) [25,27– 35,41–54,56–59,61]. Compared with the UC group, the RBPM group showed greater BP reduction (SMD, 0.315 mmHg (0.209–0.402), *p <* 0.001; WMD, 2.075 mmHg (1.399–2.750) *p* < 0.001) after conversion (Figure 6). The between-study heterogeneity was substantial (I2, 68.021%; *p <* 0.001). No outliers were detected in the sensitivity test performed through sequentially omitting each study.

**Figure 6.** A forest plot of standardized mean difference in diastolic blood pressure. Note: point estimate of individual study (•), summary effect size (◆); DBP, diastolic blood pressure; UC, usual care; RBPM, remote blood pressure monitoring.

The WMD according to time interval was 2.059 mmHg in phase I (*n* = 2, −1.143–5.262, *p =* 0.208; I2 = 0.000%, *p =* 0.45)[41,42], 1.587 mmHg in phase II (*n* = 14, 0.421–2.753, *p <* 0.001; I <sup>2</sup> = 17.407%, *p <* 0.001) [25,29,43–50], and 2.348 mmHg in phase III (*n* = 28; 1.480–3.216, *p <* 0.001; I2 = 76.230%, *p <* 0.001) [26–28,30–35,51–61].

#### 3.3.3. Target Blood Pressure Rate

To determine the effect of RBPM, the rate of BP control was estimated based on BP normalization criteria defined in each primary study. Across 16 studies (25 comparisons), the data of 2655 patients in the UC group and 2816 patients in the RBPM group were comprehensively analyzed [13,25,30–33,45–47,50–53,59,61]. Compared with the UC group, the RBPM group showed a significant effect, with an approximately 23.7% higher improvement in BP control based on RR (RR= 1.226 (1.107–1.358), *p <* 0.001; Figure 7). The between-study heterogeneity was substantial (I<sup>2</sup> = 70.656%; *p <* 0.001). No significant difference in summary effect size was found in the sensitivity test.


**Figure 7.** Risk ratio of target blood pressure using remote blood pressure monitoring. Note: point estimate of individual study (•), summary effect size (◆); BP, blood pressure; UC, usual care; RBPM, remote blood pressure monitoring.

#### *3.4. Subgroup Analysis*

#### 3.4.1. City Size

Generally accepted international criteria define city size according to population size in a given area. In this study, a metropolitan city was defined as a city with a population of at least one million. Thus, the RCT studies included in this study were categorized based on city size as either a small-to-medium-sized city study or a large city study, and the two categories were analyzed separately. Population size was estimated from the data of the latest international population survey performed in the nearest period of time to this study. Of the 48 studies, 22 were conducted in small-to-medium cities [25,28,29,32,41–43,46,47,50, 51,54–57] and 26 were conducted in large cities [26,27,30,31,33–35,44,45,48,49,52,53,58–61]. For the former, the SBP showed a WMD of 3.860 mmHg (2.271–5.450, *p <* 0.001) without between-study heterogeneity (I<sup>2</sup> = 0.000, *p =* 0.478; Tau<sup>2</sup> = 0.000). For the latter, the SBP showed a WMD of 5.056 mmHg (3.503–6.609, *p <* 0.001) with a significant level of betweenstudy heterogeneity (I2 = 82.177%, *p <* 0.001, Tau2 = 17.368); the magnitude of the effect size was above moderate.

#### 3.4.2. Medically Underserved Areas

The presence of MUAs for each group was reflected in the analysis only if the study clearly indicated the respective area. As a result, 17 studies were categorized as MUAs [28, 29,31,32,35,41,44,57,59,61] and 31 as non-MUAs [25–27,30,33,34,42,43,45–56,58,60]. In terms of MUAs, the effect of RBPM on SBP showed a WMD of 3.213 mmHg (1.521–4.905, *p <* 0.001), with I<sup>2</sup> = 48.904% (*p* = 0.012, Tau2 = 2.793), indicating a moderate degree of between-study heterogeneity based on Cohen's rule of thumb. In contrast, in non-MUAs, the effect of RBPM on SBP showed a WMD of 5.224 mmHg (3.878–6.569; *p <* 0.001), with I <sup>2</sup> = 73.152% (*p <* 0.001, Tau<sup>2</sup> = 12.943).

#### 3.4.3. Duration of Intervention

The effect of reduced SBP based on the WMD varied according to the duration of the intervention. For an intervention duration ≤3 months [27–30,33,34,49,51,52,54,58,60], the effect was a WMD of 6.219 mmHg (*n* = 15, 3.970–8.468, *p <* 0.001; I2 = 70.060, *p <* 0.001). For 6 months [26–30,32,33,35,42,45,47,48,55,59], the effect was a WMD of 4.491 mmHg (*n* = 14, 2.461–6.521, *p <* 0.001; I2 = 84.562, *p <* 0.001). For 12 months, the effect was a WMD of 3.446 mmHg (*n* = 12, 1.209–5.683, *p =* 0.003; I<sup>2</sup> = 34.656, *p* = 0.113). The rate of BP control had an RR of 1.540 (*n* = 6, 1.223–1.939, *p <* 0.001) after 3 months [27,30,33,51,52,54], an RR of 1.159 (*n* = 11, 1.002–1.341, *p =* 0.047) after 6 months [25,27,30,32,33,45,47,48,59], and an RR of 1.167 (*n* = 5, 0.930–1.464, *p =* 0.183) after 12 months [32,46,50,53,61] (Appendix D, Figure A3).

#### 3.4.4. Setting

The BP reducing effect was analyzed according to the size of the medical institution where RBPM was mainly performed. In primary care clinics, the WMD was 2.981 mmHg (*n* = 14, 1.323–4.639, *p <* 0.001; I<sup>2</sup> = 45.343, *p =* 0.034) [25,35,44,45,47,48,50,51,53,54,56]. In community health centers, the WMD was 3.512 mmHg (*n* = 12, 1.651–5.373, *p <* 0.001; I <sup>2</sup> = 31.670, *p <* 0.001) [27–30,42,57,61], and the WMD at hospital level was 6.333 mmHg (*n* = 22, 4.750–7.917, *p <* 0.001; I2 = 73.401, *p <* 0.001) [26,31–34,41,43,46,49,52,55,58–60].

#### 3.4.5. Frequency of Remote Transmission of Blood Pressure Data

In the primary studies in which the frequency of remote BP transmission was reported, when BP information was transmitted daily, the WMD was 5.881 mmHg (*n* = 13, 3.898–7.864, *p <* 0.001; I<sup>2</sup> = 14.635, *p <* 0.001) [27,34,49,53–55,59–61]. For weekly BP transmission, the WMD was 4.024 (*n* = 15, 2.641–5.406, *p <* 0.001; I2 = 54.610, *p <* 0.001) [28,30, 32,42,43,45,47,52,56–58]. For biweekly BP transmission, the WMD was 3.941 mmHg (*n* = 4, 1.428–6.454, *p <* 0.001; I<sup>2</sup> = 0.000). For monthly BP transmission, the WMD was 1.803 mmHg (*<sup>n</sup>* = 6, −0.234–3.841, *p =* 0.083; I<sup>2</sup> = 21.639, *p =* 0.056) [26,35,41,50].

#### **4. Discussion**

The development of healthcare infrastructure and physicians' preference for practice in an urban setting implies higher accessibility to healthcare and higher patient satisfaction regarding healthcare [65]. However, the COVID-19 pandemic has raised concerns regarding face-to-face care in cities being a potential infection route between healthcare professionals and patients. In this study, data published since September 2018 were included and integrated with data from previous studies to undertake an updated analysis.

Compared with UC, RBPM for urban-dwelling patients with hypertension was found to significantly reduce SBP and DBP in both statistical and clinical terms, while improving the rate of BP control. Following RBPM, SBP and DBP WMDs decreased by 4.464 mmHg and 2.075 mmHg, respectively, compared with UC. This change, observed through quantitative data, showed a greater margin of decrease than reported in a previous meta-analysis

(SBP, 3.482 mmHg; DBP, 1.638 mmHg) [20]. Moreover, according to the temporal interval, the decrease in SBP (1.515 vs. 4.719 mmHg) and DBP (2.059 vs. 2.438 mmHg) in phase III was significantly greater than that in phase I. Therefore, we consider that the demand for RBPM has increased in line with technological advancements, the increased use of mobile devices, and the acceptance of new technologies [66].

RBPM is frequently used in pilot projects preceding the full launch of telemedicine, as it is relatively simple and cost-effective compared with other types of telemedicine. However, reports on the effect of RBPM on the rate of BP control have been inconsistent across numerous previous studies [14]. In this study, where additional data were comprehensively analyzed to extend the meta-analysis, RBPM led to an approximately 20% higher rate of BP control than UC. This is a greater magnitude of improvement than the 13% figure reported in a previous analysis [20]. Considering that the rate of BP control is <50% in traditional face-to-face care, even in countries with advanced healthcare systems, an improvement of 20% is indicative of a highly significant contribution to the prevention of CVD [67].

The ultimate objective behind attempts to lower and control BP in patients with hypertension and to bring it closer to a target BP is to reduce the incidence of CVD. However, in the meta-analysis in this study, data were not analyzed in relation to cardiovascular (CV) events because the included RCTs primarily showed outcomes that targeted changes in BP or the rate of BP control, not CV events. Nevertheless, the effect of RBPM on CV events in urban-dwelling patients with hypertension can be conjectured based on the results of previous studies. In a previous large-scale meta-analysis on prospective monitoring, including randomized, controlled, placebo trials or anti-hypertensive studies, a decrease of 2–3 mmHg in SBP in patients with a moderate risk of CVD was shown to cause a 10% reduction in CV mortality and a 20–30% reduction in major adverse CV events [2,68–70]. Thus, the observed decrease in SBP of 4.464 mmHg in this study, when the WMD was compared between UC and RBPM, is clinically significant and potentially contributes to reducing CV events.

The effect size of the primary outcomes was set as the SMD and, as it showed moderateto-high heterogeneity (I2 = 70.908%; *p <* 0.001), a subgroup analysis was performed (Appendix E, Table A1). First, the analysis according to city size (based on population size) showed that the effect of RBPM was greater in cities with a population of ≥1 million (SBP, 3.860 mmHg, *p <* 0.001; I<sup>2</sup> = 0.000, *p* = 0.478) than in small-to-medium cities with a smaller population, although within-study heterogeneity was high (I<sup>2</sup> = 82.177, *p <* 0.001). The effect of RBPM in reducing SBP was statistically significant compared with UC, irrespective of city size. The rate of BP control also showed greater effects in large cities (RR, 1.268; *p <* 0.001) than in small-to-medium-sized cities (RR, 1.157; *p =* 0.094). In a previous literature review, the intervention effect was found to be smaller in larger cities (large city, 3.229 mmHg vs. small-to-medium city, 3.765 mmHg), where the difference was considered to be associated with the difference in technological utility based on acceptance [66]. In particular, there was a sudden rise in demand for telemedicine to avoid the transmission of infectious diseases in response to the COVID-19 pandemic in 2020 [71,72].

Second, subgroup analysis was also performed according to urban MUAs in terms of healthcare accessibility. The decrease in BP in relation to RBPM in non-MUAs was 5.224 mmHg (I<sup>2</sup> = 73.152%, *p <* 0.001), indicating a greater effect of RBPM in reducing SBP compared with MUAs (3.213 mmHg, *p <* 0.001; I2 = 48.904%, *p* = 0.012). The extent to which the level of within-study heterogeneity affects the summary effect size remains unclear, but the results of the analysis provided supporting evidence for determining the overall effect. Although a precise reason for this result could not be identified in this study, the following factors may be considered: changes in attitudes towards the use of mobile devices and chronic disease management and changes in economic lifestyle related to reduced opportunities for healthcare. These results may be used as evidence by healthcare policy-makers to support the need for differentiated policies for the supply of telemedicine in urban settings.

Third, a subgroup analysis was also performed concerning the duration of intervention. No optimal schedule has been established for the period of management of hypertension based on RBPM and the frequency of remote transmission of data [47,73]. Despite slight differences in the magnitude of reduction in SBP, RBPM in this study showed a consistent effect of reducing SBP, regardless of duration. Nonetheless, as the intervention duration increased, the level of BP reduction decreased. The reason for such a decrease could not be clearly identified, but possible causes may be fatigue, indifference, and inadequate level of perceived utility due to the prolonged performance of the intervention [73,74]. However, considering that it is essential to achieve a target BP as early as possible in patients with hypertension to prevent CVD, the effect of RBPM on the early outcome of BP reduction may be emphasized for its use in practice. The optimal duration of RBPM should be limited to a short period of time due to hypertension being a chronic disease requiring long-term management.

Fourth, in this updated study, subgroup analysis was undertaken according to the setting where RBPM was mainly implemented. Accordingly, when the intervention was performed at a tertiary hospital, RBPM had a significant reduction in BP (6.33 mmHg, *p <* 0.001; I<sup>2</sup> = 73.401%, *p <* 0.001). The same numerical comparison was not compared in each group and, in the case of hospitals, its size was not analyzed separately; however, the results were statistically significant and included a sufficient number of studies to support the results; therefore, the significance of the results should not be ignored. The reason that RBPM had a higher BP lowering effect in tertiary medical institutions than in primary medical institutions may be due to the greater financial and human resource capacity in tertiary medical institutions [75].

Finally, this study observed the effect of RBPM with respect to the frequency of transmission of BP data. In the case of daily transmission, the WMD decreased by 5.881 mmHg. In contrast, in the case of monthly transmission, a decrease of 1.803 mmHg was observed. Some conflicting studies show that the higher the frequency of remote transmission, the lower the BP reduction effect [60,61]. However, in our study on cities, the longer the transmission interval, the lower the effect.

In previous meta-analyses, the number of studies conducted in urban settings was insufficient, and no study showed a change according to temporal progression. In this updated research, we included a comparison of the average effect over time, which was not covered in previous studies, and the effect according to the frequency of setting and data teletransmission. In particular, in our previous meta-analysis, it was reported that the effect of RBPM on patients with hypertension in metropolitan cities was not as large as that in small and medium cities. However, in this updated study, we found that the decrease in SBP and DBP was large in cities with a population of ≥1 million. Therefore, this study addressed the limitations of previous studies. Advancements in telecommunication technology have led to increased use of remote monitoring in healthcare [76]. In situations where physical distancing is emphasized, such as in the case of COVID-19, it is essential to assess the effects of RBPM in an urban setting [77]. To our knowledge, this study is the first meta-analysis to assess the effects of RBPM in urban-dwelling patients with hypertension from RBPM inception to the end of November 2020, including during the COVID-19 pandemic period, and these comprehensive results may provide a clinical basis for developing future healthcare policies.

In this study, a structured formula was applied, and a transparent process was followed to analyze RCTs with a high level of evidence. However, this study had some limitations. First, although the final studies were selected through a structured search using reliable databases, there may have been a language barrier. No outlier was found to have an influence on the summary effect size through the "one-study removed" sensitivity test method and a cumulative meta-analysis; however, selecting articles in different languages may have prevented adequate accounting for errors. Although most abstracts included in the search were written in English, the collected data may not have been sufficient. To overcome this limitation, multiple languages need to be set in the search with a wider scope

to include gray literature. Second, the number of small-sized articles was insufficient to test for publication errors. Egger's test for the results in this study was used to determine combined two-tailed *p*-value significance, and the number of articles with a nil result in terms of a 95% CI was as high as 2898, which increased reliability. Nevertheless, there remained the possibility of publication errors. This limitation could be addressed through including a larger number of small-sized articles. Third, as the studies included in this meta-analysis varied in terms of the period when they were conducted, the criteria for target BP reflected in the rate of BP control may also have varied. Thus, further studies should set a clear BP target for collecting and synthesizing the data to produce more accurate results. Fourth, the authors categorized time intervals to compare the average SBP according to time interval and to quantify the results, which involved dividing the studies according to time based on the year 2000, when internet use expanded globally, and making simple comparisons at 10-year intervals thereafter. However, distinctions between time intervals may have been unclear. Although it is not possible to clearly divide the development time of telemedicine technology, we consider that the timeframe could be set more precisely based on historical developments in mobile communication technology and telemedicine. Finally, we examined trends in the effect of RBPM over time through categorizing studies based on their publication dates to indicate the temporal association with COVID-19. However, since differences between the actual dates of research and publication dates are possible, a future study should clarify the dates during which studies were conducted or include more studies published after the onset of the COVID-19 pandemic to address this limitation.

#### **5. Conclusions**

Our study findings indicated that RBPM for urban-dwelling patients with hypertension was a practical and clinically effective means of reducing office BP. As the cumulative analysis shows, a consistent and clear effect was found in terms of reduction in office SBP following RBPM according to the temporal progress of the primary studies included in this study; an identical trend was found for 2020.

Based on the primary findings, the effects were classified according to intervention duration, city size, setting, frequency of remote monitoring of BP data, and urban MUAs, and it is anticipated that the implementation of specific policies in relation to these factors would more effectively guide the application of efficient and successful urban remote monitoring. Future studies should analyze more specific variables and include a greater number of studies to obtain more reliable results.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/article/10 .3390/ijerph182010583/s1, Table S1: PRISMA 2020 Checklist.

**Author Contributions:** Conceptualization, W.-S.C. and J.-H.S.; methodology, W.-S.C.; software, W.- S.C.; validation, W.-S.C., S.-H.P., J.P. and J.-H.S.; formal analysis, W.-S.C.; investigation, S.-H.P. and J.-H.S.; resources, W.-S.C.; data curation, J.P.; writing—original draft preparation, S.-H.P. and J.-H.S.; writing—review and editing, W.-S.C. and S.-H.P.; visualization, J.P. and W.-S.C.; supervision, W.-S.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A. Searching Strategy via Cochrane Library**



#### **Appendix B. Sensitivity Test Based on a "One-Study Removed" Approach**

**Figure A1.** Note: point estimate of individual study (•), summary effect size (◆); SBP, systolic blood pressure; UC, usual care; RBPM, remote blood pressure monitoring.


#### **Appendix C. Cumulative Meta-Analysis of RBPM According to the SMD of SBP**

**Figure A2.** Note: point estimate of individual study excluding each individual study (•), summary effect size (◆); SBP, systolic blood pressure; SMD, standardized mean difference; UC, usual care; RBPM, remote blood pressure monitoring.

**Appendix D. Meta-Regression of Risk Ratio According to RBPM Duration**

**Figure A3.** Note: point estimate of individual study (); RBPM, remote blood pressure monitoring.

#### **Appendix E. Subgroup Analysis**


**Table A1.** CI, confidence interval; FEM, fixed effects model; SBP, systolic blood pressure; WMD, weighted mean difference.


**Table A1.** *Cont.*

#### **References**

1. Kishore, S.P.; Heller, D.J.; Vasan, A. Beyond hypertension: Integrated cardiovascular care as a path to comprehensive primary care. *B. World Health Organ.* **2018**, *96*, 219–221. [CrossRef] [PubMed]

2. Lewington, S.; Clarke, R.; Qizilbash, N.; Peto, R.; Collins, R. Age-specific relevance of usual blood pressure to vascular mortality: A meta-analysis of individual data for one million adults in 61 prospective studies. *Lancet* **2002**, *360*, 1903–1913. [PubMed]

3. Chow, C.K.; Teo, K.K.; Rangarajan, S.; Islam, S.; Gupta, R.; Avezum, A.; Bahonar, A.; Chifamba, J.; Dagenais, G.; Diaz, R.; et al. Prevalence, awareness, treatment, and control of hypertension in rural and urban communities in high-, middle-, and low-income countries. *JAMA* **2013**, *310*, 959–968. [CrossRef]

4. Egan, B.M.; Li, J.; Hutchison, F.N.; Ferdinand, K.C. Hypertension in the United States, 1999 to 2012: Progress toward Healthy People 2020 goals. *Circulation* **2014**, *130*, 1692–1699. [CrossRef]


*Systematic Review*

## **Virtual Reality in the Treatment of Adults with Chronic Low Back Pain: A Systematic Review and Meta-Analysis of Randomized Clinical Trials**

**Beatriz Brea-Gómez, Irene Torres-Sánchez \*, Araceli Ortiz-Rubio, Andrés Calvache-Mateo, Irene Cabrera-Martos, Laura López-López and Marie Carmen Valenza**

> Physical Therapy Department, Faculty of Health Sciences, University of Granada, 18016 Granada, Spain; beatrizbreagomez@gmail.com (B.B.-G.); aortiz@ugr.es (A.O.-R.); andrescalvache@ugr.es (A.C.-M.); irenecm@ugr.es (I.C.-M.); lauralopez@ugr.es (L.L.-L.); cvalenza@ugr.es (M.C.V.) **\*** Correspondence: irenetorres@ugr.es

**Abstract:** Virtual reality (VR) can present advantages in the treatment of chronic low back pain. The objective of this systematic review and meta-analysis was to analyze the effectiveness of VR in chronic low back pain. This review was designed according to PRISMA and registered in PROSPERO (CRD42020222129). Four databases (PubMed, Cinahl, Scopus, Web of Science) were searched up to August 2021. Inclusion criteria were defined following PICOS recommendations. Methodological quality was assessed with the Downs and Black scale and the risk of bias with the Cochrane Risk of Bias Assessment Tool. Fourteen studies were included in the systematic review and eleven in the meta-analysis. Significant differences were found in favor of VR compared to no VR in pain intensity postintervention (11 trials; *n* = 569; SMD = −1.92; 95% CI = −2.73, −1.11; *p* < 0.00001) and followup (4 trials; *n* = 240; SDM = −6.34; 95% CI = −9.12, −3.56; *p* < 0.00001); and kinesiophobia postintervention (3 trials; *n* = 192; MD = −8.96; 95% CI = −17.52, −0.40; *p* = 0.04) and followup (2 trials; *n* = 149; MD = −12.04; 95% CI = −20.58, −3.49; *p* = 0.006). No significant differences were found in disability. In conclusion, VR can significantly reduce pain intensity and kinesiophobia in patients with chronic low back pain after the intervention and at followup. However, high heterogeneity exists and can influence the consistency of the results.

**Keywords:** chronic low back pain; virtual reality; videogames; horse simulator riding; rehabilitation; physical therapy

#### **1. Introduction**

Chronic low back pain (CLBP) is one of the main causes of pain, dysfunction, and disability [1,2]. It is one of the most common reasons for which patients require medical attention [3]. Furthermore, it is the world's leading cause of years of life lived with disability [4]. In most cases, it is not possible to identify the specific nociceptive cause of CLBP and therefore, it is classified as nonspecific (pain not caused by a specific pathology such as infection, tumor, fracture, or inflammation) [2]. CLBP affects the physical, psychological, and social areas and carries a great socioeconomic burden, as it is the main cause of work absenteeism and the excessive use of therapeutic services [5]. For all these reasons, it is essential to establish an effective treatment.

There are many ways to treat CLBP in the clinical environment, such as surgery, medication, or physical therapy. In addition to analgesic treatment with drugs, manual therapy, pain management, and early physical exercise (coordination, strengthening, and resistance exercises) have been recommended with a strong level of evidence, as they can be beneficial in reducing pain and achieve a functional improvement [5,6]. However, in many cases the main limitation of physical exercise is lack of motivation and adherence [7]. Virtual reality (VR) can present some advantages in the face of these problems, since it

**Citation:** Brea-Gómez, B.;

Torres-Sánchez, I.; Ortiz-Rubio, A.; Calvache-Mateo, A.; Cabrera-Martos, I.; López-López, L.; Valenza, M.C. Virtual Reality in the Treatment of Adults with Chronic Low Back Pain: A Systematic Review and Meta-Analysis of Randomized Clinical Trials. *Int. J. Environ. Res. Public Health* **2021**, *18*, 11806. https:// doi.org/10.3390/ijerph182211806

Academic Editor: Paul B. Tchounwou

Received: 21 October 2021 Accepted: 7 November 2021 Published: 11 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

contributes the motivational component and interactivity to the treatment [8]. The patient is involved in their recovery in a fun and attractive way and the interactive elements and feedback offered by the virtual environment can increase adherence to the exercises [9,10]. Negative thoughts and beliefs about pain experienced by some patients can lead to pain avoidance behaviors, causing inactivity, and preventing recovery and pain reduction [11]. VR treatment is a powerful pain distraction mechanism by focusing on an external stimulus and not on body movement, reducing attention to pain by dividing attention tasks [9,12]. Furthermore, compared to traditional methods, VR is considered a cost-effective and efficient tool [13].

In the current scientific literature, we found different reviews about VR in the treatment of pain in various areas. Gumaa et al. [14] explored VR effectiveness in orthopedic rehabilitation, showing inconclusive results in low back pain. In addition, they referred to the need for higher quality studies to establish more solid conclusions. In another review, VR in spinal pain was investigated [15]. Due to the low quality of the included studies, Ahern et al. [15] concluded that higher quality studies were necessary. A recent review published by Bordeleau et al. [16] concluded that while the specific set of studies showed high heterogeneity across several methodological factors, a tentative conclusion could be drawn that VR is effective improving back pain intensity and tends to have a positive effect on improving other pain outcomes and motion function. Bordeleau et al. [16] highlights that methodology framework for the development of VR treatments should be considered.

Since the completion of the search of the review of Bordeleau et al. [16], several new randomized clinical trials have been published on this topic, so there is new evidence to contribute to this issue. Additionally, a subgroup analysis of the different interventions is needed. Whether VR is applied alone or added to a physical therapy intervention could produce different results; furthermore, the comparison should also be taken into account.

Additionally, an analysis comparing the effects of the different VR interventions, the different durations of the interventions and the effects of VR at followup should be useful. It would also be of interest to explore other variables related to pain, in addition to pain intensity.

Therefore, the objective of this systematic review and meta-analysis of randomized clinical trials was to analyze the effectiveness of VR interventions in the treatment of CLBP. Implications and considerations may arise regarding the characteristics of the intervention programs.

#### **2. Materials and Methods**

#### *2.1. Design*

A systematic review was performed to identify randomized clinical trials exploring the effects of VR on the treatment of CLBP. The guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [17] was used to carry out this systematic review. This systematic review was previously registered at the International prospective register of systematic reviews (PROSPERO) with number CRD42020222129. Available from: https://www.crd.york.ac.uk/prospero/display\_record.php?RecordID=222129.

#### *2.2. Search Strategy*

Four databases were searched from their inception up to August 2021 without language restrictions. We used PubMed, Cinahl, Scopus, and Web of Science. The full search strategy is described in Appendix A. In order to find other relevant articles to the study, the reference list of other reviews and related articles were reviewed.

Additionally, a search was conducted for ongoing randomized clinical trials, which have not yet been published, to find out if they could be included in our review. The clinical trials registries ClinicalTrials.gov, the International Standard Randomized Controlled Trial Number (ISRCTN) Registry, and the International Clinical Trials Registry Platform (ICTRP) were used. Appendix B describes the search strategy used in each database.

#### *2.3. Study Selection*

The selection of studies was conducted systematically based on the prespecified PICOS (participants, interventions, comparisons, outcome, and study design) eligibility criteria: (1) Participants: adults (≥18 years) with CLBP (12 weeks or more) [18]; (2) Interventions: interventions based on VR; length of intervention of at least four weeks; (3) Comparisons: no intervention, interventions without VR, standard treatment, usual care, placebo or control; (4) Outcomes: pain intensity and other outcomes related to pain; (5) Study design: randomized clinical trials.

For the first screening title and abstract of each article was evaluated. We excluded those that did not meet the inclusion criteria defined with the PICOS strategy. After, the full text of relevant studies was assessed to check if they met the inclusion criteria. The list of excluded studies in the last screening and reason for exclusion is described in Appendix C.

When full text was not available, we contacted the corresponding author of the study via email. Two reviewers (BBG and ITS) independently carried out the search and selection of studies. If needed, disagreements were resolved with a third reviewer.

#### *2.4. Data Extraction*

The following data were recorded from the included articles: author, year of publication, country, disease, sample size, age (years), gender (percentage of males), outcome measures, main results (outcomes that showed significant differences (*p* ≤ 0.05)), measuring instrument, and time point assessment. This information is summarized in Table 1. In addition, the score obtained on the Downs and Black methodological quality scale [19] was added. Table 2 shows the characteristics of interventions: experimental group intervention, control group interventions, session duration, frequency, program duration, supervision, and adverse events.

When the information was insufficient or unclear, we contacted the corresponding author of the study via email. If the data were still unclear after contacting the corresponding author or if contact was not possible, it was analyzed using the available data. Two reviewers (BBG and ITS) independently carried out the data extraction. If needed, disagreements were resolved with a third reviewer.

#### *2.5. Methodological Quality of Included Studies*

Downs and Black quality assessment method [19] was used to assess the methodological quality of included studies in this review. This scale is one of the six best quality assessment scales [20–22]. This method contains 27 items divided into 5 sections: study quality (10 items), external validity (3 items), study bias (7 items), confounding and selection bias (6 items), and study power (1 item). In this review, we used the modified Downs and Black scale. The scoring for item 27 was simplified to a choice of 0 ("no"/"unable to determine") or 1 point ("yes"). These scores will be the same for the rest of the items except item 5 which is valued as 0 ("no"/"unable to determine"), 1 ("partially"), or 2 ("yes"). Therefore, the scores range from 0 to 28 and the higher ones indicate a better methodological quality of the study [22,23]. According to their quality, studies can be categorized as excellent (26–28), good (20–25), fair (15–19), and poor (≤14) [22–24].

#### *2.6. Risk of Bias of Included Studies*

The Cochrane Risk of Bias Assessment Tool [25] was used to assess the risk of bias of included studies. This tool assesses seven domains: random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective reporting, and other bias. For each study, the different domains were scored as "high risk of bias", "low risk of bias", or "unclear".

Two reviewers (BBG and ITS) carried out the assessment of risk of bias, as well as the assessment of methodological quality independently, and in case of doubt or disagreement a third reviewer was consulted.

#### *2.7. Statistical Analysis*

We used the Review Manager (RevMan) software version 5.4 to perform statistical analysis and used forest plots to display the results. Analysis was performed for those outcomes repeated at least in three comparisons or studies. Regarding the period of time, the analysis was carried out after the intervention and at 6 months followup. Mean, standard deviation (SD) and sample size were extracted from included studies to estimate the overall effect. For continuous outcomes, the mean difference (MD) and the 95% confidence intervals (CI) were used when the outcomes were evaluated with the same scale and the standardized mean difference (SMD) when the scales were different. The method utilized was inverse variance. The fixed effects model was used and the random effects model was applied when heterogeneity was greater than 75%. A value of *p* ≤ 0.05 was considered statistically significant. Heterogeneity between studies was assessed using the I<sup>2</sup> test. The degree of heterogeneity was categorized as low (I2 < 25%), moderate (I <sup>2</sup> = 25–75%), and high (I<sup>2</sup> > 75%). In order to explore possible causes of heterogeneity among study results we conducted a subgroup analysis. Subgroups were performed according to the comparisons (VR vs. no intervention, VR vs. placebo, VR vs. oral treatment, VR vs. physiotherapy, VR + physiotherapy vs. physiotherapy, and VR + physiotherapy vs. no VR exercise + physiotherapy); the type of intervention with VR (Nintendo consoles, Horse Simulator Riding, and Prokin System), and the duration of the intervention (4, 8, or 12 weeks).

#### **3. Results**

#### *3.1. Search Selection*

After the initial search in the databases and reference lists, we found 1363 manuscripts. After removing duplicates, we obtained 838 potentially eligible records. After screening by title and abstract, 58 articles remained, of which the full text was assessed. Of those 58 studies, 14 randomized clinical trials met the inclusion criteria, and finally 11 were included in quantitative synthesis.

In addition, we searched for ongoing randomized clinical trials. Of the 63 studies found in the three clinical trial registries consulted, 17 finally met the inclusion criteria. Figure 1 shows the flow diagram of the articles during the study selection process in the databases and clinical trial registries. The list of ongoing randomized clinical trials that could be included in the review is shown in Appendix D. None of the ongoing randomized clinical trials were included in this review.

#### *3.2. Characteristics of Included Studies*

Table 1 shows the characteristics of included studies in this review. All studies were randomized clinical trials and are arranged chronologically from oldest to newest. The included studies were published between 2013 and 2021.

Six studies were conducted in South Korea [26–31], one in Brazil [32], one in the USA [33], one in Australia [7], three in Saudi Arabia [34–36], one in Turkey [37], and one in Japan [38]. The total number of participants was 765. The mean age of the participants was 40.04 with 62.08% men. All studies measured pain intensity. It was measured using VAS in nine studies [26–29,31,34–36,38], 11-point Numeric Pain Rating Scale (11-NPRS) in four studies [7,30,32,37], and The Defense and Veterans Pain Rating Scale (DVPRS) in one study [33]. Four studies measured disability associated with low back pain using the Oswestry Disability Index (ODI) [29–31,37], four studies measured kinesiophobia using the 17-item Tampa Scale of Kinesiophobia (17-TSK) [7,35,36,38], and four studies measured body composition using bioelectrical impedance analysis method [27,28,31,38]. Other variables assessed more frequently were severity of disability with Roland Morris Disability Questionnaire (RMDQ) in two studies [7,30], isokinetic trunk flexion/extension with a dynamometer in three studies [27,28,31], pain self-efficacy with the 10-item Pain Self-Efficacy (10-PSEQ) in two studies [7,38], pain catastrophizing with Pain Catastrophizing Scale (PCS) in two studies [33,38], and blood serum levels of stress hormones in two studies [35,36]. Variables were assessed before and after the intervention in all articles. Five

studies included followup, one at 8 weeks and 6 months [34], other at 3 and 6 months [7], and three only at 6 months [30,35,36]. In addition, one of these studies included a midterm assessment after 4 weeks [30], and one study assessed outcomes during intervention [33].

**Figure 1.** PRISMA flow diagram: database and clinical trials register search and other sources.

#### *3.3. Characteristics of Interventions*

Characteristics of the interventions of the included studies are described in Table 2. Regarding the interventions, three studies compared VR with no intervention [7,27,28], two studies with a placebo [31,33], and other study with oral treatment (Nonsteroidal Anti-Inflammatory Drugs (NSAIDs), tramadol, and duloxetine) [38]. In two studies, comparisons consisted of VR versus physiotherapy [30,37]. In addition, three studies combined VR + physiotherapy versus physiotherapy alone [26,29,32], and four studies combined those interventions and compared them with no VR exercises and physiotherapy [26,34–36].

Four of the fourteen studies used Nintendo programs and consoles [7,26,32,38], whereas three studies used other types of video games with sensors and a monitor (Prokin System) [34–36]. Five studies used a horse simulator riding [27–31]. One study used a system similar to VR, but without video games, based on biofeedback [37], and in one study, the intervention was a behavioral skills-based VR program with VR glasses [33].


**Table 1.** Characteristics of included studies.



**Table 2.**

Characteristics

 of

interventions.





The mean time using VR was 28.29 min and the mean session duration was 46.21 min. Regarding the frequency of the sessions, it varied from one weekly session [38] to seven sessions per week [33]. The duration of the program in the different studies ranged from 4 [29,34–36] to 12 weeks [31,37]. In nine studies, the interventions were supervised [27,28,30–32,34–37]. In one article, participants were contacted by phone calls [7], and one did not include any type of supervision [33]. Three studies did not report on supervision of the intervention [26,29,38]. Of all the articles, only two reported adverse events derived from the intervention with VR (e.g., nausea, motion sickness, vertigo, etc.) [32,33]; in two articles no adverse events were reported [7,30], and in the rest no information was provided.

#### *3.4. Methodological Quality of Included Studies*

Downs and Black quality assessment method [19] was used to assess the methodological quality of included studies in this review. The total score for each study is shown in Table 1, and the score for each item is summarized in Appendix E. According to their score, of the 14 articles evaluated, two were classified as excellent (26–28), seven as good (20–25), four as fair (19–15), and one as poor (≤14). The mean score of the included studies was 20.79 (range: 13–27).

#### *3.5. Risk of Bias of Included Studies*

The Cochrane Risk of Bias Assessment Tool [25] was used to assess the risk of bias of the articles included in this review. Figures 2 and 3 show the summary and the graph of the risk of bias assessment, respectively. Random sequence generation, allocation concealment, incomplete outcome data, and selective reporting did not obtain a high risk of bias in any study. In addition, other bias obtained unclear risk of bias in all of the included studies. Blinding of participants and personnel and blinding of outcome assessment was evaluated as a high risk of bias in four [7,30,32,38] and two [33,38] studies, respectively. Two studies obtained unclear risk of bias in all items [29,37] and other two studies obtained unclear risk of bias in all items, except in incomplete outcome data [26,28].

**Figure 2.** Risk of bias summary.

**Figure 3.** Risk of bias graph.

#### *3.6. Effects of Virtual Reality vs. No Virtual Reality in Chronic Low Back Pain*

For meta-analysis, we only considered the outcome pain intensity and outcomes related to pain.

Eleven studies were included in the meta-analysis. All of them were included for pain intensity postintervention; four for pain intensity at the 6 month followup; three for disability postintervention; three for kinesiophobia postintervention, and two (four comparisons) for kinesiophobia at the six months followup. Two articles were excluded from the meta-analysis because they did not express data in mean ± SD [33,37]. In addition, Yoo et al. [28] was excluded because the SD was 0, and it was not estimable by RevMan.

3.6.1. Subgroup Based on Intervention Comparisons: Virtual Reality Alone or Combined with Physiotherapy vs. Control Group Interventions

Firstly, a subgroup analysis of the different interventions was performed to know if VR applied alone or added to a physical therapy intervention could produce different results, and if it differed depending on the type of intervention of the control group. We analyzed pain intensity, disability, and kinesiophobia postintervention; and pain intensity and kinesiophobia at the 6 months followup. The Visual Analog Scale (VAS) to evaluate pain intensity was adjusted to a scale of 0–10 cm when it was expressed in millimeters.

In Figure 4a, the results show significant differences (SMD = −1.92; 95% CI = −2.73, −1.11; *p* < 0.00001) in favor of VR compared to no VR in pain intensity postintervention. When VR was compared with no intervention (SMD = −1.84; 95% CI = −3.48, −0.21; *p* = 0.03), placebo (SMD = −2.71; 95% CI = −3.33, −2.10; *p* < 0.00001), or oral treatment (SMD = −0.78; 95% CI = −1.42, −0.13; *p* = 0.02), the subgroup analysis showed significant differences in favor of VR. In addition, when VR + physiotherapy were compared with no VR exercise + physiotherapy, the subgroup analysis showed significant differences (SMD = −3.26; 95% CI = −5.08. −1.44; *p* = 0.0004) in favor of VR too. However, no significant differences were observed between VR and physiotherapy (SMD = −0.28; 95% CI = −0.85, 0.28; *p* = 0.33) or VR + physiotherapy and physiotherapy (SMD = 0.08; 95% CI = −0.42, 0.59; *<sup>p</sup>* = 0.75). Heterogeneity was high in overall effect (I<sup>2</sup> = 93%; *p* < 0.00001) and in two subgroups, VR versus no intervention (I2 = 90%; *p* < 0.00001) and VR + physiotherapy versus no VR exercise + physiotherapy (I2 = 95%; *p* < 0.00001). According to the I<sup>2</sup> statistic, 0% of variation across studies was due to heterogeneity (*p* = 0.98) in VR + physiotherapy versus the physiotherapy subgroup.

In Figure 4b, the results show significant differences (SDM = −6.34; 95% CI = −9.12, –3.56; *p* < 0.00001) in pain intensity at the six month followup in favor of VR compared to no VR. When VR was compared with physiotherapy, the subgroup analysis showed no significant differences (SDM = 0.17; 95% CI = −0.54, 0.87; *p* = 0.64). However, when VR + physiotherapy were compared with no VR exercise + physiotherapy, the subgroup analysis showed significant differences in favor of VR (SDM = −7.56; 95% CI = −10.79, –4.32; *p* < 0.00001). Heterogeneity was high in overall effect (I2 = 97%; *p* < 0.00001) and in VR + physiotherapy versus no VR exercise + physiotherapy subgroup (I<sup>2</sup> = 96%; *p* < 0.00001).

As shown in Figure 5, no significant differences were found between VR interventions and other interventions without VR (MD = 10.46; 95% CI = −30.02, 9.09; *p* = 0.29) in disability postintervention. Subgroup analysis did not show significant differences between VR and physiotherapy (MD = −3.26; 95% CI = −8.44, 1.92; *p* = 0.22) or between VR + physiotherapy and physiotherapy (MD = −0.10; 95% CI = −3.47, 3.27; *p* = 0.95). However, when VR was compared with the placebo, the subgroup analysis showed significant differences in favor of VR (MD = −27.89; 95% CI = −30.77, –25.01; *p* < 0.00001). Heterogeneity between studies was high (I<sup>2</sup> = 99%; *p* < 0.00001).


**Figure 4.** *Cont*.

**Figure 4.** Effect of virtual reality versus no virtual reality in chronic low back pain for pain intensity postintervention (**a**) and at the six month followup (**b**) based on the type of intervention. CBT: conventional balance training; IKT: isokinetic training; CPR: combined physical rehabilitation; VR: virtual reality.

(**b**)

**Figure 5.** Effect of virtual reality versus no virtual reality in chronic low back pain for disability postintervention based on the type of intervention. VR: virtual reality.

> As shown in Figure 6a, the results showed significant differences (MD = −8.96; 95% CI = −17.52, –0.40; *p* = 0.04) in favor of VR in total comparison in kinesiophobia postintervention. When VR was compared with oral treatment, the subgroup analysis showed significant differences in favor of oral treatment (MD = 3.47; 95% CI = 1.00, 5.94; *p* = 0.006). However, when VR + physiotherapy were compared with no VR exercises + physiotherapy, the subgroup analysis showed significant differences in favor of VR (MD = −12.05; 95% CI = −20.13, –3.98; *<sup>p</sup>* = 0.003). Heterogeneity was high in overall effect (I2 = 99%; *p* < 0.00001) and in VR + physiotherapy versus no VR exercise + physiotherapy subgroup (I<sup>2</sup> = 98%; *p* < 0.00001).

(**b**)

**Figure 6.** Effect of virtual reality versus no virtual reality in chronic low back pain for kinesiophobia postintervention (**a**) and at the six month followup (**b**) based on the type of intervention. IKT: isokinetic training; CBT: conventional balance training; CPR: combined physical rehabilitation; VR: virtual reality.

> All studies in this meta-analysis (Figure 6b) compared VR + physiotherapy versus no VR exercise + physiotherapy. The results showed significant differences (MD = −12.04; 95% CI = −20.58, –3.49; *p* = 0.006) in favor of VR in kinesiophobia at the 6 month followup. Heterogeneity between studies was high (I<sup>2</sup> = 99% *p* < 0.00001).

#### 3.6.2. Subgroups Based on Virtual Reality Interventions

Other subgroup analysis was based on the type of VR intervention. The studies were divided into three subgroups: Nintendo consoles, Horse Simulator Riding, or Prokin System. We analyzed pain intensity, disability, and kinesiophobia postintervention and pain intensity and kinesiophobia at the 6 months followup.

As shown in Figure 7a, the results showed significant differences (SMD = −1.92; 95% CI = −2.73, −1.11; *p* < 0.00001) in favor of VR versus no VR in pain intensity postintervention. When Nintendo consoles were compared with interventions without VR, the subgroup analysis showed no significant differences (SMD = −0.07; 95% CI = −0.57, 0.43; *p* = 0.78). However, when horse simulator riding (SMD = −1.68; 95% CI = −2.95, –0.41; *p* = 0.009) or Prokin System (SMD = −3.96; 95% CI = −5.71, –2.21; *p* < 0.00001) were compared with interventions without VR, the subgroup analysis showed significant differences in favor of VR. Heterogeneity was high in overall effect (I<sup>2</sup> = 93%; *p* < 0.00001) and in two subgroups, horse simulator riding (I<sup>2</sup> = 92%; *p* < 0.00001) and Prokin System (I <sup>2</sup> = 93%; *p* < 0.00001). According to the I2 statistic, 54% of variation across studies was due to heterogeneity (*p* = 0.07) in the Nintendo consoles subgroup.


(**b**)

**Figure 7.** Effect of virtual reality versus no virtual reality in chronic low back pain for pain intensity postintervention (**a**) and at the six months followup (**b**) based on the type of virtual reality intervention. CBT: conventional balance training; IKT: isokinetic training; CPR: combined physical rehabilitation; VR: virtual reality.

As shown in Figure 7b the results showed significant differences (SDM = −6.34; 95% CI = −9.12, –3.56; *p* < 0.00001) in favor of VR in total comparison in pain intensity at the 6 month followup. Regarding subgroup analysis, no significant differences were found between horse simulator riding and no VR interventions (SDM = 0.17; 95% CI = −0.54, 0.87; *p* = 0.64). However, significant differences in favor of VR were found in the Prokin System subgroup (SDM = −7.56; 95% CI = −10.79, –4.32; *p* < 0.00001). Heterogeneity was high in overall effect (I<sup>2</sup> = 97%; *p* < 0.00001) and in Prokin System versus no VR interventions subgroup (I<sup>2</sup> = 96%; *p* < 0.00001).

All studies in Figure 8 compared horse simulator riding interventions versus other interventions without VR. No significant differences were found between VR and no VR (MD = −10.46; 95% CI = −30.02, 9.09; *p* = 0.29) in disability post-intervention. Heterogeneity between studies was high (I<sup>2</sup> = 99%; *p* < 0.00001).

**Figure 8.** Effect of virtual reality versus no virtual reality in chronic low back pain for disability postintervention based on the type of virtual reality intervention. VR: virtual reality.

> As shown in Figure 9a, the results showed significant differences (MD = −8.96; 95% CI = −17.52, −0.40; *p* = 0.04) in favor of VR in total comparison in kinesiophobia postintervention. The results showed significant differences in favor of interventions without VR versus interventions with Nintendo consoles (MD = 3.47; 95% CI = 1.00, 5.94; *p* = 0.006). However, when the Prokin System was compared with interventions without VR significant differences were found in favor of the Prokin System subgroup (MD = −12.05; 95% CI = −20.13, −3.98; *<sup>p</sup>* = 0.003). Heterogeneity was high in overall effect (I2 = 99%; *p* < 0.00001) and in Prokin System versus interventions without VR (I<sup>2</sup> = 98%; *p* < 0.00001).

> All studies in Figure 9b compared Prokin System versus interventions without VR. The results showed significant differences (MD = −12.04; 95% CI = −20.58, −3.49; *p* = 0.006) in favor of VR in kinesiophobia at the 6 month followup. Heterogeneity between studies was high (I<sup>2</sup> = 99%; *p* < 0.00001).


(**a**)

**Figure 9.** *Cont*.


(**b**)

**Figure 9.** Effect of virtual reality versus no virtual reality in chronic low back pain for kinesiophobia postintervention (**a**) and at the six month followup (**b**) based on the type of virtual reality intervention. IKT: isokinetic training; CBT: conventional balance training; CPR: combined physical rehabilitation; VR: virtual reality.

3.6.3. Subgroups Based on the Duration of the Intervention

The last subgroup analysis was based on the duration of the intervention. The studies were divided into three subgroups: four weeks, eight weeks, or twelve weeks of intervention. We analyzed pain intensity, disability, and kinesiophobia postintervention and pain intensity and kinesiophobia at the 6 month followup.

As shown in Figure 10a, the results showed significant differences (SMD = −1.92; 95% CI = −2.73, −1.11; *p* < 0.00001) in favor of VR versus no VR in pain intensity postintervention. Subgroup analysis showed significant differences in favor of VR after 4 weeks of intervention (SMD = −3.38; 95% CI = −5.06, −1.70; *p* < 0.0001), 8 weeks of intervention (SMD = −0.65; 95% CI = −1.29, −0.00; *p* = 0.05), and 12 weeks of intervention (SMD = −2.71; 95% CI = −3.33, −2.10; *<sup>p</sup>* < 0.00001). Heterogeneity was high in overall effect (I2 = 93%; *p* < 0.00001) and in all subgroups (I2 = 94%; *p* < 0.00001) (I<sup>2</sup> = 81%; *p* < 0.00001).

As shown in Figure 10b, the results showed significant differences (SDM = −6.34; 95% CI = −9.12, −3.56; *p* < 0.00001) in favor of VR in total comparison in pain intensity at the 6 month followup. Regarding subgroup analysis, no significant differences were found between VR versus no VR after 8 weeks of intervention (SDM = 0.17; 95% CI = −0.54, 0.87; *p* = 0.64). However, significant differences in favor of VR were found after 4 weeks of intervention (SDM = −7.56; 95% CI = −10.79, −4.32; *p* < 0.00001). Heterogeneity was high in overall effect (I2 = 97%; *p* < 0.00001) and in the 4 weeks of intervention subgroup (I<sup>2</sup> = 96%; *p* < 0.00001).

No significant differences were found between VR interventions and other interventions without VR (MD = −10.46; 95% CI = −30.02, 9.09; *p* = 0.29) in disability postintervention. Subgroup analysis did not show significant differences between VR and no VR after 4 weeks (MD = −0.10; 95% CI = −3.47, 3.27; *p* = 0.95) or 8 weeks of intervention (MD = −3.26; 95% CI = −8.44, 1.92; *p* = 0.22). However, significant differences were found in favor of VR after 12 weeks of intervention (MD = −27.89; 95% CI = −30.77, –25.01; *p* < 0.00001). Heterogeneity between studies was high (I<sup>2</sup> = 99%; *p* < 0.00001). Figure 11 shows these results.

As shown in Figure 12a, the results showed significant differences (MD = −8.96; 95% CI = −17.52, −0.40; *p* = 0.04) in favor of VR in total comparison in kinesiophobia postintervention. After 8 weeks of intervention, the results showed significant differences in favor of no VR intervention (MD = 3.47; 95% CI = 1.00, 5.94; *p* = 0.006). However, significant differences in favor of VR were observed after 4 weeks of intervention (MD = −12.05; 95% CI = −20.13, −3.98; *<sup>p</sup>* = 0.003). Heterogeneity was high in overall effect (I2 = 99%; *p* < 0.00001) and in the 4 weeks of intervention subgroup (I2 = 98%; *p* < 0.00001).

**Figure 10.** Effect of virtual reality versus no virtual reality in chronic low back pain for pain intensity postintervention (**a**) and at the six month followup (**b**) based on the duration of the intervention. CBT: conventional balance training; IKT: isokinetic training; CPR: combined physical rehabilitation; VR: virtual reality.


**Figure 11.** Effect of virtual reality versus no virtual reality in chronic low back pain for disability postintervention based on the duration of the intervention. VR: virtual reality.

(**a**)

#### (**b**)

**Figure 12.** Effect of virtual reality versus no virtual reality in chronic low back pain for kinesiophobia postintervention (**a**) and at the six month followup (**b**) based on the duration of the intervention. IKT: isokinetic training; CBT: conventional balance training; CPR: combined physical rehabilitation; VR: virtual reality.

All studies shown in Figure 12b conducted a 4-week intervention. The results showed significant differences (MD = −12.04; 95% CI = −20.58, –3.49; *p* = 0.006) in favor of VR in kinesiophobia at the 6 month followup. Heterogeneity between studies was high (I<sup>2</sup> = 99%; *p* < 0.00001).

#### **4. Discussion**

The objective of this systematic review and meta-analysis was to analyze the effectiveness of VR interventions in the treatment of CLBP. Fourteen studies were included in this review and eleven of them in the meta-analysis. The results showed significant differences in favor of VR interventions in pain intensity and kinesiophobia postintervention and at the six month followup. However, no significant differences were found in disability postintervention.

#### *4.1. Pain Intensity*

Pain intensity was assessed in all of the studies included in meta-analysis. The meta-analysis showed significant differences in favor of interventions with VR versus interventions without VR in pain intensity postintervention and at the six month followup. On the one hand, the effect of VR was superior to no intervention [7,27], placebo [31], and oral treatment [38] in pain intensity postintervention, but it should be noted that there was only a study in two of these subgroups. Significant differences in favor of VR + physiotherapy were observed when we compared with no VR exercise + physiotherapy [26,34–36] in pain intensity postintervention and at the six month follow-up. Most of the studies included in this subgroup had good methodological quality and obtained significant differences in favor of VR in the rest of the variables not included in the meta-analysis. However, it must be taken into account that these results have been obtained from studies that only included young athletic men and cannot be generalized [34–36]. On the other hand, VR was not superior to physiotherapy in pain intensity postintervention or at the six month followup. Nevertheless, there was only one article (with young adults and a high dropout rate) in this subgroup [30]. Neither were significant differences found between VR + physiotherapy versus physiotherapy in pain intensity postintervention. It should be noted that these studies had a small sample size and some of them had low methodological quality [26,29,32].

Regarding the type of VR, horse simulator riding and Prokin System were superior to interventions without VR in pain intensity postintervention. However, in the horse simulator riding subgroup, most of the studies compared this type of VR with no intervention [27] or placebo [31], which can explain the good results in the analysis. Nintendo consoles did not show significant differences. This may be because the Prokin System and horse simulator riding are specialized VR devices compared to Nintendo consoles. At the six month followup, the results showed significant differences in favor of the Prokin System but not in favor of horse simulator riding. It must be taken into account that there was only one study (with young adults and a high dropout rate) in this subgroup [30]. The results showed significant differences between VR and no VR in pain intensity postintervention after 4 weeks, 8 weeks, or 12 weeks of intervention. At the six month followup, significant differences in favor of VR were found after 4 weeks of intervention but not after 8 weeks of intervention. It should be noted that there was only one study in this subgroup [30].

#### *4.2. Disability*

No significant differences were found between VR interventions (horse simulator riding) and no VR interventions in disability postintervention. However, when VR was compared with placebo and when the intervention lasted 12 weeks, the different subgroups analysis showed significant differences in favor of VR. This can be explained because the same article was included in the subgroups [31]. No significant differences were observed between VR and physiotherapy [30] or VR + physiotherapy and physiotherapy [29], or after four [29] or eight weeks of intervention [30]. It should be noted that there was only one study in each subgroup. In addition, these studies had a small sample size, poor methodological quality [29], and some limitations, such as a high dropout rate [30].

#### *4.3. Kinesiophobia*

The results showed significant differences in favor of VR when compared with no VR in kinesiophobia postintervention and at the six month followup. When VR + physiotherapy were compared with no VR exercises + physiotherapy, the subgroup analysis showed significant differences in favor of VR in kinesiophobia postintervention and at the six month followup. These studies used Prokin System as the VR intervention, so significant differences in favor of Prokin System also were observed in this subgroup. The intervention lasted 4 weeks in all these articles, thus, the same results were found in subgroups based on duration of intervention. Although these articles had a good methodological quality, they only included young athletic men and their results cannot be generalized [35,36]. The other study that assessed kinesiophobia postintervention, Sato et al. [38] compared Nintendo Switch with oral treatment and the duration of the intervention was 8 weeks. In all of the different subgroups realized significant differences were found in favor of oral treatment. It must be taken into account that there was only this study in each subgroup (Nintendo and 8 weeks).

#### *4.4. Virtual Reality in Other Populations*

Other studies explored the effects of VR in different populations (such as, patients with chronic neck pain, fibromyalgia, acute pain, Parkinson's disease, stroke, etc.) and the results differ in part from ours.

In similar chronic pathologies, for example chronic neck pain, VR showed significant differences compared with no VR in pain intensity postintervention, which is in line with our results [39,40]. In this case, interventions consisted of VR compared with physiotherapy and in our review this type of comparison did not obtain significant results. No significant differences were found between VR + physiotherapy versus physiotherapy in pain intensity postintervention [41], which coincides with our review. However, in patients with fibromyalgia, VR combined with exercises showed significant improvement compared to exercises [42,43], although, results in pain intensity are not always conclusive [42]. In addition, in chronic neck pain, significant differences were observed in favor of VR in disability postintervention [39,40] which differs with the results found in our studies. These differences can be explained; the VR interventions in chronic neck pain articles were immersive, and the articles had better methodological quality.

In acute pain pathologies, VR has proven to be an adjuvant tool that can reduce procedural pain [44], burn pain, and anxiety [45]. In addition, it can reduce the use of medication [45]. As we have mentioned previously, in our review the studies that made a comparison between VR + another treatment versus same treatment did not obtain significant differences. In this case, this may be due to differences in the duration of pain and its origin.

Regarding neurological pathologies, such as Parkinson's disease, VR rehabilitation showed better results in overall improvement than conventional rehabilitation [46]. However, in another study, VR combined with exercises was statistically as effective as each intervention alone [47]. In any case, these results do not agree with ours, since no significant differences were found for these comparisons in CLBP. In stroke patients, VR combined with conventional physical therapy obtained significantly higher improvements than conventional physical therapy [48–50]. These results are also not in line with the current review.

#### *4.5. Discussion with Other Reviews*

The results obtained in our meta-analysis differ partially from those found in other reviews. Bordeleau et al. [16] found significant differences in favor of VR versus no VR interventions for pain intensity postintervention, which is in line with our results. Nevertheless, there are differences in subgroup analysis. When we compared VR with no intervention, the subgroup analysis showed significant differences in favor of VR but in Bordeleau et al. [16] significant differences were not found. The differences found between the meta-analysis may be due to the different articles included in each one and how they was carried out. In addition, they included studies with back pain, whereas we only included studies limited to CLBP patients. In Gumaa et al. [14] the results of the meta-analysis did not show significant differences between VR interventions compared to no intervention in pain intensity postintervention. It should be noted that in one of the studies there was an intervention, since there was electrotherapy [26], and another had a short intervention [51] compared to the others, so we did not consider it comparable. Our results showed significant differences in favor of VR versus no intervention. This can be explained by the greater number of articles included in our meta-analysis and by the different comparisons realized. However, most of the studies included in this meta-analysis had a small sample size, fair methodological quality, and unclear risk of bias.

Bordeleau et al. [16] did not observe significant differences between horse simulator riding and interventions without VR, whereas in our meta-analysis significant differences in favor of horse simulator riding were obtained in pain intensity postintervention. In addition, this is consistent with the results found in two reviews. Collado-Mateo et al. [52] concluded that horse-riding simulators are a promising tool to reduce pain intensity in low back patients, but the interpretation of the results must be performed with extreme caution due to the large heterogeneity, the low number of studies, and the potential risk of bias. Ren et al. [53] also found significant differences in favor of horse simulator riding compared with control in pain intensity postintervention and severity of disability in people with CLBP. However, Ren el al. included another type of VR in addition to horse simulator riding and patients with subacute low back pain.

In Bordeleau et al. [16] the results showed that the potential beneficial effect of VR was more important when more than 12 sessions were performed. In our review, the interventions of included articles lasted 4, 8, or 12 weeks. In all of these cases significant differences in favor of VR were found in pain intensity postintervention, but it should be noted that the best results were obtained in the 12 weeks of intervention subgroup. However, only one study was included [31].

#### *4.6. Strengths and Limitations*

This review represents an update in the knowledge about the effects of VR treatment in CLBP, incorporating a meta-analysis of outcomes that could not be performed before.

The strengths of the current systematic review included following the PRISMA guidelines [17] for implementation and the use of the PICOS strategy to define the inclusion criteria. Another strength was the performance of meta-analysis. The assessment of methodological quality was carried out with the Downs and Black scale [19], one of the six best scales of methodological quality [21]. Additionally, the risk of bias was assessed with the Cochrane Risk of Bias Assessment Tool [25]. Furthermore, the review was previously registered in PROSPERO with registration number CRD42020222129.

However, although PRISMA guidelines were adhered to and the methodology was strictly followed, completely accounting for the limitations of the included studies was impossible. One of the main limitations was the high heterogeneity between included studies and the difficulties found in making comparable subgroups in order to draw solid conclusions. There were also differences in the age ranges and in the clinical profile of the participants. Regarding the characteristics of the patients, in four studies pain was defined as nonspecific [7,29,30,32], in another study pain was related to work [26]. Four studies [27,35,36,38] made reference to nonspecific pain, however, pain was not defined

as nonspecific in the inclusion criteria of these studies. Finally, in three studies pain may have been related to sports practice [34–36], as the participants were football players. These differences in the origin of pain must be taken into account, because they could influence the results. Furthermore, the sample size of the included studies was relatively small in some of the studies (19 to 179) and there are no data on long-term outcomes. Finally, three studies compared VR with no intervention and it was expected that results in favor of VR would be observed.

#### *4.7. Clinical Implications for Practice*

VR interventions could be integrated into clinical practice to reduce pain intensity and kinesiophobia in patients with CLBP, with good results in the short and midterm followup. However, its effects on midterm followup have only been analyzed in a specific population of young sportsmen and cannot be generalized to the general population. Evidence for the efficacy of VR in disability associated with low back pain remains limited. Of the different types of VR, the Prokin System and horse simulator riding have obtained the best short-term results. However, only the studies using the Prokin System showed significant differences at midterm followup (6 months). In addition, this type of VR was combined with physiotherapy. Regarding the duration of the program, an intervention of 12 weeks showed the best results. However, interventions of 4 or 8 weeks also obtained significant results in favor of VR.

#### *4.8. Future Research*

None of the included studies assessed the variables at long-term followup so future research needs to focus on long-term effects. It may be interesting to conduct more studies comparing VR and physiotherapy versus physiotherapy due to the results obtained in other populations and the low quality of the studies included in this review. Prokin System and horse simulator riding showed good results in the treatment of CLBP. However, these devices are sophisticated and specialized and can be difficult to obtain for a clinic. Therefore, more studies would be necessary to explore the effects of Nintendo consoles in the treatment of CLBP. Although its results are inconclusive, it is commercially available and easier to implement in clinical practice. In addition, there is the possibility of it being used at home. Most of the studies included in this review have been conducted in adult patients under 30 years of age, and our best results were obtained in studies that only included young sportsmen. So, studies of similar quality in other types of populations are needed.

#### **5. Conclusions**

The results suggest that VR interventions can significantly reduce pain intensity and kinesiophobia in patients with CLBP after the intervention and at the 6 month followup. However, these studies showed high heterogeneity among them, influencing the consistency of the results. VR treatment showed the best results when it was compared with no intervention, placebo, or oral treatment in pain intensity postintervention. VR combined with physiotherapy versus no VR exercise and physiotherapy obtained significant differences in pain intensity and kinesiophobia postintervention and at the six month followup. Regarding VR systems, the Prokin System and horse simulator riding were the most effective short-term. Evidence of Nintendo consoles is still inconclusive, but they present some advantages, so more research is necessary. In terms of the duration of the program, 4, 8, or 12 week interventions showed good results. Studies are needed to evaluate the long-term effects of these interventions.

**Author Contributions:** Conceptualization, idea, and research design: B.B.-G., I.T.-S., A.O.-R., A.C.-M., L.L.-L., I.C.-M., M.C.V.; data collection: B.B.-G., I.T.-S.; data analysis: B.B.-G., I.T.-S.; writing—original draft preparation: B.B.-G., I.T.-S., A.O.-R., A.C.-M., L.L.-L., I.C.-M., M.C.V.; writing—review and editing: B.B.-G., I.T.-S., A.O.-R., A.C.-M., L.L.-L., I.C.-M., M.C.V. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the Spanish Ministry of Education by a FPU ("Formación Profesorado Universitario") grant for the authors Andrés Calvache-Mateo and Laura López-López (grant number, FPU:19/02609, FPU:17/00408).

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data presented in this study are available in selected articles in the reference list.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**



#### **Appendix B**



#### **Appendix C**

**Table A3.** Excluded Studies in the Last Screening with Reasons for Exclusion (*n* = 44).


#### **Table A3.** *Cont*.


#### **Appendix D**

**Table A4.** Characteristics of Included Registry Entries or Ongoing Trials (*n* = 17).


#### **Appendix E**


**Table A5.** Methodological Quality of Included Studies.

#### **References**


### *Review* **Methods Used to Evaluate mHealth Applications for Cardiovascular Disease: A Quasi-Systematic Scoping Review**

**Felix Holl 1,2,\*, Jennifer Kircher 1, Walter J. Swoboda <sup>1</sup> and Johannes Schobel <sup>1</sup>**


**Abstract:** In the face of demographic change and constantly increasing health care costs, health care system decision-makers face ever greater challenges. Mobile health applications (mHealth apps) have the potential to combat this trend. However, in order to integrate mHealth apps into care structures, an evaluation of such apps is needed. In this paper, we focus on the criteria and methods of evaluating mHealth apps for cardiovascular disease and the implications for developing a widely applicable evaluation framework for mHealth interventions. Our aim is to derive substantiated patterns and starting points for future research by conducting a quasi-systematic scoping review of relevant peerreviewed literature published in English or German between 2000 and 2021. We screened 4066 articles and identified *n* = 38 studies that met our inclusion criteria. The results of the data derived from these studies show that usability, motivation, and user experience were evaluated primarily using standardized questionnaires. Usage protocols and clinical outcomes were assessed primarily via laboratory diagnostics and quality-of-life questionnaires, and cost effectiveness was tested primarily based on economic measures. Based on these findings, we propose important considerations and elements for the development of a common evaluation framework for professional mHealth apps, including study designs, data collection tools, and perspectives.

**Keywords:** mobile health; cardiovascular diseases; evaluation methods

### **1. Introduction**

In 2019, over 331,000 deaths in Germany were attributed to cardiovascular disease (CVD) [1], the treatment of which generates higher medical costs to the German healthcare system than any other single illness, estimated at € 46.4 billion in 2015 [2]. Similarly, in the US, CVD is among the most expensive and most frequent causes of death among the population [3]. Kvedar et al. [4] pointed out the urgent need to develop, optimize, and evaluate programs and technologies that ensure more effective care for patients, where mobile health (mHealth) concepts are likely to play a significant role [5]. The World Health Organization defines mHealth as "Medical and public health practice supported by mobile devices, such as mobile phones, patient monitoring devices, personal digital assistants (PDAs), and other wireless devices" [6].

The 2019 German Digital Healthcare Act (DVG) permitted mobile health applications (mHealth apps)that meet specific requirements to be included the list of reimbursable digital health applications (DiGA list) [7]. Germany is one of the first countries to introduce a standardized mechanism for reimbursing digital health services and its healthcare and medical insurance policy-makers are still working through several challenges. For example, the DiGA list only includes mHealth apps classified as medical devices as defined in the Medical Devices Act administered by the German Federal Institute for Drugs and Medical Devices (BfArM) [8]. While other professional mHealth apps, such as medication reminders

**Citation:** Holl, F.; Kircher, J.; Swoboda, W.J.; Schobel, J. Methods Used to Evaluate mHealth Applications for Cardiovascular Disease: A Quasi-Systematic Scoping Review. *Int. J. Environ. Res. Public Health* **2021**, *18*, 12315. https:// doi.org/10.3390/ijerph182312315

Academic Editors: Irene Torres-Sanchez and Marie Carmen Valenza

Received: 20 October 2021 Accepted: 20 November 2021 Published: 23 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

or prevention apps, demonstrate both medical benefit and positive care effects, they remain ineligible for reimbursement.

Beyond narrowly defined medical devices, the data and treatment results provided by other professional mHealth apps require equally stringent assessment to ensure reliably high-quality care. Notably, there is currently no established and broadly applicable framework for evaluating mHealth interventions [9].

As a step toward filling this gap, this study examines the criteria and methods for evaluating mHealth interventions for cardiovascular disease discussed in the published literature as a basis for developing a more broadly applicable framework.

#### **2. Materials and Methods**

In this study, we conducted a quasi-systematic scoping review of methods and criteria used to evaluate cardiovascular disease mHealth apps in the published literature. In a preliminary scoping review, we identified gaps in the literature and synthesized key concepts in a narrative review [10]. Then, in an iterative process, we scoped the literature with refined search terms, performing a final quasi-systematic search with fixed search terms [11].

#### *2.1. Preliminary Scoping Review*

We conducted a preliminary scoping review of articles of mHealth apps for CVD through an unstructured and open search to generate an overview of existing methods of evaluating mHealth apps for CVD [12] and to confirm the validity of our research objective. The results of this review informed the development of our final search strategy and analysis.

#### *2.2. Inclusion and Exclusion Criteria*

Our preliminary scoping review revealed various apps designed to reduce the users' risk of developing cardiovascular disease. These apps focus mainly on reduction and control of risk factors for CVD, such as diabetes, hypertension, chronic obstructive pulmonary disease, nutrition, and physical activity. Based on these results, we derived inclusion and exclusion criteria for the subsequent quasi-systematic scoping review of publications in German and English evaluating mHealth apps designed for adult patients diagnosed with acquired cardiovascular disease. Table A1 in the Appendix A provides a complete overview of our inclusion and exclusion criteria.

#### *2.3. Search Strategy*

Our final search followed a quasi-systematic approach. We searched the "PubMed", "Livivo", and "ProQuest" databases to identify relevant literature published between 2000 and the beginning of April 2021. The last search took place on 6 April 2021. Using keywords and index terms relevant to cardiovascular disease, mHealth, and evaluation, we developed search strings, which we adjusted for each database. Table A2 in the Appendix A provides a list of our search terms.

#### *2.4. Literature Selection*

In selecting suitable literature, we applied the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) scheme [13]. The process steps and the results of the study selection are illustrated in Figure 1 below.

**Figure 1.** PRISMA flow diagram of the study.

After importing our 5044 records into *Covidence*, we excluded 978 duplicates. Then, two scholars independently screened the titles and abstracts of the remaining 4066 entries to identify adherence to previously defined inclusion and exclusion criteria. After resolving inconsistencies by consensus, 3708 studies were excluded. We then undertook a full-text review of the remaining 358 articles, excluding an additional 320 studies because they failed to meet our inclusion criteria. Many of the articles we excluded were study protocols, focused on apps designed only to prevent risk factors, such as high blood pressure or diabetes apps, or assessed apps that rely on implanted sensor technology. Our final sample of *n* = 38 articles was included in the scoping review and approved for data extraction.

#### *2.5. Data Extraction and Analysis*

In a next step, we extracted data from the studies according to variables, in order to sort and map the literature to reveal patterns, key information, and research gaps in a data chart for subsequent evaluation. The data extraction sheet was developed by two authors based on the findings of the preliminary scoping review and adapted as part of the iterative process to ensure all relevant information from the studies were captured and included in the analysis. To identify evaluation approaches and criteria, we classified the studies into three categories. Interventions carried out using only an app are classified as "mHealth app"; interventions using an app plus at least one additional device, such as an electrocardiogram or smartwatch, are classified as "mHealth system"; and interventions using only text messages are classified as "mHealth text messaging". Table A3 in the Appendix A summarizes the extracted information as a data chart.

#### **3. Results**

#### *3.1. Characteristics of the Identified Studies*

All articles included in our study were published between 2012 and 2020, even though our search spanned 2000 to April 2021. One-third of the articles were published by scholars in the US (*n* = 13), 13% by scholars in Australia, and 10% by scholars in China. Studies with quantitative and qualitative research designs were included in our review. The largest proportion (*n* = 18) consists of randomized controlled trials (RCTs), followed by singlearm prospective studies and mixed-methods studies (each *n* = 7). Figure 2 illustrates the frequency of study designs.

**Figure 2.** Study designs of the studies identified.

Four of the studies [14–17] lasted over 12 months, while the shortest study lasted 2 weeks [18]. The largest study had 767 participants [18], while the smallest study had 8 participants [19]. Just over half (57.9%) of the studies reported a retention rate (RR) (the percentage of study participants who remained in the study until the defined end of the study process) of between 90% and 100%, while only four studies [20–23] reported an RR of below 50%. For analysis purposes, we also tracked the corresponding loss to follow-up (LTFU) (the percentage of study participants who drop out of a study before the defined end of the study process) figure for each study.

Just over half (52.6%) of the studies focused on mHealth systems (app plus device). The context includes applications for telemonitoring (*n* = 12) as well as for cardiac rehabilitation (CR) (*n* = 8). Seven studies in the mHealth apps (app only) category focused on self-management applications and five focused on CR. In contrast, the smallest share (15.7%) of studies focused on text messaging for self-management purposes (mHealth text messaging category).

#### *3.2. Methods and Measurements for Evaluating mHealth Technologies*

The studies followed qualitative, quantitative, and mixed-methods designs and the great majority (*n* = 31) analyzed data collected through standardized questionnaires. In most cases (*n* = 33), the overall aim of the research was to assess participants' perceptions of treatment and subjective health. In addition to general questionnaires on quality of life (e.g., "EQ-5D" [15], "health-related quality of life" [15], illness (e.g., "Self-Care of Heart Failure Index" [24]) or the psychological well-being of the patients (e.g., "8-item Morisky Medication Adherence Scale" [25,26], "Hospital Anxiety and Depression Scale" [20]), specific question sets for digital applications were also used. The Mobile Application Rating Scale (MARS) was frequently applied in assessing mHealth apps [27]. The "Perceived Health Web Site Usability Questionnaire" (PHWSUQ) [28] specifically addresses assessing the usability of websites among elderly participants [29]. Each questionnaire appeared

once in the analysis [18,28]. In addition to standardized question sets, self-defined questionnaires (*n* = 3), interviews (*n* = 5), and open-feedback rounds (*n* = 7) were conducted to determine perceptions.

A large proportion of the publications (63%) evaluated mHealth interventions using medical measurements (e.g., blood pressure, pulse, weight), comparing health parameters before and after the intervention. The results were often compared directly between the standard of care and the mHealth intervention (*n* = 15). The medical outcomes were used to assess, among others, the feasibility of the intervention (*n* = 16) and physical activity (*n* = 21). The measurements were either documented by the participants using the mHealth device or determined by healthcare providers using monitoring data or laboratory diagnostics.

Interactions with the mHealth app on the part of patients (*n* = 19) and health care providers (*n* = 2) were often recorded in usage protocols (*n* = 19) used to draw conclusions about participants' motivation (*n* = 17), adherence (*n* = 18), and self-efficacy (*n* = 14). In mHealth apps for CR, usage data and logging activities related to login-ins, training, or learning modules were analyzed [30,31]. In one study of an mHealth system for medication adherence [32], the number of times two electronic pill bottles were opened was documented using timestamps.

The usability of mHealth interventions (*n* = 14) was evaluated using several measurement methods and instruments, such as the PHWSUQ and the "System Usability Scale" [33]. A theoretical basis was used in two studies [34,35] to develop the intervention and measure usability. One study adapted the Unified Theory of Acceptance and Use of Technology 2 (UTAUT2) to measure various factors influencing mHealth intervention technology use behavior [36]. In another study [34], the practice of mHealth was prompted by the responsible intervention team as part of a usability test.

Over one-third of the studies (*n* = 14) investigated the effectiveness and efficiency of mHealth for new clinical treatments. Several studies relied on various key performance indicators (KPIs) in assessing mHealth effectiveness (*n* = 11), including, most frequently, hospital readmission, length of hospital stay, number of doctor visits, and hospital admittance due to heart defects. Less attention was paid to mortality and personnel resources required for monitoring. Two studies [37,38] undertook cost-effectiveness analyses. A small number of studies used application-specific indicators, such as data management [38,39], communication between users [38,40], app features [18,41], design characteristics [42], or technology and algorithm analyses [43].

#### **4. Discussion**

The integration of mHealth apps into healthcare structures is a relatively young field of investigation: the analysis shows that the oldest two studies [14,24] date back less than 10 years, probably due to relatively recent and rapid developments in mobile technologies. The relevance of the research topic of mHealth systems and their evaluation is supported by the large number of publications that we found, and a large body of research exists for health applications for certain manageable illnesses and conditions, such as diabetes, high blood pressure, and obesity-related health problems. Most of the studies included in the analysis were randomized controlled trials, thus providing high-quality evidence-based results and high proof of efficacy [44].

#### *4.1. Patient Empowerment in mHealth Interventions for CR*

Overall, our results show that mHealth interventions for cardiac rehabilitation (CR) can be used to reduce or manage coronary heart disease (CHD) and potentially contribute to secondary prevention by empowering heart attack survivors to monitor their risk factors themselves and act accordingly. We find that by using self-management functions, patients can participate actively in their care process and take more responsibility for their health [45]. We thus identify self-efficacy and motivation as key indicators for evaluating mHealth interventions and in an evaluation framework. This recommendation underscores Schwab et al.'s discussion of approaches to developing mHealth applications and the

importance they attribute to increasing awareness and empowerment among patients and healthcare professionals [46].

#### *4.2. Usage Behavior and Motivation*

Our results show that the retention rate and LTFU are suitable measures of motivation and commitment among mHealth intervention users. The fact that more than half of the studies identified had a very high retention rate indicates an overall positive perception of mHealth interventions among users. Our results indicate that usage protocols provide reliable insights into usability, acceptance, and user motivation levels. We also identify the benefits of adapting the Unified Theory of Acceptance and Use of Technology 2" (UTAUT2) to fit the mHealth application use context: the modified construct includes seven factors influencing intention to use a telemonitoring system, together with the independent variables age, gender, and experience influencing the factors.

#### *4.3. Quantitative and Qualitative Research Methods*

While both quantitative and qualitative research methods can be used to collect data, almost all included studies use standardized validated questionnaires and scales, enabling the analysis and comparison of large samples and yielding comparable quantifiable results. Using validated tools is cost and time efficient [47]. Since quantitative research methods often allow little room to interpret the questions, the research framework should include open questions, such as semi-structured interviews or focus groups [48]. Our results illustrate the benefits of combing quantitative and qualitative research methods, particularly in assessing patient satisfaction with the intervention.

#### *4.4. Quality Assessment*

The Mobile Application Rating Scale (MARS) [18] has been used as an instrument to assess the quality of mHealth apps according to the following quality indicators: engagement, functionality, aesthetics, information quality, and subjective app quality [27]. Terhorst et al. [49] demonstrated the suitability and validity of these indicators and recommended using the instrument to increase transparency for stakeholders and patients. While an mHealth intervention evaluation framework should include app quality criteria, the quality assessment should not be limited to subjective user feedback but rather should include data quality and interoperability with other devices and interfaces.

#### *4.5. Privacy and Data Security*

Data security and privacy are important to patients and legally protected. Schnall et al. [50] found a decrease in trust in mHealth solutions and data transfer over time and Zhou et al. [51] showed that some patients refuse to use mHealth applications because of security concerns, loss of interest, or hidden costs. Despite these concerns, our results show that little attention has been paid to data management, such as data transfer between health care providers and participants, data privacy, and data security. An mHealth app evaluation framework should assess the app's data protection systems carefully and communicate the results transparently.

#### *4.6. Economic Evaluation*

Performance measures, such as hospital readmissions, are an important indicator of the effectiveness and efficiency of mHealth systems and should be included in an evaluation framework as well. In the CR mHealth intervention context, our results show that mHealth apps can reduce heart failure-related hospital days and studies conducting cost-effectiveness analysis underscore that shortening out- and inpatient stays also cuts healthcare costs [52]. Similarly, Maddison et al.'s [37] post-hoc economic evaluation assessed the costs of implementing and delivering the intervention to calculate the incremental cost-effectiveness ratio (ICER) between costs and quality-adjusted life years (QALYs) gained and to compare the health benefit gains of switching from standard in- and

outpatient care to mHealth-supported care. The authors found that mHealth interventions are more cost-effective compared to the standard care and can improve health-related quality of life in an ongoing program. Martín et al. applied a "Hidden Markov Model" to measure cost-effectiveness. Long-term costs and outcomes associated with an illness and a particular health intervention can be estimated over multiple cycles, based on resource use and health outcomes [53]. Martín et al.'s [38] study modeled the different disease states of patients during the mHealth intervention, using economic parameters for the outcome analysis and aligning participants' health-specific and follow-up data with healthcare costs published by the health care system. Their cost-effectiveness analysis model showed that introducing an mHealth app lowered the overall cost of disease management by 33% of the total cost of disease management [38]. Pavlovi´c et al.'s [54] results are equally striking: introducing mHealth apps can reduce the total expenses related to data collection in medical scenarios by 50%.

#### **5. Conclusions**

Our scoping review of scholarly articles including criteria and methods of evaluating mHealth apps for cardiovascular disease makes recommendations for developing an evaluation framework for mHealth interventions. In keeping with recent research on the health benefits of active patient involvement in their treatment process, we recommend adopting a user perspective. While various methods and criteria have been used, we recommend quantitative methods using validated standardized questionnaires to generate comparable quantifiable results with a reasonable effort in terms of time commitment and cost. In addition to considering the overall effects of mHealth apps on mental and physical health, we recommend that mHealth intervention evaluations apply usage protocols to understand the patients' interaction with the application and assess their motivation, engagement, and acceptance of integrating the interventions into healthcare processes sustainably. We also recommend including the retention rate and LTFUs, and adapting use and acceptance constructs, such as UTAUT2, into the mHealth technology use setting by modifying its assessment dimensions accordingly.

In terms of the overall scope and considerations for the development of an mHealth app evaluation framework, we recommend focusing on the added value of an mHealth intervention. Specifically, we recommend laboratory diagnostics and physical tests to assess objective physical health, standardized surveys and semi-structured interviews to assess subjective quality of life, and economic performance and efficiency KPIs, such as hospital readmission data and incremental cost-effectiveness ratios between costs and quality-adjusted life years. Heterogeneity of results by using different standardized surveys and questionnaires could be a major challenge for the analysis and comparisons of the results from such a framework. Therefore, the selection of data collection tools needs to be made carefully.

mHealth app providers, patients, healthcare providers, healthcare systems, and society at large will benefit by applying these recommendations when developing a holistic framework to evaluate mHealth apps and interventions to ensure that they are effective, efficient, empowering, accurate, sustainable, and safe. Such a framework will enable an informed decision when choosing an mHealth app.

**Author Contributions:** Conceptualization, F.H., methodology, J.K. and F.H.; software, J.K. and F.H.; validation, F.H.; formal analysis, J.K.; investigation, F.H.; resources, J.K.; data curation, J.K.; writing original draft preparation, J.K.; writing—review and editing, F.H., W.J.S. and J.S.; visualization, J.K.; supervision, J.S. and W.J.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** All relevant data can be found in the Appendix A.

**Acknowledgments:** The authors would like to thank Timo Guter for his guidance and support during the literature search.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**

**Table A1.** Inclusion and exclusion criteria based on the PCC elements.


#### **Table A2.** Search strings and number of results.





**Table A3.** *Cont.*


**Table A3.** *Cont.*


#### **Table A3.** *Cont.*


**Table A3.** *Cont.*

#### **References**


## *Review* **Open Issues and Practical Suggestions for Telemedicine in Chronic Pain**

**Marco Cascella 1, Franco Marinangeli 2, Alessandro Vittori 3,\*, Cristina Scala 4, Massimo Piccinini 5, Alessandro Braga 6, Luca Miceli <sup>7</sup> and Renato Vellucci <sup>8</sup>**


**Abstract:** Telemedicine represents a major opportunity to facilitate continued assistance for patients with chronic pain and improve their access to care. Preliminary data show that an improvement can be expected of the monitoring, treatment adherence, assessment of treatment effect including the emotional distress associated with pain. Moreover, this approach seems to be convenient and costeffective, and particularly suitable for personalized treatment. Nevertheless, several open issues must be highlighted such as identification of assessment tools, implementation of monitoring instruments, and ability to evaluate personal needs and expectations. Open questions exist, such as how to evaluate the need for medical intervention and interventional procedures, and how to define when a clinical examination is required for certain conditions. In this context, it is necessary to establish dynamic protocols that provide the right balance between face-to-face visits and telemedicine. Useful tips are provided to start an efficient experience. More data are needed to develop precise operating procedures. In the meantime, the first experiences from such settings can pave the way to initiate effective care pathways in chronic pain.

**Keywords:** chronic pain; telemedicine; healthcare delivery; health-related quality of life; functionality

### **1. Introduction**

Telemedicine has been defined as the use of electronic technologies for communication and information of patients, to provide the public with remote healthcare services [1]. Although it has existed for more than two decades, its implementation has been limited for many years, until the emergency during the COVID-19 pandemic. This catastrophic event has promoted attempts to provide chronic patients with adequate care despite restrictions to in-presence activities [2]. The definition of adequate care pathways for chronic pain will need long clinical trials as the condition is complex and many different cases must be faced. As an example, patients with chronic post-surgical pain who have a certain diagnosis, a complex condition, and need careful and prolonged follow-up, and can benefit from the assistance through remote services. Recent experiences have suggested that telemedicine

**Citation:** Cascella, M.; Marinangeli, F.; Vittori, A.; Scala, C.; Piccinini, M.; Braga, A.; Miceli, L.; Vellucci, R. Open Issues and Practical Suggestions for Telemedicine in Chronic Pain. *Int. J. Environ. Res. Public Health* **2021**, *18*, 12416. https://doi.org/10.3390/ ijerph182312416

Academic Editors: Marie Carmen Valenza and Irene Torres-Sanchez

Received: 27 October 2021 Accepted: 24 November 2021 Published: 25 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

can improve access to care, facilitate continuity of care, allow better resource efficiency, and lower costs, compared with traditional in-person hospital or ambulatory visits [1,3,4]. Improved access to care is expected to enhance timely adjustment of therapy, and improved adherence, which could reduce the progression toward reduced functionality of patients with chronic pain [5]. Appropriate uses of telemedicine for patients with chronic pain have recently been described. It has opened a promising field of activity, although a business case analysis would be needed in each prospective application [2,6].

On these premises, best practice approaches for telemedicine programs in chronic pain need to be suggested, to enable clinicians to provide and patients to benefit from remote assistance.

This article is based on the direct experience of a group of clinicians and attempts to provide a framework to prepare physicians, patients with chronic pain, and caregivers to use telemedicine with satisfactory results.

#### **2. The Complexity of the Patient with Chronic Pain**

According to the International Association for the Study of Pain (IASP), chronic pain is commonly defined as persistent or recurrent pain lasting more than 3 months or beyond normal tissue healing [7]. It has been recognized as a real disease associated with multiple adaptations in the nervous, endocrine, and immune systems [8]. Consequently, chronic pain is a complex multidimensional experience severely compromising the patient's health-related quality of life (HRQoL), often limiting the ability to work, sleep, and affecting social interactions with friends and family [2,9]. Reduced functionality, emotional imbalances, and social isolation are frequently associated complaints and may exacerbate each other in a vicious circle that compromises the HRQoL and induces a progression toward disability [10,11]. Furthermore, chronic pain is often associated with multimorbidity. In particular, many patients with chronic pain have other comorbidities, such as depression, cardiovascular and pulmonary diseases, diabetes mellitus, and cancer [12,13]. Notably, proper pain control may be extremely challenging in multi-morbid patients as comorbidities and their treatments can increase the risk of side effects of analgesics and thus limit the applicability of disease-specific clinical guidelines [14]. On the other hand, chronic pain is an independent risk factor for mortality in people with other co-morbidities [15]. Moreover, patients with chronic pain require multidisciplinary, continuous, and skilled management, which may challenge healthcare system organizations. Difficulties with the traditional models of care, with in-person patient visits to their physicians inherently leading to delayed care, have further cumulated during the COVID-19 pandemic and the need to structure new care pathways to ensure appropriate treatment of chronic pain patients, became even more prominent.

The first and necessary steps for the management of chronic pain are diagnosis and assessment of all pain dimensions. Given the 11th International Classification of Diseases (ICD-11) introduction, chronic pain will be classified into primary and secondary pain. Chronic primary pain can be conceived as a disease. This is a definition, which applies to chronic pain syndromes that are best conceived as health conditions in their own right [7]. Chronic secondary pain syndromes are linked to other diseases as the underlying cause, for which pain may initially be regarded as a symptom. Secondary pain includes cancer-related, post-traumatic and postsurgical, neuropathic, visceral, musculoskeletal, and headache/orofacial pain [7]. In many cases, the secondary chronic pain may continue beyond the successful treatment of the initial cause; in such cases, the pain diagnosis will remain, even after the diagnosis of the underlying disease is no longer relevant. This distinction is particularly important because it conditions the clinical, diagnostic, and therapeutic approaches. For example, in the cancer patient, close monitoring with close controls is mandatory when opioids are used. Moreover, in primary chronic pain (e.g., fibromyalgia) a combined approach with the collaboration of different professional figures may be necessary.

#### **3. Approaches of Telemedicine for Patients with Chronic Pain**

Telemedicine has been defined as the use of electronic technologies for communication and information to provide the public with healthcare services at a distance. More recently, the American Medical Association defined "telehealth" as a general group of modalities allowing: (1) real-time audio and visual connections between patients and physicians in different locations; (2) image and data collection storage and sharing for later interpretation; (3) remote patient monitoring tools, including mobile health (mHealth) tools, wearables, and devices; and (4) virtual check-ins through voice-only patient portals, messaging technologies [16]. The Italian Ministry of Health produced guidelines for telemedicine and defined the televisit as "a health act in which the doctor interacts remotely with the patient". The definition specifies that "the health act of diagnosis that arises from the visit may give rise to the prescription of drugs or treatments" [17].

Thus, the terms "telemedicine" and "telehealth" are commonly used interchangeably and encompassed in the set of tools called "telecare" [18]. Telecare includes several possible modalities and activities, such as archiving and sharing medical images or biosignals (e.g., in the fields of radiology, or dermatology), telemonitoring, and real-time interactive services. The latter modalities include a variety of services, such as telenursing, telepharmacy, telerehabilitation, emergency counseling, and mostly online consultation via remote visits and/or multi-professional teleconsulting.

#### **4. Online Consultation Pathway**

A remote system is a great opportunity to improve access to care and continuing assistance, which may help to personalize treatments and to increase adherence. To fulfil these objectives, it is necessary to structure a defined pathway. It is divided into a series of technical processes (information technology infrastructure) and operational phases (preparation, execution, scheduling of controls) (Figure 1).

**Figure 1.** Telemedicine pathway for addressing chronic pain.

The information technology infrastructure must allow all organizational phases (reservations, contacts, links for connection, and data collection) and guarantee data security. Operation includes a first visit usually performed in person. This face-to-face assessment is followed by the preparation for telemedicine (legal and regulatory issues, patient information, technical issues). Later, telemedicine is performed, and scheduled controls are programmed. New in-person visits can be required (e.g., to carry out minimally invasive procedures). IT, information technology.

#### *4.1. IT Infrastructure Functioning*

The COVID-19 pandemic and the need to provide alternative ways for the in-person visit has led to the creation of a wide range of information technology (IT) infrastructures. On the market, there are systems which have different complexities (and costs). In general

terms, the platforms consist of an operating system for the management of the whole service, devices (e.g., laptops), and an integrated software system (software modules) for sending documents, reports, and imaging. In brief, the IT infrastructure must allow all organizational phases (reservations, contacts, links for connection, and data collection) and guarantee data security and privacy.

#### *4.2. Operational Phase: First Face-to-Face Visit*

Since programs of telemedicine should be based on good interpersonal relationships, a first visit should usually be performed in-person to:


An in-person contact provides the necessary and reciprocal confidence for further remote relationships. Indeed, patients must be evaluated for their ability to use telemedicine before a program is established. Most of the information infrastructures available for telemedicine require that patients and/or caregivers simply need to have an email address and a smartphone, iPad/tablet, or a personal computer with a camera and speakers. Based on the in-person visit, a remote follow-up and monitoring schedule can be prepared.

#### *4.3. Operational Phase: Remote Follow-Up*

For remote follow-up it is important to collect the right parameters that can be obtained remotely. All the different aspects of chronic pain must be monitored; pain intensity, therapeutic adherence, sleep quality, movement functionality, emotionality, and working abilities. Patients and caregivers need training in order enable them to focus on the relevant topics during the telemedicine visit; the physician will guide the visit and choose the relevant area to be investigated in the situation.

#### 4.3.1. Clinical Assessment

The remote evaluation involves the study of the patient's medical record (imaging, laboratory tests, other documents) and the clinical–diagnostic phase. A comprehensive pain assessment is a crucial step in the management of a patient with pain. As already stated, chronic pain is a multidimensional experience resulting in impaired functioning in daily life and reduced quality of life and well-being of the patient, as it can be observed for chronic low back pain (cLBP) [19]. We suggest that physicians could assess pain severity mainly using the parameters of pain intensity, pain-related distress, and functioning.

Generally, assessment tools for telemedicine should be validated, and suitable; moreover, the same tools should always be used. In addition, other instruments can be used, for the objective evaluation of distress and functionality through web-based use. Based on the authors' experience, a combination of unidimensional and multidimensional tools can be adopted. The numerical rating scale (NRS) may easily be used by patients to assess pain intensity. The Brief Pain Inventory (BPI) is a validated, simple, and self-completed questionnaire (visual administration) that evaluates not only pain intensity, but also functionality, and provides long-term monitoring in patients with progressive conditions. The tool is reliable and valid for many clinical situations (e.g., cancer pain, and non-cancer pain conditions) and across cultures and languages. Functionality can be assessed by an ecological matrix scale [20–22], which considers the outer environment and the personality structure of the patient, the motives, the personal expectations, and needs, and helps the patient and the physician to identify treatment objectives that may be satisfactory for the patient.

In the setting of chronic low back pain, the Oswestry Disability Index and the Roland-Morris disability questionnaire may be useful for the functional evaluation of low back pain [22,23].

Other indices are available, such as the Low Back Pain Rating Scale (LBPRS), as an example, the Progressive Isoinertial Lifting Evaluation (PILE), and the Quebec Back Pain Disability Scale (QBPDS) [24]. In this respect, clinicians should also evaluate the capability of their patients to perform daily activities, the patient's emotional status, and their strength. Up to now, different proper instruments are available to evaluate HR-QoL, such as the EQ-5D from the EuroQol Research Foundation, and the short form (SF-12 scale) of the 36-item Health Survey instrument are available [25,26].

In addition to the evaluation through tools, all the anamnestic elements must be collected. Even if at a distance, the clinician will have to investigate the clinical elements of the painful symptomatology: location, intensity, triggering factors, therapies carried out, and comorbidities. Peculiar aspects, such as breakthrough cancer pain, drug effects, and clinical conditions, which may affect the use of particular categories of drugs (e.g., organ damage), must be evaluated.

Finally, the recommendations issued by the Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials (IMMPACT) can also be used. A common tool would be used to evaluate the validity of telemedicine models. Notably, six core outcome domains were recommended by IMMPACT including pain, physical functioning, emotional functioning, participant ratings of global improvement, symptoms and adverse events, and patient disposition (adherence to the treatment regimen, reasons for withdrawal from treatment) [27]. A systematic review showed that eHealth and mHealth interventions had significant effects on multiple short- and intermediate-term outcome measures recommended in the IMMPACT guidelines [28].

#### 4.3.2. Outcomes, Therapy, and Re-Evaluation

The televisit can give rise to different outcomes; it is possible to achieve clinical stability within the already known diagnostic framework. It can bring out the need for urgent access to diagnostic or therapeutic services, which require the patient to have a face-to-face consultation with the pain specialist. The third scenario includes the need for further examination to have a diagnosis, which the specialist will manage with the prescription of the necessary services.

During the televisit, the previous therapy can be either confirmed or changed; in this case, the specialist prescribes the drugs and sends the prescription to the patient as agreed with him and his caregiver. Although the primary objective of treatment should be control of pain intensity, functional recovery and general wellbeing are the overall aims of the patient's management. This means that pharmacological and non-pharmacological treatments must be considered, and that tailored objectives are to be pursued according to the expectations and needs of each patient [20–22].

If it is impossible to reach a diagnostic or therapeutic conclusion, the doctor will propose the execution of a further check-up in the times and the ways appropriate to the clinical situation, and the follow-up pathway will be planned. In some cases, possible adverse events may be mentioned by the physician, asking the patients whether any of them occurred in the last period; this may prove a simple and effective method to detect tolerability issues. Under certain circumstances, such as drug side effects, need for a physical examination or interventional procedures, an in-person visit may be necessary.

The remote visits should be tightly planned to ensure that monitoring can be performed adequately, and therapy can be adjusted. The data collected and outcome of the televisit will need to be recorded as a routine medical visit and filed according to local customs.

#### **5. Open Issues and Suggestions**

When developing a telemedicine plan for the care of patients with chronic pain, a number of challenges need to be considered. These include the risk of adverse effects of drugs prescribed in a remote visit, the correct management of patients with advanced age, cognitive impairment, emotional frailty, and the need to alternate remote and in-person

visits. To address these challenges, some suggestions may be useful. Table 1 presents the recommended steps for the implementation of a telemedicine service for patients with chronic pain.

**Table 1.** Some recommended actions for the management of a telemedicine system.


Note: BPI, brief pain inventory.

Prerequisites for remote care systems include correct and exhaustive information about legal issues and regulation, availability of suitable technical equipment, and specific medical skills for the remote management of patients with chronic pain so that a correct assessment is performed despite the lack of a physical inspection. Specifically, all local requirements for healthcare must be fulfilled by remote systems, as well as by traditional inpresence organizations, and some technical solutions must be found to this aim. Therefore, first of all, knowledge of regulation is needed so that medico-legal problems are detected and addressed, and then rules must be adapted to remote systems [29]. The main legal issues to be faced are related to data protection, privacy, and delivery of reports and prescriptions. Several online platforms are available, which respect such requirements.

A basic suitable technical equipment for the physician and the patient includes an efficient connection to the internet, a digital device (usually a PC) with a webcam, and a customized web-based platform. In addition, to set up a telemedicine system, a large

proportion of the target population needs to have sufficient skills to use the proposed web-based platform.

This means that very simple digital tools are to be preferred. Common social media are often used, to facilitate patients and caregivers, but they would not fulfill privacy and data protection requirements. The current reference guidelines for telemedicine are obsolete [30]; some of those, for example, date back to the 1980s and were not followed by later recommendations [31–34]. Privacy regulations, technological opportunities, and problems related to the pandemic are open issues that must be urgently addressed. The use of validated and protected platforms available through public or private healthcare providers could help respond to these issues.

As an example, the Regional Health Service in Tuscany, Italy, made a digital platform for televisits available to all specialists, in June 2020. The online visit requires a PC (with Windows 7 or IOS 11, or later versions), or smartphone or tablet (with Android 5.0 or later version), or iPhone or iPad (with IOS 11 or later version). The platform was mainly used by diabetologists, rheumatologists, and cardiologists, and registered up to 5000 visits in the first month of activity. This platform is extremely user friendly, is linked to the online clinical records, and with the regional health booking service; digital prescriptions are delivered by a preexisting system, while a final report of each visit is provided on the platform, according to specific rules set by the regional healthcare system.

The last suggestion on this point is that patients and/or caregivers must be exhaustively trained to be able to use the telemedicine system.

In summary, before the initiation of the telemedicine program, the medical activities must be carefully planned and scheduled. Online performances will be very fast, and waiting times are very limited online; so, everything must be ready beforehand. In addition, it is important to identify the parameters deemed suitable for remote visits on one end and for telemonitoring on the other one. The patient and/or the caregiver must be selected and, if accepted, prepared. Selection will be based both on medical (as an example, chronic, stable conditions are more suitable than lately diagnosed, progressing patients) and cognitive qualities, and personal features. The patient/caregiver must accept the program and feel it as an opportunity for improved management, continuing assistance, and access to care. If a caregiver is necessary, the same person should be present at all remote visits, and the same person should oversee assessments for monitoring. If the patient is a child, special precautions should be used for an effective protection of her/his rights. A relationship between the patient and/or caregiver and the pain specialist should already exist, with a good therapeutic alliance. A first in-person visit will usually be performed to obtain a diagnosis, to prepare the patient, and state a reciprocal reliance. This phase is followed by telemonitoring and scheduled remote control visits. A strict and punctually respected schedule of the program will facilitate its long-term continuation.

Special attention is necessary for the organization of the clinical assessment. Firstly, suitable tools must be identified. As previously mentioned, simple tools are to be offered; NRS, BPI, self-evaluation numerical scales, and ecologic matrix scales may be used. The assessment frequency must be stated in advance and explained to the patient. A telemedicine system for chronic pain will improve continued assistance but does not usually provide an emergency service. Patients must be informed of the aim of the system and should know what to do in case of adverse events or serious pain episodes. If around-the-clock assistance is necessary according to the patient's conditions, a phone triage should be available to deliver primary information and refer the patient to the correct health operator. Finally, a telemonitoring program will be efficient if the patient and/or caregiver are empowered; so, great care is due to instruction, and information about the disease and the treatment.

During the online evaluation, the physician will not be able to perform physical maneuvers which are commonly used for in-presence assessment, but some simple ones may be proposed to the caregiver. Each visit will be mainly based on the evaluation of the report by the patient and the revision of data collected during the telemonitoring. The physician should develop his/her listening ability as much as possible, and empathy. Punctuality, respect of visit duration, and the ability to listen to the patient and/or caregiver are very important. After each visit a medical report must be delivered; predisposed forms can be designed, and an identical delivery system, such as email, should be routinely used. As an example, patients with musculoskeletal chronic pain are very likely to benefit from an assiduous follow-up performed with a multidimensional assessment made possible and affordable by the introduction of telemedicine [35–37].

#### **6. Limitations**

We acknowledge that several problems limit the possibility of achieving our aim of providing useful hints for the implementation of telemedicine for chronic pain. No evidence is available on the efficacy of this model. Regulation is widely different worldwide, and this makes it impossible to address local issues. As an example, medical responsibility is criminal and not only civil in Italy. In addition, certified online platforms are very different from each other. All these discrepancies make it difficult to design the many care pathways that are needed to address a complex condition such as chronic pain.

#### **7. Conclusions**

Telemedicine seems to be promising for the efficient management of patients with chronic pain. This approach can deliver tailored pain management, providing improved access to health services and creating and maintaining a therapeutic alliance in the long term. Nevertheless, the effects of therapies provided via telemedicine on pain and painrelated conditions, such as disability, depression, and anxiety, are promising but not well documented yet. Furthermore, when approaching remote assistance for chronic pain, several issues are to be faced such as accurate diagnosis, assessment, monitoring and need to change treatment. Consequently, the implementation of new web-based systems for the management of chronic pain needs further evaluation and well-structured pathways must be necessarily built. Finally, as privacy regulation is incomplete worldwide, clinicians are bound to be extremely cautious about respecting the patient's rights.

**Author Contributions:** Conceptualization, M.C., F.M. and A.V.; methodology, M.C., F.M., A.V., C.S. and A.B.; writing—original draft preparation, M.C., F.M. and A.V.; writing—review and editing, M.C., F.M., A.V., L.M., C.S., M.P., A.B. and R.V.; visualization, M.C.; supervision, M.C., F.M., A.V. and R.V.; project administration, F.M., M.C. and A.V. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Acknowledgments:** Editorial assistance was provided by Laura Brogelli, PhD and Aashni Shah (Polistudium SRL, Milan, Italy). This assistance was supported by Grunenthal.

**Conflicts of Interest:** R.V., A.V., F.M., C.S., M.C., L.M. and M.P. report no conflict of interest in this work. A.B. is employed in Grunenthal Italia srl, Italy.

#### **References**


### *Review* **Efficacy of Web-Based Supportive Interventions in Quality of Life in COPD Patients, a Systematic Review and Meta-Analysis**

**Andrés Calvache-Mateo, Laura López-López, Alejandro Heredia-Ciuró, Javier Martín-Núñez, Janet Rodríguez-Torres, Araceli Ortiz-Rubio and Marie Carmen Valenza \***

> Department of Physical Therapy, Faculty of Health Sciences, University of Granada, 18007 Granada, Spain; andrescalvache@ugr.es (A.C.-M.); lauralopez@ugr.es (L.L.-L.); ahc@ugr.es (A.H.-C.); javimartinn29@gmail.com (J.M.-N.); jeanette92@ugr.es (J.R.-T.); aortiz@ugr.es (A.O.-R.) **\*** Correspondence: cvalenza@ugr.es; Tel.: +34-958-248-035

**Abstract:** Background: Adults living with Chronic Obstructive Pulmonary Disease (COPD) often have difficulties when trying to access health care services. Interactive communication technologies are a valuable tool to enable patients to access supportive interventions to cope with their disease. The aim of this revision and meta-analysis is to analyze the content and efficacy of web-based supportive interventions in quality of life in COPD. Methods: Medline (via PubMed), Web of Science, and Scopus were the databases used to select the studies for this systematic review. A screening, analysis, and assessment of the methodological quality was carried out by two independent researchers. A meta-analysis of the extracted data was performed. Results: A total of 9 of the 3089 studies reviewed met the inclusion criteria. Most repeated web content elements were educational and involved communication with healthcare professional content. Finally, seven of the nine studies were included in a quantitative analysis. Web-based supportive interventions significantly improved quality of life when added to usual care (SMD = −1.26, 95% CI = −1.65, −0.86; *p* < 0.001) but no significant differences were found when compared with an autonomous pedometer walking intervention (*p* = 0.64) or a face-to-face treatment (*p* = 0.82). Conclusion: This systematic review and meta-analysis suggests that web-based supportive interventions may complement or accompany treatments in COPD patients due to the advantages of online interventions. The results obtained should be treated with caution due to the limited number of studies in this area and methodological weaknesses.

**Keywords:** communication; COPD patients; educational content; supportive interventions; web-based

#### **1. Introduction**

Chronic obstructive pulmonary disease (COPD) is a non-reversible inflammatory disease that causes progressive obstruction of the airways. According to the Global Initiative for Chronic Obstructive Lung Disease (GOLD) 2020 report, COPD is the leading lung disease in terms of mortality and morbidity worldwide [1,2]. Due to the increase in smoking and the progressive ageing of the population, the prevalence of COPD will increase in the coming years [3].

As the disease progresses, the symptoms become increasingly severe and complex. Often, the combination of psychological, emotional, and social factors with physical symptoms makes it difficult for patients and professionals to deal with the disease [4]. As a result, COPD patients experience significant impairment of disease-related quality of life as well as social isolation [5] that generates a significant burden of disability [6] and demands continuous health care [7].

Unfortunately, COPD patients face significant barriers when seeking access to appropriate health services to manage the disease, including living in medically underserved regions [8], language barriers [9], reduced mobility due to the disease itself, or other comorbidities, such as ageing and limited time [10]. In addition, due to the respiratory status of these patients and the potentially serious medical consequences for them, the risk of

**Citation:** Calvache-Mateo, A.; López-López, L.; Heredia-Ciuró, A.; Martín-Núñez, J.; Rodríguez-Torres, J.; Ortiz-Rubio, A.; Valenza, M.C. Efficacy of Web-Based Supportive Interventions in Quality of Life in COPD Patients, a Systematic Review and Meta-Analysis. *Int. J. Environ. Res. Public Health* **2021**, *18*, 12692. https://doi.org/10.3390/ijerph 182312692

Academic Editors: Shyamali Dharmage and Paul B. Tchounwou

Received: 27 September 2021 Accepted: 29 November 2021 Published: 2 December 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

COVID-19 infection should be minimized [11,12]. Despite all these obstacles, there are not many interventions to support COPD patients in dealing with their disease [8].

Technological development is a great opportunity to generate new tools to support COPD patients [8,13]. Those technologies have enabled existing therapies to be delivered online and allow for the development of new interventions tailored to patients' needs [14]. New technologies are increasingly being investigated with the aim of developing interventions that can adequately complement or replace interventions already provided in health services [15–17].

Rapid advances towards a more digitalized society as well as the rapid development of today's electronic devices have caused a significant rise in the availability of communication technologies applied to health services [18,19]. The different online health communication tools allow patients to access personalized content, disease self-management tools, and communication with healthcare professionals from the comfort and security of their own home [20–22].

The most recent systematic reviews and meta-analyses [23–33] on telehealth care analyze teleassessment, telephone assistance, mobile app development, and website assistance in depth, but they need to be analyzed separately [8].

Previous studies show chronic disease patients' need for personalized web-based interventions [34,35]. COPD patients demand access to information about their health status, related to the disease itself and to the improvement of quality of life [8]. Different mechanisms related to a perception of health-related needs, such as health education, self-management [36], and family and social support, have a significant influence on the quality of life of patients using web-based interventions [12,21,27,37–39].

Web-based interventions can encompass several distinct, often overlapping interventions, including: (1) tele-education content; (2) symptom and mood telemonitoring; (3) physical activity monitoring and personalized feedback to the patient; (4) tele-education in self-management skills; (5) tele-consultation with healthcare professionals; (6) telecommunication with other patients; (7) remote decision support systems; (8) tele-diagnosis; and (9) tele-rehabilitation [27,40,41].

The advantages offered by web-based interventions such as easy and on-demand access to health information content, interactive support with other patients, and tools for symptom self-management may have the potential to influence the different variables and symptoms of a COPD patient. There is a need to investigate whether these web-based interventions have an impact on the quality of life of COPD patients and determine which are the most appropriate contents. The aim of this revision and meta-analysis is to analyze the content and efficacy of web-based supportive interventions in quality of life in COPD.

#### **2. Materials and Methods**

#### *2.1. Search Strategy and Eligibility Criteria*

This systematic review was conducted according to the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) statement guidelines [42] and its registration number in the International Prospective Register of Systematic Review (PROSPERO) is CRD42020211978. The Cochrane Collaboration guidelines for reviewing interventions were also closely followed [43]. Three databases were used for the electronic search: Medline (via Pubmed), Web of Science, and Scopus. The screening and analysis of the studies was conducted between November 2020 and March 2021. Relevant publications from inception to 1 March 2021 were included. A search strategy was created for Medline and then modified to be specific to each of the databases. The following Medical Subject Headings (MeSH) terms were used (Appendix A).

To adequately define the research question, the impact of patient, intervention, comparison, outcome (PICOS) strategy [44] was applied.


(O) Outcome: Any outcome reporting quality of life (e.g., St. George Respiratory Questionnaire, Chronic Respiratory Disease Questionnaire).

(T) Timing: At any time.

(S) Setting: No restriction of setting.

Only full-text randomized controlled trials written in English, Spanish, and French were included in the systematic review. Systematic reviews and meta-analyses, observational studies, clinical practice guidelines, letters, abstracts, editorials, conference papers, theses, and dissertations were excluded. Studies in other languages were also considered for inclusion when translation was possible.

#### *2.2. Study Selection and Data Extraction*

After all studies had been retrieved from the three databases, duplicates were removed. To determine if the articles met the inclusion criteria for this systematic review, two independent investigators performed a first assessment of the title and abstract of all studies. If the article met the inclusion criteria, it was selected for a second phase in which the full text was analyzed and reviewed.

The Cochrane guidelines for systematic reviews were followed for data extraction [43]. A third reviewer was responsible for resolving any disagreement between the two main reviewers. The information extracted from the articles was: year of publication, main author, sample size, sample age, treatment status, severity of COPD, specific intervention for the control and experimental groups, web content elements, intervention duration, outcome measures, and main results. If the reviewers did not find any data during the analysis and review of the articles, they contacted the authors of the studies.

#### *2.3. Assessment of Methodological Quality and Risk of Bias*

The Downs and Black quality checklist was used to assess the methodological quality of the studies included in the review [45]. This assessment was carried out independently by the two principal investigators. This method contains 27 items divided into 5 subscales (study quality, external validity, study bias, confounding and selection bias, and study power). Due to its high reliability and validity, this scale is considered one of the six most appropriate scales to measure the quality of the studies included in a systematic review [46]. Studies are classified into four categories according to the score obtained: it will be classified as poor if its score is less than or equal to 14, fair if the score is between 15 and 19, good if the score is between 20 and 25, and excellent if the score is between 26 and 28 [46,47].

In addition to the methodological quality of the articles, the risk of bias was assessed using the Cochrane Risk of Bias Tool for Randomized Controlled Trials [43]. This measurement tool is divided into seven items that are subdivided into six subscales. The first subscale corresponds to the selection bias and is the only one with two items. The remaining subscales are called performance bias, detection bias, attrition bias, reporting bias, and other bias, and have only one item. When the reviewer determines that there is a low risk of bias for each of the items, the study is classified as high quality. When the reviewer determines that one of the items is not met because there is a high risk of bias or two of the items cannot be answered clearly, the study is classified as fair quality. When the reviewer determines that one of the items is not met because there is a high risk of bias or two of the items cannot be answered clearly and there are important limitations that may invalidate the results, the study is classified as poor quality. The study is also classified as poor quality when two or more items are not met.

#### *2.4. Data Synthesis and Analysis*

A meta-analysis was undertaken using Review Manager (RevMan v5.3; Cochrane Collaboration, Oxford, UK). All variables included were continuous data. Study authors were contacted by e-mail whenever data were insufficient for the purposes of meta-analysis (e.g., neither means nor standard deviation were provided). Authors were given 2 weeks to respond. If they had not responded within a week, they were written to again as a reminder. The embedded Review Manager calculator was used to calculate standard deviations whenever *p*-values or 95% confidence intervals were given [48].

The main outcome considered for this meta-analysis was quality of life. Standardized mean differences were used because all scales were assumed to measure the same underlying symptom or condition, but some studies measured outcomes on different scales and 95% confidence intervals (CI) were calculated for all outcomes [49]. Subgroup analysis was also used in this study to help clarify the different uses of web-based interventions.

When the studies presented different scales to measure quality of life, we selected the data provided by the Saint George Respiratory Questionnaire (SGRQ), since it is the most frequently used, disease-specific quality of life measure in this population group [50]. When studies did not use the SGRQ, scores from other disease-specific quality of life scales, such as the Chronic Respiratory Disease Questionnaire (CRQ), were used [50–52]. The scoring of the different scales was converted so that a lower score always indicated a better outcome.

The Q and I2 statistics were calculated to examine statistical heterogeneity, and a visual inspection of the forest plots was also performed to identify outlier studies. The I2 is a statistical value that is interpreted as the percentage of the total variation observed between studies that is due to the difference between them and not to sampling error (chance). An I<sup>2</sup> of ≥50%; I<sup>2</sup> >25% and < 50%; I2 of ≤25% were considered to indicate high, moderate, and low heterogeneity, respectively. When the I2 value is greater than 50%, the meta-analysis is considered heterogeneous and, therefore, a random effects analysis had to be used. Statistical significance was established as *p* < 0.05, which means that the effects differ significantly between the control and intervention groups. We also explored sources of heterogeneity and performed a sensitivity analysis excluding trials with high risk of attrition or detection bias [48].

#### **3. Results**

#### *3.1. Study Selection*

An initial search of the databases found 3089 records. After eliminating duplicates, a total of 1319 studies were selected. In the end, an overall total of 9 studies that analyzed a total of 1168 participants were included in this systematic review. The PRISMA flow diagram for the study selection process is shown in Figure 1.

#### *3.2. Study Characteristics*

Table 1 shows the main characteristics of the studies included in the systematic review. The included studies were published between 2013 and 2020, and assessed participants were aged between 66.1 [53] and 71.9 years [54]. All the studies except the study of Wang et al. (47.5%) [54] had a higher proportion of males than females in the study sample. Regarding COPD severity, five studies [55–59] included mild to very severe patients and four studies [53,54,60,61] included moderate to very severe patients. All studies included clinically stable patients, with the exception of Wang et al. [54] and Jiménez-Reguera et al. [61], which included patients after discharge.

The web-based supportive interventions of each study were covered in Table 2 by the content of the comparison group approach, the content of the experimental interventions, the intervention duration, the outcome measures, and main results. Table 2 also includes nine web content elements that were identified as important to the technical characteristics of internet-supported therapeutic interventions [27,62] as well as for evidence-based web interventions: 1, tele-education content; 2, symptom and mood telemonitoring; 3, physical activity monitoring and personalized feedback to the patient; 4, tele-education in self-management skills; 5, tele-consultation with healthcare professionals; 6, tele-communication with other patients; 7, remote decision support systems; 8, tele-diagnosis and 9, tele-rehabilitation [63,64].

**Figure 1.** PRISMA flow chart of the literature screening process and results.

One study compared the usual care with a comparator group who received the usual care in addition to the web-based supportive program based on tele-education and tele-consultation with healthcare professionals [54]. Four studies compared a web-based supportive pedometer walking intervention based on physical activity monitoring, personalized feedback to the patient, and tele-education, with a pedometer walking intervention without web support [56–59].

Four studies attempted to demonstrate the non-inferiority of the web-based intervention when compared to a face-to-face program. For this purpose, the same intervention was carried out in both face-to-face and online modalities. Two studies were based on a telerehabilitation program [53,60], another in a self-management program [55], and the last one was based on tele-education and symptom and mood telemonitoring [61].

Most repeated web content elements were tele-education content, self-management skills training, and tele-consultation with healthcare professionals. Only one study [57] excluded educational content. Education in self-management skills and tele-communication with healthcare professionals were excluded by Jiménez-Reguera et al. [61]. in three of the studies [53,55,60].


**Table 1.** Characteristics of the included studies.

Notes: EG: experimental group; CG: control group; SD: standard deviation.

In each study, the mean duration of intervention was 7.9 months (ranging from 6 weeks to 15 months). Most of the studies conducted an intervention over one year [54,55,57,59]. One study conducted an intervention of 10 months [61] and 4 studies conducted an intervention of less than 4 months [53,56,58,60].

The included studies evaluated quality of life using different tools. Disease-specific tools, e.g., the St. George's Respiratory Questionnaire (SGRQ), Chronic Respiratory Questionnaire (CRQ), Chronic Obstructive Pulmonary Disease Assessment Test (CAT) and general tools, e.g., the Short Form 36-Item Health Survey (SF-36) and EuroQol 5-Dimension Questionnaire (EQ- 5D) were used. The most commonly reported outcome was SGRQ, which was followed by CRQ and CAT.

Other variables used in several studies were: self-efficacy, functional capacity, dyspnea, physical activity, lung function, anxiety, and depression. Self-efficacy was measured in four studies, with the most used tool being the Exercise Self-Regulatory Efficacy Scale (Ex-SRES). Functional capacity was the second most frequently measured variable after quality of life. Five studies measured functional capacity with the 6MWT being the most used tool [54,55,58,60,61]. Four studies measured dyspnea and physical activity [54,55,58,60], three studies measured anxiety and depression [53,58,60] and two studies measured lung function [54,61].

The results obtained in the majority of included RCTs show no significant differences between groups in quality of life. Only one study reaches significant results in quality of life when compared to control intervention [54]. This result can be due to the duration of the program (12 months) and the content related to coaching. Furthermore, the majority of included studies showed significant improvements among the group in quality of life outcomes after intervention [53,55–57,61]. In addition, some studies aimed to demonstrate that web-based intervention was not inferior to face-to-face intervention and found similar results in quality of life for the intervention and control groups [53,55,60,61].

Regarding the results of other outcomes, most of the included studies in this systematic review have significant results in a functional capacity. Four studies [54,56–58] were significant between group results in favor of the web-based intervention group and three studies were significant among group improvements in a functional capacity after intervention for the web-based group [53,55,61]. Studies intended to demonstrate the non-inferiority of web-based support intervention found similar functional capacity results for the intervention and control groups.

Nguyen et al. [55] showed a significant improvement in dyspnea compared with the baseline in the experimental group and Wang et al. [54] showed a significance between the group's difference in dyspnea and lung function in favor of the experimental group.

#### *3.3. Risk of Bias*

The Downs and Blacks scale scores are presented in Table 1. The average score of the included studies in this systematic review was 21.6 points. In accordance with the suggested cut-off points to grade studies according to methodological quality, one article was rated as "fair" (15–19 points) [61] and eight were categorized as "good" (20–25 points) [53–60]. Figure 2 shows, in detail, the scoring of the studies on the different items of the Cochrane Risk of Bias Tool for randomized trials.

#### *3.4. Results of Meta-Analysis*

Data from seven RCTs reporting results obtained in quality of life were included in the meta-analysis [54–58,60,61]. All the included studies use the SGRQ to measure quality of life, except for the study conducted by Nguyen et al. [55] which used the CRQ.

All studies that did not provide sufficient data on quality of life (means and standard deviations at baseline or after the intervention) and for which no response was received from the authors were excluded. Ultimately, the analysis has been performed on a total of 873 patients (359 for control and 514 for intervention).

Figure 3 depicts the forest plot. Due to the statistical heterogeneity of the results (I <sup>2</sup> = 83%, *p* < 0.001), a statistical random effects model was applied. Patient quality of life was not significantly improved in the intervention groups in comparison with controls (SMD = −0.21, 95% CI = −0.56, 0.14).

When compared to usual care, the mean difference showed a significant overall effect with the addition of the web-based supportive program to usual care (SMD = −1.26, 95% CI = −1.65, −0.86; *p* < 0.001, one study [54]). When compared to a pedometer walking intervention without web-support with a web-based supportive pedometer walking intervention (SMD = −0.05, 95% CI = −0.28, 0.17; *p* = 0.64, three studies [56–58]) or a web-based supportive intervention with a face-to-face intervention (SMD = −0.03; 95% CI= −0.33, 0.26; *p* = 0.82, three studies [55,60,61]), the pooled SMD showed no significant overall effect.

**Figure 2.** Risk of bias assessment of included studies. Notes: red, high risk of bias; yellow, moderate risk of bias; green, low risk of bias.


**Table 2.**

Characteristics

 of the included studies in this systematic review.


compared with the baseline.


BDI-II, Beck Depression Inventory-II; Hospital Anxiety and Depression Scale; EQ- 5D, EuroQol 5-Dimension

Shuttle Walk Test; CAP FISIO, Respiratory

 BCKQ, Bristol COPD Knowledge

Questionnaire;

 questionnaire;

Physiotherapy

 Adherence self-report

 PRAISE, PR Adapted Index of Self-Efficacy;

questionnaire.

 MOS-SSS, Medical Outcomes Study Social Support Survey; CAT, COPD Assessment Test; HADS,

 ISWT, Incremental Shuttle Walk Test; ESWT, Endurance

**Figure 3.** Forest plot of the effect of web-based supportive interventions on quality of life in COPD patients compared with the control group. Notes: Risk of bias color: red, high risk of bias; yellow, moderate risk of bias; green, low risk of bias.

#### **4. Discussion**

The continuous technological growth of today's society, the increasing use of online services, and patients' need for new supportive solutions have facilitated the creation of new web-based interventions that have not been properly tested yet. To the authors' knowledge, this is the first systematic review and meta-analysis evaluating the effects of web-based supportive interventions on quality of life in COPD patients.

Our results support the idea that web-based supportive interventions can improve the quality of life in COPD patients. Nevertheless, it is important to note that the systematic review of the literature related to the design of web-based supportive interventions must be correctly interpreted, considering the different sample sizes of the studies, the differences in length of therapy and follow up, and the differences in effect size of the included studies.

Our systematic review is the first one specifically exploring the effects of web-based supportive interventions in quality of life in COPD patients, with nine RCTs [53–61] included in the qualitative analysis. Our results are consistent with those of previous systematic reviews performed in COPD patients and other telehealth systems [23,34,65–68].

Internet-based interventions can, however, present a rather confusing picture as the only common ground is the delivery medium. The interventions may range from posting pamphlets online to dynamic combinations of text-based information and communicative features, such as forums, "ask an expert", or multimedia tools, to individually computer tailored content [69].

Regarding web components, Sobnath et al. [70] described the possible features that a potential supporting tool for COPD patients should have in their systematic review. The tools must be easily accessible both for patients and health professionals. In addition, they should be adapted to elderly patients with limited experience in the use of technology and have a user-friendly interface. According to previous literature, the tool should include a customized education section for each patient, with disease-specific information and self-management material, phycological motivation to encourage good adherence, electronic coaching, comment sections, and social networks to share information with health professionals [70,71].

Among the web-based supportive interventions analyzed, the educational content was the most used alone or in combination with other contents, and the most frequent comparison treatment was the same in a face-to-face format. When compared, web-based supportive interventions showed similar results in all measured variables.

The web-based support interventions analyzed in this systematic review used a variety of components of COPD patient support tools that were described by Sobnath et al. [70], such as personalized education sections and social networks to share information with medical professionals. Our results are in line with the previous systematic reviews conducted in patients with cancer in which the most common and promising interventions include a combination of effective communication with healthcare providers, customized educational strategies based on the patient's disease and condition, ongoing symptom monitoring, disease self-management tools, and automated feedback [72,73].

It is difficult to determine exactly which web elements are most important in designing an effective disease management tool, and to determine whether the effects are due to one or some of the elements, or to all of them together. Effective communication with healthcare providers is highly recommended content for web-based support intervention since patients have different characteristics, preferences, and needs [62,74] as seen in the Norwegian WebChoice study [75].

A Cochrane review identified that in improving the quality of life of COPD patients, the effects of technology-based interventions attenuated over time. Support interventions based on new technologies were found to be more effective in improving the quality of life of COPD patients than interventions based on face-to-face education and support materials even at six months, but not at one year. This is probably due to the fact that educational and motivational content were not updated during the maintenance phase [67,76], highlighting the importance of these elements.

Our systematic revision of web-based interventions in COPD, have shown additional improvements in dyspnea and physical activity in programs which include selfmanagement components [54–56,58,59]. Different reviews [13,17,19,77] have reported the opportunities for telehealth interventions in increasing physical activity and symptoms when behavioral components are included.

Given the great heterogeneity and diversity of the studies included in this systematic review, it might not be recommended to perform a meta-analysis. However, a randomeffects model was chosen to allow the pooling of more clinically heterogeneous studies [78]. Furthermore, to adequately answer the question discussed in this review, i.e., whether web-based support interventions are effective in improving the quality of life of COPD patients, and due to the great diversity of the studies published to date, it was necessary to use a wide range of studies in which these types of interventions were used. It is therefore required to adequately justify our findings.

The findings of our meta-analysis of pooled data do not identify statistically significant differences in the quality of life of COPD patients. Even though the results of this metaanalysis suggested that there is no evidence that web-based support interventions are effective in increasing the quality of life of COPD patients, the results should be analyzed by subgroups.

This meta-analysis supports the promising role and the feasibility of web-based supportive interventions in COPD patients to improve quality of life when added to the usual care, reaching the currently minimum significant established difference for SGRQ results in a mean COPD sample population of −4 points [79], but not when compared to an autonomous pedometer walking intervention or face-to-face treatment. These results are in line with the increasing evidence in literature on the success of telehealth interventions [64–67].

Four included studies used wearable systems like the pedometer in the web-based supportive programs [56–59]. Those programs showed similar results in quality of life to those using autonomous interventions. Those results can be due to the theory of selfregulation [80], in which the use of a pedometer (either web-based or autonomous) guides the patient to their own feelings, thoughts, and behaviors to achieve specific goals. In addition, blinding patients from the web-based supportive pedometer walking interventions would require giving a pedometer to the control group; this may cause the results of the control group to be altered, since the simple fact of having control of their daily steps may promote an increase in the physical activity of the patients.

Other studies have used web-based pulmonary rehabilitation programs compared to the same program developed face-to-face. The results obtained by Bourne et al. [60] show no significant differences between groups in quality of life. In the study by Jiménez-Reguera et al. and Nguyen et al. [55,61], the results show statistically significant improvement on the quality of life of the web-based group, but no differences between groups after intervention.

These studies support the argument that comparable results between web-based and face-to-face interventions, or the absence of impairment can be considered a success as seen in previous reviews [23,81], due to the opportunities for new technologies for at risk COPD patients [23,82]. In this line of thinking, web-based supportive interventions may complement routine care as no significant differences were found between the face-to-face and online modalities [70]. Some further advantages should be derived from the use of telehealth interventions for this argument to be valid and the extensive literature on this topic leaves no doubt. Telehealth intervention groups show better results than the control group in risk of exacerbation [83], costs of health care [84], hospitalization days [83], risk of hospitalizations, and risks of the emergency department visit, without the need for travel [85].

Our results are consistent with the increasing evidence in the literature on the efficacy of telehealth supportive interventions [23,34,65–67]. The use of web-based supportive interventions for COPD patients is not recommended if based solely on quality of life data, but there is also no argument against the use of these interventions.

Regarding the methodological quality, the seven RCTs included in the metaanalysis [54–58,60,61] were classified as "poor quality" according to the Downs and Black quality checklist. The main reason for the low quality of the studies included in this systematic review lies in methodological issues. For example, it has been shown in previous studies that selection bias in interventions based on technological tools is evident. The reason for this is that some patients are already used to the use of new technologies and the Internet, leading to the automatic preference of these over other tools [86].

In addition, web-based interventions appear to be unsuitable for all patients because the level of follow-up and adherence to treatments is often low [87]. Other factors that also increase the risk of bias involve the lack of patient blinding and not adequately describing the randomization method.

#### *Strengths and Limitations*

To start with, we need to assess the strengths of the present study. First, only RCTs were included to increase the quality of evidence, and second, we were able to pool data from seven studies in a meta-analysis.

Thirdly, in previous studies on the effects of e-health's intervention, web-based supportive intervention was not separately analyzed. In this study, web-based supportive intervention was first taken as a primary intervention.

The major weakness of this systematic review is the limited number of RCTs focused on web-based supportive interventions. However, the inclusion criteria enabled us to include articles with this type of intervention even if quality of life was not the main variable. There are no obvious reasons for the lack of research on COPD web-based supportive interventions but the issue of possible facilitators, such as a decreased burden of web-based interventions and the personalized nature and possible barriers including security and technical issues, should be addressed when performing these types of health interventions [88].

Other limitations need to be reported. First, one subgroup in our meta-analysis only had one study. Second, it should be noted that the diversity of the targeted interventions makes it difficult to distinguish whether the web-based supportive intervention was solely responsible for the observed effects. Third, since the authors were only fluent in French, English, and Spanish, they were only able to review research published or translated into these languages and not studies in other languages.

#### **5. Conclusions**

This systematic review and meta-analysis show the promising potential of web-based supportive interventions for improving quality of life in COPD patients. Due to the methodological limitations, the heterogeneity, and the limited number of studies in this field, the results should be treated with caution. Further randomized controlled studies are needed to evaluate the effect of web-based supportive interventions, with larger COPD populations and using appropriate interventions to blind the control group, thus increasing the evidence in this field of research.

#### *Practical Implications*

Our findings suggest that the most common and promising web-based supportive intervention content are the educational content as well as communication with healthcare professionals. This systematic review and meta-analysis suggest that web-based supportive interventions may complement or accompany treatments in COPD patients due to the advantages of online interventions.

**Author Contributions:** Conceptualization: A.C.-M., L.L.-L., J.M.-N., J.R.-T. and M.C.V.; methodology: A.C.-M., A.H.-C., J.M.-N., J.R.-T. and M.C.V.; software: A.C.-M., A.H.-C. and A.O.-R.; validation: A.C.-M., L.L.-L. and A.H.-C.; formal analysis: A.C.-M. and A.O.-R.; investigation: A.C.-M., L.L.-L., J.M.-N.; resources: A.C.-M.; data curation: A.C.-M. and A.O.-R.; writing—original draft preparation: A.C.-M., M.C.V. and L.L.-L.; writing—review and editing: A.C.-M., J.R.-T. and M.C.V.; visualization: L.L.-L., A.H.-C. and J.M.-N.; supervision: M.C.V.; project administration: M.C.V.; funding acquisition: M.C.V. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Ministerio de Ciencia, Innovación y Universidades, grant number FPU: 19/02609, FPU: 17/00408 and FPU: 16/01531 and by University of Granada, grant number FPU: PP20/05.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**

("Chronic Obstructive Lung Disease" OR Airflow Obstruction, Chronic" OR "Chronic Airflow Obstructions" OR "Chronic Airflow Obstruction" OR "COPD" OR "Chronic Obstructive Pulmonary Disease" OR "COAD" OR "Chronic Obstructive Airway Disease" OR "Airflow Obstructions, Chronic") AND ("eHealth" OR "ehealth" OR "e-Health" OR "ehealth" OR "telemedicine" OR "tele-medicine" OR "Mobile Health" OR "Health, Mobile" OR "mHealth" OR "m-Health" OR "m-health" OR "telehealth" OR "tele-health" OR "telecare" OR "tele-care" OR "telemonitoring" OR "tele-monitoring" OR "teleconsultation" OR "tele-consultation" OR "health informatics" OR "internet" OR "mobile") AND ("Life Quality" OR "Health-Related Quality Of Life" OR "Health Related Quality Of Life" OR "HRQOL" OR "quality of life" OR "management" OR "adherence" OR "healthy lifestyle" OR "well-being").

#### **References**


## *Article* **Eliciting Requirements for a Diabetes Self-Management Application for Underserved Populations: A Multi-Stakeholder Analysis**

**Samuel Bonet Olivencia 1, Arjun H. Rao 1, Alec Smith <sup>1</sup> and Farzan Sasangohar 1,2,\***


**Abstract:** Medically underserved communities have limited access to effective disease management resources in the U.S. Mobile health applications (mHealth apps) offer patients a cost-effective way to monitor and self-manage their condition and to communicate with providers; however, current diabetes self-management apps have rarely included end-users from underserved communities in the design process. This research documents key stakeholder-driven design requirements for a diabetes self-management app for medically underserved patients. Semi-structured survey interviews were carried out on 97 patients with diabetes and 11 healthcare providers from medically underserved counties in South Texas, to elicit perspectives and preferences regarding a diabetes self-management app, and their beliefs regarding such an app's usage and utility. Patients emphasized the need for accessible educational content and for quick access to guidance on regulating blood sugar, diet, and exercise and physical activity using multimedia rather than textual forms. Healthcare providers indicated that glucose monitoring, educational content, and the graphical visualization of diabetes data were among the top-rated app features. These findings suggest that specific design requirements for the underserved can improve the adoption, usability, and sustainability of such interventions. Designers should consider health literacy and numeracy, linguistic barriers, data visualization, data entry complexity, and information exchange capabilities.

**Keywords:** diabetes mellitus; self-management; blood glucose self-monitoring; mobile applications; medically underserved area; health literacy; telemedicine; disease management

#### **1. Introduction**

In 2020, over 30 million individuals in the United States suffered from diabetes, most (about 90%) with type 2 diabetes [1]. Rural/medically underserved areas—defined as populations with low access to primary care providers, high infant mortality, high poverty, and/or high elderly population [2]—have shown relatively poor diabetes outcomes compared to the urban/well-served areas [3]. Additionally, type 2 diabetes disproportionately affects people of certain racial and ethnic groups, many of whom may live in areas identified as rural/medically underserved [4], such as Hispanics/Latinx Americans. Recent estimates from the Centers for Disease Control showed that individuals from Hispanic/Latinx American heritage were more likely (17%) to suffer from diabetes than the non-Hispanic White population (8%) [5]. Additionally, data from the U.S. Department of Health and Human Services' Office of Minority Health revealed that diabetes was among the leading causes for mortality among the non-White population [6].

Effective self-management of diabetes can have a significant impact on health outcomes. Studies have shown that patients who received training in self-management were successful in regulating their blood glucose levels, dietary habits, and glycemic control [7–11]. However, several barriers restrict the ability of underserved patients to execute self-management effectively [12]. These include limited access to timely healthcare

**Citation:** Bonet Olivencia, S.; Rao, A.H.; Smith, A.; Sasangohar, F. Eliciting Requirements for a Diabetes Self-Management Application for Underserved Populations: A Multi-Stakeholder Analysis. *Int. J. Environ. Res. Public Health* **2022**, *19*, 127. https://doi.org/10.3390/ ijerph19010127

Academic Editors: Irene Torres-Sanchez and Marie Carmen Valenza

Received: 1 November 2021 Accepted: 17 December 2021 Published: 23 December 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

services [13], limited financial resources [14], low literacy [15–17], and geographic barriers to seeking care from providers outside their community [18]. Therefore, there is a need to investigate methods or interventions that enable self-management by identifying and addressing such barriers systematically.

Telehealth, a type of health information technology, has received special attention in recent years for improving access to health care [19], and for supporting integrated care for chronic diseases by providing patient education and information transfer between patients and providers [20]. Recent advances in mobile health (mHealth) technologies, a modality of telehealth interventions, have shown promise in mitigating barriers related to accessibility. These technologies facilitate the self-management of diabetes, including discreet, costefficient, and non-invasive tools for monitoring health conditions, and a reliable platform for interactions between healthcare providers and patients [21,22]. A recent review of 11 mHealth apps for diabetes [23] revealed common features, such as setting reminders, tracking blood glucose and hemoglobin A1c (HbA1c), medication use, physical activity, and weight which support self-management. mHealth technologies, integrated with monitoring technologies, such as glucometers and continuous glucose monitors, have also shown promise in improving healthcare delivery [24]. Additionally, the recent integration of machine learning algorithms and artificial intelligence with mHealth played a vital role in the use of the data collected by these technologies for clinical decision-making [24].

While such characteristics make mHealth a promising method to address barriers to self-management in underserved populations, there is limited research documenting guidelines or mHealth design requirements for these populations. Previous research highlights the importance of supporting different languages and cultures for the improved adoption and sustainability in underserved populations. For example, Burner et al [25] and Williams et al. [26] discussed the need for providing basic features, such as educational content [25,26], reminders [26], and user interfaces in Spanish for Hispanic/Latinx users. In addition, glucometer connectivity functionalities [26], and the personalization of messages and content [25] are discussed as features that are important for users in underserved populations; however, such features are typically lacking in those apps available in the market [26]. Low health literacy and eHealth literacy have been identified as potential challenges for the sustained engagement of vulnerable populations with electronic, mobile, and telehealth tools; a systematic review shows that these factors have been underassessed in the published literature about the design of mobile interventions [27]. Additionally, research revealed that paid mobile apps are more likely to integrate strategies to engage low health literate populations, in comparison with free mobile apps [28]; however, cost has been identified as a major concern for people to download and adopt mHealth apps [29], and financial barriers can restrict underserved populations' self-management of their chronic conditions [14,30]. Age has been another factor highlighted in the literature affecting mHealth usage and adoption. While younger individuals have been identified as more likely to engage with mHealth apps, it is vital to assess design consideration for the elderly population [29]. For example, the use of simple, actionable, and information rich visualizations can help to address some of the design limitations of the low health literate and elderly population [31]. The patients' intrinsic level of motivation has also been linked to vulnerable the populations' level of engagement with mobile interventions [27]. Research has shown the need to apply design techniques, such as sequential multiple assignment randomized trials (SMART), to tailor self-monitoring mobile interventions to the patients' individual level of internal motivation [32].

Despite the evidence suggesting users' preferences for personalized nutritional and health behavior content [32], research [33] highlighted the lack of personalized feedback and significant usability issues, including ease of data entry and integration with patients and electronic health records, suggesting a potential gap in user-centered design (UCD) approaches. Indeed, usability tests on eight mHealth apps for diabetes revealed that more than two-thirds (6/8) were scored by patients as "marginal" or "not acceptable" [23]. This is supported by another study, in which about half of the participants reported stopping

their use of mHealth apps due to a high data entry burden and loss of interest, among other factors [29]. UCD has shown promise in fostering user engagement and improving the perception of app effectiveness, with positive impact on sustained behavioral change [34,35]. To our knowledge, only a few attempts have been documented to utilize UCD to inform requirements for diabetes self-management apps (e.g., [36]) and no research has focused on the needs and expectations of the underserved. To address this gap, in this paper, we document the stakeholders' needs and expectations from a diabetes self-management app, by eliciting feedback from patients with diabetes and providers in several medically underserved areas in the United States.

#### **2. Methods**

Semi-structured interviews were conducted with a convenience sample of 97 patients and 11 healthcare providers from several medically underserved counties in South Texas. The interviews with patients were conducted by four nurse educators with graduate degrees in nursing or education during diabetes self-management education sessions, held between 8 April 2019 and 3 May 2019, as part of the Healthy Texas initiative. These education sessions aimed at educating patients with diabetes on practical strategies and tips for incorporating healthy behaviors in their daily activities, including effective nutrition, general health and wellness, the role of physical activity, and ways to mitigate the financial and physical burden of diabetes. Participants were informed about the study at the end of educational sessions and were selected if they met the participation criteria (aged 18+ and had diabetes). The authors FS and AR, who held doctoral degrees in Engineering and had extensive experience in qualitative research, provided these interviewers with training on conducting interviews. Providers were recruited and interviewed by the authors AR, SB, and FS during a diabetes education conference in South Texas. The research group used a booth at the conference exhibition room to recruit the participants. The study team did not establish a relationship with the participants prior to the study and no one other than the interviewers were present at the interview sessions. No potential participant refused or withdrew mid-study, and no repeated interviews were carried out.

Two interview protocols were developed for patients and providers, respectively, to reduce individual biases and assumptions and to standardize the interviews. Interviews with patients focused on understanding their expectations from a diabetes self-management app. The questions in the interview protocol for patients included topics, such as perceived barriers and limitations for diabetes self-management, the use of technology to manage diabetes, important characteristics in a technology for diabetes self-management, and preferences on features for an app for diabetes self-management. Similarly, interviews with providers focused on their expectations from a self-management app for diabetes both from the patients' perspective and the type of information or interactions providers expected from such a tool. The questions in the interview protocol for providers included topics, such as perceived barriers and limitations for patients to adopt and app for diabetes self-management, perceived barriers and limitations for providers to monitor patient who have adopted such technology, perceived importance on feature for an app for diabetes self-management, and preferences about data representation and data communication. The interviews took approximately 45 min for both the patients and providers. The patients and providers received a USD 25 or USD 50 gift card, respectively, for participation. The Texas A&M University Review Board reviewed and approved this study (IRB Protocol #IRB2018-1503D) and all participants provided informed consent.

The interviews were audio recorded and no field notes were made by the interviewers during or after the interviews. A transcription service, Temi, was used to transcribe the audio recorded interviews preceding analysis [37]. The thematic analysis of the interviews was conducted by two coders (AS and AR) [38,39]. The two coders completed the following steps, separately and sequentially, and then met to discuss any discrepancies: code creation, initial coding, and focused coding. The thematic coding process entailed a deeper discussion of the themes and constructs that emerged from the analysis. After coming to

a consensus, the themes were discussed with the other authors (SB and FS) and changes were made, as necessary. MAXQDA 12 was used to complete the analysis [40]. AS and SB were doctoral students and had extensive experience in qualitative data analysis.

#### **3. Results**

#### *3.1. Demographics of Participants*

#### 3.1.1. Patient Demographics

Table 1 presents the key demographics of the patients. A total of 100 patients participated in the interviews. After cleaning the data, removing incomplete entries, a total of 97 interviews were analyzed. The average age of the participants was 56.07 (SD = 13.10). A vast majority of the participants were Hispanic or Latinx (90%, 87/97). Most of the patients did not have a postsecondary degree, with 73.20% (71/97) of the respondents having either some college (no degree), a high school diploma, or less. Approximately half the respondents (50.51%; 49/97) had a household income of less than USD 30,000. About a fourth of participants reported not having medical insurance (24%; 23/97). A majority of the participants had type 2 diabetes (81%, 79/97). About a fourth of participants were diagnosed with diabetes within a year of the date of their participation in the study (25%, 24/97), and about 39% (38/97) of respondents reported having diabetes for more than 10 years.

**Table 1.** Demographic information of patients.



**Table 1.** *Cont.*

#### 3.1.2. Healthcare Provider Demographics

Eleven healthcare providers serving medically underserved communities in South Texas participated in the interviews. Table 2 presents the key demographics for the healthcare providers interviewed. On average, the physicians sampled had nearly 3 decades (mean = 28.86; SD = 7.75; and range: 10 to 38) of experience in their current roles. Most participants (9/11) practiced family medicine, one practiced general medicine, and one was a pediatric nurse practitioner. Two participants held leadership roles (president/CEO) in their respective organizations.


**Table 2.** Demographic information of healthcare providers.

#### *3.2. Participant Interview Themes*

Patients were asked to specify features they desired in a diabetes self-management mobile app. A total of 97 participants responded to this question. The analysis of these responses resulted in five superordinate themes: (1) logging and tracking of blood sugar readings; (2) assistance with adopting a healthy lifestyle; (3) integration with the healthcare system; (4) reminders and alerts; and (5) usability and non-invasiveness. Almost 20% of the respondents (19.58%; 19/97) indicated that they did not know what features they would expect in a diabetes self-management app.

The healthcare providers were asked a series of questions about features they believed would benefit their patients and would improve their practice. The analysis of the responses from 11 physicians resulted in 5 superordinate themes: (1) dietary logs; (2) patient diabetes education; (3) reminders and alerts; (4) information communication and presentation; and (5) patient-related challenges and barriers. These themes and associated subthemes are discussed below. The proportion of participants whose response is captured by a theme or subtheme is presented with percentage (%) and counts (xx/XX). Some subtheme counts do not total 100% because some participants had responses in multiple subthemes.

#### 3.2.1. Functional Requirements Suggested by the Patients Logging and Tracking Blood Sugar Readings

Almost a quarter (24.74%; 24/97) of the patients in our sample expressed the need to be able to track and log their blood sugar readings. Two prominent subthemes were identified from the interviews: (a) logging readings and (b) assistance and insights from the readings.

**Logging Readings**: this subtheme captures the patients' desire for the app to help them log and recall their blood sugar readings. More than half of the patients (54.17%; 13/24) who expected this feature also highlighted the need to trace back to previous readings to check their well-being.

*"Just to be able to keep track of myself* ... *or tracking my glucose* ... *without having to write it down"—P12*

**Assistance & Insights from the Readings**: some patients (16.67%; 4/24) pointed out the need to understand what the entries mean. Specifically, they indicated the need for graphical interfaces to visualize the trends of their readings. For example, *"keeping track of history* ... *so I can monitor for trends" (P19)*. The participants mentioned familiarity with similar visual trends, such as activity and expected similar visualization for sugar levels. In addition, some of the patients mentioned that descriptive statistics about their parameters would be useful in managing their condition, such *as "the daily average, and the weekly average" (P09)*.

#### Assistance with Adopting a Healthier Lifestyle

About a third (34.02%; 33/97) of the respondents indicated the need for assistance with managing their condition and adopting healthier choices, and demonstrated a willingness to learn about tips and techniques to manage their diabetes. Specifically, their responses were categorized into three subthemes: (a) diet regulation; (b) health tips; and (c) fitness and physical activity.

**Diet Regulation:** this theme captures the patients' desire to be provided with information on regulating their eating habits and food intake. Two thirds (66.67%; 22/33) of these participants wanted diet regulation features, including access to a list of the types of foods they can consume to maintain glycemic control. Furthermore, patients also wanted the app to help them to construct and adhere to a diet plan. Finally, some patients (based on the diabetes education they had received) indicated that they could benefit from having a carbohydrate "tracker".

*"Like maybe like a diet plan, things to do or not to do you know that can lower your sugars if they're high."—P21*

*"[* ... *] and a list of dos' and do not food, you know, like a list, an actual list."—P07*

*"How many carbs, I can [eat], you know, in, um, like in the mornings [* ... *or] at lunchtime I'll have a sandwich [* ... *] I think that's one of the reasons my diabetes goes up. It scares me, you know, to eat a lot of carbs."—P16*

**Health Tips:** patients frequently (39.39%; 13/33) mentioned the need to easily access health-related resources. Although there was an interest in health resources in general, patients were particularly keen on accessing specific tips about nutrition. The participants also mentioned expecting prescriptive tips when presented with abnormal blood sugar values.

*"There's a lot of things like for your heart and stuff [* ... *] there's a lot of stuff out here that we eat and we're not supposed to because it's really damaging ourselves. So, you know some advice [* ... *] give us something like that."—P11*

*"[* ... *] to see, to measure if your sugar is high or low and to explain what things you can do to lower our sugar [* ... *]"—P08*

**Fitness and Physical Activity**: several of these patients (15.15%; 5/33) indicated that, while there are several commercial apps for fitness and activity tracking, a fitness module integrated into the diabetes self-management app would be ideal, suggesting the perceived importance of the connection between physical activity and diabetes.

#### Reminders and Alerts

Some patients (11.34%; 11/97) suggested timely alerts or reminders would help them adhere to their medication regimen. Two subthemes emerged from patient responses: (a) reminders and scheduling, and (b) predictive capability.

**Reminders and Scheduling**: several of these patients (72.72%; 8/11) highlighted their busy lifestyle as a reason for forgetting to monitor their blood sugar levels. In addition to being reminded to monitor their health, patients also suggested that a scheduling tool would help them keep track of their appointments.

**Predictive Capability**: some of these patients (36.36%; 4/11) responded that they would like predictive features, such as the early detection of warning signs and monitoring trends, so they can mitigate any problems before they occur.

*"[* ... *] maybe signs to look for, like when you're going to have maybe an [hypoglycemia] episode, so like warning signs."—P39*

#### Integration with Healthcare System

A few patients in our sample (3.09%; 3/97) highlighted the need for their diabetes selfmanagement program to be integrated into their overall care system. The patients desired easy communication of diabetic parameters and progress reports with their healthcare provider.

*"[* ... *] being able to send it to the doctor, or bring a recording of the reading. That way they could keep track of it."—P06*

#### Usability and Non-Invasiveness

Several patients (13.4%; 13/97) desired a system that would be easy to use and noninvasive. This superordinate theme can be categorized into two subthemes: (a) usability and (b) non-invasiveness.

**Usability**: Several patients raised concerns about their experiences with app usability (46.15%; 6/13) and expected an app that was reliable, accurate, and easy to use.

**Non-invasiveness**: this theme captures the patients' desire for a method of reading sugar levels without having to prick themselves, as commonly required by most glucometers. Most of these patients (53.84%; 7/13) were fatigued by the constant pricking for the blood sugar measurement and desired an app that would display blood sugar readings (potentially from an implantable continuous glucose monitor).

*"Like I said, a feature that would allow you to check your glucose level without [pricking],* ... *, I mean I don't know if they can make something like that without drawing your blood."—P14*

3.2.2. Functional Requirements Suggested by Healthcare Providers

#### Dietary Logs

Healthcare providers (72.72%; 8/11) highlighted the importance of a diet/food log to keep track of what patients are consuming and to have patients engage in their treatment. Healthcare providers also suggested that the app should provide immediate feedback to the patients about the calorie density and quality of the food they are ingesting. However, providers cautioned against using food logs in clinical assessments as patients tended to be dishonest in their logs.

*"So, I mean, if they want to write it down, that's fine [* ... *] if you're assuming perfect compliance and honesty. But my experience is that most patients aren't completely honest* *with what they do. So, I guess in the ideal setting, a food log would be great. So, you can go, I see when you have that bowl of ice cream, you know, that wasn't broccoli, you know, then food log could be really important. So, I guess we could change it."—S01*

*"Food log with [* ... *] input about calorie and everything else. So, it'd be two-way [* ... *] Immediate feedback. Get pretty much immediate feedback. If they're going to go to the trouble of entering in all that food, they need to get, I don't want it just to be written down, you know, and just stored somewhere and they look it up. They're going to enter what they're going to eat in a food log. They need to get immediate feedback about the calories or, and this is on or off their diet or something like that."—S08*

One physician alluded to the use of image processing and machine learning techniques to analyze a photograph of a plate of food. The results of the analysis provide a breakdown of calorie content and nutritional value.

*"I thought it would be fantastic if a person sets their meal, their plate down, they take a photo of it. And artificial intelligence calculates the, based on the size of the plate, I mean, [* ... *], how much potatoes take up, how much the meat takes up. And it calculates [* ... *]. We load the fat amount, the protein amount [but] I don't think they have that yet."—S04*

#### Patient Diabetes Education

Some providers (27.27%; 3/11) highlighted the low health literacy of patients they treated and encouraged the creation of an educational component in the diabetes selfmanagement app for facilitating communication with educators.

*"We have to give them the information* ... *It's like a coach. This is the game plan* ... *this is how you throw the ball and all that. [You have] repetition and they get better at it."—S08*

#### Reminders and Alerts

Most providers (72.72%; 8/11) suggested that providing periodic reminders or alerts about multiple topics, including ingesting medication, diet adherence, activity reminders, and appointments, would benefit patients and help them in self-management.

*"Self-management. So yeah, you get reminders. You got to do that for them. Probably about every two hours* ... *you remind them about if your glucose is too high or too low* ... *They could do a reading* ... *to help them for self-management."—S02*

Providers also cautioned designers about the tendency for patients to develop alarm fatigue leading to ignored reminders, thus highlighting the need to remind or alert only when appropriate.

*"[* ... *] there are patients who may feel like this is getting a little [annoying], and you're going to have to see everybody [feels] a little intruded."—S08*

*"[* ... *] when you start getting emails that are 12 different things on the same subject, you just start going through them and not reading them. And that's what we're seeing. They will gloss over them."—S05*

When discussing reminders, the healthcare providers' responses could be categorized into two subthemes: (a) medication intake recall and (b) activity reminder

**Medication Intake Recall**: several providers (50%; 4/8) indicated that some of their patients had trouble recalling the nature and amount of medication taken in a specific period. Therefore, the healthcare providers believed that including an easy to use and intuitive medication reminder feature in the app would remind patients about previously ingested or impending medication.

*"Having that in the app, so they're documenting it [* ... *] I think from a provider standpoint it would be great, but from the patient standpoint, we can't get them to write it down in a book. It would have to be very simple. Like they go in and click a button or* *two, you know, have their medications, already populated and they could just go in and go click, click, click."—S07*

**Activity Reminder**: a few of the healthcare providers (37.5%; 3/8) suggested designing a feature that would help patients log their activities and remind them to exercise/stay active while giving the provider access to that information.

*"[* ... *] we need something to help them exercise on here and way of recording it. [Even though] those are already with Fitbit's and stuff, but that needs to be sent to the physician."—S04*

#### Information Communication and Presentation

Although the healthcare providers encouraged open lines of communication with their patients, they highlighted key features relating to data communication and presentation, such as synchronous and asynchronous communication and information presentation.

**Synchronous vs. Asynchronous Communication**: two providers (18.2%; 2/11) stressed the need for both providers and patients to be able to communicate and exchange information, even for a self-management tool.

*"Is there one-way or two-way communication with this app? It could be two-way. It has to be two. If it's two-way, I'd feel comfortable. If it's only one-way, it's not worth it."*

An example of a such one-way or asynchronous communication method is the use of voice notes. The providers had mixed feelings about the use of voice notes as a means of communicating with patients. While some healthcare providers (27.27%; 3/11) believed that voice notes can be beneficial to patients who were visually challenged, the majority (72.72%; 6/11) were against the use of voice notes, citing issues with understanding patient accents and dialects.

*"[* ... *] for those with really poor eyesight, it's gonna have to be a voice [recording], in their language."—S04*

*"But you know as well as I do, there's so many dialects [* ... *] word accents. Sometimes you can't understand."—S03*

The majority of providers (63.63%; 7/11) believed that sending text messages can be a useful medium to communicate specific, personalized, and urgent messages or instructions to patients, while a few preferred a chat feature.

*"[* ... *] texts would be for urgent things like too high or too low [blood sugar]."*

*"[* ... *] what I like and what I think a lot of the younger crowd would like, would be, that "chat." [* ... *] You know, if you have questions, you're gonna chat"*

**Information Presentation**: most providers were highly supportive (72.72%; 8/11) of having graphs in the mobile app. However, they remarked that some patients in their communities had challenges in comprehending the information provided in graphs and would often require the healthcare providers to describe it to them. The providers emphasized the need for the graphs to be simple, easy to read, with clearly displayed limits, and intuitive ways to warn patients about abnormal sugar levels.

*"[* ... *] People respond visibly very easily using warning colors. Green, good, red, bad. The line where yours is. Pictures and graphs are great and probably better than texts."—S08*

Some healthcare providers indicated that adding appropriate pictures can help patients understand, interpret, and potentially maintain glycemic control.

*"[* ... *] they see somebody happy; they know their blood sugars in a happy range. Uh, see some blood sugars, they, they maybe they can follow it on a chart day to day. Happy face here means they're in control. A sad face here means they're out of control."—S04*

*"[* ... *] every picture tells a story. I think they would like pictures. See where they were and where they're going."—S09*

Of the six healthcare providers who responded to this question, 50% (3/6) were cautiously optimistic about the use of tables and charts to communicate clinical data to patients, while the other half felt that the patients might be overwhelmed.

Patient-Related Challenges or Barriers

When asked about the potential to implement a diabetes self-management app in a clinical setting, healthcare providers highlighted key barriers that could impede care. Their responses were classified into the following subthemes: (a) patient literacy levels; (b) privacy concerns; and (c) lack of motivation.

**Patient Literacy Levels**: several providers (45.45%; 5/11) emphasized the diversity in education levels of the patients and questioned the patients' ability to read and interpret graphs or other information on an application. Furthermore, providers raised concerns about the patients' general readiness to use technology-based interventions, age-related usability barriers, and the language barriers.

*"Well, like I said, the people I'm going to use it on are usually older people and those people didn't grow up with technology."—S10*

*"Most of my people speak Spanish or Spanglish."—S02*

Some providers (36.36%; 4/11) indicated that many of their patients can struggle with self-management, which can require them to visit the physician (in-person) to interpret their readings. This can in turn exacerbate issues related to access and geographical barriers. They suggested integrating the app with existing technologies, such as telemonitoring, or providing means of communicating relevant information to address this barrier.

*"If they have to come to the office [* ... *] to present the data, that's a barrier. If it can be like the telehealth telemonitoring it's transmitted and that's not a barrier for them."—S04*

*"[* ... *] Transportation is a big barrier to adopt something like this. [Because] they have to get to the office. They also have looking for rides and I'm in a neighborhood, lot of people walk to my place, well [those] people have to take a bus."—S02*

**Privacy Concerns**: a few providers (18.18%; 2/11) cited privacy concerns, highlighting that patients can be unwilling to be monitored or reluctant to share data.

*"I don't know if they'd be open to doing something like that. [* ... *] most of them don't [* ... *]. They don't want something intruding on their [* ... *] autonomy I guess."—S10*

**Lack of Motivation**: a few providers (18.18%; 2/11) also indicated that some of the patients in underserved areas can lack motivation to adhere to their treatment. This concern in turn relates to sustained app use for effective adoption of a healthier lifestyle.

#### **4. Discussion**

Our qualitative investigation into the requirements for a diabetes self-management app provided rich data on the key features and functionalities for patient adoption and engagement. These data lend insights into the facilitators and barriers that can encourage or impede the patients' sustained use of a diabetes self-management app. Although there has been research on the preferences of medically underserved patients [25,26], our study adds the multi-stakeholder perspective of providers in medically underserved communities.

Our findings are consistent with the previous literature on essential features for a diabetes self-management app [41–43]. Evidence-based guidelines suggest that logging and tracking blood glucose levels are essential elements in diabetes management [44,45]. Complementing these guidelines, Chavez and colleagues stated that physical activity, nutrition, blood glucose testing, medication or insulin dosage, health feedback, and education were key diabetes management tasks [46]. Consistent with this literature, patients in our study highlighted the need for logging and monitoring blood glucose levels, including visual trends for such values, tips about health lifestyle choices (e.g., exercise and nutrition), and reminders—all key basic features in a diabetes self-management app [42,43]. In addition, patients also requested the creation of a schedule feature, which would likely help them track their medication intake. Moreover, patients emphasized their preference to have a list of the types of foods they can consume to maintain glycemic control and to have a feature that can serve as a carbohydrate "tracker." Research has shown that people with diabetes in low income and minority neighborhoods have limited access to healthy foods and limited discussions with healthcare providers about healthy eating [47]. In addition, it has been suggested that cultural factors, such as the preference for carbohydrate-rich foods, should be considered when understanding the prevalence of diabetes in Latinx/Hispanic communities in the U.S. [48]. Moreover, eating disorders are more prevalent in individuals with type 1 diabetes, in comparison to the general population [49]. Finally, while our findings did not include the desire for mental health support, previous work suggests that patients with diabetes support the inclusion of features for assistance with the psychological and emotional aspects of diabetes self-management, such as stress management and mechanisms to cope with negative emotions [43].

Patients were particularly interested in accessing educational resources to help them better self-manage their condition. Although patients interviewed in the study were part of a program that provided diabetes education, our results indicate their preference to have access to dietary tips and educational content in a mHealth app. Such a tool can remove the barriers related to access to educational resources and can serve as a central repository of educational information that patients can access on-demand with tailored content based on their preferences. The request for educational content was also supported by the healthcare providers in our sample, which highlights a potential gap and unmet need in existing apps, with education among the most underrepresented features [26,41]. Furthermore, it is important to note that content should be provided at levels consistent with the educational background of the population (primarily high school level or less in our sample). It is imperative that the content provided is aligned to recommendations in the literature to overcome the challenges and limitations related to the patients' literacy levels. Design implications, such as minimizing technical jargon [50–55], presenting simplified language into tangible units [50–53], explaining uncommon terms [50–53], aligning content to the patients' cultural background [50–52], and implementing visual and audible features over the use of text [50–56], must be considered when creating the educational content to be integrated in the mobile app. Williams and Schroeder [26] go on to state that the use of video-based educational material can complement text-based content since Hispanics are among the major consumers of online multimedia content [57]. Additionally, the availability of educational content in multiple languages seems to be essential to overcome language barriers, especially in those underserved communities in which patients with diabetes are predominantly non-English speakers. Finally, while participants mainly emphasized the need for educational content about healthy eating, further research is warranted to understand the perspectives about additional interventions to encourage behavioral change regarding eating habits. Participants also supported the integration of a fitness and physical activity module. However, given the availability and prevalence of apps for physical activity, further work is needed to understand the utilization of such apps among underserved populations and investigate unmet needs.

Our findings showed that patients were partial to understanding and interpreting their diabetic parameters through graphs and visualization. Although healthcare providers were generally supportive of these media, they cautioned against complex displays. Healthcare providers cautioned about the patients' literacy levels and technical literacy levels, questioning their ability to read graphs and information in the app. Additionally, research suggests that the limited health literacy influence patients sustained motivation for engaging in monitoring their condition through self-management [58]. This can suggest that the design of health graphs and visualizations should account for the users' literacy levels. This is in line with evidence suggesting the benefits to the patients' health outcomes, when interventions sensitive to low health literacy limitations are used [30]. One of the healthcare providers suggested the use of colors and imagery to convey meaning and urgency, a

view supported by the findings of Desai and colleagues, who suggested the use of a traffic light representation and facial emotions among possible visualizations of blood glucose forecasts [31]. Furthermore, while some patients indicated that they would benefit from the blood sugar forecasting capabilities in the app, research has shown that individuals with low numeracy can find it difficult to interpret the uncertainties associated with a forecast, consequently leading to disengagement [31]. The research suggests that visualizations that are simple, information-rich, convey authority, and promote actionable and learning behaviors from users are more effective in assisting users with data interpretation [31]. Additionally, providing the user with step-by-step guidance (e.g., a welcome wizard) about the different screen or features in the app can help address this problem [59]. Additionally, research has shown that difficulties using a smartphone can also impact the use of mobile apps [60]. Therefore, emphasis should be given towards orienting users to the app when they seem to be newer adopters of smartphone technology.

Patients expressed an interest in logging data and notes relating to their diabetes self-management. Healthcare providers also encouraged patients to log information, in the belief that it would keep patients engaged in their treatment. However, research has shown that a common reason for abandoning the use of apps is due the time required to enter data [61], which our providers also considered as an issue. Self-management practices can be encouraged by reducing the burden of data entry, for instance through simplified interfaces with adjustable text icon sizes to cater to individuals with visual impairments [26]. Additionally, data entry tasks could be simplified by the integration of databases and auto-fill features that help to minimize the amount of information the patients must recall and enter. Furthermore, providers highlighted the importance of synchronous and asynchronous communication capabilities; however, healthcare providers highlighted difficulties that can arise due to language barriers. Therefore, the availability of communication functions should integrate translators. Additionally, ways to effectively manage synchronous or patient-initiated communications must be explored to avoid unnecessary burden on providers, since concerns about potential changes in workload and overwhelming number of notifications have been highlighted in the literature regarding provider support of mHealth interventions for diabetes self-management [62].

In our study, patients with diabetes expressed their preference for simple instructions for healthy diets (i.e., a list of do's and don'ts for healthy eating). This finding is consistent with the results of Turchioe et al. [32] that showed low-income diabetic patients' mixed attitudes towards goal setting for dietary intake, and the general lack of positive attitudes towards personalized decision support and self-discovery. This is in contrast to findings in studies with more advantaged populations (e.g., [63]), where more detailed information on the impact of dietary consumption on glucose levels were preferred. These differences warrant further research to investigate adaptive or personalized interventions that take the user's characteristic in mind and/or provide personalization capabilities for various user types.

The synergistic interplay between remote patient monitoring (RPM) systems and smartphone apps can play a key role in assisted diabetes self-management. Patients and healthcare providers agreed that seamless integration of the app with their healthcare infrastructure can improve monitoring and managing diabetes. Studies have shown that RPM systems facilitate monitoring vital signs and allow for early detection of potentially hazardous health conditions, allowing time for provider intervention and preventing expensive hospital admissions [64]. For example, a clinician can track an insulin-dependent patient's blood sugar profile, identify hypoglycemic tendencies, and make the necessary changes to the patient's insulin dosages. In addition, a recent study demonstrated the potential of an RPM-facilitated diabetes management program, which incorporated evidence-based lifestyle interventions [65]. However, enthusiasm to adopt RPM in medically underserved communities should be tempered by the patients' access to technology, user proficiency, and training requirements [66–68].

This work has certain limitations. First, our sample did not contain medically underserved patients beyond those residing in several counties in South Texas. Therefore, the results may not be generalizable to other medically-underserved communities. A similar limitation applies to the clinicians who were interviewed in our study. To overcome this limitation, we recommend carrying out the same study in multiple medically underserved communities across Texas and the U.S. Next, our sample consisted of only those patients who were seeking diabetes education and care through TAM-HST. It is conceivable that there are other patients with diabetes in that region who can have additional design requirements or have faced additional barriers not represented here. In addition, while in this study we identified patients by their diabetes type, no information was collected about usage of insulin, especially among patients with type 2 diabetes. Future work should investigate the differences between the requirements for users who do and do not use insulin. Next, our sample size for providers is small due to overall shortage of providers in the medically underserved counties and limited access. While we reached saturation with our current sample, future work should collect the perspectives of more providers in other underserved communities across Texas and the U.S. to validate and expand our findings. Finally, a significant proportion of the patients indicated that they were not knowledgeable about smartphone or self-management apps, thereby failing to provide design features. Although this is an unfortunate scenario, it presents an opportunity to educate patients about the capabilities of mHealth in assisting with diabetes self-management. It is imperative that these patients be included in formative usability tests to gather their perspective on the app design.

#### **5. Conclusions**

This paper highlights key features and functional requirements for the design of a diabetes self-management app tailored to the underserved community. Some of our findings are consistent with previous literature on essential features for a diabetes self-management app for the general population, such as including features to log and track blood glucose levels, physical activity, nutrition, and medication and insulin dosage, in addition to including reminders and educational content about healthy lifestyle choices [41,42,46]. In addition to these commonalities, with the published literature about the general guidelines for the design of self-management apps for diabetes, our review of relevant literature and interviews with patients and physicians in some representative underserved areas suggest that specific design requirements for the underserved can improve the adoption, usability, and sustainability of such interventions. Despite the prevalence of several selfmanagement apps, the emergence of patient education as a desired feature suggests the need for designers to pay closer attention to the patients' linguistic abilities and health literacy levels. Both the patients and providers also strongly desired the use of appropriate visualizations of diabetes data. In this regard, we recommend further investigations into the types of visualizations that would facilitate the easy interpretation of diabetes data. The use of simplified interfaces, adjustable icons, databases, and auto-fill features were identified to simplify information visualization, information recall, and data entry tasks. The use of formative training and technology exposure were identified to address issues of low experience with technology and low knowledge about mHealth capabilities, which can affect adoption and sustained engagement. While our data suggest that patients in underserved communities desire educational content about healthy lifestyle choices (e.g., nutrition and exercise), it is important that this content is presented in a way that is sensitive to their social and economic limitations, cultural background, and promotes a healthy attitudes towards eating. The results from our study also provided insights into perceived patient adoption barriers, including health literacy levels, motivation, and privacy concerns. To mitigate these barriers, we recommend adopting a community-based participatory research approach to facilitate a grassroots-level education about the capabilities of the app being designed.

**Author Contributions:** Conceptualization, F.S.; methodology, F.S., A.H.R. and S.B.O.; data collection, F.S., A.H.R. and S.B.O.; formal analysis, A.H.R. and A.S.; writing—original draft preparation, A.H.R. and A.S.; writing—review and editing, F.S. and S.B.O. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded partly by the Robert J. Kleberg and Helen C. Kleberg Foundation and the National Science Foundation under Grant No. 1648451.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Texas A&M University Review Board (IRB Protocol #IRB2018-1503D).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data that support the findings of this study are available from the corresponding author, FS, upon reasonable request.

**Acknowledgments:** We thank Jacob M. Kolman, MA, research associate at Texas A&M University and scientific writer at Houston Methodist, for critical review and editing.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*International Journal of Environmental Research and Public Health* Editorial Office E-mail: ijerph@mdpi.com www.mdpi.com/journal/ijerph

MDPI St. Alban-Anlage 66 4052 Basel Switzerland

Tel: +41 61 683 77 34 Fax: +41 61 302 89 18

www.mdpi.com