**Information Technology's Role in Global Healthcare Systems**

Editors

**Martin Wiesner Bj¨orn Schreiweis**

MDPI Basel Beijing Wuhan Barcelona Belgrade Manchester Tokyo Cluj Tianjin

*Editors* Martin Wiesner Department Medical Informatics Heilbronn University Heilbronn Germany

Bjorn Schreiweis ¨ Institute for Medical Informatics and Statistics Kiel University and University Hospital Schleswig-Holstein Kiel Germany

*Editorial Office* MDPI St. Alban-Anlage 66 4052 Basel, Switzerland

This is a reprint of articles from the Special Issue published online in the open access journal *International Journal of Environmental Research and Public Health* (ISSN 1660-4601) (available at: www. mdpi.com/journal/ijerph/special issues/healthcare systems).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. *Journal Name* **Year**, *Volume Number*, Page Range.

**ISBN 978-3-0365-4680-3 (Hbk) ISBN 978-3-0365-4679-7 (PDF)**

© 2022 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.

## **Contents**


Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 7147, doi:10.3390/ijerph18137147 . . . **83**

#### **Silke Schwarz, David D. Martin, Arndt B ¨ussing, Olga Kulikova, Hanno Krafft and Moritz Gwiasda et al.**

Sociodemographic Characteristics and Interests of FeverApp Users Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 3121, doi:10.3390/ijerph18063121 . . . **99**

#### **Na-Eun Cho, KiHoon Hong and Jongwha Chang**

Do Market Characteristics Matter? Factors Associated with Health Information Exchange Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 11976, doi:10.3390/ijerph182211976 . **113**

#### **Adnan Muhammad Shah, Mudassar Ali, Abdul Qayyum, Abida Begum, Heesup Han and Antonio Ariza-Montes et al.**

Exploring the Impact of Linguistic Signals Transmission on Patients' Health Consultation Choice: Web Mining of Online Reviews

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 9969, doi:10.3390/ijerph18199969 . . . **123**

#### **Sabur Safi, Gerhard Danzer, Solaiman Raha, Eyyad Nassar, Frank T. Hufert and Kurt J. G. Schmailzl**

Does Being Ill Improve Acceptance of Medical Technology?— A Patient Survey with the Technology Usage Inventory

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 9367, doi:10.3390/ijerph18179367 . . . **145**

#### **Youngcheoul Kang, Nakbum Choi and Seoyong Kim**

Searching for New Model of Digital Informatics for Human–Computer Interaction: Testing the Institution-Based Technology Acceptance Model (ITAM)

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 5593, doi:10.3390/ijerph18115593 . . . **163**

#### **Claus Zippel and Sabine Bohnet-Joschko**

Rise of Clinical Studies in the Field of Machine Learning: A Review of Data Registered in ClinicalTrials.gov

Reprinted from: *Int. J. Environ. Res. Public Health* **2021**, *18*, 5072, doi:10.3390/ijerph18105072 . . . **199**

## **Preface to "Information Technology's Role in Global Healthcare Systems"**

Over the past few decades, modern information technology has made a significant impact on people's daily lives worldwide. In the field of health care and prevention, there has been a progressing penetration of assistive health services such as personal health records, supporting apps for chronic diseases, or preventive cardiological monitoring.

In 2020, the range of personal health services appeared to be almost unmanageable, accompanied by a multitude of different data formats and technical interfaces. The exchange of health-related data between different healthcare providers or platforms may therefore be difficult or even impossible. In addition, health professionals are increasingly confronted with medical data that were not acquired by themselves, but by an algorithmic "black box". Even further, externally recorded data tend to be incompatible with the data models of classical healthcare information systems.

From the individual's perspective, digital services allow for the monitoring of their own health status. However, such services can also overwhelm their users, especially elderly people, with too many features or barely comprehensible information. It therefore seems highly relevant to examine whether such "always at hand"services exceed the digital literacy levels of average citizens.

In this context, this reprint presents innovative, health-related applications or services emphasizing the role of user-centered information technology, with a special focus on one of the aforementioned aspects.

> **Martin Wiesner and Bj¨orn Schreiweis** *Editors*

## *Article* **Patients' Access to Their Psychiatric Notes: Current Policies and Practices in Sweden**

**Annika Bärkås 1,\*, Isabella Scandurra <sup>2</sup> , Hanife Rexhepi <sup>3</sup> , Charlotte Blease <sup>4</sup> , Åsa Cajander <sup>5</sup> and Maria Hägglund <sup>1</sup>**


**Abstract:** Patients' access to electronic health records (EHRs) is debated worldwide, and access to psychiatry records is even more criticized. There is a nationwide service in Sweden which offers all citizens the opportunity to read their EHR, including clinical notes. This study aims to explore Swedish national and local policy regulations regarding patients' access to their psychiatric notes and describe to what extent patients currently are offered access to them. The rationale behind the study is that current policies and current practices may differ between the 21 self-governing regions, although there is a national regulation. We gathered web-based information from policy documents and regulations from each region's website. We also conducted key stakeholder interviews with respondents from the regions and cross-regional private care providers, using a qualitative approach. The results show that 17 of 21 regions share psychiatric notes with patients, where forensic psychiatric care was the most excluded psychiatric care setting. All private care providers reported that they mainly follow the regions' guidelines. Our findings show that regional differences concerning sharing psychiatric notes persist, despite Swedish regulations and a national policy that stipulates equal care for everyone. The differences, however, appear to have decreased over time, and we report evidence that the regions are moving toward increased transparency for psychiatry patients.

**Keywords:** mental health; psychiatry; psychiatric record; psychiatric notes; patient accessible electronic health record; PAEHR; open notes; policies

#### **1. Introduction**

The use of secure web-based portals where patients can access and read their Electronic Health Record (EHR) is referred to as Patient Accessible Electronic Health Records (PAEHR). Internationally, implementation of PAEHR-services has become more widespread [1,2] but remains far from the norm. The phenomenon of sharing clinical notes or narrative visit reports with patients [3,4] is often referred to as 'open notes'. Open notes can be considered an essential part of any PAEHR. In some countries, for example, in Sweden [2], Norway [5], Finland [6], and Estonia [7], nationwide PAEHR services, including open notes, are offered to most adult citizens. In the United States, the OpenNotes movement was initiated in 2010, providing patients access to their clinical notes [8]. Since 5 April 2021, a new federal law (21st Century Cures Act) in the US mandates all health organizations to offer patients secure online access to the information—including test results, referral information, and the notes written by clinicians—housed in their EHR [9].

However, many organizations that implement PAEHRs do not share mental health notes written by psychiatry professionals or give limited access to notes from psychiatry

**Citation:** Bärkås, A.; Scandurra, I.; Rexhepi, H.; Blease, C.; Cajander, Å.; Hägglund, M. Patients' Access to Their Psychiatric Notes: Current Policies and Practices in Sweden. *Int. J. Environ. Res. Public Health* **2021**, *18*, 9140. https://doi.org/10.3390/ ijerph18179140

Academic Editors: Martin Wiesner and Björn Schreiweis

Received: 13 July 2021 Accepted: 24 August 2021 Published: 30 August 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

clinics. Research shows that the sharing of mental health notes enhances patient empowerment [2,10,11], increases the sense of control in their care [4,12–16] and augments patient autonomy [11,16].

Most of the studies investigating mental health patients' experiences of access to their mental health notes have been conducted in the USA. These studies report that mental health patients experience increased understanding of their mental health [14,15], feeling in control of their care [16], and that they enhance trust in their clinician when reading their mental health notes. Further, the studies report that mental health patients experience feelings of greater engagement, validation [16–19], and that they acquire a better awareness about potential side effects of their medications when reading their mental health notes, as well as better remembering their care plan and obtaining a greater understanding of what goes on in therapy [15]. However, some patients perceive their mental health notes as inaccurate, disrespectful, judgmental, or report being surprised by disparities between what they read and what was communicated face-to-face [15,16,18,19]. Some patients also reported feeling more worried or offended by the content in their mental health notes [13,15–17]. In addition to deficit research being conducted on patients' experiences on reading their mental health notes, patients suffering from severe mental illness (e.g., bipolar disorders, psychotic disorders, and personality disorders) are missing from the sample sizes. Furthermore, there has been scarce research into sharing mental health notes in inpatient or emergency care settings [20]. Studies focusing on other patient groups [21,22] or more general populations [2] have been performed in Sweden; however, psychiatric patients have to date received little attention.

Clinicians remain concerned that mental health patients may become anxious, confused or offended by what they read, and that making mental health notes accessible to patients will create more clinical work [15,23–27]. In a US study at the Veterans Health Administration (Washington, DC, USA), nearly 1 in 2 clinicians admitted they would be pleased if open mental health notes were discounted [24]. Similar results have been reported in a Swedish study [25]. A Norwegian study reports that 29% of clinicians in psychiatric care do not report all relevant information in the EHR when patients have access, and they keep a "shadow record" to document information they considered should be inaccessible to the patient [5]. Many psychiatry clinicians report changing their documentation due to patient access [23–26,28,29]. On the other hand, studies from the USA found that psychiatry clinicians working in outpatient settings reported greater patient engagement and perceived enhanced trust in clinicians when patients read their mental health notes [16,17].

The context of psychiatric specialist care in Sweden is viewed holistically and includes, among other things, outpatient care, inpatient care, and psychotherapy care, and often includes professionals, such as doctors, nurses, assistant nurses, psychologists, physical therapists, occupational therapists, medical secretaries, and social workers [23,25]. Therefore, the term 'psychiatry' will continuously be used in this paper instead of 'mental health'.

#### *Sharing Psychiatric Notes in Sweden*

Sweden has a decentralized healthcare system with 21 self-governing regions and private care providers spanning multiple regions. Swedish healthcare is controlled by, among other entities, the Swedish Healthcare Act (2017:30) and the Swedish Patient Act (2014:821), both highlighting the importance of care on equal terms for the entire population. The Swedish Patient Data Act (2008:355) states that the patient must have access to information about the care and treatment in order to be able to participate. Nevertheless, each region has its own policy or regulatory documents on what data patients can access in the Swedish national PAEHR service Journalen, which offers patients access to their clinical notes and see their lab results, diagnoses, referrals, medications, etc. Therefore, there are considerable differences in what health information patients have access to in Journalen, depending upon the region in which the patient has received care [2,30]. For instance, a Swedish study from 2018 reports that only 2 of the 21 regions share psychiatric notes with patients [2].

To rectify these regional policy differences, the Swedish national eHealth organization Inera, responsible for Journalen, and the Swedish Association of Local Authorities and Regions, an association consisting of the 21 regions, established the Swedish National Regulatory Framework (NRF). NRF stipulates that citizens should have direct access to all the digital health information available and the same opportunities regardless of where the citizen lives or receives care [31,32]. Inera stresses that all the regions have endorsed NRF and, therefore, that all Swedish citizens should be offered access to all health data available. Despite the self-governing regions, all the regions have agreed to deliver data from the EHRs so that all Swedish citizens' can access their health data in Journalen. This has not been the case regarding psychiatric notes, where patient access is considered particularly controversial. In light of these concerns, this study aims to explore Swedish national and local policy regulations regarding patients' access to their psychiatric notes and describe to what extent patients are offered access to their psychiatric notes. Authorities and Regions, an association consisting of the 21 regions, established the Swedish National Regulatory Framework (NRF). NRF stipulates that citizens should have direct access to all the digital health information available and the same opportunities regardless of where the citizen lives or receives care [31,32]. Inera stresses that all the regions have endorsed NRF and, therefore, that all Swedish citizens should be offered access to all health data available. Despite the self-governing regions, all the regions have agreed to deliver data from the EHRs so that all Swedish citizens' can access their health data in Journalen*.* This has not been the case regarding psychiatric notes, where patient access is considered particularly controversial. In light of these concerns, this study aims to explore Swedish national and local policy regulations regarding patients' access to their psychiatric notes and describe to what extent patients are offered access to their psychiatric notes. This study is the first overview of how Sweden's regions and some of the largest

patients can access in the Swedish national PAEHR service Journalen, which offers patients access to their clinical notes and see their lab results, diagnoses, referrals, medications, etc. Therefore, there are considerable differences in what health information patients have access to in Journalen, depending upon the region in which the patient has

To rectify these regional policy differences, the Swedish national eHealth organization Inera, responsible for Journalen, and the Swedish Association of Local

*Int. J. Environ. Res. Public Health* **2021**, *18*, x 3 of 10

This study is the first overview of how Sweden's regions and some of the largest cross-regional private care providers share psychiatric notes. From a societal perspective, it is essential to explore current differences in Sweden, whether psychiatric notes are offered to patients or not, and why these differences occur. cross-regional private care providers share psychiatric notes. From a societal perspective, it is essential to explore current differences in Sweden, whether psychiatric notes are offered to patients or not, and why these differences occur.

#### **2. Materials and Methods 2. Materials and Methods**

In this study, we used a sequential data collection and analysis process (Figure 1). First, all web-pages from the 21 regions were analyzed, focusing on information related to psychiatry health records online. When such information was found, it was collected in an excel document to obtain an overview of the data. We studied the materials, looking for differences in how the regions provide patients with access to their psychiatric notes. We identified the following categories: (1) which region and private care provider share psychiatric notes (and for which settings), (2) if notes are shared with outpatients and inpatients, (3) if signed and unsigned notes are shared, (4) if the notes are shared with immediate access or with a delay. In this study, we used a sequential data collection and analysis process (Figure 1). First, all web-pages from the 21 regions were analyzed, focusing on information related to psychiatry health records online. When such information was found, it was collected in an excel document to obtain an overview of the data. We studied the materials, looking for differences in how the regions provide patients with access to their psychiatric notes. We identified the following categories: (1) which region and private care provider share psychiatric notes (and for which settings), (2) if notes are shared with outpatients and inpatients, (3) if signed and unsigned notes are shared, (4) if the notes are shared with immediate access or with a delay.

**Figure 1.** Overview of the Method. **Figure 1.** Overview of the Method.

Second, based on the data in the excel file, we observed differences in how the regions reported their implementation. In order to validate the gathered data and to fill out the gaps where information was missing, we performed key stakeholder email interviews with representatives of the regions (*n* = 21) and private care providers (*n* = 3). The Second, based on the data in the excel file, we observed differences in how the regions reported their implementation. In order to validate the gathered data and to fill out the gaps where information was missing, we performed key stakeholder email interviews with representatives of the regions (*n* = 21) and private care providers (*n* = 3). The interview questions can be found in Appendix A. The email interviews took place from the end of May to the end of June in 2021. We used a combination of structured and semi-structured questions to confirm information from the document analysis and clarify inconsistencies or missing information. Follow-up emails with additional clarificatory questions were solicited when necessary. The method was chosen to give the respondents more freedom to check the answers to the questions and the opportunity to attach documents. The answers were mapped according to the categories into an excel file, describing each region's implementation of patients' access to psychiatric notes. Recruitment of respondents to

the key stakeholder interviews was undertaken via a closed eService used by healthcare providers who share information with patients through the PAEHR Journalen. All regions' and the three private healthcare providers' respective responsible administrators were reachable by the eService.

Finally, we returned to the policy and regulatory documents and performed a rapid qualitative analysis focusing on content related to patients' access to their psychiatry notes. Relevant answers from the email interviews were also included.

The results presented in this paper are based on the qualitative document analysis, as well as the analysis of the answers from the email interviews, explaining in detail to what extent patients have access to their psychiatric notes in Sweden.

According to Swedish legislation, this study did not require ethical approval as no sensitive data were analyzed. Nonetheless, we followed ethical guidelines with informed consent.

#### **3. Results**

In this study, we mapped which Swedish regions and private care providers shared notes with patients in psychiatric care and conducted an analysis of *how* they are shared with patients. Additionally, we analyzed which regions currently have policies or offer regulatory documents for shared notes in psychiatric care. We also investigated which regions fully complied with the NRF.

#### *3.1. Sharing Notes in Psychiatric Care*

The results show that 17 of 21 regions in Sweden share notes with patients in psychiatric care (Table 1). The four regions that currently do not share notes in psychiatric care plan to start soon. All 17 regions that offer patients in psychiatric care access to their notes share notes from adult psychiatry, and 15 of these regions also share pediatric and adolescent psychiatry notes. Moreover, three regions state they plan to make notes available from pediatric and adolescent psychiatry. A representative from these regions says: "*We are looking for a secure solution in our medical record system to exclude individual conversations with children below 13 so that they are not shown to guardians*". Another region stated that they completely exclude notes from pediatric and adolescent psychiatry.

**Table 1.** Whether psychiatric notes are shared and, in such cases, from which psychiatric care setting for each region/private care provider. Note: (Light and dark) green colour = YES we share, (light and dark) grey colour = NO sharing, and N/A = not applicable. (Region number) 1 Blekinge, 2 Dalarna, 3 Gotland, 4 Gävleborg, 5 Halland, 6 Jämtland/Härjedalen, 7 Jönköping, 8 Kalmar, 9 Kronoberg, 10 Norrbotten, 11 Skåne, 12 Stockholm, 13 Sörmland, 14 Uppsala, 15 Värmland, 16 Västerbotten, 17 Västernorrland, 18 Västmanland, 19 Västra Götaland, 20 Örebro, 21 Östergötland. (Private care provider number) 22 Capio, 23 KRY, 24 MinDoktor.

Of the 17 regions offering shared notes in psychiatry, seven regions give patients access to notes from forensic psychiatric care. Of the ten remaining regions, two do not carry any forensic psychiatric care. Two regions plan to start sharing notes from forensic psychiatric care shortly, while two regions stated that this is currently not a priority. Four of the regions stated they decided not to share forensic psychiatric clinical notes.

The three private care providers we investigated do offer care across regions and appear to follow the regions' guidelines on sharing psychiatric notes as far as possible. Two of the private care providers are predominantly online healthcare providers, meaning

patients from all over Sweden may seek digital care from them. A representative from one of the three private care providers stated the following: "*Each region specifies if we should offer patients' shared notes, but not with exact or detailed amounts of information. Notes from physicians, psychologists and nurses are shared on Journalen*", while another private care provider explained they operate via one of the regions. Two private care providers are currently sharing notes in Journalen, while the third has ongoing work to start giving patients access to notes in Journalen. None of the private care providers carry forensic psychiatric care.

All the regions that share psychiatric notes share both outpatient and inpatient psychiatric notes and both signed notes (meaning, a note signed or validated by the provider who is responsible for the information in the note, indicating that the note is correct and complete) and unsigned notes (Table 2). An unsigned note is often a note that a clinician has dictated and has then been transcribed by a medical secretary and should then be checked by the clinician to confirm it is correct, a common practice in Swedish healthcare. One of the private care providers shares outpatient and inpatient notes and signed notes only. The other private care provider, which exclusively offers primary care, consequently shares signed outpatient notes only. Eleven regions and one of the private care providers share the notes with immediate access, while one of the regions only gives outpatients immediate access and 28 days' delay to psychiatric inpatients. Four regions only share signed notes, offering immediate patient access, and unsigned notes with 14 days' delay. Three of these regions have made it clear that they plan to make unsigned notes available immediately, while one region stated: "*Major parts of adult psychiatry routinely seal the health record established at their clinic during an ongoing care session, i.e., inpatient care. The seal is then removed in connection with the discharge. Currently, this is not going to change*". One region and one private care provider have a delay of 14 days on all types of notes regarding psychiatric care.

**Table 2.** How psychiatric notes are shared in each region/private care provider. Note: (Light and dark) blue colour = YES, and N/A = not applicable. (Region number) 1 Blekinge, 2 Dalarna, 3 Gotland, 4 Gävleborg, 5 Jönköping, 6 Kalmar, 7 Kronoberg, 8 Norrbotten, 9 Skåne, 10 Stockholm, 11 Uppsala, 12 Värmland, 13 Västernorrland, 14 Västmanland, 15 Västra Götaland, 16 Örebro, 17 Östergötland. (Private care provider number) 18 Capio, 19 KRY.


#### *3.2. Regional Policies and Regulatory Documents*

All regions except one have a Digital Agenda or Development Strategy, where regional work on how to increase implementation and use of e-health solutions is included. In almost all regions, these documents highlight that patients should be offered access to their health records online. In the emails, the majority of the regions confirmed that they agreed to the NRF, which aims to give all Swedish citizens aged 16 or older access to all their health data in Journalen. Nevertheless, none of the regional policies identified in this study focus specifically on shared notes in psychiatry. However, from the key stakeholder email interviews, we received additional information from several regions about how they approach shared notes in psychiatry. Nine of the 21 regions report that information considered sensitive to the patient is not shown in Journalen. Thus, specific keyword templates are configured not to appear in Journalen via a form of keyword filtering. The keywords that often are mentioned from the regions are: "Early hypotheses," "Violence in close relationships," "Concerns about child abuse," and "Compulsory care". One region mentions that they have routines for how healthcare providers should document certain sensitive topics, such as suicide risk assessments or other similarly sensitive information.

An interesting discovery from the web-based search for regional documents included a regional instruction document on writing notes within psychiatry. The instructions stressed the importance of being accurate with time and event dates to be able to follow up if needed, that all people present during the visit must be documented in the notes, but that the name of the patient or relative should never be written in the notes. Clinicians are also urged not to use abbreviations or medical language, and not to use euphemisms even if they are common practice in medicine.

#### **4. Discussion**

This study provides the first overview of how Sweden's regions and some of the largest cross-regional private care providers share notes from psychiatry. This study shows that all 17 regions share notes from adult psychiatry and that some regions have made a firm decision that certain psychiatric care settings are excluded in Journalen. Only one region had decided not to share notes from paediatric and adolescent psychiatry settings. The reasons behind this decision, and why it differed from other regions, are unclear.

Another interesting finding is that out of five regions that have decided not to share notes from forensic psychiatric care, one region referenced the Swedish Criminal Data Act (2018:1177) in their decisions. Only this region has interpreted that this law does not allow shared notes in forensic psychiatric care, which would be interesting to study further. The four regions and one of the private care providers that currently do not share notes from psychiatric care are planning to start soon. The results also report differences in inclusion of all psychiatric care settings (adult, forensic, paediatric, and adolescent) in the regions' decisions to share psychiatric notes since some regions currently have decided not to share notes from forensic psychiatry and paediatric and adolescent psychiatry. Correspondingly, the results of the study report differences in access to shared notes in psychiatric care nationwide in Sweden, highlighting even more the impact of the decentralized healthcare system.

A study based on Miranda Fricker's concept of 'epistemic injustice' emphasizes, among other things, denying patients access to their medical records may lead to ethical wrongs [1]. According to Fricker, the sharing and production of knowledge is a valued good; as such, inequalities in access to such knowledge and to participation in knowledge formation activities constitute an ethical wrong that can lead to primary and secondary harms [33]. In the case of shared notes, it is argued that patients who are precluded from reading their notes are thereby denied opportunities to feel more in control of their care [4,12], to better understand their mental health [15], and to facilitate patient autonomy [11,16] and empowerment [11]. Failure to access notes also means that patients cannot correct errors, omissions, or inaccuracies in their records. Blease et al. [1] argue that there is growing evidence that people with psychiatric illnesses may be more vulnerable to this type of injustice, as they are often seen negatively as unable to understand or cope with the information in their clinical notes. Should epistemic injustice indeed be an accurate portrayal of what happens when patients are denied access to their psychiatric notes, this study finds evidence of systematic structural barriers to access in Sweden. However, we also note that many, and an increasing number of, patients in Sweden do have access to their clinical notes from their psychiatric care. We emphasize that aside from the risk of ethical wrongs in denying patients to participate in their care, such as reading their clinical notes, it also violates the legal, nationally developed and agreed upon NRF. Further, more research into the practice of sharing notes is needed to confirm that any risks are minimized, for example misunderstandings due to the lack of common vocabulary between clinicians and patients [15,16,18,19].

According to Essén et al. [34], Sweden has weak legislation regarding patients' access to their health records online, as no health data is required or mandatory by law to be shared. It also means that regions or health organizations are not fined if patients are not offered access to their health records, unlike in the USA [9]. Notwithstanding, today, 17 out of 21 regions offer patients access to their psychiatric notes in Journalen, a number that has steadily increased. Conceivably, the "soft regulation" NRF may contribute to why Sweden has this slow, yet positive, development, since a "soft regulation" will not cause any penalties if not followed [34].

#### *Limitations and Future Work*

One limitation of our methodology was the selection of respondents for the key stakeholder interviews. We recruited participants through the closed eService used by regional and private healthcare providers when implementing Journalen nationally, with the assumption that the regions' representatives in this group would have the knowledge to provide us answers to our questions. Most of them had a central role in working with Journalen in their region. However, they may not always have had detailed knowledge of the specific regulations regarding psychiatric notes in their respective region. If this was the case, we encouraged them to pass the questions on to the right person; however, we were not able to control whether this was necessary and if it was done.

In the study, we limited ourselves to investigating implementation policies into sharing clinical psychiatric notes rather than other data, such as laboratory results or medications. So far, most of the concerns are related to the content of the notes, and therefore we have no reason to expect stricter limitations to other types of data from psychiatric records; however, this would be interesting to explore further. We also limited the study to examining regulations regarding patients' access to clinical notes in psychiatry at a regional level. Potentially, there may be further local, more informal practices at, e.g., the hospital, or even on individual department levels that may affect patients' access to their psychiatric notes. In Norway, shadow records have, for example, been local practice in psychiatry that is not sanctioned in formal regulations [5]. In future research, other data collection methods could be applied to determine whether such local deviance from the policies also exist in Sweden.

We have chosen to focus on psychiatry in its entirety regarding shared notes and we have not looked specifically at access to psychiatric notes for patients who have severe mental illness. Does access to their psychiatric notes differ from patients who have mental illness that is not severe? However, since both outpatient and inpatient care settings are included in the analysis, and all regions also give access to inpatient psychiatry notes, we can assume that most patients with severe mental illness will (eventually) have access to their notes. Whether this leads to further workarounds or local deviations from the regulation similar to the Norwegian shadow records, we cannot say. Since there is a lack of research today focusing on patients who have severe mental illness and their online access to their psychiatric notes, this is, along with forensic psychiatric care, another exciting area for future work.

#### **5. Conclusions**

Despite the national framework NRF, which stipulates that citizens should have direct access to their entire EHR and the same opportunities regardless of where Swedish citizens live, this study enlightens that the NRF is applied differently across the regions. The enforcement is different due to the autonomous regional system in Sweden, which is observed in our results. Nonetheless, despite different enforcement, we can see that 17 of 21 regions share psychiatric notes with patients. Clinical notes from adult psychiatry are shared by 17 regions, while forensic psychiatric care is the one psychiatric care setting that is most often excluded by the regions. However, our findings demonstrate a sustained effort to implement psychiatric notes across the regions, including those that do not yet offer patients access to notes from one or more of the psychiatric care settings.

**Author Contributions:** Conceptualization, A.B., I.S., H.R., Å.C. and M.H.; methodology, A.B., I.S., H.R., Å.C. and M.H.; formal analysis, A.B., I.S., H.R., C.B. and M.H.; investigation, A.B.; data curation, A.B.; writing—original draft preparation, A.B.; writing—review and editing, A.B., I.S., H.R., C.B., Å.C. and M.H.; funding acquisition, M.H. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by NordForsk through the funding to Nordic eHealth for Patients: Benchmarking and Developing for the Future (NORDeHEALTH), project number 100477.

**Institutional Review Board Statement:** Ethical review and approval were waived for this study, due to Swedish legislation only requiring ethical review for studies involving sensitive personal data (i.e., data related to race or ethnicity, political convictions, religious or philosophical convictions, union membership, health, or sexual orientation).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** Thanks to Inera AB (www.inera.se (accessed on 26 May 2021)) for providing access to the closed eService to recruit respondents for this study. Furthermore, thanks to all respondents for their time participating in the email interview and providing essential data to this study.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **Appendix A**

The appendix presents the questions sent as key stakeholder email interviews to respondents in each region and private care provider included in this study. Questions number 1–4 were sent to the private care providers. Questions number 1–2 were excluded from the regions (who only had to answer questions 3–4). The questions are translated from Swedish to English.

	- a. IF YES, can we take part in it/them?
	- b. IF YES, please answer the following questions (numbers 3 and 4).

#### IF YES,

	- i. If only signed notes are shared, are there any plans in the future to include unsigned notes as well?

i. If yes, are there any plans in the future to remove the delay of notes?

#### IF NO,

	- i If yes, what is your timeline?
	- a. IF YES, can we take part in it/them?
	- b. IF NO, is there anything else (document) that supports you?

#### **References**


## *Review* **Telemonitoring of Real-World Health Data in Cardiology: A Systematic Review**

**Benjamin Kinast 1,\*, Matthias Lutz <sup>2</sup> and Björn Schreiweis <sup>1</sup>**


**Abstract:** Background: New sensor technologies in wearables and other consumer health devices open up promising opportunities to collect real-world data. As cardiovascular diseases remain the number one reason for disease and mortality worldwide, cardiology offers potent monitoring use cases with patients in their out-of-hospital daily routines. Therefore, the aim of this systematic review is to investigate the status quo of studies monitoring patients with cardiovascular risks and patients suffering from cardiovascular diseases in a telemedical setting using not only a smartphone-based app, but also consumer health devices such as wearables and other sensor-based devices. Methods: A literature search was conducted across five databases, and the results were examined according to the study protocols, technical approaches, and qualitative and quantitative parameters measured. Results: Out of 166 articles, 8 studies were included in this systematic review; these cover interventional and observational monitoring approaches in the area of cardiovascular diseases, heart failure, and atrial fibrillation using various app, wearable, and health device combinations. Conclusions: Depending on the researcher's motivation, a fusion of apps, patient-reported outcome measures, and non-invasive sensors can be orchestrated in a meaningful way, adding major contributions to monitoring concepts for both individual patients and larger cohorts.

**Keywords:** telemonitoring; telemedicine; telecardiology; cardiology; wearable; sensors; consumer health devices; cardiovascular disease; heart failure; atrial fibrillation

#### **1. Introduction**

Within the last decade, advances in sensor technology have made a large number of wearables and other consumer health devices ready for the market. Both leading technology companies and specialized manufacturers have acknowledged a need for affordable and accessible integrated sensor technologies for fitness and health; they are serving this trend with significant investments in the emerging market [1]. One result is a progressive penetration of these technologies into a large proportion of the general public, given that consumer health devices allow individuals to measure cardiac vital signs while working out, or to self-monitor their own health status, potentially improving an individual's health behavior [2]. As these technologies become more widespread and sophisticated, there are many potential applications and use cases; several of these involve monitoring individual patients' and entire cohorts' physiology in the context of everyday life. This potential has been recognized by both researchers and health care professionals, as remote patient monitoring opens up new sustainable ways to support and care for patients in their homes [3–5]. In particular, the field of cardiology can be considered one of the most important fields of application, as integrated sensor technologies allow a variety of use cases, following up with a patient's cardiovascular health status under real-world conditions while avoiding clinical biases such as white coat hypertension [6–8]. On the other hand, cardiovascular diseases are the leading cause of death in the European

**Citation:** Kinast, B.; Lutz, M.; Schreiweis, B. Telemonitoring of Real-World Health Data in Cardiology: A Systematic Review. *Int. J. Environ. Res. Public Health* **2021**, *18*, 9070. https://doi.org/10.3390/ ijerph18179070

Academic Editor: Paul B. Tchounwou

Received: 29 July 2021 Accepted: 26 August 2021 Published: 27 August 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

countries and, therefore, avoiding these has a huge impact on public health and the health system. For example, heart failure affects approximately 26 million people worldwide [9]. Once hospitalized, up to 25% of heart failure patients are readmitted within 30 days [10,11]. Thus, recognizing the worsening of heart failure and avoiding hospital admissions is a key quality metric for managing heart failure patients.

This also influenced the researchers of the Use Case Cardiology (UCC) of the HiGHmed [12] consortium when planning the integration of both institutional and cross-sectional heart failure (HF)-related health care data in 2017. As part of an affiliated telemonitoring (TM) study, the application of wearables in the follow-up care of HF patients is planned. The aim is to support patients and their physicians in the disease management of HF while simultaneously aggregating health data from the "black box" home setting by equipping patients with wearables, complementary devices, and patient-reported outcome measures (PROMs). The aggregated data will then be transferred into a medical data integration center and merged with the hospitals' electronic health records (EHRs) to create a longitudinal dataset of HF patients. Therefore, it is the consortium's premise to develop and deploy low-threshold state-of-the-art solutions. In doing so, our aim is to passively observe the patients' disease progression retrospectively, without requiring any additional intervention. Thus, our study focuses on the latest consumer technologies that are suitable for everyday use.

In order to obtain an overview of recent research- and technology-related developments in the field, the main objective of this systematic review is to investigate the status quo of studies monitoring patients with cardiovascular risks and patients suffering from cardiovascular diseases in a telemedical setting, using not only a smartphone-based app, but also consumer health devices such as wearables and other sensor-based devices. With this novel approach, we aim to provide a holistic perspective on telemonitoring as we take both the general organizational and technical context as well as qualitative and quantitative aspects into consideration.

#### **2. Materials and Methods**

We performed a systematic review in order to identify published articles regarding telecardiological studies using consumer health devices to monitor patient's' health status reported via a mobile app. We identified and evaluated the available literature in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [13] though no registration of the protocol was performed.

#### *2.1. Search Strategy*

We conducted this comprehensive and systematic search of five databases for literature published between 1 January 2001 and 31 March 2021. We identified relevant Englishlanguage publications searching *PubMed*, *Web of Science*, *CINAHL*, *Cochrane Library*, and *Scopus*. The mandatory keywords ("telemedicine" OR ("telecardiology" AND "cardiology" OR "cardiovascular disease") AND "app" OR "mobile application") were used for the search. We provide detailed queries in Table 1.


#### **Table 1.** Our search queries as we executed them per database.

Proper names are shown in italics.

#### *2.2. Inclusion and Exclusion Criteria*

We intended to include articles matching the following criteria: (1) primary studies dealing with (2) telemedical concepts in (3) cardiovascular disease monitoring that used (4) consumer health devices such as wearables (5) or other noninvasive sensors to (6) track patients' health data (7) with a smartphone app as a central user interface. Studies not considering both wearable- and sensor-generated data were excluded.

#### *2.3. Selection and Data Extraction*

We managed the retrieved articles of each search in the aforementioned databases with Citavi 5 (Swiss Academic Software GmbH Citavi 5 Version 5.7.1.0., Wädenswil, Switzerland). First, we removed duplicates; then, we identified relevant articles by screening all keywords, titles, and abstracts based on our selection criteria. We excluded all records that did not clearly meet the eligibility criteria. Subsequently, one experienced expert in the field of medical informatics assessed all potentially relevant and freely available full-text publications regarding the inclusion and exclusion criteria. In case of ambiguity, the articles were discussed with a second expert in the domain to decide about inclusion or exclusion. While we conducted the full-text review, we identified potentially relevant references in the first-level results based on the context.

#### *2.4. Comparison Criteria*

In order to compare the studies, we determined various comparison criteria and divided them into the three groups: (1) study protocol, (2) technical parameters, and (3) qualitative and quantitative parameters.

#### 2.4.1. Study Protocol

This group includes the framework conditions of the publications, giving an overview of the relevant studies. As this review focuses on cardiological diseases, the disease-related use cases form an important criterion together with both sample sizes and study cohort sizes, study types, and the minimum participation duration. In addition, the country in which the study was conducted, as well as the application area distinguishing between local (e.g., Munich), regional (e.g., Bavaria), or national (e.g., Germany), were selected.

#### 2.4.2. Technical Parameters

Consisting of whether study staff monitored patients by including intervention—i.e., actively intervening by adjusting a participant's treatment/therapy plan (e.g., due to changing measures or vital signs)—or without intervention—i.e., as a passive, observing

character. Additionally, the platforms on which the patients' apps were offered were included. The third and fourth technical criteria include the applied wearables and other non-wearable consumer health devices connected to the patients' apps.

#### 2.4.3. Qualitative and Quantitative Parameters

Following the group of technical parameters, this group focuses on parameters provided by (1) the patients, (2) measured via a wearable or other consumer health device, and (3) data collected in a hospital setting by a physician, including examinations and surveys. We further divided patient-reported data into patient-reported outcome measures (PROMs) and patient-generated health data (PGHD). PROMs, following the definition of Weldring et al., describe tools or instruments (e.g., standardized questionnaires) developed to ensure a valid and reliable measurement of patient-reported outcomes [14]. Accordingly, these can further be subdivided into the PROMs that (a) measure functional status, or capture (b) health-related quality of life, (c) symptoms and symptom burden, (d) personal experience of care, and (e) health-related behaviors such as anxiety and depression, as well as PROMs that cannot be assigned to any of the above-mentioned groups because they are, for example, non-disjunct, summarized into (f) others. While, according to Sharpio et al., patient-generated health data (PGHD) are defined as "health-related data including health history, symptoms, biometric data, treatment history, lifestyle choices, and other information created, recorded, gathered or inferred by or from a patient"' in this review we focus on patient data documented via an app [15]. Finally, the specific vital signs provided by wearables and other consumer health devices were also included as a criterion, while the frequency by which device-tracked parameters were captured was also taken into account.

#### **3. Results**

We identified 166 articles in our initial search (see Supplementary Materials Table S1: Databases export). After we removed duplicates, a total of 157 articles were included for the title and abstract screening process. Among these, 30 articles seemed relevant, and we performed a full-text review/evaluation, resulting in a total of 7 articles being eligible and included in the study (see Supplementary Materials Table S2: Articles excluded) [16–22]. After we did a backward reference screening, we included one additional article [23]. Finally, eight articles were included in this systematic review. The detailed selection process is illustrated as a PRISMA flow diagram in Figure 1.

The included articles describe studies with several different types of study design, such as proof-of-concept studies (2 of 8; 25.0%) [19,22], randomized controlled trials (2 of 8; 25.0%) [20,23], cluster randomized trials (1 of 8; 12.5%), longitudinal cohort studies (1 of 8; 12.5%) [16], pilot studies (1 of 8; 12.5%) [21], and screening studies (1 of 8; 12.5%) [17]. These studies were conducted in four different countries: three in the United States (37.5%) [18,19,21], two in China (25.0%) [16,17], two in Germany (25.0%) [22,23], and one in Canada (12.5%) [20]. Four studies were enrolled on national (50.0%) [16,17,19,23] and four on local (50.0%) [18,20–22] levels, while none of the included studies were conducted at the regional level. In terms of the use cases, heart failure was represented in three (37.5%) [20,22,23], cardiovascular disease in three (37.5%) [18,19,21], and atrial fibrillation in two studies (25.0%) [16,17]. The smallest study cohort comprised 10 participants. The largest study included 246,541 participants. The minimum participation duration of all eight studies ranged from 14 days to 393 days. We provide an overview of the results in Table 2.

**Figure 1.** PRISMA flow diagram of the literature screening process. **Figure 1.** PRISMA flow diagram of the literature screening process.

The included articles describe studies with several different types of study design, **Table 2.** Overview of the studies included in the systematic review, with a focus on the study protocols.


<sup>1</sup> RCT: randomized controlled trial; <sup>2</sup> CRT: cluster randomized controlled trial.

With respect to the technical characteristics of the included studies, we identified six studies following an interventional monitoring approach (75.0%) [16,17,20–23], while two studies used the applied app and technology to log patients' health status for further research (25.0%) [18,19]. The operating systems for the patient apps included Google Android and Apple iOS; the latter was used as a platform for conducting two studies (25.0%) [18,22]: one study was carried out using multiple platforms (12.5%) [19], and one relied on the use of an Android-based app (12.5%) [17]. The remaining four articles (50.0%) provided no further information about the platform(s) used [16,20,21,23]. When it comes to the wearables used, two articles stated the use of smartwatches from Apple (25.0%) [18,22], one article reported the use of a Fitbit wearable (12.5%) [21], one study relied on the use of a Withings smartwatch and Withings fitness tracker (12.5%) [19], while two articles reported the use of the Honor Band 4, the Honor Watch, and the Huawei Watch GT (25.0%) [16,17]. Two study protocols did not plan the use of any wearables (25.0%) [20,23]. Furthermore, we analyzed whether the participants were provided with other consumer health devices connected to the patients' apps. We found that five study protocols included different types of Bluetooth blood pressure monitors (62.5%) [18–21], four involved the use of Bluetooth scales (50.0%) [19–21,23], and one study each included the use of a glucometer (12.5%) [21], a sleep tracking system (12.5%) [19], an electrocardiography device (12.5%) [23] and a pulse oximeter (12.5%) [23]. Some studies used a combination of several of the aforementioned devices. Three studies did not use additional devices aside from the wearables (37.5%) [16,17,22]. We provide an overview of the results in Table 3.



<sup>1</sup> BP: blood pressure; <sup>2</sup> SpO2: oxygen saturation.

Based on previously defined groups of PROMs, we could categorize two PROMs as outcomes measuring functional statuses (2 of 17; 11.76%) [18,23], five as describing health-related quality of life (29.41%) [20,22,23], three for symptoms and symptom burden (17.65%) [16,20], one for personal experience of care (5.88%) [20], four for health-related behaviors such as anxiety and depression (23.53%) [18,19,22], and two non-disjunct PROMs (11.76%) [18,22]. The exact allocation of the PROMs can be found in the (Supplementary Materials Table S3: Categorization PROMs).

In Table 4, we provide an overview of the quantitative and qualitative parameters described within the reviewed studies. Overall, 17 PROMs could be identified, with two studies using one PROM (25.0%) [18,21], five studies using two or more types of PROMs (62.5%) [16,18,20,23], and no PROMs reported in one study (12.5%) [17].

**Table 4.** Overview of the studies included in the systematic review, with a focus on the qualitative and quantitative parameters.



#### **Table 4.** *Cont.*

<sup>1</sup> HAS-BLED: hypertension, abnormal renal/liver function, stroke, bleeding history or predisposition, labile international normalized ratio, elderly, drugs/alcohol concomitantly score.

> When it comes to PGHD, five studies (62.5%) [16–18,21,22] collected various parameters, while three studies did not foresee the documentation of any additional data by the patient (37.5%) [19,20,23]. These five studies took into account a variety of self-documented lifestyle factors, such as diet (2 of 8; 25,0%) [16,21], smoking behavior (2 of 8; 25.0%) [18,21], and alcohol use (1 of 8; 12.5%) [18]. Furthermore, therapy compliance factors such as medication adherence (6 of 8; 75.0%) [16–18,21,22] were documented, while unspecified health surveys (1 of 8; 12.5%) [18], self-reported risk factors (1 of 8; 12.5%) [18], information about cardiovascular disease history (1 of 8; 12.5%) [18], sociodemographic data (1 of 8; 12.5%) [18], atrial-fibrillationrelated hospital visits (1 of 8; 12.5%) [17], and hospitalizations (1 of 8; 12.5%) [17] were also requested to be entered into the patients' app or paper-based questionnaire. One study asked the patients to enter their blood pressure and weight manually into the app (12.5%) [22], using non-connected conventional devices. In addition to the aforementioned patient-reported data, three studies reported the assessment of laboratory parameters at the beginning and in the course of the study (37.5%) [16,20,23]. One further study used a clinical questionnaire for the collection of data by clinical staff (12.5%) [17]. One study conducted a six-minute walk test and an ECG examination by study personnel (12.5%) [22]. Based on the wearables and devices to be found in Table 3, a wide range of self-tracked parameters could be identified, including seven studies measuring the patients' heart rate (87.5%) [16–20,22,23], six studies measuring the patients' blood pressure (75.0%) [16,18–21,23], four studies asking the patients to track their weight (50.0%) [20,21,23], and two using the devices to track the daily steps or mean daily steps (25.0%) [19,21]. Finally, the device-based self-tracking of a six-minute walk test (6MWT) [22], no further described physical activity [19], and the measurement of blood glucose [21], pulse wave velocity (PWV) [19], sleep duration [19], and oxygen saturation (SpO2) [23] were each performed in one study (12.5%).

> Werhahn et al. equipped patients with the Apple Watches to measure their heart rate. They used built-in pedometer functions of both smartphones and Apple Watches to capture daily steps, calculated as an arithmetic mean of 14 days. During three planned study site visits, the device-based 6MWT was validated by simultaneously carrying out a regular 6MWT [22]. Wenger et al. report that their trial participants measured their blood glucose levels daily using a glucometer, as well as their daily steps using the Fitbit's built-in pedometer; moreover, they collected participants' blood pressure and bodyweight once a week on the same day using a Bluetooth BP monitor and weight scale [21]. Seto et al. did not use any wearables, but did use Bluetooth BP monitors and weight scales to measure heart rate, blood pressure, and bodyweight daily [20]. Modena et al. included patients already owning a Withings fitness tracker or Withings Watch and BP monitor, weight scale, or sleep-tracking system to track their participants' pulse wave velocity, blood pressure, heart rate, and bodyweight at least two days a week, while the participants' physical activity levels were captured using the built-in activity trackers on the participants' smartphones.

Additionally, Modena et al. described measuring the participants' sleep duration via a Withings smartwatch or a sleep-tracking system if available [19]. McManus et al. report that they equipped a subpopulation of their study cohort with an Apple Watch and an additional Bluetooth BP cuff to log their blood pressure weekly as well as their daily measured heart rate [18]. Guo and Wang et al. included participants owning a Huawei Watch GT, Honor Watch, or Honor Band 4 to frequently capture their heart rate every 10 min [17]. Guo and Lane et al. used the same selection of devices to capture both heart rate and blood pressure, but did not provide further information about the frequency [16]. No other consumer health devices were used in either setting described by Guo et al. [16,17]. In contrast, Koehler et al. outline the application of only non-wearable-based sensors, including ECG monitors, BP measuring devices, weighing scales, and SpO2 sensors; these four devices were used to track the participants heart rate, blood pressure, weight, and capillary oxygen saturation daily [23].

#### **4. Discussion**

This systematic review summarizes the findings of studies using a patient app as an interface to document not only different sensor-based vital signs, but also self-tracked and self-documented real-world health data, for the purpose of telemonitoring in cardiology and observational research, including cardiological telemedicine data. The results suggest that different types of commercially available wearables and other consumer health devices can be implemented in a meaningful way in order to gain major insights into health behaviors and the course of diseases in different cardiological patient cohorts.

The comparison shows that although the studies' primary focuses were different, there are many similarities, suggesting that the symbiosis of these new technologies in a cardiological context seems to be of interest to researchers worldwide. To achieve their respective objectives, all studies relied on a combination of apps and non-invasive devices. While the interventional studies' approach was to monitor the daily management of disease progression or to provide active support, preventing deterioration when serious symptoms occurred, the observational programs aimed to provide further real-world health data for medical research, improving therapies and treatments in the long term.

Furthermore, the comparison shows that the choice of non-invasive devices is crucial when it comes to monitoring either high-frequency data or snapshots of a patient's health status. This also depends on the scientific question or the context of treatment. In the studies reviewed, sensor- and app-based monitoring was implemented on the basis of various cardiological use cases, while some had intersections when it came to the PROMs or self-tracked follow up parameters collected. As vital signs such as heart rate and blood pressure or weight were taken into account by almost all of the studies reviewed, it can be assumed that these turn out to be key physiological signals to be monitored, providing initial insights into a patient's general condition. However, this is countered by the fact that the accuracy of commercial wrist-worn devices is subject to ongoing scientific debate [25–27]. From a monitoring point of view, wearables have the advantage that they can provide high-frequency streaming data while worn. Although the market for consumer health devices is rapidly evolving, the types of sensors used in commercially available wearables are still limited, e.g., blood pressure, heart rate, SpO2, electrocardiogram, or photoplethysmography; thus, the need for both further developments in current sensors (e.g., wrist-worn ECG with more leads) and new sensor technologies was also recognized in the studies examined. This is why in some studies additional consumer health devices were applied to add follow-up parameters that generally cannot yet be captured by wearables or cannot be captured with sufficient quality. Adding to this, the review found that frequent surveys of standardized PROMs via a patient app seem to be another meaningful way to assess various aspects of a patient's health status at home by adding further assessment criteria. Moreover, the digitization of PROMs seems to be a meaningful step towards a more patient-centered treatment [28,29]. While, from our point of view, for purposes of analysis, the use of structured data acquisition is to be preferred, there is much to be said

for expanding the data basis through simple surveys, such as confirmation of medication intake or documentation of dietary behavior, as practiced in some of the programs.

Werhahn et al.'s study required patients to manually enter self-measured body weight as well as other parameters into the app without fully exploiting the possibilities of automatically transferring measurements by using existing interfaces such as Bluetooth. In contrast are Seto et al.'s, Moderna et al.'s, and Koehler et al.'s approaches to reduce the hurdle for regular data transfers to the app by equipping patients with Bluetooth scales. Thus, the manual entry of patients' medical history by the patients themselves, as described by McManus et al., has potential for improvement, as this data could already be stored in the patient's EHR or personal health record (PHR). Seto et al. describe a practical example, as they explicitly mention the import of laboratory parameters—e.g., brain natriuretic peptide (BNP) levels—from their hospital's EHRs. Furthermore, Koehler et al. also took the BNP level into account, while Guo et al. took hemoglobin, liver, and renal function in both screened studies into consideration for the prediction of deterioration of the state of health. This review did not investigate whether or how laboratory parameters were transferred to the app but, again, it seems reasonable to do so by integrating the EHRs. Among all of the studies considered, Wenger et al. were the only team to use a point-of-care test, as synchronized glucometers to measure patients' blood glucose were handed to the participants. This demonstrates that further laboratory parameters, which can currently only be measured by health care professionals, could in the future also be measured in the home setting. This would add a wider range of parameters to be monitored. The general advantages of mHealth technologies consist not only of bridging time and distance, but also offer the potential to avoid resource-intensive on-site monitoring. As soon as more over-the-counter sensors for measuring laboratory parameters reach market maturity, further scientific and clinical value could be gained from their integration in monitoring concepts. However, this is yet to be evaluated in further studies.

Finally, it is important to consider the platforms used, as the review revealed that only Modena et al. took a cross-platform approach integrating real-world health data from both Android and iOS devices. In the other studies, patients were provided with a compatible smartphone, or were only eligible for study participation if they already owned a suitable device. Consequently, this automatically leads to the exclusion of potential patients with unsupported device combinations. When considering a multiplatform approach, the corresponding effort and associated resource consumption must be taken into account. While a less complex single-platform approach allows the full exploitation of features of wearables or other devices via native interfaces, a comprehensive and elaborate integration into a multiplatform application might be associated with limited access to all device features [30]. Koehler et al., for example, integrated various consumer health devices from different manufacturers, although the underlying platform was unknown to the authors.

In summary, although consumer health devices or wearables remain evolving technologies, they are already able to offer a meaningful contribution in providing a more holistic insight into cardiological patients' health status and behavior, while at the same time bridging the distance between patient and doctor.

#### *4.1. Limitations*

The results suggest that the search terms used were appropriate for the research question, but some limitations of our study should still be considered. For instance, our keywords telemedicine or telecardiology could limit the choice to studies that focused on interventional approaches, while observational studies are left out. To weaken the impact, we added the keyword mHealth to our queries. This did not provide more results, and was therefore dismissed. Furthermore, as the title/abstract filter was not applied constantly for the PubMed query, this results in a slightly larger pool of findings, which had a positive impact on the scope of our results.

In addition to the selected search terms, the challenge was to create a category scheme in which all included studies could be meaningfully presented to provide a holis-

tic overview without excluding relevant factors. Therefore, the scheme was limited to categories that are relevant from the authors' point of view. However, all information can be found in a table in the Supplementary Materials. The separation between PGHD, PROMs, and clinical parameters was also discussed and assessed in detail between the authors to accomplish it as distinctly as possible; thus, we cannot ensure that everybody would evaluate this in the same manner. Although prominent studies such as the Apple Heart Study [5,31] were not included in the literature review, we assume that our analysis covered studies in the clinical context of telecardiology. However, this indicates that there may be other studies in the field that we did not include.

#### *4.2. Outlook*

In the context of this review, we did not address the algorithms used—for example, by Guo et al. and Seto et al. to predict AF and decompensation in HF, respectively. Although there are already internationally agreed treatment standards, there is still a lack of transparent and uniform diagnostic algorithms, as these are the subject of current research. It could be of interest to investigate which cardiological therapy guidelines or standards have been used to derive rules for algorithms, and what is the status quo in cardiologic algorithm research. Thinking beyond study situations, the possibilities of regular patients contributing their self-tracked health data into their EHRs are also of interest. In addition, as we advocate the establishment of platforms through which users can donate their wearable data for public research purposes without being tied to a specific purpose, corresponding concepts could be of interest for further research.

In future studies, it seems appropriate to replace the manual documentation of sensory data (e.g., weight by integrating consumer health Bluetooth scales). Given this, suitable solutions satisfying regulatory, technical, and medical requirements will be sought. As a second improvement, the adaption of further or different questionnaires should be investigated.

#### **5. Conclusions**

In this systematic review, we evaluated different approaches conducted by various researchers in the field of cardiological patient monitoring, which applied an integrated combination of app-based surveys, wearables, and other consumer health devices. Our review shows that, depending on the researcher's motivation, a fusion of apps, PROMs, and non-invasive sensors can be orchestrated in a meaningful way, adding major contributions to monitoring concepts for both individual patients and larger cohorts. We suggest that different combinations of device-based vital-sign monitoring combined with patient-reported outcomes and the documentation of lifestyle factors can contribute further insights into patients' disease progression, therapy compliance, and general health behavior patterns. In the medium-tolong term, disease prevention will most likely depend on consumer-health-device-based cardiovascular risk monitoring as a tool to follow patients up.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/article/10 .3390/ijerph18179070/s1, Table S1: Databases export; Table S2: Articles excluded, Table S3: Categorization PROMs.

**Author Contributions:** B.K. conducted the database search, analyzed the findings, and drafted the paper. B.S. helped with developing the methodology, advised on the preparation of the inclusions, and revised the paper. M.L. contributed medical expertise and revised the paper. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by German Federal Ministry of Education and Research, grant number 01ZZ1802T. This research was supported by an intramural grant of the Faculty of Medicine of Kiel University ("Forschungsförderung 2021").

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study, in the collection, analyses, or interpretation of data, in the writing of the manuscript, or in the decision to publish the results.

#### **References**


## *Review* **Routine Health Information Systems in the European Context: A Systematic Review of Systematic Reviews**

**Francesc Saigí-Rubió 1,2,† , José Juan Pereyra-Rodríguez 3,† , Joan Torrent-Sellens 2,4 , Hans Eguia 1,5 , Natasha Azzopardi-Muscat <sup>6</sup> and David Novillo-Ortiz 6,\***


**Citation:** Saigí-Rubió, F.; Pereyra-Rodríguez, J.J.; Torrent-Sellens, J.; Eguia, H.; Azzopardi-Muscat, N.; Novillo-Ortiz, D. Routine Health Information Systems in the European Context: A Systematic Review of Systematic Reviews. *Int. J. Environ. Res. Public Health* **2021**, *18*, 4622. https:// doi.org/10.3390/ijerph18094622

Academic Editors: Martin Wiesner, Björn Schreiweis and Paul B. Tchounwou

Received: 2 March 2021 Accepted: 22 April 2021 Published: 27 April 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

**Abstract:** (1) Background: The aim of this study is to provide a better understanding of the requirements to improve routine health information systems (RHISs) for the management of health systems, including the identification of best practices, opportunities, and challenges in the 53 countries and territories of the WHO European region. (2) Methods: We conducted an overview of systematics reviews and searched the literature in the databases MEDLINE/PubMed, Cochrane, EMBASE, and Web of Science electronic databases. After a meticulous screening, we identified 20 that met the inclusion criteria, and RHIS evaluation results were presented according to the Performance of Routine Information System Management (PRISM) framework. (3) Results: The reviews were published between 2007 and 2020, focusing on the use of different systems or technologies and aimed to analyze interventions on professionals, centers, or patients' outcomes. All reviews examined showed variability in results in accordance with the variability of interventions and target populations. We have found different areas for improvement for RHISs according to the three determinants of the PRISM framework that influence the configuration of RHISs: technical, organizational, or behavioral elements. (4) Conclusions: RHIS interventions in the European region are promising. However, new global and international strategies and the development of tools and mechanisms should be promoted to highly integrate platforms among European countries.

**Keywords:** routine health information system; health management information system; health system performance

#### **1. Introduction**

High-quality data supporting health management decisions are key to effective governance, leadership, and management [1–6]. Informational support for all levels of health management enables planning, policymaking, operational management, and continuous quality improvement [2]. A health information system (HIS) is a set of components (technical, organizational, behavioral) and procedures "organized to generate information that allows improving health management decisions at all levels of the health system" [7]. When a HIS produces high-quality, timely, and reliable data, it enables health program managers to monitor, evaluate, and improve health system performance and make evidence-based decisions. This information can then aid decision making, including the prioritization of funding and the allocation of other resources, and to assess which information or sources of in-formation are missing, uncertain, or of low quality [8]. These data can be used to

system-atically explore new ideas, while formulating basic strategies to support them (WHO Eu-ropean Health Information Initiative (EHII)) [8].

Healthcare providers routinely collect data on health services, statuses, and re-sources. In turn, public health advisors, hospital and healthcare managers, and ongoing surveys of health facilities also provide information. The data provide a snapshot of the state of health, health services, and health resources. The sources of these data are generally records of services rendered, individual medical records, and records of health resources. They provide information about the health of the patients and the type of treatments and tests they receive. Other information may be collected by managers on human resources, finances, drugs, and supply systems.

Routine medical information may originate from a variety of data sources that include information related to the provision of clinical services (e.g., clinical records, laboratory, and other diagnostic systems service records) and administrative record systems of routine (e.g., staff timesheets), which can be collected during regular periods (daily, monthly, quarterly, annually). A routine health information system (RHIS, also called a health facility and community information system) is any system of data collection, distribution, and use that provides information at regular intervals that is produced through routine mechanisms to address predictable health information needs [9]. Routine data on health service delivery, utilization, and clinical outcomes are reported more frequently, but an RHIS also includes routine data sets related to other health system functions (human resources management, finance, drug and equipment supply chains, and governance and management) [1].

RHISs generate data at regular intervals (one year or less) that have been collected from public and private health facilities and institutions, and community-level healthcare posts and clinics. An RHIS effectively and efficiently supports management decision making if it produces good quality data with timely, relevant, accurate, complete, and accessible information. If this is the case, optimal impact can be achieved in health outcomes and the functioning of health systems. The data produced by RHISs allow evidence-based decisions to be made for the governance and management of health systems and services for planning, monitoring and evaluation, and quality improvement.

An effective RHIS has two main objectives: first, to produce high-quality, routine health information; and second, the effective use of routine health information for decision making [7,10,11]. The ultimate objective of an RHIS is not information for its own sake but to "improve health services management through optimal informational support" [7]. A robust RHIS can be achieved by improving data production (data quality and accessibility) or data use (the capacity and processes for effective, data-informed decision making).

Given the centrality of routine information to management decision making and the challenge of making decisions when these systems are not optimal, we need to know what works in which settings for RHISs to support health system management decisionmaking effectively [7,12–15]. Synthesized evidence from research studies that evaluated interventions to address this challenge can help offer solutions to improve RHISs, and in turn to strengthen health system management.

An RHIS mainly focuses on high-level information management (national, regional, and district levels) without obligatory feedback to lower levels such as physicians. This stratification is counterproductive because, in cases of emergencies, pandemics, or natural disasters, those health personnel are the first in contact with the population. Thus, RHIS at local, provincial or state, and national levels need to be strengthened, so they can provide relief personnel with up-to-date information for planning [16].

RHISs also can assist physicians in making evidence-based decisions to enhance the local health system's performance. Positive health outcomes can be improved with the right implementation of an RHIS at both the hospital and primary care levels [17]. An optimally functioning RHIS could remove obstacles between individual care and public health information systems, ultimately improving individuals' health statuses and strengthening the global health system with more effective and efficient management and planning.

The Performance of Routine Information System Management (PRISM) framework is an innovative approach to designing, strengthening, and evaluating RHIS performance by incorporating organizational, technical, and behavioral determinants of performance [12]. The PRISM framework identifies two main functions of an RHIS and three key domains that are influential in shaping RHIS. The two main functions of an RHIS are the production of quality data and the effective use of data for decision making. These three key domains also represent areas for improving RHIS:

Technical: Technical interventions to improve an RHIS are usually intended to improve the design and the technical aspects of the RHIS, such as the usefulness and functionality of registers and computer hardware and software.

Behavioral: Behavioral interventions aim to improve staff motivation and skills to collect, extract, and use data effectively.

Organizational: Organizational interventions are meant to strengthen organizational rules, values, and support practices aimed at building a culture of data use for decision making.

RHIS interventions can address any of the components described in the PRISM framework [12,13]. An example of using multiple data streams for disease surveillance is influenza surveillance [18].

In this systematic review, we recognize that reliable health information and data that are embedded in a fully functioning and high-quality HIS form the foundation for sound decision making in healthcare and are essential for health system policy development. The aim of this systematic review is to provide a better understanding of the requirements to improve RHIS for the management of health systems, including the identification of best practices, opportunities, and challenges in the 53 countries and territories of the WHO European region.

The article makes a new contribution, from a number of perspectives, to the literature on this topic. Firstly, a systematic review of mostly Europe-centered literature is performed, taking into consideration the multidimensional set of routine practices undertaken within the HIS context. This has involved the conceptualization and delimitation of RHIS within the HIS family. Moreover, a PRISM framework approach has been taken to the literature review. This framework is widely used in the literature on the topic of HIS technologies but is rarely used within the context of RHIS. In this regard, a set of drivers and, in particular, barriers have been identified. These barriers limit the use of RHIS and the generation of RHIS-based outputs and outcomes. Lastly, this review makes a unique contribution because it supplements the results identified in the literature in two ways. First, it analyzes the link between RHIS and new health management systems based on big data or machinelearning behavior prediction algorithms. Second, it reflects on how RHISs have helped in managing the COVID-19 pandemic.

#### **2. Materials and Methods**

#### *2.1. Search Strategy and Inclusion Criteria*

This study is a systematic review of reviews that assessed "data collection" and "health information system assessments" with a focus on routine health information systems (RHISs). The study was conducted in accordance with the AMSTAR 2 [19] checklists and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [20] to ensure the quality of the review and the methodological considerations when using existing systematic reviews. It has been conveniently registered in the PROS-PERO database with the number CRD42020207267. The risk of bias was assessed, and disagreements regarding bias and the interpretation of results were resolved by consensus discussions.

A literature search was performed using MEDLINE (accessed through PubMed), Cochrane (Cochrane Database of Systematic Reviews, Cochrane Central Register of Controlled Trials, Cochrane Methodology Register, Database of Abstracts of Reviews of Effects, National Health Service Economic Evaluation Database), EMBASE, and Web of Science electronic databases in August 2020, using the following set of keywords:

Routine Health Information Systems:

"Health informatics" [TIAB] OR "health information system \*" [TIAB] OR "hospital information system \*" [TIAB] OR "management information system \*" [TIAB] OR "ambulatory care information system \*" [TIAB] OR "clinical laboratory information system \*" [TIAB] OR "clinical pharmacy information system \*" [TIAB] OR "radiology information system \*" [TIAB] OR "medical order entry system \*" [TIAB] OR "health information management" [TIAB] OR "decision support system \*" [TIAB] OR "health information exchange" [TIAB] OR "interoperability" [TIAB] OR "information system \*" [TIAB] OR "medical informatic \*" [TIAB] OR "dental informatic \*" [TIAB] OR "health information" [TIAB] OR "nursing informatic \*" [TIAB] OR "public health informatic \*" [TIAB] OR "medical record \*" [TIAB] OR "electronic health record \*" [TIAB] OR "personal health record \*" [TIAB] OR "individual health record \*" [TIAB] OR "RHIS" [TIAB] OR "routine health information system \*" [TIAB] OR "eHealth" [TIAB] OR "e-Health" [TIAB].

WHO European region (53 countries and territories):

"Albania" [TIAB] OR "Andorra" [TIAB] OR "Armenia" [TIAB] OR "Austria" [TIAB] OR "Azerbaijan" [TIAB] OR "Belarus" [TIAB] OR "Belgium" [TIAB] OR "Bosnia and Herzegovina" [TIAB] OR "Bulgaria" [TIAB] OR "Croatia" [TIAB] OR "Cyprus" [TIAB] OR "Czechia" [TIAB] OR "Denmark" [TIAB] OR "Estonia" [TIAB] OR "Finland" [TIAB] OR "France" [TIAB] OR "Georgia" [TIAB] OR "Germany" [TIAB] OR "Greece" [TIAB] OR "Hungary" [TIAB] OR "Iceland" [TIAB] OR "Ireland" [TIAB] OR "Israel" [TIAB] OR "italy" [TIAB] OR "Kazakhstan" [TIAB] OR "Kyrgyzstan" [TIAB] OR "Latvia" [TIAB] OR "Lithuania" [TIAB] OR "Luxembourg" [TIAB] OR "Malta" [TIAB] OR "Monaco" [TIAB] OR "Montenegro" [TIAB] OR "Netherlands" [TIAB] OR "North Macedonia" [TIAB] OR "Norway" [TIAB] OR "Poland" [TIAB] OR "Portugal" [TIAB] OR "Moldova" [TIAB] OR "Romania" [TIAB] OR "Russia" [TIAB] OR "San Marino" [TIAB] OR "Serbia" [TIAB] OR "Slovakia" [TIAB] OR "Slovenia" [TIAB] OR "Spain" [TIAB] OR "Sweden" [TIAB] OR "Switzerland" [TIAB] OR "Tajikistan" [TIAB] OR "Turkey" [TIAB] OR "Turkmenistan" [TIAB] OR "Ukraine" [TIAB] OR "United Kingdom" [TIAB] OR "Uzbekistan".

The search was restricted to systematic reviews, by publication date (from 1 January 2000 up to 15 August 2020), and by publication language (English and Spanish).

#### *2.2. Study Selection*

The systematic review includes data from reviews that covered any practice targeting any component or dimension of an RHIS, with at least one component related to health services performance or management in at least one WHO European country or territory. Exclusion criteria were (1) studies written in languages other than English, and those for which the full text was not available online; and (2) conference abstracts.

Initial screening was based on titles and abstracts by three researchers (J.J.P.-R, J.T.-S., and F.S.-R.). Disagreement on bias assessment and the interpretation of results was resolved by two investigators (D.N.-O. and H.E.). Abstracts lacking information were retrieved for full-text evaluation. Subsequently, the same investigators independently evaluated full-text articles and determined eligibility. Disagreement on bias assessment and the interpretation of results was resolved by consensus discussions. Authorship, journal, and years were not blinded.

#### *2.3. Data Extraction and Quality Assessment*

Three investigators conducted data extraction following standardized criteria, and results were reviewed by two senior researchers. The following data were extracted: journal, publication year, databases searched, time period, setting, system or technology, data type and collection, intervention type, number of studies, total number and countries of patients, study design, whether a review of systematic reviews or meta-analysis or bibliometric analysis was performed, outcomes, lessons and barriers for implementation, main results, main limitations, implications: challenges and opportunities, and information systems evaluation (see Tables S1 and S2 in Supplementary Materials).

#### **3. Results 3. Results**

A flow chart of the literature search and study selection results is shown in Figure 1. The first database search resulted in 45,614 articles; the updated search resulted in 280 articles. After exclusion of duplicates, 249 articles were screened, and 196 were excluded. Full texts of 53 eligible articles were reviewed. Out of these, 33 were excluded for not meeting the criteria relating to study type, intervention, or outcome. The 20 remaining studies were included in this systematic review. A flow chart of the literature search and study selection results is shown in Figure 1. The first database search resulted in 45,614 articles; the updated search resulted in 280 articles. After exclusion of duplicates, 249 articles were screened, and 196 were excluded. Full texts of 53 eligible articles were reviewed. Out of these, 33 were excluded for not meeting the criteria relating to study type, intervention, or outcome. The 20 remaining studies were included in this systematic review.

Three investigators conducted data extraction following standardized criteria, and results were reviewed by two senior researchers. The following data were extracted: journal, publication year, databases searched, time period, setting, system or technology, data type and collection, intervention type, number of studies, total number and countries of patients, study design, whether a review of systematic reviews or meta-analysis or bibli-

*Int. J. Environ. Res. Public Health* **2021**, *18*, x 5 of 26

tems evaluation (see Tables S1 and S2 in Supplementary Materials).

*2.3. Data Extraction and Quality Assessment* 

**Figure 1.** Flow of information through the different phases of the systematic review. **Figure 1.** Flow of information through the different phases of the systematic review.

#### *3.1. Descriptive Analysis of the Systematic Reviews*

#### *3.1. Descriptive Analysis of the Systematic Reviews*  3.1.1. General Characteristics of Reviewed Papers

3.1.1. General Characteristics of Reviewed Papers The 20 systematic reviews included in our review were published between 2007 and 2020 in 12 unique journals. In these reviews, systematic literature searches were performed from 1974 to 2019, and all reviews were international (covering between 3 and 14 The 20 systematic reviews included in our review were published between 2007 and 2020 in 12 unique journals. In these reviews, systematic literature searches were performed from 1974 to 2019, and all reviews were international (covering between 3 and 14 countries). The system or technology analyzed was varied, the most frequent being general ICT systems, medical health records, automated alert and reminder systems, and support systems for clinical decision making. The most frequently applied setting on which the technology focused and aligned were hospital care, primary care, and emergency services; two studies focused on aging, and one on AIDS and hypertension.

Almost all the studies included a multidatabase search, except for Anker et al., who only searched PsychInfo [21], and Marschollek, who searched PubMed [22]. The number of studies included in the systematic reviews ranged between 4 and 99, the majority between 20 and 40 studies. Only 2 of the 20 systematic reviews also included a meta-analysis [23,24] (see Table S1 in Supplementary Materials).

The studies included in the systematic reviews were diverse. Most included both randomized and nonrandomized clinical trials, including retrospective case series, case– controls, descriptive cohorts, and qualitative studies.

#### 3.1.2. Aims

Most of the reviews aimed to analyze the impact of interventions on the outcomes of the professional (readmission), centers (drug alerts, patient decisions), or of the patients (independent aging, healthy behaviors). Some studies analyzed factors that generally influenced practitioners in the use of patient data collection applications. One article explored the barriers and facilitators in the use of health information exchange systems.

#### 3.1.3. Intervention

The reviews included studies with interventions based on different technologies or systems. Most of the studies were based on EHR and contextual patient information in intensive and emergency care [25–27]; ambulatory or primary care [25,28,29]; healthcare settings, including hospitals [30]; patient results, performance, and safety [31]; and prescription alerts via EHR [23]. Other systems evaluated were health smart homes (HSHs) and home-based consumer health (HCH) for the activity of elderly people [32], and clinical decision support for the management of AIDS [33]. Other reviews included combinations of several systems, such as CDSS, computerized provider order entry (CPOE), and electronic prescribing [24,34]. Lastly, some reviews analyzed generic RHISs [35,36] (see Table S1 in Supplementary Materials).

#### *3.2. Outcomes*

The reviews naturally showed variability in results in accordance with the variability of interventions and target populations. Studies that focused on evaluating an intervention generally show weak evidence in favor of its use. This occurs, for example, in the Arditi [29] reminder study, in which they concluded that reminders to professionals can probably improve the quality of care in various contexts and under various conditions. Even studies about some interventions, such as technologies for independent aging [32], did not find strong evidence to support the technology.

Some reviews analyzed the use of different clinical information systems in different settings [24,25,27,30,31,33,35,36]. These will likely provide the most encouraging results. Several studies conclude that using an RHIS makes it possible to improve efficiency both in management (reduction of missed appointments, waiting times, etc.) [33] and for clinicians (better communication with patients and colleagues, patient information in real time), which allows better coordination, decision making, and health outcomes [30,36].

Some studies value RHISs as administrative, public health, or epidemiology tools and also consider them useful assets for various medical specialties such as emergency and critical care in hospital medicine or primary care (GP clinics) [23,25,26,28,29,31]. Studies also analyzed communication systems between patients and healthcare workers, particularly nurses. From the results obtained, ICTs showed to improve the nurse–patient relationship and increase empowerment, knowledge, well-being, and even the state of health [37].

A study analyzed the employment of health-smart homes (HSHs) and home-based consumer health (HCH) technologies to support aging at home [32], but due to the design and quality of the studies—sample sizes, etc.—there was insufficient evidence to support the role of these systems in improving independent living in the homes of the elderly. The systems used in monitoring older adults do not adequately collect or are not designed for the purpose of being assessed by an RHIS. The collected data mostly reflects the patient's current status and is then discarded [22].

Baysari et al. used information technology as decision-making support systems integrated into EMR decisions for prescribing antibiotics [24]. These systems can help improve the use of antibiotics in the hospital environment. However, there is mixed evidence of the impact on final health outcomes such as mortality or length of stay. Great variability was

also found in the designs of the studies; therefore, more evidence is needed to conclude that these systems can help organizations improve their prescribing. Bayoumi et al. also evaluated computerized alerts to improve prescribing [23]. Analyzed results showed a reduction of adverse events and hospitalization; clinical outcomes such as reduction in hypoglycemia and optimization in the maintenance of INR in therapeutic range for anticoagulants; and finally, changes in prescription behavior, which had the most immediate impact and evidence. This means that an RHIS can also affect medical audits by validating probable errors in medication [38], laboratory results [33], undeclared medication side effects [23], etc.

The use of different software applications for data collection [28] and the hesitancy to share health data with competitors [25]—especially in countries where health systems are private—are major drawbacks in global data generation. Hence, it should be recognized that RHISs follow strong privacy and safety protections for ethical use and collection of useful information. Unfortunately, some information sources such as EHR failed to present adequate or correctly used data [25]; in some cases, doctors inputted data poorly because of low computer literacy [33]. With these problems solved, RHISs would also be useful for improving access to information by making it more visible and contextualized [27]. Educating health personnel on the correct management of EHRs could alleviate this problem [28,37].

Other initiatives, such as the development of strategic frameworks, clinical leadership that values technology skills [31,36], financial resources for training [35], and the development of strategies to overcome resistance to change in health personnel [24,34] could improve the RHIS's ability to gather better information.

Another group of interventions analyzed focused on ICTs in general, as well as the use of the internet and social media [21,22,37].

Effective RHIS function requires the interaction between physicians, technical personnel, technology, the clinical environment, and the social system to work [27], along with the correct data input, adequate policies, and leadership from key players in the system. Table S3 in Supplementary Materials shows an overview of the attributes of the dimensions of success measured in the 20 systematic reviews.

#### *3.3. Areas for Improvement for RHISs According to the PRISM Framework*

To evaluate RHIS, we used the PRISM framework. This conceptual framework hypothesizes that technical, behavioral, and organizational determinants (inputs) influence data collection, transmission, processing, and presentation (Table 1). These, in turn, influence data quality and use (outputs), which include technical, organizational, and behavioral aspects related to the effective use of information for decision making (Table 2), health system performance (outcomes), and ultimately, health outcomes that represent a health impact (Table 3) [13]. According to the three determinants of the PRISM framework that influence the configuration of RHIS (technical, organizational, or behavioral elements) we have found the following areas for improvement for RHISs.




(+) implies acting as a driver; (-) implies acting as a barrier.










39


(+) implies acting as a driver; (-) implies acting as a barrier.

Regarding the inputs, the literature review shows that there is a set of actions that could foster more efficient and effective use of RHISs. Firstly, the use of contextual frameworks or theoretical models would enable an analysis of RHIS use-related behavior to be performed. One of the problems identified in the literature is the lack of theoretical references in the explanation of RHIS acceptance by healthcare professionals [25,36]. Linked to this first element, the review has also highlighted the need for a much better connection between RHIS use and people's skills and organizations' abilities [24–27,37,39]. Relationships of complementarity between RHIS, healthcare professionals' competencies and skills, and less bureaucratic organizational forms that are better adapted to evidence-based decision making [21,31,32,34] are also especially important when it comes to fostering RHIS use. Additionally, third, from the input perspective, the literature also highlights the need to overcome the technical and technological limitations that undermine the effective use of RHISs [25,28,30]. Among such limitations are problems associated with connectivity, bandwidth, usability, and interoperability between systems [35].

Regarding RHIS use-related outputs, the review also points to a set of elements that could facilitate more effective uses and returns. Firstly, a whole set of elements linked to data management has been emphasized. The management of privacy, security, and confidentiality of RHIS health data input and output is of vital importance [25]. Within this context, the importance of developing confidentiality protocols that are compatible with the use of data for evidence-based decision making has been noted [31]. In addition, issues linked to the security and adaptability (e.g., to generational preferences [22,32,35]) of RHIS input data collection and storage devices have also been emphasized [37,40].

Regarding RHIS outcomes, the literature review also offers some relevant conclusions. First, it is important to note that, despite the importance of using RHIS to support evidencebased decision making in health systems, the available evidence on its outcomes is very limited to analyses of effectiveness in specific areas [23,33,38]. There is little evidence of findings on the effects of RHIS use for health systems as a whole [21,23,24,29,30]. Second, and taking into account the reluctance to use RHISs and the limitations of the information obtained from them, the review also highlights the need to incorporate the needs of professionals who use RHISs [27,34,36]. Once again, this leads us to the question of relationships of complementarity with people and organizations [27,39]. To ensure that RHISs have efficient and effective outcomes, it is vital to consider both healthcare professionals' digital competencies and information management skills, as well as a flattening of organizational hierarchies and "top-down" mechanisms [27,34,35].

#### **4. Discussion**

RHISs are an evolution of HISs. Much broader in scope, they are complex, nested systems for health data collection and management. The novelty of RHISs rests on two main elements: the regularity of data captured and the effective use of these data for decision making. With these two novel elements, RHISs facilitate data production and enable isolated data-driven decisions to be made. The aim is to provide support for integral decision making in healthcare through information systems containing regular, optimal data.

To evaluate RHISs, we used the PRISM framework. This conceptual framework is useful for evaluating the effectiveness of an RHIS by defining and relating its inputs, outputs, and outcomes. The PRISM framework draws a flow diagram in which:


4. Once health information has been generated and used effectively, the RHIS is ready to generate outcomes, i.e., the results of its implementation. In general, these results refer to the effectiveness of either the information system itself or the health system in general. The ultimate intention is to improve citizens' health statuses.

Through PRISM and a systematic literature review of 20 scientific articles that reviewed the literature on the various practical dimensions of HISs, we reached the following main conclusions:

#### *4.1. Inputs*

We have found four key aspects that need to be improved:

First, RHISs need to incorporate new underlying frameworks to predict behaviors for adoption and use. In this context, and with regard to modeling, it would be useful to have updates of the unified theory of acceptance and use of technology (UTAUT) [41], the theory of planned behavior (TPB) [42,43], the theory of diffusion of innovations (DOI) [44], and the theory of organization and environment (TOE) [45]. These frameworks have all been used in literature that investigates the motivations for the use of technology in various contexts, including healthcare [46–51].

Second, the usability and interoperability between RHISs and their ability to connect with each other need to be improved considerably. In addition, the choice of the HIS provider is key for their subsequent development. In this respect, health organizations must gain a better understanding of the information systems market in general and the HIS market in particular.

Third, the changing nature of information systems and technology use suggests improvement in some aspects of organizations. One example is training professionals in digital skills, in information systems in general and HIS in particular, such as training physicians to input data more accurately into the EHR. Medical professionals must prioritize developing skills in transformational leadership and management of healthcare organizations, crucial to overcoming probable resistance present in some healthcare professionals. Additionally, collaborative networks must be created between technical and healthcare professionals in the context of HISs. Furthermore, organizational culture must be developed among healthcare personnel to make evidence-based decisions in healthcare organizations, and, in particular, in the evaluation of healthcare policies. Additionally, organizations must establish investment-financing mechanisms because of the economic effort involved in developing and maintaining RHISs, including public–private partnerships and learning from the experiences of other sectors. Furthermore, it is advisable to promote connections between medical science systems and information systems and technologies. The connection between medical research and the medical device market is well developed, but the same cannot be said for the connection between medical research and HIS development. Connecting the medical research, technology, and management sectors is crucial for the efficient and useful development and implementation of RHISs. In this sense, the creation of a specific training agreement would be useful. Finally, operational groups and tasks of the data scientists office (DSO) must be incorporated into health organizations and public health policy evaluation teams.

Fourth, RHISs also have information and communication infrastructure requirements. However advanced the HIS might be, it cannot be effective within contexts with connectivity and bandwidth problems. In this regard, 5G technology offers possibilities.

#### *4.2. Outputs*

Issues of privacy, confidentiality, and security of the data generated and used in RHISs are of vital importance. RHISs should reinforce the protocols ensuring that any data obtained are used confidentially and securely, without limiting their potential to be used to improve decision making in healthcare.

The emergence of big data and data-driven management is a great opportunity for RHISs. Unlike earlier methods, big data allows initially unstructured mass data to be

collected and processed. For example, through social listening methodologies that can be matched to clinical and behavioral data, healthcare management can have access to broader, more accurate, and robust information about any dimension of health. This is especially important in the acute management of situations such as the coronavirus pandemic, where an immense amount of patient data is being recorded that cannot feasibly be reviewed manually. A good structured and reliable system could be extremely useful for the prevention of the disease in obtaining data to avoid spreading, appropriate diagnosis, and diagnostic possibilities of proven benefit. Big data is also well known as a health management tool to prevent future risks, reduce unnecessary expenses, decrease health disparities, and encourage efficient use of material (antibiotics, beds, medications, etc.) [52]. However, without the data to generate it, its use becomes aspirational. For physicians, this tool can provide valuable information to guide options for certain patients, as shown in a study-oriented on the critical patient that observed that numerous systems can predict a wide variety of health conditions [26]. However, most of these studies were single-center studies, which limited the generalizability of results and conclusions.

The combination of RHIS and big data is especially useful for the analysis and evaluation of the health problems of, and policies for, specific groups, particularly the chronically ill and elderly. We must adapt the information technology or system to the specific needs of each group. For example, chronically ill young people might prefer wearables, whereas a combination of face-to-face care and virtual follow-up would work better for older patients. Not all technologies or information systems are equally effective for the management of health problems. Consequently, the training of health personnel accompanied by the development of appropriate programs may allow the data obtained from smart homes and wearable devices to be dedicated to casual or sporadic monitoring and be a valid source of data for establishing global strategies for specific groups.

The difficulty that older adults have in handling technology is widely known, often due to unclear instructions or poor support, and hence, their perceptions of technology must be recorded to maximize and facilitate its use in their daily activities [53]. Therefore, the data obtained from smart homes, especially those where the elderly reside, become vital to evaluate because projects can be created that allow them to better manage their problems, complications, and even comorbidities. However, a study [22] that focused on elderly individuals found limitations in the technologies and also found that their main use is monitoring healthcare and not as an intermediary for information.

#### *4.3. Outcomes*

While an array of partial evidence shows how certain HISs, PHRs, or clinical decision support (CDSS) technologies and systems have positive impacts on the effectiveness of health systems, joint (multiple information systems), representative, and longitudinal evidence from population samples is very scarce. Social research into the health, organizational, and healthcare policy effects of RHIS use should be considerably expanded. It is especially important to consider the relationships of complementarity between RHIS and the technologies of the second digital wave, such as big data or data-driven management, artificial intelligence (AI) and machine learning, and collaborative platforms, among others.

The implementation of information technologies and digital systems in healthcare tends to be rejected by the general public. This rejection is linked to the generalized idea that investment in these systems is made to the detriment of investment in people who provide face-to-face care to others (the classic model of health care). RHIS implementation is often top down, and this is rejected by professionals and patient associations, who perceive that technology is being prioritized over people. Consequently, the opinions of professionals on the timing of the RHIS implementation must be incorporated and complement the launch with the necessary information technology and IS support. Crucially, the neutrality of the technology can be affected by the implementation of a specific technique. RHIS implementation should occur while considering the maximization or minimization of any foreseeable positive or negative effects. On the other hand, we must work on the permeability and connection of health organizations regarding RHIS and its justification and explanation to society about its needs and benefits for health systems. As with other information technologies and systems, the effective implementation of an RHIS is not possible unless there is general acceptance by its potential users (healthcare professionals and the general public).

RHISs represent major cultural changes for healthcare professionals and the general public. RHISs are not developed in isolation. Rather, they are a more effective instrument for organizations and healthcare policy to promote citizen empowerment regarding their own health. Empowered by multiple practices of information generation and digital communication in healthcare, citizens seek to be cared for in accordance with new criteria governing the doctor–patient relationship, which no longer needs to be the traditional passive one. At the same time, many citizens, mainly—though not solely—older ones, will still seek traditional services. RHIS can be useful for segmenting these different needs into personal categories depending on the health status of individuals and for developing different care methodologies and policies.

RHISs also offer significantly innovative and disruptive alternatives for health system organizations. Within this context, complementarity between RHIS, big data, and AI is especially important for the development of digital health platforms. Digital spaces can provide infinite possibilities for agents to connect with one another, in which the traditional separation between the roles of professional and patient becomes blurred and the limitations of place, time, and connection between equals are largely overcome. Moreover, 20th-century hospitals and primary care centers may be partially replaced by 21st-century digital health platforms. These platforms would serve as digital intermediaries between healthcare or wellness providers (not necessarily healthcare professionals), and those seeking healthcare.

#### *4.4. Efficiency of RHISs in the Prevention/Treatment of COVID-19 Transmission*

The development of an RHIS intervention would also be useful during the COVID-19 pandemic. That is because the pandemic has generated a series of new data (data related to procedures, trips, the movement of people, immigration, etc.) on top of the data already existing in health systems. Thus, through digital surveillance evidence and unstructured data profiling, this new and large amount of raw data can be turned into useful big data. These data must also be represented in RHISs in order to make better decisions and to take advantage of other data generated by digital sources (e.g., social media, train routes, Google Trends, etc.). Thus, using the data obtained by RHIS, it would be possible, for example, to examine patterns of use in selected health services. This is the case of Singapore, where the data obtained by RHIS were used to predict the health service use levels and thus better understand the pattern and magnitude of the COVID-19 effect on the use of certain services [54]. Bangladesh used prepandemic RHIS to develop a model that would predict total health service utilization, including an estimate of health service use levels if the pandemic had not occurred [54]. This should provide very important data to assess costs and develop health policies based on the results, compared to those obtained by RHIS in the current pandemic. In China, a similar system was used to quantify the effect of the COVID-19 pandemic on the use of health services. In this case, detailed monthly data were used, which included data for previous years, for the year the pandemic started, and even for the periods after the various waves of the pandemic. These analyses show that RHIS data are of great significance for timely and effective tracking of the performance of the health system in low- and middle-income countries [55].

RHISs could be useful for COVID-19 surveillance. In Bulgaria, contact tracing has been implemented by RHIS. Therefore, when someone who has had close contact with a person with confirmed COVID-19, he or she is registered and has to be tested [56].

WHO has incorporated RHIS data standards into key projects such as immunization, HIV, malaria, tuberculosis, and reproductive, maternal, newborn, child and adolescent health (RMNCAH), and continues to include other data in its own digital health package

to be able to report health data that has proven to be a key need [57], especially in the era of the COVID-19 pandemic. However, not everything is perfect in the use of RHIS in relation to COVID-19. For example, an RHIS may not be able to capture the full impact of the COVID-19 epidemic in populations that have health services that do not report data to it (nongovernmental or religious organizations), or in those in which health services are provided by the private sector. In addition, many countries (especially low- and middleincome ones) do not have systems in place for the routine assessment of data quality. These systems are often beset with data entry errors and with an inconsistent application of reporting definitions, due to a failure to use standards [58].

#### *4.5. Policy Implications*

In 2015, the WHO Regional Office for Europe developed a tool to guide the assessment of HISs and the development of a national health information strategy [59]. According to a survey of European members, it was agreed necessary and desirable to improve the integration of HISs at the national level. Better sharing of these health data allows for more and better comparative health research, international benchmarks, and national and EU-wide public health monitoring [60]. However, some countries lack the resources to implement the program properly or even specify the financial resources for the preparation of the program in the budget, which may challenge the desired integration. Participants in one study mentioned various other challenges that have different relevance to countries, such as data availability, opportunities for linking data sources, legal restrictions, technical restrictions, and institutional issues [59].

Several European nations are considered leaders in the use of electronic medical records (especially in primary care). In these, HISs have been used for much longer than in other nations of the world [61]. Nevertheless, RHISs continue to display a gap between recording, reporting, and the effective use of data; therefore, strengthening RHISs has become a global priority for tracking and addressing national health goals [62]. The operations of RHISs in low-income countries fall below the globally expected standard due to the production and use of poor quality data, or to not using high-quality data to make informed decisions [63,64]. Despite investment in RHISs in low- and middleincome countries, several problems still persist (technical, organizational, financial), thus preventing proper use of RHIS (incorrect data and nonuse of data already in the system) [65]. The use of RHISs in various low-income countries in Latin America and in Africa is associated with the most significant local public health problems, such as interventions to improve maternal and newborn health [66], or to reduce communication delays and improve quality of care via a tuberculosis laboratory information system in Peru [67]. RHIS data from the research and health policy community in Mozambique will help build sustainable long-term capabilities to manage and evaluate health conditions effectively [68]. In Ethiopia, the Ministry of Health and the Bill & Melinda Gates Foundation (Bill & Melinda Gates Foundation) launched "Operational Research and Coaching for Analysts" (ORCA) as a method for developing data collection and reporting [62].

Despite a large number of studies and reviews on HISs, contradictory results continue to be evidenced. This is because some parts of the systems are unpredictable, such as the users, the flow of information, and the settings [61]. Even if they present a number of problems, RHISs can help to strengthen policy decision making in local health systems, especially in low-income countries. Therefore, it is necessary to establish a suitable strategy based on the digitization of data processing, which allows indicator use to be simplified and reports to be saved and delivered, thus leading to a modern and effective data use structure.

To establish improvement strategies, it is necessary to know what the current problems and weaknesses in the evaluated studies are. Indeed, we can find various aspects in need of improvement within them, including the lack of a strategy for RHIS system implementation and evaluation [25,32]; the lack of financial, personnel, and equipment resources, making it impossible to correctly collect data capable of providing the best results [31,35,40]; data capture systems (software) that are not intuitive enough and require

extra training, which hinders their use [22,25,35] and causes rejection by their users (especially doctors) [24,33,35]. Moreover, the lack of interoperability between systems (medical records, databases, etc.) further complicates the proper use of data [24,25,28,36]. Without adequate planning to overcome poor communication between technical, administrative, and health personnel [32], the results obtained from such data are only able to give an overview that is of little benefit to local entities [22,27,28,30,36].

By identifying such problems, strategies can be established to solve them. These strategies should be established and grouped by the specific determinants found: technical, organizational, and behavioral [63]. The reason for doing so is that it is practically impossible to generalize an answer within a single overall strategy.

Technical strategies: The records must be simplified in a standardized way to facilitate data entry. Ideally, creating intuitive software is an excellent choice that may even increase user acceptance. The development of tools to improve the results of poor RHIS data has been described in other studies with good results [65]. These tools could even allow data availability and usability to be improved (by both uploading new data and reviewing data that is already available), possibly by using cloud storage services to enable easy access from anywhere and by having a common standard that allows for interoperability of systems in different locations.

Organizational strategies: The management of resources is essential, of both those available and those needed, and that is why it is very important to have a protocol for project implementation before any project is actually carried out. It is necessary to assess what can be achieved with the financial and human resources that are presently available, as well as the possibility of making improvements by obtaining new resources. To solve organizational problems, projects must be correctly established from the start. This should include follow-up measures (based on variables) and evaluation so that any post-evaluation improvements can be made so as to enhance the use of the data obtained. In addition, the possibility of establishing a project monitoring and evaluation director position should be considered. This is because leadership within projects such as these is essential to guide, monitor, and resolve any issues that team members may have. These teams must be made up of professionals from the various areas participating in the project since this will help to create the right tools, which should be useful to all potential users, and also be easy to use.

Behavioral strategies: These aim to improve the staff members' competence and motivation to collect, extract, and use data effectively [63]. One of the most commonly encountered problems is the participants' (mostly health practitioners') refusal to use an RHIS because they consider that using a new tool will not bring any benefits. This can be resolved through educational interventions to show the benefits of RHISs and by training staff to use them properly. These actions have been shown to improve staff members' abilities to use data [65]. Here, a leadership figure is very important for the purpose of providing guidance during project rollout. This is because workshops and educational interventions do not always achieve the expected results, whereas the combination of leadership and motivation can have a powerful behavioral and organizational impact on data improvement [63].

In order to achieve the integration of issues, it is important to clearly understand what should be integrated, how it should be integrated, what activities should be considered, and the benefits that can be obtained. Through current technological advances, certain basic information system improvements can be demonstrated (providing quality data, data recordkeeping, legislative and technical infrastructure, and personnel improvements) that promote process integration in Europe. To accomplish this, proper leadership and good management are key to improving RHIS architecture and infrastructure [54].

#### *4.6. Limitations*

Although this systematic review was conducted according to the suggested methodology, we acknowledge that our study has some limitations. We searched four databases and focused only on systematic reviews, meta-analysis, and bibliometric analysis. Consequently, our search may not be exhaustive. On the other hand, the inferior quality scores based on AMSTAR-2 tools might reflect incomplete reports rather than unqualified review methods (see Table S4 on Supplementary Materials). Finally, the large number of publications required an optimized approach. However, we have ensured transparency by clearly outlining the process followed in the Methods Section. Therefore, we expect this review will only serve as a temporary system review and can be further updated as needed.

#### **5. Conclusions**

The use and development of plans for RHIS at the national level in European countries would also be desirable at the continental level. Our research is based on a variety of available related articles, showing the possibility of coordinating work in various areas and creating integrated recommendations.

Some strategies have been developed. However, some countries in the European region are still not working in concordance with the development of RHIS, including legislatively. To alleviate this obstacle, new global and international strategies should be planned, and the development of tools and mechanisms should be promoted in order to highly integrate platforms among European countries.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/article/10 .3390/ijerph18094622/s1. Table S1. Summary of included studies' characteristics. Table S2. Summary of included studies' results. Table S3. Attribute of different success factors. Table S4. Quality assessment judgment using the AMSTAR 2 tool.

**Author Contributions:** Conceptualization, D.N.-O., N.A.-M., F.S.-R. and J.J.P.-R.; methodology, D.N.- O., F.S.-R., J.J.P.-R. and J.T.-S.; validation, D.N.-O., F.S.-R. and J.J.P.-R.; formal analysis, F.S.-R., J.J.P.-R., J.T.-S. and H.E.; writing—original draft preparation, F.S.-R., J.J.P.-R., J.T.-S. and H.E.; writing—review and editing, D.N.-O., N.A.-M., F.S.-R. and J.J.P.-R. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** No additional data available.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Abbreviations**



#### **References**


## *Article* **Role of Individual Motivations and Privacy Concerns in the Adoption of German Electronic Patient Record Apps—A Mixed-Methods Study**

**Richard Henkenjohann 1,2**


**Abstract:** Germany's electronic patient record ("ePA") launched in 2021 with several attempts and years of delay. The development of such a large-scale project is a complex task, and so is its adoption. Individual attitudes towards an electronic health record are crucial, as individuals can reject optingin to it and making any national efforts unachievable. Although the integration of an electronic health record serves potential benefits, it also constitutes risks for an individual's privacy. With a mixed-methods study design, this work provides evidence that different types of motivations and contextual privacy antecedents affect usage intentions towards the ePA. Most significantly, individual motivations stemming from feelings of volition or external mandates positively affect ePA adoption, although internal incentives are more powerful.

**Keywords:** personal electronic health records; technology adoption; endogenous motivations; health information privacy concern; mixed-methods; ePA

#### **1. Introduction**

Providing efficient healthcare has a genuine impact on society, as it directly influences people's well-being. The use of information technology (IT), and more specifically, electronic health records (EHRs), improves provisioned quality and reduces healthcare costs in general [1]. Healthcare is information-intensive, since many activities are enabled through storing, processing, and analyzing data. An EHR increases efficiency in healthcare delivery, simplifies monitoring patient health, facilitates monetary savings, reduces paper-based errors, and improves diagnoses and treatments [2–6]. Governments and healthcare systems promote national patient health records as "a way of preserving patients' health and medical information and maintaining their data in a central facility that ideally can be shared between different healthcare providers" ([7], p. 1). An EHR offers "efficiencies in collecting and storing patient information, contributing to continuity of care and alleviating problems such as misdiagnosis or prescription errors" ([7], p. 1). Patient-administered health records, often referred to as personal health records (PHRs), aim to improve this continuity of care while simultaneously realizing the right to informational self-determination in that each patient is made the owner of all disease-related data [8]. In a PHR, citizens can add valuable self-reported information to their health records and are given control over their data by, for example, deciding who can access their health records [7,9–11]. The adoption of such systems, however, is a complex task. First, the implementation of electronic patient records has to be performed on the institutional level where all care providers have to adopt all technologies [1,12]. Secondly, the adoption of PHRs has to take place on the individual level. Individual attitudes are crucial on that level, as patients can reject opting-in to the PHR or demand to opt-out from the PHR [1]. Consequently, it is vital to understand how individuals will be willing to adopt such a system [13,14]. The patient's

**Citation:** Henkenjohann, R. Role of Individual Motivations and Privacy Concerns in the Adoption of German Electronic Patient Record Apps—A Mixed-Methods Study. *Int. J. Environ. Res. Public Health* **2021**, *18*, 9553. https://doi.org/10.3390/ijerph 18189553

Academic Editor: Paul B. Tchounwou

Received: 31 July 2021 Accepted: 26 August 2021 Published: 10 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

choice in not adopting a PHR is crucial in the diffusion process [1] as patients' resistance can result in "any national efforts unachievable" ([13], p. 360). Though the integration of an electronic health record serves potential benefits, it also constitutes risks for an individual's privacy [15]. Privacy concerns remain the significant factor for patients in terms of withholding EHR adoption [13,14,16]. In particular, individual health information can easily be de-anonymized when combined [17]. Consequently, health and medical data's highly sensitive nature results in many ethical issues when establishing a nationwide electronic health record [7].

The German healthcare system is characterized by fragmented care structures that hinder cross-sectoral care of patients and can lead to additional costs for the healthcare system, such as loss of information between practitioners, duplicated examinations, and uncoordinated treatment processes [18]. Germany's electronic patient record project (*elektronische Patientenakte—ePA*) intends to overcome these barriers while increasing transparency and efficiency [19,20]. As of 1 January 2021, statutory health insurance companies in Germany have been obligated to offer electronic patient records to their insureds (§ 341 German Social Code, Book V). In an early study, Hoerbst et al. [21] gathered attitudes towards EHRs among Austrian and German citizens and found that citizens are generally interested in managing their health data and exchanging data between healthcare providers; however, data protection concerns were often mentioned. A Eurobarometer survey [22] showed that respondents generally like to have web-based access to their medical records, depending on the possibility of limiting access. Studies showed that privacy controls determined by the patient are a prerequisite for sharing health information [23,24]. A lack of granular controls negatively influences the willingness to share health information with other health professionals [25]. Consequently, worse healthcare outcomes can be expected [26]. Additionally, Caine and Hanania [27] discussed that patients want detailed privacy controls over their data in health records. That is why we aimed to understand different antecedents that add to an individual's privacy concerns about ePA. For a more comprehensive view, this mixed-method study aimed to research individual's attitudes towards the ePA by considering different types of motivations.

#### **2. Theoretical Background and Prior Research**

#### *2.1. Endogenous Motivations in Driving Usage Intentions*

In motivational psychology, it is the consensus that individuals' motivations can be either intrinsically or extrinsically originated (i.e., [28]), which is also embodied in Davis et al.'s [29] motivational model, which represents the prevailing perspective on understanding user intentions [30]. In this model, "extrinsic motivation influences behavior due to the reinforcement value of outcomes, [while] intrinsic motivation refers to the performance of an activity for no apparent reinforcement other than the process of performing the activity per se" ([29], p. 1112). Consequently, "perceived usefulness is an example of extrinsic motivation, whereas enjoyment is an example of intrinsic motivation" ([29], p. 1112). Even though prevailing technology adoption models help to explain many antecedents to behavioral intentions, such as perceived usefulness and ease of use, traditional technology acceptance models fail to capture significance because of uncaptured "user-beliefs" [30,31]. That is why the extrinsic/intrinsic dichotomy that comes with the conceptualization of extrinsic motivation as perceived usefulness and intrinsic motivation as enjoyment may result in an incomplete understanding [31]. Therefore, a different approach for capturing user intentions was proposed by Malhotra et al. [31]. It utilizes *organismic integration theory* (OIT) [32], which is a sub-theory of *self-determination theory* (SDT) [33].

Historically, research has seen motivation as a concept that varies primarily in quantity (cf. [34]), the idea being that more motivated people "will aspire greater achievement and be more successful in their efforts than people with less motivation" ([35], pp. 221–222). In contrast, SDT argues that the different types of motivation are more vital than the level of motivation in predicting behavioral outcomes [33,36]. Thus, the "distinction between

autonomous versus controlled is more important than the distinction between intrinsic and extrinsic" ([37], p. 471). The OIT regards motivation as the level of internalization and integration of the activity's value [32]. Individuals that experience their behaviors as autonomously driven perceive volition, whereas individuals whose behaviors are linked to feelings of pressure recognize themselves as being controlled [35]. There is evidence that perceived volitional motivation has a more significant influence on the behavioral outcome than motivation through external influences [31,35,38,39].

The OIT specifies a taxonomy for the levels of perceived autonomy, referring to the *perceived locus of causality (PLOC)*. The PLOC describes the extent to which someone senses an action as being self-initiated [32]. Figure 1 shows the relations of different types of endogenous motivations to specific PLOC types. With an *internal PLOC*, individuals see themselves as the originators of their behavior, whereas with an *external PLOC*, people see themselves as being controlled by external forces [31]. For example, users may be motivated to learn how to use a new piece of technology out of self-interest or compliance with a supervisor. Internal PLOC further splits into *identified PLOC* and *intrinsic PLOC*. Feelings of volition are common to both types. Intrinsic PLOC refers to instinctive and spontaneous behavior [40] that results in actions being performed due to inherent enjoyment or fun [32]. Identified PLOC, however, refers to behavior based on individual values and meaningful goals that are performed freely and autonomously [40]. Because identified PLOC motivational behavior results from internalizing external regulations as essential values, this is a type of extrinsic motivation [31]. Both intrinsic and identified PLOC are often combined into a composite of autonomous motivation [31]. Both types imply an internal PLOC, but only identified PLOC can be directorially influenced motivation [31]. In *external PLOC*, individuals attribute the reasons for their actions to external authority or compliance [32]. A crucial characteristic of external PLOC is that perceived external influences and personal values are not conflicting [31]. *Introjected PLOC*, however, is defined by a misalignment of perceived external influence and personal values [31]. The conflict can result in affective feelings of guilt and shame or esteem-based pressure to act [32,40]. Introjected PLOC often leads to rejection of the "imposed" behavior [31]. Even though both external PLOC and introjected PLOC are linked to external influence, they result in different behavioral outcomes [32].

**Figure 1.** Different types of endogenous motivations and their relations to the PLOC notation (adapted from [31]).

The PLOC framework [31] suggests that the different types of PLOC have cumulative effects on behavioral intentions. Understanding endogenous motivation can explain and predict individual differences in usage intentions across a population. The framework can also help explain different behavioral outcomes and why some users more widely accept some technologies than others. The PLOC framework has been applied in various research, such as in sustainable consumer behavior and educational and health-related lifestyle contexts [41,42]. Existing studies demonstrate that the PLOC framework needs to be contextualized. For a context-specific study, relevant contextual variables need to be used [43,44]. Consequently, we contextualized the PLOC model by conducting a qualitative study in the first phase of the mixed-methods design [45,46].

#### *2.2. Privacy Theories and Research in the Health Context*

The ubiquitous nature of information technology led to a "privacy is dead" shockwave [47]; however, many still consider health data sensitive and believe it should still be protected [15]. Even though the privacy literature is comprehensive, research in the health context is still ongoing. Only a few studies have examined human privacy in the health context (e.g., [1,2,15,48–51]). Existing literature demonstrates that protecting health data is increasingly vital to individuals. Privacy concerns result in privacy-protective behaviors, such as rejecting to adopt health solutions, including EHRs, [13,52–54], and holding back information from health professionals [2,55], which can negatively influence diagnoses.

The privacy concept has been discussed in various ways, but most literature emphasizes the matter of *control* [56–60]. For example, building on top of Clarke [61], Bélanger and Crossler [62] defined privacy as an individual's desire for control over their personal information. Similarly, in the health context, Fox and Connollly [63] define privacy as an individual's desire to be granted greater control over collecting and disseminating personal health information via health professionals and technology vendors. As the concept of human privacy remains challenging to measure, various other concepts are used as proximal measures. As such, *privacy concern* has been established as a central measure [62,64,65]. Privacy concern is the extent of the perception of a potential loss of privacy [66], i.e., the general tendency of people to worry about the loss of their informational privacy [67,68].

To measure privacy concerns, Smith et al. [64] introduced a 4-dimensional "Concern for Information Privacy" (CFIP) scale, which queries individuals' concerns regarding the collection, errors, unauthorized secondary use, and improper access of their information. The CFIP then has been used to measure health information privacy concerns in electronic medical reports and EHRs [1,13,16,52,54]. However, Kordzadeh et al. [48] suggested acknowledging additional factors of Internet Users' Information Privacy Concerns (IUIPC) [68], namely, the sub-dimensions collection, control, and awareness. The CFIP and IUIPC were combined to a 6-dimensional "Internet Privacy Concerns" (IPC) scale by Hong and Thong [69]. Fox and Connolly [63] then rephrased the IPC measure to create the Health Information Privacy Concern scale. Hong and Thong [69] formed the IPC as a third-order construct, and as a consequence, the HIPC has three dimensions as well. Concerns about collection, secondary usage, and control form a second-order interaction management factor, and errors and improper access constitute the second-order factor information management. Both second-order factors, plus awareness, build the third-order factor (H)IPC. The HIPC is shown in Figure 2. While past studies on healthcare adoption often measured privacy concern with one dimension (cf. [14,48,70]), the complex nature of m-health technology requires a more sophisticated approach to measuring human privacy. The HIPC is multidimensional, as depicted in Figure 2, and is preferably measured as such [63,69].

**Figure 2.** Formative third-order factor to measure the health information privacy concern (adapted from [15]).

Prior research has mainly utilized the antecedents→privacy concerns→outcomes (APCO) macro model that posits that a number of *antecedents*, usually individual traits or contextual factors, form an individual's privacy concerns, which, in turn, cause behavioral outcomes [65]. However, the majority of studies focus on the outcomes rather than on the antecedents [65]. A major contribution to the research of antecedents on the HIPC has been made by Fox and James [15]. This work will further examine antecedents that are subject to impact the HIPCs while simultaneously validating the factors found by Fox and James [15].

#### *2.3. Risk and Trust Beliefs in Privacy Research*

Besides privacy concerns, *risk* and *trust* play significant roles in privacy research. *Trust beliefs* become crucial when dealing with uncertainty [71], where trust is the "belief that the trusted party will fulfill its commitments [72,73] despite the trusting party's dependence and vulnerability [74,75]" ([76], p. 54). Research shows that greater trust in the vendor's *competence*, *benevolence*, and *integrity* results in lower privacy concerns [77–79]. Research in the health context supports the influence of trust on privacy concerns. For instance, Bansal et al. [14] found that trust influences the customer's willingness to interact with health-related websites. Dinev et al. [1] also found that trust in EHR system vendors reduces privacy concerns. On the other hand, *risk* is considered an antithesis to trust and can be described as one's expectation that information disclosure will have a negative outcome [56]. Studies have shown that risk perceptions increase privacy concerns for health websites [50] and reduce usage intentions for health-promoting wearables [80].

For electronic health records, users expect a heightened probability of privacy breaches and data misuse [13]. Additionally, "the highly sensitive nature of personal medical data adds even more to the uneasiness individuals feel about the violations and misuse" ([1], p. 29). Those concerns are general and are not necessarily linked to specific systems or practices [1]. Consequently, trust is crucial in overcoming risk perceptions concerning electronic health records [1,14]. McKnight et al. [79] distinguished between institution-based trust and disposition to trust in information systems research. For EHRs, an individual can trust a health professional but may not necessarily trust EHR systems; alternatively, an individual may value an EHR but not the institutions or care providers using it [1].

#### *2.4. IT Identity in Predicting IT Adoption Intentions*

Self-categorization and social comparison shape an individual's identity [81]. Selfcategorization supports individuals in putting their social environment into order and understanding and recognizing their peers [82]. The self-identity develops over time as people observe and categorize themselves relative to others based on their goals, perceptions of how others respond to them, and their self-evaluations [83]. One type of self-categorization is to see IT as being integral to the sense of self. Carter and Grover ([84], p. 938) defined IT identity as "the extent to which a person views use of IT as integral to his or her sense of self". The concept assumes that an individual's IT usage is motivated by positive selfidentification with IT use [85]. People who highly self-identify with IT employ IT usage more often than those who do not identify with the technology [84].

Carter [86] defined three dimensions of IT identity that serve an individual's selfperception about IT: dependency, emotional energy, and relatedness. *Dependency* is specified as "the degree of reliance a person feels on a particular IT or class of ITs as a source of personal well-being" ([86], p. 115). IT is so ubiquitous that businesses and humans depend on it, so it constitutes one component of individuals' identities. People express the perception of a need for devices. *Emotional energy* is defined as "an individual's enduring feelings of emotional attachment and enthusiasm in relation to an IT or class of ITs" ([86], p. 115). For example, continuous interaction with an IT device could result in confidence, energy, and enthusiasm. Conversely, lack of these emotions can cause negative feelings, such as boredom [87]. Finally, *relatedness* refers to "a blurring of boundaries between notions of the self and an IT experienced as feelings of connectedness with an IT

or class of ITs" ([86], p. 114). When individuals incorporate their devices' characteristics within their self-identities, they feel intimately connected to these IT devices.

In the digital health context, it was proposed that IT identity influences emotions relating to IT, affecting patients' decisions on whether or not to adopt healthcare devices or applications [88]. Additionally, the literature indicates that the IT identity theory is a relevant factor in explaining patients' interaction with m-health applications [89]. Accordingly, this work adopts IT identity and its dimension to predict users' adoption intention of the ePA application.

#### **3. Prototype**

At the time of research, the ePA had not been launched, and as we write now, the ePA has not had high diffusion due to missing technical infrastructure [90]. To overcome this limitation, we reviewed the ePA and provided a prototypical ePA mobile application. Based on the findings in the literature (e.g., [91]), we developed a prototype to be used for further research and present it in Figure 3. The prototype was used to inform the participants of the following studies. Creating a distinct prototype will also help to create a common understanding of the ePA, which should be helpful, given the breadth of available ePA applications. For prototyping, we utilized Figma [92]—a "mid-fidelity" prototyping tool for creating interfaces that can be immediately tested to get practical impressions of the applications [93].

**Figure 3.** The prototype of the ePA application used in our research to inform the participants.

As each health insurer will provide its own version of an ePA application, the applications will be branded. Health insurance companies and technology vendors likely offer ePAs as parts of more comprehensive digital health applications. All electronic documents in the patient file are listed chronologically by name, and by publication date in the document view (second screen). With the ePA being patient-administered, users can upload any documents, even newspaper articles. Another view (third screen) visualizes the permissions screen where all given permissions are listed. The list gives an overview of what health providers were given access, in the past or continuing. The last screen holds a record of every action performed on the patient file. For instance, the list gives an overview of what files have been uploaded, downloaded, deleted, and by whom.

#### **4. The Mixed-Methods Design**

We applied a two-stage sequential mixed-methods design to research the intentions, attitudes, and privacy concerns towards ePAs. Mixed-methods research combines "elements of quantitative and qualitative research approaches [. . .] for the broad purposes of breadth and depth of understanding and corroboration" ([94], p. 123). A mixed-methods design approach is compelling in the ever-changing IT context, where researchers encounter problems with the explanatory power of existing theories and findings [30]. Mixed-methods research offers three main advantages: it allows one to address confirmatory and explanatory research questions simultaneously, provides more robust inferences than a single method, and can produce a more comprehensive range of divergent and complementary views [46].

The overall study made use of Venkatesh et al.'s [46] design guidelines. At the beginning of the process, we defined three research questions (one qualitative, one quantitative, and one mixed-methods; see Appendix A). The purpose of the mixed-methods is "developmental," where the findings from the first strand qualitative method are used to inform the second strand quantitative method [46]. This study followed multiple paradigms from an epistemological perspective, with the first strand being interpretive and the second strand being deductive [46]. The methodology is "mixed-methods multistrand" ([46], p. 443) with a "sequential exploratory design" [95], which is characterized by the qualitative phase followed by its quantitative phase ([46], p. 445). The research design is sequential exploratory-explanatory, as it combines exploratory and explanatory approaches [96]. The study falls into the category of a "dominant-less dominant design," with the quantitative strand being dominant in the overall design ([97], p. 44). Appendix A proves the design choices made. Figure 4 visualizes the dominant–less dominant design of our mixed-methods study.

**Figure 4.** The mixed-methods multistrand exploratory-explanatory research with the dominant-less dominant design.

#### **5. Phase 1 Qualitative Study**

The phase 1 qualitative study aimed to answer the research question: "What are the salient factors determining an individual's intentions toward using the ePA?" For answering this question, we conducted semi-structured interviews with four individuals.

#### *5.1. Research Methodology*

Before the interviews, we created a semi-structured interview guideline. Semistructured interviews encourage communication, thereby encouraging respondents to reveal underlying concepts [98]. The conversational form allows follow-up questions and prompts based on the answers [98]. This approach is particularly appropriate for generating new theories rather than confirming the established theory. The four interviewees (Appendix C) were identified through purposive sampling, i.e., non-probabilistic sampling where subjects were selected intentionally [99]. The interviews took about 30 min each and were conducted in German with a combination of open-ended and closed questions (Appendix B). All interviews were conducted remotely, recorded, and transcribed. Ethical considerations included measures such as only using encrypted communication channels, using pseudonyms in the transcripts, and not asking for health-related circumstances, such as chronic diseases. Beginning with some general questions, we presented the prototype of the ePA application from Figure 3. The prototype was explained in detail but took no longer than five minutes. Subsequently, the interviewees were asked about general attitudes toward this application and were asked to articulate how these attitudes were constituted. Subsequently, the respondents were asked about their health information privacy concerns (cf. [15]). Other questions cover the interviewee's usage intentions, perceived benefits, and perceptions of risk.

We used an inductive approach [100] to make sense of the interviews rather than quantifying the data. We started by generating a list of "start list" of codes ([101], p. 58) resulting from the literature review. Then, with a "constant comparative" ([102], p. 105) analysis, we intended to identify the initial concepts and to link them to resulting sets of broader categories [103]. In grounded theory methodology, this procedure is equivalent to the "open coding" phase ([104], p. 12), where "conceptually similar events/actions/interactions are grouped to form categories and subcategories". We used the software Atlas.ti to apply codes to the transcripts. Through constant comparison, "abstract categories" of labels were assigned to similar concepts ([101], p. 58).

#### *5.2. Findings*

The coding of the transcripts revealed different types of motivation-related variables. Identified PLOC was emergent, i.e., interest in accessing health data or more efficient treatments. Additionally, coding revealed that advice from health professionals supports ePA usage intentions, indicating that external PLOC drives adoption intentions. Introjected PLOC can result in rejecting the ePA, as respondents discussed that a negative medical history could result in feelings of shame due to conflicting external expectations in internal values. Among these motivation-related variables, respondents mentioned that a person's IT experience and age could influence the usage intentions. In the interviews, we found indicators for different privacy concerns, i.e., concern for collection, secondary usage, improper access and errors, and a general desire for privacy and control over their data. Especially regarding the desire for control, many respondents underlined the importance of granular access rights. The trade-off between privacy risk and trust was repeatedly mentioned. Respondents mentioned that the trust regarding their physicians positively influences their intentions and perceived risk, especially regarding the general data collection practices on technical devices, has a negative impact. Among these variables, some interviewees mentioned that usability might play a role in using or discontinuing use of the application. The topics and broader concepts that were emergent through the interviewees are provided in Appendix D. Additionally, Appendix E displays clarifying quotes per the interviewees.

#### **6. Research Model**

The research model used (1) the PLOC framework [31] as the underlying theory to capture individual motivations, (2) the HIPC construct with contextualized privacy antecedents, and (3) heuristically evaluated context-based constructs to develop and justify

the hypotheses. Table 1 shows the constructs and their definitions. We used *intention* as the dependent variable because an intention is the most "proximal" influence on behavior ([105], p. 76). An intention is what one plans on doing.

**Table 1.** Constructs and their definitions.


Internal PLOC is identified by the intrinsic and the identified PLOC that both. Feelings of volition characterize both states. Intrinsic PLOC refers to spontaneous behavior and performance for inherent fun, and identified PLOC refers to behavior based on personal values, goals, and outcomes [32]. For the ePA, users may adopt it if they can control it (intrinsic drivers) or be guided by internalized values such as health awareness. For example, one interviewee (I1) said that they likes to see "which current diagnoses I will have or which doctor's letters and documents come together that exist about me". Hence,

**Hypothesis 1.** *Internal PLOC positively influences one's intentions toward adopting ePA applications.*

External PLOC is perceived when one's actions are attributed to external authority [32]. There must be no conflict between the perceived external influences and an individual's internal values. The resulting behavior is usually done to comply with external demands. In the case of the ePA, such external demands could arise from recommendations by physicians or the health insurance. Hence,

**Hypothesis 2.** *External PLOC positively influences one's intentions toward adopting ePA applications.*

Introjected PLOC refers to feelings of shame and guilt that may emerge from other parties prompting them to act in a particular way (e.g., [33]). The user feels tension and confusion as introjected PLOC derives from misalignment between a user's beliefs about behavior and their self-perceived autonomy [32]. If users experience that either their health insurance or the government is exerting pressure to use the ePA but judge themselves to be autonomous, the resulting uncertainty is likely to influence the usage intentions of ePA applications negatively. Hence,

#### **Hypothesis 3.** *Introjected PLOC negatively influences one's intentions toward adopting ePA applications.*

As to Carter and Grover [84], there are three behavioral consequences of IT identity: feature use behavior, enhanced use, and resistance behavior. Consequently, mobile technology identity can lead to both resistance and adoption [84]. Higher mobile technology identity can lead to higher motivation to adopt mobile applications, since people are dependent on and enthusiastic about their phones. This enthusiasm concerning the mobile device can increase the individual's motivation to adopt m-health applications [107]. Additionally, feelings of IT dependence or relatedness can motivate people because they can link these feelings to dimensions of their identity [84]. Hence,

#### **Hypothesis 4.** *Mobile technology identity positively influences one's intentions toward adopting ePA applications.*

The interviews indicate that age has a role in the adoption of ePA applications. One interviewee, being aged 50+, expressed that they is very unfamiliar with technology and media, resulting in being conservative (I4). Another interviewee noted that older people might have problems with handling such applications and would not use the ePA. In literature, demographics, such as age, are commonly associated with privacy concerns. In Laric et al. [70], older participants expressed deeper privacy concerns regarding healthcare services. In contrast, Kordzadeh et al. [48] found younger users to have more significant privacy concerns attributed to their privacy literacy. In Vodicka et al. [108], people under 55 expressed more severe privacy concerns of the physician's notes from their treatments. Additionally, King et al. [109] found that age correlates with concern about health information privacy. The majority of studies have revealed that privacy concerns increase with age, which is in line with the findings of the phase 1 study. Hence,

#### **Hypothesis 5.** *Age positively influences the HIPC.*

The health status was an emergent theme in our interviews. While one interviewee stated that they would use the application uncoupled from their health status, other interviewees expressed concerns. For example, one interviewee stated that people with certain chronic diseases retain from using the ePA:

People with serious chronic illnesses, psychological problems, and those who fall under social taboos will hardly use the app.

(I3)

People with severe medical conditions require frequent treatments [110]. Thus, those people generate the most personal health information and are likely to express higher privacy concerns [15]. In Flynn et al. [111], people who feared mental illness's stigma were less likely to opt into an electronic psychiatric record. Other studies support that health status influences information sensitivity and privacy concerns [14,112]. Based on the literature and phase 1 study findings, we posit that severe health conditions have impacts privacy concerns. Hence,

**Hypothesis 6.** *A severe health condition positively influences the HIPC.*

Anderson and Agarwal [2] claimed that perceived information sensitivity affects privacy concerns and intentions to provide personal health information. In Dinev et al. [56], information sensitivity was associated with perceived risk. Caine and Hanania [27] found that the decision to share data in an EHR with some particular parties was based on the perceived sensitivity of personal health information. Additionally, Bansal and Davenport [14] found a positive correlation between health information sensitivity and privacy concerns. Further, the "highly sensitive nature of personal medical data" increases one's concerns of violations and data misuse ([1], p. 29). One respondent demonstrated this:

If it says in your documents, you have some sexually transmitted disease or something, you may not want everyone to access it because it's something that's only your business.

(I2)

With support from the literature and the qualitative findings, we posit that perceived sensitivity of health information impacts privacy concerns. Hence,

**Hypothesis 7.** *Perceived information sensitivity positively influences the HIPC.*

Trust and risk often are linked to privacy concerns [1,66,113]. Even though the APCO model positions risk as an outcome of privacy concerns, Smith et al. [65] recognized that prior studies support the influence of privacy risk on privacy concerns. Studies have shown that perceived privacy risk positively correlates with different websites, including healthcare ones [50]. In the case of the ePA, both health professionals and health insurance companies handle personal health information. Hence,

**Hypothesis 8a.** *Perceived risk associated with health professionals positively influences the HIPC.*

**Hypothesis 8b.** *Perceived risk associated with health institutions positively influences the HIPC.*

Additionally, trust has been shown to both an outcome and an antecedent of privacy concerns [65]. Prior studies found that trust in physicians and EHRs lowers privacy concerns [1,114]. One respondent in the qualitative study expressed that trust in their health insurer was a factor in using the ePA:

I would trust the health insurance companies. That plays an essential role for me.

(I1)

Hence,

**Hypothesis 9a.** *Trust in health professionals negatively influences the HIPC.*

**Hypothesis 9b.** *Trust in health institutions negatively influences the HIPC.*

Past studies show evidence that privacy concerns influence usage adoptions for health applications, including EHRs [15,53,115]. We consequently posit that with an increased HIPC, individuals will be less likely intend to opt-in to the ePA. Hence,

**Hypothesis 10.** *The HIPC negatively influences intentions to adopt ePA applications.*

Besides these variables mentioned, the questionnaire of the phase 2 study also covered traditional control variables such as education, employment, as the literature advocates that these elements affect behavioral intentions [30,116]. Thus, those factors were added to the questionnaire to enrich understanding of the ePA applications' usage intentions. Finally, we present the research model conceptualized from the hypotheses in Figure 5.

**Figure 5.** The research model with the different hypotheses.

#### **7. Phase 2 Quantitative Study**

The second phase of the mixed-methods study aimed to answer the question, "Does the research model explain usage intentions of the ePA?" Therefore, we conducted a survey of potential German adopters to test the research model.

#### *7.1. Research Methodology*

To gather empirical data from potential adopters, we conducted an online survey with a traditional questionnaire design consisting of a closed-questions design. The questionnaire was published on SoSci Survey. The advantages of an online survey are a potentially broader target audience, straightforward distribution and analysis, and the collection of additional measures, such as the time needed to complete the survey. The survey was distributed via e-mail to a list of acquaintances and a market research panel simultaneously to gather responses from participants with diverse socio-demographic backgrounds. To be included to the study, participants needed to be at least 18 and have a permanent residency in Germany. At the beginning of the survey, we presented the screenshot of our prototypical ePA application (Figure 3) with a brief explanation of available features to establish a common understanding of the ePA.

The research model was analyzed using partial least squares structural equation modeling (PLS-SEM). The calculations were made in SmartPLS version 3.3.3 [117]. PLS regression is often used in information systems research to understand behavioral phenomena. We applied current recommendations and validation tools to analyze our model [118].

#### *7.2. Measures and Pilot Testing*

For measuring intention to use the ePA, we used a two-item scale based on the literature [30,119]. The endogenous motivation was measured by scales based on Ryan and Connell [32] that were extended to capture ePA adoption. Those measures were greatly influenced by the findings of the phase 1 study. Items for external PLOC measured selfperceived reasons for usage intentions resulting from their health insurance or physician recommendations. The internal PLOC scale measured reasons for using an ePA characterized by self-determined choice and volition. For measuring introjected PLOC, we used items dealing with conflicts between personal values and social norms. We added two-item scales to measure both dependence and emotional energy as characteristics of IT identity [84,86]. For measuring an individual's Health Information Privacy Concern in the ePA, we added a three-item scale for each dimension—secondary usage, control, errors, and improper access from Fox and James [15,69]. We, however, omitted the *awareness* construct of the HIPC scale since we did not find evidence for this attribute in the interviews. In addition, the questioning focused on the individuals' perceptions of their concerns rather than their expectations, as proposed by Hong and Thong [69]. Capturing

the individual's health status involved a three-item scale based on Bansal et al. [14]. We utilized a two-item scale for each category to capture risk perception concerning health professionals and insurance providers and capture trust towards health professionals and insurance providers [15,53,69]. For measuring personal health information sensitivity, we utilized a 5-point-Likert-scale for different categories of health data to rate the perceived sensitivity, based on Laric et al. [70]. Finally, we added items for measuring the demographic characteristics age (offering four categories), employment (four categories), formal educational level (four categories), and prior m-health experience (yes/no). At the end of the survey, we added items that gathered self-reporting seriousness checks to improve data quality [120]. All scales are presented in detail in Appendix F.

The survey used validated construct scales from the literature where applicable. Several best practices were applied to avoid common-method bias [121–123]: The questionnaire was designed to maximize user engagement and minimize task difficulty. A "good cover story" ([123], p. 562) on the opening page of the survey aimed to engage respondents' accuracy and motivation. The introductory text was both descriptive and motivating by emphasizing the respondent's desire for self-expression [123]. We applied clear and concise language and avoided ambiguous or unfamiliar terms. We separated parts in the questionnaire and repeatedly displayed the image of the prototype in the hope of diminishing "effects of involuntary memory-based and perceptual biases" ([123], p. 563). The questionnaire was pilot tested to validate the instrument. We conducted two on-site and two remote pilot tests. The user tests provided feedback that resulted in the rewording of items and clarified descriptions. Participants reported difficulties with one item from the introjected PLOC scale that we decided to drop from the questionnaire. The questionnaire was then reviewed until judged satisfactory.

#### *7.3. Sample*

The sample was intended to be "probabilistic" [46]. The heterogeneity of the sample could be verified by the descriptive analysis of the survey data. The external validity of the sample was reasonably ascertained by assuring that the sample represented the whole German population by comparing the sample with data of German citizens (see Appendix G). The online survey was opened 480 times. A total of 289 participants commenced the survey, which makes a response rate of 60%, though the click-rate of the survey is a vague metric. Among those participants, 250 respondents finished the last page of the survey. Then, incomplete responses (*n* = 2) were removed, which resulted in a completion rate of 86%. For data cleaning, we followed the practice that all cases should be retained unless evidence suggests a case is aberrant [124]. Leiner ([125], p. 242) proposed a "relative speed index" to eliminate potentially meaningless cases by completion time. We chose a speed index of 2.00 and removed *n* = 17 responses with completion times two times faster than the median completion time. We also removed cases from respondents who did not give their consent or self-reported their answers as meaningless. After data cleaning, 222 responses were used for further analysis. Participants' demographic characteristics are shown in Appendix G. Basic descriptive characteristics (mean, standard derivation) are presented in Appendix F.

#### *7.4. Preliminary Analysis Validation*

To ascertain the quality of the quantitative results, we evaluated a range of reliability measures to test the convergent and discriminant validity of the scales [121]. We began with assessing the convergent validity by evaluating the multi-item construct quality (see Appendix H). The Cronbach's alpha revealed undesirable internal consistency (*α*<0.600) for both the introjected PLOC and HIPC-control scale. A low alpha indicates poor interrelatedness between items or heterogeneous constructs [126]. For both scales, we improved internal consistency by dropping one item from the construct. We then further assessed construct reliability by conducting the composite reliability and AVE scores. The composite reliability should exceed 0.700 and be larger than the AVE [124], which was the case for all constructs. We further obtained the outer loadings and t-statistics for all items across

each construct. Loadings above 0.700 are often recommended, but lower values can be sufficient [124]. As Appendix I reports, all items had outer loadings above 0.700 and were significant at the *p* < 0.05 level.

We also calculated the tolerance levels and variance inflation factors (VIF) to check for multi-collinearity. The threshold of 10 [127] was passed for two of the HIPC-access items. The tolerance levels were all greater than 0.10, but for the before-mentioned items, thereby indicating that multicollinearity generally is no issue. As these two items are used to form a third-order factor to measure HIPC, we did not consider the VIF problematic. Thus, all items were retained for further analysis.

To examine the discriminant validity, we conducted a Fornell–Larcker test (see Appendix J). A latent construct should better explain the variance of its indicator than the variances of other latent constructs [121,128]. The average variance extracted (AVE) from each of the latent constructs should be higher than the highest squared correlation with any other latent variable. Our test ensured that the square root of the AVE exceeded all correlations with other latent constructs, and discriminant validity was given.

#### *7.5. Model Results*

The structural model results are summarized in Table 2 and Figure 6.

**Table 2.** Results for test hypotheses and control variables.


**Figure 6.** Summary of Full Model Results. <sup>∗</sup> *p* < 0.05; ∗∗ *p* < 0.01; ∗∗∗ *p* < 0.001.

#### **8. Discussion**

The mixed-methods design aimed to discover individual's intentions toward using the ePA mobile applications. The qualitative study uncovered a range of factors influencing usage intentions to formulate 12 hypotheses. The results of the quantitative study show overall support for most hypotheses. We implemented qualitative analysis, followed by the quantitative analysis (see Table 3) [46]. The results show consistency but also reveal some incompatible findings. Overall, we found the same row of parameters were significant in both qualitative and quantitative studies. Even though the questionnaire was developed from the findings of the qualitative study, we found some significant differences in the findings of the studies: Besides an individual's health status emerging as a critical factor for ePA adoption in the qualitative study, health status was not significant in the second study. Similarly, one's positive self-identification with mobile devices ("mobile IT identity") was not significant in the quantitative study. A limitation of our study is that we did not replicate the divergent results with a new dataset [46,129]. However, we offer a theoretical explanation to remedy the inconsistent findings.

Overall, our meta-inferences are congruent with our research model. We successfully added value beyond the individual studies with the integration of the qualitative and quantitative research strands. Considering that the phase 1 and phase 2 study data were from different sets of respondents and different data-collection approaches, the similarity implies that we utilized solid theoretical models as our research foundation. The mixedmethods helped us determine and understand factors that influence ePA usage intentions. With the qualitative study, we were able to determine a set of aspects and their relevance. In contrast, the quantitative study empirically examined the research model that resulted from the qualitative study to determine what factors influence ePA usage intentions. Table 3 summarizes our meta-inferences.

In particular, the results highlight the predictive power of motivation. Some respondents from the phase 1 study expressed intrinsic PLOC, i.e., "joy" in accessing their data and using the ePA. Other respondents expressed indicators that relate to the perceived usefulness, thereby relating to identified PLOC. For example, one respondent identified that a digital health record helps them to keep track of their data, even when consulting different physicians:

I have moved several times in my life now, even long distances. In the end, I always had to have everything handed over to me in physical form by the family doctor I was seeing.

(I3)

Both intrinsic and identified PLOC were crucial factors for predicting ePA usage intentions among the studies. Some respondents indicated that they considered adopting the ePA when advised to, indicating the motivational power of external PLOC. Those findings were consistent among both strands of our mixed-methods study. However, internal PLOC was a stronger predictor than external PLOC. These findings are consistent with the literature about external rewards [39,130]. We found strong indicators for introjected PLOC, hindering ePA adoption in the qualitative study. The respondents repeatedly expressed uneasiness resulting from a misalignment of perceived social influences and personal values:

I think if you are seriously ill and you carry this application around with you all the time, it's like carrying your X-rays around with you all the time. I don't like the idea.

(I2)

People with serious chronic illnesses, psychological problems, and those who fall under social taboos will hardly use the app.

(I3)



The quantitative study supported the negative impact of introjected PLOC. In particular, political pressure and shame were two factors that hindered ePA adoption from the quantitative study.

Contrary to our expectations, the meta-analysis for the "mobile IT identity" and "health status" variables indicate the lack of influence of these factors on ePA adoption or the HIPC. We now attempt to explain these meta-analysis: (1) The low impact of one's mobile IT identity can be explained by the not-so-technical nature of a health record: Even though the ePA is distributed as a mobile application, such an application does not require a self-identity that is usually attributed to "mobile IT identity". We assume that, in contrast, ePA applications being heavily gamified might demand positive perceptions towards IT in a pronounced manner. (2) An individual's health status did not have a significant influence on the HIPC. The share of subjects with self-reported severe health status was generally low in our sample; thus, our quantitative study failed to see an effect on this variable. We argued that people with severe health conditions would express higher privacy concerns; however, none of the interviewees from Study 1 reported severe chronic diseases themselves, but thought that there might be concerns from people with such conditions. On the other hand, populations with multiple chronic conditions may have more motivation to use the ePA to facilitate patient–doctor communication and control privacy settings themselves. Whether a severe health condition has a positive impact on privacy concerns, or a positive impact on the usage intentions, or even both, is not supported by the meta-analysis.

The quantitative study showed evidence that age has an impact on the HIPC. However, contrary to our assumptions, higher age led to lower HIPC. Our literature review showed conflicting findings for the impact of age on an individual's privacy concern in the health context. One supported explanation is that younger people might have higher privacy concerns attributed to their privacy literacy, which is also supported by the literature [48].

The findings towards perceived risk and trust concerning health professionals and one's health insurance were congruent among both strands of research. This is a strong indicator of risk and trust being linked to privacy concerns. Our findings are consistent with the literature [1,66,114,131]. For instance, a satisfying experience with one's health insurance can lead to less resistance when adopting an ePA that is distributed by their health insurance, as stated by one respondent:

I have personally been very, very satisfied with my health insurance company over the years. I am sure that it works well, and I can download the application with confidence. In contrast, for third-party providers, I would have to deal with who is behind the app.

(I3)

Even though the APCO model positions risk as an outcome of the privacy concern, we demonstrated that privacy risk influences privacy concern. This impact was also theorized by Smith et al. [65].

Our findings from both research strands show that the attributed information sensitivity of health data adds to the HIPC towards the ePA. Individuals that perceived their health information as being more sensitive were less likely to adopt the ePA. This finding is consistent with the literature [2,14,27,56]. Overall, the information sensitivity and general demand for privacy differed among the respondents in the qualitative study, which was also reinforced by the quantitative findings. We discussed that the perceived privacy risk and privacy calculus are less profound where electronic patient records are relatively new. Individuals tend to weigh the benefits of the ePA more heavily than the concerns of privacy. However, those societal values may change over time during the diffusion process of the ePA. Additionally, secondary usage of one's health data can result in uneasiness, for instance, when health data are used for data mining purposes, or when the data impact the services delivered by the health insurance.

In the qualitative study, the control that one could exercise over their health data was an essential factor in ePA usage intentions:

I would like to decide what the doctor can get from me and what insight he can get from me.

(I4)

Additionally, existing literature demonstrated that "patients want granular privacy control over health information in electronic medical records" [27].

The concept of perceived ownership of data in the ePA was also present in our interviews. For instance, one respondent mentioned that they did not feel up to exerting control over their data:

Do I wish I had control over it myself when my family doctor has the data? I would like to have confidence that the control will be realized by someone else.

(I2)

Tang et al. ([132], p. 125) noted that, with the patient having data sovereignty, "different mindsets and levels of trust" will become mandatory. Fox and James [15] researched the HIPC and found that interviewees have differing perceptions of data ownership in the context of EHRs. Perceptions ranged from beliefs that the patient is the owner, to dual ownership, to the perception that the physician owns the data. Fox and James [15] also found that interviewees seeing themselves as single data owners expressed a strong desire for privacy and were highly concerned about unauthorized secondary use, improper access, and control.

Our research model could explain 77% (*R* <sup>2</sup> = 0.771) of the variance in our sample, which is a satisfactory fit of our theoretical model. Thus, our study demonstrates that we used solid underlying theory, i.e., the PLOC framework, to understand the intentions toward adopting the ePA.

#### **9. Limitations and Future Research**

Our study has some limitations, which should create opportunities for future research. First, we interviewed four individuals in the first study, making it unlikely to reach theoretical saturation. Since study 1 was less dominant in the overall study design, this limitation was maintainable. Further, we did not ask health-status-related questions out of ethical considerations, which may have resulted in an incomplete picture of the impacts of health-related factors on ePA usage intentions. Next, we showed and described the prototype from Figure 3 to the participants in a detailed manner. However, the prototype was static, and the remote setting has downsides regarding user comprehension. Further studies should prepare a high-fidelity prototype and consider the impacts of participants' digital skills and literacy levels.

Second, our sample in study 2 contained 222 usable responses from German citizens. Even though we ascertained external validity employing the demographics (see Appendix G), our sample had an imbalance in the age distribution. Further, we noticed that the share of respondents that reported severe health conditions was generally low. Additionally, an online survey requires a certain level of IT literacy. This is particularly important because a severe health status, IT literacy, and old age may co-occur. Similarly, we measured health status with a two-item scale (see Appendix F) that lacked understanding of the population's actual health conditions. The scale used in the questionnaire mixes chronic and acute diseases from participants and does not capture multimorbidity, nor polypharmacy. Further studies should improve the measures to capture health status.

Third, even though we discovered several antecedents that impact privacy concerns, we did not capture each antecedent. This limitation has been shared by other studies examining the antecedents of the HIPC [15]. We thus encourage exploring privacy antecedents more comprehensively. Lastly, an individual's perceptions evolve due to changing societal values or recent events. In addition, perceptions change over the time when a new technology is in the process of diffusion. Currently, Germany's ePA is in an early testing stage. Even though some issues were already discussed in public, the concept of the ePA is

not widespread yet. Thus, future research could apply a longitudinal study to get further insights into users' adoption intentions.

#### **10. Conclusions**

The adoption of the ePA is a complex task. With the launch of new technology, such as the ePA, its adoption faces significant challenges. With a mixed-methods design and by developing a contextual model, we gathered evidence that different types of motivation, the HIPC, and privacy antecedents affect usage intentions regarding ePA. Most importantly, a profound understanding of the different types of motivation is critical to understanding individual usage intentions, since motivational variables were shown to explain the majority of the variance in our sample.

The findings showed the integral positive effect of internal PLOC. Individuals who feel volitional about using an electronic health record are more likely to adopt it. Consequently, policymakers must understand what types of motivation are critical predictors in ePA adoption and use. The findings demonstrate that policymakers have to provide both internal and external incentives. We believe that the results of this work contribute to the growing body of research on technology adoption in the field of the ePA in a German context.

**Supplementary Materials:** The survey data are available in https://www.mdpi.com/article/10.3390/ ijerph18189553/s1.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Ethical review and approval were waived for this study because that ethics committee does not handle approval requests from students. The ethical review committee was approached and the process documented.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the studies.

**Data Availability Statement:** The survey data are available in the Supplementary Materials to enable reproduction of the study's results.

**Acknowledgments:** I thank Thomas Mandl and Joachim Griesbaum, Faculty of Linguistics and Information Science, University of Hildesheim, Germany, for supervising my final thesis, which this paper was derived from. I further thank DongBack Seo, Department of Management Information Systems, Chungbuk National University, South Korea, for giving me inspiration for this work during her mentoring.

**Conflicts of Interest:** The author declares no conflict of interest.

#### **Appendix A**

#### **Table A1.** Design Decisions for the Mixed-Methods-Design (Adapted from [46]).



#### **Table A1.** *Cont.*

#### **Appendix B. Interview Guideline**

	- How did you react/have you reacted?
	- Users: What technologies? What data? benefits? reasons for use?
	- Former users: Which technologies? Which data. Any advantages? Reasons for stopping use?
	- Non-users without experience: Would you ever use these technologies? What, why, perceived benefits.
	- Can you tell us something about your experience with the app

#### *Presentation of the prototype (Figure 3)*

	- What role does your interest in technology play?
	- What role do health factors play?
	- What role does the publisher of the app play?

10.Can you imagine your doctor prescribing via an app in the future?


12.How would you describe your concerns about protecting your health data?

13.Which groups should have access to your health data, in your opinion?


#### **Appendix C**



#### **Appendix D**

**Table A3.** Emergent Themes from the Interviews.


#### **Appendix E**

**Table A4.** Selected Quotes from the Interviews.


#### **Appendix F**

#### **Table A5.** Scale Items for Construct Measures.


**Table A5.** *Cont.*


DQMeaningless Did you perform all tasks as asked in each instruction?

(1 = I completed all tasks as required by the instructions, 2 = *Sometimes* I clicked something because I was unmotivated or just

didn't know my way around, 3 = I *frequently* clicked on something so I could finish quickly)

*a* : Dropped after preliminary analysis. *<sup>b</sup>* : Dropped after pilot study. *<sup>c</sup>* : Dropped after preliminary analysis.

#### **Appendix G**

#### **Table A6.** Distribution of Sample and German Citizens.


#### **Appendix H**

#### **Table A7.** Reliabilities of Multi-Item Constructs.


#### **Appendix I**

**Table A8.** Loadings of the Multi-Item Constructs.


#### **Appendix J**

**Table A9.** Fornell–Larcker Criterion for Discriminant Validity of Multi-Item Constructs.


Diagonal numbers represent the square-root of the AVEs. Each SQAVE exceeds all correlations with other latent constructs.

#### **References**


## *Article* **Usage of eHealth/mHealth Services among Young Czech Adults and the Impact of COVID-19: An Explorative Survey**

**Michal Dolezel \* and Zdenek Smutny**

Faculty of Informatics and Statistics, Prague University of Economics and Business, 130 67 Prague, Czech Republic; zdenek.smutny@vse.cz

**\*** Correspondence: michal.dolezel@vse.cz; Tel.: +420-224-09-5476

**Abstract:** Various mHealth/eHealth services play an increasingly important role in healthcare systems and personal lifestyle management. Yet, the relative popularity of these services among the young population of the Czech Republic was not known. Therefore, we carried out an on-line survey with a convenience sample (*n* = 299) of young adults aged 18–29 and living in the Czech Republic. To this end, we adapted the survey instrument which was previously used in a similar study conducted in a different cultural context (Hong Kong). In our study, we found out that *health tutorial* activities (i.e., acquiring information on diet, exercise, fitness) were the most common among our respondents (M = 2.81, SD = 1.14). These were followed by *health information seeking* activities (i.e., acquiring information on medical problems) (M = 2.63, SD = 0.89) and *medical services* (i.e., the eHealth/mHealth services that provide infrastructural support, such as ePrescription and doctor appointment organizers) (M = 2.18, SD = 0.97). Based on the grouping according to gender and existing health condition, pairwise comparisons showed statistically significant differences. We also briefly analyzed the influence of the COVID-19 pandemic on the examined activities. Based on their relative popularity, we suggest leveraging the potential of *health tutorial* activities to improve public health.

**Keywords:** personal health informatics; consumer health data; consumer health information; self-tracking; quantified self; mHealth apps

#### **1. Introduction**

Nowadays, individuals with a non-medical background increasingly rely on technology when they interact with existing health systems or independently consume healthrelated knowledge. In doing so, this group frequently benefits from various eHealth and mHealth services. The former term "refers to tools and services that use information and communication technologies (ICTs) to improve prevention, diagnosis, treatment, monitoring and management of health and lifestyle" [1]. Being a component of eHealth, mHealth can in turn be defined as "medical and public health practice supported by mobile devices" [2]. However, mHealth technologies also carry promising potential, disrupting and improving established healthcare routines and behaviors [3].

From a policy-based perspective, eHealth and mHealth technologies can be divided into four broad categories [4]. These are: (i) system services with a supporting role; (ii) information and communication platforms; (iii) health diaries and consumer-grade electronics for monitoring; (iv) interventional health technologies. To describe the categorial content in more detail, the *first category* includes solutions that ease the navigation within existing health systems, such as ePrescription [5] and doctor appointment organizers [6]. Within the *second category*, health information seeking [7,8] is the dominant class of activities referring to a broad range of tasks. On the one hand, this conceptual label may be used for more formal activities, such as accessing tethered electronic health records [9] via patient portals [10]. On the other hand, internet discussion forums [11], patient support

**Citation:** Dolezel, M.; Smutny, Z. Usage of eHealth/mHealth Services among Young Czech Adults and the Impact of COVID-19: An Explorative Survey. *Int. J. Environ. Res. Public Health* **2021**, *18*, 7147. https:// doi.org/10.3390/ijerph18137147

Academic Editors: Martin Wiesner and Björn Schreiweis

Received: 15 May 2021 Accepted: 29 June 2021 Published: 3 July 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

groups on social networks [12] and special patient websites [13] play an important role in the lives of many patients nowadays. In addition, secure messaging and video apps for healthcare represent an infrastructural mean essential for trustful communication with healthcare providers [14]. Evolving very rapidly, the *third category* covers fitness trackers and other consumer-grade electronics [15], such as sleep gadgets [6]. Lastly, the *fourth category* covers health technology used for complex interventions mandated by health professionals [16]. Broadly, this integrates platforms that address public health concerns or allow the self-management of existing health problems [17]; support the diagnostic processes; and enable active recording and monitoring by capturing validated data [18], etc. Importantly, in the real world, those four core categories may partly overlap due to blurry boundaries between them. For example, smart watches and other consumer-grade electronics (category No. ii) are increasingly often being examined as promising means for health intervention programs (category No. iv) [19,20]. An alternative, more simple categorization of eHealth/mHealth services has been proposed by different authors [21].

Presently, however, the relative popularity of the above services among the young population of the Czech Republic is not known. This research therefore aims to explore to what extent different eHealth and mHealth services are used by the adults aged 18–29, living in the Czech Republic. Here, we mapped the eHealth/mHealth landscape in an explorative sense. Our intention was to obtain an initial understanding of eHealth/mHealth consumption patterns by comparing the relative frequency of exercising the analyzed activities. Of note, the data collection phase overlapped with the beginning of the COVID-19 pandemic crisis in the Czech Republic. In reaction to this, we also briefly covered that aspect, although it was not the main aim of our study. As a secondary contribution, we therefore report brief quantitative and qualitative insights regarding the influence of the COVID-19 pandemic on the examined activities.

Previously, many studies have explored the topics related to eHealth/mHealth services in other countries and in different age cohorts (e.g., [21–26]). However, we are among the first to report to extent the young lay users interact with eHealth/mHealth in the context of the Czech Republic. In that sense, our perspective complements the provider perspective previously described by Klocek et al. [27] and a perspective focused on mHealth apps introduced by Smahel, Elavsky and Machackova [28]. In addition, carrying out a study which took a psychological approach, Knapova, Klocek and Elavsky [29] examined eHealth services in a cohort of older Czech adults.

#### **2. Methods**

#### *2.1. Procedure and Participants*

Our self-report, cross-sectional survey study explored eHealth/mHealth services usage among young adults aged 18–29 who live in the Czech Republic. No incentives were offered for participation. Data were collected using *1ka.si*, a survey research platform operated by the Centre for Social Informatics, at the Faculty of Social Sciences, University of Ljubljana. Considering the characteristics of the target population, we engaged students from our institution into the research process. This had also specific teaching objectives. A group of 16 master students volunteered in translation and data collection activities in return for a course credit.

The students were instructed to share the link in several Czech social network groups frequently visited by their peers (mostly other university students) with the aim to achieve a broad coverage. Attempting to extend the reach of the survey towards non-studying young adults, the students were encouraged to distribute the link on their personal profiles (snow-ball sampling). Given this approach yielded a portion of responses from a different population than our target age cohort, we filtered out those responses during the data analysis phase (Section 2.3).

The survey was active from 3 April to 6 May 2020. During this period, it was opened by 1081 individuals, of whom 606 started responding and 495 completed it (81.68% completion rate). A total of 46% of those who opened the link came from Facebook and 5%

from Instagram. Another 47% were marked by the survey platform as "direct links", which means the referral source was not recognized due to the way inter-website referral mechanisms presently operate. Table 1 summarizes the composition of our sample after data filtering (see Section 2.3).

**Table 1.** Basic demographic characteristics of respondents (N = 299).


*2.2. Survey Development*

#### 2.2.1. General Considerations

As a baseline for this exploratory descriptive survey, we used the list of health information and eHealth/mHealth activities compiled by Leung and Chen (2019), drawing on the extant literature and a focus group with students. Leung and Chen's study examined a broader issue of eHealth/mHealth technology readiness and acceptance [30], and therefore clearly went beyond the description. We did not replicate their survey instrument in full length, as our intention was not to contribute to the research field of technology acceptance processes [31] as such. Rather, we wanted to gain an initial understanding of how frequently the different types of activities by which Czech young adults use various mHealth/eHealth services. We took this route because the research on eHealth/mHealth services in the Czech Republic is sparse and the related gap in knowledge is significant. Hence, in this stage, we prioritized the simplicity and short length of our instrument, aiming to gain initial insights from a convenience sample of as many respondents as possible.

Regarding the instrument adaptation, we make use of the conceptual dichotomy differing between information-based activities and utility-based activities [21]. This dichotomy is a simplification of the more complex categorization of the eHealth/mHealth activities, as presented in the introductory part of this paper. In short, Leung and Chen [21] suggested differing rather straightforwardly between "information-based activities (e.g., health information seeking) and utility-based activities (e.g., self-monitoring)". This differentiation was used as the basic guideline for the conceptualization of the activities examined, while also keeping in mind the more intriguing view summarized above, when adding new activities.

We preserved the logic of the original study we conceptually replicated, highlighting here two important features of the original study. First, by focusing our research on the lay public and their daily activities, we refrained from a more detailed exploration of digital tools used in formalized health interventions (category No. iv, as described above). This decision was due to the target population, who was assumably mostly asymptomatic. Second, we abstracted the study from aspects such as whether a particular eHealth/mHealth service is provided by a specialized mobile application or a standard web browser [26]. While eHealth and mHealth activities are employed through different technological means [32], treating those activities as technologically agnostic helped to streamline the data collection process.

In sum, we adopted 14 original items, omitted one item and added seven new items. We describe these modifications below together with the related concepts.

#### 2.2.2. Concepts Related to Information-Based Activities

In our survey, *information-based activities* consisted of two major subtypes. Being differentiated by the source of information, this was viewed from the lay user perspective. The concepts of *health information seeking* (marked "A" in our inventory) and *health tutorial* (marked "F") measured the pole of health information *consumption*. In the context of our research, we defined *health information seeking* as the activities carried out by an eHealth/mHealth user, trying to find a possible guidance in dealing with his/her own health issues [7,8]. Adopting the original Likert scale of four items, we added an item about seeking expert consultation on-line [14]. This was due to our awareness about an on-line, quite popular tele-consultation service (*ulekare.cz*), which offers short, text-based medical advice on a pay-per-use basis. In addition, *health tutorial* covered activities related to the eHealth/mHealth user's lifestyle management through technology, an activity associated with illness prevention and maintaining overall well-being [33]. Originally having two items, this Likert scale was adapted considerably. First, we split the original item "To seek information on diet, exercise, or fitness" into two items. This was to differentiate between "diet" and "exercise and fitness", aiming to obtain more fine-grained data. In addition, deemed of high importance by the group of research students participating in instrument adaptation, a new item ("To seek a description of exercising and/or to develop an exercise plan") was added.

In contrast to information consumption, the concept of *sharing experience* (marked "C") quantifies the health information *provision* pole of the continuum. With regard to this category, the motivation behind eHealth/mHealth usage is different. Broadly, sharing health-related experience is driven by pro-social motives such as striving to help others who cope with a similar health problem [34]. No modifications were done in this scale.

#### 2.2.3. Concepts Related to Utility-Based Activities

The second categorial group, *utility-based activities,* was represented by the concepts of *medical services* (marked "B"), *reminders* (marked "D") and *recording/monitoring* (marked "E"). In line with the policy-oriented categorization outlined in the Introduction, we define *medical services* as a class of electronic eHealth/mHealth services that digitally support (rather than directly constitute the core) interactions in a healthcare system [4]. In this Likert scale, we omitted item number 5 from the original survey ("To pay medical treatment fees"), as in the Czech context, the majority of costs is paid indirectly—i.e., through a compulsory

health insurance system. Inversely, we added an item covering ePrescription (*eRecept*). In the Czech Republic, ePrescription is an eHealth solution adopted wide-scale [5], yet coupled with a strong past controversy due to the implementation strategy chosen by the state [35]. Regarding the second category, *reminders* are digital functionalities that help the eHealth/mHealth users with medication adherence [36].

Lastly, the broad category *recording/monitoring* covered selected activities carried out typically by consumer-grade electronics [15]. Conceptually, we did not differ between the monitoring activities performed by the proponents of the self-tracking movement and the monitoring activities prescribed by a health professional, as these two seemed to gradually blur with each other [19,33]. Four new items were added to the original two, using the generic prefix "To record and monitor . . . ". First, we added " . . . weight and/or related parameters", as this is a popular feature of consumer electronics for personal health use [37]. Second, we wanted to broadly cover activities related to monitoring of "heart activity" (the generic wording was chosen intentionally) with one item. This was due to recent discussions regarding the potential of using consumer electronics for atrial fibrillation screening and recording of single-lead ECG, both worldwide [37,38] and locally. Third, we added one item regarding blood sugar monitoring. This activity is also moving towards the segment of consumer electronics for diabetes self-management and prevention, including reportedly the next generation of Apple Watch [39]. On the one hand, we did not expect to see a high frequency of this activity, considering the target population demographics. On the other hand, we anecdotally noted the popularity of an open-source mobile app for diabetes management, which has been used by some tech-savvy patients in the Czech Republic and studied by a local community of medical researchers [40]. Therefore, we deemed it important not to omit technologies for diabetes management entirely. Finally, we added one more generic, broadly-worded item, considering the rapid development of the consumer recording/monitoring area [37].

#### 2.2.4. Translation Procedure

Considering the target population characteristics, the survey was prepared in Czech only. Hence, the original questions and items were firstly translated from English into Czech. For translation, we followed a committee approach [41]. Although many researchers consider instrument backtranslation to be the mainstream approach, the committee approach offers some additional advantages [42]. Firstly, we assigned the original English instrument and the proposed modifications (drafted in English by the first author) to the group of master students (the same as described in Section 2.1). All students were English proficient (B2–C1). Then, the students were instructed to translate the instrument into Czech by reaching a within-group consensus. The students were also instructed to discuss the validity of individual items from their perspective. Then, the second author repeatedly interacted with the students and guided them throughout the process. Finally, both authors carefully reviewed both the adopted and new survey items in terms of clarity of the translation and appropriateness of their cultural adaptation [41]. Inconsistencies were discussed between the stakeholders until the final consensus was reached.

It is worth noting that Leung and Chen's work indeed represents an interesting step towards a possible standardization of measuring the extent of individual eHealth/mHealth activities. However, it is important to clarify that neither their nor our aim was to create a validated cross-cultural instrument in terms of common psychometric standards. Hence, we adopted the simplified translation procedure as described above.

#### 2.2.5. Levels of Measurement and Demographics Questions

For all the activity items, we used the original 5-point quantification, ranging from 1 = "never" to 5 = "very often". Aside from the responses to these items, we collected demographics information on gender, age, education, the number of inhabitants in the respondents' city of residence, and technology ownership. Within the demographics section, we did not ask about income, as the surveyed population were mostly students, hence the information would be of questionable value.

Aiming to use this information as a filter question, we explicitly asked our respondents about the country where they currently live.

#### 2.2.6. Special Treatment Due to the COVID-19 Pandemic

Given the period when our survey started, the respondents were also asked to estimate the extent of the impact of the COVID-19 pandemic on these activities. This variable (COVID 19 impact) was measured by a four-point Likert-type scale ranging from 1 = "significantly influenced" to 4 = "not at all influenced". We also provided our respondents with the opportunity of a free-text answer. This was to detail the nature of the impact from their subjective perspective.

#### *2.3. Data Analysis*

We used a filter question ("Where do you currently live?") to exclude 59 responses, namely those of the respondents presently living in Slovakia (51), Great Britain (2) and a few other countries (1 response per country). We also excluded 122 responses of those who were 30 and older, or below 18. Finally, 15 responses exhibiting "straightlining" [43] were excluded during the data cleaning process on a case-by-case basis.

Descriptive statistics was used to report our findings in means, standard deviations (SD), and percentages. Missing values (i.e., all items marked as "don't know/cannot evaluate" by individual respondents) were replaced with means. Fewer than 4% of the individual responses were missing per any item. The only exception was item E6 "To monitor my health conditions by other means than those above", where 32 (8.9%) datapoints were missing. The mean scales were then computed by averaging the items in the six categories. Higher mean scores indicate higher intensity of conducting activities aggregated in the respective category. Cronbach alpha calculations resulted in values equal or above 0.7 for the summary means, indicating the acceptable reliability of the scales. We used Jamovi (v. 1.1.9.0, open-source) for data analysis and OriginPro 2021 (v. 9.8.0.200, OriginLab Corporation, Northampton, MA, USA) for plotting the data.

Based on previous research [22,44], we expected to find differences for sex (male/female) and for presence/absence of chronic conditions (CCs). Regarding age, it should be noted that our target population was aged 18–29, and thus felt among "digital natives" [25]. Hence, in the analysis, we treated the age of respondents as invariant. To allow testing for the subgroup differences, we created a new categorial variable by combining two demographics attributes mentioned above. Using this new variable, we classified all survey responses accordingly. Namely, we coded them as follows: 1 = man without CC, 2 = woman without CC, 3 = man with CC, 4 = woman without CC. Using a significance level of 5%, we applied the Kruskal–Wallis test to compare the mean scores among the four subgroups. This test was accompanied by applying Dwass–Steel–Critchlow–Fligner (DSCF) pairwise comparisons [45] to check for differences between the individual pairs. The error bars displayed in the figures (Figures 1, A1 and A2) represent standard deviations.

**Figure 1.** Mean scores for categories stratified according to the participant subgroups. \* *p* < 0.05.

#### **Figure 1.** Mean scores for categories stratified according to the participant subgroups. *\* p <* 0.05*.* **3. Results**

1. To do self-education about a specific disease or medical

2. To search information about a specific disease or medical

the last column.

**3. Results** In Table 2, we report the results in the form of mean scores (M) and standard deviations (SD) for all respondents, and then they are stratified into the four subgroups (men/women with/without chronic condition). Overall, the most frequent category of activities was using digital technologies for *health tutorial* (M = 2.81, SD = 1.14), followed by *health information seeking* (M = 2.63, SD = 0.89). Applying the technologies in the context of In Table 2, we report the results in the form of mean scores (M) and standard deviations (SD) for all respondents, and then they are stratified into the four subgroups (men/women with/without chronic condition). Overall, the most frequent category of activities was using digital technologies for *health tutorial* (M = 2.81, SD = 1.14), followed by *health information seeking* (M = 2.63, SD = 0.89). Applying the technologies in the context of booking *medical services* or purchasing medicines and similar products was less frequent (M = 2.18, SD = 0.97). This was followed by the *recording and monitoring* of various patient data (M = 1.95, SD = 0.68).

booking *medical services* or purchasing medicines and similar products was less frequent (M = 2.18, SD = 0.97). This was followed by the *recording and monitoring* of various patient data (M = 1.95, SD = 0.68). **Table 2.** Total mean scores (SD) and mean scores per subgroups. Significant differences (*p* < 0.05) are indicated in bold in **No Chronic Condition Chronic Condition(s) χ2 All (N = Value Men (n = Women Men (n = Women**  A Kruskal–Wallis test showed that there was a statistically significant difference in mean scores for *health information seeking*, χ 2 (3) = 9.17, *p* = 0.027, with the following results: (i) M = 2.45, SD = 0.88 for men without CC (subgroup 1); (ii) M = 2.64, SD = 0.82 for women without CC (subgroup 2); (iii) M = 2.52, SD = 1.06 for men with CC (subgroup 3); and (iv) M = 2.82, SD = 0.85 for women with CC (subgroup 4). A significant difference was similarly found between subgroups 1 and 4, with women with CC scoring higher than men without CC (*p* = 0.020). A similar trend was noted regarding *health tutorial*, χ 2 (3) = 12.14, *p* = 0.007, with (i) M = 2.46, SD = 1.05 for men without CC; (ii) M = 2.87, SD = 1.13 for women without CC; (iii) M = 2.61 SD = 1.12 for men with CC, and (iv) M = 3.11, SD = 1.15 for women with CC. Using DSCF, a significant difference was found between subgroups 1 and 4, with women with CC scoring higher than men without CC (*p* = 0.006).

**62)**

problem. 2.96 (1.10) 2.73 (1.15) 2.94 (1.03) 2.84 (1.26) 3.22 (1.01) 10.44 **0.015** 

**(n = 109)**

**45)**

**(n = 83)**

*p***-**

**299)**

3. To search the nearest hospital or clinics. 2.45 (1.14) 2.39 (1.08) 2.55 (1.12) 2.27 (1.27) 2.45 (1.15) 3.31 0.346

Cronbach's alpha .86 .85 .85 .91 .85

4. To do self-diagnosing. 2.57 (1.09) 2.24 (1.08) 2.53 (0.95) 2.56 (1.22) 2.89 (1.12) 13.88 **0.003** 5. To find expert medical opinion. 2.40 (1.07) 2.22 (1.13) 2.49 (0.99) 2.24 (1.15) 2.49 (1.06) 5.94 0.115

**A. Health information seeking** 2.63 (0.89) 2.45 (0.88) 2.64 (0.82) 2.52 (1.06) 2.82 (0.85) 9.17 **0.027**

**Table 2.** Total mean scores (SD) and mean scores per subgroups. Significant differences (*p* < 0.05) are indicated in bold in the last column.


In regard to *medical services*, a statistically significant difference was found, χ 2 (3) = 23.42, *p* < 0.001, with (i) M = 1.77, SD = 0.81 for men without CC; (ii) M = 2.26, SD = 0.95 for women without CC; (iii) M = 2, SD = 1.03 for men with CC; and (iv) M = 2.46, SD = 0.96 for women with CC. DSCF yielded the following results. The mean scores differed significantly between subgroup 1 and subgroup 2 (*p* = 0.003), subgroup 1 and subgroup 4 (*p* < 0.001), and subgroup 3 and subgroup 4 (*p* = 0.026). Figure 1 shows the mean scores per individual subgroups marked with significance lines where appropriate. Appendix A provides additional figures (Figures A1 and A2) showing the mean scores of individual Likert-type items.

As this study was conducted at the beginning of the COVID-19 worldwide pandemic crisis, we asked our respondents whether the pandemic had had impacted their behavior related to health information seeking and eHealth/mHealth use. A Kruskal–Wallis was conducted to explore these differences. There was a statistically significant difference in the impact scores (reverse scoring) for the four groups, χ 2 (3) =16.3, *p* < 0.001. Pairwise

comparisons indicated that the mean score of the least impacted group, i.e., men without CC (M = 3.13, SD = 0.78) significantly differed (*p* = 0.003) from those of women without CC (M = 2.67, SD = 0.85). Men without CC also differed significantly (*p* = 0.007, *p* = 0.005) from those of men with CC (M = 2.53, SD = 0.97) and from those of women with CC (M = 2.61, SD = 0.94). That means the activities of subgroups 2–4 were significantly more impacted by the COVID-19 pandemic crisis than those of subgroup 1.

Some of the respondents offered a short free-text clarification regarding the nature of the COVID-19 impact. This provided some interesting insights. Three core topics mentioned were as follows: (i) the change in frequency related to *recording/monitoring* and *health tutorial*; (ii) *health information seeking* associated with the COVID-19 pandemic; (iii) change in *medical services* consumption patterns.

Regarding *recording/monitoring* and *health tutorial*, many respondents tended to associate eHealth/mHealth activities primarily with physical activities. This theme represented an important framing for many free-text answers.

*I stopped wearing the sport tracker, [as] I don't track my [physical] activity anymore.*

(R191, woman)

*The closure of fitness centers makes exercising impossible, so there is nothing [no data] to track.*

(R182, man)

*[The COVID-19 pandemic] results in decreased intensity of my eHealth technologies (smart-watch) use, as I spend more time at home, not using them.*

(R82, woman)

*[Due to the pandemic,] I search more the description of exercises and [other] inspiration for exercising at home or in the park.*

(R437, woman)

*Health information seeking* was largely associated with COVID-19, and frequently intertwined with the remaining conceptual categories, illustrating the multifaceted nature of the COVID-19 impact.

*I search [on-line] for [descriptions of] symptoms [and I watch] how the disease [COVID-19] spreads. I exercise more. I also buy protective equipment [on-line].*

(R437, woman)

The impact on *medical services* can be illustrated by the following answer.

*I use telemedicine and ePrescription more, so that I can avoid visiting the doctor office.* (R149, woman)

Interestingly, some of the respondents highlighted a certain positive impact of the COVID-19 pandemic on their personal development. While this theme was only loosely associated with eHealth/mHealth services per se, we highlight its arguable importance for some respondents.

*I don't spend 24/7 in the medical school [anymore], and I dedicate the time to myself. I hold a trainer license, so that I discover and design new things [exercises?] and test them on my own.*

(R271, woman)

#### **4. Discussion**

We conducted a descriptive survey study among young Czech adults aged 18–29. In this research, we focused on their behavior related to eHealth/mHealth services usage. Following the previous research of Leung and Chen [21], the central part of our survey was structured into six activity categories, of which we briefly discuss four with the top scores below. Then, we analyze the impact of gender.

#### *4.1. Health Tutorial*

The category with the highest mean score for our survey population was *health tutorial* (M = 2.81, SD = 1.14), with females scoring higher than men. This category covered activities related to diet, exercise and fitness. The popularity of this category was also supported by the qualitative data. Free-text answers related to this category and provided by those who shared more details regarding the COVID-19 pandemic impact were frequent. We speculate that the COVID-19 pandemic caused a considerable increase mostly in health tutorial activities.

The high popularity of this category among young Czech adults confirms the findings of Leung and Chen, who similarly reported these activities being the most popular among Hong Kong respondents. What is more, the popularity of these activities was highlighted in a number of other studies [46,47]. Interestingly, recent research has uncovered how health information is often consumed through social media platforms such as YouTube and Instagram [48,49], and this seems to be an important research theme for future studies due to the growing popularity of these platforms both in the Czech Republic and abroad.

#### *4.2. Health Information Seeking*

*Health information seeking* was the second most frequent class of activities popular with our respondents (M = 2.63, SD = 0.89). Again, this relative popularity follows the relative ranking order from the original study. Health information seeking consists of activities related to self-education, self-diagnosing and, broadly, health information consumption. Today, health information is seen as playing pivotal role in the process of realizing the vision of patient empowerment [50]. However, the nature of health information seeking carried out may impact the nature of the patient–physician relationship, considering that many patients bring their lay findings into the conversation in the doctor office [7]. Inversely, having concerns about hampering the relationship with their doctor, some other patients hesitate in openly discussing health information found on-line [25,51]. Trust appears to be a prominent factor, as the frequency of health information seeking seems to increase when patients believe the official treatment given by healthcare providers is ineffective [52]. Patients also seek health information when they want to acquire additional information following a medical consultation [25]. An important prerequisite for qualified health information seeking is digital health literacy [3]. The crucial role of this factor manifested especially during the recent pandemic crisis [53,54], and more research is needed to understand how to provide sound public health advice to lay public and fight the infodemic [55].

Presently, little is known about all these problems in the context of the Czech Republic. Future research is warranted to uncover what platforms health information seekers use and what obstacles they face when bringing acquired health information into the discussion with health professionals [56].

#### *4.3. Medical Services*

As another prominent category, our survey identified *medical services* as the third most popular category (M = 2.18, SD = 0.97). In this category, activities such as ePrescription pickup (dispensation), buying medicine on-line, and booking an appointment with a doctor were grouped. Clearly, the most common activity was ePrecription pickup (M = 2.49, SD = 1.39). This is understandable, as in the Czech Republic, the use of ePrescribing and eDispensing has been enacted as mandatory for vast majority of medicament types since 2019 [5]. During the COVID-19 pandemic in 2020 and 2021, there has been a further decline in using optional, paper-based print forms together with ePrescribing and eDispensing, in favor of using SMS and QR codes [57]. This shift was due to the fact that a considerable part of communication regarding both chronic and acute diseases was not realized face to face, because of epidemiological reasons. Interestingly, according to anecdotal reports, the COVID-19 pandemic radically transformed the previously bad image of ePrescription among the Czech medical practitioners [35] virtually overnight [58].

As previously described, we found significant differences regarding the use of ePrescription between men and women. This might be related to the fact that ca. one third (34%) of Czech women aged 15–49 use contraceptive pills [59], which may be prescribed electronically as well.

#### *4.4. Recording/Monitoring*

Finally, in the *recording/monitoring category*, we covered various activities related to the use of consumer wearables producing patient generated health data [60]. Clearly, these activities are less popular among our respondents (M = 1.95, SD = 0.78), but the frequency is still roughly in line with the results of Leung and Chen. The top position of recording and monitoring of the amount of exercise confirms the findings of Smahel et al. [28]. They found out that "counting steps" occupied a position among the top (21.6% monitor daily or almost daily), preceded in popularity only by monitoring calorie intake (24.1%). Strictly speaking, however, we only included activities resulting in machine-generated health data in this category, not user-generated observations/records such as diet diaries.

Importantly, as our brief qualitative data (i.e. the free-text answers) indicated, the COVID-19 pandemic crisis had a significant impact on the behavioral patterns related to the use of *recording/monitoring* eHealth/mHealth tools.

#### *4.5. Gender Differences*

Interestingly, in three of four categories discussed above, we found some support for the hypothesis that there are differences in the popularity of health information seeking activities between men and women. Namely, in two categories (*health information seeking* and *health tutorial*), we found statistically significant differences when comparing men without chronic conditions with women with chronic conditions. The remaining comparisons were not statistically significant. In one category (*medical services*), we found a higher differentiation of the scores. Overall, we observed that woman tended to score higher in many items when compared with men. This finding confirms the conclusions of previous research studies carried out in different countries e.g., [8,22], indicating that women perform these activities more often. However, a more fine-grained perspective on this problem would be beneficial, as different health information areas attract different levels of attention from the lay public [61].

Like our findings, representative data published by the Czech Statistical Office on health information seeking confirm the above trend. In the age group 16–24, considerably more women (59%) than men (33%) use the Internet to seek information about their health [59].

Importantly, we did not find statistically significant differences between men and women when examining the *recording/monitoring* category. This observation is in-line with the study of Smahel et al., in which gender was not confirmed as a predictor for more frequent mHealth apps usage. To clarify, we used a more broadly defined and fine-grained activity categorization. Differently put, we went beyond the conceptual scope of the study of Smahel et al., which was focused mostly on patient-generated health data and apps usage.

#### **5. Limitations**

We acknowledge a number of limitations concerning the design of our study. First, we used a convenience sample of young Czech adults, arguably being mostly students. Our findings thus cannot be generalized on the whole population. Second, we prioritized the simplicity and short time needed to complete the questionnaire by respondents over obtaining deeper insights into the research problem. Therefore, we decided to omit more complex questions such as what social media platforms the respondents use to seek and consume health content, or what concrete mHealth apps they benefit from at most. Clearly, all these questions are valid and important, but this study cannot provide adequate answers to them. Third, this survey was carried out at the beginning of the COVID-19 epidemic

crisis in the Czech Republic. Considering the enormous impact the pandemic has had in different areas of people's lives, it might be impossible to generalize the findings to pre-pandemic or after-pandemic constellations. This fact is obvious from the illustrative qualitative data, showing that many respondents changed their routines significantly.

Finally, we note again that the survey instrument underwent adaptation. That is to say that some individual Likert-type items forming the scales in the reference study of Leung and Chen were removed, and new ones were added. This makes it harder, though not entirely impossible, to draw strong quantitative conclusions about the differences in eHealth/mHealth usage patterns between the Czech Republic and Hong Kong.

#### **6. Conclusions**

Patients using various eHealth/mHealth services, either within existing healthcare systems or outside of them, pose a shift in the traditional paradigm of medical care [3,62,63]. Differently from many other countries, the Czech Republic previously seemed to be among the laggards in top-down eHealth implementations driven by the state [35]. However, this lagging does not necessarily apply to the eHealth/mHealth services consumption patterns in young Czech adults, some of whom seemingly use eHealth/mHealth technologies quite intensively as a part of their lifestyle management activities. Specifically, the most popular class of activities among our respondents was *health tutorial*. Based on this finding, policy makers and health professionals in the Czech Republic should consider how various types of health tutorials can be leveraged in terms of guiding the users towards information content that is relevant, accessible and medically sound. In our view, this represents a massive opportunity for prevention-oriented health interventions [33]. By unlocking the potential of innovative eHealth/mHealth solutions and health information programs, these interventions can target public health concerns related specifically to young adults, or even children and adolescents [64]. To date in the Czech Republic, however, such official programs are scant.

**Author Contributions:** Conceptualization, M.D.; methodology, M.D. and Z.S.; software, M.D. and Z.S.; validation, M.D. and Z.S.; formal analysis, M.D.; investigation, Z.S.; resources, Z.S.; data curation, M.D. and Z.S.; writing—original draft preparation, M.D.; writing—review and editing, M.D. and Z.S.; visualization, M.D. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding. The authors received support from an institutional fund (IP400040) and an internal grant (F4/23/2019) at the Faculty of Informatics and Statistics of the Prague University of Economics and Business.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data that support the findings of this study are available from the corresponding author, M.D., upon reasonable request.

**Acknowledgments:** We are grateful to the volunteer master students for their help with data collection. We also express our appreciation to all survey participants.

**Conflicts of Interest:** The authors declare no conflict of interest.

*Int. J. Environ. Res. Public Health* **2021**, *18*, x FOR PEER REVIEW 13 of 16

*Int. J. Environ. Res. Public Health* **2021**, *18*, x FOR PEER REVIEW 13 of 16

**Appendix A**

**Figure A1.** Mean scores for individual Likert-type items (part 1). *\* p < 0.05.* **Figure A1.** Mean scores for individual Likert-type items (part 1). \* *p* < 0.05. **Figure A1.** Mean scores for individual Likert-type items (part 1). *\* p < 0.05.*

**Figure A2.** Mean scores for individual Likert-type items (part 2). *\* p < 0.05.* **Figure A2.** Mean scores for individual Likert-type items (part 2). *\* p < 0.05.* **Figure A2.** Mean scores for individual Likert-type items (part 2). \* *p* < 0.05.

Men Women With chronic conditions

#### **References**


## *Article* **Sociodemographic Characteristics and Interests of FeverApp Users**

**Silke Schwarz <sup>1</sup> , David D. Martin 1,2, Arndt Büssing <sup>1</sup> , Olga Kulikova <sup>1</sup> , Hanno Krafft <sup>1</sup> , Moritz Gwiasda <sup>1</sup> , Sara Hamideh Kerdar <sup>1</sup> , Ingo Fingerhut <sup>3</sup> and Ekkehart Jenetzky 1,4,\***


**Abstract:** The FeverApp Registry is a model registry focusing on pediatric fever using a mobile app to collect data and present recommendations. The recorded interactions can clarify the relationship between user documentation and user information. This initial evaluation regarding features of participants and usage intensity of educational video, information library, and documentation of fever events covers the runtime of FeverApp for the first 14 months. Of the 1592 users, the educational opening video was viewed by 41.5%, the Info Library was viewed by 37.5%, and fever events were documented by 55.5%. In the current sample, the role of a mother (*p* < 0.0090), having a higher level of education (*p* = 0.0013), or being registered at an earlier date appear to be cues to take note of the training video, Info Library, and to document. The FeverApp was used slightly less by people with a lower level of education or who had a migration background, but at the current stage of recruitment no conclusion can be made. The user analyses presented here are plausible and should be verified with further dissemination of the registry. Ecological momentary assessment is used more than the information option, in line with the task of a registry. Data collection via app seems feasible.

**Keywords:** fever; FeverApp; ecological momentary assessment; user behavior; sociodemographic characteristics; registry; guidelines; feasibility; usability

#### **1. Introduction**

Mobile health applications (apps) are now widespread, although their use in Germany is still relatively low [1]. The Digital Health Care Act (DVG), which came into force in 2019, entitles approximately 73 million insured individuals in the statutory health insurance system to be provided with digital health applications (DiGA), which can be prescribed by physicians as an "app on prescription" and reimbursed by the health insurance fund. The introduction of this technology and accompanying structures creates challenges for patients, providers, and the industry. At the same time, it is becoming clear how large the influence of mobile technologies can be if they are used appropriately [2].

A child having a fever is one of the most frequent reasons for parents to consult a pediatrician [3,4]. Although a fever is known as a useful body reaction to fight the underlying pathogens [5], many parents feel insecure and anxious when dealing with a child with a fever. Lack of knowledge and wrong perceptions about fevers could be potential causes of this anxiety [6]. It has been recognized that parent anxiety could lead to mismanagement of fever [4], such as overuse of antipyretics [6–8] which could lead to unintentional poisoning [9,10]. Accordingly, researchers recommend not only clear communication between

**Citation:** Schwarz, S.; Martin, D.D.; Büssing, A.; Kulikova, O.; Krafft, H.; Gwiasda, M.; Hamideh Kerdar, S.; Fingerhut, I.; Jenetzky, E. Sociodemographic Characteristics and Interests of FeverApp Users. *Int. J. Environ. Res. Public Health* **2021**, *18*, 3121. https://doi.org/10.3390/ ijerph18063121

Academic Editors: Martin Wiesner and Björn Schreiweis

Received: 22 February 2021 Accepted: 12 March 2021 Published: 18 March 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

pediatricians and parents [11], but also educating parents about fevers and its benefits as well as about the warning signs that should induce seeking help [12–15]. Several studies have examined the influence of different manners of education regarding fever and found a positive outcome of attaining better knowledge and management of fevers [16]. However, in a scoping review, Arias et al. [17] concluded that both parents and healthcare personnel should be addressed by a "more standardized educational platform".

In 2019, six model proposals to create patient-related registries to address important subjects in health service research were selected and funded by the Federal Ministry of Education and Research (BMBF) in Germany. The six model registries focus on different subjects [18,19]. One of them, the FeverApp registry, aims to advance knowledge on fevers. The multiple objectives are: To achieve ecological momentary assessment (EMA) of data on febrile illnesses, to "assess guideline adherence", to increase the knowledge about parental fever management, to improve parental skill, knowledge, and confidence in fever management, and consequently, to reduce use of medication and overuse of healthcare visits [20]. For this purpose and as a means of data collection for the registry, a smartphone application, FeverApp, was developed. A particular potential for the users of health apps lies in the education about relevant topics and the possibility of increasing their own health literacy in order to be able to cope better with the course of diseases [21,22].

The question posed by this study is the sociodemographic characteristics of current users of the FeverApp and what feature is of interest to them according to their usage. This provides a basis for understanding which target group uses the app and which groups of users do not have good access to it, as well as which features of the app are relevant to which groups of users. This will assist in not only the planning of further educational content, specifically position based on user behavior, but also similar health apps with multiple features such as, visualization, information, and documentation. As this kind of documentation feeds the central registry for research on fever events, interactions in the app are monitored and first analyses are presented.

#### **2. Materials and Methods**

The FeverApp enables a parent to document their child's febrile illnesses in real time (Ecological Momentary Assessment), to learn about fevers, and how to safely treat it [20].

The submitted entries and interactions between different pages of the app are stored locally in the app within an open-source JavaScript database, PouchDB, which synchronizes it when online with Apache CouchDB. The latter is centrally located on the University of Witten/Herdecke's servers and is transformed daily into a Mongo-DB. Several relational data tables (CSV format) are extracted on demand through structured query language scripts and are processed in SPSS V25 (IBM Corp., Armonk, NY, USA, Supplementary Materials). These data represent the registry. A positive vote by the ethics committee of the University of Witten/Herdecke on pseudonymized data collection using an app was received (#139/2018), as well as a positive vote by our data protection service.

From September 2019 to July 2020, the FeverApp was accessible only through selected pediatric and adolescent practices for the validation of data and improvement of the app's usability. Physician clearance is required to analyze the validity of the data collected. The comparison between the data systematically collected by the pediatricians in the practice and the FeverApp registry data will be published separately. Since July 2020, access was facilitated for all interested practices. Each practice received a unique code that was shared with parents. Upon entering the practice code in the app, a family code was generated for each family user. Families had the option of so-called "family sharing", which means that an individual family code could be shared within a family with any number of smartphone users. Each smartphone user creates a user profile during installation, whereby all users (roles) of a family (code) could create profiles of their observed children or view them together. For each child (profile), one or more series of entries (loops, i.e., a series of entries) could be created over the course of a fever event. In addition, the users receive information on childhood fever via educational video (Figure 1a) and a detailed guideline orientated

multimedia Info Library with 23 chapters (Figure 1b). After documentation of multiple entries (Figure 1c), users could have an overview of all entries within a fever event in the "graph view" section of the app (Figure 1d). To enhance user experience, the app has the option of night mode, in which the appearance of the app background changes from light to dark. detailed guideline orientated multimedia Info Library with 23 chapters (Figure 1b). After documentation of multiple entries (Figure 1c), users could have an overview of all entries within a fever event in the "graph view" section of the app (Figure 1d). To enhance user experience, the app has the option of night mode, in which the appearance of the app background changes from light to dark.

number of smartphone users. Each smartphone user creates a user profile during installation, whereby all users (roles) of a family (code) could create profiles of their observed

the users receive information on childhood fever via educational video (Figure 1a) and a

*Int. J. Environ. Res. Public Health* **2021**, *18*, x 3 of 12

**Figure 1.** (**a**) Educational video; (**b**) Info Library-Menu; (**c**) Entry start; and (**d**) Graph view. **Figure 1.** (**a**) Educational video; (**b**) Info Library-Menu; (**c**) Entry start; and (**d**) Graph view.

As shown in Figure 1b,d, at the bottom of the screen the user can choose to switch between the overview (i.e., Homepage) and Info Library, in which they have access to information on fevers (Figure 1b). Such navigations within the FeverApp, called interactions, are also recorded. Besides reporting FeverApp user characteristics, use of the documentation and information features is the focus of this first evaluation. For clarity, the presentation of user behavior distinguishes between those who did not use the documentation or the Info Library at all, slightly (one to two times), or intensively (more than twice). This is only possible by separately tracking the interactions made by the user. In this publication, the following were analyzed for the entire cohort and not only for selected users: The use of the opening video and guide information area as well as the documentation activity, i.e., the actual registry. The evaluation was carried out with IBM SPSS Statistics 25 (IBM Corp., Armonk, NY, USA) in the form of absolute and relative frequencies. The explorative p-value was determined with the help of the Chi² test or Fishers exact test. We will report whether the information video intended to convey the core infor-As shown in Figure 1b,d, at the bottom of the screen the user can choose to switch between the overview (i.e., Homepage) and Info Library, in which they have access to information on fevers (Figure 1b). Such navigations within the FeverApp, called interactions, are also recorded. Besides reporting FeverApp user characteristics, use of the documentation and information features is the focus of this first evaluation. For clarity, the presentation of user behavior distinguishes between those who did not use the documentation or the Info Library at all, slightly (one to two times), or intensively (more than twice). This is only possible by separately tracking the interactions made by the user. In this publication, the following were analyzed for the entire cohort and not only for selected users: The use of the opening video and guide information area as well as the documentation activity, i.e., the actual registry. The evaluation was carried out with IBM SPSS Statistics 25 (IBM Corp., Armonk, NY, USA) in the form of absolute and relative frequencies. The explorative p-value was determined with the help of the Chi<sup>2</sup> test or Fishers exact test. We will report whether the information video intended to convey the core information was used, and by whom (Table 1). Furthermore, we compare the user characteristics with the use of information (Info Library, Table 2) and the documentation (registry entries, Table 3).

#### mation was used, and by whom (Table 1). Furthermore, we compare the user characteris-**3. Results**

#### tics with the use of information (Info Library, Table 2) and the documentation (registry *3.1. Participants' Description*

entries, Table 3). For app use, 1451 families from 86 pediatric/adolescent practices registered by the time of this evaluation (31 October 2020). The largest practice recruited 45% of families (*N* = 649), the second largest 7% (*N* = 103), and 64 practices had just begun participation at the time of the evaluation and were feeding the registry with single-digit numbers of participating families.

A total of 1592 users or installations of the app from 1451 families can be identified, not all of whom provided information on their sociodemographic data (marked with asterisks in the tables). Mothers were the most common family role at 83.4%, followed by fathers at 15.4%, and 1.2% consisted of other roles. The app was mostly used by one person in a family (91.4%). In 7.8%, two users entered the data and in 0.8%, more than two users entered the data. The platform used was slightly more often Android (57.3%) than iOS (42.7%).

Out of 1494 level of education entries, the majority of the users, 48%, had higher school education ('Abitur'), i.e., general qualification for university entrance, 22.4% had 'Fachhochschulreife' (vocational diploma), 23.1% had 'Mittlere Reife' (intermediate school certificate), 5.5% had a 'Hauptschulabschluss' (basic school certificate), and 1.1% were without a school certificate. Of the users, 18.6% did not state their ethnic origin, and the remaining (*N* = 1296) were 89.0% German, 1.8% Turkish, and 1.4% Polish. The age of the users (*N* = 1503) was on average 35.5 ± 6.6 years with a range of 14–68 years (IQR = 31–39). Accordingly, 10 individuals would have been 14 years old. These are presumably misrepresentations, since the default setting indicates the year 2006, which must be actively corrected. Except for 89 users who did not indicate any age at all, 93.8% of the age data are available.

We do not know the exact number of children per family since profiles created by users in the role of "mother" (*N* = 1136) were used as a substitute for the number of children. These were categorized as "users with one child" (61.2%), "with two children" (31.5%), or "with more than two children" (7.3%). Thus, on average, 1.16 ± 0.47 individuals entered data per family and created profiles of 1.38 ± 0.72 [range: 0–6] children.

There was almost no difference between partners and single users when the user did not document at all (45.5% vs. 46%) or when they created at least some entries (53% vs. 53%). Since not all partners register, we do not know exactly how many of the families are single parent families and have therefore restrained from this analysis at this point in time.

#### *3.2. Description of Interactions*

The 1592 users performed a total of 175,564 interactions and 9275 entry series (loops). The median is 58 interactions (IQR = 30–133; range: 8–1815). The corresponding distribution is left-sloping, i.e., most users performed fewer interactions. Of the 175,564 interactions, the majority (58.2%) were related to a page selection. In 23.8%, it was starting, restarting, or pausing the app and in 9.1%, it was selecting a child's profile. Deactivating night mode (8.0%) was significantly more common than activating night mode (0.4%).

Furthermore, the 9275 series of entries were made by a total of 55.5% of all users, whereby these can be differentiated into low (17.2%) and more intensive users (38.3%) of the documentation function. The Info Library was visited by 37.5% of all users, of whom 15.0% viewed only one to two pages and 22.5% viewed more than two pages. Those who did not document anything also did not use the Info Library, while those who documented often also looked at the Info Library (Chi<sup>2</sup> value: 315; *p* < 0.001).

#### *3.3. Use of the Educational Video*

In the following, we will report whether the information video that was intended to convey the core information was used, and by whom (Table 1). The four-minute educational video started at a total of 733 times (0.4% of all interactions). Out of the 1592 participants, 41.5% (*N* = 660) watched the video, of which some users (*N* = 43; 2.7%) watched it multiple times (two to seven times). In terms of family position, 43.4% are mothers, 34.7% are fathers, and 16.7% are others. Mothers watched the input video most often (*p* = 0.009). Those who watched the video also used the Info Library more intensively (*p* = 0.010; Chi<sup>2</sup> value = 9.1) and documented in the app (*p* < 0.0001; Chi<sup>2</sup> value = 28.9) (Table 3). There were no differences in opening and viewing the educational video among the four selected age groups (<30, 30–34, 35–39, and ≥40 years), but individuals with a higher level of education (see Table 1 for definition) were more likely (*p* = 0.0013; Chi<sup>2</sup> value = 21.8) to view the video. Users

who viewed the opening video tended to use the Info Library (*p* = 0.0104; Chi<sup>2</sup> value = 9.1). Users who watched the educational video were definitely more likely to document in the FeverApp (*p* < 0.0001; Chi<sup>2</sup> value: 29.7). However, the longer the installation of the app, the more the features were used (*p* = 0.0002; Chi<sup>2</sup> value = 22.4).


**Table 1.** Viewing the opening video.

\* Missing values due to missing information up to user number 1592; # Fisher's exact test instead of Chi<sup>2</sup> , due to too small cell population;

\$ Based on 1136 mothers with profile information, not all 1592 users, therefore smaller number of cases. Bold if exploratory *p*-value < 0.01.

#### *3.4. Use of the Info Library*

The FeverApp Info Library contains 23 information pages, of which the "Warning signs of fever" section was visited most often (484 times; 11.5%). The Warning Signs section,

located directly after the Fever video in position two of the Info Library, provides parents with information about critical signs and conditions for which a medical presentation should be made. Other preferred areas of information included "1. What is fever" (9.4%) with information regarding fever definition, "3. Certificate for employers" (6.7%) with information about German regulations about sick leave when one's child is sick, and the table of contents (6.4%), also located at the top of the Info Library. Frequency of measurement (5.8%), accompanying symptoms (5.1%), and information on "correct" fever measurement (4.7%) were also visited slightly more often than the other information pages.

The use of the Info Library (Table 2) was divided into three groups: 22.5% used it more than three times, 15.0% seldom used it (once or twice), and 62.5% did not use it at all. It was mainly the fathers who did not use the Info Library (*p* = 0.0002; Chi<sup>2</sup> value = 22.4), as well as users who had only recently installed the app (*p* < 0.0001; Chi<sup>2</sup> value = 110.1). Age and number of children had no influence on the use of Info Library. The maximum level of the child's fever (*p* < 0.001) was related to the intensity of Info Library use (Figure 2). *Int. J. Environ. Res. Public Health* **2021**, *18*, x 7 of 12

**Figure 2.** Use of the Info Library and highest reported temperature (circles are outliers). **Figure 2.** Use of the Info Library and highest reported temperature (circles are outliers).



High ('Fachhoch-schulreife') 334 (22.3) 132 (39.5) 60 (18.0) 142 (42.5) Moderate ('Mittlere Reife') 345 (23.1) 149 (43.2) 63 (18.3) 133 (38.6)

**Documented So Far (0 Entries)** 

Mother 1244 (83.4) 533 (42.8) 213 (17.1) 498 (40.0)

Total 1492 \* (100) 679 (45.5) 258 (17.3) 555 (37.2)

Highest ('Abitur') 717 (48.0) 304 (42.4) 116 (16.2) 297 (41.4)

certificate) 98 (6.6) 53 (54.1) 22 (22.4) 23 (23.5) Total 1494 \* (100) 638 (42.7) 261 (17.5) 595 (39.8)

Android 874 (57.3) 363 (41.5) 145 (16.6) 366 (41.9)

iOS 651 (42.7) 295 (45.3) 122 (18.7) 234 (35.9)

Total 1525 \* (100) 658 (43.1) 267 (17.5) 600 (39.4)

2019 404 (25.4) 74 (18.3) 66 (16.3) 264 (65.3)

<30 years 221 (14.7) 102 (46.2) 47 (21.3) 72 (32.6) 0.0088 (17.1) 30–34 years 486 (32.3) 209 (43.0) 85 (17.5) 192 (39.5)

January to May 2020 306 (19.2) 90 (29.4) 50 (16.3) 166 (54.2) June to October 2020 882 (55.4) 545 (61.8) 158 (17.9) 179 (20.3) Total number 1592 (100) 709 (44.5) 274 (17.2) 609 (38.3)

35–39 years 460 (30.6) 179 (38.9) 68 (14.8) 213 (46.3)

**Documentation (1–2 Entry Series)** 

Absolute frequency N (Relative frequency %)

**the Documentation (>2 Entry Series)** 

**(Chi²-Value)** 

**<0.0001**  (32.4)

0.0316 (13.8)

0.0621 (5.56)

**<0.0001**  (310.0)

**Variable All Users** 

Low ('Hauptschulabschluss' or no

Type of role

Education status

Operating

User

Agegroup

system

registration


**Table 2.** *Cont.*

\* Missing values due to missing information up to the user count of 1592; \$ Smaller number of cases because the base is 1136 mothers with profile information and not all 1592 users. Bold if exploratory *p*-value < 0.01.

#### *3.5. Documentation Use*

Users (Table 3) who saw the educational video or consulted the Info Library were also more likely to document. These were primarily mothers and users from the early period (2019) and without a migration background. Users from 2019 also had the longest observation interval, so were more likely to have a fever event and hence documentation.


**Table 3.** Use of documentation function depending on the status of parents, education, age, and children.

\* Missing values due to missing information up to the user count of 1592; \$ Smaller number of cases because the base is 1136 mothers with profile information and not all 1592 users. Bold if exploratory *p*-value < 0.01.

#### **4. Discussion**

The FeverApp is a documentation tool for the FeverApp registry, which also provides parents with guideline information on the subject of fevers to increase parental knowledge and confidence. While other health apps with a focus on fevers emphasize on surveillance of specific illnesses in specific seasons, such as influenza [23,24], Dengue fever [25], or malaria [26], the FeverApp registry concentrates on pediatric fever as a symptom and a common reason of parental anxiety and overuse of antibiotics, antipyretics, and health services. To the best of our knowledge, there are very few research-based mobile applications that focus both on educating parents and include a diary-function to record fever episodes and the way parents manage fever.

The main users of the FeverApp are mothers (83%) in their 30s (63%), and persons with a higher level of education. These findings support the fact that younger individuals with a higher level of education are more likely to use mobile health apps [27]. The proportion of mothers with nationalities other than German using the app (11%) is lower than the reported, with 18% of non-German mothers living in Germany in 2019 [28]. However, at the current stage of recruitment no conclusion can be made. In 93% of cases, one to two child profiles are created, which reflects the reality of the number of children per family in Germany in 2019, where 88% have one or two children [29].

As indicated by other studies [30], it is important to maintain a close relationship between users and practitioners in order to achieve the best possible results from health apps. While recruitment style is possible through online advertisement [23] or word of mouth [24], in the FeverApp registry we recruited parents through pediatric and adolescent practices. More than half of the registry data collected are from the last five months, indicating evolving recruitment dynamics of providing access to the app through interested practices.

The quality of the app and its user experience [31] determine if the users continue using the app for its intended purposes and therefore require constant evaluation [32]. The FeverApp and the registry data quality are regularly assessed in three different ways: (a) A voluntary feedback function in the app, (b) active qualitative testing with selected users, and (c) analysis of user behavior, i.e., interactions, based on registered data. The last category is the focus of the current article by investigating the level of interaction between information and documentation in detail. What is unique about this form of study is that it analyzes user behavior (regarding documentation in the registry and information) in the app by observing interactions. This user testing looks at the complete collective, as opposed to the other two forms of user feedback, which could become a new standard for health apps.

Data collection via the FeverApp offers an advantage, where: (a) Data collected are pseudonymized, (b) all answers are voluntary, and (c) users are informed, by pediatricians and through the app, that their entries are used for the purpose of research. Previous studies indicate that once individuals know that the intention of data collection is for research, they do not mind sharing their information [33], despite concerns about secondary use of information, e.g., by other companies [27]. Despite the fact that each part of data entry is voluntary, the data are relatively complete.

Based on a systematic review, Young et al. [16] suggested that the best way to educate parents about fever management is a multidimensional education using different means such as text, video, and verbal materials. As mentioned before, the FeverApp's Info Library is a multimedia library, consisting of videos, images, and text (Figure 1b, in addition to the documentation tool in the app which has of itself an educative aspect. The aim of the educational fever video developed in collaboration with the German Association of Pediatric and Adolescent Doctors (Berufsverband der Kinder- und Jugendärzte, BVKJ e.V.) is to inform parents about the essential elements of recommendations of the BVKJ e.V. and the German Society of Pediatrics and Adolescent Medicine (Deutsche Gesellschaft für Kinder- und Jugendmedizin, DGKJ) [34]. So far, 41.5% of all users who installed the app took note of the short video at the beginning. Although the video can also be accessed later via the Info Library, it was not frequently used there. Considering earlier

research proposing that although parents appear to prefer videos in comparison with other means of education [17], knowledge transfer remains a challenge even with informative videos. However, it could be that previous users already have, or believe they have, a basic knowledge that is sufficient for them, so that further immersion seems unnecessary. Mothers and those with higher levels of education were more likely to watch the video than to use the Info Library. Likewise, parents who installed the app in 2019 were more likely to have watched the educational video (51%, Table 1).

It was found that people who watched the educational video also consulted the Info Library more intensively (26% versus 20%). Similarly, parents were more likely to use the Info Library if they had installed the app in 2019 (36%), i.e., the longer they had the app, the more likely they were to use it. For instance, 29% of the parents used the Info Library the first half of 2020 compared to 14% in the second half of 2020. The same trend was observed for viewing the educational video. This may be related to both the length of time these parents were observed in the registry and the type of application they used. Fathers were the least likely to use the Info Library intensively (12%). Mothers in their 30s (25%) consulted the Info Library slightly more often than those who are older or younger, although this is not statistically significant. This raises the question as to whether the younger, usually less experienced, individuals might already be well-informed and/or whether they seek their information from other sources. They may also be less aware of their need for information. It is striking that the 46% of the users who initially viewed the video about fever compared to the 36% who did not, were also those with intensive documentation. Those users (73% vs. 21%, Table 3) also used the Info Library. Both are strongly associated with intensity of documentation of fever to the maximum level of the child's fever in the app (Figure 2). This would indicate the importance of parents having sound technical information on fevers at hand. With a higher temperature, more information seems to be needed, for instance, regarding medication, warning signs, etc. The app is most heavily used by parents between the ages of 30 and 40. Individuals with higher educational levels and users of a cell phone with Android also documented more. The above-average use of the iPhone (43%) is striking, as the national percentage of iPhone users was 29% in September 2020 [35]. The distribution of educational level among the users was in accordance with that of 30- to 40-year-old parents in Germany.

One strength of this study in comparison with other studies on the subject of pediatric fever [23] is the collection of basic demographic information of users, in which the trend of fever management can be further investigated based on age, gender, etc. The strength of this registry is the current achieved number of cases. With nationwide coverage, the use of FeverApp is increasing. The extent of missing data is low despite the fact that it is completely voluntary. Although the unsystematic distribution of missing data limits the significance of the study, it does provide a naturalistic registry. The validation of the data through parallel recording in several family pediatric centers is currently taking place and will be analyzed separately.

After one year, the registry data collected with the FeverApp provides a good overview of basic user behavior. This is important both for the further development, distribution strategy, and for the interpretation of analyses of the registry data collected with the app. Furthermore, the manner in which to present information in health apps could be changed depending on the sociodemographic of its users.

#### **5. Conclusions**

The FeverApp provides the possibility to collect real-time data, educate parents about fevers, and provide insights to researchers and pediatricians regarding management of fevers at home. This study of the app's interactions offers additional information on the behavior of the app's users. It was shown that recording changes between screens could be useful. The recording and analyses of interactions could be extended, for example, more detailed analyses concerning the timing of the navigation between information and documentation. The educational opening video was viewed by 41.5% of the 1592 users, who were correspondingly more likely to also document fever events (62.0%; *p* < 0.0001) or consult the Info Library (40.9%, *p* = 0.0104). The documentation function was used more than the information option, whether via content of Info Library or an integrated educational video, in line with the task of a registry. It seems feasible to have data collection as a registry by means of an electronic case report app.

Overall, plausible trends could be demonstrated. The observation of user behavior was an important measure to further develop the registry and FeverApp. Whether people with a higher educational status, who use the app more frequently, are conversely more insecure in dealing with fever in their child than people from other educational backgrounds seems questionable, rather, there seems to be clearly a stronger thirst for information. The data collected provide an initial basis for controlling bias in this app-based registry. Communicating guideline knowledge is challenging. This specially developed app can do this in a way that is accessible to many and that can be monitored and continually optimized. User behavior of different subgroups in the registry and the potential clinical impact of information through the app will be further evaluated in the coming years.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/1660-460 1/18/6/3121/s1.

**Author Contributions:** Conceptualization, S.S. and A.B.; methodology, E.J.; software, D.D.M. and E.J.; validation, E.J. and S.H.K.; formal analysis, E.J.; investigation, S.S., D.D.M., A.B., O.K., H.K., M.G., S.H.K., I.F. and E.J.; resources, D.D.M. and I.F.; data curation, E.J.; writing—original draft preparation, E.J., S.H.K., A.B., S.S., H.K. and M.G.; writing—review and editing, S.H.K., E.J. and D.D.M.; project administration, S.S.; funding acquisition, D.D.M. and E.J. All authors have read and agreed to the published version of the manuscript.

**Funding:** A total of 20% of the funding is provided by the Software AG Foundation and 80% of the funding is provided by the German Federal Ministry of Education and Research (grant number: 01GY1905) for the years 2019–2024.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Ethics Committee of the University of Witten/Herdecke (protocol code #139/2018 and 13 December 2018).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available in SPSS format as supplementary material.

**Acknowledgments:** We thank all parents and pediatricians helping us on this project. We are thankful for language editing by Patricia Meinhardt.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Do Market Characteristics Matter? Factors Associated with Health Information Exchange**

**Na-Eun Cho 1,\*, KiHoon Hong <sup>1</sup> and Jongwha Chang <sup>2</sup>**


**Abstract:** This study explores factors associated with the breadth (extent) and depth (level of detail) of digital information exchange among stakeholders in health information technology (IT) systems. Annual and IT surveys of the American Hospital Association and the U.S. Census Bureau's smallarea income and poverty estimates from 2014–2016 were analyzed for associations between key factors and breadth and depth of information exchange. OLS Regression was used with a sample consisting of 10,040 year-hospital observations. We found that hospital-level variables such as size, ownership type, system affiliation, physician-hospital arrangement, and revenue model affect information exchange. We further found that market-level variables such as concentration ratio, urbanness, and median household income, although they directly affect information exchange, do not moderate the relationship between hospital-level variables and information exchange. Our study fills a gap in the previous literature arising from the lack of research on the determinants of health information exchange.

**Keywords:** health information technology; information exchange; hospital; market

#### **1. Introduction**

Many economically developed countries have put great effort into implementing health information technology (IT) to reduce healthcare costs and improve quality of care [1,2]. Examples include the Health Information Technology for Economic and Clinical Health (HITECH) Act passed in the U.S., which provided USD 27 billion in incentive payments to encourage adoption of electronic health records over a 10-year period [1], and the U.K. government's 2018 announcement of an approximately USD 540 million investment in hospital information technology [2]. Although the U.S. Office of the National Coordinator for Health Information Technology (2018) reported 96% of all non-federal acute care hospitals to possess a certified health IT system as of 2017 [3]; however, there exists a gap between health IT adoption and use, specifically with regard to information exchange among healthcare providers. Despite a high rate of health IT adoption, 32% of patients who visited a healthcare provider within the past 12 months reported having to repeat a procedure, provide their medical history again, or bring a test result to an appointment because of the unavailability of prior data [4]. The implication is that there exists many hospitals that have adopted health IT and do not successfully share electronic information among providers.

Scholars, taking note of the importance of information exchange, have begun to focus on how information is incorporated and shared in healthcare settings [5]. A study comparing length of stay and readmission rates between instances of assimilation of electronic health records, measured as achievement of meaningful use and mere adoption, found patient outcomes to improve only in the former case [6]. Another study found that electronic mobilization of healthcare information across organizations could save as much

**Citation:** Cho, N.-E.; Hong, K.; Chang, J. Do Market Characteristics Matter? Factors Associated with Health Information Exchange. *Int. J. Environ. Res. Public Health* **2021**, *18*, 11976. https://doi.org/10.3390/ ijerph182211976

Academic Editors: Martin Wiesner, U Rajendra Acharya, Alessio Ishizaka and Björn Schreiweis

Received: 4 August 2021 Accepted: 12 November 2021 Published: 15 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

as USD 1.9 million in emergency department settings annually [7]. These results commonly support the idea that it is the actual use of health IT, not mere adoption, that determines the extent of benefits, which a hospital gains from IT investment [8].

Increasing interest notwithstanding, little is known about what affects information exchange in the healthcare setting. Determinants of adoption have been widely studied [9–13], but information exchange has not. Exceptions, such as studies examining determinants of meaningful use [14,15], have been based on early stage of meaningful use that has not involved robust electronic health information exchange. That only the final stage of meaningful use (termed "promoting interoperability" in fall 2018) includes electronic information exchange as a major requirement, and its reporting years are ongoing (2019–2021) [16], suggests that prior studies provide limited understanding of what affects the exchange of health information. Recently, scholars have started examining the factors associated with health information exchange, but they have either focused on a few individual relevant factors such as system membership or incentives [17–19] or on one aspect of information shared electronically, e.g., volume [20].

With the goal of providing a more comprehensive picture of drivers of health information exchange, the present study examines specific factors that affect information exchange among health IT systems. We investigate, based on findings reported in the prior literature about their effect on "mere adoption" of health IT [9–12], whether factors such as size, concentration, system affiliation, ownership type, teaching status, urbanness, etc. affect information exchange similarly or differently. Responding to the increasing importance of information exchange among healthcare providers, as detailed above, our study focuses on configuration strategies found to affect hospital performance [21], specifically, what affects the breadth (i.e., the extent to which patient health information is shared electronically among stakeholders) and depth (i.e., the level of detail at which patient health information is shared electronically) of information exchange among stakeholders.

Using data sets from the American Hospital Association's (AHA's) annual and IT surveys, and the Census Bureau's small-area income and poverty estimates for the 2014–2016 period, we estimate the association of key factors with breadth and depth of information exchange and derive important policy implications from evaluations of previously studied determinants of health IT adoption. The paper is organized as follows. In Section 2, we describe our data and ex-ante predictions. Results are discussed in Section 3. We present our conclusions in Section 4.

#### **2. Materials and Methods**

We compiled 2014–2016 data from multiple sources including the Annual and IT surveys of the American Hospital Association (AHA) (https://www.ahadata.com/ accessed on 1 March 2021) and U.S. Census Bureau's small-area income and poverty estimates (https://www.census.gov/ accessed on 1 March 2021). The current research did not need to be reviewed and approved by the institutional review board because these data sets do not involve "human subjects". Annual AHA surveys include not only hospital-level information, such as bed size, hospital ownership type, teaching status, system affiliation, physician-hospital integration, and revenue models, but also market-level information, such as urbanness and market concentration, and AHA IT surveys provide detailed information regarding breadth and depth of information exchange. The U.S. Census Bureau's small-area income and poverty estimates provide information about median household income proximate to each hospital's location.

To examine the association between various factors and breadth/depth of information exchange, we selected 10 determinants of health IT adoption, such as electronic medical records or electronic health records. We chose seven hospital-level (bed size, hospital ownership type (two dummy variables to categorize for-profit, government, and nonprofit), teaching status, system affiliation, physician-hospital integration, and revenue models) and three market-level (concentration, urbanness, and median household income) variables.

#### *2.1. Hospital-Level Determinants of Information Exchange*

Our ex-ante expectation of the relation of bed size to breadth and depth of information exchange is positive and statistically significant. Just as their more abundant resources make it easier for larger than for smaller hospitals to adopt health IT (i.e., electronic medical records and electronic health records) [10–12,22], so superior technical capabilities make it easier for larger hospitals to exchange information electronically with other stakeholders.

Results for ownership type were inconsistent. Previous research has suggested that health IT is more likely to be adopted and implemented by for-profit hospitals with greater financial resources than by nonprofit or government hospitals [11,12,23]. Effects of ownership type, however, were statistically insignificant in one study [12] and significant in another [11], and other studies have found nonprofits, because they view it as part of their public service responsibility, to in fact be more likely than other hospital types to implement health IT [9]. Due to these inconsistencies, effect of ownership type on information exchange is not predicted in our study.

Prior literature also suggests that young medical students, residents, and fellows at teaching hospitals, being more comfortable with new technology, may find it easier to share information electronically with others [11]. Thus, our ex-ante expectation for the effect of a teaching hospital on information exchange is positive and statistically significant. System affiliation is also expected to increase information exchange, affiliated institutions that share organizational practices, culture, and policies [11,18] being more likely to share health information as well. Physician-hospital integration, that is, employment of physicians by hospitals, is expected to provide greater control over physician behavior and thus facilitate hospitals' implementation of new technology [24]. Thus, our ex-ante expectation for the effect of this variable on information exchange is positive and statistically significant. Different incentive models such as bundled payment or alternative payment models are expected to be one of the key organizational determinants of information exchange [17,19,20]. Similarly, we expect hospitals with capitation model-based revenue, whereby providers receive a fixed per person payment regardless of actual services provided, to be more incentivized to share detailed information in order to reduce overall costs. Our ex-ante expectation of capitation revenue is thus positive and statistically significant.

#### *2.2. Market-Level Determinants of Information Exchange*

We measured concentration, one of the environmental variables included in our study, as the Herfindahl index. Hospitals facing greater competition with attendant pressure for cost-reduction, although they might be expected to put greater effort into actively sharing information electronically, may find it difficult to do so owing to differences in IT system vendors or technical capabilities across providers and hospitals. This might explain the statistical insignificance of the effect of competition in a previous study [11]. Our ex-ante expectation of the effect of competition on information exchange is thus not predicted.

The ex-ante expectation of the effect of urbanness on information exchange is negative and statistically significant. Whereas previous literature has suggested that urban hospitals with greater resources are likely to adopt health IT [20], once equipped with IT systems, rural hospitals that share similar cultures are likely to find it easier to share patient information. That the ex-ante prediction of the relation between munificence and information exchange is positive and statistically significant corroborates our expectation that hospitals located in affluent areas, other things being constant, are more likely to appeal to patients by more effectively employing information technology, such as electronic medical records or electronic health records.

#### *2.3. Measurement*

We employed a multiple regression analysis with 10 variables, as follows.

*In f ormation Exchangeit*

= *α* + *β*1*Bed Sizeit*

+*β*2*For* − *pro f it Hospitalit* + *β*3*Government Hospital it* + *β*4*Teaching Hospitalit*

+*β*5*System A f f iliationit* + *β*6*Physician* − *Hospital Integrationit*

+*β*7*Capitation Revenueit* + *β*8*HH Iit* + *β*9*Urvannessit* + *β*10*Median Household Incomeit*

+*Yeart*+ ∈*it*

Our main dependent variable, information exchange, was measured in two ways, as breadth and depth of information exchange [21]. The breadth variable was measured based on answers to questions about whether a hospital provides or electronically shares data with (1) other hospitals in its system, (2) hospitals outside its system, (3) ambulatory providers in its system, and (4) ambulatory providers outside its system. The depth variable was measured based on answers to questions about how much detailed data is shared (five categories: patient demographics, laboratory results, medication history, radiology reports, clinical/summary care records). The values of each item were summed to generate the breadth and depth variables. While we noticed that breadth and depth of information exchange have been identified differently in previous literature [17,19], we decided to use the definition and measurement of breadth and depth following the paper that first operationalized these concepts using AHA IT surveys, to the best of our knowledge [21].

(1)

Bed size is the total number of hospital beds, and for-profit hospital and government hospital are dummy variables. Both dummies equal to zero signify a voluntary nonprofit hospital. Teaching hospital is a dummy variable that takes the value of 1 if the hospital is a teaching hospital. System affiliation is also a dummy variable. Physician-hospital integration is a binary variable that takes the value of 1 if a hospital has an arrangement whereby physicians become employees of the hospital, that is, employs an integrated salary model, and 0 otherwise [24,25]. Previous literature has shown the integration salary model to be the tightest mode of integration between physicians and hospitals. Capitation revenue is also a dummy variable that takes the value of 1 if a hospital's net revenue paid on a capitated basis or shared risk basis. We expect physicians and hospitals under predetermined basis to be encouraged to reduce overall costs and avoid unnecessary tests or procedures. HHI is the Herfindahl–Hirschman index, based on total facility admissions. A high HHI index implies that a hospital is located in a concentrated market, a low HHI index that a hospital is in a competitive market. Urbanness is a dummy variable that takes the value of 1 if a hospital is located in an urban area, and 0 if located in a rural area. Median household income is a county-level variable that describes an area's munificence.

#### **3. Empirical Results**

Table 1 reports descriptive statistics for the variables for 10,040 year-hospital observations (3258 in 2014, 3495 in 2015, and 3597 in 2016). According to Table 1, 19% of our observations are for-profit, 22% government, and the remainder nonprofit hospitals.


**Table 1.** Descriptive Statistics.

**Table 1.** *Cont*.


Table 2 shows the results of our ordinary least squares (OLS) regression analyses regarding the effect of the 10 aforementioned variables on breadth and depth of information exchange. To ensure that our models do not suffer from multicollinearity, we conducted the variance inflation factor (VIF) test. Our model has a VIF of 1.38, indicating that there is not enough evidence to determine that our model suffers from serious multicollinearity problem.

**Table 2.** Hospital- and Market-level Determinants of Information Exchange.


Standard errors (in parenthesis) are clustered at the hospital level; \*\*\* *p* < 0.01, \* *p* < 0.1.

The estimated coefficients of the variables' bed size, system affiliation, physician– hospital integration, and capitation revenue model, being consistent with our stated exante expectations, are not discussed further. Results on the effect of ownership type, not predicted in our study, indicate nonprofit to be more likely than either for-profit or government owned hospitals to share detailed information with other stakeholders. This suggests that nonprofits, being prohibited by law from making a commercial or monetary profit, take information exchange more seriously than other hospital types owing to altruistic concerns about quality. The coefficient of teaching hospital is insignificant, suggesting that the inclination to exchange information electronically is not greater among those working at teaching hospitals than among those employed by non-teaching hospitals. This might reflect lack of other organizations' ability to engage in electronic exchange despite teaching hospitals' interest in sharing information with outside stakeholders. The effect of HHI, which is not predicted in our study, is positive and statistically significant,

suggesting that hospitals in a given county with limited competition are more likely than those in highly competitive areas to share detailed information with others. The estimated coefficient of the effect of urbanness on depth of information exchange is consistent with our stated ex-ante expectation, while the one on breadth of information exchange is not. One possible explanation for the difference is that shortage of financial resources may offset the information sharing effect originated by similar cultures in rural areas. The coefficient of median household income is small but positive, consistent with our ex-ante prediction.

Examining hospital and market-level determinants of health IT-driven information exchange addresses a gap in previous literature that has not looked carefully at the actual use of health IT. More interesting, perhaps, is that although hospital-level determinants are more amenable from the perspective of hospital administrators and policymakers, marketlevel determinants are more or less deterministic. Whereas hospitals can, for example, increase or reduce numbers of beds or alter arrangements with physicians, they are unlikely to move from an urban to a rural area or be able to effect a change in the overall income level of the county in which they are situated.

This leads us to further investigate whether effects of hospital-level determinants vary with market characteristics. To this end, we divide the full sample into two groups according on HHI, Urban/Rural, and median household income, and for HHI and median household income split the sample by taking above and below the mean. Results are reported in Tables 3 and 4. The results of VIF test suggest that there is not enough empirical evidence to determine that our models suffer from serious multicollinearity (Mean VIF: 1.27 (column 1), 1.23 (column 2), 1.27 (column 3), 1.14 (column 4), 1.27 (column 5), and 1.28 (column 6) in Tables 3 and 4, respectively).


**Table 3.** Sub-sample Analysis—Breadth of Information Exchange.

Standard errors (in parenthesis) are clustered at the hospital level; \*\*\* *p* < 0.01, \*\* *p* < 0.05, \* *p* < 0.1.


**Table 4.** Sub-sample Analysis—Depth of Information Exchange.

Standard errors (in parenthesis) are clustered at the hospital level; \*\*\* *p* < 0.01, \*\* *p* < 0.05, \* *p* < 0.1.

Estimated coefficients of most hospital-level variables, including bed size, ownership type, teaching status, system affiliation, and physician-hospital integration, are the same for the sub-sample as for the full sample analysis. These results suggest that market-level variables, although they seem to affect information exchange directly, do not indirectly affect hospital-level variables' influence on information exchange. The results further imply that government does not necessarily need to modify policy according to market characteristics to encourage information exchange among providers.

The coefficient of capitation revenue differs by market characteristics, for breadth of information exchange becoming insignificant in highly concentrated and rural areas, and for depth of information exchange becoming insignificant in highly concentrated, rural, and low-income areas. A possible explanation for this result is that hospitals located in competitive, urban, high-income regions face more pressure to provide high-quality services based on health IT. As this aspect was beyond the scope of our study, we leave exploration of this possibility to future research.

#### **4. Discussion**

Our study makes a number of contributions. First, to the best of our knowledge, the current study is the first to investigate determinants of information exchange in the healthcare setting. Previous research has focused mostly on determinants of adoption, and in such studies of meaningful use as existing information exchange has only recently been added as a requirement [17–20]. Understanding of the determinants of information exchange has thus been limited. The results of our study should contribute to the realization of many of the benefits of health IT, including those associated with enhanced information exchange.

Second, our study helps to provide a more comprehensive picture of drivers of health information exchange. Rather than merely focusing on the percentage of information shared electronically, the current study focused on the relevant factors associated with the breadth and depth of information shared. Moreover, our study examines one of the key hospital-level factors not previously studied: physician-hospital integration. Even among hospital-level variables, it is relatively easier to change physician-hospital arrangements than to change, for example, ownership type. Those interested in enhancing information exchange should continue to search for other unexamined factors that might increase cross-party sharing of detailed information.

Last but not least, our analysis elucidates the effect of specific market-level determinants on information exchange. The finding of our sub-sample analysis that market-level variables such as concentration, urbanness, and median household income do not moderate the relation between hospital-level variables and information exchange is particularly important, as hospital-level determinists are more amenable and hospital administrators and policymakers thus do not need to pursue different strategies to promote the widespread use of health IT using the results of our study. The government with limited resources should thus revisit policies of increasing health information exchange and focus more on hospital-level determinants, such as revenue model or system affiliation.

Few limitations of our paper could shed light on noble opportunities for future research. In examining the determinants of information exchange, the current study only looks at two dimensions: the breadth (extent) and depth (level of detail). Recent studies suggests that four dimensions, including volume, diversity, breadth, and depth are important [17,19]. Future studies could examine the factors we studied that could also affect other dimensions of information exchange, and the varying effect by market characteristics. Furthermore, while our study focuses on the three market-level variables, HHI, urbanness, and median household income, the future researchers can examine other social contexts, such as race, patient mix, and percentage of those who receive care in an integrated system (e.g., Health Maintenance Organization).

#### **5. Conclusions**

We analyzed multiple years of national data to inform understanding of the use of health IT, specifically, information exchange, beyond mere adoption. Aided by detailed datasets rarely used previously, our study examines the relationship between select unexamined variables and breadth and depth of information exchange in healthcare settings. We found that hospital-level variables such as size, ownership type, system affiliation, physician-hospital arrangement, and revenue model affect information exchange, and that market-level variables such as concentration ratio, urbanness, and median household income, although they directly affect information exchange, do not moderate the relationship between hospital-level variables and information exchange.

This paper enhances understanding of factors associated with information exchange via health IT and its implications for the sustainability of the healthcare system. Efforts in many countries to increase information exchange using health IT effectively have gained added importance during the present pandemic. Among the insights related to theory and policy yielded by our results is that hospital administrators and policymakers who seek to increase health IT-driven information exchange should focus on hospital-level determinants and not differentially formulate strategies according to market-level characteristics.

**Author Contributions:** Conceptualization, N.-E.C.; methodology, K.H.; software, K.H.; validation, N.-E.C., K.H. and J.C.; formal analysis, K.H.; investigation, N.-E.C. and K.H.; resources, J.C.; data curation, J.C.; writing—original draft preparation, N.-E.C.; writing—review and editing, K.H.; visualization, N.-E.C.; supervision, N.-E.C.; project administration, N.-E.C.; funding acquisition, N.-E.C. and K.H. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the 2021 Hongik University Research Fund.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Annual and IT surveys of the American Hospital Association (AHA) were purchased by both N.-E.C. and J.C. at https://www.ahadata.com/ (accessed on 1 March 2021). The U.S. Census Bureau's small-area income and poverty estimates that provide information about poverty ratio are publicly available (https://www.census.gov/ (accessed on 1 March 2021).

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Abbreviations**

A list of abbreviations is provided as follows: IT (information technology); AHA (American Hospital Association); HHI (Herfindahl-Hirschman Index); OLS (ordinary least squares); VIF (variance inflation factor (VIF).

#### **References**


## *Article* **Exploring the Impact of Linguistic Signals Transmission on Patients' Health Consultation Choice: Web Mining of Online Reviews**

**Adnan Muhammad Shah 1,2 , Mudassar Ali <sup>3</sup> , Abdul Qayyum <sup>4</sup> , Abida Begum <sup>5</sup> , Heesup Han 6,\* , Antonio Ariza-Montes <sup>7</sup> and Luis Araya-Castillo <sup>8</sup>**


**Abstract:** Background: Patients face difficulties identifying appropriate physicians owing to the sizeable quantity and uneven quality of information in physician rating websites. Therefore, an increasing dependence of consumers on online platforms as a source of information for decisionmaking has given rise to the need for further research into the quality of information in the form of online physician reviews (OPRs). Methods: Drawing on the signaling theory, this study develops a theoretical model to examine how linguistic signals (affective signals and informative signals) in physician rating websites affect consumers' decision making. The hypotheses are tested using 5521 physicians' six-month data drawn from two leading health rating platforms in the U.S (i.e., Healthgrades.com and Vitals.com) during the COVID-19 pandemic. A sentic computing-based sentiment analysis framework is used to implicitly analyze patients' opinions regarding their treatment choice. Results: The results indicate that negative sentiment, review readability, review depth, review spelling, and information helpfulness play a significant role in inducing patients' decision-making. The influence of negative sentiment, review depth on patients' treatment choice was indirectly mediated by information helpfulness. Conclusions: This paper is a first step toward the understanding of the linguistic characteristics of information relating to the patient experience, particularly the emerging field of online health behavior and signaling theory. It is also the first effort to our knowledge that employs sentic computing-based sentiment analysis in this context and provides implications for practice.

**Keywords:** online review helpfulness; signaling theory; sentiment analysis; physician rating websites; consumer decision-making; COVID-19

#### **1. Introduction**

Recent developments in digitalization have opened up doors to the health sector. An increasing number of people are turning to the Internet to collect information related to their healthcare needs [1]. One of the popular sources for seeking and sharing healthrated information is the physician rating websites (PRWs). Patients search for information

**Citation:** Shah, A.M.; Ali, M.; Qayyum, A.; Begum, A.; Han, H.; Ariza-Montes, A.; Araya-Castillo, L. Exploring the Impact of Linguistic Signals Transmission on Patients' Health Consultation Choice: Web Mining of Online Reviews. *Int. J. Environ. Res. Public Health* **2021**, *18*, 9969. https://doi.org/10.3390/ ijerph18199969

Academic Editors: Martin Wiesner and Björn Schreiweis

Received: 8 July 2021 Accepted: 14 September 2021 Published: 22 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

regarding past patients' experience with physicians and their working practices from these rating sites [2]. PRWs provide patients with a unique platform to post their quality of service encounters and write reviews of healthcare providers. From a physician's perspective, PRWs are valuable because the expectations of patients about the level of care rendered by the physician are made publicly accessible. Hence, PRWs have become a powerful source of information in patients' choice for physician selection, as consumers used to rely on online reviews to make purchase decisions [3]. Recently, increasing the utilization of PRWs is becoming popular among healthcare consumers. Health professionals need to learn new ways in which different information aspects of PRWs affect patients' decisionmaking process.

Recent marketing research has reported that online reviews influence not only customer choice [4] but product sales as well [5]. Researchers have carried out several empirical studies in different domains, such as tourism [6], hotel booking [7], and e-commerce [5]. Online reviews have also gained interest from researchers in the healthcare field. A variety of investigations were performed using different PRWs in different countries, such as the average rating score of a physician, review volume [8–10], and qualitative analysis of OPRs to mine patients' interests [11]. In addition to these studies, several other studies have examined the impact of physician star ratings and textual feedback on patients' choice [10,12–14]. Similarly, online physician reviews (OPRs) posted by patients on PRWs provide recommendations for a physician in enhancing the quality of care. Despite there being few academic studies about how to apply online information to identify good doctors, it is complicated to determine how best to assess their relevance. Moreover, information quality measurement is not yet complete, and its effects on patients' consultation choice are largely ignored. Based on previous studies, healthcare practitioners still have a shortage of comprehensive frameworks to assist them in developing IT solutions to help find excellent doctors. Therefore, it is critically important for healthcare providers to evaluate whether and to what degree the various service characteristics lead to overall physician satisfaction among patients [15].

In addition, scant research has been performed on how OPRs affect patients' health consultation decision-making process. Although, academics analyzed quantitative patient ratings posted on PRWs regarding patient healthcare choice [12]; however, a patient's real behavior and emotional response could not be captured from quantitative ratings, as a result of valuable information loss. Furthermore, feedback comments are more valued than numerical ratings in reviews. Hence, it is of particular interest to the researchers that patients use online linguistic signals (unstructured comments) to evaluate the information diagnosticity in choosing a good doctor [16]. As an online signal, linguistic signaling is an appealing and multifaceted subject, particularly in an online environment where there are often considerable asymmetries to information. Patients post OPRs on PRWs regarding a physician's healthcare quality, whereas the physician invokes a response in the form of an answer to the review. This communication between doctors and patients can be a major challenge that leads to weak ties, social and geographical distance that characterizes PRWs. However, in the perspective of PRWs, there is no theoretical model available in our knowledge that considers signaling in relation to information patterns. Since linguistic patterns have not been understudied in previous e-health literature and thus require further investigation, these are likely to play an important role in evaluating the physicians' performance, in turn, lead to positive patients' behavior.

Given the motivation for our research and the nascent nature of information technology in the healthcare domain, we propose a method for identifying good physicians using big data analytics-based sentic computing framework by combining different linguistic signals to recommend suitable and high-quality physicians in PRWs. The proposed approach is based on available physician data in a big data context and connects patients to highquality physicians in order to improve care services. Identifying the right doctor can give patients peace of mind, helping them feel better about the choice they have made [17]. Involving patients in the organization and process is known to strengthen the doctor-patient

relationship, leading to fewer tensions and incidents in hospitals, resulting in fewer lawsuits. This research, therefore, aims at contributing to the literature by drawing a theoretical framework on the basis of signaling theory. The current study utilizes linguistic signaling in the form of affective features (negative sentiment) and informative features (readability, depth, spelling, and review helpfulness) to investigate their impacts on patients' treatment choice. As recommendations play an important role in patients' decision-making, we also investigated different linguistic antecedents of information helpfulness (IH) in online healthcare services. Next, the mediation effect of IH onto the relationships between linguistic signals and patients' treatment choice was also investigated. After investigating the differential effects on the outcome variable, we analyze patients' emotions implicitly from OPRs using a sentic computing framework [18]. The proposed model is tested using unique datasets from two famous PRWs in the U.S (Healthgrades and Vitals), which covers the COVID-19 outbreak period from December 2019–June 2020. This research also provides guidance for platform developers, management of PRWs, and physicians to include essential information components on PRWs, which could improve patients' behavior toward a physician.

#### **2. Theoretical Background and Research Hypothesis**

#### *2.1. Linguistic Patterns*

PRWs contain plenty of linguistic signals, which are a precious resource for people who are seeking health information and support [19]. Previous studies regarding the effects of *information quality* as a linguistic signal refer to the persuasive strength of the message, which is commonly measured in terms of its relevance, timeliness, accuracy, and comprehensiveness [20]. Reviews posted by different users are always different in length, accuracy, comprehensiveness, tone, and even logic [21]. In an online environment, users perceived the information regarding a particular activity in which they were engaged, fitted in their expectations and requirements [22].

Extent literature employed "argument quality or information quality" to measure its effect on the users' behavior [21,23,24]. Since the last few years, researchers have been struggling to better understand the impact of online rating sites on various aspects of people's choice behaviors [4,23,25]. Following this line of research, information evaluation is considered an important antecedent of the patients' healthcare decisions. From the perspective of traditional communication theories, patients evaluate information from different perspectives before making their health consultation decisions [10]. Online reviews integrate information from various sources providing online word-of-mouth (WOM) to healthcare consumers who lack WOM [13]. Extant studies on the patients' choice of physician outlined that information gathered from peers or other patients are always amongst the top influencing factors [9,26].

In line with this research, several other studies have also stated the information quality as a predictor of users' behavior in the healthcare domain [27,28]. The healthcare field is categorized by high environmental and demand uncertainty, and healthcare consumers and providers are more likely to look for appropriate and credible information [29]. Therefore, efficient information management contributes to the customer benefits in healthcare [30]. These arguments are further supported by Wu [21], who stated that information quality is highly a critical factor that influences the patients' online behavior. Moreover, Yoon [31] stated that patients tend to adopt that information, which is more factual, relevant, and useful. Hence, quality information about a doctor enables patients to obtain further information on healthcare providers and hospitals. For instance, a study by Lu and Zhang [32] proved that the perceived quality of Internet information would affect peoples' treatment decisions. Yazdinejad et al. [33] proposed a block-chain-based decentralized verification of patients in a distributed hospital set-up. Ignoring the re-authentication process among distributed affiliated hospitals, the proposed architecture will have a significant impact on network throughput, overhead reduction, response time improvement, and energy consumption. Javed et al. [34] proposed a method called the Cognitive Assessment of Smart

Home Resident (CA-SHR) that uses a neuropsychologist to quantify smart home residents' capacity to perform various tasks on a daily basis using pre-established scoring systems. Shah, Yan, Shah, Shah, and Mamirkulova [10] contributed to the unified signaling theory and Maslow's hierarchy of needs theory by combining different patient-generated and system-generated signals in order to help patients in deciding on their medical decisions based on their disease risks.

Shah et al. [35] explored the influence of different online signals (online reputation and online effort), offline signals (offline reputation), and disease risk on patients' physician selection choice for e-consultation during the COVID-19 crisis. The findings suggested that online signals have a significantly positive effect on patients' e-consultation choice than offline signals.

Although analyzing the quality of online health information is challenging [36], information quality helps to allow patients to obtain better healthcare services, improve the organization's performance and doctor-patient relationships [37]. To end with this, high information quality leads physicians to adopt a better quality system in healthcare.

#### *2.2. Signaling Theory*

Spence [38] indicated that signaling mechanism could reduce problems of information asymmetry regarding quality by maintaining equilibrium in which only high-quality seller seeks valuable information to invest in signals. Models of information asymmetry demonstrate that at least one party to a transaction has related information while others do not [39].

It is worth remembering that our research is in the context of the healthcare industry and is unique in the following ways: First, healthcare is a highly information-asymmetric sector. In the case of online healthcare, physicians have more knowledge about their own level of service than the patients. They are also more conscious than patients. Though physicians know their quality of service, patients have little knowledge about this fundamental issue. As a result, physicians will prescribe unnecessary treatments that raise their income even if it may be of little benefit. Because of the limited knowledge, patients do not have a reliable way to assess the quality of the advice they get [40]. This information asymmetry situation produces a power imbalance in transactions, and the transactions often get bugged—a kind of market failure in the worst cases [41].

Second, on PRWs, patients can put pressure on physicians by possibly obtaining a second opinion. In the context of online healthcare, physicians send signals about the quality of service to their patients. Upon receiving this information, patients can change their decision on the quality of service provided by physicians and thus modify their physician choice [40]. Hence, the selection of appropriate signals in the PRWs is vital to the success of these rating websites because different signals can express different types of information and eventually lead to uneven outcomes.

#### *2.3. Hypotheses Development*

Based on the above discussion, this study incorporates five linguistic patterns, which are negative sentiment, readability, depth, spelling, IH, and explore their influence on patients' choice. Specifically, we categorized negative sentiment as a component of affective signals and readability, depth, and spelling as informative signals. We also examined the mediating effect of IH in the given context. Figure 1 shows the research model.

#### 2.3.1. Affective Signals and Patients' Treatment Choice

Emotions (e.g., positive vs. negative) significantly influence an individual patient's ability in his/her treatment decisions [10]. Sentiment refers to the attitude, assumption, or decision stimulated by feelings [42]. User-generated content in PRWs is usually enclosed with sentiment valence (i.e., positive, negative, or neutral sentiment).

**Figure 1.** Research model. **Figure 1.** Research model.

2.3.1. Affective Signals and Patients' Treatment Choice Emotions (e.g., positive vs. negative) significantly influence an individual patient's ability in his/her treatment decisions [10]. Sentiment refers to the attitude, assumption, or decision stimulated by feelings [42]. User-generated content in PRWs is usually enclosed with sentiment valence (i.e., positive, negative, or neutral sentiment). In the case of healthcare as a credence good, people may not evaluate the quality of healthcare services even after consumption. Therefore, they seem to favor strongly in reducing losses to boost gains in healthcare decision-making. This implies that those individuals are more prone to perceived damages than benefits when making a healthcare decision in ambiguous or unpredictable circumstances. As healthcare consumers prefer to follow negative OPRs (e.g., low-quality products, monetary losses, etc.) in order to minimize their losses and become more reactive to negative facts, negative OPRs have a greater effect on the patients' treatment decisions [43]. Given that people detect negative information faster than positive information [42], negative valence PRW–OPRs should have a greater effect on perception creation [44]. Thus, healthcare consumers find negative In the case of healthcare as a credence good, people may not evaluate the quality of healthcare services even after consumption. Therefore, they seem to favor strongly in reducing losses to boost gains in healthcare decision-making. This implies that those individuals are more prone to perceived damages than benefits when making a healthcare decision in ambiguous or unpredictable circumstances. As healthcare consumers prefer to follow negative OPRs (e.g., low-quality products, monetary losses, etc.) in order to minimize their losses and become more reactive to negative facts, negative OPRs have a greater effect on the patients' treatment decisions [43]. Given that people detect negative information faster than positive information [42], negative valence PRW–OPRs should have a greater effect on perception creation [44]. Thus, healthcare consumers find negative OPRs more useful than positive OPRs in their treatment decision-making. Following this logic, we demonstrate that negative OPRs provide more predictive, informative, and reliable information regarding the patients' perceptions of physician service quality than positive OPRs. Thus, we propose that:

OPRs more useful than positive OPRs in their treatment decision-making. Following this logic, we demonstrate that negative OPRs provide more predictive, informative, and reliable information regarding the patients' perceptions of physician service quality than pos-**Hypothesis 1 (H1).** *The negative sentiment expressed in an OPR on PRW is positively related to patients' treatment choice.*

#### itive OPRs. Thus, we propose that: 2.3.2. Informative Signals and Patients' Treatment Choice

**Hypothesis 1 (H1).** *The negative sentiment expressed in an OPR on PRW is positively related to patients' treatment choice.*  2.3.2. Informative Signals and Patients' Treatment Choice *Review Readability*: Readability is the degree of comprehension of a piece of text or an article. Online feedback might be comprehensible if these are used as an input variable when making buying decisions [45]. The extant literature has established the readability of information on social media sites as important for consumer acceptance [43,45]. For customer purchase decision-making, a review is reflected to be more useful with sufficient readability than a review that is too lengthy and includes many typographical mistakes, which makes reading challenging for users [16]. Similarly, Yin et al. [46] reported that reading difficulty has a detrimental impact on consumer decision-making. In PRWs, a *Review Readability*: Readability is the degree of comprehension of a piece of text or an article. Online feedback might be comprehensible if these are used as an input variable when making buying decisions [45]. The extant literature has established the readability of information on social media sites as important for consumer acceptance [43,45]. For customer purchase decision-making, a review is reflected to be more useful with sufficient readability than a review that is too lengthy and includes many typographical mistakes, which makes reading challenging for users [16]. Similarly, Yin et al. [46] reported that reading difficulty has a detrimental impact on consumer decision-making. In PRWs, a health information seeker can readily adapt the readability of a review to evaluate the physicians' performance. Therefore, we hypothesized that the more comprehensible the text, the more helpful the review is considered to be to evaluate the physicians' performance in terms of patients' treatment choice.

health information seeker can readily adapt the readability of a review to evaluate the **Hypothesis 2 (H2).** *The readability of an OPR on PRW is positively related to patients' treatment choice.*

*Review depth*: The concept of review depth refers to the comprehensiveness/elaborateness of the information provided in a review [16]. With the advent of virtual platforms, people post a large amount of online reviews. The overloading of information makes it difficult for us to find the relevant information or ignoring important and critical information [16]. The impact of information overload on the consumer purchase process has been discussed in previous research [43,47]. Short reviews are considered superficial and provide a less detailed evaluation of product characteristics [45]. Long reviews provide more information, including a detailed product overview and its characteristics [48]. An OPR may provide a complete overview of individual health status, medications, and questions or concerns about the physicians' healthcare quality. Researchers have reported the limits or capacity of purchase decision-makers to process information if the volume is too big or very low. Inadequate information has a detrimental impact on patients' choice. A comprehensive review may enhance information diagnosticity and reduce the search costs of patients. Keeping all other factors constant, a detailed review not only invokes trust but also provides ample information for patients' consultation choice [16]. Therefore:

#### **Hypothesis 3 (H3).** *The depth of an OPR on PRW is positively related topatients' treatment choice.*

*Review spelling*: Non-Standard spelling and blank spelling errors are common in usergenerated content [44]. The situation may worsen, particularly in the case of PRWs, where users of different backgrounds and different health conditions will impair their ability to spell correctly. Therefore, spelling is another critical factor that leads to obtaining relevant information about physician performance successfully from online peers. In an online social network context, the accurate spelling in one's review will not only generate a good image of the reviewer's literacy [49]. Still, it will also effectively communicate the right meaning to the target community. Previous evidence is particularly important for this feature; Ghose and Ipeirotis [50] indicated that the presence of spelling errors in online reviews is negatively associated with the IH. Given the compelling nature of spelling errors in PRWs, we argue that the spelling of reviews on PRWs matters for those individuals who seek informational support to evaluate the physicians' performance, in turn, leads to patients' treatment choice. Therefore, we propose:

#### **Hypothesis 4 (H4).** *The degree of correct spelling in an OPR on PRW is positively related to patients' treatment choice.*

#### 2.3.3. Affective Signals and Information Helpfulness

Another element of a review is its sentiment, which is a brief overall user experience evaluation [47]. Readers can quickly identify the author's attitude and feeling on the basis of the sentiments. It has been perceived that the more negative the sentiment expressed by the author, the higher will be the IH value [51]. Such negative sentiments can either be exciting or unsatisfactory. For instance, Lee, Jeong, and Lee [42] suggested that negative reviews play a more critical role in consumers' information processing and decision making. When patients read negative online user feedback about a physician, for example, they may find out that the physicians' services are of poor quality. This example indicates that negative OPRs for the patients' consultation decision-making process are more informative and credible; thus, negative OPRs can be viewed as more beneficial than positive OPRs. OPRs with negative sentiment will be more influential. A more compelling review ensures that readers have a better chance of understanding a review. This implies that readers will consider the helpfulness of the review and vote for it. Hence, we propose that:

#### **Hypothesis 5 (H5).** *An OPR expressing more negative sentiment will receive more helpful votes.*

#### 2.3.4. Informative Signals and Information Helpfulness

The main element of a review is the review content. A review should be reliable or easy to understand without potential conflicts to provide information effectively [52]. Readability, judged by its writing style, refers to how easily readers could understand the text. The readability of the text denotes the author's social status, level of education, and social hierarchy [16]. An OPR with high readability should also be regarded as more accurate than reviews with low readability. If an OPR is accurate or easy to understand, then it would spread its meaning to more people. Thus, a more readable OPR will receive more helpful votes. Therefore:

#### **Hypothesis 6 (H6).** *A more readable OPR will receive more helpful votes.*

The quality of a review can be used as a proxy to determine its helpfulness for consumers in making informed buying decisions. Since a review is a source of information, its usefulness depends on how much information there is in its textual content [53]. In this regard, researchers have shown that the review depth was positively correlated with the perceived helpfulness of a review [48]. A possible justification for this evidence could be that longer reviews may contain more detail than shorter ones [45]. Therefore, healthcare consumers are supposed to consider longer OPRs contained detailed information as being more effective in making their health consultation decisions than shorter ones. Hence, we propose that:

#### **Hypothesis 7 (H7).** *An OPR with detailed information will receive more helpful votes.*

A review with sufficient readability is considered more valuable to users than a review, which is hard to read and contains multiple typographic errors. While many reviewers worry about the accuracy of the spelling when writing online reviews, spelling errors can lead to reading problems. Forman et al. [54] reported that spelling errors have a negative effect on the review helpfulness and readability of a review. Ghose and Ipeirotis [50] found a negative association between online reviews and helpful votes received by the review. In the online healthcare context, we claim that the spelling of an OPR matters for efficient information dissemination. An OPR with the right spelling can efficiently transfer accurate and helpful information to information seekers. Hence, we propose that:

#### **Hypothesis 8 (H8).** *An OPR with correct spelling will receive more helpful votes.*

*Information helpfulness signal*: IH is the component of information quality that reflects the user's perception regarding the reliability and relevancy of the information. Existing feedback in the form of online reviews on the Internet may be useful or not because usefulness perception leads customers' intention of information adoption for purchase decisions [23,25].

In a similar vein, IH influences the patients' consultation choice. From a physician's perspective, defining the features of an online health information retrieval experience and embedding those features in websites that are perceived to be high quality may lead to customer satisfaction [55,56]. Wu [21] indicated that the perceived usefulness of a system significantly influences the continuance use of online health communities in achieving health-related goals. A system higher in the perceived IH offers a positive user-performance relationship. Scholars also suggest that perceived usefulness/IH/value is the antecedent of consumer behavior [57,58]. For these reasons, we propose that helpful information on a PRW leads to assist users in positively evaluating the quality of physician healthcare services, lead to positive behavior toward that physician.

#### **Hypothesis 9 (H9).** *IH on PRWs is positively related to patients' treatment choice.*

Further, we argue that patients will make their treatment decisions based on helpful reviews from peer patients. Patients will decide to choose a physician if the information regarding physician service quality is really helpful. Hence, physicians can improve patients' attitudes by providing useful information to the patients on PRWs. These arguments are

consistent with prior findings in the IH literature that link different linguistic signals to review helpfulness and consumer behavior [4,5,21,23]. Hence, we propose that:

**Hypothesis 10 (H10).** *IH mediates the relationship between affective signals, information signals, and patients' treatment choice.*

#### **3. Methods**

#### *3.1. Sample and Data Collection*

We test our research hypotheses by collecting data from Healthgrades.com (accessed on 11 August 2020) and Vitals.com (accessed on 11 August 2020), included among the highest traffic ranking websites in the U.S.

In June 2020, we used an automated network spider coded in Python 3.6 to scrape all of the relevant physician profiles from Healthgrades and Vitals. Data were collected covering a period of COVID-19 epidemic outbreak in the U.S from December 2019 to June 2020. So far, this novel coronavirus has caused 2,537,636 confirmed cases with 126,203 mortalities [59]. Clearly, a significant problem during the epidemic was the lack and scarcity of healthcare services. The provision of timely and effective health services during the pandemic period proved to be overwhelming because of inadequate protective equipment, travel restrictions, lockdown, and the possibility of spreading diseases to patients and physicians [60]. Hospitals can increase the effectiveness of their healthcare facilities by replacing physical treatments with virtual technologies in order to reduce and monitor the spread of the pandemic.

For data collection, we chose 10 disease specialties that contain the highest number of active physicians in the U.S [61]. Data were collected from 10 leading U.S states based on the number of active specialist physicians (i.e., California, New York, Texas, Florida, Pennsylvania, Ohio, Illinois, Massachusetts, and New Jersey) [62]. Our final data set include a non-random selection of 5521 physicians with a total of 52,340 reviews from 10 disease specialties.

#### *3.2. Measurements*

Table 1 shows the variables in the analysis and their definitions. The unit of analysis was the individual OPRs. Figure 2 shows the overall structure of the variable measurement and proposed analysis procedure.

First, we pre-process the raw data in the form of OPRs. Second, we mine different concepts from the pre-processed OPRs using Stanford core NLP modules (parsing, chunking, normalization, lemmatization, POS tagging), and calculate the similarity match between different concepts. Third, we apply the sentiment analysis method to compute the polarity of concepts. Fourth, we use the FKRE to calculate the review readability, LIWC toll to calculate review depth, and spell checker for review spelling checking. Finally, OLS regression analysis, SEM analysis, and boot-strapping-based mediation analysis were performed to determine the impact of different linguistic features on patients' choice.

#### 3.2.1. Dependent Variable

Patients' choice is measured by four dimensions. Patients assign the quality ratings for physician services; they received in offline hospitals. These ratings ranged from 1–5 with 1–2 being negative, 3 as neutral, and 4–5 as positive. Hence, quality ratings, the number of blogs that the physician has initiated, the number of physician's healthcare articles, and the physician number of replies to patients are used to measure patients' choice. All four dimensions are averaged to obtain a composite variable.

#### 3.2.2. Independent and Mediating Variables

*Negative sentiment*: In order to determine the strength of negative opinion, an emotional response (sentiment score) of users toward a provider is computed. For this purpose, the current study adopted a hybrid approach (sentic computing) that follows the state-of-theart text mining techniques discussed in previous studies [63,64].


132

*Review readability*: Flesch–Kincaid Reading Ease (FKRE) is used to measure the readability level of a review (see Equation (1)). FKRE [65] analyses the complexity of the text in order to determine the number of years of education that would be needed for someone to understand the text being assessed.

$$\text{FKRE} = 206.835 - \left(1.015 \times \frac{\text{\textdegree of words}}{\text{\textdegree of sentences}}\right) - \left(84.6 \times \frac{\text{\textdegree of syllables}}{\text{\textdegree of words}}\right) \tag{1}$$

Score Range = 0–100

=>The score between 60 and 70 is largely considered acceptable.

*Review depth*: Review depth is the number of words in a review. Review depth is calculated using the natural language processing (NLP) tool, that is, linguistic inquiry and word count (LIWC), which is a text-mining program [66].

*Review spelling*: We use an open-source spell checker software (The software Language Tool is available at https://languagetool.org (accessed on 29 September 2020 and 2 October 2020)) to compute the Jaro–Winkler similarity [67] between the original text and the corrected text as the proxy of the PRW review's spelling. The Jaro–Winkler similarity score ranges between 0 and 1. The higher the similarity between the original and the updated text, the closer the metric reaches 1.

*Information helpfulness* (*IH*) *as a mediator*: On PRWs, there is an accumulative helpfulness vote that results from other reviewers who vote on the effectiveness level of each review. The helpfulness value of the review increases with the increase in the number of helpfulness votes for an OPR. Following the ground truth, we assume that the IH variable here has a continuous value and is measured as the ratio of helpful/useful votes to the total votes. From our dataset, the non-voted reviews were removed to reduce the noise.

#### 3.2.3. Control Variables

We controlled a number of other variables that could theoretically explain patients' treatment choice. These control variables include: (1) Physician title (*Title*) in offline hospital, (2) practical experience (*Exp.*), and (3) physician gender (*Gender*).

#### *3.3. Machine Learning Sentiment Analysis*

Choosing a suitable approach for sentiment mining relies on the needs of the analysis. Sentic computing has been used to solve various cognitive-inspired issues, such as classifying natural language text (positive or negative). A hybrid approach to sentic computing and opinion mining incorporates knowledge-based methods and statistical approaches to identify opinions and sentiment calculations from natural language text [18,63].

#### *3.4. Pre-Processing of Online Reviews and Concept Mining*

After obtaining OPRs in Section 3.1, the raw data is cleaned using Stanford NLP tools, as shown in Figure 2. To mine the common-sense concepts from the text, the Stanford Chunker [68] is used first to chunk the input text; then, semantic parsing is performed using a semantic parser [69]. The semantic parser breaks sentences into clauses first and then decomposes such clauses into bags of concepts. Concepts are transformed into vector space modeling (VSM) built from WordNet Affect and Concept Net. In VSM, each concept is represented as a point in a vector space with one dimension for a term in the vocabulary [64], as shown in Figure 3. VSM captures the semantic and affective similarities between the concepts and performs analogical reasoning quickly and efficiently [69]. The purpose of concept similarity detection is to compare a given concept with others in the database and thus minimize data sparsity. In addition, to calculate the cross-correlations between concepts, a dimensionality reduction method called truncated singular value decomposition is implemented on the matrix representation of AffectNet.

tion is implemented on the matrix representation of AffectNet.

*3.4. Pre-Processing of Online Reviews and Concept Mining* 

After obtaining OPRs in Section 3.1, the raw data is cleaned using Stanford NLP tools, as shown in Figure 2. To mine the common-sense concepts from the text, the Stanford Chunker [68] is used first to chunk the input text; then, semantic parsing is performed using a semantic parser [69]. The semantic parser breaks sentences into clauses first and then decomposes such clauses into bags of concepts. Concepts are transformed into vector space modeling (VSM) built from WordNet Affect and Concept Net. In VSM, each concept is represented as a point in a vector space with one dimension for a term in the vocabulary [64], as shown in Figure 3. VSM captures the semantic and affective similarities between the concepts and performs analogical reasoning quickly and efficiently [69]. The purpose of concept similarity detection is to compare a given concept with others in the database

concepts, a dimensionality reduction method called truncated singular value decomposi-

**Figure 3.** Affective space. **Figure 3.** Affective space.

The polarity score (*p*) of a sentence (*d*) and the overall sentiment score of a review is computed using Plutchik's wheel on human emotions [70], as shown in Equations (2) and (3), respectively. The polarity score (*p*) of a sentence (*d*) and the overall sentiment score of a review is computed using Plutchik's wheel on human emotions [70], as shown in Equations (2) and (3), respectively.

$$p = \sum\_{i=1}^{N} \frac{Pleanantness\left(c\_i\right) + \left|Attention\left(c\_i\right)\right| - \left| Sensitivity\left(c\_i\right)\right| + Aptitude\left(c\_i\right)}{3N} \tag{2}$$

$$\text{Option\\_score}(d) = \frac{\sum\_{p\_i \in d} policy(p\_i)}{N} \tag{3}$$

where:

*n* = number of concepts

3 = normalization feature

Polarity *p* ranges [−1, +1].

For a given sentence, SenticNet [18] and extreme machine learning [64] determine the opinion score between −1 to +1, where −1 denotes extremely negative, and +1 indicates extremely positive. Iterating the above procedure can result in the opinion score for other reviews in the dataset.

#### *3.5. Empirical Model*

To model the treatment choice as a function of independent variables and control variables, our empirical model is shown below:

log (*Treatment choicei*) = *β*<sup>0</sup> + *β*<sup>1</sup> (*Scorei*) + *β*<sup>2</sup> (Re*adabilityi*) + *β*<sup>3</sup> log(*Depthi*) + *β*<sup>4</sup> (*Spellingi*) + *β*<sup>5</sup> log (*IHi*) + *β*<sup>6</sup> (*Titlei*) + *β*<sup>7</sup> (*Experiencei*) + *β*<sup>8</sup> (*Genderi*) + *z<sup>i</sup>* (4)

#### **4. Results**

Tables 1 and 2 present the descriptive statistics and correlations of the variables, respectively. It is interesting to note from Table 2 that for this analysis the VIF values of our independent variables were below 10, hence multicollinearity might be ignored [71]. All our empirical data were analyzed using STATA and AMOS (version 23.0). We ran the ordinary least squares (OLS) regression on our dataset. Testing of the hypotheses was performed by exploring the structural model results using patients' treatment choice models, which are presented in Table 3 and Figure 4.


Note: Standard errors are in parentheses. \* *p* < 0.05; \*\* *p* < 0.01; \*\*\* *p* < 0.001.

**Table 2.** Variables correlations.

As shown in Model 1, it can be observed that title (*β* = 0.146, *p* < 0.001) and experience (*β* = 0.241, *p* < 0.01) have positive effects on the treatment choice, while gender (*β* = 0.010, *p* > 0.001) has an insignificant effect onto the treatment choice. From Model 2, it is notable that score (β = 0.121, *p* < 0.001), readability (β = 0.020, *p* < 0.001), depth (β = 0.089, *p* < 0.001),

through H5 generally agree with previous studies demonstrating the positive impact of different features of message on consumer choice [4,16,42–44,48,50]. In addition, the results show significant F values and adjusted-R2 values, which lie within the defined

We tested our hypotheses H5–H10 using structural equation modeling (SEM) with the AMOS 23.0. The results are shown in Table 4. The output of SEM indicates that the score (β = 0.243; *p* < 0.001) shows a significant and positive relationship with IH, in line with the previous studies of the influence of emotions on IH [73]. In contrast, insignificant relationships were found between readability (β= −0.312, *p* = insignificant), depth (β=

*4.1. Analysis Results for Direct Effects* 

threshold level [72].

*4.2. Results for Mediation Analysis* 

**Model's Goodness of Fit Hypotheses Relationship Β T** 

−0.029; *p* < 0.005), spelling (β = 0.043; *p* = insignificant) and IH.

**Table 4.** Structural equation modeling and bootstrapped mediation analysis results.

χ2/df 2.441 H5 Sentiment → IH 0.243 \*\*\* 4.021 Supported


**Table 3.** Estimation results.

Note: Standard errors are in parentheses. \*\* *p* < 0.01; \*\*\* *p* < 0.001.

#### *4.1. Analysis Results for Direct Effects*

As shown in Model 1, it can be observed that title (*β* = 0.146, *p* < 0.001) and experience (*β* = 0.241, *p* < 0.01) have positive effects on the treatment choice, while gender (*β* = 0.010, *p* > 0.001) has an insignificant effect onto the treatment choice. From Model 2, it is notable that score (β = 0.121, *p* < 0.001), readability (β = 0.020, *p* < 0.001), depth (β = 0.089, *p* < 0.001), spelling (β = 0.819, *p* < 0.001), and IH (β = 0.213, *p* < 0.001) all have a positive effect onto the treatment choice. Therefore, H1–H5, are all supported. The results found in H1 through H5 generally agree with previous studies demonstrating the positive impact of different features of message on consumer choice [4,16,42–44,48,50]. In addition, the results show significant F values and adjusted-R<sup>2</sup> values, which lie within the defined threshold level [72].

#### *4.2. Results for Mediation Analysis*

We tested our hypotheses H5–H10 using structural equation modeling (SEM) with the AMOS 23.0. The results are shown in Table 4. The output of SEM indicates that the score (β = 0.243; *p* < 0.001) shows a significant and positive relationship with IH, in line with the previous studies of the influence of emotions on IH [73]. In contrast, insignificant relationships were found between readability (β= −0.312, *p* = insignificant), depth (β= −0.029; *p* < 0.005), spelling (β = 0.043; *p* = insignificant) and IH.

The overall SEM fit provided a value of χ <sup>2</sup>/df = 2.441, which is below the recommended threshold of 3. The CFI = 0.939, NFI = 0.916, TLI = 0.925, root mean square error of approximation (RMSEA) = 0.06 RMSEA; thus, all were above the suggested threshold [71]. Hence, the SEM shows a good fit.


**Table 4.** Structural equation modeling and bootstrapped mediation analysis results.

Note: \*\*\* *p* < 0.001.

To test for hypotheses H5–H10, the mediating effect of perceived IH was investigated using the bootstrapping method as recommended by Preacher and Hayes [74], and AMOS 23.0. The bootstrapping method is a particularly effective approach to alternative methods such as the Sobel test [75]. In particular, we tested whether score, readability, depth, and spelling have indirect effects on treatment choice through the mediation of perceived IH. We first analyzed the direct effects estimation results without a mediator, the direct results after the mediator (satisfaction) are inputted, and the indirect results. If the indirect effect is significant, then a mediation effect can be established [71]. Table 5 also presents the indirect effects of four independent variables on treatment choice through perceived IH.

**Table 5.** Bootstrapped mediation analysis model.


Note: \* *p* < 0.05.

First, the indirect effect of the score on treatment choice was positive and significant (95% CI = [0.492, 0.251]). According to Hair, Black, Babin, and Anderson [71], when both the direct and the indirect effects from X to Y are significant, a partial mediation effect occurs; where the direct effect is insignificant when the mediator is introduced, and the indirect effect is significant, the full mediation effect is achieved.

If the direct effect was never significant, but there is an indirect effect, then there is an indirect effect of mediation. In our analysis, direct impact without a mediator and its direct impact with a mediator of the score were both insignificant; hence, this result shows that IH did an indirect mediation role. Second, the indirect impact of readability on treatment choice was not significant (95% CI = [0.035, −0.642]), thus suggesting that IH did not have a mediation effect. Third, the indirect impact of depth on treatment choice was significant (95% CI = [0.211, 0.069]). Since both were significant for its direct effect without a mediator and its direct effect with the mediator, it indicates that IH had a partial mediation effect. Finally, the indirect effect of spelling on IH was insignificant (95% CI = [0.089, −0.007]), indicating that IH did not have a mediation effect. These results confirm potential mediating impacts of IH raised by previous research [4].

#### **5. Discussion**

#### *5.1. Discussion of the Results*

Linguistic signaling is an interesting and complex topic, particularly in an online environment where there are often substantial asymmetries to information. Investigating the impact of different linguistic signals on patients' treatment choice during the COVID-19 epidemic period can be a major challenge. Nevertheless, we still need to see a theoretical framework considering linguistic signaling within PRWs. A comprehensive understanding of linguistic signaling in the online healthcare environment provides a theoretical foundation for patients' choice toward a specific physician.

First, H1 is supported, which means that the positive relation between negative sentiment and patients' health consultation choice has been verified. Previous studies have proven that negative emotions positively influence user behavior [49,51]. In our context, when users read negative opinions about healthcare providers, they will decide whether to consult a particular physician or not.

Second, H2, H3, and H4 are supported significantly, which means that informative signals positively influence users' decision-making process. These results are consistent with extant studies [46,49,76]. Informative signals include review readability, depth, and spelling. When users read online reviews and find a review to be more readable, comprehensive, and accurate about a particular provider, they will trust in this online review to consult a particular physician.

Third, H5 is significantly supported, that is to say, patients' negative sentiments about physician's service quality have significant effects on perceived IH. The positive relation between user sentiments and IH has been tested before [48,76]. The results of the present study showed that negative emotions in reviews showed a positive effect on review helpfulness. This finding can be considered in light of various prior investigations. When a service fails, customers may vent their frustrations in online reviews to let others know about their bad experience [42,77]. Potential customers who read reviews may appear to place a higher value on losses than on profits, based on their perceptions [49]. This argument is in line with the findings of Cacioppo and Berntson [78], who found that negative input, has a stronger impact on behavioral expressions than positive input.

Fourth, H6, H7, and H8 are not supported, which means that readability, depth, and spelling are not supported toward the IH. These results are not in line with the previous findings [16,44,79]. The probable explanation behind these results is that when consumers search information online regarding the provider for their health consultation, they do not perceive the readability, depth, and spellings of online reviews to be helpful; thus contradicting the earlier findings. However, regarding H9, our study findings showed that IH is significantly and positively related to treatment choice, in line with previous findings [4,80,81]. Patient decision-making can be considered as perfect if a large amount of electronic word of mouth activity exists and these reviews are helpful at the same time, which indicates the physician is popular and attracts patients' attention more than the other physicians.

Finally, regarding H10 the current study explores the mediation effect of IH between affective and informative signals and patients' health consultation choice. In our context, the PRWs aim to provide users with more helpful or useful information regarding disease consulting services and providers. If users experience high quality of doctor's service and health information, they will think it more useful to consult a particular physician [82].

#### *5.2. Theoretical Contributions*

This work adds to theoretical knowledge in a number of ways:

First, we present signaling theory as a relevant theoretical underpinning for our proposed linguistic signals-based paradigm for holistically addressing patients' information demands. In specific, we examined the affective signals in terms of the negative sentiment and informative signals as the readability, depth, and spelling of online reviews. Prior research in various domains has mainly focused on the effects of one or two signals in

isolation [4,73,83]; all signals are not systematically studied in one study, and hence we cannot compare these effects directly due to varied research settings and isolated study outcomes. Therefore, we incorporate work from well-grounded quantitative research and also includes qualitative analysis. Mining review text posted during the novel coronavirus period, the current study helps to bridge the gap from both content and linguistic aspects to the emerging research body. Since limited studies have examined review content focusing on cognitive perceptions (e.g., readability, depth), this study is one of the first to examine precisely how individuals' emotions implicitly serve as signals for readers to assess patients' choice toward a physician. When we compare our proposed framework to other physician recommendation systems, we find that ours can assist patients in making better physician selections.

Furthermore, this is the first study to examine the influence mechanism of patients' treatment decision-making process. On the foundation of different features of messages in the form of OPRs, the OLS regression model, SEM analysis, and mediation analysis are used to analyze the influencing message features on patients' decision-making. We found that the factors which influence patients' health consultation choice include negative sentiment, readability, depth, spelling, and IH. Based on the results from the mixed methods, we incorporate those key elements into the patients' choice model for identifying high-quality physicians. Since we introduced the concept of IH in online healthcare services, we also proposed and tested for the first time a new mediator, IH [4], which explains the effect of linguistic signals on patients' choice toward physician services. The results showed that the IH mediates the relationships between negative sentiment, review depth, and patients' treatment choice.

Finally, our information technology-based linguistic signals transmission method is a big data technology, which is more than just an internet technology. It contributes to the theoretical literature on the use of information technology and signaling theory by providing new insights. Moreover, our study is a theoretical extension of information processing, allowing patients and healthcare organizations to exchange information more easily, and enhance information sharing among online healthcare platforms. Furthermore, it expands the application of signaling theory and responds to calls for theory development in healthcare management.

#### *5.3. Practical Implications*

The study's findings indicate that technologies such as big data, machine learning, regression analysis, and multi-method analysis can be integrated to benefit the health domain. The findings have important managerial implications for patients and healthcare practitioners, and firms that provide similar high involvement services can benefit from this technology.

First, identifying high-quality providers with the use of internet-based health information technology brings tremendous benefit to patients. The proposed linguistic signals model can help in the improvement of patients' healthcare service experiences by allowing them to select the most appropriate physicians; hence minimizing patient dissatisfaction caused by sub-optimal medical professionals. Thus, the dispute between physicians and patients may be reduced. Furthermore, the proposed information technology paradigm has the potential to save patients search costs, time, and money.

Second, using information technology to find good doctors may benefit healthcare administrators. There is a significant gap between the vast amount of online healthcare information and the information requirements of users. To bridge this gap, we provide a physician selection choice model for patients that healthcare platforms can utilize to select the best physicians, to increase the user experience, and to increase their website's efficiency. In practice, this study also provides web designers with an overview of the features of the most valuable knowledge that could help them in designing signals for patients and online healthcare industry reviewers to use during COVID-19 pneumonia [84]. Web designers and moderators can actively examine the linguistic characteristics that invoke

useful responses toward providers. In particular, most reviewers use a similar writing style that first helps to communicate information effectively and includes opinions and emotional content. In addition, where certain styles are normative, moderators can educate new users about the language features that the majority prefer (e.g., emotion, length, style, etc.), and/or provide ideas in real-time as reviews are posted, such as different wordings or styles that more closely fit reviews in the PRWs previously. Designers can even take into account an innovative characteristic that can analyze the review content (e.g., length, spelling errors, etc.) to provide an opportunity for evaluating physicians' performance. Through providing information on crowd assessments and crowd behavior, web designers can make the assessment of healthcare services simpler for customers, thus encouraging buying decisions.

Lastly, this study makes vital contributions to the medical and healthcare industries; the effective and efficient use of negative feedback is of utmost importance for physicians during the COVID-19 pandemic. As patients find negative reviews to be more influential in their treatment decisions than positive reviews, physicians should take decisive steps to resolve the depressed feelings of patients before they become more disenchanted. Physicians may improve review management practices by classifying and/or ranking feedback from the patients' eyes through their value or helpfulness; for example, usually, first favorable reviews, and then critical reviews with strong emotional statements. Similar to the service quality and recovery paradigm in other domains suggests [85] that patients think positively about a physician after their mistakes have been corrected, compared to how they would look at the physician if high-quality care were delivered. Further, physicians should be vigilant to negative feedback with intense emotional expressions. Such kind of feedback would influence the review site's profile, which could possibly cause clients to leave for other sites. To maintain strong ties with patients in the digital world, it would be important for the review sites to have clear guidance on posting reviews and online protocols.

#### *5.4. Limitations and Future Research*

This research has certain limitations that could be addressed in future works. First, we only examined the linguistic features and their impact on consumers' choice. There are some other characteristics (i.e., review valance and review volume), which may also affect consumers' choice. Future research should consider these characteristics and their impact on consumers' behavior. Second, in this study, we collected data for 6 months during the epidemic period; however, research must be carried out over a substantially longer period. As part of our further research, we will continue collecting data at different time points when the epidemic comes to an end.

#### **6. Conclusions**

With the implementation of digital technology in healthcare, the utilization of PRWs becomes increasingly popular during the COVID-19 epidemic crisis. An OPR that evaluates a physician's competence would inevitably impact the patients' choice of a physician on PRWs. This research examined the impact of affective signals and informative signals on patients' physician selection. Further, the mediation effect of IH was investigated in the relationship between the linguistic signals and patients' treatment choice. Our findings revealed that negative sentiment, review readability, review depth, review spellings, and IH have positive effects on patients' treatment choice. Moreover, the IH mediates the relationship between negative sentiment, review depth, and treatment choice. Theoretically, this paper establishes a research model based on signaling theory to understand the linguistic signaling mechanism on patients' behaviors and adds significantly to the literature concerning PRWs. In practice, our research findings suggest that web designers and physicians better deal with the impact of OPRs on them.

**Author Contributions:** Conceptualization, M.A.; methodology, A.M.S.; software, A.Q.; validation, A.B. and A.A.-M.; formal analysis, A.M.S.; investigation, L.A.-C.; resources, A.Q.; data curation, A.M.S.; writing—original draft preparation, A.M.S.; writing—review and editing, A.M.S.; visualization, A.A.-M.; supervision, H.H.; project administration, L.A.-C.; funding acquisition, H.H. All authors have read and agreed to the published version of the manuscript.

**Funding:** There is no external funding source.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Does Being Ill Improve Acceptance of Medical Technology?— A Patient Survey with the Technology Usage Inventory**

**Sabur Safi 1,\* , Gerhard Danzer <sup>2</sup> , Solaiman Raha <sup>3</sup> , Eyyad Nassar <sup>3</sup> , Frank T. Hufert <sup>4</sup> and Kurt J. G. Schmailzl <sup>3</sup>**


**Abstract:** Acceptance of new medical technology may be influenced by social conditions and an individual's background and particular situation. We studied this acceptance by hypothesizing that current and former COVID-19 patients would be more likely to accept an electrocardiogram (ECG) "patch" (attached to the chest) that allows continuous monitoring of the heart than individuals who did not have the disease and thus the respective experience. Currently infected COVID-19 patients, individuals who had recovered from COVID-19, and a control group were recruited online through Facebook (and Instagram) and through general practitioners (GPs). Demographic information and questions tailored to the problem were collected via an online questionnaire. An online survey was chosen in part because of the pandemic conditions, and Facebook was chosen because of the widespread discussions of health topics on that platform. The results confirmed the central hypothesis that people who had experienced a disease are more willing to accept new medical technologies and showed that curiosity about new technologies and willingness to use them were significantly higher in the two groups currently or previously affected by COVID-19, whereas fears of being "monitored" (in the sense of surveillance) were significantly higher among people who had not experienced the disease and threat. Experiencing a serious disease ("patient experience") promotes acceptance of new medical technologies.

**Keywords:** COVID-19; technology acceptance; user survey; wearable health monitor; ECG patch

#### **1. Introduction**

People's attitudes toward new medical technologies are driven in part by their cultural background and their confidence in the existing health care system and its effectiveness and equity. However, there is a two-fold problem regarding medical technology—its availability and, perhaps more critically, people's acceptance of technology in general and their willingness to accept monitoring or even accompanying inconveniences caused by new, unfamiliar technologies. The Unified Theory of Acceptance and Use of Technology (UTAUT) is a theory used to predict consumer acceptance of technology. It assumes four independent factors influence acceptance [1,2]. These four factors are performance expectancy, effort expectancy, social influence and facilitating conditions.

This theory, originally formulated in computer science, can also be applied to predicting user acceptance of medical technology. As in computer science, the acceptance of medical technology by its users, consumers, and patients depends on many different underlying conditions. These include device features (performance expectation of the technology), ease of use, costs, whether the medical device constrains its user or how this

**Citation:** Safi, S.; Danzer, G.; Raha, S.; Nassar, E.; Hufert, F.T.; Schmailzl, K.J.G. Does Being Ill Improve Acceptance of Medical Technology?— A Patient Survey with the Technology Usage Inventory. *Int. J. Environ. Res. Public Health* **2021**, *18*, 9367. https:// doi.org/10.3390/ijerph18179367

Received: 26 June 2021 Accepted: 24 August 2021 Published: 5 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

constraint relates to the expected benefits of the device (effort expectation), and appearance and other socially determined features and circumstances [3]. These conditions are significantly shaped by the patient's overall background, including their medical history, and their current health status.

For example, GPS trackers are commercially available for older people with mild to moderate dementia to allow them to go out on their own and, should they become lost, to be found easily. GPS trackers can be used for many purposes outside of medicine, but can be used in medicine for people with cognitive impairment and dementia-related syndromes. Caregivers or family members involved in the care can use a smartphone app to locate the patient is in real time. This allows the person being monitored to actually be more independent. At the same time, however, it is possible that the monitoring function may be perceived as an invasion of privacy, and this, along with a threshold fear of the new technology, may result in the tool not being adopted. Threshold anxiety is a term used to explain the psychological barriers people experience, or the fear people feel when faced with something unknown, in this case a new technology. Acceptance depends on how the patient views the advantage of the device, in this case the ability to go out alone, versus the disadvantage of the impairment caused by the device [4,5].

The outcome of this cost–benefit balance depends on the particular circumstances. In the case of the GPS tracker, the average healthy user would most likely feel they were subject to excessive surveillance. In contrast, for a patient with cognitive impairment, the tracker may provide a level of security that is perceived as reassuring and/or makes unaccompanied walks possible, i.e., expands rather than restricts the user's freedom [3]. Therefore, technologies can be useful in the medical field for a variety of purposes.

Numerous other examples show that patients are more likely to accept a medical device that would be completely unacceptable to healthy individuals because they expect to receive benefits in their situation that would otherwise not be available [6–8]. These include technologies for detecting falls or epileptic seizures, continuous blood glucose monitors, and other more or less invasive devices for diabetics [9–11]. Acceptance of reusable respiratory filters increased in the wake of the current COVID-19 pandemic [12]. As more people began to use different types of face masks, the threat of shortages became greater and some countries even stopped the export of face masks so they were able to meet the demand within their countries [13]. Consequently, it can be deduced that there is a correlation between the demand and the shortages, and between the more widespread use of face masks and the higher demand for them.

It is therefore evident that the acceptance of a medical device depends on the individual and changing circumstances of the user. However, there is little literature that has directly compared the acceptability of a medical device in patients and healthy individuals or has investigated whether healing from an acute illness affects acceptability. For this reason, the pilot study presented here was conducted. This study can be a step forward in filling the research gap in the area of the relationship between the medical condition and patients' acceptance of new medical technology. In the present paper, patients with active COVID-19 disease (Group 1—COVID), people who had survived COVID-19 disease (Group 2—recovered), and Group 3—healthy people (regardless of the possible presence of other comorbidities) were asked about their acceptance of medical technology using the example of an electrocardiogram (ECG) patch. Acceptance of medical technology is important for several reasons. First, medical technology can improve the likelihood of recovery for people who are overcoming illnesses that need further medical support. Second, medical technology can help in the diagnosis and early detection of illnesses. Finally, medical technology can help in monitoring medical conditions. Consequently, acceptance of medical technology can significantly improve the overall medical health of people at the different stages of a medical condition—either for prevention, recovery, or monitoring. The present study aimed to compare the attitude towards medical technology in general, and towards the electrocardiogram (ECG) patch in particular, of healthy people, people suffering from COVID-19, and people who have survived COVID-19. The study

maintained the thesis that being a patient, that is, suffering from a medical condition or illness, increases the level of acceptance of medical technology. The study is innovative in the manner that it collects data from healthy people, which was not undertaken in most of the previous research. Collecting and analyzing data from healthy people is important in evaluating the general attitude towards medical technology and not only the attitude of those suffering from an illness. By being aware of healthy people's attitudes towards new medical technology, we can bring about an overall positive change by educating healthy people about the importance of medical technology. It is clearly preferable that this change in attitude occurs prior to individuals becoming ill, given the time it takes to bring about a change in their acceptance may prove fatal.

#### **2. Material and Methods**

#### *2.1. Recruitment*

The survey was conducted online from April to November 2020 using Unipark academic survey software, that runs on Questback EFS (Winter 2019 v.32.2). Unipark was the preferred method for data collection due to its user-friendly interface, which ensured positive participant attitudes toward the survey, and its flexibility in creating question types and structure. Overall, the method ensured that data would be collected objectively and securely, because Unipark offers a high level of data security. In addition, Unipark provides the ability to extract data in a variety of formats. For the present study, the data were stored in Excel. Overall, Unipark provided participant satisfaction and ease of subsequent analysis of the data collected. The survey was placed on Facebook in various COVID-19 groups, and participants were asked to respond anonymously. Data collection took place in Germany and neighboring countries, because the COVID-19-related groups in which the survey was distributed included individuals from these countries. Data were collected anonymously and voluntarily, and the analysis did not include an assessment of the particular situation or condition of individual participants, but simply their opinions on medical technology issues. Therefore, the survey posed an extremely low risk to the participants and the data they provided, and participant consent was not required. The data collected were not used to link them to individuals, nor was it of interest to determine who the individuals were. In addition, two bloggers on Instagram helped distribute the survey and two general practitioners (GPs) distributed the survey to their respective patients. This aimed to reach many and varied people, and recruit them to complete the survey. The survey distributed by the bloggers and the GPs was not different from the other survey posted in the Facebook groups. In each case, the survey was distributed via a link and subject participation was voluntary and anonymous. Study participants were divided into three study groups: (a) COVID-19 patients with active symptoms, (b) individuals who had recovered from COVID-19, and (c) healthy individuals without current and previous COVID-19 disease. The inclusion criterion was a clinically confirmed diagnosis, i.e., COVID-19 confirmed by PCR assay. Exclusion criteria included misunderstanding the purpose of the survey and lack of access to technology, which effectively meant that these individuals could not participate in the survey. Patients with severe cases of COVID-19 who were being treated in an intensive care unit and were unable to actively participate were also excluded from the survey.

#### *2.2. Data Collection*

All study participants were given a questionnaire, included in the Appendix A, to collect basic demographic data and data on their attitudes toward technology in general, and finally to collect data on their attitudes toward technology in light of the current COVID-19 pandemic. The questionnaire started by collecting general demographic and social data such as gender, age, place of residence, marital status, and current health status in relation to COVID-19 (whether the person has had it before, currently has it, is hospitalized, or has never been exposed to the disease). The following questions about participants' general attitudes toward technologies and technology aimed to measure the

level of understanding of new technologies, whether participants see technology as a facilitator and a means to make daily life easier, how affordable they think new technology is and should be, and whether they associate technologies with dangers and risks. The questions deal with how they learn about new technologies. Each question was answered on a seven-point Likert scale, with half points also possible as an answer. The following ten questions on COVID-19 contained the same scale. The questions aimed to measure correlations between whether a person is or has been affected by COVID-19 and his or her attitude toward technology and technical aids, and any change in attitude toward medical technology resulting from the individual being affected. The questionnaire also elicited attitudes toward the eight general factors of curiosity, fear of technology, interest, ease of use, usefulness, skepticism, accessibility, and "intention to use". The questionnaire was based on the Technology Usage Inventory (TUI), although one dimension of the instrument (intention to use) was omitted because it was not relevant to this case study.

In addition, ten other questions developed by the authors were included that aimed to measure attitudes specifically of individuals who had been exposed to COVID-19.

All participants completed the survey online. However, the number of individuals >65 years of age who completed the survey was lower. Nonetheless, the data collected from individuals >65 years of age were included in the data set and were not treated as a subgroup despite the lower number of participants.

#### *2.3. Statistical Analysis*

The dataset was analyzed using SPSS (Statistical Product and Service Solutions), first looking for relationships between participants and the expressions of each variable. Following this, the variables were analyzed among themselves. This included the internal consistency (Cronbach's alpha) and the discriminatory power of the items (ITC = item-total correlation); a value of >0.7 was considered acceptable. For the discriminatory power, the correlation between the measured value of an item and the result of the measured value of the variable without the respective item under consideration was determined. Items with a correlation of r < 0.3 were excluded. For the analysis of the present data, the TUI was a particularly appropriate instrument because it provides a different scale for assessing both general attitudes toward technology and attitudes toward a specific device.

For the descriptive analysis, the median and the 25% and 75% quartiles, the mean, the standard deviation, and the minimum and maximum values were calculated. Histograms of response frequencies were created for graphical representation. Differences between the means of the three study groups were tested for significance using ANOVA, and a *p*-value of *p* < 0.05 was considered significant.

#### **3. Results**

#### *3.1. Basic Demographic Parameters*

This study was conducted to test the hypothesis that there is a correlation between experience of a condition and attitudes toward medical technology.

The study compared attitudes toward medical technology in general and, specifically, acceptance of an ECG patch in a group of current COVID-19 patients (COVID), a second group who had survived COVID-19 (COVID-recovered), and healthy individuals (healthy).

A total of 607 participants took part in the study, including 155 men (25.5%). The COVID group included 130 participants, of whom 70 (53.7%) were male. The COVIDrecovered group consisted of 127 participants, of whom 59 (46.5%) were male. Group 3, in which participants had no personal current or previous experience with COVID-19, consisted of 350 participants, of whom 26 (7.4%) were male. Four (0.7%) study participants reported "diverse" as their gender (one each in the COVID- and COVID-recovered groups and two in the healthy group). Thus, the healthy group had a statistically significant excess of females compared to the other two study groups (*p* < 0.001). The two COVID groups also differed from group 3 in terms of age distribution. Group 3 had a significantly higher proportion of study participants in the 18–24 and 25–34 age groups, whereas Groups 1 and

2 had a higher proportion of participants in the 35–44 and 45–54 age groups, respectively (*p* < 0.001).

Other baseline demographic parameters of the study participants are listed in Table 1. Significant differences between groups existed in terms of marital status, smoking status, education level, and place of residence, in addition to gender and age. Groups 1 and 2 were relatively similar to each other and different from Group 3, but the groups did not differ with respect to the proportion of study participants with minor children (*p* = 0.380).


**Table 1.** Demographic parameters of the study participants.


**Table 1.** *Cont.*

#### *3.2. Technology Acceptance*

The study showed that a significant difference exists among the three groups in terms of various acceptance parameters for the ECG patch. Thus, scores for both curiosity about the technology and interest in using it were highest in the COVID group and lowest in the healthy group, with group differences reaching significance only for curiosity (A, *p* < 0.001 and B, *p* = 0.739). Conversely, an opposite tendency was found for fear of technology and skepticism toward technology. Here, the mean values were highest in the healthy group and low in COVID-19 patients and those who had recovered from COVID-19. Again, group differences reached significance for only one of the two factors, skepticism (C, *p* = 0.148 and D, *p* = 0.001). The COVID group also had the highest expectations concerning ease of use (*p* = 0.003), accessibility (*p* = 0.001), and usefulness (*p* = 0.005) of the ECG patch. Here, too, the recovered study participants had values that lay very close to those of the COVID group, whereas the values of the healthy people were much lower.

#### *3.3. COVID-Specific Questions*

The third part of the study examined how attitudes toward modern medical technology have changed as a result of the COVID-19 pandemic. A questionnaire with 10 questions was set up for this purpose. Again, there were some significant differences between the groups. Interestingly, the recovered group took an intermediate position between the diseased and healthy groups on five of these questions, showing surprisingly high or low scores in three cases (Figures 1 and 2).

**Figure 1.** Average attitudes toward different aspects of technology acceptance in the three study groups. The *y*-axis shows the mean result of participants choice on the Likert scale: (**A**) Curiosity about the technology; (**B**) Interest in using it; (**C**) **Figure 1.** Average attitudes toward different aspects of technology acceptance in the three study groups. The *y*-axis shows the mean result of participants choice on the Likert scale: (**A**) Curiosity about the technology; (**B**) Interest in using it; (**C**) Fear of technology; (**D**) Skepticism about the technology. **Figure 1.** Average attitudes toward different aspects of technology acceptance in the three study groups. The *y*-axis shows the mean result of participants choice on the Likert scale: (**A**) Curiosity about the technology; (**B**) Interest in using it; (**C**) Fear of technology; (**D**) Skepticism about the technology.

**Figure 2.** Expectation scores related to ease of use (**A**), accessibility (**B**), and usefulness (**C**) of the ECG patch in the three study groups. The *y*-axis shows the mean result of participants choice on the Likert scale. **Figure 2.** Expectation scores related to ease of use (**A**), accessibility (**B**), and usefulness (**C**) of the ECG patch in the three study groups. The *y*-axis shows the mean result of participants choice on the Likert scale.

The group acutely affected by COVID-19 agreed most strongly with the statement

that the COVID-19 pandemic could be a major reason for ambulatory use of medical devices to monitor their health status, followed by the recovered group (*p* = 0.001; Figure 3). Interestingly, however, both COVID-affected and healthy individuals on average agreed more strongly than the recovered group with the statement that multiple sensors on the

vices to monitor their health status, followed by the recovered group (*p* = 0.001; Figure 3). Interestingly, however, both COVID-affected and healthy individuals on average agreed more strongly than the recovered group with the statement that multiple sensors on the

The group acutely affected by COVID-19 agreed most strongly with the statement that the COVID-19 pandemic could be a major reason for ambulatory use of medical devices to monitor their health status, followed by the recovered group (*p* = 0.001; Figure 3). Interestingly, however, both COVID-affected and healthy individuals on average agreed more strongly than the recovered group with the statement that multiple sensors on the body could make a person appear "older" or "sicker" than they are (*p* = 0.823; Figure 4), although group differences on this statement did not reach statistical significance. In contrast, both COVID patients and convalescents appeared to be more open-minded and tolerant of body-worn sensory technology and showed generally positive attitudes toward modern medical technology, but at the same time agreed significantly less often and less than Group 3 (healthy individuals) with the statement that such wearables would always remind them of "sickness" and thus would certainly establish mental distress (*p* = 0.001; Figure 5). The somewhat opposite statement that wearables and sensor technology could help reassure family and friends was answered accordingly by the three groups, with the COVID group having the highest mean, followed by the recovered group (*p* < 0.001; Figure 6). COVID group having the highest mean, followed by the recovered group (*p* < 0.001; Figure 6). All three groups tended to agree with the statement that it was important for sensory equipment to be aesthetically pleasing or as unobtrusive as possible. Group differences regarding this statement were not statistically significant (*p* = 0.547; Figure 7). The healthy group was least likely to agree that medical records should be available to treating physicians, but again the group differences were not significant (*p* = 0.194, Figure 8). Interestingly, it was the recovered group that was most willing to share contact tracing data with government agencies while having the least concern about data security (*p* = 0.028 and *p*  = 0.001, Figures 9 and 10, respectively). The differences in these two statements reached statistical significance. The recovered group was also most likely to agree that patients should make private co-payments for such sensor technology, but these differences were not significant (*p* = 0.176, Figure 11). However, both the COVID-affected and recovered groups were significantly more likely than the healthy group to agree that their attitudes toward medical technology had changed since the COVID 19 pandemic (*p* < 0.001, Figure 12).

body could make a person appear "older" or "sicker" than they are (*p* = 0.823; Figure 4), although group differences on this statement did not reach statistical significance. In contrast, both COVID patients and convalescents appeared to be more open-minded and tolerant of body-worn sensory technology and showed generally positive attitudes toward modern medical technology, but at the same time agreed significantly less often and less

Figure 5). The somewhat opposite statement that wearables and sensor technology could help reassure family and friends was answered accordingly by the three groups, with the

*Int. J. Environ. Res. Public Health* **2021**, *18*, x FOR PEER REVIEW 8 of 17

**Figure 3.** Average agreement with statement 1 of the COVID-specific questionnaire: "The Covid-19 pandemic is a reason for outpatient use of medical devices to monitor health conditions." The *y*-axis shows the mean result of participants choice on the Likert scale. **Figure 3.** Average agreement with statement 1 of the COVID-specific questionnaire: "The Covid-19 pandemic is a reason for outpatient use of medical devices to monitor health conditions." The *y*-axis shows the mean result of participants choice on the Likert scale.

*Int. J. Environ. Res. Public Health* **2021**, *18*, x FOR PEER REVIEW 9 of 17

**Figure 4.** Average agreement with statement 2 of the COVID-specific questionnaire: "Multiple sensors on the body could make a person appear older or more ill than they are." The *y*-axis shows the mean result of participants choice on the Likert scale. **Figure 4.** Average agreement with statement 2 of the COVID-specific questionnaire: "Multiple sensors on the body could make a person appear older or more ill than they are." The *y*-axis shows the mean result of participants choice on the Likert scale. **Figure 4.** Average agreement with statement 2 of the COVID-specific questionnaire: "Multiple sensors on the body could make a person appear older or more ill than they are." The *y*-axis shows the mean result of participants choice on the Likert scale.

**Figure 5.** Average agreement with statement 3 of the COVID-specific questionnaire: "Body-worn sensor technology would always remind me of illness and thus would certainly be a psychological burden." The *y*-axis shows the mean result of participants choice on the Likert scale.

participants choice on the Likert scale.

**Figure 5.** Average agreement with statement 3 of the COVID-specific questionnaire: "Body-worn sensor technology would

**Figure 6.** Average agreement with statement 4 of the COVID-specific questionnaire: "Body-worn sensor technology for monitoring could help reassure family and friends as my health would always be medically monitored." The *y*-axis shows the mean result of participants choice on the Likert scale. **Figure 6.** Average agreement with statement 4 of the COVID-specific questionnaire: "Body-worn sensor technology for monitoring could help reassure family and friends as my health would always be medically monitored." The *y*-axis shows the mean result of participants choice on the Likert scale.

All three groups tended to agree with the statement that it was important for sensory equipment to be aesthetically pleasing or as unobtrusive as possible. Group differences regarding this statement were not statistically significant (*p* = 0.547; Figure 7). The healthy group was least likely to agree that medical records should be available to treating physicians, but again the group differences were not significant (*p* = 0.194, Figure 8). Interestingly, it was the recovered group that was most willing to share contact tracing data with government agencies while having the least concern about data security (*p* = 0.028 and *p* = 0.001, Figures 9 and 10, respectively). The differences in these two statements reached statistical significance. The recovered group was also most likely to agree that patients should make private co-payments for such sensor technology, but these differences were not significant (*p* = 0.176, Figure 11). However, both the COVID-affected and recovered groups were significantly more likely than the healthy group to agree that their attitudes toward medical technology had changed since the COVID 19 pandemic (*p* < 0.001, Figure 12).

the mean result of participants choice on the Likert scale.

participants choice on the Likert scale.

**Figure 5.** Average agreement with statement 3 of the COVID-specific questionnaire: "Body-worn sensor technology would always remind me of illness and thus would certainly be a psychological burden." The *y*-axis shows the mean result of

**Figure 7.** Average agreement with statement 5 of the COVID-specific questionnaire: "It's important to me that such sensor technology looks aesthetically pleasing or is completely unobtrusive." The *y*-axis shows the mean result of participants choice on the Likert scale. choice on the Likert scale.

**Figure 8.** Average agreement with statement 6 of the COVID-specific questionnaire: "Thanks to new digital technologies, large amounts of data can be stored and retrieved at any time. My medical records should be available to my treating physicians." The *y*-axis shows the mean result of participants choice on the Likert scale. **Figure 8.** Average agreement with statement 6 of the COVID-specific questionnaire: "Thanks to new digital technologies, large amounts of data can be stored and retrieved at any time. My medical records should be available to my treating physicians." The *y*-axis shows the mean result of participants choice on the Likert scale.

**Figure 9.** Average agreement with statement 7 of the COVID-specific questionnaire: "In the fight against pandemics, movement data can help—for example, to trace chains of infection. Medical data collected through body-worn sensor

ticipants choice on the Likert scale.

choice on the Likert scale.

large amounts of data can be stored and retrieved at any time. My medical records should be available to my treating

physicians." The *y*-axis shows the mean result of participants choice on the Likert scale.

**Figure 7.** Average agreement with statement 5 of the COVID-specific questionnaire: "It's important to me that such sensor technology looks aesthetically pleasing or is completely unobtrusive." The *y*-axis shows the mean result of participants

**Figure 9.** Average agreement with statement 7 of the COVID-specific questionnaire: "In the fight against pandemics, movement data can help—for example, to trace chains of infection. Medical data collected through body-worn sensor **Figure 9.** Average agreement with statement 7 of the COVID-specific questionnaire: "In the fight against pandemics, movement data can help—for example, to trace chains of infection. Medical data collected through body-worn sensor technology should be available to government institutions." The *y*-axis shows the mean result of participants choice on the Likert scale. technology should be available to government institutions." The *y*-axis shows the mean result of participants choice on the Likert scale.

**Figure 10.** Average agreement with statement 8 of the COVID-specific questionnaire: "I have security concerns (e.g., hacking attacks/manipulation) about such sensor technology to monitor my health." The *y*-axis shows the mean result of par-**Figure 10.** Average agreement with statement 8 of the COVID-specific questionnaire: "I have security concerns (e.g., hacking attacks/manipulation) about such sensor technology to monitor my health." The *y*-axis shows the mean result of participants choice on the Likert scale.

ticipants choice on the Likert scale.

the Likert scale.

**Figure 10.** Average agreement with statement 8 of the COVID-specific questionnaire: "I have security concerns (e.g., hack-

technology should be available to government institutions." The *y*-axis shows the mean result of participants choice on

**Figure 11.** Average agreement with statement 9 of the COVID-specific questionnaire: "If health insurance companies do not cover the entire cost of such sensor technology, the cost should be offset by private co-payments." The *y*-axis shows the mean result of participants choice on the Likert scale. **Figure 11.** Average agreement with statement 9 of the COVID-specific questionnaire: "If health insurance companies do not cover the entire cost of such sensor technology, the cost should be offset by private co-payments." The *y*-axis shows the mean result of participants choice on the Likert scale.

**Figure 12.** Average response to question 10 of the COVID-specific questionnaire: "Has your attitude toward outpatient medical technology changed since the COVID-19 pandemic?" The *y*-axis shows the mean result of participants choice on the Likert scale. **Figure 12.** Average response to question 10 of the COVID-specific questionnaire: "Has your attitude toward outpatient medical technology changed since the COVID-19 pandemic?" The *y*-axis shows the mean result of participants choice on the Likert scale.

The influence of various confounding parameters on participants' responses between all three groups (COVID, recovered, and healthy) was examined. An influence of gender,

cation level on some of the responses was found, but this influence was small. These influences were examined in terms of interactions between Groups 1–3 (COVID, recovered, and healthy, respectively) and sociodemographic indicators when answering the questions. Results showed that, of 136 analyses in which the results for the questionnaire scales (i.e., curiosity, technology anxiety, etc.) were compared to sociodemographic factors, only 19 showed significant differences, whereas 117 show no statistical differences (Table A1 in Appendix A). There was a significant interaction between Groups 1–3 and gender for technology anxiety (*p* = 0.043) and for statement 2 of the COVID-specific questions (*p* = 0.028). Furthermore, significant interactions between Groups 1–3 and marital status were evident for curiosity (*p* = 0.023), skepticism (*p* = 0.039), and statement 10 of the COVIDspecific questions (*p* = 0.041). A significant interaction between Groups 1–3 and minor children was shown for interest (*p* = 0.011) and for statement 10 of the COVID-specific questions (*p* = 0.030). Smoking status also had an influence on some responses in Groups 1–3. There were significant interactions between Groups 1–3 and smoking status for accessibility (*p* = 0.008) and for statements 6 (*p* = 0.033) and 9 (*p* = 0.007) of the COVID-specific questions. For education level, a significant interaction with Groups 1–3 was only found for statement 7 of the COVID-specific questions (*p* = 0.028). In contrast, for place of residence, significant interactions between Groups 1–3 were demonstrated for curiosity (*p* =

*3.4. Confounding Parameters* 

#### *3.4. Confounding Parameters*

The influence of various confounding parameters on participants' responses between all three groups (COVID, recovered, and healthy) was examined. An influence of gender, marital status, having minor children, smoking status, place of residence, state, and education level on some of the responses was found, but this influence was small. These influences were examined in terms of interactions between Groups 1–3 (COVID, recovered, and healthy, respectively) and sociodemographic indicators when answering the questions. Results showed that, of 136 analyses in which the results for the questionnaire scales (i.e., curiosity, technology anxiety, etc.) were compared to sociodemographic factors, only 19 showed significant differences, whereas 117 show no statistical differences (Table A1 in Appendix A). There was a significant interaction between Groups 1–3 and gender for technology anxiety (*p* = 0.043) and for statement 2 of the COVID-specific questions (*p* = 0.028). Furthermore, significant interactions between Groups 1–3 and marital status were evident for curiosity (*p* = 0.023), skepticism (*p* = 0.039), and statement 10 of the COVID-specific questions (*p* = 0.041). A significant interaction between Groups 1–3 and minor children was shown for interest (*p* = 0.011) and for statement 10 of the COVID-specific questions (*p* = 0.030). Smoking status also had an influence on some responses in Groups 1–3. There were significant interactions between Groups 1–3 and smoking status for accessibility (*p* = 0.008) and for statements 6 (*p* = 0.033) and 9 (*p* = 0.007) of the COVID-specific questions. For education level, a significant interaction with Groups 1–3 was only found for statement 7 of the COVID-specific questions (*p* = 0.028). In contrast, for place of residence, significant interactions between Groups 1–3 were demonstrated for curiosity (*p* = 0.021), interest (*p* = 0.021), and for statements 1 (*p* = 0.005), 2 (*p* = 0.035), and 10 (*p* = 0.010) of the COVID-specific questions. For place of living, significant interactions within Groups 1–3 were found for interest (*p* = 0.028) and accessibility (*p* = 0.012). Within Groups 1–3 there was no significant interaction found when comparing the age groups with the different scales. All *p*-values for the interactions are listed in Table A1 in the Appendix A. Although it is important to mention that the sociodemographic variety between Groups 1–3 had some influence on the responses to the surveyed questions, in the majority of cases, the demographic characteristics did not have a significant impact on the participants' answers. This should be taken into account when interpreting the group differences.

The presence of various comorbidities, such as hypertension, respiratory disease, diabetes mellitus, and cardiovascular disease, had a significant influence on some, but not all, of the responses and generally increased the response tendencies of COVID-affected and recovered individuals.

#### **4. Discussion**

The study presented here investigated the hypothesis that being or having been ill increases acceptance of medical technology. For this purpose, current COVID-19 patients (Group 1), people who had recovered from COVID-19 (Group 2), and people with no previous or current COVID-19 disease (Group 3) completed a questionnaire on different aspects of medical technology acceptance. The responses confirmed the hypothesis in several aspects, e.g., people diagnosed with COVID-19 and those who had recovered from it were more curious about modern medical technology and had a higher interest in using it, whereas healthy people were at the same time more fearful and skeptical about technology and its monitoring potential. Understanding the acceptance factors for medical technology is important to improve the use and benefits of new technologies. A study with scoliosis patients showed that aesthetic factors in the design of corsets were an important contributor to acceptance [14]. A comparable result was seen in the study presented here, where all three groups placed importance on the used sensor technology either looking aesthetically pleasing or being as unobtrusive as possible. Product design is relevant to how people view a technology and their willingness to use it.

However, most studies published to date have investigated the acceptability of a particular medical technology only in patients who use the particular technology as part of

their treatment, and not in healthy people. The present study fills this gap by also looking at healthy individuals. By comparing the opinions of healthy people with those of people who suffer from or have survived COVID-19, the work enables a conclusion to be drawn about preventive medicine. Personal experience of health and illness leads people to trust or distrust technologies and value intrusions into their privacy.

Previous research shows that the stigma is particularly high for obvious medical aids, such as wheelchairs, and is associated with depression and limitations of use in patients with recent spinal cord injuries [15]. The observations of the present study show that even healthy individuals can be skeptical of monitoring and new technologies. As the wheelchair example shows, acceptance toward medical technology is important because it can significantly improve people's lives and overcome stigma by promoting autonomy and self-determination.

Another interesting aspect of this study is that individuals who had recovered from COVID-19 were more accepting of using data from wearables or smartphones to track contacts and had fewer data security concerns than current COVID sufferers or healthy individuals. This may reflect the fact that COVID-19 is a highly contagious infectious disease with a relatively high case fatality rate for at-risk populations, and that those who have recovered may better appreciate the value of contact tracing based on their own experience. A further line of inquiry would be to investigate whether the risk of infection and/or the severity of the disease play a role. Why actively ill people are less "openminded" is not clear. It is conceivable that being ill itself is a greater psychological burden at this time and thus altruistic motives, which may be present in recovered individuals, are relegated to the background.

#### *Limitations*

This study had several limitations. First, recruitment was conducted via social media, which alters the age composition of potential study participants compared with the general population. Data collection via social media also raises questions about misinformation in and through social media. Participants' sources of information were not queried. In addition, the study excluded individuals who are technophobic and do not use computers or social media from participation. Finally, the recruitment method excluded ill persons if they were too ill to use a computer at the time of recruitment. However, because this study was designed to identify differences in technology acceptance between patients, former patients, and healthy individuals, these limitations are of rather minor importance. Furthermore, due to the diverse nature of participants there were many sociodemographic differences between Groups 1–3, making results of comparative analysis between these groups difficult to interpret. To determine the extent to which these differences impacted our comparative analysis, we cross-examined all groups and their respective sociodemographic categories. These results, presented in Section 3.4 and listed in Table A1 in the Appendix A, show that, of 136 analyses, only 19 showed significant differences. Although it is important to mention these sociodemographic differences as possible limitations, these results further strengthen the validity of the data and conclusions.

#### **5. Conclusions**

This study confirmed the initial hypothesis that being ill or having recently been seriously ill increases open-mindedness toward new technologies and acceptance of medical technology related to the respective illness, and that this acceptance slightly decreases after recovery. This should have implications for preventive approaches in medicine.

**Author Contributions:** Conceptualization, S.S., K.J.G.S.; Methodology, S.S., F.T.H.; Software, S.S., S.R.; Validation, S.S., G.D., K.J.G.S.; Formal analysis, S.S.; Investigation, S.S.; Resources, S.S., G.D.; Data curation, S.S., E.N.; Writing—original draft preparation, S.S.; Writing—review and editing, K.J.G.S., F.T.H.; Visualization, S.S., S.R., E.N.; Supervision, G.D., K.J.G.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research project was privately funded by the authors.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the or Ethics Committee Medical School Hamburg (MSH-2021/121, 3 March 2021).

**Informed Consent Statement:** Patient consent was given before the questionnaire was filled out. The survey was conducted unanimously and the data collected did not aim to connect the particular participant to their answers but aimed to collect opinions in terms of agreement or disagreement with different statements. Moreover, the survey was voluntary and did not target specific people; instead, participants took part in the survey at their own will.

**Data Availability Statement:** The data used for all statistical analysis and calculations are stored in the research data repository PsychArchives and can be found here: https://hdl.handle.net/20.500.1 2034/4223 (accessed on 25 July 2021). Data storage was performed as stated in the ethical application under the number MSH-2021/121.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**

**Table A1.** *p*-values of interactions between Group 1 (COVID), Group 2 (Recovered), and Group 3 (Healthy) and sociodemographic indicators for all dependent variables. ANOVA comparing Groups 1–3 to the different scales of the TUI questionnaire.


Significant *p*-values are marked in bold.

#### **References**


## *Article* **Searching for New Model of Digital Informatics for Human–Computer Interaction: Testing the Institution-Based Technology Acceptance Model (ITAM)**

**Youngcheoul Kang <sup>1</sup> , Nakbum Choi 2,\* and Seoyong Kim <sup>3</sup>**


**Abstract:** The fourth industrial revolution has produced new information technology (IT) that is widely used in the healthcare industry. Although the nature of the institution affects IT adoption, the Technology Acceptance Model (TAM), a dominant theory, has dismissed its role and influence. Our research investigates how institutions influence the adoption of new IT by using the Institutionbased Technology Acceptance Model (ITAM). We conducted an empirical test by using survey data collected from 300 employees in the public sector. Structural equation modeling was applied to test the proposed hypotheses. The results showed the total effect of institutions on the intention to use new IT is positive and significant. Second, IT adoption is not only affected by external institutions but also by type of institution; the external institution takes a greater role in inducing perceived usefulness, perceived ease of use, and intention to use than does the internal. Third, perceived ease of use and perceived usefulness play mediating roles between institution and IT adoption. Fourth, an alternative expanded model to which more individual and organizational factors were added confirmed the results of the base model. We concluded that institutions have a strong impact on the level of intention for IT use through perceived ease of use and perceived usefulness.

**Keywords:** new information technology; Technology Acceptance Model (TAM); institution; institutionalism; digital informatics; human–computer interaction

#### **1. Introduction**

The rapid development of digital technologies has led to many ideas and attempts to change developmental paths in society [1]. These recent developments, along with the advancement of social media tools and IT, provide innovative and intense motivations to strengthen interactions between organizational boundaries and individuals [2,3]. Some argue that new technological developments could pave the way for long-term growth [4]. The development of digital technologies transforms conventional industry manufacturing, service delivery, policy decision-making, public participation and also influences the digital health management related to public health. The force of technologies leads to innovative approaches in many spheres of society, which motivates many companies and governments to follow these technological advancements. Because recent IT growth has been based on digital data availability and interconnections among individuals, every person has their own preference for using IT platforms, which require extensive resources to set up [2,5–7]. Thus, it is necessary to use new technology that guarantees organizational process optimization through interactive communications with various stakeholders. It is believed that these technological progresses could facilitate sustainable development since they afford many stakeholders real-time opportunities to voice their grievances on day-to-day policy issues in their community; moreover, these developments advance digital data integration [3]. As its development could enhance technological usability, economic wealth, and political

**Citation:** Kang, Y.; Choi, N.; Kim, S. Searching for New Model of Digital Informatics for Human–Computer Interaction: Testing the Institution-Based Technology Acceptance Model (ITAM). *Int. J. Environ. Res. Public Health* **2021**, *18*, 5593. https:// doi.org/10.3390/ijerph18115593

Academic Editors: Martin Wiesner and Björn Schreiweis

Received: 20 April 2021 Accepted: 18 May 2021 Published: 24 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

**<sup>\*</sup>** Correspondence: nakbum.choi@gmail.com

participation, new technology has become a major social and economic vehicle to change the ways people work, learn, and socialize [8]. With advancing developments of digital data, IT could lead to a break with precedent and prompt remarkable productivity and economic growth.

Applications of the fourth industrial revolution are considered to be revolutionizing structural elements in current society and have garnered attention in many countries [9]. As the scale and scope of change are vast and diverse, the impact of these new interactive technological developments is immense. The innovative development and diffusion of the fourth industrial revolution in the business sector is occurring faster than expected and many countries are trying to keep up with the pace of change. These changes are based on the idea that digital data sharing and interconnectedness among manufacturing stakeholders open new aspects of availability of technology, and many of these digitized devices will provide new approaches to acquiring individual demands and connecting to suppliers more conveniently [1]. Digitalization is becoming a major theme of development in every aspect as it provides a fundamental basis for recent technological innovations [9–12]. The fourth industrial revolution comprises new and transformative technologies, such as the internet of things, cloud computing, big data analytics, and automotive robotics, which can accelerate the pace of change in society.

Digital innovation is changing the fundamental structure of the public sector. Therefore, innovation strategies that consider the specific contexts and characteristics that exist in the public sector are necessary. Kromidha and Córdoba-Pachón ([13], p. 16) effectively explained the meaning of digital technologies in the public sector through adopting discursive institutionalism; digital innovation projects in the public sector are important strategic points of interaction for multifaceted stakeholders whose ideas and discourses could converge. In a transition context, coordinate discourse dominates thorough transformation projects, communicative discourse is valuable for change, participation in discourse is influenced by the power position, and the leadership philosophies' value-led discourses are tools to justify desirable change. Moreover, Ashok et al. [14] empirically showed that when adopting innovative technology, organizational culture and transformational leadership traits positively affect the adoption of knowledge management, which has a positive effect on organizational culture to counter organizational inertia.

In order to succeed in the innovation of information and communication technology in the public domain, it is important to overcome the inertia and failure factors that exist inside. Hannan and Freeman [15] define inertia from a population ecology perspective as a persistent organizational resistance to changing architectural features. If there is inertia, management requires only minimal discretion to adapt to environmental changes. To overcome this inertia, an organization must have the ability to quickly respond to new opportunities, which presumably competes with the ability to execute consistently and accurately. Moreover, Leonard-Barton [16] found that four core capabilities—employee knowledge and skills, technical systems, managerial systems, and the values and norms—have paradoxical attributions that facilitate and block innovation. Frost [17] classified the failure factors in knowledge management implementation into causal factors (lack of performance indicator, inadequate skill of knowledge managers and workers, problems with organization culture) and resultant failure culture (lack of widespread contribution, overemphasis on formation learning, and improper implementation of technology). How does one overcome those inertia and failure factors in public sectors? Hannan and Freeman [15] stressed continued learning to adapt to the new environment. Moreover, Leonard-Barton [16] argued that to succeed in new technology innovation, one must manage the paradox of conflicting capabilities. To overcome inertia in public sectors, Ashok et al. [14] stressed organizational culture and transformational leadership in knowledge management adoption. All of these are closely related to the institutional context.

Therefore, it is necessary to consider the impact of institutions and the role of governments as key factors to accelerate the usefulness of new technology and regulate basic principles. Governments are positioned to increase data availability and usage and public

employees play a critical role for innovation in government [18–20]. Consequently, understanding the attitudes and behavioral intentions of public employees is vital for the successful adoption and implementation of new technology. The fourth industrial revolution has diverse focus areas [21] because it includes various fields related to advanced IT developments and includes digitalization in the workplace and utilization of digital interactions in community activities [21]. As a result, the fourth industrial revolution is characterized as a collection of technologies that incorporate numerous digital transformative approaches in both the physical and biological spheres, particularly in fields such as public health [21,22]. This study is especially concerned with the examination of the relationship between institutions and the adoption of the fourth industrial revolution in the public sector.

In the new information age, new theoretical concepts and models related to informatics are required. In the literature regarding new technology adoption, the Technology Acceptance Model (TAM) has been widely used as a theoretical framework for predicting individuals' acceptance of technology. Since the TAM was introduced by Davis [23], great theoretical progress has been made by adding more contextual variables regarding individuals' behavioral intention to use technology [23,24]. TAM has been applied in many IT studies and has received substantial theoretical and empirical support from many researchers. Thus, TAM has the practical advantage of being extended to diverse contexts and settings. Therefore, we will base our argument on the idea of TAM that posits individual adoption of technology is mostly determined by two perceptions: usefulness and ease of use. In addition, we examine how these two core variables predict individuals' decisionmaking with regard to adopting new technology in institutional settings. However, the TAM model overlooks the macro-level structure and system in that it mainly focuses on the individuals' utility from the information system at the micro-level. Human–computer interactions differ depending on not only the direct relationship between the two but also the intervening variables. Therefore, it is necessary to consider structural variables at the macro level in TAM. In this study, we examine how not only micro-level individuals' utility but also macro-level institutional factors affect the acceptance of new technology and extend knowledge on technology adoption behaviors. We believe our findings can contribute to the extant literature of this domain and to understand critical factors when adopting new technology.

Because IT has been extensively used in the health field, it is very important to understand the fundamental structure of technology acceptance. Technology acceptance will be critical for providing better healthcare services through online IT, especially during pandemics like COVID-19. Although new technologies help healthcare services through telemedicine and online databases, some barriers to broader adoption of new technology persist. In particular, institutional readiness and the adoption of new technology by employees are necessary. Since healthcare service in nature is a system of combining microlevel individual actors and macro-level intuitional structures, the acceptance of new digital technology for health care service should consider not only humane side factors but also institutional settings which constrain people's attitude and perception. Digital technologies, the core of new technologies, cannot operate in isolation and require integrated institutional systems. In particular, digital data sources need to be integrated and interoperable into broader institutional settings. On the other hand, the government is one of the main actors in the institutional setting for healthcare services. The use and analysis of this data heavily depends on digital infrastructure and strong support from the public health system by governments. Governments' responsibility and strategic foresight are especially important in high-risk situations such as a pandemic. In this regard, it is necessary to understand the adoptive mechanism of new technology in the public sector, especially focusing on the impact of institutions. Therefore, our study is expect to contribute to understanding IT services for the healthcare field.

The remainder of this study is divided into four sections. First, based on previous research, the authors propose theoretical reviews and hypotheses on technology adoption

behaviors of employees and focus on the relationships with institutions and TAM, in particular. The next section discusses the methodology of this study, the sample and data used, and the variables and measurement. Next, the results are presented in the following section to show the relationships among factors affecting technology adoptions. The final section discusses the implications and limitations of the study with a conclusion.

#### **2. Theoretical Background and Hypotheses**

#### *2.1. New Information Technology in the Fourth Industrial Revolution and Its Adoption*

The increased and intensive use of IT requires governments to develop IT-based tools to directly deliver public service. The recent technological developments of the fourth industrial revolution emphasize the interconnectivity and reciprocity of the digital revolution and can lead to transformational changes at a societal level [25].

While recent IT developments bring many benefits, it is necessary to recognize the challenges of these new technologies in the public sector. The advantages of implementing new technologies are immeasurable, especially in terms of assisting public organizations in improving their tasks. The new technologies have high usability and flexibility and provide high-performance service delivery at low cost [26]. However, there are challenges regarding security, privacy, trust, and lack of understanding [26]. New technology adoption can create structural inertial pressures within organizations [14]. As March [27] suggested, internal organizational change was difficult to manipulate. Since the administrative system regulates resources, any external or internal institutional forces can cause resistance on adopting the new technical systems [15]. Also, when technological innovation enters an organization, the administration should consider how to recognize and manage the acceptable behavior of employees [16]. Therefore, considering the impact of institutions at the initial stage of technology diffusion, it is crucial to examine the employees' perception and contextual understanding of technology adoption for the success of new technology [17]. IT development provides innovative and intense contingencies that allow societies to strengthen their positions using IT platforms [2,5–7]. Among these trends, there have been intense and innovative attempts to reshape structures and improve performance in the public sector using interactive technology. For example, confronted with the current COVID-19 pandemic in many countries, we recognize the potential role of new technology in the fourth industrial revolution in alleviating and resolving issues [28]. However, the usage of this technology is still a long way off and each country has different institutional arrangements and resources to utilize the benefits of new technology.

With the unprecedented efforts to utilize digital technologies in the field of government and work processes, the advent of the fourth industrial revolution ushers people into digital transformation in all areas of their personal lives. Due to advanced internet technologies, this movement mainly includes ubiquitous data communication, managerial flexibility, disruptive innovations, and autonomous systems [29]. The current fourth industrial revolution movement involves the distribution and implementation of new technologies in accordance with digital interconnectivity. Many countries also plan to launch industry programs for the new technologies and have created intelligent technical stimuli to associate key actors with their implementation, including business, university, and government counterparts. For example, as initiated in 2006 by the German federal government, the fourth industrial revolution mainly presents a strategic vision and goals of economic and social transformation through digital innovation. Represented as hightech strategies and innovation, the key distinction of the fourth industrial revolution is its participatory decision-making processes involving multiple stakeholders [30]. Along with these movements, the German government also launched an IT-related government framework that emphasizes the role of government as a facilitator to construct sustainable digital infrastructure for the IT industry.

Many other countries also made strategic preparations to support the implementation of a wide range of innovations of the fourth industrial revolution. For example, Canada launched "Digital Canada", which includes five critical categories in new technology: connection, protection, cybersecurity, economic opportunities, and digital government [31–33]. Likewise, the Korean central government has tried to establish broad strategies to establish a universal digital platform [34–37]. To consider establishing a platform that can develop a network among related stakeholders involved in the fourth industrial revolution, the Presidential Committee for the fourth industrial revolution was launched in 2017 to bolster government support for related businesses [11,36]. In particular, South Korea effectively halted COVID-19 transmission during the pandemic's early period. It was about twice as effective as the U.S. The key to South Korea's success mainly came from blending technology and physical testing, institutional adjustment for centralized control, and coordination between actors. As a result, we concluded that South Korea is an appropriate location for studying the fourth industrial revolution's new technology adoption.

#### *2.2. Technology Acceptance Model (TAM) and Its Limitation*

Any innovative or transformative movement in large organizations, especially governments, should consider many relevant factors in their implementation. Even though the significant factors of IT implementation in government are diverse and complicated, they could be categorized as technological, structural, and behavioral factors [38–40].

Among the causal factors for successful application, institutions play a significant role when policymakers try to frame and introduce an IT policy [41–43]. The diffusion of technological innovation is up to actual users such as employees, stakeholders, and citizens, who are directly influenced by institutional arrangements. Institutional structures usually include legislative frameworks, managerial guidance, administrative rules, procedural standards, and conventional decision-making procedures. With this background, we seek to understand the behavioral factors of employees on new technology adoption and development in the context of the application of the fourth industrial revolution in the South Korean government. Because of its integrative and systemic approach toward a digitalized society, the government's support and drives are critical for successful implementation [10].

TAM has been continuously studied to estimate users' acceptance and behaviors; its application has expanded [24]. As a theoretical extension of the theory of reasoned action (TRA) [44], TAM provides a unified theory of technology acceptance and usage and persuasive explanations for individual motives for utilizing IT and its adoption [45,46]. TAM states that two fundamental perceptions determine technology acceptance behavior: perceived ease of use and usefulness. Perceived usefulness is defined as "the extent to which a person believes that using IT will enhance their performance" ([24], p. 187). Perceived ease of use is defined as "the degree to which a person believes that using an IT will be free of effort" ([24], p. 187). Therefore, TAM is recognized as a stable and manageable framework to understand user acceptance of new technologies in various organizations [47].

TAM has been continuously revised and new variables have been added to increase its theoretical explanatory power. Davis [23]'s original research showed that TAM fully explained the impact of system characteristics on users' behavior. However, this model only accounted for 36% of the variance and thus needed additional variables to increase the explanatory power. Dishaw and Strong [48] suggest an integration of TAM and tasktechnology fit structures in this direction. They argue that a more integrated model will contribute to explaining more choices regarding IT use. Venkatesh and Davis [24,49] showed that the extended model accounted for 40–60% of the variance in usefulness perceptions and 34–52% of the variance in use intentions. In particular, they found a significant role of both social influence processes, e.g., subjective norm and cognitive instrumental processes, e.g., job relevance. Lucas and Spitler [50] demonstrated that organizational variables such as social norms and the characteristics of job are more important in influencing use of the technology than users' perceptions about the technology.

However, TAM has several limitations. First, TAM overlooks various preferences because it emphasizes individual utility. According to Chtourou and Souiden [51], while the utility aspect of TAM is important, the hedonic aspect should be considered. The results suggest that product designers should develop interfaces and products that not only satisfy utilitarian needs but also hedonic and enjoyment motivations. This requires considering not only utility at the individual level but also the variables of the individual ideal. Moreover, Beglaryan et al. [52] argued that TAM research ignores factors beyond the individual, including group and social processes related to IT implementation and technology's organizational and social consequences.

Second, there are objective variables, not just subjective, among the causal factors affecting technology acceptance. On the other hand, TAM disregards these objective variables at the initial stage of development. Since previous research had focused on the variables existing inside the model, it overlooked the external variables. For example, Hu et al. [53] and Davis [23,54] did not consider external variables in TAM. However, recently, as interest in objective variables has increased, research on external variables has increased. Venkatesh et al. [55] focused on facilitating conditions as objective factors in the environment that an individual agrees to make an act easy to accomplish. Moreover, Agarwal and Prasad [56] tested their significant role with regard to technology, tenure in the workforce, level of education, and prior similar experiences. Igbaria et al. [57] investigated internal computing support, internal computing training, management support, external computing support, and external computing training participation in training.

Third, TAM neglects the conditions, environment, and context that promote the perceived usefulness and perceived ease of use it IT adoption. Maranguni´c and Grani´c [58] indicated that a field with considerable potential is the study of various information systems and environments. Nevertheless, more efforts to examine various environmental factors, including emotion, habits, personality differences, and technology change, are necessary. Recently, more research has focused on facilitating conditions or structural constraints in TAM. For example, Ngai, Poon, and Chan [59] investigated facilitating conditions and showed how they affect technology acceptance. Venkatesh and Bala [46] elaborated on facilitating conditions that are related to individuals' control beliefs regarding the availability of organizational resources and support structures to facilitate system use. They also found that perceived enablers or barriers in the environment influence a person's perception of ease or difficulty of performing a task [60]. Ajibade [61] argued that IT experiences promote the ease of use of technology, while technology acceptance and intention is moderated by more structural constraints like the company's rules, policies, and IT guidelines. Similarly, Lin and Wu [62] confirmed the role of intra-and extraorganizational factors as causal factors of end-user computing perception.

Following previous research, this study utilizes the extended TAM to explain the adoptive behavior and influencing factors of the fourth industrial revolution in governments. This study seeks to validate TAM in research on emerging new technology movement by analyzing factors that influence adoptive behavior in the central government of South Korea. Generally seen as a forerunner in the adoption and implementation of IT policies, South Korea is driving IT-based policy initiatives toward the successful transformation of the fourth industrial revolution [11]. Therefore, South Korea provides a useful field to carry out this study.

There have been many debates on the definition and concept of the fourth industrial revolution because of variation in countries' focus on applications and content [11,21]. As the application of core components in the fourth industrial revolution requires a comprehensive understanding of its application and support from various stakeholders, the role of public employees is especially critical. For successful implementation, it is necessary to develop appropriate operating systems and institutional arrangements to increase adaptability and flexibility in various polices, especially complex fields like environment and public health. In this regard, the current study applies institutional theory to the basic TAM for a better understanding on the impact of external arrangements that are embedded in individuals' usage of new technology. Because the interplay between technology usage and social contexts is complex and recursive, institutional aspects of IT contribute significantly to technology selection and use [63]. Institutions strongly influence peoples' procedural

decisions in the workplace [64] and employee behavior because their work procedure contents and standards are guidelines for the application of decision rules.

#### *2.3. Hypothesis Development*

TAM and extended ITAM posit complex human behaviors and decision-making processes that are subject to human–computer interaction. Researchers should take into account these intertwining relations in the model. This studies examine not only perceptional factors about institutional ones. A neoinstitutional framework makes links between the individual/behavioral and social/contextual sides of technology acceptance. Neoinstitutional theory applies its theories to explain technology implementation in an institutional environment. It regards informationization as a kind of institutional change and a process of institutionalization [65,66]. Jepperson ([67], pp. 145–152) defined an institution as "a social order or pattern that has attained a certain state or property". Institutionalism is "a theoretical strategy that features institutional theories and seeks to develop and apply them". According to institutionalism, institutions constrain and exert on an organization and organizational factors [68,69]. DiMaggio and Powell [68] suggested the three institutional pressures in organization: coercive, normative, and mimetic pressures. Similarly, Scott [69,70] proposed three institutional pillars (regulative, normative, and cultural-cognitive) that constrain and normalize individuals' behavior. Institutions then "represent constraints on the options that individuals and collectives are likely to exercise, albeit constraints that are open to modification over time" ([69], p. 94).

In an information system (IS) context, institutional approaches focus not only on "how institutions influence the design, use, and consequences of technologies, either within or across organizations" ([71], p. 153) but also how they have an impact on users' behavior in IS. Institutional theory sets the importance of contextualizing IT within the wider socioeconomic and political landscape [66]. Institutionalism focuses on the impact of institutions on technology users. Institutionalists believed in the formative power of institution and context in organization [72,73], which influences the way people think and constrains the way they act. Ciborra and Lanzara ([74], p. 70) describe formative contexts as "the set of institutional arrangements and cognitive imageries that inform the actors' practical and reasoning routines in organizations". Jensen et al. [75] addressed the electronic patient record (EPR) implementation works at three levels: the organizational field, the organizational/group, and the individual/sociocognitive level. In particular, macro-level structures, as well as individual interpretations, influence the adoption of IS. In the course of adoption, the institution significantly influences individual perceptions. Beglaryan et al. [52] examined the barriers of information system implementation concern and found that there are not only individual level factors: (a) perception, (b) expectancy and (c) utility of IS; however, there are also more institutional factors: (a) financial, (b) structural, (c) technical barriers (lack of infrastructure and suboptimal nature of the applied technology and solutions), (d) unavailability of facilitating conditions (involvement, training, organizational support, technical and expert support, and (e) lack of legal framework in IS implementation processes.

In the TAM model, institutional factors are regarded as external or social process elements. For example, Igbaria et al. [57] demonstrated that exogenous variables such as management support and external support have a significant impact on both perceived ease of use and perceived usefulness. Social influence processes (subjective norm, voluntariness, and image) and cognitive instrumental processes (job relevance, output quality, result demonstrability, and perceived ease of use) significantly influenced user acceptance [24]. Although institutions are not directly discernible in TAM, several variables are made available for attributions for institutions. For example, Venkatesh and Davis [24] identified significant antecedents such as subjective norm, perceived behavioral control, and selfefficacy. Subjective norms are shared collective beliefs that are regarded as an institution in institutionalism [76].

TAM also focuses on culture, which is a type of institution. The cultural-cognitive approach of neoinstitutional theory emphasizes the role of shared beliefs and ideas in shaping individual and organizational behaviors ([52], p. 52). When Straub et al. [77] compared the TAM model across three different countries, they found that TAM holds for both the United States and Switzerland but not for Japan, implying that TAM may not predict technology usage across all cultures. Beglaryan et al. [52] showed that administrative monitoring with an institutional character has a positive influence on perceived usefulness. Among the various institutional frameworks of TAM, culture is one of the challenging contexts that affect technology adoptive behavior [78]. Culture is shared with people who live within the same social environments. It consists of formal and informal rules of social interactions [79]. Therefore, culture may play a critical role in reducing uncertainty or risk avoidance behavior when confronting new technology. For example, individuals with the same culture who scored high on the uncertainty avoidance dimension may tend to seek ways to reduce risk and heavily rely on institutional arrangements [80].

However, with the wide and different ways of defining culture, it is difficult to examine and measure culture in an observable and constant manner to all individuals with the same cultural backgrounds [81]. Therefore, researchers usually regard culture as a type of collective institution or embedded institutional arrangement [82]. In this regard, McCoy et al. [81] suggested that the TAM model was considered to fit well in several countries, but some individual variations appeared as well. Thus, the TAM model might depend on culture as an institution collectively. Individual differences in the same cultural backgrounds, on the other hand, would be investigated for further consideration. Based on this discussion and previous studies, it can be assumed that institutions affect individuals' thinking and behavior, suggesting the following hypothesis.

#### **Hypothesis 1A.** *Institution is positively related to the perceived usefulness (PU) and the perceived ease of use (PEOU) of new technology in the fourth industrial revolution*.

As organizations and their members generally adapt to and operate in institutional arrangements (e.g., laws, orders, procedures, and instructions), institutions influence employees' behavior and define appropriate behavior in these practices [64,77]. As March and Olson ([83], p. 22) have argued, the institution defines the "logic of appropriateness of behavior" that clarifies appropriate behavior and the individual's role. Therefore, institutions regulate as a means of interlinking policy goals with individual actions [55]. Because institutions define responsibilities and goals, it is necessary for employees to accept institutional logic and applicable behaviors. This logical mechanism creates organizational and social conditions for the adoption of new technology in the workplace. The behavior of public employees can be more bounded by external institutions, such as laws and formal government procedures that consist of complex rule systems [84]. Thus, we attempted to develop an extended TAM model by adding institutional arrangement variables to further understand new technology adoption behavior of employees. Based on these arguments, we propose the following hypotheses to test the aforementioned logic.

#### **Hypothesis 1B.** *External institution is more positively related to the perceived usefulness (PU) and the perceived ease of use (PEOU) of new technology than internal institution.*

The perceived ease of use is the degree to which the person believes that using the particular system would be free of effort [23,54]. Davis [23] originally hypothesized that the user's attitude toward the information system was a major influential factor in whether the user would actually accept or reject the system. The attitude of the user was considered to be influenced by perceived usefulness and perceived ease of use. Subramanian [85] showed that perceived usefulness, not ease of use, is a main determinant of predicted future use of information systems. In addition, Chau [86] found that ease of use has the largest impact on specific software acceptance; perceived usefulness, not perceived ease of use, was found to be a significant determinant of attitude and intention for technology use. Igbaria et al. [57] indicated that perceived ease of use is a main factor in explaining perceived usefulness and system use. However, although perceived ease of use significantly predicts intention to use, the explanation power is secondary, following perceived usefulness [23].

#### **Hypothesis 2.** *Perceived usefulness (PU) is positively related to the intention to use new technology in the fourth industrial revolution.*

Davis [23] suggested that "ease of use operates through usefulness" ([23], p 332). According to Davis [54], perceived ease of use has a direct influence on perceived usefulness. Davis [23] demonstrated that perceived ease of use fully mediated the effects on use intentions of perceived output quality. Davis et al. [87] also explained that perceived ease of use generally affects IT adoption indirectly through its effect on perceived usefulness because perceived ease of use is instrumental in making new IT more useful. Legris et al. [88]'s meta-study reported that the correlation between perceived ease of use and use intention had a significant positive relationship in 16 of 28 studies. However, Gefen and Straub [89] comment that the role of perceived ease of use in TAM remains controversial in that some studies show that perceived ease of use does directly affect either self-reported use or intended IT use.

#### **Hypothesis 3A.** *Perceived ease of use (PEOU) is positively related to the intention to use new technology in the fourth industrial revolution*.

Davis [23] defined perceived usefulness as the degree to which a person believes that using the particular system would enhance her/his job performance. Igbaria et al. [57] and Abdullah et al. [90] showed that perceived usefulness is a variable that directly affects the intention to use an information system. When comparing the explanation of perceived usefulness and perceived ease of use, Davis et al. [23] reported that the perceived usefulness had 50% more impact than ease in determining information system use. Similarly, Keil et al. [91] demonstrated that perceived usefulness is a more important variable than ease of use in influencing system use. Legris et al. [88]'s meta-study showed the relationships between perceived ease of use and perceived usefulness demonstrated statistical significance in 21 of 28 studies.

#### **Hypothesis 3B.** *Perceived ease of use (PEOU) is positively related to perceived usefulness (PU).*

According to Venkatesh and Bala [46]'s TAM 3 study, perceived ease of use and perceived usefulness play mediating roles in the relationship between their determinants and outcome factors. In particular, since perceived ease of use and perceived usefulness have different antecedents, the roles of both are very different, and therefore, they have differentiation and uniqueness as a path of influence. Abdullah et al. [90] showed that perceived ease of use and perceived usefulness mediated the relationships between external variables and students' intention to use the e-portfolio. However, Beglaryan et al. [52] showed that perceived usefulness plays a mediating role between administrative monitoring and intention to use, but perceived ease of use does not perform such a function. They explained that this is because the perceived ease of use directly affects perceived usefulness and plays a different role as a variable.

#### **Hypothesis 4.** *Perceived ease of use (PEOU) and perceived usefulness (PU) positively mediate the relationship between institution and intention to use new technology in the fourth industrial revolution.*

The following equations represent the proposed hypotheses. The baseline model posits that the adoption of new technologies is a function of the institutional space that surrounds the organization as well as individual elements.

Based on these assumptions, the structural model was constructed to test the logic and our hypotheses. Based on these assumptions, the structural model was constructed to test the logic and our hypotheses.

its that the adoption of new technologies is a function of the institutional space that sur-

intention to use, but perceived ease of use does not perform such a function. They explained that this is because the perceived ease of use directly affects perceived usefulness

**Hypothesis 4**. *Perceived ease of use (PEOU) and perceived usefulness (PU) positively mediate the relationship between institution and intention to use new technology in the fourth industrial* 

$$\text{PEOU} = f(\text{institution}), \tag{1}$$

$$\text{PU} = f(\text{institution}, \text{PEOU}), \tag{2}$$

Intention to use = ƒ{institution, PEOU, PU, M(PEOU, PU)} (3) Intention to use = ƒ{institution, PEOU, PU, M(PEOU, PU)} (3)

where: where:

*revolution*.


*Int. J. Environ. Res. Public Health* **2021**, *18*, x FOR PEER REVIEW 10 of 37

rounds the organization as well as individual elements.

and plays a different role as a variable.

Figure 1 shows the conceptual model and proposed relationships between the variables. In Structural Equation Modelling (SEM), latent variables are not directly observed but are inferred by the covariation among a set of observed variables (also called reflective indicators). SEM combines factor analysis and regression, which provides far greater flexibility to the modeler than either of these two analysis methods. This is distinct from doing a factor analysis and then inputting the factor scores into a multiple regression. Figure 1 shows the conceptual model and proposed relationships between the variables. In Structural Equation Modelling (SEM), latent variables are not directly observed but are inferred by the covariation among a set of observed variables (also called reflective indicators). SEM combines factor analysis and regression, which provides far greater flexibility to the modeler than either of these two analysis methods. This is distinct from doing a factor analysis and then inputting the factor scores into a multiple regression.

**Figure 1.** Conceptual and hypothesized model for new technology adoption during the fourth industrial revolution.

The variables in ellipses represent latent constructs that are explained by the reflective indicators depicted with rectangles. In our proposed model, latent variables IU\_1 through IU\_5 stand for the items in the test for intention to use and e13 through e17 stand for measurement errors (unreliability) in each item. In addition, SEM considers measurement error by modeling it explicitly when estimating latent variables from indicators [92]. These error terms are depicted with circles.

At the center of the model lies the assumption that perceived ease of use and perceived usefulness have mediating roles that have an effect on attitudes and behavioral intentions. The model suggests that the indirect pathway, from institution to behavioral intention to use, is determined by a set of personal beliefs that influence usage behavior.

As the purpose of our study is to suggest a constructive artifact of digital informatics of human–computer interaction, it is worthwhile evaluating our proposal in terms of information system perspective [93,94]. Information system research should analyze the interplay between information technology policy, organizational architecture and contexts, and user behavior [94]. In this regard, our conceptual focus is on the interplay between organizationbased artifacts and technology adoption behavior. We seek to build a unified model for the development of design and behavioral science [94,95]. To evaluate the appropriateness of our proposal, we tried to apply the guidelines and principles of information system research because our study is an integrative attempt between behavioral science and design science research [95]. Hevner et al. [93] suggested the guidelines and recommendations that include design of an artifact, problem relevance, design evaluation, research contributions, research rigor, design as a search process, and communication of research. The guidelines and principles of information systems by Hevner et al. [93] and its adequacy to this study are outlined in Table 1. Although it is difficult to address all the guidelines [95], we tried to apply these criteria in our proposed model to identify the important considerations for developing a proactive framework while understanding the requirements in planning for new information technology in the fourth industrial revolution in various countries. However, even though our proposed model mostly fulfills these criteria, there are a few guidelines that are partially adequate. Thus, there are partially adequate fulfillments for our model according to Henver et al. [93]'s guidelines.


**Table 1.** Guidelines and principles of information systems by Hevner et al. [93] and its application in this study.

New technology in the fourth industrial revolution is highly based on interactions among individuals, technologies, organizations, and people. Therefore, it is critical to investigate the causal factors to adopt and use IS. In an IS, technology and behavior are not mutually exclusive [96,97]; research in IS is often categorized into two frameworks: behavior and design science [93]. Because our study is trying to find new digital informatics, the conceptual model is to examine the interconnection between system design factors (institution) and behavioral factors (intention to use with perceived ease of use and usefulness) to explain the adoption of new technology in the fourth industrial revolution. Our conceptual model particularly focuses on the organization and human-based artifacts that address relationships between institutions and technology adoption behavior in the fourth industrial revolution. Specifically, we introduce the concept of institutions as an enabler that can facilitate the usage behavior of new technology. It is critical to analyze institutional structures in operational processes and policy implementation because of the fundamental role of the institution as a driving factor for new technology adoption behavior. Since our model suggests a more extended IT adoption model that is broadly applicable in relevant settings, it can extend the scope of research on information system and design science. It can also contribute to top managers of organizations by suggesting technology-based solutions to solve business problems. In considering the guidelines and principles of information system perspectives, it can be concluded that the presented model and its implications will provide further explanation for IT adoption phenomena in the field of artifacts that solve important organizational problems.

#### **3. Data and Methods**

#### *3.1. Research Method*

TAM is widely applied across various fields in IS research. Many researchers have tried to provide an extension of TAM by integrating contextual and external factors. To offer a deeper explanation of technology adoption behavior, both empirical testing of the hypothesized model and investigating new influencing factors are needed. Qualitative research can provide in-depth understanding of IT adoption; however, exploring subjective experiences and subtle aspects of behaviors is a more effective way to learn the underlying reasons and motivations for IT adoption. This study focuses on the growing and recent phenomenon of TAM application in the latest industry techniques. Therefore, a quantitative research approach was chosen for this study. SEM was employed to examine our proposed hypotheses because it is the preferred method of testing a series of complex and multiple relationships constituting a set of an entire theory [92,98]. In SEM, the simultaneous effects of direct and indirect relations between variables are reflected. In addition, subjective variables such as people's attitudes are often inaccessible to direct measures. SEM ties multiple observed measures to the substantive latent variables and takes account of measurement errors in observed measures. SEM can appropriately reduce measurement errors of constructs by estimating error variance parameters for the latent variables. Therefore, it is a suitable method to test complicated causal relationships between human perceptions and attitudes measured by multiple indicators. Compared to common quantitative methods, such as correlation and regression, the strength of SEM is that researchers can specify an a priori relationship between variables [99,100]. Researchers can test whether hypothesized relationships derived from theory are reflected in the sample data [92]. In SEM, two approaches are widely used: covariance-based SEM (CB-SEM) and partial least squares SEM (PLS-SEM). CB-SEM adopts a common factor model approach, which calculates covariance between the variables, and only that variance is included in estimating the construct measures that can be explained by the common factor (one unobserved variable) and individual random error. In contrast, PLS-SEM utilizes all the variance of independent variables that explains the variance of dependent variables. CB-SEM is preferred for theory confirmation and theory testing, while PLS-SEM is primarily used for exploratory research and hypothesis development [101–104]. In addition, CB-SEM is recommended when the sample size is large (more than 100). In this study, the CB-SEM

method (specifically, maximum likelihood estimation) was chosen because our conceptual model was based on the established TAM theory and the sample size is 300.

The number of samples is a critical issue in SEM. Bentler and Yuan ([105], p. 181) explained that the most natural method for analyzing non-normal data, the asymptotically distribution-free procedure, is not defined when the sample size is less than the number of nonduplicated elements in the sample covariance. Bentler and Chou [106] believe that the number of cases needs to be five times the number of free parameters. On the other hand, Jöreskog and Sörbom [107] state that if the measured variable is less than 12, 200 cases are required, and if the number of measured variables is 12 or more, 1.5q (q + 1) equation is applied, where q represents the number of items that are used in the analysis. Mitchell [108] suggested the rule of thumb that there should be 10 to 20 times as many cases as variables. Similarly, according to Stevens [99], 15 cases per measured variable or indicator are needed. We added this discussion to the method section of the paper.

#### *3.2. Data*

To address the research questions and hypotheses, our study used a dataset from an open archive of the Korea Institute of Public Administration (KIPA). The KIPA is one of the leading public research organizations funded by the central government that mainly focuses on administrative developments and practices affecting public employees. Various surveys on policy and practical issues were exclusively conducted by KIPA to support government policymaking and implementation. Thus, datasets from the surveys are provided by formal request and review for further research [109]. According to the regulations of the KIPA on consent ownership and usage, it has been authorized for use in our research. The independent variables, mediating variables, dependent variables, and control variables used in this study were measured through the survey data on the perception of new technology in the fourth industrial revolution.

The survey into the state of new digital technology in the fourth industrial revolution was designed to identify primary factors that influence the adoption of new technologies in the public sector. The survey was conducted by KIPA through an online web page from 5 to 28 September 2018. Public employees who work in IT services in the central governments of South Korea were the target population of the survey. Random and cluster sampling methods were used for data collection. A total of 333 target respondents were randomly selected from employees of IT service divisions from 18 central government ministries, which are regulated by the Government Organization Act of South Korea. The candidates were contacted by phone prior to the survey to ensure that they agreed to participate in the survey. A website address was sent to them via e-mail and respondents logged onto the website to conduct the online survey; 300 respondents participated in the survey (90.09% of response rate). The total population size of the survey was 132,422 and our sample size was 300 with 95% confidence internal and 5.6% error. Determining an appropriate sample size for SEM is debatable where there is no prevailing consensus. Many guidelines have been suggested for the optimal sample size, such as the N: q rule [110] which uses the ratio of observations to estimated parameters, where N represents the number of cases and q represents the number of parameters that should be estimated. In this heuristic approach, the recommended sample-size-to-parameters ratio ranged from 5:1 to 20:1 [106].

It depends on not only the complexity of the model but also additional factors (e.g., the number of parameters, normality of the data). CB-SEM requires a larger sample size than PLS-SEM [92], and the median sample size is about 200 cases in studies where maximum likelihood estimation is used [111]. Barrett [112] also suggested that SEM analysis with a small sample (*N* < 200) may be problematic. Given these requirements, even though archival data was used, we concluded that 300 participants was appropriate to test the conceptual model.

The survey sample reflects the proportion of the total number of central government public employees and each proportion by ranking. Table 2 shows the characteristics of the respondents. The hierarchical position of respondents ranged from 3rd (managers) to 9th (lowest-level employees). Respondents included managers (grades 3–4), middle-managers (5–7), and team members (8–9, and others). In terms of gender, 71.7% of the respondents were male and 28.3% were female.


**Table 2.** Characteristics of the respondents.

#### *3.3. Measures*

The questionnaires used in this study were designed by KIPA [109]. The items were derived from validated measurements that were used and verified in previous TAM research. To ensure the construct and content validity of the measures, items selected for the variables in this study were adapted from the KIPA reports. Multiple items were used for all of the measures in order to improve reliability and validity.

*Institution.* Institution can be conceived as laws, norms, and systems that confine the way people select, implement, and use IT in public organizations [70,113]. Public officials are restrained by not only laws but also administrative rules, manuals, or plans that governmental organizations accept. Therefore, broader aspects of general rules and regulations should be adequately addressed in this study. In this sense, we used indicators of "institution" that were developed by KIPA to evaluate the level of administrative innovation using the fourth industrial revolution technology in government services. The concept of institution used in this study consists of six components shown in Figure 2. Respondents were asked to rate the degree to which they agreed or disagreed with the following statement: "There are laws/acts, ordinances, work guidance, manuals, plans/strategies, and work processes that can facilitate the use of the fourth industrial revolution technologies in the administrative service delivery."

Perceived ease of use related to computer self-efficacy, perception of external control, and computer playfulness [23,46] were measured using a five-point Likert scale. Perceived usefulness (PU) refers to subjective norm, image, job relevance, output quality, and result demonstrability [23,46]. Intention to use's measurement focused on purpose to use, motivation to use, and sustainable effort to use [47,51].

*Perceived ease of use*. Perceived ease of use was measured by seven items adapted from previous research by Venkatesh and Bala [46] who suggested TAM3. However, four items (computer anxiety, perceived enjoyment, objective usability, and compatibility) were excluded due to low factor loadings. As a result, the determinants of perceived ease of use in this study consisted of three items: computer self-efficacy, perception of external control, and computer playfulness. Computer self-efficacy refers to individuals' beliefs about their ability to use and control a computer system. Perception of external control refers to individuals' beliefs about sufficient organizational and technological support for facilitating and adopting new technology in their organization. Computer playfulness refers to intrinsic motivation and willingness to use any new system.

intrinsic motivation and willingness to use any new system.

*Institution.* Institution can be conceived as laws, norms, and systems that confine the way people select, implement, and use IT in public organizations [70,113]. Public officials are restrained by not only laws but also administrative rules, manuals, or plans that governmental organizations accept. Therefore, broader aspects of general rules and regulations should be adequately addressed in this study. In this sense, we used indicators of "institution" that were developed by KIPA to evaluate the level of administrative innovation using the fourth industrial revolution technology in government services. The concept of institution used in this study consists of six components shown in Figure 2. Respondents were asked to rate the degree to which they agreed or disagreed with the following statement: "There are laws/acts, ordinances, work guidance, manuals, plans/strategies, and work processes that can facilitate the use of the fourth industrial revolution

Perceived ease of use related to computer self-efficacy, perception of external control, and computer playfulness [23,46] were measured using a five-point Likert scale. Perceived usefulness (PU) refers to subjective norm, image, job relevance, output quality, and result demonstrability [23,46]. Intention to use's measurement focused on purpose to use, moti-

*Perceived ease of use*. Perceived ease of use was measured by seven items adapted from previous research by Venkatesh and Bala [46] who suggested TAM3. However, four items (computer anxiety, perceived enjoyment, objective usability, and compatibility) were excluded due to low factor loadings. As a result, the determinants of perceived ease of use in this study consisted of three items: computer self-efficacy, perception of external control, and computer playfulness. Computer self-efficacy refers to individuals' beliefs about their ability to use and control a computer system. Perception of external control refers to

**Figure 2.** The measurement and components of institution. **Figure 2.** The measurement and components of institution.

technologies in the administrative service delivery."

vation to use, and sustainable effort to use [47,51].

*Perceived usefulness.* Perceived usefulness (PU) was measured by five items adapted from the study by Venkatesh and Bala [46]. They suggested five types of determinants of PU: subjective norm, image, job relevance, output quality, and result demonstrability. Subjective norm refers to "the degree to which an individual perceives that people who are important to him/her think one should or should not use the technology" ([46], p. 277). Image refers to "an enhancement of one's social system" ([114], p. 195). Job relevance refers to the applicability of technology in one's work. Output quality refers to "the degree to which a person believes that the technology performs a job's tasks well" ([24], p. 191). *Perceived usefulness.* Perceived usefulness (PU) was measured by five items adapted from the study by Venkatesh and Bala [46]. They suggested five types of determinants of PU: subjective norm, image, job relevance, output quality, and result demonstrability. Subjective norm refers to "the degree to which an individual perceives that people who are important to him/her think one should or should not use the technology" ([46], p. 277). Image refers to "an enhancement of one's social system" ([114], p. 195). Job relevance refers to the applicability of technology in one's work. Output quality refers to "the degree to which a person believes that the technology performs a job's tasks well" ([24], p. 191). Result demonstrability refers to a tangible, observable, and communicable improvement of work outcome caused by the use of technology.

*Intention to use*. Behavioral intention predicts and facilitates actual usage behavior. Respondents were asked questions about their attitude toward the use of new technologies (such as artificial intelligence, internet of things, and big data). Intention to use was measured using three items that refer to purpose and motivation to use new technologies and intention to put effort into using new technologies.

#### *3.4. Analysis Method: Structural Equation Modeling*

We use SEM as main analysis method. In terms of the statistical approach, SEM has good advantages for testing a series of complex and simultaneous relationships. SEM is a multivariate technique used to test all coefficients (relationships) in the complete model simultaneously. Simple ordinary least squares (OLS) regression cannot assess the significance of particular relationships between variables when moderating effects are hypothesized. In SEM, path coefficients (also called connection strengths) are parameters of the structural model and are estimated through a series of algebraic manipulations. The maximum likelihood fit function is used to calculate the path coefficients which are iteratively modified [107]. The path coefficient represents the change of the dependent (responding) variable according to a unit change of explanatory variable when all other variables in the model remain constant. The path coefficients are similar to b (unstandardized coefficient) or beta (β, standardized coefficient) in the regression model. The standard error (SE) of the coefficient (or standard deviation of the estimate) is taken into account in order to indicate how the estimated sample means precise an unknown mean of a population. This allows researchers to calculate a confidence interval that provides the range of observed effect size or *p*-value that assesses whether the estimates are significantly different from some reference value.

The purpose of SEM is to specify a model derived from theory and the estimation of the parameters of the model is derived from the conventional approach of SEM. The overall objective of the conventional approach is to connect the theory and specification of the model. Within this approach, exploratory factor analysis and confirmatory factor analysis are performed to test the proposed model. At this stage, the measurement model is estimated and the structural model can be estimated later. The full model can be structured through a continuous model modification and evaluation of goodness-of-fit until the model meets the criterion of adequate fit. This is discussed in more detail in the data and methods chapter. The meaning and interpretation of parameters in the SEM are outlined in Table 3.

**Statistics Meaning Interpretation** *S.E* (Standard error) The standard error means an estimate of the standard deviation (S.D) of the coefficient. The standard error allows us to identify the magnitude of error which is made in estimating an outcome variable from an independent variable. Estimate (β, standardized coefficient) The standardized coefficient is calculated by multiplying the unstandardized coefficient by the ratio of standard deviation of explanatory variable and outcome variable. Each of the estimated parameters represents the amount of change in the dependent variable as a function of a single unit change in the explanatory variable. *AVE* (Average Variance Extracted) AVE is the level of variance that is captured by a construct compared to the level of variance. The suggested threshold that is normally higher than 0.50 would be acceptable. *R* 2 (R-Square) As the independent variables are correlated in SEM, the R<sup>2</sup> of each estimate indicates the partial effect of each variable on the dependent variable. The R<sup>2</sup> of the structural model can be interpreted as a proportion of variance explained. The full structural model relationships between latent variables and direct variables have the highest partial coefficient. *CFI* (The Comparative Fit Index) CFI is an incremental relative fit index that measures the relative improvement in the fit of the researcher's model. CFI is a revised form of NFI. It ranged from 0 to 1. The recommended threshold is 0.9 or more. *GFI* (The Goodness of Fit) GFI is the proportion of variance accounted for by the estimated population covariance. GFI indicates the proportion of variance explained by the estimated population covariance. The recommended threshold is 0.9 or more. *NFI* (The Normed-Fit Index)/ *TLI* (Tucker–Lewis Index) NFI indicates whether the proposed model improves the fit compared to the null model. TLI (also called non-Normed-Fit) is preferable for a small sample. NFI ranges between 0 and 1. The recommended threshold for NFI is 0.9 or more. *SRMR* (Standardized Root Mean Square Residual) SRMR is the standardized difference between the residuals of the observed sample covariance matrix and the predicted hypothesized model. SRMR ranges between 0 and 1. The recommended threshold for SRMR is 0.08 or less. *RMSEA* (Root Mean Square Error of Approximation) RMSEA represents a parsimony-adjusted index. RMSEA ranges from 0 to 1. Values closer to 0 represent a good fit. The recommended threshold is 0.08 or less.

**Table 3.** The meanings and interpretations of statistics in the SEM.

#### **4. Results**

SEM consists of two powerful statistical approaches: exploratory factor analysis (EFA) and structural modeling where structural path analysis is applied. EFA is used to identify the number of hypothetical factors or latent variables that can be explained by covariance among a set of observed variables. Therefore, a priori specification of the number of variables is not required in EFA [111,115]. Instead, the structural model (also referred to as confirmatory factor analysis (CFA)) is designed to confirm priori conceptualized and hypothesized models; therefore, the exact number of variables should be specified in order to conduct CFA [116]. First, as a primary test, we conducted Harman's single factor test by loading all variables onto a single factor without a rotation using SPSS. 12.0 [117]. Harman's single factor test is a widespread statistical technique to identify common method bias. It is designed to test whether one component will explain more than 50% of the covariance among the measures. If common method variance exists, one factor accounts for the majority of the covariance between the variables. Results showed that a common latent factor explained 30.24% of the covariance, which suggested that there was no common methods bias.

The construct validity of measures was investigated in two steps: EFA and CFA. The characteristics of these two steps are outlined in Table 4. EFA based on principal component

analysis with varimax rotation was conducted. Exploratory factor analysis, also referred to as the unrestricted factor model, is a multivariate statistical method used to uncover the underlying structure of measures and identify latent variables [92,111,115]. Common factors among measured variables are identified while the structure and correlations among these observed variables are explained. In EFA, varimax rotations are used to enhance the interpretability of retained factors and clarify the relationship among factors. In this step, potential factors were affirmed on theoretical backgrounds and the basic construct was developed according to the following results. Items with lower factor loadings were extracted (the lowest factor loading was 0.588) and a total of 17 items were selected for the measurement model. In EFA, all observed measures depend on all factors, which were not specified by researchers beforehand; thus, EFA may generate several possible structures and models, from a one-factor model to multiple-factor model [11,111,115,116]. As a result, four factors were derived and the cumulative percentage of variance explained by them was 65. The Bartlett's test of sphericity was performed to identify whether the variances were equal for all samples. The chi-squared value for the test was high and significant (approximate χ <sup>2</sup> = 2498.853, *p* < 0.0001). The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was 0.849, rejecting the null hypothesis that there was no difference in variances between the groups [107].


**Table 4.** The characteristics of explanatory factor analysis (EFA) and confirmatory factor analysis (CFA).

#### *4.1. Measurement Model*

We followed the two-step procedure proposed by Anderson and Gerbing [119], which consists of a measurement model and a structural model. In this stage, after obtaining initial instruments, we developed a measurement model and performed a CFA using AMOS 18.0. In CFA, each variable is derived from the factors specified by EFA and the researcher's theoretical expectation; that is, restricted measurement models are identified and analyzed [111]. SEM allows researchers to evaluate relationships among variables comprehensively by providing a transition from EFA to CFA. The restricted model reflects a priori hypotheses and knowledge of the theory, which in turn allows researchers to test the relationship between observed variables and underlying latent constructs. Six common model-fit indicators were used to assess the goodness-of-fit of the measurement model. As presented in Table 5, all fit indices exceeded the recommended threshold. The comparative fit index (CFI), goodness of fit index (GFI), normed fit index (NFI), and Tucker–Lewis index (TLI) were all above 0.90. The standardized root mean residual (SRMR) was less than 0.08 [107], and the root mean square error of approximation (RMSEA) was less than 0.08 [120].


**Table 5.** Fit indices for the measurement and structural models.

Contrary to EFA, the exact number of factors should always be defined in CFA before analysis. However, CFA does not necessarily confirm the initial restricted model that fits the data. In this case, the modified hypotheses and specified models should be proposed as alternative models [111]. In this study, CFA confirmed the hypothesized restricted model and initial constructs derived from EFA. The results of the CFA are presented in Table 6. All selected items had an acceptable value of factor loadings above 0.5 [120,121]. In addition, average variance extracted (AVE) was calculated in order to assess discriminant validity. AVE shows the level of variance that is captured by a construct compared to the level of variance due to measurement error. The values of AVE of all four factors were greater than the suggested threshold of 0.5 [112]. Overall, the measurement model indicated reasonable and acceptable convergent validity and discriminant validity.


**Table 6.** Results of confirmatory factor analysis (CFA).

Note: PU: perceived usefulness; PEOU: perceived ease of use; IU: intention to use.

#### *4.2. Reliability Analysis and Correlations*

Table 7 summarizes the descriptive statistics, the measure of scale reliability (Cronbach's alpha), and the result of correlation analysis between the variables. The values of Cronbach's alpha for all the factors were above the criterion of 0.70 [121], which means that scale items had high internal consistency. In addition, we calculated Pearson's correlation coefficient (*r*) in order to evaluate the association between the variables. As shown in Table 7, correlations between the factors were statistically significant and showed predicted

directions in our hypotheses. The dependent variable, intention to use, was positively related to institution (*r* = 0.276, *p* < 0.01), perceived usefulness (*r* = 0.551, *p* < 0.01), and perceived ease of use (*r* = 0.557, *p* < 0.01).

**Table 7.** Descriptive statistics, reliabilities, and correlations.


Note: \*\* *p* < 0.01, PU: perceived usefulness; PEOU: perceived ease of use; IU: intention to use.

#### *4.3. Hypothesis Tests*

#### 4.3.1. Structural Model

The structural model was identified in order to test the hypotheses. Beginning with the hypothesis tests, multiple fit statistics were examined for model fit. As shown in Table 5, the results showed a reasonably good model fit, supporting the conceptual model we suggested (CFI = 0.957; GFI = 0.924; NFI = 0.917; SRMR = 0.058; and RMSEA = 0.056). Thus, we could proceed to an analysis of the path coefficients. First, we focused on the standardized direct effect of predictor variables on the response variables in the model. As Figure 3 outlines, overall results were in the predicted directions in accordance with the aforementioned hypothesis. Results showed that institutions had a positive and significant effect on perceived ease of use (β = 0.327, *p* < 0.001) and perceived usefulness (β = 0.142, *p* < 0.05), in support of Hypothesis 1A. This suggests that institutions can play an important role in the adoption and diffusion of new technology, supporting previous studies that focused on the institutional perspective [122,123]. Institution explained 10.7% of the variance in perceived ease of use (squared multiple correlations or *R* <sup>2</sup> = 0.107). In terms of Hypothesis 2, the effect of perceived usefulness on intention to use was positive and significant (β = 0.329, *p* < 0.001). The results also showed that perceived ease of use had a strong and positive impact on intention to use (β = 0.547, *p* < 0.001), supporting Hypothesis 3A. The results indicate that those who perceive that the fourth industrial revolution technology is useful and simple are more likely to have a higher intention to use it. The path coefficient between perceived ease of use and perceived usefulness was positive and statistically significant (β = 0.554, *p* < 0.001), in support of Hypothesis 3B. In terms of perceived usefulness, 37.8% of variance in perceived usefulness was explained by predictor variables (perceived ease of use and institution). *Int. J. Environ. Res. Public Health* **2021**, *18*, x FOR PEER REVIEW 21 of 37 and perceived ease of use. We focused particularly on the impact of institutions and lend support to the positive relations in TAM paths. The summary of the hypothesis test results is presented in Table 8 and Figure 3. **Table 8.** Summary of the hypothesis test results (standardized direct effect). **Path Estimate Hypotheses Test Results** *R***<sup>2</sup>** Institution → PEOU 0.327 (*p* = 0.000) H 1\_1 Supported 0.107 Institution PEOU → → PU PU 0.142 (*p* = 0.020) 0.554 (*p* = 0.000) H 1\_1 H3\_2 Supported Supported 0.378 PU PEOU → → IU IU 0.329 (*p* = 0.000) 0.547 (*p* = 0.000) H2 H3\_1 Supported Supported 0.624 Note: PU: perceived usefulness; PEOU: perceived ease of use; IU: intention to use.

**Figure 3.** Results of the baseline model. Note: *N* = 300. \* *p* < 0.05, \*\*\* *p* < 0.001. **Figure 3.** Results of the baseline model. Note: *N* = 300. \* *p* < 0.05, \*\*\* *p* < 0.001.

In order to explore more specific institutional settings and to further establish the

By adopting externally legitimized formal frameworks and internal administrative process, organizations may strengthen the engagement of their internal members in change and innovations [125]. It is noted that the adoption of new technology frequently implements similar practices in the environment without clear usefulness [126]. In this respect, the diffusion of change within the organization is influenced by external forces; formation and establishment are influenced by internal forces [127]. Institutions enforce diffusion from the top down, while individuals are immersed in innovation from the bottom up. Considering this perspective, it can be assumed that institutional features might influence new technology adoption differently in the fourth industrial revolution. Thus, we are trying to examine the influence of specific settings in institutions. Institutional change often arises in the integration of top-down and bottom-up frameworks; it requires

An attempt to compare alternative models lets us not only review the variety of institutional systems but also determine the superiority of the conceptual modes. These two dimensions generated two-factor institutional models illustrated in Figure 4. External institutions (laws, acts, and ordinances) are enacted by legislators who can exercise political control over the operation of public organizations. Regarding path dependence and feed-

synthetic efforts toward technological change and innovation [128].

enforced by entities inside or outside the organization.

SEM also allows one to examine the total effect of each independent variable on any dependent variable. Total effects are calculated through the decomposition of effects, where the direct path coefficient between the two variables and the indirect effect between the two through other mediating variables are summed [124]. Total effects consider the mediating effects of the intervening variables that might potentially influence the direct effect. In doing so, total effects can provide an understanding of which factors are more important in determining the level of dependent variable. The indirect effect of institutions on intention to use through perceived ease of use and perceived usefulness was tested using the bootstrapping method. The indirect effect was positive and statistically significant (β = 0.116, *p* < 0.01) in support of Hypothesis 4. In addition, the total effect of institution on (Institution→ perceived ease of use→ intention to use) and (Institution→ perceived ease of use→ perceived usefulness→ intention to use) was positive and significant (β = 0.267, *p* < 0.01) (Table 7). The result is not surprising given the body of work in technology studies that asserted intention to use is shaped both by perceived usefulness and perceived ease of use. We focused particularly on the impact of institutions and lend support to the positive relations in TAM paths. The summary of the hypothesis test results is presented in Table 8 and Figure 3.

**Table 8.** Summary of the hypothesis test results (standardized direct effect).


Note: PU: perceived usefulness; PEOU: perceived ease of use; IU: intention to use.

In order to explore more specific institutional settings and to further establish the measure of institution, we additionally adopted two-factor institutional models. The focus of the alternative models is the dimensions of institutions: whether they are created and enforced by entities inside or outside the organization.

By adopting externally legitimized formal frameworks and internal administrative process, organizations may strengthen the engagement of their internal members in change and innovations [125]. It is noted that the adoption of new technology frequently implements similar practices in the environment without clear usefulness [126]. In this respect, the diffusion of change within the organization is influenced by external forces; formation and establishment are influenced by internal forces [127]. Institutions enforce diffusion from the top down, while individuals are immersed in innovation from the bottom up. Considering this perspective, it can be assumed that institutional features might influence new technology adoption differently in the fourth industrial revolution. Thus, we are trying to examine the influence of specific settings in institutions. Institutional change often arises in the integration of top-down and bottom-up frameworks; it requires synthetic efforts toward technological change and innovation [128].

An attempt to compare alternative models lets us not only review the variety of institutional systems but also determine the superiority of the conceptual modes. These two dimensions generated two-factor institutional models illustrated in Figure 4. External institutions (laws, acts, and ordinances) are enacted by legislators who can exercise political control over the operation of public organizations. Regarding path dependence and feedback frameworks, external institutions have a strong effect on early moments in new technology adoption. Because of their strong effect, these external institutions might bring inflexibility and rigidness in utilization of new technology: just follow the paths [129].

**Figure 4.** Results of the external institution model. Note: *N* = 300. \*\*\* *p* < 0.001. **Figure 4.** Results of the external institution model. Note: *N* = 300. \*\*\* *p* < 0.001.

Internal institutions (work guidance, manuals, and plans/strategies) are implemented by the top of the bureaucratic hierarchy as a means of agency management. As the interrelations between organization and institution are complicated and diverse, actors in organizations tend to choose institutional suboptimal solutions or arrangements to reduce uncertainty and ambiguity in the implementation process [130]. Therefore, we tried to examine external and internal institutional influence on the intention to use new technology in the fourth industrial revolution.

**Figure 5.** Results of the internal institution model. Note: *N* = 300. \* *p* < 0.05, \*\*\* *p* < 0.001. Our results showed that the directions of path coefficients adhere to the general TAM hypotheses. In terms of model fit, the external institution model showed adequate fit (CFI = 0.975, NFI = 0.942, SRMR = 0.043, RMSEA = 0.047) and the internal institution model also showed adequate fit (CFI = 0.940, NFI = 0.903, SRMR = 0.052, RMSEA = 0.069). In the external institution model (Figure 4), external institutions had a positive and significant effect on perceived ease of use, and the standardized total effect on intention to use was 0.291 (*p* < 0.01). Regarding the internal institution model (Figure 5), perceived ease of use and perceived usefulness had positive and significant relationships with internal institutions. However, the internal institution explained less variance of dependent variables than did the external institution. The standardized total effect of internal institution on intention to use was 0.210 (*p* < 0.01). The findings may indicate that institutional arrangements made by external entities can be more powerful than managerial actions when adopting new technologies in the public sector, therefore confirming our Hypothesis 1B.

4.3.2. Alternative Expanded Model In SEM studies, the standard procedures provide a baseline model based on general theory and test a modified model based on possible relationships between other indicators [99]. During the process, theoretical support and key conceptual assumptions of the model should be maintained. Researchers can change paths or variables and observe the coefficient and model fit changes compared to other models. This can be done by adding new variables or removing problematic latent variables. All steps need to be backed up with Regarding the characteristics of organizational structure, it should be considered that there are possible differences in institutional influences between organizations with high and low hierarchical levels. Because our study was based on a high hierarchical organization (central government), it is worthwhile to discuss possible differences when we apply our two-factor institutional models to a low hierarchical organization (for example, a public expert organization). Regarding organizational members' tendency to reduce uncertainty in the decision-making process [131], the members in low hierarchical organization might show the same results of institutional influences when adopting new technologies.

logical reasoning and theory. After several iterations of paths and variables, we decided to use the alternative expanded model, explained in Figure 6. However, in terms of an organization's complexity, centralization, and formalization, it would be considered that the influence of internal institutions would have a strong effect on new technology adoption in low hierarchical organization [132,133]. Because a low hierarchical organization has fewer managerial layers, organizational members could easily participate in the decision-making process, especially when making internal institutions like manuals and operational standards. Therefore, when adopting new technologies, internal institutions would show strong influence unlike high hierarchical organizations. Furthermore, hierarchical levels represent the system of authority for coordinating tasks and ordering the specialization of functions. These systems use varied amounts of hierarchy and control to shape behaviors; decisions are made from above and organizational members mainly receive orders. Therefore, when adopting new technologies, external institutions have strong impacts with well-defined links in the chain of command in high hierarchical organizations. Meanwhile, low hierarchical organizations have relatively few links; they might more focus on their internal institutions than external [134]. Therefore, it is possible to assume that the result of two-factor institution models in our study might be different if we apply this model in low hierarchical organization. **Figure 4.** Results of the external institution model. Note: *N* = 300. \*\*\* *p* < 0.001.

*Int. J. Environ. Res. Public Health* **2021**, *18*, x FOR PEER REVIEW 23 of 37

**Figure 5.** Results of the internal institution model. Note: *N* = 300. \* *p* < 0.05, \*\*\* *p* < 0.001. **Figure 5.** Results of the internal institution model. Note: *N* = 300. \* *p* < 0.05, \*\*\* *p* < 0.001.

#### 4.3.2. Alternative Expanded Model

4.3.2. Alternative Expanded Model In SEM studies, the standard procedures provide a baseline model based on general theory and test a modified model based on possible relationships between other indicators [99]. During the process, theoretical support and key conceptual assumptions of the model should be maintained. Researchers can change paths or variables and observe the coefficient and model fit changes compared to other models. This can be done by adding new variables or removing problematic latent variables. All steps need to be backed up with logical reasoning and theory. After several iterations of paths and variables, we decided In SEM studies, the standard procedures provide a baseline model based on general theory and test a modified model based on possible relationships between other indicators [99]. During the process, theoretical support and key conceptual assumptions of the model should be maintained. Researchers can change paths or variables and observe the coefficient and model fit changes compared to other models. This can be done by adding new variables or removing problematic latent variables. All steps need to be backed up with logical reasoning and theory. After several iterations of paths and variables, we decided to use the alternative expanded model, explained in Figure 6.

to use the alternative expanded model, explained in Figure 6. In doing so, we tried to give a more comprehensive explanation of complex human behaviors and attitudes that are constrained by various conditions and circumstances. The purpose of examining an additional alternative expanded model becomes clear when considering the fundamental nature of technology diffusion and adoption in organizations. Technology adoption and diffusion occur as a continuous process based on a series of individual decisions that are mostly calculated by the incremental benefits of using the technology. In addition, technology adoption is usually requested by a top-down management and often seen as an investment decision made by managers [70]. Researchers suggested that technology adoption occurs in stages according to individual differences. It is often modified by environmental conditions and uncertainty [71].

0.001.

*Int. J. Environ. Res. Public Health* **2021**, *18*, x FOR PEER REVIEW 24 of 37

**Figure 6.** Alternative expanded model for the acceptance of new information technology. Note: *N* = 300. \*\* *p* < 0.01, \*\*\* *p* < **Figure 6.** Alternative expanded model for the acceptance of new information technology. Note: *N* = 300. \*\* *p* < 0.01, \*\*\* *p* < 0.001.

In doing so, we tried to give a more comprehensive explanation of complex human behaviors and attitudes that are constrained by various conditions and circumstances. The purpose of examining an additional alternative expanded model becomes clear when considering the fundamental nature of technology diffusion and adoption in organizations. Technology adoption and diffusion occur as a continuous process based on a series of individual decisions that are mostly calculated by the incremental benefits of using the technology. In addition, technology adoption is usually requested by a top-down management and often seen as an investment decision made by managers [70]. Researchers suggested that technology adoption occurs in stages according to individual differences. It is often modified by environmental conditions and uncertainty [71]. Therefore, the alternative expanded model posits that contextual factors may have direct and indirect effects on intention to use. In order to give a more comprehensive explanation of complex new technology phenomena, we constructed an alternative expanded model and tested non-mediated models. The proposed baseline model only considered causal path relationships of institutions, perceived ease of use, and perceived usefulness to intention to use of new technology in the fourth industrial revolution. To exam-Therefore, the alternative expanded model posits that contextual factors may have direct and indirect effects on intention to use. In order to give a more comprehensive explanation of complex new technology phenomena, we constructed an alternative expanded model and tested non-mediated models. The proposed baseline model only considered causal path relationships of institutions, perceived ease of use, and perceived usefulness to intention to use of new technology in the fourth industrial revolution. To examine the influence of individual factors, we included intrinsic factors (experience, voluntariness), organizational factors (culture, security), and task-related factors (specialty, manager's concern) [46,134]. Among individual intrinsic factors, experience might be important for adopting new technology [135]. New ways of collecting and sharing digital information have already been established by digital-based ubiquitous technologies, users who are accustomed to these trends might easily accept new technology. Thus, experience may be important for adopting new technology [136]. The degree to which prospective users perceive their decision is voluntary is important as it can affect when they adopt new technologies [56]. When technology adoption is organizationally imperative, user motivations vary and certain users restrain such type of mandate [128]. In this respect, voluntariness may affect adoptive behavior of new technology.

ine the influence of individual factors, we included intrinsic factors (experience, voluntariness), organizational factors (culture, security), and task-related factors (specialty, manager's concern) [46,134]. Among individual intrinsic factors, experience might be important for adopting new technology [135]. New ways of collecting and sharing digital information have already been established by digital-based ubiquitous technologies, users who are accustomed to these trends might easily accept new technology. Thus, experience may be important for adopting new technology [136]. The degree to which prospective When introducing new technologies, the ability for individuals to adapt and modify their patterns is challenged. In this respect, a strong cultural foundation will facilitate the acceptance and success of new technologies [14,137]. Hence, culture can have an effect on new technology adoption. Because of the importance of new technology for social and economic growth, as well as an increasing focus on data collection, privacy has become a public concern [138]. It is critical to find the right balance between technology deployment and data security with user privacy, especially in the public domain [139]. Consequently, privacy can have an effect on new technology adoption.

users perceive their decision is voluntary is important as it can affect when they adopt new technologies [56]. When technology adoption is organizationally imperative, user motivations vary and certain users restrain such type of mandate [128]. In this respect, voluntariness may affect adoptive behavior of new technology. New technology in the fourth industrial revolution might influence the types of job tasks required of individuals, which health professionals during the current pandemic have experienced [140]. Thus, we tried to include task-related factors (specialty, manager's

concern) in the alternative expanded model. In recent technological advances, including social networking, smartphones, sensors, and embedded systems, different attitudes and skills have been required [141], which has added to the burden of some employees. Consequently, specialty might have an effect on new technology adoption. Regarding managers' organizational role, manager behavior has been discussed as one of the most significant factors in adopting new technologies [142]. Managers can assist in the technology adoption process by advising all users in the organization [143]. In this respect, managers' concern will impact the adoption of new technology in the fourth industrial revolution.

Therefore, we examined the possible impact of individual features (experience, voluntariness, and specialty) and organizational conditions (culture, security, and managers' concern). Individual experience was measured by "I have experienced public services provided by new technologies in the fourth industrial revolution (such as artificial intelligence, internet of things, and big data)". Voluntariness was measured by "I think it is important to foster public service innovation using new technologies in the fourth industrial revolution". Specialty was measured by "I have job specialties so that I can adopt new technologies in the fourth industrial revolution". Organizational conditions were measured by the following: "My organization is trying to foster a culture of new technology in the fourth industrial revolution utilization", "My organization provides technical security measures to protect IT-enabled public services", and "Managers in my organization try to foster IT-enabled public services innovation."

Our approach to the expanded modification model with individual and organizational factors can contribute to the development of information system research. From the perspective of "good design science" [93], studies must be presented effectively to both technology-oriented and management-oriented audiences. In this regard, results must be addressed to both the technology scholars and management communities with adequate objectivity and significance. Following this standard, a modified model can provide a thorough discussion of the behavioral processes for new technology adoption and managerial actions that are necessary. Our study results can provide useful solutions for managers and employees when they face obstacles to adopting new technology.

We tested whether significant improvement in model fit was made when adding new variables [92]. In order to investigate the differential effects of each condition and to compare model fits among the proposed models, changes of path and variables were added in stages from Models 1 to 4. In the first modification, a direct relationship between institution and intention to use was presumed (Submodel 1). The second modification was made to the structural model by adding individual factors (Submodel 2). The third modification was made by adding organizational factors to the baseline model (Submodel 3). In the final modification in Model 4, both individual and organizational factors were added and tested. Overall model fit indices are presented in Table 9.


**Table 9.** Comparison of the goodness-of-fit.

Note: The dependent variable is intention to use. Bootstrap, bias-corrected two-tailed tests used to calculate significance of the total effects. *N* = 300.

Generally, the proposed alternative models were acceptable; however, the model fits were not greatly improved over the baseline model. Thus, we concluded that alternative models may not be preferable to the baseline model. Although alternative models did not appear superior in terms of model fits, it should be noted that more conditional variables were considered and the simultaneous effects were examined.

Beyond these model fit results, significant path coefficients were found in most of the expected directions and the results supported the verified causal relationships of the baseline model. Table 9 represents the total effects of each variable on intention to use and Figure 6 only represents direct effects between the variables. The total effect is combined with direct and indirect effects. In SEM, the pathway from the exogenous latent variable to endogenous outcome variable through the mediator calculates indirect or mediating effect. As shown in Table 10, the significant and positive total effects of institution and perceived ease of use were maintained in all of the alternative models. More importantly, coefficients of both variables were raised incrementally as alternative factors were added, thus confirming the positive relation between institution and technology adoption behavior. This also suggests that the impacts of institution and perceived ease of use on intention to use remains dominant when other individual and work conditions are controlled. In addition, the direct effect of institution on intention to use did not approach significance in Figure 6; however, the total effect was significantly mediated by perceived ease of use and perceived usefulness. This suggests institutions can affect intention to use only through mediating effects of perceived ease of use and perceived usefulness. Again, perceived ease of use and perceived usefulness had full mediating effects on the relationship between institution and intention to use. Perceived ease of use also had a significant and positive effect on intention to use in Submodel 2 and Submodel 3; however, significance was not found in the full Submodel 4. Of the three predictors, the improvement of the coefficient of perceived ease of use was noticeable compared to other variables, indicating that perceived ease of use is relatively more important in determining an outcome. Thus, it may be concluded that institutions strengthen the level of intention to use mainly through perceived ease of use. The summary of the hypotheses test results are as follows.


**Table 10.** Path coefficients for the alternative models (standardized total effects).

Note: The dependent variable is intention to use. Bootstrap, bias-corrected two-tailed tests used to calculate significance of the total effects. *N* = 300. \* < 0.05, \*\* *p* < 0.01.

As with any previous TAM studies, the effects of both perceived ease of use and perceived usefulness on intention to use were positive and significant in the basic parsimonious model. Furthermore, institutions significantly affect perceived ease of use and perceived usefulness. Institutions have a significant and strong indirect influence on intention to use, which would lead one to expect that institutional factors may have a crucial role in new technology acceptance of public employees in South Korean central government agencies. The test results of the alternative models lend support to the important role of institutions as an external facilitating factor, thus confirming Hypothesis 1. While the direct effects of institutions on perceived usefulness did not show significance in submodel 4, its total influence on intention to use was still reinforced by mediating variables. In terms of the total effects of contextual variables, specialty and security showed a negative and significant effect on intention to use in submodel 2 and 3, respectively. However, direct effects were not significant (Figure 6). In submodel 4, only specialty had a significant and negative effect on institution (total effect) and the direct effect was also significant as shown in Figure 6.

#### **5. Discussion and Implication**

#### *5.1. Discussing the Main Results*

Our study aimed to analyze the links between the impact of institutions, perceived ease of use, perceived usefulness, and intention to use of new technology adoption in the fourth industrial revolution [11,12,20,31,144]. By using individual-level data from a survey of public employees in the Korean central government conducted in 2018, this study empirically examined the important factors for adopting new technology. The understanding and recognition of public employees engaging in new emerging technology is important because of their role in decision-making and value creation in collective issues like the environment and public health [145,146].

The contributions of our study to the healthcare field are suggested as follows: First, the usage of digital technology and developments in the healthcare field are expected to increase significantly following COVID-19. Regarding the role of information systems in COVID-19, this study implies that institutional environments should be considered when initiating digital health related policies or projects. After tracking websites' digital communication strategies in Latin America hospitals during the COVID-19 pandemic, Tejedor et al. [147] suggested that digital media could constitute legitimate resources for healthcare information consumption, so their accuracy and proper development seem to be significant to becoming a genuine source. In addition, after studying Electronic Health Record (EHR), Beglaryan et al. [52] indicated that institutional factors as well as personal benefits at the micro level should be considered. In particular, they suggest that the critical role of social factors, like institutions, hinders success of informatization at the organizational level. Moreover, Ahmad et al. [148] found that social influence had a positive impact on patients' continued intention to use digital health wearables. These studies suggest not only utility-based managerial factors but also social structural ones should be considered in investigating new technology adoption. While it is important to emphasize the convenience and benefits of operationalization of new technologies, it is also important to construct basic institutional arrangements for their utilization. The usage of information technology for pandemic management is still a long way from achieving an integrated system focused on effective cooperation between humans and machines for disease prevention and treatment [28]. As legal frameworks have been critical to more promptly responding to the COVID-19 crisis, the need for more comprehensive regional and national disease registries is required. It is also emphasized that coordination between agencies and organizations is required for successful testing and quarantine for individuals with COVID symptoms using new technology such as contact tracing [139]. However, without clear guidance and recommendations, successful coordination among stakeholders is difficult to achieve. Therefore, the results of this study suggest that institutional adjustments are required to promote the acceptance of new technology, especially during crisis like COVID-19.

Second, our findings suggest that institutions had an indirect impact on the intention to use new technology of the fourth industrial revolution. It is important to consider what factors affect government employees' intentions to use new technology as they introduce and organize various collective strategies on public issues such as pandemics. The institutions do not ensure that the actual application of emerging technologies can occur without the mediation of presumed ease and usefulness. As a result, when we introduce new technologies, we must understand how people actually perceive the ease of use and usefulness of such technologies. In particular, the COVID-19 pandemic involved huge demands that necessitated the collection and analysis of digital health data using new technology. As a result, the role of government employees is critical. Thus, it is worthwhile to investigate the factors of new technology adoption, especially when it is intended to be used by public employees confronting this unprecedented issue.

Third, when considering the design of information systems for healthcare services, it needs to consider whether external and internal institutions had a differential effect on the intention to use new technology [128]. In the results of our study, in terms of

uncertainty reduction, the effect of internal institutions is stronger. As a consequence, we would conclude that when implementing new technologies, it is important to provide employees with a standardized procedure and best practices [149]. However, external institutions had a stronger total effect on intention to use than internal institutions, where perceived ease of use and perceived usefulness have mediating roles in our result. Thus, we can conclude that official government schemes and systems may have superiority over managerial actions when implementing new technology. Therefore, it is necessary to provide adequate external institutions (like laws and ordinances) for effective adopting new technology in healthcare fields.

In Table 11, we summarized the contributions of our study in the fourth industrial revolution and TAM studies. This section will provide a concise description of the experimental results, their interpretation, and the experimental conclusions that can be drawn.


**Table 11.** Theoretical contributions of this study to representative previous research.


#### **Table 11.** *Cont.*

#### *5.2. Practical Implications*

Our study empirically investigated the influence of institutions on new technology adoption of the fourth industrial revolution. The characteristics of these new technological developments are complicated [21], and it is expected that the fourth industrial revolution will bring new changes by growing networking, data-based intelligence, and knowledge sharing between humans by real-time computer interaction [153]. When new technology is initiated in an organization, institutional influence is normally based on centralization; however, simply adopting external requirements could result in short-term usage of new technology like perceived ease of use but not long-term goals like usefulness and intention to use in our study [13].

New technological advancements like the fourth industrial revolution should be closely interconnected with institutional arrangements such as strategies that allow a structure to adjust or preserve stabilization after experiencing periods of instability [154]. For instance, in the case of South Korea, after the Middle East Respiratory Syndrome (MERS) outbreak in 2015, structures and processes such as strengthened legislative authority for quarantine and surveillance systems encouraged people to promptly adapt to digital technologies in response to the pandemic [155]. Thus, these institutional facilitators and

the groundwork helped the new technology to be quickly used for public health. These institutional arrangements for disease control that include relative digital data management have enabled South Korea to extend accelerated testing and case recording to a greater portion of the population when faced with COVID-19 [155]. Contrariwise, the United States, where lengthy lines for coronavirus testing wrapped around the block early in the pandemic, demonstrated a defective monitoring process even several months after the pandemic outbreak.

Countries like Singapore and Australia launched a digital app and QR tracing, while data encrypted technology was reserved for government-based utilization [156,157]. In Singapore's attempts to control the COVID-19 pandemic, technologies such as artificial intelligence and data analytics have played a critical role. The Australian experience also shows that specific institutional responses are crucial for public health prevention. For example, with the launch of a monitoring and tracing application for smartphones, COVIDSafe, the Australian federal government rapidly replaced the conventional contact tracing methods. These strong institutional adjustments increase user safety [158]; the application was downloaded by six million people in a short period.

Given the pandemic's uncertain nature and the risks it presents, it made sense for government authorities to first centralize decision making, such as school openings, and create or implement strategies for delivering instruction to fulfill regulatory requirements [159]. Nevertheless, as we can see with cases in various countries, administrative regulatory decisions during the pandemic require unambiguous evidence of the reasons and necessities for the public. Without a large amount of obvious data from reliable objective technological procedures, public uncertainty and doubt cannot be easily alleviated. We should consider public confidence in public health officials and other policy makers. While the latest pandemic is unsettling, we can envision a worse situation in the future.

As we have seen in the previous examples of response to the COVID-19 pandemic in various countries, new technology adoption does not always lead to complete adoption and use. One of the main reasons for the failure of U.S. governance was not due to technological difficulty but rather to institutional instability and perceived utility [160]. Thus, as our study suggested, detailed considerations should be prepared when established institutions adopt new technology in the fourth industrial revolution.

#### **6. Conclusions**

In this study, we assessed the causal path relationships of new technology adoption in the fourth industrial revolution by an empirical survey of public employees of the central government in South Korea. Since government employees play such a significant role in the decision to construct new technology deployment [161,162], they are in a key position to introduce and utilize the advantages of new technology. The convergence of physical and digital technologies is one of the key features of the fourth industrial revolution [153] and it could open new opportunities to transform conventional human–computer interaction, as we have witnessed during the current global COVID-19 pandemic [155].

We identified that institutions influence perceived ease of use and usefulness, which mediate the impact of institutions on intention to use new technology [163]. According to our findings, institutions have a significant influence on the intention of public employees to use new technology in terms of ease of use and usefulness, which has a mediating effect on intention to use. This means that if digital-based new technology is assumed to be the new means of information usage, the behavioral patterns of new technology adoption would not be strongly differentiated from conventional technology use [23,152], except for institutional impact. In addition, as we have witnessed, the applications of new technology in the fourth industrial revolution to confront the pandemic are constantly increasing. However, without appropriate consideration of users' perception and behavior, this technological use would not be as effective [164].

In addition, as the results of our alternative expanded model showed, institutions cannot directly impact intention to use new technology without the mediating effect on perceived ease of use and usefulness if we do not control the causal path relationship [165]. It is important to note that the casual path direction of the influence of institutions is only through ease of use, not usefulness in the expanded alternate model, implying that perceived ease of use should receive significant consideration when new technology is introduced in a society [165,166]. In conclusion, this study specified the determinants of new technology adoption in the fourth industrial revolution and, in particular, sought a causal path between institutions and the intention to use new technology with perceived ease of use and usefulness as mediators.

The findings of this study could be meaningful in many respects. However, they also present certain limitations that will provide a basis for future research. This study performed statistical tests and analyses based on cross-sectional survey data collected at a single point in time. Thus, it is not possible to use a panel analysis methodology that examines the trend of usage over time. In addition, even though we conducted several statistical tests to avoid possible methodological bias in our model, mono source bias or correlated measurement error might not be completely excluded.

Because of the contextual features of the South Korean public sector, it is necessary to consider biases from contextual factors like culture in further research [167]. Furthermore, given that our alternative two-factor institution models are based on high hierarchical level organization, the results might show differently if they were applied to low hierarchical level organization. In addition, to offer greater insight into the phenomenon of usage and utilization, the use of case studies should be considered to investigate more interpretive research of new technology in the fourth industrial revolution phenomena [13,14]. Future studies are encouraged to explore the issues and practices of new technology and its effectiveness in the context of determinants.

**Author Contributions:** Conceptualization Y.K. and N.C.; data acquisition, N.C.; writing—theoretical framework and analysis, Y.K. and S.K.; writing—methodology and analysis, N.C. and Y.K.; writing review and editing, S.K and N.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This article is financially supported by the Graduate School of Public Administration at Korea University in 2021. Nakbum Choi is financially supported by Seowon University.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to regulations and guideline of data open policy according to the Korea University.

**Acknowledgments:** This paper makes use of research material produced by the Korea Institute of Public Administration (KIPA) and has been authorized for use according to KIPA's regulations on the ownership and use of said research material.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Rise of Clinical Studies in the Field of Machine Learning: A Review of Data Registered in ClinicalTrials.gov**

**Claus Zippel and Sabine Bohnet-Joschko \***

Chair of Management and Innovation in Health Care, Faculty of Management, Economics and Society, Witten/Herdecke University, 58448 Witten, Germany; claus.zippel@uni-wh.de **\*** Correspondence: Sabine.Bohnet-Joschko@uni-wh.de; Tel.: +49-(0)-2302-926-592

**Abstract:** Although advances in machine-learning healthcare applications promise great potential for innovative medical care, few data are available on the translational status of these new technologies. We aimed to provide a comprehensive characterization of the development and status quo of clinical studies in the field of machine learning. For this purpose, we performed a registry-based analysis of machine-learning-related studies that were published and first available in the ClinicalTrials.gov database until 2020, using the database's study classification. In total, *n* = 358 eligible studies could be included in the analysis. Of these, 82% were initiated by academic institutions/university (hospitals) and 18% by industry sponsors. A total of 96% were national and 4% international. About half of the studies (47%) had at least one recruiting location in a country in North America, followed by Europe (37%) and Asia (15%). Most of the studies reported were initiated in the medical field of imaging (12%), followed by cardiology, psychiatry, anesthesia/intensive care medicine (all 11%) and neurology (10%). Although the majority of the clinical studies were still initiated in an academic research context, the first industry-financed projects on machine-learning-based algorithms are becoming visible. The number of clinical studies with machine-learning-related applications and the variety of medical challenges addressed serve to indicate their increasing importance in future clinical care. Finally, they also set a time frame for the adjustment of medical device-related regulation and governance.

**Keywords:** machine learning; digital health; registry analysis; ClinicalTrials.gov; device regulation

#### **1. Introduction**

#### *1.1. Background*

Before medical innovations can be implemented in daily clinical routine, it takes more than a decade from research and development to market approval [1–3]. In this translation phase, a multitude of challenges and specifications have to be overcome so that a device can successfully be brought to the market, from patient recruitment, data consolidation and fragmented infrastructures to regulatory hurdles and (start-up) financing of research costs [4,5]. Examining the literature, it is noticeable that so far, there are hardly any data on the specific translation process of medical–digital applications that are increasingly being developed and that promise great benefits and potentials for health prevention, diagnostics, and therapy [6–10].

#### *1.2. Research Motivation and Objective*

Against this background, it was our aim to explore the development and current translation status of medical–digital applications in the field of machine learning (ML), a sub-area of artificial intelligence in which computer algorithms and statistical models are trained based on large datasets to independently link and predict abnormalities and correlations in a self-learning manner [11–15]. We focused on ML, as there are already a wide range of ML-based approaches and innovative developments for health care reported

**Citation:** Zippel, C.; Bohnet-Joschko, S. Rise of Clinical Studies in the Field of Machine Learning: A Review of Data Registered in ClinicalTrials.gov. *Int. J. Environ. Res. Public Health* **2021**, *18*, 5072. https://doi.org/10.3390/ ijerph18105072

Academic Editors: Martin Wiesner, Björn Schreiweis and Corneel Vandelanotte

Received: 7 April 2021 Accepted: 7 May 2021 Published: 11 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

in the literature, from image diagnostics and processing [16–20], personalized medicine and genomics [21–23] to clinical data analysis for decision support and training in surgery, therapy planning or patient management [24–28].

In view of the research question, we decided to analyze study register data as they offer a glance into the research pipeline of universities, university clinics and research institutions as well as pharmaceutical, medical device, and biotech companies, and thus, provide first insights into the clinical translation process of ML-related applications and software. This registry-based approach also allows us to cluster and identify fields with increased research and investment that might be of clinical significance in the next decade. Considering legislative delays, our results may support health decision- and policymakers struggling with challenges in the regulation and governance of ML-applications [29–31].

#### **2. Materials and Methods**

#### *2.1. Data Acquisition and Processing*

For our study, we used datasets from ClinicalTrials.gov, one of the most comprehensive databases for clinical studies worldwide with over 360,000 planned, ongoing and completed clinical studies published at the time of access [32–35]. The register is freely accessible via https://clinicaltrials.gov [36]. For each study, (i) a given set of study characteristics is compulsory, and (ii) study-specific details are requested, using free text fields, such as title or individual short description. The ClinicalTrials.gov database and methodological approach have already been chosen frequently in other research studies to characterize study populations and trends in clinical care and research [37,38], for example in the areas of medical imaging [39–41], rare diseases [42] or oncology [43,44].

In view of the research question, the "advanced search function" was used to filter the register data records for which "Machine Learning" (a MeSH term introduced in 2016, [12]) had been entered in the report form and which were published by the end of 2020 (search term: "Machine Learning"|First posted on or before 31 December 2020). The dataset was retrieved on 7 January 2021 and exported in CSV file format [36]. In a second step, the authors scanned the dataset and included all study entries that clearly focused on the use or testing of ML-based algorithms, approaches or applications in a clinical setting. Entries on clinical trials that, according to the reporting party, were "withdrawn" or "terminated" or clearly did not primarily focus on the use of ML-related approaches or applications in clinical care were excluded from the study. In order to be able to filter and subgroup the studies in detail, the authors scanned the free text information of the study entries. Figure 1 shows the methodical procedure for the selection process of the study dataset considered for the register data analysis in the form of a flowchart.

**Figure 1.** Flowchart for the selection procedure of the ML-related clinical study entries considered for the quantitative registry analysis. Source: Own figure based on the evaluation of the ClincalTrials.gov dataset [36].

#### *2.2. Data Evaluation and Analysis*

In order to provide an overview of the development and status quo of ML-related software approaches and applications in the clinical setting, the study entries were sorted in ascending order according to the date of which the study record was first available on ClinicalTrials.gov. Furthermore, common standardized study parameters, such as study type, recruitment status, age group or funding source, were evaluated [45]. In order to achieve a more in-depth characterization of the dataset, the authors scanned, evaluated and subcategorized the study entries according to further parameters, such as recruiting country, academic/industry sponsor or clinical study-initiating medical specialty/field. Further free text information, such as intervention arms, inclusion criteria or end points of the trials, were not part of the study.

In view of the explorative nature of the study objective, we evaluated the registry dataset descriptively. One-dimensional frequency distributions (absolute, relative) were determined for the analyzed study characteristics. The development of the published studies per year over time was shown graphically using a bar chart, and the description of all other parameters was summarized in tables. The quantitative acquisition, processing and statistical evaluation of the dataset was carried out, using Microsoft Excel® software for Microsoft Windows®.

#### **3. Results**

#### *3.1. Registration of ML-Related Studies over Time*

For our study, *n* = 358 study entries in the field of ML were included (see Figure 1). Sorted by year of first publication in the ClinicalTrials.gov register, a continuous rise in ML-related study entries could be seen since 2015, with a particularly significant increase between 2019 and 2020, from *n* = 89 to *n* = 149 posted studies (see Figure 2).

**Figure 2.** Number of clinical studies related to ML by year of publication on ClinicalTrials.gov (*n* = 358). Source: Own figure based on the evaluation of the ClincalTrials.gov dataset [36].

#### *3.2. Medical Field of Application*

The registered studies focused on a broad spectrum of different topics from a wide range of medical specialties. The majority of the posted studies in the field of machine learning was initiated by experts from the field of imaging (diagnostic radiology, nuclear medicine, radiation oncology; 12%), followed by cardiology, psychiatry, anesthesia/intensive care medicine (all 11%), neurology (10%), medical oncology (8%) and infectious disease medicine (6%) (see Figure 3). The latter mainly included studies that were published in 2020 on COVID-19-related issues. *Int. J. Environ. Res. Public Health* **2021**, *18*, x 3 of 13

latter mainly included studies that were published in 2020 on COVID-19-related is-**Figure 3.** Study entries in the field of ML by study-initiating medical specialty/field (*n* = 358). Source: Own figure based on the evaluation of the ClincalTrials.gov dataset [36]. \* Dianostic Radiology/Biomedical Imaging, Radiation Oncology, Nuclear Medicine.

**Figure 3.** Study entries in the field of ML by study-initiating medical specialty/field (*n* = 358).

Source: Own figure based on the evaluation of the ClincalTrials.gov dataset [36].

sues.

#### *3.3. Patient Recruitment and Study Organization*

About half of the listed clinical studies were open (55%) or closed (45%) for patient enrollment. A total of 27% of the studies had already completed the recruitment phase. The vast majority of studies (98%) did not yet have any results. A total of 80% of the studies in the dataset were single-center, 13% multi-center studies. Seven percent could not be classified because of missing information (see, for this and the following, Table 1). Of the studies, 96% were national and 4% international. Of these, by far the most studies had a last recruiting location in the U.S.A. (40%), followed by China (9%), the United Kingdom (8%), Canada (6%), France (5%), Switzerland and Germany (each 4%). Across all study entries, and with a view to the major global regulatory regions, most of the published studies recruited patients in a country in North America (47%), followed by Europe (37%) and Asia (15%; other 6%).

**Table 1.** Recruitment and organizational parameters of the included ML-related trials from the ClinicalTrials.gov registry (*n* = 358).


**Table 1.** *Cont.*


\* Sum partly 6= 100 due to rounding; \*\* More than one choice possible; \*\*\* Subcategories in italics; Source: Own table based on the evaluation of the ClincalTrials.gov dataset [36].

In 82% of the studies, a university (hospital) and/or research institution was named as the organization/person responsible for the study (so-called "lead sponsor"), and in 18%, an industrial company. The majority of trials (88%) was (co-) funded by individuals, universities or organizations themselves, 24% of trials were (co-) funded by the industry and 5% had a public (government) sponsorship.

#### *3.4. Study Type and Design*

Of the *n* = 358 clinical studies categorized, around two thirds (64%) were reported as observational studies and around one third (36%) as interventional studies (see, for this and the following, Table 2). Among the observational studies, the majority of the studies were designed as prospective cohort studies. The majority of the interventional studies was open label/non-masked and single-armed. Over 90% of the studies planned to enroll (elderly) patients of both genders.

**Table 2.** Study type and study design specific parameters of the included ML-related clinical trials from the ClinicalTrials.gov registry (*n* = 358).



#### **Table 2.** *Cont.*

\* Sum partly 6= 100 due to rounding; \*\* More than one choice possible; \*\*\* Subcategories in italics; Source: Own table based on the evaluation of the ClincalTrials.gov dataset [36].

#### **4. Discussion and Conclusions**

Recent improvements and innovative approaches in the field of artificial intelligence promise high potential for the diagnosis and treatment of patients [46–49]. The subarea of ML in which self-learning algorithms (such as convolutional neural network, random forest, support vector machine, etc. [50–52]) are trained on large datasets and used to make predictions independently when exposed to new data, is particularly advancing [11,13,14,17,19,20]. More and more research is showing that newly developed algorithms can process specialized tasks just as well as experienced health professionals or can increase their efficiency and performance in daily care [53–56]. A crucial factor for the successful development of ML-based software and assistance systems is—besides medical and technological expertise—in particular, the testing and use of these applications in daily clinical routine [57,58]. With this in mind, it was our goal to find out more about the recent development and status of the clinical translation of ML-related software and applications into the clinical setting. The translation and market approval of ML-based algorithms represent a major challenge in terms of legislation and regulation. Using the example of register data, the results show how dynamically this area is developing across medical disciplines. As a result, questions about governance and clinical testing will have to be answered in the near future (cf. for example [29–31]). In the following sections, we will summarize the main results of the registry data analysis on ML-related clinical studies, discuss this with reference to the regulatory environment and point out the methodological limitations of the study.

#### *4.1. Studies in the Field of ML*

The study data show that the number of ML-related studies in ClinicalTrials.gov has increased continuously from year to year since 2015, with a particular increase between

2019 and 2020 (see Figure 1). From a methodological point of view, it should be noted that the MeSH term "machine learning", which was crucial for the study search in the registry database, was introduced in 2016 by the U.S. National Library of Medicine [12]. This could have influenced the search and selection procedure (especially for the period before 2015), as this MeSH term was probably only systematically reported and checked as a quality control review criteria for the clinical study registration from this point in time [59]. For the last few years, however, a visible increase in the number of published studies can be determined. This could be an indicator for the growing potential that is associated with the use and application of ML-related software/algorithms for medical care and research.

In addition, it was found that the majority of the analyzed studies in the field of ML were initiated and led by (university) hospitals or academic/research institutions (82%) and were (co-) financed from university (88%) or public/government funds (5%) (see Table 1). Among the academic institutions, most of the registered studies were reported by the Mayo Clinic (U.S.), Maastricht University Medical Center (NL), Sun Yat-Sen University (CN) and University of California (U.S.). In this context, the authors assume, that the number and proportion of academically initiated ML-related studies is likely to be underestimated here since the sponsor or PI in some cases does not necessarily have to register an academic study in a database such as ClinicalTrials.gov. This is especially the case for studies in the preclinical development stage or if only retrospective data are used. In comparison, fewer studies were initiated (18%) or (co-) financed (24%) by an industry sponsor. The proportion of studies with an industrial study sponsor is (still) relatively low, compared to other publications on ClinicalTrials.gov study data. For example, a cross-sectional analysis by Ross et al., published in 2009, showed a proportion of 40% in studies with industry sponsors [38] and a study by Bell and Smith from 2014 on over 24 thousand clinical studies on rare and non-rare conditions showed a proportion of more than 30 percent [42].

Among the industry sponsors were several comparatively small companies and startups with a focus on the development of algorithms in medicine (e.g., Dascena® and Eko Devices®). In general, it can therefore be assumed that the ML-related approaches reported were still mainly initiated and used in an academic/research context but could gradually be transferred to clinical translation and early clinical study development phases with increasing support from the industry, which sees investment potentials in this area.

Moreover, the analyzed studies were initiated from a variety of different medical fields and disciplines (Figure 3). Looking at the dataset, it could be seen that the ML-related approaches in the clinical studies used different types of training data. This included image data (e.g., in radiomics studies), sensor data (e.g., ECG signals), video data, text data and audio data (e.g., monitor audio signals). Furthermore, the registered studies used a wide range of different types and approaches of ML algorithms, such as (un-) supervised or reinforced learning. In order to illustrate this heterogeneity, we show selected study approaches from different medical application areas and fields. We hereby focus on advanced clinical studies for which the recruitment phase was reported as completed and at least one scientific publication was available.


In summary, the results of the registry data analysis show that the registered studies in the field of ML were very heterogeneous, both from an organizational and study design perspective. Against this background, it would make sense to carry out further (especially multivariate) sub-evaluations of the dataset for selected study groups, for example, with large cohort radiomics studies, etc. Finally, it should be noted that the imaging disciplines in particular are involved in many studies, both as a study-initiating discipline and as a clinical partner, for example, for CT, MRI or PET scans. Since only the respective, study-initiating department was focused on for the register analysis, it can be assumed that the proportion of ML-related studies in which imaging experts are centrally integrated is significantly higher than the 12% shown in Figure 3.

#### *4.2. Regulatory Framework and Aspects*

With regard to the dataset, it is essential to point out, from a regulatory point of view, that the posted studies in the field of ML always address software that, in many cases, functions or is used (directly) in connection with a medical device. This is of central importance since from a regulatory point of view, software is considered a medical product in many regulatory areas, such as the U.S. or the European Union [68], and is, therefore, subject to the associated regulatory requirements, such as conformity assessment, registration, clinical evaluation or post-market surveillance [69]. In the EU, for example, software is considered a medical device according to the European Medical Device Regulation (MDR), which will come into force in May 2021, "when specifically intended by the manufacturer to be used for one or more [ . . . ] medical purposes [ . . . ], independent of the software's location or the type of interconnection between the software and a device" [70]. The risk classification is based on the diagnostic and therapeutic intension of the software from risk classes I (lowest risk class) to III (highest risk class).

In this context, it should be pointed out that for ML-related software, primarily the general regulatory requirements for software apply and that there are hardly any laws or harmonized standards for the specific use of ML-software and applications in healthcare. With this in mind, it is of great interest that the U.S. Food and Drug Administration (FDA) has published a discussion paper on "Artificial Intelligence/Machine Learning (AI/ML)- Based Software as a Medical Device (SaMD) Action Plan", which is continuously updated and currently making proposals with regard to the following areas:


In view of the increasing amounts of clinical studies in the field of ML (Figure 1), it will be interesting to see how the regulatory framework will adapt, worldwide, to AI- and ML-related software and applications as well as the specifics associated with them. Aspects that have not yet been clarified, such as changes in ML-related software over time due to changing datasets, should be of particular interest. In the literature, suggestions are increasingly being submitted and discussed [30,72], both on general regulatory aspects [29,73,74] and on device- or subject-specific features, e.g., in view of medical imaging [75,76].

In addition, it becomes clear how important it will be in the future to pool patient data for clinical studies in the field of machine learning across multiple locations. The reason for this is that access to large amounts of data will be essential for the further development of the approaches in prospective clinical studies. An example of how this could work in view of strict data protection requirements is shown by the Joint Imaging Platform for Federated Clinical Data Analytics for the application of medical algorithms across study sites in the field of medical imaging [77].

#### *4.3. Methodological Notes*

The evaluation of registry data from ClinicalTrials.gov enables a broad and detailed analysis of a multitude of systematically collected, study-specific entries of high quality over a period of time. However, a number of limitations to this study approach need to be noted. Firstly, a method-inherent error of this approach is that the register dataset only represents a subset of all initiated ML-related studies around the globe. The reasons for this are that in some cases, the PI or sponsor does not necessarily have to register the study (see Section 4.2) or may as well choose a different registry to list the study accordingly [78–80]. In this context, it should also be pointed out that data and information specifically relating to research in the field of machine learning are also published in other digital archives or specific registers and research platforms, such as the platform of the Association for Computing Machinery (ACM) or the Institute of Electrical and Electronics Engineers (IEEE). This both illustrates the importance of harmonizing the fairly large number of registries and archives to prospectively create (also linguistically) more uniform data, a process that is focused on by projects such as the "Research Data Alliance" or the "Open Data Institute". Secondly, the registry search only took into account register entries in which the search term "machine learning" was explicitly specified in the study title or free text. Since the use of study-specific MeSH terms when registering studies in ClinicalTrials.gov is recommended but not mandatory, it can be assumed that studies that used other MeSH terms or were registered with terms related in taxonomy were not taken into account for the dataset. This may well lead to the fact that the actual number of ML-related clinical studies published, and thus the clinical development in this field, is probably underestimated. Thirdly, common limitations of clinical registry (meta-) data analyses apply, which can lead to inaccuracy and inconsistencies, and thus may impair the data quality. This includes, in particular, incorrect or not-at-all answered sections of the registry form. In addition, the study text information (some of which vary in scope and content) can be interpreted differently, which could reduce the validity of the results [37,39–45]. Fourthly, the subgrouping of studies into medical specialties was not always clear; for example, when experts from two or more medical specialties were involved. In order to avoid this methodological problem, the medical specialty of the PI responsible for the trial and named in the study entry was used for subgrouping in case of doubt. As a result, medical specialties that are often involved in ML-related studies but tend to initiate fewer studies as the lead medical specialty were probably counted less (e.g., (neuro-) pathology [81]). Fifthly, it has to be assumed that since ClinicalTrials.gov is an American registry, there is a disproportionately high number of registered clinical trials conducted in North America. Our study results strongly support this hypothesis, seeing that the vast majority of studies included those recruited in the U.S.A. and Canada (see Table 1). This may possibly lead to distortions in comparison to the status and characteristics of ML-related trials in other regions, such as Europe or Asia.

In view of the limitations, the present study cannot represent a complete, detailed picture of the status quo. However, since ClinicalTrials.gov is by far the biggest and most renowned registry for clinical trials, the authors conclude that this approach allows a good first overview on the current status of clinical development and translation of ML-based approaches and applications in health care. This could provide an impetus for decisionmakers in healthcare facilities and policy as well as regulatory discussions.

#### **5. Summary for Decisionmakers**


increased from year to year since 2015, with a particularly significant increase in the last two years.


**Author Contributions:** Both authors developed the idea of the paper/research method, conducted a search of the literature and wrote, reviewed, edited and formatted the draft of the manuscript. All authors have read and agreed to the published version of the manuscript.

**Funding:** The APC is paid for under the ATLAS project which is funded by the Ministry of Economic Affairs, Innovation, Digitalization and Energy of North Rhine-Westphalia (funding code: ITG-1-1).

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** For the study, data from the website ClinicalTrials.gov were used [36]. The registry for clinical studies is available online https://clinicaltrials.gov (accessed on 7 January 2021).

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*International Journal of Environmental Research and Public Health* Editorial Office E-mail: ijerph@mdpi.com www.mdpi.com/journal/ijerph

MDPI St. Alban-Anlage 66 4052 Basel Switzerland

Tel: +41 61 683 77 34

www.mdpi.com

ISBN 978-3-0365-4679-7