Toward a Comprehensive Understanding and Evaluation of the Sustainability of E-Health Solutions

Abstract

Digital health transformation (DHT) has been deployed rapidly worldwide, and many e-health solutions are being invented and improved on an accelerating basis. Healthcare already faces many challenges in terms of reducing costs and allocating resources optimally, while improving provided services. E-solutions in healthcare can be a key enabler for improvements while controlling the budget; however, if the sustainability of those solutions is not assessed, many resources directed towards e-solutions and the cost of adoption/implementation will be wasted. Thus, it is important to assess the sustainability of newly proposed or already in-use e-health solutions. In the literature, there is a paucity of empirically driven comprehensive sustainability models and assessment tools to guide practices in real-world cases. Hence, this study proposes a comprehensive sustainability model for e-health solutions to assess the essential sustainability aspects of e-health solutions and anticipate the likelihood of their sustainability. To build the model, a systematic literature review (SLR) was conducted to extract the e-health sustainability dimensions and elements. In addition, the SLR analyzes the existing definitions of sustainability in healthcare and sustainability assessment methods. The proposed sustainability model has five dimensions, namely; technology, organization, economic, social, and resources. Each dimension has aspects that provide another level of required detail to assess sustainability. In addition, an assessment method was developed for this model to assess the aspects of each dimension, resulting in the overall prediction of the e-health solution’s sustainability level. The sustainability model and the assessment method were validated by three experts in terms of comprehensiveness and applicability to be used in healthcare. Furthermore, a case study was conducted on a Hospital Information System (HIS) of a hospital in Saudi Arabia to evaluate the sustainability model and its assessment method. The sustainability model and assessment method were illustrated to be effective in evaluating the sustainability of e-solutions and more comprehensive and systematic than the evaluation used in the hospital.

1. Introduction

Sustainability has been a challenge in many domains and development contexts. In e-health, in particular, it is challenging to define and apply sustainability to real projects due to several reasons [1]. Some of these reasons are the complex nature of healthcare, the increasing demand for resources [2], and the lack of a clear and consensus definition of sustainability in healthcare [3]. The importance of sustaining digital solutions in healthcare is laid upon the fact that healthcare is a critical sector involving many obstacles, in which implementing changes is difficult due to the strict regulations and complex, interrelated processes [4].

Some studies proposed sustainability models; however, the models lack important aspects, such as economic and funding estimation, change impact on current employees, and patient and staff acceptance [3,5]. Furthermore, several studies stated that the absence of a clear definition of sustainability in the literature results in the absence of rigorous evaluation methods of sustainability [3,6,7,8,9] and hinders the achievement of assessing e-health sustainability [10]. In addition, the absence of a comprehensive view of the sustainability of e-health solutions remains the main challenge in having sustainability frameworks and assessment methods [3].

Hence, the main objective of this study is to provide a comprehensive sustainability model of e-health solutions and its assessment method. This is achieved by conducting a systematic literature review (SLR) that investigates and analyzes existing studies about the sustainability of e-health solutions. The SLR focuses on analyzing sustainability definitions, models, and assessment methods. Investigating existing definitions of the sustainability of e-health solutions will provide a general and wide overview of how sustainability is perceived in the context of e-health solutions and its main, high-level, dimensions. Additionally, the SLR results are used to propose a comprehensive sustainability model, dimensions, and aspects for e-health solutions that lay the foundations for sustainability analysis and assessment methods. Finally, the third part of the SLR focuses on analyzing existing sustainability assessment methods in the context of e-health solutions. That is, the SLR aims to learn the dimensions of sustainability that have been assessed and how they have been assessed in existing work to develop a comprehensive assessment method to the proposed sustainability model. The comprehensiveness and applicability of the proposed model was validated by three IT experts in the healthcare domain. The main contributions of this work are (1) providing an overview of sustainability definitions and proposing a comprehensive definition accordingly, (2) a thorough analysis of sustainability dimensions and aspects, (3) a comprehensive sustainability model for e-health solutions, and (4) an assessment method that is used to assess the sustainability of e-health solutions.

The rest of the paper is organized as follows: Section 2 provides an overview of related work on sustainability models in healthcare. Section 3 illustrates the study protocol followed in conducting the SLR. Section 4 presents the results of the SLR. Section 5 presents our proposed sustainability model for e-health solutions, and Section 6 includes the assessment method. Section 7 presents the conducted case study. Section 8 discusses the results and important challenges in this area, while Section 9 states the limitations. Finally, Section 10 concludes and identifies future work items.

2. Related Work

Most of the existing studies discuss sustainability in healthcare from a managerial level only. Doyle et al. [11] proposed a sustainability model for changes in healthcare. The model was in the form of a self-assessment tool, which consisted of 10 factors that were prioritized over 100 identified factors. These factors were categorized into three main domains and given weights as follows: process (31%), staff (52%), and organization (17%). Each of the ten factors has four levels, by which the user chooses one level for each factor that describes the current situation. However, some important aspects were missing in these factors, such as the economic and funding estimation, the probability of laying off staff and its impact on the organization, and the social-cultural aspects of the organization and the staff. Molfenter et al. [12] constructed a model to predict the sustainability of changes in healthcare. The study proposed nine factors associated with multiple levels for each factor. However, this model lacks factors concerning patients, such as their satisfaction with the change. One of the important aspects of sustaining changes in healthcare, which is rarely addressed in the existing literature, is the willingness of healthcare stakeholders (e.g., patients, healthcare providers, families, and managers) to adjust themselves to the change [13].

Penzenstadler and Femmer [14] proposed a sustainability model consisting of three levels for software products. In their work, they defined the sustainability dimensions to be individual, social, economic, environmental, and technical. The dimensions were at the top level of the model. At the same time, the middle level included the values, indicators, and regulations. The activities of the model, such as reducing waste, are placed in the lower level. This model focused on environmental sustainability and lacked a steady assessment method that can measure the impact of each factor in anticipating the sustainability of the solution. A systematic integrative review [3] on healthcare sustainability revealed that out of 92 studies, only 6 studies used purpose-designed tools to evaluate sustainability. Braithwaite et al. [3] emphasized the lack of a unified and rigorous sustainability definition and inconsistency in the way it is conceptualized [3]. Another study [6] concluded the lack of a comprehensive definition of sustainability was the essential challenge that hindered the field from moving forward. Other studies discuss sustainability in healthcare with a focus on developing tools [1] and frameworks [2,15] for sustaining the healthcare organizations rather than focusing on the changes within these organizations. Furthermore, another study discussed the role of digital solutions in sustaining healthcare in general by presenting different types of interactions that can happen via digital platforms [16].

A study conducted by A. Bartosiewicz et al. [17] found the importance of IT competence among nurses for the quality of using new technologies in work. This indicates that there are different aspects that should be considered when evaluating the quality of an e-health solution.

3. Study Protocol

This section contains the designed protocol that was followed when conducting the SLR. It presents the research questions, search strategy, quality assessment, data extraction, and data synthesis. The protocol was validated by an expert in software engineering.

3.1. Research Questions

This SLR was intended to answer the following research questions:

• RQ1: Are there specific definitions of sustainability of e-solutions in the context of healthcare?

• Objective 1: Analyze the sustainability definitions found in the literature for e-solutions in healthcare and provide a comprehensive definition.

• RQ2: What are the existing dimensions and elements of sustainability for e-solutions in the healthcare context?

• Objective 2: Analyze the dimensions and aspects of sustainability found in the literature for e-solutions in healthcare.

• RQ3: Are there specific metrics, measures, and assessment methods/models of evaluating sustainability for e-solutions in the context of healthcare?

• Objective 3: Analyze the assessment methods of sustainability models found in the literature for e-solutions in healthcare.

3.2. Search Strategy Design

3.2.1. Search Process

The search was conducted over three rounds, as shown in Figure 1. First, the search string was used to examine the selected databases and filter the studies based on the relevance of their titles and abstracts. Second, the full text of the resulting studies from the first round was skimmed in order to exclude papers that did not meet the inclusion criteria or matched one of the exclusion criteria. Third, the studies that passed the second round were evaluated using the quality assessment criteria. All papers whose score in the quality assessment was less than the predetermined threshold were excluded from the set of selected studies. Finally, all papers that passed the third round were considered in the final set of studies and used in data extraction and synthesis.

3.2.2. Search String

Table 1. Search string.

Scope	String
Sustainability	(sustainability or long-term or continuous or lasting) and
Main issue	((model or definition or elements or dimension* or component*) or (assess* or evaluate* or measure* or metric*)) and
Technology	(electronic* or digital or technology* or software or service* or smart) and
Healthcare	(healthcare or e-health or m-health)

contains the search string that was used to find the primary studies. It consists of all keywords that are relevant to the focus of the study. The keywords terms were derived from two SLRs [18,19]. Both SLRs investigated the sustainability issue in healthcare in general [19] and in sustaining hospital-based interventions [18]. Furthermore, keywords and search strategy in the study [18] were peer-reviewed by an academic librarian in accordance with PRESS guidelines [20].

The keywords were then refined to match the scope of this work. Boolean operators and wildcards (e.g., “*” and “?”) were used to account for plurals and variations in spelling.

3.2.3. Selected Digital Libraries

In order to select the relevant databases, we ran the search string on several of them. Based on that, the databases of three digital libraries were chosen, which were found to have more relevant results to the research scope than other libraries (PubMed, Medline, SpringerLink, and ACM). The selected libraries are Web of Science, IEEE Xplore, and Scopus. In addition, a snowballing technique was used to collect relevant studies.

3.3. Study Selection Criteria

All collected studies were evaluated by the inclusion and exclusion criteria before adding them to the set of selected studies. The inclusion and exclusion criteria are presented in the following subsections.

3.3.1. Inclusion Criteria

A paper was selected if it met all inclusion criteria presented in

Table 2. Inclusion criteria.

No.	Criteria
IC1	The paper was published between 2000 and 2021.
IC2	The paper discusses sustainability in healthcare with respect to e-health solutions.
IC3	The paper includes at least one of the following: sustainability model or components (dimensions or elements), sustainability measurements or assessment/evaluation methods, or sustainability definition.
IC4	The paper must be a full research paper (not an abstract).
IC5	The paper is written in English.
IC6	The paper is available in full-text version.

. The year range was chosen upon a previous SLR conducted in 2018 [19]. The SLR included studies from 1980 until 2017. The studies in the SLR [19] that are relevant to our research scope were found between 2002 and 2014. Therefore, we decided to expand the range to be from year 2000 to year 2021.

3.3.2. Exclusion Criteria

The papers that met one or more of the following exclusion criteria in

Table 3. Exclusion criteria.

No.	Criteria
EC1	Meeting abstracts, book chapters, workshop descriptions, tutorials, posters, progress reports, masters and doctoral dissertations.
EC2	Papers discuss sustainability from a purely environmental perspective, such as recycling or energy conservation, etc.
EC3	Papers discuss sustainability of the healthcare system/organization instead of the sustainability of e-health solutions within healthcare system.
EC4	Papers discuss sustainability of a medical device/tool instead of e-health solution related to healthcare system and its processes.
EC5	Papers discuss sustainability from a general perspective without proposing or providing results related to constructs of sustainability models, definitions, or assessment methods.

were excluded.

3.4. Quality Assessment

The purpose of quality assessment, which is presented in

Table 4. Quality assessment checklist.

No.	Quality Assessment Question	Score
QA1	Does the paper have a clear statement (definition) of the aims (goals, purposes, problems, and research questions)?	(+1) Yes/(+0) No/Partially (+0.5)
QA2	Does the paper explicitly propose/discuss the sustainability aspects of e-health solutions?	(+1) Yes/(+0) No/Partially (+0.5)
QA3	Does the paper evaluate/investigate the sustainability of e-health solutions through using case studies, surveys, etc.?	(+1) Yes/(+0) No/Partially (+0.5)
QA4	Are conclusions, implications for practice, and future research, reported suitably for its audience?	(+1) Yes/(+0) No/Partially (+0.5)
QA5	Were the aspects extracted from interviews, case studies, etc.?	(+1) Yes/(+0) No/Partially (+0.5)

, is to assess the quality of each selected paper to be included in the final set. Questions were deduced from Zhou, et al. [21], which is a tertiary study about the quality assessment criteria used in software engineering SLRs. The maximum score a study could obtain by meeting the quality criteria, which are illustrated in five quality questions, is 5 points. The threshold is 2.5.

3.5. Data Extraction

The data extraction form was filled out using the relevant data of each selected study. To maintain ease of subsequent data synthesis, we grouped the items in the data extraction form based on definition, sustainability constructs, assessment methods, e-health solution, validation, and limitations.

Table 5. Data extraction form.

1. Data Extraction Log

Data extractor Data reviewer

2. Paper Information

Study ID Title Author(s) Publication year Publication type (conf/journal) Database Summary

3. Extracted Data

3.1 Sustainability Definition

Definition(s) Target (General sustainability definition or specific definition for e-health solutions?) Source (whether proposed or cited) Focus/scope of definition (e.g., technical, health provision, stakeholders, etc.)

3.2 Sustainability Aspects

Number of constructs The aspects Details/notes Extraction method (literature, case study, etc.) Target (general/customized)

3.3 E-health Solution

E-health solution type Focus of e-health solution Replace an existing process? (Y/N) – if no, it means it works to align with a process, or provides a new service. Status (running, proposed, etc.) Users (e.g., patients, nurses, physicians, staff, etc.) Benefits

3.4 Assessment Methods

Existence of assessment method (yes/no) Method (model, framework, diagram, etc.) Type of the assessment methods (standards, formula, checklist, etc.) List of assessment methods Used for which aspects? Source (literature, proposed, etc.)

3.5 Evaluation/Validation

Evaluated/validated? Method (with experts, case study, etc.) Number of iterations Duration Context

3.6 Additional Information

presents the data extraction form.

3.6. Data Synthesis

Data synthesis was conducted on the extracted data in order to analyze the results and draw conclusions. The method used for synthesizing the data was narrative synthesis [22]. By using this method, the data was tabulated in a consistent manner in three main categories: sustainability definitions, sustainability dimensions, and sustainability assessment methods. After that, the data was visualized using tables.

4. Results

In this section, we present the findings of the SLR. First, an overview of the selected studies is presented. Then, the findings are discussed in terms of e-health sustainability definitions, dimensions and elements, and assessment methods. The study selection process took place in February 2021 and then updated in November 2022.

In the first search round, a total of 4811 papers were obtained by applying the search string in three chosen digital libraries (i.e., Web of Science, IEEE Xplore, and Scopus) in addition to 16 studies retrieved by the snowballing technique, as shown in Figure 2. The title and abstracts of all papers that appeared in the search results of the selected libraries were checked for relevancy, and if found to be relevant, they passed to the second round. Of the 4811 papers, 125 papers passed to the second round, that is, the reading of the entire paper (see Figure 1), as many papers found were deemed irrelevant. This is due to the polysemy of some keywords in the search string, such as the term “long-term”, that was used in this SLR as a synonym of sustainability. Nonetheless, “long-term” is used in the healthcare context to refer to the technologies that are used to help people with chronic diseases and need a “long-term” healthcare provision. Therefore, such papers were excluded in the first round.

Simultaneously, the backward snowballing approach was conducted in the first and second rounds to collect relevant studies that were not included in the digital libraries’ results. In the second round, all 125 papers that passed the first round were read fully and then compared against the inclusion and exclusion criteria. Only 22 out of 125 papers passed the second round. The rest were eliminated because they did not meet the inclusion and exclusion criteria presented in Table 2 and Table 3. The third round was conducted, and all 22 papers passed the quality assessment.

4.1. Overview of The Search Results

The primary studies reviewed in this work are presented in

Table 6. The final set of the relevant studies reviewed in this work.

Study ID	Ref.	Title	Database	Year	Citation Index
PS1	[23]	A global socio-economic-medico-legal model for the sustainability of longitudinal electronic health records - Part 1	Web of Science	2006	79
PS2	[24]	A global socio-economic-medico-legal model for the sustainability of longitudinal electronic health records Part 2	Web of Science	2006	79
PS3	[10]	A multi-phased QFD based optimization approach to sustainable service design	Web of Science	2016	124
PS4	[7]	A Qualitative Study of Sustainability and Vulnerability in Australian Telehealth Services	Web of Science	2010	45
PS5	[25]	Assessing the suitability of smart technology applications for e-health using a judgment-decomposition analytic hierarchy process approach	Web of Science	2020	6
PS6	[26]	Challenges, Alternatives, and Paths to Sustainability for Health Information Exchange Efforts	Web of Science, Scopus	2013	87
PS7	[27]	Complexities in securing sustainable IT infrastructures in Hospitals: The many faces of Local Technical Support	Web of Science	2010	16
PS8	[28]	Efficacy and sustainability of a telerehabilitation program	IEEE	2003	16
PS9	[29]	EMR continuance usage intention of healthcare professionals	Web of Science	2017	64
PS10	[30]	How to Sustain Smart Connected Hospital Services: An Experience from a Pilot Project on IoT-Based Healthcare Services	Web of Science	2018	8
PS11	[31]	Information systems in healthcare - state and steps towards sustainability.	Web of Science	2009	18
PS12	[32]	Information Technology in Healthcare: HHC-MOTES, a Novel Set of Metrics to Analyse IT Sustainability in Different Areas	Web of Science	2018	14
PS13	[33]	Organizational Dynamics of Sustainable eHealth Implementation: A Case Study of eHMIS	IEEE	2017	3
PS14	[34]	Proposed model for evaluation of m-health systems	IEEE	2015	2
PS15	[8]	Sustainability requirements for connected health applications	Web of Science	2018	21
PS16	[35]	An evaluation of eHealth systems implementation frameworks for sustainability in resource constrained environments	snowball	2015	29
PS17	[36]	Business Models for Sustained eHealth Implementation: Lessons from Two Continents	Snowball	2012	10
PS18	[37]	Clinician acceptance is the key factor for sustainable telehealth services	Snowball	2014	369
PS19	[38]	From health technology assessment to health technology sustainability	Snowball	2018	11
PS20	[9]	Towards sustainability of health information systems: How can we define, measure and achieve it?	Snowball	2007	33
PS21	[39]	Exploring factors that affect the uptake and sustainability of videoconferencing for healthcare provision for older adults in care homes: a realist evaluation	Web of Science	2021	5
PS22	[40]	Identification of critical quality dimensions for continuance intention in m-health services: Case study of onecare service	Scopus	2019	105

. The studies discuss the sustainability of different e-health solutions, such as telehealth and electronic health records (EHRs).

Table 7. Primary Studies and Proposed E-Health Solutions.

Study ID	E-Health Solutions
PS1, PS2, PS6, PS9	Electronic health records (EHRs), electronic medical records (EMRs), Health information exchange (HIE)
PS3, PS14	m-Health
PS4, PS8, PS18	Telehealth
PS5	Smart e-health applications
PS7, PS11, PS13, PS20	Hospital/health information systems (HIS), Electronic Health Management Information System (eHMIS)
PS10, PS15	Internet of Things (IoT) technology, connected health applications
PS12, PS16, PS17, PS19	General e-health systems and application

demonstrates the primary studies and the domain e-health solutions discussed in each paper.

4.2. The Sustainability of E-Health Solutions

In this section, the results related to each research question are presented. The first subsection provides the existing definitions of sustainability of e-health solutions. The second and third subsection discusses the sustainability dimensions and elements, and assessment methods.

4.2.1. Sustainability Definitions

Five studies defined sustainability, in which most of them were cited definitions. The scope of the definitions was centered around demand, continuity, and cost. Three studies [8,9,35] used the general sustainability definition that is provided by the United Nations (UN): “meets the needs of the present without compromising the ability of future generations to meet their own needs” [41]. Both Wade et al. [7] and Garde et al. [9] defined the sustainability of e-health solutions as the ability of the solution to continue a defined behavior and functioning into the future. Moreover, Ouhbi et al. [8] and Fanta et al. [35] referred to sustainability as continuous use, and addressed the three dimensions of sustainability and other important factors of the intended system. On the other hand, Lenz [31] associated sustainability of IT systems in healthcare with the cost needed for replacements over the years; “a sustainable IT systems in healthcare should prevent the necessity of costly system replacements every five to ten years” [31].

4.2.2. Sustainability Dimensions

The studies discussed the sustainability of e-health solutions by presenting some of its dimensions or aspects. In the data extraction, the constructs of the sustainability mentioned in the studies were categorized into dimensions and aspects. The dimensions are the essential pillars of sustainability, which are high-level categories that involve a set of aspects, which influence the sustainability of e-health solutions. The aspects are a fine-grained level of detail of the dimensions that include elements and items through which data can be collected and sustainability evaluation is enabled.

Table 8. The mentioned sustainability dimensions of e-health solutions.

Dimension	Study ID
Technical	PS2, PS15, PS17
Economic	PS13, PS15, PS17, PS19
Social/Human	PS5, PS17, PS13, PS15, PS19
Organization/Management/Environment	PS3, PS4, PS6, PS8, PS10, PS11, PS12, PS13, PS15, PS16, PS17, PS20
Political/legal	PS1, PS2, PS5, PS6, PS13, PS17, PS19
Resourcing	PS4, PS5, PS8, PS13, PS18

presents the dimensions mentioned in the studies, which are technology, economic, social/human, organization/management/environment, political/legal, and resourcing.

Table 9. The sustainability aspects of e-health solutions as mentioned in the studies.

Aspect	Study ID
Functional suitability	PS14, PS13, PS16
Performance efficiency	PS12, PS13, PS14, PS16
Usability	PS12, PS13, PS16, PS22
Security	PS14, PS13, PS16, PS22
Modifiability/changeability, accessibility, reusability	PS12
Interoperability	PS11, PS13, PS14, PS16
Portability	PS12, PS13, PS16, PS20
Flexibility	PS16, PS13
Reliability	PS12, PS16, PS13, PS22
Scalability	PS16
Accuracy	PS13, PS16
Completeness, consistency, data accessibility, confidentiality, relevance	PS13
Timeliness	PS13, PS16
Semantic interoperability	PS1, PS20
Electromagnetic compatibility, power, physical parameters, data exchange, compatible architecture	PS14
Mobility and unobtrusiveness	PS5, PS14
Supporting online social networking	PS5
Attitude	PS9, PS16
Perceived ease of use	PS9, PS14, PS16
Perceived usefulness	PS9, PS14, PS16
User satisfaction	PS9, PS3, PS8, PS16
Confirmation	PS9, PS16
Trust in using e-health, readiness to make changes, human computer interaction, benefit in learning, satisfaction and willingness, users’ technology adoption, awareness about e-health	PS16
Acceptance (patients and clinicians)	PS8, PS14, PS18
Social	PS5, PS17, PS13, PS15, PS19
Workforce availability	PS3, PS18, PS21
Technical support	PS7, PS16, PS13, PS12
Technical training	PS8, PS3, PS20, PS16, PS13
Physical structures	PS13, PS3
Network management and communication	PS14
Environment (ecological: natural resources)	PS15, PS12
Support and commitment of top management	PS4, PS6, PS8, PS10, PS13, PS16, PS21
Organizational structures	PS4, PS8, PS13, PS16, PS20
Process workflow and process integration	PS13, PS17, PS11
Laws and standards related to ICT and e-health	PS16, PS2, PS5, PS6
Financial policy of e-health	PS16, PS12
Agreement on clinical content	PS20
Privacy and security policy	PS16, PS1
Data management	PS14, PS2, PS13, PS16
Stakeholders’ involvement/engagement	PS4, PS8, PS13, PS16, PS22
Direct benefit to users in routine work	PS16
Incentives and recognition	PS3
Organizational culture of electronic information use	PS16, PS13
(Outcomes of the solution are defined by the organization itself and needed to be evaluated regularly. They vary whether they are clinical, financial, technical, or social outcomes)	PS8
Sufficient demand and establishment of need	PS4, PS5, PS8, PS16, PS18
User and organizational readiness, eGovernment policy, efficiency of legal systems in settling disputes, intellectual property protection, procedures to enforce contracts	PS16
Policy	PS17, PS20
Available and sustainable funding	PS4, PS6, PS16, PS20
Economic feasibility	PS10
Affordability of e-solution, return on investment	PS16
Cost-effectiveness/cost minimization	PS16, PS8
Revenue and reducing the cost of data archiving	PS2, PS8

contains all aspects mentioned in the studies.

Two studies [8,35] discussed the issue of sustaining the implementation of e-health solutions and identified similar sustainability dimensions, which are social, environmental, economic, technological, and individual dimensions. It is worth noting that the focus of the environment dimension in both studies was different. In the study of Ouhbi et al. [8], the focus was on the sustainability from an ecological perspective, by raising environmental issues such as energy consumption and emissions of carbon dioxide. The study underlined how the e-health solution should consider this aspect in order for it to be sustainable. The same conception of the environment dimension was found in study [32]. On the other hand, study [35] focused on the work and organizational environment, such as organizational culture, values, leadership, and resources.

Joshua et al. [26] concluded that some issues impact the sustainability of health information exchange (HIE). These are financial, political, and organizational issues. In their study, they discussed the strengths and weaknesses of organizational models used for HIE in New York State, such as regional health information organization (RHIO). In addition, they provided suggestions to ensure the sustainability and effectiveness of exchange efforts. Financial issues were perceived as upfront costs, decreased profits, and funding, while political and organizational issues were summarized as the difficulty of cooperation and collaboration among healthcare organizations (e.g., other competitive hospitals). The authors suggested that existing public/government funding and codifying health laws that enforce and facilitate the HIE would help in solving these issues. Another study [30] investigated the sustainability of a pilot project for smart connected health services that used Internet of Things (IoT) technology and were provided for both patients and health professionals. The results showed that in order to sustain smart connected hospital services, it is necessary to consider the technical feasibility, economic feasibility, and organizational commitment and support.

Iandolo et al. [38] argued that the focus of healthcare technology assessment should be shifted from efficiency and effectiveness to sustainability. By proposing a reference framework, they stated that the sustainability of the technology can be achieved by assessing three aspects, along with considering the relations and interactions between them [25]. These aspects are economic, social, and political. Garde et al. [9] analyzed the inhibitors and enablers that influence the sustainability of health information systems (HIS). They proposed a high-level sustainability framework based on four pillars, which are clinical, technical, socio-technical, and political/business. Wade et al. [7] investigated 54 telehealth services in Australia to understand the reasons behind the continuing and ceased services, and to assess their sustainability. The results revealed that insufficient demand, issues with technology, funding, loss of senior management support, and loss of the interest of clinicians are the failure reasons among the ceased telehealth services. On the other hand, the success factors of sustainable healthcare services are sufficient demand, available funding, and developing organizational structures that help to integrate telehealth into their regular operations. Furthermore, they demonstrated the importance of routinization and integration of the telehealth service, especially when it is executed within an existing clinical unit. Moreover, they stated that in the public sector, the funding is not a major issue; as long as the need and the support could be demonstrated, then the funds would follow. Additionally, the sustainability of a telehealth application called TeleRehab was discussed by Scheideman-Miller et al. [28] and concluded that six factors impact the sustainability of TeleRehab. These factors are organizational strategy, establishment of need, organizational stakeholders, organizational commitment, human factors, revenue/cost savings, and outcomes. Organizational strategy addresses the customer, clinical operations, financial, and clinical development for health care providers and the organization. The human factors involve acceptance, long-term commitment of healthcare providers, specialized training, and user satisfaction. In addition, the outcomes of the e-solution vary whether they are clinical, financial, technical, or social outcomes. Therefore, they should be defined by the organization itself and need to be evaluated regularly. It was stated that a partnership of collaborators within an organization, including clinical, technical, and administrative professionals, will help in ensuring the success and sustainability of a telemedicine program.

Similar dimensions were mentioned in other studies that presented frameworks and methods to assess e-health sustainability. Fanta et al. [35] evaluated the contribution of four frameworks for the implementation of e-health systems in a resource-constrained environment. The evaluation was carried out in terms of four sustainability dimensions. These are social, environmental, economic, and technological. They underlined that it is essential to consider the alignment of social, economic, and environmental factors with the technology factor to ensure the sustainability of e-health. In a following study [33], Fanta et al. presented the topic of sustaining the e-health implementation and emphasized the importance of the technological, social, organizational, and economic dimensions in this regard. The study proposed a conceptual framework for the sustainability of e-health implementation that focused on both organizational and technological dimensions. The organizational dimension comprises some aspects such as resources (i.e., human, physical, and financial resources), workflow process, and management support, while the technological dimension encompasses system quality, information quality, and service quality. Furthermore, they underlined the techno-organizational processes that are essential for the success of e-health implementation. These processes are user training, data management, stakeholder engagement, project management, and organizational communications. Remondino [32] proposed a framework that includes five dimensions, which are management, organization, technology, environment, and social. The framework helps in the long-term assessment of the sustainability of IT applications and projects in healthcare and their implications on the healthcare organization, with respect to the five mentioned dimensions.

With regard to engineering requirements, Ouhbi et al. [8] have demonstrated the need for defining sustainability software requirements that are not focused only on the energy consumption of the software. Instead, they developed a design catalog called SCH-CAT for sustainable connected health applications. The catalog contains five dimensions: economic, individual, social, environment, and technical; all of these have their own sustainability requirements except for the economic dimension, whose requirements were not covered because they are not considered inherent requirements. The catalog was used in the study to evaluate the sustainability of a connected health app for blood donation. Dyk et al. [36] discussed two models that aim to assist in the successful implementation and sustainability of e-health solutions. The technology, users, work processes, finances, and policy categories were included in both models, as they contribute to the sustainability of e-health solutions. Along with that, Chowdhury et al. [10] proposed an optimization integrated fuzzy QFD methodology to determine the optimal strategies that can be used to mitigate a service’s barriers and achieve an organization’s sustainability goals. They emphasized three dimensions that must be considered and balanced for sustainable service design. These dimensions are social, environmental, and economic. The proposed methodology was applied to an m-health services setting in Bangladesh, by which lists of barriers and strategies to mitigate them were identified regarding m-health service sustainability. The results of the study emphasized the importance of training the service providers and collecting customers’ feedback to improve the service quality. Furthermore, they stated that mitigating a service’s barriers has a substantial impact on the sustainable health service.

Moreover, some studies argued that user acceptance is significant to sustain e-health solutions. Wade et al. [37] developed an explanatory model for the telehealth services’ sustainability. In their work, they argued that the acceptance of clinicians is the key factor that influences the sustainability of telehealth services. The model shows how other components, such as workforce availability, clinician demand, adequate technology, and resourcing are related to clinician acceptance. The two factors, clinician demand for the service and adequate technology, impact the clinician acceptance of the service, which in turn influences the clinician workforce availability and leads to the telehealth service being sustained. To achieve clinician acceptance, the study suggested two strategies, legitimation and relationship building. Legitimation consists of promoting positive beliefs about telehealth among clinicians, while relationship building is concerning with maintaining good relationships with those who provide the care, both of which are shown in the model as parts of champions promoting telehealth. The continuous usage intention of electronic medical records (EMRs) among healthcare professionals was evaluated using technology continuance theory (TCT) in [29]. With continuance of usage being one factor of sustainability, the study focused on the six acceptance constructs that influence continuance usage. The six constructs are confirmation, satisfaction, perceived usefulness (PU), perceived ease of use (PEU), attitude, and continuance intention. Spies [34] identified the concerns of m-health implementation when sustainable use is the intended goal for the system. The concerns were discussed in terms of the patient’s and health practitioner’s perspectives (e.g., acceptance, usefulness, wearability, and quality of service), as well as the considerations of hardware and software design (e.g., compatibility, data management, functionality, and interoperability).

Technical, political, and management dimensions were the focus of other studies. Petersen [27] addressed the role of technical support for hardware, software, and people-ware in sustaining the hospital information system (IS) and information technology (IT) infrastructure. Lenz [31] stated that two goals must be considered equally for the sustainability of healthcare information systems, which are process integration and systems integration. Chen and Wu [25] proposed a judgment-decomposition analytic hierarchy process (JD-AHP) approach to help decision-makers in assessing the sustainability of smart e-health applications, such as the smart body analyzer. The main factors that critically influence the sustainability of a smart technology application are unobtrusiveness, supporting online social networking, relaxation of the related medical laws, future e-health market size, correct identification of need and situation. Shabo [23,24] discussed the sustainability of electronic health records (EHRs). The first study discussed sustainability issues of three different models of EHR from management (i.e., conflicts of interests and access restriction for some special cases), legal (i.e., patients’ bill of rights), and technological (i.e., semantic interoperability standards) aspects. Furthermore, the study proposed the Independent Health Record Banks (IHRBs) model that would help in sustaining the her, while the following paper [24] discussed sustainability considerations of IHRBs, which are legal and ethical (e.g., authentication, access control, privacy, data integrity), business (i.e., revenue and reducing the cost of data archiving), and technological (i.e., information exchange standards). These considerations act as guidelines for the new legislation needed for IHRBs to be established and operate successfully.

4.2.3. Evaluation/Assessment Methods

Eight of the primary studies [8,25,29,32,33,34,35,36] discussed a method, a framework, a model, or a theory to evaluate the sustainability of e-health solutions. These are the JD-AHP approach [25], the technology continuance theory (TCT) model [29], the HCC-MOTES framework [32], a conceptual framework to sustain e-health implementation [33], a model to evaluate m-health systems [34], the SCH-CAT catalog [8], a conceptual framework for e-health systems in a resource-constrained environment [35], and a model for sustaining telemedicine projects [36]. The e-health solutions mentioned in the studies are smart e-health applications, EMRs, information technology in healthcare, the Electronic Health Management Information System (eHMIS), m-health systems, and telehealth.

Table 10. Tools and assessment methods proposed by the primary studies.

Study ID	Method of Assessment	Type of Assessment Methods	Source	E-Health Solution	Evaluated/Validated in Real Context?
PS5	AHP method	Mathematical equations	Proposed	Smart e-health applications	No
PS9	Model	Theory	Technology Continuance Theory (TCT)	Electronic medical records (EMRs)	Yes, evaluated
PS12	Framework	Qualitative metrics and KPIs	Existing literature and validated sets	Information Technology (IT) in Healthcare	No
PS13	Conceptual framework	NA	Proposed	Electronic Health Management Information System (eHMIS)	Yes, evaluated
PS14	Model	Standards	Clinical and Laboratory Standards Institute (CLSI) “Institute of Electrical and Electronics Engineers Standards Association (IEEE)” “International Electrotechnical Commission Technical Report (IEC/TR)” “International Organization for Standardization -Technical Report (ISO/TR)” “German Institute for Standardization (DIN)	m-health systems	No
PS15	Catalog	Checklist of software quality requirements	Elicited from several sources in software engineering	Connected health apps (e.g., telemedicine, m-health, etc.)	Yes, evaluated
PS16	Conceptual framework	Qualitative indicators	Literature and international agencies’ reports	e-health	Yes, evaluated
PS17	Model	Qualitative feedback and quantifiable scale	Combination of two existing models (CeHRes roadmap and TMMM)	Telehealth	Yes, evaluated

presents a summary of the proposed evaluation/assessment methods.

Chen and Wu [25] identified several factors that are critical to the sustainability of technology applications in healthcare. By using the JD-AHP method proposed in their study, the priority of critical sustainability factors in two or more technologies are compared by a decision-maker to determine which technology is more likely to be sustained. Using 10 smart technologies in e-health, the proposed method was compared against existing methods, which are AHP, OWA, and MACBETH. However, as stated in the study, this method works for one decision-maker, if more were involved, then the sustainability problem becomes considerably complicated.

Gilani et al. [29] used the TCT model to evaluate the continuance usage intention of an EMR system among users (i.e., healthcare professionals) in the post-adoption phase. In their study, the hypotheses discussed the positive effect of satisfaction, attitude, perceived usefulness, and perceived ease of use for healthcare professionals on their continuance intention to use the EMR system. In order to test the hypotheses, a survey was carried out on 195 respondents consisting of various healthcare professionals from five hospitals. Results showed that all hypotheses in the TCT model were supported, which indicates the significant power of TCT to explain EMR system continuance usage intention of both short- and long-term users. As indicated in their study, TCT will help policymakers and managers in the healthcare sector to understand the factors that result in the ongoing use of an EMR system after the adoption phase.

Remondino [32] developed a framework, named HHC-MOTES, that contains a novel set of key performance indicators (KPIs) to evaluate the sustainability of healthcare IT projects in terms of five dimensions (management, organization, technology, environment, and social). The framework aimed to provide a qualitative measurement of various aspects related to the sustainability of IT applications in healthcare. It is stated in the study that the HHC-MOTES framework can be used by healthcare decision-makers for strategic decisions regarding introducing or updating IT systems or examining current systems and identify limitations and shortcomings. However, it worth noting that the mentioned KPIs of all dimensions, other than the technology dimension, were focused on assessing the positive impact of IT systems on the efficiency of the healthcare system. For example, in the environment dimension, the indicators evaluated the reduction in paper use and energy consumption as a result of using the IT system. However, the technological KPIs discussed the aspects that should be considered to ensure the sustainability of IT systems in healthcare. Examples of these indicators are modifiability, reusability, portability, and supportability of the IT system.

Fanta et al. [33] developed a conceptual framework that describes how organizational and technical factors influence the sustainability of eHealth implementation. The framework has four sections. These sections are (1) inputs, which contains organizational and technological factors, (2) processes, which deals with activities executed to achieve the desired outputs, (3) outputs, which consists of intention to use, actual use, and user satisfaction, and (4) outcomes, which represents the organizational benefits (i.e., improving the decision-making process and internal communications in the organization). By using a causal loop diagram (CLD), the study presented the dynamic interactions between organizational factors and the technological dimensions. Additionally, the study drew insights from the literature about measuring the technological aspects. For system quality, it could be measured by ease of use, functionality, reliability, flexibility, portability, integration, and importance of the system [42,43,44]. The measures of information quality consist of accuracy, timeliness, completeness, relevance, and consistency of information produced by the IT system [42,43,44,45]. Moreover, the service quality measures include up-to-date hardware and software, reliability, responsiveness, assurance, and empathy of technical support [43,44].

The model proposed by Spies [34] aimed to be used as an evaluation method for developed and under-developed m-health systems to assess the sustained use of them. The model focuses on evaluating aspects related to the user (i.e., patients and physicians) and system design (i.e., hardware and software), each of which are intended to be measured by endorsed standards. The standards are included in Table 10.

As requirements engineering plays a vital role in software sustainability, Ouhbi et al. [8] designed the SCH-CAT catalog for software quality requirements that are critical to the sustainability of connected health (CH) applications. Although the SCH-CAT catalog targets the design phase of CH applications, a checklist can be generated from it, which includes a set of requirements to be used to evaluate the sustainability of CH applications in the implementation phase. The catalog was applied to assess the sustainability of a running CH application called Blood Donor+. A questionnaire containing requirements as checklist questions was used to evaluate the CH app, in which the answer is one of the three options: yes (+1), no (+0), and partially (+0.5). The overall score was calculated and the threshold score that determines the app’s sustainability was set to be 50%.

Based on a literature survey, Fanta et al. [35] constructed a conceptual framework that can support the efforts of developers, researchers, managers, and policy makers to ensure the implementation of sustainable e-health systems in resource-constrained settings. In the study, the framework was used to assesses the contribution of four e-health frameworks. The framework consists of four sustainability aspects, namely, social, environmental, economic, and technological, all of which contain indicators that can be used to evaluate the sustainability of e-health systems.

Dyk et al. [36] discussed two models, the TeleMedicine Maturity Model (TMMM) and the CeHRes roadmap, that were demonstrated to be effective in facilitating, implementing, and sustaining telemedicine projects. The CeHRes roadmap serves as a guide to the establishment of a telemedicine service, whilst the TMMM is concerned with maturation of the existing telemedicine service. In the study, the authors proposed a new model by combining these two models. In the proposed model, it was suggested that each maturity level of TMMM serves as a stage gate for each of the phases of the CeHRes roadmap. Maturity levels are initial, managed, defined, quantitatively managed, and optimized. The maturity levels are used to evaluate the maturity categories, which are technology, users, work processes, finances, and policies. By applying the proposed model to an e-health solution, the resulting maturity level of each category determines its sustainability.

Iandolo et al. [38] argued that the focus of healthcare technology assessment should be shifted from efficiency and effectiveness to sustainability. They proposed a reference framework containing efficiency and effectiveness dimensions to achieve the sustainability dimension of healthcare technology. They defined efficiency as the plans (i.e., things are done in the right way), effectiveness to be the goals (i.e., the right things get done.), and sustainability as when the right relationships exist with other service systems. As they indicated, efficiency can be measured with reference to the structure, whilst the effectiveness of a system is measured with reference to its specific context and in relation to the general environment; sustainability therefore can be achieved.

5. E-Health Sustainability Model

Based on the SLR results, we built an E-Health Sustainability Model. The process of building the proposed e-health sustainability model is presented in this section, which includes the phases of building the model, the definitions of the sustainability aspects, and the validation of the model.

5.1. Phases of Designing and Validating the E-Health Sustainability Model

Figure 3 presents the phases of building the model. Based on the SLR results, the extracted sustainability aspects were grouped into five dimensions according to the categorizing criteria (refer to

Table 11. Aspects’ categorizing criteria.

Dimension	Criteria of Aspect Categorization
Technology	Any aspect that fits under the technology category is an external entity/attribute that is associated with the new technology itself and does not depend on or belong to the organization. The aspect in this category should contribute directly to the overall soundness of an e-health solution that is under evaluation OR be a part of hardware or software of an e-health solution that is under evaluation (i.e., it could be input/output, components, properties, or attributes, etc.).
Organization	The aspects in this category are associated with the readiness of the organization to adopt an e-solution. The aspect regulates, defines, and impacts the way an e-solution is adopted in the organization.
Resources	Any aspect belongs to this category represents or belongs to an organization’s resources that affect the deployment of an e-solution. The aspect should contribute to the operating environment of the e-health solution AND help in maintaining acceptable level of using e-health solution.
Social	The aspects belong to this category is concerned with the user interaction and use of an e-solution. The aspect should directed to the user of e-health. contribute to assessing the users’ perception and behavior towards the use of the e-health.
Economic	The aspect in this category is associated with the financial affordability and benefits associated with an e-solution. The aspect should contribute to covering the costs of adopting and operating e-health solution. concern with the financial benefits gained from adopting and using the e-health solution.

) to form the e-health sustainability model. These dimensions are technology, organization, resources, social, and economic. The categorization of the dimensions was deduced from the literature, as they were mentioned to be the main dimensions of e-health sustainability [10]. In addition, to make sure that the grouping of dimensions is consistent and comprehensive, we used qualitative coding analysis where each aspect/dimension mentioned in the SLR results was given a code describing its content. Hence, if a dimension was mentioned in a number of papers with almost similar definitions and if the given codes were similar, such as in technology, we added a dimension named technology. On the other hand, if two dimensions were given two similar definitions but one definition is more general, we checked the codes and grouped them into one group or separate groups accordingly. For example, regarding the political/legal dimension, it was mentioned as a dimension in some papers. However, because its definition overlaps partially with the organization’s definition, we included political/legal as an aspect under the organization dimension (as in PS15 and PS16). The significance of representing the sustainability of e-health solutions through multiple dimensions arises from the fact that focusing on only one dimension is insufficient for evaluating software’s sustainability [46]. Following that, the extracted aspects (refer to Table 9) were categorized into the five sustainability dimensions based on the criteria of each dimension that are presented in Table 11. As shown in

Table 12. Developed definitions of sustainability dimensions.

Dimension	Definition
Technology	The technology dimension includes the aspects that determine the quality of the system itself regardless of the context it is applied to [35].
Organization	The organization dimension is concerned with the aspects that influence the readiness of the organization to adopt the system and accommodate the changes that follow the adoption [35].
Resources	The resources dimension involves all aspects that facilitate the installation of the e-health solution and assist the e-health to operate and be used at the optimal level [32].
Social	The social dimension comprises all aspects that are related to the users’ perspectives and behaviors towards adopting and using the system [32].
Economic	The economic dimension is related to the funding directed to e-health solutions, and the financial benefits gained from using the system [35].

, the definitions of the dimensions were obtained from SLR studies and were used as guidance to the development of the categorization criteria that is used to map the aspects with dimensions in this study. Each dimension is explained in detail in the following sections.

Figure 4 illustrates the three levels (i.e., dimensions, aspects, and elements) of the sustainability model. Dimensions are in the first level of the sustainability model that cannot be eliminated as each one of them influences the sustainability of e-health. Each dimension contains aspects, which represent the second level in the model. The fulfillment of these aspects is reflected in their dimension and indirectly impacts the sustainability of e-health. In the third level, some of the aspects contain elements that may, potentially, be used as tools to assess the sustainability of e-health. The importance of having the dimensions decomposed into aspects and elements is to support a fine-grained analysis of the sustainability level in future. After that, the model was validated by experts in healthcare. The validation phase is described in Section 5.3.

5.2. E-health Sustainability Model

This section describes the dimensions of the e-health sustainability model. Some of the reviewed papers mentioned the dimensions explicitly. For the rest, a coding analysis was run on the extracted dimensions/aspects and then grouped based on repetition and relevancy. Figure 5 presents the e-health sustainability model with all its components. Technology, organization, resources, social, and economic dimensions are explained in detail in the following subsections.

5.2.1. Technology

The technology dimension contains three aspects: system quality, data quality, and hardware quality. All these aspects are concerned with the quality of the e-health solution. The system quality aspect refers to the soundness of the e-solution, which comprises the system’s functional suitability, performance efficiency, usability, security, maintainability, compatibility, portability, flexibility, reliability, scalability, and integrability. The existence of these elements ensures that the system operates efficiently and therefore contributes positively to the solution sustainability. Similarly, the data quality aspect refers to the quality of the content information produced by the e-health solution. Elements of this aspect, such as data confidentiality, accuracy, and completeness are critical in the context of e-health solutions, as they deal with sensitive health data such as drug prescriptions and patients’ information. The hardware quality aspect refers to the suitability of the hardware components of the e-health solution to the users’ needs and comfort. Some elements of this aspect are the amount of electrical power consumed by e-health and the degree of mobility it provides to the user. This aspect is mostly applicable to e-health solutions that are related to m-health contexts, or in case of the existence of special hardware components required to operate the e-health (i.e., hardware components used directly by the users, such as a microphone to dictate medical reports). Although some unmentioned elements in the technology dimension can be used to determine the quality of a software product, the elements outlined in the model were found in the literature to significantly affect the sustainability of e-health solutions.

5.2.2. Organization

The organization dimension addresses changes introduced to the organization because of the adoption of the e-health solution. The organization dimension is important for the sustainability of the e-health solution because it ensures the availability of plans, policies, and practices that streamline the workflow integration of the e-health solutions and acceptance of users. In the results of the SLR (refer to Table 8), some studies (i.e., PS1, PS2, PS5, PS6, PS13, PS17, PS19) represented the political/legal as aspects while in other studies they were mentioned as dimensions. As mentioned in Section 5.1, because the definition of political/legal overlaps partially with the organization’s definition, we included political/legal as part of Planning and Policies aspect under the organization dimension. The aspects of this dimension are support and commitment, organizational structures and strategy, planning and policies, stakeholders, the culture of electronic information use, outcomes identification, and demand and need.

The support and commitment of the top management and their awareness of the importance of adopting the e-health solution are essential to its sustainability. The importance of this aspect is that the top management always has the final decision on whether to continue running and funding the e-health or not, and whether to enforce the changes associated with the e-solution or not. The absence of their support may result in an unplanned suspension of the e-solution; hence, the money and resources that were allocated to it are wasted. Similarly, the support and commitment of the e-health solutions’ vendors to continue providing the services impact the sustainability of the e-health solution. Therefore, it is necessary to consider this when assessing sustainability.

Furthermore, having well-established plans and policies that deal with the e-health regulations provides a solid foundation to adopt and sustain any e-health solution. In addition, it is the responsibility of the healthcare organization to identify the outcomes of adopting and using e-health and to have an evaluation plan to assess these outcomes regularly. Moreover, determining the need and demand on e-health plays a key role in its sustainability because low demand or unfulfilled needs will be a waste of the organization’s resources, leading to stopping the e-health solution.

5.2.3. Resources

The resources dimension is concerned with the aspects needed for the continuity of e-health’s operation. These aspects are human resources, financial resources, availability of technical support and training, and the readiness of the ICT infrastructure within the healthcare organization. Human resources refers to the availability of enough users to use and support the e-health solution to provide the healthcare service or complete its tasks. Even with the existence of needs and demands for e-health, a shortage in human resources will affect the work process and cause the e-health solution to be suspend or stopped. The financial resources aspect refers to the availability of financing that supports the operation of the e-health solution, such as covering the expenses of purchasing a new system feature or upgrading the software license to cover even more features, such as technical support and maintenance. The technical support and training aspect involves the technical support and maintenance for the e-health solution’s components and users, and providing them with the necessary knowledge to use the e-health solution. Maintaining the soundness of the e-health solution and technical skills among users will contribute to using the system, hence, a higher sustainability of the e-health solution is expected. The readiness of the ICT infrastructure is reflected in the e-health performance and therefore has an impact on its sustainability, because even with an efficiently designed e-health solution, its capabilities will not be optimally utilized if the ICT infrastructure within the organization is not ready to operate the e-health (e.g., no steady networking system, no sufficient servers’ cabalists, or outdated equipment.), hence affecting its performance.

5.2.4. Social

The social dimension consists of all elements that are related to the users’ perceptions towards the e-health solution. These elements were grouped into two aspects, which are the continuous usage intention and the user behavior. The intention of the users to continue using e-health must be highly considered, especially if the regulations do not enforce the use of the e-health solution. This interrelates with the human resources aspect in the resources dimension, having enough users using the system to achieve its outcomes. Furthermore, users’ behavior towards e-health influences the work process and e-health productivity. The users who trust the use of e-health and have the requisite skills and training are most likely to be ready to make the necessary adjustments corresponding to the adoption of e-health, which contributes to the achievement of the outcomes and goals of the e-health solution.

5.2.5. Economic

The economic dimension contains three aspects, which are the affordability of the e-health solution, sustainable funding, and financial benefits. The affordability of e-health refers to the degree to which all expenses of the e-health solution are covered. This is not limited to the technical expenses but extends to operational costs such as staff salaries and required equipment. However, even in the case of the capability of affording the e-health solution, it is of great importance to have sustainable funding for the e-health solution. Therefore, the sustainability of funding must be considered and ensured when approving the e-health affordability, as it is not a one-time activity. The financial benefits aspect concentrates on the direct or indirect pecuniary or material benefit that is obtained by implementing and using the e-health solution [47]. This can be in a form of direct cost-saving (cost reduction) or indirect cost-saving through operative, regulative, or social aspects, that eventually affects the financial business aspects [47]. This aspect can interrelate with the outcome’s identification aspect in the organization dimension. However, it is worth noting that it depends on the healthcare organization’s goals and objectives for using the e-health solution. In other words, if the healthcare organization does not consider financial benefits in the adoption of e-health, then this aspect can be eliminated in the context of the healthcare organization.

5.3. Experts Evaluation

To verify the validity and the comprehensiveness of the e-health sustainability model, interviews were conducted with three experts in IT from three different hospitals. One hospital is in the eastern province of Saudi Arabia and the other two are in the western province of Saudi Arabia. The experts are healthcare providers and staff, who have been working for an average of 10 years in the healthcare industry. They have direct experience in managing the adoption of e-health solutions within their hospitals. The sampling technique used is judgmental sampling [48]. The duration of the interview discussions was from 45 to 60 min.

The objective of meeting experts was to collect their feedback on (1) the validity and correctness of the dimensions and their elements and aspects, (2) the comprehensiveness of the model in terms of representing the e-health sustainability dimensions, (3) and the applicability of each dimension and their aspects to assess the sustainability of e-health solutions. In terms of the validity and comprehensiveness of the proposed model, the three experts agree that the model is comprehensive, and the construct is correct. In addition, two experts suggested (1) adding software integration as an element in the system quality aspect in the technology dimension, and (2) considering the commitment and support of the e-health vendor as an element that impacts the sustainability. Accordingly, these two aspects were added to the model as the experts had suggested. In terms of the applicability, the experts found the sustainability model applicable in real-world cases of evaluating sustainability. However, one expert suggested that there has to be a list of tools to assess each dimension and its aspects in order use the model in practice.

6. E-Health Sustainability Assessment Method

To use the model for identifying the sustainability levels of digital solutions, an assessment method was designed. It consists of the tools used to evaluate the model’s aspects and elements, and the assessment procedures that propagate the evaluation of the model’s elements to compute the overall sustainability level.

Saputri and Lee proposed a framework for evaluating software sustainability from a software engineering perspective, which focused on analyzing software artifacts [46]. Their framework consists of four major phases that combine static analysis, dynamic analysis, and machine learning (ML) approaches. The evaluation results determine the sustainability achievement of the aspects and which aspects require improvements. This is carried out based on the baseline scenarios (i.e., below, same, and above) defined by the knowledge experts according to the context. Other work carried out by Abdellatif et al. [49] focuses on software product maturity. In their work, they developed a Technical-Capability Maturity Model Integration (TCMMI) framework to assess the software product quality, which consists of two components: a reference model and an assessment method.

A study conducted by Siedler et al. proposed a model that assesses companies’ digitalization maturity level [50]. These levels are explorer, beginner, intermediate, and expert, with a scale from 1 to 4 for the digitalization maturity level (ML_total). ML_total is calculated by obtaining the arithmetic mean of the achieved maturity level of the four product lifecycle phases and the Corporate Level [50]. The maturity level of the four product lifecycle phases and the Corporate Level is achieved through calculating the mean of the belonging criterion digitization that is specified in [50].

6.1. Sustainability Evaluation Tools

The elements of each aspect are assessed by either checklist or survey questions. Both checklists and surveys have a five-point Likert scale. The scales ranges from strongly agree, agree, neutral, disagree, to strongly disagree, which is equivalent to the numerical scale 5-1, respectively. The questions that were used to assess the model elements were gathered from different resources found in the literature. It is worth mentioning that most of the resources focus on evaluating software products in general, and in this work, they have been adapted for the context of e-health solutions. To illustrate, some of the questions in the evaluation tools were altered in terms of the wordings to fit the context of e-health solution evaluation. Furthermore, the questions’ focus of some aspects was deduced from the literature, and based on that, the questions were developed by the author.

Table 13. Types of evaluation tools of each sustainability aspect.

Dimensions	Aspects	Type of Evaluation Tool	Customization	Evaluation Frequency	References
Technology	System quality	Checklist	Change of wording	On system change or update	[51,52,53]
	Data quality	Checklist	Change of wording	On system change or update	[54]
	Hardware quality	Checklist	Developed *	On system change or update	[34]
Organization	Support and commitment	Checklist	Change of wording	Annually	[55]
	Organizational structures and strategy	Checklist	Change of wording	Once	[19,55]
	Planning and policies	Checklist	Developed *	Annually	[55,56,57,58]
	Stakeholders	Checklist	Developed *	Annually	[19,55,56,59]
	Culture of electronic information use	Checklist	Change of wording	Once	[60]
	Outcome identification	Checklist	Developed *	Annually	[28]
	Demand and need	Checklist	Developed *	Annually	[55]
Resources	Human resources	Checklist	Developed *	Annually	[19,56]
	Technical support and training	Checklist	Change of wording	Annually	[19,55,61,62]
	ICT infrastructure	Checklist	Change of wording	Annually	[35,55,59,62]
	Financial resources	Checklist	Developed *	Annually	NA
Social	Continuance usage intention	Survey	Change of wording	Periodically	[29]
	User behavior	Survey	Change of wording	Periodically	[63,64,65]
Economic	Sustainable funding	Checklist	Developed *	Annually	[7,26]
	Affordability of e-solution	Checklist	Developed *	Once	[35]
	Financial benefits	Checklist	Developed *	Annually	[47]

* Developed by the author based on the aspect’s definition.

shows how each aspect’s questions were customized.

In addition, some aspects require regular evaluation (e.g., every year or after a major change/update on the e-health). The importance of regular evaluation is that some changes might happen to the e-health solution after the assessment phase, and thus the predicted sustainability level may change as a result. Furthermore, with the rapid changes in demands on healthcare environments, the goals and needs might change quickly, hence, a deployed e-solution might not be the optimal solution anymore. Thus, it is better to assess the sustainability regularly in order to avoid wasting money and other resources, or to invest more in highly sustainable systems. The sustainability assessment is expected to be run at least annually. Table 13 shows the types of evaluation tools for each aspect along with their evaluation frequency.

6.2. Procedures of Sustainability Assessment

The designed sustainability assessment method in this research was found to be aligned with the work of Abdellatif et al. [49]. The e-health sustainability model is comparable to the reference model of the TCMMI framework, in which the quality attributes in their work correspond with the sustainability aspects, and the product quality dimensions correspond with sustainability dimensions. Hence, we designed the sustainability assessment method to be aligned with previously published work (TCMMI) [49], and it is also aligned with the ISO/IEC 15504 standard (Guidance on performing an assessment) [66]. The sustainability assessment method propagates the scores of each level of the sustainability model to produce the overall sustainability level of the e-health solution. All assessment method procedures and steps are explained in the following subsections. Figure 6 presents the main phases of sustainability assessment procedures.

6.2.1. Assessment Team

The ideal assessment is conducted by at least three expert assessors: one expert from the IT department, another from the finance department, and one from hospital management. The inclusion of experts from the three domains will ensure a high level of accuracy and validity in the sustainability assessment [46]. The IT expert shall assess the technology dimension, the finance expert shall assess the economic dimension, and the hospital management expert shall assess the organization, resources, and social dimensions. It is worth noting that the assessment of the social dimension must be completed by the users of the e-health solution. Therefore, the selection of the reflective sample is important for results’ reliability. Furthermore, as all three expert assessors work in one team, each expert shall help in assessing aspects of other dimensions that are related to their specialty.

6.2.2. Assessment Steps

The e-health sustainability model consists of three levels, as demonstrated in Figure 4. The assessment tools are applied to the elements (third level) of the sustainability model. The assessment results of the elements will be reflected in the aspects’ fulfillment of achieving sustainability, which in the end contributes to predicting the overall sustainability of the e-health solution. Figure 7 shows the assessment steps, which are explained in the following subsections.

• Step 1: Dimensions’ Weights

As the dimensions represent the first level of the e-health sustainability model, they will be given weights based on the assessors’ judgment. The given weights reflect the importance of each dimension according to the context in which e-solution is going to be used [25]. For example, if the proposed system is going to be used by healthcare providers, the social dimension becomes very important compared to a situation where the system is going to be used by the IT department only. The weight of each dimension must be between 0 and 1, in which the sum weights of all dimensions must add up to one.

Because the assessment will be conducted by a team, it is recommended to use the group analytic hierarchy process (AHP) method. This method assists the team to decide on the weights that will be assigned to each dimension in the sustainability model [67].

• Step 2: Selection of Sustainability Elements

Software products vary and there is no unified context of use for all of them [49]. Therefore, the elements under each aspect will be selected based on their applicability to the e-health solution’s context. The selection of aspects and elements is carried out by the assessment team and must be based on the applicability, not the availability, of the aspects. For example, if the e-health solution should be used with special hardware components, then the hardware quality aspect in the technology dimension is applicable to be a part of the evaluation, even if the required information about the hardware quality is not available. In the case where hardware quality is required but the relevant data is not available, the hardware quality aspect will give a very low score that means its assessment is not fulfilled. To differentiate between the assessment of an aspect with a low score that results from unavailable data and low scores resulting from available data, when experts decide that an aspect is relevant, they also have to indicate whether data is going to be available for the evaluation or not. Hence, if data can be collected or it is already available, the experts can proceed with filling out the survey or the checklist; otherwise, the aspect will be assigned the lowest score until data becomes available and then assessment is run again.

After selecting the applicable elements, the checklists and surveys will be completed by the assessment team and the overall sustainability level will be calculated based on the following steps and formulas.

• Steps 3–7: Calculating Evaluation Score

As each aspect has a set of five-point Likert scale questions, to determine the score of each aspect, the following equation is applied:

$$\frac{\sum \mathrm{E}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n} \mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e} \mathrm{o}\mathrm{f} \mathrm{e}\mathrm{a}\mathrm{c}\mathrm{h} \mathrm{q}\mathrm{u}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n} \mathrm{o}\mathrm{f} \mathrm{e}\mathrm{l}\mathrm{e}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s} \mathrm{o}\mathrm{f} \mathrm{t}\mathrm{h}\mathrm{e} \mathrm{s}\mathrm{e}\mathrm{l}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d} \mathrm{a}\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{s}}{\#\mathrm{o}\mathrm{f} \mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l} \mathrm{q}\mathrm{u}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s} \mathrm{i}\mathrm{n} \mathrm{t}\mathrm{h}\mathrm{e} \mathrm{a}\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\times 5}$$

This step is applied to all aspects within the dimension. Lastly, the overall score of the e-health solution sustainability is the aggregation of the weighted scores of the dimensions, which is calculated by the below equation. It is worth mentioning that this assessment method does not include any mathematical model. The equations used are for weighting the scores of each aspect and dimension.

$$\sum \mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}\left(D_n\right)\times \mathrm{W}\mathrm{e}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}\left(D_n\right)$$

• Step 8: Mapping Evaluation Score to Sustainability Level

The sustainability levels indicate the prediction of the extent to which the e-health solution is considered to be sustainable. There are four levels for e-health sustainability: high, moderate, low, and unknown, all of which have a specific range of values that is used to map the aggregated score of the sustainability assessment to the corresponding level. As presented in

Table 14. Sustainability levels.

Scale of Sustainability Level
Level	Unknown	Low	Moderate	High
Overall description of sustainability levels	There is no/rare contribution of the dimensions to the overall sustainability	There is limited contribution of some dimensions only to the overall sustainability	Some of dimensions and their selected elements highly contributed to the overall sustainability	Most of dimensions and their selected elements contributed to the overall sustainability
Aggregated score (x)	x ≤ 35%	35% < x ≤ 60%	60% < x ≤ 85%	85% < x ≤ 100%

, the aggregated sustainability score that is less than or equal to 35% is mapped as an unknown sustainability level, while a low sustainability level is assigned to scores between 35% and 61%. The aggregated score that is between 60% and 86% is mapped as moderate level, and any score that is greater than 85% is mapped as a high sustainability level. Table 14 presents the four sustainability levels along with their threshold scale.

The levels and their scale values were inspired by Siedler et al., in which they used a scale from 1 to 4 to assess the maturity level of digitalization in a company [50]. We adopted the definitions that describe each digital maturity level (explore, beginner, intermediate, and expert) for the sustainability levels in the study (unknown, low, moderate, and high). Additionally, we converted their numerical scale to a percentage scale. Table 14 presents the resulting sustainability levels, their definitions, and the corresponding percentage scale. This aligned with the threshold scale that was proposed in another study conducted by Ouhbi et al., in which it was stated that the minimum score for a sustainability level is above 50% [8].

7. Case Study

The e-health sustainability model and its assessment method were evaluated using a case study conducted to assess the sustainability of a hospital information system (HIS) of a public hospital located in Saudi Arabia. The evaluation was in terms of comprehensiveness, effectiveness, and applicability to be used in real-world contexts.

7.1. Case Study Protocol

The required information was collected through an interview discussion with a pharmacy quality specialist who collected relevant data from other experts in the hospital, namely, a hospital management representative, and a hospital IT representative. The pharmacy expert worked with the other experts in the domain to collect the needed data. The sustainability model was introduced to them, and they answered the questions in each dimension, from which we calculated the dimensions’ scores. The interview focused on collecting the data needed to assess the applicable aspects of the five dimensions of the sustainability model. In addition, the expert was asked to provide weights for each dimension based on their importance and relevancy to the system and context under evaluation. The description of the gathered information is grouped below according to the dimensions.

7.1.1. Characteristics of the Case Study

The case study is applicable to be conducted on an e-health solution that is (1) proposed or already in use in a healthcare institution, and (2) used by staff in a healthcare institution, healthcare practitioners, or patients. The selection of the e-health solution is based on availability.

7.1.2. Units under Test

There are two units under test: (1) the sustainability model, and (2) the assessment method that is proposed to evaluate the sustainability of e-solutions in healthcare. The model is assessed in terms of its comprehensiveness in covering all essential dimensions and aspects that technology experts consider in practice. The assessment method is evaluated in terms of its applicability and effectiveness in anticipating the sustainability levels of e-solutions in healthcare.

7.1.3. Data Gathering

Qualitative data is gathered from interviews and discussions with stakeholders and technology experts in the hospital’s management, who are involved in the decision of adopting and using the e-health solutions. Two types of interviews were conducted, which are pre-implementation and post-implementation of the case study. The pre-implementation interviews’ questions focused on extracting information about the e-health solution to be used to evaluate the sustainability model and assessment method. The post-implementation interviews aim to gather stakeholders’ feedback on the evaluation’s results and to compare their sustainability evaluation, if it exists, to the sustainability evaluation generated by our assessment method. The social dimension is expected to be evaluated by the intended users of the e-solutions; hence, an online survey was designed and distributed to the users.

7.2. Case Study: Evaluating the Hospital Information System

The selected system (i.e., the hospital information system) had been running for six years, and it was adopted two years after the opening of the hospital. The main goal of adopting the system was to digitalize the work process in the hospital’s pharmacy, such as dispensing medicines and laboratory tests processes. Additionally, another intended use of the system was to manage healthcare records and information within the hospital. Hence, the system was expanded to allow physicians to write the prescriptions of medicines in the patient’s report, and place the order for medication with the hospital’s pharmacy. However, as stated in the interview discussion, the system was found to be insufficient regarding meeting the minimum users’ needs and allowing a full or efficient level of digitalization within the hospital, as some tasks were still paper-based.

7.2.1. Technology

The weight assigned to this dimension was 0.3, and the selected aspects were the system quality and data quality. The hardware quality aspect was inapplicable to the context of HIS as there was no specially designed hardware required to use this system. The system is operated and used through regular computer devices.

Regarding the system quality aspect, the system did not meet the users’ needs. Although the initial goal was to serve and facilitate the pharmacy work, after the system was adopted, it was found that it had limited features and the essential ones were not provided. This was because the adopted version of the system was the basic license that did not include customization or extension of the basic features. Furthermore, maintainability of the system was not fulfilled because after each update, some the systems’ permissions changed without the user’s request. To illustrate, in the medicines dispensing features, the input fields of the patient information was changed from mandatory to optional after each update. The users needed to ask for IT support to reset the input fields to mandatory. Therefore, some of the elements in this aspect were not fulfilled, which resulted in this aspect receiving an assessment score of 67%.

Regarding the data quality, the accessibility, confidentiality, consistency, relevance, and timeliness of the information produced by and stored in the system passed the evaluation. However, because the system had limited features, the produced information was not complete, nor sufficiently accurate to cover the needs of users’ tasks.

Table 15. Assessment results of case study.

Dimension	Dimension’s Weight	Aspects	Evaluation Score	Dimension’s Score	Dimension’s Weighted Score
Technology	0.3	System quality	67%	71%	21%
		Data quality	75%
		Hardware quality	NA
Organization	0.25	Support and commitment	20%	42%	10%
		Organizational structures and strategy	31%
		Planning and policies	69%
		Stakeholders	40%
		Culture of electronic information use	80%
		Outcome identification	30%
		Demand and need	20%
Resources	0.15	Human resources	65%	58%	9%
		Technical support and training	45%
		ICT infrastructure	100%
		Financial resources	20%
Social	0.05	Continuance usage intention	69%	67%	3%
		User behavior	64%
Economic	0.25	Sustainable funding	20%	35%	9%
		Affordability of e-solution	50%
		Financial benefits	NA
Total					52%

presents the evaluation score of each aspect that was achieved by filling out the assessment checklist for each of them.

7.2.2. Organization

The assigned weight for the organization dimension was 0.25. All aspects were selected as applicable. The top management supported the adoption of the system at first. However, this support was not sustained after the system was found to have limited features and more funds were needed to upgrade it. Furthermore, this system was supposed to work as a comprehensive health information system for the hospital. However, the features of the adopted basic version were limited to tasks in specific departments in the hospital, which hindered the integration of work processes. Although the IT department requested to purchase the HIS and got the approval on the basic version, they did not know that its features were limited to only pharmacists and laboratory tests without covering the physicians’ and nurses’ needs. This indicated a lack of proper planning, and that the important stakeholders were not involved in the planning and decision on the appropriate system to choose. Instead, the decision was taken based on the system with the lowest price and meeting the minimum users’ needs, without consulting all stakeholders. The culture of information use among users and in the organization was ready to adopt the system. In addition, the needs, demands, and outcomes of adopting the system were not identified and analyzed. Therefore, the goals of adopting the system were not achieved. Instead, some of the tasks in the hospital were still paper-based, because the system did not have all features that could replace the paperwork. The received score of each aspect in the organization dimension is presented in Table 15.

7.2.3. Resources

The resources dimension was assigned a weight of 0.15, and all its aspects were selected. The human resources aspect lacked the availability of individuals to provide technical support and training. The ICT infrastructure was fully ready to adopt the system. Furthermore, the technical support and maintenance was not available because the basic version of the system did not cover these services. New users did not receive any technical training. Furthermore, the financial resources were not available to upgrade the system to the full version. Table 15 presents the evaluation score of each aspect in the resources dimension that was achieved by filling out the assessment checklist for each of them.

7.2.4. Social

The weight that was given to the social dimension was 0.05, and all aspects were selected. Although the social dimension is assessed by the intended users through a survey, in this case study the survey was filled out based on the expert judgment. This was because reaching users in a short period was challenging.

Based on the estimation, the users were found to be unsatisfied with using the system and they did not perceive the usefulness of using it. This inconvenience was because the tasks were partially completed by the system, and the users were required to perform extra work to complete the tasks, which consumed more effort and resources. Although the users were ready to adopt the system, they did not trust the use of it, as it was not reliable to complete their work. As stated by the expert, if the current system continued to run despite all these limitations, the healthcare provision would be negatively impacted and the work productivity among healthcare practitioners and staff would decrease as their efforts were wasted on work processes that could be easily carried out with the full version of the system. Table 15 presents the aspects’ results of the social dimension.

7.2.5. Economic

The economic dimension’s weight was 0.25, and most aspects were selected, except for the financial benefits aspect, as the hospital was not interested in any type of financial benefits that could be provided by the system.

There was sustainable funding that covered the expenses of the basic version of the system. Although purchasing the basic version of the system was affordable, the accumulative cost, including upgrade cost, was not analyzed. Table 15 presents the aspects’ results of the economic dimension.

7.3. Sustainability Assessment Results

Based on the gathered data, the sustainability assessment method was applied to the HIS. Table 15 presents the dimensions’ weights and the assessment scores of the selected aspects after applying the assessment method. In the results, the aggregated score was found to be 52%. Hence, the sustainability level of the HIS is considered low according to the e-solution sustainability levels.

8. Discussion

Although sustainability is a critical issue in all sectors, it is more important in healthcare due to the complexity and sensitivity of this sector [8,24,31,33,68]. As e-health solutions are involved in many processes of healthcare provision, the sustainability of these solutions should be particularly considered by healthcare decision-makers and management. However, the results of this SLR indicated a significant scarcity of studies investigating the sustainability of e-health solutions and providing methods to assess and evaluate their sustainability.

The issue of not having a unified sustainability definition in healthcare was mentioned in most of the primary studies, yet their scope did not focus on resolving this issue or referring to other studies that tackled it. There exist some studies that focused on defining sustainability in healthcare [6,69]. However, the scope of these studies was defining the sustainability of healthcare innovation generally, and not focusing on e-health solutions. Therefore, a unified definition of the sustainability of e-health solutions remains a need, and further studies are required to investigate how e-health sustainability is perceived and defined in practice and in the literature. Based on the focus of the definitions elicited from the literature, our proposed definition of the sustainability of e-health solutions is the ability of e-health to continue meeting the expanding demand for facilitating healthcare provision through the cost-efficient use of digital technologies.

Moreover, the studies emphasized the importance of having multidimensions to assess e-health sustainability, as it is not enough to focus on only one dimension such as financial, environmental, or technology [7,32,38]. Correspondingly, the organization dimension was found to dominate more than the technology dimension and aspects related to e-health solutions [33]. This is because the same e-health solution can exhibit different levels of success when implemented in different healthcare organizations [33]. As the budget for e-health is a global issue in all healthcare systems, top management and decision-maker support is one of the organizational factors that help overcome this issue [25]. This is because when the top management supports the adoption of the e-health solution, then the funding will be readily facilitated [25], which contributes to its sustainability. These findings demonstrate the alignment of the dimensions in the developed model with the sustainability of e-health solutions.

Furthermore, it is important to consider the variation of the influences on sustainability among the dimensions, in which sustainability should not be an all-or-nothing concept [25]. Instead, each dimension impacts the sustainability of e-health to a different degree according to the context of the e-health solution [25]. Although all dimensions are intrinsically interconnected and must be considered as they influence each other [32], most studies that stated the dimensions of e-health sustainability did not explore the influence degree of dimensions on the sustainability and on each other, except for study [37], in which it argued that the clinical acceptance has the greatest influence on sustaining telehealth, which was the focus of the study. Determining the degree of influence of the dimension reflects the importance of each dimension and therefore helps to assess the sustainability of the e-health solution, which was not discussed in depth in the primary studies [38]. In addition, two experts mentioned that when evaluating sustainability using our proposed model, healthcare IT experts should decide on the weights of the dimensions and select the applicable aspects based on their judgment. The aspects and elements with high importance should be mandatory in assessment, while other aspects with less importance can be eliminated based on the context.

Some of the selected studies in this review have proposed a model or framework containing dimensions of e-health sustainability. However, all the mentioned evaluation methods were either too narrow, in that they were designed to evaluate a specific type of e-health (e.g., health connected applications), or too general and did not comprehensively evaluate the sustainability of the e-health solutions. Therefore, there is a need for a sustainability assessment method that comprehensively assesses all sustainability aspects of e-health solutions.

To cover a wider range of studies, we used the OpenGrey (http://www.opengrey.eu/ accessed on 7 November 2021) database to search for relevant papers in the grey literature. However, the papers found in the results were irrelevant to the focus of this SLR. For example, some papers discussed sustainability from a manufacturing perspective rather than focusing on the sustainability of digital solutions. Furthermore, some of them focused on software sustainability from developers’ perspectives, in which no relevant aspects can be extracted and used to reflect the sustainability of e-health solutions. The rest of the results discussed sustainability from an ecological perspective. In addition, we searched other types of documents (i.e., other than articles and conference papers) in the three libraries (Scopus, Web of Science, and IEEE), and no relevant documents were found. Therefore, the grey literature was not included in this review. We believe that our proposed sustainability model lays the foundation for developing a sustainability analysis and assessment method. Furthermore, there is a potential to use the sustainability model as a practical guide that supports healthcare institutions in reasoning about e-health solution sustainability.

The model and the assessment method were demonstrated to assess the sustainability level of an e-health solution (i.e., HIS) successfully. The sustainability assessment in the case study received a low score of 52%, which matched the actual status of the e-health solution. Before applying the sustainability assessment method, the gathered information indicated a low level of sustainability, which means the e-health solution will not be sustained. A similar sustainability evaluation was predicted by the sustainability assessment method, which means only few aspects contributed to the sustainability; hence, the e-solution will not be sustained in the long-term unless most of the performance in most of the aspects are improved. The results revealed the unsustainability risk of this e-health solution and highlighted the aspects in which improvements could be introduced.

Furthermore, it was observed that the organization dimension has the greatest influence on the other dimensions. This conclusion aligned with the results of the SLR, in which the organization dimension was frequently found in the studies to be an important dimension that influences sustainability the most. This was apparent in the case study, in which the organization dimension received a score of 42%. This score resulted from the unfulfillment of most of the organizational aspects, such as the top management support and stakeholders’ involvement. By this, the other dimensions (i.e., resources and economic dimensions) were negatively impacted, in that the funds were not provided, which impacted the availability of the resources. Furthermore, the unfulfillment of the resources and economic dimensions negatively impacted the social dimension, as users were not satisfied with using a system that had many limitations and did not meet their needs. Moreover, the lack of sustainable funding is an issue that directly impacts the sustainability of e-health solutions, because e-health solutions are improving rapidly and the demand for new changes and updates is increasing correspondingly. In order to keep up with the demand, adequate sustainable funding is required to continue the use of the system. The resistance to providing funds is hindering the efficient employment of technology in healthcare and increasing the burden on healthcare practitioners, which does not achieve the digital health transformation goals.

This work contributes to the body of knowledge by providing a comprehensive definition and model of the sustainability of e-health solutions that was derived empirically. In addition, it proposes an assessment method to assess the sustainability of e-health solutions. In practice, the proposed model and its assessment method could be used as a tool to support decision making regarding e-solution alternatives. Furthermore, it could be used to give an early estimate of whether an e-health solution would be sustainable or not. This work lays the foundation for many interesting research directions and industry innovations, such as automated estimation of the sustainability of products using AI and existing data. The model will be the base on which relevant data will be collected and analyzed. AI methods will replace the assessment method if sufficient data exist.

9. Limitations

Despite the perceived effectiveness of the proposed sustainability model and its assessment method, there are some threats that are worth mentioning. In terms of the SLR construct validity, the SLR protocol, especially the search string, might be biased or might not cover all relevant studies. However, to mitigate this threat, we had the SLR reviewed by an expert and the search string’s terms were extracted from relevant studies. Another threat was that some of the evaluation tools, surveys and checklists of the aspects were gathered from studies that discuss the evaluation of the intended aspects without relating it to the sustainability of digital solutions. To mitigate those risks, the model and the assessment method were validated by IT and domain experts in the healthcare field. Lastly, as there is no clear evaluation scale for the sustainability of e-solutions in healthcare in the literature, we proposed an evaluation scale, which might impact its validity. However, to mitigate this risk, the proposed scale was inspired from other studies that were published to assess the maturity level of the digitalization of companies, which is close to the subject of this study. For the internal validity, data extraction might be biased, or some relevant data might be missed, as the data was extracted by the researchers. However, having the template that includes what data to extract, and documentation of extracted data might reduce the risk if this issue (see Table 5). Additionally, some of the questions in the quality assessment are not decisive, as they depend on how the researchers interpreted the studies. Hence, quality assessment criteria were adopted from a published study. In terms of building the sustainability model, grouping the aspects and elements under the dimensions might be biased towards the authors’ understanding. To mitigate this risk, we used thematic and coding analysis where dimensions were given clear definitions and criteria, which were used to code and group the aspects and dimensions. In addition, three IT experts in the healthcare domain wholly validated the constructed model.

10. Conclusions

This study aimed to investigate e-health sustainability in terms of the definitions, dimensions, and aspects, and to develop a sustainability model and assessment method for e-health solutions. The results showed that empirical work related to e-health sustainability models and assessment is scarce. Hence, this paper proposes a sustainability model that encompasses all dimensions and aspects that impact the sustainability of e-health solutions. The proposed e-health sustainability model and assessment method support decision making in adopting new digital technology in healthcare. The sustainability model captures the dimensions and aspects of e-health sustainability and provides a systematic method to evaluate and predict the sustainability of e-health solutions. The model was constructed based on the dimensions and aspects extracted from the SLR conducted in this paper. The model was built and designed through multiple iterations and was validated, in terms of comprehensiveness and applicability, by three experts in healthcare, and their comments were accommodated to refine the model and produce the final and validated version. Furthermore, using the validated version of the model and its assessment method, a case study was conducted on an e-health solution to assess its sustainability. The assessment method considers the singularity of each healthcare context by allowing each dimension to be assigned different weights based on its importance in the context of the evaluation. The contribution of this work is that there is no existing sustainability model that opens the door for more automation. When practiced, this could be used to help professionals to evaluate the sustainability of e-health solutions. The findings discovered the need for more case studies and experiments on assessing e-health solutions. The sustainability model and its assessment are expected to provide a holistic evaluation and insights about the enablers and challenges in adopting e-solutions. In addition, they are expected to reduce wasted resources, as they support the assessment of the overall sustainability level of an e-solution before adoption or deployment.