1. Introduction
The one of the most common chronic disorders that affect people today is diabetes mellitus, and its incidence is quickly increasing around the world (According to WHO report on diabetes (2021), it has been mentioned that, in 2014, 8.5% of adults aged 18 years and older had diabetes, and in 2019, diabetes was the direct cause of 1.5 million deaths, and 48% of all deaths due to diabetes occurred before the age of 70 years). [
1,
2]. Nowadays, we are living in the era of smart technology in healthcare domain. As a result of mobile technology such as smart phones, wearable devices, and other smart devices, the use of mHealth applications is growing at an exponential rate. Hence, most poor countries have also embraced the use of mobile health applications to improve the delivery of health services [
3]. Establishing a specific treatment plan, taking medicines, and adhering to regular blood glucose (BG) monitoring and medical nutrition therapy are all critical steps in improving diabetes control [
4]. However, usability testing can be used by researchers to uncover flaws in the products and systems that exist and gain ideas about how to improve them. The ease with which a product can be controlled by users to achieve a particular objective in terms of efficiency, effectiveness, and satisfaction is referred to as usability [
5]. When it comes to T2DM mHealth apps, the main goal for users is to find reliable, accurate, and timely information with minimal effort. As a result, quality is a critical aspect for T2DM mHealth applications, which must be monitored in order to achieve the stated goal. T2DM mHealth applications, on the other hand, are built to deliver accurate, reliable, and adequate information with high efficiency and low effort. To do this, T2DM mHealth applications must be more user-friendly. The usability of five T2DM mHealth applications namely, Glucose Buddy, mySugr, Diabetes: M, Blood Glucose Tracker, and OneTouch Reveal, is evaluated in this paper. All five T2DM mHealth applications are part of the same platform and offer outstanding and widespread assistance to type 2 diabetic patients with diabetes self-management. As a result, T2DM mHealth applications should provide users with useful information that takes minimal effort and time. The usability of five T2DM mHealth applications that may meet quality criteria in a broad sense is evaluated in this paper.
A pilot analysis of the study was carried out for recognizing the usability attributes that should be included when evaluating usability, taking into account the various types of users who access the T2DM mHealth applications. For the assessment of the utility of T2DM mHealth applications, several criteria and sub-criteria should be met. All five T2DM mHealth applications are evaluated for usability in order to determine the most usable T2DM mHealth applications for achieving user expectations and demands.
From the perspective of the user, the best T2DM mHealth applications are determined by the total usability score for each T2DM mHealth application, which is in turn determined by the quality of the T2DM mHealth applications. MCDM approaches are used to make the selection process easier. Due to its capacity to use specific criteria to evaluate numerous alternatives, MCDM is an excellent methodology for evaluating and analyzing difficult and complicated real-world situations [
6]. The multi-criteria decision-making (MCDM) algorithms described in this work include TOPSIS, VIKOR, and PROMETHEE II. Criteria importance through the inter-criteria correlation (CRITIC) methodology is applied in this study. For evaluating the usability of T2DM mHealth applications, the CRITIC methodology is first utilized in order to find objective weights of all related criteria. CRITIC can effectively differentiate the properties of individual criteria by analyzing their respective strengths and evaluating criteria weights based on the relationships between them. The best alternative was chosen using a combination of criteria and sub-criteria. A variety of usability attributes and sub-attributes were employed as criteria and sub-criteria in this study. The main purpose of this study is to determine which of the T2DM mHealth apps is the most useful among the five alternatives—Glucose Buddy, mySugr, Diabetes: M, Blood Glucose Tracker, and One Touch Reveal—based on the ranking assigned by the different MCDM methods adopted in this paper, which include TOPSIS, VIKOR, and PROMETHEE II. The major contributions of this study are as follows:
Identifying T2DM mHealth applications’ attributes and sub-attributes.
Collection of feedback using questionnaire-based evaluation. Questionnaires are framed based on attributes and sub-attributes using a pilot study and expert opinions.
Obtain objective weights of all connected criteria using the CRITIC technique.
Evaluate the usability of the mHealth applications using three popular MCDM methodologies like TOPSIS, VIKOR, and PROMETHEE II.
Ranking the alternatives based on the obtained usability score.
Comparing the ranking of the alternatives across the three MCDM methods to check the consistency of the results.
The following is the organization of this paper.
Section 2 presents related works on usability evaluation and MCDM techniques. Then, the paradigm and methodologies proposed for the process of usability evaluation are presented in
Section 3. The results and analysis are discussed in
Section 4, and in
Section 5 the work is concluded.
2. Related Works
For plant site selection challenges and problems, the fuzzy technique for order preference by similarity to ideal solution (TOPSIS) methodology was introduced by Chu [
7]. Firstly, the decision makers’ ratings and weights are standardized to develop a scale that is comparable. For all alternative sites and for every rating that is normalized, the membership function is developed for each criterion. The ranking of the alternatives is determined by a closeness or proximity coefficient. Amiri et al. [
8] employed the TOPSIS technique in conjunction with heuristics based on fuzzy goal programming to discover the ideal position. The challenge of selecting the facility location is handled using three stages: (a) choosing the smallest range of possible distribution centers, (b) situating those in the largest location that is feasible, and (c) locating these facilities at the lowest possible cost. Under a fuzzy context, Lo et al. [
9] used TOPSIS to rank web services. They used five core criteria (cost, runtime, configuration management, transaction, and security). They also used 17 sub-criteria to choose the ideal web service based on the team’s performance. For evaluating and analyzing the overall performance of eHealth systems, Lohan et al. [
10] developed a MCDM model. The eHealth efficiency was evaluated by authors using five criteria that include ease of use, cost, acceptability, false alarm, and accuracy.
Grigoroudis et al. [
11] created a method for evaluating performance as well as a review system based on a balanced scorecard for public health care services. In order to preserve service quality, customer contentment, the organization’s self-improvement system, and the organization’s ability to adapt and evolve, both the financial and the non-financial performance metrics are included in this evaluation method. Researchers compared TOPSIS, VIKOR, and ELECTRE, the three multi-criteria decision-making (MCDM) methodologies, to see how well they could evaluate and assess the performance and the quality of 21 food companies [
12]. A further, comparative analysis of AHP, TOPSIS, and PROMETHEE, [
13] observed that certain decision-making approaches are still susceptible to confusion where a choice has to be made between two or more alternatives in relation to the subjectivity. Mahmoodzadeh et al. [
14] used a hybrid of fuzzy AHP and the TOPSIS techniques in order to develop a project-selecting strategy that employs the enhanced technique for obtaining the attribute or criterion weights, and then for ranking the projects, the algorithm of TOPSIS was chosen.
Gavade [
15] looked into a variety of multi-criteria decision-making situations. As examples of MCDM techniques for various cloud services, they studied AHP, TOPSIS, VIKOR, ELECTRE, and PROMETHEE. TOPSIS could be used for PaaS decision making, according to the authors. “The Secure use score to achieve system usability” was proposed by Peikari, H. R. et al. [
16] in 2018. Every characteristic in this measure was given a score by the authors. However, only three usability parameters were examined for this metric, and no case study was used to evaluate it. In 2019, Hsieh, M.H. et al. [
17] assessed the utility of the available three diabetic self-management smartphone applications. In this study, the usability of diabetes applications was also examined with the help of usability testing. The usability study involved a total of 30 participants having type 2 diabetes (15 men and 15 women). The participants’ average age was 60.03 years, having a standard deviation of 8.92 years. Jayant. A. et al. [
18] classified major works involving MCDM methodologies that include PROMETHEE, VIKOR, ELECTRE, and TOPSIS, and provided help in identifying the literature and future research opportunities. PROMETHEE, VIKOR, ELECTRE, and TOPSIS are all MCDM strategies that are studied and implemented in this research. Wang, F.K. et al. [
19] analyzed and enhanced six sigma projects for minimizing overall performance and quality shortfalls within every criterion and parameter, and created a hybrid MCDM model that included the DEMATEL technique, the VIKOR method, and the analytic network process (ANP).
Ghaleb, A.M. et al. [
20] established an approach for evaluating and comparing three different selection methods: the technique for order of preference by similarity to ideal solution (TOPSIS), VIKOR (stepwise procedure), and the analytic hierarchy process (AHP). This review procedure took into account the number of distinct techniques and criteria, efficiency in the decision-making procedure, computing difficulty, and sufficiency for the support of a group choice, and criterion of inclusion or deletion. This work included a case study to examine the evaluation process. The criteria or attributes employed to examine and selecting the best or the finest manufacturing process included accuracy, flexibility, quality, productivity, complexity, operation cost, and material use.
Wu. Z. et al. [
21] compared and evaluated four multi-criteria decision-making methods including the analytic hierarchy process (AHP), elimination et choix traduisant la realité (ELECTRE III), the technique for order of preference by similarity to ideal solution (TOPSIS), and Preference Ranking Organization Methods for Enrichment Evaluations (PROMETHEE II) for one sewer network decision made by the group in the early stages for sewer water infrastructure’s asset management. Furthermore, the Delphi methodology is used to manage and organize talks across all decision makers during the implementation of numerous MCDM approaches. PROMETHEE II is the most popular approach among decision makers; AHP takes much time and effort, and there may be several irregularities as a result; while vector normalization for multi-dimension criteria is being done, TOPSIS loses the information, and the results of ELECTRE III are unclear. Bratati et al. [
22] did a study related to the usability measures and various evaluation methodologies that aid in user satisfaction. The authors also explored a number of issues concerning usability and the need for usability models in the field of cloud computing. A survey was conducted by Roy and Pattnaik [
23] to evaluate various usability criteria and different evaluation methods used to assess the usability and acceptability of web apps and websites. Based on the relevance of the online internet facility for all sorts of users, the authors recommended two new measures to demonstrate the usefulness of websites, including device independence and assistance for people who are physically challenged.
Liew et al. [
24] in their research work conducted a qualitative study to provide a deep understanding of the usability parameter of different mHealth applications and also incorporate suggestions for upgrading the experience of users in terms of usability aspect. The authors explore the various alignments between the users and mHealth practitioners for conducting the study. It was observed that, based on the five major themes selected from 20 different applications, satisfaction is the top-ranked attribute, whereas intuitiveness was least preferred by the users.
Zhou et al. [
25] and other authors have conducted a research study on developing a reliable usability questionnaire, especially for evaluating the usability of mHealth app popularly known as MAUQ, which has three subscales and is tested in both standalone as well as interactive mHealth applications. The obtained responses were compared with the standard Post Study System Usability Questionnaire (PSSUQ) as well as System Usability Scale (SUS), which seem to be correlated with each other.
Islam et al. [
26] used a three-stage approach. The first stage was to do a keyword-based app search on the most popular app stores. The affinity diagram method was used, and the applications discovered were divided into nine groups. Secondly, four apps were chosen at random from each group (a total of 36 apps) and a heuristic evaluation was performed. Finally, in the third stage, the most downloaded app from each group was chosen, and user studies with 30 people were undertaken.
Isaković et al. [
27] studied the diabetes monitoring app named DeStress Assistant (DeSA), which was created as part of an EU project and evaluated in a hospital context. An assessment of an available diabetes app was carried out in two test trials with older users, utilizing various questionnaires. Since the number of older persons is rising, the app is designed with their population in mind. The app, which was built with the support of workshops and comments from tech-savvy patients and healthcare professionals, is challenging to use by elderly users, according to a number of supervised tests.
Georgsson et al. [
28] conducted a study to see if a multi-method technique of data collection and analysis for patients’ experience with a mobile health system for diabetes type 2 diabetes self-management is feasible. From a wider clinical trial, a random sample of 10 users was chosen. User testing involving eight typical tasks and the Think Aloud protocol, a semi-structured interview, and a questionnaire on patients’ experiences with the system were all used to obtain data. The results were structured, coded, and evaluated using the framework analysis (FA) approach and the usability problem taxonomy (UPT). After classification, a usability severity rating was applied.
Eberle et al. [
29] in their study reviewed the clinical effectiveness of mHealth apps in managing diabetes mellitus patients of type 1 (T1DM), type 2 (T2DM), and gestational DM. A system review was conducted from a literature review carried out between January 2008 and October 2020, which was categorized based on the type of DM and results obtained. A meta-analysis report was prepared to measure the impact of glycated hemoglobin (HbA1c) on the different types of diabetes mellitus mHealth apps.
Teng et al. [
30] studied how various authentication methods affect the usability of mHealth apps. Secondly, new metrics for evaluating ease of use were introduced, and thirdly, the usability of two prevalent authentication systems for mHealth apps was evaluated using numerous key process features and their influence on users. Based on the findings, a QR-code-based authentication method for mHealth apps was proposed, which would help users to overcome frequent barriers.
A systematic review of diabetes management apps for the iOS platform is described by Martin et al. [
31]. The KLM review revealed several usability difficulties related to data entry and personalized settings, while the heuristic evaluation revealed additional issues related to devising loss, aesthetics, learn ability, error management, and security.
Timurtas and Polat [
32] conducted a study to perform a comparison of the usability parameter of smartphone and smart watch devices that aims to help users suffering from type 2 diabetes as well as clinicians focusing on T2DM. Usability was measured by using the System Usability Scale (SUS), and a t-test was conducted to compare the scores obtained by SUS for both devices. It was observed that the usability of smartphone devices is higher when compared with smart watch devices, but overall both the devices have high usability scores (SUS score > 80.8) measured by users as well clinicians.
5. Conclusions
As we all know, the present doctor-to-patient ratio in our country is low, so it is very much essential to digitalize the health services. The only way to do so is through mHealth applications, which have become the medium for delivering health services in real time and also for facilitating consultation from remote locations. The use of mHealth applications for self-management of T2DM patients has saved various lives and has become the need of the modern world. However, all these facilities require efficient mHealth application with good user interface design for improving task effectiveness and user satisfaction. Thus, usability evaluation will surely help in identifying the best mHealth applications, which can, in turn, improve the health outcomes by improving patient experience of care and facilities provided, saving cost, and time while visiting a doctor’s office, facilitating real-time health monitoring, and other benefits. Hence, improving the usability and efficiency of the mHealth applications is essential for better customer satisfaction and trust. This research work identifies 10 criteria (attributes) and 29 sub-criteria (sub-attributes) based on the features of mHealth applications and expert opinions, which are used in questionnaire-based evaluations. For measuring the criteria weights, the CRITIC method was used. Data obtained from the feedback mechanism were then analyzed using three popular MCDM models, namely TOPSIS, VIKOR, and PROMETHEE II. All the three MCDM models were used to evaluate the usability score of five mHealth applications (which include Glucose Buddy, mySugr, Diabetes: M, Blood Glucose Tracker, and OneTouch Reveal) and provide the ranking of those alternatives. The key findings of the proposed research work are summarized below:
The relative proximity () obtained from TOPSIS method ranges between 0.271650464 and 0.748031419. The mHealth applications with the maximum () value is considered as the best alternative. Here, the mySugr application is considered as the best mHealth application among the rest.
The aggregating index () calculated from VIKOR method shows that the alternative having the least () value is the best among all. The () value of the five alternatives taken in our study ranges between 0 and 0.898883249. It was observed that the mySugr application has the least () value among all; thus, it is the best mHealth application as far as usability is concerned.
PROMETHEE II is used for calculating the net outranking flow (φ(i)) of T2DM applications. The (φ(i)) value obtained in our study ranges between −0.26391893 and 0.344439094. A higher (φ(i)) value indicates best alternative. From result analysis, it was observed that, in PROMETHEE II, the mySugr application is the best mHealth application with the maximum (φ(i)) value.
A comparison study was carried out among the three MCDM models to check the consistency of the result. It was observed that all the three models show almost the same ranking for the five alternatives, with the mySugr application as the best and the Blood Glucose Tracker as the least preferred application among the users.
This study proposes a very useful methodology for evaluating the usability score of mHealth applications and support decision making in the selection of the best mHealth applications focusing on T2DM patients. This research work is beneficial to patients for their day-to-day health monitoring and recording blood sugar levels, which can also help medical practitioners for further analysis.
Limitations: One of the limitations of the research study is the sample size of the population, which needs to be increased for better efficacy in the decision-making process. This is because the research work was conducted on rare chronic diseases, i.e., T2DM patients, whose population is less in a smaller region of the country, like Jharkhand. Another limitation of the research work is the number of alternatives, because with the increase in the number of alternatives, the MCDM methods may produce varied results.
Future Scope: The future scope of the research work is to develop a hybrid or novel model considering the popular MCDM methods that can provide better result in terms of accuracy. Special attention should be given on fuzzy-based MCDM approaches, which improve expert judgement by removing vagueness and human error. Moreover, feedback will be collected from all groups of users (especially medical experts/doctors/practitioners, etc.) to improve the effectiveness of the decision-making process.