Construction and Ranking of Usability Indicators for Medical Websites Based on Website User Experience

Abstract

In the era of digitalization, medical websites have rapidly expanded their healthcare market share due to their convenience. However, with this user-base expansion, issues with poor user experience have surfaced. To address this, we developed and ranked usability indicators for medical websites, aiming to improve their design and development from a user experience perspective, thereby improving user satisfaction and the website’s usability. Initially, we reviewed the relevant literature and summarized 30 usability indicators. Subsequently, we formed a Delphi panel of 20 experts and preliminarily identified 24 usability indicators through the Delphi survey method. Using data from 300 valid user surveys, we applied the Exploratory Factor Analysis (EFA) method to categorize these 24 indicators into four groups. Finally, we assessed the relative importance and priorities of these indicators using the Analytic Hierarchy Process (AHP) method. The results showed that, in terms of criterion layer weight priorities, Trust and Security (0.5494), Basic Performance (0.2710), and Features and Technology (0.1355) exhibited higher proportions. For the solution layer, Property Protection (0.1894), Credibility (0.1852), Privacy Protection (0.1194), Effectiveness (0.0932), and Findability (0.0579) exhibited higher weight proportions. The findings of this study will assist in future usability assessments and enhancements of medical websites. By optimizing the usability, we can both advance the digitalization of medical websites and improve the usability of medical websites, and enhance the service experience and satisfaction of your users.

1. Introduction

In the digital age, the Internet has become an indispensable dimension of global infrastructure [1]. In the field of healthcare, the internet has demonstrated unique value and potential. For example, with the widespread adoption of mobile internet technology, users can access medical information and a variety of services anytime and anywhere through mobile devices, significantly enhancing the accessibility and convenience of medical websites [2]. According to a report by GlobalMed, nearly three-quarters of millennials prefer the convenience and immediacy of teleconsultations over in-person appointments [3], indicating a substantial user market for telemedicine. Furthermore, telemedicine can provide better services for patients in regions with poor medical conditions, significantly alleviating the imbalance in medical resource allocation and improving public health levels [4]. Research by Gao, J., Fan, C., et al., highlights that telemedicine offers a feasible solution to the unequal distribution of healthcare resources, making it an increasingly popular option for bridging the gap in healthcare service capacity and quality between urban and rural areas [5]. During the COVID-19 pandemic, the public’s demand for online remote medical services surged dramatically. Governments and relevant departments launched policies to support the development of online healthcare services, leading to rapid market recognition and the acceptance of this emerging service model [6]. According to McKinsey & Company, online medical trends have stabilized 38 times higher than pre-pandemic levels [7]. In addition, the online medical market is expected to grow to USD 225 billion by 2030 [8].

Despite the number of medical website users rapidly increasing [9] and the market showing strong growth momentum, a series of existing issues have been exposed. These include disorganized medical web pages [10] on which users struggle to find the required information [11], insufficient user-friendliness [12], complex content information [13], and the lack of secure payment capabilities on medical webpages [14]. All these issues severely affect user experience and satisfaction.

According to research by Gale, J.J., and Black, K.C., the usability of online healthcare directly impacts user engagement and satisfaction, as well as the ability of the service to achieve its goals [11,15]. Therefore, to address these issues and enhance user experience, it is crucial to improve the usability of these medical websites. However, current research on medical websites mainly focuses on usability testing [16,17,18], their acceptance [19,20,21], and telemedicine services [19,22,23]. Although the aforementioned studies positively promote the development of medical websites, research on the development and prioritization of usability metrics for online medical websites from a user experience perspective is quite limited.

In light of this, the aim of this study is to develop a set of user experience-based usability metrics for medical websites. This involves systematically identifying and prioritizing key usability issues to address those that most significantly impact user experience. By doing so, we seek to enhance website utilization and enable a broader user base to access online medical services. The results of this study are anticipated to provide practical references and guidance for the design of future medical websites. Furthermore, by improving user experience, the study aims to increase user engagement and retention with online medical websites, thereby promoting the healthy development of digital healthcare.

2. Research Methodology

In this study, we selected usability indicators related to medical websites by analyzing the literature on website interaction, usability, web usability, and online medical services, we have compiled indicators of usability for medical websites. Then, the final indicators were determined using the Delphi method. Thereafter, dimensionality reduction was performed using Exploratory Factor Analysis (EFA). Finally, weights were assigned to these indicators using the Analytic Hierarchy Process (AHP). A flow chart of our methodology is shown in Figure 1.

2.1. Literature Review

2.1.1. Websites User Experience

With the technological revolution and the rise of the internet, websites designed with user experience in mind have been shown to significantly enhance user satisfaction and loyalty [24]. Consequently, the concept of user experience (UX) has become increasingly crucial [25].

According to ISO 9241-210 [26], user experience encompasses a user’s perceptions and responses before, during, or after using a product, system, or service. These experiences can be direct, as in the operation of device interfaces, or indirect, such as the feelings, thoughts, and perceptions elicited by interacting with a website [25]. Hussain et al. define user experience in terms of the emotions and behaviors people exhibit when interacting with a page [27]. In terms of websites, user experience refers to whether the website is easy to navigate, whether the information is clearly presented in an easy-to-understand language, and whether the design effectively supports users in completing tasks [28]. The research of Zlokazova, T., Blinnikova, I., et al. shows that the structure and format of the web page also affect user experience [29]. The studies by Casalo, L., and Flavian, C., suggest that considering user experience, website design should be simple, direct, and easy to use [30]. Alben’s research suggests that a website’s user experience is influenced not only by technical and objective factors but also by the user’s emotional state; thus, according to the literature review, user experience encompasses Effectiveness, Efficiency, Readability, Screen Design and Layout, and Satisfaction.

2.1.2. Usability

Usability was first introduced in the 1970s, with the concept varying among researchers and target groups [31].

IEEE Std.610.12 (1990) defines usability as an attribute that facilitates system input and output and makes it easy to learn how to operate the system [32]. Nielsen (1993) defines usability as how easily users can utilize system functionalities, setting five criteria for usability evaluation: Learnability, Efficiency, Memorability, Errors, and Satisfaction [33]. Duma and Redish (1993) define usability as the degree to which a user of a product can quickly and easily complete tasks, considering good usability as user-centered, effective, efficient, and above all easy to use [34]. ISO 9241-11 (1998) [35] describes it as the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use. Iwarsson et al. (2003) view usability as the ability of users to instinctively and effectively use a product, interface, or system [36]. Hu (2006) [37] considers usability as the degree to which specific users can efficiently, effectively, and satisfactorily implement a system, product, or service in a specific environment.

Numerous studies measure product usability based on these attributes, which serve as benchmarks in the evaluation process, such as those by Man Lee and Maeng Ho Kim on the usability of smart home apps [38] and Zhang Chi and Chung Gunjang on the factors influencing user experience of smartphone travel apps [39]. Thus, in this paper, we also use these standards as benchmarks for measuring the usability indicators of medical websites.

Moreover, the literature indicates that the concept of usability has evolved with time. Where it was initially mainly considered to be a product or system’s ease of use and learning, usability has now expanded to include personal evaluations and users’ subjective feelings. In particular, modern usability indicators not only focus on technical operability but also emphasize user experience, making the concept more comprehensive [40]. Therefore, in this paper, we also consider the characteristics of user experience in constructing usability indicators for medical websites.

2.1.3. Web Page Usability

Web page usability refers to the functionality of a website stemming from a design approach that focuses on user needs. In this area, a user-centered design process is employed to ensure that websites are efficient and easy to use for users [41].

Lindgaard (1994) defines the field of website usability evaluation and proposes usability assessment criteria for different website developmental stages. These criteria include Navigation, Screen Design and Layout, Terminology, Feedback, Consistency, Sensory Forms, Redundancy, User Control, and Task Conformity [42]. Richardson, B. and Campbell-Yeo, M., et al. describe the usability of a page through a framework of attributes, including usability, usefulness, desirable, findability, accessibility, credibility, and value [43]. Lee and Kozar propose 10 dimensions for assessing the usability and user experience of websites, i.e., Consistency, Navigability, Supportability, Learnability, Simplicity, Interactivity, Emotional Engagement, Credibility, Content Relevance, and Readability [44]. Through a series of three studies, Palmer, J.W. demonstrates that website usability and success are commonly associated with download speed, layout organization, information ordering, the type and amount of content, interactivity, customization, and responsiveness [45]. These indicators are widely used in web usability research. Therefore, in this study, we also adopted these assessment methods as the evaluation benchmarks for our research methodology.

2.1.4. Online Medical Services

Online medical services are a form of telemedicine that provides healthcare services via the Internet. Consultations and treatments are provided through online consultations for individuals who are unable or unwilling to visit medical facilities due to time or location constraints. These services can help treat various conditions and improve access to high-quality medical care in remote areas [46,47].

Medical websites are considered convenient and efficient platforms [48], similar to primary care providers [49]. This is because they enable patients to access required medical services from anywhere at any time. Moreover, medical websites can reduce costs, enhance patient engagement, and facilitate easier access to information resources [50].

However, as noted in a study by Meszaros, J. and Buchalcevova, A. [51], there are also downsides to Internet-based services, such as the risk of information leakage, financial risks, performance risks, and ineffective information. Christensen et al. believe that the credibility of medical websites determines patient choices [52]. Similarly, studies by Eysenbach G, Powell J, Bernstam EV, and Walji MF highlight consumer concerns about the quality of online health information [53,54]. In summary, we considered convenience, efficiency, privacy protection, property protection, effectiveness, credibility, and user engagement as the usability benchmarks for online medical services.

2.2. The Delphi Survey Method

The Delphi method was developed in the 1950s to obtain reliable consensus opinions from a group of experts through a series of questionnaires [55,56,57]. It has been applied in many health-related fields, including clinical medicine and public health research [58,59].

In the Delphi method, questionnaires are scored anonymously, and participants are encouraged to add new ideas, amend existing responses, or suggest the removal of redundant answers across several iterations until a consensus is reached [52,60]. Typically, a carefully selected anonymous panel of experts undergoes two to three rounds of structured surveys, concluding when consensus is achieved [61,62].

The Delphi method is a commonly used research technique, widely regarded as a part of survey research [63,64]. This paper chooses the Delphi method not only because it is extensively applied in health science research to determine priorities and reach consensus on important issues, addressing fundamental problems in healthcare [65], but also due to its features of anonymity, controlled feedback, flexible statistical analysis options, and the ability to gather participants from different geographical regions [66]. These characteristics enable experts from diverse fields to conduct a comprehensive and multi-faceted evaluation of the research subject.

The number of members in a Delphi panel is generally between 8 and 20 [65]. In the study by L., Taylor, H., and Reyes, H., the number of Delphi experts was 12 [67]. In the study by Zhang, Y., Hamzah, H., and Adam, M., the number of Delphi experts was 15 [68]. To maximize the diversity of the sample, in this Delphi survey, twenty experts were invited to participate, including eight industrial designers with over 5 years of experience, two physicians with over 10 years of experience, four web designers with over 5 years of experience, and six interaction designers with over 5 years of experience. The survey was conducted anonymously using the Questionnaire Star (wenjuanxing: https://www.wjx.cn/ accessed on 15 April 2024) software over two rounds on 20 April 2024 and 23 April 2024, with consensus among the experts being reached by the end of the second round.

2.3. Exploratory Factor Analysis

Exploratory Factor Analysis (EFA) was initially proposed by Charles Spearman in 1904 [69]. It is a multivariate technique that addresses questions related to the possibility that several underlying variables explain many individual variables [70].

This study employs Exploratory Factor Analysis (EFA) because it allows researchers to identify underlying dimensions or factors within a dataset and decompose items into discrete dimensions that can be summed or aggregated [71]. This aligns with the needs of the research, making EFA the chosen method.

In this study, we utilized the usability indicators derived from the Delphi method to create an online survey on the Questionnaire Star platform, collecting data from users with actual experience for the Exploratory Factor Analysis (EFA). This online survey was conducted from 1 May 2024 to 3 May 2024, during which 327 questionnaires were distributed. Subsequently, 300 were judged to be valid and 27 invalid. The survey explored users’ opinions on the importance of design indicators for medical websites. A Likert 5-point scale was used for measurement.

2.4. Analytic Hierarchy Process

The Analytic Hierarchy Process (AHP) is a popular group decision-making method that has been applied in various fields [72], including healthcare [73,74,75], education [76,77], and business [78,79]. The AHP is used to evaluate options, allocate resources, compare benefits and costs, and perform system management [80].

In the product development process, making the right decisions is crucial, as inaccurate decisions can lead to product redesigns. An effective tool for determining the most suitable decision-making scheme is the Analytic Hierarchy Process (AHP) [81]. According to the research by Nukman, Y., Ariff, H., et al. AHP has been applied in nearly all decision-related applications [81]. Therefore, this study employs the AHP method to continue decision-making regarding the importance of various indicators.

In this study, we first constructed a three-level framework of indicators, namely, the overall goal level, the criteria level, and the alternatives level. Subsequently, a pairwise comparison matrix questionnaire was developed based on the structural hierarchy. This questionnaire was administered to 15 industry experts with more than five years of experience [82], including two web designers, eight industrial designers, and five interaction designers. The Saaty 1–9 scale method [83] was employed to score the indices in the matrix. The questionnaire was distributed from 5 May 2024 to 7 May 2024, achieving a 100% return and efficiency rate.

3. Research Execution and Analysis

3.1. Derivation of Usability Indicators

Following the literature review on user experience, usability, web page usability, and online medical services, redundant and semantically similar indicators were removed. Ultimately, a set of 30 usability indicators was compiled, as shown in

Table 1. Indicators after Delphi questionnaire.

Compilation of Usability Indicators for Medical Websites
Indicator	Description	Indicator	Description
Effectiveness	Whether the website helps users successfully achieve their goals	Content Relevance	Whether the content is relevant to users’ needs and searches
Learnability	How quickly a new user can learn to use the website’s functions	Screen Design and Layout	How information is displayed on the screen
Efficiency	The time and resources required to complete tasks	Readability	Ease of understanding text, appropriate formatting
Controllability	The degree of control users has over website operations	Terminology	Understandability of professional medical terms used on the website
Memorability	Whether users can easily remember how to use the website	Responsiveness	Response speed and layout adaptability of the website on different devices
Task Consistency	Whether different parts of the website maintain task consistency	Feedback	Quality of system feedback after user actions
Error	Frequency and severity of errors encountered while using the website	Convenience	Convenience and ease of use of the website
Findability	Whether users can easily find the information they need	Consistency	Consistency of interfaces and operations across different pages
Satisfaction	Users’ satisfaction with using the website	Privacy Protection	How the website protects user information
Accessibility	Addressing the needs of people with disabilities and patients when accessing the webpage	Remote Presentation	Ability of the website to support remote services
User-Centered	Whether the website design considers users’ needs and experiences	Property Protection	Website safeguards users’ personal information and medical conditions
Interactivity	Facilitates effective communication between users and the platform	Customization	Providing different experiences based on individual needs and preferences
Easy to use	Ease of use of the website for users	Download Speed	Speed at which website content is downloaded to the user’s device
Credibility	Reliability and reputation of the website and its medical information	Unnecessariness	Whether the website has unnecessary content or features
Navigation	How smoothly users can move between systems and modules	Sensory Forms	Whether the web page design is visually appealing

.

Based on these indicators, we conducted a survey, the results of which are presented in the following table. Data analysis for this survey was performed using SPSS 27.0 (https://www.ibm.com/support/pages/downloading-ibm-spss-statistics-27, accessed on 30 April 2024). According to the research by Preece, J., Rogers, Y., and others, indicators were considered to have high consensus among experts when the p-value (p) was less than 0.05, the mean (M) was greater than 3.5, and the coefficient of variation (CV) was less than 0.3 [84]. Therefore, these criteria were used to evaluate the data from this survey. It is important to note that when calculating the coefficient of variation (CV), the standard deviation (SD) must be divided by the mean (M). Therefore, the SD values have been included in the table. After the first round of the Delphi survey, while the scores for Terminology met the standards, Readability was found to encompass the meaning of Terminology; therefore, Terminology was removed and not included in the second round of voting. Following the completion of the second round, a consensus was reached among the experts, and no third round of survey testing was conducted. Both rounds of the survey achieved a 100% response rate. The Delphi survey data are shown in

Table 2. Data from two rounds of the Delphi survey.

N = 20	Round One Delphi Data			Round Two Delphi Data
Indicator	M	SD	CV	M	SD	CV
Effectiveness	4.700	0.470	0.100	4.250	0.444	0.104
Learnability	4.750	0.444	0.093	4.250	0.444	0.104
Efficiency	4.600	0.503	0.109	4.150	0.366	0.088
Memorability	2.600	0.995	0.383	-	-	-
Error	3.650	0.813	0.223	4.100	0.447	0.109
Satisfaction	4.400	0.503	0.114	4.350	0.489	0.112
User-Centered	4.900	0.308	0.063	4.600	0.503	0.109
Easy to use	4.200	0.410	0.098	4.100	0.447	0.109
Navigation	4.250	0.550	0.129	4.000	0.324	0.081
Screen Design and Layout	4.150	0.745	0.180	3.750	0.550	0.147
Terminology	4.100	0.447	0.109	-	-	-
Feedback	4.150	0.745	0.180	3.800	0.410	0.108
Consistency	4.250	0.550	0.129	4.000	0.459	0.115
Remote Presentation	3.200	0.616	0.193	3.650	0.489	0.134
Customization	1.750	0.851	0.486	-	-	-
Sensory Forms	3.200	0.768	0.240	2.350	0.671	0.286
Unnecessariness	3.900	0.308	0.079	3.950	0.224	0.057
Controllability	4.050	0.394	0.097	3.900	0.447	0.115
Task Consistency	2.300	1.031	0.448	-	-	-
Findability	4.650	0.489	0.105	4.250	0.550	0.129
Accessibility	4.650	0.489	0.105	4.100	0.553	0.135
Interactivity	3.850	0.875	0.227	4.000	0.324	0.081
Credibility	4.900	0.308	0.063	4.900	0.308	0.063
Content Relevance	4.350	0.489	0.112	4.050	0.224	0.055
Readability	4.650	0.489	0.105	4.100	0.308	0.075
Responsiveness	3.750	0.910	0.243	3.500	0.607	0.173
Convenience	3.950	0.394	0.100	3.950	0.224	0.057
Privacy Protection	4.950	0.224	0.045	4.850	0.366	0.075
Property Protection	4.950	0.224	0.045	4.900	0.308	0.063
Download Speed	1.750	1.020	0.583	-	-	-

.

Ultimately, after two rounds of the Delphi survey and subsequent analysis, the indicators obtained are as shown in

Table 3. Usability indicators for medical websites.

Effectiveness	Learnability	Efficiency	Error	Satisfaction	User-Centered
Easy to use	Navigation	Screen Design and Layout	Feedback	Consistency	Remote Presentation
Unnecessariness	Controllability	Findability	Accessibility	Interactivity	Credibility
Content Relevance	Readability	Responsiveness	Convenience	Privacy Protection	Property Protection

.

3.2. Dimension Reduction and Naming of Usability Indicators

Before conducting an Exploratory Factor Analysis (EFA), it is essential to assess the validity and reliability of the data obtained from the online survey to ensure their suitability. A Kaiser–Meyer–Olkin (KMO) value greater than 0.7 and Bartlett’s test of sphericity significance value less than 0.05 indicate good validity [82]. A Cronbach’s alpha coefficient between 0.7 and 0.95 suggests good reliability of the scale [85,86].

Consequently, in this study, we calculated the KMO value, Bartlett’s test of sphericity, and Cronbach’s alpha coefficient for the survey indicators using SPSS 27.0, as shown in

Table 4. Results of KMO and Bartlett’s test result.

Test		Test Value
KMO		0.924
Bartlett’s	Approximate Chi-Square	4255.067
	Degrees of Freedom (df)	276
	Significance	0.00
Cronbach’s Alpha		0.921

. The results yielded a KMO of 0.923, Bartlett’s test of sphericity significance < 0.05, and a Cronbach’s alpha of 0.923. These results confirmed that the survey data were both reliable and valid, making it suitable for EFA analysis.

In the Exploratory Factor Analysis (EFA), the criteria for factor extraction were set such that only factors with eigenvalues (λ) greater than 1 were considered, and any factor composed solely of a single item was excluded. If the loading difference between two items on the same factor was less than 0.05, one of the items was removed and the analysis was re-run. Items associated with a factor with a communality of less than 0.4 or a maximum loading of less than 0.35 were also excluded [87]. After multiple rounds of selection and rotation, four common factors with eigenvalues greater than 1 were ultimately identified, and 24 indicators related to the usability of medical websites were retained.

The aforementioned methodology determined the proportion of each factor in the total variance, as shown in

Table 5. Total variance explained.

Total Variance Explained
Element	Initial Eigenvalues			Sum of Squared Loadings for Extraction			Rotated Sum of Squared Loadings
	Total	Percentage of Variance Explained	Cumulative Percentage of Variance Explained	Total	Percentage of Variance Explained	Cumulative Percentage of Variance Explained	Total	Percentage of Variance Explained	Cumulative Percentage of Variance Explained
1	8.547	35.612	35.612	8.547	35.612	35.612	4.472	18.632	18.632
2	2.683	11.177	46.789	2.683	11.177	46.789	4.075	16.978	35.611
3	2.655	11.063	57.852	2.655	11.063	57.852	4.072	16.968	52.579
4	2.264	9.433	67.285	2.264	9.433	67.285	3.529	14.706	67.285
5	0.614	2.560	69.845
6	0.589	2.453	72.297
7	0.565	2.356	74.653
8	0.508	2.117	76.770
9	0.490	2.044	78.813
10	0.460	1.917	80.731
11	0.435	1.814	82.545
12	0.419	1.748	84.293
13	0.408	1.699	85.992
14	0.391	1.629	87.621
15	0.384	1.601	89.222
16	0.362	1.508	90.731
17	0.340	1.415	92.145
18	0.316	1.318	93.463
19	0.310	1.290	94.754
20	0.286	1.192	95.946
21	0.270	1.125	97.070
22	0.263	1.096	98.166
23	0.225	0.936	99.102
24	0.216	0.898	100.000

. The scree plot illustrated in Figure 2 further elucidates the contribution of each factor to the total variance. The rotated component matrix in

Table 6. Rotated component matrix.

Rotated Component Matrix
	Element
	B1	B2	B3	B4
Effectiveness: C1	0.812
Convenience: C2	0.770
Easy to use: C3	0.768
Efficiency: C4	0.763
Error: C5	0.757
Satisfaction: C6	0.745
Learnability: C7	0.739
Feedback: C8		0.797
Navigation: C9		0.785
Screen Design and Layout: C10		0.782
Consistency: C11		0.781
User-Centered: C12		0.780
Interactivity: C13		0.776
Accessibility: C14			0.821
Remote Presentation: C15			0.820
Findability: C16			0.801
Unnecessariness: C17			0.783
Controllability: C18			0.782
Responsiveness: C19			0.726
Readability: C20				0.816
Credibility: C21				0.807
Privacy Protection: C22				0.797
Content Relevance: C23				0.789
Property Protection: C24				0.787

Extraction Method: Principal Component Analysis. Rotation Method: Kaiser Normalization Varimax Method.

provides a detailed depiction of the correlations between the extracted factors (such as B1, B2, etc.), and the indicators (such as Effectiveness, Convenience, etc.).

Based on the results of the rotated component matrix and the characteristics of the indicators in each dimension, we denoted the reduced common factors as follows:

The first group of common factors, B1, includes the following indicators: Effectiveness, Convenience, Usability, Efficiency, Errors, Satisfaction, and Learnability. These indicators primarily measure the basic experience and efficacy of users when using products or services. They relate to the fundamental functions of the product and are, thus, named Basic Performance.

The second group of common factors, B2, comprises the following indicators: Feedback, Navigation, Screen Design and Layout, Consistency, User-Centered, and Interactivity. These indicators focus on the quality of product design and aspects of user interaction and are, therefore, named Design and Interface.

The third group of common factors, B3, includes the following indicators: Accessibility, Remote Presentation, Findability, Unnecessariness, Controllability, and Responsiveness. These indicators focus on the technical and functional aspects of the product, ensuring that technological support meets user needs; hence, they are named Features and Technology.

The fourth group of common factors, B4, contains the following indicators: Readability, Credibility, Privacy Protection, Content Relevance, and Property Protection. These indicators relate to the trust and security users feel towards the product or service, including the protection of user information, the relevance, and the accuracy of content, and are, thus, named Trust and Security.

3.3. Calculation of Usability Metric Weights

3.3.1. Building the Hierarchical Model

The entire hierarchical model is divided into three levels: the overall objective layer, the criterion layer, and the solution layer, as shown in

Table 7. AHP hierarchical decision-making framework.

Overall Objective Layer	Criterion Layer	Solution Layer
Constructing Usability Indicators for Medical Websites A	B1	C1
		C2
		C3
		C4
		C5
		C6
		C7
	B2	C8
		C9
		C10
		C11
		C12
		C13
	B3	C14
		C15
		C16
		C17
		C18
		C19
	B4	C20
		C21
		C22
		C23
		C24

.

3.3.2. Constructing the Judgment Matrix

The hierarchical model was input into the YAAHP 10.1 (https://www.metadecsn.com/yaahp/, accessed on 30 April 2024) software to conduct verification based on the AHP hierarchical model. Following the verification of the hierarchical model, a judgment questionnaire was developed. The scoring criteria for the questionnaire are shown in

Table 8. Judgment matrix scoring criteria.

Saaty’s 1–9 Scale Assignment Method
Scale	Meaning
1	Indicators i and j are of equal importance.
3	Indicator i is slightly more important than indicator j.
5	Indicator i is moderately more important than indicator j.
7	Indicator i is strongly more important than indicator j.
9	Indicator i is absolutely more important than indicator j.
2, 4, 6, 8	The importance of the indicators falls between the above scales.
Reciprocal	If the comparison between factors i and j results in the judgment matrix entry Cij, then the comparison of factor j to i is given as Cij = 1/Cij.

.

After scoring, the results of the questionnaire, which was completed by 15 experts, were converted into a judgment matrix. This matrix was used to conduct pairwise comparisons of the various indicators at each level. The method for constructing the judgment matrix is as follows:

$$A=\left(a_{ij}\right)n\times n=\left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{n1}& a_{n2}& \cdots & a_{nn}\end{array}\right]$$

(1)

In the matrix, a_ij denotes the outcome of comparing the significance of indicators i and j within the same subgroup. Here, a_ij is greater than zero, a_ij is the reciprocal of a_ji, and a_jj equals 1, for, i, j = 1, 2, 3, …, n, where n represents the total number of subgroups included in A.

3.3.3. Calculation of Weights

By standardizing the judgment matrix through its eigenvector computation, we progressively determined the relative importance of each component. The weight values accumulated at each hierarchical level were measured in relation to the overarching objective and were calculated progressively from the upper to lower tiers. Within the framework of a layered decision-making process, the weight designated to each level underpins the assessment of the elements’ relative significance within that level [88].

The procedure for computing the sorting method for individual levels is outlined as follows:

(1)

Calculate the product of each row’s indicators in the judgment matrix M_i. m is the total number of indicators in the judgment matrix.

$$M_i={{\displaystyle \prod }}_{j=1}^m{a}_{ij}\left(j=1,2,\cdots ,m\right)$$

(2)

(2)

Calculate the nth root of M_i.

$$W_i=\sqrt[n]{M_i} (i=1,2,\cdots ,n)$$

(3)

(3)

Normalize W_i to obtain the eigenvector ω_i.

$${\omega }_i=\frac{W_i}{{{\displaystyle \sum }}_{j=1}^m{W}_j}\left(j=1,2,\cdots ,m\right)$$

(4)

(4)

The formula for the maximum value of the judgment matrix is as follows:

$${\lambda }_{max}={{\displaystyle \sum }}_{i=1}^n\frac{{\left[A\omega \right]}_i}{n{\omega }_i}$$

(5)

(5)

Consistency Test

The Consistency Ratio (CR) is defined as the ratio of the Consistency Index (CI) of the judgment matrix to the Random Consistency Index (RI). The judgment matrix is deemed consistent if the CR is less than 0.1. If the CR exceeds 0.1, the judgment matrix needs to be reconstructed [89].

The formula for calculating the CR is as follows:

$$\mathrm{CI}=\frac{{\mathsf{\lambda }}_{\max }-n}{n-1}$$

(6)

$$\mathrm{CR}=\frac{\mathrm{CI}}{\mathrm{RI}}$$

(7)

where the RI value in the equation is based on

Table 9. Random index (RI) value.

n	3	4	5	6	7	8	9	10
RI Value	0.52	0.89	1.12	1.26	1.36	1.41	1.46	1.49

.

The Analytic Hierarchy Process (AHP) utilizes Saaty’s 1–9 scale for pairwise comparisons. Accordingly, we first organize the data scored by 15 experts using Saaty’s rating scale into Table 10, Table 11, Table 12, Table 13, Table 14 and Table 15. The evaluation method can be illustrated using the primary indicators B1 and B2 from Table 10. The ratio of the B1 indicator on the vertical axis to the B1 indicator on the horizontal axis is 1 since they are the same indicator. The ratio of the B1 indicator on the vertical axis to the B2 indicator on the horizontal axis is 7, indicating that B1 is significantly more important when compared to B2. Conversely, the ratio of the B2 indicator on the vertical axis to the B1 indicator on the horizontal axis is 1/7, which also indicates that B2 is significantly less important. Similarly, in pairwise comparisons of B2 and B3 indicators, the ratio of the B2 indicator on the vertical axis to the B3 indicator on the horizontal axis is 1/5, showing that B3 is more important than B2. By analogy, all pairwise comparison data in the table are derived from experts’ evaluations of the importance of each pair of indicators using Saaty’s rating scale. Finally, based on the experts’ scores, Formulas (2)–(7) are applied to determine the weights at each level, which are then annotated in Table 10, Table 11, Table 12, Table 13, Table 14 and Table 15.

According to the above rules, the criterion layer indicator weights are as follows:

Table 10. Weights of the criteria layer indicators data.

A	B1	B2	B3	B4	Wi	CR	λmax
B1	1	7	3	1/3	0.2710	0.0772	4.2063
B2	1/7	1	1/5	1/8	0.0442
B3	1/3	5	1	1/5	0.1355
B4	3	8	5	1	0.5494

Table 10. Weights of the criteria layer indicators data.

A	B1	B2	B3	B4	Wi	CR	λmax
B1	1	7	3	1/3	0.2710	0.0772	4.2063
B2	1/7	1	1/5	1/8	0.0442
B3	1/3	5	1	1/5	0.1355
B4	3	8	5	1	0.5494

According to the above rules, the solution layer indicator weights are determined as follows:

Table 11. Weights of the solution layer under the B1 indicator.

B1	C1	C2	C3	C4	C5	C6	C7	Wi	CR	λmax
C1	1	5	4	4	4	3	3	0.3440	0.0892	7.7278
C2	1/5	1	1/5	1/2	3	2	1/3	0.0760
C3	1/4	5	1	2	3	5	1/2	0.1785
C4	1/4	2	1/2	1	4	3	2	0.1479
C5	1/4	1/3	1/3	1/4	1	1	1/3	0.0482
C6	1/3	1/2	1/5	1/3	1	1	1/3	0.0533
C7	1/3	3	2	1/2	3	3	1	0.1521

Table 11. Weights of the solution layer under the B1 indicator.

B1	C1	C2	C3	C4	C5	C6	C7	Wi	CR	λmax
C1	1	5	4	4	4	3	3	0.3440	0.0892	7.7278
C2	1/5	1	1/5	1/2	3	2	1/3	0.0760
C3	1/4	5	1	2	3	5	1/2	0.1785
C4	1/4	2	1/2	1	4	3	2	0.1479
C5	1/4	1/3	1/3	1/4	1	1	1/3	0.0482
C6	1/3	1/2	1/5	1/3	1	1	1/3	0.0533
C7	1/3	3	2	1/2	3	3	1	0.1521

Table 12. Weights of the solution layer under the B2 indicator.

B2	C8	C9	C10	C11	C12	C13	Wi	CR	λmax
C8	1	1	3	3	1/7	1	0.1172	0.0657	6.4142
C9	1	1	3	2	1/8	3	0.1295
C10	1/3	1/3	1	1/3	1/8	1/2	0.0410
C11	1/3	1/2	3	1	1/6	1/2	0.0720
C12	7	8	8	6	1	7	0.5529
C13	1	1/3	2	2	1/7	1	0.0873

Table 12. Weights of the solution layer under the B2 indicator.

B2	C8	C9	C10	C11	C12	C13	Wi	CR	λmax
C8	1	1	3	3	1/7	1	0.1172	0.0657	6.4142
C9	1	1	3	2	1/8	3	0.1295
C10	1/3	1/3	1	1/3	1/8	1/2	0.0410
C11	1/3	1/2	3	1	1/6	1/2	0.0720
C12	7	8	8	6	1	7	0.5529
C13	1	1/3	2	2	1/7	1	0.0873

Table 13. Weights of the solution layer under the B3 indicator.

B3	C14	C15	C16	C17	C18	C19	Wi	CR	λmax
C14	1	1/2	1/5	1/3	1/4	1/2	0.0503	0.0931	6.5866
C15	2	1	1/3	1/3	1/2	2	0.0970
C16	5	3	1	5	5	6	0.4272
C17	3	3	1/5	1	4	6	0.2352
C18	4	2	5	1/4	1	2	0.1282
C19	2	1/2	1/6	1/6	1/2	1	0.0620

Table 13. Weights of the solution layer under the B3 indicator.

B3	C14	C15	C16	C17	C18	C19	Wi	CR	λmax
C14	1	1/2	1/5	1/3	1/4	1/2	0.0503	0.0931	6.5866
C15	2	1	1/3	1/3	1/2	2	0.0970
C16	5	3	1	5	5	6	0.4272
C17	3	3	1/5	1	4	6	0.2352
C18	4	2	5	1/4	1	2	0.1282
C19	2	1/2	1/6	1/6	1/2	1	0.0620

Table 14. Weights of the solution layer under the B4 indicator.

B4	C20	C21	C22	C23	C24	Wi	CR	λmax
C20	1	1/7	1/7	1/3	1/8	0.0358	0.0323	5.1445
C21	7	1	2	6	1	0.3370
C22	7	1/2	1	5	1/2	0.2172
C23	3	1/6	1/5	1	1/6	0.0653
C24	8	1	2	6	1	0.3447

Table 14. Weights of the solution layer under the B4 indicator.

B4	C20	C21	C22	C23	C24	Wi	CR	λmax
C20	1	1/7	1/7	1/3	1/8	0.0358	0.0323	5.1445
C21	7	1	2	6	1	0.3370
C22	7	1/2	1	5	1/2	0.2172
C23	3	1/6	1/5	1	1/6	0.0653
C24	8	1	2	6	1	0.3447

Considering that each indicator at a lower level operates within a framework set by a higher level, assessing the relative values of weights within a single layer alone is insufficient. To calculate the overall weights, we adopted a method from the literature [90], which involves multiplying the weights of lower-level indicators by the weights of their respective higher-level indicators. This approach determines the relative importance of each factor in the overall decision-making process. The formula for calculating the overall weights is as follows:

Let W_i represent the weight of a primary indicator, and w_ij represent the weight of the jth secondary indicator under the ith primary indicator. The composite weight OW_ij for the jth secondary indicator can be calculated using the formula below:

$$O{W}_{ij}=W_i\times w_{ij}$$

(8)

where i is the index for the primary indicators, and j is the index for the secondary indicators given i.

Based on Formula (8), the overall weights for all criteria on the solution level were calculated. The single-layer weight values and the total weight values for each are compiled in Table 15.

Table 15. Weight summary.

Criterion Layer	Weights	Rank	Solution Layer	Weights	Rank	Overall Weights	Rank
B1	0.2710	2	C1	0.3440	1	0.0932	4
			C2	0.0760	5	0.0206	12
			C3	0.1785	2	0.0484	6
			C4	0.1479	4	0.0401	8
			C5	0.0482	7	0.0131	17
			C6	0.0533	6	0.0144	15
			C7	0.1521	3	0.0412	7
B2	0.0442	4	C8	0.1172	3	0.0052	21
			C9	0.1295	2	0.0057	20
			C10	0.0410	6	0.0018	24
			C11	0.0720	5	0.0032	23
			C12	0.5529	1	0.0244	11
			C13	0.0873	4	0.0039	22
B3	0.1355	3	C14	0.0503	6	0.0068	19
			C15	0.0970	4	0.0131	16
			C16	0.4272	1	0.0579	5
			C17	0.2352	2	0.0319	10
			C18	0.1282	3	0.0174	14
			C19	0.0620	5	0.0084	18
B4	0.5494	1	C20	0.0358	5	0.0197	13
			C21	0.3370	2	0.1852	2
			C22	0.2172	3	0.1194	3
			C23	0.0653	4	0.0359	9
			C24	0.3447	1	0.1894	1

Table 15. Weight summary.

Criterion Layer	Weights	Rank	Solution Layer	Weights	Rank	Overall Weights	Rank
B1	0.2710	2	C1	0.3440	1	0.0932	4
			C2	0.0760	5	0.0206	12
			C3	0.1785	2	0.0484	6
			C4	0.1479	4	0.0401	8
			C5	0.0482	7	0.0131	17
			C6	0.0533	6	0.0144	15
			C7	0.1521	3	0.0412	7
B2	0.0442	4	C8	0.1172	3	0.0052	21
			C9	0.1295	2	0.0057	20
			C10	0.0410	6	0.0018	24
			C11	0.0720	5	0.0032	23
			C12	0.5529	1	0.0244	11
			C13	0.0873	4	0.0039	22
B3	0.1355	3	C14	0.0503	6	0.0068	19
			C15	0.0970	4	0.0131	16
			C16	0.4272	1	0.0579	5
			C17	0.2352	2	0.0319	10
			C18	0.1282	3	0.0174	14
			C19	0.0620	5	0.0084	18
B4	0.5494	1	C20	0.0358	5	0.0197	13
			C21	0.3370	2	0.1852	2
			C22	0.2172	3	0.1194	3
			C23	0.0653	4	0.0359	9
			C24	0.3447	1	0.1894	1

4. Results and Discussion

4.1. Criterion Layer Weights

As shown in Figure 3, Trust and Security emerges as the most critical indicator within the criterion layer, followed by Basic Performance (B1, 0.2710), Features and Technology (B3, 0.1355), and Design and Interface (B2, 0.0442). This reflects the high level of concern users have regarding the protection of personal information and health data when using medical websites. Additionally, Basic Performance and Features and Technology ensure that the website both meets the basic operational needs of users and offers technological functionality and services. Although Design and Interface carry a lower weight, they still play a significant role in enhancing user experience, boosting user satisfaction, and establishing brand identity. Therefore, in the design and development of medical websites, there should be a greater emphasis on protecting user health information and providing an efficient and convenient service experience, rather than solely focusing on visual appeal.

4.2. Solution Layer Weights

As illustrated in Figure 4, within the Basic Functionality B1 level, Effectiveness (C1, 0.3440) holds the highest weight, followed by Learnability (C7, 0.1521), Easy to use (C3, 0.1785), and Efficiency (C4, 0.1479). Next, we have Satisfaction (C6, 0.0533), Errors (C5, 0.0482), and Convenience (C2, 0.0760). These rankings indicate that, within Basic Functionality, users primarily focus on whether the platform can accurately and effectively perform its intended functions. Additionally, users care about the ability to quickly learn how to use a platform and the efficiency of the service. Therefore, when optimizing the basic functionalities of a medical website, priority should be given to enhancing the platform’s effectiveness, ease of learning, and usability.

As shown in Figure 5, within the Design and Interface B2 level, the indicator User-Centric Design (C12, 0.5529) carries the highest weight, followed by Navigation (C9, 0.1295), Feedback (C8, 0.1172), Interactivity (C13, 0.0873), Consistency (C11, 0.0720), and finally, Screen Design and Layout (C10, 0.0410). These rankings highlight that, in Design and Interface, users are most concerned with whether the interface is user-centric, i.e., whether the website provides an interactive experience that meets user needs and expectations. Additionally, good navigation and timely feedback are crucial factors in enhancing usability, playing a significant role in ensuring a smooth and intuitive user experience. Therefore, when optimizing the design and interface of a medical website, the principles of user-centric design should be prioritized, along with ensuring good navigation and effective interactivity, to enhance the overall user experience.

As depicted in Figure 6, within the Features and Technology B3 level, Searchability (C16, 0.4272) holds the highest weight, followed by Redundancy (C17, 0.2352), Controllability (C18, 0.1282), Remote Presentation (C15, 0.0970), Responsiveness (C19, 0.0620), and finally, Accessibility (C14, 0.0503). These rankings indicate that, in Features and Technology, users are most concerned about their ability to quickly and accurately find the information and services they need. Additionally, users care about the redundancy and controllability of the webpage, i.e., avoiding redundant functions on the platform and the degree of control users have over platform operations, such as font size and selecting specific operational processes. Therefore, when optimizing the features and technology of a medical website, priority should be given to enhancing the searchability of information and reducing the redundancy of the interface.

As shown in Figure 7, within the Trust and Security B4 level, Property Protection (C24, 0.3447) has the highest weight, followed by Credibility (C21, 0.3370), Privacy Protection (C22, 0.2172), Content Relevance (C23, 0.0653), and finally, Readability (C20, 0.0358). These rankings demonstrate that, in Trust and Security, users are most concerned about the protection of their assets. Additionally, the credibility of the information provided by the website and privacy protection are key factors that users consider, which are directly related to the site’s reputation, user trust, and privacy. Therefore, when optimizing trust and security on a medical website, priority should be given to strengthening measures for property and privacy protection. It is also essential to ensure the credibility and relevance of the medical information provided, to enhance the overall trust and security felt by users.

4.3. Overall Weights

As shown in Figure 8, in the total weight analysis, Property Protection (C24, 0.1894) is considered the most important factor, indicating that ensuring the security of users’ financial assets is the most significant factor for medical websites. This is followed by Credibility (C21, 0.1852), which emphasizes the critical importance of establishing and maintaining users’ trust in medical websites. Privacy Protection (C22, 0.1194), ranking third, highlights the importance of safeguarding users’ personal data, a key component that cannot be overlooked for medical websites. Effectiveness (C1, 0.0932) and Findability (C16, 0.0579), ranking fourth and fifth, respectively, stress the importance of the efficient delivery of medical information and services and their accessibility to users. Effectiveness focuses on whether services meet the actual needs of users, while Findability concerns whether users can easily locate the information and features they need. Lower-ranking indicators, such as Screen Design and Layout (C10, 0.0018), Consistency (C11, 0.0032), and Interactivity (C13, 0.0039), performed poorly. The low importance of these indicators might suggest that, although they have some impact on the overall user experience, they are not the primary factors considered by users when assessing the usability of medical websites.

These findings provide valuable insights for the future design and improvement of online medical platforms. To enhance user satisfaction and overall website usability, developers and designers should focus on these high-weight indicators. Balancing and optimizing these factors will help create safer, more trustworthy, and user-friendly medical websites.

5. Conclusions

In this study, we aimed to develop and rank a framework of usability indicators for medical websites, grounded in user experience. Initially, 30 potential usability indicators related to user experience, usability, and medical websites were selected from the literature. Through two rounds of Delphi expert panel surveys, 24 key indicators were selected. These indicators were then reduced and categorized into four primary categories using Exploratory Factor Analysis and were subsequently named. The Analytic Hierarchy Process (AHP) was applied to hierarchically organize these indicators and calculate their weights, reflecting the level of user attention to each indicator during usage. The results of this study serve as a framework for evaluating the usability of medical websites, which is crucial for assessing and improving their usability.

However, to address the rapidly changing usability trends and user needs in the digital healthcare field, we have also considered relevant dynamic demands while developing the usability evaluation criteria. Future research can investigate the current literature in the online healthcare domain to identify and summarize the latest indicators that align with the current needs of the healthcare field and its users. Based on this, reasonable additions and adjustments can be made to the usability evaluation system summarized in this study. This approach ensures that the evaluation system maintains the practicality and authority of the usability criteria, thereby better adapting to the ever-changing digital healthcare environment and user expectations.

Despite this, our study does have certain limitations. For example, there was a limited number of experts involved in the survey, which may have resulted in an insufficiently diverse array of opinions. Moreover, the selection of usability assessment indicators for medical websites was solely based on a review of the literature. Considering these issues, future research should consider increasing the number of experts involved in the surveys and expanding the range of selected indicators. The goal would be to more comprehensively and systematically integrate and collect a wider spectrum of opinions and indicators.