An Experience-Based Framework for Evaluating Tourism Mobile Commerce Platforms

Abstract

This research presents and studies an evaluation framework for tourism mobile commerce platforms based on tourists’ experience. Synthesizing from prior literature, relevant theories, and the results of online questionnaires, we select 24 evaluation indices for preliminary evaluation. Using exploratory factor analysis method, we then extract from these indices the following five principal factors: interactive experience, infrastructure experience, personalization experience, product or service quality experience, and product operation experience. We further employ the confirmatory factor analysis to test the construction of the evaluation framework and demonstrate that the evaluation framework is both robust and effective. Finally, based on our proposed evaluation framework, we empirically evaluate the most popular mobile commerce platforms (Ctrip and Qunaer) in China by using fuzzy comprehensive evaluation method.

1. Introduction

The proliferating wireless technologies have enabled consumers to increasingly interface and interact with mobile commerce (m-commerce) systems for transactions. In China, mobile devices have become ubiquitous in people’s daily activities, resulting in about 50% of e-commerce transactions completed through mobile platforms in comparison to about 20% in the United States and 33% in the United Kingdom.

Among all the m-commerce transactions, online bookings through mobile devices have become increasingly popular. For instance, 25% of total online bookings were made from mobile terminals in 2016 in the United Kingdom, up from 12% three years ago [1]. In the United States, the digital travel sales through mobile platforms were expected to exceed 50 billion in 2016 and reach 70 billion by 2018 with 35% of online bookings being mobile [2]. Among all the markets, China is the leader in mobile bookings with a projected 60% of online bookings being made on a mobile device by 2017 [3].

Since mobile terminals are driving the increase in the overall traffic of online travels, it is important for tourism companies to ensure the appropriate functioning of their m-commerce platforms so as to optimize the allocation of tourism resources and present their products and services to tourists in a meaningful way to attract more online transactions. Therefore, evaluating different m-commerce platforms used in the tourism industry and analyzing the appraisal results will help m-commerce platform providers to ensure the quality of tourists’ experience and the improvement of their customer loyalty.

An effective evaluation system will allow tourism service providers to reduce unnecessary costs, improve their efficiencies, and better understand the inherent needs of their tourists, so that they can remain innovative and competitive on the market by providing more convenient, personalized, and meaningful products and services to their customers [4]. Therefore, how to evaluate the performance of m-commerce tourism platforms so as to enhance tourists’ satisfaction has recently become the focus of academic and business communities of the tourism industry.

Prior studies have proposed some metrics to measure the efficacy of online traveling services (e.g., [5,6,7]). Nevertheless, very few studies have developed a comprehensive evaluation system that can be effectively used to quantitatively evaluate the performance of existing m-commerce tourism platforms. Our research attempts to bridge this gap by making the following contribution to the literature. First, the factors impacting the service quality are identified from related literature, and then used to construct the framework to evaluate m-commerce tourism platforms. Second, a survey is conducted with our purposely designed questionnaire to test the reliability of the proposed evaluation framework. Finally, our framework is applied to evaluate some most popular mobile tourism platforms in China.

The rest of the paper proceeds as follows. Next section reviews prior literature related to our research. Section 3 presents our evaluation framework. Section 4 demonstrates an application of our model with actual examples. The last section concludes the entire paper with insights.

2. Prior Literature

This section reviews prior literature with a focus on the use and efficacy of m-commerce systems in the tourism industry as well as the factors that influence online tourism services. In addition, the emphasis and contribution of our study are also highlighted in this section.

The growing prevalence of smart phones, tables, and other types of mobile devices has enabled them to be increasingly used in tourism, requiring the platforms or systems to be designed with a user-centered approach [8]. Recent studies have further explored the issues and performance of mobile tourism systems and services from users’ perspectives. For instance, Wang and Liao [9] assess the effective design of an m-commerce system through conceptualizing and measuring m-commerce user satisfaction construct. Kenteris, Gavalas, and Economou [10] empirically evaluate the user experience of their proposed mobile tourism prototype. Based on a case study of online ticketing services, Mallat et al. [11] suggest evaluating the needs derived from a user’s context in order to assess the benefits of mobile systems. Using a factor analysis approach, Goh et al. [12] identify important types of mobile services from tourists’ perspectives including transportation, accommodation, and food. Douglas and Lubbe [13] verify mobile devices as useful tools for booking services and indicate the satisfaction level of visitors’ experience with their mobile applications. However, prior studies have not explicitly constructed any evaluation for mobile tourism platforms based on user experience.

User experience is defined as “a person’s perceptions and responses that result from the use and/or anticipated use of a product, system or service” by International Standardization Organization [14]. Recent studies have constructed and investigated user experience in different contexts. For instance, Park et al. [15] classify user experience with mobile phones into three categories (present, brand, and product/service experience) and identify specific elements in each category by using survey, interview, and observation methods. Pu et al. [16] evaluate the perceived quality of recommendations from a recommendation system by using their proposed evaluation framework consisting of four basic constructs: user perceived qualities, user beliefs, user attributes, and behavioral intensions. Xiong et al. [17] construct an evaluation framework based on user experience in the future 5G systems from a technical perspective. Analyzing the results from interviews and workshops, Vermeeren et al. [18] identify the needs for user experience evaluation methods such as those for early phases of development, for social and collaborative user experience evaluation, and for practicability. Nevertheless, very few prior studies have incorporated user experience in the context of mobile tourism and its platforms.

In order to identify and synthesize the elements in our evaluation framework based on user experience and apply the framework in mobile tourism platforms, we further review prior research that has studied the important factors impacting the performance of online services in a broad tourism context. Kaynama and Black [19] develop seven dimensions of online travel agency service quality: content, access, navigation, design, response, background information, and personalized. Zeithaml, Parasuraman, and Malhotra [20] categorize website features into reliability, access, response, effectiveness, easy navigation, flexible, trust, security, price, website design, and personalization, and explore the indicators of e-commerce services including reliability, accessibility, responsiveness, effectiveness, flexibility, price, trust, beauty, security, and personalization. Extending their model, Parasuraman, Zeithaml, and Malhotra [21] constructs a 22-item scale in four dimensions: efficiency, fulfillment, system availability, and privacy, and establishes a second scale that contains 11 items in three dimensions: responsiveness, compensation, and contact. Kim, Kim, and Lennon [22] evaluate online traveling websites with the following nine indicators: security, ease of use, low cost, website design and appearance, speed and useful information, booking service ability, pre-booking flexibility and classification. Based on fuzzy theory, Hu [23] evaluates service quality by using dimensioned criteria such as effectiveness, availability, compensatory, reactivity, integrity, contact, security, benefit, and personalized service. Kim and Lee [24] find that online travel agencies and suppliers share similar commonalities with regard to information content, reputation and security, structure and ease of use, and usefulness. Ho and Lee [25] investigate online tourism by grouping e-service quality constructs into five core components: information quality, security, website functionality, customer relationships, and responsiveness. Ghose and Han [26] investigate users’ behavior on mobile devices and identify some influential factors to users’ mobile Internet usage, such as social network, extend of geographical mobility, and user mobility. Bernardo, Marimon, and del Mar Alonso-Almeida [27] confirm that both functional and hedonic quality are two important dimensions significant influencing the perceived value with respect to the performance of e-services in online traveling agencies.

In summary, most of the prior research on m-commerce for the tourism industry is restricted to the development of technical models and prototypes. Although some studies attempt to use quantitative methods to construct system models in the tourism industry, very few of them have applied quantitative methods to conduct comprehensive analysis. Furthermore, most of the prior research related to user experience is based on website design, recommendation systems, and technology products; the effects of the tourists’ experience have not been formally incorporated into mobile travel services. Our study addresses this gap by formally proposing an evaluation framework based on tourists’ experience and using the framework to empirically evaluate two most popular m-commerce tourism platforms in China.

3. Evaluation Framework

Identifying appropriate evaluation indices is essential for constructing the evaluation framework. Selecting and incorporating different evaluation indices in the framework will have different influences on its accuracy and practicability. Although there lacks a common standard for choosing the evaluation indices for m-commerce tourism application platforms, prior studies show that they all follow some similar principles. Following upon these principles, we collect user experience-based influential factors of m-commerce and online travel services used by many researchers, extract online travel service quality influence indices according to the empirical factors, and then continue to summarize these collected indicators for evaluating m-commerce platforms and websites. We summarize the specific procedure as follows.

3.1. Selecting Preliminary Evaluation Indices

Based on prior literature, we categorize all the relevant experience-based factors into the following five preliminary first-level indices: user interface experience, product content experience, software security experience, service quality experience, and personalization experience. The second-level indicators are then listed in each category accordingly.

(1) The user interface experience describes how visitors feel when they browse a mobile application platform (e.g., [28,29]). A visitor’s first good impression to the mobile application can improve the visitor’s stickiness to the application. The preliminary second level indicators include six indices: interface layout, interface navigation, interaction, APP loading/login time cost, efficiency of operations, smooth guidance of the purchase process, and evaluation feedback.

(2) The product content experience can directly influence a tourist’s decision to purchase products and services (e.g., [30,31,32]). Good contents can improve customer loyalty to m-commerce platforms. Many important functions are offered by various m-commerce tourism platforms. For instance, visitors can use the query searching function from mobile service providers to search for the information about tourism products and services, and then continue to booking and payment. They can also share their experiences of offline consumptions after their purchases with other tourists in the community of the m-commerce platforms. All of these behaviors are based on product contents. Therefore, tourists’ experience and product contents are closely related. Here, we choose the following seven aspects as the second level indices: product price, product timeliness, product coverage, product content authenticity, product diversity, product booking availability, and membership rebate.

(3) The experience of software security is a crucial concern to users regardless of the PC or mobile terminals they use (e.g., [33,34]). Tourists’ willingness to fulfill their m-commerce tourism transactions are contingent on the software security affiliated with the m-commerce platforms as their bank accounts and other personal information must be under good protection. Therefore, the security issues are fundamental to mobile e-commerce operators before they can provide other services. We choose the following three second-level indices for software-security experience: the security and convenience of payment, the authenticity of transaction, and the confidentiality of data information.

(4) A good service-quality experience can improve transaction rate, attract potential offline users, and promote customer loyalty (e.g., [35,36,37]). One of the important reasons to attract visitors to download mobile software applications and further to purchase tourism products is an m-commerce provider’s popularity and reputation. Tourists’ good offline consumer experience will further contribute to the provider’s reputation, which is the best way to further publicize its products and services with the word-of-mouth effect. The second level indices we choose for service-quality experience are follows: visibility and reputation, service friendliness, offline service quality, emergency remedial capacity, advisory hotline, and complaint channel.

(5) Personalization experience is referred as the needs and expectations for different individuals in terms of tourism products and services (e.g., [38,39,40]). Therefore, m-commerce tourism providers should take into account the differences among users’ demands and preferences for products and services. In order to meet the needs of different tourists, they will have to continuously improve their mobile traveling service functions. We identify the second-level indices for personalization experience as personalized service, timeliness of information update, and users’ expectations.

3.2. Determining the Index System

In order to make the identified indices more scientifically rigorous so they can be applied in generic situations, we design a questionnaire to survey and verify the indicators, and then use SPSS software to further analyze the data.

3.2.1. Questionnaire

(1) Design of the Questionnaire

The questionnaire consists of two sections. The first section is the main part, including a five point Likert Scale, which is used to measure the importance of the evaluation indicators of the selected indices in the process of their experience. The second section is the basic personal information. It helps analyze the different education, income, occupation of different proportions of the population and their impact on the evaluation indices (See details of the questionnaire in Appendix A).

(2) Distribution of the Questionnaire

The targeting group of our questionnaire includes the tourists who have used m-commerce tourism platforms to query information or book traveling products. In order to get the sufficient number of responses in a certain period of time, we adopt the format of e-questionnaire by using the specific tool called “Questionnaire Star”. Unlike traditional online questionnaires that can be easily distributed but are not effective, “Questionnaire Star” can improve the effectiveness of questionnaires by inhibiting the repetition of the same IP addresses and sources of information.

In order to obtain effective responses to the questions in the questionnaire, we piloted the survey in a small scale. After adjusting some of the choices based on the results, we then distributed the survey through QQ, WeChat, and some other popular social media apps in China to ensure that the questionnaire can be widely disseminated.

(3) Collecting Questionnaire Results

The survey was distributed through “Questionnaire Star” for five days between 7 December 2015 and 15 December 2015 with a total of 310 responses. After discarding those responses with a completion time less than one minute and repeated IP addresses, we finally obtained 184 valid questionnaire responses. Descriptive statistics of the effective responses is summarized in Appendix F.

3.2.2. Reliability Analysis

We use the SPSS20.0 to test the reliability of the questionnaire based on the 184 valid responses. The statistical results show that the Cronbach’s Alpha values of the questionnaire are almost all greater than 0.8, inferring that the questionnaire is highly reliable. See

Table 1. Reliability statistics.

	Cronbach’s Alpha	Number of Indices
interactive experience	0.794	3
infrastructure experience	0.830	3
personalization experience	0.866	3
product or service quality experience	0.959	10
product operation experience	0.893	4
Total	0.967	24

.

3.2.3. Exploratory Factor Analysis

Applying the exploratory factor analysis method, we analyze the 24 indices in the questionnaire to investigate the effect of m-commerce tourism platforms on visitors’ experience. In our analysis, we use the principal component analysis approach to extract five immobilization factors and then use the maximum variance method to rotate the factors’ load matrix.

(1) Descriptive statistics

We summarize the details of the descriptive statistics in Appendix G.

(2) KMO and Bartlett testing

Table 2. Kaiser-Meyer-Olkin and Bartlett testing.

Kaiser-Meyer-Olkin Measure of Sampling Adequacy		0.955
Bartlett’s Test of Sphericity	Approx. chi-Square	3871.842
	Df	276
	Sig.	0.000

shows the results of the KMO and Bartlett testing, in which KMO value is 0.955, Bartlett’s test of sphericity approximate Chi-Square value is 3871, Degree of freedom is 276, and Significance is 0.000. The significant probability is less than 0.001, indicating that there is a correlation among the variables, so they are suitable for factor analysis.

(3) Explanation of factor analysis

Table A1. Contribution to total variance.

Elements	Initial Factor Value			Square Extraction and Loading			Square Rotation and Loading
	Total	Variance %	Cum. %	Total	Variance %	Cum. %	Total	Variance %	Cum. %
1	13.940	58.084	58.084	13.940	58.084	58.084	6.773	28.223	28.223
2	1.350	5.625	63.709	1.350	5.625	63.709	3.166	13.192	41.414
3	1.134	4.726	68.435	1.134	4.726	68.435	3.159	13.163	54.578
4	1.023	4.263	72.698	1.023	4.263	72.698	2.863	11.928	66.506
5	0.834	3.476	76.175	0.834	3.476	76.175	2.320	9.669	76.175
6	0.595	2.481	78.656
7	0.518	2.158	80.814
8	0.475	1.979	82.792
9	0.458	1.910	84.702
10	0.433	1.804	86.506
11	0.360	1.499	88.005
12	0.348	1.450	89.455
13	0.332	1.383	90.839
14	0.318	1.324	92.163
15	0.286	1.193	93.356
16	0.251	1.047	94.403
17	0.234	0.975	95.377
18	0.214	0.893	96.270
19	0.198	0.824	97.094
20	0.184	0.766	97.860
21	0.158	0.659	98.520
22	0.143	0.597	99.116
23	0.130	0.543	99.659
24	0.082	0.341	100.000

(shown in Appendix B) displays the total-variance of the extracted factors to the original variables. The first factor contributes 28.223%, second factor 13.192%, third factors 13.163%, fourth factors 11.928%, and fifth factors 9.669% to the original variables. The cumulative variance contribution rate of the five factors is 76.175%. From the sixth factor to the last one, its characteristic value becomes smaller, which means its contribution rate to the original variance is less important. Therefore, the extraction of these five factors is sufficient for factor analysis.

(4) Factors’ load matrix

Table A2. The load on each of the five factors in the factors’ load matrix.

	Component
	1	2	3	4	5
A20 emergency recovery capability	0.823	0.299	0.227	0.102	0.135
A16 transaction authenticity	0.821	0.133	0.211	0.247	0.191
A15 data privacy	0.809	0.145	0.192	0.292	0.237
A18 consultation hotline	0.778	0.312	0.247	0.156	0.163
A17 visibility and credibility	0.704	0.190	0.313	0.314	0.120
A19 complaining methods	0.690	0.406	0.155	0.254	0.106
A11 product content authenticity	0.656	0.225	0.296	0.330	0.307
A21 service friendship	0.649	0.456	0.243	0.192	0.194
A13 product reservation possibility	0.572	0.301	0.331	0.212	0.302
A07 product price	0.555		0.462	0.385	0.302
A14 payment safety and convenience	0.503	0.238	0.368	0.435	0.143
A22 personalized service	0.233	0.812	0.184	0.154	0.170
A23 upgrade/update timeliness	0.362	0.736	0.138	0.243	0.169
A24 user preferences and expectations	0.303	0.709	0.189	0.105	0.342
A09 product timeliness	0.313	0.197	0.777	0.110	0.163
A08 product diversity	0.303	0.151	0.765	0.152	0.235
A10 product coverage	0.466	0.177	0.587	0.189	0.257
A12 membership rebate		0.515	0.580	0.369
A04 APP load/login time	0.355	0.215		0.751	0.237
A06 evaluation feedback	0.210	0.276	0.365	0.632	0.141
A05 operation processing convenient	0.543	0.177	0.171	0.629	0.199
A01 interface layout	0.200	0.254	0.181		0.787
A02 interface navigation	0.221	0.195	0.190	0.275	0.713
A03 interactive humanization	0.452	0.117	0.199	0.496	0.507

(in Appendix C) shows the load on each of the five factors in the factors’ load matrix. Before rotation, although there exists orthogonality between the factors, it is still difficult to explain them. After rotation, the load matrix structure can be simplified, making it easier to explain the practical significance of the common factors.

3.2.4. Reconstruction of m-Commerce Tourism Evaluation Framework

Five principal component factors (first level indices) and their influencing factors (second level indices) can be obtained from the rotated component matrix (in Table A2 of Appendix C). For example, the influencing factors of the first principal component factor include those from emergency recovery capability (A20) to payment security and convenience (A14). Although the results are a little bit different between the expected and evaluation indicators, the overall indicators are able to evaluate m-commerce platforms in a good extent. After adjusting the second level indicators, we obtain the final evaluation framework in

Table A3. Evaluation framework with indices.

Target	First Level Index	Second Level Index
Tourism M-commence platform service quality	interactive experience	interface layout
		interface navigation
		humanized interaction
	infrastructure experience	APP load/login time
		evaluation feedback
		convenience of processing operations
	personalization experience	personalization service
		timeliness of upgrade/update
		user preferences and expectations
	product or service quality experience	emergency recovery capability
		transaction authenticity
		data privacy
		consultation hotline
		visibility and credibility
		complaining methods
		product content authenticity
		service friendliness
		product reservation possibility
		product price
		payment safety and convenience
	product operation experience	product timeliness
		product diversity
		product coverage
		membership rebate

(see Appendix D).

(1) The first first-level indicator interprets product or service quality experience which includes the following 11 secondary indices: emergency recovery capability, transaction authenticity, data privacy, consultation hotline, visibility and credibility, complaining methods, product content authenticity, service friendliness, product reservation possibility, product price, payment safety and convenience. These indicators are related to the product and service quality for mobile e-commerce platforms, as well as their security issues. These are the primary factors affecting the application software.

(2) The second first-level indicator explains product operation experience by including these four secondary indices: product timeliness, product diversity, product coverage, and membership rebate. These indices evaluate the effective factors that can attract tourists to purchase and improve customer loyalty.

(3) The third first-level indicator deals with personalization experience that includes the following three secondary indices: personalization service, timeliness of upgrade/update, and user preferences and expectations. These indicators reflect the needs and expectations of providing different service information for different users.

(4) The fourth first-level indicator focuses on infrastructure experience with three secondary indices: APP load/login time, evaluation feedback, and convenience of processing operations. These indicators assess the quality of mobile traveling e-commerce application software, not that for products and services.

(5) The fifth first-level indicator describes interactive experience by incorporating three secondary indices: interface layout, interface navigation, humanized interaction. These indices can be utilized by users to develop a self perception for mobile application software. Good interactive design can enhance the browsing and reading experience, highlighting the characteristics of a brand and its public image.

4. Application of the Evaluation Framework

Having established the formal evaluation framework, we next apply this framework to investigate some of the most popular m-commerce platforms so as to demonstrate the applicability of our proposed evaluation framework and further test its robustness.

4.1. Selection of M-Commerce Platforms

According to the 184 effective responses to our questionnaire, the most popular tourism M-Commerce platforms are Ctrip and Qunaer in China (See

Table A4. The most popular tourism m-commerce platforms in China.

	Frequency	Percentage (%)	Effective Percentage (%)	Accumulative Percentage (%)
Ctrip	70	38.0	38.0	38.0
Qunaer	71	38.6	38.6	76.6
Tongcheng	18	9.8	9.8	86.4
Tuniu	9	4.9	4.9	91.3
Lvmama	1	0.5	0.5	91.8
Mafengwo	2	1.1	1.1	92.9
others	13	7.1	7.1	100.0
total	184	100.0	100.0

in Appendix E). They account for 38.0% and 38.6% of the total, respectively, followed by Tongcheng 9.8%, Tuniu 4.9%, Mafengwo 1.1%, lvmama 0.5%, and other 7.1%. Therefore, we select Qunaer and Ctrip as our empirical research target because of their popularity. Using our proposed evaluation framework and Fuzzy Comprehensive Evaluation method [41,42], we next evaluate these two tourism m-commerce platforms.

4.2. Application of Fuzzy Comprehensive Evaluation Model

Because it is not easy to accurately quantify each evaluation index in our framework, the instrument of fuzzy mathematics can be applied to the evaluation framework. Specifically, we use the fuzzy comprehensive evaluation method to test the second level indices with a bottom-up evaluation process. Synthesizing the single factor evaluation matrix and the weight vector on each layer, we then conclude the testing results.

4.2.1. Determining the Weight Set

(1) Weight set determination of the first level indices

We use contribution rate as the weight for the five main factors extracted by principal components analysis method. If the contribution ratio for factor $u_i$ is $a_i$, the weight of $a_i$ is

$$a_i=\frac{a_i}{{\displaystyle {\sum }_{i=1}^n{a}_i}},n=5$$

(1)

The weight of each main factor is obtained accordingly and displayed in

Table 3. First level factors and their weights.

First Level Factors	Eigen Value after Rotate	Weight
quality of product and service experience	6.773	0.370
personalized experience	3.166	0.173
product operations experience	3.159	0.173
basic construction experience	2.863	0.157
interactive experience	2.320	0.127

.

Therefore, the first level index weight vector is A = (0.370  0.173  0.173  0.157  0.127), which shows that the product and service quality experience is the most important factor for m-commence tourism platforms, followed by the personalization experience and product operations experience both as the second most important factors. The third most important factor is the infrastructure experience and the least important is the interface interaction experience.

(2) Weight sets determination of the second level index

The weight set of the second level indices is determined according to the statistical output of the communalities. (See

Table 4. Second level factors and its weight.

Indices Weight of Quality of Product and Service Experience
	Initial Value	Extract Value	Weight
A20 emergency recovery capability	1.000	0.847	0.100
A16 transaction authenticity	1.000	0.833	0.099
A15 data privacy	1.000	0.853	0.101
A18 consultation hotline	1.000	0.815	0.096
A17 visibility and credibility	1.000	0.743	0.088
A19 complaining methods	1.000	0.740	0.088
A11 product content authenticity	1.000	0.771	0.091
A21 service friendship	1.000	0.764	0.090
A13 product reservation possibility	1.000	0.664	0.079
A07 product price	1.000	0.763	0.090
A14 payment safety and convenience	1.000	0.655	0.078
Indices Weight of Personalized Experience
	Initial Value	Extract Value	Weight
A22 personalized service	1.000	0.801	0.342
A23 upgrade/update timeliness	1.000	0.780	0.333
A24 user preferences and expectations	1.000	0.759	0.324
Indices Weight of Product Operations Experience
	Initial Value	Extract Value	Weight
A09 product timeliness	1.000	0.779	0.260
A08 product diversity	1.000	0.778	0.260
A10 product coverage	1.000	0.694	0.231
A12 membership rebate	1.000	0.747	0.249
Indices Weight of Basic Construction Experience
	Initial Value	Extract Value	Weight
A04 APP load/login time	1.000	0.801	0.354
A06 evaluation feedback	1.000	0.673	0.297
A05 operation processing convenient	1.000	0.790	0.349
Indices Weight of Interactive Experience
	Initial Value	Extract Value	Weight
A01 interface layout	1.000	0.765	0.343
A02 interface navigation	1.000	0.707	0.317
A03 humanized interaction	1.000	0.761	0.341

). The communality of each second level index represents its contribution rate, which reflects the importance of each second level index in the first level index it belongs to. We consider the communality as the weight and then use Equation (1) to calculate values. In particular, we fix the extracted factor as one and then normalize the extracted value to obtain the weight.

Therefore, we obtain the second level index weight vectors as follows:

• A₁ = (0.100  0.099  0.101  0.096  0.088  0.088  0.091  0.090  0.079  0.090  0.078);
• A₂ = (0.342  0.333  0.324);
• A₃ = (0.260  0.260  0.231  0.249);
• A₄ = (0.354  0.297  0.349); and
• A₅ = (0.343  0.317  0.341).

4.2.2. Determining Factor Set

(1) Construction of the first-level factor set

We use U to denote the tourism m-commerce platform overall service quality:

U = {U₁, U₂, U₃, U₄, U₅}

(2)

where U₁ represents product and service quality experience, U₂ personalization experience, U₃ product operations experience, U₄ infrastructure experience, and U₅ interactive experience.

(2) Construction of the second level factor sets.

We first construct the second level factors as follows for each first level factor.

U₁ = {u₁₁, u₁₂, u₁₃, u₁₄, u₁₅, u₁₆, u₁₇, u₁₈, u₁₉, u₁₁₀, u₁₁₁}

(3)

• U₁—the product and service quality experience;
• u₁₁—emergency recovery capability;
• u₁₂—transaction authenticity;
• u₁₃—data privacy;
• u₁₄—consultation hotline;
• u₁₅—visibility and credibility;
• u₁₆—complaining methods;
• u₁₇—product content authenticity;
• u₁₈—service friendship;
• u₁₉—product reservation possibility;
• u₁₁₀—product price; and
• u₁₁₁—payment safety and convenience.

U₂ = {u₂₁, u₂₂, u₂₃}

(4)

• U₂—personalized experience;
• u₂₁—personalized service;
• u₂₂—upgrade/update timeliness; and
• u₂₃—user preferences and expectations.

U₃ = {u₃₁, u₃₂, u₃₃, u₃₄}

(5)

• U₃—product operations experience;
• u₃₁—product timeliness;
• u₃₂—product diversity;
• u₃₃—product coverage;
• u₃₄—membership rebate.

U₄ = {u₄₁, u₄₂, u₄₃}

(6)

• U₄—basic construction experience;
• u₄₁—APP load/login time;
• u₄₂—evaluation feedback; and
• u₄₃—operation processing convenient.

U₅ = {u₅₁, u₅₂, u₅₃}

(7)

• U₅—interactive experience;
• u₅₁—interface layout;
• u₅₂—interface navigation; and
• u₅₃—humanized interaction.

4.2.3. Determining Comment Sets

The fuzzy evaluation of tourism m-commerce platforms is a collection of different tourists’ satisfaction levels to a specific platform. Based on the evaluation results given by tourists, we set up five levels of fuzzy evaluations as

V = {v₁, v₂, v₃, v₄, v₅}

(8)

where v₁ is very unsatisfied, v₂ unsatisfied, v₃ normal, v₄ satisfied, and v₅ very satisfied.

4.2.4. Determining Judgment Matrix

We select the first 40 responses as samples to the questionnaires of Ctrip and Qunaer to calculate the rating score with Equation (8), and then divide the scores by 40 to get the membership grade influencing factors. Finally, we obtain the evaluation matrix based on the selected second-level indices.

(1) Ctrip’s evaluation matrix

For Ctrip’s membership statistics, see Appendix H.

According to the evaluation index system and membership statistics, we derive the evaluation matrix for Ctrip as:

$$R_1^1=\left(\begin{array}{ccccc}0.025& 0.075& 0.050& 0.300& 0.550\\ 0.025& 0.050& 0.050& 0.175& 0.700\\ 0.025& 0.050& 0.050& 0.150& 0.725\\ 0.025& 0.075& 0.025& 0.325& 0.550\\ 0.000& 0.050& 0.075& 0.275& 0.600\\ 0.050& 0.050& 0.075& 0.275& 0.550\\ 0.075& 0.000& 0.100& 0.275& 0.550\\ 0.050& 0.025& 0.100& 0.375& 0.450\\ 0.050& 0.050& 0.100& 0.350& 0.450\\ 0.075& 0.000& 0.125& 0.225& 0.575\\ 0.050& 0.050& 0.100& 0.275& 0.525\end{array}\right)$$

$$R_2^1=\left(\begin{array}{ccccc}0.025& 0.075& 0.200& 0.400& 0.300\\ 0.000& 0.075& 0.200& 0.350& 0.400\\ 0.025& 0.025& 0.225& 0.350& 0.375\end{array}\right)$$

$$R_3^1=\left(\begin{array}{ccccc}0.000& 0.075& 0.200& 0.350& 0.400\\ 0.050& 0.025& 0.150& 0.375& 0.400\\ 0.025& 0.025& 0.225& 0.350& 0.375\\ 0.075& 0.075& 0.275& 0.275& 0.300\end{array}\right)$$

$$R_4^1=\left(\begin{array}{ccccc}0.075& 0.025& 0.100& 0.300& 0.500\\ 0.050& 0.000& 0.050& 0.325& 0.575\\ 0.075& 0.050& 0.150& 0.350& 0.375\end{array}\right)$$

$$R_5^1=\left(\begin{array}{ccccc}0.050& 0.025& 0.125& 0.525& 0.275\\ 0.050& 0.000& 0.200& 0.425& 0.325\\ 0.075& 0.000& 0.050& 0.350& 0.525\end{array}\right)$$

(2) Qunaer’s judgment matrix

Qunaer’s membership statistics can be seen in Appendix I. Based on the evaluation index system and membership statistics, we get the evaluation matrix of Qunaer as:

$$R_1^2=\left(\begin{array}{ccccc}0.050& 0.025& 0.100& 0.450& 0.375\\ 0.025& 0.050& 0.050& 0.275& 0.600\\ 0.050& 0.025& 0.050& 0.275& 0.600\\ 0.050& 0.075& 0.075& 0.350& 0.450\\ 0.050& 0.050& 0.025& 0.375& 0.500\\ 0.050& 0.025& 0.025& 0.425& 0.475\\ 0.075& 0.000& 0.100& 0.350& 0.475\\ 0.050& 0.025& 0.150& 0.475& 0.300\\ 0.075& 0.000& 0.075& 0.475& 0.375\\ 0.075& 0.000& 0.075& 0.375& 0.475\\ 0.025& 0.050& 0.100& 0.400& 0.425\end{array}\right)$$

$$R_2^2=\left(\begin{array}{ccccc}0.050& 0.000& 0.225& 0.500& 0.225\\ 0.050& 0.025& 0.175& 0.500& 0.250\\ 0.050& 0.100& 0.175& 0.525& 0.150\end{array}\right)$$

$$R_3^2=\left(\begin{array}{ccccc}0.050& 0.025& 0.150& 0.400& 0.375\\ 0.050& 0.050& 0.175& 0.425& 0.300\\ 0.050& 0.000& 0.100& 0.575& 0.275\\ 0.050& 0.125& 0.250& 0.350& 0.225\end{array}\right)$$

$$R_4^2=\left(\begin{array}{ccccc}0.050& 0.050& 0.150& 0.375& 0.375\\ 0.075& 0.050& 0.150& 0.450& 0.275\\ 0.050& 0.050& 0.075& 0.350& 0.475\end{array}\right)$$

$$R_5^2=\left(\begin{array}{ccccc}0.075& 0.075& 0.225& 0.375& 0.250\\ 0.075& 0.025& 0.025& 0.525& 0.350\\ 0.075& 0.025& 0.025& 0.550& 0.325\end{array}\right)$$

4.3. Results of Fuzzy Comprehensive Evaluation Analysis

4.3.1. First-Level Index Fuzzy Comprehensive Evaluation

(1) Ctrip

Based on the individual factor of the second level indices, we calculate the comprehensive evaluation value. For instance, the fuzzy comprehensive evaluation set for product and service quality experience can be obtained as follow:

$$\begin{array}{c}{B}_1^1={\mathrm{A}}_1\ast R_1^1=\left(0.1000.0990.1010.0960.0880.0880.0910.0900.0790.0900.078\right)\\ \ast \left(\begin{array}{ccccc}0.025& 0.075& 0.050& 0.300& 0.550\\ 0.025& 0.050& 0.050& 0.175& 0.700\\ 0.025& 0.050& 0.050& 0.150& 0.725\\ 0.025& 0.075& 0.025& 0.325& 0.550\\ 0.000& 0.050& 0.075& 0.275& 0.600\\ 0.050& 0.050& 0.075& 0.275& 0.550\\ 0.075& 0.000& 0.100& 0.275& 0.550\\ 0.050& 0.025& 0.100& 0.375& 0.450\\ 0.050& 0.050& 0.100& 0.350& 0.450\\ 0.075& 0.000& 0.125& 0.225& 0.575\\ 0.050& 0.050& 0.100& 0.275& 0.525\end{array}\right)\\ =\left(0.0400.0360.0760.2700.570\right)\end{array}$$

Similarly, we can get the other four evaluation sets:

• $B_2^1={\mathrm{A}}_2\ast R_2^1=\left(0.0170.0590.2080.3670.357\right),$
• $B_3^1={\mathrm{A}}_3\ast R_3^1=\left(0.0370.0500.2110.3380.369\right),$
• $B_4^1={\mathrm{A}}_4\ast R_4^1$ $=\left(0.0680.0260.1030.3250.479\right)$, and
• $B_5^1={\mathrm{A}}_5\ast R_5^1=\left(0.0590.0090.1230.4340.376\right).$

According to the maximum membership grade principle, in the five Ctrip’s first-level indices, the product and service quality experience and the infrastructure experience are “v₅” (very satisfied), and the personalization experience, product operations experience, and the interactive experience are “v₄” (satisfied).

(2) Qunaer’s

Similar to the procedure applied for Ctrip, we get the evaluation sets for Qunaer as:

• $B_1^2={\mathrm{A}}_1\ast R_1^2=\left(0.0520.0300.0750.3810.462\right),$
• $B_2^2={\mathrm{A}}_2\ast R_2^2=\left(0.0500.0410.1920.5080.209\right),$
• $B_3^2={\mathrm{A}}_3\ast R_3^2=\left(0.0500.0510.1700.4340.295\right),$
• $B_4^2={\mathrm{A}}_4\ast R_4^2$ $=\left(0.0570.0500.1240.3890.380\right)$, and
• $B_5^2={\mathrm{A}}_5\ast R_5^2=\left(0.0750.0420.0940.4830.308\right).$

Among Qunaer’s five first-level indices, the product and service quality experience is “v₅” (very satisfied) and the other four are “v₄” (satisfied).

4.3.2. Second-Level Index Fuzzy Comprehensive Evaluation

(1) Ctrip

We construct Ctrip’s second-level fuzzy comprehensive evaluation single factor matrix $R^1$ as

$$R^1=\left(\begin{array}{l}{B}_1^1\\ B_2^1\\ B_3^1\\ B_4^1\\ B_5^1\end{array}\right)=\left(\begin{array}{ccccc}0.040& 0.036& 0.076& 0.270& 0.570\\ 0.017& 0.059& 0.208& 0.367& 0.357\\ 0.037& 0.050& 0.211& 0.338& 0.369\\ 0.068& 0.026& 0.103& 0.325& 0.479\\ 0.059& 0.009& 0.123& 0.434& 0.376\end{array}\right)$$

Then, continue to get the second level fuzzy comprehensive evaluation set as

$$\begin{array}{ll}{B}^1=\mathrm{A}\ast R^1& =\left(0.3700.1730.1730.1570.127\right)\ast \left(\begin{array}{ccccc}0.040& 0.036& 0.076& 0.270& 0.570\\ 0.017& 0.059& 0.208& 0.367& 0.357\\ 0.037& 0.050& 0.211& 0.338& 0.369\\ 0.068& 0.026& 0.103& 0.325& 0.479\\ 0.059& 0.009& 0.123& 0.434& 0.376\end{array}\right)\\ & =\left(0.0420.0370.1320.3280.460\right)\end{array}$$

According to the maximum membership grade principle, Ctrip’s second-level indices are “v₅” (very satisfied).

(2) Qunaer

Qunaer’s second level fuzzy comprehensive evaluation single factor matrix $R^2$ is

$$R^2=\left(\begin{array}{l}{B}_1^2\\ B_2^2\\ B_3^2\\ B_4^2\\ B_5^2\end{array}\right)=\left(\begin{array}{ccccc}0.052& 0.030& 0.075& 0.381& 0.462\\ 0.050& 0.041& 0.192& 0.508& 0.209\\ 0.050& 0.051& 0.170& 0.434& 0.295\\ 0.057& 0.050& 0.124& 0.389& 0.380\\ 0.075& 0.042& 0.094& 0.483& 0.308\end{array}\right)$$

Its second-level fuzzy comprehensive evaluation set is

$$\begin{array}{ll}{B}^2=\mathrm{A}\ast R^2& =\left(0.3700.1730.1730.1570.127\right)\ast \left(\begin{array}{ccccc}0.052& 0.030& 0.075& 0.381& 0.462\\ 0.050& 0.041& 0.192& 0.508& 0.209\\ 0.050& 0.051& 0.170& 0.434& 0.295\\ 0.057& 0.050& 0.124& 0.389& 0.380\\ 0.075& 0.042& 0.094& 0.483& 0.308\end{array}\right)\\ & =\left(0.0550.0400.1220.4260.357\right)\end{array}$$

According to the maximum membership grade principle, Qunaer’s second level indexes are also “v₅” (very satisfied).

4.3.3. Fuzzy Comprehensive Evaluation Score

Finally, we normalize the vector of the evaluation matrix by setting different values for v according to five levels respectively, i.e., “v₁” = 1, “v₂” = 2, “v₃” = 3, “v₄” = 4, and “v₅” = 5. Therefore, obtaining and using the score vector S = (1 2 3 4 5), we multiple it to the fuzzy comprehensive evaluation matrix and get the final score.

(1) Ctrip’s final fuzzy comprehensive evaluation score is

$$Y^1=R^1\ast S^T=\left(\begin{array}{ccccc}0.040& 0.036& 0.076& 0.270& 0.570\\ 0.017& 0.059& 0.208& 0.367& 0.357\\ 0.037& 0.050& 0.211& 0.338& 0.369\\ 0.068& 0.026& 0.103& 0.325& 0.479\\ 0.059& 0.009& 0.123& 0.434& 0.376\end{array}\right)\ast \left(\begin{array}{l}1\\ 2\\ 3\\ 4\\ 5\end{array}\right)=\left(\begin{array}{l}4.272\\ 4.011\\ 3.971\\ 4.121\\ 4.064\end{array}\right)$$

which shows that Ctrip’s product and service quality experience score is 4.272, personalization experience is 4.011, product operations experience is 3.971, infrastructure experience is 4.121, and interactive experience is 4.064. Therefore, Ctrip’s final score of fuzzy comprehensive evaluation is

$$Z^1=B^1\ast S^T=\left(0.0420.0370.1320.3280.460\right)\ast \left(\begin{array}{l}1\\ 2\\ 3\\ 4\\ 5\end{array}\right)=4.125.$$

(2) Qunaer’s final fuzzy comprehensive evaluation score is

$$Y^2=R^2\ast S^T=\left(\begin{array}{ccccc}0.052& 0.030& 0.075& 0.381& 0.462\\ 0.050& 0.041& 0.192& 0.508& 0.209\\ 0.050& 0.051& 0.170& 0.434& 0.295\\ 0.057& 0.050& 0.124& 0.389& 0.380\\ 0.075& 0.042& 0.094& 0.483& 0.308\end{array}\right)\ast \left(\begin{array}{l}1\\ 2\\ 3\\ 4\\ 5\end{array}\right)=\left(\begin{array}{l}4.172\\ 3.782\\ 3.874\\ 3.984\\ 3.908\end{array}\right)$$

which indicates that Qunaer’s product and service quality experience score is 4.172, personalization experience is 3.782, product operations experience is 3.874, infrastructure experience is 3.984, and interactive experience is 3.908. Therefore, Qunaer’s final score of fuzzy comprehensive evaluation is

$$Z^2=B^2\ast S^T=\left(0.0550.0400.1220.4260.357\right)\ast \left(\begin{array}{l}1\\ 2\\ 3\\ 4\\ 5\end{array}\right)=3.990.$$

4.4. Analysis of Results

Summarizing the results derived from the fuzzy vectors of Ctrip’s and Qunaer’s m-commerce platforms,

Table 5. Comparison of the evaluation results.

Evaluation Target	Ctrip Travel		Qunaer Travel
	Result	Score	Result	Score
product and service quality experience	Very satisfied	4.272	Very satisfied	4.172
personalization experience	Very satisfied	4.011	satisfied	3.782
product operations experience	satisfied	3.971	satisfied	3.874
infrastructure experience	Very satisfied	4.121	satisfied	3.984
interactive experience	Very satisfied	4.064	satisfied	3.908
overall	Very satisfied	4.125	satisfied	3.990

demonstrates that both Ctrip and Qunaer perform well in terms of product and service quality experience as they both get a high score. Ctrip is better than Qunaer in the aspect of personalization experience, infrastructure experience, and interactive experience.

The overall score can be seen as a fuzzy measurement of a platform’s performance in general. Ctrip scores 4.125, higher than Qunaer’s score (3.990), but the difference is quite small. Ctrip Travel, the most authoritative tourism m-commerce company in China, has an excellent reputation, which is why it can continuously attract tourists and increase customer loyalty. Originated from the early development of mobile terminals, Qunaer Travel started to compete in the tourism market later than Ctrip. However, by fully exploiting the opportunities in the m-commerce market, Qunaer Travel has quickly caught up and diminished its distance with the traditional online enterprises represented by Ctrip Travel.

All membership degrees of the first-level and second-level indices are better than “normal”. Since Ctrip Travel and Qunaer Travel are the leading enterprises in China’s online travel market, our results show that the consumers in this market are overall satisfied. When China’s tourism m-commerce progresses toward its maturity, we will continue to observe the improvement with respect to the quality of tourism products and services to meet the diverse needs of tourists.

5. Conclusions

Prior research on user experience has mostly focused on website design, recommendation systems, and technology products; the effects of the tourists’ experience have not been formally incorporated into mobile travel services. This research makes contribution to the literature by presenting and studying a tourism m-commerce platform evaluation framework. In particular, based on prior literature and relevant theories, we identify 24 preliminary evaluation indices. Using online questionnaires and exploratory factor analysis method, we extract from the 24 preliminary evaluation indices five experience-based principal components, including interactive, infrastructure, personalization, product and service quality, and product operations experience. In addition, we apply the confirmatory factor analysis method to test the robustness of the proposed evaluation framework. Our test result shows that the evaluation framework is both robust and effective. Finally, we empirically evaluate the m-commerce platforms of Ctrip and Qunaer by using our proposed evaluation framework in combination with the fuzzy comprehensive evaluation method. The insights derived from our study, however, are only our initial attempt to understand the factors influencing the performance of tourism m-commerce platforms. Future research may overcome some of the limitations to further extend and improve our evaluation framework. For instance, most of the respondents to our questionnaire were college students, which might result in the partiality of the survey results and our analysis. In addition, we may need to further refine the process of identifying and selecting the preliminary factors to make our evaluation framework more comprehensive.