CHECKHI: A Checklist for Evaluating the Customer Experience of Tourism in the Hotel Industry

Abstract

Evaluating the customer experience (CX) when guests interact with different products or services helps to understand their perceptions and responses during their journey. Nevertheless, most evaluation methods/instruments used in the hotel sector only focus on individual touchpoints of a customer’s journey, not the whole CX. This article presents CHECKHI: a novel checklist for evaluating the CX of the tourism area, specifically in the hotel industry. CHECKHI is composed of 102 items divided into seven touchpoints that cover the entire guest journey (before, during, and after the stay). We adapted and applied an eight-stage methodology to develop, validate, and refine CHECKHI, within two iterations. We performed several experiments to verify the content validity of CHECKHI using the Delphi method and interviews. The results obtained in the validations of CHECKHI items demonstrate its representativeness, clarity, and usefulness; this makes it possible to evaluate the CX at the touchpoints identified, regardless of the hotel type. CHECKHI could be useful for companies and/or organizations that offer hotel services and require evaluations of the CX; academics who need to create new evaluation instruments and can use CHECKHI as a reference; and professionals who need to learn about CX evaluation in the hotel sector.

1. Introduction

The tourism area represents a service industry composed of different sectors, such as accommodation, food and beverage, recreation and entertainment, transportation, and travel services. These multiple sectors are the main source of income for various countries and companies [1]. In the hotel industry, the customer experience (CX) through products, systems, or services directly influences how a guest feels and will behave in their following interactions with a hotel, especially when addressing issues related to sustainability, such as inclusion and accessibility. Therefore, hotels have started to be more concerned about the CX at their multiple touchpoints throughout a customer’s journey [2].

There are several instruments that can be used for evaluating the CX in the hotel industry, such as interviews, questionnaires, focus groups, and scales [3]. Nevertheless, most of these instruments are focused on evaluating only the experience of certain specific touchpoints, not the entire CX and its dimensions [4]. For this reason, we propose CHECKHI, a novel checklist consisting of seven touchpoints and 102 items for evaluating the customer experience in the hotel industry. The objective of the study is to propose an instrument that facilitates the detection of problems that generate a bad experience for hotel guests, improve services, and, thus, offer better quality and sustainable services. This instrument evaluates the CX through multiple touchpoints existing in the hotel industry. The touchpoints considered in CHECKHI are: (1) search for information (18 items); (2) book a room (11 items); (3) check-in (17 items); (4) room stay (17 items); (5) order and receive food (17 items); (6) checkout (16 items); (7) report experience (6 items).

We adapted the methodology propose by Quiñones et al. [5,6] to develop, validate, refine, and improve CHECKHI, in two iterations. Although this methodology is focused on developing usability/user experience heuristics, we decided to use this methodology because all its stages are very detailed: (1) a definition; (2) inputs; (3) activities or tasks; (4) outputs; (5) a BPMN diagram to guide development. Due to its flexibility and the way it was designed, the methodology can be used to create checklists. The results obtained in the validations of CHECKHI items demonstrate its representativeness, clarity, and usefulness; this makes it possible to evaluate the CX at the touchpoints identified, regardless of the hotel type.

We consider that this new evaluation instrument will be useful for companies and organizations that offer hotel services and require evaluations of the CX; academics who need to create new evaluation instruments and can use CHECKHI as a reference; and professionals who need to learn about the CX’s evaluation in the hotel sector. CHECKHI contributes to the generation of new approaches and more accurate evaluations, since it incorporates key element related to the CX: touchpoints, channels, and CX dimensions. We assert that the use of CHECKHI will facilitate the timely detection of problems that generate a negative experience and, with that information, pursue effective solutions to create a more pleasant guest experience.

This article is organized as follows: Section 2 introduces the theoretical background; Section 3 presents the methodology applied to develop CHECKHI; Section 4 explores the process followed to develop CHECKHI; Section 5 describes the validations of CHECKHI; Section 6 shows the final version of CHECKHI; Section 7 presents the contributions of this research; Section 8 explains the limitations; Section 9 summarizes the conclusions and future work.

2. Background

2.1. Customer Experience

Customer experience (CX) is a concept that has aroused the interest of companies from different sectors [7]. Several authors have proposed different definitions, due to the different areas in which CX can be evaluated and its holistic nature (Lemon and Verhoef [8]). Specifically, in the hotel sector, Pen et al. [9] define the CX as “guests’ emotional evaluations of their consumption episode”.

Meyer and Schwager [10] define CX as “the internal and subjective response customers have to any direct or indirect contact with a company. Direct contact generally occurs in the course of purchase, use, and service and is usually initiated by the customer. Indirect contact most often involves unplanned encounters with representations of a company’s products, services, or brands and takes the form of word-of-mouth recommendations or criticisms, advertising, news reports, reviews, and so forth”. Meyer and Schwager [10] stress the subjective character of CX and declare that it occurs with both direct and indirect contact with a company.

On the other hand, Gentile et al. [11] states that the CX “originates from a set of interactions between a customer and a product, a company, or part of its organization, which provoke a reaction. This experience is strictly personal and implies the customer’s involvement at different levels (rational, emotional, sensorial physical and spiritual)”. Gentile et al. [11] approach their definitions from the creation of the CX and emphasize the customer’s involvement at different levels. In this sense, several authors have proposed dimensions or attributes to describe the CX (see

Table 1. Customer experience dimensions.

Author (Year)	Domain	Dimensions
Schmitt (1999) [12]	Marketing	Sense, Feel, Think, Act, Relate
Gentile et al. (2007) [11]	General	Sensorial, Emotional, Cognitive, Pragmatic, Lifestyle, Relational
Verhoef et al. (2009) [13]	Retail	Cognitive, Affective, Emotional, Social, Physical
Hosany and Witham (2010) [14]	Tourism and Cruising	Education, Entertainment, Esthetics, Escapism
Nambisan and Watt (2011) [15]	Online Environments	Pragmatic, Hedonic, Sociability, Usability

).

We used the dimensions proposed by Gentile et al. [11] to develop CHECKHI, since they are defined in a general way, allowing their application to different contexts. Each dimension is briefly described below.

• Sensorial: Dimension that aims to provide a good sensorial experience through any of the five senses (sight, hearing, touch, taste, and smell). For instance: aesthetical pleasure, excitement, satisfaction, or sense of beauty.
• Emotional: Dimension related to the generation of moods, feelings, or emotions to create an affective relation with the company, its brand, or its products.
• Cognitive: Dimension related to the customer’s thinking and conscious mental processes, engaging customers by using their creativity or problem-solving abilities.
• Pragmatic: Dimension associated to practical act of doing something and the concept of usability through all the product life-cycle stages.
• Lifestyle: Dimension that refers to values and beliefs shared by the company and the customer through the product, its consumption, and/or its use.
• Relational: Dimension that involves the customer’s social context, their relationships with other people, and their own ideals.

The experience that a customer may have arises when they interact with an organization or company through products, systems, or services. These interactions are called “touchpoints” and can occur through multiple channels [16,17]. Stein and Ramaseshan [18] state that “customers have experiences every time they ‘touch’ any part of the product, service, brand or organization, across multiple channels and at various points in time. Such moments of truth between the customer and any part of the company are known as touchpoints”. We developed CHECKHI by incorporating items that allow for evaluating the CX through different touchpoints of the hotel industry (see Section 6). In addition, when creating CHECKHI, we considered a channel as the means by which an interaction occurs, and, therefore, an experience is produced (physical channels such as: customer service and a physical reservation; digital channels such as: websites, email, and social networks). For each touchpoint raised in CHECKHI, the channel through which the interaction occurs is included (see Section 6).

2.2. Customer Experience Evaluation

There are multiple methods, instruments, and metrics to evaluate the CX (see Table 2). It is common to use methods designed to evaluate more particular aspects (such as usability, user experience, satisfaction, etc.), since the CX is considered a broader concept composed of various factors or dimensions. Due to the above, it may be difficult to identify which of them is the most effective for evaluating the CX or, even more so, which of them allows for evaluating the total experience of the customer at different moments of interaction.

On the other hand, the literature related to the constructs that allow for evaluating the CX in the tourism industry is limited [19]. To deliver a positive guest experience, academics and practitioners need to know how to assess the CX to capture all facets of the experience [19]. We propose CHECKHI to cover this gap, including items that allow for evaluating different CX dimensions related to the hotel industry.

Table 2. Methods, instruments, and metrics used to evaluate the CX.

Instruments	Type	Description
Interviews [3]	Method	A series of questions is compiled and then asked to participants, usually in person or online.
Questionnaires [3]	Method	These are lists of questions. There are mainly two different approaches: fixed-answer questionnaires and open-ended questionnaires.
Focus Group [3]	Method	A group of participants discuss a particular topic. A moderator guides the activity, and a small group of people (6–12 people) interacts with themselves.
Heuristic Evaluation [20]	Method	An inspection method that allows for finding the potential usability/UX problems in the design of a user interface. A small group of usability experts judges a user interface to check if it complies with the principles of usability design regarding certain heuristics.
SERVQUAL [21]	Instrument	A 22-item instrument used for assessing customer perceptions of service quality in service and retailing organizations. These items are distributed in five dimensions: Reliability, Responsiveness, Assurance, Empathy, and Tangibles.
System Usability Scale (SUS) [22]	Instrument	Instrument used to measure usability perception. It is composed of 10 Likert-scale questions and produces a score from 0–100, where individual item scores are not significant on their own.
Set of Heuristics [23]	Instrument	Principles used in heuristic evaluation as an instrument for detecting usability/UX problems. Nielsen’s 10 heuristics are widely used to evaluate any kind of system.
Net Promoter Score (NPS) [24]	Metric	NPS is an index ranging from −100 to 100 that measures the willingness of customers to recommend a product, service, or company to others, allowing companies to know their customers’ loyalty.
Customer Satisfaction (CSAT) [25]	Metric	Metric that measures customer satisfaction with a product or service on a set scale during consumption or post-consumption.
Customer Effort Score (CES) [26]	Metric	Metric that measures how easy it was for a customer to interact with a product or service.

Table 2. Methods, instruments, and metrics used to evaluate the CX.

Instruments	Type	Description
Interviews [3]	Method	A series of questions is compiled and then asked to participants, usually in person or online.
Questionnaires [3]	Method	These are lists of questions. There are mainly two different approaches: fixed-answer questionnaires and open-ended questionnaires.
Focus Group [3]	Method	A group of participants discuss a particular topic. A moderator guides the activity, and a small group of people (6–12 people) interacts with themselves.
Heuristic Evaluation [20]	Method	An inspection method that allows for finding the potential usability/UX problems in the design of a user interface. A small group of usability experts judges a user interface to check if it complies with the principles of usability design regarding certain heuristics.
SERVQUAL [21]	Instrument	A 22-item instrument used for assessing customer perceptions of service quality in service and retailing organizations. These items are distributed in five dimensions: Reliability, Responsiveness, Assurance, Empathy, and Tangibles.
System Usability Scale (SUS) [22]	Instrument	Instrument used to measure usability perception. It is composed of 10 Likert-scale questions and produces a score from 0–100, where individual item scores are not significant on their own.
Set of Heuristics [23]	Instrument	Principles used in heuristic evaluation as an instrument for detecting usability/UX problems. Nielsen’s 10 heuristics are widely used to evaluate any kind of system.
Net Promoter Score (NPS) [24]	Metric	NPS is an index ranging from −100 to 100 that measures the willingness of customers to recommend a product, service, or company to others, allowing companies to know their customers’ loyalty.
Customer Satisfaction (CSAT) [25]	Metric	Metric that measures customer satisfaction with a product or service on a set scale during consumption or post-consumption.
Customer Effort Score (CES) [26]	Metric	Metric that measures how easy it was for a customer to interact with a product or service.

2.3. Tourism and Hotel Industry

The tourism industry is one of the most important economic activities for many countries, since it create jobs, drives exports, and generates prosperity across the world [1]. Tourism includes “the industry of travel, hotels, transportation, and all other components, including promotion, that serves the needs and wants of travelers” [27].

Within tourism, the accommodation sector is composed of different establishments: hotels, motels, inns, bed and breakfasts, guest houses, and hostels. A hotel is a building that provides guests with a place to stay overnight in exchange for money [28]. Hotels can provide different additional services according to their classification (star rating). Nonetheless, they share similar features regardless of their classification. Customer interactions with hotels can take place across different channels. For the digital channel (hotel websites), Law et al. [29] propose five features for the hotel industry.

• Reservation information: Related to the information and facilities presented when customers make room reservations at the hotel.
• Facilities information: Related to the general information of the hotel, information about the services being offered, and the amenities available to customers.
• Contact information: Related to the information available for direct communication between the hotel and customers.
• Website management: Related to the maintenance of the website, the present functionalities, and the efficient and effective use of this for customers.
• Surrounding area information: Related to tourist information about places of interest in the city or near the hotel for customers.

For the physical channels (e.g., hotel facilities), Rojas et al. [30] propose nine features for the hotel industry.

• Cleanliness: Related to the hotel, which needs to be completely tidy and clean, especially those sectors where customers interact, for instance, the front desk, restaurant, and hotel rooms, among others.
• Speed of service: Related to all the actions in which the customers interact with the hotel, which are to be carried out in the most effective and efficient way possible.
• Staff professionalism: Related to the behavior of the staff toward the customer, which must be totally professional, both in the manner of communicating and in the intention of fulfilling the needs of the customers.
• Comfort: Related to how the hotel and mainly its rooms should create an atmosphere of ease and harmony for customers during their stay.
• Food service: Related to the different characteristics of the food that customers need when interacting with the hotel.
• Amenities: Related to the products/utensils that the hotel make available to the customers to facilitate their stay and make it more comfortable and pleasant.
• Security: Related to the common areas of the hotel and its rooms, which should provide security to the customers, ensuring that they do not feel any danger or hostility during their stay.
• Accessibility: Related to the different facilities and areas of the hotel, which must be designed to be used by any type of customer equally regardless of their condition.
• Hotel professionalism: Related to the additional activities/tasks carried out by the hotel, with the aim of meeting the needs of its customers for a specific interaction and help generate a better CX.

We considered the features presented above to develop CHECKHI. We present the items categorized by touchpoints and digital/physical channels (see Section 6).

2.4. Related Work

After reviewing the literature, we noticed that there are no checklists for evaluating the CX in the hotel industry. Several studies propose scales or questionnaires for assessing the CX, but mainly focus on service quality or satisfaction [31,32,33,34,35,36,37,38]. The CX involves several other dimensions in addition to quality and satisfaction [39,40], such as innovation [41], sustainability [42], and accessibility [43]. Therefore, these scales may not fully evaluate the entire experience a customer has throughout their journey and the moments of interaction with a product, system, or service being offered by the hotel [44].

For instance, Klaus [31] proposes EXQ, a scale to measure the CX quality. The instrument was validated and refined several times [32,33], and it includes 25 items grouped into three dimensions: brand experience (with 7 items), service purchase experience (with 11 items), and post-purchase experience (with 7 items). It is interesting to highlight that this instrument has been applied in several sectors (such as tourism, retail, and sporting events, among others), and the scale covers the three stages of the CX (pre-purchase, purchase, and post-purchase). In addition, it has been validated and refined in several stages, making it an instrument with a validity and effectiveness that have been proven. However, it focuses on the areas of marketing and quality of service, leaving out other CX dimensions. On the other hand, it is not particularized for hotels, so some features of this domain may not be covered by the scale.

The 102 items that CHECKHI contains have been proposed in such a way that they consider: (1) seven moments of interaction (touchpoints) that cover the entire guest journey; (2) six CX dimensions that describe the holistic guest experience; and (3) nine hotel features that characterize this type of sector. After validations and refinements were performed, we conclude that CHECKHI holistically evaluates the CX in hotels throughout all its stages, including the interactions between the guest and the hotel in its different channels.

Regarding scales aimed at evaluating the service quality in hotels, Getty and Getty [34] developed the Lodging Quality Index (LQI), a scale composed of 26 items for measuring customers’ perception of service quality in the lodging industry; Knutson et al. [35] created LODGSERV, a 26-item index designed to measure customers’ expectations for service quality in the hotel experience; Mei et al. [36] proposed HOLSERV, a 27-item scale for measuring service quality in the hospitality industry; Hahn et al. [37] established a scale to measure e-Service Quality (e-SQ) with hotel websites; and Minh et al. [38] proposed a 21-item scale to investigate the impact of service quality dimensions on customer satisfaction in hotels.

Scriven [45] describes a checklist as “a list of factors, properties, aspects, components, criteria, tasks, or dimensions, the presence or amount of which are to be separately considered, in order to perform a certain task”. Moreover, the author identifies several types of checklists: laundry lists, sequential checklists (strongly or weakly sequential), iterative checklists, diagnostic checklists, and criteria of merit checklists (COMlist). This checklist can be used as a main or supporting instrument to evaluate the different interactions by customers with a hotel. However, reviewing the literature, we could not find any checklists for evaluating the CX in any specific domain. Instead, we note that it is common to develop and apply scales for evaluating customers perceptions with hotels.

Getty and Getty [34] developed the Lodging Quality Index (LQI) for measuring customers’ perception of service quality (SQ) in the lodging industry. The first version of this scale took as reference the original 10 dimensions proposed in SERVQUAL [21]. However, after reliability and validity tests, the scale comprised 26 items grouped in five dimensions: (1) tangibility; (2) reliability; (3) responsiveness; (4) confidence; (5) communication. Knutson et al. [35] created LODGSERV, a 26-item index designed to measure customers’ expectations for service quality in the hotel experience. Since it is a variation of SERVQUAL, it uses the same five dimensions: (1) tangibility; (2) reliability; (3) responsiveness; (4) assurance; (5) empathy. Nevertheless, it changes several measurement questions of the index.

Mei et al. [36] proposed HOLSERV, a 27-item scale for measuring service quality in the hospitality industry. This scale was developed using SERVQUAL dimensions and items and extends those by including eight new items to cover additional aspects. Hahn et al. [37] presented a scale to measure e-Service Quality (e-SQ) with hotel websites. To achieve this, the authors defined 24 items across six dimensions: functionality, atmospheric quality, reliable information, locality information, customer reviews, and emotional engagement. Minh et al. [38] proposed a scale to investigate the impact of service quality dimensions on customer satisfaction in hotels. This scale takes as a reference four scales: SERVQUAL [21], LODGSERV [35], HOLSERV [36], and the Lodging Quality Index [34]. Thus, this scale consists of 21 items distributed in the five dimensions of SERVQUAL.

Due to the lack of a checklist for evaluating the CX in the hotel industry, we used the scales proposed by Hahn et al. [37] and Minh et al. [38] for developing CHECKHI. This is because their dimensions and items can be applied for evaluating the CX through the digital and physical channels of a hotel. The dimensions present in each scale are briefly presented in

Table 3. Scales selected for creating CHECKHI.

Scale	Dimension	Description
e-Service Quality [37]	Functionality	Related to the ease of use and the different functionalities present on the hotel website.
	Reliable Information	Related to the information on the hotel website, which must be accurate, up-to-date, specific, and diverse.
	Locality Information	Related to the information present on the hotel website regarding the location and the area surrounding the hotel.
	Atmospheric Quality	Related to the atmosphere perceived by the customer when entering the hotel website.
	Customer Reviews	Related to the presence of customer reviews on the hotel website.
	Emotional Engagement	Related to the emotions present in customers when using the hotel website.
Service Quality [38]	Tangible	Related to the appearance of the hotel, the staff, and the facilities present in the common sectors, rooms, and bathrooms, among others.
	Reliability	Related to the hotel’s ability to perform its services effectively and efficiently.
	Responsiveness	Related to the hotel’s willingness and flexibility to serve and help its customers.
	Assurance	Related to the hotel’s ability to build trust in customers through its services being offered or the knowledge and professionalism of the staff.
	Empathy	Related to the attention and care of the hotel to each customer.

.

3. Methodology Applied to Develop CHECKHI

We developed CHECKHI using the eight-stage methodology proposed by Quiñones et al. [5,6]. We decided to use this methodology because of the following reasons.

• All its stages are very detailed: (1) a definition; (2) inputs; (3) activities or tasks; (4) outputs; (5) a BPMN diagram to guide the development. Additionally, the way it was designed allows for it to be easily iterated to enhance the instrument through new validations or refinements.
• It is one of the most detailed and specified methodologies for developing heuristics [46].
• Although it is focused on developing usability/user experience heuristics [5,6], it is flexible and can be adapted: (1) to develop other types of instruments (such as principles [47] and guidelines [48]); for use in different domains (such as usability [47] and UX [48]); (3) to incorporate several of its stages in other methodologies or processes in a simple way [46].
• Several studies have used it and have obtained good results, creating new evaluation instruments such as heuristics and checklists [49,50,51,52].

For example, Granollers [47] adapted the methodology [5,6] to propose a list of heuristic principles (heuristics with associated questions) to evaluate user interfaces, defining three stages: review, compare similarities, and integration. The authors conclude that the proposed principles provide a complete usability/UX evaluation methodology for modern and future user interfaces. On the other hand, Vi et al. [48] adapted the methodology [5,6] to create guidelines for designing extended reality (XR) applications that serve as a reference to build this type of application. Further, Kaya et al. [46] applied the specification, validation, and refinement stages of the methodology [5,6] together with other steps to develop their heuristics for the set-top box and TV interfaces.

To establish CHECKHI, we adapted some of the original stages by changing the approach from the usability/user experience to the CX and by considering other inputs, activities, and outputs. The methodology adaptations are shown in

Table 4. Methodology applied to develop CHECKHI.

Methodology Stage	Description of the Adapted Stage
Step1: Exploratory Stage	We conducted a literature review about hotel features, CX/SQ/e-SQ dimensions, and touchpoints.
Step2: Experimental Stage	We collected additional information through a survey.
Step 3: Descriptive Stage	We selected, prioritized, and grouped the information collected about hotel features, CX/SQ/e-SQ dimensions, and touchpoints.
Step 4: Correlational Stage	We performed correlations among touchpoints, features, and CX/SQ/e-SQ dimensions.
Step 5: Selection Stage	We chose an action for each item of the SQ/e-SQ scales: Keep, Eliminate, Adapt, and Create.
Step 6: Specification Stage	We formally specified CHECKHI by defining touchpoints, channels, website/hotel features, CX dimensions, SQ/e-SQ dimensions, items, and hotel response (yes, partially, no, and N/A).
Step 7: Validation Stage	We verified the content validity (representativeness, clarity, and usefulness) of CHECKHI through the Delphi method and interviews.
Step 8: Refinement Stage	We defined CHECKHI items as Keep, Eliminate, Adapt, and Create.

.

4. Development of CHECKHI

We developed CHECKHI through two iterations: (1) the first iteration aimed to propose the first two versions of CHECKHI by performing the eight stages defined in the methodology; (2) the second iteration consisted of validating and refining the second version of CHECKHI to present the third and final version. Figure 1 illustrates the stages and iterations conducted to develop CHECKHI.

The development of CHECKHI was based on the following information: (1) touchpoints; (2) channels; (3) website/hotel features; (4) CX/SQ/e-SQ dimensions. Our strategy consisted of defining stages (before, during, and after the stay) where we identified touchpoints and channels through a literature review and a survey (see Figure 2). Subsequently, each touchpoint was associated with the different website/hotel features and the CX/SQ/e-SQ dimensions affecting the touchpoint and its channels. During the development of CHECKHI, we created three versions, where each had a different ID for items at its touchpoints (see

Table A1. IDs used in each version of CHECKHI.

Touchpoint	First Version	Second Version	Third Version
TP1: Search for information	SS	SE	SI
TP2: Book a room	BB	BO	BR
TP3: Check-in	IN	CHI	CI
TP4: Room stay	RR	RO	RS
TP5: Order and receive food	OO	OF	OR
TP6: Checkout	OU	CHO	CO
TP7: Report experience	RP	RX	RE

). The three versions of CHECKHI are briefly described below.

• First version of CHECKHI: We developed this version in the first iteration of “Step 6: Specification stage”. It was designed by taking as a reference the items of the e-SQ [37] and SQ scales [38]. As result, this version is composed of 106 items grouped into seven touchpoints.
• Second version of CHECKHI: We established this version at the end of the first iteration. It was created by validating and refining the first version of CHECKHI using the Delphi method. As result, after multiple modifications, this version is composed of 104 items in seven touchpoints.
• Third and final version of CHECKHI: We proposed this version at the end of the second iteration. It was developed by validating and refining the second version of CHECKHI by conducting interviews with practitioners. The third and final version of CHECKHI consists of 102 items in 7seventouchpoints.

4.1. First Iteration

In the first iteration, we performed all the stages of the methodology to develop the first and second versions of CHECKHI. In “Step 1: Exploratory stage”, we carried out a literature review to obtain information regarding the different key concepts involved in the creation of CHECKHI: CX, touchpoints, CX evaluation, tourism and hotel industry, and related work (see Section 2). In “Step 2: Experimental stage”, we conducted a survey to identify the touchpoints, channels, and emotions in the customer journey. This survey was focused on obtaining information about the CX at the different touchpoints present before, during, and after the stay in a hotel. We identified 13 touchpoints on the customer journey at each stage related to hotel stay, which were grouped and renamed into 7 touchpoints to evaluate the CX in hotels using CHECKHI (see Figure 2). In “Step 3: Descriptive stage”, we selected, prioritized, and grouped the information collected in the previous stages (touchpoints, website/hotel features, and CX/SQ dimensions) using a three-level scale (1—not important to 3—highly important). We selected the following information: (1) touchpoints (search for information, book a room, check-in, room stay, order and receive food, checkout, and report experience); (2) CX dimensions (sensorial, cognitive, pragmatic, lifestyle, and relational); (3) website features (facilities information, contact information, website management, surrounding area information, and reservation information); (4) hotel features (comfort, cleanliness, speed of service, amenities, food service, hotel professionalism, staff professionalism, security, and accessibility); (5) SQ dimensions (tangibility, reliability, responsiveness, assurance, and empathy); (6) e-SQ dimensions (functionality, reliable information, locality information, and customer reviews).

In “Step 4: Correlational stage”, we matched each CX/SQ/e-SQ dimension and website/hotel feature with the selected touchpoints. We performed this match to determine which hotel features (or hotel website) and which CX/e-SQ dimensions would be assessed with CHECKHI at each touchpoint. Regarding e-SQ dimensions and website features, we associated them with the following touchpoints: search for information, book a room, and report experience. Concerning SQ dimensions and hotel features, we relates them to the following touchpoints: check-in, room stay, order and receive food, and checkout. For the CX dimensions, they were related to the touchpoints, regardless of the channel.

Table A2. Correlations made in “Step 4: Correlational stage”.

Touchpoint	Website/Hotel Features	CX Dimensions	SQ Dimensions	E-SQ Dimensions
TP1: Search for information	Facilities Information	Cognitive	-	Functionality
	Contact Information	Pragmatic		Reliable Information
	Website Management	Lifestyle		Locality Information
	Surrounding Area Information			Customer Reviews
TP2: Book a room	Facilities Information	Cognitive	-	Functionality
	Reservation Information	Pragmatic		Reliable Information
	Website Management
TP3: Check-in	Cleanliness	Sensorial	Tangible	-
	Service Speed	Relational	Reliability
	Staff Professionalism		Responsiveness
	Accessibility		Assurance
TP4: Room stay	Amenities	Sensorial	Tangible	-
	Cleanliness	Pragmatic	Assurance
	Comfort		Empathy
	Security
	Accessibility
TP5: Order and receive food	Food Service	Sensorial	Reliability	-
	Service Speed	Cognitive	Responsiveness
	Staff Professionalism	Relational	Assurance
			Empathy
TP6: Checkout	Cleanliness	Sensorial	Tangible	-
	Service Speed	Relational	Reliability
	Staff Professionalism		Responsiveness
	Accessibility		Assurance
TP7: Report experience	Hotel Professionalism	Lifestyle	Empathy	Customer Reviews

shows an overview of the correlations made at this stage. In “Step 5: Selection stage”, we decided to eliminate, adapt, keep, or create the items from the SQ/e-SQ scales to develop the first version of CHECKHI. Regarding the e-SQ scale (24 items), we kept 6 items, adapted 8 items, eliminated 10 items (since these were already considered in other items or were related to subjective factors that cannot be evaluated using a checklist), and created 10 items (see

Table A3. Actions chosen for E-SQ scale items in “Step 5: Selection stage”.

ID	Item Name/Explanation	Action	Feature Covered	Applicability
FU1	Booking system of the site is easy to use	Eliminate (considered in another item)	-	-
FU2	Site provides a service guarantee for online booking	Adapt	Reservation Information	(3) Critical
FU3	Site gives confirmation promptly when I book	Adapt	Contact Information	(2) Important
FU4	Site does not share my personal information with other sites	Adapt	Website Management	(3) Critical
FU5	I can check booking availability in real time	Adapt	Reservation Information	(3) Critical
FU6	Site provides a function to calculate the price	Adapt	Reservation Information	(3) Critical
FU7	Site makes it easy to find what I need quickly	Adapt	Facilities Information	(2) Important
RI1	Site provides accurate information	Adapt	Facilities Information	(3) Critical
RI2	Site provides up-to-date information	Keep	Website Management	(3) Critical
RI3	Site does not exaggerate its information	Eliminate (considered in another item)	-	-
RI4	Site provides room rate information	Adapt	Reservation Information	(3) Critical
LI1	Site provides sightseeing/surroundings information	Keep	Surrounding Area Information	(2) Important
LI2	Site provides a map of the area around the hotel	Keep	Contact Information	(2) Important
LI3	Site provides transportation information	Keep	Surrounding Area Information	(2) Important
AT1	Site has a lively atmosphere	Eliminate (item related to subjective factor)	-	-
AT2	Site has a unique atmosphere	Eliminate (item related to subjective factor)	-	-
AT3	Site has a modern atmosphere	Eliminate (item related to subjective factor)	-	-
AT4	Site has a luxurious atmosphere	Eliminate (item related to subjective factor)	-	-
CR1	Information given by other customers that can increase my trust in the expected hotel service	Eliminate (item related to subjective factor)	-	-
CR2	Site shows feedback from other customers	Keep	Contact Information	(3) Critical
CR3	Site provides both customers’ positive and negative reviews	Keep	Contact Information	(2) Important
EE1	Pictures or music make me feel happy	Eliminate (item related to subjective factor)	-	-
EE2	Site’s clean graphic style keeps me calm	Eliminate (item related to subjective factor)	-	-
EE3	Opportunity to take a virtual tour makes me excited	Eliminate (item related to subjective factor)	-	-
N1	Website provides visual content related to the hotel	Create	Facilities Information	(3) Critical
N2	Website has a multilingual version	Create	Website Management	(3) Critical
N3	Website provides contact information	Create	Contact Information	(2) Important
N4	Website clearly indicates the booking terms	Create	Reservation Information	(3) Critical
N5	Website has an aesthetic and minimalist design	Create	Website Management	(2) Important
N6	Website indicates the general terms and conditions	Create	Facilities Information	(3) Critical
N7	Website allows different payment options	Create	Reservation Information	(2) Important
N8	Website indicates the details of the booking before completing the process	Create	Reservation Information	(2) Important
N9	Website provides detailed room rate information	Create	Reservation Information	(3) Critical
N10	Website allows for different booking options	Create	Reservation Information	(3) Critical

). Regarding the SQ scale (21 items), we kept zero items, adapted 17 items, eliminated 4 items (since these were already considered in other items or were not originally considered to be evaluated on CHECKHI), and created 19 items (see

Table A4. Actions chosen for SQ scale items in “Step 5: Selection stage”.

ID	Item Name/Explanation	Action	Feature Covered	Applicability
TAN1	Convenient location	Eliminate (location is not considered in this checklist)	-	-
TAN2	Comfortable facilities and equipment	Adapt	Comfort	(3) Critical
TAN3	Appealing decoration	Eliminate (decoration is not considered in this checklist)	-	-
TAN4	Neat appearance of the staff	Adapt	Staff Professionalism	(3) Critical
TAN5	Visually presented brochures and directories	Adapt	Hotel Professionalism	(2) Important
TAN6	Appropriate environment for taking a rest	Adapt	Comfort	(3) Critical
REL1	Performs service accurately	Adapt	Service Speed	(3) Critical
REL2	Provides service on time	Adapt	Service Speed	(3) Critical
REL3	Solves problem sincerely	Adapt	Staff Professionalism	(3) Critical
REL4	Keeps records confidential	Adapt	Security	(3) Critical
RES1	Willingness to serve guests	Eliminate (considered in another item)	-	-
RES2	Availability to respond to guests’ requests	Eliminate (considered in another item)	-	-
RES3	Flexibility according to guests’ demands	Adapt	Staff Professionalism	(3) Critical
ASS1	Guests feel safe and secure in their stay	Adapt	Security	(3) Critical
ASS2	Staff with knowledge to provide guests information about surrounding areas (shopping, museums, places of interest, etc.)	Adapt	Staff Professionalism	(3) Critical
ASS3	Staff has occupational skills	Adapt	Staff Professionalism	(3) Critical
ASS4	Staff is courteous and polite	Adapt	Staff Professionalism	(3) Critical
EMP1	Provides guests with individual attention	Adapt	Staff Professionalism	(3) Critical
EMP2	Understands guests’ specific needs	Adapt	Staff Professionalism	(3) Critical
EMP3	Positive attitude when receive feedback from guests	Adapt	Hotel Professionalism	(2) Important
EMP4	Healthy menu	Adapt	Food Service	(2) Important
N1	Hotel rooms have the amenities promised	Create	Amenities	(3) Critical
N2	Hotel bathrooms have the amenities/toiletries promised	Create	Amenities	(3) Critical
N3	Hotel rooms’ technological amenities are modern and easy to use	Create	Amenities	(2) Important
N4	Hotel rooms are accessible inside	Create	Accessibility	(3) Critical
N5	Hotel bathrooms are accessible	Create	Accessibility	(3) Critical
N6	Hotel rooms are accessible via the elevator/lift	Create	Accessibility	(3) Critical
N7	Fresh and tasty food is prepared as requested	Create	Food Service	(3) Critical
N8	Food delivered has an appropriate temperature	Create	Food Service	(3) Critical
N9	Checkout staff clearly explains any additional charges	Create	Staff Professionalism	(3) Critical
N10	Checkout staff clearly explains the total charges	Create	Staff Professionalism	(3) Critical
N11	Checkout staff provides a copy of the invoice	Create	Staff Professionalism	(3) Critical
N12	Payment method is checked and processed discreetly	Create	Staff Professionalism	(2) Important
N13	Unnecessary red tape is eliminated	Create	Staff Professionalism	(2) Important
N14	Checkout staff provides a customer experience form	Create	Staff Professionalism	(3) Critical
N15	Hotel sends surveys to learn about customer experiences	Create	Hotel Professionalism	(3) Critical
N16	Hotel reviews customer feedback on both internal and external platforms	Create	Hotel Professionalism	(2) Important
N17	Hotel responds to customer feedback	Create	Hotel Professionalism	(1) Useful
N18	Hotel considers customer feedback to improve	Create	Hotel Professionalism	(3) Critical
N19	Hotel surveys indicate contact information for customers	Create	Hotel Professionalism	(2) Important

).

In “Step 6: Specification stage”, we designed the first version of CHECKHI, which was composed of 106 items grouped into seven touchpoints. The template suggested in the methodology proposed by Quiñones et al. [5,6] was adapted for CHECKHI specification, including six elements: (1) touchpoints; (2) channels; (3) website/hotel features; (4) CX/SQ/e-SQ dimensions; (5) items; (6) hotel response (yes, partially, no, and N/A).

In “Step 7: Validation stage”, we verified the content validity (items) and refined the first version of CHECKHI using the Delphi method. We defined three stages, adapting the application of Delphi by Kamioka et al. [53]. The details of the validation performed can be reviewed in Section 5.1.

Finally, “Step 8: Refinement stage” was conducted in parallel with “Step 7: Validation stage”, since the Delphi method has to apply, as a requirement, the changes suggested by the participants before moving onto the next Delphi stage. After the first Delphi stage, we decided to keep 43 items, modify 48 items, eliminate 15 items, and create 13 items to evaluate the CX in hotels at the proposed touchpoints. After the second Delphi stage, we considered it necessary to keep 100 items and modify 4 items. After the third Delphi stage, we received good results regarding CHECKHI items, so it was not necessary to make any changes, resulting in the second version of CHECKHI, with 104 items grouped into seven touchpoints. Further details of the validations and refinements carried out are presented in Section 5.1.

4.2. Second Iteration

In the second iteration, we performed steps 7 and 8 of the methodology (see Figure 1). This second iteration aimed to validate and refine the second version of CHECKHI. In “Step 7: Validation stage”, we validated CHECKHI through practitioners’ interviews. The details of the validation performed can be reviewed in Section 5.2.

Finally, in “Step 8: Refinement stage”, we refined the items based on the qualitative and quantitative analyses obtained in step 7. We decided to check 13 items that presented relevant feedback and/or had a low score (mean < 4). We kept six items, adapted five items, and eliminated two items. Additionally, the touchpoint “Order and receive food” became optional for hotels offering this service. Further details of the refinements carried out are presented in

Table A5. Refinement made for CHECKHI items in second iteration—“Step 8: Refinement stage”.

Touchpoint	Item	Experts’ Comments	Action
TP1: Search for information	SE1: Website has an aesthetic and minimalist design	- Item very subjective, some customers only look for images while others want as much information as possible - This item is very subjective - It would be good to consider but it is not important	Change
	SE3: Website has the option to choose a currency	- Not very relevant if the hotel is in the same country of the customer - There are more important things than this item	Keep
TP4: Room stay	RO1: Hotel has elevators near the accessible rooms	- It depends on the type of hotel; however, all accessible rooms are usually on the first floor - It would be ideal but depends on the type of hotel	Change
	RO2: Hotel rooms are spacious	- This item depends on the price paid by customers - As long as you have free space to move is enough - Customers are more concerned about cleanliness than space because it is a price factor	Eliminate
	RO9: Hotel can provide additional amenities/toiletries	- Not relevant in my opinion - If not on the website explicitly this is not offered - If the hotel offer this is fine, but this is no really relevant, you can’t offer everything	Keep
	RO12: Hotel rooms have temperature control	- We have stoves, besides this factor depends on the location of the hotel - We have stoves in every room, we increase the blankets, but we have no temperature control.	Change
TP5: Order and receive food	OF1: Room service is available 24 × 7	- Due to logistical issues this is very complicated - Depends a lot on the hotel category - Depends too much on the stars of the hotel - This is excessive	Keep
	OF2: Hotel offers a variety of food and beverages	- The variety often depends on the type of hotel - Depends too much on the stars of the hotel	Keep
	OF3: Hotel offers vegetarian and vegan dishes	- It doesn’t matter if it’s not considered since customers care about it - It would be ideal but in general people carry their own food - Usually asked when check-in at the hotels that offer breakfast - This is not very common	Eliminate
	OF5: Minimal background noise when ordering	- It’s a factor that can’t always be controlled, especially if you call the restaurant	Keep
	OF15: Fresh and tasty food is prepared as requested	- “Tasty” is very subjective	Change
	OF18: Hotel offers dishes for people with special needs/diets (e.g., celiac or diabetic)	- That kind of customer should bring their own special food - It should be merged with the item that talks about vegans and vegetarians	Change
TP7: Report experience	RX1: Hotel contacts the customer to learn about their experience (e.g., survey via email or phone)	- It is not so necessary although it would be good It is better that customers use external website like Booking	Keep

. The third and final version of CHECKHI consists of 102 items grouped into seven touchpoints (see Section 6).

5. Validation of CHECKHI

5.1. First Iteration: Delphi Method

The Delphi method aims to acquire the most reliable consensus of a group of experts on some topic [54]. It has been widely used in multiple studies as a tool for the creation of checklists [53,55,56,57,58,59]. The Delphi method allows for a review and validation of an instrument from a quantitative and qualitative approach, as participants assign a score and provide multiple comments on each item.

We decided to perform three stages of the Delphi method to validate the items (content validity) proposed in the first version of CHECKHI, by considering the consensus of a panel of six experts. In different studies, this number of experts has made it possible to achieve consensus in such a group, as it depends more on experience than on the number of participants [60,61,62]. All experts in the study have extensive experience in the following areas: tourism, HCI, and the CX; working in academics, industries, and consulting services from 2 to 20 years. We took the application of Delphi made by Kamioka et. al. [42] as a reference and modified it, for validating CHECKHI through three stages.

5.1.1. Delphi Method: Stage 1

In this stage, a survey was sent to each expert. Experts assigned a score individually to each item (106) within the touchpoints (7) using a nine-level Likert scale (1—disagree to 9—agree) regarding the question “Do you think this item would have to be included in the checklist to evaluate the touchpoint [touchpoint name]?”. We calculated the mean and standard deviation, where any item with a mean <7 was removed immediately.

Table 5. Summary of actions taken in stage 1 of Delphi method.

Touchpoint	Initial Items	To Keep	To Modify	To Eliminate	To Create	Final Items
TP1: Search for information	17	8	7	2	3	18
TP2: Book a room	15	6	5	4	0	11
TP3: Check-in	18	4	10	4	3	17
TP4: Room stay	16	8	6	2	4	18
TP5: Order and receive food	18	10	7	1	1	18
TP6: Checkout	17	5	10	2	1	16
TP7: Report experience	5	2	3	0	1	6
Total	106	43	48	15	13	104

shows the actions taken regarding the score and comments of the experts.

As can be seen in Table 5, multiple actions were performed to refine the first version of CHECKHI before proceeding to stage 2 of the Delphi method. We decided to keep 43 items, as they were considered necessary by the participants; modify 48 items, because they were unclear or subjective; eliminate 15 items, since they did not meet the established criteria (mean > 7.00) or they were very difficult to evaluate with a checklist; and create 13 items to better evaluate the CX in hotels at the proposed touchpoints.

Regarding the comment obtained by experts, we defined four actions for refining CHECKHI: (1) “change”, where the changes were applied directly based on the experts’ comments; (2) “separate”, where items that were so general or difficult to evaluate were divided; (3) “keep”, where items were not modified because the suggestions were not supported by the other experts or were out of focus; (4) “eliminate”, when items were difficult to evaluate using a checklist, or another item already evaluated the same feature.

Table 6. Example of some expert comments and the actions for refining CHECKHI in stage 1 of Delphi method.

Touchpoint	Item	Experts’ Comments	Action
TP1: Search for information	SS6: Website presents the hotel rating.	Do you mean the hotel category or the customer rating?	Separate
TP2: Book a room	BB9: Website does not ask for unnecessary information.	The item is subjective, define unnecessary!	Eliminate
TP3: Check-in	IN18: Check-in staff offers luggage assistance.	Include parking assitance to the item.	Keep
TP4: Room stay	RR13: Hotel rooms are completely secure.	How can I evaluate this one?	Separate
TP5: Order and receive food	OO8: Room service staff is totally focused on the customer.	What does it mean “totally focused”?	Change
TP6: Checkout	OU10: Checkout adapts to the needs of the customer.	What kind of needs?	Eliminate
TP7: Report experience	RP1: Hotel sends surveys to learn about the customer experience.	There are other instruments that could be used to evaluate the CX.	Change

presents some comments from the experts regarding the items in the first version of CHECKHI.

Regarding the items suggested by the experts (see

Table 7. New items suggested by experts in stage 1 of Delphi method.

Touchpoint	Items Suggested by Experts	Action
TP1: Search for information	Website displays the schedule of activities for customers.	Take into account
TP3: Check-in	Check-in staff communicates information effectively (i.e., clear and concise).	Take into account
	Check-in staff is able to resolve issues quickly.	Take into account
	Hotel has a luggage storage service for early check-in.	Take into account
TP4: Room stay	Hotel rooms have a telephone to communicate with reception or request services.	Take into account
	Hotel must have an infirmary.	Reject (out of scope)
TP5: Order and receive food	Hotel offers dishes for people with diseases (e.g., celiac or diabetic).	Take into account
TP6: Checkout	Hotel has a luggage storage service for late checkout.	Take into account
TP7: Report experience	Hotel indicates that the information provided by customers will be kept confidential.	Take into account

), we defined two actions: (1) “take into account”, an expert’s suggestion was considered; (2) “reject”, a suggestion was not taken into consideration, since it was out of scope. Regarding the nine items suggested by the experts, eight items were adapted and subsequently included in the corresponding touchpoint in CHECKHI, and one item was rejected.

5.1.2. Delphi Method: Stage 2

As in stage 1, a survey was sent to each of the participants. Unlike in the previous stage, experts had to reassign a score to the refined items of CHECKHI, knowing the perception of each item by the group of experts as a whole (i.e., mean and SD) regarding each item (104) using the same question and Likert scale (1—disagree to 9—agree).

Table 8. Summary of actions taken in stage 2 of Delphi method.

Touchpoint	Initial Items	To Keep	To Modify	To Eliminate	To Create	Final Items
TP1: Search for information	18	17	1	0	0	18
TP2: Book a room	11	10	1	0	0	11
TP3: Check-in	17	16	1	0	0	17
TP4: Room stay	18	18	0	0	0	18
TP5: Order and receive food	18	17	1	0	0	18
TP6: Checkout	16	16	0	0	0	16
TP7: Report experience	6	6	0	0	0	6
Total	104	100	4	0	0	104

summarizes the actions taken in response to the experts’ comments in stage 2.

As can be observed in Table 8, a few actions were taken before proceeding to stage 3. These actions included: keep 100 items, which were considered by the experts as necessary for CHECKHI; modify 4 items, because they had minor drafting errors. At this stage, the experts did not propose any new items to be included, and none were deleted because they all met the criteria (mean > 7.00).

Regarding the comments obtained from the experts, only five suggestions for improvements were provided, and we carried out the same actions as in stage 1. We accepted and included four suggestions because they improved the wording of the item, while one was rejected because the scores given by the experts indicate that the item was necessary and should not be eliminated or modified.

Table 9. Expert comments and the actions for refining CHECKHI in stage 2 of Delphi method.

Touchpoint	Item	Experts’ Comments	Action
TP1: Search for information	SS18: Website displays the schedule of activities for customers.	I don’t understand what you want to evaluate.	Change
TP2: Book a room	BB8: Website displays the total room rate in multiple currencies.	Change the item to something similar to “The system offers the possibility to display the total amount in multiple currencies”.	Change
TP3: Check-in	IN9: Check-in staff provides information of interest related to the hotel.	What kind of information exactly?	Change
TP4: Room stay	RR14: Hotel rooms have safety deposit boxes.	Is this necessary?	Keep
TP5: Order and receive food	OO18: Hotel offers dishes for people with diseases (e.g., celiac or diabetic).	I would change “diseases” to “special needs/diet” I wouldn’t treat them like “sick”!	Change

presents the comments of the experts regarding the refined version of CHECKHI.

5.1.3. Delphi Method: Stage 3

In this stage, we decided to modify the application of the Delphi method proposed by Kamioka et al. [53], with the aim of demonstrating with other statistical analyses the content validity of the items presented in CHECKHI. In addition to the previous descriptive statistics, the reliability or interrater agreement (IRA) and the content validity index (CVI) [63] were calculated.

The IRA represents the degree of agreement among the participants regarding the representativeness and clarity. The representativeness indicates that the items evaluated correctly represent the domain, in this case, the corresponding touchpoint. The clarity refers to whether the items are correctly worded and can be easily understood [63]. The IRA is calculated by counting the number of items with 100% agreement (i.e., rated either 3 or 4 by all the experts) and dividing it by the total number of items (see Formula (1)).

$${\mathrm{IRA}}_{\mathrm{touchpoint}}=\frac{\mathrm{number}\mathrm{of}\mathrm{items}\mathrm{with}100\%\mathrm{expert}’\mathrm{s}\mathrm{agreement}}{\mathrm{total}\mathrm{number}\mathrm{of}\mathrm{items}}$$

(1)

where 100% experts’ agreement means items that all experts assigned a score of 3 or 4.

The CVI indicates the degree of agreement among the experts concerning the items considered as content valid for the touchpoints, by using the representativeness. This is obtained by calculating the CVI for each element at a touchpoint by counting the experts’ agreement (i.e., rated either 3 or 4 by all the experts) and dividing that score by the total number of experts. Then, the CVI of the touchpoint is measured by calculating the average CVI across the items (see Formula (2)).

$$\begin{array}{l}{\mathrm{CVI}}_{\mathrm{touchpoint}}=\frac{{\sum }_{\mathrm{i}=1}^{\mathrm{T}}{\mathrm{CVI}}_{{\mathrm{item}}_{\mathrm{i}}}}{\mathrm{total}\mathrm{number}\mathrm{of}\mathrm{items}}\\ {\mathrm{CVI}}_{\mathrm{item}}=\frac{\mathrm{expert}’\mathrm{s}\mathrm{agreement}}{\mathrm{total}\mathrm{number}\mathrm{of}\mathrm{experts}}\end{array}$$

(2)

where:

-

T: number of items in the touchpoint,

-

experts’ agreement: the number of experts giving an item a score of 3 or 4.

In order to verify the content validity calculating the IRA and CVI, a survey was sent to six experts, which received responses from only five of them. In this stage, the experts were asked to assign a score to the 104 items resulting from stage 2 using a four-level Likert scale (1—worst to 4—best) regarding two criteria: (1) representativeness and (2) clarity. Therefore, upon receiving the answers from the experts, the mean, SD, IRA, and CVI were calculated for both the items and touchpoints.

Table 10. Summary of statistical results for each touchpoint at stage 3 of Delphi method.

Touchpoint	Representativeness (R)		Clarity (C)		IRA		CVI
	Mean	SD	Mean	SD	R	C
TP1: Search for information	3.84	0.28	3.91	0.15	0.89	1	0.98
TP2: Book a room	3.93	0.16	3.95	0.09	1	1	1
TP3: Check-in	3.89	0.22	3.91	0.19	1	0.94	1
TP4: Room stay	3.94	0.12	3.91	0.16	1	1	1
TP5: Order and receive food	3.93	0.13	3.90	0.22	1	0.94	1
TP6: Checkout	3.98	0.06	3.91	0.20	1	1	1
TP7: Report experience	3.90	0.22	4.00	0.00	1	1	1

provides a summary of the statistical results for each touchpoint used by CHECKHI.

As seen in Table 10, excellent results were obtained for all indicators at each touchpoint; a brief analysis of this is presented below.

• Representativeness: The experts consider that it makes sense to evaluate the items within the proposed touchpoints. This was supported, as the mean was in a range of values between 3.84 and 3.98, while the SD varied between 0.06 and 0.28. Although excellent results were obtained for both indicators, it should be mentioned that the touchpoint with the most problems was “TP1: Search for information”, which obtained the worst mean and standard deviation.
• Clarity: Regarding this indicator, the experts indicate that the items present at the touchpoints have no wording problems nor are difficult to understand. This was supported by analyzing the mean and standard deviation, which are in a range between 3.90 to 4.00 and 0.00 to 0.22, respectively. In this case, the touchpoint “TP5: Order and receive food” had the worst results; nonetheless, these remained excellent.
• Interrater agreement (IRA): The results indicated that the experts are almost completely in agreement on the scores assigned for both representativeness and clarity. Thus, it can be observed that excellent results were obtained because the lowest value for representativeness was 0.89 for the touchpoint “TP1: search for information”, while clarity was 0.94 for the touchpoints “TP3: Check-in” and “TP5: Order and receive food”.
• Content validity index (CVI): The results indicated that there was an almost total degree of agreement among the experts considering the items as content valid for the proposed touchpoints. Finally, with values above 0.80, we can state that excellent results were obtained [63], [64] since the lowest value corresponds to 0.98 for the touchpoint “TP1: search for information”, which is consistent with the results obtained for the representativeness.

Due to the positive results obtained in the validations of the third stage of the Delphi method, it was not necessary to further refine the CHECKHI items. Thus, at the end of the first iteration, we obtained the second version of CHECKHI, which was composed of 104 items grouped into seven touchpoints.

5.2. Second Iteration: Validation through Interviews

In the second iteration, we conducted four structured interviews, to collect and analyze the perceptions of practitioners to validate and refine the second version of CHECKHI. The main objective of this iteration was to include the perspective of experts who had worked in hotels. The interviewees had worked in different types of hotels interacting with customers (i.e., hotel reception, room cleaning, etc.). from a range to 2 to 4 years.

In the interview, practitioners assigned a score for each item (104) of the second version of CHECKHI using a five-level Likert scale (1—worst to 5—best) regarding the question “How useful is it to include this item to evaluate the CX in hotels?”. As practitioners assigned a score to the items, they were asked to comment on the items evaluated with low scores and those most relevant to them. These interviews allowed us to identify that some items were very specific and rare, so they were eliminated, while others were confusing or used very technical language, so their wording was improved. A detailed analysis of the comments given by the experts, classified by each touchpoint, are presented below.

• TP1—Search for information:

This touchpoint was one of the most commented on by the practitioners. This is mainly due to factors such as subjectivity, value judgments, and benefits to the hotel. The latter factor arose because certain practitioners gave their point of view as hotel workers and did not put themselves in the “customer’s shoes” in order to satisfy the customer’s needs. Due to this, multiple comments were obtained that classify the items as: “it’s an extra, but it’s not important” or “there are more important things”.

The most commented item in this touchpoint was “SE1: Website has an aesthetic and minimalist design”, where three of the four practitioners indicated that it would be difficult or irrelevant to evaluate this item. The practitioners indicated that the aesthetics and minimalism of a website is subjective and not very important, which was observed in comments such as: “this item is very subjective, some customers only look for images while others want as much information as possible” and “it would be good to consider but it is not important”. These comments allowed us to understand that it would be necessary to modify the wording, change the focus, or eliminate SE1 from CHECKHI.

In addition, the other most discussed items were “SE3: Website has the option to choose a currency” and “SE10: Website provides sightseeing/surroundings information (e.g., distance to city center or attractions)”, where half of the participants mention that it is not useful to evaluate these items. The main reason for these comments was because practitioners thought like a worker instead of a customer, which was shown in comments such as: “not very relevant if the hotel is in the same country of the customer”, “it’s an extra, but it’s not important”, and “it can be negative to have a lot of information… customers do not measure distances well or have a lot of expectation”. Due to this, we decided not to consider making changes or removing them, unless the score given by each participant was very critical both in this validation and in the previously performed Delphi method.

There were other items with comments that reflect this thought (i.e., “SE5: Website indicates when the information was last updated”, “SE6: Website presents the hotel’s star rating”, “SE7: Website presents a rating made by customers”, “SE12: Website shows information about the proximity to public transport services”, and “SE16: Website provides both customers’ positive and negative reviews). However, as they were mentioned by only one of the four interviewees, these were not classified as relevant for further analysis. Finally, in this touchpoint, 10 items were considered as essential and relevant by experts to evaluate the CX in this interaction.

• TP2—Book a room:

This touchpoint was the best perceived by the practitioners. There were no particular comments for each item present in this interaction, since they were asked to comment only on the most relevant and lowest-scoring items. Nevertheless, after assigning a score to each of the items, the interviewees gave comments such as: “all these items are absolutely necessary”, “in my opinion it fulfills everything”, or “I cannot say that any of these items is not useful to evaluate as they are all very important”.

The only comments given for the items present at this touchpoint were from a practitioner regarding “BO4: Website allows different payment options” and “BO8: Website offers the possibility to display the total amount in multiple currencies”. In the first one, the interviewee indicated that the item was “not relevant in my opinion”, while in the second one they mentioned that the item was “very extra in my opinion, does not affect the experience”. Analyzing the comments, we observed that the participant did not think about relevant customer’s needs, as these items were evaluated both in this interview and in the Delphi method as very important and necessary.

• TP3—Check-in:

This touchpoint was also widely discussed by the practitioners. This is because all practitioners have worked at the front desk of their hotels, so they have experience dealing with customers in this interaction. Thus, feedback was obtained from the experts, but from the point of view of them as workers, so the results were contrasted with the assigned score and other experiments to decide whether to make modifications or not.

The most commented items were: “CHI6: Check-in staff is not distracted by other tasks when serving a customer” and “CHI14: Check-in staff offers luggage assistance”, where three of four practitioners gave their opinion indicating that this is very difficult to handle. This statement was based on the experience that the participants have at this touchpoint and the environmental and/or working conditions of the hotel. Thus, for the first item, it was stated that often due to the flow of customer and the number of workers, it is not possible to focus on a single customer, with comments such as: “it is difficult to be 100% aware of customers” and “this can often not be controlled if you work alone at the front desk”. While, for the second item, it was discussed for different reasons; on the one hand, this service is not offered all types of hotels, and on the other hand, they indicate that some of the staff cannot help with luggage due to their age.

Another item commented on by two practitioners corresponded to “CH12: Check-in staff performs the registration process quickly”, where they indicated that this item depends mainly on a factor related to information: both the information available of the customers and the information that should be provided to them, so that they have no doubts during their stay. Furthermore, there were other items with comments (i.e., “CHI5: Check-in staff is dressed within the established dress code”, “CHI7: Check-in staff speaks at least English and the native language of the country”, and “CHI17: Hotel has a luggage storage service for early check-in”). However, they were only commented on by one participant, and the information given is not relevant enough to consider making modifications or deleting them. Finally, we observed that 11 of 17 items were considered useful to evaluate the CX in this touchpoint.

• TP4—Room stay:

Similar to TP1 and TP3, this touchpoint received multiple comments by the practitioners for its items. These comments mainly focused on factors that depend on the type of hotel and customer, so they may not apply in every case or situation. For this reason, it will be necessary to analyze and determine if the wording and approach of these need to be modified, or to eliminate them definitively given the results obtained in this validation and those previously realized.

In this touchpoint, the most commented items were “RO2: Hotel rooms are spacious” and “RO9: Hotel can provide additional amenities/toiletries”, where three of the four practitioners gave their opinion. Regarding the first one, it was indicated that this item depends mainly on the price that customers pay, i.e., customers pay to make their room more spacious; this is reflected in comments such as: “This item depends on the price paid by customers” or “…it is a price factor”. Regarding the second one, it was mentioned that this should be previously indicated as a service on the website or check-in in order to not cause a bad impression to customers, by informing them what is offered because, as said by a participant, “hotels can’t offer everything”.

In addition, the other most discussed items were “RO1: Hotel has elevators near the accessible rooms”, “RO10: Hotel rooms’ technological amenities are modern and easy to use”, and “RO12: Hotel rooms have temperature control”, where half of the practitioners mention that these items should be modified. Regarding RO1, it was mentioned that this factor depends mainly on the type of hotel, where these rooms are usually located on the first floor of the hotel to facilitate access. Regarding RO10, it was noted that customers are not always looking for modern products, as one practitioner mentioned “modernity is important, but it depends a lot on the type of hotel and customer. In large cities customers seek modernity but in smaller cities they seek warmth and tranquility”. Concerning RO12, both practitioners commented that the hotels where they worked had no temperature control, but they had stoves and extra blankets in each room in cold seasons.

As for previous touchpoints, there were certain items (i.e., “RO6: Hotel rooms are free from any annoying noises” and “RO14: Hotel rooms have safety deposit boxes”) that obtained observations from only one practitioner; nonetheless, these were personal considerations that were not critical enough to consider any modifications to these items. Finally, this touchpoint contained 11 of 18 items that were classified as essential for evaluating the CX in this interaction.

• TP5—Order and receive food:

This touchpoint was the most commented on by the practitioners. The practitioners commented that not all hotels had room service with the possibility of ordering food freely, and this service depends mainly on the type and category of the hotel. This is so uncommon that none of the practitioners provide this service in the hotels where they have worked. For this reason, this touchpoint will be optional for evaluation by hotels offering this service/interaction.

The most discussed items in this touchpoint were “OF1: Room service is available 24 × 7” and “OF3: Hotel offers vegetarian and vegan dishes”, where all the experts mentioned the difficulty of considering these aspects. Regarding the first, it was mentioned that it is extremely complicated to perform because it only applies to luxurious hotels, so the practitioners commented: “due to logistical issues this is very complicated”, “this is excessive” or “depends a lot on the hotel category”. Regarding the second, it was commented that it is not very common to offer vegan and vegetarian dishes, and, for that reason, customers concern themselves with bringing their own food; nevertheless, when analyzing these comments, it can be noticed that the experts are not thinking about satisfying the needs of this type of customer.

Furthermore, there were other items from where feedback was obtained from half of the practitioners: “OF2: Hotel offers a variety of food and beverages” and “OF18: Hotel offers dishes for people with special needs/diets (e.g., celiac or diabetic)”. For the first item, the practitioners indicated that usually the variety within the menu depends on the type or classification of the hotel, so in some hotels with many customers this may be possible but not in others. While, for the second item, the comment of one practitioner was highlighted, which suggested merging this item with “OF3: Hotel offers vegetarian and vegan dishes”, since both mention special needs/diets.

There were items where only an interviewee gave their opinion (i.e., “OF5: Minimal background noise when ordering”, “OR6: Room service staff speaks at least English and the native language of the country”, “OF7: Room service staff has knowledge of the menu items and preparation”, and “OF15: Fresh and tasty food is prepared as requested”), which as for previous touchpoints, would not be taken into consideration for in-depth analysis. However, in this case, an exception was made for OF15, because that comment mentions that: “tasty is very subjective”, which is why the item will be analyzed to be modified or deleted if necessary. Finally, we identified 10 of 18 items that do not present any problem and were considered useful when evaluating the CX for this touchpoint.

• TP6—Checkout:

This touchpoint was one of the best perceived by the practitioners because they did not mention many problems. This is because 11 of 16 items were declared as useful and important when evaluating the CX at checkout. In addition, only three items (i.e., “CHO4: Checkout staff speaks at least English and the native language of the country”, “CHO5: Checkout staff is not distracted by other tasks when serving a customer”, and “CHO11: Checkout staff provides a customer experience form”) were commented on by a practitioner; nevertheless, the comments were not severe enough to modify or remove any items.

For this touchpoint, the most discussed items were “CHO6: Checkout staff discreetly checks and processes payments” and “CHO10: Checkout staff performs the departure process quickly”, which received feedback from half of the practitioners. Regarding CHO6, it is not possible to interpret much of their opinions because the practitioners think differently: one mentioned that “it is important to be discreet because sometimes there are problems with payments”, while the other said that “it should be the opposite; it is better to be very transparent than discreet”. Concerning CHO10, both practitioners indicated that this item depends a lot on the type of customer, as there are some who just want to leave as quickly as possible; nevertheless, there are also others who prefer to get as many details as possible when they must pay additional charges. Thus, for these two items, the assigned scores will be checked and compared with previous results to decide whether to keep them in CHECKHI or not.

• TP7—Report experience:

Due to the limited number of items proposed at this interaction, it was difficult to evaluate whether this touchpoint was well-perceived by practitioners. However, two of the four practitioners did not indicate any problems with the set of items, classifying these as being completely essential to evaluate the CX. Furthermore, there were two items that would not require modifications (i.e., “RX2: Hotel surveys indicate contact information for customers” and “RX3: The hotel answers customer feedback on their platforms”) because these were mentioned as necessary by practitioners.

Thus, the most commented on item was “RX4: The hotel offers compensation to dissatisfied customers”, by two of the four practitioners. For this item, both practitioners agreed that it could be useful; however, the problem must first be verified as real. This is mentioned in the following comments: “the compensation should only be made when the hotel is responsible or has made a mistake” and “hotels should always verify that the problem was real”.

Finally, there were three items that received a comment from a practitioner (i.e., “RX1: Hotel contacts the customer to learn about their experience (e.g., survey via email or phone)”, “RX5: Hotel considers customer feedback to improve”, and “RX6: Hotel indicates that the information provided by customers will be confidential”); however, they were also analyzed. Regarding RX1, a practitioner indicated that customers could be asked not only to answer their surveys but also to comment about their experience on external websites. Regarding RX5, another practitioner indicated that not all comments received by hotels would have to be considered, only the most repeated or common. Concerning RX6, it was indicated that this information would not have to be confidential; nevertheless, it was not possible to obtain more details about this comment because the interview with this practitioner ended prematurely. Thus, the feedback given for RX1 and RX5 was analyzed, to decide whether or not to include it based on the results obtained from the other analyses and validations.

Furthermore, practitioners were asked to rate each item using a Likert scale of five levels (1—worst to 5—best) regarding the question “How useful do you think it is to evaluate this item to improve the CX in hotels?”.

Table 11. Descriptive statistics of practitioners’ responses.

Touchpoint	Items	Items with: Mean = 5	Items with: 5 > Mean ≥ 4	Items with: Mean < 4	Touchpoint Mean	Touchpoint SD
TP1: Search for information	18	3 (16.7%)	13 (72.2%)	2 (11.1%)	4.47	0.62
TP2: Book a room	11	7 (63.6%)	4 (36.4%)	0 (0%)	4.80	0.36
TP3: Check-in	17	6 (35.3%)	11 (64.7%)	0 (0%)	4.65	0.52
TP4: Room stay	18	7 (38.9%)	7 (38.9%)	4 (22.2%)	4.44	0.59
TP5: Order and receive food	18	3 (16.7%)	11 (61.1%)	4 (22.2%)	4.31	0.71
TP6: Checkout	16	2 (12.5 %)	14 (87.5%)	0 (0%)	4.41	0.82
TP7: Report experience	6	0 (0%)	5 (83.3%)	1 (16.7%)	4.29	0.84
Total	104	28 (26.9%)	65 (62.5%)	11 (10.6%)

shows the descriptive statistics calculated for each touchpoint, where the items that had perfect (mean = 5), good (5 > mean ≥ 4), and bad (mean < 4) scores are indicated, next to the mean and SD of the touchpoint.

Since an in-depth analysis and interpretation of the comments given by the practitioners has already been carried out, brief observations on the quantitative results are mentioned below.

Regarding the mean of the touchpoints, we observed that excellent results were obtained in this experiment, as all the means had a score greater than 4.0. These positive results are mainly due to the validations carried out by the Delphi method, which allowed for multiple refinements to CHECKHI. The best evaluated touchpoints correspond to “TP2: Book a room”, with a mean of 4.80, followed by “TP3: Check-in”, with a value of 4.65. On the other hand, the lowest rated touchpoints were “TP5: Order and receive food”, with a mean of 4.31, while “TP7: Report experience”, with a mean of 4.29, received the most critical feedback of all.

Regarding the SD, we obtained acceptable results, since four of the seven items had an SD of less than 0.7. As mentioned in the qualitative analysis, this is mainly because some practitioners evaluated multiple items thinking as hotel workers and did not put themselves in the “customer’s shoes”. Thus, the touchpoints where the practitioners presented less difference of opinion were again “TP2: Book a room”, with an SD of 0.36, followed by “TP3: Check-in”, with a score of 0.52. The touchpoints where the greatest difference of opinion was presented were “TP6: Checkout”, with a value of 0.82, and again “TP7: Report experience”, with an SD of 0.84.

Finally, when observing the results regarding the mean of the items, we state that CHECKHI obtained good valuations about its usefulness from the practitioners, where 26.9% of the items received perfect scores (mean = 5), 62.5% of these received good values (5 > mean ≥ 4), and only 10.6% received bad scores (mean < 4). Dichotomizing the results and analyzing them with respect to good scores (mean ≥ 4) and bad scores (mean < 4), we observed that 89.4% of the items received good scores, but 10.6% received bad scores. In addition, a general analysis was performed regarding the perception of the practitioners concerning the items at each touchpoint. Thus, among the best rated touchpoints were “TP2: Book a room”, “TP3: Check-in”, and “TP6: Checkout”, where all their items were ranked with good scores, followed by “TP1: Search for information”, with 88.9% of its items in this category ranked with good scores. Lastly, the touchpoints with the worst evaluated items were “TP7: Report experience”, where 16.7% of its items were poorly evaluated, and “TP4: Room stay” and “TP5: Order and receive food”, where 22.2% of their items received a bad score.

6. CHECKHI: A Checklist for Evaluating the Customer Experience in the Hotel Industry

Based on the results obtained from both the iterations and validations, CHECKHI was refined and improved. CHECKHI contains 102 items grouped into seven touchpoints: (1) search for information (18 items); (2) book a room (11 items); (3) check-in (17 items); (4) room stay (17 items); (5) order and receive food (17 items); (6) checkout (16 items); (7) report experience (6 items). We formally specified CHECKHI by defining 7 elements: (1) touchpoints; (2) channels; (3) website/hotel features; (4) CX dimensions; (5) SQ/e-SQ dimensions; (6) items; (7) hotel response (yes, partially, no, and N/A). The third and final version of CHECKHI is presented in

Table 12. Third and final version of CHECKHI.

CHECKHI: A Checklist for Evaluating the Customer Experience in the Hotel Industry		Yes	Partially	No	N/A
TP1: Search for information Channel: Hotel Website—External Website Website Hotel Features: Facilities Information, Contact Information, Website Management, Surrounding Area Information CX Dimensions: Cognitive, Pragmatic, Lifestyle E-SQ Dimensions: Functionality, Reliable Information, Locality Information, Customer Reviews
SI1	Website has an intuitive design (e.g., information easy to find)
SI2	Website has a multilingual version
SI3	Website has the option to choose a currency
SI4	Website provides visual content related to the hotel
SI5	Website indicates when the information was last updated
SI6	Website presents the hotel’s star rating
SI7	Website presents a rating made by customers
SI8	Website indicates the location of the hotel
SI9	Website provides a map of the area around the hotel
SI10	Website provides sightseeing/surroundings information (e.g., distance to city center or attractions)
SI11	Website provides the contact information for the hotel
SI12	Website shows information about the proximity to public transport services
SI13	Website indicates free and paid services
SI14	Website indicates room amenities
SI15	Website indicates bathroom amenities/toiletries
SI16	Website provides both customers’ positive and negative reviews
SI17	Website displays links to social networks
SI18	Website displays the schedule of activities for customers (e.g., breakfast, housekeeping, and/or fitness center)
TP2: Book a room Channel: Hotel Website—External Website Website Hotel Features: Facilities Information, Reservation Information, Website Management CX Dimensions: Cognitive, Pragmatic E-SQ Dimensions: Functionality, Reliable Information
BR1	Website indicates booking availability in real time
BR2	Website indicates the services and products included in the booking
BR3	Website indicates the hotel policies (i.e., reservation, payment, and cancellation)
BR4	Website allows different payment options
BR5	Website has secure payment systems
BR6	Website indicates information related to check-in and checkout (i.e., times, early check-in, and late checkout)
BR7	Website provides a form to indicate additional customer needs (usually called a “special request form”)
BR8	Website offers the possibility to display the total amount in multiple currencies
BR9	Website provides detailed room rate information
BR10	Website indicates the details of the booking before completing the process
BR11	Website gives a prompt booking confirmation
TP3: Check-in Channel: Hotel Front Desk Hotel Features: Cleanliness, Service speed, Staff Professionalism, Accessibility CX Dimensions: Sensorial, Relational SQ Dimensions: Tangible, Reliability, Responsiveness, Assurance
CI1	Hotel entrances are completely accessible
CI2	Front desk is clean and tidy
CI3	Check-in staff is courteous and friendly
CI4	Check-in staff has a good presentation (i.e., neat, clean, and tidy)
CI5	Check-in staff is dressed within the established dress code
CI6	Check-in staff is not distracted by other tasks when serving a customer
CI7	Check-in staff speaks at least English and the native language of the country
CI8	Check-in staff has knowledge about the surrounding areas and encourages customers to visit them
CI9	Check-in staff provides information of interest related to the hotel (e.g., breakfast hours)
CI10	Check-in staff is trained to answer questions about the hotel
CI11	Check-in staff does not publicly display customers’ personal information
CI12	Check-in staff performs the registration process quickly
CI13	Check-in staff indicates the location of the room that has been booked
CI14	Check-in staff offers luggage assistance
CI15	Check-in staff communicates information effectively (i.e., clear and concise)
CI16	Check-in staff is able to resolve issues quickly
CI17	Hotel has a luggage storage service for early check-in
TP4: Room stay Channel: Hotel Room Hotel Features: Amenities, Cleanliness, Comfort, Security, Accessibility CX Dimensions: Sensorial, Pragmatic SQ Dimensions: Tangible, Assurance, Empathy
RS1	Hotel has accessible rooms
RS2	Accessible hotel rooms are easily reachable (e.g., these are on the first floor or near the elevators)
RS3	Hotel rooms are clean and tidy
RS4	Ensuite bathrooms are clean and tidy
RS5	Hotel rooms have sufficient natural light and ventilation
RS6	Hotel rooms are free from any annoying noises
RS7	Hotel rooms have the promised amenities
RS8	Ensuite bathrooms have the promised amenities/toiletries
RS9	Hotel can provide additional amenities/toiletries
RS10	Hotel rooms’ technological amenities are modern and easy to use
RS11	Hotel rooms are free of bad smells
RS12	Hotel rooms are equipped to control the temperature (e.g., air conditioning, stoves, or blankets)
RS13	Hotel rooms’ windows can be locked
RS14	Hotel rooms have safety deposit boxes
RS15	Hotel rooms’ entrances are restricted by cards, passwords, or keys
RS16	Hotel indicates the emergency exits and safe areas
RS17	Hotel rooms have a telephone to communicate with reception or request services
TP5: Order and receive food (optional, only to be evaluated if the hotel offers this service) Channel: Room Service Hotel Features: Food Service, Service Speed, Staff Professionalism CX Dimensions: Sensorial, Cognitive, Relational SQ Dimensions: Reliability, Responsiveness, Assurance, Empathy
OR1	Room service is available 24 × 7
OR2	Hotel offers a variety of food and beverages
OR3	Hotel offers dishes for people with special needs/diets (e.g., vegan, vegetarian, celiac, and/or diabetic dishes)
OR4	Room service staff is courteous and friendly
OR5	Minimal background noise when ordering
OR6	Room service staff speaks at least English and the native language of the country
OR7	Room service staff has knowledge of the menu items and preparation
OR8	Room service staff is not distracted by other tasks when serving a customer
OR9	Room service staff gives menu recommendations (e.g., recommend items, daily specials, or promotions)
OR10	Room service staff verifies the order made by the customer
OR11	Room service staff indicates approximate delivery time
OR12	Room service staff delivers the order efficiently
OR13	Room service staff has a good presentation (i.e., neat, clean, and tidy)
OR14	Room service staff is dressed within the established dress code
OR15	Dishes are prepared as requested
OR16	Dishes that are delivered have the right temperature
OR17	Room service order is removed efficiently
TP6: Checkout Channel: Hotel Front Desk Hotel Features: Cleanliness, Service Speed, Staff Professionalism, Accessibility CX Dimensions: Sensorial, Relational SQ Dimensions: Tangible, Reliability, Responsiveness, Assurance
CO1	Checkout staff has a good presentation (i.e., neat, clean, and tidy)
CO2	Checkout staff is dressed within the established dress code
CO3	Checkout staff is courteous and friendly
CO4	Checkout staff speaks at least English and the native language of the country
CO5	Checkout staff is not distracted by other tasks when serving a customer
CO6	Checkout staff discreetly checks and processes payments
CO7	Checkout staff explains additional charges
CO8	Checkout staff explains total charges
CO9	Checkout staff provides a physical and/or digital invoice
CO10	Checkout staff performs the departure process quickly
CO11	Checkout staff provides a customer experience form
CO12	Checkout staff is able to resolve issues quickly
CO13	Checkout staff has knowledge about the surrounding areas and encourages customers to visit them
CO14	Checkout staff offers luggage assistance
CO15	Hotel departures are completely accessible
CO16	Hotel has a luggage storage service for late checkout
TP7: Report experience Channel: Hotel Website—External Website—Social networks—Email Hotel Website Features: Hotel Professionalism CX Dimensions: Lifestyle E-SQ/SQ Dimensions: Empathy, Customer Reviews
RE1	Hotel contacts the customer to learn about their experience (e.g., survey via email or phone)
RE2	Hotel surveys indicate contact information for customers
RE3	Hotel answers customer feedback on their platforms
RE4	Hotel offers compensation to dissatisfied customers
RE5	Hotel considers customer feedback to improve
RE6	Hotel indicates that the information provided by customers will be confidential

.

7. Contributions

As result of this research, we create CHECKHI: a novel checklist for evaluating the customer experience of tourism in the hotel industry. The validations performed by the CHECKHI items demonstrate its representativeness, clarity, and usefulness, making it possible to evaluate the CX at the identified touchpoints regardless of the hotel type. We consider that this new evaluation instrument may be useful for:

• Companies, entities, and/or organizations that offer tourism services related to hotels and accommodation industries and require an evaluation of the CX to improve the quality of their products, systems, or services;
• Academics, researchers, and/or experts in the areas of HCI, service science, customer experience, tourism, and the hotel sector, who need to create and/or adapt new instruments—such as checklists, scales, and questionnaires—for evaluating the customer experience in several domains, not only in the tourism area but also in other sectors such as retail, banks, and airlines, among others;
• Students in training and (future) professionals who work in service science and tourism and need to evaluate the CX for the products, systems, and/or services that they offer (or will offer) to their customers in the hotel sector.

It has been detected that the existence of instruments to evaluate the CX in the hotel industry is limited [19], so CHECKHI contributes by generating new approaches and more accurate evaluations. We incorporated key elements related to the CX to create CHECKHI: (1) customer traits and profiles, by considering their characteristics and emotions (user profiles and customer journey maps); (2) touchpoints and related channels; (3) CX dimensions. This is an interesting and novel starting point, since there are no proposals that consider these three key elements when developing instruments to evaluate the CX in the hotel industry. We assert that the use of CHECKHI will facilitate the timely detection of problems that generate a negative experience, thus pursuing effective solutions to create a more pleasant guest experience.

8. Limitations

As this research progressed and CHECKHI was developed, different limitations were identified. Firstly, the interviews conducted for the second validation and refinement of CHECKHI were carried out with only four practitioners. However, although the number of participants was limited, they provided valuable feedback based on their perspective, due to their experience working directly with customers in different types of hotels. Secondly, despite the fact that we performed several validations in two iterations, practitioners could not actually apply CHECKHI in hotels due to the COVID-19 outbreak. Thirdly, when conducting the latest experiment with practitioners, it was detected that the touchpoint “TP5: Order and receive food” was a very specific interaction that does not usually occur in any type of hotel. Therefore, it was decided to change this touchpoint to be optional, to be evaluated only by hotels offering this service, to avoid losing valuable results from the previous validations. Finally, since CHECKHI aims to evaluate the CX in any type of hotel at the specific touchpoints, it is suggested to perform additional validations and refinements to particularize CHECKHI to specific types of hotels. This is because it may be necessary to include new touchpoints and/or eliminate some items that do not apply to specific hotels, in order to obtain more accurate results when evaluating the CX in this industry.

9. Conclusions

We proposed CHECKHI, a novel checklist composed of 102 items grouped into seven touchpoints that make it possible to evaluate the CX in hotels. The touchpoints correspond to: (1) search for information, (2) book a room, (3) check-in, (4) room stay, (5) order and receive food, (6) checkout, and (7) report experience. The touchpoints are related to different CX dimensions and SQ/e-SQ dimensions, together with different aspects of both hotels and hotel websites regarding the type of interaction and the channel. CHECKHI items are oriented to evaluate the CX in hotels, and several of them address aspects related to sustainability, especially elements of inclusion and accessibility (such as CI1, CI14, RS1, RS2, OR3, CO14, and CO15).

CHECKHI was developed by adapting the methodology proposed by Quiñones et al. [5,6], where two iterations were performed. Regarding the first iteration, the eight stages of the methodology were conducted to propose the first and second versions of CHECKHI; these versions were validated using the Delphi method, so we verified the content validity of CHECKHI over three stages. Concerning the second iteration, the last two stages of the methodology were performed by validating and refining CHECKHI through a series of interviews with practitioners, to acquire their perceptions about the usefulness of the items at each touchpoint to evaluate the CX in hotels.

Based on the results obtained from the validations performed, we conclude that CHECKHI items were recognized as representative and clear by the experts, since the indicators used (i.e., IRA and CVI) showed excellent results after we performed multiple refinements to CHECKHI in the first iteration. In addition, the CHECKHI items were perceived as useful by the practitioners; by dichotomizing the results obtained and analyzing the good scores (mean ≥ 4) and bad scores (mean < 4), we observed that 89.4% of the items obtained good scores. Therefore, we state that CHECKHI makes it possible to evaluate the CX at the different touchpoints identified regardless of the type of hotel.

We consider that this new evaluation instrument will be useful for companies and/or organizations that offer hotel services and require an evaluation of the CX; academics who need to create new evaluation instruments and can use CHECKHI as a reference; and professionals who need to learn about CX evaluation in the hotel sector.

For future work, we aim to use and apply CHECKHI in several hotels to validate and refine the items. Based on the results obtained in these experiments, distinct versions of CHECKHI will be developed for different types of hotels, identifying and incorporating new touchpoints to achieve the creation of a complete/holistic evaluation instrument. Finally, we plan to use the information obtained from this research to propose formal recommendations to improve the CX in the hotel industry through products, systems, and services.