An Aspect-Based Review Analysis Using ChatGPT for the Exploration of Hotel Service Failures

Abstract

In this study, we employed ChatGPT, an advanced large language model, to analyze hotel reviews, focusing on aspect-based feedback to understand service failures in the hospitality industry. The shift from traditional feedback analysis methods to natural language processing (NLP) was initially hindered by the complexity and ambiguity of hotel review texts. However, the emergence of ChatGPT marks a significant breakthrough, offering enhanced accuracy and context-aware analysis. This study presents a novel approach to analyzing aspect-based hotel complaint reviews using ChatGPT. Employing a dataset from TripAdvisor, we methodically identified ten hotel attributes, establishing aspect–summarization pairs for each. Customized prompts facilitated ChatGPT’s efficient review summarization, emphasizing explicit keyword extraction for detailed analysis. A qualitative evaluation of ChatGPT’s outputs demonstrates its effectiveness in succinctly capturing crucial information, particularly through the explicitation of key terms relevant to each attribute. This study further delves into topic distributions across various hotel market segments (budget, midrange, and luxury), using explicit keyword analysis for the topic modeling of each hotel attribute. This comprehensive approach using ChatGPT for aspect-based summarization demonstrates a significant advancement in the way hotel reviews can be analyzed, offering deeper insights into customer experiences and perceptions.

1. Introduction

Service failure in the hospitality industry refers to situations where there is a discrepancy between expected and delivered services [1]. These failures, ranging from errors, deficiencies, or problems during service delivery, often result in customer dissatisfaction and can negatively impact a business’s objectives, like survival and growth [2]. The perception of service failure not only arises from direct service inadequacies but also from a mismatch between customer expectations and the actual service provided [3]. This discrepancy leads to negative reviews and opinions, which significantly influence potential customers’ decision-making processes and their overall perception of a hotel [4].

With the advent of the internet and the proliferation of consumer-generated content, the approach to assessing service quality in hotels has shifted from traditional quantitative methods, like using guest comment cards and questionnaires, to a more qualitative analysis performed through online reviews [5,6]. These reviews are increasingly influencing consumer decisions and hotel selection. Studies have shown that negative reviews tend to have a greater impact on potential customers than positive ones. Service failures, as reported in these reviews, often stem from employee behavior, core service inadequacies, or unexpected situations during service delivery [7]. The repercussions of these service failures manifest in various forms, including customer dissatisfaction, negative word of mouth, customer defection, loss of revenue, and even declining employee morale [8,9,10,11].

The traditional approach to customer complaint analysis was manual coding that categorizes reviews based on predefined frameworks [12]. While this method offers strong interpretive power, it is labor-intensive and highly dependent on the analysts’ knowledge, making the results hard to replicate and limiting the scope of the corresponding research [13]. This approach, despite its detailed analysis, struggles with scalability and objectivity, prompting the need for more efficient and comprehensive methodologies to understand and address customer feedback in the hospitality sector [14]. Given the limitations of traditional manual coding in customer complaint analysis, natural language processing (NLP) is emerging as a highly effective alternative [15]. NLP is a machine learning-based approach that automates the analysis of textual data, efficiently extracting meaningful insights [16]. By adopting NLP, the analysis of customer feedback becomes not only more structured and less labor-intensive but also broader in scope and more objective, addressing the critical shortcomings of the traditional manual coding method. In hotel review analysis, NLP techniques streamline the process by identifying specific aspects of customer feedback, such as service quality or room cleanliness, using targeted keywords [17]. This is followed by an advanced stage of keyword and sentiment analytics, where the frequency of keywords is assessed (word counting) and sentiments are analyzed to determine whether they are positive or negative [18].

Despite the advancements in NLP techniques, there are inherent limitations, particularly when dealing with the complex and ambiguous nature of hotel review texts. These reviews often contain irrelevant or biased information and sometimes present mixed sentiments within a single sentence, making it challenging to extract clear insights using simple NLP methods [19]. The process of aspect determination in NLP still largely relies on the manual identification of keywords, introducing subjectivity and limiting scalability, which can vary significantly and affect the consistency of the analysis [20,21]. Another major challenge is the labor-intensive and costly process of creating supervised datasets, a process that is difficult to generalize across different contexts. These limitations highlight the need for more sophisticated NLP methodologies capable of handling the subtleties and complexities of consumer feedback in the hospitality industry.

The recent emergence of large language models (LLMs) like ChatGPT represents a breakthrough in the application of NLP, particularly in tasks like aspect-based review analysis. Trained on massive datasets with extensive parameters, these models are capable of human-like text generation and understanding context, significantly enhancing the accuracy and depth of analysis [22]. ChatGPT’s ability to generalize across numerous tasks without specific fine-tuning makes it a promising tool for automated text analysis in various domains, including hospitality [23]. Its application in aspect-based review summarization can potentially overcome the limitations of traditional NLP approaches, providing more accurate, context-aware, and comprehensive insights into customer opinions and experiences [24].

The objective of this study is to propose a novel approach for analyzing aspect-based hotel complaint reviews using ChatGPT. Our methodology involved using a TripAdvisor dataset and defining ten hotel attributes to create aspect–summarization pairs for each attribute. We designed prompts that enabled ChatGPT to perform aspect-based review summarization with respect to these predefined attributes. The output produced by ChatGPT was in a format conducive to further analysis. Our qualitative analysis of the sampled results showed that ChatGPT could successfully summarize the reviews, providing concise and relevant information about each comment. Further analysis included examining the distribution of topics with respect to different market segments (budget, midrange, and luxury hotels). Additionally, we conducted topic modeling for each hotel attribute to understand the variance in customer feedback across different hotel categories. To the best of the authors’ knowledge, this study is the first to exploit ChatGPT for aspect-based review analysis. This comprehensive approach using ChatGPT for aspect-based summarization demonstrates a significant advancement in the way hotel reviews can be analyzed, offering deeper insights into customer experiences and perceptions.

2. Related Works

2.1. Aspect-Based Sentiment Analysis

One of the widely used methods for NLP-based hotel review analysis is aspect-based sentiment analysis (ABSA). ABSA is designed to connect sentiments to specific aspects or features within a text. This method provides a detailed sentiment breakdown for different aspects of a product or service, offering deeper insights compared to general sentiment analysis. For example, in a restaurant review, sentiments about food and service can be distinctly analyzed (“The pasta was delicious” vs. “The service was slow”).

A key step in ABSA is aspect identification, in which various methods are used for effective detection. Frequency-based methods focus on word occurrence, particularly nouns, assuming that frequently mentioned words indicate key text aspects [16]. Syntax-based methods rely on syntactical word relationships for aspect identification, adeptly linking sentiment words with known aspects [17]. Supervised machine learning techniques treat aspect identification as a classification problem, using labeled datasets to train algorithms to learn discerning aspects based on word context and parts of speech [25]. Conversely, unsupervised methods like Latent Dirichlet Allocation (LDA) [26] do not require pre-labeled data, instead analyzing word distributions to uncover hidden patterns and topics, forming clusters representing different aspects [27].

Regarding hotel reviews, ABSA has been applied in various innovative ways. Ray et al. [28] developed an ensemble-based hotel recommender system combining sentiment analysis with aspect categorization using BERT models and a Random Forest classifier. Akhtar et al. [29] created a hotel recommender system using LDA, sentiment analysis, and summarization for hotel review analysis. Sann and Lai [18] applied NLP-ABSA to understand service failures in the hotel guest cycle, comparing experiences among different guest demographics. Al-Smadi et al. [17] enhanced ABSA for Arabic hotel reviews with supervised classifiers, focusing on aspect category identification and sentiment polarity. Lastly, Hegde and Seema (2017) [30] explored opinion summarization using machine learning, including an iterative decision tree method for aspect-based feature extraction from customer reviews. These studies demonstrate ABSA’s potential in extracting valuable insights from customer feedback.

2.2. Aspect-Based Text Summarization

Text summarization is the process of condensing the main points of a source text into a shorter version, preserving its key information and overall meaning. There are two primary approaches: extractive and abstractive. The extractive approach involves selecting a subset of existing words, phrases, or sentences in the original text to form a summary. The main objective is to identify and extract the most significant portions of the text that encapsulate the core message or the main points. It focuses on identifying and extracting the most important parts of a text. On the other hand, the abstractive approach involves creating a new, concise version of a text by paraphrasing and summarizing the main ideas. It generates a summary that captures the essence of the original text but in new words, often requiring more complex language-processing techniques [31].

The methodologies for the text summarization of hotel reviews in studies employ both extractive and abstractive approaches. Tsai et al. [19] used an extractive method beginning with the identification of helpful reviews, followed by sentiment analysis to enhance summary relevance. Ma and Li [32] introduced a weakly supervised extractive framework that relies on sentiment labels, using a hierarchical document encoder and a deep neural network for sentiment-preserving document summarization. Abdi et al. [33] combined machine learning with linguistic analysis for sentiment-oriented summarization, incorporating sentiment scoring and word-embedding models. Amplayo and Song [34] developed an adaptable sentiment analysis model for short reviews, emphasizing domain and language versatility. Hu et al. [35] proposed a multi-text summarization technique using content and sentiment similarities and a k-medoids clustering algorithm, considering author credibility. Tan et al. [36] presented a two-step extractive method, topic-anchoring-based review summarization (TARS), focusing on sentiment-specific topic aspects. Conversely, Li et al. [37] proposed an abstractive approach with the user-aware sequence network (USN), which personalizes summaries by integrating user preferences and styles; its usefulness was demonstrated effectively on the Trip dataset with over 536,000 reviews.

2.3. Natural Language Processing Using Large Language Model

The advent of large language models (LLMs) has significantly impacted the field of natural language processing (NLP). These advancements introduce new capabilities and challenges in text analysis and summarization. Recent research has focused on exploring the effectiveness of these LLMs in various NLP tasks such as information extraction or text summarization, highlighting both the strengths and limitations of LLMs in processing and summarizing complex textual data. Zhong et al. [38] found that ChatGPT excels in inference and sentiment analysis, similar to BERT models, but struggles with paraphrasing and similarity tasks, impacting its ability to accurately summarize complex texts like hotel reviews. Yang et al. [39] reported ChatGPT’s effectiveness in text summarization, matching traditional methods across various datasets, indicating its efficiency in condensing lengthy texts. Wei et al. [40] highlighted ChatGPT’s proficiency in zero-shot information extraction from unannotated texts, a crucial task for summarizing diverse hotel reviews without extensive pre-training. Törnberg [41] discussed the versatility and ease of use of LLMs like ChatGPT in various text analysis tasks, including sentiment and discourse analysis. Qin et al. [42] presented a nuanced view, noting ChatGPT’s strengths in reasoning but limitations regarding specific tasks like sequence tagging (which is important for structuring summaries). Koubaa et al. [24] reviewed ChatGPT’s technical advancements and future challenges, while Han et al. [43] identified strengths in generating responses but difficulties with irrelevant context and complex relations, underscoring the need for cautious application in hotel review summarization. Kocoń [22] observed a decline in quality and potential bias in complex tasks but noted ChatGPT’s personalization capabilities, suggesting its usefulness in customizing summaries.

3. Methodology

The overall flow of our methodology is shown in Figure 1. As shown in this figure, the first step of the method was the collection of hotel reviews from TripAdvisor, followed by preprocessing to prepare the text for analysis. Using ChatGPT, we automatically identified aspects and sentiments within the reviews, specifically focusing on negative sentiments, with the aid of a pre-trained BERT model. This approach allowed us to pinpoint the key areas of customer dissatisfaction accurately. Subsequently, ChatGPT was employed to generate abstractive summaries of the reviews, ensuring that the essence of customer feedback was concisely captured. For a deeper dive into the themes present in the feedback, Bertopic was utilized to perform advanced topic modeling. This step uncovered the predominant topics of discussion among customers, providing insights into specific areas of concern.

Our approach uniquely combines the capabilities of ChatGPT for aspect and sentiment identification with those of BERT for sentiment filtering and topic modeling. This methodology not only enhanced the precision of our analysis but also allowed for a comprehensive understanding of customer feedback in a nuanced manner.

Table 1. Methodological advancement in this study.

Feature	Previous Works	This Study
Aspect identification	Various methods ranging from frequency-based to unsupervised learning like LDA [16,26,27].	Automated detection using ChatGPT, enhancing precision and reducing subjectivity.
Sentiment identification	Primarily relies on manual coding or machine learning models, facing challenges regarding scalability and subjectivity [17,25].	Automated detection via ChatGPT combined with BERT for negative sentiment filtering, offering a comprehensive and nuanced sentiment analysis.
Text summarization	Both extractive and abstractive methods, limited by their reliance on manual input for effectiveness [19,32,33,34,35,36,37].	Automated summarization through ChatGPT, offering a novel approach to creating concise, meaningful summaries from extensive review data.
Topic modelling	LDA and other traditional methods for topic detection, often limited by the need for extensive manual tuning [26,27].	Advanced modeling with Bertopic, leveraging deep learning to uncover nuanced insights and trends in customer feedback, surpassing traditional LDA’s capabilities.

highlights the distinctions between our methodology and other previously employed methodologies from the literature.

3.1. Data Collection

In this study, we use a dataset that was obtained from a consumer review of TripAdvisor, which is one of the most popular travel and hotel review platforms globally. The dataset is publicly available at the cited source [44], which includes 878,561 hotel reviews, constituting a total of 1.3GB of data from 4,333 different hotels listed on TripAdvisor. These hotels are located in 25 major cities, including well-known U.S. cities like New York City and Los Angeles. We chose the TripAdvisor dataset because it is a popular online platform that collects reviews and ratings from users for various hospitality services, mainly hotels and restaurants. This dataset has many features that give a complete view of consumer ratings and reviews, as well as hotel information. Its important features include the following:

• Ratings—Detailed ratings are provided, covering overall satisfaction and specific areas like cleanliness, location, rooms, service, sleep quality, and value, with ratings ranging from 0 to 5;
• Review Title and Text—Each review has a title and a detailed description of the customer’s experience;
• Author Information—This includes the user’s name, the number of cities they have visited, the number of helpful votes they have received, the total number of reviews they have written, and the types of reviews;
• Hotel and Review Identifiers—Each hotel and review has its own unique identifier;
• Location—The physical location of the hotel;
• Helpful Votes—The number of helpful votes a review has received.

In our research, we concentrate on analyzing hotel review texts from New York City, a well-known tourist destination in the U.S. Our focus is specifically on understanding negative guest experiences, leading us to exclusively consider reviews with an overall rating of 3 or lower. While we acknowledge that reviews with ratings above 3 might also contain critical feedback on certain hotel service aspects, our decision to limit our analysis to lower-rated reviews was primarily motivated by the need to control the operational costs associated with utilizing ChatGPT. Despite its impressive efficiency and effectiveness in processing and analyzing text, ChatGPT requires significant quantities of computational and financial resources, necessitating the more focused approach used in our research.

Also, to make sure the reviews we analyzed were reliable and relevant, we only included reviews that had more than 5 helpful votes. This helped us to pick reviews that other users found useful and trustworthy, reducing the chance of including spam or irrelevant reviews. As a result, out of 267,057 total reviews of New York City hotels, we used 8539 reviews for further analysis.

3.2. Determination of Hotel Service Aspects

To effectively conduct aspect-based text summarization, we first established the specific hotel attributes that guided our analysis. This determination process involved a thorough literature review, ensuring that our attribute list was both comprehensive and precise. Our objective was to cover all significant customer criteria while avoiding redundancy and confusion in the summarization process. Thus, we aimed to acquire a list that was mutually exclusive and as concise as possible. After careful consideration of the related literature [18,45,46,47], we identified the following ten key attributes:

• Cleanliness—Evaluating the overall cleanliness of a hotel, including its rooms, bathrooms, common areas, and dining facilities;
• Room Quality—Assessing factors like room size, comfort of beds, in-room amenities, soundproofing (noise level), air conditioning, and maintenance;
• Service Quality—Reviewing the professionalism, friendliness, and responsiveness of the hotel staff, including reception, housekeeping, and other staff members;
• Food and Dining—Analyzing the quality, variety, and pricing of food and beverages offered in a hotel’s dining areas;
• Location—Considering the convenience of a hotel’s location in relation to attractions, transport, and business areas;
• Shared Facilities—The quality of shared facilities such as swimming pools, fitness centers, spas, business centers, elevators and parking;
• Check-in/Check-out—Assessing the efficiency and ease of the check-in and check-out processes;
• Safety and Security—Evaluating concerns related to personal safety, security measures at the hotel, and the safety of the area;
• Internet Connectivity—Analyzing the quality and reliability of Wi-Fi or other internet services;
• Décor and Design—Reviewing guests’ perceptions of a hotel’s design, decor, and overall atmosphere, which contribute to its character and mood.

These carefully selected attributes serve as the foundation for our aspect-based text summarization, allowing us to effectively analyze and interpret the vast array of customer reviews.

3.3. Design of Prompt Template for ChatGPT

A LLM’s task-specific performance crucially depends on the quality of the prompts used. Thus, we designed a dedicated prompt template, focusing on the following key elements: instruction, context, output, and input data. The prompt used for querying ChatGPT in this study is shown in Figure 2. We used ChatGPT-3.5 Turbo in this analysis. We utilized the standard ChatGPT model without applying any fine-tuning techniques. Our approach predominantly involved the zero-shot learning method, where we crafted single, specific questions to retrieve answers directly from ChatGPT. This method was chosen to explore the model’s inherent ability to understand and summarize complex customer feedback without the need for extensive training or customization.

3.3.1. Context

The context part usually includes background details and specific constraints, which are crucial to allow a model to understand and effectively respond to a given task. The first part of our prompt imposes a persona on GPT, giving it a more profound context and thus potentially enhancing the quality of the analysis. The second part provides the hotel service aspects slated to be analyzed.

3.3.2. Instruction

The instruction part defines the specific task or command that we wanted ChatGPT to execute. Detailed descriptions about each part of the prompt are given below:

• “Classify this complaint into the appropriate category and provide keywords or a reason for your classification with a concise single sentence”: This prompt directly instructs ChatGPT to focus on relevant hotel review criteria and summarize the reasoning behind its categorization, thus aiding in aspect-based summarization.
• “Categorization should not be overlapped”: This instruction directs ChatGPT to classify aspects in a MECE (Mutually Exclusive, Collectively Exhaustive) manner, ensuring that each category is distinct and that there is no redundancy in the responses.
• “Sentence also should be U.S. English if it is not”: Recognizing that TripAdvisor is a multilingual platform, this instruction allowed us to include reviews in various languages by translating non-English reviews into English.
• “Do not include other categories except defined above”: This instruction helped in minimizing response variation, ensuring ChatGPT adhered closely to our specified instructions and stayed aligned with this study’s objectives.
• “Do not include positive comments”: Our research concentrates on analyzing negative comments for each hotel attribute. This instruction directs ChatGPT to specifically focus on negative feedback, reducing the chance of irrelevant or off-topic responses and sharpening the precision of our sentiment analysis.

3.3.3. Output Indicator

This element defines the desired format for ChatGPT’s output. In our study, we chose the JSON format for presenting the aspect-based text summarization results. JSON is a widely used format, known for its compatibility with various databases and programming languages, making it an ideal choice for our analysis.

3.3.4. Input Data

This component of our prompt pertains to the actual data or questions that we aimed to analyze. In the context of our study, each record from our dataset, representing a consumer’s hotel review, was treated as an individual input. We have structured this input in a standardized format to ensure uniformity in processing.

3.4. Refining ChatGPT-Generated Result

ChatGPT, a generative model, inherently operates on a probability distribution, resulting in slight variations in its responses to the same question. This feature is a fundamental aspect of its design, enabling it to generate diverse answers. While the ChatGPT API offers controls allowing the adjustment of the degrees of freedom in its responses, there are instances where the answers may deviate from what we intended.

In this study, some variation in the responses that are not aligned with the prompt were observed. Firstly, on rare occasions, ChatGPT produced responses with uninformative comments (i.e., not available, N/A, no specific complaints) or included positive comments for each aspect. To resolve this, we measured the sentiment and its polarity for each sentence using FLAIR [48], an open-source, BERT-based pre-trained model specialized in sentiment analysis. Afterwards, we filtered the sentences whose sentiments were positive or neutral. Secondly, there were also instances of malformed output structures, including problems with quotation marks, unclosed brackets, and other issues that did not adhere to the standard JSON format. This issue was resolved by deriving manually crafted rules for correcting the formatting errors. This process resulted in a refined dataset, which better aligned with the objectives of our study.

3.5. Analytical Strategy

3.5.1. Complaint Frequency Analysis

In our analysis of the ChatGPT-generated responses, we conducted a thorough examination focused on understanding the distribution of customer feedback across various hotel categories. Initially, we delved into examining the frequency distribution of each aspect mentioned in the reviews. Also, we examined how complaint distributions differed across various hotel classes. To facilitate a structured analysis, we categorized hotels into three distinct classes based on their star ratings: hotels rated between 1 and 2.5 stars were classified as ‘Budget’, those rated between 3 and 4 stars were classified as ‘MidRange’, and the ones rated between 4.5 and 5 stars were classified as ‘Luxury’. This categorization was pivotal in understanding the expectations and experiences of guests across different levels of service quality and pricing. We hypothesized that there would be notable distributional differences in customer feedback across these different hotel classes. This assumption was based on the premise that guest expectations and experiences tend to vary significantly with the level of luxury and service offered by a hotel. By analyzing the ChatGPT-generated responses through this lens, we aimed to gain deeper insights into how customer perceptions and satisfaction levels differed among budget, mid-range, and luxury hotels, thereby enabling a more nuanced understanding of the hospitality industry from the customer’s perspective.

3.5.2. Topic Modelling

In our study, to gain a deeper understanding of service failures in relation to each aspect, topic modelling was applied. This process involved analyzing the summary text collected from each service aspect. Through topic modeling, we were able to identify common themes or topics within the data. These themes represented the recurring patterns, concerns, or notable aspects mentioned in customer feedback. By uncovering these commonalities, topic modeling facilitated a more straightforward analysis of patterns or trends in service failures. To perform topic modelling, we utilized BERTopic [49], a sophisticated topic-modeling technique based on BERT (Bidirectional Encoder Representations from Transformers) [50]. The pre-training of BERT on diverse text collections allows BERTopic to understand context and word nuances more effectively than would be possible using traditional methods like Latent Dirichlet Allocation (LDA). BERTopic’s reliance on BERT’s pre-trained knowledge enables it to interpret subtle language elements with less of a need for fine-tuning or domain-specific adjustments. This makes BERTopic particularly useful for our research, allowing for precise analysis and extraction of insights from complex language data. The overall procedure of BERTopic is shown in Figure 3.

In this study, BERTopic was applied to the collection of summarized texts for each hotel service aspect. BERTopic typically identified 5 to 10 distinct topics per aspect, each defined by specific keywords, capturing the primary themes and concerns from the reviews. We then conducted a frequency analysis to pinpoint the most prevalent topics for each service attribute. Finally, we analyzed how these topics varied across different hotel service classes, providing insights into the varying focus on certain aspects depending on a hotel’s service level.

4. Analysis Result

4.1. Aspect-Based Summarization Result Obtained Using ChatGPT

To evaluate the quality of ChatGPT’s responses, we conducted an analysis wherein several comments were randomly selected, and their aspect-based summarizations provided by ChatGPT were compared to the original text. The result is shown in

Table 2. Comparison of original consumer review and aspect-based summarization result obtained using ChatGPT.

Topic	Aspect-Based Summarization Result produced by ChatGPT
We stayed at the CP during our recent trip to NYC. The location is great and most of the people working there were very helpful and polite. (the lone exception was the balding guy with glasses at the information desk). But if you are going to stay here, be prepared for the rooms to be average at best. Worn furniture and carpet, not overly clean, and run down in general. We had a great time and used the room only at night, so if the room is not that big of a deal this hotel is OK, but if you are looking for nice, go elsewhere.	-room_quality: worn furniture and carpet, not overly clean, and run down in general
This place was tiny, run down and dirty. The staff was rude and it was close to nothing that was pen late night. It was better than Motel 6 but not much. oh and the elevator broke down.	-location: close to nothing that was open late night -room_quality: tiny, run down
Pulled up...valet came out to greet me. Turns out he was the valet AND the bellhop. After the 10 min it took to get luggage to the room, he went downstairs to presumably park the car. About 10 min later got a call that my car had been hit by a truck (which was under vontract to the hotel to remove laundry). No apology was made for the incident, no offer for compensation given. Later, after dealing with the front desk and on-call manager, neother of whom made any effort to make my stay better, I was given the General Managers number. He made the call to compensate me for my room, albeit reluctantly. Additionally, folks above me were playing music and throwing stuff off their patio at at 4 AM. If you are teir on the weekend have to fight through a velvet rope to get back to your room amongst the desperate ho hos that skank around its first floor club.	-service_quality: Unprofessional and unresponsive staff, no apology or compensation for car incident -location: Inconvenience of fighting through a velvet rope to access rooms -room_quality: Noise disturbance from guests above and lack of security in the hotel
Stayed 26–28 December 2005. A modest hotel on the upper West side, the rate of $330 this time for 3 people in a “suite” came too close to better hotels mid-town. Staff were great as usual, and the room is large by NYC standards. But the old HVAC system, which worked on a whim, soured the stay as the room temp alternately went from very cold to too hot.\nThe ajoining resturant, where Lucerne provides a 20% breakfast discount, still suffers from very poor service. Marybeth’s, one block away, is far superior.	-room_quality: The old HVAC system, which worked on a whim, soured the stay as the room temp alternately went from very cold to too hot. -food_dining: The adjoining restaurant, where Lucerne provides a 20 breakfast discount, still suffers from very poor service.
Stayed in the Wellington Hotel for two nights only, thankfully…Took ages to check in, there was a queue a mile long with two persons on the desk. Whilst I was checking in, two groups approached the reception desk to make complaints. The room we were allocated was small, carpet was filthy, there were chips gone out of furniture and the air conditioning units outside the windows were awfully noisy. We refused to stay in this room, and all that was available seemingly was a smaller room, however it was much cleaner so we agreed to stay there. We were only in new york for two days and most certainly did not want to have to go searching for another hotel!!! The only thing going for this hotel is it’s location, it’s just a couple of mins away from central park/carnegie hall/time square!	-check-in/check-out: long wait to check in with complaints from other guests -room_quality: small room with noisy air conditioning units

. The aspect-based summarization exhibited accuracy in aligning with the primary concerns highlighted in the reviews. For instance, for room quality, specific issues such as “worn furniture and carpet” were precisely identified, reflecting a clear understanding of the guests’ feedback regarding their accommodation conditions. Concerns about location, like being “far from late-night venues,” were also accurately captured, indicating an effective interpretation of guests’ inconvenience. Furthermore, the summarization adeptly noted service quality issues, consistently pointing out instances of unresponsive staff and incidents like car accidents, without deviation from the original sentiments expressed in the reviews. These examples support the conclusion that the aspect-based summarization accurately mirrored the key themes and specific concerns raised by the guests, demonstrating its effectiveness in capturing the essence of guest experiences.

4.2. Service Failure Frequency Analysis

First of all, we examined the service failure distribution across different hotel failure aspects. The analysis result is shown in Figure 4. This graph depicts the distribution of service failure aspects mentioned in the consumer reviews, where each review can correspond to one or more of the ten service aspects. The graph aggregates these reviews, counting each occurrence of a service failure aspect in a review as 1, and displays the total count of each aspect. Following a similar approach, we also looked at how these service failures differed across various hotel categories, namely, budget, mid-range, and luxury. The findings from this part of our study are displayed in Figure 5.

One of the most striking observations was the high frequency of complaints related to room quality in Budget hotels, with a value of approximately 0.72, suggesting a critical area for improvement within this segment. This is closely followed by the mid-range and luxury segments, indicating that regardless of the price point, guests consistently expect high standards in their accommodation spaces. Another notable aspect is service quality, which has a high complaint frequency in luxury hotels, corresponding to around 0.68, possibly reflecting the elevated expectations that guests have for service in high-end establishments. Interestingly, the budget segment, while having fewer complaints in this category, still registered a significant frequency, suggesting that service quality is a universal concern throughout the industry. When examining complaints about cleanliness, budget hotels again show a higher frequency, suggesting a need for these establishments to invest more in maintenance and cleaning services to meet guest expectations. In contrast, the luxury segment, with the lowest frequency of cleanliness complaints, presents a relative strength in this aspect, which is often a selling point for high-tier hotels. Despite this, luxury hotels see a higher frequency of complaints regarding decor and design, an area where guests’ expectations are presumably influenced by the premium they pay for their stay.

Our analysis further reveals that certain aspects such as safety and security and internet connectivity do not vary significantly in complaint frequency across hotel classes, indicating these are areas of less concern for guests or that hotels are generally meeting expectations in these domains. However, the mid-range segment shows a marginally higher frequency of complaints about internet connectivity, hinting at an opportunity for differentiation by improving this service.

4.3. Topic Analysis accross Hotel Aspects

In this section, we present the results of our topic-modeling analysis for each service aspect. Utilizing the advanced capabilities of Bertopic, we meticulously analyzed the aggregated and summarized text to discern the most frequent and relevant topics within each service aspect. Detailed figures and tables illustrating the topic distributions, their keywords, and interpretations are shown in the appendix due to the extensive quantity of data. Instead, we summarize the findings from the keywords and their distributions across hotel segments in

Table 3. Highlighted topics across market segments.

Hotel Aspect	Budget Hotels	Mid-Range Hotels	Luxury Hotels
Room Quality	Concerns about size, comfort, and cleanliness.	Air conditioning and noise levels are problematic.	High standards for size and comfort are prioritized.
Service Quality	Front desk interactions are the most problematic.	Similar concerns, with emphasis on staff responsiveness.	High expectations for staff interaction and responsiveness.
Cleanliness	Issues with mold, peeling paint, and odor are prominent.	General cleanliness concerns, with odor issues noted.	Highest concern for room and bathroom cleanliness.
Check-in/Check-out	Inefficiencies in processes; rude staff.	Similar issues, with slightly lower emphasis.	Slightly higher percentages of check-in process issues.
Shared Facilities	Noise disturbances; elevator service issues are prominent.	Elevator speed and wait times are major concerns.	Expectations for high-quality gym and pool facilities.
Food and Dining	Limited breakfast options are a primary issue.	Concerns about value and unexpected closures of restaurants.	Overpriced food and dissatisfaction with service quality.
Décor and Design	Focus on room decor and comfort; hotel atmosphere issues.	Equal dissatisfaction with room decor and hotel atmosphere.	Dissatisfaction with atmosphere surpasses room decor concerns.
Location	Inconvenience of location in relation to attractions noted.	Highest emphasis on location convenience and proximity.	Proximity to attractions/business centers is crucial.
Internet Connectivity	Internet speed and access are critical, with moderate concerns.	Highest amount of scrutiny directed toward internet access and speed.	Costs associated with internet services are a standout issue.
Safety and Security	General safety and security concerns are significant.	Similar emphasis on safety, slightly less than that for luxury hotels.	Safety and security are the paramount concerns.

.

5. Discussion

This section provides a detailed comparison of our findings with those from prior research on hotel service failures and customer dissatisfaction. We chose notable studies employing text analyses, ranging from basic frequency analysis to advanced topic-modeling techniques, to identify key areas for service improvement within the hotel industry.

Table 4. Customer dissatisfaction factors found in previous studies.

Related Study	Data Source	Methods	Findings
Manickas and Shea (1997) [12]	220 complaints	Frequency-based methods	● Customers generally complained about fundamental service failures, such as declined service quality and rude employees.
H. Li et al. (2013) [51]	42,668 online reviews	Frequency-based methods	● Customers complained frequently about beds, reception services, and room size and decoration.
Berezina et al. (2016) [52]	2510 online reviews	Frequency-based methods	● Complaints were about tangible aspects such as facility-related issues and cleanliness.
Hu et al. (2019) [45]	27,864 online reviews	Topic Modelling (STM)	● High-end Hotels: Complaints were focused on intangible service issues and perceived value for the amount of money paid. ● Low-end Hotels: Complaints were about tangible aspects such as facility-related issues and cleanliness.
Xu and Lee (2016) [53]	3480 online reviews	Topic Modelling (LSA)	● Full-Service Hotels: Wi-Fi quality, staff helpfulness, facilities’ condition, parking, and bathroom cleanliness are major dissatisfaction points. ● Limited-Service Hotels: Noise is the main dissatisfaction factor, along with smoking, Wi-Fi, and vending machine issues. ● Suite Hotels with F&B: The cleanliness of rooms is a significant complaint, followed by issues with restaurants, parking, swimming pools, and air conditioning. ● Suite Hotels without F&B: Dissatisfaction stems from smoking/polluted air, noise, dirty bathrooms, slow Wi-Fi, and outdated facilities.
Guo et al. (2017) [46]	266,544 online reviews	Topic Modelling (LDA)	● One-Star Hotels: Room experience and communication are prevalent complaints. ● Mid-Range Hotels: Focus on basic aspects like parking, check-in/out process, staff service, bathroom facilities, and price. ● Four/Four-and-a-Half-Star Hotels: Emphasis on the quality of hotel and resort facilities, food, room size, and decoration. ● Five-Star Hotels: Homeliness, event management, pet friendliness, and luxury differentiation are key attributes.
Büschken and Allenby (2016) [54]	4467 online reviews of hotels in New York	Topic Modelling (LDA)	● Upscale Hotels in Manhattan: Check-in problems, room issues, noise, Wi-Fi charges, and inadequate amenities are dissatisfaction drivers. Positive factors include staff friendliness and willingness to return. ● Midscale Hotels near JFK International Airport: Complaints focus on noise, unpleasant odors, front desk service, and carpet cleanliness. Positive aspects are good service responses and proximity to JFK International Airport.
Mankad et al. (2016) [55]	5830 online reviews of 57 hotels in Moscow, Russia	Topic Modelling (LDA)	● Reviews mentioning location and experience are associated with higher ratings. Reviews discussing transactions and value are associated with lower ratings. ● Amenities are less influential on ratings when sentiment is factored in, implying a more objective viewpoint on this aspect.

summarizes the research on customer (dis)satisfaction using text analysis, highlighting the methodologies and key findings of each study.

5.1. Service Failure Frequency Analysis

This section aligns our analysis of service failure distribution, outlined in Section 4.2, with findings from previous research, as summarized in Table 4. It draws comparisons and highlights consistencies and deviations in customer dissatisfaction across different hotel classes.

Notably, our analysis identified room quality as a paramount concern across all hotel segments, with budget hotels exhibiting a particularly high frequency of complaints (approximately 0.72). This finding aligns with the results reported by Xu and Lee (2016) [53], who identified room quality issues such as Wi-Fi quality and bathroom cleanliness as being major dissatisfaction points across different hotel types, but it especially echoes the concern for tangible aspects like facility-related issues and cleanliness highlighted by Berezina et al. (2016) [52] and Hu et al. (2019) [45] for lower-end accommodations.

Our study further highlights a notable discrepancy in service quality complaints, with luxury hotels experiencing a higher frequency (around 0.68), reflecting guests’ elevated expectations. This finding resonates with the results reported by Hu et al. (2019) [45], who observed that high-end hotel complaints often revolve around intangible service issues and perceived value for money. Interestingly, despite the budget segment registering fewer complaints regarding service quality, its significant frequency suggests a universal demand for exemplary service across the industry, reinforcing the importance of staff interactions noted by Büschken and Allenby (2016) [54] in upscale Manhattan hotels. In terms of cleanliness, our analysis reveals a stark contrast between budget and luxury hotels, with budget accommodations showing a higher frequency of complaints. This discrepancy underscores the critical role of cleanliness in guest satisfaction, particularly in lower-tier hotels, and aligns with the emphasis on facility cleanliness in previous studies by H. Li et al. (2013) [51] and Guo et al. (2017) [46].

Moreover, our findings indicate that concerns about safety and security and internet connectivity do not significantly differ across hotel classes, suggesting that these are baseline expectations for guests. However, the midrange segment’s marginally higher frequency of complaints about internet connectivity hints at a specific area for differentiation, echoing Xu and Lee’s (2016) [53] discussion on Wi-Fi quality as a major dissatisfaction point. In summary, our analysis actively corroborates and contrasts with prior research, confirming the universal importance of room quality, service quality, and cleanliness across hotel classes while also revealing hotel-segment-specific areas for improvement. This comprehensive comparison provides actionable insights for hotel managers aiming to enhance guest satisfaction tailored to their hotel’s market segment.

5.2. Topic Modelling Analysis

This section juxtaposes the insights derived from our topic modeling analysis with those obtained in previous studies, as depicted in Table 4. Through this comparison, we uncover both alignments and variations in customer dissatisfaction trends across different hotel classes.

The studies by Manickas and Shea (1997) [12] and H. Li et al. (2013) [25] laid the groundwork by highlighting fundamental service failures, including declined service quality and concerns regarding room features such as beds and decoration. Our analysis takes these concerns further by examining specific aspects like air conditioning, noise levels, and Wi-Fi quality, thereby uncovering a broader spectrum of service expectations and failures. This addition of granularity to the understanding of service failures illustrates the evolution and expansion of guests’ expectations to encompass modern amenities and environmental comfort.

In the domain of cleanliness and facility-related issues, the work of Berezina et al. (2016) [29] identified complaints focusing on tangible aspects such as facility conditions and cleanliness. Our findings echo the significance of cleanliness across all hotel segments while shedding light on specific issues like mold and odors, which are more pronounced in budget hotels. This suggests a segmentation in cleanliness standards and expectations, indicating the necessity for hotel management to adopt tailored cleanliness and maintenance strategies to meet varying guest expectations.

Our study also navigates the dichotomy between intangible service issues and tangible aspects as delineated by Hu et al. (2019) [26], who observed a differentiation in complaints between high-end and low-end hotels. We extend this analysis across budget, midrange, and luxury hotels, demonstrating that both intangible and tangible aspects critically influence guest experiences across segments, with varying emphases. This elucidation of the complexity of customer dissatisfaction suggests an interrelation between tangible and intangible service failures, affecting guest experiences in a more interconnected manner than previously thought.

Further, aligning with the segment-specific concerns identified in studies such as Xu and Lee’s (2016) [27] and Guo et al.’s (2017) [28], our findings confirm these concerns while offering deeper insights into emerging dissatisfaction factors like internet connectivity and shared facility expectations. This showcases the evolving nature of guest expectations, where traditional concerns are complemented with new factors reflecting the dynamic landscape of the hospitality industry.

Lastly, in line with the research conducted by Mankad et al. (2016) [31], who highlighted the significant influence of location and experience on guest ratings, our study reaffirms the importance of location across all segments and introduces the critical role of digital amenities and experiential factors in shaping modern hotel experiences. This suggests a shift towards a more holistic view of guest satisfaction, where both physical and digital amenities play pivotal roles.

In conclusion, our research builds upon the foundational work of previous studies to offer fresh perspectives on the multifaceted nature of hotel service failures and customer dissatisfaction. By providing a nuanced analysis across different hotel market segments, we equip hoteliers with actionable insights allowing them to effectively address both traditional and emerging guest concerns, fostering an environment that meets and exceeds the evolving expectations of today’s travelers.

6. Conclusions

In concluding our study, we have demonstrated the significant potential of ChatGPT, a state-of-the-art large language model (LLM), in conducting advanced aspect-based analysis of hotel reviews. Utilizing a dataset from TripAdvisor, our methodology involved categorizing reviews according to ten predefined hotel attributes and employing ChatGPT for the aspect-based summarization of these reviews.

From a methodological point of view, this research represents a methodological innovation in the analysis of hotel service failures and customer dissatisfaction by integrating advanced natural-language-processing (NLP) technologies, notably leveraging ChatGPT for aspect and sentiment detection, and employing a pre-trained BERT model for focused negative sentiment analysis. Distinguishing itself from traditional methodologies, it combines the efficiency of extractive summarization with the depth of abstractive techniques through ChatGPT, enhancing the accuracy and context-awareness of text summarization. Additionally, it advances topic modeling by utilizing BERTopic, surpassing conventional approaches like LDA in identifying nuanced topics within customer feedback. This comprehensive and sophisticated analytical framework not only improves efficiency and reduces subjectivity but also provides a deeper, more precise understanding of customer dissatisfaction, setting a new benchmark for research in hotel review analysis.

Our findings revealed the model’s proficiency in extracting relevant information efficiently, providing a detailed and insightful overview of customer opinions on various service aspects. Our analysis underscores the critical importance of addressing room quality, service quality, and cleanliness across all hotel segments to enhance guest satisfaction while also recognizing segment-specific areas such as internet connectivity and facility-related issues for targeted improvements. By aligning with and expanding upon previous research, it highlights a dynamic hospitality landscape where both tangible and intangible service failures significantly impact guest experiences. This comprehensive approach provides hoteliers with actionable insights allowing them to tailor their strategies effectively, ensuring that both traditional and emerging guest concerns are addressed to meet the evolving expectations of today’s travelers.

For future work, there are several promising directions that can be explored to enhance our understanding of customer feedback in the hospitality industry. One area of focus could be the exploration of how cultural differences and gender perspectives influence guest feedback, which could uncover nuanced patterns in guest experiences and expectations. Another avenue is the identification of dynamic patterns in guest feedback over time, an approach that has the potential to reveal evolving trends in customer expectations and service quality, thereby offering valuable insights for ongoing improvement in the hospitality sector. Furthermore, we plan to conduct more comprehensive evaluations of ChatGPT’s performance, including detailed comparisons with other methods. This will involve exploring various prompt strategies, such as zero-shot, few-shot, and chain of thought, to determine their effectiveness in different contexts. Additionally, we are considering the fine-tuning of ChatGPT to enhance the precision of aspect-based text summarization further. This process would likely include developing a supervised dataset, carefully annotated by experts, to train this language model.