Review Evaluation for Hotel Recommendation

Abstract

With the prevalence of backpacking and the convenience of using the Internet, many travelers like sharing their experiences in online communities. The development of online communities has changed the decision-making process of consumer purchasing, especially for travel, i.e., some travelers reconsider their decisions because they believe that the reviews of online communities are more valuable than advertisements. However, these reviews are not completely reliable since most reviews are provided without specific author information and the review data are too large to be observed. In this paper, we propose a novel approach (named ET) to evaluate the trustworthiness of reviews in online travel communities. Our method considers three concepts, including the sentiment similarity of reviewers in the social network, features of the reviews, and behaviors of the reviewers. The experimental results demonstrate that our method is effective in evaluating the trustworthiness of reviews.

1. Introduction

With the prevalence of backpacking and the convenience of using the Internet, many tourists like sharing their experiences about their travels in online communities, e.g., Tripadvisor.com, travellerspoint.com, etc. Online travel communities [1,2,3,4] provide a platform where some tourists can post their opinions about the beauty of tourist attractions, the service quality of hotels and restaurants, and their impressions of the travel facilities. Other potential tourists, who have not yet taken that trip, may browse the opinions and experiences before their tour [5]. This information, though huge and continuously generated, can affect the decisions of these first-time visitors, especially their decisions about which hotels they will stay at. According to the study [6,7], a large portion of travel consumers browse hotel and restaurant reviews on message boards or online communities before booking the hotel or restaurant. In fact, consumer reviews, compared to traditional advertisements, have a strong influence on potential consumers’ choices about hotels and restaurants.

In general, online travel communities provide useful information for global tourists. Many tourists make decisions about choosing a hotel, restaurant, or entertainment venue based on reviews from reviewers who have already visited these places. However, in some cases, the review data are large and may involve deceptive activities. This is because anyone can post reviews in online travel communities in an anonymous way. For example, hotel staff can pretend to be tourists and write fake reviews in order to promote their own business or defame other competitors. In this situation, spam reviews are a big problem for tourists making decisions about choosing places to visit. It is not so much about the cost of visiting so much as about the possible spoiled impression and wasted time, for which often cannot be compensated. Currently, it is difficult to distinguish whether a review is fake or not. Recently, several studies have focused on online review spam. Most of them are aimed at tangible products, e.g., books and clothes, which consumers can roughly evaluate before inquiring. Reviews cannot dramatically change the minds of consumers when they purchase a tangible product. Nevertheless, regarding intangible products, such as a tour experience, consumers are more likely to be influenced by word-of-mouth [8,9]. The reason is that word-of-mouth has more persuasive power than advertisements, especially for intangible products [10]. Thus, many consumers pursue the opinions of experts or past users before making a purchase. As a result, the trustworthiness of the reviews for these products is more important [11,12,13].

Techniques for detecting review spam of tangible products can be categorized into two types: the supervised-learning-based classification approach (SLC) [14,15,16,17]. Most SLC-like methods adopt logistic regression or support vector machines to determine whether the review is malicious or not. These SLC-like methods need to train a model beforehand; indeed, the training sets are hard to obtain because review spam is difficult to define. Regarding the NLP-like methods, they infer review spam from their semantic similarities. Although NLP-like methods can derive the sentiments and abstracts of reviews, the processing speed of these methods is slow. In addition, NLP-like methods cannot identify review spam if there are slight similarities among them.

Some online travel communities, like Tripadvisor.com (the largest online travel review community [10,18], have announced that they can filter fake reviews by analyzing reviews and comments. However, they do not disclose their criteria for judgment. Other communities have only declared that they apply three approaches to detecting review spam, namely, an expert audit, impeachment, and a heuristic approach. Clearly, it is necessary to expend a lot of human resources if the detection of spam is performed by expert audits. Regarding the impeachment approach, a new problem probably arises when the impeachment is issued by malicious users. Regarding the heuristic approach, it is stated that they can filter fake reviews through semantic analyses or automatic summarization; however, the detailed process is secret owing to business interests.

In this paper, we propose a novel approach to evaluate the trustworthiness of reviews in online travel communities. Our method considers three concepts, i.e., the sentiment similarity of reviewers in the social network, features of the reviews, and behaviors of the reviewers. We constructed a social network among the reviewers according to their relationships. For example, if the reviewers posted a review about the same hotel, their score increased on the social network. In this case, the amount of approval for all reviews of a specific reviewer was high, and thus, the score of this reviewer was also high. We analyzed the behavior of reviewers to understand their enthusiasm, including their participation degree and active time. For example, we evaluated the participating degree of each reviewer according to the time and effort they spent [19]. It indicates enthusiasm if a certain reviewer’s participating degree is higher than others. In general, more enthusiastic reviewers contribute reviews with higher reliability. According to several studies [10,20,21,22], review features can influence the trustworthiness of reviews. For example, if a review contains a specific hotel name many times, we can suspect the review is spam because the review intends to make an impression of the hotel on potential customers. This paper utilizes the above three concepts and maps them onto a metric to measure the trustworthiness of reviews in online travel communities.

We conducted several experiments to test the effectiveness of our method in comparison with the ranking results of experts. The experimental analysis, i.e., Spearman’s rank correlation coefficient, demonstrates that the ranking results of our method and those of the experts have an acceptable positive correlation.

2. Literature Review

2.1. Trustworthiness of Social Networks

Social networks can represent the social relationships of human society. The technology of social networks provides an effective method to analyze the relationships among persons, groups, and organizations. Many studies have applied social theory, mathematics, and statistics to analyze social networks to understand human activity through large interactivity data [23,24,25]. A social network consists of nodes and links. A node denotes an individual object, like a person, group or organization, and a link between two nodes represents the relationship between the two corresponding individual objects, such as friends or a borrower and lender. There are three types of links: context, strength, and direction. The context link shows the cause of a relationship between two corresponding individuals, such as friendship, kinship, or the same interests between the two individuals. The strength link indicates the power of the link in the socio-gram, which is used to measure the intensity of a relationship between two nodes. A link is undirected or directed. An undirected link shows a reciprocal relationship, like friendship between two persons; while a directed link forms a one-way relationship, like a borrower and lender.

In online traveling communities, users can browse, comment, and post their reviews to share with each other. By definition, such a social relationship can form a social network. For example, in Figure 1, there is a directed social network for two hotels α and β. Every node refers to a reviewer who posted a review or commented certain review. An example of a directed link, from node A to node C, shows that user A posted a review, and then user C commented on the review of user A. Regarding the no-response nodes, i.e., H and I, the status indicates that they posted reviews, but other users did not comment on their reviews. Regarding the width of the directed link, if it is wider, the two nodes are more similar. For example, the sentiment of the link D-to-A is wider than that of C-to-A, so the two nodes D and A are more similar. By using the technology of social networks, we can analyze the properties of nodes and links to obtain the trustworthiness of reviews and reviewers in online travel communities.

Trust is an optimistic expectation of an individual about the outcome of an event or behavior of a person [26]. The trust of an individual can be further quantified as the trust degree of the individual [27,28,29], which represents the expectation degree of the individual who will produce the intended results. There are three types of trust for an individual, namely, calculative trust, relational trust, and institutional trust [30]. We can determine calculative trust by analyzing the past behaviors of a person. Relational trust involves the interactions between people. For example, a person evaluates their degree of trust in another person through their long-term interaction with them, and their impression of previous interactions can strongly influence their trust degree in the other. Institutional trust imposes the regulation of involved institutions. For example, companies sign a contract to ostensibly establish trust.

Relational trust is a common type of trust and people usually have face-to-face interactions with each other. With the rapid development of technology, relational trust is difficult to build or maintain since many people communicate with each other just through the Internet or cell phones without face-to-face interactions. Thus, calculative trust becomes the major way that people evaluate trust in modern society. Calculative trust estimates the trust of people by analyzing their past online behaviors. For example, in online shopping, sellers can accumulate their own trust degrees by using customer feedback, i.e., past transactions between customers and sellers. If a seller has a higher trust degree, it indicates that customers probably trust the seller. Likewise, we can evaluate the trust of reviewers by analyzing their past behaviors in online communities. In this paper, we adopt the concept of calculative trust to measure the trustworthiness of reviews and reviewers in online travel communities.

2.2. Review Spam Detection

According to previous studies [31,32,33], review spam can be classified into three types. The first type, called an untruthful review, contains either malicious reviews from competitors or boasts reviews from conspiracy persons. The second type, called a brand-only review, means the review is provided for the purpose of embedded marketing. The third type, called a non-review, means that the reviews are unconnected to the major topic; instead, they include advertising slogans or similar purposes. In general, brand-only reviews and non-reviews have more significant features, such as the amount of feedback and the length of the title; therefore, these two types of spam reviews can be detected easily. In comparison, untruthful reviews are more difficult to detect because they have insignificant features and look like normal reviews.

In general, there are two kinds of approaches to detecting the three types of review spam. The first one is supervised learning-based (SL) classification [31,34], which considers the problem of detecting review spam as a classification problem. SL-based approaches classify reviews as review spam or non-spam according to the different features between them. The SL-based approach needs training data (i.e., the features of untruthful reviews) in order to construct a model, like a neural network. Studies [31,35] used manually labeled training sets to construct a model. By using the trained model in [31], the classifier could effectively identify brand-only and non-review spam; however, the precision rate of recognizing untruthful reviews was low. This is because the training data used for the classifier did not include the features of untruthful reviews. Specifically, the study [31] processed a characteristic of untruthful reviews while the reviews had similar contexts to others, and hence, the reviews had a very high probability of being untruthful ones. In other words, the study [31] collected duplicate and approximately duplicated reviews and then fed them into a model, which served as a training set and was used to build the classifier. Such a model could only detect a small portion of untruthful reviews, especially when the reviews had duplicate characteristics.

In addition to spam reviews, other studies [36,37] have also adopted the technology of a classifier to detect spammers. The joint detection of spam and spammers is formalized as an optimization problem, and the relationship between reviewers and their reviews is represented as constraints in the graph regularization for ensuring whether the result of detecting spam reviews is consistent with that of spammers. Then, the least squares support vector machine (LS-SVM), based on the maximum margin principle, is used to classify the spam and spammer. In comparison to traditional SVM, the LS-SVM-like approach could find more spam and spammers, but there was still an insignificant improvement in accuracy.

Taking plagiarized reviews into consideration, a study [20] applied Kullback–Leibler divergence to evaluate the overlapping degree between two reviews for detecting review spam. In order to avoid the synonym problem, the study used the service of WordNet v.2.1 to compare the overlapping degrees. Nevertheless, such a WordNet-based method encountered another problem with substitute words. For example, spammers can use the symbol “$” to substitute the word “dollar”. Moreover, spammers can use “fantasabctic” to substitute for the word “fantastic”. These words have semantic relationships in spammers’ and readers’ minds, but this is irrelevant in WordNet.

In sum, the previous methods, based on supervised learning-based classifications, are inadequate to prepare canonical training sets to train their classifiers. Most of these methods adopted insufficient review features as the factors in determining the trustworthiness of reviews. For example, a review has a high probability of being trusted if its context is richer. In addition, these methods seldom consider the behaviors of the reviewers, which can significantly affect the trustworthiness of reviews. For example, if a review is written by a reviewer who frequently writes or replies to other reviews, the review is supposed to have a high probability of being trustworthy. On the other hand, the natural language processing (NLP) approach has been applied to understand the semantics of reviews, extract summaries of reviews, and calculate the distribution of the sentiments of reviews. In this paper, we consider not only review features but also reviewers’ behaviors as influencing factors. Furthermore, based on the technology of social networks, we investigated the interaction between the reviewers and their commenters. Thus, we could evaluate the influential reviewers and their reviews.

3. Evaluation of Review Trustworthiness

In online travel communities, many users are enthusiastic about sharing their experiences with others by writing and posting reviews; however, malicious users can take advantage of anonymity to post fraudulent reviews. The review features and reviewers’ behaviors imply valuable clues to evaluate the trustworthiness of the reviews. In addition, the interaction between the reviewer and commenters is another important clue to evaluate the reviews.

In this study, we first constructed a social network with respect to the commented hotels, their associated commenters, and the followers of the comments. Through the segmentation of the social network, we evaluated reviews residing in an ego-centric social network rooted in a specific hotel. Three ideas, namely, the similarity of the reviewer’s sentiments about the social network, the reviewer’s behavior, and the semantic features of the reviews, are formulated as a final mathematical formula. The experimental study justifies the proposed approach to evaluating reviews, indicating the reliability of the authors’ approach. Three modules, shown in Figure 2, are used to implement the proposed method. The first module is the crawler module, which is equipped with a spider to collect the comments for the hotels from specific travel websites. The collected comments and the links embedded in the comments for hotels are used as the data to construct the ego-centric social networks rooted in each hotel. A pair of reviewers residing in the same ego-centric network can have a directed link between them if one’s comment is re-commented by the other. For example, reviewer D has a direct link to reviewer A if reviewer D re-comments the comment issued by reviewer A about the same hotel. The last part of the second model is to use the proposed mathematical formula (discussed later) to evaluate the trustworthiness degree of each collected comment. The three ideas mentioned above are used to calculate the final score of each comment. The last module is an interaction interface to reply to the ranked comments of the user for a queried hotel.

3.1. Construct of Social Network of Reviewers

The review features and reviewers’ behaviors imply valuable clues to evaluate the trustworthiness of the reviews. We applied the technology of social networks to model their relationship. The interaction between reviewers and commenters forms a social network that consists of nodes (i.e., reviewers or commenters) and direct links (i.e., feedback given by a commenter to a review posted by a reviewer). After browsing the review, commenters can choose to agree or disagree with its contents. For example, they press the “helpful” or similar button provided by online communities when they feel the review is useful. In the present case, the online community counts the “helpful” approvals of the same user once in order to avoid receiving indiscriminate feedback. Formally, we used NHG(r_i,j) to denote the number of “helpful” approvals received by the review r_i,j, where i represents the reviewer and j means the review posted by reviewer i. If the value of NHG(r_i,j) is large, it indicates that the review received many “helpful” approvals from its readers; as a result, we can assert that their review is highly trustworthy.

In some cases, users post a comment against or for the review if they appreciate, supplement, or disagree with the review. When a review is untruthful, the attitude towards the review will be opposite to that of its comments, and certain commenters will even criticize it. For identifying untruthful reviews, we applied the technique of the NLP approach to parse the comments for each review and comment. In addition, we have adopted the semantic orientation concept [38] to calculate the sentiment similarity between a review and each of its comments. Without loss of generality, we define that the sentiment of a review or comment is only divided into two modes, i.e., positive and negative modes. The part-of-speech tagger is used to locate the adjective and adverb words in the review and its related comments, and then the semantic orientations (abbreviated as SO) of these words are calculated. For each of these words (denoted as w), we calculated the semantic orientation of the word w against the reference words of positive and negative modes through the function of pointwise mutual information (abbreviated as PMI), which is used to evaluate the association degree of two words in an article. The two words are related if the value of PMI is high. The function of PMI is defined as:

$$PMI(w_s,w_t)={\log }_2(p(w_s,w_t)/p(w_s)\times p(w_t)),$$

(1)

where $p(w_s)$ and $p(w_t)$, respectively, denote the two probabilities that words $w_s$ and $w_t$ occur alone in an article, $p(w_s,w_t)$ is the probability that words $w_s$, and $w_t$ co-occur in the article. By definition [38], the semantic orientation of the word w in a review and its comments is calculated by:

$$\begin{gathered} SO(w)=PMI (w, “reference words of positive mode”) - \\ PMI(w, “reference words of negative mode”) \end{gathered}$$

(2)

where the two parameters of the PMI function, “reference words of positive mode” and “reference words of negative mode”, are the thesaurus of the two modes derived from WordNet.

In the travel review community, it is usual that the term “excellent” has the highest rating, and the term “poor” has the lowest rating; thus, we select the two terms as the standard words of positive and negative modes, respectively. With the help of WordNet, we can easily acquire all the reference words for “excellent” and “poor”, like fantabulous for excellent and miserable for poor. Thus, we evaluate the semantic orientations of the related words through Equation (2). Next, we sum the semantic orientation of all related words and then calculate the average in each review for the semantic orientation of the review, which is defined as:

$$SO(r_{i,j})=({\displaystyle \sum _{w\in E{r}_{i,j}}SO}(w))/\left|E{r}_{i,j}\right|$$

(3)

where $E{r}_{i,j}$ is the set of the extracted adjectives and adverbs in review r_{i j}, and $\left|E{r}_{i,j}\right|$ is the cardinality of words in set $E{r}_{i,j}$.

Similarly, for the kth comment for the jth review written by reviewer i (denoted as C_i,j,k), we can calculate its sentiment orientation as follows:

$$SO(C_{i,j,k})=({\displaystyle \sum _{w\in E{C}_{i,j,k}}SO}(w))/\left|E{C}_{i,j,k}\right|$$

(4)

where $E{C}_{i,j,k}$ is the set of the extracted adjectives and adverbs in comment C_i,j,k, and $\left|E{C}_{i,j,k}\right|$ is the cardinality of words in the set $E{C}_{i,j,k}$.

After obtaining the semantic orientation of each comment related to review $r_{i,j}$ (see Equation (4)), we compute the average semantic orientation of the comments for review $r_{i,j}$, which is denoted as $AvgCSO(r_{i,j})$. Assume there are $S{r}_{i,j}$’s comments on review $r_{i,j}$; then, the average semantic orientation of the comments for review $r_{i,j}$ is defined as follows:

$$AvgCSO(r_{i,j})=({\displaystyle \sum _{k=1}^{S{r}_{i,j}}SO(C_{i,j,k})})/S{r}_{i,j}$$

(5)

According to the rule of thumb, the review will have high trustworthiness in the viewpoint of semantic orientation if the similarity between the semantic orientation of a review and the average semantic orientation of all its comments is high. Such trustworthiness, denoted as $TSO(r_{i,j})$, refers to review $r_{i,j}$, and the value of $TSO(r_{i,j})$ is computed by:

$$TSO(r_{i,j})=1-\left|SO(r_{i,j})-AvgCSO(r_{i,j})\right|/\left|SO(r_{i,j})+AvgCSO(r_{i,j})\right|$$

(6)

The value of $TSO(r_{i,j})$ in Equation (6) falls into the range from 0 to 1. If the semantic orientation of review $r_{i,j}$ is close to the average semantic orientation of all its comments, i.e., $\left|SO(r_{i,j})-AvgCSO(r_{i,j})\right|\approx 0$, the value of $TSO(r_{i,j})$ will be close to 1; on the contrary, the value of $TSO(r_{i,j})$ will be close to 0 because the value of the equation $\left|SO(r_{i,j})-AvgCSO(r_{i,j})\right|$ is large. We can project the trustworthiness of the review by counting the amount of helpful approvals and calculating the sentiment of the comments from commentators for a review. For a review, the two metrics, NHG for counting the approval and TSO for counting the comments, can be merged into a single value, which is called the likelihood degree of sentiment (abbreviated as LDS).

 Definition 1. 

Given the trustworthiness $TSO(r_{i,j})$in the viewpoint of semantic orientations of review  $r_{i,j}$ against all its comments, and the number of helpful approval  $NHG(r_{i,j})$ received by this review. The two metrics,  $TSO(r_{i,j})$ and $NHG(r_{i,j})$ , are weighted to emphasize the importance of the trustworthiness of the review. Then, the likelihood degree of sentiment (LDS) of review  $r_{i,j}$ is the sum of the two weighted metrics as: 

$$LDS(r_{i,j})=\alpha \times TSO(r_{i,j})+\beta \times NHG(r_{i,j}),$$

where  $\alpha +\beta =1 \mathrm{and} \alpha >\beta$ holds. Note that the value of  $\alpha$ is larger than that of  $\beta$ because the procedure of writing a comment takes more time and effort than that of clicking the helpful button by a commentator.

3.2. Formulating the Trustworthiness Equation

3.2.1. Trustworthiness of Review Features

After discovering the factors in the trustworthiness of reviews, we now evaluate the trustworthiness of a review from its own features such as the length, original degree, outlier of sentiment distribution, and other characteristics.

Generally, if the content length of a review is long, reviewers will take the time to produce the materials of the review. We consider that the trustworthiness degree of a review is high if its content length is long. The title length of a review is another important feature to determine the trustworthiness of a review; similarly, a review is trustworthy if the title length is long. We denote the content length and title length of the review $r_{i,j}$ as $CL(r_{i,j})$ and $TL(r_{i,j})$, respectively. The trustworthiness degree of review $r_{i,j}$ is proportional to the magnitudes of $CL(r_{i,j})$ and $TL(r_{i,j})$. Since the content length and title length of reviews are very varied, we applied the nature logarithm to transfer these values to condense the variety of lengths among the reviews. Next, we normalize these values. The factors of content length (abbreviated as $FCL(r_{i,j})$) and title length (abbreviated as $FTL(r_{i,j})$), influence the trustworthiness degree of review $r_{i,j}$, and are defined as follows:

$$FCL(r_{i,j})=(\log (CL(r_{i,j}))-{\overline{x}}_{CL}^{H_{i,j}})/{\sigma }_{CL}^{H_{i,j}},$$

(7)

where $H_{i,j}$ is the hotel that review $r_{i,j}$ mentioned of, ${\overline{x}}_{CL}^{H_{i,j}})$ is the average of the logarithmic values of the population of context lengths in the hotel $H_{i,j}$, ${\sigma }_{CL}^{H_{i,j}}$ is the standard deviation of the population of context lengths in the hotel $H_{i,j}$.

$$FTL(r_{i,j})=(\log (TL(r_{i,j}))-{\overline{x}}_{TL}^{H_{i,j}})/{\sigma }_{TL}^{H_{i,j}},$$

(8)

where ${\overline{x}}_{TL}^{H_{i,j}}$ is the average of the logarithmic values of the population of title lengths in the hotel $H_{i,j}$, ${\sigma }_{TL}^{H_{i,j}}$ is the standard deviation of the logarithmic values of the population of title lengths in the hotel $H_{i,j}$. Note that we adopt the same procedure for other factors. The normalization operation for the variable $Y$ under the population $X$ is denoted as:

$$FCL(r_{i,j})=Nor{m}_{R{H}_{i,j}}(\log (CL(r_{i,j}))),$$

(9)

$$FTL(r_{i,j})=Nor{m}_{R{H}_{i,j}}(\log (TL(r_{i,j}))),$$

(10)

where $R{H}_{i,j}$ is the set of all reviews for the hotel $R{H}_{i,j}$, i.e., the population under the normalization equation.

In the travel community, spammers perhaps read others’ reviews and plagiarize the contents of these reviews [31]. In this manner, spammers can finish spam reviews in a short time. Thus, we consider that a review has a high probability of being unreliable if its content is similar to others. In other words, the review has a higher probability of being trustworthy if its content is more dissimilar to that of others. The original degree serves as a useful factor to determine whether a review is spam. The higher the original degree, the higher the probability of being trustworthy the review has. Based on the Kullback-Leibler (KL) divergence [20], we then evaluate the original degree between the two reviews. The KL divergence is used to evaluate the dissimilar degree $KL({\theta }_f||{\theta }_g)$ between two articles ${\theta }_f$ and ${\theta }_g$. The detail to calculate the value of KL-divergence is beyond the scope of this paper. In order to obtain the factor of the original degree of review $r_{i,j}$ (denoted as $FOD(r_{i,j})$), we initially calculate the KL-divergence of the review against other reviews of the same hotel and then compute the average of all KL-divergences, which is defined as:

$$FOD(r_{i,j})=Nor{m}_{R{H}_{i,j}}(({\displaystyle \sum _{{}_{r_{x,y}\in R{H}_{i,j}}^{r_{x,y}\ne r_{\hspace{0.17em}i,j} \mathrm{and}}}KL(r_{i,j}||r_{x,y})})/(\left|R{H}_{i,j}\right|-1)),$$

(11)

where $R{H}_{i,j}$ is the set of all reviews for hotel $H_{i,j}$, $\left|R{H}_{i,j}\right|$ is the number of reviews for hotel $H_{i,j}$. Note that the KL-divergences of review $r_{i,j}$ against other reviews (involved in the same hotel) do not count the KL-divergence against the review $r_{i,j}$ itself. Thus, the calculation: ${\displaystyle \sum _{{}_{r_{x,y}\in R{H}_{i,j}}^{r_{x,y}\ne r_{\hspace{0.17em}i,j} \mathrm{and}}}KL(r_{i,j}||r_{x,y})}$ excludes the KL-divergences of review $r_{i,j}$ against itself and then is divided by the value of $(\left|R{H}_{i,j}\right|-1)$. For standardizing the KL-divergence of all reviews for the same hotel, the factor of the original degree is normalized.

According to the study [31], two types of review spam “brands-only review” and “non-review” occur when the review contains a specific brand name many times and many links to advertisements or email addresses that reinforce the impression of the hotel or provide convenient ways of visiting the hotel. Hence, we define that the review has a higher probability of being unreliable if the content of a review has more hotel names and links than the average of reviews. For the review $r_{i,j}$, we count the number of hotel names $N(r_{i,j})$, hyperlink $L(r_{i,j})$, and email addresses $A(r_{i,j})$ that form the factor of advertisement characteristic $FDA(r_{i,j})$ as follows:

$$FDA(r_{i,j})=Nor{m}_{R{H}_{i,j}}(N(r_{i,j})+L(r_{i,j})+A(r_{i,j})).$$

(12)

The four factors, i.e., content length, title length, original degree, and advertisement characteristic, are under consideration in the review itself. The viewpoint of peers is another factor in evaluating the trustworthiness of the review. Specifically, if the sentiment of a review is very different from that of others in the same hotel, we suspect that it is review spam because of malicious or boasting. Initially, we calculate the semantic orientation of each review in the same hotel, which has been described in Section 3.1. Next, we apply the z-score to locate the anomalous review (also called outlier). According to the law of large numbers, we define a review as belonging to an outlier if the z-score of the semantic orientation of a review does not fall within the range from −3 to 3. In particular, we decrease the trustworthiness degree of a review only if the review is an outlier in the viewpoint of the semantic orientation of peer reviews; for other cases, the trustworthiness degree of the review is not changed. For review $r_{i,j}$, the factor of anomalous sentiment orientation $FOSO(r_{i,j})$ is determined by the peer reviews, which is defined as:

$$FOSO(r_{i,j})=\left\{\begin{array}{l}0, \mathrm{if} \mathrm{the} \mathrm{expression} 3\ge Norm(SO(r_{i,j})\ge -3 \mathrm{holds};\hfill \\ \left|Nor{m}_{R{H}_{i,j}}(SO(r_{i,j}))\right|, \mathrm{otherwise}.\hfill \end{array}\right.$$

(13)

Based on the proposed review features, we now formulate the trustworthiness degree of review $r_{i,j}$ as the Definition 2.

 Definition 2. 

The trustworthiness degree of review $r_{i,j}$,$TS(r_{i,j})$, is computed by the weighted values of factors: content length  $w_{fcl}$, title length  $w_{ftl}$, originality  $w_{fod}$, advertisement characteristics  $w_{fda}$, and extremity of sentiment orientation  $w_{foso}$.

$$\begin{array}{l}TS(r_{i,j})=w_{fcl}\times FCL(r_{i,j})+w_{ftl}\times FTL(r_{i,j})+w_{fod}\times FOD(r_{i,j})-\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{w}_{fda}\times FDA(r_{i,j})-w_{foso}\times FOSO(r_{i,j}).\end{array}$$

Note that we subtract the two values $FDA(r_{i,j})$ and $FOSO(r_{i,j})$ because they have negative influence on the trustworthiness degree of review $r_{i,j}$.

3.2.2. Trustworthiness of Reviewers’ Behaviors

In addition to the feedback from commentators and the review’s own features, it is important to notice the reviewer’s behaviors because their behaviors affect the trustworthiness of reviews, which includes the reputation degree, the prejudice degree, and the contribution degree.

In the online community, a mutual-rating mechanism arises in a stratified society. For example, in an auction website, the value of credit, such as 1, 0, or −1, is given by the final bidder. Usually, a good buyer or seller has a high credit, and vice versa. A similar mutual-rating mechanism is also provided in online travel communities. A reviewer earns “helpful” feedback from other readers if the reviewer writes a good review. The ratio of helpful feedback against the number of reviews written by a reviewer can be regarded as the credit of the reviewer. The reviewer with a high credit usually indicates his/her reviews are trustworthy. In addition to the helpful feedback, the reviewer has a higher reputation degree if his/her reviews obtain a high ratio of comments with the same sentiment orientation. The reputation degree of the reviewer $u_i$ is defined as:

$$FRD(u_i)=Nor{m}_{R{H}_{i,j}}({\displaystyle \sum _{j=1}^{NR(u_i)}NHG(r_{i,j})}/NR(u_i))+Nor{m}_{R{H}_{i,j}}({\displaystyle \sum _{j=1}^{NR(u_i)}TSO(r_{i,j})}/NR(u_i)),$$

(14)

where $NR(u_i)$ denotes the number of reviews written by reviewer $u_i$, $NHG(r_{i,j})$ is the number of helpfulness received from review $r_{i,j}$.

In online travel communities, reviewers are usually requested to rate the hotel before finishing their reviews. According to the study [39], review spam probably has an extreme rating such as one point (the poorest) or five points (the best) because spammers want to boast about certain hotels or defame competitor’s hotels. In fact, it is possible that a reviewer gives an extreme rating when a hotel is fabulous or poor in truth. We observe that the reality of extreme ratings can be justified by other ratings. For example, a reviewer gave most hotels one point since he/she usually met poor hotels, and other reviewers lived in the same hotels and also gave them low ratings. Thus, we compare a rating against others of the same hotel to measure whether the reviewer is prejudiced, which is defined as the prejudice degree of the reviewer $u_i$ and denoted as:

$$FPD(u_i)=Nor{m}_{R{H}_{i,j}}({\displaystyle \sum _{j=1}^{NR(u_i)}\left|Z(r_{i,j})\right|/NR}(u_i)),$$

(15)

where $Z(r_{i,j})$ is the z-score of review $r_{i,j}$’s rating under the population of all reviews of the hotel that review $r_{i,j}$ referred.

In general, the number of reviews written by a reviewer positively influences the credibility of the reviewer since the reviewer spends much time sharing reviews in the community. Thus, we consider that it is trustworthy if the reviewer $u_i$ makes substantial contributions. We classify a contribution into three types, namely, the number of reviews shared by the reviewer $u_i$ (denoted as $NR(u_i)$), the number of comments shared by the reviewer $u_i$ (denoted as $NC(u_i)$), and the number of photos and videos shared by the reviewer $u_i$ (denoted as $NP(u_i)$). The contribution degree of the reviewer $u_i$ (denoted as $FCD(u_i)$) is computed by:

$$FCD(u_i)=Nor{m}_{R{H}_{i,j}}(NR(u_i))+Nor{m}_{R{H}_{i,j}}(NC(u_i))+Nor{m}_{R{H}_{i,j}}(NP(u_i)).$$

(16)

The study [40] states that spammers usually post a lot of review spam in communities in a short time after they just registered. In addition, spammers seldom return to these websites again. As a result, the contribution degree, shown in Equation (16), needs to be degraded according to the passing time. We adopt the time gap (called idle time) to evaluate the decline of the contribution degree, where idle time refers to the time that a review appeared until now. If the value of idle time is long, the contribution degree is low. On the other hand, the registration time is another important factor for the contribution degree. In general, normal reviewers share reviews or comments continuously after they register in the online community. Thus, we evaluate the contribution degree of reviewers by using the period from their registration to their latest activity, which is called active time. The contribution degree is high if the value of active time is large.

As mentioned, we use two factors idle time and active time to evaluate the contribution degree of a reviewer. Specifically, the contribution degree of a reviewer $u_i$ is inversely proportional to the idle time (denoted as $IT(u_i)$) but is proportional to the active time (denoted as $AT(u_i)$). For example, Figure 3 shows that three reviewers $u_a$,$u_b$, and $u_c$ have the respective idle time $IT(u_a)$, $IT(u_b)$, $IT(u_c)$ active time $AT(u_a)$, $AT(u_b)$, and $AT(u_c)$. Suppose that $IT(u_a)$ is longer than $IT(u_b)$, but the active time of the two reviewers $u_a$ and $u_b$ are the same. In this case, the contribution degree of the reviewer $u_a$ is lower than that of the reviewer $u_b$. In other words, the reviewer $u_a$ has a higher probability of being a spammer than the reviewer $u_a$ because the idle time of the reviewer $u_a$ is longer than that of the reviewer $u_b$. Let us consider other cases. Suppose that $AT(u_b)$ is longer than $AT(u_c)$, but the idle time of the two reviewers are the same. In this case, reviewer $u_b$ has a higher probability of being normal reviewer than reviewer $u_c$ since the active time of reviewer $u_b$ is longer than that of reviewer $u_c$. Taking idle time and active time into consideration, we revise the measurement of contribution degree as:

$$FCD\prime (u_i)=Nor{m}_{R{H}_{i,j}}(AT(u_i)/IT(u_i))\times FCD(u_i).$$

(17)

Based on the proposed behavior features, we now formulate the trustworthiness degree of reviewer $u_i$ as the Definition 3.

 Definition 3. 

Let $TU(u_i)$ denotes the trustworthiness degree of the reviewer  $u_i$, which is computed by the weighted values of factors: reputation degree, prejudice degree, and contribution degree. That is, $TU(u_i)$ is defined as the following equation.

$$TU(u_i)=w_{frd}\times FRD(u_i)-w_{fpd}\times FPD(u_i)+w_{fcd\prime }\times FCD\prime (u_i).$$

(18)

Note that we subtract the value FPD(u_i) since it has a negative influence on the trustworthiness degree of the reviewer $u_i$.

As discussed, the trustworthiness of a review can be evaluated from its likelihood degree of sentiment against its peers, review features, and the behaviors of reviewers, which measurements are integrated as well as Definition 4.

 Definition 4. 

Let $T(r_{i,j})$ denotes the final trustworthiness degree of the jth review written by the reviewer  $u_i$ , which is calculated by:

$$T(r_{i,j})=w_{lds}\times LDS(r_{i,j})+w_{ts}\times TS(r_{i,j})+w_{tu}\times TU(u_i),$$

(19)

where w_lds, w_ts, and w_tu are the weights given to the factors of likelihood degree of sentiment, trustworthiness degree of review features, and trustworthiness degree of reviewer, respectively; in addition, the expression  $w_{lds}+w_{ts}+w_{tu}=1$ holds.

4. Evaluation and Empirical Studies

To evaluate our method, we performed several experiments with the online travel community at Tripadvisor.com, which is the largest online travel community. The trustworthiness degree of a review influenced by the weights of the factors was formulated heuristically in the set of experiments. In addition, a group of experts were invited to evaluate the appropriateness of our method.

4.1. Empirical Studies

We collected reviews of Taiwan Hotels from Tripadvisor.com. There were 17,234 records related to 1285 hotels in Taiwan, which were accessed from March 2020 to March 2023. The record information contains the details of the reviews and reviewers, such as the review title, review content, comments for the review, register date of the corresponding reviewer, and so on. In the data pre-processing, we selected the top three hotels that had the maximum number of reviews for representative purposes, that is, to avoid manipulation. There were 258 reviews of Les Suites Taipei, 152 reviews of the San Want Residences, and 22 reviews of the Dandy Hotel–Tianmu. Moreover, we randomly selected 10 reviews of each of the hotels. Without loss of generality, we gave the same weight to every factor, as shown in

Table 1. The value of every weight.

Value	Weight Type
1/10	wtso
1/10	wnhg
1/10	wfcl
1/10	wftl
1/10	wfod
1/10	wfda
1/10	wfoso
1/10	wfrd
1/10	wfpd
1/10	wfcd’

. Next, we computed the final trustworthiness degree by referring to the equation in Definition 4, and then we invited three experts who are experienced in the three hotels to rank these reviews.

The ranking results are presented in

Table 2. The review ranking in Les Suites Taipei Hotel: the difference between the experts and our method.

Reviewer	Our Ranking	The Average of Ranking by Experts	The Difference of Ranking
Kexxxxxx	1	2.33	1.33
pexxxx	2	3.00	1.00
Stxxx	3	8.67	5.67
Daxxxxxxxxxxxxxxx	4	3.33	0.67
Trxxxxxxxxx	5	2.33	2.67
Maxxxxxx	6	6.33	0.33
ztxxxx	7	5.33	1.67
puxxxxxxxxxx	8	7.33	0.67
anxx_xxxxxxxxxx_xx	9	5.33	3.67
Erxxxxxx	10	10.00	0.00
Average of difference (μ)			1.766667
Variance of difference (σ2)			3.112346

,

Table 3. The review ranking in San Want Residences: the difference between the experts and our method.

Reviewer	Our Ranking	The Average of Ranking by Experts	The Difference of Ranking
Bixx-xxxx-xxxxxxx	1	1.67	0.67
JAxxxxxx	2	5.00	3.00
paxxxxxx	3	6.00	3.00
phxxxxxxx	4	3.67	0.33
cmxxx	5	4.00	1.00
arxxxxxxxxxxx	6	4.67	1.33
Frxxx	7	7.33	0.33
Caxxxx	8	7	1.00
trxxxxxxx	9	7	2.00
gixxxxx	10	8.67	1.33
Average of difference (μ)			1.40
Variance of difference (σ2)			0.958025

and

Table 4. The review ranking in Dandy Hotel–Tianmu: the difference between the experts and our method.

Reviewer	Our Ranking	The Average of Ranking by Experts	The Difference of Ranking
Wixxx x	1	2.67	1.67
raxxxxxxxxxxxxxx	2	2.67	0.67
Vaxxxxxxxxx	3	5.00	2.00
SGxxxxxx	4	1.00	3.00
Ccxx	5	6.33	1.33
Mjxxxxxx	6	9.33	3.33
toxxxxxxxxx	7	8.33	1.33
Chxxxxxxx	8	4.67	3.33
esxxxxxxx	9	8.33	0.67
noxxxxxxx	10	6.67	3.33
Average of difference (μ)			2.066667
Variance of difference (σ2)			1.204938

. The first column shows the authors’ IDs related to their reviews crawled from Tripadvisor.com; the second column reveals the scores of the reviews that were evaluated by our method; the third column presents the average scores of these reviews that were assessed by the experts; the fourth columns show the difference in the value between that of the second columns and the third. A small difference implies that the experts and our method had similar judgments. In the case of the San Want Residences, the similarity was higher than in the other two cases because the average difference was the smallest. Regarding the average difference, the case of Les Suites Taipei had higher similarity than that of the Dandy Hotel–Tianmu. Regarding the variance in difference, the case of Les Suites Taipei had a higher divergence than that of the Dandy Hotel–Tianmu.

Here, we discuss the reasons behind the results. We discovered that in the review of the Dandy Hotel–Tianmu, posted by SGxxxxxx, our method underestimated the trustworthiness degree compared to the score given by the experts. In this case, the score of the reviewer’s behavior was relatively low, and so far, this reviewer has seldom participated in the community. Hence, the trustworthiness degree from the viewpoint of the reviewer’s behavior is much lower than others. Regarding the case of overestimation, e.g., the review of Les Suites Taipei posted by Stxxx contains extremely (positive) sentiments in spite of other reviews that have different sentiments. The experts suspect that this review was written by the hotel manager, and therefore, the experts gave a relatively low score. With our method, this review had a slightly low score because the extreme score was offset against other factors.

In order to demonstrate the effectiveness of our method, we compared the results of our method with those of the experts’ judgments. Specifically, we adopted Spearman’s rank correlation coefficient [41] (abbreviated as SRCC) to evaluate the similarity of the ranking results. The values of Spearman’s rank correlation coefficient range from −1 to 1. A value of 1 indicates that the two objects have the largest positive correlation; while a value of −1 means that the two objects have the largest negative correlation. In Figure 4, the value of the SRCC is 0.778 in the review ranking of Les Suites Taipei, which means a highly positive correlation between the results of the experts and our method. Figure 5 and Figure 6 show that the values of the SRCC are 0.604 and 0.634 in the review ranking of the San Want Residences and Dandy Hotel-Tianmu, respectively, which indicates a moderately positive correlation between the results of experts and our method. In sum, the experimental results show that the evaluation of our method is closely correlated to that of the experts.

To evaluate the accuracy of our method in a fair way, we repeated the same procedure for the experiments at least five times, and each round was performed by randomly selecting ten reviews of each of the three hotels. When finishing the task of each hotel, we could obtain Spearman’s rank correlation coefficient, the average ranking difference, and its variance. Then, we computed the average of five Spearman’s rank correlation coefficients, the average ranking difference, and its variance for each hotel. Figure 7 presents that the average differences for the three hotels ranged from 1.7 to 2.3, while their variances ranged from 1.6 to 3.3. Further, the average values of the correlation coefficients in the three hotels are around 0.6, that is, in the three hotels, there is a moderately positive correlation between the results using our method and those of the experts. Therefore, it would seem that our method is effective and approximate to that of the experts in evaluating the trustworthiness degree of reviews.

The above experimental results show that there is a high correlation between the proposed method and an expert in ranking the hotels; the former is from an evaluation of the comments, while the latter is from the viewpoints of experts. It is clear that the proposed method can act as a decision support system to help customers choose a valued hotel, restaurant, or entertainment venue. Expert comments for hotels are scarce and hard to obtain; trustworthy comments are a fast and good way for a tourist to make a good choice if they have difficulty choosing their desired hotel. The proposed method involves crawling a travel website, analyzing the trustworthiness of each comment, and ranking them according to their degree of trustworthiness. The method can subsequently filter biased and collusive comments and lead to a wonderful trip.

4.2. Limitations of the Research

We investigated the comments on a famous traveling website. The proposed trustworthiness equation is based on the unveiled information and comments of reviewers and social relationships crawled from the website. The disclosed information about the trails of the comments, the characteristics of the commenters, and linguistic considerations, such as the compatibility of the language, the judgment of sentiments, etc., significantly influenced the precision of the formulas to evaluate the trustworthiness degree of comments.

Because of the confidentiality and protection policy for the reviewers of each traveling website, the limitation of the research is the degree of data disclosed to the public. Furthermore, in cases where the unveiled data are de-identified or even when comments are suppressed into statistical data, the proposed method cannot work well. However, website owners have the ability to access the full data. The proposed evaluation criterion can be adopted by the tourist website to rank and prompt comments according to their reliability to avoid spoiled impressions and wasted time. Spam comments can be marked or deleted to help tourists choose a hotel, restaurant, or entertainment venue based on unbiased comments.

5. Conclusions

In this paper, we propose a trustworthiness evaluation method of comments based on the trails of comments and the social networks among them. The authors of this paper investigated a lot of not fully reliable review data for hotels from the online travel community. For such intangible products, customers demand more opinions and trustworthy reviews. We considered three concepts, i.e., the sentiment similarity of reviewers in the social network, reviewer behaviors, and review features, to construct an assessment model for evaluating the trustworthiness of reviews in the online travel community. We conducted several experiments with real-world data. The experimental results demonstrate that our method is effective in ranking reviews. It is worth noting that perceptions of bias occur if a certain review contains extreme sentiments. In future work, we will explore the decisive factors that are used to adjust the sheer weight of evaluation. From our research, we can learn that a reasonable mechanism to evaluate the trustworthiness of comments can help users make decisions about choosing satisfactory commodities and make it easier to detect biased, malicious, or even collusive comments. In addition, this paper also advocates for applying the proposed concepts to the evaluation of comments generated by customers on other topic-specific websites. These might include evaluations of responses from shopping websites, like eBay and TaoBao, for example.