A Natural Language Processing Model for Predicting Five-Star Ratings of Video Games on Short-Text Reviews ^†

Abstract

The gaming industry is one of the most important and innovative subfields in the field of technology, which boasts a staggering USD 200 billion in annual revenue and stands as a behemoth. It has an immense effect on popular culture, social networking, and the entertainment industry. Continuous advances in technology are the primary factor fueling the industry’s expansion, and these innovations are also revolutionizing the design of games and improving the overall gaming experience for players. The growing number of people who have access to the internet, the widespread use of smartphones, and the introduction of high-bandwidth networks such as 5G have all contributed to an increase in the demand for gaming around the world. It is essential to perform consumer feedback analysis if one wants to appreciate market requirements, evaluate game performance, and realize the effect that games have on players. On the other hand, short-text reviews frequently lack grammatical syntax, which makes it difficult for standard natural language processing (NLP) models to effectively capture underlying values and, as a result, compromises the accuracy of these models. This research focuses on determining which natural language processing model is the most accurate at forecasting five-star ratings of video games based on brief reviews. We make use of natural language processing (NLP) to avoid the constraints that are imposed on us by the linguistic structure of short-text reviews. The findings of our research have led to several important contributions, one of which is the creation of an innovative model for reviewing and grading short writings. The accuracy is improved by employing different machine learning models, which enables game creators and other industry stakeholders to identify patterns about the behavior and preferences of the users.

1. Introduction

The plethora of user-generated content on online platforms has experienced significant exponential growth in recent years. This growth has resulted in the accumulation of a substantial amount of valuable information, such as reviews and ratings pertaining to a wide range of products and services [1,2,3]. The gaming industry is a highly lucrative sector with a market value in the billions, demonstrating its sustained growth and success. To maintain a competitive edge, it is imperative for game developers, publishers, and marketers to possess a comprehensive understanding of gamers’ preferences and sentiments. There are some confounding factors that can impact the review–rating relationship of video games such as genre, platform, audience type, release date, player expectation, etc. Customer reviews serve as a valuable resource for gathering information regarding market demands, assessing the performance of games, and evaluating their impact on users. Nevertheless, the composition of these reviews typically consists of brief texts, posing a distinctive obstacle for conventional natural language processing (NLP) models. The reason for this is that short texts frequently lack grammatical syntax, which can pose challenges when attempting to accurately extract meaningful insights [4,5,6].

This study presents a novel methodology for forecasting five-star ratings of video games by utilizing various NLP models, including naive Bayes, SVM, logistic regression, BERT, TextCNN, Word2Vec, and Doc2Vec. By harnessing the progress made in deep learning and machine learning methodologies, the aim is to create an accurate and effective model capable of automatically assigning a five-star rating to a video game by analyzing the accompanying textual review. In a previous study, ref. [4] introduced a framework for predicting review ratings using deep learning. They emphasized the capability of deep learning models to effectively capture intricate patterns within textual data. Refs. [5,6,7] employed machine learning methods to forecast business ratings using Yelp reviews. Their research demonstrated the efficacy of these models across various domains.

Our work aims to conduct an analysis of various machine learning (ML) algorithms to predict the five-star rating of video games by utilizing short-text game reviews and measuring the efficacy of these models.

According to Kibriay et al. [8], the multinomial naive Bayes classifier has gained significant popularity in the field of text categorization because of its simplicity and efficiency. The empirical results presented in this study compare the results of the standard multinomial naive Bayes classifier with the TWCNB approach, which was introduced in a recent study [9]. The results indicate that specific modifications in TWCNB are not necessary to achieve optimal performance on certain datasets. The experiments conducted indicate that the accuracy of the multinomial naive Bayes classifier is significantly influenced by TF-IDF conversion and document length normalization. The significance of preprocessing steps in enhancing the performance of the classifier is emphasized by these results. The present discovery presents a challenge to the commonly held belief that naive Bayes classifiers consistently exhibit superior performance compared to SVMs in text categorization tasks. Therefore, it is essential to prioritize the exploration of alternative classification algorithms to attain the highest level of accuracy [9]. To achieve better results for the multinomial naive Bayes classifier, researchers have investigated the use of locally weighted learning techniques as described in the literature.

The literature review by Liu Zefang et al. [10] examines the prediction of restaurant ratings using the Yelp open dataset. It analyzes the distribution of data, the process of creating a training dataset that is balanced, and the utilization of different ML models and transformer-based models.

The literature examines two distinct vectorization techniques, although it does not provide specific details about the vectorizers used. The purpose of these techniques is to extract the pertinent characteristics from Yelp reviews, thereby enabling the models to generate precise predictions. The selection of a vectorizer has a direct impact on the quality of the features that are extracted from the text. This, in turn, influences the overall performance of the models.

The implementation includes four models: BERT, DistilBERT, RoBERTa, and XLNet [11]. The models utilize pre-trained language models to effectively capture contextual information and semantic meaning in Yelp reviews. The evaluation of the model is performed by utilizing accuracy as the primary metric. According to the abstract, the XLNet model demonstrates a 70% accuracy rate in five-star classification, outperforming logistic regression, which achieved a score of 64% [12].

2. Methodology

In this section, the methodology employed for this research is presented, with the aim of identifying the most effective natural language processing (NLP) model for the video game-specific domain. As part of our research objectives, we strive to develop a robust approach for representing short texts. This approach is based on insights gained from previous research, which has explored various methods such as Unigram, Bigram, and their combinations. Figure 1 shows the method for the selection of model using natural language processing (NLP).

The predictive model utilized in this research focuses on predicting ratings for individual reviews. In addition to the review content, the model can also incorporate sentiment analysis, metadata, or topic data as additional features during the prediction process. These features provide valuable insights and contribute to the overall prediction of a game’s rating. The final prediction for a game is obtained by averaging the predicted review ratings.

2.1. Data Collection

The Amazon video game review dataset is an extensive compilation of user reviews and ratings for video games that are accessible on the Amazon platform. The dataset holds significant value for researchers, developers, and data enthusiasts who are interested in analyzing and comprehending user sentiments, preferences, and trends pertaining to video games. The idea is to use the above-mentioned dataset and provide an in-depth analysis of the essential characteristics, practical uses, and important factors to consider when working with it.

2.1.1. Applications of the Dataset

• Sentiment Analysis: Researchers can employ natural language processing techniques to perform sentiment analysis on the review texts. This analysis can help identify positive and negative sentiments towards different video games, enabling companies to gauge customer satisfaction and make improvements accordingly.
• Recommender Systems: The dataset can be utilized to build and train recommendation algorithms that suggest video games based on user preferences. By understanding the preferences and feedback of a large user base, personalized recommendations can be generated to enhance the user experience on platforms like Amazon.
• Market Research: The dataset allows for market research to highlight popular trends, preferences, and patterns in the video game industry. Companies can utilize this information to make informed decisions about game development, marketing strategies, and product positioning.

2.1.2. Considerations and Challenges

• Data Bias: It is essential to acknowledge and address potential biases in the dataset, such as selection bias, review manipulation, or review bombing. Researchers should carefully consider these factors and apply appropriate data preprocessing techniques to minimize bias and ensure accurate analysis.
• Data Volume: While the dataset contains a substantial number of reviews, it may not capture the entire user sentiment landscape for video games on Amazon. Researchers should be cautious while generalizing findings from this dataset and consider its limitations.
• Data Preprocessing: The dataset may contain noise, spam, or uninformative reviews. Preprocessing steps like text cleaning, removing duplicates, and outlier detection should be employed to ensure the quality and integrity of the dataset.

2.1.3. Key Features of the Dataset

Table 1. Amazon Game Review dataset description.

Column	Description
overall	Rating of the review text (int)
verified	The reviewer is verified user (bool)
reviewTime	Formatted time of the review (string)
reviewerID	ID of the reviewer (string)
asin	Unique ID of the review (string)
reviewText	Review text submitted by the user (string)
Summary	Small annotated summary of the review test (string)
unixReviewTime	Unix timestamp of the review submission (string)

presents the features of the Amazon video game review dataset, which were taken into consideration during the model training and evaluation process.

2.2. Text Preprocessing

Text preprocessing focuses on transforming raw data into a format suitable for analysis using text cleaning, tokenization, stop word removal, lemmatization, etc. The Amazon video game review dataset exhibits a bias towards 5-star reviews, which account for 1.4 million out of 2.7 million reviews. To normalize the dataset, each rating was associated with 260,000 reviews by calculating the average number of reviews excluding 5-star ratings. Figure 2 shows the raw and normalized data set.

The language structure of short-text reviews, like those frequently seen in the game industry, can have limitations. The following potential effects on the analysis and comprehension of these reviews may result from these limitations such as ambiguity and lack of content, spelling mistakes and grammar errors, lack of structure, etc. There are different techniques which have been employed to address these challenges. Tokenization techniques were employed to efficiently process textual data in the context of machine learning and neural network models. They helped in breaking down textual information into meaningful units, facilitating subsequent analysis and feature extraction. By employing tokenization, the data were effectively prepared for further exploration. Stopword removal was performed to eliminate common words, known as stopwords, which do not carry significant meaning and can lead to noise in the analysis. Examples of stopwords include “a”, “an”, “the”, and other frequently occurring words in the English language. By removing these stopwords, the data became cleaner and more focused, allowing for the subsequent analysis to be concentrated on the essential content of the text. Lemmatization or stemming was used to reduce words to their base or root forms, thereby avoiding redundancy and improving analysis accuracy. Lemmatization transformed words to their dictionary or base form, while stemming reduced words to their stem form by removing prefixes or suffixes. By normalizing the words in this manner, variations of the same term were unified, leading to a more consistent representation of the text. These methods efficiently processed the textual data, breaking it down into meaningful units, and preparing it for further exploration and feature ex-traction.

2.3. Feature Engineering

TF-IDF (term frequency–inverse document frequency) was utilized as a statistical measure to assess the relevance of terms within a collection of documents. It aided in identifying important terms and filtering out common or irrelevant ones by assigning weights based on their frequency and rarity. TF-IDF scores provided a numerical representation of term importance, enabling the prediction of review ratings for video games within the specific domain. Additionally, word embeddings played a crucial role in enhancing the effectiveness of neural network models for processing textual data. These embeddings represented words or phrases as dense vectors, capturing their semantic and syntactic relationships. By incorporating word embeddings into the research, the neural network model’s ability to understand the meaning and context of input text was improved. Word embedding models, including Word2Vec and Doc2Vec, were explored to determine the most suitable approach. The evaluation process involved utilizing domain-specific metrics and benchmarks to assess the quality and usefulness of the word embeddings within the video game domain. The combined use of tokenization, TF-IDF, and word embeddings enabled us to derive valuable insights and enhance the accuracy of predicting review ratings for video games.

2.4. Models

For the classification of short-text reviews, several models were employed to predict review ratings based on the content of the reviews. The naive Bayes classifier, specifically the multinomial variant, was utilized as a straightforward and effective approach for assigning predefined categories to text documents. The classifier worked on the concept of conditional probability, provided that the features were conditionally independent. The multinomial variant represented feature vectors as counts of word occurrences within the documents. The algorithm constructed probability models for each class by estimating conditional probabilities using techniques like maximum likelihood estimation or smoothing methods such as Laplace smoothing.

For multi-class classification, the linear support vector machine (SVM) algorithm was applied. The SVM sought to identify the best hyperplane with the greatest margin between data points from various classifications. The original textual data were converted into numerical representations using feature extraction techniques including bag-of-words, TF-IDF, and word embeddings (like Word2Vec). To assess the effectiveness of the SVM model, the dataset was split into training and testing sets and tested against parameters like accuracy.

The logistic regression model was used to estimate the probability of each review rating class. It employed a logistic function to map the linear combination of input features and their corresponding weights to a probability score between 0 and 1. The model was trained on a labeled dataset, and optimization algorithms like gradient descent were used to adjust the weights and minimize the error between predicted and actual ratings. Relevant features, including textual sentiment analysis scores, reviewer demographics, and game attributes, were considered in the logistic regression model. Preprocessing steps such as tokenization, stopword removal, and dataset splitting were performed, followed by evaluation.

Moreover, advanced models such as BERT and Text-CNN were explored. BERT, a pre-trained language model based on the Transformer architecture, captured complex contextual relationships, while Text-CNN employed convolutional filters to extract local features from text. Both models showed effectiveness in predicting review ratings on the game dataset, with BERT excelling at capturing contextual relationships and Text-CNN performing well on shorter texts.

Finally, Word2Vec and Doc2Vec models were utilized to learn distributed representations of words and documents, respectively. Word2Vec, a widely used NLP technique developed by Tomas Mikolov et al. (2013) at Google, was utilized to learn distributed representations (word embeddings) of words in a large corpus of text, i.e., the Amazon video game review dataset. The Word2Vec model, operating on the principle that words appearing in similar contexts have similar meanings, was trained using two architectures: continuous bag-of-words (CBOW) and skip-gram. By leveraging the learned word embeddings, the reviews were represented as vectors by averaging the embeddings of the words within each review. These review vectors were then fed into a machine learning algorithm, such as a regression model, to predict the corresponding review ratings. Evaluation of the Word2Vec model’s performance involved metrics like mean squared error (MSE) or accuracy, employing techniques like dataset splitting and cross-validation. In addition, the study explored the Doc2Vec model, an extension of Word2Vec introduced by Le and Mikolov in 2014. Doc2Vec incorporates document-level context by representing documents as fixed-length vectors, known as “document embeddings.” By training the Doc2Vec model on the preprocessed Amazon video game review dataset and adjusting hyperparameters, including vector size, window size, and training epochs, document embeddings were learned. These embeddings captured the semantic meaning of the corresponding reviews and facilitated various downstream tasks, including sentiment analysis and classification. To predict review ratings, ML algorithms such as SVMs, random forests, or gradient boosting were employed, and the performance was evaluated using metrics such as accuracy, precision, recall, and F1 score.

Table 2. Different models for predicting the rating of review texts.

Model	Accuracy
Naïve Bayes	48%
SVM	50%
Logistic Regression	70%
BERT	23%
TextCNN	56%
Word2Vec	73%
Doc2Vec	79%

shows the accuracy of different models for predicting the rating of review texts.

3. Results and Discussion

This research explored the application of NLP models in evaluating short-text reviews to predict five-star video game ratings. A comprehensive analysis was conducted on various models, including logistic regression, naive Bayes, support vector machine, and neural networks, using Amazon video game reviews as the dataset. Among these models, the neural network approaches, particularly the Word2Vec and Doc2Vec algorithms, demonstrated the best performance in predicting five-star evaluations [13], as highlighted in the report. The Word2Vec algorithm, renowned for learning vector representations of words that effectively capture their meaning and contextual information, proved valuable in vectorizing each review and generating rating forecasts. However, the study also revealed that by employing a larger dataset and pre-training the model on generic text, the Word2Vec technique exhibited enhanced accuracy, achieving 73% accuracy. This approach enabled the model to acquire more generalized principles for predicting five-star video game ratings through the incorporation of a broader range of data and pre-training techniques. Notably, the research placed particular emphasis on the effectiveness of the Doc2Vec model, which outperformed all other implemented models with an accuracy of 79%. By incorporating document-level context alongside word-level information, the Doc2Vec algorithm successfully captured the intricate nuances present in the reviews, enabling accurate prediction of their corresponding five-star ratings. The utilization of document embeddings proved to be a critical factor in achieving superior performance within the context of the research.

Furthermore, the research findings highlighted that BERT performed notably below expectations, exhibiting the lowest accuracy of only 23%, which contrasts with its typical good performance such as in the study [14], wherein BERT outperformed the traditional NLP approach and proved to be better in terms of accuracy and implementation. A similar poor performance of BERT was observed in research [15] based on clinical text classification, where possible reasons for this outcome were the increased complexity due to BERT’s WordPiece tokenizer and the inclusion of irrelevant keywords instead of the critical ones resulting from the model’s learning process. SVMs also exhibited suboptimal performance, achieving a mere 50% accuracy in predicting five-star ratings. Despite being widely regarded for their efficacy in certain tasks, such as product review classification [16], sentiment analysis [17,18], and text classification for multi-lingual datasets [19] where they have outperformed models like logistic regression and naïve bayes, the SVM models employed in this study struggled to adequately capture the complexities and nuances present in the short-text reviews. As a result, their accuracy was comparatively lower. Logistic regression showed better results with 70% accuracy, while TextCNN fell short with a less satisfactory accuracy of 56%, which is significantly low if compared to previous works [20], indicating a subpar predictive capability in this context. However, the clear standout was the Doc2Vec model, providing the best performance among all models with an impressive accuracy of 79% for predicting five-star game reviews. This comparison underscores the significance of using Doc2Vec for capturing semantic representations and gaining valuable insights into user sentiments within online review systems. The combined utilization of Word2Vec and Doc2Vec algorithms showcased their respective strengths and contributed to the enhanced accuracy in predicting five-star video game ratings [21]. NLP models that predict five-star video game ratings from short-text reviews have showed promise, although they have limits and biases. The models must be generalizable to varied games and player demographics and address reviewer bias. To make gaming industry models useful and practical, interpretability and regression vs. classification should be carefully studied. NLP models can help developers and publishers improve player experiences and game quality by tackling these concerns. The potential ethical implications of user feedback or any kind of privacy or consent-related concerns are addressed by using anonymized data, monitoring the model for bias, gaining consent from the user, or using a transparent model.

4. Conclusions

The study conducted a thorough performance comparison of several NLP models using a dataset comprising Amazon video game reviews. The research revealed that neural network models, particularly the Word2Vec and Doc2Vec algorithms, demonstrated superior accuracy in predicting five-star ratings. Moreover, this research paper sheds light on the challenges associated with the implementation and deployment of these models. Ethical considerations, data biases, interpretability, and computational requirements are among the critical factors that researchers and practitioners must address to ensure responsible and effective utilization of ML technology. By analyzing and comprehending the results from the short text classification of game reviews, game creators and industry professionals can leverage these insights to improve existing technologies. The ability to identify trending genres, such as intense real-world experiences becoming popular, empowers developers to cater to user demands effectively. Moreover, the models’ insights enable the identification of issues and the exploration of new concepts, thereby fostering continuous improvements in game development and enhancing the overall user experience. In addition to the valuable insights gained, the research suggests potential avenues for future endeavors. The paper could investigate the utilization of various pre-training techniques, such as masked language modeling (MLM) and self-supervised learning (SSL), to further enhance the accuracy and performance of the models. Furthermore, the study may explore the applications of these models in diverse domains, including but not limited to recommending video games to users and identifying emerging trends in video game reviews. In conclusion, this research not only highlights the effectiveness of neural network models for predicting five-star ratings in video game reviews but also emphasizes the broader implications and opportunities for improvement within the gaming industry. With responsible and insightful utilization of ML technology, game developers can continually enhance their creations, providing an exceptional gaming experience to users and staying ahead in the ever-evolving world of video games.

5. Future Work

In this research, the performance of several text classification models, including BERT, TextCNN, and SVM, did not yield the expected results achieved in previous research works on other datasets. BERT, which has revolutionized NLP with its ‘state of the art’ performance, performed poorly with a meagre accuracy of 23%. TextCNN, known for its computational efficiency and competitive performance in various text classification tasks, achieved an accuracy of 56%. Similarly, SVM provided an accuracy of only 50%. Previous research has demonstrated that BERT excels at capturing complex contextual relationships, making it highly effective in various NLP tasks. TextCNN, on the other hand, has shown competitive performance and computational efficiency, making it suitable for shorter texts. However, in the context of the Amazon game review dataset, these models did not deliver the desired accuracy. For future research, it is crucial to explore ways to improve the accuracy of these models on the Amazon game review dataset. One approach is to consider variants of BERT and models infused with BERT to leverage its contextual-understanding capabilities. By fine-tuning and optimizing these models specifically for the unique characteristics of the Amazon game review dataset, it is anticipated that better performance can be achieved. Such efforts will contribute to advancing the field of text classification and enable more accurate analysis of game reviews. In summary, despite the underperformance of BERT, TextCNN, and SVM on the Amazon game review dataset compared to previous research works on other datasets, future research will focus on implementing strategies to enhance their accuracy.