Exploring Mobile Application User Experience Through Topic Modeling

Abstract

Exploring user satisfaction and experience is the first step of software product improvement and business sustainability. The primary goal of this research is to investigate how companies can use topic modeling to understand mobile application user experience and offer methodological steps for identifying factors shaping it. SalesForce was selected for this case study as it was the most widely used CRM application in 2023. This study aims to uncover factors influencing positive and negative user experience, to compare and systematize them, and to indicate the business implications of topic modeling findings. For this study, authors collected 9081 online reviews of the SalesForce application from Google Play Store. The corpus is divided into three subsets based on the associated star ratings, where four and five stars indicate positive sentiment, three mixed sentiment, and two and one negative. Each subset is analyzed using the Latent Dirichlet Allocation algorithm, hyper-parameters were fine-tuned, and the experimental models were evaluated with coherence measures to determine the model with the optimal number of topics. The results indicate that the driving factors of positive experience are seamless functionality and reliability, design flaws and performance issues shape negative experiences, and mixed experiences arise from inconsistencies in usability and authentication challenges.

1. Introduction

The widespread use of digital technologies across all business sectors, such as mobile, Internet of Things, artificial intelligence, and social platforms, is driven by the Fourth Industrial Revolution. Company adoption of these technologies is known as digitalization, which enhances business operations, sales, productivity, innovative value creation, and customer relationships. Product and service lifecycles have shortened while customer expectations have increased, requiring companies to anticipate future needs as dissatisfied customers can easily switch providers [1]. Digitalization is closely related to digital business transformation, the process through which companies leverage various digital technologies to achieve better business results and market positions [1,2]. This transformation involves changes in business due to the advantages of modern digital technologies [3]. The extent of the transformation varies by industry and digitalization level, affecting business models, customer experiences, processes, decision making, skills, talents, organizational culture, and value creation systems [2].

Today, approximately 6.4 billion users of mobile devices benefit from the portability and accessibility of these devices, allowing their use anytime and anywhere [4]. This vast user base has prompted modern companies to shift their business strategies towards the mobile market and development of mobile business applications. Digital transformation, driven by the adoption of mobile technology, has led to a significant number of business applications available in the Google Play and Apple application stores. In addition to enabling users to find, purchase, and install mobile applications, these stores serve as the reviewing platform for applications based on comments and star ratings.

Evaluation of the user experience of mobile business applications is a key factor for sustainability in a competitive market, as it directly relies on positive experience, user satisfaction, and loyalty, indicating the importance and topicality of the research topic in this paper. Satisfied users remain loyal, promote products or services, and influence others’ purchasing decisions. Monitoring user satisfaction helps companies identify customer needs and areas of improvement. Addressing user needs helps build a loyal customer base and enhances company resilience and adaptability, contributing to long-term sustainability [5]. Online reviews have emerged as a modern form of digital Word-of-Mouth (WoM) marketing, with the potential to attract new users through the satisfaction of existing ones. They serve as a primary channel for mobile application users to articulate their needs, frustrations, and challenges, necessitating their continuous evaluation to enhance the products or services. As such, they contain valuable information about the user experience, which can serve as essential input for companies to improve their applications.

User experience with mobile business applications through online reviews is a trending research topic across various business domains, such as mobile banking [6,7,8,9], healthcare [10,11,12], and education [4,13], underscoring the importance of the chosen research topic. Across these domains, machine learning and natural language processing are widely adopted approaches in exploring user experience with mobile applications. The dominant technique for identifying dimensions affecting user satisfaction and experience is topic modeling [4,7,8,9,10,11,13,14], which uncovers prevailing topics recognized as (dis)satisfaction cues. Authors also use sentiment analysis to detect the polarity of expressions—whether the author writes positively or negatively about their user experience with the mobile application [6,7,12] or neural networks for evaluating the relationships between identified dimensions and user experience [14].

Business-wise, customer management has been supported through customer relationship management (CRM) software for over two decades, helping all types of companies achieve success. With the rise in the mobile market, companies have replaced traditional desktop CRM software with mobile CRM applications, and user satisfaction monitoring has shifted towards user experience monitoring with mobile applications. Although highly important from a business perspective and sustainability, the literature review indicates a single paper analyzing user experience with CRM mobile applications [14]. The gap identified between the critical role of mobile CRM applications supporting business operations and the relatively unexplored nature of user satisfaction with these business applications in the academic literature motivated this study presented in this paper primarily. The research presented in this paper aims to investigate the factors shaping CRM mobile application user experience and satisfaction by utilizing topic modeling on the SalesForce online reviews collected from Google Play Store. User satisfaction exploration and its integral part—the user experience—is also the first step for improving any software product and the key factor for business sustainability. SalesForce was selected for the case study as it is the most widely used CRM application, with a 21.7% market share in 2023 [15]. To the best of our knowledge, there is no similar research exploring user experience with the SalesForce CRM mobile application, although it is the most common application on the market of these products [15]. It provides companies access to various functionalities for the effective management of interactions with current and potential customers to establish long-term relationships and improve sales.

The literature review highlights several research gaps that have informed the proposed research questions and the design of empirical research and methodology. Studies on mobile application user experiences with topic modeling often suffer from insufficiently documented methodologies [8,12,14] or a limited understanding of how specific techniques impact data and the interpretability of results [11,13]. This issue is particularly evident in pre-processing, a critical step influencing topic quality and interpretability, as emphasized by Camacho-Collados and Camacho-Collados [16], Fakhitah and Wan Mohd Nazmee [17], Hickman et al. [18], and Laureate et al. [19]. Variations in pre-processing practices further reveal a gap in understanding the role of data treatment in shaping topic structures and preserving semantic meaning [20], reinforcing the need for methodological advancements, such as those proposed in our study, which emphasize informed pre-processing decisions to improve the quality of data used as inputs for topic modeling. Few studies addressed multilingual data collection [9,11] despite its critical role in capturing diverse user perspectives, reflecting a wide range of linguistic and cultural backgrounds, and enhancing data size, contributing to the improved generalization of insights.

Furthermore, there is a noticeable scarcity of research exploring the business implications of obtained results [12,21], particularly in their practical application to support business sustainability. This lack of focus on the intersection between research findings and their real-world business relevance was recognized as a critical gap in the existing literature.

In the context of the research topic and identified research gaps, the authors aimed to answer the following research questions through the results of empirical research:

• RQ1: How can companies use topic modeling to understand mobile application user experience?
• RQ2: What are the main factors impacting the SalesForce mobile application user experience?
• RQ3: How do the factors influencing the SalesForce mobile application user experience compare to those discussed in the literature on user satisfaction with CRM applications?
• RQ4: What are the business implications of topic modeling findings?

To address the research questions, the authors developed a methodology for extracting generalizable insights from topic modeling analysis, comprising six key tasks. Data collection involved scraping 9308 SalesForce reviews from Google Play Store, which formed the corpus. Data cleaning and pre-processing enhanced the quality of the corpus, transforming it into a format suitable for machine learning algorithms. Pre-processed texts were then vectorized to obtain their numerical representation. Corpus exploration followed, using n-gram analysis, statistical techniques, and word clouds to gain insights and refine further analysis based on associated star ratings indicating expressed sentiments. Reviews were categorized into three subsets: negative (one or two stars), mixed sentiment (three stars), and positive (four or five stars). The selection of the appropriate topic modeling approach refers to the task of determining the suitable topic modeling algorithm. The authors utilized Latent Dirichlet Allocation (LDA). Experimentation with the number of topics and hyperparameters was conducted on each subset to identify the optimal model, which was then interpreted from a business perspective.

The paper is structured as follows: the Literature Review Section reviews the existing literature on user experience as a tool for monitoring business sustainability. Next, the Materials and Methods Section outlines the research framework, detailing the methodology, data acquisition process, pre-processing techniques, and analytical approach. The Results Section introduces the generated topic models and offers an analysis of user experience and satisfaction with the SalesForce mobile application. The Discussion Section addresses each of the research questions individually. Finally, the Conclusion provides a summary of this study’s key findings, contributions, and implications, reflecting on limitations as well.

2. Literature Review

The literature review is organized into two subsections: the first situates our research within the existing body of the relevant literature, while the second examines the methodologies employed in topic modeling research on mobile application user experience. This approach aims to guide the design of our empirical research and methodology, highlighting areas requiring methodological refinement and increased rigor.

2.1. Research on Mobile Application User Experience

The Fourth Industrial Revolution and modern digital technologies have empowered customers, allowing them to switch providers of goods and services within a click. As a result, companies prioritize understanding and predicting customer needs to foster loyalty. The consistent choice of a particular brand or company is referred to as loyalty. It plays a crucial role in driving revenue, enhancing WoM marketing, and securing a competitive edge [22,23], and it is closely tied to satisfaction and user experience. Satisfied customers are more likely to repeat purchases, contributing to stable revenue streams through frequent and larger transactions. They also play a key role in WoM marketing by recommending products or services to others, attracting new customers, and increasing brand visibility and recognition. Moreover, loyal customers provide valuable feedback, helping companies refine their products or services [22,23]. User experience encompasses all customer interactions with a company’s products or services. It is a comprehensive concept that includes sensory, emotional, cognitive, social, and physical responses to all interactions with the company. A positive user experience enhances customer satisfaction by creating emotional value, ultimately fostering deeper connections with the brand [24].

Exploration of user experience with mobile applications is a trending research topic due to the rapidly growing market, evidenced by the number of available applications on Google Play or Apple application stores. A consumer’s decision to buy and use business mobile applications is rooted in various features perceived as relevant, such as design, usability, or attractiveness. With continuous user experience monitoring, companies can track sources of dissatisfaction and direct improvements of products and services. Poor user experience will lead to decreased revenue, customer dissatisfaction, negative ratings and comments, negative WoM, and a negative impact on the brand [25,26,27].

Table 1. Summary of research papers in domain of mobile application user experience.

Research Domain	Study	Publication Year	Applied Techniques	Factors Driving User Experience
Mobile banking	[6]	2023	machine learning, sentiment analysis, thematic analysis (TA)	ease of use|helpful|reliable|user friendly|good aesthetics|convenience|secured
	[9]	2023	machine learning, topic modeling (LDA)	usefulness|time-saving|convenience|processing fee|login|security
	[8]	2021	topic modeling (LDA)	ease of use|security|convenience|customer support
	[7]	2020	sentiment analysis, topic modeling (LDA)	ease|simplicity|helpfulness|account registration and login|performance issues|network connectivity
Healthcare	[11]	2023	topic modeling, sentiment analysis	functionality|usability|reliability|compatibility|user interface|price|sleep improvement
	[21]	2023	topic modeling, sentiment analysis	time and money|convenience|responsiveness|availability|seamless payment system|transparent refund policy|video consultation|doctor availability through online booking
	[12]	2022	sentiment analysis, topic modeling (LDA)	functionalities|timeliness of responses|reasonableness of fees|system stability|smoothness of use|accuracy and comprehensiveness of information|service quality|content quality|technical quality|management quality
	[10]	2021	topic modeling, topical n-gram.	usability|functionality|helpful|easy to use|technical quality and shortcomings|payments
Education	[4]	2023	topic modeling (LDA)	gamification|usefulness|freeness|help feature
	[13]	2021	topic modeling (LDA), sentiment analysis.	financial issue|technical-non-functional features|design|video and multimedia
CRM	[14]	2023	topic modeling (LDA), learning vector quantization (LVQ), adaptive neuro fuzzy inference system (ANFIS).	System quality: usability|interface design|reliability|performance|system updates|compatibility|adaptability|system integration|access control|security|user permissions Service quality: user training|support|service improvement Information quality

summarizes research centered on user experience with mobile applications. It presents the business domain, publication year, applied analytical techniques, and the key findings on factors influencing user experience. We focused on the recent literature published within the last five years, covering the period from 2020 to 2024. Authors researched user experience in several business domains: (a) mobile banking [6,7,8,9], (b) healthcare [10,11,12,21], (c) education [4,13], and (d) CRM [14]. Prevailing analytical approaches are topic modeling for uncovering hidden topics from texts and sentiment analysis for exploring the sentiment polarity of reviews. The literature indicates that the factors influencing the user experience differ depending on the business domain to which the application belongs, such as users of educational mobile applications, in particular, the Duolingo learning application, who value video content the most [4], while users of banking mobile applications place the safety factor at the top of the ladder [6,7,9]. Findings like these point to the need to conduct individual case studies that engage in focused research on user experiences and the identification of the unique factors that shape them.

2.1.1. Mobile Banking Domain

The results of exploratory research aimed at understanding user opinions and attitudes towards mobile banking applications are consistent. Authors Mahmood et al. [6], Permana et al. [7], Oh and Kim [8], and Çallı [9] identified that the main factors driving the positive experience of mobile application users, as presented in Table 1, are ease of use of the application, usefulness, convenience, and time savings that these applications provide to users. Security is identified as the factor present in positive reviews within the research conducted by Mahmood et al. [6] and Permana et al. [7], while Çallı [9] identified that technical and security issues are highly correlated with low ratings and negative reviews of mobile banking applications in Turkey. Among factors influencing negative user experience, Mahmood et al. [6] and Permana et al. [7] identified account registration and login problems, updates to new versions, OTP code delivery, performance issues, and network connectivity.

2.1.2. Healthcare Domain

Empirical research focusing on examining user experience and satisfaction through online reviews has been conducted in the domain of various types of healthcare mobile applications, such as mobile nutrition applications [10], sleep monitoring applications [11], or general mobile medical applications [12] or platforms [21]. The findings are consistent regarding factors influencing negative user experience. As illustrated in Table 1, Zečević et al. [10] and Zhai et al. [12] indicate that technical quality and shortcomings are the key sources of user dissatisfaction. On the other hand, functionalities, responsiveness, service and content quality, cost, convenience, and availability are driving factors of positive user experience identified across studies [10,12,21].

Vargas-Calderón et al. [28] underscored the importance of investigating data originating from non-English speaking countries, as they can indicate distinctions in characteristics, cultural differences, or manifest variations or the relevance of sociological aspects between countries of origin. Nuo et al. [11]’s findings are in line with Vargas-Calderón et al. [28]. Authors Nuo et al. [11] identified geo-location variations within factors influencing user satisfaction with sleep applications. They conducted a comparative analysis of online reviews written by USA and Chinese users of 10 different sleep monitoring applications. They considered various factors, such as functionality, usability, reliability, compatibility, user interface, price, and sleep improvement effects, and identified that the most influential factors of satisfaction vary depending on the country of origin. Factors that USA users consider relevant are related to the application functionalities, such as sleep tracking options (24.8%) and sound recording (10.7%), while user interface design (8.9%), price (3.3%), and compatibility (2.2%) are less influential. Contrary to these findings, to Chinese users, the price of the application (16.9%), reliability (16.5%), and usability (13.4%) are the most valued factors. Although price is relevant in both groups of users, the data indicate that Chinese users put more emphasis on price than USA users.

2.1.3. Education Domain

Mobile learning applications are highly popular and widely used, predominantly for language learning and gaining or improving technical skills. Author Polatgil [4] identified that users of Duolingo, the language-learning application, most value video content as the type of content through which they learn and retain information most effectively. Okuboyejo [13] leveraged factors influencing English-speaking user satisfaction with the most prominent learning applications for up-skilling, such as Udemy, Coursera, edX, LinkedIn Learning, Lynda, Skillshare, Khan Academy, and Pluralsight. By examining negative reviews, the author identified desirable features that applications should have to increase user satisfaction. These characteristics were categorized into three groups: financial, technical, and design-related [13] (Table 1). They indicate variations between relevant factors depending on the type of learning application.

2.1.4. Customer Relationship Management

A single study focuses on the same research domain, the assessment of user satisfaction, and experience with CRM mobile applications. Authors Nilashi et al. [14] used 5.172 online reviews of 8 CRM mobile applications collected from the Google Play Store platform. They used LDA to identify key topics affecting user satisfaction, which were categorized into three dimensions—information quality, system quality, and service quality, which are presented in more detail in Table 1. The LDA output was subsequently utilized as input in a prediction model of user satisfaction.

Although the results justified the combined approach in terms of the performance of the prediction model, the research has certain limitations in terms of topic modeling applications. According to [29], the Pollyanna principle, a human tendency to remember pleasant moments more accurately than unpleasant ones [30,31,32], is highly pronounced in online reviews and associated with a higher frequency of positive reviews [31,32,33,34,35]. Nilashi et al. [14] do not state which specific applications are included in the research or the information on the number of collected reviews per application. The average amounts to 647 reviews per application (5.172/8). Encompassed applications can be of varying quality in terms of the variety of functionalities they offer, as well as in terms of average ratings. The Pollyanna principle is more pronounced in higher-rated applications where it is expected to have more positive reviews than negative. When a dataset is strongly unbalanced, negative cues and cues of dissatisfaction can easily be omitted, so without additional context related to the Nilashi et al. [14] research, particularly without information on encompassed applications, their overall ratings on the Google Play Store and the amount of data collected per application, it is challenging to evaluate the justification of collectively analyzing reviews of all applications as a homogeneous set in the topic modeling. In addition to this, authors Kirilenko, Stepchenkova, and Dai [36] and Hu, Zhang, Gao, and Bose [37] suggested that the use of the overall corpus for topic modeling analysis can lead to the omission of certain dissatisfaction cues, indicating the need for extraction and isolated analysis of the content with negative sentiment. Therefore, combined analysis, such as the one in Nilashi et al. [14] may omit cues of (dis)satisfaction. In addition, the authors encompassed only English reviews and thereby reduced the diversity of the sample, as user experience can vary significantly depending on cultural and regional factors [28].

Based on the limitations of the research presented in the work of Nilashi et al. [14], the authors identify that it is necessary to include a larger amount of data on individual applications, as well as the reviews written in different languages, to gain deeper and more precise insights into user satisfaction. For improved understanding of the characteristics of positive and negative user experience, it is necessary to partition the dataset according to the sentiment polarity expressed in the reviews. This is the main contribution of our research, compared to the one conducted by authors Nilashi et al. [14]. A broader representation of user perceptions, needs, and attitudes can be achieved by collecting non-restricted data concerning language, as well as more generalizable conclusions. Such insights can subsequently inform localized analyses, enabling a more nuanced understanding of specific cultural contexts when needed, and, therefore, the inclusion of multilingual data is intended to complement localized aspects by expanding the range of perspectives considered in the analysis.

2.2. Topic Modeling in Mobile Application User Experience Research

Studies researching mobile application user experience predominantly explore user satisfaction through the analysis of online reviews using either topic modeling, sentiment analysis, or combining both techniques. Among the papers detailing the selected research approach, authors predominantly use Latent Dirichlet Allocation (LDA) as the approach for topic modeling [4,7,8,9,11,12,13,14].

Topic modeling, as a text mining technique, operates on a word level. It uses vocabulary from the corpus to obtain groups of keywords that form a topic. The quality of input data, i.e., keywords used in topic modeling, is of utmost importance and achieved through text pre-processing. Text pre-processing is conducted to raise the data quality and prepare it for further analysis [38]. The output is usually a corpus comprising documents, where each document is represented as a list of words.

Decisions made during text pre-processing play a crucial role in capturing language content and style, influencing the statistical power of analyses and the validity of insights derived from text mining [18]. Within the topic modeling methodology, we can argue that pre-processing represents a crucial step that strongly influences the quality of obtained topics, as well as their interpretability, as recognized in the computer science literature [16,17,18,19]. However, though critical, text pre-processing is frequently overlooked and inadequately documented in text mining research [19,39]. Authors [19] emphasize the need for more thorough attention and the documentation of pre-processing steps, suggesting that it could bolster confidence in topic models.

Similar to the findings of [19], we identified an absence of uniformity in data pre-processing protocols across studies.

Table 2. Overview of text pre-processing techniques.

Pre-Processing Technique	Number of Studies	Studies
stopword removal	10	[4,6,7,8,9,10,11,12,14,21]
tokenization	8	[4,6,7,8,9,11,12,13]
case folding	6	[4,6,7,9,10,13]
punctuation removal	6	[4,6,7,9,11,21]
stemming	6	[7,9,11,13,14,21]
lemmatization	4	[4,6,11,13]
number removal	3	[10,11,21]
n-gram features	2	[7,9]
length-based filtering	2	[8,9]
special character removal	2	[9,10]
PoS tagging	2	[10,13]
extra whitespace removal	2	[6,21]
frequency-based analysis	1	[8]
negation handling	1	[8]
PoS filtering	1	[11]
null data or record removal	1	[11]
spell check	1	[10]
removal of irrelevant content	1	[12]
duplicate removal	1	[12]
format standardization	1	[12]
abnormal data removal	1	[12]
text cleaning	1	[14]
special character replacement	1	[9]
domain-specific stopword removal	1	[9]

summarizes the pre-processing steps identified in the studies analyzed in the literature review of this paper, as well as the variations in applied pre-processing techniques. We can identify three types of text pre-processing techniques. The first group refers to those aiming at cleaning text from noise, redundancies, or grammar, spelling, and typographical mistakes. The second is focused on the reduction in variations and dimensionality, and the third at feature engineering for topic modeling.

Text cleaning techniques refer to the removal of stopwords, punctuation numbers, special characters, extra whitespaces, and null or duplicate records, as well as the correction of grammar and spelling and special character replacement. Stopword removal is the most prominent pre-processing technique, applied in 10 studies (90.91%). It is an important technique for topic modeling pre-processing as it removes frequent words that do not contribute to the meaning of the text. In addition to stopwords, corpora are characterized by domain stopwords as well. These words are often mentioned in the texts and are specific to the domain, business area, or industry. They do not contribute to the context. The author of only one study removed them based on a custom list mostly containing companies and city names, Çallı [9], for maintaining semantically meaningful words for further analysis. Punctuation removal is another frequently used technique, applied in six studies (54.55%). While each of the text cleaning techniques enhances the quality of keywords retained for topic modeling, Table 2 indicates their sporadic use in user experience topic modeling research.

Effective data preparation is crucial for reducing dimensionality, which characterizes corpora, and simplifying attributes [40]. Techniques directed towards variations and dimensionality reduction refer to case folding, which, in the case of user experience topic modeling research, mostly refers to the lowercasing of all capitalized letters, text normalization (stemming and/or lemmatization), and length-based filtering. Case folding and stemming are used in six studies each (54.55%), while lemmatization is used in four studies (36.37%), out of which two apply stemming as well. Although stemming alone or in combination with lemmatization are commonly adopted approaches to text normalization in user experience topic modeling research, we can argue the validity of this approach. Authors [19] highlight that the choice of stemming or lemmatization often reflects a limited understanding of how data processing affects model behavior and human interpretation. They caution against stemming utilization for topic modeling, noting its potential to introduce lexical ambiguity, senseless terms, and semantic inaccuracies, ultimately impairing topic interpretability and model quality [41]. While stemming may improve joint probability scores, it compromises meaningful semantic relationships, aggressively reduces vocabulary, and is unsuitable for short, noisy texts, such as online reviews [19]. The authors also caution that failing to balance vocabulary reduction with topic interpretability may result in poorly performing models in applied contexts requiring meaningful insights.

The third group of pre-processing techniques refers to feature engineering for topic modeling. Two studies [7,9] use n-grams to generate bigrams and trigrams, two and three subsequent words naturally occurring in texts and are used for topic modeling. Ambiguities in the use of part-of-speech (PoS) tagging are present. In two studies, authors state utilizing PoS tagging without documenting the purpose [10,13], while only one study uses PoS tags to filter features used in topic modeling to adverbs, adjectives, and nouns that carry opinionated meaning [11].

Incomplete documentation is notable across the studies. To promote a thorough understanding and to enhance the replicability of research, authors should provide clear descriptions of pre-processing techniques. However, the authors of [12] noted the removal of “irrelevant content” and “abnormal data” during pre-processing; however, they did not provide clear definitions or criteria for what constitutes “irrelevant” or “abnormal”. Similarly, [14] refers to conducting text cleaning, but, without detailed documentation, the specific tasks performed under this broad term are unclear. Authors of [8] employed frequency-based filtering to identify common banking vocabulary frequently appearing in the corpus and words prevalent in positive and negative sections of their dataset. However, they did not provide further documentation on how these findings were utilized or incorporated into their analysis. The same authors also address negation handling by removing phrases containing negators without providing a rationale or justification for this specific approach.

Variations in pre-processing practices indicate an insufficient understanding of the critical role that data treatment plays in shaping topic structures and preserving semantic meaning [20]. Research on user experiences with topic modeling could greatly benefit from the adoption of more formalized pre-processing protocols. In this regard, the authors of this paper have sought to advance the field by proposing a methodology that emphasizes informed and deliberate choices in pre-processing techniques aimed at producing high-quality inputs for topic modeling.

3. Materials and Methods

The research presented in this study is focused on a single CRM application, contrary to the research of Nilashi et al. [14], aiming to offer a methodology for extracting generalizable knowledge from topic modeling analysis. Achieving such generalizability might be challenging when working with a limited dataset, such as the one utilized in Nilashi et al.’s [14] study. The methodology proposed in this research is rooted in the CRISP-DM (CRoss Industry Standard Process for Data Mining) process model for planning data mining projects, structuring communication within and outside the project team, and documentation of the project [42]. The methodology comprises six steps, presented in Figure 1: (1) data collection, (2) data cleaning and pre-processing, (3) corpus exploration, (4) the selection of topic modeling algorithm, (5) the evaluation of topic models, which facilitates identification of the optimal number of topics, and (6) the interpretation of topics with discussion of the results. As the literature on topic modeling of user experience with mobile applications indicated the most variability and sub-optimal uses within the data cleaning and pre-processing, special emphasis is put on this step in the methodology. With the proposed steps, authors strived to structure pre-processing activities for topic modeling and indicate the relevance of each step. The ultimate goal is to achieve high-quality inputs for topic modeling, meaning retainment of the semantics in the pre-processed texts and reduction in dimensionality of the corpus.

Data collection involves selecting sources aligned with the defined analytical objectives and gathering the relevant data. Data cleaning and pre-processing are conducted to increase data quality and transform data into an adequate format suitable for the chosen machine learning technique or analytical approach. Corpus exploration is part of exploratory text analysis conducted to gain intuition about the corpus and guide decisions about additional data pre-processing and further analysis. The selected analytical approach, the topic modeling, discovers topics hidden in a corpus [43], connections between topics, and their evolvement in time. Approaches to topic modeling are various, such as LDA, structural topic model, and non-negative matrix factorization. Selecting the topic modeling algorithm ensures alignment with the analytical objectives. As CRISP-DM suggests, models should be evaluated to assess their quality and goodness of fit to particular data or a problem. The fourth step of the proposed methodology, the Evaluation of topic models, implies the assessment of built topic models that facilitate the identification of the model with an optimal number of topics and the selection of the resulting model. The final step of the proposed methodology, involving the Interpretation of topics from the resulting model and discussion of the generated results, enhances the business value of the findings. The following sections present methodological steps in detail.

3.1. Data Collection

Data collection involves activities related to the dataset, i.e., corpus development, which pertains to determining and selecting data and data sources aligned with the analytical or research goals. Data collection implies activities related to the development of a dataset, i.e., corpus, which pertains to the identification of data and data sources aligned with the analytical or research goals. This paper focuses on identifying factors influencing satisfaction and dissatisfaction with the SaleForce mobile application that impact the user experience and, consequently, loyalty. Since online reviews reflect positive, negative, or neutral opinions, attitudes, and sentiments towards a certain entity [29], we restricted our research to online reviews of the SalesForce application. An online review can target the entity as a whole—the overall user experience or some of its aspects. A review reflects the subjective view of a person, while an analysis of a collection of online reviews provides a comprehensive picture of the user experience [44].

The data are collected from the Google Play Store (The URL location of SalesForce mobile application on Google Play Store: https://play.google.com/store/apps/details?id=com.salesforce.chatter&hl=en&gl=US, accessed on 16 January 2024) using a custom-written Python code for scraping the websites’ contents. The authors collected a corpus comprising 9.308 online reviews of the mobile application, which reflect the opinions and attitudes of users. The structure of the collected data is presented in

Table 3. An excerpt of data collection.

Review ID	User Name	User Image	Content	Score	Thumbs Up Count	Review Created Version	Time	Reply Content	Replied Time	Application Version
c19b1abc-5b0f-4721-b0b3-1a1fdbb8c9ea	Bijoy Shikari	https://play-lh.googleusercontent.com/a-/ALV-UjWVAV9UUHWQiwLy-M0q9MSYE8kI_NowZCw7Jv-mZ6xhfnM	Very well	5	0	246.030.0	16 January 2024 14:39:24	/	/	246.030.0
d72220d6-7f0f-4854-af1a-1d9070fc4d87	PRADEEP GUPTA	https://play-lh.googleusercontent.com/a-/ALV-UjVG045XEcDPwgIxewYY6dxjJH3Y4s1qxtXGQLJb2dPZ2JY	Nice app	5	0	246.040.0	16 January 2024 14:24:35	/	/	246.040.0

. The corpus includes the following attributes: Review ID—a unique identifier assigned to each review, User name—the username chosen by the reviewer, User image—a profile image associated with the user, Content—the text of the review, Score—star ratings given by the user, ranging from 1 to 5, Thumbs Up count—the number of likes a review receives, reflecting agreement with its content, Review created version—the version of the application being reviewed, and Time—the date and time when the review was published. Additionally, three attributes refer to replies made to the review: Reply content—the text of the reply, Replied time—the data and time the reply was posted, and Application version—the version of the application used by the person responding to the review.

As the goal of this research is to explore user experience and satisfaction, for further analysis, the authors used attribute content and scores, i.e., review texts with associated star ratings, while other attributes were ignored during the analysis.

3.2. Data Cleaning and Pre-Processing

Data pre-processing, combined with data cleaning, aims to enhance corpus quality and transform the data into a format optimized for the chosen machine learning algorithms. This process focuses on removing noise and preparing a refined dataset suitable for an effective machine learning algorithm application.

The proposed methodology, presented in Figure 1, implies seven major pre-processing steps applied sequentially on collected online reviews, explained in more detail in the subsequent of the paper. The effects of pre-processing are illustrated in

Table 4. Illustration of the text pre-processing.

Original Review

Pre-Processed Review

To Start off, Salesforce provides most of the functionality that is available in the web in Salesforce1 app right off the bat. Pros—Chatter in the phone—Report Notifications and Events & Today application help a lot—Anything developed in the Desktop Salesforce ports over easily Cons (wishlist)—I would like the UX to improve slightly better to reduce number of clicks for related list records etc—Search & Install for application exchange—Tight Dialer Integration Great application all in all ! Looking forward to future enhancements.

‘start’, ‘salesforce’, ‘provide’, ‘functionality’, ‘available’, ‘web’, ‘salesforce’, ‘application’, ‘right’, ‘bat’, ‘pro’, ‘chatter’, ‘phone’, ‘report’, ‘notification’, ‘event’, ‘today’, ‘application’, ‘help’, ‘lot’, ‘develop’, ‘desktop’, ‘salesforce’, ‘port’, ‘easily’, ‘con’, ‘wishlist’, ‘like’, ‘user’, ‘experience’, ‘improve’, ‘slightly’, ‘better’, ‘reduce’, ‘number’, ‘click’, ‘related’, ‘list’, ‘record’, ‘search’, ‘install’, ‘application’, ‘exchange’, ‘tight’, ‘dialer’, ‘integration’, ‘great’, ‘application’, ‘look’, ‘forward’, ‘future’, ‘enhancement’

.

Multilingualism handling: Data collection was not limited to a single language, resulting in a corpus that included reviews written in multiple languages. Figure 2 illustrates the distribution of non-English reviews in the dataset, amounting to 3079 instances.

Without implementing a multilingual management strategy, the corpus would have been significantly reduced. By not restricting data collection to English, this study ensured the inclusion of diverse user perspectives, reflecting a broad range of linguistic and cultural backgrounds.

To address the challenges posed by multilingual data and enable consistent pre-processing, all reviews were translated into English using the Google Translate API. This approach aligns with methodologies identified by Laureate, Buntine, and Linger [19], as well as other studies examining user perceptions and preferences [36,45,46]. Translating the textual data into a single language allowed for applying standardized NLP tools, avoiding the complexities associated with processing multilingual data. This unified approach preserved the diversity of user feedback, avoiding the common practice of excluding non-English content, which often results in a loss of valuable insights.

Following translation, a manual inspection was conducted to identify incomplete translations, such as untranslated Hindi phrases (e.g., hota hai). These instances were either manually translated to ensure consistency or excluded if translation was not feasible. This step was essential for maintaining dataset integrity and ensuring uniform pre-processing during topic modeling.

While translating all reviews into a single language provided advantages in dataset size and uniformity, alternative strategies for managing multilingual data in topic modeling were also considered. Laureate, Buntine, and Linger [19] identified two common approaches for short-text data, such as online reviews: (1) restricting the retrieval process to a single language, which simplifies analysis but significantly reduces dataset diversity and size, and (2) removing documents written in undesired languages, typically non-English, which can be resource-intensive but maintains linguistic focus. Both strategies have merits, particularly for studies prioritizing the preservation of linguistic context or targeting specific populations. However, for this study, which sought to analyze user feedback on the SalesForce application from a broad and diverse user base, the translation-based approach was deemed the most effective. This method ensured the widest possible representation of user perspectives while facilitating a consistent and streamlined analysis.

The second pre-processing step, defined by the proposed methodology in Figure 1, refers to spelling and grammar handling. This step involves activities designed to maintain the integrity of the text:

• Correction of misspellings, colloquial, or slang words, such as gud to good or awsm to awesome.
• Manual correction of abbreviated or contracted words, such as app to application, I’ve to I have, can’t to cannot, as illustrated in Table 4 with purple text.

Case standardization: Different forms of the same word, capitalized, uppercase, or lowercase, are considered by algorithms a different feature, increasing the dimensionality of the data. The authors standardized the case by lowercasing the text of all online reviews. In Table 4, the difference between the original and lowercase text is illustrated with orange text.

Noise reduction is an essential pre-processing step aimed at removing text elements that do not contribute to the contextual relevance or intended meaning. It encompass several activities:

• Removal of URLs, hashtags, and emails using regular expressions.
• Removal of the punctuation, alphanumerical values, numeric values, and redundant characters, such as square brackets, parentheses, and curly brackets, as they do not add new knowledge or information about the text but introduce the noise. For this purpose, the authors used regular expressions. In Table 4, in the column Original review, the removal of punctuation is illustrated by green text, while the removal of numerical values is illustrated by red text.
• Newline and extra white spaces are considered noise as they do not carry meaning. They are removed from the text.
• Stopwords are commonly used words, such as and, or, but, that often appear in text but have no semantical meaning. For topic modeling, it is important to retain only words that carry information, add new knowledge, or have semantical meaning, as topic modeling generates topics based on the keywords. Stopwords are removed from the texts. In Table 4, in the column Original review, stopwords are illustrated in blue.

Length-based filtering. Similar to [47,48], the authors removed short words containing less than 3 characters, as they do not carry meaning and can be considered stopwords. The authors defined exceptions for this pre-processing step. The words bad, not, and no were retained in the texts.

Tokenization divides the text into smaller units called tokens, such as words or symbols. It facilitates further text processing steps, such as lemmatization.

Lemmatization simplifies words by reducing them to their base or root form, known as the lemma, thereby decreasing data dimensionality. For instance, the verb “developed” and its variations, such as “develops” and “developing”, are reduced to their base form, “develop”, as demonstrated in pink text in Table 4. As a preparation for this step, authors conducted Part-of-Speech (PoS) tagging using Penn Treebank. PoS tagging is a process that searches for tokens with a particular part of speech and labels it. The authors tagged all adjectives, adverbs, nouns, and verbs, which are consequently lemmatized.

The authors managed missing values by removing all instances that did not contain review text, which may be because the user left only a star rating, or the review was of a modest size consisting of either stopwords, emoticons, or short-length words, and, during pre-processing, its content was removed. After pre-processing, corpus comprises 9.106 SalesForce mobile application reviews.

Algorithms are unable to understand and process texts in their raw format, and pre-processed texts should be vectorized before the modeling phase [38]. Text vectorization implies transforming text into numerical format. Available approaches to text vectorization vary from simple ones that rely on the raw frequency of word appearance or contribution of words to the context, such as the bag of words (BoW) or Term Frequency—Inverse Document Frequency (TF-IDF), to more advanced ones that take context into account, such as word2Vec, FastText, or BERTembeddings. In the research presented in this paper, authors used Python implementation of the bag of words, the CountVectorizer, which takes each online review and converts it into a vector composed of the number of word occurrences in the document, i.e., the real frequencies. The length of a vector is equal to the number of unique words comprising the vocabulary generated from the corpus. A document-term matrix is formed by combining all vectorized documents. The matrix contains documents in the rows, and each vocabulary element in the columns. The values of the matrix are the numbers indicating the frequencies of occurrence of the vocabulary elements (words) in the document—the online review.

3.3. Corpus Exploration

N-grams represent n words that naturally occur in the text. To gain intuition regarding the leading topics in SalesForce online reviews, authors analyze the most frequent n-grams in the corpus. Unigrams comprise a single word. Figure 3 represents the twenty most frequent unigrams in the corpus. We can observe that words, such as application, salesforce, and mobile, often appear in the corpus, but they can be considered domain stopwords. Domain stopwords are words that often appear in text without adding new knowledge about the subject. Reviews are purposefully collected to characterize the user experience with the SalesForce mobile application and text with different sentiment expressions targeting various aspects of this mobile application. For this reason, keywords application, mobile, and salesforce can be considered domain-dependent stopwords.

Bigrams comprise two consequent words and provide more context. Figure 4 illustrates the twenty most frequent bigrams in the corpus, from which we can observe that users most frequently refer to the application as good, great, nice, best, excellent, and user-friendly; however, a certain percentage of users considered the application bad. Adverbs used to describe user experience indicate that not all mentions of word application are domain-dependent stopwords, and the authors managed them in the following way: all mentions of word application preceded by an adjective are joined into the bigram using the underscore character, such as good_application. The authors used PoS tagging to detect adjectives in the text. After their manual inspection, authors defined exceptions for this task, a list of adjectives that should be ignored during the process of joining the adjectives with the word application, such as install, update, inspect, etc. (Words are lemmatized and converted to their root form after PoS tagging, and, therefore, the illustrative words are in their root form that is not adjective. Examples of the adjective form would be installed, updated, and inspected—the past participle form of a verb.). This step allowed for words to join into bigrams to become new features for topic modeling, while other instances of the unigram application were removed from the corpus, as well as unigram Salesforce and bigrams mobile application and salesforce application.

Trigrams comprise three consecutive words. The most frequent trigrams in the corpus are presented in Figure 5. The leading trigram bad application ever indicates the absolute dissatisfaction of users. Trigram analysis provides an initial intuition about specific causes of user dissatisfaction, such as take much time, take forever load, take long time, blank white screen. Based on the analysis of bigrams and trigrams, authors identified variations in the way that the company name is written; some users write SalesForce, while others write Sales force. This requires unification. All instances of sales force were rewritten into salesforce before removing domain-dependent stopwords, while the trigram salesforce mobile application was considered a domain stopword and was removed from texts.

Authors Kirilenko, Stepchenkova, and Dai [36] and Hu, Zhang, Gao, and Bose [37] suggested that the use of the overall corpus for topic modeling analysis can lead to the omission of certain dissatisfaction cues, indicating the need for extraction and isolated analysis of the content with negative sentiment. This motivated the authors of this paper to partition the collected dataset into subsets using star ratings.

Table 5. Review distribution by star ratings.

Star Rating	Number of Reviews	% of Total Reviews
5	4.565	50.27%
4	1.357	14.94%
3	684	7.53%
2	561	6.18%
1	1.914	21.08%
Total	9.081	100%

provides data on review distribution by star ratings, indicating the prevalence of strongly emotional reviews. Strongly positive reviews (5 stars) comprise 50.27% of the corpus, followed by strongly negative reviews (1 star) which comprise 21.09% of the corpus.

The partition of the corpus according to star ratings was conducted as follows: The first dataset, referred to as C1 in

Table 6. Overview of the main corpus statistics.

	C1	C2	C3
Number of reviews	5.922	684	2.475
Distribution by ratings	4.565 (5 stars) + 1.357 (4 stars)	684 (3 stars)	561 (2 stars) + 1.914 (1 star)
Minimal length in tokens	3	3	3
Maximal length in tokens	378	512	660
Average length in tokens	23	58	67

, comprises reviews rated with four or five stars, denoting reviews with positive sentiment. The second dataset, referred to as C2 in Table 6, comprises reviews with three stars, denoting reviews with mixed sentiment. The third data subset, referred to as C3 in Table 6, comprises reviews rated with one or two stars, denoting reviews with negative sentiment. Table 6 provides an overview of the main corpus statistics.

Although people tend to express positive sentiment more often, based on the minimal, maximal, and average lengths of reviews presented in Table 6, we can postulate, similar to the authors Kovačević et al. [29], that unpleasant emotions, negative experiences, and interactions with the application or customer service, when they do occur, have a stronger effect on users of the SalesForce mobile application, which, in turn, leads to broader vocabulary in use in expressions of dissatisfaction as opposed to satisfaction. The maximal length of negative reviews measured in the number of tokens is 660, while, on average, reviews contain 67 tokens. In positive sentiment, the average number of tokens is approximately three times smaller than in negative sentiment (23 tokens).

Word cloud is a visual representation of the most prominent words in the corpus. Words that best capture a specific topic are visually accentuated through the use of enlarged font sizes. The size of the word indicates its frequency in the corpus. Larger words appear more frequently in the corpus, whereas smaller words are less frequent. We used the Python library WordCloud to generate word clouds for each corpus subset, C1–C3, as presented in Figure 6. The colors are randomly assigned to differentiate the words in the word cloud, and they are not bearing any particular meaning. A positive word cloud, illustrated in Figure 6a, indicates what constitutes a positive user experience, such as good, efficient, nice, great, or excellent. A negative word cloud, depicted in Figure 6c, indicates what constitutes a negative user experience, such as login issues, authenticate, error, terrible, garbage, awful, or meaningless. In the mixed sentiment word cloud, in Figure 6b, words such as glitch, problem, slow query, refresh, and start interval appear to be focal points of user experience. As single words provide only intuition on sources of satisfaction and dissatisfaction among SalesForce mobile application users, further modeling is required to obtain a better understanding of user experience. The authors of this paper used topic modeling.

3.4. Method

Online reviews are considered short texts [49,50,51]. They are commonly analyzed using Latent Dirichlet Allocation (LDA), a widely adopted topic modeling approach in scholarly research. Laureate et al. [19] report, in their systematic literature review, that 79.79% of studies on short-text topic modeling research employed LDA [18]. The same method was applied in this study for its proven effectiveness in similar domains [4,7,8,9,12,13,14]. It is designed to identify clusters of keywords, referred to as topics that frequently co-occur in textual data [38]. The LDA framework operates on three core assumptions: (1) frequently co-occurring words are indicative of a shared topic, (2) individual documents are characterized as a blend of various topics, and (3) each topic is represented by a probabilistic distribution of keywords [38]. By assuming that documents are mixtures of latent topics and that words are sampled from these topics, LDA aligns well with many real-world text datasets, including those in the context of user experience, because most documents naturally cover multiple topics. For instance, a single-user review often encompasses multiple topics, such as mobile application usability (e.g., the ease of navigation), design aesthetics (e.g., the appeal of the interface), and functionality (e.g., the responsiveness of the features). LDA’s ability to capture these overlapping topic distributions makes it particularly effective in analyzing user experience data, where insights often span multiple, interrelated aspects of the user journey. In addition to this, the motivation for selecting LDA as the method refers to its provision of interpretable, understandable, and applicative results, given the nature of the representation of documents, topics, and words (topics are represented as distributions over words and documents as distributions over topics). The availability of the resources supporting LDA analysis and the visualization of the results additionally motivated its use.

The method operates iteratively, as illustrated in Figure 7 [52], where the input comprises M documents, denoted as {d₁, d₂, …, d_M}. Each document is expressed as a sequence of N words (w₁, w₁, …, w_N), with w_N representing the nth word. In the figure, W signifies the collection of words within the documents, while Z represents the associated topics. The core objectives of LDA are as follows: (1) to determine the optimal number of topics (K), (2) to estimate Φ, the distribution of words over topics, which indicates the likelihood of a word belonging to a specific topic, and (3) to compute θ, the distribution of topics across documents, which reflects the probability of each topic’s presence in a document.

LDA employs two key hyperparameters—alpha (α) and beta (β), which significantly influence the model’s output [53]. The parameter α controls the allocation of topics across documents, while β governs the selection of words within each topic. Fine-tuning these hyperparameters is essential, as their optimal values are not predefined and require experimental validation. To assess the quality of the generated topics, various statistical metrics are utilized, including model coherence, stability, and coverage [38]. Among these, topic coherence is the most widely applied measure, particularly in research focused on the application of LDA over online reviews [28,54,55,56,57,58,59,60]. This metric, as applied in this study, evaluates the frequency of the co-occurrence of prominent words within a topic [53]. For a given set of words T = {w₁, …, w_n}, the coherence C(T) is defined mathematically as follows [53,61]:

$$C\left(T\right)=\sum _{m=2}^M\sum _{l=1}^{m-1}\log {\displaystyle \frac{p\left(w_m,w_l\right)+\frac{1}{D}}{p\left(w_l\right)}}$$

where p(w_m, w_l) represents the probability of both words w_m and w_l co-occurring in the same document, calculated as the ratio of documents containing both words to the total number of documents in the corpus D. Similarly, p(w_l) denotes the probability of a single word’s occurrence across the corpus. A smoothing factor of 1/D was incorporated to avoid issues arising from zero-probability cases.

The evaluation process involves constructing multiple LDA models with different configurations of hyperparameters and selecting the one with the highest coherence score [43]. This approach ensures that the final model captures meaningful and consistent topics that align with the business or analytical goal.

4. Results

The authors conducted topic modeling on three distinct datasets, referred to as C1, C2, and C3. For each dataset, several LDA models were developed by varying the key parameters K, α, and β. The number of topics (K) was tested across values ranging from 2 to 15, while the hyperparameters α and β were evaluated at values of 0.01, 0.31, 0.61, and 0.91. Both symmetric and asymmetric topic distributions were considered during the modeling process. Symmetric distributions assume an even spread of topics across all documents, while asymmetric distributions allow certain topics to dominate [62].

The quality of each model was assessed using coherence scores, as there was no prior knowledge regarding the underlying topic distributions. Higher coherence scores indicate better alignment between the generated topics and the semantic structure of the data.

For the C1 dataset, the optimal model was identified at K = 14, as this configuration yielded the highest coherence score. Similarly, for the C2 dataset, the best results were achieved with K = 3. In the case of the C3 dataset, K = 10 was determined to be the most suitable number of topics. These findings highlight the importance of tailoring model parameters to each dataset to capture its unique characteristics effectively.

Table 7. Overview of optimal model parameters.

Data Subset	Coherence	K	α	β
C1	0.657	12	0.61	0.61
C2	0.684	3	asymmetric	0.91
C3	0.671	10	0.61	0.91

provides an overview of the parameter settings that resulted in the highest coherence scores for each dataset. These configurations serve as benchmarks for future studies aiming to apply LDA to similar types of short-text data.

4.1. Topic Model of Positive Reviews

The optimal model for reviews with positive semantics resulted in 12 topics.

Table 8. Positive sentiment topics.

Topic Number	Topic Name	Keywords	Topic Coverage
Topic 1	Seamless experience	nice_application, experience, best_application, ever, smooth, informative, flow, opening, worth, ultimate, truly, finance, perfect_application, india, nicely, fully, business, kindly, solid_application, enquiry, limit, awesome, maruti, beautiful, target, choice, learner, bless, guide, supportive	5.7%
Topic 2	Effective business management	great, best, tool, business, sale, platform, customer, manage, people, system, management, company, wonderful, efficient, communication, know, recommend, world, software, organization, amazing, product, hand, small, manager, cloud, professional, communicative	9.5%
Topic 3	User-friendly learning and exploration	user_friendly, amaze, learn, still, faster, bug, rock, super_application, explore, login, cool_application, brilliant, colleague, every, come, entire, definitely, alot, functional, friend, training, week, effort, house, install, ask, certificate, curve, animation	6.1%
Topic 4	Usability	excellent, super, sometimes, helpful_application, excite, level, bad, properly, extend, member, room, beneficial, tracking, comfortable, family, practical, true, seamlessly, perform, indeed, alright, deep, besides, next, educate, usability, material, fabulous, another, application appointment	5.7%
Topic 5	Speed and reliability	awesome, update, fast, slow, superb, fantastic, start, process, little, always, speed, simply, often, response, require, easily, resource, high, accessible, quicker, error, password, brilliant_application, connection, kill, huge, late, bring, lag, plan	6.8%
Topic 6	Performance and flexibility	work, love, need, well, time, everything, get, do, thank, friendly, performance, first, cool, connect, fine, nothing, take, understand, company, fix, expect, dashboard, place, save, wait, run, whole, stuff, perfectly, flexible	10.3%
Topic 7	Functionality and operability	good, working, vary, limitation, monkey, concept, outage, congratulation, noon, front, vein, really, basic_application, limited, consume, associate, getter, real, employees, very, initiative, exhaustive, shadow, fresher, maintain, extension, easy, application, working	6.3%
Topic 8	Service improvement and training	nice, great_application, service, awesome_applicaiton, improvement, nausea, helpful, useful, site, slowly, previous, cant, technician, project, junk, training, host, blackberry, expectation, senior, guess, previously, classic, limitation, very, finish, beyond, invaluable, incentive	5.9%
Topic 9	User experience and technical feedback	like, would, really, phone, version, able, could, report, can, not, issue, chatter, view, call, please, time, option, star, give, find, notification, task, also, allow, android, update, wish, look, note, case	18.8%
Topic 10	Support and integration	good_application, useful, excellent_application, support, solution, market, amazing_application, amaze_application, okay, incredibly, fool, hope, single, private, interest, reporting, line, integrate, power, automatically, powerful_application, satisfied, employee, yeah, efficiency, various, company, log, free, convenience	5.9%
Topic 11	Intuitiveness and reliability	useful_application, perfect, problem, pretty, enjoy, intuitive, go, capability, crash, easy_application, purpose, wonderful_application, admin, flood, develop, fantastic_application, learning, secure, solve, reliable, past, dirty, thumb, want, lot, offline, login, review, hate, lovely	5.9%
Topic 12	Simplicity and convenience of navigation	easy, make, keep, use, help, much, helpful, access, team, feature, user, data, track, interface, information, quick, improve, simple, thing, navigate, activity, handy, daily, desktop, lead, thanks, convenient, stay, provide, field	13.1%

provides an overview of topics, their indicative names generated by authors based on the most salient keywords presented in the second column in Table 8, and topic coverage.

Uncovered topics indicate people positively talk about four aspects of the SalesForce mobile application:

• User experience and usability—Topic 1: Seamless experience; Topic 3: User-friendly learning and exploration; Topic 4: Usability; Topic 9: User experience and technical feedback; and Topic 12: Simplicity and convenience of navigation.
• Performance and reliability—Topic 5: Speed and reliability; Topic 6: Performance and flexibility; and Topic 11: Intuitiveness and reliability.
• Service and support—Topic 8: Service improvement and training and Topic 10: Support and integration.
• Business management—Topic 2: Effective business management and Topic 7: Functionality and operability.

Discovered topics indicate sources of satisfaction among SalesForce mobile application users that influence and elevate user experience. The satisfaction factors are reflected in the simplicity of the use of the application and navigation, efficiency, and reliability of the application, support for business operations and integration with other systems, and quality customer support and support throughout learning pathways. Among discovered topics, the users consider as the most relevant possibility to communicate regarding technical issues with SalesForce representatives (18.8% coverage for Topic 9), simple and convenient navigation through the application (13.1% coverage for Topic 12), and good performance and flexibility of the application (10.3% coverage for Topic 6).

4.2. Topic Model of Mixed Sentiment Reviews

The optimal model for mixed sentiment reviews resulted in three topics.

Table 9. Mixed sentiment topics.

Topic Number	Topic Name	Keywords	Topic Coverage
Topic 1	Performance and updates	work, good, need, would, time, slow, can, not, update, well, like, use, phone, version, load, android, make, able, please, great, could, page, really, notification, easy, calendar, seem, take, open, miss	78.2%
Topic 2	Interface and usability issues	tablet, landscape, mode, open, case, would, support, work, keyboard, android, view, make, feature, rotate, portrait, user, screen, many, still, issue, know, sale, sometimes, please, need, account, problem, quick, cant, install	15.7%
Topic 3	Authentication challenges and system constraints	option, case, text, comment, view, difficult, must, multiple, quality, tool, login, post, go, accept, type, internal, maintain, cost, filter, impossible, field, payslip, print, find, logout, respond, common, resize, sure, report	6.1%

presents topics, the most salient keywords per topic, and topic coverage.

The SalesForce mobile application users center their attention on the performance and updates of the application (Topic 1), interface and usability (Topic 2), and authentication (Topic 3). They are satisfied with the application speed and update regularity while expressing dissatisfaction with the slow loading of the application and notifications. Interface and usability issues are predominantly notable on tablets, in particular when switching from landscape to portrait regime and issues with feature installations. The third topic refers to the issues related to login difficulties. The most relevant topic for the mixed sentiment is related to performance (78.2% coverage of Topic 1), followed by interface and usability issues (15.7% coverage of Topic 2), and authentication issues (6.1% coverage of Topic 3).

4.3. Topic Model of Negative Reviews

The optimal topic model for reviews with negative semantics resulted in 10 topics.

Table 10. Negative sentiment topics.

Topic Number	Topic Name	Keywords	Topic Coverage
Topic 1	Compatibility issues	work, version, android, screen, like, phone, chatter, bad, please, error, company, update, keep, install, use, good, make, sale, look, galaxy, crash, issue, go, device, blank, would, well, get	21.3%
Topic 2	Design and usability	properly, good_application, slow_application, design, permission, automatically, waste, auto, include, useless_application, logout, camera, functional, offer, word, feature, head, screw, possibly, high, forget, rotate, pixel, shake, landscape, student, could, door, ferrari, passenger	4.6%
Topic 3	Customer support	calendar, private, enquiry, support, post, public, promise, much, delivers, except, feedback, remote, message, date, price, bad, answer, today, point, zero, custom, idea, ultra, massive, group, so, either, extra, show, forum	4.2%
Topic 4	Access and functionality	can, not, case, open, useless, view, access, button, lead, need, tablet, task, landscape, anything, change, seem, object, want, able, note, link, feed, upgrade, list, without, chatter, pretty, desktop, detail, find	12.1%
Topic 5	Performance optimization	improve, soon, process, hope, difficult, opportunity, navigate, everything, shot, performance, dashboard, expense, report, hint, navigation, able, move, picture, line, start, know, create, care, sync, video, portal, customize, point, tabs	4.6%
Topic 6	Device compatibility and battery issues	class, dont, third, battery, understand, tough, motorola, term, slowly, ipad, licensing, supplemental, obliterate, reveal, one, dreamforce, android, incredibly, lightning, tailhead, still, approve, contract, friendly, chalte, counterintuitive, eats, store, bullshit	3.9%
Topic 7	Reliability	bad_application, ever, worst, rely, best, world, software, piece, better, trash, excuse, monkey, useless, plan, see, unreliable_application, widely, force, unacceptable, expensive, business, upon, customer, marketing, design, know, help, adopt, attrition, iphones	4.9%
Topic 8	Login and load time	time, work, slow, load, even, would, login, need, take, never, update, contact, use, account, page, really, open, phone, access, many, data, always, try, email, crash, browser, download, search, enter	27.4%
Topic 9	Poor user experience	experience, poor, awful, horrible, clunky, simply, pathetic, location, money, learn, great, develop, every, interface, waste, proper, frustrating, fill, unsubscribe, beyond, cycle, subpar, frequently, poorly, quote, organization, garbage, everyday, low, imagine	5.2%
Topic 10	Account management and functionality	make, account, comment, view, call, record, use, slow, option, would, phone, functionality, also, easy, page, create, field, support, good, like, feature, basic, buggy, update, nice, available, useful, device, site, browser	11.8%

provides an overview of topics, their indicative names generated based on the most salient keywords presented in the second column in Table 10, and topic coverage.

The dissatisfaction of SalesForce mobile application users is primarily centered on four aspects:

• Design and usability—Topic 2: Design and usability and Topic 9: Poor user experience.
• Performance and reliability—Topic 5: Performance optimization, Topic 7: Reliability, and Topic 8: Login and load time.
• Compatibility and functionality—Topic 1: Compatibility issues, Topic 6: Device compatibility and battery issues, and Topic 4: Access and functionality.
• Customer interactions and support—Topic 3: Customer support and Topic 10: Account Management and Functionality.

Sources of dissatisfaction among SalesForce mobile application users predominantly lie in performance and compatibility issues, which are the two most valued aspects in negative feedback (Topic 8 with 27.4% of coverage and Topic 1 with 21.3% of coverage), followed by issues with changes from landscape to portrait view, and account management.

5. Discussion

The section is organized to provide answers to each of the proposed research questions.

5.1. RQ1: How Can Companies Use Topic Modeling to Understand Mobile Application User Experience?

Topic modeling is an analytical approach that enables the extraction of hidden semantic structures from texts and a powerful analytical tool that facilitates companies in identifying sources of user (dis)satisfaction. It helps in understanding the most valued and undervalued aspects of a product or service. For any business, insights into users’ discontent and frustration are valuable as they can guide the company toward customer-centric product improvements. However, several authors highlighted the challenges if topic modeling is conducted over diverse reviews or the overall corpus, such as Kirilenko, Stepchenkova, and Dai [36] and Hu, Zhang, Gao, and Bose [37], as it can lead to the oversight of certain dissatisfaction cues. For this reason, we suggested one possible approach towards extracting content for topic modeling based on star ratings associated with the textual reviews of products. The proposed approach allows companies to identify exclusively positive or negative aspects of a product or service, which can be used as roadmaps for defining priorities in product development suited to the current needs and requirements of the customer base. By applying topic modeling over subsets of reviews (negative: one and two-star ratings, mixed sentiment: three-star ratings, and positive: four and five-star ratings), a company distinguishes factors contributing to positive user experiences and negative ones. Discovered knowledge can be used to design strategies aimed at proactively addressing identified challenges or defining priorities for improvements, aligning updates with user expectations that will improve company offer, gain a competitive advantage in the market, and enhance customer support.

5.2. RQ2: What Are the Main Factors Impacting the Salesforce Mobile Application User Experience?

The results of topic modeling indicate that the users perceived the application as an effective business management solution. They emphasize how the offered tools for business management enhance business organization. Users value the application because of the seamless user experience, informative content, learning support, simple navigation, and practical application options. Discovered knowledge indicates that users consider as relevant the application performance and its reliability by indicating that application speed and regular updates increase their satisfaction and impact positive user experience, while intuitiveness and ease of use are appreciable. In general, users value customer support, and the SalesForce mobile application users particularly value the support that the company provides regarding the integration of the application with other systems. On the contrary, users are unsatisfied with the design of the application interface and user experience related to the usability of the application. The SalesForce mobile application users are experiencing difficulties while navigating through the application and switching from landscape to portrait view. Poor design and usability are leading to their frustration and inclination to unsubscribe. Negative factors associated with performance and reliability are related to the technical performance of the application, particularly speed, reliability, and loading time. Users with certain devices or operating systems are excluded given the incompatibility of the application with devices or operating systems, which consequently limits the reach of the application.

Based on the aspects addressed within the topic model for positive sentiment (Section 4.1) and for negative sentiment (Section 4.3), the authors of this paper derived the set of aspects that reflect the influential factors of user experience with the mobile application.

Table 11. Overview of user experience factors.

Aspect	Positive User Experience	%	Negative User Experience	%	Mixed User Experience	%
User experience, application design, and usability	Seamless experience (5.7%) User-friendly learning and exploration (6.1%) Usability (5.7%) Simplicity and convenience of navigation (13.1%)	30.6%	Design and usability (4.6%) Poor user experience (5.2%)	9.8%	Interface and usability issues (15.7%)	15.7%
Performance and reliability	Speed and reliability (6.8%) Performance flexibility (10.3%) Intuitiveness and reliability (5.9%)	23%	Performance optimization (4.6%) Reliability (4.9%) Login and load time (27.4%)	36.9%	Performance and updates (78.2%)	78.2%
Service and support	Service improvement and training (5.9%) Support and integration (5.9%) User experience and technical feedback (18.8%)	30.6%	Customer support (4.2%)	4.2%	/	/
Business management	Effective business management (9.5%) Functionality and operability (6.3%)	15.8%	/	/	/	/
Compatibility and functionality	/	/	Compatibility issues (21.3%) Device compatibility and battery issues (3.9%) Access and functionality (12.1%)	37.3%	/	/
Account management	/	/	Account Management and Functionality (11.8%)	11.8%	Authentication (6.1%)	6.1%

summarizes distinctions among factors influencing positive, negative, and mixed user experience and provides aspect coverage, which is generated by aggregating the individual topic coverage data from positive, negative, or mixed sentiment topics.

Figure 8 comparatively illustrates the contributions of identified aspects to different levels of user experience. Compatibility issues and ill functionalities associated with incompatibility of the application with other devices or operating systems and authentication issues are exclusively characterizing negative user experience. Unsatisfied users of the SalesForce mobile application cannot value the business management contributions of the application, contrary to satisfied ones.

5.3. RQ3: Comparison of Identified Factors Influencing User Experience of the Salesforce Mobile Application and Factors Identified in the Literature

Insights obtained throughout our study are compared with the results of Nilashi et al.’s [14] research and systematized to answer the third research question: how do the factors influencing the SalesForce mobile application user experience compare to those discussed in the literature on user satisfaction with CRM applications?

The main difference between the research presented in this paper and that of Nilashi et al. is reflected in the fact that we directed the analysis of factors that influence user satisfaction toward the sources of positive and negative user experience by deriving subsets of data based on the associated star ratings. We aggregated identified factors into six aspects (user experience, application design, and usability, performance and reliability, service and support, business management, compatibility and functionality, and account management), while Nilashi et al. [14] grouped factors into three dimensions (information quality, system quality, and service quality). In the subsequent analysis, we strived to map the factors comprising each dimension in Nilashi et al.’s [14] research on user experience aspects derived from our research. The results are presented in

Table 12. Comparison of SalesForce user experience factors and factors identified in the literature.

SalesForce User Experience Aspects	Nilashi et al. [14] User Experience Dimensions	Shared Factors
User experience, application design, and usability	System quality	usability interface design
Performance and reliability		reliability performance system updates
Compatibility and functionality		compatibility adaptability system integration access control
Account management		security user permissions
Service and support	Service quality	user training support service improvement
Business management	Information quality	-

.

The following aspects of user experience, application design and usability, performance and reliability, compatibility and functionality, and account management align with Nilashi et al.’s [14] dimension System quality. The service support aspect is associated with the Service quality dimension, identified by Nilashi et al. [14]. Although we did not identify explicit matching topics, business management could be aligned with Information quality as effective management is supported by quality information for reporting and data analytics. Similarities between identified factors influencing user experience are reflected in the shared topics presented in Table 12, column Shared factors. However, our approach to user experience exploration, which is focused on one particular CRM solution and centered on a separate analysis of subsets of reviews according to the expressed sentiments, indicates that more nuanced factors can be detected. Contrary to Nilashi et al.’s [14] findings that usability, performance, and reliability are important factors, we identified more precisely which aspects of mobile applications these factors are directed, such as the association of usability with navigation and the design of the application. While performance and reliability are covered in Nilashi et al. [14], our results expand on flexibility, speed, and intuitiveness, which add more nuance to these factors. The same applies to compatibility, which is expressed with more detailed factors in the findings presented in our study, focusing on operability, device compatibility, and real-time functionality.

5.4. RQ4: What Are the Business Implications of Topic Modeling Findings?

Discovered knowledge about aspects influencing positive and negative user experience can direct business strategies that will impact user satisfaction and trust, help in building positive customer relationships, and expand the user base, as presented in

Table 13. User experience insights and business impact.

Aspect	In-Focus	Business Impact
User experience, application design, and usability	Intuitive and user-centric design Usability improvements	user satisfaction retention
Performance and reliability	Performance optimization Assurance of system reliability	user trust churn
Service and support	Enhance service support and user training	user satisfaction user loyalty user trust positive customer relationship
Compatibility and functionality	Resolving incompatibility issues	expansion of user base user satisfaction
Account management	Simplification of account management	user satisfaction retention

. Table 13 summarizes insights about user experience and indicates how acting upon the insights might impact the business:

• User experience is largely rooted in the user interface design and ease of navigation [63], which is supported by topic modeling findings, indicating that the SalesForce mobile application users require a user-friendly and intuitive interface. By addressing their needs, e.g., by implementing sophisticated drag-and-drop dashboard builders or simplified navigation bars, companies can directly enhance usability, leading to a decrease in frustration, an increase in overall satisfaction, and an increase in retention rates.
• The enhancement of user satisfaction could be achieved by addressing account management and authentication issues, which users of the SalesForce mobile application mention as issues. According to the findings of topic modeling analysis, enhancement in this area requires simplification, which will reduce user frustration, improve the usability of the application, and affect user satisfaction and retention. Companies could consider options, such as Google’s One-Tap Sign-In or biometric logins that ease the login process and reduce frustration rates, contributing to reduced drop-off rates.
• User trust is enforced with fast and reliable applications [64], while topic modeling analysis identified that slow loading time influences the users’ satisfaction with the SalesForce application performance. The company might conduct more frequent performance tests and invest more in optimization of the backend architecture and infrastructure that will ensure that the company meets customer expectations related to performance and reliability.
• User satisfaction and experience are shaped by the support they receive [65]. Quality customer interactions help a company build loyalty and trust that will contribute to maintaining positive relationships with users. By implementing features like chatbot integration and personalized responses, companies could improve satisfaction and maintain a strong customer relationship. The SalesForce user indicates services and support as a major issue, which the company should consider addressing to prevent further dissatisfaction. Our model helps to identify aspects considered positive within the services and support, such as training support, which should be fostered.
• The expansion of the user base can be achieved by ensuring the mobile application is compatible with a wide range of devices and operating systems. Another issue users are indicating in negative reviews is related to the resource intensiveness of the application and the effect it has on battery consumption, implying the need for resource optimization strategies, which is something that the SalesForce might consider to enhance user experience.

To ensure the continuous monitoring of user preferences and perception, companies should ensure a continuous feedback loop that implies a continuous collection of online reviews from relevant sources, as well as the incorporation of real-time feedback mechanisms into the application (e.g., pop-up surveys or chat support within the application), which will allow businesses to address concerns proactively, preventing dissatisfaction from escalating due to negative reviews and comments spread. This allows companies to allocate resources and prioritize investments based on the most impactful factors identified from the analyzed data. In addition to this, companies might consider analyzing reviews segmented according to geographic regions to understand cultural nuances and adapt product strategies accordingly.

6. Conclusions

Intensive competition in the mobile application market has influenced the dependence of long-term business sustainability on positive user experience. Users repeatedly choose products with which they are satisfied and have a positive user experience. A satisfied user becomes loyal to the company, from which the company has multiple benefits. They provide a secure and continuous income, while their positive WoM contributes to the acquisition of new users. Therefore, the goal of every company is to secure a loyal customer base. In this aspect, user experience research should be carried out continuously, as it provides guidelines for product improvements and thus the sustainability of the company in the rapidly growing market. In this paper, the authors research the possibilities and the reach of applying topic modeling to reveal information on user experience and inform the business decision-making process. It is illustrated in the example of the SalesForce mobile application. The research process is planned and rooted in the CRISP-DM methodology. The authors use the collection of online reviews of the mobile application populated from the Google Play Store.

Based on the associated star ratings, the collection is split into three subsets. A negative user experience is represented by online reviews rated with one or two stars. Three-star rated online reviews constitute a mixed user experience, while those with four and five stars represent a positive user experience. After extensive text pre-processing, directed towards enhancing data quality and text vectorization, the LDA approach to topic modeling was applied, and a series of models were developed and evaluated using the coherence measure. The results indicate that the users of the SalesForce mobile application express satisfaction through 12 different topics, which are grouped into four segments constituting the positive user experience—good user experience and usability, performance and reliability, service and support, and business management solutions. Mixed user experience was denoted by performance, update, interface and reusability issues, authentication challenges, and system constraints. Negative user experience is reflected in 10 topics. The most common refers to compatibility, problems of access and functionality, login and loading time, and account management and functionality challenges.

6.1. Theoretical and Methodological Contributions

The topic modeling of user experience with CRM mobile applications is a relatively unexplored field. By providing case-based research and the methodology on how to apply topic modeling for the evaluation of user satisfaction and experience, this paper contributes to expanding the existing body of the relevant literature. The methodological framework proposed in our work was applied to one CRM mobile application, contrary to [14]. Also, our study processed 9.081 user reviews for one application, unlike [14] where the total number of processed reviews for 8 applications was 5172. By encompassing more data that are narrowly focused on a single application, our research resulted in the identification of more detailed and granular (dis)satisfaction cues, reflected in a wider range of identified factors influencing user satisfaction with CRM mobile applications, such as flexibility, speed, and intuitiveness. Also, our study more precisely describes which aspects of CRM mobile applications are focused on usability, performance, and reliability, compared to the research conducted by Nialshi et al. [14].

This study presented in this paper also highlights the importance of multilingual data collection, which broadens the representation of user perception and enhances the generalizability of findings. Unlike predominant approaches identified by [19] that restrict data collection to English, our multilingual strategy preserved one-third of the dataset that would have otherwise been excluded, offering a more comprehensive understanding of global user needs and experiences. Such insights can subsequently inform localized analyses, enabling a more nuanced understanding of specific cultural contexts when needed. In addition to this, the authors Nialshi et al. [14] encompassed only English reviews and thereby reduced the diversity of the sample, as user experience can vary significantly depending on cultural and regional factors [28].

The key theoretical implication and scientific contribution of the paper is the proposed methodological framework for implementing the topic modeling approach in the evaluation of user experience. The proposed framework emerged as a response to an identified gap in the literature, pointing to numerous shortcomings of topic modeling approaches applied in research dealing with user experience with mobile applications. According to the observed shortcomings, we systematized and organized methodological steps, particularly focusing on the phase of data cleaning and pre-processing where the most variations or sub-optimal utilization of techniques are detected in the literature review. The proposed methodology strives to ensure a higher quality of the data that will be used in topic modeling.

6.2. Practical Implications

As digital transformation accelerates, understanding the factors influencing user satisfaction is vital for organizations integrating mobile technologies into their business models. Online reviews have a crucial role in reputation management, enabling businesses to monitor and enhance their brand perception. This study proposes a methodology for online review analysis with topic modeling, identifying key drivers of user experience. The insights derived inform decisions on application design, feature prioritization, and resource allocation, ultimately shaping strategies that enhance user satisfaction, trust, and loyalty while promoting business sustainability.

Using the SalesForce case study, this paper demonstrates the practical application of the methodology and the challenges that might emerge in its application in real-world scenarios while revealing actionable insights. For instance, users value intuitive interfaces and robust customer support, while issues such as authentication difficulties, slow load times, and high battery usage cause frustration. These granular findings align with six overarching user experience factors: (1) usability and design, (2) performance and reliability, (3) service and support, (4) business management, (5) compatibility and functionality, and (6) account management. Addressing these areas directly impacts retention strategies and strengthens user trust and satisfaction.

6.3. Limitations and Future Research Directions

Limitations of the proposed methodology can occur with applications emerging on the market, which suffer from an insufficient number of reviews. It is tied to textual data, so comments that include images, videos, or emoticons are a priori excluded. Although methodology offers the possibility to extract valuable insights, as demonstrated through the practical case study, data can be unrepresentative of all users. Not all users of mobile applications share their opinions in the form of reviews. Additionally, the analysis excludes traditional factors such as demographics, location, and consumer roles, as these attributes are not provided by the data source. Expanding future research to incorporate diverse data sources, such as other review platforms, social media, surveys, and interviews, could address these limitations. A comparative analysis of competitors, such as SAP CRM, could also provide valuable insights into SalesForce’s market position, highlighting strengths and weaknesses relative to competitors.

Future research should explore multilingual data utilization to derive localized insights and personalized software recommendations. Segmenting reviews by geographic region can reveal cultural nuances, allowing businesses to adapt strategies accordingly.

Topic modeling is the analytical approach fully based on the keywords present in the texts. Data pre-processing and feature engineering highly influence the quality of the model. As the data encompass negative reviews, where negation is widely used, future research will encompass evaluation of the influence of the negation on the resulting topics, such as joining negation with negated words and evaluating the interpretability of the resulting topics. Currently, authors use CountVectorizer for text vectorization, and future research might include other approaches to vectorizing texts, such as Term Frequency-Inverse Document Frequency, which indicates the contribution of particular words to the overall meaning of a text document. Future research may include new algorithms for topic modeling, such as non-negative matrix factorization, probabilistic latent semantic analysis, or structural topic modeling.