ChurnKB: A Generative AI-Enriched Knowledge Base for Customer Churn Feature Engineering

Abstract

Customers are the cornerstone of business success across industries. Companies invest significant resources in acquiring new customers and, more importantly, retaining existing ones. However, customer churn remains a major challenge, leading to substantial financial losses. Addressing this issue requires a deep understanding of customers’ cognitive status and behaviours, as well as early signs of churn. Predictive and Machine Learning (ML)-based analysis, when trained with appropriate features indicative of customer behaviour and cognitive status, can be highly effective in mitigating churn. A robust ML-driven churn analysis depends on a well-developed feature engineering process. Traditional churn analysis studies have primarily relied on demographic, product usage, and revenue-based features, overlooking the valuable insights embedded in customer–company interactions. Recognizing the importance of domain knowledge and human expertise in feature engineering and building on our previous work, we propose the Customer Churn-related Knowledge Base (ChurnKB) to enhance feature engineering for churn prediction. ChurnKB utilizes textual data mining techniques such as Term Frequency-Inverse Document Frequency (TF-IDF), cosine similarity, regular expressions, word tokenization, and stemming to identify churn-related features within customer-generated content, including emails. To further enrich the structure of ChurnKB, we integrate Generative AI, specifically large language models, which offer flexibility in handling unstructured text and uncovering latent features, to identify and refine features related to customer cognitive status, emotions, and behaviours. Additionally, feedback loops are incorporated to validate and enhance the effectiveness of ChurnKB.Integrating knowledge-based features into machine learning models (e.g., Random Forest, Logistic Regression, Multilayer Perceptron, and XGBoost) improves predictive performance of ML models compared to the baseline, with XGBoost’s F1 score increasing from 0.5752 to 0.7891. Beyond churn prediction, this approach potentially supports applications like personalized marketing, cyberbullying detection, hate speech identification, and mental health monitoring, demonstrating its broader impact on business intelligence and online safety.

1. Introduction

Customer churn refers to a customer’s tendency to discontinue business with a company within a given period or contract [1]. Nearly all businesses encounter customer churn, often exacerbated by the limited visibility of customer data. This lack of insight leads to customer relationship crises and churn, making it difficult for companies to predict when customers are likely to leave [2]. Retaining existing customers is significantly more profitable than acquiring new ones, as retained customers generate higher profit margins [3]. Moreover, attracting a new customer costs 5–10 times more than retaining an existing one [4]. Therefore, preventing customer churn is critical for businesses offering ongoing services. Addressing churn effectively requires a deep understanding of customers’ cognitive status and behaviour, enabling businesses to anticipate and mitigate churn risks proactively. To this end, leveraging analytical technologies to identify early signs of churn can be of great benefit.

Extensive research has been conducted on customer churn analysis, utilizing various analytical and machine learning (ML) techniques to gain insights into customer churn tendencies. Most of these studies rely on feature engineering as a fundamental prerequisite for model training and predictive analysis [3,5,6]. However, traditional churn-related analytical studies have predominantly focused on demographic, product usage, and revenue-based features. Some recent studies have incorporated social network analysis features into churn models, which yield promising results  [5,6,7].While some prior studies have explored social media sentiment analysis [8], call transcripts [9], and support tickets [10] for churn prediction, there is still a gap in systematically integrating a structured knowledge base and generative AI into feature engineering for churn analysis. Our approach, ChurnKB, extends prior work by combining knowledge-driven text mining with generative AI to enhance feature extraction and improve predictive performance [11].

Leveraging knowledge bases, which integrate domain knowledge and experts’ experience, can enhance the feature engineering process [12,13,14]. Human expertise and intuition play a crucial role in feature engineering, a fundamental step in ML projects that involves designing and selecting relevant features [15]. In our previous work [16], we explored various psychological and cognitive science sources to develop a domain-specific knowledge base related to mental health disorders, referred to as the Mental Disorder Knowledge Base (mKB). Our findings indicated that depression is closely linked to negative emotions such as sadness, anger, and disgust. Building upon the promising results of that study and recognizing that negative cognitive status, emotions, and feelings (e.g., dissatisfaction with a product or service) increase the likelihood of churn [1], we propose the development of the Customer Churn-related Knowledge Base (ChurnKB) to enhance customer churn analysis.

Recent advancements in generative AI (GenAI) have expanded the opportunities available to businesses, allowing them to enhance various aspects of their operations beyond what was previously considered achievable [17]. GenAI is already transforming industries such as journalism, visual arts, and customer service by enabling automated content creation [18].

ChurnKB leverages generative AI in conjunction with textual data mining techniques to extract churn-related features from customer-generated content, such as emails, chat logs with support agents, website reviews, and social media feedback. Unlike traditional NLP-based approaches (e.g., rule-based text mining, sentiment analysis, and domain-specific lexicons), which rely on predefined heuristics, generative AI offers the flexibility to handle unstructured text, uncover latent features, and generate contextually enriched representations of customer behaviour [19].

As the first contribution of this study, we propose the development of ChurnKB and leverage generative AI, specifically Large Language Models (LLMs), to refine and enhance its structure. This includes utilizing LLMs for feature generation to build a more comprehensive and adaptable knowledge base. To ensure its robustness, we implement a feedback loop approach that validates both the structural integrity of ChurnKB and the effectiveness of GenAI in improving its framework.

A key factor in effectively utilizing ML algorithms is the selection of meaningful features that capture essential aspects of the data. In customer churn analysis, behavioural patterns that indicate a propensity to churn are particularly important. However, despite the availability of numerous tools for handling ML data and algorithms, support for cognitive and behavioural features remains limited [20]. As our second contribution, we explore the integration of ChurnKB into ML models by leveraging cosine similarity and Term Frequency-Inverse Document Frequency (TF-IDF) techniques. These techniques facilitate the quantification of churn-related features, allowing ML classifiers to better capture subtle emotional cues and behavioural signals indicative of churn. Additionally, we assess the effectiveness of ChurnKB-derived features by comparing model performance with and without knowledge-enhanced feature representations.

The remainder of this paper is organized as follows: Section 2 provides background studies on customer churn, feature engineering, and generative AI. Section 3 details our proposed approach for developing ChurnKB and its integration into the customer churn classification pipeline. Section 4 presents the evaluation results of the proposed approach, followed by a discussion of the findings in Section 5. Finally, Section 6 concludes the paper with remarks on future research directions.

2. Background

2.1. Customer Journey, Cognitive Status, and Behaviours

The customer journey encompasses the entire experience a customer has with a business, including interactions at various touchpoints (i.e., points of interaction) such as purchasing and post-purchase services [1]. Optimizing these touchpoints is essential in enhancing customer experience. Customer service, as a key component of the journey, involves communication with customers, follow-ups after sales, and gathering qualitative data on customer sentiments. Combining these data with quantitative factors provides valuable insights into customer feedback and trends [1].

The customer journey significantly influences customer cognitive status, satisfaction, and overall churn risk [1,21]. Meanwhile, customer behaviour refers to the activities and decision-making processes customers undergo when purchasing and consuming products [22,23,24]. Various factors, including personal attributes (e.g., gender, age, education, and income), psychological influences (e.g., motivation, perception, and attitudes), social elements (e.g., family and reference groups), and cultural aspects (e.g., social class), shape customer cognitive status and behaviour [25]. Emotional responses such as pleasure, fear, anger, and sadness further impact the experiences and purchasing decisions of customers, ultimately influencing their likelihood of staying with or leaving a business [26].

2.2. Customer Churn from an Analytical Perspective

Understanding and predicting customer churn is crucial in today’s data-driven world. Churn analysis aims to identify patterns and reasons behind customer departures. By analysing data from various sources, businesses can gain insights into why customers leave. ML models help predict potential churners, enabling companies to take action and retain valuable customers [27]. An ML workflow relies on a good combination of data, features, and models [20]. Features act as a bridge between raw data and insights. Selecting the right features simplifies model training and improves performance. Poor feature selection may require more complex models to achieve the same accuracy [28].

A feature is a measurable property of data used in ML. Features can be numerical (e.g., age and income), categorical (e.g., gender and country), textual (e.g., TF-IDF and word embeddings), temporal (e.g., date and time), geospatial (e.g., location and distance), or transactional (e.g., product prices) [28]. Some models work better with specific feature types. The number of features also matters. Too few may lead to poor predictions, while too many can increase complexity and costs [28]. Feature engineering is the process of transforming raw data into useful features for ML models. It involves creating new features, modifying existing ones, and ensuring they are informative and relevant [28,29,30]. Effective feature engineering enhances a model’s predictive power and performance [31].

Feature engineering transforms raw data into structured representations for ML, optimizing model performance [27,29,32]. It encompasses feature generation, preprocessing, extraction, and selection. Feature generation derives new attributes from existing data, enhancing interpretability and prediction performance [33]. Preprocessing ensures data consistency through cleaning, transformation, and aggregation, supporting meaningful feature extraction [34]. Feature extraction identifies informative patterns, leveraging techniques like embeddings and named entity recognition [12]. Feature selection refines model inputs by prioritizing relevant attributes, reducing dimensionality and noise [35]. Additionally, domain knowledge-based feature engineering integrates expert insights to improve feature relevance, with approaches such as rule-based engineering and external knowledge incorporation [16,36]. Studies have explored advanced feature engineering strategies, including data curation [12], weak supervision [16], and visualization-driven ideation [20], further enhancing interpretability and predictive power.

Artificial Intelligence (AI) refers to the simulation of human intelligence in computer systems. A key advancement in this field is GenAI, which mimics human creativity and problem solving by learning patterns from existing data [37]. As shown in Table 1, compared to traditional feature engineering techniques, which often rely on domain-specific heuristics or handcrafted rules, GenAI provides a more adaptive approach by leveraging deep learning to dynamically extract meaningful features from unstructured data. While rule-based text mining and sentiment analysis require predefined keywords and lexicons, generative AI can identify implicit signals, sentiment nuances, and context-aware patterns that may be overlooked by static methods [38]. GenAI uses deep learning techniques to generate human-like content in various domains, including text, images, and structured data. Its ability to produce realistic and innovative outputs has made it valuable in multiple industries.

GenAI has revolutionized various tasks by providing models like Generative Adversarial Networks (GANs) [39] and Variational Autoencoders (VAEs) [40]. These models are particularly useful for feature engineering, especially when dealing with unstructured data [41], anomaly detection [42], data augmentation [43], synthetic data generation [43,44], imbalanced data [45,46], and data imputation [41,47]. Combining generative models with traditional feature engineering methods can improve the overall efficiency of machine learning models.

As GenAI continues to evolve, its impact on technology and creativity is expanding. Well-known applications include DALL-E (https://openai.com/dall-e-2 (accessed on 13 April 2025)); VALL-E (https://www.microsoft.com/en-us/research/project/vall-e-x/ (accessed on 13 April 2025)); ProcessGPT [18]; and ChatGPT (https://openai.com/blog/chatgpt (accessed on 13 April 2025)),a widely recognized and conversational AI tool. Introduced by OpenAI in 2018, ChatGPT is based on transformer architectures, which excel in natural language processing (NLP) tasks such as text generation and question answering [48].

Building on our previous research [16] and the increasing influence of GenAI in various industries [49], we propose the use of generative models to improve feature engineering for customer churn prediction. One of the key challenges in this approach is Prompt Engineering (PE), which involves designing strategic queries to maximize the quality of AI-generated responses. To address this challenge, we conducted a comprehensive literature review on customer journey analysis, with a focus on behavioural and cognitive aspects. Using the acquired knowledge, we developed structured prompts and fed them into a generative model. Since these models have inherent limitations [50,51], such as generating incorrect information, we integrated a feedback loop where domain experts verified the reliability of AI-generated content.

Table 1. Comparison of generative AI vs. traditional NLP feature engineering.

Method	Strengths	Limitations
Rule-based text mining	Explicit, interpretable rules	Requires manual rule creation, limited generalization [52]
Sentiment analysis	Captures basic sentiment polarity (positive/negative)	May miss subtle or domain-specific sentiment nuances [53]
Domain-specific lexicon	Provides structured vocabulary for churn indicators	Static and requires regular updates to remain relevant [52]
Generative AI (proposed approach)	Extracts latent features, dynamically adapts to text, and generates new insights	Computationally expensive, potential risk of hallucination [54]

Table 1. Comparison of generative AI vs. traditional NLP feature engineering.

Method	Strengths	Limitations
Rule-based text mining	Explicit, interpretable rules	Requires manual rule creation, limited generalization [52]
Sentiment analysis	Captures basic sentiment polarity (positive/negative)	May miss subtle or domain-specific sentiment nuances [53]
Domain-specific lexicon	Provides structured vocabulary for churn indicators	Static and requires regular updates to remain relevant [52]
Generative AI (proposed approach)	Extracts latent features, dynamically adapts to text, and generates new insights	Computationally expensive, potential risk of hallucination [54]

2.3. Churn-Related Analysis in Various Domains

This section reviews customer churn studies across different industries, focusing on their feature engineering approaches. Churn is commonly defined as inactivity for a long time [55], though the definition varies across businesses due to market competition and flexible service contracts. As an example, in industries such as Internet services (e.g., online banking, education, and entertainment) churn is frequent due to low switching costs and minimal investment requirements [56,57,58].

Most churn prediction models use log data, which record user interactions within a system (https://www.sumologic.com/glossary/log-file/ (accessed on 13 April 2025)). Businesses employ log-based churn models to analyse transaction logs (e.g., purchases and cancellations), communication logs (e.g., emails and calls), and activity logs (e.g., login data) [3]. Various industries, including telecommunication, banking, and insurance, rely on churn analysis to improve customer retention.

In social media churn prediction, Amiri et al. were the first to explore churn in microblogs (e.g., Twitter). Their study suggested that identifying churn-related content enables companies to offer personalized assistance to at-risk customers. Their main feature categories were ‘demographic churn indicators’, ‘content churn indicators’, and ‘context churn indicators’ [6]. For telecom churn analysis, Agrawal et al. found that long-term contract customers were less likely to churn compared to those on short-term contracts. They also noted that high monthly charges increase churn likelihood. Their study used demographic data, subscription details, user billing profile, and tenure as key predictive features [59].

In the banking sector, churn is a major issue due to the availability of multiple service providers [60]. Customers who close all accounts and cease banking relationships pose financial risks to institutions [61]. Studies suggest that frequent credit card use is linked to lower churn rates, with factors such as total transaction amount; transaction frequency; and revolving balance, i.e., the portion of credit card spending that goes unpaid at the end of a billing cycle, being significant predictors [62].

For insurance churn prediction, Zhang et al. proposed a dynamic modelling technique to assess monthly churn likelihood. Their model incorporated features such as partner status, gender, customer lifetime, age, premium amounts, discounts, and insurance type count [63]. They highlighted that discount possession and variations over time were key churn indicators. The study also suggested that external factors like competitor campaigns and final price offers could influence customer retention.

While previous studies have employed sentiment analysis and text mining techniques for churn prediction [8,64], these approaches often focus on individual feature extraction. Our approach builds upon these works by integrating a domain-specific knowledge base enriched with generative AI to enhance feature engineering by capturing features related to customer behavioural and cognitive patterns.

Despite extensive research, there are limited studies leveraging GenAI for feature engineering in churn prediction. Existing works mainly utilize GANs [45,46] and VAEs [65] to address a major challenge in churn analysis, i.e., the imbalance between churned and non-churned data, which can affect prediction accuracy. GANs are employed to generate synthetic customer data that mimic real churn patterns, helping to balance the dataset and improve model performance. Similarly, VAEs capture the underlying distribution of customer behaviours and generate diverse yet realistic synthetic samples, enhancing the robustness of churn prediction models [66].

In this study, we employ GenAI to enhance ChurnKB by integrating features related to customer cognitive status, emotions, and behaviours. This approach aims to improve churn prediction by capturing more meaningful and personalized insights into customer retention.

3. Method

3.1. Developing a Customer ChurnKB

In the first section of the method, a pipeline to construct ChurnKB is proposed. In this pipeline, previous studies on customer journey, cognitive science, and customer behaviour are leveraged to understand the concepts and behavioural patterns associated with customer churn. We also discuss how we utilise GenAI to enhance the structure of the proposed ChurnKB. As illustrated in Figure 1A, the process of developing ChurnKB involves seven steps: (1) a comprehensive literature review, (2) feature identification, (3) taxonomy development, (4) taxonomy validation/modification (Feedback Loop), (5) the development of a sub-instance to phrasal/lexical list connector API, (6) linked lists validation (feedback loop), and (7) the development of a sub-instance score calculator API. Each of these steps is briefly explained in the following sections and discussed in more detail in Section 4.

3.1.1. Comprehensive Literature Review

A review of previous studies on customer experience, customer journey, and customer churn was conducted to understand the emotional and behavioural aspects of churn. This analysis provides a holistic view of the customer experience, identifying potential dissatisfaction sources, pain points, emotions, and behaviours. Customer interactions and communications with a company influence their perceived value, which, in turn, affects customer satisfaction and the decision to continue or discontinue their relationship with the company [1,67].

3.1.2. Feature Identification

The previous step generates a set of churn-related features. Based on this, in the current step, various concepts and sub-concepts, as well as relationships, that may indicate a customer’s propensity to churn are identified. In developing ChurnKB, two primary concepts that suggest customers’ likelihood of leaving a company are identified as their feelings and the resulting behaviours. Among these, dissatisfaction is one of the most significant emotions contributing to churn. Factors such as poor customer service, low product or service quality, ineffective communication, and lack of value for money are commonly associated with dissatisfaction [1].

However, this research requires more detailed emotional and behavioural features related to these factors, which are not comprehensively covered in the existing literature.We use GenAI to extract granular churn-related features. Unlike rule-based NLP, it detects hidden patterns in customer interactions. It adapts to evolving language, making it valuable for churn prediction. GenAI also generates new insights beyond heuristics, improving the understanding of customer dissatisfaction and churn.

By interacting with a generative model, we retrieve detailed features to be incorporated into ChurnKB. For example, following query “In addition to dissatisfaction, what are the primary emotions that might lead customers to leave a company or business?” produces the following outcome:

• Frustration: Frustration arises when customers encounter obstacles, delays, or challenges in their interactions with a company. This emotion may be triggered by poor customer service or unresolved issues.
• Disconnection: Disconnection occurs when customers no longer feel emotionally or personally connected to a brand. This feeling may result from shifts in company values or inconsistent communication.

As another example, the query “What are the main probable behaviours of customers with a tendency to churn, derived from those feelings?” produces the following outcome:

• Reduced Usage: Customers who are dissatisfied or frustrated or feel disconnected from a company are likely to reduce their usage of the company’s products or services.
• Seeking Alternatives: Dissatisfied, frustrated, and disconnected customers are prone to seek alternatives actively. They might research and explore other companies or businesses offering similar products or services.

Continuing the question-answering process with GenAI yields a comprehensive set of features related to customer churn. By integrating insights from customer journey studies with GenAI interactions, we can identify that disconnection might be caused by factors such as no emotional connection and a lack of interest in the product/service. Similarly, frustration could arise from difficulty in using the product/service, a complicated processes, or frequent disruptions or downtime.

Additionally, behaviours like cancelling a contract might result from an intention to churn. These feelings and emotions can be reflected in or predicted by the content of customers’ interactions and communications with the company. Such communications might reveal specific emotions like anger, disgust, and sadness. They might also indicate intent to churn, the customer’s personality type, or negative feedback. Consequently, any combination of these emotions and behaviours could lead to outcomes such as decreased frequency and/or duration of product/service usage, the seeking of alternatives, or negative word-of-mouth communication, eventually resulting in churn.

3.1.3. Taxonomy Development

In the third step, a churn taxonomy is developed based on the features identified in the previous steps. A taxonomy is a hierarchical structure that organises concepts, instances, and their relationships [12,36]. This taxonomy comprises four levels: churn-related concepts (level 1), sub-concepts (level 2), instances (level 3), and sub-instances (level 4). At level 1, the taxonomy includes two primary concepts: customer feelings and customer behaviour. Level 2 includes sub-concepts such as dissatisfaction and reduced usage, while level 3 includes instances such as poor customer service and decreased frequency of use and sub-instances such as anger and disgust associated with the feelings and behaviours of customers likely to churn.

3.1.4. Taxonomy Validation/Modification (Feedback Loop)

Generative AI is known to hallucinate or generate misleading information. To verify the quality of AI-generated features to enhance the taxonomy structure, the fourth step implements a feedback loop to collect domain experts’ insights on the relevance of the included elements. A questionnaire is designed and reviewed by an expert for relevance. She also suggests a threshold for modifying the taxonomy based on expert feedback. If a certain percentage marks an item as ‘irrelevant’, it is removed; otherwise, it stays. This ensures only elements with strong expert consensus are retained, refining the taxonomy for churn analysis.

3.1.5. Developing a Sub-Instance to Phrasal/Lexical List Connector API

In textual documents, there is a clear link between certain words and phrases and the presence of distinctive content or emotion (e.g., profanity [68], self-criticism [69], negative emotion [70], and suicidal ideation or attempts [71]). Consequently, we propose the use of lexical/phrasal resources to identify customers who are experiencing emotions such as dissatisfaction or frustration. It could be done by analysing text from various forms of communication between customers and a business (e.g., voice call transcripts, emails, feedback, or posts on business-related online platforms).

In this step, each sub-instance (level-4) is provided by a ‘sub-instance to phrasal/lexical list connector API’. Such APIs are developed and embedded into the corresponding ChurnKB sub-instances. Each API is an ML algorithm that connects the corresponding sub-instance to the appropriate lexical or phrasal category.

There are several lexical sources available, including the LIWC2015 lexicon [72], which contains categories related to churn, such as anger, sadness, disgust, and fear, along with the corresponding group of words related to each category. Additionally, the NRC emotional lexicon [73] includes categories like negative emotion and their associated words. On the other hand, since some churn-related sub-instances, such as ask for compensation, frustration, neglect, and intent to churn, are not included in either of the available lexical sources, we leveraged GenAI to help us create lists words/phrases that are related to a specific sub-instance indicative of a customer’s propensity to churn, e.g., I’m leaving, discontinue, and This is unacceptable.

All of the churn-related categories included in the lexical sources are joined together to create the main database in the form of an Excel file. As shown on Figure 2, each of the columns of the Excel file acts as a category related to one of the churn sub-instances. For example, there is a special API that connects the sadness sub-instance to the sadness category. Another API links the intent to churn sub-instance to the list of words/phrases showing the intention to churn, etc.

3.1.6. Linked List Validation (Feedback Loop)

Similar to Section 3.1.4, in the sixth step, we implement a feedback loop to gather domain experts’ opinions on the relevance of the elements included at the sub-instance level (i.e., Level 4) of the taxonomy, as well as the corresponding lists of words/phrases. A questionnaire is used to facilitate this. An expert is also asked to confirm the appropriateness of the questions before distribution among other experts.

3.1.7. Developing a Sub-Instance Score Calculator API

In the final step, several APIs are developed, each corresponding to a sub-instance in the customer churn taxonomy. These APIs employ ML algorithms to calculate sub-instance scores (e.g., an anger score) by analysing the customer’s chat logs (i.e., textual input data). The resulting scores, represented as cosine similarity values, form a key component of our proposed ChurnKB, facilitating automated knowledge-based analysis. Algorithm 1 outlines the process of calculating sub-instance scores for APIs using NLP and text mining techniques such as TF-IDF, regular expressions, word tokenization, and stemming. The input for this process is curated textual data, with details of the curation process provided in Section 3.2.

Algorithm 1 The process of sub-instance score calculation in APIs that work based on textual data mining techniques.

Input: Curated input textual data Output: Calculated sub-instance scores The ChurnKB sub-instance; Linked sub-instance-related lexicons; for Each sub-instance in the ChurnKB do    Reading the curated input textual data;    Reading the linked lexicon;    Applying proper text mining techniques (e.g., Cosine Similarity Calculation); end for

We employ term frequency-inverse document frequency (TFIDF) and cosine similarity to quantify textual features related to customer churn. TF-IDF identifies key terms in customer interactions, while cosine similarity measures textual similarity between customer-generated content and churn-related lexicons. These methods allow us to capture churn signals efficiently without requiring extensive domain-specific rules.

In the realm of text classification, it serves to quantify the similarity between two documents. The range of values spans from 0 to 1, with 0 denoting no similarity and 1 signifying identical documents. For a pair of documents, such as “doc1” and “doc2”, the similarity is expressed through the following formula:

$$\mathrm{Cosine}\mathrm{Sim}(\mathrm{doc}1,\mathrm{doc}2)={\displaystyle \frac{\mathrm{doc}1·\mathrm{doc}2}{\left|\right|\mathrm{doc}1\left|\right|\times \left|\right|\mathrm{doc}2\left|\right|}}$$

(1)

which is equal to:

$$\frac{{\displaystyle \sum _{i=1}^n}{A}_i{B}_i}{\sqrt{{\displaystyle \sum _{i=1}^n}{A}_i^2}\times \sqrt{{\displaystyle \sum _{i=1}^n}{B}_i^2}}$$

(2)

where $A_i$ and $B_i$ represent the components of vectors doc1 and doc2, respectively [74]. In this study, doc1 and doc2 correspond to a customer-related document and a lexical/phrasal source document, respectively.

The final version of the churn taxonomy which is embedded with sub-instance score calculator APIs, is called ChurnKB. Figure 1B illustrates a snapshot of the customer ChurnKB, indicative of a small part of the inter-connected feelings and behaviours of customers at risk of churn. ChurnKB contains different concepts and sub-concepts associated with churn-related feelings and behaviours, instances and sub-instances, and the relationships among them. Experiencing various feelings could end in different behaviours in a customer. In this figure, solid connections mainly indicate relationships between concepts, sub-concepts, and corresponding instances. On the other hand, dotted arrows indicate the connection between the instances and the corresponding sub-instances. Examples of existing relationships in ChurnKB include the following: poor customer service could cause customers to feel unimportant, which, in turn, may lead to a lack of interest in the service/product. As a reflection of this lack of interest, different changes in customers’ behaviour or consumption patterns may arise, such as decreased frequency of use, decreased duration of use, decreased spend, or decreased purchase frequency. Additionally, a sense of unimportance could directly contribute to decreased spend or decreased purchase frequency.

3.2. Developing a Knowledge Base-Enhanced Classifier for Identifying Customer Churn-Related Patterns

This work enhances ML classification using ChurnKB-driven features. By combining cognitive science and data science, key features of customer behaviour are identified. ChurnKB’s insights on customer feelings and behaviours are used for feature extraction from textual data.

Figure 3 shows the pipeline for a KB-enhanced churn classifier. The method adopts a scalable, service-oriented architecture, allowing for seamless integration of new analytical features.

• The first step involves data curation, defined as the process of transforming raw data into contextualised data and knowledge [12], thereby enhancing the efficiency of ML algorithms. Drawing inspiration from two recent studies [12,75], it is crucial to prepare and curate raw textual data before proceeding with further analysis. The curation process involves three steps: (i) data cleaning, (ii) feature extraction, and (iii) feature enrichment. Customer interactions include communications such as reviews and chat logs in a desired period. Cleaning includes removing punctuation, stop words, and special characters while normalizing text. Next, feature extraction applies part-of-speech tagging, named entity recognition, and keyword identification to capture key linguistic patterns. Finally, Feature enrichment incorporates synonym expansion (e.g., WordNet [76]) and stemming to standardize word forms, enhancing the dataset for churn prediction analysis.
• Each customer feeling and behaviour represented in ChurnKB (e.g., dissatisfaction) may be caused by various reasons (e.g., poor customer service). These reasons, in turn, may lead to several feelings or behaviours (e.g., anger), which are considered sub-instances within ChurnKB. In the second step, as outlined in Algorithm 2, these sub-instances are linked to the curated data and extracted features by initialising an empty list for each instance. Extracted features (e.g., stemmed keywords or phrases) are then added to the corresponding sub-instance list.
• In the third step, the extracted features from the previous step are used as input for the corresponding sub-instance-related APIs, enabling the score calculation process for each sub-instance.
• In the fourth step, the calculated scores are used as input features for the churn classifier. ML algorithms like Random Forest, Logistic Regression, and XGBoost can be applied. The model is trained as a binary classifier to predict if a customer is likely to churn. A feedback loop evaluates the KB’s performance, as described in Section 4.1.

Algorithm 2 Linking extracted features from input textual data to the sub-instances in ChurnKB.

Input: Curated input textual data Output: List of features that are linked to the sub-instances of the ChurnKB Extract features from textual data; Churn-related sub-instances = Set up and Array for sub-instances in the ChurnKB; % (e.g., Negative Feedback and Anger, etc.) for Each Churn-related sub-instances do    Generate an empty list;    for Each feature in extracted-feature do      Add the feature to the corresponding sub-instance list;      Link extracted features to the sub-instances in ChurnKB;    end for end for

4. Evaluation

To address potential limitations of generative models [50,51] and assess the effectiveness of our ChurnKB, we conducted a survey and gathered feedback from experts in the field. Additionally, we evaluated our approach by applying features derived from ChurnKB to customer churn classification tasks using various ML algorithms and evaluation metrics, as detailed in the following sections.

4.1. Evaluating the ChurnKB Development Approach

We formulated four hypotheses to test and validate the information in the knowledge base and the proposed method. For this purpose, we used an iterative approach that closely mirrors reinforcement learning with human feedback.

To test the hypotheses, we designed and distributed anonymous questionnaires consisting of multiple-choice items. Participants were asked to indicate their level of agreement with each statement using a Likert scale ranging from 1 to 5 (1: irrelevant; 2: weakly relevant; 3: neutral; 4: relevant; 5: strongly relevant). No identifying information was collected, and the identities of participants remained undisclosed throughout the study. This approach focuses on refining the knowledge base through multiple feedback loops, gathering domain experts’ opinions on the relevance of the taxonomy’s elements.

Initially, the included questions were reviewed by domain experts to ensure their relevance to the churn problem. Once the questionnaires were finalized, it was recommended by the experts that a 20% threshold be applied in the analysis of the collected responses. All responses were provided anonymously, and no identifying information about the participants was collected or disclosed.

On the questionnaire, an explanation of its purpose and goals is first provided. This is followed by a target hypothesis. Next, a snapshot of the ChurnKB components related to the hypothesis questions is presented. Finally, the corresponding questions are provided.

Four hypotheses, designed to validate our approach on developing ChurnKB, are outlined as below:

• H1: The structure of the initial churn taxonomy, including concepts and instances, etc., is relevant to customer churn.
• H2: The use of the pronoun I), absolute words, and certainty words by customers in their communications or feedback is relevant to identifying customers’ churn-related cognitive and behavioural patterns.
• H3: Churn-related features (i.e., sub-instances) and the corresponding phrasal/lexical list are relevant to identifying customers’ churn-related cognitive and behavioural patterns.
• H4: The developed classifier leads to reliable results derived from the application of ChurnKB to enhance the feature engineering phase.

The initial version of the churn taxonomy was developed based on customer journey studies and GenAI capabilities. To test H1, we presented various parts of the initial churn taxonomy, including different concepts and instances, to the experts. We asked them to evaluate the relevance of each part to customer churn. An example question designed for this purpose is shown in Figure 4A.

To test H2, various questions were posed to validate an assumption, leading to modifications in the initial churn taxonomy. During the development of the initial churn taxonomy, we noted that several sub-instances, such as sadness and negative emotions, were present in a KB developed in previous work [16]. This KB includes different feelings and mental statuses that individuals suffering from mental health issues, such as depression, are likely to experience. Based on this, the use of terms like I, absolute words, and certainty words in communication could reflect specific psychological and emotional statuses.

Motivated by that understanding, we hypothesised that dissatisfied customers may experience a negative emotional status (e.g., similar to depressed individuals) that confirms their decision to leave a company. Therefore, those individuals may use the pronoun I, absolute words, and certainty words in their feedback, reviews, and/or interactions with a company to emphasise a specific decision of theirs. Hence, we came up with the idea of adding those three as sub-instances of negative word of mouth, assumed to be probable behavioural signs of unhappy customers who wish to discourage others from using the product/service.

To validate the H2, we asked experts about their opinions of adding those sub-instances to the latest version of the churn taxonomy. As shown in Figure 4E, we showed experts part of the taxonomy with those three sub-instances added to it. An example question related to the relevance of using absolute words to customer dissatisfaction is illustrated on Figure 4B. Analysis of the experts’ responses showed that the majority of experts found all of the three sub-instances to be relevant to the churn taxonomy. Hence, we incorporated them into it.

To test H3, we focused on questions related to the sub-instances (level 4), as well as the relevance of the generated lists, which include related words and phrases corresponding to some of the sub-instances. Figure 4C illustrates an example question asked in this regard. After analysing expert responses, as shown in Figure 5A,B, we noticed that there were two sub-instances that received a corresponding ‘irrelevant’ rate of more than 20% from the experts. Hence, we removed them. These sub-instances are embarrassment and envy. Embarrassment refers to feeling self-conscious or ashamed about a situation. Envy, on the other hand, refers to the feeling of jealousy towards others who might have a better experience.

To test H4, as shown in Figure 4D, we asked questions of experts to get their ideas regarding the reasonableness/reliability of churn classifier performance based on the corresponding results derived by applying ChurnKB to enhance feature engineering. For this purpose, examples of customer chat logs and the churn status of the corresponding customers were shown to the experts. The analytical results of the classifier are were also presented to the experts. They were asked about their ideas regarding the results. In Section 5, more discussion is provided regarding the final evaluation results of the four hypotheses that were tested in this study.

4.2. Evaluating Churn Classifier Performance

4.2.1. Data

For our experiments and to evaluate the effectiveness of applying the feature engineering derived from ChurnKB, we need a churn-related dataset that includes both textual and other data types, such as categorical and numerical data. The only available dataset that meets these criteria is the churn prediction with text and interpretability dataset (https://github.com/aws-samples/churn-prediction-with-text-and-interpretability/blob/main/README.md (accessed on 13 April 2025)). While the dataset is specific to the telecom industry, the underlying methodology of ChurnKB is designed to be adaptable across various sectors. Customer churn behaviours, though differing in industry-specific manifestations, share common cognitive and behavioural indicators, such as dissatisfaction, reduced engagement, and negative sentiment patterns, that ChurnKB could potentially capture.

The dataset contains 3333 rows and 21 columns. Of these, 2850 samples (85.50%) belong to the ‘churn = no’ class, while 483 samples (14.50%) belong to the ‘churn = yes’ class. Each column represents a different feature type. The features include categorical variables such as state, international plan, and voice-mail plan, and numerical variables like account length and total length of day calls. Additionally, the dataset includes a specific column for text data, which contains chat logs between customers and agents.

4.2.2. Evaluation Metrics

This study employs the F1 score as the primary evaluation metric, given its effectiveness in handling imbalanced datasets. The F1 score balances precision and recall, which is crucial for accurately identifying churners and optimizing retention strategies. Incorporating both precision and recall ensures reliable and actionable predictions for churn mitigation. Precision measures how many predicted churners actually churned, while recall assesses how well the model captures actual churn cases. The F1 score is calculated as follows:

$$F1=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall},}}$$

where

$$\mathrm{Precision}={\displaystyle \frac{\mathrm{Number}\mathrm{of}\mathrm{correctly}\mathrm{predicted}\mathrm{churn}\mathrm{customers}}{\mathrm{Total}\mathrm{number}\mathrm{of}\mathrm{customers}\mathrm{predicted}\mathrm{as}\mathrm{churn},}}$$

whereas

$$\mathrm{Recall}={\displaystyle \frac{\mathrm{Number}\mathrm{of}\mathrm{correctly}\mathrm{predicted}\mathrm{churn}\mathrm{customers}}{\mathrm{Total}\mathrm{number}\mathrm{of}\mathrm{actual}\mathrm{churn}\mathrm{customers}.}}$$

4.2.3. Results

For experimentation and evaluation, the data were processed through a PyCaret (https://github.com/pycaret/pycaret (accessed on 13 April 2025)) pipeline to compare different machine learning models and assess the effect of ChurnKB-enabled features on their analytical performance. PyCaret does not provide built-in support for deep learning models such as TensorFlow or PyTorch-based neural networks in its standard modules. However, it includes various machine learning models via scikit-learn, XGBoost, LightGBM, CatBoost, logistic regression, and decision trees. To further validate the effectiveness of ChurnKB features, we also experimented separately with neural network-based models. For performance comparison, we report only the three top-performing models in each scenario due to space limitations, along with Multilayer Perceptron (MLP) and TabNet, which are designed for tabular data. These deep learning models are more suitable for tabular data than traditional recurrent architectures like Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRUs), which are optimized for sequential data. In addition, we utilised 31 sub-instances from ChurnKB, which can be supplied with an appropriate lexical source.

We compared three scenarios, i.e., (A) a dataset with only numerical data, (B) a dataset with only textual data, and (C) a dataset that includes both textual and numerical data. The evaluation results are reported in

Table 2. Model performance comparison across different data scenarios, organized in ascending order, with XGBoost consistently appearing in all scenarios and showing performance improvement from A to C.

Model	Recall	Precision	F1 Score
(A) Scenario One: Numerical Data Only.
Tabnet	0.5000	0.7800	0.6094
Gradient Boosting	0.4970	0.7446	0.5894
XGBoost	0.5005	0.6830	0.5752
MLP	0.5065	0.6190	0.5571
LightGBM	0.4823	0.6711	0.5567
(B) Scenario Two: Textual Data Only.
Linear Discriminant Analysis	0.6308	0.7966	0.7021
MLP	0.6410	0.7463	0.6897
AdaBoost	0.6229	0.7661	0.6850
Tabnet	0.6301	0.7188	0.6715
XGBoost	0.6057	0.7565	0.6698
(C) Scenario Three: Numerical + Textual Data.
XGBoost	0.7085	0.8946	0.7891
LightGBM	0.7013	0.8963	0.7842
AdaBoost	0.7193	0.8531	0.7754
Tabnet	06623	0.7969	0.7234
MLP	0.6494	0.7812	0.7092

(A), (B), and (C), respectively. Among various classification models, XGBoost appears consistently across all comparisons, with its performance improving when textual features are included. In the first scenario, XGBoost achieves an F1 score of 0.5752, while in the second scenario, its performance improves to 0.6698. The most notable performance boost occurs in the third scenario, where the inclusion of both numerical and textual features leads to an F1 score of 0.7891. These results highlight the positive impact of textual features on predictive performance. Given XGBoost’s strong and consistent performance across all settings, we selected it as the target classifier for our research.

All the comparison results indicate that applying ChurnKB-enabled feature engineering positively impacts the performance of different classifiers, specifically the XGBoost model. While MLP [77] and TabNet [78] demonstrate competitive performance, XGBoost remains the highest-performing model across all evaluation metrics, including F1 score. The superior performance of XGBoost can be attributed to its ability to efficiently capturing complex feature interactions. Additionally, XGBoost requires fewer computational resources and less training time compared to deep learning models, making it a more practical choice for real-world churn prediction applications.

4.2.4. Statistical Validation

To verify that ChurnKB’s improvements are not due to random variations, a paired t-test was conducted.

Table 3. Paired t-test results comparing model performance with and without ChurnKB-derived features across multiple evaluation metrics. Statistically significant improvements (p-value < 0.05) in all metrics suggest that ChurnKB features enhance model performance.

Metric	T-Statistic	p-Value	Metric	T-Statistic	p-Value
AUC	5.297949	0.000253	Recall	7.819004	0.000008
Precision	6.428685	0.000049	F1	8.786979	0.000003

presents the results, comparing model performance with and without ChurnKB-derived features. The T-statistic measures the magnitude of performance differences, while the p-value indicates statistical significance. A larger absolute T-statistic strengthens evidence against random chance, and a smaller p-value (typically < 0.05) suggests the improvement is unlikely to be random. The results show statistically significant gains across all metrics, confirming that ChurnKB-derived features enhance model performance.

For the purpose of conducting an ablation study and measuring the impact of ChurnKB-enabled features, we considered the results derived from XGBoost, which demonstrated the strongest predictive performance.

Table 4. Ablation study results, measuring the impact of ChurnKB-enabled features.

Model Setup	F1 Score	Improvement over Baseline
Baseline (A)—numerical Only	0.5752	—
ChurnKB-enabled (B)—textual features only	0.6698	+9.4%
Full model (C)—numerical + textual features	0.7891	+21.3%

presents the impact of ChurnKB-enhanced features in three experimental setups (A) a baseline model, using only numerical features (without ChurnKB); (B) a ChurnKB-enabled model, using only ChurnKB-derived textual features (lexicon-based churn indicators from ChurnKB); and (C) a full model, combining both numerical and ChurnKB-enhanced features.

The ablation study confirms that ChurnKB contributes significantly to churn prediction performance. The ChurnKB-derived textual features alone improve the F1 score by +9.4% compared to the baseline. When combined with numerical features, ChurnKB enhances overall performance by +21.3%, demonstrating its effectiveness in feature enrichment.

5. Discussion

As illustrated in Figure 6, ChurnKB was validated in terms of its elements, content, and performance, leveraging human expertise for refinement. Participants’ responses support Hypotheses 1–4. Part (i) confirms that all participants find the components of the developed knowledge base relevant to customer churn. In Part (ii), ten participants recognize the three sub-instances (i.e., the use of the pronoun I, absolute words, and certainty words) as relevant to churn, while the rest perceive them as neutral. Since the majority consider these sub-instances relevant, we decided to incorporate them into ChurnKB.

Furthermore, Part (iii) shows that all experts, except one, recognize the relevance of the sub-instances used in the developed classifier for identifying churn-related behavioural patterns. One expert, however, considers them only weakly relevant. Part (iv), which assesses the churn classifier’s outcomes using the modified version of ChurnKB, indicates that all but one participant find the results relevant.

However, it is essential to note that there are some validity concerns that should be considered before drawing conclusive results from our study. Although the overall survey results support Hypotheses 1, 2, 3, and 4, there is still room for improvement in the proposed approach—for instance, considering weights for some of the concepts and instances to be indicative of being more influential with respect to churn decisions of customers. These matters will be further investigated in our future works.

On the other hand, the classifier was developed and evaluated using the extracted features from ChurnKB. Although the study is currently limited to a telecom dataset, the framework can be applied to other industries, such as banking, insurance, and e-commerce. For instance, in banking, customer churn often manifests through declining transaction frequency, reduced credit card usage, and negative sentiment in customer support interactions. In insurance, churn indicators include policy cancellations, inquiries about competitor policies, and dissatisfaction with claim processes. ChurnKB can be extended to these domains by adapting its feature extraction pipeline to industry-specific text sources (e.g., customer service transcripts, complaint records, and online reviews). Additionally, integrating industry-specific lexicons and knowledge bases would further refine ChurnKB’s feature representation, enhancing its applicability across sectors.

6. Conclusions

In this study, we introduced ChurnKB, a knowledge base for churn analysis, integrating customer-generated textual data, such as chat logs, with churn analysis. By doing so, it captures cognitive and behavioural patterns indicative of churn risk. To enhance ChurnKB, we developed an interactive GenAI-enabled feature engineering approach, leveraging generative AI to refine its structure and content.

We validated our approach through expert evaluation, using a structured questionnaire to assess ChurnKB’s structure, content, and analytical outcomes. The results confirm its relevance and potential in improving churn prediction. However, some limitations were identified. Some features had weak relevance for detecting churn-related patterns, and linguistic indicators like I, absolute words, and certainty words were seen as neutral.

Despite these challenges, integrating customer–company interactions into churn analysis improves upon traditional approaches based on demographic and product usage features. Our results demonstrate the effectiveness of ChurnKB-derived features, significantly boosting model performance, with XGBoost’s F1 score increasing from 0.5752 to 0.7891 when combining textual and numerical data. This underscores the value of cognitive and behavioural insights in churn analysis. Future work will refine ChurnKB by addressing limitations and optimizing feature selection for further accuracy improvements.

Future research could explore ChurnKB’s application in banking, insurance, and e-commerce to assess its generalizability. Integrating real-time interactions, such as live chat and social media feedback, may enhance early churn detection. Additionally, explainable AI techniques could improve transparency and help businesses justify predictions. AI-driven knowledge bases like ChurnKB may shift churn management from reactive to proactive. They could also enhance customer experience and automate large-scale churn risk detection. These advancements may lead to more effective and responsible business decision making.