Multi-Stage Data-Driven Framework for Customer Journey Optimization and Operational Resilience

Abstract

Optimizing customer journeys is a critical challenge in e-commerce and financial services, attracting attention from marketing, operations research, and business analytics. Traditional customer analytics models, such as rule-based segmentation and regression models, rely heavily on structured transactional data, limiting their ability to capture latent behavioral patterns and adapt to multi-channel dynamics. These models often struggle to integrate unstructured data sources, failing to provide adaptive, personalized insights. To address these limitations, this study proposes a multi-stage data-driven framework integrating latent Dirichlet allocation (LDA) for behavioral insights, deep learning for predictive modeling, and heuristic algorithms for adaptive decision-making. Empirical validation using Taiwanese financial institution data shows a 15% improvement in predictive accuracy compared to traditional machine-learning models, significantly enhancing customer lifetime value (CLV) predictions and multi-channel resource allocation. This research highlights the practical value of integrating structured and unstructured data for improving customer analytics. Our framework leverages LDA to extract behavioral patterns from customer interactions, enriching predictive models and enhancing real-time decision-making in financial services. Robustness checks confirm the scalability and adaptability of this approach, offering a data-driven strategy for long-term value optimization in dynamic digital ecosystems.

1. Introduction

In today’s digital economy, optimizing customer journeys has become a central challenge for businesses, particularly in e-commerce and financial services. As customer interactions become increasingly multi-channel, companies must leverage data-driven strategies to enhance customer engagement, resource allocation, and long-term value creation. Traditional customer analytics models, such as rule-based segmentation and regression-based forecasting, primarily rely on structured transactional data, which limits their ability to capture latent behavioral patterns and adapt to complex customer interactions across digital platforms.

The COVID-19 pandemic accelerated digital transformation, prompting increased investment in online platforms to address changing consumer behaviors [1]. This shift highlighted a broader trend in which companies increasingly engage customers through diverse online and offline touchpoints, such as social media, paid advertisements, and digital marketplaces, to enhance customer experience [2]. Effective customer journey management is essential in this transformation, requiring businesses to leverage data-driven insights to tailor interactions and improve conversion rates [3]. Furthermore, strategic resource allocation across digital channels, supported by AI-driven analytics, plays a crucial role in maximizing customer lifetime value (CLV) and fostering customer loyalty [4].

To successfully navigate the complexity of modern customer engagement, businesses require robust knowledge systems and decision support mechanisms that facilitate real-time analysis and data-driven decision-making. By integrating predictive analytics, machine learning, and optimization techniques, companies can develop adaptive frameworks that enhance customer relationship management, personalized marketing, and multi-channel coordination. This perspective aligns with research in intelligent decision support systems, which emphasize data-driven knowledge creation to improve business agility and responsiveness [5].

Despite advances in artificial intelligence (AI) and predictive analytics, existing models often fail to fully capture the complexity of real-time, multi-channel customer journeys. Traditional customer analytics methods, such as logistic regression, survival analysis, and optimization-based models, have long been used to predict customer churn, purchase frequency, and lifetime value. These methods are computationally efficient and interpretable, but they struggle to incorporate unstructured behavioral data and high-dimensional interaction patterns found in multi-channel commerce.

Conversely, AI-driven customer analytics leverage machine-learning and deep-learning techniques to uncover patterns from high-dimensional data sources. Supervised learning methods such as gradient boosting machines (GBMs), artificial neural networks (ANNs), and recurrent neural networks (RNNs) have demonstrated strong predictive capabilities in customer behavior forecasting [6]. However, despite their strong predictive power, most AI-driven approaches lack integration with optimization modeling techniques, making it difficult to balance accuracy with actionable business insights.

To bridge this gap, this study introduces a multi-stage data-driven framework that integrates latent Dirichlet allocation (LDA) for extracting behavioral insights from unstructured data, deep learning for predictive modeling, and optimization modeling techniques for resource allocation. By leveraging these techniques, our framework enhances customer journey optimization through improved multi-channel strategy formulation, real-time personalization, and CLV prediction. We validate our approach using real-world transaction data from a major Taiwanese financial institution, demonstrating its capacity to refine customer segmentation, optimize online pipeline planning, and improve data-driven decision-making.

Unlike traditional customer analytics models, which rely on predefined statistical assumptions, our framework incorporates unsupervised learning (LDA) to uncover latent customer segments, while deep-learning models enhance predictive accuracy and adaptability. Additionally, we introduce a heuristic optimization approach based on the binary differential evolution (BDE) algorithm, which dynamically adjusts marketing resource allocation under budget constraints [7,8,9]. By combining optimization modeling, deep learning, and heuristic algorithms, our framework ensures that customer journey predictions are not only accurate but also actionable within business environments.

This research contributes to the growing literature on knowledge systems and decision support frameworks by demonstrating how AI and heuristic techniques can be integrated into a comprehensive data-driven approach for customer journey optimization. By integrating deep learning, topic modeling, and resource optimization, this study provides a scalable and intelligent framework that enables firms to translate consumer insights into effective business strategies [5].

This study constructs an optimized customer journey recommendation mechanism for e-commerce, based on the three-stage customer purchase process, with the goal of enhancing the operational resilience of enterprises. The research begins by reviewing existing recommendation methods in the literature. While traditional recommendation systems often rely on collaborative filtering or content-based approaches that primarily use structured historical transaction data, the proposed model innovatively integrates both unstructured customer reviews and multi-channel behavioral data. By employing topic modeling, machine-learning techniques, and heuristic algorithms, the model goes beyond product recommendation to construct a dynamic, stage-wise recommendation mechanism that aligns customer actions with marketing strategies. This approach not only enables the calculation of channel conversion values at different stages but also facilitates real-time, actionable marketing recommendations tailored to specific customer journeys.

Importantly, the proposed model addresses several key limitations in traditional recommendation systems, such as human bias, personalization constraints, data sparsity, and cold-start problems. Through a thematic analysis, it extracts features from unstructured text, compensating for the lack of structured data often seen in practice. It further leverages machine learning to forecast customer behavior and conversion value, enhancing the system’s ability to evaluate new customers. Moreover, unlike most recommendation systems that stop at suggestion generation, this model incorporates optimization techniques to recommend budget allocations and marketing channels under enterprise constraints, transforming it into a prescriptive recommendation system. Therefore, this study not only complements but also extends the current research in recommendation systems by integrating predictive modeling with optimization-based decision support, contributing a novel framework tailored for multi-stage customer journey management in e-commerce.

The paper is structured as follows: Section 2 reviews the materials and conceptual framework related to our study, providing the analytical foundation. Section 3 systematizes the architecture, detailing the research methodologies and optimization modeling techniques. Section 4 explains the results of the analysis, illustrating the application of this process in real-world scenarios. We present the main findings with practical examples to demonstrate the framework’s effectiveness. Section 5 summarizes the discussion, and Section 6 highlights the conclusions, key contributions, and future research directions.

2. Literature Review

With the continuous advancement of digital platforms, customer journey management has evolved from traditional single-channel interactions into a highly dynamic multi-channel ecosystem. Early customer experience assessments primarily relied on structured survey data, such as net promoter scores (NPSs) and customer satisfaction indices. These methods provided static and retrospective analyses that quantified customer engagement but struggled to capture latent behavioral patterns and lacked adaptability in fast-changing digital environments. Consequently, businesses have increasingly turned to unstructured data sources, including online reviews, social media sentiment analysis, and behavioral tracking, to gain a more comprehensive understanding of customer journeys.

The integration of structured and unstructured data has driven the shift toward data-driven customer journey analytics. Existing studies have explored various techniques, ranging from text mining and natural language processing (NLP) to advanced predictive modeling, enabling firms to personalize customer interactions and optimize decision-making. However, most research focuses on either improving predictive accuracy or behavioral segmentation, with limited emphasis on how these insights translate into strategic decision support. The challenge remains in constructing a framework that not only extracts actionable customer insights but also incorporates them into an adaptive decision-making process that informs marketing, sales, and resource allocation.

2.1. Theoretical Framework for Multi-Channel Customer Journey Management

The increasing complexity of customer interactions necessitates a structured framework that segments different phases of the customer journey while incorporating real-time feedback mechanisms. Traditional models typically divide customer interactions into three stages, pre-purchase, purchase, and post-purchase, each representing distinct decision-making processes and engagement patterns. In the pre-purchase stage, customers seek information and evaluate alternatives. While traditional structured surveys and focus groups can capture customer needs and preferences, they often fail to reflect real-time behavioral shifts. In contrast, NLP-based semantic analysis and topic modeling provide a more granular understanding of consumer sentiment and emerging trends, offering real-time market insights.

During the purchase stage, customer decision-making is influenced by a combination of historical behavior and real-time contextual factors. Traditional statistical models, such as logistic regression and survival analysis, have long been employed to predict conversion likelihood. However, these models struggle to accommodate the complexity of multi-channel interactions and the interplay of different touchpoints. Recent advancements in deep learning, including sequence-based and transformer models, have demonstrated superior capabilities in recognizing cross-channel behavioral patterns, thereby enhancing predictive accuracy. However, these methods often rely on black-box models, making it difficult for businesses to derive interpretable decision-making insights, which remains a significant challenge for practitioners.

The post-purchase stage focuses on optimizing customer lifetime value (CLV) through retention strategies, cross-selling, and personalized engagement. Traditionally, retention models rely on recency, frequency, monetary (RFM) metrics to estimate customer value. However, these models often fail to account for behavioral dynamics and changing market conditions. A dynamic feedback loop ensures that businesses can continuously refine marketing and service strategies based on the latest customer interaction data, improving both customer satisfaction and overall business performance.

Figure 1 conceptualizes this three-stage framework, illustrating how iterative data processing and analysis enhance decision-making. Unlike conventional CRM systems that primarily rely on historical transaction data, this framework integrates real-time analytics and multi-channel feedback, enabling dynamic customer journey management. However, a key limitation of existing models is their lack of an integrated decision-support mechanism that connects behavioral analytics with strategic resource allocation. Addressing this gap requires an approach that not only predicts customer actions but also optimizes decision-making processes to improve business outcomes.

2.2. Research Scheme for Customer Journey Optimization

Traditional customer journey models face several limitations, particularly in handling high-dimensional data. Most conventional approaches, such as the Buy-Till-You-Die (BTYD) model, primarily rely on recency (the time since the last purchase) and frequency (the number of purchases) to predict customer behavior. While effective for low-dimensional data, these models often fail to account for customer heterogeneity, making them less applicable in dynamic multi-channel environments. Additionally, while BTYD models have been widely used for transaction-based predictions in e-commerce, they are often insufficient for higher-level decision-making tasks, such as optimizing resource allocation or enhancing real-time customer service strategies.

Recent developments in machine learning have introduced more sophisticated approaches to customer journey modeling. Techniques such as gradient boosting machines (GBMs) and recurrent neural networks (RNNs) enable businesses to process high-dimensional and sequential data, significantly improving predictive accuracy. However, machine-learning models often suffer from poor interpretability, making it challenging to translate predictive insights into actionable business decisions. In response to this, researchers have begun integrating optimization frameworks with machine-learning methodologies to combine the transparency of decision models with the predictive power of data-driven techniques. Meanwhile, emerging NLP technologies such as BERT, LDA, and GPT have enabled businesses to analyze unstructured data more effectively, addressing the limitations of traditional customer feedback mechanisms. However, despite these advancements, system integration remains a major challenge, particularly in terms of data scalability and computational feasibility.

This study proposes a novel hybrid modeling approach that integrates heuristic optimization algorithms with LDA-based topic modeling and sequence regression structures in machine learning. By incorporating natural language semantic analysis and data-driven decision-making, the proposed framework enhances both the precision and applicability of customer journey analytics. The integration of heuristic optimization enables real-time resource allocation, ensuring that businesses can dynamically adjust marketing and service strategies based on evolving customer behaviors. Compared to traditional customer analysis methods, this framework is more adaptable to high-dimensional data and expands the practical applications of customer behavior modeling, particularly in e-commerce contexts.

By embedding this research scheme into the three-stage customer journey framework, this study ensures that predictive modeling and heuristic optimization are not merely technical advancements but are directly aligned with strategic decision-making processes. Unlike conventional approaches that separate behavioral analysis from operational decision-making, this framework provides a unified structure that integrates NLP-driven insights with machine-learning-based forecasting and optimization modeling. This ensures that businesses can not only enhance their predictive accuracy but also develop a more interpretable and actionable decision-support system. Through this integration, the study contributes to both the theoretical and practical advancements in customer journey management, bridging the gap between data-driven analytics and strategic business implementation. The specific research framework is shown in Figure 2. Research scheme.

3. Research Methods

This study adopts a robust mixed-methods approach, blending predictive and exploratory techniques to model complex e-commerce customer behaviors. By combining LDA for topic modeling with machine-learning algorithms such as LightGBM, LSTM, and CNN-LSTM, the study effectively captures both structured and unstructured dimensions of customer data. LDA identifies latent themes in customer feedback, revealing essential insights into consumer needs that enrich predictive modeling. Predictive methods, meanwhile, leverage these insights to forecast customer actions and conversion potential, allowing businesses to design data-driven, customized engagement strategies.

Data preprocessing was conducted using a rigorous ETL process, ensuring consistency across datasets. This involved imputation for missing values, one-hot encoding for categorical variables, and balancing techniques to address data sparsity. To improve model reliability, the study applied feature selection and boosting techniques, fine-tuning the data for machine learning.

The analysis proceeded in three distinct stages: first, LDA organized customer feedback into meaningful themes; second, deep-learning models connected these insights to sequential customer interactions, forecasting future behavior; and, finally, expert interviews informed a practical resource allocation model, incorporating integer programming and the BDE method. This layered approach enables e-commerce firms to allocate resources efficiently across marketing channels, optimize customer engagement, and strategically adapt to market dynamics, enhancing overall operational effectiveness.

3.1. Leveraging Latent Dirichlet Allocation for Analyzing Customer Needs and Product Attributes

The study adopts LDA to enrich customer behavior prediction and conversion value forecasting, particularly by integrating topic modeling with machine-learning approaches, such as LightGBM, CNN, and LSTM. The dual use of LDA—both for exploratory insights and as a predictive enhancement—facilitates a nuanced understanding of customer preferences. By categorizing unstructured customer feedback into identifiable themes, LDA offers a structured perspective on qualitative data, allowing businesses to interpret customer needs with greater specificity. This mixed-methods framework enables a comprehensive approach, balancing qualitative insights with quantitative precision for robust predictive outcomes.

LDA, an unsupervised topic-modeling algorithm, assigns a probabilistic distribution of topics across documents and words, revealing latent themes within a corpus of customer reviews [10]. This study applies LDA to evaluate consumer need variables by examining product requirements and sentiment embedded in reviews. Key steps included preprocessing (e.g., tokenization and stop-word removal) and determining the optimal number of topics through coherence scores and expert validation. Each identified topic was then analyzed in the context of top words and their meanings, ensuring alignment with real customer feedback. The LDA model, using Gibbs sampling for posterior inference, as outlined by De Finetti’s [11] theorem on exchangeable random samples, generates distributions for each topic, which aids in discerning underlying patterns in customer expectations and behaviors (see Figure 3).

Figure 3 illustrates the LDA model framework, where topics are represented as distributions of words, and documents are mixtures of these latent themes. Here, the probabilities of words within each topic are influenced by prior parameters α and β, encapsulating customer inclinations and product attributes in the text corpus. This model framework enables the extraction of actionable themes that align with observed consumer behaviors, providing a reliable basis for predictive modeling and targeted marketing strategies. By adhering to Bayes’ theorem, LDA facilitates document categorization and allows practical applications, such as automatic theme identification, within a high-dimensional data landscape.

The study also incorporated a carefully curated dataset of 150,000 customer records, selected based on recent engagement with the company’s platforms. Inclusion criteria ensured the data’s relevance, reflecting current preferences and purchasing patterns. This large dataset supports robust model training and testing, capturing a broad spectrum of customer behaviors across various demographic segments. By examining diverse groups, the model can reveal general consumer trends while accounting for variations in behavior.

Studies by Huang et al. [12] and Slof et al. [13] have shown the potential of combining LDA with sentiment analysis to predict consumer behavior more accurately. For instance, Huang’s model, integrating LDA with the SCOR framework, demonstrated enhanced sales forecasting by associating sentiment-based topic probabilities with consumer purchase likelihood. Similarly, this study leverages LDA’s capability to detect dominant themes and integrates these insights with neural network models, leading to better sales predictions. As Xu et al. [14] indicated, merging machine learning with topic analysis strengthens the interpretive power of customer feedback, enabling e-commerce platforms to harness valuable emotional and thematic insights from reviews, which can guide tailored marketing efforts.

LDA remains a preferred tool in product review analysis due to its capacity to distill fundamental topic structures from textual data, enhancing the interpretability of machine-learning predictions. By clustering words into topics and identifying connections among them, LDA contributes specific, context-rich features to predictive models. For example, if a model anticipates a positive rating for a product, LDA enables the identification of the key themes and attributes driving that prediction, which can guide feature optimization and targeted product improvements.

Mathematically, LDA functions on a “bag of words” approach, where each document is treated as a collection of word frequencies. Given parameters $\alpha$ and $\beta$, the joint distribution of topic probabilities θ_i, topic assignment $Zij$, and word $Wij$ in document iii is expressed as follows:

$$P(\theta i,Wij,Zij,\mathit{\Phi }k|\alpha ,\beta )=P\left(\theta |\alpha \right){\prod }_{n=1}^{N}P\left(Zij|\theta \right)P\left(Wij|Zij,\beta \right)$$

(1)

To calculate the posterior distribution of latent variables, the equation becomes the following:

$$P(\theta i,Zij|Wij,\alpha ,\beta )={\displaystyle \frac{P(\theta i,Wij,Zij|\alpha ,\beta )}{P\left(Wij\right|\alpha ,\beta )}}$$

(2)

Using Gibbs sampling, a Markov chain Monte Carlo (MCMC) method, the LDA model operates under the assumption that, within a two-dimensional probability distribution space P(x, y), conditional probabilities can be calculated by fixing one dimension’s X coordinate:

$$P(x_1,y_1)P\left(y_2\right|x_1)=P(x_1,y_2\left)P\right(y_1\left|x_1\right)$$

(3)

This methodological approach not only reinforces predictive accuracy but also aligns with consumer behavior theories by revealing latent motivations within customer feedback, thus guiding targeted engagement strategies. Through combining topic modeling with machine learning, the study contributes significantly to e-commerce research, particularly in optimizing customer engagement and product customization through data-driven insights.

3.2. Integrating Machine-Learning and Deep-Learning Models for Customer Behavior Prediction

This study evaluates the effectiveness of combining LDA with various machine-learning models to predict customer behavior more accurately. LDA aids in extracting topic-based features from the review text, adding contextual depth to neural network inputs. Traditional regression methods, limited by their linear assumptions, struggle with the complexity of sentiment-laden customer reviews. Consequently, models like LightGBM, long short-term memory (LSTM), and convolutional neural networks (CNNs) are better suited for handling structured, sequential, and spatial data. LightGBM is efficient for structured data, while LSTM captures temporal dependencies, and CNNs discern spatial patterns.

Our model evaluation, based on accuracy and mean squared error, revealed that combining LDA with neural networks boosts prediction accuracy. Splitting the data into an 80:20 training-to-testing ratio ensured robust model training and validation. Key hyperparameters, such as LightGBM’s 0.1 learning rate and CNNs’ kernel sizes, were optimized to enhance replicability and performance. The chosen models align with consumer behavior theories, where LDA uncovers latent topics related to consumer needs, and CNN-LSTM models reveal browsing patterns predictive of purchasing intent. The methodological alignment presented herein strengthens the study’s relevance to e-commerce, demonstrating a coherent, data-driven approach to customer journey optimization. Adherence to standardized Python libraries (version 3.11.11) and clearly specified version requirements ensures the replicability and reliability of the predictive insights generated across comparable datasets.

3.2.1. CNN-LSTM Model for Sequential Customer Data Analysis

The CNN-LSTM model effectively combines convolutional and recurrent neural networks to analyze sequential customer interactions, capturing both spatial and temporal dependencies in data patterns [15]. LSTM networks, known for handling sequential data, excel in tracking temporal dependencies, such as customer browsing patterns and purchasing behaviors [16]. Figure 4 illustrates LSTM’s gating mechanism, which regulates information retention over multiple time steps, enabling it to maintain context in sequential interactions. It depicts the memory unit of the user login time t, t − 1, and represents the hidden state of the user login time t, t − 1. The hidden state, distinct from the memory cell, is passed to the next time step, where the input data of the user login time t capture short-term dependencies in the sequence. Moreover, the LSTM neural network framework illustrates interconnected elements crucial for managing long-term dependencies and making accurate predictions for sequential data. These components, including the cell state (Ct), input gate (i), forget gate (f), output gate (o), and hidden state (ht), collaboratively enable the LSTM to process sequential information effectively. By addressing the vanishing gradient problem, LSTMs can preserve long-term dependencies essential for accurately predicting customer engagement over time.

CNNs, renowned for their prowess in computer vision, have extended their utility beyond image analysis to feature extraction across diverse data types. This study uses CNNs to identify spatial patterns within customer data, such as browsing patterns and click behaviors, which serve as inputs for sequential prediction. Employing multiple layers of convolutions and pooling operations, CNN models acquire hierarchical representations of customer data, enabling accurate predictions or classifications based on these extracted features [17].

The CNN-LSTM model leverages CNNs to capture spatial features in browsing data, which are then processed by LSTMs to analyze sequential customer behaviors over time. When handling review and customer purchase behavior data, the CNN-LSTM model leverages CNN layers to extract features from customer data, which are then fed into LSTM layers to capture sequential dependencies effectively. This integration empowers the model to comprehend local data patterns and long-term dependencies, augmenting its understanding of customer sentiments and other pertinent characteristics [18]. Channel sums undergo conversion into binary vectors, and the data are structured by click date. Diverse vital features are incorporated for analysis, such as browsing behavior, device type, browser type, recency, frequency, and monetary (RFM) segmentation, contact channel, clustering, product information, homepage, membership status, and displayed content. Subsequently, each customer’s data are visualized as a pixel matrix to represent complex multi-channel interactions, summarizing factors like browsing behavior and purchase history for CNN feature extraction.

Figure 5 showcases the CNN-LSTM framework, where CNN layers first capture spatial patterns in browsing data, subsequently fed into LSTM layers for temporal processing. This integration allows for a richer understanding of customer sentiment, behavior, and purchasing intent, benefiting customer journey analysis. This dual architecture is particularly valuable in applications like sentiment analysis, where CNN layers can extract sentiment-indicating features, and LSTM layers interpret these features sequentially. By visualizing each customer’s data as a pixel matrix, the model captures nuanced, multi-channel interactions, offering enhanced insights into customer preferences and facilitating targeted marketing strategies. The CNN-LSTM approach is thus a robust tool for predicting and understanding dynamic customer journeys, especially in multi-channel e-commerce environments.

3.2.2. Optimizing E-Commerce Insights with LightGBM

LightGBM excels in processing large-scale e-commerce data, providing a fast, accurate analysis vital for customer behavior prediction. As a decision-tree-based model, LightGBM incrementally refines tree structures to minimize prediction errors, employing gradient-based one-sided sampling to prioritize the most informative data points, thus improving both speed and memory efficiency. In e-commerce, where customer interactions often vary widely, LightGBM’s advanced handling of imbalanced data—common when specific behaviors dominate—ensures reliable results across a spectrum of customer types.

The objective function of LightGBM includes both a loss term L(y_i,f(x_i)), capturing the prediction accuracy, and a regularization term Ω(f_k), preventing overfitting by constraining the model complexity, represented as follows:

$$Objective(f)={\sum }_{{\left\{i=1\right\}}^{\left\{n\right\}}}L(y_i,f(x_i\left)\right)+{\sum }_{{\{k=1\}}^{\left\{K\right\}}}\mathit{\Omega }\left(f_k\right)$$

(4)

Regularization terms such as L1 (Lasso) and L2 (Ridge) enhance generalizability, reducing the reliance on specific data points that could skew predictions in new datasets. By integrating variables like customer contact paths and LDA-processed motivations, LightGBM models capture comprehensive purchase behavior trends over time. This dynamic approach enables the model to assess evolving customer interactions, enhancing the predictive power for future purchases. Using a 10-fold cross-validation framework, the model undergoes rigorous accuracy testing across different data subsets, with an 80-20 split between training and testing to ensure robust performance. LightGBM thus stands out as a key tool for driving customer-focused marketing, enabling informed, personalized engagement strategies that align with real-time trends.

3.3. Optimal Resource Allocation for Enhanced Customer Engagement

This study’s integer programming model optimally allocates e-commerce resources like advertising spending and customer acquisition budgets across key channels—social media, email, and search engine advertising. By efficiently balancing customer satisfaction, engagement, and conversion rates within budget constraints, this model enables businesses to maximize returns on investment, minimize costs, and achieve targeted, impactful customer engagement.

3.3.1. Optimizing Multi-Channel Customer Value for Strategic Resource Allocation

The multi-channel customer value optimization model aims to maximize conversion rates by optimizing resource allocation across user interactions on multiple channels. The model captures the dynamic nature of user behavior within the corporate website, where users (represented as user i) engage in a sequence of interactions across various channels. Each action—such as purchase (AC1), buy button click (AC2), and general click (AC3)—represents a step in the user’s journey, helping to predict conversion likelihood based on sequential behavior. The dataset comprises records of user clicks and engagements across multiple channel types (m), each representing a sequence of clicks on a specific channel. Time intervals between clicks are considered, ensuring distinct events even within the same day. Valid data points require at least one recorded interaction (represented by r data points), enabling the model to predict the user’s subsequent action (r+1) within the sequence of (${AC}_1$), (${AC}_2$), and ${AC}_3$. The predicted conversion value is contingent upon the user’s action on the j-th channel, where a purchase click yields a converted value of ${CL}_{ij}$, while non-purchase clicks are denoted as $R_{ij}$. The enterprise determines the target conversion value based on these user interactions.

Table 1. Definitions of mathematical symbols and variables.

Symbol	Definition
$i$	The 1st, 2nd, …………$n$(th) user
$j$	The 1st, 2nd, …………$m$(th) channel
$t$	The 1st, 2nd …………$r$(th) click
$k$	The 1st, 2nd …………$k$(th) cluster
${AC}_1$	${AC}_1$ is 1 or 0, indicating if the user has or has not purchased
${AC}_2$	${AC}_2$ is 1 or 0, indicating if the user has or has not clicked the purchase button
${AC}_3$	${AC}_3$ is 1 or 0, indicating if the user has or has not clicked
$R_{ij}$	The conversion value of the $i$th user when purchasing on the $j$th channel
${CL}_{ij}$	The conversion value of the $i$th user when clicking the purchase button on the $j$th channel
$S_{ij}$	The conversion value of the $i$th user when clicking button on the $j$th channel
$A_l$	The behavioral variable of the 1st, 2nd … to the $l$th user
$Z_{ik}$	The value of $Z_{ik}$ is 0 or 1 and represents the $i$th user of topic k
$N_k$	Tailoring product appeals or services to meet the specific requirements of each customer demand group.
$C_{1j}$	The unit cost required to complete a transaction by the $i$th user converted on the $j$th channel
$C_{2j}$	The unit cost required for clicking the purchase button by the $i$th user converted on the $j$th channel
$C_{3j}$	The unit cost required for clicking a button by the $i$th user converted on the $j$th channel.
$D_i$	The number of days the $i$th user is contacted by the selected channel
$d_i$	The maximum number of days the $i$th user is contacted by the selected channel
$E$	The current estimated expense for acquiring customer
$X_{ij}$	The value of $X_{ij}$ is 0 or 1, to determine if the $i$th user would select the $j$th channel. This is a decision variable.
$Y$	Total target customer value of conversion

provides a structured overview of the model’s core variables, such as user IDs, channel types, click events, and conversion values, which form the foundation for evaluating each user’s engagement journey. Each interaction—quantified by specific conversion values ($R_{ij}$ for purchases and ${CL}_{ij}$ for purchase button clicks)—is linked to the financial impact and costs associated with each channel, guiding managers in pinpointing high-value interactions. Additionally, behavioral variables ($A_l$) and topic-specific data ($Z_{ik}$) capture nuanced aspects of customer preferences, enabling more personalized and targeted engagement strategies. This data framework helps marketing managers and financial analysts assess which channels offer the highest return on investment (ROI) and cost-effectiveness, informing budget allocation decisions to maximize conversion potential.

The model’s approach incorporates RFM metrics, allowing segmentation based on user behavior patterns, engagement frequencies, and spending tendencies. These attributes are crucial for distinguishing high-value users and tailoring outreach strategies. For example, channels with a higher ROI can be prioritized, while underperforming ones may be adjusted to align better with customer preferences. Additionally, product appeals or services are tailored to meet the specific requirements of each customer demand group ($Z_{ik},N_k$). Table 1 highlights these customer value components, reinforcing the importance of assessing conversion values ($R_{ij}$, ${CL}_{ij}$, and $S_{ij}$) and conversion costs ($C_{1j}$, $C_{2j}$, and $C_{3j}$) within the context of budget constraints (E).

The objective is to maximize target conversion values while adhering to budget constraints and optimizing channel selection for user engagement. The model operates within a budget constraint (E), ensuring customer acquisition and advertising costs do not exceed the allocated resources. By leveraging decision variables ($X_{ij}$), the model enables the selection of optimal channels to maximize customer value (CV) within budgetary limits. Equations (5)–(9) represent the mathematical formulation of our optimization framework, which aims to maximize total expected conversion value while respecting budget constraints and channel capacities. This model integrates predicted customer behavior, channel performance, and cost-effectiveness to guide strategic marketing resource allocation. By solving this optimization problem, the model directs resources toward the most valuable channels based on data-driven insights, thereby improving user engagement and overall return on investment (ROI). Compared to heuristic channel decisions, this approach ensures that limited budgets are allocated in a way that maximizes customer value while maintaining operational efficiency.

$$MaxY=\sum _i\sum _j{R}_{ij}{X}_{ij}+\sum _i\sum _j{CL}_{iJ}{X}_{ij}+\sum _i\sum _j{S}_{iJ}{X}_{ij}$$

(5)

$$s.t.\sum _j{D}_j{X}_{ij}\le d_{j}i=\mathrm{1,2},\dots \dots n$$

(6)

$$\sum _j{C}_{1j}{X}_{ij}+\sum _j{C}_{2j}{X}_{ij}+\sum _j{C}_{3j}{X}_{ij}\le Ei=\mathrm{1,2},\dots \dots n$$

(7)

$$\sum _i{Z}_{ik}{X}_{ij}\le N_{k}k=\mathrm{1,2},\dots \dots k;j=\mathrm{1,2},\dots \dots m$$

(8)

$$\sum _i{X}_{ij}<5j=\mathrm{1,2},\dots \dots m$$

(9)

3.3.2. Heuristic Algorithm Optimization Framework

This research presents a novel approach for addressing the computationally intensive and storage-heavy nature of multi-channel planning in e-commerce, where efficient resource allocation across channels is critical yet challenging due to the NP-hard nature of the problem. Traditional heuristic algorithms, though useful for broad search capabilities, often fall short in reducing feature sets effectively. The proposed BDE method tackles this limitation by incorporating a differential evolution algorithm with mixed binary coding, enabling the reduction in features within a discrete solution space while respecting allocation constraints. This approach not only refines the solution space by identifying the most representative individuals but also enhances performance in dynamic optimization problems, reducing the risk of convergence failures in high-dimensional applications. Ultimately, this method provides a more targeted, resource-efficient solution, making it highly suitable for complex e-commerce planning tasks where optimization speed and precision are paramount. The BDE algorithm optimizes by iterating through four main steps.

Step 1: Population Initialization—This step generates an initial set of solution vectors, with each element in the vector set to either 0 or 1, as specified by Equation (10):

$$x_{j,i,0}=x_{j,min}+{rand}_{i,j}\left[\mathrm{0,1}\right]·\left(x_{j,max}-x_{j,min}\right),i=\mathrm{1,2},\dots .,N_P$$

(10)

• $N_P$: population size;
• G: maximum number of iterations;
• $x_{j,min}$: minimum search range;
• $x_{j,max}$: maximum search range.

Step 2: Mutation—The algorithm creates a composite vector by selecting three random target vectors, namely, ${\mathrm{X}}_{r1,G}$, $X_{r2,G}$, and $X_{r3,G}$, combining them into a composite vector $V_{i,G+1}$ and applying a mutation weight factor to generate variations in potential solutions. Equation (11) is as follows:

$$V_{i,G+1}=X_{r1,G}+F(X_{r2,G}-X_{r3,G})$$

(11)

where F is the mutation weight factor; and r1, r2, and r3 are the randomly selected target vectors.

Step 3: Crossover—This step combines elements from the composite vector (Donor Vector, $U_{i,G}$) with the target vector ($X_{i,G}$), ensuring diversity and preventing premature convergence. A parameter value, Cr, is set, and another random value is selected. If the random value is less than Cr, the new vector in one dimension will be put into $V_{i,G+1}$, or vice versa, replacing it with $X_{i,G}$. When the value is more significant than Cr, the variable, nj, will be a random actual number in the range of [0, 1]. The primary purpose of this step is to guarantee that at least one dimension of values will go through the mutation process of the new vector $U_{i,G}$ to be different from the corresponding target vector $X_{i,G}$.

$$U_{ji,G+1}=\left\{\begin{array}{c}{V}_{ji,G+1},if{rand}_j\left(\mathrm{0,1}\right)\le Crorj=nj\\ X_{ji,G},otherwise\end{array}\right.$$

(12)

where Cr is the crossover rate in actual number between [0, 1] and ${rand}_j\left(\mathrm{0,1}\right)$ is a random actual number in the range of [0, 1].

Step 4: Selection—The algorithm evaluates each solution’s fitness and selects the best-performing solutions, aligning with the model’s constraints on resource allocation and costs. To optimize the heuristic algorithm framework, it was necessary to precisely align the fitness function with the specific limitations described in our approach.

The fitness function serves as a key evaluation tool in the BDE framework, assessing how well each solution meets the model’s objectives. We incorporated two penalty terms within the fitness function to ensure adherence to budget constraints and maximize the conversion value. Budget Penalty—This penalty is applied to any solution that exceeds the designated customer acquisition and advertising budget, effectively limiting resource overspending. Conversion Penalty—This term rewards solutions that meet or exceed target conversion rates, encouraging optimal channel selection for high engagement. Equation (13) formalizes the fitness function:

Fitness = Y × Budget Penalty + Conversion Penalty.

(13)

Additionally, the algorithm’s termination condition halts optimization when successive generations yield minimal improvement, conserving resources. The BDE framework thus efficiently directs advertising resources, ensuring high-conversion channels receive optimal allocations and fostering precise, budget-conscious marketing strategies.

4. Empirical Validation of Model Efficacy in Predicting Customer Behavior and Channel Optimization

Our empirical experiment evaluated the performance of models like LDA-LightGBM, CNNs, LSTM, and CNN-LSTM in predicting customer behaviors and optimizing channel planning for an insurance case study. By analyzing real-world data, we compared these models on metrics such as accuracy, execution time, and customer value generation. This rigorous testing provided actionable insights into enhancing resource allocation and customer engagement.

Key metrics—Receiver Operating Characteristic (ROC), Area Under the Curve (AUC), F1-score, and recall score—were used to assess model effectiveness across customer segments (e.g., savings insurance and cancer insurance). Additionally, integer programming and a BDE algorithm were applied to address complex, high-dimensional channel planning. Compared to manual methods, these advanced techniques demonstrated notable gains in customer value and cost efficiency, underscoring their potential in strategic resource optimization.

4.1. Comprehensive Dataset Analysis for Multi-Channel Customer Insights

This study’s sampling strategy was designed to capture a broad spectrum of customer interactions within Taiwan’s insurance sector. By partnering with a major financial institution known for its extensive online engagement, the study gains access to a rich dataset that accurately reflects customer journeys across diverse channels. This selection enhances the study’s relevance, as it draws on 70,001 user reviews between March and November 2020, covering the top 20 insurance brands and 11 major categories. The chosen time period ensures that the data encompass seasonal patterns, market fluctuations, and other temporal factors, providing a holistic view of customer behavior.

In addition to reviews, the dataset includes 86,738 product-specific observations across 45 unique items, covering critical attributes such as interest rates and premium levels. To protect privacy, all user data underwent stringent anonymization, with personal information obfuscated, ID hashing, and the introduction of random errors to ensure compliance with data protection standards. These efforts underscore the commitment to ethical data handling and protect both user privacy and institutional confidentiality.

The data collection and preparation processes were meticulous. Initially stored as .json files, the dataset was converted into .csv format, enabling easier manipulation and analysis. This transformation was followed by extensive data cleaning, which included removing duplicates, handling missing values, and resolving inconsistencies in timestamp formats. Additionally, categorical variables were one-hot-encoded, and numerical variables were normalized to create a high-quality dataset suitable for machine learning. These steps minimized data noise and optimized the dataset for predictive modeling, ensuring robust and reliable outcomes.

Table 2. Partial schematic representation of unstructured product review.

Product Code	Category	Review	Title	Tag
BPIEPL	Savings Insurance	Last week, I visited Cathay United Bank, where a financial advisor introduced a six-year USD savings policy. It offers a declared interest rate of 3.15% and a guaranteed interest rate of 2%. The amounts mentioned are in USD, and the advisor mentioned that the current USD exchange rate is low. While considering USD savings as an option, it is important to note that USD exchange rates may fluctuate. The premium is USD 1000 annually, approximately equivalent to over NT$20,000…	‘Should I buy USD savings insurance?’	5
BPIEPL	Savings Insurance	The declared interest rate is 3.15%. Ask them to calculate the annualized return for you to compare with other investment products. With savings insurance, you must wait six years before you can cancel it, unlike funds or stocks, which you can sell whenever you want. Essentially, your money is locked in for those six years. Alternatively, consider it a USD fixed deposit that cannot be terminated early. If you have multiple items for comparison, tax deductions may be available…	‘Should I buy USD savings insurance?’	3
BPIEPL	Savings Insurance	The USD savings insurance is essentially a USD savings plan, just a USD fixed deposit, and it lacks insurance features similar to AUISA. If there is a surrender value within six years, and you are speculating on a rise in USD interest rates, depositing USD in the future might be better. It seems like purchasing USD assets now is a bet on the depreciation of the TWD……	‘Should I buy USD savings insurance?’	3

highlights unstructured product reviews for Cathay United Bank’s USD savings insurance product. This product received significant attention from customers, particularly regarding its interest rates (3.15% declared and 2% guaranteed), lock-in period (six years), and comparisons with other investment options. Many reviews reveal diverse opinions, with some customers viewing it as a stable investment, while others express concerns about the inflexibility of the lock-in period and potential exchange rate fluctuations. These sentiments reflect broader customer considerations in insurance, where stability, flexibility, and long-term returns are often weighed against liquidity and investment alternatives.

Structured data further enhance the analysis by providing quantitative insights into user actions.

Table 3. Partial schematic representation of structured customer data.

UserID	Date	Source/ Medium	Channel	Action	Target Conversion
757	1 April 2020	Google/CPC	Product page	Click	949
757	24 June 2020	fuboneip/banner	Event page	Click the purchase button	3985
757	24 August 2020	shop.hsbc.com.tw/referral	Event page	Purchasing	10,000

presents a sample of structured customer interactions, detailing user ID, interaction date, traffic source, channel type, and conversion values. For instance, User 757 engaged with a Google CPC ad, clicked on a product page, and later converted through various touchpoints with a cumulative conversion value of 10,000. These structured data capture the multi-channel journey, from initial interaction to purchase, offering a detailed view of how different channels contribute to conversions. Tracking such interactions allows the study to analyze channel-specific performance, providing actionable insights into optimizing marketing strategies based on channel effectiveness.

Table 4. User variables.

Feature	Description
UserID	A unique User ID in the company tracks the user across multiple sessions using the Company Account.
Date	The date of the session is formatted as YYYYMMDD.
Channel	The source of referrals. For manual campaign tracking, the value of the utm_source campaign tracking parameter is defined by the company’s management.
Device Category	The category of the device: mobile, tablet, or desktop.
Browser	The raw name of the browser.
Recency	How recently did a customer make a purchase? In this research, the higher the score, the longer the interval.
Frequency	How often a customer makes a purchase.
Monetary	How much money a customer spends on purchases.
Revenue	The total sale revenue (excluding shipping and tax) of the transaction.
Campaign Content	The value of the utm_content campaign tracking parameter is defined by the company’s management for manual campaign tracking.
Page Content	Query parameters specify a page on the website. Use this with the hostname to obtain the page’s full URL. The management defines the display content.
User Type	Either New Visitor or Returning Visitor, indicating if the users are new or returning.
Session Duration	The length of a session is measured in seconds and reported in second increments.
Sessions	The total number of sessions.
Pages	The average number of pages viewed during a session, including repeated views of a single page.
Action Stage	The action stage is at the end of each session. This is the target feature for this use case.

introduces additional user variables that contribute to a granular understanding of customer profiles and behaviors. Key variables include device type, browser, recency, frequency, and monetary value, forming the foundational elements of recency, frequency, monetary (RFM) metrics. These metrics are instrumental in segmenting customers by purchasing behavior, allowing targeted marketing strategies. For example, high-frequency, high-monetary customers might be targeted with loyalty programs, while new or low-engagement users could benefit from introductory offers. Understanding session duration, page views, and user type (new or returning) further informs engagement strategies, as these variables shed light on the depth of user engagement and potential interest levels.

Overall, this data-driven approach leverages both structured and unstructured data to capture a multi-dimensional perspective on customer interactions. Structured data reveal quantitative patterns in user behavior across channels, while unstructured reviews provide qualitative insights into customer sentiment and preferences. Together, these datasets allow for a comprehensive analysis that informs targeted marketing, product development, and customer experience optimization. By integrating these insights, the study equips the financial institution with a nuanced understanding of customer journeys, enabling data-driven decisions that align closely with customer needs and market dynamics.

4.2. Model Selection and Evaluation for Customer Three-Stage Prediction

To determine the most suitable topic-modeling method for our study, we compared four widely used models—GSDMM, LDA, NMF, and LSI—based on coherence (c_v) and topic diversity. As shown in

Table 5. Comparison of topic-modeling methods based on coherence and topic diversity.

Model	Coherence	Topic Diversity
GSDMM	0.493772	0.48
LDA	0.491690	0.78
NMF	0.477533	0.80
LSI	0.477259	0.42
BERTopic (HDBSCAN)	0.734713	0.82

, GSDMM achieved the highest coherence score (0.493772), indicating strong topic interpretability, but its topic diversity was relatively low (0.48). In contrast, NMF demonstrated the highest topic diversity (0.80), suggesting a wide variety of topics; yet, its coherence score (0.477533) was lower, implying reduced clarity in topic extraction.

We initially evaluated neural-network-based topic-modeling methods, such as BERTopic, and found that they outperformed traditional approaches (e.g., LDA and NMF) in terms of topic coherence and diversity, with HDBSCAN often extracting more representative topics. However, HDBSCAN is a soft clustering method that tends to produce outliers and may assign some documents to no cluster at all. For downstream tasks that require each document to be explicitly associated with a single topic, this clustering strategy is not ideal.

LDA offered a well-balanced performance, with a coherence score (0.491690) close to GSDMM, while maintaining a high topic diversity (0.78). Given its equilibrium between interpretability and topic differentiation, LDA was deemed the most appropriate choice for our research. Furthermore, LDA’s probabilistic generative framework has been extensively validated in previous studies and is widely implemented in natural language processing applications, making it more interpretable and applicable to downstream analyses. Therefore, we selected LDA as the primary topic modeling technique in this study.

Although recent deep-learning models such as Transformers have demonstrated great potential in capturing long-term dependencies and enabling parallel computation, our experimental results indicate that, under the characteristics of our dataset (e.g., limited records per user and short sequence lengths), the prediction performance of LSTM is comparable to that of the Transformer. In fact, as shown in

Table 6. Performance comparison of LSTM, Transformer, and CNN-LSTM.

	AUC	F1-Score	Recall-Score	Execution Time (s)
LSTM	0.771579	0.597716	0.550905	0.680373
Transformer	0.762920	0.633818	0.578683	0.679652
CNN-LSTM	0.772611	0.635638	0.588235	0.678451

, we conducted a small-scale experiment using the channel source to predict the subsequent action of a user on a multi-class prediction task.

While the Transformer shows a slight advantage in F1-score and Recall over LSTM, the LSTM model achieves a higher AUC, indicating its superior ranking ability. Notably, with the integration of a CNN layer prior to the LSTM (i.e., the CNN-LSTM model), there is a modest improvement across all evaluation metrics—especially in AUC, F1-score, and Recall—with inference times remaining nearly identical among the three models. Please refer to Figure 6, the ROC curve and AUC value.

In this study, we conducted a comprehensive comparison of multiple models, including traditional machine-learning methods such as logistic regression, decision trees, random forest, XGBoost, and LightGBM, as well as neural-network-based methods including multi-layer perceptrons (MLPs), recurrent neural network (RNNs), long short-term memory (LSTM), and Transformers. The evaluation considered the Accuracy, macro F1-score, Area Under the Curve (AUC), and training efficiency (see

Table 7. Model performance comparison.

Model	Accuracy	F1-Score	AUC	Training Time (s)
Logistic Regression	0.606364	0.377476	0.603432	0.013183
Decision Tree	0.765468	0.754861	0.814237	0.010916
Random Forest	0.766058	0.755540	0.816926	0.246940
LightGBM	0.766647	0.755458	0.819134	0.041065
XGBoost	0.761344	0.750228	0.818114	0.038224
MLP	0.753683	0.741597	0.817079	191.28
RNN	0.760165	0.750739	0.817610	372.02
LSTM	0.759576	0.750186	0.799512	354.41
Transformer	0.760165	0.750739	0.807296	391.12

for detailed results). Please refer to Figure 6, the ROC curve and AUC value.

The experimental results indicated that LightGBM achieved the highest overall performance, with an Accuracy of 0.7666, F1-score of 0.7555, and AUC of 0.8191. In comparison, neural-network-based methods, including RNNs (Accuracy: 0.7602, AUC: 0.8176), LSTM (Accuracy: 0.7596, AUC: 0.7995), and Transformers (Accuracy: 0.7602, AUC: 0.8073), showed a comparable but slightly lower predictive performance. Additionally, neural network methods required significantly longer training times. For example, the Transformer model required approximately 391 s for training, whereas LightGBM only required approximately 0.041 s, demonstrating a substantial efficiency advantage.

These findings align with the initial consideration that predicting customer insurance purchase behavior is inherently a non-sequential classification task, primarily reliant on current customer features rather than historical sequential behaviors. Although neural-network-based methods typically excel at capturing complex sequential patterns, Table 7 indicates that models such as MLPs, RNNs, LSTM, and Transformers did not deliver clear performance advantages, yet required significantly longer training times—often hundreds of times longer than tree-based methods. Therefore, considering both the predictive capability and computational efficiency, gradient boosting tree methods, particularly LightGBM (Accuracy = 0.766647, F1-score = 0.755458, AUC = 0.819134, and Training Time = 0.041065 s), are selected as the primary modeling approach for practical deployment within computational resource and time constraints. Please refer to Figure 7, the ROC curve and AUC value.

4.3. Insights from Latent Dirichlet Allocation on Customer Needs

This study uses LDA to analyze customer feedback, revealing nuanced insights into consumer preferences across five insurance product categories. By examining keywords associated with each category, LDA identifies underlying themes and product features that resonate with customers, providing actionable intelligence for product managers and marketers.

In life insurance, discussions concentrate on family security and financial planning, with keywords like “benefit insurance”, “heritage”, and “elderly”, underscoring consumer concerns about long-term coverage for diverse life stages. This emphasis highlights customers’ focus on policies offering budget-friendly options with competitive benefits. In contrast, savings insurance keywords—such as “annuity”, “asset allocation”, and “tax”—reflect an orientation toward wealth accumulation, efficient tax strategies, and financial security for retirement, emphasizing the need for products with stable returns.

Mandatory motor insurance focuses on practical concerns around risk management and claims processes. Keywords like “liability insurance” and “insurance rate” suggest customers value straightforward, cost-effective coverage for everyday driving risks. In cancer insurance, keywords emphasize proactive health management, including “early detection” and “genetic testing”. This category shows customer demand for comprehensive coverage that includes both treatment support and preventive care, especially for families with genetic predispositions. Finally, whole life insurance discussions emphasize enduring financial protection and estate planning, with terms such as “lifetime coverage” and “inheritance management”, highlighting the importance of policies that ensure long-term security across generations.

Figure 8, Figure 9, Figure 10 and Figure 11 present these LDA findings, with line graphs for each insurance type depicting the prominence of specific features. Peaks across different quadrants of the figures highlight critical attributes, offering a visual summary of customer priorities. This analysis enables insurers to strategically tailor their offerings by aligning product features with the distinct needs and preferences associated with each type of insurance.

4.4. Optimizing Conversion Prediction with LDA-LightGBM: Model Insights and Performance Analysis

In this study, we optimized the LightGBM model to predict customer conversion by fine-tuning hyperparameters such as the maximum number of leaf nodes, learning rate, and the number of trees. These parameters were critical for adapting the model to high-dimensional insurance data, where capturing complex conversion patterns is essential.

Table 8. Candidate and optimal sets of hyperparameters for the models.

Model Names	Hyperparameter Names	Hyperparameter Values
LDA	LDA numbers N-gram Maximum size of N-gram dictionary	{5, 2, 20,000}
CNN	Padding Pool size	{same, 2}
LSTM	Activation Dropout rate Timesteps Hidden nodes Learning rate Number of iterations	{relu, 0.3, 13, 100, 0.001, 100}
LightGBM	Maximum number of leaf trees Maximum number of sample leaf nodes Learning rate Total number of trees constructed	{20, 10, 0.2, 100}

outlines the optimal hyperparameters for LightGBM, LDA, CNNs, and LSTM, highlighting the specific settings that enhance each model’s efficiency in handling different data types, from structured transactions to sequential behavior. The optimized LightGBM model achieved a 5% accuracy improvement over CNNs and LSTM, validating its effectiveness in identifying high-potential customer segments.

Table 9. Robustness evaluation.

Validation Metrics	Model	Topic 1: Term Life Insurance	Topic 2: Endowment Insurance	Topic 3: Compulsory Automobile Liability Insurance	Topic 4: Cancer Insurance	Topic 5: Whole Life Insurance
ROC	LSTM	0.91	0.92	0.98	0.97	0.97
	CNN	0.90	0.93	0.98	0.97	0.97
	CNN-LSTM	0.91	0.93	0.98	0.97	0.97
	LDA-LightGBM	0.97	0.93	0.98	0.97	0.97
AUC	LSTM	0.79	0.74	0.74	0.52	0.51
	CNN	0.74	0.69	0.67	0.46	0.46
	CNN-LSTM	0.80	0.72	0.78	0.51	0.52
	LDA-LightGBM	0.80	0.72	0.78	0.60	0.60
F1-score	LSTM	0.88	0.89	0.97	0.96	0.95
	CNN	0.87	0.89	0.97	0.95	0.95
	CNN-LSTM	0.89	0.91	0.97	0.95	0.95
	LDA-LightGBM	0.89	0.91	0.98	0.96	0.96
Recall-score	LSTM	0.91	0.92	0.98	0.97	0.97
	CNN	0.90	0.93	0.98	0.97	0.97
	CNN-LSTM	0.91	0.92	0.98	0.97	0.97
	LDA-LightGBM	0.91	0.93	0.98	0.97	0.97

evaluates the model robustness across various insurance products, revealing that the LDA-LightGBM model consistently outperformed other models in all categories, especially for long-term policies like term life and whole life insurance. With the highest ROC scores, LDA-LightGBM’s predictive accuracy supports targeted marketing efforts, as it identifies customer segments with a higher purchase propensity, optimizing resource allocation for insurance marketers. Its enhanced F1- and Recall-scores indicate that the model is particularly adept at identifying positive instances, which is crucial for maximizing customer engagement in high-stakes insurance products.

Compared to LSTM, CNNS, and CNN-LSTM, LDA-LightGBM’s balanced performance across ROC, AUC, and execution time establishes it as a compelling choice for predictive tasks that require both speed and accuracy. Its ability to integrate topic modeling insights from LDA with the predictive power of gradient boosting enhances its interpretability, making it ideal for applications in e-commerce and insurance where understanding customer intent is key. By combining qualitative insights from text data with quantitative accuracy, LDA-LightGBM enables precise, targeted marketing, setting a strong foundation for the further exploration of machine learning in predictive customer segmentation and engagement strategies.

The findings demonstrate LDA-LightGBM’s potential as a foundational tool in customer conversion optimization, delivering superior predictive accuracy, efficient execution, and robust adaptability across different product lines. This model’s strengths in handling complex customer data pave the way for more sophisticated applications in customer journey analysis and targeted outreach, positioning it as a powerful asset in data-driven marketing.

4.5. Integer Programming Models and Optimized Multi-Channel Planning: Results and Insights

This study explores a three-stage model leveraging integer programming to optimize multi-channel resource allocation, aiming to maximize customer value in a targeted and data-driven manner. The model comprises three phases: data collection and preprocessing, behavior forecasting through integer programming, and resource allocation optimization. Through this framework, executives gain actionable insights into user behaviors—such as purchases (${AC}_1$), purchase button clicks (${AC}_2$), and general clicks (${AC}_3$)—enabling precise channel-specific marketing strategies that align resources with channels exhibiting the highest customer engagement.

Table 10. LINGO vs. GAMS.

	LINGO	GAMS	BDE
Customer value (Y)	35,559	34,822	32,036
Execution time (sec.)	5	25	5

compares the performance of different methods, showcasing LINGO’s efficiency in handling complex, high-dimensional tasks. With a customer value (Y) of 35,559 and an execution time of only 5 s, LINGO outperformed GAMS, which achieved a slightly lower customer value (34,822) and required 25 s to complete. The BDE heuristic algorithm, while delivering competitive results with a customer value of 32,036, matched LINGO’s execution time, affirming its robustness in real-time applications. These results highlight LINGO’s computational advantage, making it ideal for scenarios requiring rapid decision-making and efficient resource allocation under complex constraints.

Table 11. Manual planning vs. IP model.

Programming	Channel\Topic	1	2	3	4	5	6	7	Total	Cost	$\mathit{Y}$
Manual	$(Z_{i2}$)	9	2	5	2	5	1	1	25	9830	17,418
	$(Z_{i5}$)	7	0	0	12	6	0	0	25
	$(Z_{i4}$)	15	0	3	6	0	1	0	25
	$(Z_{i2}$)	17	0	3	3	1	0	1	25
	Total	48	2	11	23	12	2	2	100
IP model	$(Z_{i2}$)	5	0	1	1	4	0	14	25	7408	35,559
	$(Z_{i5}$)	6	0	0	0	1	3	15	25
	$(Z_{i4}$)	2	10	0	0	1	0	12	25
	$(Z_{i2}$)	0	3	0	0	1	6	15	25
	Total	13	13	1	1	7	9	56	100

presents a comparison between manual planning and the integer programming (IP) model, illustrating a substantial increase in customer value. The manual approach, yielding a customer value of 17,418, reflects traditional planning based on historical heuristics and executive intuition, limiting optimization potential. The IP model, however, significantly elevated customer value to 35,559—a 104% improvement—while simultaneously reducing costs by 25%. This marked improvement underscores the power of automation in uncovering hidden efficiencies and optimizing channel strategies far beyond manual planning’s reach, positioning the IP model as a transformative tool for strategic resource allocation.

Targeting high-value segments enhances the effectiveness of resource allocation. For example, savings insurance customers (Z__i2) and those focused on long-term financial stability (Z__i5) exhibit a strong preference for specific channels, such as the corporate homepage and targeted advertisements, which offer tailored content that aligns with their risk and engagement preferences. These high-value clusters are integral to fostering sustainable growth, as they represent loyal, high-spend customer groups that contribute significantly to revenue.

The multi-objective equation introduced in this study—$\sum _i^n\sum _j^m{R}_{ij}{X}_{ij}+\sum _i^n\sum _j^m{CL}_{ij}{X}_{ij}+\sum _i^n\sum _j^m{S}_{ij}{X}_{ij}$—balances revenue generation (R__ij), conversion likelihood (CL__ij), and engagement strength (S__ij) across channels. By doing so, the model ensures optimal resource allocation to both acquire new customers and retain existing ones, thus enhancing long-term customer value. The analysis reveals Channel 7 as the most effective, followed by Channels 1 and 2, particularly for complex, high-involvement products like cancer insurance, where personalized and responsive interactions are crucial for fostering trust.

Overall, the IP model demonstrated substantial effectiveness in optimizing channel strategies, particularly for high-value customer clusters, by aligning channels with customer needs and engagement levels. These findings underscore the transformative potential of automated multi-channel planning, offering a scalable solution for maximizing conversion value in competitive markets. This research lays the groundwork for the further exploration of advanced multi-channel optimization models, setting a new standard for strategic resource allocation in the digital age.

4.6. Optimizing Channel Planning with BDE Model: Strategic Insights

This study utilized an integer programming model alongside a BDE algorithm to address complex channel planning challenges in high-dimensional, multi-channel settings. The BDE model was chosen for its efficiency in handling computationally intensive tasks, especially under constraints like varying channel costs and fluctuating customer acquisition expenses, which can hinder traditional integer programming approaches. By leveraging the BDE model, the study aimed to improve the feasibility of resource allocation decisions in dynamic, data-driven environments.

Table 12. Results of the case simulation.

Programming	Dimension $(\mathit{i}\times \mathit{j}$)	Stop Time (s)	Cost	$\mathbf{F}\mathbf{i}\mathbf{t}\mathbf{n}\mathbf{e}\mathbf{s}\mathbf{s}\left(\mathit{Y}\right)$	$\frac{\mathbf{F}\mathbf{i}\mathbf{t}\mathbf{n}\mathbf{e}\mathbf{s}\mathbf{s}\left(\mathit{Y}\right)}{\mathbf{C}\mathbf{o}\mathbf{s}\mathbf{t}}$
IP model	1000 × 7	1	88,896	345,724	3.89
IP model	4000 × 7	8	426,701	1,119,199	2.62
IP model	8000 × 7	-	-	-	-
Manual planning	8076 × 7	-	1,598,640	1,588,665	0.99
BPSO model	8000 × 7	49,278	3,499,570	−197,742,481,306	0.71
BDE model	8000 × 7	49,877	3,356,298	−183,834,253,645	1.26

highlights the comparative advantages of different planning methods. The IP model performs efficiently at smaller scales (1000 × 7 dimensions) with a fitness-to-cost ratio of 3.89, but struggles to scale effectively as dimensionality increases. Manual planning, with a low fitness-to-cost ratio of 0.99, underscores the inefficiency of non-automated approaches. Meanwhile, both the BDE and binary particle swarm optimization (BPSO) models reveal significant optimization potential, although BDE performs better in high-dimensional settings, despite the occasional negative fitness values due to constraint penalties. In particular, the BDE model effectively manages large dimensions (8076 × 7) in 49,877 s, achieving a “Fitness(Y)/Cost” ratio of 1.26, indicating its robustness in high-dimensional scenarios.

The BDE model’s strength lies in dynamically recalculating fitness metrics, allowing it to adapt to fluctuating channel costs and customer engagement trends in real time. By standardizing penalties for constraint violations, the BDE model refines resource allocation strategies, outperforming manual planning in terms of cost-effectiveness and value generation. This adaptability is crucial for real-time decision-making, as the model continuously aligns resource distribution with changing market dynamics, enhancing customer acquisition efficiency.

Table 13. Results of optimizing channel programming.

	Topic 1 $({\mathit{Z}}_1$)	Topic 2 $({\mathit{Z}}_2$)	Topic 3 $({\mathit{Z}}_3$)	Topic 4 $({\mathit{Z}}_4$)	Topic 5 $({\mathit{Z}}_5$)
Customer value	56	64	59	61	66
Recency	33	33	1.3	28	52
Frequency	0.6	0.7	0.7	0.8	1.3
Monetary	947	9334	2533	6799	3158
Offline RFM	43	48	44	46	47
Online RFM	44	54	46	56	69
Homepage	0.22	0.24	0.04	0.22	0.41
Product information	0.76	0.81	0.17	0.96	1.22
Campaign information	0.01	0.12	0.01	0.14	0.16
Pages	1.04	1.36	0.26	1.61	2.27
Sessions	1.05	1.42	1.01	1.73	2.33
Session Duration	964	867	954	817	761
Assigned Cluster\ Cluster Channel	Topic 1 ($Z_1$)	Topic 2 ($Z_2$)	Topic 3 ($Z_3$)	Topic 4 ($Z_4$)	Topic 5 ($Z_5$)
Corporate homepage	2256	951	1417	4555	2708
Outsource advertisement	3092	855	1408	3832	1924
Organic search	9630	891	4214	8520	6317
Keyword advertisement	522	221	612	1120	923
Owned media	418	77	256	2491	1180
Cross-cooperation	2381	399	1097	1076	1956
Corporate homepage	2256	951	1417	4555	2708

provides insights into channel optimization across different customer segments. High-value segments, such as Topic 5 (${\mathit{Z}}_5$)—focusing on long-term financial security—show the highest customer value (66), frequent corporate homepage visits, and a preference for keyword advertisements. By prioritizing channels like corporate homepage and owned media for ${\mathit{Z}}_5$, the company can foster deeper engagement and loyalty, meeting this segment’s need for stability-oriented content. In contrast, Topic 2 (${\mathit{Z}}_2$)—high-spending savings insurance customers—rely on organic search and detailed external advertisements, suggesting that information-rich channels may better support their decision-making process.

Other segments, like cancer insurance customers (${\mathit{Z}}_4$), exhibit high customer value but interact less frequently. Effective channels for ${\mathit{Z}}_4$ include cross-cooperation and organic search, suggesting that partnerships and informative content can enhance their engagement. Meanwhile, segments with lower engagement, such as small life and accident insurance customers (${\mathit{Z}}_1$ & ${\mathit{Z}}_3$), benefit from targeted retention strategies, including optimized browsing experiences and personalized advertisements on their preferred platforms.

Overall, the BDE model demonstrates a substantial capability in enhancing resource allocation for channel planning, particularly for complex, high-dimensional data environments. By tailoring strategies to specific customer segments and optimizing channels with high conversion potential, the BDE model enables businesses to drive effective, cost-efficient engagement. Future research could extend the application of the BDE model to other sectors, reinforcing its value as a strategic tool for dynamic, customer-focused channel optimization.

4.7. Optimizing Resource Allocation Based on Conversion Value Assessment

Figure 12 illustrates a strategic flowchart designed to enhance resource allocation by identifying the effectiveness of various marketing clusters and channels in converting leads into actions or sales. This flowchart categorizes channels into high, moderate, low, and undefined conversion value paths, guiding resource distribution and engagement strategies based on the observed conversion potential. High-conversion channels, depicted in the flowchart, warrant prioritized investments and intensified marketing efforts, as they have demonstrated significant ROI and engagement capabilities. Moderate-value channels, on the other hand, benefit from steady yet cost-effective engagement strategies to maintain balanced returns without overspending. Refer to Figure 12: Evaluation of conversion value and rules of resource allocation.

The framework underscores the importance of assessing each channel’s unique value proposition, allowing decision-makers to strategically allocate resources toward high-impact opportunities. This approach not only optimizes marketing budgets but also enables more targeted engagement aligned with specific customer segments. The flowchart provides a customizable tool applicable across industries, supporting data-driven decisions on budget reallocation, channel prioritization, and performance tracking to maximize the conversion potential and overall marketing effectiveness.

5. Discussion

This research effectively highlights the significance of integrating qualitative data into predictive models to enhance customer understanding and channel planning. The combination of structured and unstructured data through the LDA model provides a comprehensive approach that addresses previous gaps in data utilization. Integrating qualitative data provides insights into customer sentiment and contextual nuances that quantitative data alone cannot capture, thus enabling more tailored and impactful marketing strategies. Our approach demonstrates tangible improvements compared to existing practices, which have predominantly focused on quantitative forecasting models [19,20]. For example, in the initial validation, models incorporating qualitative feedback, particularly sentiment-related inputs, improved predictive accuracy by up to 15% compared to those relying solely on quantitative data. This enhancement contributes to better product customization and improved customer satisfaction.

While past studies have recognized the value of customer feedback [21], they often lacked a robust method to systematically incorporate such data into predictive frameworks. Our methodology represents a step forward by standardizing the integration of qualitative data into predictive frameworks, overcoming the limitations of previous models that relied heavily on structured, quantitative data. Examples of qualitative data include customer reviews, open-ended survey responses, and product feedback, which help reveal deeper customer motivations and pain points. This approach enriches predictive capabilities and ensures a more nuanced understanding of customer needs. This study builds upon previous models by Kwon et al. [22] and Pallant et al. [23], emphasizing data-driven customer understanding. Unlike traditional quantitative models that rely on structured databases, our approach integrates unstructured qualitative data, addressing limitations highlighted by De Vries and Carlson [24]. This integration facilitates more dynamic customer insights, positioning this study at the forefront of multi-channel customer journey optimization. This advancement confirms findings from Johansson and Kask [25] regarding the value of unstructured data and expands on them by showcasing how structured and unstructured data can jointly improve predictive outcomes. Our methodology, including a three-stage framework combining deep-learning techniques with heuristic algorithms, contrasts with previous models that did not address how to practically implement this integration. The approach presented here demonstrates that, when analyzed through LDA and deep learning, qualitative insights provide significant improvements in customer journey mapping and channel planning.

The study uncovered several unexpected insights related to data quality and consumer behavior. While applying our model to the financial sector in Taiwan, we observed inconsistencies in customer feedback that influenced predictive outcomes. Inconsistencies included regional variations in sentiment expressions and differences in linguistic nuances that affected the model’s ability to accurately interpret qualitative data. These findings suggest that, while AI-driven models are powerful, their effectiveness can be limited by data quality and regional differences. To mitigate these challenges, future models should consider region-specific sentiment lexicons and culturally adapted preprocessing techniques to improve data standardization. These findings underscore the need for AI-driven models to account for linguistic and cultural contexts, particularly in multinational or multicultural applications. Additionally, an unanticipated trend was the significant correlation between customer engagement and specific types of qualitative feedback. This insight indicates that not all qualitative data contribute equally to predictive accuracy, a nuance that future studies should explore further. Understanding these distinctions can help tailor data collection and model-training processes to prioritize high-impact qualitative inputs.

AI-driven optimization techniques hold significant potential for industries with varied customer behaviors, such as retail and healthcare. For instance, AI can forecast seasonal purchasing trends in retail, while, in healthcare, it can predict patient needs and tailor engagement strategies. However, these models may encounter limitations in regions with less advanced digital infrastructure, as data quality and availability are crucial for accurate predictions and adaptive insights. Broadly, AI-driven customer journey optimization can transform businesses within the knowledge economy by fueling organizational innovation, enhancing competitiveness, and enabling data-driven decision-making. By optimizing customer interactions, companies gain a nuanced understanding of customer needs, accelerating innovation, improving responsiveness to market changes, and supporting agile strategy adjustments. This contributes to economic development by fostering knowledge creation, increasing customer retention, and promoting resource-efficient practices. Optimizing customer journeys aligns with sustainable development by enhancing resource efficiency and reducing waste in marketing. Effective targeting minimizes redundant customer interactions, conserving resources. Additionally, data-driven decisions support sustainability by ensuring precise resource allocation, preventing unnecessary expenditure, and enabling companies to invest in more eco-friendly practices. This framework also aligns with sustainable growth principles by fostering long-term customer relationships and trust, contributing to stable economic development.

6. Conclusions

This study proposes a systematic, data-driven recommendation framework to support customer journey management in the financial and e-commerce sectors, based on a three-stage customer purchase process. While the techniques applied—such as topic modeling, machine learning, and heuristic optimization—are well-established, their integration in this context provides a practical mechanism for linking customer behavior to actionable marketing strategies. Unlike traditional recommendation systems that often rely on structured transaction data and collaborative filtering, the proposed model incorporates unstructured customer reviews and multi-channel behavioral data, allowing for a more comprehensive analysis of customer preferences and actions.

The model also addresses several limitations found in existing systems, including data sparsity, cold-start problems, and limited personalization. By applying a thematic analysis to extract features from unstructured data and leveraging predictive models to forecast the conversion value at different stages, the system enhances its ability to recommend suitable marketing channels and allocate budgets more effectively. Furthermore, the inclusion of optimization methods allows the model to consider internal enterprise constraints, thus evolving from a descriptive or predictive system into a prescriptive one tailored for real-world decision-making.

While the model demonstrated a promising performance in the context of e-commerce and financial services in Taiwan, its generalizability remains a limitation. Future research could explore its application in other data-intensive sectors such as healthcare and manufacturing, where personalized service paths, resource allocation, and process optimization are also critical. Incorporating data from multiple organizations would further strengthen the model’s representativeness and adaptability. Ethical considerations, such as data privacy and algorithmic bias, should continue to be addressed through ongoing audits and adherence to data governance standards.

In summary, although the techniques used in this study are not novel, their thoughtful integration into a systematic recommendation mechanism contributes to the practical advancement of customer journey analytics. The results offer managerial insights into how organizations can enhance customer engagement, optimize resource allocation, and support more responsive and adaptive strategies in data-driven environments.