Hidden Variable Models in Text Classification and Sentiment Analysis

Abstract

In this paper, we are proposing extensions to the multinomial principal component analysis (MPCA) framework, which is a Dirichlet (Dir)-based model widely used in text document analysis. The MPCA is a discrete analogue to the standard PCA (it operates on continuous data using Gaussian distributions). With the extensive use of count data in modeling nowadays, the current limitations of the Dir prior (independent assumption within its components and very restricted covariance structure) tend to prevent efficient processing. As a result, we are proposing some alternatives with flexible priors such as generalized Dirichlet (GD) and Beta-Liouville (BL), leading to GDMPCA and BLMPCA models, respectively. Besides using these priors as they generalize the Dir, importantly, we also implement a deterministic method that uses variational Bayesian inference for the fast convergence of the proposed algorithms. Additionally, we use collapsed Gibbs sampling to estimate the model parameters, providing a computationally efficient method for inference. These two variational models offer higher flexibility while assigning each observation to a distinct cluster. We create several multitopic models and evaluate their strengths and weaknesses using real-world applications such as text classification and sentiment analysis.

1. Introduction

In this fast-paced world of technological advances, one of the most significant contributing factors has been the emergence of various digital data forms, opening opportunities in different fields to gather helpful information. Everyday, massive amounts of digital data are stored in digital data archives. The same distinction can be made for the enormous quantity of textual data available on the Internet. Therefore, it is critical to developing effective and scalable statistical models to extract hidden knowledge from such rich data [1].

One of the main challenges in the statistical analysis of textual data is capturing and representing their complexity. Different approaches have been applied to deal with this problem. Furthermore, due to information technology’s rapid development, vast quantities of scientific documents are now freely available to be mined. Thus, the analysis and mining of scientific documents have been very active research areas for many years.

Data projection and clustering are crucial for document analysis, with projections aimed at creating low-dimensional, meaningful data representations and clustering and grouping similar data patterns [2,3]. Traditionally, these methods are studied separately, but they intersect in many applications [3]. K-means clustering, though widely used for creating compact cluster representations, does not fully capture document semantics. This gap has led to the adoption of machine learning and deep learning for text mining challenges, including text classification [4], summarization [5], segmentation [6], topic modeling [7], and sentiment analysis [8].

In this paper, we will focus on topic modeling aspects. Topic models are generally classified into two categories: those based on matrix decomposition, like singular value decomposition (SVD), and generative models [9]. The matrix decomposition approach, such as probabilistic latent semantic analysis (PLSA) [10,11], analyzes text via mining and requires a deep understanding of the corpus structure. PLSA, also known as probabilistic latent semantic indexing (pLSI) [11], represents documents as a mix of topics by performing matrix decomposition on the term–document matrix and is effective in identifying relevant words for each topic. In contrast, the generative approach of topic modeling focuses on the context of words across the entire document corpus. These models use latent variable models, and a document is treated as a combination of various topics, each represented by a random vector of words [3].

Meanwhile, the research by [12] indicates that while the probabilistic latent semantic indexing (pLSI) model offers some insights, it falls short in clustering and as a generative model due to its inability to generalize to new documents. To address these limitations, latent Dirichlet allocation (LDA) [12] was introduced, enhancing pLSI using a Dirichlet distribution for topic mixtures. LDA stands out as a more effective generative model, though it still lacks robust clustering capabilities [3]. The integration of clustering and projection into a single framework has been a recent focus in this field, recognizing the need to combine these two approaches [13,14].

The LDA model [12] has been proposed to solve these shortcomings, and this model has proved to be an efficient and scalable data processing method [15,16].

The main issue with current text analysis models is their failure to clearly define a probability model encompassing hidden variables and assumptions [11,17,18,19]. To address this, variational Expectation Maximization (EM) has been utilized, notably in Multinomial PCA (MPCA), which links topics to latent mixture proportions in a probabilistic matrix factorization framework [19,20]. Extensions of LDA, like its hierarchical [21] and online versions [22], have been developed, although they lack the integration of Dirichlet priors in modeling. Researchers have explored alternative models using conjugate priors and methods, like Gibbs sampling and Markov Chain Monte Carlo (MCMC) methods [23], which, despite their effectiveness, require longer convergence times compared to the variational Bayes approach.

In this paper, we introduce two novel models, GDMPCA and BLMPCA, that significantly improve text classification and sentiment analysis by combining generalized Dirichlet (GD) and Beta-Liouville (BL) distributions for a more in-depth understanding of text data complexities [16,24,25]. Both models employ variational Bayesian inference and collapsed Gibbs sampling for efficient and scalable computational performances, which is critical for handling large datasets.

The generalized Dirichlet (GD) distribution, introduced in [26], exhibits a more flexible covariance structure than its Dirichlet counterpart. Similarly, the Beta-Liouville (BL) distribution, enriched with additional parameters, offers improved adjustments for data spread and modeling efficiency. Our contribution was validated through a rigorous empirical evaluation on real-world datasets, which demonstrated our models’ superior accuracy and adaptability. This work represents a significant step forward in text analysis methodologies, bridging theoretical innovation with practical application, with experimental results demonstrating the relationships between these models.

The structure of the rest of this paper is as follows. In Section 2, we cover the related work. Section 3 introduces the extension of MPCA with generalized Dirichlet and Beta-Liouville distributions with all the details about the parameters estimation. Section 4 is devoted to the discussion of the experimental results. Finally, we conclude our work in Section 6.

2. Related Work

In this section, we delve into the vast array of the literature on topic modeling approaches. The foundation of this field is built upon traditional topic modeling techniques [10,11], with significant contributions from topic-class modeling [27,28,29] and the nuanced exploration of global and local document features [30,31].

Innovative strides have been made with the introduction of a two-stage topic extraction model for bibliometric data analysis, employing word embeddings and clustering for a more refined topic analysis [32]. This approach provides a nuanced lens with which to view the thematic undercurrents of scholarly communication.

The landscape of sentiment analysis is similarly evolving, with breakthroughs like a term-weighted neural language model paired with a stacked bidirectional LSTM (long short-term memory) framework, enhancing the detection of subtle sentiments like sarcasm in text [33]. Such advancements offer deeper insights into the complexities of language and its sentiments.

Cross-modal sentiment analysis also takes center stage with deep learning techniques, as seen in works that identify emotions from facial expressions [34]. These studies, which utilize convolutional neural networks and Inception-V3 transfer learning [35], pave the way for multimodal sentiment analysis, potentially influencing strategies for textual sentiment analysis.

A hybrid deep learning method has been introduced for analyzing sentiment polarities and knowledge graph representations, particularly focusing on health-related social media data, like tweets on monkeypox [36]. This underscores the importance of versatile and dynamic models in interpreting sentiment from real-time data streams.

Collectively, these contemporary works highlight the expansive applicability and dynamic nature of deep learning across various domains and data types. Their inclusion in our review underlines the potential for future cross-disciplinary research, expanding the scope of sentiment analysis to include both text and image data.

Alongside these emerging approaches, well-established techniques such as principal component analysis (PCA) and its text retrieval counterpart, latent semantic indexing [37], continue to be pivotal. Probabilistic latent semantic indexing (pLSI) [11] and latent Dirichlet allocation (LDA) [12] further enrich the discussion on discrete data and topic modeling. Non-negative matrix factorization (NMF) [17] has also demonstrated effectiveness, emphasizing the need for models that can simultaneously handle clustering and projection. In addressing a gap in the literature, a multinomial PCA model has been proposed to offer probabilistic interpretations of the relationships between documents, clusters, and factors [19].

Our focus on the MPCA model and its extensions aims to consolidate these disparate strands of research, presenting a comprehensive framework for topic modeling that accounts for both clustering and projection, reflecting the ongoing dialogue within the research community on these topics.

2.1. Multinomial PCA

Probabilistic approaches to reducing dimensions generally hypothesize that each observation $x_i$ corresponds to a hidden variable, referred to as a latent variable ${\theta }_i$. This latent variable exists within a subspace of dimension K. Typically, the relationship involves a linear mapping ($\beta$) within the latent space coupled with a probabilistic mechanism.

In the probabilistic PCA (pPCA) framework, as detailed in the work in [38], it is posited that each observation $x_i$ originates from a standard Gaussian distribution $N_K(0_K;Z_K)$. The assumption of a Gaussian distribution is also employed for the conditional distribution of the observations:

$$x_i|{\theta }_i\sim {\mathcal{N}}_v(\beta {\theta }_i+\mu ,{\sigma }^2{\mathcal{Z}}_V)$$

(1)

where Z is a “standard” normal distribution, ($\beta$, $\mu$) are the model parameters, and ${\sigma }^2$ is the variance that is learned using maximum likelihood inference.

The Gaussian assumption is suitable for real-valued data, yet it is less applicable to non-negative count data. Addressing this, [19] introduced a variant of pPCA where the latent variables are modeled as a discrete probability distribution, specifically using a Dirichlet distribution, where as $m\sim Dir\left(\alpha \right)$,

$$\mathcal{D}(m;\alpha )=\frac{1}{Z\left(\alpha \right)}\prod _{k=1}^K{m}_k^{{\alpha }_k-1}\left(m\right)$$

(2)

where $\alpha =({\alpha }_1,\dots ,{\alpha }_k)\ge 0$.

Then, the probabilistic function is assumed to be multinomial:

$$\begin{array}{c}\hfill m\sim Dirichlet\left(\alpha \right)\\ \hfill C\sim Multinomial(m,L),\\ \hfill w_k\sim Multinomial({\Omega }_k,c_k)\end{array}$$

(3)

The variables m and w are assumed to be hidden parameters for each document. For the parameter estimation of MPCA, first, the variable $\Omega$ is estimated using the Dirichlet prior on m using parameters $\alpha$ [19]. The likelihood model for the MPCA is given as follows [20]:

$$p(m,w|\alpha ,\Omega )=\frac{\Gamma \left({\sum }_k{\alpha }_k\right)}{{\prod }_k\Gamma \left({\alpha }_k\right)}{C}_{w_{1,1},w_{1,2}\dots ,w_{k,1},w_{1,J}\dots w_{K,J}}^L\prod _K{m}_K^{a_k-1}\prod _{k,j}{m}_k^{w_{k,j}}{\Omega }_{k,j}^{w_{k,j}}$$

(4)

In the MPCA model, it is assumed that each observation $x_i$ can be broken down into a probabilistic mixture of K topics that represent the whole corpus. Then, m indicates the observation with mixture weights in the latent space, and $\Omega$ is a global parameter that encapsulates all the information at the corpus level.

As a result, the following equation is derived when the hidden variables have a Dirichlet prior [19]:

$$\begin{array}{c}\hfill m\sim Dirichlet\left(\alpha \right)\\ \hfill {\Omega }_k\sim Dirichlet\left(2f\right)\end{array}$$

(5)

The following updated formula converges to the local maximum $\log p(\Omega ,{\alpha }_m|r)$, where $\frac{\Gamma \left({\sum }_k{\alpha }_k\right)}{{\prod }_k\Gamma \left({\alpha }_k\right)}$ is a normalizing constant for the Dirichlet, and r is the total row-wise number of words in the document representation with the k component [19]:

$${\gamma }_{j,k,\left[i\right]}=\frac{\Gamma \left({\sum }_k{\alpha }_k\right)}{{\prod }_k\Gamma \left({\alpha }_k\right)}\frac{1}{{\Omega }_{k,j}{m}_{k,\left[i\right]}}$$

(6)

$$m_{k,\left[i\right]}=\frac{\Gamma \left({\sum }_k{\alpha }_k\right)}{{\prod }_k\Gamma \left({\alpha }_k\right)}(a_k-1+\sum _j{r}_{j,\left[i\right]}{\gamma }_{j,k,\left[i\right]})$$

(7)

Equations (8) and (9) are the parameters for a multinomial and a Dirichlet, respectively.

$${\Omega }_{k,j}=\frac{\Gamma \left({\sum }_k{\alpha }_k\right)}{{\prod }_k\Gamma \left({\alpha }_k\right)}(2f+\sum _i{r}_{j,\left[i\right]}{\gamma }_{j,k,\left[i\right]})$$

(8)

$${\Psi }_0\left(a_k\right)-{\Psi }_0(\sum _k{a}_k)=\frac{\log (1/k)+{\sum }_i\log \left(m_{k,\left[i\right]}\right)}{1+I}$$

(9)

According to the exponential family definition (Appendix A), Equation (9) rewrites $\alpha$ in terms of its dual representation. Minka’s approach is used to derive $\alpha$, where $n_k$ is the number of times that the outcome was k [39]:

$$\begin{array}{cc}\hfill & n_k=\sum _i\delta (x_i-k)\hfill \\ \hfill & n_i=\sum _k{n}_{i}k\hfill \end{array}$$

(10)

$${\alpha }_k^{new}=a_k\frac{{\sum }_i\Psi (n_{ik}+a_k)-\Psi \left(a_k\right)}{{\sum }_i\Psi (n_{i}k+{\sum }_k{a}_k)-\Psi \left({\sum }_k{a}_k\right)}$$

(11)

Connection between MPCA and LDA

The multinomial PCA model is closely connected to LDA [12] and forms the foundation over several topic models.

In text analysis, an observation typically refers to a document represented by a sequence of tokens or words, denoted as $w_i=w_{in},$ where $n=1\dots L_i$. Each word $w_{in}$ within a document i is initially linked to a topic, which is specified by a vector $z_{in}$ that is derived from a $Multinomial(1,{\beta }_k)$ distribution. The model for any given document i can be described as follows:

$$\begin{array}{cc}\hfill & {\theta }_k\sim Dirichlet\left(\alpha \right)\hfill \\ \hfill & z_{in}|{\theta }_i\sim Multinomial(1,{\theta }_i)\hfill \\ \hfill & w_{in}|z_{ink}\sim Multinomial(1,{\beta }_k)\hfill \end{array}$$

(12)

At the word level, marginalizing on $z_{in}$ yields a distribution similar to Equation (3):

$$w_{in}|{\theta }_i\sim Multinomial(1,\beta {\theta }_i)$$

(13)

Furthermore, the distinction between LDA and MPCA is that LDA is a word-level model, whereas MPCA is a document-level model. Since GDMPCA and BLMPCA are new variations of MPCA, both new models are assumed to be document-level in the following proposed approaches.

3. Proposed Models

In this section, we present two pioneering models, generalized Dirichlet Multinomial Principal Component Analysis (GDMPCA) and Beta-Liouville Multinomial Principal Component Analysis (BLMPCA), which were designed to revolutionize text classification and sentiment analysis. At the core of our approaches is the integration of generalized Dirichlet and Beta-Liouville distributions, respectively, into the PCA framework. This integration is pivotal, as it allows for a more nuanced representation of text data, capturing the inherent sparsity and thematic structures more effectively than traditional methods.

The GDMPCA model leverages the flexibility of the generalized Dirichlet distribution to model the variability and co-occurrence of terms within documents, enhancing the model’s ability to discern subtle thematic differences. On the other hand, the BLMPCA model utilizes the Beta-Liouville distribution to precisely capture the polytopic nature of texts, facilitating a deeper understanding of sentiment and thematic distributions. Both models employ variational Bayesian inference, offering a robust mathematical framework that significantly improves computational efficiency and scalability. This approach not only aids in handling large datasets with ease but also ensures that the models remain computationally viable without sacrificing accuracy.

To elucidate the architecture of our proposed models, we delve into the algorithmic underpinnings, detailing the iterative processes that underlie the variational Bayesian inference technique. This includes a comprehensive discussion of the optimization strategies employed to enhance convergence rates and ensure the stability of the models across varied datasets. Moreover, we provide a comparative analysis, drawing parallels and highlighting distinctions between our models and existing text analysis methodologies. This comparison underscores the superior performances of GDMPCA and BLMPCA in terms of accuracy, adaptability, and computational efficiency, as evidenced by an extensive empirical evaluation on diverse real-world datasets.

Our exposition on the practical implications of these models reveals their broad applicability across numerous domains, from automated content categorization to nuanced sentiment analysis in social media texts. The innovative aspects of the GDMPCA and BLMPCA models, coupled with their empirical validation, underscore their potential to set a new standard in text analysis, offering researchers and practitioners alike powerful tools for uncovering insights from textual data.

Table 1. Parameters of generalized Dirichlet and Beta-Liouville distributions.

Parameter	Generalized Dirichlet (GDMPCA)	Beta-Liouville (BLMPCA)
$\xi$	Parameters of GD distribution	Not applicable
${\rm Y}$	Not applicable	Parameters of BL distribution
m	Mixture weights (GD)	Mixture weights (BL)
z	Topic assignments	Topic assignments
w	Words in documents	Words in documents
$\Omega$	Multinomial parameters (words)	Multinomial parameters (words)
L	Number of words per document	Number of words per document
$C,{\Omega }_k,c_k$	Multinomial parameters for topics	Multinomial parameters for topics

summarizes the relevant variables for the proposed models.

3.1. Generalized Dirichlet Multinomial PCA

Bouguila [40] demonstrated that when mixture models are used, the generalized Dirichlet (GD) distribution is a reasonable alternative to the Dirichlet distribution for clustering count data.

As we mentioned previously, the GD distribution, like the Dirichlet distribution, is a conjugate prior to the multinomial distribution. Furthermore, the GD has a more general covariance matrix [40].

Therefore, the variational Bayes approach will be utilized to develop an extension of the MPCA model incorporating the generalized Dirichlet assumption. GDMPCA is anticipated to perform effectively because the Dirichlet distribution is a specific instance of the GD [41]. Like MPCA, GDMPCA is a fully generative model applied to a corpus. It considers a collection of M documents represented as the corpus, denoted by $D=\{w_1,w_2,\dots ,w_M\}$. Each document $w_m$ consists of a sequence of $N_m$ words, expressed as $w_m=(w_{m1},\dots ,w_{m{N}_m})$. Words within a document are represented by binary vectors from a vocabulary of V words, where if the j-th word is selected, $w_j^n=1$, and if not, $w_j^n=0$ [42]. The GDMPCA model then describes the generation of each word in the document through a series of steps involving c, a $d+1$ dimensional binary vector of topics:

$$\begin{array}{c}\hfill m\sim GD\left(\xi \right)\\ \hfill z\sim Multinomial(m,L)\\ \hfill w_k\sim Multinomial({\Omega }_k,c_k)\end{array}$$

(14)

If the i-th topic is chosen, $z_i^n=1$, and in other cases, $z_i^n=0$. $m=(m_1,\dots ,m_{d+1})$, where $m_{d+1}=1-{\sum }_{i=1}^d{m}_i$.

The multinomial probability $p(w_n\mid z_n,{\Omega }_w)$ is conditioned on the variable $z_n$. The distribution $GD\left(\xi \right)$ is a d-variate generalized Dirichlet distribution characterized by the parameter set $\xi =(a_1,b_1,\dots ,a_d,b_d)$, with its probability distribution function denoted by p, where ${\gamma }_i=b_i-a_{i+1}-b_{i+1}$ [42]:

$$p(m_1,\dots ,m_d|\xi )=\prod _{i=1}^d\frac{\Gamma (a_l+b_l)}{\Gamma \left(a_l\right)\Gamma \left(b_l\right)}{m}_i^{a_l-1}{(1-\sum _{j=1}^i{m}_j)}^{{\gamma }_i}$$

(15)

The GD distribution simplifies to a Dirichlet distribution when $b_i=a_{(i+1)}+b_{(i+1)}$.

The mean and the variance matrix of the GD distribution are as follows [41]:

$$E\left(m_i\right)=\frac{a_l}{a_l+b_l}\prod _{k=1}^{i-1}\frac{b_k}{a_k+b_k}$$

(16)

$$var\left(m_i\right)=E\left(m_i\right)\left(\frac{a_l+1}{a_l+b_l+1}\prod _{k=1}^{i-1}\frac{b_k+1}{a_k+b_k}+1-E\left({\theta }_i\right)\right)$$

(17)

and the covariance between $m_i$ and $m_j$ is given by

$$cov(m_i,m_j)=E\left(m_j\right)\left(\frac{a_l}{a_l+b_l+1}\prod _{k=1}^{i-1}\frac{b_k+1}{a_k+b_k}+1-E\left(m_i\right)\right)$$

(18)

The covariance matrix of the GD distribution offers greater flexibility compared to the Dirichlet distribution, due to its more general structure. This additional complexity allows for an extra set of parameters, providing $d-1$ additional degrees of freedom, which enables the GD distribution to more accurately model real-world data. Indeed, the GD distribution fits count data better than the commonly used Dirichlet distribution [43]. The Dirichlet and GD distributions are both members of the exponential family (Appendix A). Furthermore, they are also conjugate priors to the multinomial distribution. As a result, we can use the following method to learn the model.

The likelihood for the GDMPCA is given as follows:

$$\begin{array}{cc}\hfill p(m,w|\xi ,\Omega )& =\frac{\Gamma (a_i+b_i)}{\Gamma \left(a_i\right)\Gamma \left(b_i\right)}{z}_{w_{1,1},w_{1,2}\dots ,w_{k,1},w_{1,J}\dots w_{K,J}}^L{m}_k^{b_{k-1}-1}\prod _{i=1}^{k-1}[m_i^{a_{i-1}}\hfill \\ & {(\sum _{j=1}^k{m}_j)}^{b_{i-1}+(a_i+b_i)}]\prod _{k,j}{m}_k^{w_{k,j}}{\Omega }_{k,j}^{w_{k,j}}\hfill \end{array}$$

(19)

Hence, when hidden variables are assigned GD priors, and given a defined universe of words, we use an empirical prior derived from the observed proportions of words in the universe, denoted by f, where ${\sum }_k{f}_k=1$. The equation, then, is structured as follows:

$$\begin{array}{c}\hfill m\sim GD\left(\xi \right)\\ \hfill {\Omega }_k\sim GD\left(2f\right)\end{array}$$

(20)

where 2 shows the small size of the prior sample size.

First, we will calculate the parameters of GD utilizing the Hessian matrix as described in Appendix B.1.2, following Equations (19) and (20). To find the optimal variational parameters, we minimize the Kullback–Leibler (KL) divergence between the variational distribution and the posterior distributions $p(m,w|\Omega ,\xi )$. This is achieved through a repetitive fixed-point method. We specify the variational parameters as follows:

$$q(m,c|\gamma ,\Phi )=q\left(m\right|\gamma )\prod _{k=1}^{K}q\left(c_k\right|{\Phi }_k)$$

(21)

As an alternative to the posterior distribution $p(m,c,w,\xi ,\Omega )$, we determine the variational parameters $\gamma$ and $\Phi$ through a detailed optimization process outlined subsequently. To simplify, Jensen’s inequality is applied to establish a lower bound on the log likelihood, which allows us to disregard parameters $\gamma$ and $\Phi$ [44]:

$$\begin{array}{cc}\hfill \log p\left(w\right|\xi ,\Omega )& =\log \int \sum _{z}p(m,c,w|\xi ,\Omega )dm\hfill \\ \hfill & =\log \int \sum _z\frac{p(m,c,w|\xi ,\Omega \left)q\right(m,c)}{q(m,c)}dm\hfill \\ \hfill & \ge \int \sum _z\log p(m,c,w|\xi ,\Omega )q(m,c)dm\hfill \\ \hfill & -\int \sum _{z}q(m,c)\log q(m,c)dm\hfill \\ \hfill & =E\left[\log p\right(m,c,w|\xi ,\Omega )]-E[\log q(m,c)]\hfill \end{array}$$

(22)

Consequently, Jensen’s inequality provides a lower bound on the log likelihood for any given variational distribution $q(m,c|\gamma ,\Phi )$.

If the right-hand side of Equation (22) is denoted as $\mathcal{L}(\gamma ,\Phi ;\xi ,\Omega )$, the discrepancy between the left and right sides of this equation represents the KL divergence between the variational distribution and the true posterior probabilities. This re-establishes the importance of the variational parameters, leading to the following expression:

$$\log p\left(w\right|\xi ,\Omega )=\mathcal{L}(\gamma ,\Phi ;\xi ,\Omega )+D(q(m,c|\gamma ,\Phi \left)\right)\left|p\right(m,c|x,\xi ,\Omega )$$

(23)

As demonstrated in Equation (23), maximizing the lower bound $\mathcal{L}(\gamma ,\Phi ;\xi ,\Omega )$ with respect to $\gamma$ and $\Phi$ is equivalent to minimizing the Kullback–Leibler (KL) divergence between the variational posterior probability. By factorizing the variational distributions, we can describe the lower bound as follows:

$$\begin{array}{cc}\hfill \mathcal{L}(\gamma ,\Phi ;\xi ,\Omega )& =E_q\left[\log p\left(m\right|\xi )\right]+E_q\left[\log p\left(c\right|m)\right]+E_q\left[\log p\left(w\right|c,\Omega )\right]\hfill \\ & -E_q\left[\log q\left(m\right)\right]-E_q\left[\log q\left(c\right)\right]\hfill \end{array}$$

(24)

After that, we can extend Equation (A7) in terms of the model parameters $(\xi ,\Omega )$ and variational parameters $(\gamma ,\Phi )$ (A13).

In order to find ${\varphi }_{nl}$, we proceed to maximize with the respect to ${\varphi }_{nl}$, so we have the following equations:

$$\begin{array}{cc}\hfill \mathcal{L}\left[m_{nl}\right]& =m_{nl}(\Psi \left({\gamma }_l\right)-\Psi ({\gamma }_l+\Phi ))+m_{nl}\log {\Omega }_{w\left(lv\right)}-m_{nl}\log m_{nl}\hfill \\ & +{\lambda }_n(\sum _{ll=1}^{d+1}{m}_{n\left(ll\right)}-1)\hfill \end{array}$$

(25)

and therefore, we have

$$\frac{\partial \mathcal{L}}{\partial {\varphi }_{nl}}=(\Psi \left({\gamma }_l\right)-\Psi ({\gamma }_l+\Phi ))+\log {\Omega }_{lv}-\log {\varphi }_{nl}-1+{\lambda }_n$$

(26)

Setting the above equation to zero leads to

$$m_{nl}={\Omega }_{lv}{e}^{({\lambda }_n-1)}{e}^{(\Psi \left({\gamma }_l\right)-\Psi ({\gamma }_l+\Phi ))}$$

(27)

Next, we maximize Equation (A13) with respect to ${\gamma }_i$. The terms containing ${\gamma }_i$ are

$$\begin{array}{cc}\hfill \mathcal{L}\left[{\xi }_q\right]& =\sum _{l=1}^d[a_l(\Psi \left({\gamma }_l\right)-\Psi ({\gamma }_l+\Phi ))+(\Psi \left({\gamma }_l\right)-\hfill \\ \hfill & \Psi ({\gamma }_l+\Phi )\left)(b_l-a_{l+1}-b_{l+1})\right]\hfill \\ \hfill & +\sum _{n=1}^N{m}_{nl}(\Psi \left({\gamma }_l\right)-\Psi ({\gamma }_l+\Phi )+\sum _{n=1}^N{m}_{n(d+1)}(\Psi \left({\gamma }_d\right)-\Psi ({\gamma }_d+{\Phi }_d))\hfill \\ \hfill & -\sum _{l=1}^d(\log \Gamma ({\gamma }_l+{\Phi }_l)-\log \Gamma \left({\gamma }_l\right)-\log \Gamma \left({\Phi }_l\right))\hfill \\ \hfill & +\sum _{l=1}^d(\Psi \left({\gamma }_l\right)-{\gamma }_l(\Psi ({\gamma }_l+{\Phi }_l))\hfill \\ \hfill & +(\Psi (\Phi )-\Psi (\Phi +{\gamma }_l))(\Phi -{\gamma }_{l+1}-{\Phi }_{l+1})\left)\right)\hfill \end{array}$$

(28)

Setting the derivative of the above equation to zero leads to the following updated parameters:

$${\gamma }_l=a_l+\sum _{n=1}^N{m}_{nl}$$

(29)

$${\Phi }_l=b_l+\sum _{n=1}^N\sum _{l=l+1}^{d+1}{m}_{n\left(l\right)}$$

(30)

The challenge of deriving empirical Bayes estimates for the model parameters $\xi$ and $\Omega$ is tackled by utilizing the variational lower bound as a substitute for the marginal log probability, using variational parameters $\gamma$ and $\Phi$. The empirical Bayes estimates are then determined by maximizing this lower bound in relation to the model parameters. Until now, our discussion has centered on the log probability for a single document; the overall variational lower bound is computed as the sum of the individual lower bounds from each document. In the M-step, this bound is maximized with respect to model parameters $\xi$ and $\Omega$. Consequently, the entire process is akin to performing a coordinate ascent as outlined in Equation (31). We formulate the update equation for estimating $\Omega$ by isolating terms and incorporating Lagrange multipliers to maximize the bound with respect to $\Omega$:

$$\mathcal{L}[\Omega ]=\sum _{d=1}^M\sum _{n=1}^{N_s}\sum _{l=1}^{K+1}\sum _{j=1}^V{m}_{dnl}{w}_{dn}^j\log {\Omega }_{\left(lj\right)}+\sum _{l=1}^{K+1}{\lambda }_l(\sum _{j=1}^V{\Omega }_{w\left(ij\right)})$$

(31)

To derive the update equation for ${\Omega }_{\left(lj\right)}$, we take the derivative of the variational lower bound with respect to ${\Omega }_{\left(lj\right)}$ and set this derivative to zero. This step ensures that we find the point where the lower bound is maximized with respect to the parameter ${\Omega }_{\left(lj\right)}$.

$${\Omega }_{\left(lj\right)}\propto \sum _{d=1}^M\sum _{n=1}^{N_d}{m}_{dnl}{w}_{dn}^j$$

(32)

The updates mentioned lead to convergence at a local maximum of the lower bound of $\log p(\Omega ,\xi |r)$, which is optimal for all product approximations of the form $q\left(m\right)q\left(w\right)$ for the joint probability $p(m,w|\Omega ,\xi ,r)$. This approach ensures that the variational parameters are adjusted to optimally approximate the true posterior distributions within the constraints of the model.

$${\Phi }_l=\frac{\Gamma (a_i+b_i)}{\Gamma \left(a_i\right)\Gamma \left(b_i\right)}{m}_{nl}(\Psi \left({\gamma }_l\right)-\Psi ({\gamma }_l+\Phi ))$$

(33)

$${\gamma }_l=a_l+\sum _{n=1}^N{m}_{nl}$$

(34)

$${\Omega }_{\left(lj\right)}=\frac{\Gamma (a_i+b_i)}{\Gamma \left(a_i\right)\Gamma \left(b_i\right)}(2f_j\sum _{d=1}^M\sum _{n=1}^{N_d}{m}_{dnl}{w}_{dn}^j)$$

(35)

Collapsed Gibbs Sampling Method

Utilizing the fundamental procedure of the GD distribution as delineated in the all-encompassing generative formula $p(c,z,\theta ,\phi ,w|,\Omega ,\xi ,\mu )$ within our innovative methodology, we can express it in the following manner:

$$p(c,z,\theta ,\phi ,w|,\Omega ,\xi ,\mu )=p\left(w\right|\mu )p\left(\theta \right|\Omega )p\left(\phi \right|\xi )\times \prod _{n=1}^{N}p\left(z_n\right|\theta )p\left(x_n\right|z_{nn},\phi )$$

(36)

Here, $p\left(\theta \right|\Omega )$ signifies the GD document prior distribution, where $\Omega =(a_1,b_1,\dots ,a_n,b_n)$ serves as a hyperparameter. Simultaneously, $p\left(\phi \right|\xi )$, with $\xi =({\alpha }_1,{\beta }_1,\dots ,{\alpha }_d,{\beta }_d)$ as its hyperparameters, represents the GD corpus prior distribution. The process of Bayesian inference seeks to approximate the posterior distribution of hidden variables z by integrating out parameters, which can be mathematically depicted as follows:

$$p(c,z|w,\Omega ,\xi )=W{\int }_{\theta }{\int }_{\phi }p(c,z,\theta ,\phi ,|\Omega ,\xi )d\phi d\theta$$

(37)

Crucially, the joint distribution is expressed as a product of Gamma functions, as highlighted in prior research [12,45,46]. This expression facilitates the determination of the expectation value for the accurate posterior distribution:

$$p(z_{ij}=k|c,w,\Omega ,\xi )={\mathrm{E}}_{p\left(z^{-ij}\right|w,c,\Omega ,\xi )}\left[p(z_{ij}=k|z^{-ij},c,w,\Omega ,\xi )\right]$$

(38)

Employing the GD prior results in the posterior calculation as outlined below:

$$\begin{array}{cc}\hfill p(z_{ij}=k|z^{-ij},c,w,\Omega ,\xi )& \propto \left[\frac{(N_{jk}^{-ij}+{\alpha }_{wk})({\beta }_{wk}+{\sum }_{l=k+1}^{K+1}{N}_{jl}^{-ij})}{({\alpha }_{wk}{\beta }_{wk}+{\sum }_{l=k+1}^{K+1}{N}_{jl}^{-ij})}\right]\hfill \\ & \times \left[\frac{(N_{k{v}_{ij}}^{-ij}+a_v)(b_v+{\sum }_{d=v}^{V+1}{N}_{k{d}_{ij}}^{-ij})}{a_v+b_v+{\sum }_{d=v}^{V+1}{N}_{k{d}_{ij}}^{-ij})}\right]=A\left(K\right)\hfill \end{array}$$

(39)

This leads to a posterior probability normalization as follows:

$$p(z_{ij}=k|z^{-ij},x,\Omega ,\xi )=\frac{A\left(k\right)}{{\sum }_{k^{\prime }=1}^{K}A\left(k^{\prime }\right)}$$

(40)

The sequence from Equation (38) to Equation (40) delineates the complete collapsed Gibbs sampling procedure, encapsulated as follows:

$$p(z_{ij}=k|c,w,\Omega ,\xi )={\mathrm{E}}_{p\left(z^{-ij}\right|w,c,\Omega ,\xi )}\left[\frac{A\left(k\right)}{{\sum }_{k^{\prime }=1}^{K}A\left(k^{\prime }\right)}\right]$$

(41)

The implementation of collapsed Gibbs sampling in our GD-centric model facilitates sampling directly from the actual posterior distribution p, as indicated in Equation (41). This sampling technique is deemed more accurate than those employed in variational inference models, which typically approximate the distribution from which samples are drawn [46,47]. Hence, our model’s precision is ostensibly superior.

Upon the completion of the sampling phase, parameter estimation is conducted using the methodologies discussed.

3.2. Beta-Liouville Multinomial PCA

For the Beta-Liouville Multinomial PCA (BLMPCA) model, we define a corpus as a collection of documents with the same assumption described in the GDMPCA section. Hence, we have the following procedure for the model for every single word of the document. The BLMPCA model proceeds with generating every single word of the document with the following steps, where c is a $d+1$-dimensional binary vector of topics defined:

$$\begin{array}{c}\hfill m\sim BL({\rm Y})\\ \hfill z\sim Multinomial(m,L),\\ \hfill w_k\sim Multinomial({\Omega }_k,c_k)\end{array}$$

(42)

In the model described, each topic is represented by a binary variable, where $z_i^n=1$ indicates that the i-th topic is chosen for the n-th word, and $z_i^n=0$ indicates it is not chosen. The vector $z_n$ is a $(D+1)$-dimensional binary vector representing the topic assignments across all $D+1$ topics for a given word. The vector m is defined as $m=(m_1,m_2,\dots ,m_{D+1})$, where $m_{D+1}=1-{\sum }_{i=1}^D{m}_i$ captures the distribution of topic proportions across the document, ensuring that the sum of proportions across all topics equals 1.

A chosen topic is associated with a multinomial prior w over the vocabulary, where ${\Omega }_{w_{ij}}=p(w^j=1|z^i=1)$ describes the probability of the j-th word being selected given that the i-th topic is chosen. This formulation allows for each word in the document to be drawn randomly from the vocabulary conditioned on the assigned topic.

The probability $p\left(w_n\right|z_n,{\Omega }_w)$ is a multinomial probability that conditions on the topic assignments $z_n$ and the topic-word distributions ${\Omega }_w$, effectively modeling the likelihood of each word in the document given the topic assignments.

Additionally, $BL({\rm Y})$ represents a d-variate Beta-Liouville distribution with parameters ${\rm Y}=({\alpha }_1,...,{\alpha }_D,\alpha ,\beta )$. The probability distribution function of this Beta-Liouville distribution encapsulates the prior beliefs about the distribution of topics across documents, accommodating complex dependencies among topics and allowing for flexibility in modeling topic prevalence and co-occurrence within the corpus.

$$\begin{array}{cc}\hfill P({\theta }_1,\dots ,{\theta }_D|{\rm Y})& =\frac{\Gamma \left({\sum }_{d=1}^D{\alpha }_d\right)\Gamma (\alpha +\beta )}{\Gamma \left(\alpha \right)\Gamma \left(\beta \right)}\prod _{d=1}^D\frac{{\theta }_d^{{\alpha }_d-1}}{\Gamma \left({\alpha }_d\right)}\times \hfill \\ \hfill & {\left(\sum _{d=1}^D{\theta }_d\right)}^{\alpha -{\sum }_{l=1}^D{\alpha }_l}\times {\left(1-\sum _{l=1}^D{\theta }_l\right)}^{\beta -1}\hfill \end{array}$$

(43)

A Dirichlet distribution is the special case of BL if ${\beta }_d={\alpha }_{d+1}+{\beta }_{d+1}$ [42,45]. The mean, the variance, and the covariance in the case of a BL distribution are as follows [45]:

$$E\left({\theta }_d\right)=\frac{\alpha }{\alpha +\beta }\frac{{\alpha }_d}{{\sum }_{d=1}^D{\alpha }_d}$$

(44)

$$\begin{array}{cc}\hfill var\left({\theta }_d\right)& ={(\frac{\alpha }{\alpha +\beta })}^2\frac{{\alpha }_d({\alpha }_d+1)}{\left({\sum }_{m=1}^D{\alpha }_m\right)({\sum }_{m=1}^D{\alpha }_m+1)}\hfill \\ \hfill & -E{\left({\theta }_d\right)}^2\frac{{\alpha }_d^2}{{\left({\sum }_{m=1}^D{\alpha }_m\right)}^2}\hfill \end{array}$$

(45)

and the covariance between ${\theta }_l$ and ${\theta }_k$ is given by

$$Cov({\theta }_l,{\theta }_k)=\frac{{\alpha }_l{\alpha }_k}{{\sum }_{d=1}^D{\alpha }_d}\left(\frac{\frac{(\alpha +1)\left(\alpha \right)}{(\alpha +\beta +1)(\alpha +\beta )}}{{\sum }_{d=1}^D{\alpha }_d+1}-\frac{\frac{\alpha }{\alpha +\beta }}{{\sum }_{d=1}^D{\alpha }_d}\right)$$

(46)

The earlier equation illustrates that the covariance matrix of the Beta-Liouville distribution offers a broader scope compared to the covariance matrix of the Dirichlet distribution. For the parameter estimation of BLMPCA, first, the parameter $\Omega$ is estimated using the Beta-Liouville prior on m using parameter ${\rm Y}$ [19]. The likelihood model for the BLMPCA is given as follows:

$$\begin{array}{cc}\hfill p(m,w|{\rm Y},\Omega )& =\frac{\Gamma \left(\alpha \right)\Gamma \left(\beta \right)}{\Gamma \left({\sum }_{d=1}^D{\alpha }_d\right)\Gamma (\alpha +\beta )}{z}_{w_{1,1},w_{1,2}\dots ,w_{k,1},w_{1,J}\dots w_{K,J}}^L[\frac{1}{\Gamma \left({\alpha }_d\right)}{m}_k^{{\alpha }_d-1}+\hfill \\ & \sum _k{m}_k^{\alpha -{\sum }_d{\alpha }_d}+{(1-\sum _k{m}_k)}^{\beta -1}]\prod _{k,j}{m}_k^{w_{k,j}}{\Omega }_{k,j}^{w_{k,j}}\hfill \end{array}$$

(47)

For the Beta-Liouville priors, we have the following:

$$\begin{array}{c}\hfill m\sim BL({\rm Y})\\ \hfill {\Omega }_k\sim BL\left(2f\right)\end{array}$$

(48)

In the following step, we will estimate the parameters for $\Omega$ using the Beta-Liouville prior and the Hessian matrix (Appendix C). As we explained in the previous Section 3.1, we should estimate the model parameters $({\rm Y},\Omega )$ and the variational parameters $(\gamma ,\Phi )$ according to Equations (21), (22) and (A7) to find $m_{nl}$, and we proceed to maximize with respect to $m_{nl}$; so, we have the following equations:

$$\begin{array}{cc}\hfill \mathcal{L}(\gamma ,\Phi ;{\rm Y},\Omega )& =\log (\Gamma (\sum _{d=1}^D{\alpha }_d))+\log (\Gamma (\alpha +\beta ))-\log (\Gamma (\alpha ))\hfill \\ \hfill & -\log (\Gamma (\beta ))-\sum _{d=1}^D\log \Gamma ({\alpha }_d)+\sum _{d=1}^D{\alpha }_d(\Psi ({\gamma }_d)-\Psi (\sum _{l=1}^D{\gamma }_l)\hfill \\ & +\alpha (\Psi ({\alpha }_{\gamma })-\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma }))+\beta (\Psi ({\beta }_{\gamma })\hfill \\ & -\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma }))+\beta (\Psi ({\beta }_{\gamma })-\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma }))\hfill \\ & +\sum _{n=1}^N\sum _{d=1}^D{m}_{nd}(\Psi ({\gamma }_d)-\Psi (\sum _{l=1}^D{\gamma }_l)+\Psi ({\alpha }_{\gamma })-\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma }))\hfill \\ & +\sum _{n=1}^N{m}_{n(D+1)}(\Psi ({\beta }_{\gamma })-\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma }))\hfill \\ \hfill & +\sum _{n=1}^N\sum _{l=1}^{D+1}\sum _{j=1}^V{m}_{nl}{w}_n^j\log ({\Omega }_{lj})\hfill \\ \hfill & -(\log (\Gamma (\sum _{l=1}^D{\alpha }_l))+\log (\Gamma (\alpha +\beta ))-\log \Gamma (\alpha )-\log \Gamma (\beta )\hfill \\ \hfill & -\sum _{i=1}^D\log \Gamma ({\alpha }_i)\hfill \\ \hfill & +\sum _{i=1}^D{\alpha }_i(\Psi ({\gamma }_{mi})-\Psi (\sum _{l=1}^D{\gamma }_{m(l)}))+{\alpha }_{(}\Psi ({\alpha }_{m\gamma })\hfill \\ & -\Psi ({\alpha }_{m\gamma }{\beta }_{m\gamma }))+\beta (\Psi ({\beta }_{m\gamma })-\Psi ({\alpha }_{m\gamma }+{\beta }_{m\gamma })))\hfill \\ \hfill & -(\sum _{n=1}^N\sum _{l=1}^{D+1}{m}_{nl}\log (m_{nl}))\hfill \end{array}$$

(49)

To find $m_{nl}$, we proceed to maximize with respect to ${\varphi }_{nl}$:

$$\begin{array}{cc}\hfill \mathcal{L}\left[m_{nl}\right]& =m_{nl}(\Psi ({\gamma }_i)-\Psi (\sum _{l=1}^D{\gamma }_l))+m_{nl}\log {\beta }_{w(iv)}-m_{nl}\log (m_{nl})\hfill \\ \hfill & +{\lambda }_n(\sum _{l=1}^D{m}_{nl}-1)\hfill \end{array}$$

(50)

Therefore, we have

$$\frac{\partial \mathcal{L}}{\partial {\varphi }_{nl}}=(\Psi ({\gamma }_d)-\Psi (\sum _{l=1}^D{\gamma }_l))+\log {\beta }_{w(iv)}-\log {\varphi }_{nl}-1+{\lambda }_n$$

(51)

The next step is to optimize Equation (49) to find the update equations for the variational; we separate the terms containing the variational Beta-Liouville parameters once more.

$$\begin{array}{cc}\hfill \mathcal{L}\left[{\xi }_q\right]& ={\alpha }_d(\Psi ({\gamma }_d))-\Psi (\sum _{l=1}^D{\gamma }_l)+\alpha (\Psi ({\alpha }_{\gamma })-\Psi ({\alpha }_{\gamma }\hfill \\ \hfill & +{\beta }_{\gamma }))+\beta (\Psi ({\alpha }_{\gamma })-\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma }))\hfill \\ \hfill & +\sum _{n=1}^N{\varphi }_n(\Psi ({\gamma }_l)-\Psi (\sum _{l=1}^D{\gamma }_l)+\Psi ({\alpha }_{\gamma })-\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma }))\hfill \\ & +\sum _{n=1}^N{\varphi }_{n(D+1)}(\Psi ({\beta }_{\gamma })-\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma }))\hfill \\ \hfill & -(\log (\Gamma (\sum _{l=1}^D{\gamma }_l))+\log (\gamma ({\alpha }_{\gamma }+{\beta }_{\gamma })-\log (\Gamma ({\alpha }_{\gamma }))\hfill \\ \hfill & -\log (\Gamma ({\beta }_{\gamma }))-\log (\Gamma ({\gamma }_l))\hfill \\ \hfill & +{\gamma }_l(\Psi ({\gamma }_l)+\Psi ({\alpha }_{\gamma })-\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma }))-\Psi (\sum _{l=1}^D{\gamma }_l)\hfill \\ \hfill & +{\alpha }_{\gamma }(\Psi ({\alpha }_{\gamma })-\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma }))\hfill \\ \hfill & +{\beta }_{\gamma }(\Psi ({\beta }_{\gamma })-\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma })))\hfill \end{array}$$

(52)

Selecting the terms containing variational Beta-Liouville variables ${\gamma }_i,{\alpha }_{\gamma },$ and ${\beta }_{\gamma }$, we have

$$\begin{array}{cc}\hfill \mathcal{L}({\gamma }_i)& ={\alpha }_i(\Psi ({\gamma }_i))-(\sum _{l=1}^D{\alpha }_l)(\Psi (\sum _{l=1}^D{\gamma }_l))+\sum _{n=1}^N{\varphi }_{ni}(\Psi ({\gamma }_i)-\Psi (\sum _{l=1}^D{\gamma }_l))\hfill \\ \hfill & -(\log \Gamma (\sum _{l=1}^D)-\log \Gamma ({\gamma }_i)+{\gamma }_i(\Psi (\sum _{l=1}^D{\gamma }_l)\sum _{d=1}^D{\gamma }_d)\hfill \end{array}$$

(53)

and

$$\begin{array}{cc}\hfill \mathcal{L}\left[{\alpha }_{\gamma }\right]& =\alpha (\Psi ({\alpha }_{\gamma })-\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma }))+\beta (-\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma }))\hfill \\ & +(\Psi ({\alpha }_{\gamma })-\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma }))\sum _{n=1}^N\sum _{i=1}^D{\varphi }_{ni}\sum _{n=1}^N{\varphi }_{n(D+1)}(-\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma }))\hfill \\ \hfill & -(\log ({\alpha }_{\gamma }+{\beta }_{\gamma })-\log (\Gamma ({\alpha }_{\gamma }))+{\alpha }_{\gamma }(\Psi ({\alpha }_{\gamma })-\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma }))\hfill \\ & +{\beta }_{\gamma }(-\Psi ({\alpha }_{\gamma }+{\beta }_{\gamma })))\hfill \end{array}$$

(54)

Setting Equations (52)–(54) to zero, we have the following update parameters:

$${\gamma }_i=\alpha +\sum _{n=1}^N{\varphi }_{ni}$$

(55)

$${\alpha }_{\gamma }=\alpha +\sum _{n=1}^N\sum _{d=1}^D{\varphi }_{nd}$$

(56)

$${\beta }_{\gamma }=\beta +\sum _{n=1}^N{\varphi }_{n(D+1)}$$

(57)

We address the challenge of deriving empirical Bayes estimates for the model parameters ${\rm Y}$ and $\Omega$ by utilizing the variational lower bound as a substitute for the marginal log likelihood. This approach fixes the variational parameters $\gamma$ and $\Phi$ at values determined through variational inference. We then optimize this lower bound to obtain the empirical Bayes estimates of the model parameters.

To estimate ${\Omega }_w$, we formulate necessary update equations. The process of maximizing Equation (52) with respect to $\Omega$ results in the following equation:

$$\mathcal{L}\left[{\Omega }_w\right]=\sum _{d=1}^M\sum _{n=1}^{N_s}\sum _{l=1}^{D+1}\sum _{j=1}^V{\varphi }_{dnl}{w}_{dn}^j\log ({\Omega }_{w(lj)})+\sum _{l=1}^{D+1}{\lambda }_l(\sum _{j=1}^V{\Omega }_{w(lj)}-1)$$

(58)

Taking the derivatives with the respect to ${\beta }_{w(lj)}$ and setting it to zero yields in Appendix C.1:

$${\Omega }_{w(lj)}\propto \sum _{d=1}^M\sum _{n=1}^{N_d}{m}_{dnl}{w}_{dn}^j$$

(59)

Beta-Liouville Parameter

The objective of this subsection is to determine the estimates of the model’s parameters using variational inference techniques [48].

$$\begin{array}{cc}\hfill \mathcal{L}\left[\xi \right]& =\sum _{m=1}^M(\log (\Gamma (\sum _{l=1}^D{\alpha }_l))+\log (\Gamma (\alpha +\beta ))-\log \Gamma (\alpha )-\log \Gamma (\beta )\hfill \\ \hfill & -\sum _{i=1}^D\log \Gamma ({\alpha }_i)+\sum _{i=1}^D{\alpha }_i(\Psi ({\gamma }_{mi})-\Psi (\sum _{l=1}^D{\gamma }_{m(l)}))\hfill \\ \hfill & +\alpha (\Psi ({\alpha }_{m\gamma })-\Psi ({\alpha }_{m\gamma }{\beta }_{m\gamma }))+\beta (\Psi ({\beta }_{m\gamma })-\Psi ({\alpha }_{m\gamma }+{\beta }_{m\gamma })))\hfill \end{array}$$

(60)

The derivative of the above equation with respect to the BL parameter is given by

$$\begin{array}{cc}\hfill & \frac{\partial \mathcal{L}\left[\xi \right]}{\partial {\alpha }_l}=M(\Psi (\sum _{l=1}^D)-\Psi ({\alpha }_l))+\sum _{m=1}^M({\Psi }^{\prime }({\gamma }_{ml})-\Psi (\sum _{l=1}^D{\gamma }_{m(l)}))\hfill \\ \hfill & \frac{\partial \mathcal{L}\left[\xi \right]}{\partial \alpha }=M[\Psi (\alpha +\beta )-\Psi (\alpha )]+\sum _{m=1}^M(\Psi ({\alpha }_{m\gamma })-\Psi ({\alpha }_{m\gamma }+{\beta }_{m\gamma }))\hfill \\ \hfill & \frac{\partial \mathcal{L}\left[\xi \right]}{\partial \beta }=M[\Psi (\alpha +\beta )-\Psi (\beta )]+\sum _{m=1}^M(\Psi ({\beta }_{m\gamma })-\Psi ({\alpha }_{m\gamma }+{\beta }_{m\gamma }))\hfill \end{array}$$

(61)

From the equations presented earlier, it is evident that the derivative in Equation (52) with respect to each of the BL parameters is influenced not only by their individual values but also by their interactions with one another. Consequently, we utilize the Newton–Raphson method to address this optimization problem. To implement the Newton–Raphson method effectively, it is essential to first calculate the Hessian matrix for the parameter space, as illustrated below [49]:

$$\begin{array}{cc}\hfill & \frac{{\partial }^2\mathcal{L}\left[\xi \right]}{\partial {\alpha }_l{\alpha }_j}=M(-\delta (i,j){\Psi }^{\prime }({\alpha }_i)+{\Psi }^{\prime }(\sum _{l=1}^D{\alpha }_l))\hfill \\ \hfill & \frac{{\partial }^2\mathcal{L}\left[\xi \right]}{\partial {\alpha }^2}=M({\Psi }^{\prime }(\alpha +\beta )-{\Psi }^{\prime }(\alpha ))\hfill \\ \hfill & \frac{{\partial }^2\mathcal{L}\left[\xi \right]}{\partial \alpha \partial \beta }=M{\Psi }^{\prime }(\alpha +\beta )\hfill \\ \hfill & \frac{{\partial }^2\mathcal{L}\left[\xi \right]}{\partial {\beta }^2}=M({\Psi }^{\prime }(\alpha +\beta )-{\Psi }^{\prime }(\beta ))\hfill \end{array}$$

(62)

The Hessian matrix shown above is very similar to the Hessian matrix of the Dirichlet parameters in the MPCA model and generalized Dirichlet parameters in GDMPCA. In fact, the above matrix can be divided into two completely separate matrices using parameters ${\alpha }_d$, $\alpha$, and $\beta$. Each of the two parts’ parameter derivation will be identical to the Newton–Raphson model provided by MPCA and GDMPCA.

3.3. Inference via Collapsed Gibbs Sampling

The collapsed Gibbs sampler (CGS) contributes to the inference by estimating posterior distributions through a Bayesian network of conditional probabilities, which are determined through a sampling process of hidden variables. Compared to the traditional Gibbs sampler that functions in the combined space of latent variables and model parameters, the CGS offers significantly faster estimation times. The CGS operates within the collapsed space of latent variables, where, in the joint distribution $p(X,z,\theta ,\varphi ,w|\Omega ,{\rm Y},\mu )$, the model parameters $\theta$ and $\varphi$ are marginalized out. This marginalization leads to the marginal joint distribution $p(X,z,w|\Omega ,{\rm Y},\mu )$, which is defined as follows:

$$p(x,z,w|\Omega ,{\rm Y})=W{\int }_{\theta }{\int }_{\phi }p(X,z,\theta ,\phi ,w|\Omega ,\xi )d\phi d\theta$$

(63)

In using Equation (63), the method calculates the conditional probabilities of the latent variables $z_{ij}$ by considering the current state of all other variables while excluding the specific variable $z_{ij}$ itself [50]. Meanwhile, the collapsed Gibbs sampler (CGS) determines the topic assignments for the observed words by employing the conditional probability of the latent variables, where “$-ij$” indicates counts or variables with $z_{ij}$ excluded [50]. This specific conditional probability is defined as follows [51]:

$$p(z_{ij}=k|z^{-ij},X,w,\Omega ,{\rm Y})=\frac{p(z_{ij},z^{-ij},X,w|\Omega ,{\rm Y})}{p(z^{-ij},X,w|\Omega ,{\rm Y})}$$

(64)

The sampling mechanism of the collapsed Gibbs approach can be summarized as an expectation problem:

$$p(z_{ij}=k|X,w,\Omega ,{\rm Y})={\mathrm{E}}_{p(z^{-ij}|w,X,\Omega ,{\rm Y})}\left[p(z_{ij}=k|z^{-ij},X,w,\Omega ,{\rm Y})\right]$$

(65)

The collapsed Gibbs sampling Beta-Liouville multinomial procedure consists of two phases for assigning documents to clusters. First, each document is assigned a random cluster for initialization. After that, each document is assigned a cluster based on the Beta-Liouville distribution after a specified number of iterations.

The goal is to use a network of conditional probabilities for individual classes to sample the latent variables from the joint distribution $p(X,z|w,\Omega ,{\rm Y})$. The assumption of conjugacy allows the integral in Equation (63) to be estimated.

$$\begin{array}{c}\hfill p(X,z|w,\upsilon )=C\prod _{j=1}^M\left[\frac{\Gamma ({\sum }_{i=1}^k{\alpha }_i)\Gamma (\alpha +\beta )}{{\prod }_{i=1}^k\Gamma ({\alpha }_i)\Gamma (\alpha )\Gamma (\beta )}\right]\times \frac{{\prod }_{i=1}^k\Gamma ({\alpha }_i^{\prime })\Gamma ({\alpha }^{\prime })\Gamma ({\beta }^{\prime })}{\Gamma ({\alpha }^{\prime }+{\beta }^{\prime })\Gamma ({\sum }_{i=1}^K{\alpha }_i^{\prime })}\end{array}$$

(66)

The likelihood of the multinomial distribution, defined by the parameter ${\rm Y}$, and the probability density function of the Beta-Liouville distribution can be expressed as follows:

$$\begin{array}{cc}\hfill p(X|{\rm Y})& =\int p(X|\theta )p(\theta |{\alpha }_1,\dots ,\beta ,\alpha )d\theta \hfill \\ \hfill & =\int \prod _{k=1}^k{\theta }_k^{m_k}\frac{\Gamma ({\sum }_{k=1}^k{\alpha }_k)\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\beta )}\prod _{k=1}^K\frac{{\theta }_k^{{\alpha }_k-1}}{\Gamma ({\alpha }_k)}\hfill \\ \hfill & \times {(\sum _{k=1}^K{\theta }_k)}^{\alpha -\sum {\alpha }_k}{(1-\sum _{k=1}^K{\theta }_k)}^{\beta -1}d\theta \hfill \end{array}$$

(67)

By integrating the probability density function of the Beta-Liouville distribution over the parameter $\theta$ and incorporating updated parameters derived from the remaining integral in Equation (69), we are able to express it as a fraction of Gamma functions. The following shows the updated parameters, where $N_{jk}$ represents counts corresponding to variables [45,51]:

$$\begin{array}{cc}\hfill & {\alpha }_K^{\prime }={\alpha }_k+\sum _{j=1}^k{N}_{jk}\hfill \\ \hfill & {\alpha }^{\prime }=\alpha +N_{jk}\hfill \\ \hfill & {\beta }^{\prime }=\beta +N_{jk}\hfill \end{array}$$

(68)

Equation (67) is then equivalent to

$$\begin{array}{cc}\hfill p(k|{\alpha }_1,\dots ,{\alpha }_k,\beta ,\alpha )& =\frac{\Gamma ({\sum }_{k=1}^K{\alpha }_k)\Gamma (\alpha +\beta )\Gamma (\alpha +{\sum }_{k=1}^{k-1}{m}_k)\Gamma (\beta +m_k)}{\Gamma (\alpha )\Gamma (\beta ){\prod }_{k=1}^K\Gamma ({\alpha }_k)\Gamma ({\sum }_{k=1}^K({\alpha }_k+m_k))}\hfill \\ \hfill & \frac{{\prod }_{k=1}^K\Gamma ({\alpha }_k+m_k)}{\Gamma (\alpha +{\sum }_{k=1}^{K-1}{m}_k+\beta +m_k)}\hfill \end{array}$$

(69)

The parameters ${\alpha }_1,\dots ,{\alpha }_k,\alpha ,$ and $\beta$ correspond to the Beta-Liouville distribution, while $m_k$ represents the number of documents in cluster k.

After the sampling process, parameter estimation is performed. Subsequently, the empirical likelihood method [47] is utilized to validate the results using a held-out dataset. Ultimately, this process leads to the estimation of the class conditional probability $p(X|w,\Omega ,{\rm Y})$ within the framework of collapsed Gibbs sampling:

$$p(X|w,\Omega ,{\rm Y})=\prod _{ij}\sum _{k=1}^K\frac{1}{S}\sum _{s=1}^S{\tilde{\theta }}_{jks}{\tilde{\phi }}_{kws}$$

(70)

The parameters are then computed as follows:

$${\tilde{\theta }}_{jks}=\frac{(N_{jk}+{\alpha }_k)({\alpha }_{jk}+{\sum }_{l=k+1}^{K+1}{N}_{jl})(N_{jk}+{\beta }_k)}{(a_k{b}_k+{\sum }_{l=k+1}^{K+1}{N}_{jl})({\alpha }_j+{\sum }_{l=k+1}^{K+1}{N}_{jl})}$$

(71)

$${\tilde{\phi }}_{kws}=\frac{(N_{jk}+{\alpha }_w)({\alpha }_{jw}+{\sum }_{l=k+1}^{K+1}{N}_{jl})(N_{jk}+{\beta }_w)}{({\alpha }_w{b}_w+{\sum }_{l=k+1}^{K+1}{N}_{jl})({\alpha }_{wj}+{\sum }_{l=k+1}^{K+1}{N}_{jl})}$$

(72)

where S is the size of a sample.

4. Experimental Results

In this section, we validate our proposed algorithms’ efficiency for two distinct and challenging applications, namely, topic modeling for medical text and sentiment analysis. Each model’s evaluation is based on the success rate for each dataset and the perplexity [3,9,52,53], which is a common measure used in language modeling and is defined as:

$$prep({\mathcal{D}}_{test})=\exp \left(\frac{-\ln p({\mathcal{D}}_{test})}{{\sum }_d\left|w_d\right|}\right)$$

(73)

where $|w_d|$ is the length of document d. A lower perplexity score indicates better generalization performance. In addition to the perplexity metric, the success rate is employed as a key performance indicator to evaluate our models, reflecting the proportion of correctly identified topics within a corpus in topic modeling. The success rate serves as a straightforward measure of a model’s efficacy, capturing its ability to accurately classify documents into the correct topical categories, which is essential for effective information retrieval and knowledge discovery in the domain of text analysis. The main goal of both applications is to compare the GDMPCA, BLMPCA, and MPCA performances. The choice of these datasets is pivotal to our research as they offer a broad spectrum of analytical scenarios, from topic modeling for medical text to sentiment analysis, thus enabling a thorough investigation into the models’ adaptability and accuracy. By encompassing datasets with distinct characteristics, we are able to demonstrate the strengths of our proposed models in varied contexts, highlighting their potential as a versatile tool in the field of text analysis.

4.1. Topic Modeling

The goal of text classification is to assign documents to predefined subject categories, a problem extensively researched with various approaches [42,54,55]. Topic modeling, a common application in natural language processing, is used for analyzing texts from diverse sources and for document clustering [56]. It identifies key “topics” in a text corpus using unsupervised statistical methods, where topics are keyword mixtures with a probability distribution, and documents are composed of topic mixtures [12]. The “CMU Book Summary Dataset” was used to validate the model performance, containing plot summaries and metadata for 16,559 books [57]. The models’ accuracy was tested by training on various document numbers and observing the impact of latent topics on the classification accuracy. Using variational Bayes inference, the models showed similar performances, but BLMPCA excelled, particularly in classifying similar classes.

In

Table 2. Common topics identified with the BLMPCA model on the CMU Book dataset, each defined by a set of keywords.

Topic Number	Topics
Topic 1	girl, tells, find, two, man, when, return, after, also, finds, time, kill, later, help, killed
Topic 2	he, one, back, man, time, house, father, police, story, mother, young, school, love, time, first
Topic 3	tells, they, return, find, girl, back, one, house, story, after, dragon, find, schools, boy, jack
Topic 4	earth, world, one, human, ship, book, planet, space, human, systems, time, years, in, people, would
Topic 5	war, novel, new, world, army, story, one, group, book, states, general, british, president, first, american

,

Table 3. A comparison of the perplexity of the MPCA, GDMPCA, and BLMPCA models, indicating the model fit quality across different topic numbers (K) on the CMU Book dataset.

K	5	10	15	20
MPCA	1455	1422	1320	1215
GDMPCA	1326	1430	1190	1178
BLMPCA	1319	1203	1198	1177

and

Table 4. Time complexity comparison for MPCA, GDMPCA, and BLMPCA at varying topic levels (K) on the CMU Book dataset.

K	5	10	15	20
MPCA	107.803	140.1439	150.9242	161.7045
GDMPCA	225.04	230.544	347.056	408.064
BLMPCA	251.64	327.132	352.296	377.46

, we present the first three topics, the perplexity measurements, and time complexity for all models compared in this study. The success rates obtained using GDMPCA, BLMPCA, and MPCA are depicted in Figure 1. These examples demonstrate that our proposed models, which incorporate Generalized Dirichlet and Beta-Liouville distributions, yield more accurate classifications in scenarios where distinct classes exhibit similarities, in contrast to the traditional MPCA which is a Dirichlet-based model. Additionally, in

Table 5. Comparison perplexity scores of MPCA, GDMPCA, and BLMPCA, reflecting the model fit as the topic count (K) increases on the CMU Book dataset with CGS inference.

K	5	10	15	20
MPCA	1391.5	1448.6	1516	1580
GDMPCA	1291.2	1316	1428	1413
BLMPCA	1310.4	1324.8	1416	1483.2

and

Table 6. Time complexity comparison for MPCA, GDMPCA, and BLMPCA with increasing topics (K) using CGS inference on the CMU Book dataset.

K	5	10	15	20
MPCA	431.212	536.57	634.69	687.818
GDMPCA	19125.2	1138.264	2429.392	2964.51
BLMPCA	1998.84	2289.924	3018.368	3497.14

, we show the results for the collapsed Gibbs sampling.

4.2. Topic Modeling for Medical Text

Topic modeling plays a crucial role in navigating the complexities of health and medical text mining, despite the inherent challenges of data volume and redundancy in this domain. The study by Onan et al. [58] marked a significant advancement, presenting an optimized topic modeling approach that utilizes ensemble pruning. This method significantly improves the categorization of biomedical texts by enhancing precision and managing the computational challenges posed by the extensive data typical of medical documents. With vast amounts of health-related data, specialists struggle to find pertinent information, exemplified by the millions of papers on PubMed and hospital discharge records in the United States in 2015. This study utilized the TMVAr corpus from PubMed and the TMVAr-Dataset containing health-related Twitter news to evaluate models [59,60,61,62,63,64].

TMVAr Dataset

The TMVar Corpus dataset, comprising 500 PubMed papers with manual annotations of various mutation mentions, was utilized to evaluate our models.

Table 7. Comparison of the perplexity of the MPCA, GDMPCA, and BLMPCA models, indicating the model fit quality across different topic numbers (K) on the TMVAR dataset with variation EM inference.

K	5	10	15	20
MPCA	2115	2083	1984	1977
GDMPCA	1996	1989	1968	1959
BLMPCA	1983	1965	1954	1949

and

Table 8. Time complexity comparison for MPCA, GDMPCA, and BLMPCA with increasing topics (K) using variation EM inference on the TMVAR dataset.

K	5	10	15	20
MPCA	9.53	22.543	26.092	28.458
GDMPCA	11.83	24.843	28.392	30.758
BLMPCA	18.57	38.997	44.568	48.282

elucidate the perplexity comparison and time complexity for the TMVAR dataset, offering insight into the performances of our proposed methods. Moreover,

Table 9. Comparison of the perplexity of the MPCA, GDMPCA, and BLMPCA models, indicating the model fit quality across different topic numbers (K) on the TMVAR dataset with CGS inference.

K	5	10	15	20
MPCA	2132.5	2232.8	2376.0	2460
GDMPCA	1360.9	1182.4	1345.6	1938
BLMPCA	1938.5	11350.5	1340.5	1440

and

Table 10. Time complexity comparison for MPCA, GDMPCA, and BLMPCA with increasing topics (K) using CGS inference on the TMVAR dataset.

K	5	10	15	20
MPCA	45.74	62.89	108.20	200.63
GDMPCA	56.74	163.95	252.35	273.70
BLMPCA	165.57	336.93	376.45	392.71

present the outcomes of the collapsed Gibbs sampling. As indicated in the tables, the time complexity of this method is higher, yet the perplexity is lower.

Furthermore, as shown in

Table 11. Common topics identified with the BLMPCA model in the TMVAR dataset, each defined by a set of keywords.

Topic Number	Topics
Topic 1	mutations, mutation, gene, family, patients, iron, exon, novel, autosomal, associated
Topic 2	gene, p, cancer, polymorphism, expression, patients, associated, deletion, study, region
Topic 3	gene, patients, dna mutation, polymorphism, detected, samples, family, study, results, dna
Topic 4	dna mutation, mutations, homozygous, variants, family, ct, position, methods, associated, substitution
Topic 5	gene, patients, protein mutation, dna, exon, study, genetic, cancer, substitution, genotype

, the BLMPCA model successfully extracts pertinent topics, which is indicative of the model’s nuanced analytical capabilities. Figure 2 further illustrates the success rate of our proposed models in comparison to the traditional MPCA, highlighting the enhanced classification accuracy achieved by our methods.

4.3. Sentiment Analysis

Sentiment analysis, crucial for interpreting emotions in texts from various sources, benefits from advanced methodologies beyond mere word analysis [65]. Recent studies, such as [66,67], have demonstrated the effectiveness of deep learning and text mining in capturing nuanced sentiment expressions. Additionally, the authors of [68] highlighted the potential of ensemble classifiers in improving the sentiment classification accuracy. These innovations showcase the shift toward more complex analyses that consider semantics, context, and intensity for a more accurate sentiment understanding.

The “Multi-Domain Sentiment Dataset”, containing Amazon.com product reviews across various domains, was used for analysis [69]. This dataset, with extensive reviews on books and DVDs, provides data for basic analysis. The applied model, using K = 8 topics, assumed that each topic comprises a bag of words with specific probabilities, and each document is a mix of these topics. The model’s goal was to learn the distributions of words and topics in the corpus.

We demonstrated that the overall sentiment of the dataset tends to be positive, influenced by the presence of high-frequency words with positive connotations within the corpus. This observation is substantiated by the sentiment analysis framework we employed.

Table 12. Comparison of the perplexity of the MPCA, GDMPCA, and BLMPCA models, indicating the model fit quality across different topic numbers (K) on sentiment data with variation EM inference.

K	2	3	5	8
MPCA	1551	1531	1542	1529
GDMPCA	1549	1539	1524	1521
BLMPCA	1448	1540	1531	1518

and

Table 13. Time complexity comparison for MPCA, GDMPCA, and BLMPCA with increasing topics (K) using variational EM inference on the sentiment analysis application.

K	5	10	15	20
MPCA	130.54	169.702	182.756	195.81
GDMPCA	142.876	185.7388	200.0264	214.314
BLMPCA	158.23	205.699	221.522	237.345

provides a detailed explanation of the perplexity measures and time complexity tested for sentiment analysis. Furthermore, the findings from the topic modeling of eight emotions and two sentiments are displayed in

Table 14. Frequency of emotions identified in text data via topic modeling.

Emotions	Count
satisfied	78,901
angry	21,345
happy	6521
joy	82,345
disgust	7125
Perfect	45,459
Tearful	3451
sad	4387

and

Table 15. The counts of positive, negative, and unlabeled sentiments identified through sentiment analysis.

Sentiment	Count
Positive	213,232
Negative	36,308
Unlabeled	23,451

. Figure 3 shows that our proposed models outperform the previous model. Figure 3 shows the success rates for MPCA, GDMPCA, and BLMPCA on sentiment analysis, with GDMPCA and BLMPCA outperforming MPCA as the number of emotions analyzed increases. This indicates their better suitability for complex emotion detection tasks in practical applications.

Furthermore,

Table 16. Comparison of the perplexity of the MPCA, GDMPCA, and BLMPCA models, indicating the model fit quality across different topic numbers (K) on sentiment data with CGS inference.

K	2	3	5	8
MPCA	1451	1511	1589	1639
GDMPCA	1332	1393	1422	1502
BLMPCA	1316	1401	1413	1498

and

Table 17. Time complexity comparison for MPCA, GDMPCA, and BLMPCA with increasing topics (K) using CGS inference on the sentiment analysis application.

K	5	10	15	20
MPCA	830.54	1069.702	1282.756	1495.81
GDMPCA	924.451	1258.78	1319.46	1383.17
BLMPCA	1085.42	1264.24	1390.12	1473.623

present the results for the collapsed Gibbs sampling. Additionally,

Table 18. Accuracy comparisons for sentiment analysis classifiers.

Classifier	SVM	NaiveBayes	MLP
MPCA	0.62	0.68	0.67
GDMPCA	0.80	0.85	0.87
BLMPCA	0.83	0.88	0.88

and

Table 19. Recall metrics for SVM, Naive Bayes, and MLP classifiers using MPCA, GDMPCA, and BLMPCA in sentiment analysis.

Classifier	SVM	NaiveBayes	MLP
MPCA	0.61	0.59	0.66
GDMPCA	0.79	0.76	0.85
BLMPCA	0.85	0.82	0.89

display the accuracy and recall of various classifiers utilized for emotion detection.

Table 20. F1-score metrics for SVM, Naive Bayes, and MLP classifiers using MPCA, GDMPCA, and BLMPCA in sentiment analysis.

Classifier	SVM	Naive Bayes	MLP
MPCA	0.6195	0.6041	0.6697
GDMPCA	0.7999	0.7701	0.8593
BLMPCA	0.8593	0.8313	0.8999

shows the F1-scores for various classifiers, indicating the balanced harmonic mean of the precision and recall for SVM, Naive Bayes, and MLP classifiers when applied with MPCA, GDMPCA, and BLMPCA models in sentiment analysis.

5. Discussion

We delved into the comparative advantages of the GDMPCA and BLMPCA models over existing methods in text classification and sentiment analysis. The superior performances of our proposed models can be attributed to several key factors. Firstly, the incorporation of Generalized Dirichlet and Beta-Liouville distributions allows for a more nuanced modeling of text data, which captures the intricacies of word distributions more effectively than traditional methods. This results in a more accurate representation of the underlying thematic structures in the data. For instance, in the CMU Book Summary Dataset, the intricacies of literary themes were better represented, showcasing the models’ aptitude for multifaceted textual analysis. This was attributed to the models’ ability to account for the co-occurrence and complex interrelationships of terms within documents, a feature less emphasized in MPCA due to its assumption of component independence.

In the TMVAR Corpus from PubMed, the medical text presented a challenge due to its specialized lexicon and the density of information. The BLMPCA model excelled by exploiting its additional parameters, optimizing data representation in this high-dimensional space, thus underscoring the importance of model selection aligned with dataset characteristics. The sentiment analysis on the Multi-Domain Sentiment Dataset further emphasized the adaptability of our models. Here, BLMPCA demonstrated its finesse in discerning subtle sentiments from Amazon.com reviews, outperforming traditional approaches that may not have captured the emotional granularity present in user-generated content.

However, the sophistication of GDMPCA and BLMPCA comes with greater computational demands, as reflected in longer convergence times. This trade-off between accuracy and computational efficiency underscores the necessity of careful model selection in practice, considering the scale of the data and available computational resources. Although our proposed models signify a leap forward in text analysis methodologies, they are not without limitations. The reliance on variational inference and assumptions specific to the models may not be universally applicable to all types of textual data, suggesting room for future refinement and the exploration of alternative distributions or learning strategies.

The findings of this study illuminate the potential of integrating advanced probabilistic distributions into PCA to uncover deeper insights within text data. It is a testament to the evolution of statistical models in text analysis, pointing toward an exciting trajectory for future research in the field. The ongoing dialogue within the academic community on these topics is reflective of the dynamic nature of machine learning and its applications to natural language processing. As we continue to push the boundaries, it is imperative to balance innovation with practicality, ensuring that our models are not only theoretically robust but also computationally viable and accessible for varied applications.

6. Conclusions

In this paper, two novel models, generalized Dirichlet Multinomial PCA and Beta-Liouville Multinomial PCA, were proposed to improve the accuracy of the MPCA model for multi-topic modeling and text classification. We followed a Bayesian analysis that considers the generalized Dirichlet and Beta-Liouville assumptions. We demonstrated that our two proposed models have more flexibility. The models were used in two separate applications: text classification and sentiment analysis. The results show that the two proposed models, in all applications, achieved superior performances, as represented by the high prediction accuracy in comparison to that of the MPCA. It could be claimed that the proposed models, using different prior assumptions, yield better results than the standard methods. Specially, the BLMPCA provides the best improvement compared to the GDMPCA and MPCA for all the tested data. Crucially, the employment of collapsed Gibbs sampling for parameter inference was proven efficient and effective, despite its time-consuming nature. This method substantially boosts our models’ computational capabilities, allowing for the superior discovery of latent topics in text corpora and marking a noteworthy advancement over the MPCA model. Future approaches for research will concentrate on model modifications and improvements to achieve greater precision in topic modeling. In addition, future works could be devoted to extending the proposed models to other applications and significantly extending the proposed model to fit a variety of data as well as real-time streaming data.