Interpretable Topic Extraction and Word Embedding Learning Using Non-Negative Tensor DEDICOM

Abstract

Unsupervised topic extraction is a vital step in automatically extracting concise contentual information from large text corpora. Existing topic extraction methods lack the capability of linking relations between these topics which would further help text understanding. Therefore we propose utilizing the Decomposition into Directional Components (DEDICOM) algorithm which provides a uniquely interpretable matrix factorization for symmetric and asymmetric square matrices and tensors. We constrain DEDICOM to row-stochasticity and non-negativity in order to factorize pointwise mutual information matrices and tensors of text corpora. We identify latent topic clusters and their relations within the vocabulary and simultaneously learn interpretable word embeddings. Further, we introduce multiple methods based on alternating gradient descent to efficiently train constrained DEDICOM algorithms. We evaluate the qualitative topic modeling and word embedding performance of our proposed methods on several datasets, including a novel New York Times news dataset, and demonstrate how the DEDICOM algorithm provides deeper text analysis than competing matrix factorization approaches.

1. Introduction

Matrix factorization methods have always been a staple in many natural language processing (NLP) tasks. Factorizing a matrix of word co-occurrences can create both low-dimensional representations of the vocabulary, so-called word embeddings [1,2], that carry semantic and topical meaning within them, as well as representations of meaning that go beyond single words to latent topics.

Decomposition into Directional Components (DEDICOM) is a matrix factorization technique that factorizes a square, possibly asymmetric, matrix of relationships between items into a loading matrix of low-dimensional representations of each item and an affinity matrix describing the relationships between the dimensions of the latent representation (see Figure 1 for an illustration).

We introduce a modified row-stochastic variation of DEDICOM, which allows for interpretable loading vectors and apply it to different matrices of word co-occurrence statistics created from Wikipedia based semi-artificial text documents. Our algorithm produces low-dimensional word embeddings, where one can interpret each latent factor as a topic that clusters words into meaningful categories. Hence, we show that row-stochastic DEDICOM successfully combines the task of learning interpretable word embeddings and extracting representative topics.

We further derive a similar model for factorization of three-dimensional data tensors, which represent word co-occurrence statistics for text corpora with intrinsic structure that allows for some separation of the corpus into subsets (e.g., a news corpus structered by time).

An interesting aspect of this type of factorization is the interpretability of the affinity matrix. An entry in the matrix directly describes the relationship between the topics of the respective row and column and one can therefore use this tool to extract topics that a certain text corpus deals with and analyze how these topics are connected in the given text.

In this work we first describe the aforementioned DEDICOM algorithm and provide details on the modified row-stochasticity constraint and on optimization. We further expand our model to factorize three-dimensional tensors and introduce a multiplicative update rule that facilitates the training procedure. We then present results of various experiments on both semi-artificial text documents (combinations of Wikipedia articles) and real text documents (movie reviews and news articles) that show how our approach is able to capture hidden latent topics within text corpora, cluster words in a meaningful way and find relationships between these topics within the documents.

This paper is an extension of previous work [3]. In addition to the algorithms and experiments described there, we here add the extension of the DEDICOM algorithm to three-dimensional tensors, introduce a multiplicative update rule to increase training stability and present new experiments on two additional text corpora (Amazon reviews and New York Times news articles).

2. Related Work

Matrix factorization describes the task of compressing the most relevant information from a high-dimensional input matrix into multiple low-dimensional factor matrices, with either approximate or exact input reconstruction (see for example [4] for a theoretical overview of common methods and their applications). In this work we consider the DEDICOM algorithm, which has a long history of providing an interpretable matrix or tensor factorization, mostly for rather low-dimensional tasks.

First described in [5], it since has been applied to analysis of social networks [6], email correspondence [7] and video game player behavior [8,9]. DEDICOM also has successfully been employed to NLP tasks such as part of speech tagging [10], however to the best of our knowledge we provide the first implementation of DEDICOM for simultaneous word embedding learning and topic modeling.

Many works deal with the task of putting constraints on the factor matrices of the DEDICOM algorithm. In [7,8], the authors constrain the affinity matrix R to be non-negative, which aids interpretability and improves convergence behavior if the matrix to be factorized is non-negative. However, their approach relies on the Kronecker product between matrices in the update step, solving a linear system of $n^2\times k^2$, where n denotes the number of items in the input matrix and k the number of latent factors. These dimensions make the application on text data, where n describes the number of words in the vocabulary, a computationally futile task. Constraints on the loading matrix, A, include non-negativity as well (see [7]) or column-orthogonality as in [8].

In contrast, we propose a new modified row-stochasticity constraint on A, which is tailored to generate interpretable word embeddings that carry semantic meaning and represent a probability distribution over latent topics.

The DEDICOM algorithm has previously been applied to tensor data as well, for example in [11], in which the authors apply the algorithm on general multirelational data by computing an exact solution for the affinity matrix. Both [6,7] explore a slight variation of our tensor DEDICOM approach to analyze relations in email data and [12] apply a similar model on non-square input tensors.

Previous matrix factorization based methods in the NLP context mostly dealt with either word embedding learning or topic modeling, but not with both tasks combined.

For word embeddings, the GloVe [2] model factorizes an adjusted co-occurrence matrix into two matrices of the same dimension. The work is based on a large text corpus with a vocabulary of $n\approx$ 400,000 and produces word embeddings of dimension $k=300$. In order to maximize performance on the word analogy task, the authors adjusted the co-occurrence matrix to the logarithmized co-occurrence matrix and added bias terms to the optimization objective.

A model conceived around the same time, word2vec [13], calculates word embeddings not from a co-occurrence matrix but directly from the text corpus using the skip-gram or continuous-bag-of-words approach. More recent work [1] has shown that this construction is equivalent to matrix factorization on the pointwise mutual information (PMI) matrix of the text corpus, which makes it very similar to the glove model described above.

Both models achieve impressive results on word embedding related tasks like word analogy, however the large dimensionality of the word embeddings makes interpreting the latent factors of the embeddings impossible.

On the topic modeling side, matrix factorization methods are routinely applied as well. Popular algorithms like non-negative matrix factorization (NMF) [14], singular value decomposition (SVD) [15,16] and principal component analysis (PCA) [17] compete against the probabilistic latent dirichlet allocation (LDA) [18] to cluster the vocabulary of a word co-occurrence or document-term matrix into latent topics. (More recent expansions of these methods can be found in [19,20].) Yet, we empirically show that the implicitly learned word embeddings of these methods lack semantic meaning in terms of the cosine similarity measure.

We benchmark our approach qualitatively against these methods in Section 4.3 and the Appendix A and Appendix B.

3. Constrained DEDICOM Models

In this section we provide a detailed theoretical view at different constrained DEDICOM algorithms utilized for factorizing word co-occurrence based positive pointwise mutual information matrices and tensors.

We first consider the case of a two-dimensional input matrix S (see Figure 1a) in Section 3.1. We then present an extension of the algorithm for three-dimension input tensors S (see Figure 1b) in Section 3.2. Finally we derive a multiplicative update rule for non-negative tensor DEDICOM.

3.1. The Row-Stochastic DEDICOM Model for Matrices

For a given language corpus consisting of n unique words $X=x_1,\dots ,x_n$ we calculate a co-occurrence matrix $\mathit{W}\in {ℝ}^{n\times n}$ by iterating over the corpus on a word token level with a sliding context window of specified size. Then

$${\mathit{W}}_{ij}=\#\mathrm{word}i\mathrm{appears}\mathrm{in}\mathrm{context}\mathrm{of}\mathrm{word}j.$$

(1)

Note that the word context window can be applied symmetrically or asymmetrically around each word. We choose a symmetric context window, which implies a symmetric co-occurrence matrix, ${\mathit{W}}_{ij}={\mathit{W}}_{ji}$.

We then transform the co-occurrence matrix into the pointwise mutual information matrix (PMI), which normalizes the counts in order to extract meaningful co-occurrences from the matrix. Co-occurrences of words that occur regularly in the corpus are decreased since their appearance together might be nothing more than a statistical phenomenon, the co-occurrence of words that appear less often in the corpus give us meaningful information about the relations between words and topics. We define the PMI matrix as

$${\mathbf{PMI}}_{ij}:=log{\mathit{W}}_{ij}+logN-log{N}_i-log{N}_j$$

(2)

where $N:={\sum }_{ij=1}^n{\mathit{W}}_{ij}$ is the sum of all co-occurrence counts of W, $N_i:={\sum }_{j=1}^n{\mathit{W}}_{ij}$ the row sum and $N_j:={\sum }_{i=1}^n{\mathit{W}}_{ij}$ the column sum.

Since the co-occurrence matrix W is symmetrical, the transformed PMI matrix is symmetrical as well. Nevertheless, DEDICOM is able to factorize both symmetrical and non-symmetrical matrices. We expand details on symmetrical and non-symmetrical relationships in Section 3.4.

Additionally, we want all entries of the matrix to be non-negative, our final matrix to be factorized is therefore the positive PMI (PPMI)

$${\mathit{S}}_{ij}={\mathbf{PPMI}}_{ij}=max\{0,{\mathbf{PMI}}_{ij}\}.$$

(3)

Our aim is to decompose this matrix using row-stochastic DEDICOM as

$$\mathit{S}\approx \mathit{A}\mathit{R}{\mathit{A}}^T,\mathrm{with}{\mathit{S}}_{ij}\approx \sum _{b=1}^k\sum _{c=1}^k{\mathit{A}}_{ib}{\mathit{R}}_{bc}{\mathit{A}}_{jc},$$

(4)

where $\mathit{A}\in {ℝ}^{n\times k}$, $\mathit{R}\in {ℝ}^{k\times k}$, ${\mathit{A}}^T$ denotes the transpose of A and $k\ll n$. Literature often refers to A as the loading matrix and R as the affinity matrix. A gives us for each word i in the vocabulary a vector of size k, the number of latent topics we wish to extract. The square matrix R then provides possibility for interpretation of the relationships between these topics.

Empirical evidence has shown that the algorithm tends to favor columns unevenly, such that a single column receives a lot more weight in its entries than the other columns. We try to balance this behavior by applying a column-wise z-normalization on A, such that all columns have zero mean and unit variance.

In order to aid interpretability we wish each word embedding to be a distribution over all latent topics, i.e., entry ${\mathit{A}}_{ib}$ in the word-embedding matrix provides information on how much topic b describes word i.

To implement these constraints we therefore apply a row-wise softmax operation over the column-wise z-normalized A matrix by defining ${\mathit{A}}^{\prime }\in {ℝ}^{n\times k}$ as

$$\begin{array}{c}\hfill {\mathit{A}}_{ib}^{\prime }:=\frac{exp\left({\overline{\mathit{A}}}_{ib}\right)}{{\sum }_{b^{\prime }=1}^{k}exp\left({\overline{\mathit{A}}}_{i{b}^{\prime }}\right)},{\overline{\mathit{A}}}_{ib}:=\frac{{\mathit{A}}_{ib}-{\mu }_b}{{\sigma }_b},\\ \hfill {\mu }_b:=\frac{1}{n}\sum _{i=1}^n{\mathit{A}}_{ib},{\sigma }_b:=\sqrt{\frac{1}{n}{\sum }_{i=1}^n{\left({\mathit{A}}_{ib}-{\mu }_b\right)}^2}\end{array}$$

(5)

and optimizing A for the objective

$$\mathit{S}\approx {\mathit{A}}^{\prime }\mathit{R}{\left({\mathit{A}}^{\prime }\right)}^T.$$

(6)

Note that after applying the row-wise softmax operation all entries of ${\mathit{A}}^{\prime }$ are non-negative.

To judge the quality of the approximation (6) we apply the Frobenius norm, which measures the difference between S and ${\mathit{A}}^{\prime }\mathit{R}{\left({\mathit{A}}^{\prime }\right)}^T$. The final loss function we optimize our model for is therefore given by

$$\begin{array}{cc}\hfill \mathcal{L}(\mathit{S},\mathit{A},\mathit{R})& ={∥\mathit{S}-{\mathit{A}}^{\prime }\mathit{R}{\left({\mathit{A}}^{\prime }\right)}^T∥}_F^2\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill & =\sum _{i=1}^n\sum _{j=1}^n{\left({\mathit{S}}_{ij}-{\left({\mathit{A}}^{\prime }\mathit{R}{\left({\mathit{A}}^{\prime }\right)}^T\right)}_{ij}\right)}^2\hfill \end{array}$$

(8)

with

$${\left({\mathit{A}}^{\prime }\mathit{R}{\left({\mathit{A}}^{\prime }\right)}^T\right)}_{ij}=\sum _{b=1}^k\sum _{c=1}^k{\mathit{A}}_{ib}^{\prime }{\mathit{R}}_{bc}{\mathit{A}}_{jc}^{\prime }$$

(9)

and ${\mathit{A}}^{\prime }$ defined in (5).

To optimize the loss function we train both matrices using alternating gradient descent similar to [8]. Within each optimization step we apply

$$\begin{array}{cc}\hfill \mathit{A}& ↤\mathit{A}-f_{\theta }({\nabla }_{\mathit{A}},{\eta }^{\mathit{A}}),\mathrm{where}{\nabla }_{\mathit{A}}=\frac{\partial \mathcal{L}(\mathit{S},\mathit{A},\mathit{R})}{\partial \mathit{A}}\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill \mathit{R}& ↤\mathit{R}-f_{\theta }({\nabla }_{\mathit{R}},{\eta }^{\mathit{R}}),\mathrm{where}{\nabla }_{\mathit{R}}=\frac{\partial \mathcal{L}(\mathit{S},\mathit{A},\mathit{R})}{\partial \mathit{R}}\hfill \end{array}$$

(11)

with ${\eta }^{\mathit{A}},{\eta }^{\mathit{R}}>0$ being individual learning rates for both matrices and $f_{\theta }(·)$ representing an arbitrary gradient based update rule with additional hyperparameters $\theta$. For our experiments we employ automatic differentiation methods. For details on the implementation of the algorithm above refer to Section 4.2.

3.2. The Constrained DEDICOM Model for Tensors

In this section we extend the model described above to three-dimensional tensors as input data. As above, the input describes the co-occurrences of vocabulary items in a text corpus. However, we consider additionally structured text: Instead of one matrix describing the entire corpus we unite multiple $n\times n$ matrices of co-occurrences into one tensor $\underset{¯}{\mathit{S}}\in {ℝ}^{t\times n\times n}$. Each of the t slices then consists of an adjusted PPMI matrix for a subset of the text corpus. This structure could originate for instance from different data (e.g., different Wikipedia articles), different topical subsets of the data source (e.g., reviews for different articles) or describe time-slices (e.g., news articles for certain time periods).

To construct the PPMI tensor we again take a vocabulary $X=x_1,\dots ,x_n$ over the entire corpus. For each subset l we then calculate a co-occurrence matrix ${\underset{¯}{\mathit{W}}}_l\in {ℝ}^{n\times n}$ as described above. Stacking these matrices yields the co-occurrence tensor $\underset{¯}{\mathit{W}}\in {ℝ}^{t\times n\times n}$.

When transforming slice ${\underset{¯}{\mathit{W}}}_l$ into a PMI matrix we want to apply information from the entire corpus. We therefore do not only calculate the column, the row and the total sums on the corresponding subset but on the entire text corpus. Therefore

$${\underset{¯}{\mathbf{PMI}}}_{\mathit{lij}}:=log{\underset{¯}{\mathit{W}}}_{\mathit{lij}}+logN-log{N}_i-log{N}_j,$$

(12)

where $N:={\sum }_{l=1}^k{\sum }_{ij=1}^n{\underset{¯}{\mathit{W}}}_{\mathit{lij}}$ is the sum of all co-occurrence counts of $\underset{¯}{\mathit{W}}$, $N_i:={\sum }_{l=1}^k{\sum }_{j=1}^n{\underset{¯}{\mathit{W}}}_{\mathit{lij}}$ the row sum and $N_j:={\sum }_{l=1}^k{\sum }_{i=1}^n{\underset{¯}{\mathit{W}}}_{lij}$ the column sum.

Finally we define the positive pointwise mutual information tensor as

$${\underset{¯}{\mathit{S}}}_{\mathit{lij}}={\underset{¯}{\mathbf{PPMI}}}_{\mathit{lij}}=max\{0,{\underset{¯}{\mathbf{PMI}}}_{\mathit{lij}}\}.$$

(13)

We decompose this input tensor into a matrix $\mathit{A}\in {ℝ}^{n\times k}$ and a tensor $\underset{¯}{\mathit{R}}\in {ℝ}^{t\times k\times k}$, such that

$$\underset{¯}{\mathit{S}}\approx \mathit{A}\underset{¯}{\mathit{R}}{\mathit{A}}^T$$

(14)

where we multiply each slice of $\underset{¯}{\mathit{R}}$ with A and ${\mathit{A}}^T$ to reconstruct the corresponding slice of $\underset{¯}{\mathit{S}}$:

$${\underset{¯}{\mathit{S}}}_{lij}\approx \sum _{b=1}^k\sum _{c=1}^k{\mathit{A}}_{ib}{\underset{¯}{\mathit{R}}}_{lbc}{\mathit{A}}_{jc}.$$

(15)

We keep our naming convention for A as the loading matrix and $\underset{¯}{\mathit{R}}$ as the affinity tensor, since again A gives us for each word i in the vocabulary a vector of size k and for each slice l the square matrix ${\underset{¯}{\mathit{R}}}_l:={\left({\underset{¯}{\mathit{R}}}_{lij}\right)}_{i,j=1}^k$ provides information on the relationships between the topics in the l-th input slice.

Analogous to (7) we construct a loss function

$$\begin{array}{cc}\hfill \mathcal{L}(\underset{¯}{\mathit{S}},\mathit{A},\underset{¯}{\mathit{R}})& ={∥\underset{¯}{\mathit{S}}-\mathit{A}\underset{¯}{\mathit{R}}{\left(\mathit{A}\right)}^T∥}_F^2\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill & =\sum _{l=1}^t\sum _{i=1}^n\sum _{j=1}^n{\left({\underset{¯}{\mathit{S}}}_{lij}-{\left(\mathit{A}{\underset{¯}{\mathit{R}}}_l{\left(\mathit{A}\right)}^T\right)}_{ij}\right)}^2\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & =\sum _{l=1}^t\mathcal{L}({\underset{¯}{\mathit{S}}}_l,\mathit{A},{\underset{¯}{\mathit{R}}}_l)\hfill \end{array}$$

(18)

with

$${\left(\mathit{A}{\underset{¯}{\mathit{R}}}_l{\left(\mathit{A}\right)}^T\right)}_{ij}=\sum _{b=1}^k\sum _{c=1}^k{\mathit{A}}_{ib}{\underset{¯}{\mathit{R}}}_{lbc}{\mathit{A}}_{jc}.$$

(19)

Note that in this framework, the DEDICOM algorithm described in the previous section is equivalent to tensor DEDICOM with $t=1$.

Update steps can then be taken via alternating gradient descent on A and $\underset{¯}{\mathit{R}}$. As in the previous section, one can now add additional constraints to A and $\underset{¯}{\mathit{R}}$ and calculate the gradients as in (10), using automatic differentiation methods. Taking update steps of size ${\eta }^{\mathit{A}}$ and ${\eta }^{\underset{¯}{\mathit{R}}}$ respectively leads to an eventual convergence to some local or global minimum of the loss (16) with respect to the original or constrained A and $\underset{¯}{\mathit{R}}$ .

Alternatively, constraints can be added to A and $\underset{¯}{\mathit{R}}$ by methods like projected gradient descent and the Frank-Wolfe algorithm [21] which either adjust the respective matrix or tensor to be constrained after the gradient step or apply the general gradient step such that the matrix or tensor never leaves the constrained area.

However, empirical results show that automatic differentiation methods lead to slow and unstable training convergence and worse qualitative results when applying the mentioned constraints on the factor matrices and tensors. We therefore derive an alternative method of applying alternating gradient descent to A and $\underset{¯}{\mathit{R}}$ based on multiplicative update rules. This does not only improve training stability and convergence behavior but also lead to better qualitative results (see Section 4.3 and Figure 2).

We derive the gradients for A and R analytically and set the learning rates ${\eta }^{\mathit{A}}$ and ${\eta }^{\underset{¯}{\mathit{R}}}$ individually for each element $(i,j)$ as ${\eta }_{ij}^{\mathit{A}}$ for matrix A and for each element $(l,i,j)$ as ${\eta }_{lij}^{\underset{¯}{\mathit{R}}}$ for tensor $\underset{¯}{\mathit{R}}$, such that the resulting update step is an element-wise multiplication of the respective matrix or tensor.

We derive the updates for the matrix algorithm first and later extend them for the tensor case. For detailed derivations refer to Appendix B. For A we derive the gradient analytically as

$$\begin{array}{c}\hfill \frac{\partial \mathcal{L}(\mathit{S},\mathit{A},\mathit{R})}{\partial \mathit{A}}=-2\left(\mathit{S}\mathit{A}{\mathit{R}}^T+{\mathit{S}}^T\mathit{A}\mathit{R}-\mathit{A}\left(\mathit{R}{\mathit{A}}^T\mathit{A}{\mathit{R}}^T+{\mathit{R}}^T{\mathit{A}}^T\mathit{A}\mathit{R}\right)\right).\end{array}$$

(20)

Therefore the update step is

$$\begin{array}{c}\hfill {\mathit{A}}_{ij}\leftarrow {\mathit{A}}_{ij}+{\eta }_{ij}^{\mathit{A}}2\left({\left[\mathit{S}\mathit{A}{\mathit{R}}^T+{\mathit{S}}^T\mathit{A}\mathit{R}\right]}_{ij}-{\left[\mathit{A}\left(\mathit{R}{\mathit{A}}^T\mathit{A}{\mathit{R}}^T+{\mathit{R}}^T{\mathit{A}}^T\mathit{A}\mathit{R}\right)\right]}_{ij}\right).\end{array}$$

(21)

If we now chose ${\eta }^{\mathit{A}}$ as

$${\eta }_{ij}^{\mathit{A}}:=\frac{{\mathit{A}}_{ij}}{2{\left[\mathit{A}({\mathit{R}}^T{\mathit{A}}^T\mathit{A}\mathit{R}+\mathit{R}{\mathit{A}}^T\mathit{A}{\mathit{R}}^T)\right]}_{ij}}.$$

(22)

the update (21) becomes

$${\mathit{A}}_{ij}\leftarrow {\mathit{A}}_{ij}\frac{{[{\mathit{S}}^T\mathit{A}\mathit{R}+\mathit{S}\mathit{A}{\mathit{R}}^T]}_{ij}}{{\left[\mathit{A}({\mathit{R}}^T{\mathit{A}}^T\mathit{A}\mathit{R}+\mathit{R}{\mathit{A}}^T\mathit{A}{\mathit{R}}^T)\right]}_{ij}}.$$

(23)

For R we derive the gradient analytically as

$$\begin{array}{c}\hfill \frac{\partial \mathcal{L}(\mathit{S},\mathit{A},\mathit{R})}{\partial \mathit{R}}=-2({\mathit{A}}^T\mathit{S}\mathit{A}-{\mathit{A}}^T\mathit{A}\mathit{R}{\mathit{A}}^T\mathit{A}).\end{array}$$

(24)

Therefore the update step is

$$\begin{array}{cc}\hfill {\mathit{R}}_{ij}& ={\mathit{R}}_{ij}+{\eta }_{ij}^{\mathit{R}}2({\left[{\mathit{A}}^T\mathit{S}\mathit{A}\right]}_{ij}-{\left[{\mathit{A}}^T\mathit{A}\mathit{R}{\mathit{A}}^T\mathit{A}\right]}_{ij}).\hfill \end{array}$$

(25)

Choose

$$\begin{array}{c}\hfill {\eta }_{ij}^{\mathit{R}}:=\frac{{\mathit{R}}_{ij}}{2{\left[{\mathit{A}}^T\mathit{A}\mathit{R}{\mathit{A}}^T\mathit{A}\right]}_{ij}},\end{array}$$

(26)

and the update (25) becomes

$${\mathit{R}}_{ij}\leftarrow {\mathit{R}}_{ij}\frac{{\left[{\mathit{A}}^T\mathit{S}\mathit{A}\right]}_{ij}}{{\left[{\mathit{A}}^T\mathit{A}\mathit{R}{\mathit{A}}^T\mathit{A}\right]}_{ij}}.$$

(27)

Since ${\mathit{S}}_{ij}\ge 0$ for all $i,j$, in both (23) and (27) each element of the multiplier matrix is positive if both $\mathit{A}\ge 0$ and $\mathit{R}\ge 0$ in all entries. Therefore, initializing both matrices with positive values results in an update step that keeps the elements of A and R positive.

To extend this rule to tensor DEDICOM, consider that the analytical derivatives hold for $\underset{¯}{\mathit{R}}$ and A by considering each slice ${\underset{¯}{\mathit{S}}}_l$ and ${\underset{¯}{\mathit{R}}}_l$ individually:

$$\begin{array}{cc}\hfill \frac{\partial \mathcal{L}({\underset{¯}{\mathit{S}}}_l,\mathit{A},{\underset{¯}{\mathit{R}}}_l)}{\partial {\underset{¯}{\mathit{R}}}_l}& =-2({\mathit{A}}^T{\underset{¯}{\mathit{S}}}_l\mathit{A}-{\mathit{A}}^T\mathit{A}{\underset{¯}{\mathit{R}}}_l{\mathit{A}}^T\mathit{A}),\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill \frac{\partial \mathcal{L}({\underset{¯}{\mathit{S}}}_l,\mathit{A},{\underset{¯}{\mathit{R}}}_l)}{\partial \mathit{A}}& =-2({\underset{¯}{\mathit{S}}}_l^T\mathit{A}{\mathit{R}}_l+{\underset{¯}{\mathit{S}}}_l\mathit{A}{\underset{¯}{\mathit{R}}}_l^T-\mathit{A}({\underset{¯}{\mathit{R}}}_l^T{\mathit{A}}^T\mathit{A}{\underset{¯}{\mathit{R}}}_l+{\underset{¯}{\mathit{R}}}_l{\mathit{A}}^T\mathit{A}{\underset{¯}{\mathit{R}}}_l^T)).\hfill \end{array}$$

(29)

Since by (18) we have $\mathcal{L}(\underset{¯}{\mathit{S}},\mathit{A},\underset{¯}{\mathit{R}})={\sum }_{l=1}^t\mathcal{L}({\underset{¯}{\mathit{S}}}_l,\mathit{A},{\underset{¯}{\mathit{R}}}_l)$ we can derive the full gradients as

$$\begin{array}{cc}\hfill \frac{\partial \mathcal{L}(\underset{¯}{\mathit{S}},\mathit{A},\underset{¯}{\mathit{R}})}{\partial \underset{¯}{\mathit{R}}}& =({\mathit{A}}^T\underset{¯}{\mathit{S}}\mathit{A}-{\mathit{A}}^T\mathit{A}\underset{¯}{\mathit{R}}{\mathit{A}}^T\mathit{A}),\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill \frac{\partial \mathcal{L}(\underset{¯}{\mathit{S}},\mathit{A},\underset{¯}{\mathit{R}})}{\partial \mathit{A}}& =\sum _{l=1}^t-2({\underset{¯}{\mathit{S}}}_l^T\mathit{A}{\underset{¯}{\mathit{R}}}_l+{\underset{¯}{\mathit{S}}}_l\mathit{A}{\underset{¯}{\mathit{R}}}_l^T-\mathit{A}({\underset{¯}{\mathit{R}}}_l^T{\mathit{A}}^T\mathit{A}{\underset{¯}{\mathit{R}}}_l+{\underset{¯}{\mathit{R}}}_l{\mathit{A}}^T\mathit{A}{\underset{¯}{\mathit{R}}}_l^T)).\hfill \end{array}$$

(31)

For A we set ${\eta }^{\mathit{A}}$ as

$${\eta }_{ij}^{\mathit{A}}:=\frac{{\mathit{A}}_{ij}}{2{\sum }_{l=1}^t{\left[\mathit{A}({\underset{¯}{\mathit{R}}}_l^T{\mathit{A}}^T\mathit{A}{\underset{¯}{\mathit{R}}}_l+{\underset{¯}{\mathit{R}}}_l{\mathit{A}}^T\mathit{A}{\underset{¯}{\mathit{R}}}_l^T)\right]}_{ij}}.$$

(32)

Then the update step is

$$\begin{array}{cc}\hfill {\mathit{A}}_{ij}& \leftarrow {\mathit{A}}_{ij}-{\eta }_{ij}^{\mathit{A}}\frac{\partial \mathcal{L}(\underset{¯}{\mathit{S}},\mathit{A},\underset{¯}{\mathit{R}})}{\partial \mathit{A}}\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill & ={\mathit{A}}_{ij}\frac{{\sum }_{l=1}^t[{\underset{¯}{\mathit{S}}}_l^T\mathit{A}{\underset{¯}{\mathit{R}}}_l+{\underset{¯}{\mathit{S}}}_l\mathit{A}{\underset{¯}{\mathit{R}}}_l^T]}{{\sum }_{l=1}^t{\left[\mathit{A}({\underset{¯}{\mathit{R}}}_l^T{\mathit{A}}^T\mathit{A}{\underset{¯}{\mathit{R}}}_l+{\underset{¯}{\mathit{R}}}_l{\mathit{A}}^T\mathit{A}{\underset{¯}{\mathit{R}}}_l^T)\right]}_{ij}}.\hfill \end{array}$$

(34)

For $\underset{¯}{\mathit{R}}$ we again set

$$\begin{array}{c}\hfill {\eta }_{lij}^{\underset{¯}{\mathit{R}}}:=\frac{{\underset{¯}{\mathit{R}}}_{lij}}{2{\left[{\mathit{A}}^T\mathit{A}{\underset{¯}{\mathit{R}}}_l{\mathit{A}}^T\mathit{A}\right]}_{ij}},\end{array}$$

(35)

and the update (25) becomes

$${\underset{¯}{\mathit{R}}}_{lij}\leftarrow {\underset{¯}{\mathit{R}}}_{lij}\frac{{\left[{\mathit{A}}^T{\underset{¯}{\mathit{S}}}_l\mathit{A}\right]}_{ij}}{{\left[{\mathit{A}}^T\mathit{A}{\underset{¯}{\mathit{R}}}_l{\mathit{A}}^T\mathit{A}\right]}_{ij}}.$$

(36)

Equations (23) and (27) provide multiplicative update rules that ensure the non-negativity of A and R without any additional constraints. Equations (33) and (36) provide the corresponding rules for matrix A and tensor $\underset{¯}{\mathit{R}}$ in tensor DEDICOM.

3.3. On Symmetry

The DEDICOM algorithm is able to factorize both symmetrical and asymmetrical matrices S. For a given matrix A, the symmetry of R dictates the symmetry of the product $\mathit{A}\mathit{R}{\mathit{A}}^T$, since

$$\begin{array}{cc}\hfill {\left(\mathit{A}\mathit{R}{\mathit{A}}^T\right)}_{ij}& =\sum _{b=1}^k\sum _{c=1}^k{\mathit{A}}_{ib}{\mathit{R}}_{bc}{\mathit{A}}_{jc}=\sum _{b=1}^k\sum _{c=1}^k{\mathit{A}}_{ib}{\mathit{R}}_{cb}{\mathit{A}}_{jc}\hfill \end{array}$$

(37)

$$\begin{array}{cc}\hfill & =\sum _{c=1}^k\sum _{b=1}^k{\mathit{A}}_{jc}{\mathit{R}}_{cb}{\mathit{A}}_{ib}={\left(\mathit{A}\mathit{R}{\mathit{A}}^T\right)}_{ji}\hfill \end{array}$$

(38)

if ${\mathit{R}}_{cb}={\mathit{R}}_{bc}$ for all $b,c$. We therefore expect a symmetric matrix S to be decomposed into $\mathit{A}\mathit{R}{\mathit{A}}^T$ with a symmetric R, which is confirmed by our experiments. Factorizing a non-symmetric matrix leads to a non-symmetric R, the asymmetric relation between items leads to asymmetric relations between the latent factors. The same relations hold for each slice ${\underset{¯}{\mathit{S}}}_l$ and ${\underset{¯}{\mathit{R}}}_l$ in tensor DEDICOM.

3.4. On Interpretability

We have

$${\mathit{S}}_{ij}\approx \sum _{b=1}^k\sum _{c=1}^k{\mathit{A}}_{ib}{\mathit{R}}_{bc}{\mathit{A}}_{jc},$$

(39)

i.e., we can estimate the probability of co-occurrence of two words $w_i$ and $w_j$ from the word embeddings ${\mathit{A}}_i$ and ${\mathit{A}}_j$ and the matrix R, where ${\mathit{A}}_i$ denotes the i-th row of A.

If we want to predict the co-occurrence between words ${\mathit{w}}_i$ and ${\mathit{w}}_j$ we consider the latent topics that make up the word embeddings ${\mathit{A}}_i$ and ${\mathit{A}}_j$, and sum up each component from ${\mathit{A}}_i$ with each component ${\mathit{A}}_j$ with respect to the relationship weights given in R.

Two words are likely to have a high co-occurrence if their word embeddings have larger weights in topics that are positively connected by the R matrix. Likewise a negative entry ${\mathit{R}}_{b,c}$ makes it less likely for words with high weight in the topics b and c to occur in the same context. See Figure 3 for an illustrated example.

Having an interpretable embedding model provides value beyond analysis of the affinity matrix of a single document. The worth of word embeddings is generally measured in their usefulness for downstream tasks. Given a prediction model based on word embeddings as one of the inputs, further analysis of the model behavior is facilitated when latent input dimensions easily translate to semantic meaning.

In most word embedding models, the embedding vector of a single word is not particularly useful in itself. The information only lies in its relationship (i.e., closeness or cosine similarity) to other embedding vectors. For example, an analysis of the change of word embeddings and therefore the change of word meaning within a document corpus (for example a news article corpus) can only show how various words form different clusters or drift apart over time. Interpretabilty of latent dimensions would provide tools to also consider the development of single words within the given topics.

All considerations above hold for the three-dimensional tensor case, in which we analyze a slice ${\underset{¯}{\mathit{R}}}_l$ together with the common word embedding matrix A to gain insight into the input data slice ${\underset{¯}{\mathit{S}}}_l$.

4. Experiments and Results

In the following section we describe our experimental setup in full detail (Our Python implementation to reproduce the results is available on https://github.com/LarsHill/text-dedicom-paper. Additionally, we provide a snapshot of our versions of the applied public datasets (Wikipedia articles and Amazon reviews). ) and present our results on the simultaneous topic (relation) extraction and word embedding learning task. We compare these results against competing matrix and tensor factorization methods for topic modeling, namely NMF (including a Tucker-2 variation compatible with tensors), LDA and SVD.

4.1. Data

We conducted our experiments on three orthogonal text datasets which cover different text domains and allow for a thorough empirical analysis of our proposed methods.

The first corpus leveraged triplets of individual Wikipedia articles. The articles were retrieved as raw text via the official Wikipedia API using the wikipedia-api library. We differentiated between thematically similar (e.g., “Dolphin” and “Whale”) and thematically different articles (e.g., “Soccer” and “Donald Trump”). Each article triplet was categorized into one of three classes: All underlying Wikipedia articles were thematically different, two articles were thematically similar and one was different, and all articles were thematically similar. The previous paper [3] contained an extensive evaluation over 12 triples of articles in the supplementary material. In this work we focused on the three triples described in the previous main paper, namely

• “Soccer”, “Bee”, “Johnny Depp”,
• “Dolphin”, “Shark”, “Whale”, and
• “Soccer”, “Tennis”, “Rugby”.

Depending on whether the article triplets were represented as input matrix or tensor they were processed differently. In the case of a matrix input all three articles got concatenated to form a new artificially generated document. In the case of a tensor input the articles remained individual documents later which later represented slices in the tensor representation.

To analyze the topic extraction capability of constrained DEDICOM also on text which was rather prone to grammatical and syntactical errors, we utilized a subset of the Amazon review dataset [22]. In particular, we restricted ourselves to the “movie” product category and created a corpus consisting of six text documents holding the concatenated reviews from the following films respectively, “Toy Story 1”, “Toy Story 3”, “Frozen”, “Monsters, Inc.”, “Kung Fu Panda” and “Kung Fu Panda 2”. Grouping the reviews by movie affiliation enabled us to generate a tensor representation of the corpus which we factorized via non-negative tensor DEDICOM to analyze topic relations across movies.

Table 1. Amazon movie review corpus grouped by movie and number of reviews per slice of input tensor.

Movie	# Reviews
Toy Story 1	2491
Monsters, Inc.	3203
Kung Fu Panda 1	6708
Toy Story 3	1209
Kung Fu Panda 2	1208
Frozen	1292

lists the number of reviews per movie and shows that based on review count “Kung Fu Panda 1” was the most popular among the six films.

The third corpus represented a complete collection of New York Times news articles ranging from 1st September 2019 to 31st August 2020. The articles were taken from the New York Times website and covered a wide range of sections (see

Table 2. New York Times news corpus composition by section and number of articles.

Section	# Articles
Politics	3204
U.S.	2610
Business	1624
New York	1528
Europe	988
Asia Pacific	839
Health	598
Technology	551
Middle East	443
Science	440
Economy	339
Elections	240
Climate	239
World	233
Africa	124
Australia	113
Canada	104

).

Instead of grouping the articles by section we binned and concatenated them by month yielding 12 news documents containing monthly information (see

Table 3. New York Times news corpus grouped by month and number of articles. This corresponds to the number of articles per slice of input tensor.

Month	# Articles
September 2019	1586
October 2019	1788
November 2019	1623
December 2019	1461
January 2020	1725
Febuary 2020	1602
March 2020	1937
April 2020	1712
May 2020	1713
June 2020	1828
July 2020	1814
August 2020	1886

for details on the article count per month). Thereby, the factorization of tensor DEDICOM allowed for an analysis of topic relations and their changes over time.

Before transforming the text documents to matrix or tensor representations we applied the following textual preprocessing steps. First, the whole text was made lower-cased. Second, we tokenized the text making use of the word-tokenizer from the nltk library and removed common English stop words, including contractions such as “you’re” and “we’ll”. Lastly we cleared the text from all remaining punctuation and deleted digits, single characters and multi-spaces (see

Table 4. Overview of word count statistics after preprocessing for all datasets. Columns represent from left to right the total number of words per corpus, total number of unique words per corpus, average number of total words per article, average number of unique words per article and the cutoff frequency of the 10,000th most common word. Wikipedia article combinations: Dolphin, Shark, Whale (DSW), Soccer, Bee, Johnny Depp (SBJ), Soccer, Tennis, Rugby (STR).

	Total	Unique	Avg. Total	Avg Unique	Cutoff
Amazon Reviews	252,400	15,560	15.2	13.4	1
Wikipedia DSW	14,500	4376	4833.3	2106.0	1
Wikipedia SBJ	10,435	4034	3478.3	1600.3	1
Wikipedia STR	11,501	3224	3833.7	1408.0	1
New York Times	12,043,205	141,591	582.5	366.5	118

for an overview of corpora statistics after preprocessing).

Next, we utilized all preprocessed documents in a corpus to extract a fixed size vocabulary of n = 10,000 most frequent tokens. Since our dense input tensor was of dimensionality $t\times n\times n$, a larger vocabulary size led to a significant increase in memory consumption. Based on the total number of unique corpora words reported in Table 4, a maximum vocabulary size of n = 10,000 was reasonable for the three Wikipedia corpora and the Amazon reviews corpus. Only the New York Times dataset could potentially have benefited from a larger vocabulary size.

Based on this vocabulary a symmetric word co-occurrence matrix was calculated for each of the corpus documents. When generating the matrix we only considered context words within a symmetrical window around the base word. Analysis in [2,3] showed that the window size in the range of 6 to 10 had little impact on performance. Thus, following our implementation in [3], we chose a window size of 7, the default in the original glove implementation. Like in [2], each context word only contributed $1/d$ to the total word pair count, given it was d words apart from the base word. To avoid any bias or prior information from the structure and order of the concatenated Wikipedia articles, reviews or news articles, we randomly shuffled the vocabulary before creating the co-occurrence matrix. As described in Section 3 we then transformed the co-occurrence matrix to a positive PMI matrix. If the corpus consisted of just one document the generated PPMI matrix functions as input for the row-stochastic DEDICOM algorithm. If the corpus consisted of several documents (e.g., one news document per month) the individual PPMI matrices were stacked to a tensor which in turn represented the input for the non-negative tensor DEDICOM algorithm.

The next section sheds light upon the training process of row-stochastic DEDICOM, non-negative tensor DEDICOM and the above mentioned competing matrix and tensor factorization methods, which will be benchmarked against our results in Section 4.3 and in the Appendix A and Appendix B.

4.2. Training

As thoroughly outlined in Section 3 we trained both the row-stochastic DEDICOM and non-negative tensor DEDICOM with the alternating gradient descent paradigm.

In the case of a matrix input and a row-stochasticity constraint on A we utilize automatic differentiation from the PyTorch library to perform update steps on A and R. First, we initialized the factor matrices $\mathit{A}\in {ℝ}^{n\times k}$ and $\mathit{R}\in {ℝ}^{k\times k}$, by randomly sampling all elements from a uniform distribution centered around 1, $\mathcal{U}(0,2)$. Note that after applying the softmax operation on A all rows of A were stochastic. Therefore, scaling R by

$$\overline{s}:=\frac{1}{n^2}\sum _{ij}^n{\mathit{S}}_{ij},$$

(40)

would result in the initial decomposition ${\mathit{A}}^{\prime }\mathit{R}{\left({\mathit{A}}^{\prime }\right)}^T$ yielding reconstructed elements in the range of $\overline{s}$, the element mean of the PPMI matrix S, and thus, speeding up convergence. Second, A and R were iteratively updated employing the Adam optimizer [23] with constant individual learning rates of ${\eta }^{\mathit{A}}=0.001$ and ${\eta }^{\mathit{R}}=0.01$ and hyperparameters ${\beta }_1=0.9$, ${\beta }_2=0.999$ and $ϵ=1\times {10}^{-8}$. Both learning rates were identified through an exhaustive grid search. We trained for $\mathtt{num}\_\mathtt{epochs}$ = 15,000 until convergence, where each epoch consisted of an alternating gradient update with respect to A and R. Algorithm 1 illustrates the just described training procedure.

Algorithm 1 The row-stochastic DEDICOM algorithm
1: initialize $\mathit{A},\mathit{R}\leftarrow U(0,2)·\overline{s}$	⊳See Equation (40) for the definition of $\overline{s}$
2: initialize ${\beta }_1$, ${\beta }_2$, $ϵ$	⊳Adam algorithm hyperparameters
3: initialize ${\eta }^{\mathit{A}}$, ${\eta }^{\mathit{R}}$	⊳Individual learning rates
4: for i in $1,\dots$, num_epochs do
5:    Calculate loss $\mathcal{L}=\mathcal{L}(\mathit{S},\mathit{A},\mathit{R})$	⊳See Equation (7)
6:    $\begin{array}{c}\hfill \mathit{A}↤\mathit{A}-{\mathrm{Adam}}_{{\beta }_1,{\beta }_2,ϵ}({\nabla }_{\mathit{A}},{\eta }^{\mathit{A}}),\mathrm{where}{\nabla }_{\mathit{A}}=\frac{\partial \mathcal{L}}{\partial \mathit{A}}\end{array}$
7:    $\begin{array}{c}\hfill \mathit{R}↤\mathit{R}-{\mathrm{Adam}}_{{\beta }_1,{\beta }_2,ϵ}({\nabla }_{\mathit{R}},{\eta }^{\mathit{R}}),\mathrm{where}{\nabla }_{\mathit{R}}=\frac{\partial \mathcal{L}}{\partial \mathit{R}}\end{array}$
8: return ${\mathit{A}}^{\prime }$ and R, where ${\mathit{A}}^{\prime }=\mathrm{row}\_\mathrm{softmax}\left(\mathrm{clo\_norm}\right(\mathit{A}\left)\right)$	⊳See Equation (5)

In the case of a tensor input and an additional non-negativity constraint on $\underset{¯}{\mathit{R}}$ we noticed an inferior training performance with automatic differentiation methods. Hence, due to faster and more stable training convergence and improved qualitative results, we updated A and $\underset{¯}{\mathit{R}}$ iteratively via derived multiplicative update rules enforcing non-negativity. Again, we initialized $\mathit{A}\in {ℝ}^{n\times k}$ and $\underset{¯}{\mathit{R}}\in {ℝ}^{t\times k\times k}$, by randomly sampling all elements from a uniform distribution centered around 1, $\mathcal{U}(0,2)$. In order to ensure that the initialized components yielded a reconstructed tensor whose elements were in the same range of the input, we calculated an appropriate scaling factor for each tensor slice ${\underset{¯}{\mathit{S}}}_l$ as

$${\alpha }_l:={\left(\frac{{\overline{s}}_l}{k^2}\right)}^{\frac{1}{3}},\mathrm{where}{\overline{s}}_l:=\frac{1}{n^2}\sum _{ij}^n{\mathit{S}}_{lij}.$$

(41)

Next, we scaled A by $\overline{\alpha }=\frac{1}{t}{\sum }_{l=1}^t{\alpha }_l$ and each slice ${\underset{¯}{\mathit{R}}}_l$ by ${\alpha }_l$ before starting the alternating multiplicative update steps for $\mathtt{num}\_\mathtt{epochs}=300$. The detailed derivation of the update rules is found in Section 3.2 and their iterative application in the training process is described in Algorithm 2.

Algorithm 2 The non-negative tensor DEDICOM algorithm
1: initialize $\mathit{A},\underset{¯}{\mathit{R}}\leftarrow U(0,2)$
2: scale A by $\overline{\alpha }$ and ${\underset{¯}{\mathit{R}}}_l$ by ${\alpha }_l$	⊳See Equation (41) for the definitions of $\overline{\alpha }$ and ${\alpha }_l$
3: for i in $1,\dots$, num_epochs do
4:     Calculate loss $\mathcal{L}=\mathcal{L}(\underset{¯}{\mathit{S}},\mathit{A},\underset{¯}{\mathit{R}})$	⊳See Equation (17)
5:      $\begin{array}{c}\hfill {\mathit{A}}_{ij}↤{\mathit{A}}_{ij}\frac{{\left[{\sum }_{l=1}^t\left({\underset{¯}{\mathit{S}}}_l\mathit{A}{\underset{¯}{\mathit{R}}}_l^T+{\underset{¯}{\mathit{S}}}_l^T\mathit{A}{\underset{¯}{\mathit{R}}}_l\right)\right]}_{ij}}{{\left[\mathit{A}{\sum }_{l=1}^t\left({\underset{¯}{\mathit{R}}}_l{\mathit{A}}^T\mathit{A}{\underset{¯}{\mathit{R}}}_l^T+{\underset{¯}{\mathit{R}}}_l^T{\mathit{A}}^T\mathit{A}{\underset{¯}{\mathit{R}}}_l\right)\right]}_{ij}}\end{array}$
6:      $\begin{array}{c}\hfill {\underset{¯}{\mathit{R}}}_{lij}↤{\underset{¯}{\mathit{R}}}_{lij}\frac{{\left[{\mathit{A}}^T{\underset{¯}{\mathit{S}}}_l\mathit{A}\right]}_{ij}}{{\left[{\mathit{A}}^T\mathit{A}{\underset{¯}{\mathit{R}}}_l{\mathit{A}}^T\mathit{A}\right]}_{ij}}\end{array}$
7: return A and $\underset{¯}{\mathit{R}}$

We implemented NMF, LDA and SVD using the sklearn library. In all cases the learnable factor matrices were initialized randomly and default hyperparameters were applied during training. For NMF the multiplicative update rule from [14] was utilized.

Figure 4 shows the convergence behavior of the row-stochastic matrix DEDICOM training process and the final loss of NMF and SVD. Note that LDA optimized a different loss function, which is why the calculated loss was not comparable and therefore excluded. We see that the final loss of DEDICOM was located just above the other losses, which is reasonable when considering the row stochasticity contraint on A and the reduced parameter amount of $nk+k^2$ compared to NMF ($2nk$) and SVD ($2nk+k^2$).

To also have a benchmark model for our constrained tensor DEDICOM methods to compare against, we implemented a Tucker-2 variation of NMF, named tensor NMF (TNMF), which factorized the input tensor $\underset{¯}{\mathit{S}}$ as

$$\begin{array}{c}\hfill {\underset{¯}{\mathit{S}}}_l\approx \underset{¯}{\mathit{W}}{H}_l.\end{array}$$

(42)

Its training procedure closely followed the above described alternating gradient descent approach for non-negative tensor DEDICOM. However, due to the two-way factorization (three-way for DEDICOM) the scaling factor ${\alpha }_l$ to properly initialize $\underset{¯}{\mathit{W}}$ and H had to be adapted to

$${\alpha }_l:={\left(\frac{{\overline{s}}_l}{k}\right)}^{\frac{1}{2}},\mathrm{where}{\overline{s}}_l:=\frac{1}{n^2}\sum _{ij}^n{\underset{¯}{\mathit{S}}}_{lij}.$$

(43)

Analogous to Figure 4, we compared the training stability and convergence speed of our implemented tensor factorization methods. In particular, Figure 2 visualizes the reconstruction loss development for non-negative tensor DEDICOM trained via multiplicative update rules, row-stochastic tensor DEDICOM trained with automatic differentiation and the Adam optimizer and tensor NMF. It could be clearly observed that row-stochastic tensor DEDICOM converged much slower than the other two models trained with multiplicative update rules (learning rates were implicit here and did not have to be tuned).

4.3. Results

In the following, we present our results of training the above mentioned constrained DEDICOM factorizations on different text corpora to simultaneously learn interpretable word embeddings and meaningful topic clusters and their relations.

First, we focused our analysis on row-stochastic matrix DEDICOM applied to the synthetic Wikipedia text documents described in Section 4.1. For compactness reasons we primarily considered the document “Soccer, Bee and Johnny Depp”, set the number of topics to $k=6$ and refer to Appendix A.1 for the other article combinations and competing matrix factorization results. Second, we extended our evaluation to the tensor representation of the Wikipedia documents ($t=3$, one article per tensor slice) and compared the performance of non-negative (multiplicative updates) and row-stochastic (Adam updates) tensor DEDICOM. Lastly, we applied non-negative tensor DEDICOM to the binned Amazon movie and New York Times news corpora to investigate topic relations across movies and over time. We again point the interested reader to Appendix A for additional results and the comparison to tensor NMF.

In the first step, we evaluated the quality of the learned latent topics by assigning each word embedding ${\mathit{A}}_i^{\prime }\in {ℝ}^{1\times k}$ to the latent topic dimension that represents the maximum value in ${\mathit{A}}_i^{\prime }$, e.g.,

$$\begin{array}{cc}\hfill {\mathit{A}}_i^{\prime }& =\left[\begin{array}{cccccc}0.05& 0.03& 0.02& 0.14& 0.70& 0.06\end{array}\right],\mathrm{argmax}\left({\mathit{A}}_i^{\prime }\right)=5,\hfill \end{array}$$

and thus, ${\mathit{A}}_i^{\prime }$ was matched to Topic 5. Next, we decreasingly sorted the words within each topic based on their matched topic probability.

Table 5. The top 10 representative words per dimension of the basis matrix ${\mathit{A}}^{\prime }$, trained on the Wikipedia data as input matrix using automatic gradient methods.

	Topic 1 #619	Topic 2 #1238	Topic 3 #628	Topic 4 #595	Topic 5 #612	Topic 6 #389
1	ball	film	salazar	cup	bees	heard
	(0.77)	(0.857)	(0.201)	(0.792)	(0.851)	(0.738)
2	penalty	starred	geoffrey	football	species	court
	(0.708)	(0.613)	(0.2)	(0.745)	(0.771)	(0.512)
3	may	role	rush	fifa	bee	depp
	(0.703)	(0.577)	(0.2)	(0.731)	(0.753)	(0.505)
4	referee	series	brenton	world	pollen	divorce
	(0.667)	(0.504)	(0.199)	(0.713)	(0.658)	(0.454)
5	goal	burton	hardwicke	national	honey	alcohol
	(0.66)	(0.492)	(0.198)	(0.639)	(0.602)	(0.435)
6	team	character	thwaites	uefa	insects	paradis
	(0.651)	(0.465)	(0.198)	(0.623)	(0.576)	(0.42)
7	players	played	catherine	continental	food	relationship
	(0.643)	(0.451)	(0.198)	(0.582)	(0.536)	(0.419)
8	player	director	kaya	teams	nests	abuse
	(0.639)	(0.45)	(0.198)	(0.576)	(0.529)	(0.41)
9	play	success	melfi	european	solitary	stating
	(0.606)	(0.438)	(0.198)	(0.57)	(0.513)	(0.408)
10	game	jack	raimi	association	eusocial	stated
	(0.591)	(0.434)	(0.198)	(0.563)	(0.505)	(0.402)

shows the overall number of allocated words and the resulting top 10 words per topic together with each matched probability.

Indicated by the high assignment probabilities, one can see that columns 1, 2, 4, 5 and 6 represent distinct topics, which can easily be interpreted. Topic 1 and 4 were related to soccer, where 1 focused on the game mechanics and 4 on the organizational and professional aspect of the game. Topic 2 and 6 clearly referred to Johnny Depp, where 2 focused on his acting career and 6 on his difficult relationship to Amber Heard. The fifth topic obviously related to the insect “bee”. In contrast, Topic 3 did not allow for any interpretation and all assignment probabilities were significantly lower than for the other topics.

Further, we analyzed the relations between the topics by visualizing the trained R matrix as a heatmap (see Figure 5c).

One thing to note was the symmetry of R which was a first indicator of a successful reconstruction, ${\mathit{A}}^{\prime }\mathit{R}{\left({\mathit{A}}^{\prime }\right)}^T$, (see Section 3.3). In addition, the main diagonal elements were consistently blue (positive), which suggested a high distinction between the topics. Although not very strong one could still see a connection between Topic 2 and 6 indicated by the light blue entry ${\mathit{R}}_{26}={\mathit{R}}_{62}$. While the suggested relation between Topic 1 and 4 was not clearly visible, element ${\mathit{R}}_{14}={\mathit{R}}_{41}$ was the least negative one for Topic 1. In order to visualize the topic cluster quality we utilized Uniform Manifold Approximation ad Projection (UMAP) [24] to map the k-dimensional word embeddings to a 2-dimensional space. Figure 5a illustrates this low-dimensional representation of ${\mathit{A}}^{\prime }$, where each word is colored based on the above described word to topic assignment. In conjunction with Table 5 one could nicely see that Topic 2 and 6 (Johnny Depp) and Topic 1 and 4 (Soccer) were close to each other. Hence, Figure 5a implicitly shows the learned topic relations as well.

As an additional benchmark, Figure 5b plots the same 2-dimensional representation, but now each word is colored based on the original Wikipedia article it belonged to. Words that occurred in more than one article were not considered in this plot.

Directly comparing Figure 5a,b shows that row-stochastic DEDICOM did not only recover the original articles but also found entirely new topics, which in this case represented subtopics of the articles. Let us emphasize that for all thematically similar article combinations, the found topics were usually not subtopics of a single article, but rather novel topics that might span across multiple Wikipedia articles (see for example

Table A2. Top half lists the top 10 representative words per dimension of the basis matrix A, bottom half lists the five most similar words based on cosine similarity for the two top words from each topic.

	Topic 1 #460	Topic 2 #665	Topic 3 #801	Topic 4 #753	Topic 5 #854	Topic 6 #721
1	shark	calf	ship	conservation	water	dolphin
	(0.665)	(0.428)	(0.459)	(0.334)	(0.416)	(0.691)
2	sharks	months	became	countries	similar	dolphins
	(0.645)	(0.407)	(0.448)	(0.312)	(0.374)	(0.655)
3	fins	calves	poseidon	government	tissue	captivity
	(0.487)	(0.407)	(0.44)	(0.309)	(0.373)	(0.549)
4	killed	females	riding	wales	body	wild
	(0.454)	(0.399)	(0.426)	(0.304)	(0.365)	(0.467)
5	million	blubber	dionysus	bycatch	swimming	behavior
	(0.451)	(0.374)	(0.422)	(0.29)	(0.357)	(0.461)
6	fish	young	ancient	cancelled	blood	bottlenose
	(0.448)	(0.37)	(0.42)	(0.288)	(0.346)	(0.453)
7	international	sperm	deity	eastern	surface	sometimes
	(0.442)	(0.356)	(0.412)	(0.287)	(0.344)	(0.449)
8	fin	born	ago	policy	oxygen	human
	(0.421)	(0.355)	(0.398)	(0.286)	(0.34)	(0.421)
9	fishing	feed	melicertes	control	system	less
	(0.405)	(0.349)	(0.395)	(0.285)	(0.336)	(0.42)
10	teeth	mysticetes	greeks	imminent	swim	various
	(0.398)	(0.341)	(0.394)	(0.282)	(0.336)	(0.418)
0	shark	calf	ship	conservation	water	dolphin
	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)
2	sharks	calves	dionysus	south	prey	dolphins
	(0.981)	(0.978)	(0.995)	(0.981)	(0.964)	(0.925)
3	fins	females	riding	states	swimming	sometimes
	(0.958)	(0.976)	(0.992)	(0.981)	(0.959)	(0.909)
4	killed	months	deity	united	allows	another
	(0.929)	(0.955)	(0.992)	(0.978)	(0.957)	(0.904)
5	fishing	young	poseidon	endangered	swim	bottlenose
	(0.916)	(0.948)	(0.987)	(0.976)	(0.947)	(0.903)
0	sharks	months	became	countries	similar	dolphins
	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)
2	shark	born	old	eastern	surface	behavior
	(0.981)	(0.992)	(0.953)	(0.991)	(0.992)	(0.956)
3	fins	young	later	united	brain	sometimes
	(0.936)	(0.992)	(0.946)	(0.989)	(0.97)	(0.945)
4	tiger	sperm	ago	caught	sound	various
	(0.894)	(0.985)	(0.939)	(0.987)	(0.968)	(0.943)
5	killed	calves	modern	south	object	less
	(0.887)	(0.984)	(0.937)	(0.979)	(0.965)	(0.937)

in the Appendix A). As mentioned at the top of this section, we are not only interested in learning meaningful topic clusters, but also in training interpretable word embeddings that capture semantic meaning.

Hence, we selected within each topic the two most representative words and calculated the cosine similarity between their word embeddings and all other word embeddings stored in ${\mathit{A}}^{\prime }$.

Table 6. For the most significant two words per topic, the four nearest neighbors based on cosine similarity are listed. Matrix ${\mathit{A}}^{\prime }$ trained on the wikipedia data as input matrix using automatic gradient methods.

	Topic 1	Topic 2	Topic 3	Topic 4	Topic 5	Topic 6
0	ball	film	salazar	cup	bees	heard
	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)
1	penalty	starred	geoffrey	fifa	bee	court
	(0.994)	(0.978)	(1.0)	(0.995)	(0.996)	(0.966)
2	referee	role	rush	national	species	divorce
	(0.992)	(0.964)	(1.0)	(0.991)	(0.995)	(0.944)
3	may	burton	bardem	world	pollen	alcohol
	(0.989)	(0.937)	(1.0)	(0.988)	(0.986)	(0.933)
4	goal	series	brenton	uefa	honey	abuse
	(0.986)	(0.935)	(1.0)	(0.987)	(0.971)	(0.914)
0	penalty	starred	geoffrey	football	species	court
	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)
1	referee	role	rush	fifa	bees	divorce
	(0.999)	(0.994)	(1.0)	(0.994)	(0.995)	(0.995)
2	goal	series	salazar	national	bee	alcohol
	(0.998)	(0.985)	(1.0)	(0.983)	(0.99)	(0.987)
3	player	burton	brenton	cup	pollen	abuse
	(0.997)	(0.981)	(1.0)	(0.983)	(0.99)	(0.982)
4	ball	film	thwaites	world	insects	settlement
	(0.994)	(0.978)	(1.0)	(0.982)	(0.977)	(0.978)

shows the four nearest neighbors based on cosine similarity for the top two words in each topic. We observed a high thematical similarity between words with large cosine similarity, indicating the usefulness of the rows of ${\mathit{A}}^{\prime }$ as word embeddings.

In comparison to DEDICOM, other matrix factorization methods also provided a useful clustering of words into topics, with varying degree of granularity and clarity. However, the application of these methods as word embedding algorithms mostly failed on the word similarity task, with words close in cosine similarity seldom sharing the same thematical similarity we have seen in DEDICOM. This can be seen in

Table A1. For each evaluated matrix factorization method we display the top 10 words for each topic and the five most similar words based on cosine similarity for the two top words from each topic.

		Topic 1	Topic 2	Topic 3	Topic 4	Topic 5	Topic 6
NMF		#619	#1238	#628	#595	#612	#389
	1	ball	bees	film	football	heard	album
	2	may	species	starred	cup	depp	band
	3	penalty	bee	role	world	court	guitar
	4	referee	pollen	series	fifa	alcohol	vampires
	5	players	honey	burton	national	relationship	rock
	6	team	insects	character	association	stated	hollywood
	7	goal	food	films	international	divorce	song
	8	game	nests	box	women	abuse	released
	9	player	solitary	office	teams	paradis	perry
	10	play	eusocial	jack	uefa	stating	debut
	0	ball	bees	film	football	heard	album
	1	invoke	odors	burtondirected	athenaeus	crew	jones
	2	replaced	tufts	tone	paralympic	alleging	marilyn
	3	scores	colour	landau	governing	oped	roots
	4	subdivided	affected	brother	varieties	asserted	drums
	0	may	species	starred	cup	depp	band
	1	yd	niko	shared	inaugurated	refer	heroes
	2	ineffectiveness	commercially	whitaker	confederation	york	bowie
	3	tactical	microbiota	eccentric	gold	leaders	debut
	4	slower	strategies	befriends	headquarters	nonindian	solo
LDA		#577	#728	#692	#607	#663	#814
	1	film	football	depp	penalty	bees	species
	2	series	women	children	heard	flowers	workers
	3	man	association	life	ball	bee	solitary
	4	played	fifa	role	direct	honey	players
	5	pirates	teams	starred	referee	pollen	colonies
	6	character	games	alongside	red	food	eusocial
	7	along	world	actor	time	increased	nest
	8	cast	cup	stated	goal	pollination	may
	9	also	game	burton	scored	times	size
	10	hollow	international	playing	player	larvae	egg
	0	film	football	depp	penalty	bees	species
	1	charlie	cup	critical	extra	bee	social
	2	near	canada	february	kicks	insects	chosen
	3	thinking	zealand	script	inner	authors	females
	4	shadows	activities	song	moving	hives	subspecies
	0	series	women	children	heard	flowers	workers
	1	crybaby	fifa	detective	allison	always	carcases
	2	waters	opera	crime	serious	eusociality	lived
	3	sang	exceeding	magazine	allergic	varroa	provisioned
	4	cast	cuju	barber	cost	wing	cuckoo
SVD		#1228	#797	#628	#369	#622	#437
	1	bees	depp	game	cup	heard	beekeeping
	2	also	film	ball	football	court	increased
	3	bee	starred	team	fifa	divorce	honey
	4	species	role	players	world	stating	described
	5	played	series	penalty	european	alcohol	use
	6	time	burton	play	uefa	paradis	wild
	7	one	character	may	national	documents	varroa
	8	first	actor	referee	europe	abuse	mites
	9	two	released	competitions	continental	settlement	colony
	10	pollen	release	laws	confederation	sued	flowers
	0	bees	depp	game	cup	heard	beekeeping
	1	bee	iii	correct	continental	alleging	varroa
	2	develops	racism	abandoned	contested	attempting	animals
	3	studied	appropriation	maximum	confederations	finalized	mites
	4	crops	march	clear	conmebol	submitted	plato
	0	also	film	ball	football	court	increased
	1	although	waters	finely	er	declaration	usage
	2	told	robinson	poised	suffix	issued	farmers
	3	chosen	scott	worn	word	restraining	mentioned
	4	stars	costars	manner	appended	verbally	aeneid

, which shows for each method, NMF, LDA and SVD, the resulting word to topic clustering and the cosine nearest neighbors of the top two word embeddings per topic. While the individual topics extracted by NMF looked very reasonable, its word embeddings did not seem to carry any semantic meaning based on cosine similarity; e.g., the four nearest neighbors of “ball” were “invoke”, “replaced”, “scores” and “subdivided”. A similar nonsensical picture can be observed for the other main topic words. LDA and SVD performed slightly better on the similar word task, although not all similar words appeared to be sensible, e.g., “children”, “detective”, “crime”, “magazine” and “barber”. In addition, some topics could not be clearly defined due to mixed word assignments, e.g., Topic 4 for LDA and Topic 1 for SVD.

Before shifting our analysis to the Amazon movie review and the New York Times news corpus we investigated factorizing the tensor representation of the “Soccer, Bee and Johnny Depp” Wikipedia documents. In particular, we compared the qualitative factorization results of row-stochastic and non-negative tensor DEDICOM trained with automatic differentiation and multiplicative update rules, respectively.

Table 7. Top 10 representative words per dimension of the basis matrix ${\mathit{A}}^{\prime }$, trained on the wikipedia data as input tensor using automatic gradient methods.

	Topic 1 #481	Topic 2 #661	Topic 3 #414	Topic 4 #457	Topic 5 #316	Topic 6 #1711
1	hind	game	film	heard	bees	disorder
	(0.646)	(0.83)	(0.941)	(0.844)	(0.922)	(0.291)
2	segments	football	starred	court	bee	collapse
	(0.572)	(0.828)	(0.684)	(0.566)	(0.868)	(0.29)
3	bacteria	players	role	divorce	honey	attrition
	(0.563)	(0.782)	(0.624)	(0.51)	(0.756)	(0.285)
4	legs	ball	series	depp	insects	losses
	(0.562)	(0.777)	(0.562)	(0.508)	(0.68)	(0.284)
5	antennae	team	burton	sued	food	invertebrate
	(0.555)	(0.771)	(0.547)	(0.48)	(0.634)	(0.283)
6	females	may	character	stating	species	rate
	(0.549)	(0.696)	(0.499)	(0.45)	(0.599)	(0.283)
7	wings	play	success	alcohol	nests	businesses
	(0.547)	(0.692)	(0.494)	(0.449)	(0.596)	(0.282)
8	small	competitions	played	paradis	flowers	virgil
	(0.538)	(0.677)	(0.483)	(0.446)	(0.571)	(0.282)
9	groups	match	films	alleged	pollen	iridescent
	(0.527)	(0.672)	(0.482)	(0.445)	(0.56)	(0.282)
10	males	penalty	box	stated	larvae	detail
	(0.518)	(0.664)	(0.465)	(0.444)	(0.529)	(0.281)

and

Table 8. Top 10 representative words per dimension of the basis matrix A, trained on the wikipedia data as input tensor using multiplicative update rules.

	Topic 1 #521	Topic 2 #249	Topic 3 #485	Topic 4 #871	Topic 5 #445	Topic 6 #1469
1	species	game	honey	allow	insects	depp
	(3.105)	(3.26)	(2.946)	(0.668)	(2.794)	(2.419)
2	eusocial	football	bee	organised	pollen	film
	(2.524)	(3.05)	(2.01)	(0.662)	(2.239)	(2.115)
3	solitary	players	beekeeping	winner	flowers	role
	(2.279)	(2.699)	(1.933)	(0.632)	(2.019)	(1.32)
4	nest	ball	bees	officially	nectar	starred
	(2.118)	(2.475)	(1.704)	(0.626)	(1.656)	(1.3)
5	females	may	increased	wins	wasps	actor
	(1.993)	(2.447)	(1.589)	(0.617)	(1.602)	(1.155)
6	workers	team	humans	level	wings	series
	(1.797)	(2.424)	(1.515)	(0.613)	(1.588)	(1.126)
7	nests	association	wild	free	many	burton
	(1.787)	(1.92)	(1.415)	(0.6)	(1.588)	(1.112)
8	colonies	play	mites	constitute	hind	played
	(1.722)	(1.834)	(1.4)	(0.596)	(1.577)	(1.068)
9	egg	referee	colony	regulation	hairs	heard
	(1.692)	(1.809)	(1.35)	(0.595)	(1.484)	(1.005)
10	males	laws	beekeepers	prestigious	pollinating	success
	(1.664)	(1.792)	(1.332)	(0.594)	(1.467)	(0.981)

in conjunction with Figure 6 and Figure 7 show the extracted topics and their relations for both methods.

It could be seen that non-negative tensor DEDICOM yielded a more interpretable affinity tensor $\underset{¯}{\mathit{R}}$ (Figure 7) due to its enforced non-negativity. For example, it clearly highlighted the bee related Topics 1, 3 and 5 in the affinity tensor slice corresponding to the article “Bee”. Moreover, all extracted topics in Table 8 were distinct and their relations were well represented in the individual slices of $\underset{¯}{\mathit{R}}$. In contrast, Topic 6 in Table 7 did not represent a meaningful topic, which was also indicated by the low probability scores of the ranked topic words. Although the results of the similar word evaluation were arguably better for row-stochastic tensor DEDICOM (see

Table 9. For the most significant two words per topic, the four nearest neighbors based on cosine similarity are listed. Matrix ${\mathit{A}}^{\prime }$, trained on the wikipedia data as input tensor using automatic gradient methods.

	Topic 1	Topic 2	Topic 3	Topic 4	Topic 5	Topic 6
0	hind	game	film	heard	bees	disorder
	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)
1	segments	football	starred	court	bee	collapse
	(0.995)	(1.0)	(0.968)	(0.954)	(0.999)	(1.0)
2	legs	players	role	divorce	honey	losses
	(0.994)	(0.999)	(0.954)	(0.925)	(0.99)	(1.0)
3	antennae	ball	series	sued	insects	attrition
	(0.993)	(0.999)	(0.951)	(0.907)	(0.976)	(0.999)
4	wings	team	burton	alleged	food	businesses
	(0.992)	(0.998)	(0.945)	(0.897)	(0.97)	(0.999)
0	segments	football	starred	court	bee	collapse
	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)
1	antennae	game	role	divorce	bees	disorder
	(1.0)	(1.0)	(0.993)	(0.996)	(0.999)	(1.0)
2	wings	players	series	sued	honey	losses
	(0.999)	(0.999)	(0.978)	(0.991)	(0.995)	(0.999)
3	bacteria	ball	burton	alleged	insects	pesticide
	(0.999)	(0.999)	(0.975)	(0.981)	(0.984)	(0.998)
4	legs	team	film	alcohol	food	businesses
	(0.998)	(0.999)	(0.968)	(0.981)	(0.976)	(0.998)

and

Table 10. For the most significant two words per topic, the four nearest neighbors based on cosine similarity are listed. Matrix A, trained on the wikipedia data as input tensor using multiplicative update rules.

	Topic 1	Topic 2	Topic 3	Topic 4	Topic 5	Topic 6
0	species	game	honey	allow	insects	depp
	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)
1	easier	football	boatwrights	emancipation	ultraviolet	charlie
	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)
2	tiny	players	glade	broadly	mechanics	infiltrate
	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)
3	halictidae	association	tutankhamun	disabilities	exploit	thenwife
	(0.999)	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)
4	provision	team	oracle	total	swallows	tourist
	(0.999)	(0.997)	(1.0)	(1.0)	(1.0)	(1.0)
0	eusocial	football	bee	organised	pollen	film
	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)	(1.0)
1	oligocene	players	subfamilies	comes	honeybees	starred
	(1.0)	(1.0)	(0.995)	(1.0)	(1.0)	(1.0)
2	architecture	game	internal	shows	enlarged	smoking
	(1.0)	(1.0)	(0.994)	(1.0)	(0.998)	(0.999)
3	uncommon	association	studied	attention	simple	dislocated
	(1.0)	(1.0)	(0.994)	(1.0)	(0.998)	(0.999)
4	termed	team	cladogram	deductions	drove	injuries
	(1.0)	(0.997)	(0.99)	(1.0)	(0.998)	(0.999)

) we prioritized topic extraction and relation quality. That is why in the further analysis of the Amazon review and New York Times news corpus we restricedt our evaluation to non-negative tensor DEDICOM.

As described in Section 4.1 our Amazon movie review corpus comprised human written reviews for six famous animation films. Factorizing its PPMI tensor representation with non-negative tensor DEDICOM and the number of topics set to $k=10$ revealed not only movie-specific subtopics but also general topics that spanned over several movies. For example, Topics 1, 9 and 10 in

Table 11. Top 10 representative words per dimension of the basis matrix A, trained on the Amazon review data as input tensor using multiplicative update rules.

	Topic 1 #528	Topic 2 #445	Topic 3 #1477	Topic 4 #1790	Topic 5 #1917	Topic 6 #597	Topic 7 #789	Topic 8 #670	Topic 9 #1599	Topic 10 #188
1	anna	shen	legendary	lasseter	disc	screams	code	mike	woody	po
	(4.215)	(4.21)	(1.459)	(2.887)	(1.367)	(3.779)	(3.292)	(4.325)	(6.12)	(5.737)
2	elsa	peacock	valley	director	birds	energy	email	crystal	buzz	master
	(4.087)	(2.668)	(1.448)	(2.392)	(1.343)	(3.315)	(2.781)	(4.055)	(5.484)	(5.276)
3	olaf	oldman	temple	andrew	widescreen	monstropolis	promo	billy	andy	shifu
	(2.315)	(2.627)	(1.31)	(2.158)	(1.327)	(3.13)	(2.645)	(3.911)	(4.355)	(4.707)
4	trolls	gary	kim	stanton	outtakes	world	free	goodman	toys	dragon
	(2.241)	(2.423)	(1.307)	(2.119)	(1.238)	(3.109)	(2.343)	(3.812)	(4.119)	(4.344)
5	frozen	lord	fearsome	special	extras	monsters	promotion	sully	lightyear	warrior
	(2.196)	(2.201)	(1.288)	(1.892)	(1.185)	(3.047)	(2.279)	(3.728)	(3.334)	(4.274)
6	kristoff	weapon	teacher	pete	dvd	city	promotional	wazowski	allen	tai
	(2.155)	(1.469)	(1.288)	(1.612)	(1.142)	(2.994)	(2.266)	(3.513)	(2.752)	(4.082)
7	queen	wolf	battle	ranft	included	monster	amazon	randall	tim	lung
	(2.055)	(1.405)	(1.264)	(1.564)	(1.13)	(2.978)	(2.129)	(3.404)	(2.609)	(3.993)
8	hans	inner	duk	joe	short	power	click	sulley	hanks	five
	(2.054)	(1.38)	(1.257)	(1.564)	(1.101)	(2.919)	(2.024)	(3.396)	(2.471)	(2.952)
9	sister	yeoh	train	feature	games	scare	download	buscemi	cowboy	oogway
	(1.904)	(1.359)	(1.221)	(1.555)	(1.1)	(2.614)	(1.889)	(3.266)	(2.407)	(2.918)
10	ice	michelle	warriors	ralph	tour	closet	instructions	james	space	furious
	(1.839)	(1.354)	(1.22)	(1.518)	(1.031)	(2.555)	(1.877)	(3.264)	(2.375)	(2.806)

could uniquely be related to the films “Frozen”, “Toy Story 1” and “Kung Fu Panda 1”, respectively, whereas Topic 5 constituted bonus material on a DVD which held true for all films. The latter could also be seen in Figure 8 where Topic 5 was highlighted in each movie slice (strongly in the top and lightly in the bottom row). In the same sense one could observe that Topic 3 was present in both “Kung Fu Panda 1” and “Kung Fu Panda 2”, which is reasonable considering the topic depicted the general notion of a fearsome warrior.

Figure 9 and

Table 12. Top 10 representative words per dimension of the basis matrix A, trained on the New York Times news article data as input tensor using multiplicative update rules.

	Topic 1 #454	Topic 2 #5984	Topic 3 #567	Topic 4 #562	Topic 5 #424	Topic 6 #330	Topic 7 #515	Topic 8 #297	Topic 9 #431	Topic 10 #436
1	suleimani	loans	masks	floyd	contributed	confederate	ukraine	storm	restaurants	weinstein
	(2.812)	(0.618)	(3.261)	(3.376)	(4.565)	(3.226)	(3.191)	(2.76)	(2.948)	(3.442)
2	iran	university	protective	minneapolis	reporting	statue	sondland	hurricane	bars	sexual
	(2.593)	(0.551)	(2.823)	(2.551)	(2.788)	(2.649)	(2.881)	(2.622)	(2.021)	(2.71)
3	iraq	oil	gloves	police	michael	statues	zelensky	winds	reopen	rape
	(2.453)	(0.549)	(2.516)	(2.255)	(2.707)	(2.416)	(2.133)	(1.715)	(1.684)	(2.102)
4	iranian	billion	ventilators	george	katie	monuments	ambassador	tropical	gyms	assault
	(2.408)	(0.54)	(2.22)	(2.088)	(2.324)	(1.815)	(1.976)	(1.606)	(1.654)	(1.861)
5	iraqi	loan	surgical	protests	alan	monument	ukrainian	storms	stores	jury
	(1.799)	(0.468)	(2.032)	(1.936)	(2.292)	(1.375)	(1.789)	(1.439)	(1.638)	(1.513)
6	baghdad	bonds	gowns	brutality	emily	flag	giuliani	coast	theaters	charges
	(1.604)	(0.466)	(1.965)	(1.765)	(2.165)	(1.206)	(1.755)	(1.415)	(1.627)	(1.409)
7	qassim	payments	equipment	racism	nicholas	richmond	volker	laura	salons	predatory
	(1.599)	(0.456)	(1.86)	(1.579)	(2.096)	(1.109)	(1.754)	(1.259)	(1.438)	(1.387)
8	strike	edited	supplies	knee	cochrane	symbols	investigations	isaias	closed	harvey
	(1.597)	(0.452)	(1.816)	(1.435)	(1.934)	(1.089)	(1.602)	(1.217)	(1.424)	(1.35)
9	gen	trillion	gear	killing	rappeport	remove	testified	category	shops	guilty
	(1.513)	(0.451)	(1.742)	(1.429)	(1.613)	(1.058)	(1.584)	(1.192)	(1.325)	(1.312)
10	maj	graduated	mask	officers	maggie	removal	testimony	landfall	indoor	sex
	(1.504)	(0.449)	(1.502)	(1.405)	(1.529)	(1.003)	(1.558)	(1.106)	(1.247)	(1.3)

refer to our experimental results on the dataset of New York Times news articles. We saw a diverse array of topics extracted from the text corpus, ranging from US-politics (Topics 4, 6, 7) to natural disasters (Topic 8), Hollywood sexual assault allegations (Topic 10) and the COVID epidemic both from a medical view (Topic 3) and a view on resulting restrictions to businesses (Topic 9).

The corresponding heatmap allowed us to infer when certain topics were most relevant in the last year. While the entries relating to the COVID pandemic remain light blue for the first half of the heatmap we sawa the articles picking up on the topic around March 2020, when the effects of the Coronavirus started hitting the US. Even comparatively smaller events like the conviction of Harvey Weinstein and the death of George Floyd triggering the racism debate in the US could be recognized in the heatmap, with a large deviation of Topic 10 around February 2020 and Topic 4 around June 2020.

Further empirical results on the Amazon review and New York Times news corpora, such as two-dimensional UMAP representations of the embedding matrix A and extracted topics from tensor NMF, can be found in Appendix A.3 and Appendix A.4, respectively. For example,

Table A21. Each column lists the top 10 representative words per dimension of the basis matrix A.

	Topic 1 #977	Topic 2 #360	Topic 3 #420	Topic 4 #4192	Topic 5 #489	Topic 6 #405	Topic 7 #1135	Topic 8 #748	Topic 9 #108	Topic 10 #1166
1	floyd	contributed	iran	masks	ship	syria	senator	restaurants	bloom	ukraine
	(82.861)	(137.058)	(80.007)	(40.463)	(87.178)	(94.686)	(77.285)	(93.511)	(110.77)	(79.611)
2	police	reporting	suleimani	patients	crew	syrian	storm	bars	julie	sondland
	(64.649)	(84.889)	(78.78)	(34.77)	(70.694)	(82.565)	(43.439)	(64.889)	(103.282)	(61.099)
3	protesters	michael	iranian	ventilators	aboard	kurdish	hurricane	reopen	edited	testimony
	(63.588)	(76.156)	(72.581)	(34.299)	(67.464)	(82.013)	(42.213)	(57.541)	(100.159)	(49.982)
4	minneapolis	katie	iraq	protective	passengers	turkey	iowa	stores	los	testified
	(63.216)	(63.146)	(63.27)	(33.719)	(65.895)	(80.374)	(41.985)	(55.435)	(95.747)	(49.959)
5	protests	emily	gen	loans	cruise	turkish	republican	gyms	graduated	zelensky
	(61.585)	(60.696)	(50.966)	(28.178)	(63.535)	(75.912)	(40.993)	(50.487)	(93.51)	(48.427)
6	george	alan	strike	supplies	princess	kurds	gov	theaters	angeles	ambassador
	(53.378)	(59.499)	(49.026)	(27.15)	(45.714)	(63.639)	(37.689)	(49.866)	(92.349)	(46.086)
7	brutality	nicholas	iraqi	gloves	flight	fighters	buttigieg	closed	berkeley	weinstein
	(44.051)	(55.899)	(46.027)	(26.724)	(45.375)	(62.313)	(37.1)	(46.544)	(85.653)	(45.053)
8	officers	cochrane	qassim	equipment	nasa	forces	democrat	indoor	grew	ukrainian
	(43.581)	(52.045)	(45.861)	(26.252)	(43.306)	(57.659)	(37.087)	(44.541)	(84.145)	(43.716)
9	racism	ben	maj	respiratory	navy	troops	representative	salons	today	giuliani
	(43.457)	(41.414)	(44.921)	(25.447)	(40.396)	(54.045)	(35.807)	(41.99)	(41.818)	(42.674)
10	demonstrations	maggie	baghdad	testing	astronauts	isis	bernie	shops	california	sexual
	(42.547)	(41.328)	(44.867)	(24.179)	(37.073)	(53.743)	(35.228)	(40.311)	(38.807)	(39.966)

shows that the tensor NMF factorization also extracted high quality topics but lacked the interpretable affinity tensor R which was crucial in order to properly comprehend a topic development over time.

5. Conclusions and Outlook

We propose a constrained version of the DEDICOM algorithm that is able to factorize the pointwise mutual information matrices of text documents into meaningful topic clusters all the while providing interpretable word embeddings for each vocabulary item. Our study on semi-artificial data from Wikipedia articles has shown that this method recovers the underlying structure of the text corpus and provides topics with thematic granularity, meaning the extracted latent topics are more specific than a simple clustering of articles. A comparison to related matrix factorization methods has shown that the combination of relation aware topic modeling and interpretable word embedding learning given by our algorithm is unique in its class.

Extending this algorithm to factorize three-dimensional input tensors allows for the study of changes in the relations between topics across subsets of a structured text corpus, e.g., news articles grouped by time period. Algorithmically, this can be solved via alternating gradient descent by either automatic gradient methods or by applying multiplicative update rules which decrease training time drastically and enhance training stability.

Due to memory constraints from matrix multiplications of high dimensional dense tensors our proposed approach is currently limited in vocabulary size or time dimension.

In further work we aim for developing algorithms capable of leveraging sparse matrix multiplications to avoid the above mentioned memory constraints. In addition, we plan to expand on the possibilities of constraining the factor matrices and tensors when applying a multiplicative update rule and further analyze the behavior of the factor tensors, for example by utilizing time series analysis to discover temporal relations between extracted topics and to potentially identify trends. Finally, further analysis may include additional quantitative evaluations of our proposed methods’ topic modeling performance with competing approaches.