Entry-Wise Eigenvector Analysis and Improved Rates for Topic Modeling on Short Documents

Abstract

Topic modeling is a widely utilized tool in text analysis. We investigate the optimal rate for estimating a topic model. Specifically, we consider a scenario with n documents, a vocabulary of size p, and document lengths at the order N. When $N\ge c·p$, referred to as the long-document case, the optimal rate is established in the literature at $\sqrt{p/\left(Nn\right)}$. However, when $N=o\left(p\right)$, referred to as the short-document case, the optimal rate remains unknown. In this paper, we first provide new entry-wise large-deviation bounds for the empirical singular vectors of a topic model. We then apply these bounds to improve the error rate of a spectral algorithm, Topic-SCORE. Finally, by comparing the improved error rate with the minimax lower bound, we conclude that the optimal rate is still $\sqrt{p/\left(Nn\right)}$ in the short-document case.

1. Introduction

In today’s world, an immense volume of text data is generated in scientific research and in our daily lives. This includes research publications, news articles, posts on social media, electronic health records, and many more. Among the various statistical text models, the topic model [1,2] stands out as one of the most widely used. Given a corpus consisting of n documents written on a vocabulary of p words, let $X=[X_1,X_2,\dots ,X_n]\in {\mathbb{R}}^{p\times n}$ be the word-document-count matrix, where $X_i\left(j\right)$ is the count of the jth word in the ith document, for $1\le i\le n$ and $1\le j\le p$. Let $A_1,A_2,\dots ,A_K\in {\mathbb{R}}^p$ be probability mass functions (PMFs). We call each $A_k$ a topic vector, which represents a particular distribution over words in the vocabulary. For each $1\le i\le n$, let $N_i$ denote the length of the ith document, and let $w_i\in {\mathbb{R}}^K$ be a weight vector, where $w_i\left(k\right)$ is the fractional weight this document puts on the kth topic, for $1\le k\le K$. In a topic model, the columns of X are independently generated, where the ith column satisfies:

$$X_i\sim \mathrm{Multinomial}(N_i,d_i^0),\mathrm{with}{d}_i^0=\sum _{k=1}^K{w}_i\left(k\right)A_k.$$

(1)

Here $d_i^0\in {\mathbb{R}}^p$ is the population word frequency vector for the ith document, which admits a convex combination of the K topic vectors. The $N_i$ words in this document are sampled with replacement from the vocabulary using probabilities in $d_i^0$; as a result, the word counts follow a multinomial distribution. Under this model, $\mathbb{E}\left[X\right]$ is a rank-K matrix. The statistical problem of interest is using X to estimate the two parameter matrices $A=[A_1,A_2,\dots ,A_K]$ and $W=[w_1,w_2,\dots ,w_n]$.

Since the topic model implies a low-rank structure behind the data matrix, spectral algorithms [3] have been developed for topic model estimation. Topic-SCORE [4] is the first spectral algorithm in the literature. It conducts singular value decomposition (SVD) on a properly normalized version of X, then uses the first K left singular vectors to estimate A, and finally uses $\widehat{A}$ to estimate W by weighted least-squares. Ref. [4] showed that the error rate on A is $\sqrt{p/\left(nN\right)}$ up to a logarithmic factor, where N is the order of document lengths. It matches with the minimax lower bound [4] when $N\ge c·p$ for a constant $c>0$, referred to as the long-document case. However, there are many application scenarios with $N=o\left(p\right)$, referred to as the short-document case. For example, if we consider a corpus consisting of abstracts of academic publications (e.g., see [3]), N is usually between 100 and 200, but p can be a few thousands or even larger. In this short-document case, ref. [4] observed a gap between the minimax lower bound and the error rate of Topic-SCORE. They posted the following questions: Is the optimal rate still $\sqrt{p/\left(Nn\right)}$ in the short-document case? If so, can spectral algorithms still achieve this rate?

In this paper, we give answers to these questions. We discovered that the gap between the minimax lower bound and the error rate of Topic-SCORE in the short-document case came from the unsatisfactory entry-wise large-deviation bounds for the empirical singular vectors. While the analysis in [4] is effective for long documents, there is considerable room for improvement in the short-document case. We use new analysis to obtain much better large-deviation bounds when $N=o\left(p\right)$. Our strategy includes two main components: one is an improved non-stochastic perturbation bound for SVD allowing severe heterogeneity in the population singular vectors, and the other is leveraging a decoupling inequality [5] to control the spectral norm of a random matrix with centered multinomial-distributed columns. These new ideas allow us to obtain satisfactory entry-wise large-deviation bounds for empirical singular vectors across the entire regime of $N\ge {log}^3\left(n\right)$. As a consequence, we are able to significantly improve the error rate of Topic-SCORE in the short-document case. This answers the two questions posted by [4]: The optimal rate is still $\sqrt{p/\left(Nn\right)}$ in the short-document case, and Topic-SCORE still achieves this optimal rate.

Additionally, inspired by our analysis, we have made a modification to Topic-SCORE to better incorporate document lengths. We also extend the asymptotic setting in [4] to a weak-signal regime allowing the K topic vectors to be extremely similar to each other.

1.1. Related Literature

Many topic modeling algorithms have been proposed in the literature, such as LDA [2], the separable NMF approach [6,7], the method in [8] that uses a low-rank approximation to the original data matrix, Topic-SCORE [4], and LOVE [9]. Theoretical guarantees were derived for these methods, but unfortunately, most of them had non-optimal rates even when $N\ge c·p$. Topic-SCORE and LOVE are the two that achieve the optimal rate when $N\ge c·p$. However, LOVE has no theoretical guarantee when $N=o\left(p\right)$; Topic-SCORE has a theoretical guarantee across the entire regime, but the rate obtained by [4] is non-optimal when $N=o\left(p\right)$. Therefore, our results address a critical gap in the existing literature by determining the optimal rate for the short-document case for the first time.

Entry-wise eigenvector analysis [10,11,12,13,14,15] provides large-deviation bounds or higher-order expansions for individual entries of the leading eigenvectors of a random matrix. There are two types of random matrices, i.e., the Wigner type (e.g., in network data and pairwise comparison data) and the Wishart type (e.g., in factor models and spiked covariance models [16]). The random matrices in topic models are the Wishart type, and hence, techniques for the Wigner type, such as the leave-one-out approach [15], are not a good fit. We cannot easily extend the techniques [11,14] for spiked covariance models either. One reason is that the multinomial distribution has heavier-than-Gaussian tails (especially for short documents), and using the existing techniques only give non-sharp bounds. Another reason is the severe word frequency heterogeneity [17] in natural languages, which calls for bounds whose orders are different for different entries of an eigenvector. Our analysis overcomes these challenges.

1.2. Organization and Notations

The rest of this paper is organized as follows. Section 2 presents our main results about entry-wise eigenvector analysis for topic models. Section 3 applies these results to obtain improved error bounds for the Topic-SCORE algorithm and determine the optimal rate in the short-document case. Section 4 describes the main technical components, along with a proof sketch. Section 5 concludes the paper with discussions. The proofs of all theorems are relegated to the Appendix A, Appendix B, Appendix C, Appendix D and Appendix E.

Throughout this paper, for a matrix B, let $B(i,j)$ or $B_{ij}$ represent the $(i,j)$-th entry. We denote $\parallel B\parallel$ as its operator norm and ${\parallel B\parallel }_{2\to \infty }$ as the 2-to-∞ norm, which is the maximum ${\ell }_2$ norm across all rows of B. For a vector b, $b\left(i\right)$ or $b_i$ represents the i-th component. We denote ${\parallel b\parallel }_1$ and $\parallel b\parallel$ as the ${\ell }_1$ and ${\ell }_2$ norms of b, respectively. The vector ${\mathbf{1}}_n$ stands for an all-one vector of dimension n. Unless specified otherwise, $\{e_1,e_2,\dots ,e_p\}$ denotes the standard basis of ${\mathbb{R}}^p$. Furthermore, we write $a_n\gg b_n$ or $b_n\ll a_n$ if $b_n/a_n=o\left(1\right)$ for $a_n,b_n>0$; and we denote $a_n\asymp b_n$ if $C^{-1}{b}_n<a_n<C{b}_n$ for some constant $C>1$.

2. Entry-Wise Eigenvector Analysis for Topic Models

Let $X\in {\mathbb{R}}^{p\times n}$ be the word-count matrix following the topic model in (1). We introduce the empirical frequency matrix $D=[d_1,d_2,\dots ,d_n]\in {\mathbb{R}}^{p\times n}$, defined by:

$$d_i\left(j\right)=N_i^{-1}{X}_i\left(j\right),1\le i\le n,1\le j\le p.$$

(2)

Under the model in (1), we have $\mathbb{E}\left[d_i\right]=d_i^0={\sum }_{k=1}^K{w}_i\left(k\right)A_k$. Write $D_0=[d_1^0,d_2^0,\dots ,d_n^0]\in {\mathbb{R}}^{p\times n}$. It follows that:

$$\mathbb{E}D=D_0=AW.$$

We observe that $D_0$ is a rank-K matrix; furthermore, the linear space spanned by the first K left singular vectors of $D_0$ is the same as the column space of A. Ref. [4] discovered that there is a low-dimensional simplex structure that explicitly connects the first K left singular vectors of $D_0$ with the target topic matrix A. This inspired SVD-based methods for estimating A.

However, if one directly conducts SVD on D, the empirical singular vectors can be noisy because of severe word frequency heterogeneity in natural languages [17]. In what follows, we first introduce a normalization on D in Section 2.1 to handle word frequency heterogeneity and then derive entry-wise large-deviation bounds for the empirical singular vectors in Section 2.2.

2.1. A Normalized Data Matrix

We first explain why it is inappropriate to conduct SVD on D. Let $\overline{N}=n^{-1}{\sum }_{i=1}^n{N}_i$ denote the average document length. Write $D=AW+Z$, with $Z=[z_1,z_2,\dots ,z_n]:=D-\mathbb{E}D$. The singular vectors of D are the same as the eigenvectors of $D{D}^{\prime }=AW{W}^{\prime }{A}^{\prime }+AW{Z}^{\prime }+Z{W}^{\prime }{A}^{\prime }+Z{Z}^{\prime }$. By model (1), the columns of Z are centered multinomial-distributed random vectors; moreover, using the covariance matrix formula for multinomial distributions, we have $\mathbb{E}\left[z_i{z}_i^{\prime }\right]=N_i^{-1}[\mathrm{diag}\left(d_i^0\right)-d_i^0{\left(d_i^0\right)}^{\prime }]$. It follows that:

$$\begin{array}{c}\mathbb{E}\left[D{D}^{\prime }\right]=AW{W}^{\prime }{A}^{\prime }+\sum _{i=1}^n{N}_i^{-1}\left[\mathrm{diag}\left(d_i^0\right)-d_i^0{\left(d_i^0\right)}^{\prime }\right]\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}=AW{W}^{\prime }{A}^{\prime }+\mathrm{diag}\left(\sum _{i=1}^n{N}_i^{-1}{d}_i^0\right)-A\left(\sum _{i=1}^n{N}_i^{-1}{w}_i{w}_i^{\prime }\right)A^{\prime }\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}=n·A\underset{\equiv {\Sigma }_W}{\underbrace{\left(\sum _{i=1}^n\frac{N_i-1}{n{N}_i}{w}_i{w}_i^{\prime }\right)}}{A}^{\prime }+\frac{n}{\overline{N}}·\underset{\equiv M_0}{\underbrace{\mathrm{diag}\left(\sum _{i=1}^n\frac{\overline{N}}{n{N}_i}{d}_i^0\right)}}.\hfill \end{array}$$

(3)

Here $A{\Sigma }_W{A}^{\prime }$ is a rank-K matrix whose eigen-space is the same as the column span of A. However, because of the diagonal matrix $M_0$, the eigen-space of $\mathbb{E}\left[D{D}^{\prime }\right]$ is no longer the same as the column span of A. We notice that the jth diagonal of $M_0$ captures the overall frequency of the jth word across the whole corpus. Hence, this is an issue caused by word frequency heterogeneity. The second term in (3) is larger when $\overline{N}$ is smaller. This implies that the issue becomes more severe for short documents.

To resolve this issue, we consider a normalization of D to $M_0^{-1/2}D$. It follows that:

$$\mathbb{E}\left[M_0^{-1/2}D{D}^{\prime }{M}_0^{-1/2}\right]=n·M_0^{-1/2}A{\Sigma }_W{A}^{\prime }{M}_0^{-1/2}+\frac{n}{\overline{N}}{I}_p.$$

(4)

Now, the second term is proportional to an identify matrix and no longer affects the eigen-space. Furthermore, the eigen-space of the first term is the column span of $M_0^{-1/2}A$, and hence, we can use the eigenvectors to recover $M_0^{-1/2}A$ (then A is immediately known). In practice, $M_0$ is not observed, so we replace it by its empirical version:

$$M=\mathrm{diag}\left(\sum _{i=1}^n\frac{\overline{N}}{n{N}_i}{d}_i\right).$$

(5)

We propose to normalize D to $M^{-1/2}D$ before conducting SVD. Later, the singular vectors of $M^{-1/2}D$ will be used in Topic-SCORE to estimate A (see Section 3).

This normalization is similar to the pre-SVD normalization in [4] but not exactly the same. Inspired by analyzing a special case where $N_i=N$, ref. [4] proposed to normalize D to ${\tilde{M}}^{-1/2}D$, where $\tilde{M}=\mathrm{diag}\left(n^{-1}{\sum }_{i=1}^n{d}_i\right)$. They continued using $\tilde{M}$ in general settings, but we discover here that the adjustment of $\tilde{M}$ to M is necessary when $N_i$’s are unequal.

Remark 1.

For extremely low-frequency words, the corresponding diagonal entries of M are very small. This causes an issue when we normalize D to $M^{-1/2}D$. Fortunately, such an issue disappears if we pre-process data. As a standard pre-processing step for topic modeling, we either remove those extremely low-frequency words or combine all of them into a single “meta-word”. We recommend the latter approach. In detail, let $\mathcal{L}\subset \{1,2,\dots ,p\}$ be the set of words such that $M(j,j)$ is below a proper threshold $t_n$ (e.g., $t_n$ can be $0.05$ times the average of diagonal entries of M). We then sum up all rows of D with indices in $\mathcal{L}$ to a single row. Let $D^{*}\in {\mathbb{R}}^{(p-|\mathcal{L}|+1)\times n}$ be the processed data matrix. The matrix $D^{*}$ still has a topic model structure, where each new topic vector results from a similar row combination on the corresponding original topic vector.

Remark 2.

The normalization of D to $M^{-1/2}D$ is reminiscent of the Laplacian normalization in network data analysis, but the motivation is very different. In many network models, the adjacency matrix satisfies that $B=B_0+Y$, where $B_0$ is a low-rank matrix and Y is a generalized Wigner matrix. Since $\mathbb{E}\left[Y\right]$ is a zero matrix, the eigen-space of $\mathbb{E}B$ is the same as that of $B_0$. Hence, the role of the Laplacian normalization is not correcting the eigen-space but adjusting the signal-to-noise ratio [15]. In contrast, our normalization here aims to turn $\mathbb{E}\left[Z{Z}^{\prime }\right]$ into an identity matrix (plus a small matrix that can be absorbed into the low-rank part). We need such a normalization even under moderate word frequency heterogeneity (i.e., the frequencies of all words are at the same order).

2.2. Entry-Wise Singular Analysis for $M^{-1/2}D$

For each $1\le k\le K$, let ${\widehat{\xi }}_k\in {\mathbb{R}}^p$ denote the kth left singular vector of $M^{-1/2}D$. Recall that $D_0=\mathbb{E}D$. In addition, define:

$$M_0:=\mathbb{E}M=\mathrm{diag}\left(\sum _{i=1}^n\frac{\overline{N}}{n{N}_i}{d}_i^0\right).$$

(6)

Then, $M_0^{-1/2}{D}_0$ is a population counterpart of $M^{-1/2}D$. However, the singular vectors of $M_0^{-1/2}{D}_0$ are not the population counterpart of ${\widehat{\xi }}_k$’s. In light of (4), we define:

$${\xi }_k:\mathrm{the} k\mathrm{t}\mathrm{h} \mathrm{eigenvector} \mathrm{of} M_0^{-1/2}\mathbb{E}\left[D{D}^{\prime }\right]M_0^{-1/2},1\le k\le K.$$

(7)

Write $\widehat{\Xi }:=[{\widehat{\xi }}_1,\cdots ,{\widehat{\xi }}_K]$ and $\Xi :=[{\xi }_1,\cdots ,{\xi }_K]$. We aim to derive a large-deviation bound for each individual row of $(\widehat{\Xi }-\Xi )$, subject to a column rotation of $\widehat{\Xi }$.

We need a few assumptions. Let $h_j={\sum }_{k=1}^K{A}_k\left(j\right)$ for $1\le j\le p$. Define:

$$H=\mathrm{diag}(h_1,\cdots ,h_p),{\Sigma }_A=A^{\prime }{H}^{-1}A,{\Sigma }_W=\frac{1}{n}\sum _{i=1}^n(1-N_i^{-1})w_i{w}_i^{\prime }.$$

(8)

Here ${\Sigma }_A$ and ${\Sigma }_W$ are called the topic-topic overlapping matrix and the topic-topic concurrence matrix, respectively, [4]. It is easy to see that ${\Sigma }_W$ is properly scaled. We remark that ${\Sigma }_A$ is also properly scaled, because ${\sum }_{\ell =1}^K{\Sigma }_A(k,\ell )={\sum }_{j=1}^p{\sum }_{\ell =1}^K{h}_j^{-1}{A}_k\left(j\right)A_{\ell }\left(j\right)=1$.

Assumption 1.

Let $h_{max}={max}_{1\le j\le p}{h}_j$, $h_{min}={min}_{1\le j\le p}{h}_j$ and $\overline{h}=\frac{1}{p}{\sum }_{j=1}^p{h}_j$. We assume:

$$h_{min}\ge c_1\overline{h}=c_{1}K/p,\mathit{for}a\mathit{constant}{c}_1\in (0,1).$$

Assumption 2.

For a constant $c_2\in (0,1)$ and a sequence ${\beta }_n\in (0,1)$, we assume:

$${\lambda }_{min}\left({\Sigma }_W\right)\ge c_2,{\lambda }_{min}\left({\Sigma }_A\right)\ge c_2{\beta }_n,\underset{1\le k,\ell \le K}{min}{\Sigma }_A(k,\ell )\ge c_2.$$

Assumption 1 is related to word frequency heterogeneity. Each $h_j$ captures the overall frequency of word j, and $\overline{h}=p^{-1}{\sum }_j{h}_j=p^{-1}{\sum }_k{\parallel A_k\parallel }_1=K/p$. By Remark 1, all extremely low-frequency words have been combined in pre-processing. It is reasonable to assume that $h_{min}$ is at the same order of $\overline{h}$. Meanwhile, we put no restrictions here on $h_{max}$, so that $h_j$’s can still be at different orders.

Assumption 2 is about topic weight balance and between-topic similarity. ${\Sigma }_W$ can be regarded as an affinity matrix of $w_i$’s. It is mild to assume that ${\Sigma }_W$ is well-conditioned. In a special case where $N_i=N$ and each $w_i$ is degenerate, ${\Sigma }_W$ is a diagonal matrix whose kth diagonal entry is the fraction of documents that put all weights on topic k; hence, ${\lambda }_{min}\left({\Sigma }_W\right)\ge c_2$ is interpreted as “topic weight balance”. Regarding ${\Sigma }_A$, we have seen that it is properly scaled (its maximum eigenvalue is at the constant order). When K topic vectors are exactly the same, ${\lambda }_{min}\left({\Sigma }_A\right)=0$; when the topic vectors are not the same, ${\lambda }_{min}\left({\Sigma }_A\right)\ne 0$, and it measures the signal strength. Ref. [4] assumed that ${\lambda }_{min}\left({\Sigma }_A\right)$ is bounded below by a constant, but we allow weaker signals by allowing ${\lambda }_{min}\left({\Sigma }_A\right)$ to diminish as $n\to \infty$. We also require a lower bound on ${\Sigma }_A(k,\ell )$, meaning that there should be certain overlaps between any two topics. This is reasonable as some commonly used words are not exclusive to any one topic and tend to occur frequently [4].

The last assumption is about the vocabulary size and document lengths.

Assumption 3.

There exists $N\ge 1$ and a constant $c_3\in (0,1)$ such that $c_{3}N\le N_i\le c_3^{-1}N$ for all $1\le i\le n$. In addition, for an arbitrary constant $C_0>0$:

$$min\{p,N\}\ge {log}^3\left(n\right),max\{log\left(p\right),log\left(N\right)\}\le C_{0}log\left(n\right),p{log}^2\left(n\right)\le Nn{\beta }_n^2.$$

In Assumption 3, the first two inequalities restrict that N and p are between ${log}^3\left(n\right)$ and $n^{C_0}$, for an arbitrary constant $C_0>0$. This covers a wide regime, including the scenarios of both long documents ($N\ge c·p$) and short documents ($N=o\left(p\right)$). The third inequality is needed so that the canonical angles between the empirical and population singular spaces converge to zero, which is necessary for our singular vector analysis. This condition is mild, as $Nn$ is the order of total word count in the corpus, which is often much larger than p.

With these assumptions, we now present our main theorem.

Theorem 1

(Entry-wise singular vector analysis). Fix $K\ge 2$ and positive constants $c_1,c_2,c_3$, and $C_0$. Under the model (1), suppose Assumptions 1–3 hold. For any constant $C_1>0$, there exists $C_2>0$ such that with probability $1-n^{-C_1}$, there is an orthogonal matrix $O\in {\mathbb{R}}^{K\times K}$ satisfying that simultaneously for $1\le j\le p$:

$$\parallel e_j^{\prime }(\widehat{\Xi }-\Xi O^{\prime })\parallel \le C_2\sqrt{\frac{h_{j}plog\left(n\right)}{nN{\beta }_n^2}}.$$

The constant $C_2$ only depends on $C_1$ and $(K,c_1,c_2,c_3,C_0)$.

In Theorem 1, we do not assume any gap among the K singular values of $M_0^{-1/2}{D}_0$; hence, it is only possible to recover $\Xi$ up to a column rotation O. The sin-theta theorem [18] enables us to bound $\parallel \widehat{\Xi }-\Xi O^{\prime }{\parallel }_F^2={\sum }_{j=1}^p{\parallel e_j^{\prime }(\widehat{\Xi }-\Xi O^{\prime })\parallel }^2$, but it is insufficient for analyzing spectral algorithms for topic modeling (see Section 3). We need a bound for each individual row of $(\widehat{\Xi }-\Xi O^{\prime })$, and this bound should depend on $h_j$ properly.

We compare Theorem 1 with the result in [4]. They assumed that ${\beta }_n^{-1}=O\left(1\right)$, so their results are only for the strong-signal regime. They showed that when n is sufficiently large:

$$\parallel e_j^{\prime }(\widehat{\Xi }-\Xi O^{\prime })\parallel \le C\left(1+min\left\{\frac{p}{N},\frac{p^2}{N\sqrt{N}}\right\}\right)\sqrt{\frac{h_{j}plog\left(n\right)}{nN}}.$$

(9)

When $N\ge c·p$ (long documents), it is the same bound as in Theorem 1 (with ${\beta }_n=1$). However, when $N=o\left(p\right)$ (short documents), it is strictly worse than Theorem 1. We obtain better bounds than those in [4] because of new proof ideas, especially the use of refined perturbation analysis for SVD and a decoupling technique for U-statistics (see Section 4.2).

3. Improved Rates for Topic Modeling

We apply the results in Section 2 to improve the error rates of topic modeling. Topic-SCORE [4] is a spectral algorithm for estimating the topic matrix A. It achieves the optimal rate in the long-document case ($N\ge c·p$). However, in the short-document case ($N=o\left(p\right)$), the known rate of Topic-SCORE does not match with the minimax lower bound. We address this gap by providing better error bounds for Topic-SCORE. Our results reveal the optimal rate for topic modeling in the short-document case for the first time.

3.1. The Topic-Score Algorithm

Let ${\widehat{\xi }}_1,{\widehat{\xi }}_2,\dots ,{\widehat{\xi }}_K$ be as in Section 2. Topic-SCORE first obtains word embeddings from these singular vectors. Note that $M^{-1/2}D$ is a non-negative matrix. By Perron’s theorem [19], under mild conditions, ${\widehat{\xi }}_1$ is a strictly positive vector. Define $\widehat{R}\in {\mathbb{R}}^{p\times (K-1)}$ by:

$$\widehat{R}(j,k)={\widehat{\xi }}_{k+1}\left(j\right)/{\widehat{\xi }}_1\left(j\right),1\le j\le p,1\le k\le K-1.$$

(10)

Let ${\widehat{r}}_1^{\prime },{\widehat{r}}_2^{\prime },\dots ,{\widehat{r}}_p^{\prime }$ denote the rows of $\widehat{R}$. Then, ${\widehat{r}}_j$ is a $(K-1)$-dimensional embedding of the jth word in the vocabulary. This is known as the SCORE embedding [20,21], which is now widely used in analyzing heterogeneous network and text data.

Ref. [4] discovered that there is a simplex structure associated with these word embeddings. Specifically, let ${\xi }_1,{\xi }_2,\dots ,{\xi }_K$ be the same as in (7) and define the population counterpart of $\widehat{R}$ as R, where:

$$R(j,k)={\xi }_{k+1}\left(j\right)/{\xi }_1\left(j\right),1\le j\le p,1\le k\le K-1.$$

(11)

Let $r_1^{\prime },r_2^{\prime },\dots ,r_p^{\prime }$ denote the rows of R. All these $r_j$ are contained in a simplex $\mathcal{S}\subset {\mathbb{R}}^{K-1}$ that has K vertices $v_1,v_2,\dots ,v_K$ (see Figure 1). If the jth word is an anchor word [6,22] (an anchor word of topic k satisfies that $A_k\left(j\right)\ne 0$ and $A_{\ell }\left(j\right)=0$ for all other $\ell \ne k$), then $r_j$ is located at one of the vertices. Therefore, as long as each topic has at least one anchor word, we can apply a vertex hunting [4] algorithm to recover the K vertices of $\mathcal{S}$. By definition of a simplex, each point inside $\mathcal{S}$ can be written uniquely as a convex combination of the K vertices, and the K-dimensional vector consisting of the convex combination coefficients is called the barycentric coordinate. After recovering the vertices of $\mathcal{S}$, we can easily compute the barycentric coordinate ${\pi }_j\in {\mathbb{R}}^K$ for each $r_j$. Write $\Pi ={[{\pi }_1,{\pi }_2,\dots ,{\pi }_p]}^{\prime }$. Ref. [4] showed that:

$$A_k\propto M_0^{1/2}\mathrm{diag}\left({\xi }_1\right)\Pi e_k,1\le k\le K.$$

Therefore, we can recover $A_k$ by taking the kth column of $M_0^{1/2}\mathrm{diag}\left({\xi }_1\right)\Pi$ and re-normalizing it to have a unit ${\ell }^1$-norm. This gives the main idea behind Topic-SCORE (see Figure 1).

The full algorithm is given in Algorithm 1. It requires plugging in a vertex hunting (VH) algorithm. A VH algorithm aims to estimate $v_1,v_2,\dots ,v_K$ from the noisy point cloud ${\left\{{\widehat{r}}_j\right\}}_{1\le j\le p}$. There are many existing VH algorithms (see sec 3.4 of [21]). A VH algorithm is said to be efficient if it satisfies that ${max}_{1\le k\le K}\parallel {\widehat{v}}_k-v_k\parallel \le C{max}_{1\le j\le p}\parallel {\widehat{r}}_j-r_j\parallel$ (subject to a permutation of ${\widehat{v}}_1,{\widehat{v}}_2,\dots ,{\widehat{v}}_K$). We always plug in an efficient VH algorithm, such as the successive projection algorithm [23], the pp-SPA algorithm [24], and several algorithms in sec 3.4 of [21].

Algorithm 1 Topic-SCORE

Input: D, K, and a vertex hunting (VH) algorithm. (Word embedding) Let M be as in (5). Obtain ${\widehat{\xi }}_1,{\widehat{\xi }}_2,\cdots ,{\widehat{\xi }}_K$, the first K left singular vectors of $M^{-1/2}D$. Compute $\widehat{R}$ as in (10) and write $\widehat{R}={[{\widehat{r}}_1,{\widehat{r}}_2,\cdots ,{\widehat{r}}_p]}^{\prime }$. (Vertex hunting). Apply the VH algorithm on ${\left\{{\widehat{r}}_j\right\}}_{1\le j\le p}$ to get ${\widehat{v}}_1,\cdots ,{\widehat{v}}_K$. (Topic matrix estimation) For $1\le j\le p$, solve ${\widehat{\pi }}_j^{*}$ from: $$\left(\begin{array}{ccc}1& \dots & 1\\ {\widehat{v}}_1& \dots & {\widehat{v}}_K\end{array}\right){\widehat{\pi }}_j^{*}=\left(\begin{array}{c}1\\ {\widehat{r}}_j\end{array}\right).$$ Let ${\tilde{\pi }}_j^{*}=max\{{\widehat{\pi }}_j^{*},0\}$ (the maximum is taken component-wise) and ${\widehat{\pi }}_i={\tilde{\pi }}_j^{*}/{\parallel {\tilde{\pi }}_j^{*}\parallel }_1$. Write $\widehat{\Pi }={[{\widehat{\pi }}_1,\dots ,{\widehat{\pi }}_p]}^{\prime }$. Let $\tilde{A}=M^{1/2}\mathrm{diag}\left({\widehat{\xi }}_1\right)\widehat{\Pi }$. Obtain $\widehat{A}=\tilde{A}{\left[\mathrm{diag}\left({\mathbf{1}}_p^{\prime }\tilde{A}\right)\right]}^{-1}$. Output: the estimated topic matrix $\widehat{A}$.

Additionally, after $\widehat{A}$ is obtained, ref. [4] suggested to estimate $w_1,w_2,\dots ,w_n$ as follows. We first run a weighted least-squares to obtain ${\widehat{w}}_i^{*}$:

$${\widehat{w}}_i^{*}={\mathrm{argmin}}_{w\in {\mathbb{R}}^K}{\parallel M^{-1/2}(d_i-A{w}_i)\parallel }^2,1\le i\le n.$$

(12)

Then, set all the negative entries of ${\widehat{w}}_i^{*}$ to zero and re-normalize the vector to have a unit ${\ell }^1$-norm. The resulting vector is ${\widehat{w}}_i$.

Remark 3.

In real-world applications, both n and p can be very large. However, since $\widehat{R}$ is constructed from only a few singular vectors, its rows are only in dimension $(K-1)$. It suggests that Topic-SCORE leverages a ‘low-dimensional’ simplex structure and is scalable to large datasets. When K is bounded, the complexity of Topic-SCORE is at most $O(npmin\{n,p\left\}\right)$ [4]. The real computing time was also reported in [4] for various values of ($n,p$). For example, when both n and p are a few thousands, it takes only a few seconds to run Topic-SCORE.

3.2. The Improved Rates for Estimating A and W

We provide the error rates of Topic-SCORE. First, we study the word embeddings ${\widehat{r}}_j$. By (10), ${\widehat{r}}_j$ is constructed from the jth row of $\widehat{\Xi }$. Therefore, we can apply Theorem 1 to derive a large-deviation bound for ${\widehat{r}}_j$.

Without loss of generality, we set $C_1=4$ henceforth, transforming the event probability $1-n^{-C_1}$ in Theorem 1 to $1-o\left(n^{-3}\right)$. We also use C to denote a generic constant, whose meaning may change from one occurrence to another. In all instances, C depends sorely on K and the constants $(c_1,c_2,c_3,C_0)$ in Assumptions 1–3.

Theorem 2

(Word embeddings). Under the setting of Theorem 1, with probability $1-o\left(n^{-3}\right)$, there exists an orthogonal matrix $\Omega \in {\mathbb{R}}^{(K-1)\times (K-1)}$ such that simultaneously for $1\le j\le p$:

$$\parallel {\widehat{r}}_j-\Omega r_j\parallel \le C\sqrt{\frac{plog\left(n\right)}{nN{\beta }_n^2}}.$$

Next, we study the error of $\widehat{A}$. The ${\ell }^1$-estimation error is $\mathcal{L}(\widehat{A},A):={\sum }_{k=1}^K{\parallel {\widehat{A}}_k-A_k\parallel }_1$, subject to an arbitrary column permutation of $\widehat{A}$. For ease of notation, we do not explicitly denote this permutation in theorem statements, but it is accounted for in the proofs. For each $1\le j\le p$, let ${\widehat{a}}_j^{\prime }\in {\mathbb{R}}^K$ and $a_j^{\prime }\in {\mathbb{R}}^K$ denote the jth row of $\widehat{A}$ and A, respectively. We can re-write the ${\ell }^1$-estimation error as $\mathcal{L}(\widehat{A},A)={\sum }_{j=1}^p{\parallel {\widehat{a}}_j-a_j\parallel }_1$. The next theorem provides an error bound for each individual ${\widehat{a}}_j$, and the aggregation of these bounds yields an overall bound for $\mathcal{L}(\widehat{A},A)$:

Theorem 3

(Estimation of A). Under the setting of Theorem 1, we additionally assume that each topic has at least one anchor word. With probability $1-o\left(n^{-3}\right)$, simultaneously for $1\le j\le p$:

$$\parallel {\widehat{a}}_j-a_j{\parallel }_1\le {\parallel a_j\parallel }_1·C\sqrt{\frac{plog\left(n\right)}{nN{\beta }_n^2}}.$$

Furthermore, with probability $1-o\left(n^{-3}\right)$, the ${\ell }^1$-estimation error satisfies that:

$$\mathcal{L}(\widehat{A},A)\le C\sqrt{\frac{plog\left(n\right)}{nN{\beta }_n^2}}.$$

Theorem 3 improves the result in [4] in two aspects. First, [4] assumed ${\beta }_n^{-1}=O\left(1\right)$, so their results did not allow for weak signals. Second, even when ${\beta }_n^{-1}=O\left(1\right)$, their bound is worse than ours by a factor similar to that in (9).

Finally, we have the error bound for estimating $w_i$’s using the estimator in (12).

Theorem 4

(Estimation of W). Under the setting of Theorem 3, with probability $1-o\left(n^{-3}\right)$, subject to a column permutation of $\widehat{W}$:

$$\underset{1\le i\le n}{max}{\parallel {\widehat{w}}_i-w_i\parallel }_1\le C{\beta }_n^{-1}\left(\sqrt{\frac{plog\left(n\right)}{nN{\beta }_n^2}}+C\sqrt{\frac{log\left(n\right)}{N}}\right).$$

In Theorem 4, there are two terms in the error bound of ${\widehat{w}}_i$. The first term comes from the estimation error in $\widehat{A}$, and the second term is from noise in $d_i$. In the strong-signal case of ${\beta }_n^{-1}=O\left(1\right)$, we can compare Theorem 4 with the bound for ${\widehat{w}}_i$ in [4]. The bound there also has two terms: its second term is similar to ours, but its first term is strictly worse.

3.3. Connections and Comparisons

There have been numerous results about the error rates of estimating A and W. For example, ref. [6] provided the first explicit theoretical guarantees for topic modeling, but they did not study the statistical optimality of their method. Recently, the statistical literature aimed to understand the fundamental limits of topic modeling. Assuming ${\beta }_n^{-1}=O\left(1\right)$, refs. [4,9] gave a minimax lower bound, $\sqrt{p/\left(Nn\right)}$, for the rate of estimating A, and refs. [25,26] gave a minimax lower bound, $1/\sqrt{N}$, for estimating each $w_i$.

For estimating A, when ${\beta }_n^{-1}=O\left(1\right)$, the existing theoretical results are summarized in

Table 1. A summary of the existing theoretical results for estimating A (n is the number of documents, p is the vocabulary size, N is the order of document lengths, and $h_{max}$ and $h_{min}$ are the same as in (8)). Cases 1–3 refer to $N\ge p^{4/3}$, $p\le N<p^{4/3}$, and $N<p$, respectively. For Cases 2–3, the sub-cases ‘a’ and ‘b’ correspond to $n\ge max\{N{p}^2,p^3,N^2{p}^5\}$ and $n<max\{N{p}^2,p^3,N^2{p}^5\}$, respectively. We have translated the results in each paper to the bounds on $\mathcal{L}(\widehat{A},A)$, with any logarithmic factor omitted.

	Case 1	Case 2a	Case 2b	Case 3a	Case 3b
Ke & Wang [4]	$\sqrt{\frac{p}{Nn}}$	$\sqrt{\frac{p}{Nn}}$	$\frac{p^2}{N\sqrt{N}}\sqrt{\frac{p}{Nn}}$	$\frac{p}{N}\sqrt{\frac{p}{Nn}}$	$\frac{p^2}{N\sqrt{N}}\sqrt{\frac{p}{Nn}}$
Arora et al. [6]	$\frac{p^4}{\sqrt{Nn}}$	$\frac{p^4}{\sqrt{Nn}}$	$\frac{p^4}{\sqrt{Nn}}$	$\frac{p^4}{\sqrt{Nn}}$	$\frac{p^4}{\sqrt{Nn}}$
Bing et al. [9]	$\sqrt{\frac{p}{Nn}}·\frac{h_{max}}{h_{min}}$	$\sqrt{\frac{p}{Nn}}·\frac{h_{max}}{h_{min}}$	$\sqrt{\frac{p}{Nn}}·\frac{h_{max}}{h_{min}}$	NA	NA
Bansal et al. [8]	$N\sqrt{\frac{p}{n}}$	$N\sqrt{\frac{p}{n}}$	$N\sqrt{\frac{p}{n}}$	$N\sqrt{\frac{p}{n}}$	$N\sqrt{\frac{p}{n}}$
Our results	$\sqrt{\frac{p}{Nn}}$	$\sqrt{\frac{p}{Nn}}$	$\sqrt{\frac{p}{Nn}}$	$\sqrt{\frac{p}{Nn}}$	$\sqrt{\frac{p}{Nn}}$

. Ours is the only one that matches the minimax lower bound across the entire regime. In the long-document case ($N\ge c·p$, Cases 1–2 in Table 1), the error rates in [4,9] together have matched the lower bound, concluding that $\sqrt{p/\left(Nn\right)}$ is indeed the optimal rate. However, in the short-document case ($N=o\left(p\right)$, Case 3 in Table 1), there was a gap between the lower bound and the existing error rates. Our result addresses the gap and concludes that $\sqrt{p/\left(Nn\right)}$ is still the optimal rate. When ${\beta }_n=o\left(1\right)$, the error rates of estimating A were rarely studied. We conjecture that $\sqrt{p/\left(Nn{\beta }_n^2\right)}$ is the optimal rate, and the Topic-SCORE algorithm is still rate-optimal.

We emphasize that our rate is not affected by severe word frequency heterogeneity. As long as $h_{min}/\overline{h}$ is lower bounded by a constant (see Assumption 1 and explanations therein), our rate is always the same, regardless of the magnitude of $h_{max}$. In contrast, the error rate in [9] is sensitive to word frequency heterogeneity, with an extra factor of $h_{max}/h_{min}$ that can be as large as p. There are two reasons that enable us to achieve a flat rate even under severe word frequency heterogeneity: one is the proper normalization of data matrix, as described in Section 2.1, and the other is the careful analysis of empirical singular vectors (see Section 4).

For estimating W, when ${\beta }_n^{-1}=O\left(1\right)$, our error rate in Theorem 4 matches the minimax lower bound if $n\ge plog\left(n\right)$. Our approach to estimating W involves first obtaining $\widehat{A}$ and then regressing $d_i$ on $\widehat{A}$ to derive ${\widehat{w}}_i$. The condition $n\ge plog\left(n\right)$ ensures that the estimation error in $\widehat{A}$ does not dominate the overall error. This condition is often met in scenarios where a large number of documents can be collected, but the vocabulary size remains relatively stable. However, if $n<plog\left(n\right)$, a different approach is necessary, requiring the estimation of W first. This involves using the right singular vectors of $M^{-1/2}D$. While our analysis has primarily focused on the left singular vectors, it can be extended to study the right singular vectors as well.

4. Proof Ideas

Our main result is Theorem 1, which provides entry-wise large-deviation bounds for singular vectors of $M^{-1/2}D$. Given this theorem, the proofs of Theorems 2–4 are similar to those in [4] and thus relegated to the appendix. In this section, we focus on discussing the proof techniques of Theorem 1.

4.1. Why the Leave-One-Out Technique Fails

Leave-one-out [13,15] is a common technique in entry-wise eigenvector analysis for a Wigner-type random matrix $B=B_0+Y\in {\mathbb{R}}^{m\times m}$, where $B_0$ is a symmetric non-stochastic low-rank matrix and Y is a symmetric random matrix whose upper triangle consists of independent mean-zero variables. One example of such matrices is the adjacency matrix of a random graph generated from the block-model family [20].

However, our target here is the singular vectors of $M^{-1/2}D$, which are the eigenvectors of $B:=M^{-1/2}D{D}^{\prime }{M}^{-1/2}$. This is a Wishart-type random matrix, whose upper triangular entries are not independent. We may also construct a symmetric matrix:

$$\mathcal{G}:=\left(\begin{array}{cc}0& M^{-1/2}D\\ D^{\prime }{M}^{-1/2}& 0\end{array}\right)\in {\mathbb{R}}^{(p+n)\times (p+n)}.$$

The eigenvectors of $\mathcal{G}$ take the form ${\widehat{u}}_k={({\widehat{\xi }}_k^{\prime },{\widehat{\eta }}_k^{\prime })}^{\prime }$, $1\le k\le K$, where ${\widehat{\xi }}_k\in {\mathbb{R}}^p$ and ${\widehat{\eta }}_k\in {\mathbb{R}}^n$ are the kth left and right singular vectors of $M^{-1/2}D$, respectively. Unfortunately, the upper triangle of $\mathcal{G}$ still contains dependent entries. Some dependence is from the normalization matrix M. It may be addressed by using the techniques developed by [15] in studying graph Laplacian matrices. A more severe issue is the dependence among entries in D. According to basic properties of multinomial distributions, D only has column independence but no row independence. As a result, even after we replace M by $M_0$, the jth row and column of $\mathcal{G}$ are still dependent of the remaining ones, for each $1\le j\le p$. In conclusion, we cannot apply the leave-one-out technique on either B or $\mathcal{G}$.

4.2. The Proof Structure in [4] and Why It Is Not Sharp for Short Documents

Our entry-wise eigenvector analysis primarily follows the proof structure in [4]. Recall that ${\widehat{\xi }}_k\in {\mathbb{R}}^p$ is the kth left singular vector of $M^{-1/2}D$. Define:

$$G:=M^{-1/2}D{D}^{\prime }{M}^{-1/2}-\frac{n}{\overline{N}}{I}_p,G_0:=n·M_0^{-1/2}A{\Sigma }_W{A}^{\prime }{M}_0^{-1/2}.$$

(13)

Since the identify matrix in G does not affect the eigenvectors, ${\widehat{\xi }}_k$ is the kth eigenvector of G. Additionally, it follows from (7) and (4) that ${\xi }_k$ is the kth eigenvector of $G_0$. By (4):

$$G-G_0=M^{-1/2}D{D}^{\prime }{M}^{-1/2}-M_0^{-1/2}\mathbb{E}\left[D{D}^{\prime }\right]M_0^{-1/2}.$$

(14)

The entry-wise eigenvector analysis in [4] has two steps. Step 1: Non-stochastic perturbation analysis. In this step, no distributional assumptions are made on G. The analysis solely focuses on connecting the perturbation from $\Xi$ to $\widehat{\Xi }$ with the perturbation from $G_0$ to G. They showed in Lemma F.1 [4]:

$$\parallel e_j^{\prime }(\widehat{\Xi }-\Xi O^{\prime })\parallel \le C\parallel G_0{\parallel }^{-1}\left(\parallel e_j^{\prime }\Xi \parallel \parallel G-G_0\parallel +\sqrt{K}\parallel e_j^{\prime }(G-G_0)\parallel \right).$$

(15)

Step 2: Large-deviation analysis of $G-G_0$. In this step, ref. [4] derived the large-deviation bounds for $\parallel G-G_0\parallel$ and $\parallel e_j^{\prime }(G-G_0)\parallel$ under the multinomial model (1). For example, they showed in Lemma F.5 [4] that when n is properly large, with high probability:

$$\parallel G-G_0\parallel \le C\left(1+N^{-1}\sqrt{p}\right)\sqrt{\frac{nplog\left(n\right)}{N}}.$$

(16)

However, when $N=o\left(p\right)$ (short documents), neither step is sharp. In (15), the second term $\parallel e_j^{\prime }(G-G_0)\parallel$ was introduced as an upper bound for $\parallel e_j^{\prime }(G-G_0)\widehat{\Xi }\parallel$, but this bound is too crude. In Section 4.3, we will conduct careful analysis of $\parallel e_j^{\prime }(G-G_0)\widehat{\Xi }\parallel$ and introduce a new perturbation bound which significantly improves (15). In (16), the spectral norm is controlled via an $\epsilon$-net argument [27], which reduces the analysis to studying a quadratic form of Z; ref. [4] analyzed this quadratic form by applying martingale Bernstein inequality. Unfortunately, in the short-document case, it is hard to control the conditional variance process of the underlying martingale. In Section 4.4, we address it by leveraging the matrix Bernstein inequality [28] and the decoupling inequality [5,29] for U-statistics.

4.3. Non-Stochastic Perturbation Analysis

In this subsection, we abuse notations to use G and $G_0$ to denote two arbitrary $p\times p$ symmetric matrices, with $\mathrm{rank}\left(G_0\right)=K$. For $1\le k\le K$, let ${\widehat{\lambda }}_k$ and ${\lambda }_k$ be the kth largest eigenvalue (in magnitude) of G and $G_0$, respectively, and let ${\widehat{\xi }}_k\in {\mathbb{R}}^p$ and ${\xi }_k\in {\mathbb{R}}^p$ be the associated eigenvectors. Write $\widehat{\Lambda }=\mathrm{diag}({\widehat{\lambda }}_1,{\widehat{\lambda }}_2,\dots ,{\widehat{\lambda }}_K)$, $\widehat{\Xi }=[{\widehat{\xi }}_1,{\widehat{\xi }}_2,\dots ,{\widehat{\xi }}_K]$, and define $\Lambda$ and $\Xi$ similarly. Let $U\in {\mathbb{R}}^{K\times K}$ and $V\in {\mathbb{R}}^{K\times K}$ be such that its columns contain the left and right singular vectors of ${\widehat{\Xi }}^{\prime }\Xi$, respectively. Define $\mathrm{sgn}({\widehat{\Xi }}^{\prime }\Xi )=U^{\prime }V$. For any matrix B and $q>0$, let ${\parallel B\parallel }_{q\to \infty }={max}_i{\parallel e_i^{\prime }B\parallel }_q$.

Lemma 1.

Suppose $\parallel G-G_0\parallel \le (1-c_0)|{\widehat{\lambda }}_K|$, for some $c_0\in (0,1)$. Consider an arbitrary $p\times p$ diagonal matrix $\Gamma =\mathrm{diag}({\gamma }_1,{\gamma }_2,\dots ,{\gamma }_p)$, where:

$${\gamma }_j>0 is an upper bound for \parallel e_j^{\prime }\Xi \parallel \parallel G-G_0\parallel +\parallel e_j^{\prime }(G-G_0)\Xi \parallel .$$

If $\parallel {\Gamma }^{-1}(G-G_0){\Gamma \parallel }_{1\to \infty }\le (1-c_0)|{\widehat{\lambda }}_K|$, then for the orthogonal matrix $O=\mathrm{sgn}({\widehat{\Xi }}^{\prime }\Xi )$, it holds simultaneously for $1\le j\le p$ that:

$$\parallel e_j^{\prime }(\widehat{\Xi }-\Xi O^{\prime })\parallel \le c_0^{-1}{|{\widehat{\lambda }}_K|}^{-1}{\gamma }_j.$$

Since ${\gamma }_j$ is an upper bound for $\parallel e_j^{\prime }\Xi \parallel \parallel G-G_0\parallel +\parallel e_j^{\prime }(G-G_0)\Xi \parallel$, we can interpret the result in Lemma 1 as:

$$\parallel e_j^{\prime }(\widehat{\Xi }-\Xi O^{\prime })\parallel \le C|{\widehat{\lambda }}_K{|}^{-1}\left(\parallel e_j^{\prime }\Xi \parallel \parallel G-G_0\parallel +\parallel e_j^{\prime }(G-G_0)\Xi \parallel \right).$$

(17)

Comparing (17) with (15), the second term has been reduced. Since $\Xi$ projects the vector $e_j^{\prime }(G-G_0)$ into a much lower dimension, we expect that $\parallel e_j^{\prime }(G-G_0)\Xi \parallel \ll \parallel e_j^{\prime }(G-G_0)\parallel$ in many random models for G. In particular, this is true for the G and $G_0$ defined in (13). Hence, there is a significant improvement over the analysis in [4].

4.4. Large-Deviation Analysis of $(G-G_0)$

In this subsection, we focus on the specific G and $G_0$ as defined in (13). The crux of proving Theorem 1 lies in determining the upper bound ${\gamma }_j$ as defined in Lemma 1. This is accomplished through the following lemma.

Lemma 2.

Under the settings of Theorem 1, let G and $G_0$ be as in (13). For any constant $C_1>0$, there exists $C_3>0$ such that with probability $1-n^{-C_1}$, simultaneously for $1\le j\le p$:

$$\parallel G-G_0\parallel \le C_3\sqrt{\frac{pnlog\left(n\right)}{N}},\parallel e_j^{\prime }(G-G_0)\Xi \parallel \le C_3\sqrt{\frac{h_{j}nplog\left(n\right)}{N}}.$$

The constant $C_3$ only depends on $C_1$ and $(K,c_1,c_2,c_3,C_0)$.

We compare the bound for $\parallel G-G_0\parallel$ in Lemma 2 with the one in [4] as summarized in (16). There is a significant improvement when $N\le p^2$. This improvement primarily stems from the application of a decoupling inequality for U-statistics, as elaborated below.

We outline the proof of the bound for $\parallel G-G_0\parallel$. Let $Z=D-\mathbb{E}\left[D\right]=[z_1,z_2,\dots ,z_n]$. From (A24) and (A25) in Appendix A, $G-G_0$ decomposes into the sum of four matrices, where it is most subtle to bound the spectral norm of the fourth matrix:

$$E_4:=M_0^{-1/2}(Z{Z}^{\prime }-\mathbb{E}\left[Z{Z}^{\prime }\right])M_0^{-1/2}.$$

Define $X_i=\left(M_0^{-1/2}{z}_i\right){\left(M_0^{-1/2}{z}_i\right)}^{\prime }-\mathbb{E}\left[{\left(M_0^{-1/2}{z}_i\right)\left(M_0^{-1/2}{z}_i\right)}^{\prime }\right]$. It is seen that $E_4={\sum }_{i=1}^n{X}_i$, which is a sum of n independent matrices. We apply the matrix Bernstein inequality [28] (Theorem A1) to obtain that if there exist $b>0$ and ${\sigma }^2>0$ such that $\parallel X_i\parallel \le b$ almost surely for all i and $\parallel {\sum }_{i=1}^n\mathbb{E}{X}_i^2\parallel \le {\sigma }^2$, then for every $t>0$,

$$\mathbb{P}\left(\parallel \sum _{i=1}^n{X}_i\parallel \ge t\right)\le 2pexp\left(-\frac{t^2/2}{{\sigma }^2+bt/3}\right).$$

Determination of b and ${\sigma }^2$ requires upper bounds for $\parallel X_i\parallel$ and $\parallel \mathbb{E}{X}_i^2\parallel$. Since each $X_i$ is equal to a rank-1 matrix minus its expectation, it reduces to deriving large-deviation bounds for $\parallel M_0^{-1/2}{z}_i{\parallel }^2$. Note that each $z_i$ can be equivalently represented by $z_i=N_i^{-1}{\sum }_{m=1}^N(T_{im}-\mathbb{E}{T}_{im})$, where ${\left\{T_{im}\right\}}_{m=1}^{N_i}$ are i.i.d. Multinomial $(1,d_i^0)$. It yields that $\parallel M_0^{-1/2}{z}_i{\parallel }^2={\mathcal{I}}_1+{\mathcal{I}}_2$, where ${\mathcal{I}}_2$ is a term that can be controlled using standard large-deviation inequalities, and:

$${\mathcal{I}}_1:=N_i^{-2}\sum _{1\le m_1\ne m_2\le N_i}(T_{i{m}_1}-\mathbb{E}{T}_{i{m}_1})M_0^{-1}(T_{i{m}_2}-\mathbb{E}{T}_{i{m}_2}).$$

The remaining question is how to bound $|{\mathcal{I}}_1|$. We notice that ${\mathcal{I}}_1$ is a U-statistic with degree 2. The decoupling inequality [5,29] is a useful tool for studying U-statistics.

Theorem 5

(A special decoupling inequality [29]). Let ${\left\{X_m\right\}}_{m=1}^N$ be a sequence of i.i.d. random vectors in ${\mathbb{R}}^d$, and let ${\left\{{\tilde{X}}_m\right\}}_{m=1}^N$ be an independent copy of ${\left\{X_m\right\}}_{m=1}^N$. Suppose that $h:{\mathbb{R}}^{2d}\to \mathbb{R}$ is a measurable function. Then, there exists a constant $C_4>0$ independent of $n,m,d$ such that for all $t>0$:

$$\begin{array}{c}\hfill \mathbb{P}\left(\left|\sum _{m\ne m_1}h(X_m,X_{m_1})\right|\ge t\right)\le C_4\mathbb{P}\left(C_4\left|\sum _{m\ne m_1}h(X_m,{\tilde{X}}_{m_1})\right|\ge t\right).\end{array}$$

Let ${\left\{{\tilde{T}}_{im}\right\}}_{m=1}^{N_i}$ be an independent copy of ${\left\{T_{im}\right\}}_{m=1}^{N_i}$. By Theorem 5, the large-deviation bound of ${\mathcal{I}}_1$ can be inferred from the large-deviation bound of:

$${\tilde{\mathcal{I}}}_1:=N_i^{-2}\sum _{1\le m_1\ne m_2\le N_i}{(T_{i{m}_1}-\mathbb{E}{T}_{i{m}_1})}^{\prime }{M}_0^{-1}({\tilde{T}}_{i{m}_2}-\mathbb{E}{\tilde{T}}_{i{m}_2}).$$

Using $h(T_{i{m}_1},{\tilde{T}}_{i{m}_2})$ to denote the summand in the above sum, we have a decomposition: ${\tilde{\mathcal{I}}}_1=N_i^{-2}{\sum }_{m_1,m_2}h(T_{i{m}_1},{\tilde{T}}_{i{m}_2})-N_i^{-2}{\sum }_{m}h(T_{im},{\tilde{T}}_{im})$. The second term is a sum of independent variables and can be controlled by standard large-deviation inequalities. Hence, the analysis of ${\tilde{\mathcal{I}}}_1$ reduces to the analysis of ${\tilde{\mathcal{I}}}_1^{*}:=N_i^{-2}{\sum }_{m_1,m_2}h(T_{i{m}_1},{\tilde{T}}_{i{m}_2})$. We re-write ${\tilde{\mathcal{I}}}_1^{*}$ as:

$${\tilde{\mathcal{I}}}_1^{*}=N_i^{-2}{y}^{\prime }\tilde{y},\mathrm{with}y:=\sum _{m=1}^{N_i}{M}_0^{-1/2}(T_{im}-\mathbb{E}{T}_{im}),\tilde{y}:=\sum _{m=1}^{N_i}{M}_0^{-1/2}({\tilde{T}}_{im}-\mathbb{E}{\tilde{T}}_{im}).$$

Since $\tilde{y}$ is independent of y, we apply large-deviation inequalities twice. First, conditional on $\tilde{y}$, ${\tilde{\mathcal{I}}}_1^{*}$ is a sum of $N_i$ independent variables (randomness comes from $T_{im}$’s). We apply the Bernstein inequality to get a large-deviation bound for ${\tilde{\mathcal{I}}}_1^{*}$, which depends on a quantity ${\sigma }^2\left(\tilde{y}\right)$. Next, since ${\sigma }^2\left(\tilde{y}\right)$ can also be written as a sum of $N_i$ independent variables (randomness comes from ${\tilde{T}}_{im}$’s), we apply the Bernstein inequality again to obtain a large-deviation bound for ${\sigma }^2\left(\tilde{y}\right)$. Combining two steps gives the large-deviation bound for ${\tilde{\mathcal{I}}}_1^{*}$.

Remark 4.

The decoupling inequality is employed multiple times to study other U-statistics-type quantities arising in our proof. For example, recall that $(G-G_0)$ decomposes into the sum of four matrices, and we have only discussed how to bound $\parallel E_4\parallel$. In the analysis of $\parallel E_2\parallel$ and $\parallel E_3\parallel$, we need to bound other quadratic terms involving a sum over $(i,m)$, with $1\le i\le n$ and $1\le m\le N_i$. In that case, we need a more general decoupling inequality. We refer readers to Theorem A3 in Appendix A for more details.

Remark 5.

The analysis in [4] uses an ϵ-net argument [27] and the martingale Bernstein inequality [30] to study $\parallel E_4\parallel$. In our analysis, we use the matrix Bernstein inequality [28], instead of the ϵ-net argument. The matrix Bernstein inequality enables us to tackle each quadratic term related to each i separately instead of handling complicated quadratic terms involving summation over i and m simultaneously. Additionally, we adopt the decoupling inequality for U-statistics [5,29], instead of the martingale Bernstein inequality, to study all the quadratic terms arising in our analysis. The decoupling inequality converts the tail anaysis of quadratic terms into tail analysis of (conditionally) independent sums. It provides sharper bounds when the variables have heavy tails (which is the case for the word counts in a topic model, especially when documents are short).

4.5. Proof Sketch of Theorem 1

We combine the non-stochastic perturbation result in Lemma 1 and the large-deviation bounds in Lemma 2 to prove Theorem 1. By Lemma A2, $|{\lambda }_K|\ge C^{-1}n{\beta }_n$. It follows from Weyl’s inequality, the first claim in Lemma 2, and the assumption of $p{log}^2\left(n\right)\le Nn{\beta }_n^2$ that with probability $1-n^{-C_1}$:

$$|{\widehat{\lambda }}_K|\ge |{\lambda }_k|·\left[1-O\left({[log\left(n\right)]}^{-1/2}\right)\right]\ge C^{-1}n{\beta }_n.$$

In addition, it can be shown (see Lemma A2) that $\parallel e_j^{\prime }\Xi \parallel \le C{h}_j^{1/2}$. Combining this with the two claims in Lemma 2 gives that with probability $1-n^{-C_1}$:

$$\parallel e_j^{\prime }\Xi \parallel \parallel G-G_0\parallel +\parallel e_j^{\prime }(G-G_0)\Xi \parallel \le C\sqrt{\frac{h_{j}nplog\left(n\right)}{N}}:={\gamma }_j.$$

We hope to apply Lemma 1. This requires obtaining a bound for $\parallel {\Gamma }^{-1}(G-G_0){\Gamma \parallel }_{1\to \infty }$. Since $\Gamma \propto H^{1/2}$, it suffices to study $\parallel H^{-1/2}(G-G_0)H^{1/2}{\parallel }_{1\to \infty }$. Similar to the analysis of $\parallel e_j^{\prime }(G-G_0)\Xi \parallel$, we can show (see the proofs of Lemmas A5 and A6, such as (A58)) that $\parallel e_j^{\prime }(G-G_0)H^{1/2}{\parallel }_1\le C{N}^{-1/2}{[h_{j}nplog\left(n\right)]}^{1/2}\le C\sqrt{h_j/log\left(n\right)}·n{\beta }_n$, where the last inequality is because of $p{log}^2\left(n\right)\le Nn$. We immediately have:

$$\parallel H^{-1/2}(G-G_0)H^{1/2}{\parallel }_{1\to \infty }=\underset{j}{max}\left\{h_j^{-1/2}{\parallel e_j^{\prime }(G-G_0)H^{-1/2}\parallel }_1\right\}\le \frac{Cn{\beta }_n}{\sqrt{log\left(n\right)}}\le \frac{|{\widehat{\lambda }}_K|}{2}.$$

We then apply Lemma 1 to get $\parallel e_j^{\prime }(\widehat{\Xi }-\Xi O^{\prime })\parallel \le C|{\widehat{\lambda }}_K{|}^{-1}{\gamma }_j\le C{\left(n{\beta }_n\right)}^{-1}{\gamma }_j$. The claim of Theorem 1 follows immediately by plugging in the value of ${\gamma }_j$ as given above.

5. Summary and Discussion

The topic model imposes a “low-rank plus noise” structure on the data matrix. However, the noise is not simply additive; rather, it consists of centered multinomial random vectors. The eigenvector analysis in a topic model is more complex than standard eigenvector analysis for random matrices. Firstly, the entries of the data matrix are weakly dependent, making techniques such as leave-one-out inapplicable. Secondly, due to the significant word frequency heterogeneity in natural languages, entry-wise eigenvector analysis becomes much more nuanced, as different entries of the same eigenvector have significantly different bounds. Additionally, the data exhibit Bernstein-type tails, precluding the use of random matrix theory tools that assume sub-exponential entries. While we build on the analysis in [4], we address these challenges with new proof ideas. Our results provide the most precise eigenvector analysis and rate-optimality results for topic modeling, to the best of our knowledge.

A related but more ambitious goal is obtaining higher-order expansions of the empirical singular vectors. Since the random matrix under study in the topic model is the Wishart type, we can possibly borrow techniques in [31] to study the joint distribution of empirical singular values and singular vectors. In this paper, we assume the number of topics, K, is finite, but our analysis can be easily extended to the scenario of a growing K (e.g., $K=O(log(n\left)\right)$). We assume $min\{p,N\}\ge {log}^3\left(n\right)$. When $p<{log}^3\left(n\right)$, it becomes a low-dimensional eigenvector analysis problem, which is easy to tackle. When $N<{log}^3\left(n\right)$, it is the extremely short documents case (i.e., each document has only a finite length, say, fewer than 20, as in documents such as Tweets). We leave it to future work.