Understanding Hierarchical Processes

Abstract

Hierarchical stochastic processes, such as the hierarchical Dirichlet process, hold an important position as a modelling tool in statistical machine learning, and are even used in deep neural networks. They allow, for instance, networks of probability vectors to be used in general statistical modelling, intrinsically supporting information sharing through the network. This paper presents a general theory of hierarchical stochastic processes and illustrates its use on the gamma process and the generalised gamma process. In general, most of the convenient properties of hierarchical Dirichlet processes extend to the broader family. The main construction for this corresponds to estimating the moments of an infinitely divisible distribution based on its cumulants. Various equivalences and relationships can then be applied to networks of hierarchical processes. Examples given demonstrate the duplication in non-parametric research, and presents plots of the Pitman–Yor distribution.

1. Introduction

The hierarchical Pitman–Yor process (HPYP) was first presented as a solution to n-gram language models [1] where it mimics the behavior of the Kneser–Ney algorithm [2]. It is an extension of the hierarchical Dirichlet process (HDP) [3]. The HPYP has since been used in a wide variety of ways, including for previously state-of-the-art and competitive algorithms for topic models [4] and text compression [5]. The HDP has been used for previously state-of-the-art and competitive algorithms for tweet clustering [6] and document segmentation [7]. Many more novel and creative uses of these processes exist, for instance, hierarchical topic models [8]. More general reviews are given by Teh and Jordan [9] and Jordan [10]. The gamma process can also be used hierarchically [11] and provides an alternative scheme for handling the HDP. The notion of hierarchical models fits in well with the computational approach to statistical modelling adopted in the machine learning community.

However, what exactly is the HPYP? A key concept for understanding the HDP and the HPYP is the notion of a discrete base probability measure. The base measure is a source measure for sampling points of the HDP or HPYP. These are discrete just when they have a countable number of possible points (the set on which the measure is based is countable). When finite, the base probability measure is just a probability distribution, usually represented as a vector. However, in non-parametric modelling, we seek to model structured objects for which the dimension may be unknown ahead of time: the number of clusters for points, the depth of a tree, the number of atoms in a molecule, the number of words in a sentence. Allowing the base measure to be countably infinite is a useful abstraction in this situation. Moreover, being able to generate an infinite discrete base probability measure provides us with the ability to model prior distributions for our structured objects without fixing dimensions ahead of time. The above models for text and clustering give examples.

It is known that the hierarchical Dirichlet process, when applied to a finite discrete base distribution, is just a Dirichlet distribution. Indeed, this property is the axiomatic definition of the process [12]. So, applications of and inference with the HDP are really just using hierarchical Dirichlet distributions, requiring no non-parametric theory to describe, although algorithms may be using non-parametric methods.

So, there is a clear concept of what the HDP model is. What is the corresponding result for the hierarchical Pitman–Yor process? For all the algorithms using the HPYP, it would be nice to know what their actual model is! Teh first referred to the hierarchical version of the PYP as the Pitman–Yor distribution [in talks accompanying] [1], saying it has “no known analytical form”. Moreover, is there a more general theory of hierarchical processes, and why does this case (the HDP) come out so neatly? These questions for hierarchical processes have been addressed in recent theory [13,14,15].

Note the Bayesian theory of non-hierarchical processes is extensive. A comprehensive analysis of different processes is developed by James [16], in the more general context of the generalised Indian buffet process [17]. The general posterior analysis of their normalised versions, including the DP, is developed by James et al. [18]. A useful review of theory and a slice sampler for the case of the normalised generalised gamma is given in Lomeli et al. [19]. A study of some of the processes considered here can also be found in Zhou and Carin [11], focusing on gamma processes and their relationships.

However, these treatments are grounded in extensive probability theory and assume the reader is already familiar with Poisson point processes, Lévy processes, subordinators and other advanced areas [20,21]. Some of these details are not strictly necessary for the understanding of the basic ideas. This paper presents the relevant background theory in a self-contained way to develop models for hierarchical processes generally based on the theory of subordinators and completely random measures [20,21]. The theory for the most part reinterprets results from the Bayesian non-parametric and statistical communities [18,22,23], though some related ideas can also be found in machine learning [11]. However, the answers to the questions about the nature of the HPYP and general application to hierarchical processes, networks of hierarchical processes and generalised Chinese restaurants are not well-known outside the Bayesian non-parametric community, so we present them here in a unified manner.

2. Background Theory

A formal theory of Poisson point processes (PPP), Lévy processes and completely random measures (CRMs) with treatment of measure theory is needed to rigorously cover this area [20,21]. Here, an informal summary is given, though trying to maintain a degree of precision, for instance keeping adequate rigor in the statement of results.

2.1. Completely Random Measures

A CRM is a discrete measure $\mu (\mathrm{d}x)$ on a space $\mathcal{X}$ constructed as

$$\mu (x)=C_0+\sum _{i=1}^{\infty }{\lambda }_i{\delta }_{x_i}(x)$$

(1)

where the $x_i\in \mathcal{X}$ are called atoms and are assumed distinct, the ${\lambda }_i\in {\mathcal{R}}^{+}$ called jumps, and the background constant $C_0$ is zero in our use. This means that $\mu (x_i)={\lambda }_i$, evaluated at atoms, and $\mu (x_i)=0$ otherwise. The $({\lambda }_i,x_i)$ are mutually independent random variables, and a finite number of the $x_i$ can also be fixed. These conditions ensure the measure is completely random, that is for $A,B\subset \mathcal{X}$, if $A\cap B=\varnothing$ then $\mu (A)\perp \perp \mu (B)$.

Moreover, suppose the class of CRMs where $C_0=0$ in Equation (1) can be normalised, so $\mu (\mathcal{X})={\sum }_{i=1}^{\infty }{\lambda }_i<\infty$. This yields discrete probability distributions on $\mathcal{X}$ represented as $\mu (x)/\mu (\mathcal{X})$. These are referred to as normalised random measures with independent increments (NRMIs) [18], a concept developed by Kingman [24], and are a general class of discrete probability distributions.

2.2. Poisson Point Process

A Poisson point process (PPP) is a stochastic process whose samples represent sets of independent events on a measurable space $\mathcal{X}$. For a sample, the count of events in $A\subseteq \mathcal{X}$ is denoted $N(A)\in \mathcal{N}$. Events are considered to be a countable subset of $\mathcal{X}$, only significant if $\mathcal{X}$ is not countable, for instance the real line. The PPP has complete independence, so for $A,B\subset \mathcal{X}$, if $A\cap B=\varnothing$ then $N(A)\perp \perp N(B)$ and $N(A\cup B)=N(A)+N(B)$. The sample is specified by a rate $\rho (\mathrm{d}x)$ which is any measure on $\mathcal{X}$. In PPP theory, the rate is referred to as a Lévy measure. The PPP has the defining property that $N(A)\sim \mathrm{Poisson}(\rho (A))$, and samples can be generated from this by working with an ever finer partition of the space $\mathcal{X}$.

A special class of PPP can be used as a family of priors for a CRM. Assume a PPP has rate $\rho (\mathrm{d}\lambda )\mu (\mathrm{d}x)$ for $\lambda \in {\mathcal{R}}^{+}$ and $x\in \mathcal{X}$. This is called homogeneous because the terms in $\lambda$ and x are independent [18]. In the case considered here, the $\mu (\mathrm{d}x)$ is a measure on $\mathcal{X}$ called a base measure, and the rate $\rho (\mathrm{d}\lambda )$ has the condition ${\int }_0^{\infty }min(1,\lambda )\rho (\mathrm{d}\lambda )<\infty$ to make everything work neatly [20], as follows: This condition is equivalent to ${\int }_0^{\infty }min(ϵ,\lambda )\rho (\mathrm{d}\lambda )<\infty$ for any $0<ϵ<\infty$. As a consequence, $\rho ([ϵ,\infty ))$ is bounded, meaning there will be a finite number of points with $\lambda >ϵ$ in the sample of the PPP (within a finitely measured subset of $\mathcal{X}$) and ${\int }_0^{ϵ}\lambda \rho (\mathrm{d}\lambda )$ is bounded, meaning the sum of the $\lambda$’s less the $ϵ$ in the sample of the PPP (within a finitely measured subset of $\mathcal{X}$) will be finite even if there is an infinite number of them. Then, a sample from the PPP is a countable set of points which can be used to constuct a CRM.

2.3. Example Processes

Consider a number of standard PPPs used to construct CRMs [21]: the generalised (three-parameter) beta process [25], the generalised (three-parameter) gamma process [26] and the stable process. These have the forms given in

Table 1. General processes. Marginal is the corresponding infinitely divisible distribution for the total rate developed, for instance, using Theorem 1.

Name	Domain	Parameters	Rate (Lévy Measure)	Marginal
$\mathrm{b}\mathrm{e}\mathrm{P}(M,\alpha ,\beta )$	$0<\lambda <1$	$0\le \alpha <1$, $\beta >0$	${\displaystyle M\frac{{\lambda }^{-\alpha -1}}{\Gamma (1-\alpha )}{(1-\lambda )}^{\alpha +\beta -1}}$	for $\alpha$ = 0, $\beta$ = 1: $\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{k}\mathrm{m}\mathrm{a}\mathrm{n}(M)$
$\mathrm{G}\mathrm{P}(M,\beta )$	$\lambda >0$	$\beta >0$	${\displaystyle M{\lambda }^{-1}{e}^{-\lambda \beta }}$	$\mathrm{g}\mathrm{a}\mathrm{m}\mathrm{m}\mathrm{a}(M,\beta )$
$\mathrm{G}\mathrm{G}\mathrm{P}(M,\alpha ,\beta )$	$\lambda >0$	$0<\alpha <1$, $\beta >0$	${\displaystyle M\frac{\alpha }{\Gamma (1-\alpha )}{\lambda }^{-\alpha -1}{e}^{-\lambda \beta }}$	$\mathrm{T}\mathrm{w}\mathrm{e}(\alpha ,M^{1/\alpha },\beta )$
$\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{P}(M,\alpha )$	$\lambda >0$	$\alpha >0$	${\displaystyle \frac{M\alpha }{\Gamma (1-\alpha )}{\lambda }^{-\alpha -1}}$	$\mathrm{p}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{b}\mathrm{l}\mathrm{e}(\alpha ,M^{1/\alpha })$
$\mathrm{P}\mathrm{P}(M)$	$\lambda =1$		${\displaystyle M}$	$\mathrm{P}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{o}\mathrm{n}(M)$
$\mathrm{N}\mathrm{B}\mathrm{P}(M,\rho )$	$\lambda \in {\mathcal{N}}^{+}$	$0<\rho <1$	${\displaystyle M\frac{-{\rho }^{\lambda }}{\lambda log(1-\rho )}}$	$\mathrm{N}\mathrm{B}(M,\rho )$

, where M is a constant background rate. They are given without specifying a base measure on $\mathcal{X}$, which could be given as a final parameter.

The Poisson process and the negative binomial process [11] are also included in Table 1. Both are used in the hierarchical context in Section 4.

The first three processes in Table 1 are widely used in various forms in the non-parametric Bayesian and machine learning communities. From a Bayesian perspective, they are best thought of as improper priors corresponding to the beta, gamma and gamma distributions, respectively. This analysis is presented later in Section 3.4.

NRMIs can be created by normalising CRMs. These are sometimes generated directly from distributions consisting of a normalised discrete set of weights as probabilities. So, generating the $\overrightarrow{\lambda }$ according to a generalised (or three parameter) gamma process, $\mathrm{GGP}(M,\alpha ,\beta )$, and then normalising yields, what is called a normalised generalised gamma process (NGG). The normalised generalised gamma process (NGG) is constructed analogously to the Dirichlet process, which normalises the gamma process. They represent the main examples of NRMIs. These NRMIs, however, are not paired with base measures when forming a discrete process on $\mathcal{X}$, rather they need to be paired with base distributions $Pr(x)$ since only one point is generated per sample. Denote the NGG process as $\mathrm{NGG}(\alpha ,\beta ,M)$ or $\mathrm{NGG}(\alpha ,\beta ,M,h(·))$, where $\alpha ,\beta ,M$ are as described for the GGP, line 3 of Table 1, and $h(·)$ is a base distribution. The DP is effectively the case when $\alpha =0$.

Traditionally, the parameter vector part of the DP in Equation (1) is called a GEM distribution (specifically, when a size-biased order is used [27]), named after Griffiths, Engen and McCloskey [28]. This can be represented as an infinite vector $\overrightarrow{\lambda }=({\lambda }_1,{\lambda }_2,\dots )$. Correspondingly, there is a two-parameter version of $\overrightarrow{\lambda }$ corresponding to the PYP, $\mathrm{GEM}(\alpha ,\beta )$, which has discount $0\le \alpha <1$ and concentration $\beta >-\alpha$. Then, $\mathrm{GEM}(0,\beta )$ is the original GEM. Including the base distribution $h(·)$ yields $\mathrm{DP}(\beta ,h(·))$ and $\mathrm{PYP}(\alpha ,\beta ,h(·))$.

The Pitman–Yor process itself was developed by Pitman and Yor [28], and a general scheme for developing related models is by Pitman [29], called Poisson–Kingman models. However, as to be shown, the hierarchical PYP is very different from the PYP, so this theory is not entirely relevant for the hierarchical case. Alternatively, in Pitman and Yor [28] ([Proposition 21]), it was shown that a PYP can be developed by marginalising out a parameter of the NGG as follows.

Lemma 1.

(Deriving a PYP from a NGG) Let $\mu (x)\sim NGG\left(\alpha ,M,h(·)\right)$ for $\alpha ,M>0$ and suppose $M\sim gamma(\beta /\alpha ,1)$ for $\beta >0$, then it follows that $\mu (x)\sim PDP(\alpha ,\beta ,h(·))$.

The result is presented rather indirectly in Pitman and has been re-expressed by several authors [23] ([Section 3.1.1]), [30] ([Corollary 1]), and leads to a different class of models to the Poisson–Kingman models called Poisson-gamma models [23].

Notice the lemma restricts the PYP to the case where the concentration is positive. More generally, PYPs can have concentration $\beta >-\alpha$. When $\beta =0$ and $\alpha >0$, then the PYP is formed from normalising a positive stable distribution.

3. Defining Processes Axiomatically

This section gathers together some definitions and theory in order to present a general class of processes built on CRMs that can be treated hierarchically analogous to the Dirichlet process.

3.1. Subordinators

A simple useful case of these PPPs has the domain $\mathcal{X}$ being ${\mathcal{R}}^{+}$, the positive real line, and is constant for $\mathcal{X}$, so the rate is $\rho (\mathrm{d}\lambda )$ for $\lambda ,x\in {\mathcal{R}}^{+}$. For this, define a new process for our case $C_0=0$ given by the cumulative values,

$${\sigma }_t=\mu ((0,t])=\sum _{i=1}^{\infty }{\lambda }_i{\delta }_{x_i\le t}$$

So, ${\sigma }_0=0$ and ${\sigma }_t$ increases in steps as each distinct $x_i$ is passed. This ${\sigma }_t$ corresponds to the class of so-called pure jump driftless subordinators, which are a kind of nondecreasing Lévy process, which in turn are processes with stationary independent increments [20]. The key relationship that underlies the general theory of these processes is that ${\sigma }_t$ is distributed according to a particular infinitely divisible non-negative distribution, explained in Theorem 1. Examples are given in Table 1. So, for instance, for the generalised gamma process with parameters $(M,\alpha ,\beta )$, the total ${\sigma }_1={\sum }_{i=1}^{\infty }{\lambda }_i{\delta }_{x_i\le 1}$ is distributed as a Tweedie distribution with parameters $(\alpha ,M^{1/\alpha },\beta )$.

The basic connection is given as follows, a special case of the Lévy–Khintchine formula for subordinators. This uses the Laplace exponent of a 1D random variable y defined as the function (of u) $\epsilon [e^{-uy}]$, which is related to the characteristic function.

Theorem 1.

Consider ${\sigma }_t$ defined as previously by a PPP with rate $\rho (d\lambda )$ for $\lambda ,x\in {\mathcal{R}}^{+}$ and $\rho (d\lambda )$ satisfying ${\int }_0^{\infty }min(1,\lambda )\rho (d\lambda )<\infty$. The Laplace exponent of ${\sigma }_t$ is given by

$$\epsilon [e^{-u{\sigma }_t}]=e^{-t\psi (u)}$$

where $\psi (u)={\int }_{(0,\infty )}(1-e^{u\lambda })\rho (d\lambda )$. This form means that ${\sigma }_t$ has an infinitely divisible non-negative distribution. The t here can be referred to as the parameter for divisibility, occurring in any infinitely divisible distribution.

Thus, given a rate $\rho (\mathrm{d}\lambda )$ defining a particular ${\sigma }_t$, one can derive its Laplace exponent $\psi (u)$ and then infer the distribution on ${\sigma }_t$ (where analytically possible). Note the scaling term M in Table 1 plays the role of t.

Some instances of this pairing, an infinitely divisible non-negative distribution with a corresponding rate are given in the last two columns of Table 1. Note that distributions corresponding to the generalised beta process are not well-known. Other distributions that could be included in the table are the inverse beta distribution (the beta distribution is not infinitely divisible but its inverse is), which includes the Pareto and F-distributions, and the generalised inverse gamma distribution [31].

3.2. Axiomatic Definitions

To extend Theorem 1 to broader classes of base distributions on general domains $\mathcal{X}$, not just the positive real line with constant measure used in subordinators, one can give an axiomatic definition of a process based on an infinitely divisible non-negative distribution:

• The derived process is a CRM,
• The process behaves like the given infinitely divisible distribution on subsets of $\mathcal{X}$.

Definition 1.

(Axiomatic definition of a CRM process) Consider an infinitely divisible non-negative distribution $\mathrm{G}(\mu )$, where μ is the parameter for divisibility. Further assume its Laplace exponent has zero drift. Given a measurable space $\mathcal{X}$, positive intensity M and measure $h(dx)$ on $\mathcal{X}$, consider a stochastic process denoted $\mathrm{GP}(M,h(·))$ induced by $\mathrm{G}(\mu )$ as follows. $X\sim \mathrm{G}\mathrm{P}(M,h(·))$ yields a CRM on $\mathcal{X}$ such that

 1. 

For $A,B\subset \mathcal{X}$, if $A\cap B=\varnothing$ then $X(A)\perp \perp X(B)$,

 2. 

For $A\subseteq \mathcal{X}$, $X(A)\sim \mathrm{G}(Mh(A))$.

The first condition implies that the measures are CRMs as per Equation (1). The second condition implies one can construct the discrete measures iteratively, on an ever finer, nested sequence of partitions using the distribution $G()$. Alternatively, one can use the Lévy–Khintchine formula of Theorem 1 to show the existence of a corresponding rate yielding a CRM with rate $Mh(\mathrm{d}x)\rho (\mathrm{d}\lambda )$ which must then satisfy the conditions.

Note that the Dirichlet process can be defined axiomatically [12], akin to Definition 1 with the Dirichlet distribution used instead of the gamma distribution, and base probability distribution used instead of a base measure. This axiomatic construction generalises for any infinitely divisible non-negative distribution as follows:

Definition 2.

(Axiomatic definition of an NRMI process) Consider an infinitely divisible non-negative distribution $\mathrm{G}(\mu )$, where μ is the parameter for divisibility. Further assume its Laplace exponent has zero drift. Consider as well the distribution on probability vectors induced by generating K values ${\zeta }_k\sim \mathrm{G}({\mu }_k)$ and normalising to obtain

$$\left(\frac{{\zeta }_1}{{\sum }_{k=1}^K{\zeta }_k},\cdots ,\frac{{\zeta }_K}{{\sum }_{k=1}^K{\zeta }_k}\right).$$

Denote this distribution by ${\mathrm{N}\mathrm{G}}_K(\overrightarrow{\mu })$, where $\overrightarrow{\mu }$ is the vector of K value ${\mu }_k$, given a measurable space $\mathcal{X}$, positive intensity M and probability distribution $h(dx)$ on $\mathcal{X}$. A process denoted $NGP(M,h(·))$, developed from $\mathrm{G}(\mu )$, is defined as follows. It is a stochastic process whose sample is a probability measure on $\mathcal{X}$ such that if $C\sim NGP(M,h(·))$ then for any finite partition $A_1,\dots ,A_K$ of $\mathcal{X}$, and count $N>0$, $(C(A_1),\dots ,C(A_K))\sim multinomial\left(N,{NG}_K(Mh(A_1),\dots ,Mh(A_K))\right)$.

In this way, a multinomial process can be defined axiomatically, as done by Zhou and Carin [11] [Corollary IV2]. One uses $\mathrm{MP}(N,h(·))$ where $N\in {\mathcal{N}}^{+}$ is the total count and $h(·)$ a probability measure. The axiomatic part is $(X(A_1),\dots ,X(A_K))\sim \mathrm{multinomial}(N,$$(h(A_1),\dots ,h(A_K)))$. Similarly, a Dirichlet compound multinomial (DCM) process can be defined, denoted as $\mathrm{DCMP}(N,h(·))$, where the axiomatic part is $(X(A_1),\dots ,X(A_K))\sim \mathrm{DCM}(N,(h(A_1),\dots ,h(A_K)))$. These correspond to a PPP and a NBP, respectively, both given in Table 1, where one has also conditioned on the total count being N.

3.3. On the Tweedie Distribution

From Table 1, the marginal distribution for the generalised gamma process is the Tweedie distribution [32] with exponent $\alpha$, or sometimes expressed as index $p=1+\frac{1}{1-\alpha }$ which has $p>2$ necessarily. For $\alpha =0$, the Tweedie distribution becomes a gamma distribution.

The Tweedie distribution with exponent $0<\alpha <1$ is formed from a positive stable distribution defined in terms of the stable distribution with characteristic exponent $\alpha$, scale parameter $s=M^{1/\alpha }$ location zero and symmetry one [33]. This distribution, denoted as $\mathrm{pstable}(\alpha ,s)$, has the functional form [adding a scale to the standard formula of] [34] given by the remarkable formula

$$\begin{array}{ccc}\hfill Pr(x|\mathrm{p}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{b}\mathrm{l}\mathrm{e}(\alpha ,s))& =& \frac{\alpha }{1-\alpha }\frac{1}{s\pi }{(x/s)}^{-\frac{1}{1-\alpha }}{\int }_0^{\pi }{a}_{\alpha }(\nu )e^{-{(x/s)}^{-\frac{\alpha }{1-\alpha }}{a}_{\alpha }(\nu )}d\nu \hfill \\ \hfill \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e} a_{\alpha }(\nu )& =& \frac{sin((1-\alpha )\nu ){(sin(\alpha \nu ))}^{\alpha /(1-\alpha )}}{{sin(\nu ))}^{1/(1-\alpha )}},\hfill \end{array}$$

which yields a simple ingenious sampling formula [34]. To obtain a Tweedie distribution, “exponentially tilt” the $\mathrm{p}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{b}\mathrm{l}\mathrm{e}(\alpha ,s)$, calculated by multiplying by $e^{-\beta x}$ and renormalising. The construction of exponentially tilting the distribution (see for instance Pitman [29]) gives the following:

$$Pr(x|\mathrm{Twe}(\alpha ,s,\beta ))=e^{{(s\beta )}^{\alpha }-\beta x}Pr(x|\mathrm{p}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{b}\mathrm{l}\mathrm{e}(\alpha ,s)).$$

Here, the term $e^{{(s\beta )}^{\alpha }-\beta x}$ is added to achieve normalisation.

3.4. Bayesian Analysis

A complete Bayesian analysis of CRMs and NRMIs has been developed by James [16] and James et al. [18], respectively, in the non-hierarchical context. This models the standard framework in which hierarchical DPs or hierarchical PYPs are used, but also applies to the Indian buffet process [17]. This is informally developed below so that their theoretical results can be subsequently used. By Bayesian analysis, the following is meant: one has an infinitely divisible distribution suitable for use with Theorem 1. One samples a CRM from this with unknown parameters of rates $\overrightarrow{\lambda }$ and atoms $x_i$. Now, hierarchically sample sets of atoms from this CRM using a PPP. Each hierarchical sample from the CRM is a discrete set $A\subseteq \mathcal{X}$, and multiple samples are drawn. Then, the task is to estimate the parameters of the parent CRM.

A CRM is represented in the form $\mu (x)={\sum }_{i=1}^{\infty }{\lambda }_i{\delta }_{x_i}(x)$ for $x\in \mathcal{X}$ where the $x_i$ are distinct and is generated according to a homogeneous PPP with rate $\rho (\mathrm{d}\lambda )\omega (\mathrm{d}x)$ where $\rho (\mathrm{d}\lambda )$ is a rate satisfying the conditions of Theorem 1. One then takes J samples from this according to a PPP, so ${\overrightarrow{n}}_j\sim \mathrm{PPP}(\mu (·))$ for $j=1,\dots ,J$. Each sample will be a finite subset of the atoms, some possibly occurring multiple times. For representational purposes, post hoc reorder the atoms of $\mu (x)$ so that only the first I have non-zero counts. So, for $I<i\le \infty$, none of the samples ${\overrightarrow{n}}_j$ contain $x_i$. The count of atom $x_i$ in sample j is represented as $n_{j,i}$, so the condition ${\overrightarrow{n}}_{1:J,i}\ne \overrightarrow{0}$ means that at least one of the J samples contains an atom $x_i$.

The following informal analysis is offered as an explanation, but formal proofs are in James [16]. To make analysis feasible, we have to convert the rate $\rho (\mathrm{d}\lambda )$ to one with finite total measure. James [16] ingeniously and elegantly presents this by viewing the posterior for ${\mu }_i$ after seeing the evidence of having at least one non-zero value in the J values, so ${\overrightarrow{n}}_{1:J,i}\ne \overrightarrow{0}$. For the particular sampling distribution of $n_{j,i}$, in our case a $\mathrm{Poisson}({\lambda }_i)$,

$$Pr({\overrightarrow{n}}_{1:J,i}\ne \overrightarrow{0}|{\lambda }_i)=1-e^{-J{\lambda }_i}$$

which has a term in ${\lambda }_i$ so the posterior rate $Pr({\overrightarrow{n}}_{1:J,i}\ne \overrightarrow{0}|{\lambda }_i)\rho (\mathrm{d}{\lambda }_i)$ obtains finite total measure. Denote this total by ${\mathsf{\Psi }}_J=\int Pr({\overrightarrow{n}}_{1:J,i}\ne \overrightarrow{0}|{\lambda }_i)\rho (\mathrm{d}{\lambda }_i)$. Then, working entirely with finite PPPs, one can compute the marginal. First, we generate the number of non-zero atoms I (for the given sample count J) by a Poisson and then generate the vector of counts for each atom ${\overrightarrow{n}}_{1:J,i}$, like so

$$\begin{array}{ccc}\hfill Pr({\overrightarrow{n}}_1,\dots ,{\overrightarrow{n}}_J|\rho (\mathrm{d}\lambda ),\mathrm{PPP})& =& e^{-{\mathsf{\Psi }}_J}\frac{{\mathsf{\Psi }}_J^I}{I!}\prod _{i=1}^{I}Pr({\overrightarrow{n}}_{1:J,i}|{\overrightarrow{n}}_{1:J,i}\ne \overrightarrow{0})\hfill \\ & =& e^{-{\mathsf{\Psi }}_J}\frac{{\mathsf{\Psi }}_J^I}{I!}\prod _{i=1}^I\frac{\int Pr({\overrightarrow{n}}_{1:J,i}|\lambda )\rho (\mathrm{d}\lambda )}{\int Pr({\overrightarrow{n}}_{1:J,i}\ne \overrightarrow{0}|\lambda )\rho (\mathrm{d}\lambda )}\hfill \\ & =& e^{-{\mathsf{\Psi }}_J}\frac{1}{I!}\prod _{i=1}^I\int Pr({\overrightarrow{n}}_{1:J,i}|\lambda )\rho (\mathrm{d}\lambda ),\hfill \end{array}$$

(2)

where the term $I!$ can be removed if one considers that the atoms are ordered. With similar reasoning, one obtains:

• the posterior rate of ${\mathbf{\lambda }}_{\mathbf{i}}$: for $i\le I$ has rate $Pr({\overrightarrow{n}}_{1:J,i}|{\lambda }_i)\rho (\mathrm{d}{\lambda }_i)$,
• the posterior rate of the remainder CRM:${\mu }_R(x)={\sum }_{i=I+1}^{\infty }{\lambda }_i{\delta }_{x_i}(x)$, has rate
  $Pr({\overrightarrow{n}}_{1:J}\ne \overrightarrow{0}|\lambda )\rho (\mathrm{d}\lambda )\omega (\mathrm{d}x)$,
• the total rate of the remainder CRM:$T_R={\sum }_{i=I+1}^{\infty }{\lambda }_i$ as given by Theorem 1.

The key formula for this kind of analysis is given in our context in

Table 2. Key formula for posterior analysis of CRMs, ${\mathsf{\Psi }}_J=\int Pr({\overrightarrow{n}}_{1:J}\ne \overrightarrow{0}|\lambda )\rho (\mathrm{d}\lambda )$, and the distribution on the remainder $T_R={\sum }_{i=I+1}^{\infty }{\lambda }_i$.

Name	${\mathsf{\Psi }}_{\mathit{J}}$	Remainder ${\mathit{T}}_{\mathit{R}}$
beP$(M,\alpha ,\beta )$-BP	${\displaystyle M\sum _{j=0}^{J-1}\frac{\Gamma (\alpha +\beta +j)}{\Gamma (1+\beta +j)}}$	${\mu }_R\sim \mathrm{beP}(M,\alpha ,J+\beta )$
GP$(M,\beta )$-PP	$M\left(log(J+\beta )-log\beta \right)$	$\mathrm{g}\mathrm{a}\mathrm{m}\mathrm{m}\mathrm{a}(M,J+\beta )$
GGP$(M,\alpha ,\beta )$-PP	$M\left({(J+\beta )}^{\alpha }-{\beta }^{\alpha }\right)$	$\mathrm{T}\mathrm{w}\mathrm{e}(\alpha ,M^{1/\alpha },J+\beta )$
GP$(M,\beta )$-NBP$(\rho )$	$M\left(log(Jlog\frac{1}{1-\rho }+\beta )-log\beta \right)$	$\mathrm{g}\mathrm{a}\mathrm{m}\mathrm{m}\mathrm{a}(M,Jlog\frac{1}{1-\rho }+\beta )$
GGP$(M,\alpha ,\beta )$-NBP$(\rho )$	$M\left({(Jlog\frac{1}{1-\rho }+\beta )}^{\alpha }-{\beta }^{\alpha }\right)$	$\mathrm{T}\mathrm{w}\mathrm{e}(\alpha ,M^{1/\alpha },Jlog\frac{1}{1-\rho }+\beta )$
staP$(M,\alpha )$-PP	$M{J}^{\alpha }$	$\mathrm{T}\mathrm{w}\mathrm{e}(\alpha ,M^{1/\alpha },J+\beta )$
staP$(M,\alpha )$-NBP$(\rho )$	$M{(Jlog\frac{1}{1-\rho })}^{\alpha }$	$\mathrm{T}\mathrm{w}\mathrm{e}(\alpha ,M^{1/\alpha },Jlog\frac{1}{1-\rho })$

.

The first line, the beP-BP case, is the three parameter beta process with Bernoulli data, which is the three parameter Indian buffet process. The second line is the gamma process with Poisson data. Note the data marginals $\int Pr({\overrightarrow{n}}_{1:J,i}|\lambda )\rho (\mathrm{d}\lambda )$ in our context can be obtained more directly, developed in Section 4.2, so formulas are not given.

4. Using Discrete Base Distributions

It is important to understand what happens when you use a discrete distribution as a base distribution to a CRM, since this is what happens when hierarchical constructions of these processes are made. Let the base measure on $\mathcal{X}$ have the form $\mu (x)={\sum }_{i=1}^{\infty }{\lambda }_i{\delta }_{x_i}(x)$, and the CRM is constructed using a homogeneous PPP with rate $\rho (\mathrm{d}\lambda )\omega (\mathrm{d}x)$. What happens? This section considers various implications of this. Note different but more extensive treatment of this scenario for the results on moments, Section 4.2, and the generalised Chinese restaurant process, Section 4.4, is given by Camerlenghi et al. [14], Argiento et al. [15]. They also include example MCMC sampling algorithms.

4.1. General Results

Superposition of PPPs says to decompose a discrete CRM into a union of trivial PPPs each with rate in the form ${\mu }_i\rho (\lambda ){\delta }_{x_i}$, so the $\mathcal{X}$ component is a delta function. The resultant CRM is also trivial and takes the form, using Definition 1, $\Lambda {\delta }_{x_i}$, where $\Lambda$ is the total of the ${\lambda }_k$ generated using the rate ${\mu }_i\rho (\lambda )$. This total is distributed as the corresponding marginal distribution for the subordinator with intensity parameter ${\mu }_i$, as per Theorem 1.

Lemma 2.

(CRM when base measure is discrete) Let a discrete measure on $\mathcal{X}$ have the form $\mu (x)={\sum }_{i=1}^{\infty }{\mu }_i{\delta }_{x_i}(x)$ for $x\in \mathcal{X}$ where the $x_i$ are distinct, and a homogeneous CRM is constructed by sampling using a PPP with rate $\rho (d\lambda )\mu (dx)$ on ${\mathcal{R}}^{+}\times \mathcal{X}$. Let $\Gamma (t)$ be the marginal total distribution for the corresponding subordinator, where t is the parameter of divisability. Then, the CRM has the form

$$\gamma (x)=\sum _{i=1}^{\infty }{\gamma }_i{\delta }_{x_i}(x)$$

(3)

where the random variable ${\gamma }_i\sim \Gamma ({\mu }_i)$, and the $x_i$ are inherited from $\mu (·)$.

The CRM $\mu (·)$ when used as a base distribution for a PPP is mapped element-wise to form a new CRM $\gamma (·)$. So, no PPP modelling is required if you know the form of the element-wise distribution.

There are a number of very convenient and well-known properties of the Dirichlet that allow it to be used in hierarchical contexts. As it happens, most of these properties also hold for other NRMIs with discrete base measures, and some for CRMs, so these results are developed here. The first property is aggregation. This has that if $(x_1,x_2,x_3)\sim \mathrm{Dirichlet}({\alpha }_1,{\alpha }_2,{\alpha }_3)$, then $(x_1,x_2+x_3)\sim \mathrm{Dirichlet}({\alpha }_1,{\alpha }_2+{\alpha }_3)$, and this applies for a Dirichlet of any dimension. The second property is renormalisation and has that if $(x_1,x_2,x_3)\sim \mathrm{Dirichlet}({\alpha }_1,{\alpha }_2,{\alpha }_3)$ then $(x_1,x_2)/(x_1+x_2)\sim \mathrm{Dirichlet}({\alpha }_1,{\alpha }_2)$. Both properties clearly follow from the fact that a Dirichlet is a normalised Gamma, and by analogy hold for NRMIs too.

Definition 3.

(Aggregation property) Consider a process that takes a measure as an input parameter and outputs another measure. The process has the aggregation property if when ${\sum }_{i=1}^{\infty }{\gamma }_i{\delta }_{x_i}(x)$ is a sample from the process with a discrete input measure ${\sum }_{i=1}^{\infty }{\mu }_i{\delta }_{x_i}(x)$ where the $x_i$ are distinct, then ${\sum }_{i=3}^{\infty }{\gamma }_i{\delta }_{x_i}(x)+({\gamma }_1+{\gamma }_2){\delta }_{x_1}(x)$ is a sample from the process with input measure ${\sum }_{i=3}^{\infty }{\mu }_i{\delta }_{x_i}(x)+({\mu }_1+{\mu }_2){\delta }_{x_1}(x)$.

The aggregation property can be used to form arbitrary groupings of the dimensions.

Definition 4.

(Renormalisation property) Consider a process that takes a measure as an input parameter and outputs a probability measure. The process has the renormalisation property if when ${\sum }_{i=1}^{\infty }{\gamma }_i{\delta }_{x_i}(x)$ is a sample from the process with a discrete input measure ${\sum }_{i=1}^{\infty }{\mu }_i{\delta }_{x_i}(x)$ where the $x_i$ are distinct, then $\frac{1}{{\sum }_{i=2}^{\infty }{\gamma }_i}{\sum }_{i=2}^{\infty }{\gamma }_i{\delta }_{x_i}(x)$ is a sample from the process with input measure ${\sum }_{i=2}^{\infty }{\mu }_i{\delta }_{x_i}(x)$.

The renormalisation property then yields probability measures on subsets of the discrete domain, so it can be used for incremental sampling.

Lemma 3.

(Aggregation and renormalisation) Consider the context of Lemma 2. The aggregation property holds for all CRMs and NRMIs. In the case of an NRMI, the renormalisation property holds. For the PYP, the aggregation property holds but not the renormalisation property.

The results for the PYP can be developed using Lemma 1. The aggregation and renormalisation properties together mean that efficient size-biased samplers can be developed for NRMIs by sampling one dimension at a time according to a two-dimensional version of the NRMI, which is effectively the stick breaking construction (although, only a few explicit cases of this are known). Alternatively, one can sample the underlying CRM according to its corresponding infinitely divisible distribution.

A third property of the Dirichlet is neutrality, which applies in the context of renormalisation and requires that the part taken away is independent of the remainder: if $(x_1,x_2,x_3)\sim \mathrm{Dirichlet}({\alpha }_1,{\alpha }_2,{\alpha }_3)$, then $(x_1,x_2)/(x_1+x_2)$ is independent of $x_3$.

Definition 5.

(Neutrality property) Consider a process that outputs a finite discrete probability measure, and without loss of generality let ${\sum }_{i=1}^I{\gamma }_i{\delta }_{x_i}(x)$ be a sample from the process where the $x_i$ are distinct. The process is completely neutral if there exists mutually independent non-negative variables ${\lambda }_1,\dots ,{\lambda }_I$ such that $({\gamma }_1,\dots ,{\gamma }_K)$ and $\left({\lambda }_1,{\lambda }_2(1-{\lambda }_1),\dots ,{\lambda }_I{\prod }_{i=1}^{I-1}(1-{\lambda }_i)\right)$ have the same distribution.

It is known that the only distribution on finite probability vectors with complete neutrality is the Dirichlet distribution [35].

4.2. Results on Moments

Moments of CRMs are critical quantities for their posterior analysis [18,36] to be developed in Section 5 and seen in Section 3.4. The generalised version is derived by unfolding the recursion that relates the moments of a distribution to its cumulants. In the context of Lemma 2, where ${\gamma }_i\sim \Gamma ({\mu }_i)$, various moments such as $\epsilon [{\gamma }_i^n|{\mu }_i]$ and $\epsilon [{\gamma }_i^n{e}^{-U{\gamma }_i}|{\mu }_i]$ can be computed recursively from the moments of the PPP rate $\rho (\mathrm{d}\lambda )$ [22] ([Section 1.3]) and its exponentially titled form. Note these moments compute the marginals one needs for multinomial and Poisson data, respectively, hence their importance.

In the theorem, the notation ${\mathcal{P}}^n$ is used to represent all possible non-empty partitions of n items, the set $\{1,\dots ,n\}$. As an example, ${\mathcal{P}}^3$ is the set

$$\left\{\right\{\{1,2,3\}\},\{\left\{1\right\},\{2,3\}\},\{\{1,2\},\left\{3\right\}\},\{\{1,3\},\left\{2\right\}\left\}\right\},\left\{\right\{1\},\{2\},\{3\left\}\right\}\},$$

so it contains the partition $\left\{\right\{1\},\{2,3\left\}\right\}$ as an element, for instance. Moreover, ${\mathcal{P}}_K^n\subseteq {\mathcal{P}}^n$ are all members are of size K, so $|{\mathcal{P}}_1^n|=|{\mathcal{P}}_n^n|=1$ and $|{\mathcal{P}}_2^3|=3$.

The following Lemma is a corollary the major result by Pitman [22], and some related results appear in Camerlenghi et al. [14], as proven in Appendix A.

Lemma 4.

(CRM moments when base measure is discrete) Consider the context of Lemma 2. Let ${\kappa }_n={\int }_0^{\infty }{\lambda }^n\rho (d\lambda )$ be the n-th moment for rate $\rho (\lambda )$, where it exists for $n\in {\mathcal{N}}^{+}$. Let $\psi (t)$ be the Laplace exponent for the rate. Then, the n-th cumulant of ${\gamma }_i$ can be re-expressed as a moment of the original rate $\rho (\lambda )$, and the n-th moment of ${\gamma }_i$ is computed recursively from it.

$$\begin{array}{ccc}\hfill {\kappa }_n& =& {(-1)}^{n+1}{\left.\frac{d{}^n\psi (t)}{d{t}^n}\right|}_{t=0}\hfill \end{array}$$

(4)

$$\begin{array}{ccc}\hfill {cumulant}_n({\gamma }_i)& =& {\mu }_i{\kappa }_n\hfill \end{array}$$

(5)

$$\begin{array}{ccc}\hfill \epsilon [{\gamma }_i^n|{\mu }_i]& =& \sum _{\Pi \in {\mathcal{P}}^n}{\mu }_i^{|\Pi |}\prod _{C\in \Pi }{\kappa }_{\left|C\right|}\hfill \end{array}$$

(6)

$$\begin{array}{ccc}& =& \sum _{K=1}^n{\mu }_i^K{T}_K^n\hfill \end{array}$$

(7)

$$\begin{array}{ccc}\hfill where T_K^n& =& \sum _{\Pi \in {\mathcal{P}}_K^n}\prod _{C\in \Pi }{\kappa }_{\left|C\right|}\hfill \\ & =& \sum _{k=1}^{n-K+1}{T}_{K-1}^{n-k}\left(\genfrac{}{}{0pt}{}{n-1}{k-1}\right){\kappa }_k.\hfill \end{array}$$

(8)

Note the recursion for $T_K^n$ starts at $T_1^n={\kappa }_n$ derived from the non-recursive form.

Thus, if the Laplace exponent is known, one can usually compute the moments of the process and hence the cumulants and evidence terms for its corresponding infinitely divisible distribution. When one has Poisson data, required moments need to include an exponential term, as proven in Appendix B.

Corollary 1.

(Adding an exponential term) Consider the context of Lemma 4 with rate $\rho (\lambda )$. To obtain exponentiated moments of the form $\epsilon [{\gamma }_i^n{e}^{-U{\gamma }_i}|{\mu }_i]$, complete the following steps.

 1. 

Use rate $e^{-U\lambda }\rho (\lambda )$, and the Laplace exponent is given by $\psi (U+t)-\psi (U)$, so the corresponding moments are given by

$${\kappa }_{n,U}={(-1)}^{n+1}{\left.\frac{d{}^n\psi (t)}{d{t}^n}\right|}_{t=U}$$

 2. 

Obtain the corresponding $T_K^n$ using Equation (8) with the ${\kappa }_{n,U}$, denoted $T_{K,U}^n$.

 3. 

Consequently,

$$\epsilon [{\gamma }_i^n{e}^{-U{\gamma }_i}|{\mu }_i]=e^{-{\mu }_i\psi (U)}\sum _{K=1}^n{\mu }_i^K{T}_{K,U}^n.$$

The components from Lemma 4 for the processes in Table 1 are given in

Table 3. Properties of processes.

Name	${\mathit{\kappa }}_{\mathit{n}}$	$\mathit{\psi }(\mathit{t})$	${\mathit{T}}_{\mathit{K}}^{\mathit{n}}$
$\mathrm{beP}(M,\alpha ,\beta )$	$M\frac{\Gamma (n-\alpha )}{\Gamma (1-\alpha )}\frac{\Gamma (\alpha +\beta )}{\Gamma (n+\beta )}$	$M\frac{\Gamma (\alpha +\beta )}{\Gamma (\beta )\alpha }({}_1{F}_1(1-\alpha ,\beta ,t)-1$	use Equation (8)
(for $\beta >1-\alpha$)		$+\frac{1}{\beta }{}_1{F}_1(1-\alpha ,\beta +1,t))$
$\mathrm{GP}(M,\beta )$	$M\frac{\Gamma (n)}{{\beta }^n}$	$Mlog(1+t/\beta )$	$\frac{M^K}{{\beta }^n}{S}_K^n$
$\mathrm{GGP}(M,\alpha ,\beta )$	$M\frac{\Gamma (n-\alpha )}{\Gamma (1-\alpha )}\frac{\alpha }{{\beta }^{n-\alpha }}$	$M\left({(\beta +t)}^{\alpha }-{\beta }^{\alpha }\right)$	$\frac{{(M\alpha {\beta }^{\alpha })}^K}{{\beta }^n}{S}_{K,\alpha }^n$
$\mathrm{staP}(M,\alpha )$	NA	$M{t}^{\alpha }$	NA

. These appear in various places in the broader statistical literature. The Laplace exponent is usually computed using integration by parts. The form $S_{s,\alpha }^n$ is the second order generalised Stirling number used in PYP inference [1,37], a generalized Stirling number of type $(-1,-d,0)$ [38]. It can be verified using its recursion [37] with Equation (8).

Note the general beta process has no simple analytic form for either $\psi (t)$ or its marginal distribution. Fortunately, is is difficult to envisage a situation where it would be used hierarchically.

4.3. The Gamma Process

Let us consider the simple example of a gamma process, $\mathrm{GP}(M,\beta )$ and assume data yields Poisson likelihoods in the form ${\prod }_{i=1}^I{\gamma }_i^{n_i}{e}^{-U{\gamma }_i}$ for dimensions $i=1,\dots ,I$ in the context of Lemma 2. The marginal likelihood then, for the $\mathrm{data}=\{n_i,x_i:i=1,\dots ,I\}$ is given by

$$Pr(data|\mu (·))=\epsilon [e^{-U{\gamma }_R}|{\mu }_R]\prod _{i=1}^I\epsilon [{\gamma }_i^{n_i}{e}^{-U{\gamma }_i}|{\mu }_i]$$

where the expectation is taken with respect to $\gamma (·)\sim \mathrm{GP}(M,\beta ,\mu (·))$, which has ${\gamma }_i\sim \mathrm{gamma}(M{\mu }_i,\beta )$. Note, in this case, the exact solution is known since the data marginals of the gamma distribution have a simple closed form,

$$\epsilon [{\gamma }_i^{n_i}{e}^{-U{\gamma }_i}|{\mu }_i]={\int }_0^{\infty }{\gamma }^{n_i}{e}^{-U\gamma }\frac{{\beta }^{M{\mu }_i}}{\Gamma (M{\mu }_i)}{\gamma }^{M{\mu }_i-1}{e}^{-\beta \gamma }\mathrm{d}\gamma =\frac{\Gamma (M{\mu }_i+n_i)}{{(U+\beta )}^{M{\mu }_i+n_i}}\frac{{\beta }^{M{\mu }_i}}{\Gamma (M{\mu }_i)}$$

(9)

Consider, however, using Corollary 1. In this case, moments including $e^{-U{\gamma }_i}$ are found to be ${\kappa }_n={\int }_0^{\infty }{\gamma }^n{e}^{-U\gamma }\rho (\mathrm{d}\gamma )=M\frac{\Gamma (n)}{{(U+\beta )}^n}$, and the Laplace exponent can be obtained using integration by parts as $Mlog(1+t/\beta )$. One can confirm that the corresponding index $T_K^n=\frac{1}{{(U+\beta )}^n}{S}_K^n{M}^K$ where $S_K^n$ is an unsigned Stirling number of the first kind, an index that is found in collapsed versions of the CRP. Equation (8) yields the standard recurrence for it. So, by Equation (7), and adding back the term $e^{-{\mu }_i\psi (U)}={\left(\frac{\beta }{U+\beta }\right)}^{M{\mu }_i}$ as per Corollary 1, obtain for atom index i the moment

$$Pr({\gamma }_i^n{e}^{-U{\gamma }_i}|{\mu }_i)=\epsilon [{\gamma }_i^n{e}^{-U{\gamma }_i}|{\mu }_i]=\frac{{\beta }^{M{\mu }_i}}{{(U+\beta )}^{M{\mu }_i+n_i}}\sum _{K=1}^{n_i}{S}_K^{n_i}{(M{\mu }_i)}^K.$$

(10)

The sum can be converted using a standard identity [37] ([Lemma 16]) to get back The sum in Equation (10) has an interpretation as a form of Chinese restaurant process for the dimension i. Each partition of the set $\{1,\dots ,n_i\}$, given by ${\Pi }_i\in {\mathcal{P}}^{n_i}$ corresponds to a configuration of the $n_i$ data in $|{\Pi }_i|$ tables. For any table with participants $C\in {\Pi }_i$, the probability of the table is $M{\mu }_i\frac{\Gamma (|C|)}{{(U+\beta )}^{\left|C\right|}}$. The probability of this configuration ${\Pi }_i$ is then ${\prod }_{C\in {\Pi }_i}M{\mu }_i\frac{\Gamma (|C|)}{{(U+\beta )}^{\left|C\right|}}$. So, introducing the partition ${\Pi }_i$ or its size as an additional variable,

$$\begin{array}{ccc}\hfill Pr({\gamma }_i^n{e}^{-U{\gamma }_i},{\Pi }_i|{\mu }_i)& =& \frac{{\beta }^{M{\mu }_i}}{{(U+\beta )}^{M{\mu }_i+n_i}}{(M{\mu }_i)}^{|{\Pi }_i|}\prod _{C\in {\Pi }_i}\Gamma (|C|)\hfill \\ \hfill Pr({\gamma }_i^n{e}^{-U{\gamma }_i},|{\Pi }_i|=K|{\mu }_i)& =& \frac{{\beta }^{M{\mu }_i}}{{(U+\beta )}^{M{\mu }_i+n_i}}{S}_K^{n_i}{(M{\mu }_i)}^K.\hfill \end{array}$$

The second form, the probability of all configurations of size $K(=|{\Pi }_i|)$, follows from Equation (8).

4.4. General Chinese Restaurant Processes

Motivated by the gamma process example just given, now construct a generalised CRP interpretation of the results in Section 4.2. The marginals have an interpretation as generalised versions of Chinese restaurants, including the more efficient collapsed versions [6], both developed in this section. This is intended to complement the comprehensive Bayesian analysis already developed for the non-hierarchical cases by [16,18].

The significance of the formula in Lemma 4 is that the sum in Equation (6) is over partitions $\Pi$ of the n data points, and ${\kappa }_{\left|C\right|}$ represents the probability of generating a single element C of size $\left|C\right|$ (in the partition $\Pi$) according to the rate $\rho (\lambda )$. The sum in Equation (7) is now over partition sizes K, and $T_K^n$ is the probability of generating a partition of K non-empty sets according to the rate $\rho (\lambda )$.

Lemma 5.

(General Chinese restaurant processes for CRMs) Consider the posterior data marginal for $\gamma (·)$, as in Corollary 2, where data is in the form of a Poisson likelihood with counts $n_i>0$ at each atom $x_i$:

$$Pr(\{n_i,x_i:i=1,\dots ,I\}|\gamma (·),U)=\prod _{i=1}^I{\gamma }_i^{n_i}{e}^{-U{\gamma }_i}$$

One can treat ${\Pi }_i\in {\mathcal{P}}^{n_i}$ as a latent variable, which represents the seating configuration for instances of the atom. Then, the data marginal using ${\Pi }_1,\dots ,{\Pi }_I$ takes the form:

$$Pr(\{n_i,x_i,{\Pi }_i:i=1,\dots ,I\}|\mu (·),U(=e^{-\psi (U){\sum }_{i=1}^{\infty }{\mu }_i}\prod _{i=1}^I\left({\mu }_i^{|{\Pi }_i|}\prod _{C\in {\Pi }_i}{\kappa }_{|C_i|,U}\right).$$

(11)

Moreover, for any j (including $j>I$),

$$Pr(x_j|\{n_i,x_i,{\Pi }_i:i=1,\dots ,I\},\mu (·))={\mu }_j{\kappa }_{1,U}+\sum _{C\in {\Pi }_j}\frac{{\kappa }_{\left|C\right|+1,U}}{{\kappa }_{\left|C\right|,U}},$$

(12)

where the convention is used that ${\Pi }_j=\varnothing$ for $j>I$ (when there is no data). Alternatively, if $K_i$, the number of tables for atom index i is handled as a latent variable, then the data marginal given table numbers takes the form:

$$Pr(\{n_i,x_i,K_i:i=1,\dots ,I\}|\mu (·),U)=e^{-\psi (U){\sum }_{i=1}^{\infty }{\mu }_i}\prod _{i=1}^I{\mu }_i^{K_i}{T}_{K_i,U}^{n_i}.$$

(13)

Equation (12) is related to the generalized Blackwell–MacQueen sampling scheme by James et al. [18] [Section 3.3]. The data marginals in Equations (11) and (13) have a simple Poisson likelihood in $\overrightarrow{\mu }$. Thus, a CRP interpretation of a Gamma process can be used for hierarchical inference with a Gamma distribution, as used by Zhou and Carin [11], for instance.

To develop a corresponding formula for NRMIs where they are generated by normalising a CRM, we use an ingenious technique for normalising a CRM within a posterior analysis from [18] The basic idea is to convert multinomial sampling into Poisson sampling (without normalisation) but require some post manipulation to derive the results. A generative variation of this goes as follows:

• For each multinomial $\overrightarrow{n}$ according to the unnormalised values $\overrightarrow{\lambda }$, introduce a scale-free latent relative mass denoted U, with the scale-invariant improper prior $\frac{dU}{U}$.
• Generate the data needed according to Poisson $n_i\sim \mathrm{Poisson}(U{\lambda }_i)$ for $i=1,\dots ,\infty$, noting that $n_i=0$ for $i>I$.
• Then, the joint posterior on $\overrightarrow{n},\overrightarrow{\lambda },U$ becomes quite concentrated for U and can be marginalised out.
• To correct the formulas, multiply the marginal by $N={\sum }_{i=1}^I{n}_i$ to obtain a conversion to a multinomial.

To see that this indeed does what is required, one needs to verify the following identity.

$$N{\int }_{{\mathcal{R}}^{+}}\prod _{i=1}^{\infty }\left(e^{-U{\lambda }_i}\frac{{(U{\lambda }_i)}^{n_i}}{n_i!}\right)\frac{dU}{U}=\left(\genfrac{}{}{0pt}{}{N}{\overrightarrow{n}}\right)\prod _{i=1}^I{\left(\frac{{\lambda }_i}{{\sum }_i{\lambda }_i}\right)}^{n_i}.$$

Note the product ${\prod }_{i=1}^{\infty }$ is well-defined because ${\sum }_{i=1}^{\infty }{\lambda }_i$ is finite.

Corollary 2.

(General Chinese restaurant processes for NRMIs) Consider the posterior data marginal for $\gamma (·)$ as given in Lemma 2, where data is in the form of a multinomial likelihood with counts $n_i>0$ at each atom $x_i$:

$$Pr(\{n_i,x_i:i=1,\dots ,I\}|\gamma (·))=\prod _{i=1}^I{\left(\frac{{\gamma }_i}{{\sum }_{i=1}^{\infty }{\gamma }_i}\right)}^{n_i},$$

and let $N={\sum }_{i=1}^I{n}_i$ be the total count. Let $U\sim gamma(N,{\sum }_{i=1}^{\infty }{\gamma }_i)$. Then, the data marginal using ${\Pi }_1,\dots ,{\Pi }_I$, similarly to Lemma 5, takes the form:

$$Pr(\{n_i,x_i,{\Pi }_i:i=1,\dots ,I\},U|\mu (·))=\frac{U^{N-1}}{\Gamma (N)}{e}^{-\psi (U){\sum }_{i=1}^{\infty }{\mu }_i}\prod _{i=1}^I\left({\mu }_i^{|{\Pi }_i|}\prod _{C\in {\Pi }_i}{\kappa }_{|C_i|,U}\right).$$

(14)

Moreover, for any j (including $j>I$),

$$Pr(x_j|\{n_i,x_i,{\Pi }_i:i=1,\dots ,I\},U,\mu (·))={\mu }_j{\kappa }_{1,U}+\sum _{C\in {\Pi }_j}\frac{{\kappa }_{\left|C\right|+1,U}}{{\kappa }_{\left|C\right|,U}}.$$

(15)

Alternatively, if each $K_i$ is handled as a latent variable, then the data marginal given table numbers takes the form:

$$Pr(\{n_i,x_i,K_i:i=1,\dots ,I\},U|\mu (·))=\frac{U^{N-1}}{\Gamma (N)}{e}^{-\psi (U){\sum }_{i=1}^{\infty }{\mu }_i}\prod _{i=1}^I{\mu }_i^{K_i}{T}_{K_i,U}^{n_i}.$$

(16)

Note, to complete the analysis, one needs to model the unseen parts of the processes. So, while it is assumed ${\mu }_i$ for $i=1,\dots ,I$ is being sampled or estimated, of ${\mu }_i$ and ${\gamma }_i$ for $i=I+1,\dots ,\infty$ only a finite number, if any, can be sampled or estimated. Handling these is illustrated in Section 5 using a remainder term ${\mu }_R={\sum }_{j=I+1}^{\infty }{\mu }_j$.

In general, then, there are two different levels of inference one can use when the marginal does not have a simple closed form and must instead be computed using the latent forms in Lemma 5 or Corollary 2:

Sampling over table configurations:

For the DP, this is exhibited by the standard CRP. One can see from Equations (6) and (12) that to resample which table a point belongs to, one would use the following proportionalities:

$$Pr(C|\Pi ,{\mu }_k,\dots )\propto \left\{\begin{array}{cc}{\mu }_k{\kappa }_1\hfill & \hfill \mathrm{s}\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{t} \mathrm{a} \mathrm{n}\mathrm{e}\mathrm{w} \mathrm{t}\mathrm{a}\mathrm{b}\mathrm{l}\mathrm{e}\\ \frac{{\kappa }_{\left|C\right|+1}}{{\kappa }_{\left|C\right|}}\hfill & \hfill \mathrm{a}\mathrm{d}\mathrm{d} \mathrm{t}\mathrm{o} \mathrm{t}\mathrm{a}\mathrm{b}\mathrm{l}\mathrm{e} C\end{array}\right..$$

(17)

Sampling over table sizes:

For the PYP, this is demonstrated by table indicator sampling methods [6,39] and “direct” Gibbs sampling of Gasthaus and Teh [5], though subsequently not used because in their context they needed to constantly resample discount $\alpha$. This is a collapsed sampler that instead samples K, the number of tables using Equation (7):

$$Pr(K|{\mu }_k,\dots )\propto {\mu }_k^K{T}_K^n$$

(18)

This is only efficient when $T_K^n$ can be tabulated. In the general case, this requires $O(n^{2}K)$ steps to follow using Equation (8) and $O(nK)$ for cases such as the gamma process above where a simpler double recursion is available for $T_K^n$ since they are generalised second-order Stirling numbers.

5. Variants of the Generalised Gamma Process

In this section, we develop both the CRM and NRMI variants of the generalised gamma process in the hierarchical context. Using the generalised gamma process in an NRMI yields an NGG or a PYP. When the NGG process and the PYP are supplied discrete base distributions as input, they behave analogously to the Dirichlet distribution, as illustrated with Lemma 3. In this discrete context, refer to the corresponding distributions as the NGG distribution and the Pitman–Yor distribution (PYD). Here analytical forms of the PY distribution are developed.

The Hierarchical Context

Consider an NRMI in the context of the base distribution $\mu (x)$, as before. Suppose multinomial type data is observed in the form of counts $n_k$ associated with the atoms $x_k$ of $\mu (·)$ for $k=1,\dots ,K$, with total count $N={\sum }_{k=1}^K{n}_k$, where all others are zero. The latent relative mass trick of James et al. [18] can be used to include U as a latent variable in the likelihood for the NGG and the PYD. Setting $U=1$ and dividing by N in this case restores the posterior to the original Poisson version. The likelihood for a PYD also includes M (via Lemma 1). To express this, the remainder terms for both the base distribution and the CRM need to be represented.

$$\begin{array}{ccc}\hfill {\lambda }_R& =& \sum _{k=K+1}^{\infty }{\lambda }_k=\Lambda -\sum _{k=1}^K{\lambda }_k\hfill \\ \hfill {\mu }_R& =& \sum _{k=K+1}^{\infty }{\mu }_k.\hfill \end{array}$$

The joint posterior for the NGG is now

$$\begin{array}{c}{\displaystyle Pr(\{{\lambda }_k,n_k,x_k:k=1,\dots ,K\},U,{\lambda }_R|\mathrm{GGP},M,\alpha ,\beta ,N,\mu (·))}\hfill \\ =\frac{1}{N^{1_{U\ne 1}}\Gamma (N)}{e}^{-U\Lambda }{U}^{N-1}Pr({\lambda }_R|\mathrm{Twe}(\alpha ,{(M{\mu }_R)}^{1/\alpha },1))\hfill \\ \prod _{k=1}^K{\lambda }_k^{n_k}Pr({\lambda }_k|\mathrm{Twe}(\alpha ,{(M{\mu }_k)}^{1/\alpha },1))\hfill \\ =\frac{1}{\Gamma (N)}{e}^{-M\left({(1+U)}^{\alpha }-1\right)}{U}^{N-1}Pr({\lambda }_R|\mathrm{Twe}(\alpha ,{(M{\mu }_R)}^{1/\alpha },1+U))\hfill \\ \prod _{k=1}^K{\lambda }_k^{n_k}Pr({\lambda }_k|\mathrm{Twe}(\alpha ,{(M{\mu }_k)}^{1/\alpha },1+U)),\hfill \end{array}$$

(19)

where the second line is obtained by applying the exponential tilting formula. Note, Lemma 2 means element-wise application of a distribution to the parameter vector $\overrightarrow{\mu }$ inside $\mu (·)$. Forms for the PYD are obtained by adding the prior for M. For the normalised stable process, denoted NSP, one obtains

$$\begin{array}{c}\hfill {\displaystyle Pr(\{{\lambda }_k,n_k,x_k:k=1,\dots ,K\},U,{\lambda }_R|\mathrm{NSP},M,\alpha ,N,\mu (·))}\\ =\frac{1}{\Gamma (N)}{e}^{-U\Lambda }{U}^{N-1}Pr({\lambda }_R|\mathrm{pstable}(\alpha ,{(M{\mu }_R)}^{1/\alpha }))\hfill \\ \prod _{k=1}^K{\lambda }_k^{n_k}Pr({\lambda }_k|\mathrm{pstable}(\alpha ,{(M{\mu }_k)}^{1/\alpha }))\hfill \\ =\frac{1}{\Gamma (N)}{e}^{-M{U}^{\alpha }}{U}^{N-1}Pr({\lambda }_R|\mathrm{Twe}(\alpha ,{(M{\mu }_R)}^{1/\alpha },U))\hfill \\ \prod _{k=1}^K{\lambda }_k^{n_k}Pr({\lambda }_k|\mathrm{Twe}(\alpha ,{(M{\mu }_k)}^{1/\alpha },U)).\hfill \end{array}$$

(20)

From this, one can derive an integral formula for the PYD. Details are in Appendix C, and the result is original.

Lemma 6.

(Integral formula for the PY distribution) Let $\overrightarrow{\mu }$ be a K-dimensional non-zero probability vector. Then, consider $\overrightarrow{\theta }\sim PYD(\alpha ,\beta ,\overrightarrow{\mu })$ for $\alpha >0$ and $\beta \ge 0$. To express the probability of $\overrightarrow{\theta }$, introduce corresponding latent variables $\overrightarrow{\nu }=({\nu }_1,\dots ,{\nu }_K)\in {[0,\pi ]}^K$:

$${\displaystyle Pr(\overrightarrow{\theta }|PYD,\alpha ,\beta ,\overrightarrow{\mu })=}\underset{{[0,\pi ]}^K}{\int }\frac{{\alpha }^{K-1}\Gamma (1+\beta )}{{(1-\alpha )}^{K-1}{\pi }^K\Gamma (1+\beta /\alpha )}\frac{\Gamma (K+\beta (1-\alpha )/\alpha ){\prod }_{k=1}^K{a}_{\alpha }({\nu }_k){\left(\frac{{\mu }_k}{{\theta }_k}\right)}^{1/(1-\alpha )}}{{\left({\sum }_{k=1}^K{a}_{\alpha }({\nu }_k){\left(\frac{{\mu }_k}{{\theta }_k}\right)}^{1/(1-\alpha )}{\theta }_k\right)}^{K+\beta (1-\alpha )/\alpha }}d\overrightarrow{\nu }.$$

(21)

This can be readily evaluated using numerical integration for small K. Plots of the marginal for ${\theta }_1$ for different parameter settings are given in Figure 1 and Figure 2.

Due to the aggregation property of the PYD, these are representative marginals of the distribution for all dimensions. One can see the distributions becoming increasingly skewed as $\alpha$ increases.

6. Networks of Processes

The next natural question to consider is how the above results apply to networks of processes. Several general schemes have been developed for inference in more general networks [3,6,11,19,39,40]. General networks for HPYPs have been demonstrated to scale [4,6], in contrast to earlier Gibbs schemes [3,40], and arguably the HGP has advantages over the HDP [11]. This section is a review of related material with regards to hierarchical processes.

6.1. Identifiability

One important question is the issue of statistical identifiability, and an underlying issue here is whether the parametric structure admits a unique representation [41]. In our case, some simple classes of non-uniqueness are easily identified and avoided. For instance, in Poisson matrix factorisation, if the matrix entry $x_{i,j}\sim \mathrm{P}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{o}\mathrm{n}\left({\sum }_{k=1}^K{\theta }_{i,k}{\varphi }_{k,j}\right)$, then one can insist that the scale of one of the matrices $\overrightarrow{\mathsf{\Theta }}$ or $\overrightarrow{\Phi }$ (comprising the entries ${\theta }_{i,k}$ and ${\varphi }_{k,j}$ respectively) needs to be anchored somehow so that the scale of the Poisson parameter is uniquely determined by just the other one. So, the rows of one of the matrices should normalise.

6.2. Equivalences

Another issue is that in some cases, networks can be transformed from one case to another. For instance, Zhou and Carin [11] ([Section VB]) show that a Poisson gamma-gamma process construction is equivalent to a HDP construction with an independent Poisson-gamma on the total. Given that there are significant differences between the corresponding algorithms in this case, and there are many more in the literature, what other equivalences are there?

Normalising processes are conducted to convert a CRM into an NRMI and in some cases, independence between the parts yields an equivalence between the CRM form and the NRMI form augmented with a total. This has major implications to networks of such processes, presented in the following subsection, so the results are summarised here.

The first results are on discrete processes and are well-known, some for instance reproduced by Zhou and Carin [11].

Lemma 7.

(Equivalent processes) Let $\overrightarrow{\mu }$ be a probability vector (possibly infinite), and M be a constant positive background rate. Let $X={\sum }_{i=1}^{\infty }{x}_i$, the sum of entries of the non-negative integer vector $\overrightarrow{x}$. The following equivalences between (A) and (B) hold:

• Conditioning the PPP,

  $$(A)\overrightarrow{x}\sim PP(M\overrightarrow{\mu })(B)X\sim Poisson(M)and\overrightarrow{x}\sim MP(X,\overrightarrow{\mu }).$$
• NBP as a Poisson-gamma mixture,

  $$(A)\overrightarrow{x}\sim NBP(M,\rho ,\overrightarrow{\mu })(B)\overrightarrow{x}\sim PP\left(GP\left(M,\frac{1-\rho }{\rho },\overrightarrow{\mu }\right)\right).$$
• DCMP, given $X\in {\mathcal{N}}^{+}$, as a multinomial-Dirichlet mixture,

  $$(A)\overrightarrow{x}\sim DCMP(X,M\overrightarrow{\mu })(B)\overrightarrow{x}\sim MP(X,DP(M\overrightarrow{\mu })).$$
• Conditioning the NBP,

  $$(A)\overrightarrow{x}\sim NBP(M,\rho ,\overrightarrow{\mu })(B)X\sim NB(M,\rho ) and \overrightarrow{x}\sim DCMP(X,M\overrightarrow{\mu }).$$

The conditioned versions of the PPP and NBP are used to decompose a likelihood into a total count and the vector of counts for atoms, given the total. Notice, while the conditioned version of the PPP yields a likelihood where the normalised measure ($\overrightarrow{\mu }$) and its total (M) are independent, the same does not hold for the conditioned NBP.

6.3. Normalisation and Independence

On non-discrete processes, some independences apply.

Lemma 8.

(Normalised processes and independence) Let $\Lambda ={\sum }_{i=1}^{\infty }{\lambda }_i$, the sum of entries of the infinite non-negative real vector $\overrightarrow{\lambda }$. The following two pairs (A) and (B) are equivalent:

• For the gamma process:

   (A) 

  $\overrightarrow{\lambda }\sim GP(M,\beta )$;

   (B) 

  $\Lambda \sim ga(M,\beta )$ and $\overrightarrow{\lambda }/\Lambda \sim GEM(0,M)$, where Λ and $\overrightarrow{\lambda }/\Lambda$ are independent.

• For the generalised gamma process where $0<\alpha <1$, marginalising M

   (A) 

  $\overrightarrow{\lambda }\sim GGP(M,\alpha ,\beta )$ where $M\sim ga(\delta /\alpha ,{\beta }^{\alpha })$;

   (B) 

  $\Lambda \sim ga(\delta ,\beta )$ and $\overrightarrow{\lambda }/\Lambda \sim GEM(\alpha ,\delta )$, where Λ and $\overrightarrow{\lambda }/\Lambda$ are independent.

Moreover, the gamma process is the only case of such independence possible for pure NRMIs (this excludes the second case as it is marginalised).

Independence in the PYP case (represented as $\mathrm{GEM}(\alpha ,\delta )$ in the lemma) is shown by Pitman and Yor [28] ([Proposition 21]).

That the gamma process is the only independence case for CRMs and their NRMIs is a result by Perman et al. [27] ([Corollary 2.3]). This is equivalent to the neutrality of the Dirichlet distribution, again the only distribution on probability vectors exhibiting neutrality. Neutrality and independence in this case can be shown to be equivalent properties. Independence in both these cases is also a consequence of the fact that so-called sized-biased sampling for the cases is independent of the total [27,29]. Independence properties such as in Lemma 8 do not hold generally, as indeed sized-biased sampling is not generally independent of the total.

Lemma 9.

(Normalisation of other process) Let $\Lambda ={\sum }_{i=1}^{\infty }{\lambda }_i$, the sum of entries of the infinite vector $\overrightarrow{\lambda }$.

• For the generalised gamma process, if $\overrightarrow{\lambda }\sim GGP(M,\alpha ,\beta )$ then $\Lambda \sim Twe(\alpha ,M^{1/\alpha },\beta )$ and $\overrightarrow{\lambda }/\Lambda \sim NGG(\alpha ,M)$.
• For the stable process, if $\overrightarrow{\lambda }\sim staP(M,\alpha )$ then $\Lambda \sim pstable(\alpha ,M^{1/\alpha })$ and $\overrightarrow{\lambda }/\Lambda \sim PYP(\alpha ,0)$,

Λand $\overrightarrow{\lambda }/\Lambda$ are not independent in either case.

6.4. Modelling LDA Using HDP

Consider models for the HDP variant of LDA [3], called HDP-LDA, which has been the subject of extensive research. There is a wide variation in the literature of how these are to be represented by graphical model and for statistical inference. Figure 3 shows two equivalent models for HDP-LDA. Figure 3a gives the original model as formulated by Teh et al. [3], and Figure 3b shows the modification used here. Authors sometimes use a more complicated formulation in terms of the underlying stick-breaking model.

In this problem, there are D documents and $N_d$ words in each document for $d=1,\dots ,D$, where the words $w_{d,n}$ are modelled with an admixture. The probabilistic specification for the corresponding models are given in Figure 4.

Figure 4a shows the probabilistic specification with full base distributions. While this follows the theory directly, it is a fairly large departure from the original representation of LDA. The reformulation in Figure 4b is a direct analogue of the original representation of LDA with two modifications essential for the treatment of a HDP, discussed below as the root node and the non-root node.

The root node of the DP hierarchy is represented as a GEM, which generates the infinite vector. In practice, this can be represented using size-biased sampling [27] formulations, and in the simplest and popular cases this corresponds to stick-breaking methods [42]. In implementation, however, there is no need for this as posterior formulations for the processes are well understood and require no implicit ordering constraints as in stick-breaking.

Non-root nodes down the hierarchy are represented using their underlying infinitely divisible non-negative distribution, in this case the Dirichlet. Note, however, this extends the standard definition of a Dirichlet as the input parameter is an infinite dimensional vector. In implementation, this is no impediment as only a finite amount of data is ever dealt with, although it does require modelling the current number of non-empty dimensions. This can be readily handled using standard parametric techniques [6] or by using truncation [4].

Note Figure 4a also uses a nested construction [43] with the expression $\mathrm{DP}(c_{\alpha },\mathrm{Dirichlet}(c_{\beta }\overrightarrow{\beta }))$. Here a distribution, in this case a Dirichlet, but it could also be a GP, a DP or any other process, is used as the base distribution. This nesting construction is exactly what is needed to model matrix and tensor factorisation using hierarchical processes.

The nested, hierarchical equivalent to Figure 5b is as follows:

$$\begin{array}{ccc}\hfill \overrightarrow{\beta }& \sim & \mathrm{G}\mathrm{E}\mathrm{M}(d_{\beta },c_{\beta })\hfill \\ \hfill G_0& \sim & \mathrm{G}\mathrm{P}\left(c_{\alpha },1,\mathrm{PYD}\left(d_{\varphi },c_{\varphi },\overrightarrow{\beta }\right)\right)\hfill \\ \hfill G_d& \sim & \mathrm{G}\mathrm{P}(c_{\theta },s_{\theta },G_0)\hfill \\ \hfill {\varphi }_{d,n}& \sim & G_d\hfill \\ \hfill {\overrightarrow{n}}_d& \sim & \mathrm{P}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{o}\mathrm{n}\left({\overrightarrow{\varphi }}_d\right)\hfill \end{array}$$

The background word probabilities $\overrightarrow{\beta }$ are generated, then used as the base distribution for a PYD which then creates variants ${\overrightarrow{\varphi }}_k$ as each atom of the gamma process $G_0$. The mixture weights of $G_0$ correspond to $\overrightarrow{\alpha }$ from Figure 5b. Variants of this, $G_d$, are then created which modify the mixture weights $\overrightarrow{\alpha }$ but leave the atoms constant. So, $G_d$ is a weighted sum of the original ${\overrightarrow{\varphi }}_k$, as is the case in Figure 5b. This is very elegant, but Figure 5b better exposes the detail needed for implementation.

6.5. Example Equivalences with Non-Parametric LDA

Consider extending HDP-LDA to include a Pitman–Yor distribution on the word side. This model, termed NP-LDA Buntine and Mishra [4], has been demonstrated using a truncated approximation. To bring out equivalences, the multinomial form of the topic model is given, and both are defined in Figure 5.

The gamma scale parameter on ${\alpha }_0$ is one as it has an equivalent affect to $c_{\theta }$. So, it needs to be made a constant for identifiability. The equivalence is obtained by noting, from Lemmas 7 and 8, and many such results exist for the finite case, for instance by [44]. One can introduce a total rate for documents, ${\mathsf{\Theta }}_d$, and model the count, $N_d$, entirely independently:

$$\begin{array}{ccc}\hfill {\alpha }_0& \sim & \mathrm{g}\mathrm{a}\mathrm{m}\mathrm{m}\mathrm{a}(c_{\alpha },1)\hfill \\ \hfill {\mathsf{\Theta }}_d& \sim & \mathrm{g}\mathrm{a}\mathrm{m}\mathrm{m}\mathrm{a}(c_{\theta }{\alpha }_0,s_{\theta })\hfill \\ \hfill N_d& \sim & \mathrm{P}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{o}\mathrm{n}({\mathsf{\Theta }}_d).\hfill \end{array}$$

If the concentration parameters are estimated during learning, which is the common case, and recommended for topic models, then equivalence does not hold.

Experimental evidence [4] shows the following:

• The topic side, ${\overrightarrow{\theta }}_d$, is best not modelled using PYPs because experiments indicate that this gives no performance improvement. The non-Zipfian DPs work best, probably because of the smaller dimensions for number of topics.
• Modelling the word side, ${\overrightarrow{\varphi }}_k$, using PYPs systematically outperforms HDP-LDA by a moderate margin in perplexity and yields more explainable topics because the overall “background” words are separately modelled using $\overrightarrow{\beta }$.

Several model equivalences hold with regard to these kinds of models.

• The asymmetric-symmetric version of LDA [45] is a truncated version, not well understood in the community.
• The asymmetric-asymmetric version of LDA, evaluated by Wallach et al. [45], is a truncated version of the model in Figure 5a.
• Hierarchical Poisson factorisation [46] (HPF) is a non-parametric formulation of Poisson-gamma matrix factorisation using stick-breaking, and thus is equivalent to HDP-LDA above (when augmented with a gamma model of the total counts).
• Robust (negative binomial) Poisson factorisation by Zhou et al. [47] is related (ignoring some issue with hyperparameters) to bursty topic models by Doyle and Elkan [48], which has a non-parametric extension in Buntine and Mishra [4].

7. Conclusions

Discrete base distributions make CRMs behave like vectors of infinitely divisible distributions, where application is element-wise without the non-parametrics. So, the gamma process becomes an element-wise gamma distribution, and the generalised gamma process becomes an element-wise Tweedie distribution. This was presented in Lemma 2, Lemma 4 and Corollary 1 and accompanying tables. Similarly, discrete base distributions make NRMIs and related processes behave as normalised versions of the above, sharing some properties of the DP such as renormalisation. So, the HPYP becomes the PY distribution, whose form was developed in Section 5.

If closed forms for analysis of the infinitely divisible distributions do not exist, the generalised versions of Chinese restaurant process (CRP) sampling, given in Equation (17), can be used instead, including versions of the more recent, efficient collapsed samplers for CRPs [6,39], given in Equation (18). Similar formulations also appear in [14,15]. Note many of these quantities, for instance in Table 1, can be derived from the Laplace exponent of the CRM, so a convenient form of the distribution is not needed. The CRPs come about when unfolding the recursion that relates the cumulants of a distribution to the moments of the distribution, a simple result in basic statistics. In this way, known CRPs for the gamma process follow a general scheme that also applies for the generalised gamma process, the generalised beta process and others.

While most of these results follow fairly simply from general results in the non-parametric Bayesian community, some have not yet seen use in the Bayesian machine learning community.

As a specific example of hierarchical distributions, it was also shown in Section 5 that the NGG and PY distributions, for the case where discount $\alpha >0$ and concentration $\beta >0$, are behaving like normalised Tweedie variables, and for the case where concentration $\beta =0$ like normalised positive stable variables. Moments of the Tweedie distribution show how the standard hierarchical likelihood for the HPYP used to date [5,37] can be directly derived from this framework without considering non-parametric theory. A novel integral expression for the PY distribution for discount $\alpha >0$ and concentration $\beta \ge 0$ was also developed in Equation (21). This answers the question, “what is a hierarchical PYP”?

There are a rich number of variations of matrix factorisation and topic models that exist, for instance, see ([11] Table 1) with seven different versions of negative binomial matrix factorisation, and the software used in Buntine and Mishra [4] has seven different non-parametric versions of LDA. This is ignoring the extensions of the model where the problem is changed significantly: document segmentation [7], hierarchical topics [8], supervised topic models, etc., and these extensions no doubt have their own rich variety of versions and equivalences. Moreover, some of the known equivalences between processes, when applied in the hierarchical case, yield relationships between models and algorithms in the machine learning community that deserve further investigation, discussed in Section 6. This is confounded by the fact that variants are evaluated using significantly different methodologies; compare, for instance, topic modelling evaluation with recommender systems evaluation. It is an open question as to what other significant equivalences exist in the literature, and the implications this has to the algorithms one can use.