Probability Distribution on Full Rooted Trees

Abstract

The recursive and hierarchical structure of full rooted trees is applicable to statistical models in various fields, such as data compression, image processing, and machine learning. In most of these cases, the full rooted tree is not a random variable; as such, model selection to avoid overfitting is problematic. One method to solve this problem is to assume a prior distribution on the full rooted trees. This enables the optimal model selection based on Bayes decision theory. For example, by assigning a low prior probability to a complex model, the maximum a posteriori estimator prevents the selection of the complex one. Furthermore, we can average all the models weighted by their posteriors. In this paper, we propose a probability distribution on a set of full rooted trees. Its parametric representation is suitable for calculating the properties of our distribution using recursive functions, such as the mode, expectation, and posterior distribution. Although such distributions have been proposed in previous studies, they are only applicable to specific applications. Therefore, we extract their mathematically essential components and derive new generalized methods to calculate the expectation, posterior distribution, etc.

1. Introduction

1.1. Review of Literature and Motivation

Full rooted trees are utilized in various fields of study. For example, for text compression in information theory, a full rooted tree represents a set of contexts, which are strings of the most recent symbols at each time point, and it is known as a context tree [1]. In image processing, it represents a variable block-size segmentation, and it is known as quadtree block partitioning [2]. In machine learning, it represents a nonlinear function that comprises many conditional branches and is known as a decision tree [3]. In most of these studies, the rooted tree is not a random variable and serves as an index of a statistical model or function; i.e., one full rooted tree $\tau$ corresponds to one statistical model $p(x;\tau )$ or one function $f_{\tau }\left(x\right)$.

Full rooted trees’ recursive and hierarchical structures are suitable for representing complex statistical models or functional structures. For example, the expansion of the leaf nodes represents an increase in the contexts of a context tree [1], a division of a block on the image in quadtree partitioning [2], or the addition of a conditional branch in a decision tree [3]. The expressive ability and extensibility of full rooted trees render them widely applicable in various fields.

However, this hierarchical expressive capability causes a problem in tree selection, i.e., the selection of one statistical model or function. This is because the optimal tree under the criterion of the likelihood or squared loss for training data is inevitably the deepest one. This phenomenon is called overfitting in the field of machine learning. Therefore, most previous studies have applied a stopping rule for node expansion [2,3], introduced a normalization term into the objective function [4], or averaged the statistical models or the functions with some weights [1,4,5]. However, these algorithmic modifications are heuristic at times.

A theoretical way to solve this problem is to consider the full rooted tree as a random variable and assume a prior distribution on it. An appropriate prior distribution provides a unified method for selecting one full rooted tree or combining them based on Bayes decision theory (see, e.g., [6]). Although Bayes decision theory is typically applied to statistical models with unknown continuous parameters, it is also applicable to statistical models with unknown discrete random variables, such as full rooted trees (see, e.g., [7]). By assigning a high prior probability to a shallow tree and a low prior probability to a deep tree, we can avoid the complex statistical model corresponding to a deep tree.

As mentioned above, most previous studies regard a full rooted tree as a non-stochastic variable. However, a few studies adopted the above-mentioned approach. In terms of text compression, the complete Bayesian interpretation of the context tree weighting method was first investigated by the authors of [8]. Not only the theory, but also the associated algorithm have been improved during the decade they were first investigated (see, e.g., [9]). Moreover, similar results obtained from rich real data analysis have been reported recently [10,11] (note that the prior form reported in [10,11] is extremely restricted and cannot be updated as a posterior, in contrast to that reported in [8,9]). In image processing, the authors of [12] were the first to regard the quadtree as a stochastic model, and its optimal estimation was derived under the Bayesian criteria. In machine learning, the authors of [13] redefined the decision tree as a stochastic generative model and improved most tree weighting methods (e.g., [5]).

However, these studies depend on specific data or generative models. This might have been the reason that more than 25 years passed before the first study [8] pertaining to text compression was applied to image processing [12] and machine learning [13].

1.2. The Objective of This Study

Therefore, we separate the mathematically essential component of the discussion from the modifiable component based on specific data or the generative model. Mathematically, a tree is defined as a connected graph without cycles (see, e.g., [14]). A rooted tree is a tree that has one node known as a root node, and a full rooted tree is a rooted tree in which each inner node has the same number of child nodes. Subsequently, we can define a finite set of subtrees of a full rooted tree. This full rooted tree, which contains all the subtrees in the finite set, is denoted as a base tree herein.

A trivial method to define a probability distribution for this set is to assign occurrence probabilities to all subtrees and regard these values as parameters. In other words, we can define the categorical distribution for the finite set of subtrees of the base tree. However, this definition requires the same number of parameters as the subtrees, which increases in a doubly exponential order with the depth of the base tree.

Therefore, we propose an efficient parametric representation of the probability distribution of a set of subtrees. It is suitable for the recursive structure of full rooted trees and allows the number of parameters to be reduced. Moreover, it enables us to calculate its mode, expectation, posterior distribution, etc., using recursive functions. Therefore, it is efficient from a computational viewpoint. Furthermore, we expect these recursive functions to be effective as a subroutine of the variational Bayesian method and the Markov chain Monte Carlo method in hierarchical Bayesian modeling (see, e.g., [15]).

Strictly speaking, our distribution has already been proposed independently in source coding [8], image processing [12], and machine learning [13], as mentioned above. The substantial novelty of our study is the extraction of the essence from the previous discussion, which depends on the applicational objects, and its representation as a clear mathematical theory. This theoretically expands the potential application of probability distributions on full rooted trees. Subsequently, we derive new generalized methods to evaluate the characteristics of the probability distribution of full rooted trees, which could not be derived in previous studies pertaining to real-world applications. More precisely, only Theorems 1 and 3 and Corollary 2 has been used in previous studies. Meanwhile, the other methods expand the possibility of applying the probability distribution on full rooted trees.

1.3. Organization of This Paper

The remainder of this paper is organized as follows: In Section 2, we present the notation used herein. In Section 3, we define the prior for full rooted trees. In Section 4, we describe the algorithms for calculating the properties of the proposed distribution—e.g., a marginal distribution for each node, an efficient calculation of the expectation, mode, and the posterior distribution. In Section 5, we discuss the usefulness of our distribution in statistical decision theory and hierarchical Bayesian modeling. In Section 6, we propose some future work. In Section 7, we conclude the paper.

2. Notations Used for Full Rooted Trees

In this section, we define notation for the rooted trees. It is shown in Figure 1. Let $k\in \mathbb{N}$ denote the maximum number of child nodes and $d_{\max }\in \mathbb{N}$ denote the maximum depth. Let ${\tau }_{\mathrm{p}}=({\mathcal{V}}_{\mathrm{p}},{\mathcal{E}}_{\mathrm{p}})$ denote the perfect (“perfect” means that all inner nodes have exactly k children and all leaf nodes have the same depth) k-ary rooted tree whose depth is $d_{\max }$, and root node is $v_{\lambda }$. ${\mathcal{V}}_{\mathrm{p}}$ and ${\mathcal{E}}_{\mathrm{p}}$ denote the sets of the nodes and edges of it, respectively. Then, let ${\mathcal{I}}_{\mathrm{p}}\subset {\mathcal{V}}_{\mathrm{p}}$ and ${\mathcal{L}}_{\mathrm{p}}\subset {\mathcal{V}}_{\mathrm{p}}$ denote the set of inner nodes and the set of leaf nodes of ${\tau }_{\mathrm{p}}$, respectively. For each node $v\in {\mathcal{V}}_{\mathrm{p}}$, ${\mathrm{Ch}}_{\mathrm{p}}\left(v\right)\subset {\mathcal{V}}_{\mathrm{p}}$ denotes the set of child nodes of v on ${\tau }_{\mathrm{p}}$. Notation about the relation between two nodes $v,v^{\prime }\in {\mathcal{V}}_{\mathrm{p}}$ is as follows. Let $v\succ v^{\prime }$ denote that v is an ancestor node of $v^{\prime }$ ($v^{\prime }$ is a descendant node of v), $v⪰v^{\prime }$ denote that v is an ancestor node of $v^{\prime }$ or $v^{\prime }$ itself ($v^{\prime }$ is a descendant node of v or v itself), $\mathrm{An}\left(v\right):=\{v^{\prime }\in {\mathcal{V}}_{\mathrm{p}}\mid v^{\prime }\succ v\}$, and ${\mathrm{De}}_{\mathrm{p}}\left(v\right):=\{v^{\prime }\in {\mathcal{V}}_{\mathrm{p}}\mid v\succ v^{\prime }\}$.

Subsequently, we consider rooted subtrees of ${\tau }_{\mathrm{p}}$ in which their root nodes are the same as $v_{\lambda }$ and all inner nodes have exactly k children. They are called full rooted subtrees and ${\tau }_{\mathrm{p}}$ is called a base tree. Let $\mathcal{T}$ denote the set of all full rooted subtrees of ${\tau }_{\mathrm{p}}$. Let ${\mathcal{V}}_{\tau }$ and ${\mathcal{E}}_{\tau }$ denote the set of the nodes and the edges of $\tau \in \mathcal{T}$, respectively. Let ${\mathcal{I}}_{\tau }\subset {\mathcal{V}}_{\tau }$ and ${\mathcal{L}}_{\tau }\subset {\mathcal{V}}_{\tau }$ denote the set of the inner nodes and the set of leaf nodes of $\tau \in \mathcal{T}$, respectively.

3. Definition of Probability Distribution on Full Rooted Subtrees

In this section, we define a probability distribution of full rooted subtrees $\mathcal{T}$. Let T denote the random variable on $\mathcal{T}$ and $\tau$ denote its realization.

Definition 1.

For ${\left({\alpha }_v\right)}_{v\in {\mathcal{V}}_{\mathrm{p}}}\in {[0,1]}^{|{\mathcal{V}}_{\mathrm{p}}|}$, we define the probability distribution $p\left(\tau \right)$ on $\mathcal{T}$ as below.

$$\begin{array}{c}\hfill p\left(\tau \right):=\prod _{v\in {\mathcal{I}}_{\tau }}{\alpha }_v\prod _{v^{\prime }\in {\mathcal{L}}_{\tau }}(1-{\alpha }_{v^{\prime }}),\end{array}$$

(1)

where ${\alpha }_v=0$ for $v\in {\mathcal{L}}_{\mathrm{p}}$.

Intuitively, ${\alpha }_v$ represents the probability that v has child nodes under the condition that v is contained in the tree (it will be proved as a theoretical fact in Remark 2). Therefore, the occurrence probability of a full rooted subtree exponentially decays as its depth increases.

Example 1.

An example of the probability distribution on full rooted subtrees for $k=2$ and $d_{\max }=2$ is shown in Figure 2.

Theorem 1.

The quantity $p\left(\tau \right)$ defined as in (1) fulfills the condition of the probability distribution, that is, ${\sum }_{\tau \in \mathcal{T}}p\left(\tau \right)=1$.

Example 2.

Before the proof of Theorem 1 for the general case, we describe an example where $d_{\max }=2$ and $k=2$ (see Figure 2). First, we factorize the sum as below.

$$\begin{array}{cc}\hfill & {\displaystyle \sum _{\tau \in \mathcal{T}}}p\left(\tau \right)\hfill \\ \hfill & =(1-{\alpha }_{v_{\lambda }})\hfill \\ \hfill & +{\alpha }_{v_{\lambda }}(1-{\alpha }_{v_0})(1-{\alpha }_{v_1})\hfill \\ \hfill & +{\alpha }_{v_{\lambda }}{\alpha }_{v_0}(1-{\alpha }_{v_{00}})(1-{\alpha }_{v_{01}})(1-{\alpha }_{v_1})\hfill \\ \hfill & +{\alpha }_{v_{\lambda }}(1-{\alpha }_{v_0}){\alpha }_{v_1}(1-{\alpha }_{v_{10}})(1-{\alpha }_{v_{11}})\hfill \\ \hfill & +{\alpha }_{v_{\lambda }}{\alpha }_{v_0}(1-{\alpha }_{v_{00}})(1-{\alpha }_{v_{01}}){\alpha }_{v_1}(1-{\alpha }_{v_{10}})(1-{\alpha }_{v_{11}})\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill & =(1-{\alpha }_{v_{\lambda }})\hfill \\ \hfill & +{\alpha }_{v_{\lambda }}\{(1-{\alpha }_{v_0})(1-{\alpha }_{v_1})\hfill \\ \hfill & +{\alpha }_{v_0}(1-{\alpha }_{v_{00}})(1-{\alpha }_{v_{01}})(1-{\alpha }_{v_1})\hfill \\ \hfill & +(1-{\alpha }_{v_0}){\alpha }_{v_1}(1-{\alpha }_{v_{10}})(1-{\alpha }_{v_{11}})\hfill \\ \hfill & +{\alpha }_{v_0}(1-{\alpha }_{v_{00}})(1-{\alpha }_{v_{01}}){\alpha }_{v_1}(1-{\alpha }_{v_{10}})(1-{\alpha }_{v_{11}})\}\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill & =(1-{\alpha }_{v_{\lambda }})\hfill \\ \hfill & +{\alpha }_{v_{\lambda }}\{(1-{\alpha }_{v_0})\left[(1-{\alpha }_{v_1})+{\alpha }_{v_1}(1-{\alpha }_{v_{10}})(1-{\alpha }_{v_{11}})\right]\hfill \\ \hfill & +{\alpha }_{v_0}(1-{\alpha }_{v_{00}})(1-{\alpha }_{v_{01}})\left[(1-{\alpha }_{v_1})+{\alpha }_{v_1}(1-{\alpha }_{v_{10}})(1-{\alpha }_{v_{11}})\right]\}\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill & =(1-{\alpha }_{v_{\lambda }})\hfill \\ \hfill & +{\alpha }_{v_{\lambda }}\{\left[(1-{\alpha }_{v_0})+{\alpha }_{v_0}(1-{\alpha }_{v_{00}})(1-{\alpha }_{v_{01}})\right]\hfill \\ \hfill & \times \left[(1-{\alpha }_{v_1})+{\alpha }_{v_1}(1-{\alpha }_{v_{10}})(1-{\alpha }_{v_{11}})\right]\}.\hfill \end{array}$$

(5)

Here, ${\alpha }_{v_{00}}={\alpha }_{v_{01}}={\alpha }_{v_{10}}={\alpha }_{v_{11}}=0$ since $v_{00},v_{01},v_{10},v_{11}\in {\mathcal{L}}_{\mathrm{p}}$. Then,

$$\begin{array}{cc}\hfill \sum _{\tau \in \mathcal{T}}p\left(\tau \right)& =(1-{\alpha }_{v_{\lambda }})+{\alpha }_{v_{\lambda }}\left\{\left[(1-{\alpha }_{v_0})+{\alpha }_{v_0}\right]·\left[(1-{\alpha }_{v_1})+{\alpha }_{v_1}\right]\right\}\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill & =(1-{\alpha }_{v_{\lambda }})+{\alpha }_{v_{\lambda }}\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill & =1.\hfill \end{array}$$

(8)

The general proof of Theorem 1 is in the following. That also consists of two parts, namely, factorization and substitution. We will first prove Lemma 1, which is the essential lemma, since it is not used only in the proof of Theorem 1 but also in the proof of other theorems later.

Lemma 1.

Let $F:\mathcal{T}\to \mathbb{R}$ be a real-valued function on the set $\mathcal{T}$ of the full rooted subtrees of the base tree ${\tau }_{\mathrm{p}}$. If F has the form

$$\begin{array}{c}\hfill F\left(\tau \right)=\prod _{v\in {\mathcal{I}}_{\tau }}G\left(v\right)\prod _{v^{\prime }\in {\mathcal{L}}_{\tau }}H\left(v^{\prime }\right),\end{array}$$

(9)

where $G:{\mathcal{V}}_{\mathrm{p}}\to \mathbb{R}$ and $H:{\mathcal{V}}_{\mathrm{p}}\to \mathbb{R}$ are real-valued functions on ${\mathcal{V}}_{\mathrm{p}}$, then the summation ${\sum }_{\tau \in \mathcal{T}}F\left(\tau \right)$ can be recursively decomposed as follows.

$$\begin{array}{c}\hfill \sum _{\tau \in \mathcal{T}}F\left(\tau \right)=\varphi \left(v_{\lambda }\right),\end{array}$$

(10)

where $\varphi :{\mathcal{V}}_{\mathrm{p}}\to \mathbb{R}$ is defined as

$$\begin{array}{c}\hfill \varphi \left(v\right):=\left\{\begin{array}{cc}H\left(v\right),\hfill & v\in {\mathcal{L}}_{\mathrm{p}},\hfill \\ H\left(v\right)+G\left(v\right)\prod _{v^{\prime }\in {\mathrm{Ch}}_{\mathrm{p}}\left(v\right)}\varphi \left(v^{\prime }\right),\hfill & v\in {\mathcal{I}}_{\mathrm{p}}.\hfill \end{array}\right.\end{array}$$

(11)

Proof. 

Let $\left[v_{\lambda }\right]$ denote the tree that consists of only the root node $v_{\lambda }$ of the base tree ${\tau }_{\mathrm{p}}$. Then, the cases of the sum is divided as follows.

$$\begin{array}{cc}\hfill \sum _{\tau \in \mathcal{T}}F\left(\tau \right)& =\sum _{\tau \in \mathcal{T}}\prod _{v\in {\mathcal{I}}_{\tau }}G\left(v\right)\prod _{v^{\prime }\in {\mathcal{L}}_{\tau }}H\left(v^{\prime }\right)\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill & =\prod _{v\in {\mathcal{I}}_{\left[v_{\lambda }\right]}}G\left(v\right)\prod _{v^{\prime }\in {\mathcal{L}}_{\left[v_{\lambda }\right]}}H\left(v^{\prime }\right)+\sum _{\tau \in \mathcal{T}\setminus \left\{\left[v_{\lambda }\right]\right\}}\prod _{v\in {\mathcal{I}}_{\tau }}G\left(v\right)\prod _{v^{\prime }\in {\mathcal{L}}_{\tau }}H\left(v^{\prime }\right)\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill & =H\left(v_{\lambda }\right)+\sum _{\tau \in \mathcal{T}\setminus \left\{\left[v_{\lambda }\right]\right\}}\prod _{v\in {\mathcal{I}}_{\tau }}G\left(v\right)\prod _{v^{\prime }\in {\mathcal{L}}_{\tau }}H\left(v^{\prime }\right)\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill & =H\left(v_{\lambda }\right)+G\left(v_{\lambda }\right)\sum _{\tau \in \mathcal{T}\setminus \left\{\left[v_{\lambda }\right]\right\}}\prod _{v\in {\mathcal{I}}_{\tau }\setminus \left\{v_{\lambda }\right\}}G\left(v\right)\prod _{v^{\prime }\in {\mathcal{L}}_{\tau }}H\left(v^{\prime }\right),\hfill \end{array}$$

(15)

where (14) is because $\left[v_{\lambda }\right]$ has no inner node and its leaf node is only $v_{\lambda }$; (15) is because every tree in $\mathcal{T}\setminus \left[v_{\lambda }\right]$ has $v_{\lambda }$ and the corresponding factor $G\left(v_{\lambda }\right)$.

We have already pointed out that each tree $\tau \in \mathcal{T}\setminus \left\{\left[v_{\lambda }\right]\right\}$ contains $v_{\lambda }$ as its inner node. The other structure of $\tau$ is determined by the shape of k subtrees whose root nodes are the child nodes of $v_{\lambda }$ (see Figure 3). We index them in an appropriate order. Then, let $v_{\lambda i}$ denote the i-th child node of $v_{\lambda }$ for $i\in \{0,1,\dots ,k-1\}$; i.e., $\{v_{\lambda 0},\dots ,v_{\lambda k-1}\}={\mathrm{Ch}}_{\mathrm{p}}\left(v_{\lambda }\right)$. Let ${\mathcal{T}}^{v_{\lambda i}}$ denote the set of subtrees whose root node is $v_{\lambda i}$. Then, there is a natural bijection from $\mathcal{T}\setminus \left\{\left[v_{\lambda }\right]\right\}$ to ${\mathcal{T}}^{v_{\lambda 0}}\times \cdots \times {\mathcal{T}}^{v_{\lambda k-1}}$.

Therefore, the summation of (15) is further factorized. Consequently, we have

$$\begin{array}{cc}\hfill \sum _{\tau \in \mathcal{T}}& F\left(\tau \right)=H\left(v_{\lambda }\right)+G\left(v_{\lambda }\right)\hfill \end{array}$$

$$\begin{array}{cc}\hfill & \times \sum _{({\tau }_0,\cdots ,{\tau }_{k-1})\in {\mathcal{T}}^{v_{\lambda 0}}\times \cdots \times {\mathcal{T}}^{v_{\lambda k-1}}}\left[\prod _{v\in {\mathcal{I}}_{{\tau }_0}}G\left(v\right)\prod _{v^{\prime }\in {\mathcal{L}}_{{\tau }_0}}H\left(v^{\prime }\right)\dots \prod _{v\in {\mathcal{I}}_{{\tau }_{k-1}}}G\left(v\right)\prod _{v^{\prime }\in {\mathcal{L}}_{{\tau }_{k-1}}}H\left(v^{\prime }\right)\right]\hfill \\ \hfill =& H\left(v_{\lambda }\right)+G\left(v_{\lambda }\right)\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill & \times \sum _{{\tau }_0\in {\mathcal{T}}^{v_{\lambda 0}}}\cdots \sum _{{\tau }_{k-1}\in {\mathcal{T}}^{v_{\lambda k-1}}}\left[\prod _{v\in {\mathcal{I}}_{{\tau }_0}}G\left(v\right)\prod _{v^{\prime }\in {\mathcal{L}}_{{\tau }_0}}H\left(v^{\prime }\right)\cdots \prod _{v\in {\mathcal{I}}_{{\tau }_{k-1}}}G\left(v\right)\prod _{v^{\prime }\in {\mathcal{L}}_{{\tau }_{k-1}}}H\left(v^{\prime }\right)\right]\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill =& H\left(v_{\lambda }\right)+G\left(v_{\lambda }\right)\prod _{i=0}^{k-1}\sum _{\tau \in {\mathcal{T}}^{v_{\lambda i}}}\prod _{v\in {\mathcal{I}}_{\tau }}G\left(v\right)\prod _{v^{\prime }\in {\mathcal{L}}_{\tau }}H\left(v^{\prime }\right).\hfill \end{array}$$

(18)

Then, from (12) and (18), we have

$$\begin{array}{c}\hfill \underset{\left(a\right)}{\underbrace{\sum _{\tau \in \mathcal{T}}\prod _{v\in {\mathcal{I}}_{\tau }}G\left(v\right)\prod _{v^{\prime }\in {\mathcal{L}}_{\tau }}H\left(v^{\prime }\right)}}=H\left(v_{\lambda }\right)+G\left(v_{\lambda }\right)\prod _{i=0}^{k-1}\underset{\left(b\right)}{\underbrace{\sum _{\tau \in {\mathcal{T}}^{v_{\lambda i}}}\prod _{v\in {\mathcal{I}}_{\tau }}G\left(v\right)\prod _{v^{\prime }\in {\mathcal{L}}_{\tau }}H\left(v^{\prime }\right)}}.\end{array}$$

(19)

The underbraced parts $\left(a\right)$ and $\left(b\right)$ have the same structure except for the depth of the root node of the subtree. Therefore, $\left(b\right)$ can be decomposed in a similar manner from (12) to (18). We can continue this decomposition to the leaf nodes.

Then, let ${\mathcal{T}}^v$ denote the set of subtrees whose root node is $v\in {\mathcal{V}}_{\mathrm{p}}$ in general; i.e., we define a notion similar to ${\mathcal{T}}^{v_{\lambda i}}$ for not only $v_{\lambda 0},v_{\lambda 1},\dots ,v_{\lambda k-1}$ but also any other nodes $v\in {\mathcal{V}}_{\mathrm{p}}$. Finally, we have an alternative definition of $\varphi \left(v\right):{\mathcal{V}}_{\mathrm{p}}\to \mathbb{R}$, which is equivalent to (11).

$$\begin{array}{c}\hfill \varphi \left(v\right):=\sum _{\tau \in {\mathcal{T}}^v}\prod _{v^{\prime }\in {\mathcal{I}}_{\tau }}G\left(v^{\prime }\right)\prod _{v^{″}\in {\mathcal{L}}_{\tau }}H\left(v^{″}\right).\end{array}$$

(20)

The equivalence is confirmed by substituting it into both sides of (19). Therefore, Lemma 1 is proved. □

Then, the proof of Theorem 1 is as follows.

Proof of Theorem 1.

Using Lemma 1, we can divide the cases of the sum and factorize the common terms of ${\sum }_{\tau \in \mathcal{T}}p\left(\tau \right)$ in the following recursive manner.

$$\begin{array}{c}\hfill \sum _{\tau \in \mathcal{T}}p\left(\tau \right)=\varphi \left(v_{\lambda }\right),\end{array}$$

(21)

where

$$\begin{array}{c}\hfill \varphi \left(v\right):=\left\{\begin{array}{cc}1-{\alpha }_v,\hfill & v\in {\mathcal{L}}_{\mathrm{p}},\hfill \\ (1-{\alpha }_v)+{\alpha }_v\prod _{v^{\prime }\in {\mathrm{Ch}}_{\mathrm{p}}\left(v\right)}\varphi \left(v^{\prime }\right),\hfill & v\in {\mathcal{I}}_{\mathrm{p}}.\hfill \end{array}\right.\end{array}$$

(22)

Then, we prove $\varphi \left(v\right)=1$ for any node $v\in {\mathcal{V}}_{\mathrm{p}}$ by structural induction. For any leaf node $v\in {\mathcal{L}}_{\mathrm{p}}$, ${\alpha }_v=0$ from Definition 1. Therefore,

$$\begin{array}{c}\hfill \varphi \left(v\right)=1-{\alpha }_v=1,v\in {\mathcal{L}}_{\mathrm{p}}.\end{array}$$

(23)

For any inner node $v\in {\mathcal{I}}_{\mathrm{p}}$, assuming $\varphi \left(v^{\prime }\right)=1$ as the induction hypothesis for any descendant nodes $v^{\prime }\in {\mathrm{De}}_{\mathrm{p}}\left(v\right)$,

$$\begin{array}{cc}\hfill \varphi \left(v\right)& =(1-{\alpha }_v)+{\alpha }_v\prod _{v^{\prime }\in {\mathrm{Ch}}_{\mathrm{p}}\left(v\right)}\varphi \left(v^{\prime }\right)\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill & =(1-{\alpha }_v)+{\alpha }_v\prod _{v^{\prime }\in {\mathrm{Ch}}_{\mathrm{p}}\left(v\right)}1\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill & =1.\hfill \end{array}$$

(26)

Therefore, ${\sum }_{\tau \in \mathcal{T}}p\left(\tau \right)=\varphi \left(v_{\lambda }\right)=1$ since $v_{\lambda }$ is also in ${\mathcal{V}}_{\mathrm{p}}$. □

Remark 1.

Although Theorem 1 is also proved in [12,13], we extract the essential part of them as Lemma 1. In [10,11], a restricted case of Theorem 1 is proved, in which ${\alpha }_v$ has a common value for all $v\in {\mathcal{I}}_{\mathrm{p}}$.

4. Properties of Probability Distribution on Full Rooted Subtrees

In this section, we describe properties of the probability distribution on full rooted subtrees and methods to calculate them. All the proofs are in Appendix A. Note that the motivation and usefulness of Conditions 1, 2, 3, and 4 in this section will be described in Section 5.

4.1. Probability of Events on Nodes

At the beginning, we explain why $v\in {\mathcal{V}}_T$ determines a probabilistic event. We consider any $v\in {\mathcal{V}}_{\mathrm{p}}$ is given as a non-stochastic constant and fixed. After that, a full rooted subtree is randomly chosen according to the probability distribution proposed in Section 3. Then, ${\mathcal{V}}_T$ sometimes contains v and sometimes not, depending on the realization $\tau$ of random variable T. Therefore, $v\in {\mathcal{V}}_T$ determines a probabilistic event on $p\left(\tau \right)$. Although the probability of such events are trivially represented as ${\sum }_{\tau \in \mathcal{T}}I\{v\in {\mathcal{V}}_{\tau }\}p\left(\tau \right)$, where $I\{·\}$ denotes the indicator function, we derive computationally efficient forms without the summation about $\tau$ in the following.

Theorem 2.

For any $v\in {\mathcal{V}}_{\mathrm{p}}$, we have the following:

$$\begin{array}{cc}\hfill & \mathrm{Pr}\{v\in {\mathcal{V}}_T\}=\prod _{v^{\prime }\in \mathrm{An}\left(v\right)}{\alpha }_{v^{\prime }},\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill & \mathrm{Pr}\{v\in {\mathcal{I}}_T\}={\alpha }_v\prod _{v^{\prime }\in \mathrm{An}\left(v\right)}{\alpha }_{v^{\prime }},\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill & \mathrm{Pr}\{v\in {\mathcal{L}}_T\}=(1-{\alpha }_v)\prod _{v^{\prime }\in \mathrm{An}\left(v\right)}{\alpha }_{v^{\prime }}.\hfill \end{array}$$

(29)

Example 3.

Let us consider $p\left(\tau \right)$ shown in Figure 2. Trivially, $\mathrm{Pr}\{v_{01}\in {\mathcal{V}}_T\}$, $\mathrm{Pr}\{v_1\in {\mathcal{I}}_T\}$, and $\mathrm{Pr}\{v_0\in {\mathcal{L}}_T\}$ are calculated as

$$\begin{array}{cc}\hfill & \mathrm{Pr}\{v_{01}\in {\mathcal{V}}_T\}=p\left({\tau }_2\right)+p\left({\tau }_4\right)=0.28,\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill & \mathrm{Pr}\{v_1\in {\mathcal{I}}_T\}=p\left({\tau }_3\right)+p\left({\tau }_4\right)=0.56,\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill & \mathrm{Pr}\{v_0\in {\mathcal{L}}_T\}=p\left({\tau }_1\right)+p\left({\tau }_3\right)=0.42.\hfill \end{array}$$

(32)

The same probabilities are also given by

$$\begin{array}{cc}\hfill & \mathrm{Pr}\{v_{01}\in {\mathcal{V}}_T\}={\alpha }_{v_{\lambda }}{\alpha }_{v_0}=0.28,\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill & \mathrm{Pr}\{v_1\in {\mathcal{I}}_T\}={\alpha }_{v_{\lambda }}{\alpha }_{v_1}=0.56,\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill & \mathrm{Pr}\{v_0\in {\mathcal{L}}_T\}={\alpha }_{v_{\lambda }}(1-{\alpha }_{v_1})=0.42.\hfill \end{array}$$

(35)

Remark 2.

Probabilities of many other events on nodes are derived from Theorem 2. For example,

$$\begin{array}{cc}\hfill \mathrm{Pr}\{v\in {\mathcal{I}}_T\mid v\in {\mathcal{V}}_T\}& =\frac{\mathrm{Pr}\{(v\in {\mathcal{I}}_T)\wedge (v\in {\mathcal{V}}_T)\}}{\mathrm{Pr}\{v\in {\mathcal{V}}_T\}}\hfill \end{array}$$

(36)

$$\begin{array}{cc}\hfill & =\frac{\mathrm{Pr}\{v\in {\mathcal{I}}_T\}}{\mathrm{Pr}\{v\in {\mathcal{V}}_T\}}\hfill \end{array}$$

(37)

$$\begin{array}{cc}\hfill & ={\alpha }_v.\hfill \end{array}$$

(38)

4.2. Mode

We describe an algorithm to find the mode of $p\left(\tau \right)$ with $O\left(k^{d_{\max }+1}\right)$ computational cost. ($O(·)$ denotes the Big-O notation, i.e., $f\left(n\right)=O\left(g\right(n\left)\right)$ means that ${}^{\exists }k>0{,}^{\exists }{n}_0>0{,}^{\forall }n>n_0,\left|f\left(n\right)\right|\le k·g\left(n\right)$). Note that the size of search space $\mathcal{T}$ is of the order of $\mathsf{\Omega }\left(2^{k^{d_{\max }-2}}\right)$ in general. ($\mathsf{\Omega }(·)$ denotes the Big-Omega notation in complexity theory; i.e., $f\left(n\right)=\mathsf{\Omega }\left(g\right(n\left)\right)$ means that ${}^{\exists }k>0{,}^{\exists }{n}_0>0{,}^{\forall }n>n_0,f\left(n\right)\ge k·g\left(n\right)$. $\left|\mathcal{T}\right|=\mathsf{\Omega }\left(2^{k^{d_{\max }-2}}\right)$ is proved by substituting $G\left(v\right)\equiv H\left(v\right)\equiv 1$ in Lemma 1). First, by replacing all the sum in the proof of Lemma 1 with the maximum, we can derive the following recursive expression of ${max}_{\tau \in \mathcal{T}}p\left(\tau \right)$.

Proposition 1.

$$\begin{array}{c}\hfill \underset{\tau \in \mathcal{T}}{max}p\left(\tau \right)=\psi \left(v_{\lambda }\right),\end{array}$$

(39)

where

$$\begin{array}{c}\hfill \psi \left(v\right):=\left\{\begin{array}{cc}1,\hfill & v\in {\mathcal{L}}_{\mathrm{p}},\hfill \\ max\left\{1-{\alpha }_v,{\alpha }_v\prod _{v^{\prime }\in {\mathrm{Ch}}_{\mathrm{p}}\left(v\right)}\psi \left(v^{\prime }\right)\right\},\hfill & v\in {\mathcal{I}}_{\mathrm{p}}.\hfill \end{array}\right.\end{array}$$

(40)

Example 4.

On $p\left(\tau \right)$ shown in Figure 2, the maximum probability is $p\left({\tau }_3\right)=0.336$. It is also calculated as follows.

$$\begin{array}{cc}\hfill & max\{1-{\alpha }_{v_{\lambda }},{\alpha }_{v_{\lambda }}max\{1-{\alpha }_{v_0},{\alpha }_{v_0}\}·max\{1-{\alpha }_{v_1},{\alpha }_{v_1}\}\}\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill & =max\{0.3,0.7max\{0.6,0.4\}·max\{0.2,0.8\left\}\right\}\hfill \end{array}$$

(42)

$$\begin{array}{cc}\hfill & =max\{0.3,0.336\}=0.336.\hfill \end{array}$$

(43)

In addition, we define a flag variable ${\delta }_v\in \{0,1\}$ as follows.

Definition 2.

For any $v\in {\mathcal{V}}_{\mathrm{p}}$, we define

$$\begin{array}{c}\hfill {\delta }_v:=\left\{\begin{array}{cc}1,\hfill & 1-{\alpha }_v<{\alpha }_v\prod _{v^{\prime }\in {\mathrm{Ch}}_{\mathrm{p}}\left(v\right)}\psi \left(v^{\prime }\right),\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(44)

We can calculate $\psi \left(v\right)$ and ${\delta }_v$ simultaneously. Then, the mode of $p\left(\tau \right)$ is given by the following proposition.

Proposition 2.

$\arg {max}_{\tau \in \mathcal{T}}p\left(\tau \right)$ is identified as the tree that satisfies

$$\begin{array}{c}\hfill v\in {\mathcal{I}}_{\tau }\Rightarrow {\delta }_v=1,\end{array}$$

(45)

$$\begin{array}{c}\hfill v\in {\mathcal{L}}_{\tau }\Rightarrow {\delta }_v=0.\end{array}$$

(46)

Then, the following theorem holds.

Theorem 3.

The mode of $p\left(\tau \right)$ can be found via backtracking search from $v_{\lambda }$ after the calculation of $\psi \left(v\right)$ and ${\delta }_v$. It is detailed in Algorithm A1 in Appendix B.

Remark 3.

In [10,11], Papageorgiou et al. proposed the same algorithm as Algorithm A1 and an algorithm to find multiple most likely trees on the background of text compression.

Example 5.

See Figure 4. The parameters are the same as those in Figure 2. The mode ${\tau }_3$ is found by the proposed algorithm.

4.3. Expectation

Let $f:\mathcal{T}\to \mathbb{R}$ denote a real-valued function on $\mathcal{T}$. Here, we discuss sufficient conditions of f, under which the following expectation can be calculated efficiently with $O\left(k^{d_{\max }+1}\right)$ cost.

$$\begin{array}{c}\hfill \mathbb{E}\left[f\left(T\right)\right]:=\sum _{\tau \in \mathcal{T}}f\left(\tau \right)p\left(\tau \right).\end{array}$$

(47)

Note that the size of $\mathcal{T}$ is of the order of $\mathsf{\Omega }\left(2^{k^{d_{\max }-2}}\right)$ in general.

Condition 1.

There exist $g:{\mathcal{V}}_{\mathrm{p}}\to \mathbb{R}$ and $h:{\mathcal{V}}_{\mathrm{p}}\to \mathbb{R}$ such that

$$\begin{array}{c}\hfill f\left(\tau \right)=\prod _{v\in {\mathcal{I}}_{\tau }}g\left(v\right)\prod _{v^{\prime }\in {\mathcal{L}}_{\tau }}h\left(v^{\prime }\right).\end{array}$$

(48)

Theorem 4.

Under Condition 1, we define a recursive function $\varphi :{\mathcal{V}}_{\mathrm{p}}\to \mathbb{R}$ as

$$\begin{array}{c}\hfill \varphi \left(v\right):=\left\{\begin{array}{cc}h\left(v\right),\hfill & v\in {\mathcal{L}}_{\mathrm{p}},\hfill \\ (1-{\alpha }_v)h\left(v\right)+{\alpha }_{v}g\left(v\right)\prod _{v^{\prime }\in {\mathrm{Ch}}_{\mathrm{p}}\left(v\right)}\varphi \left(v^{\prime }\right),\hfill & v\in {\mathcal{I}}_{\mathrm{p}}.\hfill \end{array}\right.\end{array}$$

(49)

Then, we can calculate $\mathbb{E}\left[f\right(T\left)\right]$ as $\mathbb{E}\left[f\left(T\right)\right]=\varphi \left(v_{\lambda }\right)$.

Example 6.

Theorem 2 can be regarded examples of Theorem 4.

Condition 2.

There exist $g:{\mathcal{V}}_{\mathrm{p}}\to \mathbb{R}$ and $h:{\mathcal{V}}_{\mathrm{p}}\to \mathbb{R}$ such that

$$\begin{array}{c}\hfill f\left(\tau \right)=\sum _{v\in {\mathcal{I}}_{\tau }}g\left(v\right)+\sum _{v^{\prime }\in {\mathcal{L}}_{\tau }}h\left(v^{\prime }\right).\end{array}$$

(50)

Theorem 5.

Under Condition 2, we define a recursive function $\xi :{\mathcal{V}}_{\mathrm{p}}\to \mathbb{R}$ as

$$\begin{array}{c}\hfill \xi \left(v\right):=\left\{\begin{array}{cc}h\left(v\right),\hfill & v\in {\mathcal{L}}_{\mathrm{p}},\hfill \\ (1-{\alpha }_v)h\left(v\right)+{\alpha }_v\left(g\left(v\right)+\sum _{v^{\prime }\in {\mathrm{Ch}}_{\mathrm{p}}\left(v\right)}\xi \left(v^{\prime }\right)\right),\hfill & v\in {\mathcal{I}}_{\mathrm{p}}.\hfill \end{array}\right.\end{array}$$

(51)

Then, we can calculate $\mathbb{E}\left[f\right(T\left)\right]$ as $\mathbb{E}\left[f\left(T\right)\right]=\xi \left(v_{\lambda }\right)$.

Remark 4.

Theorem 5 is useful to calculate the Shannon entropy (see, e.g., [16]) of $p\left(\tau \right)$. It is described in Section 4.4.

4.4. Shannon Entropy

Corollary 1.

By substituting $g\left(v\right)=-log{\alpha }_v$ and $h\left(v\right)=-log(1-{\alpha }_v)$ into (51), the Shannon entropy $H\left[T\right]:=-{\sum }_{\tau \in \mathcal{T}}p\left(\tau \right)logp\left(\tau \right)$ can be recursively calculated as follows.

$$\begin{array}{c}\hfill H\left[T\right]=\xi \left(v_{\lambda }\right),\end{array}$$

(52)

where

$$\begin{array}{c}\hfill \xi \left(v\right):=\left\{\begin{array}{cc}0,\hfill & v\in {\mathcal{L}}_{\mathrm{p}},\hfill \\ -(1-{\alpha }_v)log(1-{\alpha }_v)+{\alpha }_v\left(-log{\alpha }_v+\sum _{v^{\prime }\in {\mathrm{Ch}}_{\mathrm{p}}\left(v\right)}\xi \left(v^{\prime }\right)\right),\hfill & v\in {\mathcal{I}}_{\mathrm{p}}.\hfill \end{array}\right.\end{array}$$

(53)

Remark 5.

Kullback–Leibler divergence (see, e.g., [16]) between two tree distributions $p\left(\tau \right)$ and $p^{\prime }\left(\tau \right)$ can be calculated in a similar manner to Corollary 1. This fact may be useful for variational Bayesian inference, in which the Kullback–Leibler divergence is minimized. This will be future work.

4.5. Conjugate Prior for a Probability Distribution on Full Rooted Subtrees

Here, we consider that ${\alpha }_v\in [0,1]$ is also a realization of a random variable. Let $\mathbf{\alpha }$ denote ${\left\{{\alpha }_v\right\}}_{v\in {\mathcal{V}}_{\mathrm{p}}}$, and we describe $p\left(\tau \right)$ as $p\left(\tau \right|\mathbf{\alpha })$ to emphasize the dependency of $\mathbf{\alpha }$ in the following theorem. Then, a conjugate prior for $p\left(\tau \right|\mathbf{\alpha })$ is as follows.

Theorem 6.

The following probability distribution is a conjugate prior for $p\left(\tau \right|\mathbf{\alpha })$.

$$\begin{array}{c}\hfill p\left(\mathbf{\alpha }\right):=\prod _{v\in {\mathcal{V}}_{\mathrm{p}}}\mathrm{Beta}\left({\alpha }_v\right|{\beta }_v,{\gamma }_v),\end{array}$$

(54)

where $\mathrm{Beta}(·|{\beta }_v,{\gamma }_v)$ denotes the probability density function of the beta distribution whose parameters are ${\beta }_v$ and ${\gamma }_v$. More precisely,

$$\begin{array}{c}\hfill p\left(\mathbf{\alpha }\right|\tau )=\prod _{v\in {\mathcal{V}}_{\mathrm{p}}}\mathrm{Beta}\left({\alpha }_v\right|{\beta }_{v|\tau },{\gamma }_{v|\tau }),\end{array}$$

(55)

where

$$\begin{array}{cc}\hfill {\beta }_{v|\tau }:=& \left\{\begin{array}{cc}{\beta }_v+1,\hfill & v\in {\mathcal{I}}_{\tau },\hfill \\ {\beta }_v,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(56)

$$\begin{array}{cc}\hfill {\gamma }_{v|\tau }:=& \left\{\begin{array}{cc}{\gamma }_v+1,\hfill & v\in {\mathcal{L}}_{\tau },\hfill \\ {\gamma }_v,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(57)

4.6. Probability Distribution on a Full Rooted Tree as a Conjugate Prior

We define another random variable X on a set $\mathcal{X}$ and assume X depends on T; i.e., it follows a distribution $p\left(x\right|\tau )$. Here, we discuss a sufficient condition of $p\left(x\right|\tau )$, under which $p\left(\tau \right)$ becomes a conjugate prior for it, and we can efficiently calculate the posterior $p\left(\tau \right|x)$.

Condition 3.

There exist two functions $g:{\mathcal{V}}_{\mathrm{p}}\times \mathcal{X}\to \mathbb{R}$ and $h:{\mathcal{V}}_{\mathrm{p}}\times \mathcal{X}\to \mathbb{R}$, and $p\left(x\right|\tau )$ has the following form.

$$\begin{array}{c}\hfill p\left(x\right|\tau )=\prod _{v\in {\mathcal{I}}_{\tau }}g(x,v)\prod _{v^{\prime }\in {\mathcal{L}}_{\tau }}h(x,v^{\prime }).\end{array}$$

(58)

Note that g and h are not necessarily probability density functions.

Example 7.

For given ${\mu }_1,{\mu }_2\in \mathbb{R}$ and ${\sigma }_1,{\sigma }_2\in {\mathbb{R}}_{>0}$, let $\mathcal{N}\left(x\right|{\mu }_1,{\sigma }_1^2)$ and $\mathcal{N}\left(x\right|{\mu }_2,{\sigma }_2^2)$ denote the probability density functions of the normal distributions governed by them. Let $\mathit{x}:={\left(x_v\right)}_{v\in {\mathcal{V}}_{\mathrm{p}}}$. If we assume

$$\begin{array}{cc}\hfill & g(\mathit{x},v)=\mathcal{N}\left(x_v\right|{\mu }_1,{\sigma }_1^2),\hfill \end{array}$$

(59)

$$\begin{array}{cc}\hfill & h(\mathit{x},v)=\prod _{v^{\prime }\in {\mathrm{De}}_{\mathrm{p}}\left(v\right)\cup \left\{v\right\}}\mathcal{N}\left(x_{v^{\prime }}\right|{\mu }_2,{\sigma }_2^2),\hfill \end{array}$$

(60)

we can construct $p\left(\mathit{x}\right|\tau )$ that satisfies Condition 3. In other words, the elements of the $|{\mathcal{V}}_{\mathrm{p}}|$ dimensional vector $\mathit{x}$ follow the mixture of two normal distributions, and either of the two is chosen by τ.

Theorem 7.

Under Condition 3, we define $q\left(x\right|v)$ and ${\alpha }_{v|x}$ as follows.

$$\begin{array}{cc}\hfill q\left(x\right|v):=& \left\{\begin{array}{cc}h(x,v),\hfill & v\in {\mathcal{L}}_{\mathrm{p}},\hfill \\ (1-{\alpha }_v)h(x,v)+{\alpha }_{v}g(x,v)\prod _{v^{\prime }\in {\mathrm{Ch}}_{\mathrm{p}}\left(v\right)}q\left(x\right|v^{\prime }),\hfill & v\in {\mathcal{I}}_{\mathrm{p}},\hfill \end{array}\right.\hfill \end{array}$$

(61)

$$\begin{array}{cc}\hfill {\alpha }_{v|x}:=& \left\{\begin{array}{cc}{\alpha }_v,\hfill & v\in {\mathcal{L}}_{\mathrm{p}},\hfill \\ \frac{{\alpha }_{v}g(x,v){\prod }_{v^{\prime }\in {\mathrm{Ch}}_{\mathrm{p}}\left(v\right)}q\left(x\right|v^{\prime })}{q\left(x\right|v)},\hfill & v\in {\mathcal{I}}_{\mathrm{p}}.\hfill \end{array}\right.\hfill \end{array}$$

(62)

Note that ${\alpha }_v=0$ for $v\in {\mathcal{L}}_{\mathrm{p}}$ (see Definition 1). Then, the posterior $p\left(\tau \right|x)$ is represented as follows.

$$\begin{array}{c}\hfill p\left(\tau \right|x)=\prod _{v\in {\mathcal{I}}_{\tau }}{\alpha }_{v|x}\prod _{v^{\prime }\in {\mathcal{L}}_{\tau }}(1-{\alpha }_{v^{\prime }|x}).\end{array}$$

(63)

It should be noted that the calculation of $q\left(x\right|v)$ and ${\alpha }_{v|x}$ requires $O\left(k^{d_{\max }+1}\right)$ cost, and it requires $\mathsf{\Omega }\left(2^{k^{d_{\max }-2}}\right)$ cost in general.

Moreover, if we assume the following condition to be stronger than Condition 3, we can calculate the posterior $p\left(\tau \right|x)$ more efficiently with $O\left(d_{\max }\right)$ cost.

Condition 4.

In addition to Condition 3, we assume that there exists a path from $v_{\lambda }$ to a leaf node $v_{\mathrm{end}}\in {\mathcal{L}}_{\mathrm{p}}$ and another function $h^{\prime }:{\mathcal{V}}_{\mathrm{p}}\times \mathcal{X}\to \mathbb{R}$, which satisfy

$$\begin{array}{cc}\hfill g(x,v)& \equiv 1,\hfill \end{array}$$

(64)

$$\begin{array}{cc}\hfill h(x,v):=& h^{\prime }{(x,v)}^{I\{v⪰v_{\mathrm{end}}\}}.\hfill \end{array}$$

(65)

Here, $I\{·\}$ denotes the indicator function. In other words, only $h(x,v)$ on the path from $v_{\lambda }$ to $v_{\mathrm{end}}$ takes a value different of 1.

Corollary 2.

Under Condition 4, $q\left(x\right|v)$ and ${\alpha }_{v|x}$ are calculated as follows, more efficiently than (61) and (62).

$$\begin{array}{cc}\hfill q\left(x\right|v)& =\left\{\begin{array}{cc}{h}^{\prime }(x,v),\hfill & v=v_{\mathrm{end}},\hfill \\ (1-{\alpha }_v)h^{\prime }(x,v)+{\alpha }_{v}q\left(x\right|v_{\mathrm{ch}}),\hfill & v\succ v_{\mathrm{end}},\hfill \\ 1,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(66)

$$\begin{array}{cc}\hfill {\alpha }_{v|x}& =\left\{\begin{array}{cc}{\alpha }_v,\hfill & v\nsucc v_{\mathrm{end}},\hfill \\ \frac{{\alpha }_{v}q\left(x\right|v_{\mathrm{ch}})}{q\left(x\right|v)},\hfill & v\succ v_{\mathrm{end}},\hfill \end{array}\right.\hfill \end{array}$$

(67)

where $v_{\mathrm{ch}}$ is a child node of v on the path from $v_{\lambda }$ to $v_{\mathrm{end}}$. Note that we need not calculate $q\left(x\right|v)$ for $v⋡v_{\mathrm{end}}$ to update the posterior, and it costs only $O\left(d_{\max }\right)$.

Remark 6.

Condition 4 is effective at representing the generation of sequential data $x_1,x_2,\dots ,x_N$, in which there exists a path from root node $v_{\lambda }$ to a leaf node $v_{\mathrm{end}}^n\in {\mathcal{L}}_{\mathrm{p}}$ for each $n\in \{1,2,\dots ,N\}$ ($v_{\mathrm{end}}^n$ and $v_{\mathrm{end}}^{n^{\prime }}$ may different each other for $n\ne n^{\prime }$). The remarkable previous studies using Corollary 2 are [8,9,10,11,12,13] (In [10,11], only (66) is used but (67) is not). In other words, they treat only the case under Condition 4. The other theorems in this paper have potential applications to broader fields of study.

5. Discussion

In this section, we describe the usefulness of our results in statistical decision theory (see, e.g., [6]) and hierarchical Bayesian modeling (see, e.g., [15]). First, our results are useful in model selection and model averaging under the Bayes criterion in statistical decision theory (see, e.g., [6]). The proposed probability distribution $p\left(\tau \right)$ is a conjugate prior for stochastic models $p\left(x\right|\tau )$ satisfying Condition 3, as shown in Theorem 7, and the MAP estimate $\arg {max}_{\tau }p\left(\tau \right|x)$ can be efficiently calculated by applying Theorem 3 to the posterior distribution $p\left(\tau \right|x)$ obtained by Theorem 7. This is the Bayes optimal model selection based on the posterior distribution. Furthermore, we can calculate ${\sum }_{\tau }p\left(x_{\mathrm{new}}\right|\tau )p\left(\tau \right|x)$, i.e., the weighting of the stochastic models based on the posterior distribution, by using Theorems 4 and 7, since the stochastic models $p\left(x\right|\tau )$ satisfying Condition 3 also satisfy Condition 1. This is model averaging of all possible trees with Bayes optimal weights. This corresponds to the methodologies in which they do not select a single tree but aggregate several trees, such as [4,5]. It should be noted that the occurrence probability of a deep tree exponentially decays in our proposed probability distribution. Therefore, we can avoid the deep tree, which often corresponds to a complex statistical model, as mentioned in Section 1.

Second, one example of the applications derived from our results is hyperparameter learning. As mentioned in Remark 6, Condition 4 has been applied to various stochastic models $p\left(x\right|\tau )$ in previous studies [8,9,10,11,12,13]. Conditions 1 and 3 are more generalized conditions than Condition 4, since the stochastic model $p\left(x\right|\tau )$ satisfying Condition 4 also satisfies Conditions 1 and 3. In addition, the logarithm of a function $f\left(\tau \right)$ satisfying Conditions 1 and 3 (and a stochastic model $p\left(x\right|\tau )$ satisfying Condition 4) satisfies Condition 2. Therefore, we can calculate ${\sum }_{\tau \in \mathcal{T}}p\left(\tau \right|x)logp\left(x\right|\tau )$ by using Theorems 7 and 5. In particular, the fact that we can calculate the expectations $\mathbb{E}\left[p\left(x\right|T)\right]={\sum }_{\tau \in \mathcal{T}}p\left(\tau \right|x)p\left(x\right|\tau )$ and $\mathbb{E}[logp\left(x\right|T)]={\sum }_{\tau \in \mathcal{T}}p\left(\tau \right|x)logp\left(x\right|\tau )$ of the stochastic model $p\left(x\right|\tau )$ satisfying Condition 4 implies that we can learn hyperparameters of the stochastic models in [8,9,10,11,12,13] by hierarchical Bayesian modeling with variational Bayesian methods (see, e.g., [15]). To the best of our knowledge, there are no unified studies treating hyperparameter learning for these models.

6. Future Work

Since the present study is a theoretical study, the theorems derived will be applied in future studies. Theorems 1 and 3 and Corollary 2 have been used in previous studies [8,9,10,11,12,13]. Therefore, the other theorems can be applied.

In this study, we did not use approximative algorithms such as the variational Bayes or Markov chain Monte Carlo method (see, e.g., [15]). Such algorithms are required for learning hierarchical models that contain the probability distribution on full rooted subtrees. The methods proposed herein may serve as subroutines. The expansion of our methods to approximative algorithms is another future work.

In this study, the class of trees is restricted to that of full trees, in which every inner node has the same number of child nodes. Hence, another the generalization of the class to that of any rooted tree can be considered in future studies.

7. Conclusions

In this paper, we discussed the probability distributions on full rooted subtrees. Although such a distribution has been used in many fields of study, such as information theory [8,9,10,11], image processing [12], and machine learning [13], it depends significantly on the specific applications and data generative models. By contrast, we discussed it theoretically, collectively, and independently from a specific data generative model. Subsequently, we derived new generalized methods to evaluate the characteristics of the probability distribution on full rooted subtrees, which have not been performed in previous studies. The derived methods are efficient for calculating the events on the nodes, the mode, the expectation, the Shannon entropy, and the posterior distribution of a full rooted subtree. Therefore, this study expands the possibility of the applying the probability distribution on full rooted subtrees.