Consideration on Singularities in Learning Theory and the Learning Coefficient

Abstract

We consider the learning coefficients in learning theory and give two new methods for obtaining these coefficients in a homogeneous case: a method for finding a deepest singular point and a method to add variables. In application to Vandermonde matrix-type singularities, we show that these methods are effective. The learning coefficient of the generalization error in Bayesian estimation serves to measure the learning efficiency in singular learning models. Mathematically, the learning coefficient corresponds to a real log canonical threshold of singularities for the Kullback functions (relative entropy) in learning theory.

1. Introduction

The purpose of a learning system is to estimate an unknown true density function (a probability model) that generates the data. Real data associated with, for example, genetic analysis, data mining, image or speech recognition, artificial intelligence, the control of a robot and time series prediction are very complicated and usually are not generated by a simple normal distribution. In Bayesian estimation, we set a learning model that is written in probabilistic form with parameters, and our goal is to estimate the true density function by a predictive function constructed with the learning model and such data. Therefore, the learning model should be abundant enough to capture the true density function’s structure. Hierarchical learning models, such as the layered neural network, the Boltzmann machine, the reduced rank regression and the normal mixture model, are known to be effective learning models for analyzing such data. These are, however, singular learning models, which cannot be analyzed using the classic theory of regular statistical models, because singular learning models have a singular Fisher metric that is not always approximated by any quadratic form [1,2,3,4]. Therefore, it is difficult to analyze their generalization errors, which indicate how precisely the predictive function approximates the true density function.

In recent studies, Watanabe showed using algebraic geometry that the generalization and training errors are subject to a universal law and defined the model selection method “widely applicable information criterion” (WAIC ) as a generalized Akaike information criterion (AIC) [5,6,7,8,9]. WAIC can even be applied to singular learning models, whereas AIC cannot. Using the WAIC, we can estimate the generalization errors from the training errors without any knowledge of the true probability density functions. The generalization errors relate to the generalization losses via the entropy of the true distribution. Thus, we can select a suitable model from among several statistical models by this method.

Computing the WAIC requires the values of the learning coefficient and the singular fluctuation, which are both birational invariants. Mathematically, the learning coefficient is the log canonical threshold (Definition 1) of the Kullback function (relative entropy), and the singular fluctuation is known as a statistically generalized log canonical threshold, which is obtained theoretically from the learning coefficient (Equation (1) in Section 2). These values can be obtained by Hironaka’s Theorem (Appendix A). However, it is still difficult to obtain these within learning theory for several reasons, such as degeneration with respect to their Newton polyhedra and non-isolation of their singularities [10]. Moreover, in algebraic geometry and algebraic analysis, these studies are usually done over an algebraically closed field [11,12]; many differences exist for real and complex fields. For example, log canonical thresholds over the complex field are less than one, whereas those over the real field are not necessarily so. We, therefore, cannot apply results over an algebraically closed field to our current situation directly (Appendix B). One of the bottlenecks in learning theory is to obtain the learning coefficients and the singular fluctuation.

In this paper, we consider the learning coefficient of “Vandermonde matrices-type singularities” in statistical learning theory. The reason why we contribute only to such singularities is that the Vandermonde matrix type is generic and essential in learning theory. These log canonical thresholds give the learning coefficients of normal mixture models, three-layered neural networks and mixtures of binomial distributions, which are widely used as effective learning models (Section 3.1 and Section 3.2 and [13]). Moreover, we prove Theorem 2 (the method for finding a deepest deepest singular point) and Theorem 3 (the method to add variables), which are very beneficial to obtain the log canonical threshold for the homogeneous case. Theorem 2 indicates the best point of singularities that gives the log canonical threshold. Therefore, this theorem is useful for the reduction of the number of blowup processes. Theorem 3 improves our recursive blowup method by simplifying coordinate system changes with added variables. These two theorems enable us to obtain a new bound for the log canonical thresholds of Vandermonde matrix-type singularities in Theorem 5. These bounds are much tighter than those in [14].

In the past few years, we have obtained the learning coefficients for reduced rank regression [15], for the three-layered neural network with one input unit and one output unit [16,17], and for the normal mixture models with a dimension of one [18]. The paper [14] derived bounds on the learning coefficients for the Vandermonde matrix-type singularities and explicit values under some conditions. The learning coefficients for the restricted Boltzmann machine [19] have also been considered recently. Ref [20,21,22], respectively, obtained these for naive Bayesian networks and for directed tree models with hidden variables. These results give partial answers for the learning coefficients.

The rest of the paper is in three sections. Section 2 summarizes the framework of Bayesian learning models. In Section 3, we demonstrate our main theorems and consider the log canonical threshold of Vandermonde matrix-type singularities (Definition 3). We finish with our conclusions in Section 4.

2. Learning Coefficients and Singular Fluctuations

In this section, we present the theory of learning coefficients and singular fluctuations. Let $q(x)$ be a true probability density function of variables, $x\in {\mathbb{R}}^N$, and let $x^n:={\{x_i\}}_{i=1}^n$ be n training samples selected from $q(x)$ independently and identically. Consider a learning model that is written in probabilistic form as $p(x|w)$, where $w\in W\subset {\mathbb{R}}^d$ is a parameter. The purpose of the learning system is to estimate $q(x)$ from $x^n$ using $p(x|w)$. Let $\psi (w)$ be an a priori probability density function on the parameter set, W, and $p(w|x^n)$ be the a posteriori probability density function:

$$p(w|x^n)=\frac{1}{Z_n}\psi (w)\prod _{i=1}^{n}p(x_i|w)$$

where:

$$Z_n={\int }_W\psi (w)\prod _{i=1}^{n}p(x_i|w)\mathrm{d}w$$

Let us define for the inverse temperature, β:

$$E_w[f(w)]=\frac{\int dwf(x)\psi (w){\prod }_{i=1}^{n}p{(x_i|w)}^{\beta }}{\int dw\psi (w){\prod }_{i=1}^{n}p{(x_i|w)}^{\beta }}$$

We usually set $\beta =1$.

We then have a predictive density function, $p(x|X^n)=E_w[p(x|w)]$, which is the average inference of the Bayesian density function.

We next introduce the Kullback function, $K(q||p)$, and the empirical Kullback function, $K_n(q||p)$, for density functions $p(x),q(x)$:

$$K(q||p)=\sum _{x}q(x)log\frac{q(x)}{p(x)}$$

$$K_n(q||p)=\frac{1}{n}\sum _{i=1}^{n}log\frac{q(x_i)}{p(x_i)}$$

The function, $K(p||q)$, always has a non-negative value and satisfies $K(q||p)=0$, if and only if $q(x)=p(x)$.

The Bayesian generalization error, $B_g$, Bayesian training error, $B_t$, Gibbs generalization error, $G_g$, and Gibbs training error, $G_t$, are defined as follows:

$$B_g=K(q(x)||E_w[p(x|w)]))$$

$$B_t=K_n(q(x)|E_w[p(x_i|w)])$$

$$G_g=E_w[K(q(x)||p(x|w))]$$

and

$$G_t=E_w[K_n(q(x)||p(x|w))]$$

The most important of these is the Bayesian generalization error. This error describes how precisely the predictive function approximates the true density function.

Watanabe [6,7,23] proved the following four relations:

$$E[B_g]=\frac{\lambda +\nu \beta -\nu }{n\beta }+o(\frac{1}{n})$$

$$E[B_t]=\frac{\lambda -\nu \beta -\nu }{n\beta }+o(\frac{1}{n})$$

$$E[G_g]=\frac{\lambda +\nu \beta }{n\beta }+o(\frac{1}{n})$$

$$E[G_t]=\frac{\lambda -\nu \beta }{n\beta }+o(\frac{1}{n})$$

Thus we have:

$$E[B_g]=E[B_t]+2\beta (E[G_t]-E[B_t])+o(\frac{1}{n})$$

and

$$E[G_g]=E[G_t]+2\beta (E[G_t]-E[B_t])+o(\frac{1}{n})$$

Eliminating the expectation of the true probability density function from the above four errors and setting:

$$B{L}_g=-\sum _{x}q(x)log{E}_w[p(x|w)]$$

$$B{L}_t=-\frac{1}{n}\sum _{i=1}^{n}log{E}_w[p(x_i|w)]$$

$$G{L}_g=-E_w[\sum _{x}q(x)logp(x|w)]$$

$$G{L}_t=-E_w[\frac{1}{n}\sum _{i=1}^{n}logp(x_i|w)]$$

we then have:

$$E[B{L}_g]=E[B{L}_t]+2\beta (E[G_t]-E[B_t])+o(\frac{1}{n})$$

and

$$E[G{L}_g]=E[G{L}_t]+2\beta (E[G_t]-E[B_t])+o(\frac{1}{n})$$

These two equations constitute the WAIC and show that we can estimate the Bayesian and Gibbs generalization errors from the Bayesian and Gibbs training errors without any knowledge of the true probability density functions. Training errors are calculated from training samples, $x_i$, using a learning model, p. In real applications or experiments, we usually do not know the true distribution, but only the values of the training errors. Our purpose is to estimate the true distribution from the training samples, showing that these relations are effective. We can select a suitable model from among several statistical models by observing these values.

Let λ denote a learning coefficient and ν a singular fluctuation, both of which are birational invariants. Mathematically, λ is equal to the log canonical threshold introduced in Definition 1 and Appendix B. For regular models, $\lambda =\nu =d/2$ holds, where d is the dimension of the parameter space.

The difference between the Bayesian and Gibbs training errors converges to $\nu /n$:

$$n\beta (E[G_t]-E[B_t])\to \nu$$

These relations were shown using the resolution of singularities and the Schwarz distribution.

From the learning coefficient, λ, and its order, θ, the value, ν, is obtained theoretically as follows. Let $\xi (u)$ be an empirical process defined on the manifold obtained by a resolution of singularities, and ${\sum }_{u^{*}}$ denote the sum of local coordinates that attain the minimum λ and the maximum θ. We then have:

$${\displaystyle \nu =\frac{1}{2}{E}_{\xi }\frac{{\int }_0^{\infty }dt{\sum }_{u^{*}}\int du\xi (u)t^{\lambda -1/2}{e}^{-\beta t+\beta \sqrt{t}\xi (u)}}{{\int }_0^{\infty }dt{\sum }_{u^{*}}\int du{t}^{\lambda -1/2}{e}^{-\beta t+\beta \sqrt{t}\xi (u)}}}$$

(1)

$\xi (u)$ is a random variable of a Gaussian process with mean zero and variance two. Our purpose in this paper is to obtain λ.

To assist in achieving this aim, we use the desingularization approach from algebraic geometry (cf. Appendix A). It is a new problem in algebraic geometry to obtain the desingularization of the Kullback functions, because the singularities of these functions are very complicated, and as such, most of these have not yet been investigated.

3. Main Theorems and Vandermonde Matrix-Type Singularities

We denote constants, such as $a^{*}$, $b^{*}$ and $w^{*}$, by the suffix ∗. Additionally, for simplicity, we use the notation: $w={\{a_{ki},b_{ij}\}}_{1\le i\le H}$ instead of: $w={\{a_{ki},b_{ij}\}}_{1\le k\le M,1\le i\le H,1\le j\le N},$ because we always have $1\le k\le M$ and $1\le j\le N$ in this paper.

Define the norm of a matrix, $C=(c_{ij})$, by $||C||=\sqrt{{\sum }_{i,j}{|c_{ij}|}^2}$. Set ${\mathbb{N}}_{+0}=\mathbb{N}\cup \{0\}$.

Definition 1 

For a real analytic function, f, in a neighborhood, U, of $w^{*}$ and a $C^{\infty }$ function ψ with a compact support, let ${\lambda }_{w^{*}}(f,\psi )$ be the largest pole of ${\int }_U{|f|}^z\psi dw$ and ${\theta }_{w^{*}}(f,\psi )$ be its order. If $\psi (w^{*})\ne 0$, then we denote ${\lambda }_{w^{*}}(f)={\lambda }_{w^{*}}(f,\psi )$ and ${\theta }_{w^{*}}(f)={\theta }_{w^{*}}(f,\psi )$, because the log canonical threshold and its order are independent of ψ.

Definition 2 

Fix $Q\in \mathbb{N}$. Define: ${[b_1^{*},b_2^{*},\cdots ,b_N^{*}]}_Q={\gamma }_i(0,\cdots ,0,b_i^{*},\cdots ,b_N^{*})$ if $b_1^{*}=\cdots =b_{i-1}^{*}=0$, $b_i^{*}\ne 0$, and ${\gamma }_i=\left\{\begin{array}{cc}1\hfill & ifQisodd,\hfill \\ |b_i^{*}|/b_i^{*}\hfill & ifQiseven.\hfill \end{array}\right.$

Definition 3 

Fix $Q\in \mathbb{N}$.

Let $A=\left(\begin{array}{cccccc}{a}_{11}& \cdots & a_{1H}& a_{1,H+1}^{*}& \dots & a_{1,H+r}^{*}\\ a_{21}& \cdots & a_{2H}& a_{2,H+1}^{*}& \dots & a_{2,H+r}^{*}\\ & ⋮& & & ⋮& \\ a_{M1}& \cdots & a_{MH}& a_{M,H+1}^{*}& \dots & a_{M,H+r}^{*}\end{array}\right)$, $I=({\ell }_1,\dots ,{\ell }_N)\in {{\mathbb{N}}_{+0}}^N$

$$B_I=(\prod _{j=1}^N{b}_{1j}^{{\ell }_j},\prod _{j=1}^N{b}_{2j}^{{\ell }_j},\cdots ,\prod _{j=1}^N{b}_{Hj}^{{\ell }_j},\prod _{j=1}^N{b_{H+1,j}^{*}}^{{\ell }_j},\cdots ,\prod _{j=1}^N{b_{H+r,j}^{*}}^{{\ell }_j}{)}^t$$

and

$$B={(B_I)}_{{\ell }_1+\cdots +{\ell }_N=Qn+1,0\le n\le H+r-1}=(B_{(1,0,\cdots ,0)},B_{(0,1,\cdots ,0)},\cdots ,B_{(0,0,\cdots ,1)},B_{(1+Q,0,\cdots ,0)},\cdots )$$

(the superscript, t, denotes matrix transposition).

$a_{ki}$ and $b_{ij}$ $(1\le k\le M,1\le i\le H,1\le j\le N)$ are variables in a neighborhood of $a_{ki}^{*}$ and $b_{ij}^{*}$, where $a_{ki}^{*}$ and $b_{ij}^{*}$ are fixed constants.

Let $\mathcal{I}$ be the ideal generated by the elements of $AB$.

We call singularities of $\mathcal{I}$ Vandermonde matrix-type singularities.

To simplify, we usually assume that

$${(a_{1,H+j}^{*},a_{2,H+j}^{*},\cdots ,a_{M,H+j}^{*})}^t\ne 0,(b_{H+j,1}^{*},b_{H+j,2}^{*},\cdots ,b_{H+j,N}^{*})\ne 0$$

for $1\le j\le r$ and

$${[b_{H+j,1}^{*},b_{H+j,2}^{*},\cdots ,b_{H+j,N}^{*}]}_Q\ne {[b_{H+j^{\prime },1}^{*},b_{H+j^{\prime },2}^{*},\cdots ,b_{H+j^{\prime },N}^{*}]}_Q$$

for $j\ne j^{\prime }$.

Example 1 

If $N=Q=1$ and $r=0$, then we have: $B=\left(\begin{array}{cccc}{b}_{11}& b_{11}^2& \cdots & b_{11}^H\\ b_{21}& b_{21}^2& \cdots & b_{21}^H\\ & & ⋮& \\ b_{H1}& b_{H1}^2& \cdots & b_{H1}^H\end{array}\right)$.

This matrix is a Vandermonde matrix.

Example 2 

If $Q=1$, $M=1$, $H=2,N=2$ and $r=1$, then we have: $A=\left(\begin{array}{ccc}{a}_{11}& a_{12}& a_{1,3}^{*}\end{array}\right)$ and $B=\left(\begin{array}{ccccccccc}{b}_{11}& b_{12}& b_{11}^2& b_{11}{b}_{12}& b_{12}^2& b_{11}^3& b_{11}{b}_{12}^2& b_{11}^2{b}_{12}& b_{12}^3\\ b_{21}& b_{22}& b_{21}^2& b_{21}{b}_{22}& b_{22}^2& b_{21}^3& b_{21}{b}_{22}^2& b_{21}^2{b}_{22}& b_{22}^3\\ b_{31}^{*}& b_{32}^{*}& {b_{31}^{*}}^2& b_{31}^{*}{b}_{32}^{*}& {b_{32}^{*}}^2& {b_{31}^{*}}^3& b_{31}^{*}{b_{32}^{*}}^2& {b_{31}^{*}}^2{b}_{32}^{*}& {b_{32}^{*}}^3\end{array}\right)$.

In this paper, we denote:

$$\begin{array}{ccc}& & A_{M,H}=\left(\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1H}\\ a_{21}& a_{22}& \cdots & a_{2H}\\ & & ⋮& \\ a_{M1}& a_{M2}& \cdots & a_{MH}\end{array}\right),B_{H,N,I}=\left(\begin{array}{c}\prod _{j=1}^N{b_{1j}}^{{\ell }_j}\\ \prod _{j=1}^N{b_{2j}}^{{\ell }_j}\\ ⋮\\ \prod _{j=1}^N{b_{Hj}}^{{\ell }_j}\end{array}\right)and\hfill \end{array}$$

$B_{H,N}^{(Q)}={(B_{H,N,I})}_{{\ell }_1+\dots +{\ell }_N=Qn+1,0\le n\le H-1}$.

Furthermore, we denote: ${\mathbf{a}}^{*}=\left(\begin{array}{c}{a}_{1,H+1}^{*}\\ ⋮\\ a_{M,H+1}^{*}\end{array}\right)$ and

$$\begin{array}{ccc}& & (A_{M,H},{\mathbf{a}}^{*})=\left(\begin{array}{ccccc}{a}_{11}& a_{12}& \cdots & a_{1H}& a_{1,H+1}^{*}\\ a_{21}& a_{22}& \cdots & a_{2H}& a_{2,H+1}^{*}\\ & & ⋮& \\ a_{M1}& a_{M2}& \cdots & a_{MH}& a_{M,H+1}^{*}\end{array}\right)\hfill \end{array}$$

Theorem 1 

([18]) Consider a sufficiently small neighborhood, U, of

$$w^{*}={\{a_{ki}^{*},b_{ij}^{*}\}}_{1\le i\le H}$$

and variables, $w={\{a_{ki},b_{ij}\}}_{1\le i\le H}$, in the set, U.

Set: $(b_{01}^{**},b_{02}^{**},\cdots ,b_{0N}^{**})=(0,\dots ,0)$.

Let each: $(b_{11}^{**},b_{12}^{**},\cdots ,b_{1N}^{**})$, …, $(b_{r^{\prime }1}^{**},b_{r^{\prime }2}^{**},\cdots ,b_{r^{\prime }N}^{**})$ be a different real vector in:

$${[b_{i1}^{*},b_{i2}^{*},\cdots ,b_{iN}^{*}]}_Q\ne 0,fori=1,\dots ,H+r$$

That is:

$$\{(b_{11}^{**},\cdots ,b_{1N}^{**}),\dots ,(b_{r^{\prime }1}^{**},\cdots ,b_{r^{\prime }N}^{**});{[b_{i1}^{*},\cdots ,b_{iN}^{*}]}_Q\ne 0,i=1,\dots ,H+r\}$$

Then, $r^{\prime }$ is uniquely determined, and $r^{\prime }\ge r$ by the assumption in Definition 3. Set: $(b_{i1}^{**},\cdots ,b_{iN}^{**})={[b_{H+i,1}^{*},\cdots ,b_{H+i,N}^{*}]}_Q,$ for $1\le i\le r$.

Assume that:

$${[b_{i1}^{*},\cdots ,b_{iN}^{*}]}_Q=\left\{\begin{array}{cc}0,& 1\le i\le H_0\hfill \\ (b_{11}^{**},\cdots ,b_{1N}^{**}),& H_0+1\le i\le H_0+H_1,\hfill \\ (b_{21}^{**},\cdots ,b_{2N}^{**}),& H_0+H_1+1\le i\le H_0+H_1+H_2,\hfill \\ ⋮\\ (b_{r^{\prime }1}^{**},\cdots ,b_{r^{\prime }N}^{**}),& H_0+\cdots +H_{r^{\prime }-1}+1\le i\le H_0+\cdots +H_{r^{\prime }},\hfill \end{array}\right.$$

and $H_0+\cdots +H_{r^{\prime }}=H$.

We then have:

$$\begin{gathered} {\lambda }_{w^{*}}{(||AB||}^2)=\frac{M{r}^{\prime }}{2}+{\lambda }_{{w_1^{(0)}}^{*}}(||A_{M,H_0}{B}_{H_0,N}^{(Q)}{||}^2) \\ +\sum _{\alpha =1}^r{\lambda }_{{w_1^{(\alpha )}}^{*}}(||(A_{M,H_{\alpha }-1},{{\mathbf{a}}^{(\alpha )}}^{*})B_{H_{\alpha },N}^{(1)}{||}^2)+\sum _{\alpha =r+1}^{r^{\prime }}{\lambda }_{{w_1^{(\alpha )}}^{*}}(||A_{M,H_{\alpha }-1}{B}_{H_{\alpha }-1,N}^{(1)}{||}^2) \end{gathered}$$

where: ${w_1^{(0)}}^{*}={\{a_{k,i}^{*},0\}}_{1\le i\le H_{\alpha }},$ ${w_1^{(\alpha )}}^{*}={\{a_{k,H_0+\cdots +H_{\alpha -1}+i}^{*},0\}}_{2\le i\le H_{\alpha }}$ and ${{\mathbf{a}}^{(\alpha )}}^{*}=\left(\begin{array}{c}{a}_{1,H+\alpha }^{*}\\ ⋮\\ a_{M,H+\alpha }^{*}\end{array}\right)$ for $\alpha \ge 1$.

Theorem 2 (Method for finding a deepest singular point) 

Let $f_1(w_1,\dots ,w_d)$, …, $f_m(w_1,\dots ,w_d)$ be homogeneous functions of $w_1,\cdots ,w_j$ $(j\le d)$ with the degree, $n_i$, of $w_1,\cdots ,w_j$. Furthermore, let ψ be a $C^{\infty }$ function, such that ${\psi }_{(0,\cdots ,0,w_{j+1}^{*},\cdots ,w_d^{*})}\ge$ ${\psi }_{(w_1^{*},\cdots ,w_d^{*})}$ and ${\psi }_w$ is homogeneous of $w_1,\cdots ,w_j$ in a small neighborhood of $(0,\cdots ,0,w_{j+1}^{*},\cdots ,w_d^{*})$.

Then, we have:

$${\lambda }_{(0,\cdots ,0,w_{j+1}^{*},\cdots ,w_d^{*})}(f_1^2+\cdots +f_m^2,\psi )\le {\lambda }_{(w_1^{*},\cdots ,w_j^{*},w_{j+1}^{*},\cdots ,w_d^{*})}(f_1^2+\cdots +f_m^2,\psi )$$

(Proof)

Let d be the degree of $w_1,\cdots ,w_j$ for ψ in a neighborhood of $(0,\cdots ,0,w_{j+1}^{*},\cdots ,w_d^{*}).$ Let us construct the blowup of $f_1$, …, $f_m$ along the submanifold, $\{v=0,w_i=0,1\le i\le j\}$. Let $w_i=v{w}_i^{\prime }$ for $1\le i\le j$. We have: $v^{n_i}{f}_i(w_1^{\prime },\dots ,w_d^{\prime })$ and ${(f_1{(w)}^2+f_2{(w)}^2+\cdots +f_m{(w)}^2)}^z\psi dwdv={(v^{2n_1}{f}_1^2(w^{\prime })+v^{2n_2}{f}_2^2(w^{\prime })+\cdots +v^{2n_m}{f}_m{(w^{\prime })}^2)}^z\psi (w^{\prime })v^{d+j}d{w}^{\prime }dv$. Because: $v^{2n_i}{f}_i^2(w_1^{\prime },\dots ,w_d^{\prime })\le f_i^2(w_1^{\prime },\dots ,w_d^{\prime })$ for $|v|<1$, we have: $v^{2n_1}{f}_1^2+\cdots +v^{2n_m}{f}_m^2\le f_1^2+\cdots +f_m^2$, and, hence, by Lemma 1 in Appendix C:

$${\lambda }_{(0,\cdots ,0,w_{j+1}^{*},\cdots ,w_d^{*})}(f_1^2+\cdots +f_m^2,\psi )\le min\{d+j+1,{\lambda }_{w^{*}}(f_1^2+\cdots +f_m^2,\psi )\}$$

Furthermore, we consider the construction of the blowup of: $f_1$, …, $f_m$ along the submanifold: $\{w_i=0,1\le i\le j\}$, for which we have

$$\begin{array}{ccc}& {\lambda }_{(0,\cdots ,0,w_{j+1}^{*},\cdots ,w_d^{*})}(f_1^2+\cdots +f_m^2,\psi )\le d+j\hfill & \\ & & \hfill \text{\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}Q.E.D.}\end{array}$$

In general, it is not true that:

$${\lambda }_{w_0}(f_1^2+\cdots +f_m^2,\psi )\le {\lambda }_{w^{*}}(f_1^2+\cdots +f_m^2,\psi )$$

even if $w_0\in {\mathbb{R}}^d$ satisfies:

$$f_i(w_0)=\frac{\partial f_i}{\partial w_j}(w_0)=0,1\le i\le m,1\le j\le d$$

Example 3 

Let $f_1=x{(x-1)}^2$, $f_2=(y^2+{(x-1)}^2)({(y-1)}^6+x)$ and $f_3=(z^2+{(x-1)}^2)({(z-1)}^6+x)$. Then, we have: $f_1=f_2=f_3=\frac{\partial f_1}{\partial x}=\frac{\partial f_2}{\partial y}=\frac{\partial f_2}{\partial x}=\frac{\partial f_3}{\partial z}=\frac{\partial f_3}{\partial x}=0$ if and only if $x=1,y=0,z=0$.

In this case, we have ${\lambda }_{(1,0,0)}(f_1^2+f_2^2+f_3^2)=1/4+1/4+1/4>{\lambda }_{(0,1,1)}(f_1^2+f_2^2+f_3^2)=1/2+1/12+1/12.$

Theorem 3 (Method to add variables) 

Let $f_1(w_1,\dots ,w_d)$, …, $f_m(w_1,\dots ,w_d)$ be homogeneous functions of $w_1,\cdots ,w_d$ of the degree, $n_i$, in $w_1,\cdots ,w_d$. Set: $f_1^{\prime }(w_2,\dots ,w_d)=f_1(1,w_2,\dots ,w_d)$, …, $f_m^{\prime }(w_2,\dots ,w_d)=f_m(1,w_2,\dots ,w_d)$. If $w_1^{*}\ne 0$, then we have:

$${\lambda }_{(w_1^{*},\cdots ,w_d^{*})}(f_1^2+\cdots +f_m^2)={\lambda }_{(w_2^{*}/w_1^{*},\cdots ,w_d^{*}/w_1^{*})}(f_1^{\prime 2}+\cdots +f_m^{\prime 2})$$

(Proof) Set $w_2^{\prime }=w_2/w_1,\dots ,w_d^{\prime }=w_d/w_1$. Then, we have:

$$f_i(w_1,w_2,\dots ,w_d)=w_1^{n_i}{f}_i(1,w_2^{\prime },\dots ,w_d^{\prime })=w_1^{n_i}{f}_i^{\prime }$$

Since $w_1^{n_i}\ne 0$ on a small neighborhood of $w_1^{*}$, there exist positive real numbers, $C,C^{\prime }$, such that:

$$C(f_1^2+\cdots +f_m^2)\le f_1^{\prime 2}+\cdots +f_m^{\prime 2}\le C^{\prime }(f_1^2+\cdots +f_m^2)$$

This completes the proof by Lemma 1 in Appendix C. Q.E.D.

Remark 1

The above theorem shows that we can set nonzero constants as variables to obtain the same log canonical threshold. However, in general, this is not true.

(1)

Consider the function $f(x,y)={(y-1+x^2)}^2$. We have ${\lambda }_{f=0}(f)=1/2$, whereas ${\lambda }_{f(x,1)=0}(f(x,1))=1/4$.

(2)

Consider the function $f(x_1,x_2,x_3,x_4,x_5,y)={(x_1^2+x_2^2+x_3^2+x_4^2+x_5^2+y-1)}^2$. We have ${\lambda }_{f=0}(f)=1/2$, whereas ${\lambda }_0(f(x_1,x_2,x_3,x_4,x_5,1))=5/4$.

The second example shows that the following theorem over the complex field is not true over the real field.

Theorem 4 [11]

Let $f(x_1,\cdots ,x_d,y)$ be a holomorphic function near zero, and for a hyperplane H, let ${g=f|}_{y=0}$ (or $g=f_H$) denote the restriction of f to $y=0$ (or H). Then, ${\lambda }_{g=0}(g)\le {\lambda }_{f=0}(f)$.

Define: ${\displaystyle \langle \genfrac{}{}{0pt}{}{k}{l}\rangle =\frac{k!}{l!(k-l)!}}$,

Theorem 5 

We use the same notation as in Theorem 1. Let:

$${\mathrm{bound}}_1=min\{{\displaystyle \frac{(H-i+1)N+d_i(s)+d_i^{\prime }(s)+d_i^{″}(s)}{2(count(i,s,k(s))-1)Q+2}}:1\le i\le s,1\le k(1),\cdots ,k(s)\le N,1\le s\le H\}$$

where: $count(i,s,j)=\#\{i_1:i\le i_1\le s,k(i_1)=j\}$, $C(i,s)=\#\{count(i,s,j)=0,1\le j\le N\},$

$$\begin{array}{ccc}\hfill d_i(s)& =& (N-1)Q\sum _{s_1=i}^s(count(i,s_1,k(s_1))-1)\hfill \\ \hfill d_i^{\prime }(s)& =& M(i-1)\{(count(i,s,k(s))-1)Q+1\}\hfill \\ & & +QM\sum _{\genfrac{}{}{0pt}{}{s_1=i,}{count(i,s,k(s))>count(i,s_1,k(s_1))}}^{s-1}(count(i,s,k(s))-count(i,s_1,k(s_1))),\hfill \\ \hfill d_i^{″}(s)& =& \left\{\begin{array}{c}0,ifcount(i,s,k(s))=1,\hfill \\ (H-s)\{C(i,s)Q+(N-1)Q(count(i,s,k(s))-2)\},\hfill \\ ifcount(i,s,k(s))\ge 2,N-1\le M,\hfill \\ (H-s)\{C(i,s)Q+MQ(count(i,s,k(s))-2)\},\hfill \\ ifcount(i,s,k(s))\ge 2,C(i,s)\le M<N-1,\hfill \\ (H-s)\{MQ(count(i,s,k(s))-1)\},\hfill \\ ifcount(i,s,k(s))\ge 2,M\le C(i,s).\hfill \end{array}\right.\hfill \end{array}$$

Furthermore, let: ${\displaystyle {\mathrm{bound}}_2=\frac{NH+{\sum }_{i=0}^{k^{\prime }-1}MQ(k^{\prime }-i)\langle \genfrac{}{}{0pt}{}{N+Qi}{N-1}\rangle }{2+2Q{k}^{\prime }},}$ where: $k^{\prime }=max\{i\in \mathbb{Z};NH\ge M{\sum }_{i^{\prime }=0}^{i-1}(1+Q{i}^{\prime })\langle \genfrac{}{}{0pt}{}{N+Q{i}^{\prime }}{N-1}\rangle \}$, and let:

$${\mathrm{bound}}_3=\frac{MH}{2}$$

We have

$$\begin{array}{ccc}& & {\lambda }_0(||A_{M,H}{B}_{H,N}^{(Q,m)}{||}^2)\le min\{{\mathrm{bound}}_1,{\mathrm{bound}}_2,{\mathrm{bound}}_3\}\hfill \\ & & {\lambda }_0(||(A_{M,H-1},{\mathbf{a}}^{*})B_{H,N}^{(Q,m)}{||}^2)\le min\{{\mathrm{bound}}_1,{\mathrm{bound}}_2\}\hfill \end{array}$$

The proof appears in Appendix C.

Figure 1a–d show the values of new bounds, $min\{{\mathrm{bound}}_1,{\mathrm{bound}}_2,{\mathrm{bound}}_3\}$, for (a) $H=7,N=6$, (b) $H=8,N=6$, (c) $H=7,N=7$ and (d) $H=8,N=7$ with $Q=2$, respectively. We compare these values with those obtained by the past work in [14]. In the figures, the horizontal axis is the number, M, and the vertical one, the value of such bounds. The dashed lines indicate the bounds obtained by the past work. These figures show that new bounds are not greater than old ones.

Figure 1. The values of new bounds, $min\{{\mathrm{bound}}_1,{\mathrm{bound}}_2,{\mathrm{bound}}_3\}$, for (a) $H=7,N=6$; (b) $H=8,N=6$; (c) $H=7,N=7$ and (d) $H=8,N=7$ with $Q=2$, compared with the bounds obtained by the past work in [14].

Figure 1. The values of new bounds, $min\{{\mathrm{bound}}_1,{\mathrm{bound}}_2,{\mathrm{bound}}_3\}$, for (a) $H=7,N=6$; (b) $H=8,N=6$; (c) $H=7,N=7$ and (d) $H=8,N=7$ with $Q=2$, compared with the bounds obtained by the past work in [14].

In paper [24], we had exact values for $N=1$:

$${\lambda }_0(||A_{M,H}{B}_{H,1}^{(Q)}{||}^2)=\frac{MQk(k+1)+2H}{4(1+kQ)}$$

where: $k=max\{i\in \mathbb{Z}:2H\ge M(i(i-1)Q+2i)\},$ and we had:

$$\theta =\left\{\begin{array}{cc}1,\hfill & \text{if }2H>M(k(k-1)Q+2k)\hfill \\ 2,\hfill & \text{if }2H=M(k(k-1)Q+2k)\hfill \end{array}\right.$$

We had other exact values when H is small on paper [14]. Both sets of exact values are the bounded values in Theorem 5.

3.1. A Learning Coefficient for a Three-Layered Neural Network

Consider the three-layered neural network with N input units, H hidden units and M output units, which are trained for estimating the true distribution with r hidden units. Their learning coefficients, λ, are as follows [14,24]:

$$\lambda =min\{{\lambda }_0(||A_{M,H_0}{B}_{H_0,N}^{(2)}{||}^2)+{\sum }_{\alpha =1}^r{\lambda }_0(||(A_{M,H_{\alpha }-1},{\mathbf{a}}^{*})B_{H_{\alpha },N}^{(1)}{||}^2):H_0+H_1+\cdots +H_r=H,{\mathbf{a}}^{*}\ne 0\}.$$

3.2. A Learning Coefficient for a Normal Mixture Model

Consider normal mixture models with H peaks and the true distribution with r peaks. Then, their learning coefficients, λ, are as follows [14,18]:

$$\lambda =\frac{r-1}{2}+min\{{\sum }_{\alpha =1}^r{\lambda }_0(||(A_{M,H_{\alpha }-1},{\mathbf{a}}^{*})B_{H_{\alpha },N}^{(1)}{||}^2):{\sum }_{\alpha =1}^r{H}_{\alpha }=H,{\mathbf{a}}^{*}\ne 0\}$$

In particular, we have for $N=1$:

$$\lambda =r-1+\frac{i+i^2+2(H-(r-1))}{4(i+1)},\theta =\left\{\begin{array}{cc}1,\hfill & \text{if }{i}^2+i<2(H-(r-1)),\text{or }H=r\hfill \\ 2,\hfill & \text{if }{i}^2+i=2(H-(r-1)),\hfill \end{array}\right.$$

where $i=max\{j\in \mathbb{Z};j^2+j\le 2(H-(r-1))\}$.

4. Conclusions

In this paper, we prove two theorems, Theorem 2 (the method for finding a deepest singular point) and Theorem 3 (the method to add variables) for obtaining learning coefficients in a homogeneous case. By applying these methods to Vandermonde matrix-type singularities and using the inclusion of ideals and recursive blowup from algebraic geometry, we found new bounds on learning coefficients for Vandermonde matrix-type singularities. These bounds are much tighter than those in [14]. Our future research aim is to improve our methods and to obtain exact values for the general machine model.

The learning coefficients from our recent results have been used very effectively by Drton [25,26] for model selection, using a method called “singular Bayesian information criterion (sBIC)”, which can be applied to singular models, where the assumptions supporting the use of the standard BIC do not hold. Our theoretical results introduce a mathematical measure of precision to numerical calculations, such as Markov chain Monte Carlo (MCMC). Nagata and Watanabe [27,28] gave a mathematical foundation for analyzing and developing the precision of the MCMC method using our theoretical values of marginal likelihoods.