Regularizing Neural Networks via Retaining Confident Connections

Abstract

Regularization of neural networks can alleviate overfitting in the training phase. Current regularization methods, such as Dropout and DropConnect, randomly drop neural nodes or connections based on a uniform prior. Such a data-independent strategy does not take into consideration of the quality of individual unit or connection. In this paper, we aim to develop a data-dependent approach to regularizing neural network in the framework of Information Geometry. A measurement for the quality of connections is proposed, namely confidence. Specifically, the confidence of a connection is derived from its contribution to the Fisher information distance. The network is adjusted by retaining the confident connections and discarding the less confident ones. The adjusted network, named as ConfNet, would carry the majority of variations in the sample data. The relationships among confidence estimation, Maximum Likelihood Estimation and classical model selection criteria (like Akaike information criterion) is investigated and discussed theoretically. Furthermore, a Stochastic ConfNet is designed by adding a self-adaptive probabilistic sampling strategy. The proposed data-dependent regularization methods achieve promising experimental results on three data collections including MNIST, CIFAR-10 and CIFAR-100.

1. Introduction

Neural networks (NNs) that consist of multiple hidden layers can automatically learn effective representation for a learning task, such as, speech recognition [1,2,3], image classification [4,5,6], and natural language processing [7]. However, a neural network with too many layers or units, especially deep neural networks (DNNs) [8], would easily overfit in the training phase and lead to a poor predictive performance in the testing phase. In order to alleviate the overfitting problem in DNNs, many regularization methods have been developed, including data augmentation [9], early stopping, amending cost functions with weight penalties (${\ell }_1$ or ${\ell }_2$), and modifying networks by randomly dropping a certain percentage of units (Dropout [10]) or connections (DropConnect [11]). This paper will focus on the last method.

The Dropout strategy randomly drops units (along with their connections) in a neural network during training. A large sum of sub-networks that randomly dropped units would be trained. While in the testing phase, a scaled-down weights of networks is used to approximatively achieve an averaged prediction from these sub-networks. The performance of DNNs have significantly improved by such ensemble strategy. In DropConnect [11], a network is regularized by randomly drawing a subset of connections independently from a uniform prior in the training phrase and using a Gaussian sampling procedure for activations in the inferencing phrase. Both Dropout and while inferencing DropConnect assume a uniform prior prior for the dropping strategy.

From the density estimation point of view, the hidden layers of DNNs can be interpreted as an attempt to recover a set of parameters for a generative model that describes the underlying distribution of the observed data [12]. A well-designed prior is needed for incorporating the degree of importance in describing the data distribution into the network. Specifically, we would need an efficient approach to recognize how much a unit or connection is useful for revealing the underlying structure of the current data. Recently, a general parametric reduction criterion, named the Confident-Information-First principle (CIF) [13], has been proposed, in the theoretical framework of Information Geometry (IG) [14]. From a model selection perspective, they proved that both the fully Visible Boltzmann Machine (VBM) and the Boltzmann Machine (BM) with hidden units can be derived from the general multivariate binary distribution using the CIF principle. Such a geometric method offers a more intuitive criterion for parametric reduction on the parameter space.

In this paper, we study the regularization networks for DNNs in the theoretical framework of IG. Every fully connected neighboring layer in a DNN would be assigned a BM that shares the same topology with it [15]. The confidence level of each connection in a BM is evaluated according to the sample data, and the confident connection would be retained in the DNN. Differing from the mechanism of uniform prior, which Dropout and DropConnect adopt, Confident Network (ConfNet in short) is a reduced model that reduces the number of free parameters by a data-dependent method. In addition, we adopt a self-adaptive dynamic sampling mechanism to reduce the network, and the re-adjusted network is called the Stochastic Confident Network (Stochastic ConfNet in short). Data-independent regularization and data-dependent are compared in several image datasets.

2. Theoretical Foundation of IG

2.1. Parametric Coordinate Systems

A family of probability distributions is considered as a differentiable manifold with certain parametric coordinate systems. Let n be the number of variables and S denote the open simplex of all probability distributions over binary vector $x\in {\{0,1\}}^n$. Four basic coordinate systems are often used [16,17] to characterize multivariate binary distributions.

$p-$coordinates $\left[p\right]$: the probability distribution over $2^n$ states of x can be completely specified by any $2^n-1$ positive numbers indicating the probability of the corresponding exclusive states on n binary variables. For example, the $p-$coordinates of $n=2$ variables could be $\left[p\right]=(p_{01},p_{10},p_{11})$. Use the capital letters $I,J,\dots$ to index the coordinate parameters of probabilistic distribution. An index I can be regarded as a subset of $\{1,2,\dots ,n\}$. Additionally, $p_I$ stands for the probability that all variables indicated by I equal to one and the complemented variables are zero. For example, if $I=\left\{2\right\}$ and $n=2$, we have:

$$p_I=p_{01}=Prob(x_1=0,x_2=1).$$

(1)

Note that the null set can also be a legal index of the $p-$coordinates, which indicates the probability that all variables are zero, denoted as $p_{0\dots 0}$.

$\eta -$coordinates $\left[\eta \right]$:

$${\eta }_I=E\left[X_I\right]=Prob\{\prod _{i\in I}{x}_i=1\},$$

(2)

where the index I is the nonempty subset of $\{1,2,\dots ,n\}$ and the value of $X_I$ is given by ${\prod }_{i\in I}{x}_i$ and the expectation is taken with respect to the probability distribution over x. For example, if $n=2$, the $\eta -$coordinates should be $\left[\eta \right]=({\eta }_1,{\eta }_2,{\eta }_{12})$.

$\theta -$coordinates $\left[\theta \right]$ are defined by:

$$logp(x)=\sum _{I\subseteq \{1,2,\dots ,n\},I\ne Nullset}{\theta }^I{X}_I-\psi (\theta ),$$

(3)

where $\psi (\theta )=log({\sum }_{x}exp\left\{{\sum }_I{\theta }^I{X}_I(x)\right\})$ is the cumulant generating function and it equals $-logProb\{x_i=0,\forall i\in \{1,2,\dots ,n\}\}$. By solving the linear system (3), we have ${\theta }^I={\sum }_{K\subseteq I}{(-1)}^{|I-K|}log(p_K)$, where $|·|$ denotes the cardinality operator. For example, if $n=2$, the $\theta -$coordinates should be $\left[\theta \right]=({\theta }^1,{\theta }^2,{\theta }^{12})$.

$l-$mixed $\zeta -$coordinates ${\left[\zeta \right]}_l$ are defined by:

$${\left[\zeta \right]}_l=[{\eta }^{l-},{\theta }_{l+}]=({\eta }_i^1,\dots ,{\eta }_{i,j,\dots }^l,{\theta }_{l+1}^{i,j,\dots },\dots ,{\theta }_n^{i,\dots ,n}),$$

(4)

where the first part consists of $\eta -$coordinates with order less than or equal to l, and the second part consists of $\theta -$coordinates with order greater than l, $l\in \{1,\dots ,n-1\}$. The order of the coordinate is equal to the number of variables in its index I. For example, if $n=2$ and $l=1$, the $\zeta -$coordinates should be ${\left[\zeta \right]}_1=({\eta }_1,{\eta }_2,{\theta }^{12})$.

2.2. The Fisher Information Matrix

Previously, we have introduced four commonly used coordinates. Another important concept in IG is the Fisher information matrix for parametric coordinates. For a general coordinate system $\left[\xi \right]$, the $i-$th row and $j-$th column element of the Fisher information matrix for $\left[\xi \right]$ (denoted by $G_{\xi }$) is defined as the covariance of the scores of $\left[{\xi }_i\right]$ and $\left[{\xi }_j\right]$ [18]:

$$g_{i,j}(\xi )=E[\frac{\partial logp(x;\xi )}{\partial {\xi }_i}·\frac{\partial logp(x;\xi )}{\partial {\xi }_j}]$$

(5)

under the regularity condition that the partial derivatives exist. The Fisher information measures the amount of information in the data that a statistic carries about the unknown parameters [19]. The Fisher information between ${\theta }_I$ and ${\theta }_J$ in $\left[\theta \right]$ is given by [20]:

$$g_{I,J}(\theta )={\eta }_{I\cup J}-{\eta }_I{\eta }_J.$$

(6)

The Fisher information between ${\eta }_I$ and ${\eta }_J$ in $\left[\eta \right]$ is given by [20]:

$$g_{I,J}(\eta )=\sum _{K\subseteq I\cap J}{(-1)}^{|I-K|+|J-K|}·\frac{1}{p_K},$$

(7)

where $|·|$ denotes the cardinality operator.

Another important concept related to our analysis is the orthogonality defined by Fisher information. Two coordinate parameters ${\xi }_i$ and ${\xi }_j$ are called orthogonal if and only if their Fisher information vanishes, i.e., $g_{ij}=0$, meaning that their influences on the log likelihood function are uncorrelated. Based on $G_{\eta }$ and $G_{\theta }$, we can calculate the Fisher information matrix $G_{\zeta }$ for the mixed coordinates ${\left[\zeta \right]}_l$ [13]:

$$G_{\zeta }=\left(\begin{array}{cc}A& 0\\ 0& B\end{array}\right),$$

(8)

where $A={({(G_{\eta }^{-1})}_{I_{\eta }})}^{-1}$, $B={({(G_{\theta }^{-1})}_{I_{\theta }})}^{-1}$, $G_{\eta }$ and $G_{\theta }$ are the Fisher information matrices of $\left[\eta \right]$ and $\left[\theta \right]$, respectively, $I_{\eta }$ is the index set of the parameters shared by $\left[\eta \right]$ and ${\left[\zeta \right]}_l$, i.e., $\{{\eta }_i^1,\dots ,{\eta }_{i,j,\dots }^l\}$ and $J_{\theta }$ is the index set of the parameters shared by $\left[\theta \right]$ and ${\left[\zeta \right]}_l$, i.e., $\{{\theta }_{l+1}^{i,j},\dots ,{\theta }_n^{1,\dots ,n}\}$. From $G_{\zeta }$, we can see that Fisher information between lower-order $\eta -$coordinates ${\eta }_I$ and higher-order $\theta -$coordinates ${\theta }_J$ are all zero in the mixed coordinate $\left[\zeta \right]$. For example, in $\left[\zeta \right]=({\eta }_1,{\eta }_2,{\theta }^{12})$, the Fisher information between ${\theta }^{12}$ and ${\eta }_1$ (or ${\eta }_2$) is zero. This indicates that ${\theta }^{12}$ is orthogonal to both ${\eta }_1$ and ${\eta }_2$ in the $\left[\zeta \right]$. The orthogonal property of $\left[\zeta \right]$ allows us to decompose the distance between two distributions into unrelated parts.

2.3. The Fisher Information Distance

The Fisher information distance (FID), i.e., the Riemannian distance induced by the Fisher–Rao metric [14], is adopted as the distance measure between two distributions, since it is shown to be the unique metric meeting a set of natural axioms for the distribution metric [14,21,22], e.g., the invariant property with respect to reparametrizations and the monotonicity with respect to the random maps on variables. Let $\xi$ be the distribution parameters. For two close distributions $p_1$ and $p_2$ with parameters ${\xi }_1$ and ${\xi }_1$, the Fisher information distance between $p_1$ and $p_2$ is:

$$D(p_1,p_2)=\sqrt{{({\xi }_1-{\xi }_2)}^T{G}_{\xi }({\xi }_1-{\xi }_2)},$$

(9)

where $G_{\xi }$ is the Fisher information matrix [17], and $p_1$ and $p_2$ are close.

3. Motivation

Considering a fully connected layer of a DNN with input $x={[x_1,x_2,\dots ,x_n]}^T$, weight parameters W (of size $n\times m$) and bias (of size $m\times 1$). The output of this layer, $y={[y_1,y_2,\dots ,y_m]}^T$ is computed as: $y=a(W^T\times x+b)$, where a is a non-linear activation function.

3.1. Data-Independent Regularization

Dropout is a widely used form of regularization on the structure of neural networks. It regularizes the network as follows: activation of each output unit is kept with probability p, otherwise set to 0 with probability $1-p$. Given sample x, the activations of output units can be calculated by

$$y=a(W^T\times (M\otimes x)+b).$$

(10)

The operator ⊗ denotes the element-wise product and M is a $(m\times 1)$ binary mask vector. The element of M is randomly drawn from Bernoulli prior with parameter p. This breaks up co-adaption of feature detectors since the dropped-out units cannot influence other retained units [23]. In addition, the number of trained models is $2^m$ , and these models share the same parameters; thus, the final trained NN is an ensemble NN.

For DropConnect, connections are chosen at random using the Bernoulli prior during the training stage. The activation function of DropConnect is modified as:

$$y=a({(M\otimes W)}^T\times x+g\otimes b),$$

(11)

where M is a $(n\times m)$ binary mask for weights and g is a $(m\times 1)$ mask for biases. The way to interpret DropConnect is also model averaging.

Both Dropout and DropConnect provide a data-independent strategy that assumes a uniform prior on the model parameters to sample a sub-network. Such strategy does not take into consideration the importance of the units or connections. In this paper, we focus on the selection of retained connections. Intuitively, the prior probability to keep a certain connection should be proportional to its importance in estimating the data distribution. One simple solution is to estimate the data distribution using both the fully connected NN and the sub-network that removes certain connections. The likelihood loss could be evaluated after removing them. However, it is generally infeasible in practice since we have to investigate all sub-networks. For example, if the NN has K free parameters, we need to exhaustively test all possible sub-networks $(2^K-1)$ and calculate the log likelihood value. Therefore, a heuristic data-dependent method is imperative to alleviating computation complexity and improving regularization effectiveness.

3.2. Data-Dependent Regularization

The problem of modifying networks would be restated in the theoretical framework of IG as follows. A general model S (with K free parameters) could be seen as a $K-$dimensional manifold. T (with $k<K$ free parameters) is a smoothed sub-manifold of S. The motivation of modifying DNNs is that the original geometric structure of S could be preserved as much as possible after projecting on the sub-manifold T.

Let $p_t,p_s\in S$ be the true distribution and the sampling distribution, respectively. Then, the choice of sub-manifold can be defined as the optimization problem to maximally preserve the expectation of the Fisher information distance with respect to the constraint of the parametric number, when projecting distributions from the parameter space of S onto that of the reduced sub-manifold T:

$$\begin{array}{c}\underset{T}{\mathrm{maximize}}{D}^2({\Gamma }_T(p_t),{\Gamma }_T(p_s)),\hfill \\ \mathrm{subject}\mathrm{to}T\mathrm{having}k\mathrm{free}\mathrm{parameters}.\hfill \end{array}$$

(12)

The reason for selecting the Fisher information distance as the distance measure has been referred to in Section 2.3.

CIF principle is proposed to solve the optimization problem in Equation (12), as follows. The Fisher information distance $D(p_t,p_s)$ can be decomposed into the distances of two orthogonal parts [17]. Moreover, it is possible to divide the system parameters in S into two categories (corresponding to the two decomposed distances), i.e., the parameters with major variations and the parameters with minor variations, according to their contributions to the whole information distance. The former refers to parameters that are important for reliably distinguishing the true distribution from the sampling distribution, while the parameters with minor contributions can be considered as less reliable.

Regularization of neural networks could be regarded as the optimization problem in Equation (12) with $p_s$ is known. In a data-dependent regularization, confidence of a connection is defined as its contributions to the whole information distance. The confident connections will be preserved, while less confident connections will be ruled out. By this strategy, we can directly decide the reliable set of parameters from all $k-$dimensional sub-models by selecting the top$-k$ confident parameters. Then, we only need K trials to conclude to the reliable solution.

4. Model

4.1. Restricted Boltzmann Machine in IG

Restricted Boltzmann Machine (RBM) is a special case of general BM, for which RBM only has connections between visible and hidden units, while it does not have connections between visible and visible units, or hidden and hidden units. Let $v=(v_i)$, $v_i\in \{0,1\}$ be the state of visible units, and $h=(h_i)$, $h_i\in \{0,1\}$ be the state of hidden units. The entire state of an RBM can be represented as $\{v,h\}$. The energy function of RBM is as follows:

$$E_{RBM}(v,h;\xi )=-\{v^{T}Wh+b_v^{T}v+b_h^{T}h\},$$

(13)

where $\xi =\{W,b_v,b_h\}$ are the parameters. W are the connection weights between visible and hidden units. $b_v$ and $b_h$ are the visible units thresholds and hidden units thresholds, respectively. An RBM produces a stationary distribution $p(v,h;\xi )\in S_{vh}$ over $\{v,h\}$. Let B denote the manifold with probability distributions $p(v,h;\xi )$ realizable by RBM. The distribution $p(v,h;\xi )$ is as below:

$$p(v,h;\xi )=\frac{1}{Z}exp\{-E_{RBM}(v,h;\xi )\},$$

(14)

where Z is a normalization factor. The $\theta -$coordinates (Equation (3)) for B are: ${\left[\theta \right]}_{RBM}=({\theta }_1^i=b_{v_i},{\theta }_1^j=b_{h_j},{\theta }_2^{ij}=W_{ij},0,\dots ,0)$. $\left[\zeta \right]=({\eta }_i,{\eta }_j,{\theta }^{ij})$ is the $1-$mixed $\zeta -$coordinates for B. Each ${\theta }^{ij}$ corresponds to one connection $W_{ij}$. Based on the Theorem 1 in [14], $W_{ij}(i.e.,{\theta }^{ij})$ could be expressed as follows: $W_{ij}=logp(v_i=h_j=1|A)+logp(v_i=h_j=0|A)-logp(v_i=1,h_j=0|A)-logp(v_i=0,h_j=1|A)$, where the relation holds for any conditions A on the rest variables. However, it is often infeasible for us to calculate the exact value of $W_{ij}$ because of data sparseness. To tackle this problem, we propose to approximate the value of $W_{ij}$ by using the marginal distribution $p(v_i,h_j)$ to avoid the effect of condition A. According to Equation (2), ${\eta }_i^1=E\left[v_i\right],{\eta }_j^1=E\left[h_i\right]$. Then, the parametrization for RBM in $1-$mixed $\zeta -$coordinates could be calculated.

4.2. The Confidence of a Connection

Each ${\theta }^{ij}$ corresponds to one connection $W_{ij}$. The calculations of confidence over individual connections are not conditionally independent, and a greedy way is adapted to measure the confidence of each connections independently. For the purpose of deciding to keep or drop $W_{ij}$, we consider the two close distributions $p_1$ and $p_2$ with coordinates ${\zeta }_1=\{{\eta }_i,{\eta }_j,{\theta }^{ij}\}$ and ${\zeta }_2=\{{\eta }_i,{\eta }_j,0\}$, respectively. Each $p_1$ represents the case that $W_{ij}$ is kept and $p_2$ indicates the case that $W_{ij}$ is dropped.

Definition 1.

The confidence of ${\theta }^{ij}$denoted as $\rho ({\theta }^{ij})$ can be defined as the Fisher information distance between $p_1$ and $p_2$:

$$\rho ({\theta }^{ij})={({\zeta }_1-{\zeta }_2)}^T{G}_{\zeta }({\zeta }_1-{\zeta }_2)={\theta }^{ij}·g_{\zeta }({\theta }^{ij})·{\theta }^{ij},$$

(15)

where $G_{\zeta }$ is the Fisher information matrix of $\left[\zeta \right]$ in Section 2.2 and $g_{\zeta }({\theta }^{ij})$is the Fisher information for${\theta }^{ij}$.

Note that the second equality holds since ${\theta }^{ij}$ is orthogonal to ${\eta }_i$ and ${\eta }_j$. Therefore, the Fisher information distance between two distributions can be decomposed into two independent parts: the information distance contributed by $\{{\eta }_i,{\eta }_j\}$ and $\{{\theta }^{ij}\}$. The detailed definition and calculation for the Fisher information matrix $\left[\zeta \right]$ are described in Equation (5).

From the point of Maximum Likelihood Estimate (MLE) and Akaike information criterion (AIC), respectively, the rationality of retaining confident connections would be interpreted.

The log-likelihood value of a NN mentioned above is:

$$l=logp(x;\zeta ).$$

(16)

The log-likelihood values of including $W_{ij}$ and excluding $W_{ij}$ in coordinates $\left[\zeta \right]$, respectively, are $l_1=logp(x;{\zeta }_1)$ and $l_2=logp(x;{\zeta }_2)$. The gap between $l_1$ and $l_2$ is estimated as

$$\begin{array}{cc}\hfill \Delta l_{12}& =logp(x;{\zeta }_1)-logp(x;{\zeta }_2)\hfill \\ & =\sum _{i=1}^{N}log\frac{p(x_i;{\zeta }_1)}{p(x_i;{\zeta }_2})\hfill \\ & \approx NE\left[log\frac{p(x;{\zeta }_1)}{p(x;{\zeta }_2}\right]\hfill \\ & \approx ND\left[p(x;{\zeta }_1):p(x;{\zeta }_2)\right]\hfill \\ & \approx \frac{1}{2}N{g}_{\zeta }({\theta }^{ij}){{\theta }^{ij}}^2.\hfill \end{array}$$

(17)

Then, we have

$$\rho ({\theta }^{ij})\approx \frac{2}{N}\Delta l_{12},$$

(18)

where N is the number of samples, and $D(·,·)$ denotes the Kullback–Leibler divergence. The first approximation holds in an asymptotical sense, and the third approximation is entailed by the approximate relation between Kullback–Leibler divergence and Riemannian distance induced by the metric tensor $G_{\zeta }$ [24]. Obviously, as the number of samples is fixed as N, the confidence of a connection is approximately equal to $\frac{2}{N}$ times the log-likelihood gap that the connection undertakes. Hence, the higher confidence means the bigger log-likelihood gap, i.e., preserving the more primary likelihood-structure.

In model selection, given a model with K parameters ${\xi }_K$, the model selection would select a sub-model with k parameters ${\xi }_k$. AIC is a common model selection criterion. In AIC [25], $AIC=-2logL({\widehat{\xi }}_k)+2k$. L is the likelihood function given the observed samples and ${\widehat{\xi }}_k$ is the maximum likelihood estimate of ${\xi }_k$. Geometrically speaking, when k is fixed, the minimization of AIC, i.e., maximizing the log likelihood $logL({\widehat{\xi }}_k)$, is asymptotically equivalent to the minimization of the Kullback–Leibler divergence $KL(p_s;p_{{\xi }^k})$ [26]. The consistency of the data-dependent regularization and AIC is revealed, when k is fixed.

4.3. ConfNet

In this section, we introduce the implementation details of ConfNet, as shown in Algorithm 1. As mentioned, the confidence of connections between the fully connected neighboring layers equals to the confidence of connections in corresponding RBM. First, for each input v, we could sample the hidden state h from their activation probabilities. Then, we get an extended dataset of N samples from the joint space of $\langle v,h\rangle$. Let $p(v_i,h_j)$ denote the marginal sampling distribution of input unit $v_i$ and hidden unit $h_j$. To estimate $p(v_i,h_j)$, we need to go through all N samples and count the number of samples for each assignment of $v_i$ and $h_j$. For example, $p(v_i=0,h_j=0)=\frac{count(v_i=0,h_j=0)}{N}$, $p(v_i=0,h_j=1)=\frac{count(v_i=0,h_j=1)}{N}$, etc. The $1-$mixed $\zeta -$coordinates for RBM could be estimated from the marginal distribution $p(v_i,h_j)$ as mentioned in Section 4.1. Then, the Fisher information is calculated by Equation (7), and the confidence is calculated by Equation (15).

Algorithm 1: Data-dependent Regularization.

For deciding connections to retain or discard, we need to judge whether the confidence of ${\theta }^{ij}$, i.e., the Fisher information distance in the coordinate direction of $\{{\theta }^{ij}\}$, is significant or nonsignificant. We set up the hypothesis test for confidence $\rho$, i.e., null hypothesis $\rho =0$ versus alternative $\rho \ne 0$. Based on the analysis in [27], we have $N·\rho \sim {\chi }^2(1)$ asymptotically, where the ${\chi }^2(1)$ is chi-square distribution with degree of freedom 1 and N is the sampling number. Then, we could calculate the confidence probability to reject the null hypothesis as follows:

$$\pi \leftarrow cd{f}_{{\chi }^2(1)}(N·{\rho }_{ij}),$$

(19)

where $cdf$ is the cumulative distribution function. Following the conventions in the hypothesis test, we set the significant level $\alpha$ to $0.05$. If $(1-\pi )·2<\alpha$, then the connection $W_{ij}$ will be kept in the DNN; otherwise, we will set $W_{ij}$ to zero. Based on a hypothesis test scheme, we could directly derive the mask M.

4.4. Stochastic ConfNet

In ConfNet, the confident connections are retrained while the less confident are dropped without discrimination. In addition, due to the fact that the calculations of confidence over individual connections are not conditionally independent, such a greedy strategy usually can only find the local optimal solution instead of a global one. In particular, the final performance is likely to be worse with the greed process if the previous selected sub-network is far away from the optimal.

To relieve such a problem, a probabilistic sampling strategy in ConfNet (Stochastic ConfNet) is adopted to choose a subset of confident connections over a biased distribution. The retained probability of a connection in such a distribution is proportional to the confidence. Based on such a sampling strategy, connections with higher confidence would be more likely to be chosen than the lower confidence ones, while the binary strategy directly drops the connections with lower confidence out. All the connections could be chosen in Stochastic ConfNet, even if it has a low confidence, which leads to a more robust optimal process.

The number of retained connections is dynamic self-adaptive along with the training process. Figure 1 explains the intentions of the self-adaptive sampling probability. The average confidence of all connections tends to be higher and higher with the training process, while the network carries more variations by training. Intuitively, the quality of the connections should be better and better from the random initialization of weights to the well-trained weights. Thus, the self-adaptive sampling probability is in view. Min-Max normalization of confidence $\rho$ is carried out, and then the sampling probability is calculated as follows:

$$\tilde{p}=p^{i-1}-\beta \left(\frac{average({\rho }^i)-average({\rho }^{i-1})}{average({\rho }^{i-1})}\right),$$

(20)

$$p^i=mean(\tilde{p},p^{*}),$$

(21)

where $p^i$ and $p^{i-1}$ are the dropping ratio at the $i-$th iteration and $(i-1)$-th the iteration, respectively, while $1-p^i$ and $1-p^{i-1}$ are the ratio for retaining connections. $\beta$ and $p^{*}$ is a constant between 0 and 1. The initial value $p^0$ could be set as the dropping ratio of ConfNet at the first iteration, or 0.5, for simplicity’s sake.

4.5. Training with Back-Propagation

In Algorithm 1, a regularization mask M for weights W is calculated. A reduced NN is obtained by $W\leftarrow M\otimes W$ while b remain unchanged. Then, the reduced NN would be trained by back-propagation. The training of ConfNet or Stochastic ConfNet stops when the classification error converges on the training dataset. The above approach is described in Algorithm 2.

Algorithm 2: Training with Back-propagation.
	Input: A fully connected layer of a DNN with input $\{x_1,x_2,\dots ,x_n\}$; N input samples $D_x$
	Output: weights W and biases b
1	Initialize W and b
2	while classification error converges do
3	Calculate the mask M via Algorithm 1
4	Reduce NN by $W\leftarrow M\otimes W$
5	Feed-forward: $y=a(W^T\times x+b)$
6	Differentiate loss with respect to W and b
7	Update W and b using the back-propagated gradients
8	end
9	returnW and b

5. Experiments

5.1. Experimental Setup

The MNIST dataset [28] consists of $28\times 28$ pixel handwritten digit images. The task is to classify the images into 10 digit classes. Each digit has the 60,000 training images and 10,000 test images. We scale the pixel values to the $[0,1]$ range before inputting to our models. The model includes an 800-unit fully connected layer, denoted as ➀, and the sigmoid activation function is adopted, with pre-training [15]. We also test a model with two 800-unit fully connected layers, denoted as ➁, and sigmoid activation function is adopted, without pre-training. Back-propagation is used to train neural networks with the learning rate setting at 0.1. No data augmentation is utilized in the experiments.

The CIFAR-10 dataset [29] consists of $32\times 32$ natural RGB images. The task is to classify the images into 10 classes. Each class has 50,000 training images and 10,000 testing images. We also scale the pixel values to the $[0,1]$ and subtract the mean value of each channel computed over the dataset for each image. The feature extractor is: a convolutional layer with 32 feature maps and $5\times 5$ filters, a Max pooling layer with region $3\times 3$ and stride 2, a convolutional layer with 32 feature maps and $5\times 5$ filters, a Max pooling layer with region $3\times 3$ and stride 2, a convolutional layer with 64 feature maps and $5\times 5$ filters, and a Max pooling layer with region $3\times 3$ and stride 2. After that, we input the extracted features into a fully connected layer with 64 units. Sigmoid activation functions is used. Regularization is applied on the fully connected layer. No pre-training is used.

The CIFAR-100 dataset [29] is similar to the CIFAR-10, while the task is to classify images into 100 classes. Each class has 600 images, including 500 training images and 100 testing images. The feature extractor, parameter and training setting are the same as CIFAR-10 except that there are two fully connected layers with 128 units, respectively, on the top of the feature extractor. Sigmoid activation functions are used. Regularization is applied on both of the two fully connected layers. No pre-training is used.

5.2. Experimental Results

The classification results on MNIST, CIFAR-10 and CIFAR-100 are shown in

Table 1. Comparison of test classification errors (%).

Model	MNIST		CIFAR-10		CIFAR-100
	Top-1 ➀	Top-1 ➁	Top-1	Top-5	Top-1	Top-5
FullConnect	1.84	1.56	20.09	1.32	52.02	23.26
Dropout	1.79	1.43	19.87	1.24	51.18	23.02
DropConnect	1.77	1.59	18.53	1.19	50.87	22.79
Confnet	1.62	1.19	19.48	1.31	50.06	22.74
Stochastic ConfNet	1.50	1.19	18.25	1.12	48.39	19.96

The results in bold are the results with lowest error rate. On MNIST, the retained probability p of Dropout is $0.5$; the retained probability p of DropConnect is $0.5$; the hyper-parameters $\beta$ and $p^{*}$ of Stochastic ConfNet are $0.025$ and $0.1$, respectively; and the initial value $p^0$ is $0.6$. On CIFAR-10, the retained probability p of Dropout is $0.5$; the retained probability p of DropConnect is $0.5$; the hyper-parameters $\beta$ and $p^{*}$ of Stochastic ConfNet are $0.1$ and $0.4$, respectively; and the initial value $p^0$ is $0.6$. On CIFAR-100, the retained probability p of Dropout is $0.5$; the retained probability p of DropConnect is $0.5$; the hyper-parameters $\beta$ and $p^{*}$ of Stochastic ConfNet are $0.1$ and $0.4$, respectively; and the initial value $p^0$ is $0.6$.

. On all experimental datasets, the DNNs with regularization achieve better performances than the DNNs without regularization. This could reflect the existence of overfitting during training, and the effectiveness of regularization method (reducing DNN). Moreover, in most cases, the test classification errors of the DNNs with data-dependent regularization (i.e., Confnet, Stochastic ConfNet) are obviously lower than the DNNs with data-independent regularization (i.e., Dropout, DropConnect). Among all experimental datasets, the Stochastic ConfNet gives the best performances, whether for the top-1 classification errors or for the top-5 classification errors. The Confnet also performs well.

The next experimental analyses are all based on the experimental phenomena on MNIST. Figure 2a displays the train error curve and test error curve along with the training iteration. We can see that the gap between train error curve and test error curve is reduced by regularization, and all DNNs with regularization (Dropout, DropConnect, ConfNet and Stochastic ConfNet) significantly outperform the fully connected DNNs. The ConfNet and Stochastic ConfNet achieve superior performance.

Figure 2b shows the performance of different regularization methods with the increase of network size. Seven setups for the number of hidden units, i.e., $[200,400,600,800,1000,1200,1400]$, are tested. For almost all model sizes, ConfNet and Stochastic ConfNet consistently give a lower error rate than the fully connected DNNs, especially for larger model sizes (e.g., 800, 1000).

In Figure 3a, the average confidence of each model increases steadily over epochs, especially ConfNet and Stochastic ConfNet. It is reasonable that the network would carry more useful information with training, and the regularization by ConfNet and Stochastic ConfNet achieve it better. Figure 3b is about the standard deviation of confidence of the connections. The large standard deviation means a strong ability of discovering the significant connections. Stochastic ConfNet is most outstanding on these two indices.

In Figure 4, the confidence of connections is processed by normalization.The x-axis means confidence with normalization, and distribution of confidence on several different epochs are shown in Figure 4. With the increase of training iterations, the distribution curve of confidence moves to the right continually. It means that the confidence becomes larger with training, while the network carries more useful information.

A checking experiment that removes confident connections at each epoch is also implemented. Figure 5 shows the results. The test error remains high, and model can not be convergent. At about 150 epochs, the test error fluctuates at around $18.5\%$ test errors.

6. Conclusions

In conventional regularization methods (Dropout, DropConnect), the network is regularized by randomly drawing a sub-network from a uniform prior. In this paper, we propose a data-dependent regularization method inspired by the Confident Information First (CIF) principle based on Information Geometry. We have proven that retaining highly confident connections means that the more primary likelihood-structure is preserved, and such a strategy is consistent with the AIC. Moreover, a self-adaptive probabilistic sampling process is used to fit the change of confidence along training to obtain more effective regularization. Empirical evaluation results have demonstrated the superiority of our approach.

Our future research will be focused on the following directions. First, a more generalized theory to demonstrate the relationships among CIF, compressed sensing and random projection will be explored. Second, the ConfNet is currently obtained by a greedy algorithm, so the relative error and time complexity need to be measured. Third, there is a sampling process before calculating the confidence of a connection, which has a negative impact on efficiency. Therefore, a more efficient method will be investigated. Fourth, the application and evaluation of the proposed methods on extra large networks and extending them onto the convolution layer will also be considered in future work.