Finding the Optimal Topology of an Approximating Neural Network

Abstract

A large number of researchers spend a lot of time searching for the most efficient neural network to solve a given problem. The procedure of configuration, training, testing, and comparison for expected performance is applied to each experimental neural network. The configuration parameters—training methods, transfer functions, number of hidden layers, number of neurons, number of epochs, and tolerable error—have multiple possible values. Setting guidelines for appropriate parameter values would shorten the time required to create an efficient neural network, facilitate researchers, and provide a tool to improve the performance of automated neural network search methods. The task considered in this paper is related to the determination of upper bounds for the number of hidden layers and the number of neurons in them for approximating artificial neural networks trained with algorithms using the Jacobi matrix in the error function. The derived formulas for the upper limits of the number of hidden layers and the number of neurons in them are proved theoretically, and the presented experiments confirm their validity. They show that the search for an efficient neural network can focus below certain upper bounds, and above them, it becomes pointless. The formulas provide researchers with a useful auxiliary tool in the search for efficient neural networks with optimal topology. They are applicable to neural networks trained with methods such as Levenberg–Marquardt, Gauss–Newton, Bayesian regularization, scaled conjugate gradient, BFGS quasi-Newton, etc., which use the Jacobi matrix.

1. Introduction

Neural networks are widely used in various fields: education, healthcare and medicine, insurance and financial activities, business and industrial production, etc. They are used to solve various practical tasks: data classification, predictions, sound and image recognition, nonlinear control, system analysis and diagnosis, fault detection analysis, etc.

A major difficulty in solving problems with neural networks is to determine the topology of the network—how many hidden layers and how many neurons the neural networks should have. This question is of great importance because finding an efficient solution depends on the choice of topology. The random selection of a number of hidden neurons might cause either overfitting or underfitting problems.

Some applications use neural networks with a single hidden layer. In cases where there are multiple input data, the number of neurons in such shallow architecture is large. In many cases, it is more appropriate to use alternative architectures—deep neural networks (DNNs), with more hidden layers and fewer neurons. DNNs have much more power and can be used to solve much more complex tasks. On the other hand, in order to have good generalization abilities, the network must be as compact as possible [1]. The topology of the neural network is of key importance in the processes of memorization and retrieval of patterns [2]. Emmert-Streib conducted multiple experiments to solve a single task by using multiple neural networks with different topologies. He proved that the neural network topology has a great influence on its learning dynamics [3].

Many scholars have worked on the task of determining the optimal network topology. Various formulas have been published that relate to solving specific problems or specific neural networks. The wide variety of different types of neural networks and their use make it impossible to find a universal solution for choosing an appropriate topology.

Automating the process of finding neural networks with a suitable topology is usually done by a procedure that iteratively constructs different neural networks and checks whether they are acceptable depending on a predetermined tolerance. This is done by changing certain parameters, e.g., the number of neurons, the number of hidden layers, epochs, etc. Criteria are needed to determine within what limits these parameters should be changed.

This paper derives formulas for determining the maximum number of hidden layers and the maximum number of neurons in a neural network design. They provide researchers with starting points and guidance in building efficient neural networks. The subject of research is neural networks that have the same number of neurons in the hidden layers and are trained with algorithms using the Jacobi matrix. Some of the widely used training methods are Levenberg–Marquardt [4,5], Gauss–Newton [6,7], Bayesian regularization [8,9], scaled conjugate gradient [10,11], BFGS quasi-Newton [12,13], etc.

2. Determining the Topology of an Artificial Neural Network—State of the Art

Many scientists are working on the task of determining an appropriate neural network topology [14,15,16]. They use different approaches and reach different conclusions applicable to different situations.

One of the problems is related to determining the number of neurons that an artificial network should have [17,18]. Single-parameter approaches where the number of neurons depends on a single parameter—the number of network inputs—can be seen in the works of Chow et al. [19], Tamura and Tateshi [20], and Sheela and Deepa [21]. For example, Chow et al. proposed the following formula to determine the number of neurons needed:

$$N_h=\frac{\sqrt{1+8{\mathrm{N}}_{\mathrm{i}}}-1}{2}$$

where $N_i\left(i\in N\right)$ is the number of input stimuli [19].

Sheela and Deepa researched Elman neural networks. To fix the number of hidden neurons as a function of the input parameters $n$, they examined 101 various criteria and compared statistical errors [21]. They arrived at the following formula:

$$a_n=\left(4n^2+3\right)/\left(n^2-8\right)$$

Madhiarasan and Deepa developed a methodology for estimating the number of hidden neurons in a multilayer perceptron NN. They derived the following formula:

$$u_n=\left(4n-2\right)/\left(n-3\right)$$

where $n$ is the number of input parameters. They performed a comparative analysis for estimating the number of hidden neurons and validated the applicability of the formula with the convergence theorem [22].

Other scientists define the number of neurons as a function of two parameters. For example, in the work of Xu and Chen [23], the number of neurons is determined by two parameters—$N_t$ and $N_i\left(i,t\in N\right)$—the number of input–output training samples and the number of network inputs, respectively:

$$N_h=\{\begin{array}{c}\frac{1}{2}\frac{N_t}{N_i\log {\mathrm{N}}_{\mathrm{t}}},\frac{N_t}{N_i}>30\\ \frac{N_t}{N_i},\frac{N_t}{N_i}\le 30\end{array}$$

A similar two-parameter approach can be found in [14,24].

In one of our research projects, we are looking for an optimal number of neurons for the cases in which neural network training methods use the Jacobi matrix in their error function [25]. Such training methods are Levenberg–Marquardt, Gauss–Newton, Bayesian regularization, scaled conjugate gradient, BFGS quasi-Newton, etc. As a result, we have derived formulas with which we determine an upper bound for the number of neurons $q$ in the hidden layers for single-layer and multi-layer neural networks, as follows:

In single-layer neural networks:

$$q\le \frac{m-1}{n+2}$$

(1)

and in multi-layer neural networks with r hidden layers in which there is an equal number of neurons in each layer:

$$q\le \frac{\sqrt{{\left(r+n+1\right)}^2+4\left(r-1\right)\left(m-1\right)}-\left(r+n+1\right)}{2\left(r-1\right)}$$

(2)

In both cases, $m$ is the number of input–output training samples, and $n$ is the number of input stimuli. However, even in this case, it is a matter of situation in which the considered neural networks meet certain conditions:

(1)

Neural networks are trained with an algorithm using the Jacobi matrix in the error function;

(2)

There is an equal number of neurons in each of the hidden layers;

(3)

The network has one output neuron.

Other scientists have directed their efforts to search for an optimal neural network model from the point of view of layered architecture. For example, Ref. [26] discusses the possibilities of finding the number of layers of networks with backpropagation of the error. It has also been emphasized that the number of hidden layers is related to the number of neurons and thus affects the time of training the neural network and the magnitude of the error. This once again confirms that the choice of a number of hidden layers, respectively neurons, has a great influence on the performance of the entire neural network. On the other hand, Choldun et al. address this issue from the perspective of classification tasks [27]. Stathakis looks for a solution for the design of topology in neural networks for classification problems. He proposes a method based on the synergy of genetic algorithms and neural networks. The method is only applicable to topologies with up to two hidden layers [28].

Hanay et al. propose a method for network topology selection based on the attractor selection-based (ASB) algorithm. They perform virtual topology reconfiguration using multistate neural memories [29].

Perzina works on the problem of neural network topology optimization. He offers a self-learning genetic algorithm with the self-adaptation of all its parameters. This is an interesting possibility for self-adaptation, not only for one parameter or several ones, but for all possible parameters of genetic algorithms at the same time. This allows the algorithm to be used to solve a wide range of optimization problems without setting parameters for each type of problem in advance [30].

Vizitiu and Popescu propose an interesting approach to improve the quality of neural network architecture. They develop a genetic algorithm, which simultaneously optimizes both the topology and neural weights of a feedforward neural network. The Gann system architecture presented by them consists of two main modules. The first genetic module has the task of optimizing the connectivity (i.e., the number of neurons, layers, and neural weights) of a feedforward neural network. The second module is responsible for optimizing the distribution of the neural weight which are assigned to this [31]. Other solutions for optimizing NN topologies based on genetic algorithms are proposed by White and Ligomenides [32], Arena et al. [33], etc.

Leon proposes a method for optimizing the topology of neural networks based on the Shapley value: a game-theoretic solution concept that estimates the contribution of each network element to the overall performance. With it, more network elements can be simultaneously pruned. After each network simplification, an evolutionary hill-climbing procedure is used to fine-tune the network [34].

A method for estimating the neural network topology by examining multiple spike sequences has been proposed by Kuroda and Hasegawa. In the proposed method, they used the SPIKE distance, which is a parameter-free measure for qualifying the distance between spike sequences, and leveraged applied partialization analysis to the SPIKE distance [35].

While working on a specific task, some scientists create different topologies using various methods and methodologies, such as the methodology based on optimizing neural network parameters, methods based on genetic algorithms, differential evolution, etc. After that, they examine these methodologies and select the one with the best performance [36].

Other research related to neural network architectures aims at comparing the performance of neural networks with one and two hidden layers. Guliyev and Ismailov explore possibilities for the approximation of multivariate functions by single-layer and two-layer feedforward neural networks with fixed weights. They prove that single hidden layer networks cannot approximate all continuous multivariate functions. This is also a criterion for choosing an NN topology [37]. Thomas et al. conducted research to determine whether feedforward neural networks with two hidden layers generalize better than a network with a single hidden layer. They propose a method called “transformative optimization” through which they perform a hidden-node-by-hidden-node comparison of single-layer and double-layer feedforward neural networks across ten separate datasets. They determined that in nine out of ten cases, the two-layer neural networks had a better performance [38]. Nakama uses another approach to compare single- and multiple-hidden-layer networks. For both types of neural networks, he uses the same activation and output functions, number of nodes, feedforward connections, parameters, and inputs. The networks are trained by the gradient descent algorithm to approximate linear and quadratic functions, and their convergence properties are examined. It has been established that the single-hidden-layer network converges faster than the multiple-hidden-layer network only in cases where a linear function is approximated. In all other cases, it is recommended to use multilayer neural networks [39].

Other approaches to determining the topology of neural networks are systematized by Choldun, Santoso, and Surendro [40].

Thus, in the end, the answer to the question of the required number of layers remains open. However, at the same time, it is important enough to be one of the most popular topics in specialized forums, training seminars, and scientific conferences, and the answer is almost always related to the need for experimentation in each specific case or task. The difficulty of determining the optimal number of layers also stems from the fact that the influence of the number of layers on the network performance is ambiguous.

The experiments conducted within the current study show that expanding the layered architecture often leads to a decrease in network efficiency. A formula is derived for the upper bound of the number of neurons in the generalized case of neural networks with several output neurons, and it has been proved that this formula can help determine the upper limit of the number of layers, thus facilitating the search for the optimal neural network.

3. Experiments to Search for a Relationship between the Number of Layers and the Efficiency of the Network

For the purpose of this study, experiments were conducted to search for approximating neural networks using different types of objective functions: linear, quadratic, trigonometric, logarithmic, exponential, and mixed. Using the MatLab scripting language [41], fitting neural nets were modeled for these functions with different numbers of layers under the following conditions:

(1)

A total of 286 input–output pairs were used, with approximately 70% of them (200 samples) for training, 15% (43 samples) for validation, and 15% (43 samples) for an independent test of the neural networks.

(2)

The transfer function in the body of the hidden neurons is a hyperbolic tangent:

$$Tanh\left(x\right)=\frac{2}{1+e^{-2x}}-1.$$

(3)

Training is done with the Levenberg–Marquardt algorithm.

(4)

The number of layers in the network, in each particular approximation case, is changed sequentially from 1 to 100 using a purpose-built algorithm implemented through the MatLab scripting language.

(5)

The number of neurons for each hidden layer in all neural networks is three.

(6)

Uniform, randomly generated input–output datasets were used for training, validation, and testing. This eliminates the risk of influencing the neural networks’ performance being affected by dataset randomization.

Neural network efficiency was determined using this formula:

$$efficiency=\frac{1}{performance}$$

where $performance$ is the aggregate performance of the network during training, validation, and testing. The results of the conducted experiment on how the number of layers influences the accuracy when neural networks are used to approximate the objective functions are presented in Figure 1.

It is obvious that there is no universal rule that relates the performance of a neural network to the number of hidden layers. In some cases, increasing the layers leads to a drastic decrease in accuracy right after the first layer (e.g., when approximating functions such as $y=2x+1$ and $y=x^2$). In other cases, the optimal number of layers is reached later, after which the efficiency remains at a certain level (e.g., functions $y=\cos \left(x\right),y=\sin \left(x\right))$. In the third cases (e.g., $y=e^x$), after the peak in efficiency, multiple secondary lower highs and lows are observed. This suggests that finding the optimal number of layers is a strictly individual decision that depends on the specific task and is a matter of intuition and experimentation on the part of the researcher.

On the other hand, with a fixed number of hidden neurons, increasing the number of layers leads to a decrease in efficiency. Figure 2 shows some of the results for neural networks with different numbers of neurons in the hidden layers approximating the two-dimensional function $y=2x_1+3x_2+1$.

The Levenberg–Marquardt algorithm was used to train the considered neural networks, using 100 (or 70%) of a total of 143 input–output samples. For validation and testing, 15% of the sample was applied for validation, and the remaining 15% was used for testing, and the transfer function in the neuron body is a hyperbolic tangent. The obtained results lead to the conclusion that after a certain number of layers, the efficiency of neural networks decreases significantly. In the cases described here, for neural networks with up to six neurons in each layer, the numerical value of the efficiency after a certain expansion of the layered architecture goes through some short fluctuations and approaches zero. Similar results were obtained with the experiments when the number of neurons in each layer was increased, as well as when trying to approximate the other investigated functions.

Even if we make the objective function a bit more complicated, for example:

$$y=\sin \left(x_1\right)+\cos \left(x_2\right)+1$$

the observation is again that after the optimal network is modeled at the 11th layer, with

$$efficiency=0.1639$$

the efficiency drops, then begins to maintain a lower level characterized by very sharp and significant drops (Figure 3).

The results of the experiments raise two questions: 1. What is the reason for the unpredictable change in neural network efficiency? 2. How to explain the fact that, in some cases, too many hidden layers lead to low network efficiency, which makes it pointless to use huge approximating architectures?

4. Generalized Formula for Determining the Upper Bound of the Number of Neurons in Artificial Neural Networks

In a previous study, we derived formulas for the upper bounds of the number of neurons for neural networks using a Jacobi matrix with a single output neuron (Formulas 1 and 2) [25]. Let us now consider the more general case of a neural network with $k$ number of output neurons (Figure 4) and look for analogous formulas for an optimal upper bound on the number of neurons.

Theorem 1 (upper bound for the number of neurons in hidden layers). 

For an approximating neural network that is trained with an algorithm using the Jacobi matrix for the error function and that has the same number of neurons$q$in each layer, the highest approximation efficiency is achieved for some$q$:

$$q\le \frac{\sqrt{{\left(r+n+k\right)}^2+4\left(r-1\right)\left(m-k\right)}-\left(r+n+k\right)}{2\left(r-1\right)},$$

where:

$m$is the number of training samples,

$n$—the number of input stimuli,

$r$—the number of hidden layers,

$k$—the number of output neurons ($m,n,r,k\in N)$.

Proof. 

Let us consider an artificial neural network for which the following conditions are met:

(1)

There are $q$ number of neurons in each layer ($q\in N)$.

(2)

The network is trained with an algorithm using the Jacobi matrix of the error function using $m$ number of training examples:

$${\left\{\left[x_{j1},x_{j2},\dots x_{jn};t_j=F\left(x_{j1},x_{j2},\dots x_{jn}\right)\right]\right\}}_{j=1}^m$$

(3)

Let $n$ input stimuli propagate along the dendritic tree of the network, and it has $r$ hidden and $k$ output neurons ($n,r,k\in N)$.

A commonly used method for finding a set of weights that achieve a minimum of the error function is the iterative procedure in the Gauss–Newton algorithm. It is aimed at finding the minimum of the function

$$S\left(z\right)={\displaystyle \sum }_{i=1}^m{\left[{\phi }_i\left(z\right)\right]}^2.$$

In our case, $z$ are $p$-dimensional points:

$$z=\left(w_1,w_2,w_3,\dots w_p\right)$$

where $p$ is the number of all neurons (with weights $w_i,i=1,2,\dots p)$ in the network, and the number of

$${\phi }_i\left(z\right)=t_i-f(x_{i1},x_{i2},\dots x_{in};w_1,w_2,\dots w_p),\mathrm{for}i=1,2,\dots m.$$

is determined by the number of training examples.

After initializing the network with a random set of weights

$$z_0=z_0(w_{01},w_{02},w_{03},\dots w_{0p})$$

at the $k$-th iteration we have:

$$z_{\left(k\right)}=z_{\left(k-1\right)}-{\left[J_{\phi }{\left(z_{\left(k-1\right)}\right)}^T{J}_{\phi }\left(z_{\left(k-1\right)}\right)\right]}^{-1}{J}_{\phi }{\left(z_{\left(k-1\right)}\right)}^T\phi \left(z_{\left(k-1\right)}\right),$$

(3)

where

$$\phi =\left({\phi }_1,{\phi }_2,\dots {\phi }_m\right),\forall {\phi }_i\left(z\right):$$

$${\phi }_i\left(z\right)=t_i-f(x_{i1},x_{i2},\dots x_{in},w_1,w_2,\dots w_p),\mathrm{and}$$

$$J_{\phi }\left(z_{\left(k-1\right)}\right)\mathrm{is}\mathrm{the}\mathrm{Jacobi}\mathrm{matrix}\mathrm{in}{z}_{\left(k-1\right)}$$

The algorithm based on the Gauss–Newton method works correctly if one important condition is met: the number of variables in the functions ${\phi }_i$ in the sum

$$S\left(z\right)={\displaystyle \sum }_{i=1}^m{\left[{\phi }_i\left(z\right)\right]}^2$$

must be lower than the number of these functions [42,43,44], i.e.,

$$p\le m.$$

(4)

In cases where this condition is violated, the matrix

$$J_{\phi }{\left(z_{\left(k-1\right)}\right)}^T{J}_{\phi }\left(z_{\left(k-1\right)}\right)$$

is irreversible and iterations (3) cannot return unambiguous answers.

With the considered architecture (Figure 4), we have

$$p=\left(r-1\right)q^2+\left(r+n+k\right)q+k$$

which means that the number $q$ of neurons in each layer is the root of the quadratic equation:

$$\left(r-1\right)q^2+\left(r+n+k\right)q+k-p=0.$$

(5)

The root of (5), which has a real physical meaning for $r,n,k>0,$ is

$$q=\frac{\sqrt{{\left(r+n+k\right)}^2+4\left(r-1\right)\left(p-k\right)}-\left(r+n+k\right)}{2\left(r-1\right)}.$$

(6)

Thus, from condition (4), for $m\ge p,$ it follows that

$$q\le \frac{\sqrt{{\left(r+n+k\right)}^2+4\left(r-1\right)\left(m-k\right)}-\left(r+n+k\right)}{2\left(r-1\right)}.$$

(7)

This proves the theorem. □

In many cases, approximating neural networks have a single hidden layer. An analogous formula for this particular case can be derived as a consequence of Theorem 1.

Corollary 1 (upper bound for the number of neurons in hidden layers for one hidden layer). 

For an approximating neural network with one hidden layer that is trained with an algorithm using the Jacobi matrix for the error function and that has the same number of neurons$q$in each layer, the highest approximation efficiency is achieved for some$q$:

$$q\le \frac{m-k}{n+k+1}.$$

where:

$m$is the number of training samples,

$n$—the number of input stimuli,

$k$—the number of output neurons ($m,n,k\in N)$.

Proof. 

Let us consider the function $F\left(r,n,m,k\right),$ formed by the expression on the right-hand side of (7):

$$F\left(r,n,m,k\right)=\frac{\sqrt{{\left(r+n+k\right)}^2+4\left(r-1\right)\left(m-k\right)}-\left(r+n+k\right)}{2\left(r-1\right)}.$$

(8)

Obviously, for a neural network with one layer, at $r=1$, the numerator and denominator in (8) are equal to 0. To find a formula for this case, we need to find the limit of $F\left(r,n,m,k\right)$ with $r$ tending to 1:

$$G\left(n,m,k\right)=\underset{r\to 1}{\lim }F\left(r,n,m,k\right)$$

A standard approach, with a numerator and denominator equal to 0, is to use l’Hôpital’s rule and find the limit of the quotient of the first derivatives of the numerator and denominator. The solution is a little easier to reach when we multiply the numerator and denominator by the multiplier $\sqrt{{\left(r+n+k\right)}^2+4\left(r-1\right)\left(m-k\right)}+\left(r+n+k\right)$. In this way, we get a variant for $F\left(r,n,m,k\right)$ in which there is no uncertainty at $r=1$:

$$F\left(r,n,m,k\right)=\frac{2\left(m-k\right)}{\sqrt{{\left(r+n+k\right)}^2+4\left(r-1\right)\left(m-k\right)}+\left(r+n+k\right)}$$

For single-layer neural networks, by replacing r with 1, we easily arrive at the final result for an optimal upper bound on the number of neurons in an ANN trained by methods using the Jacobi matrix and for which the number of training samples is $m$, the number of input stimuli is $n$, the hidden layer is 1, and the number of output neurons is $k$($m,n,k\in N)$, which is the function $G\left(n,m,k\right)=F\left(1,n,m,k\right)$:

$$G\left(n,m,k\right)=F\left(1,n,m,k\right)=\frac{m-k}{n+k+1}.$$

(9)

Ergo,

$$q\le \frac{m-k}{n+k+1}.$$

By this, Corollary 1 is proved. □

Corollary 2 (upper bound for the number of neurons in hidden layers, given one hidden layer and one output neuron). 

For an approximating neural network with one hidden layer that is trained with an algorithm using the Jacobi matrix for the error function and that has the same number of neurons$q$in each layer, the highest approximation efficiency is achieved for some$q$:

$$q\le \frac{m-1}{n+2}.$$

where:

$m$is the number of training samples,

$n$—the number of input stimuli ($m,n\in N)$.

The proof is trivial. The formula follows from Corollary 1, after replacing $k$ with 1.

5. Upper Bound on the Number of Layers for Neural Networks Containing an Equal Number of Neurons in Each Layer

The formulas for upper bounds on the number of neurons presented in the previous part can be a good guide in the search for neural networks. The question remains of finding a similar guideline for the number of layers of a single ANN. We will prove the following theorem:

Theorem 2 (upper bound for the number of hidden layers). 

For an approximating neural network that is trained with an algorithm using the Jacobi matrix for the error function and that has$r$hidden layers with the same number of neurons in each layer, the highest approximation efficiency is achieved for some$r$:

$$r\le \frac{m-2k-n+1}{2},$$

where:

$m$is the number of training samples,

$n$—the number of input stimuli,

$k$—the number of output neurons ($m,n,k\in N)$.

Proof. 

When solving a specific task, we can assume that $\left(n,m,k\right)=\overrightarrow{const},atn,m,k\in N$. Then we can consider F as a function of only the number of layers r, considering all other parameters as previously known constants: $\left(n,m,k\right)=\overrightarrow{const}$:

$$F\left(r\right)=\frac{\sqrt{{\left(r+n+k\right)}^2+4\left(r-1\right)\left(m-k\right)}-\left(r+n+k\right)}{2\left(r-1\right)}$$

(10)

The study of the partial derivative of $F\left(r,n,m,k\right)$:

$$\frac{\partial F}{\partial r}=\frac{\frac{4\left(m-k\right)+2\left(k+n+r\right)}{2\sqrt{{\left(k+n+r\right)}^2+4\left(m-k\right)\left(r-1\right)}}-1}{2\left(r-1\right)}-\frac{\sqrt{{\left(k+n+r\right)}^2+4\left(m-k\right)\left(r-1\right)}-\left(k+n+r\right)}{2{\left(r-1\right)}^2}$$

shows that $\frac{\partial F}{\partial r}<0,$ i.e., this derivative has a negative value for all $r,n,m,k\in N.$ At the same time, as can be seen from Figure 5a,

$$\underset{r\to \infty }{\lim }\frac{\partial F}{\partial r}=0.$$

In turn, this means that the function $F\left(r\right)$ is decreasing. Moreover, just like its derivative (Figure 5b), its limit at $r$ trending to $\infty$ is equal to 0:

$$\underset{r\to \infty }{\lim }F\left(r,n,m,k\right)=0.$$

As a result of the monotonicity of $F\left(r\right)$ and its bound tending to zero, with an unlimited increase in the number of layers, it becomes clear that to preserve the network optimality, an increase in the number of hidden layers must be accompanied by a decrease in the number of neurons in each of them. Thus, the expansion of the layered architecture of the network cannot grow infinitely because, after a certain value of $r$, the number of neurons will be $q<1$, which deprives the network of practical meaning. In practice, the maximum number of hidden layers $r_{max}$ satisfies the equation:

$$F\left(r_{max}\right)=1,\mathrm{i}.\mathrm{e}.,$$

$$\frac{\sqrt{{\left(r_{max}+n+k\right)}^2+4\left(r_{max}-1\right)\left(m-k\right)}-\left(r_{max}+n+k\right)}{2\left(r_{max}-1\right)}=1,r_{max},n,m,k\in N$$

(11)

Here again, $n$ is the number of inputs to the neural network, $m$ is the number of samples used to train it, and $k$ is the number of output neurons.

The definition area of the equation includes the requirements:

$$\begin{array}{c}{r}_{max}\in \left(-\infty ,\sqrt{k^2+\left(m-k\right)\left(m+n-1\right)}-\left(2m+n-k\right)\right]{{\displaystyle \cup }}^{}\\ \left[\sqrt{k^2+\left(m-k\right)\left(m+n-1\right)}-\left(2m+n-k\right),+\infty \right)\mathsf{и}{r}_{max}\ne 1\end{array}$$

(12)

One of the roots of (11) is $r_{max}=1$, which, however, does not belong to the definition area. The other root sets the solution of the given problem, i.e.,

$$r_{max}=\frac{m-2k-n+1}{2},$$

(13)

ergo,

$$r\le \frac{m-2k-n+1}{2}$$

with which the theorem is proved. □

6. Experimental Verification of the Formula for the Upper Limit of the Number of Layers

Let’s go back to the example we looked at—the function approximation of

$$y=2x_1+3x_2+1$$

with a multilayer $(r>1)$ neural network, for which we have 100 input–output samples for training $\left(n=2,m=100,\mathrm{and}k=1\right)$. In this particular task, Equation (13) gives us the maximum optimal number of required layers $r_{max}=48$, which confirms the results shown in Figure 2.

For the one-dimensional functions presented in Figure 1, we have $n=1,m=200,\mathrm{and}k=1$. According to these values, it follows from (13) that we should expect an optimal network with a number of layers not greater than $r_{max}=99.$ The results shown in Figure 1 confirm this value, and for all the optimal neural networks presented in the figure, the number of layers is less than the specified number.

Similar experiments were conducted with multivariate mixed functions. We used 200 training samples using the Levenberg–Marquardt algorithm. For example, for the three-dimensional mixed function

$$y=x_1^2+e^{x_2}+tang\left(x_3\right)$$

we have $n=3,m=200\mathrm{and}k=1,$ which means an optimal number of layers not greater than $r_{max}=98$. The variation in the efficiency of the approximating network, with different numbers of layers, is shown in Figure 6a. The peak in efficiency is reached with an artificial neural network with 59 hidden layers. When experimenting with the randomly composed five-dimensional mixed function

$$y=x_1+x_2+\cos \left(x_3\right)+x_4{x}_5$$

the task comes down to the situation $n=5,m=200,\mathrm{and}k=1$. In this case, according to (13), we expect the optimal number of layers to be no greater than $r_{max}=97$.

After the experiment, it was found that the optimal neural network approximating the described five-dimensional function has nine hidden layers (Figure 6b).

Another part of the conducted experiments is related to checking the validity of Formula (13) when reducing the number of training examples, which obviously has a strong impact on the maximum number of layers. In general, this sample size must be large enough to provide training, validation, and testing of the out-of-the-box network. When we reduce the training examples, we should also expect a decrease in the value of $r_{max}$.

In these experiments (Figure 7), we used the selected features $y=2x_1+3x_2+1$, $y=x_1^2+e^{x_2}+tang\left(x_3\right)$, $y=x_1+x_2+\cos \left(x_3\right)+x_4{x}_5$. Using only 50 training samples, for example, for the two-dimensional linear function (Figure 7a), the maximum upper threshold of the number of layers is $r_{max}=24$, and the experimentally determined most efficient number of layers is $r_{optimum}=2$. Similarly, when examining the same function and 10 training samples available, it is once again observed that the established effective number of layers, $r_{optimum}=1$, is below the calculated maximum number $,r_{max}=4$ (Figure 7b).

Experiments with the other considered functions also confirm the expectations laid down in Formula (13) (Figure 7c–f). Although there are efficient neural networks with a number of layers greater than the calculated maximum number, it is clear from the presented graphical results (e.g., Figure 7c,d) that the search for neural networks with a number of layers greater than $r_{max}$ is unnecessary.

Similar experiments conducted with other randomly composed multidimensional functions also confirm the validity of Formula (13).

7. Conclusions

One of the important questions in the theory of artificial neural networks is that of determining the topology of the network. There are many different types of neural networks for different purposes, and therefore no universal answer can be provided to this question. In some cases, a suitable neural network can be searched for automatically by iteratively changing certain network parameters. In such cases, it is useful to determine in advance the upper limit of the number of layers and the number of hidden neurons of the neural network. This allows for the minimization of computations and facilitates the finding of a suitable neural network.

This paper derives formulas for the maximum number of layers and neurons in neural networks trained with algorithms that use the Jacobi matrix for error estimates. The subject of research is neural networks that have the same number of neurons in the hidden layers. The proposed formulas are verified by experimental verification with multilayer artificial neural networks approximating both one-dimensional and multidimensional objective functions of varied uniform nature and those of mixed type. They can be used as a criterion when searching for the optimal topology of approximating neural networks.

Finding stricter rules for determining the number of layers and neurons of an ANN can significantly optimize the process of automated ANN construction. Our future work is aimed at looking for additional factors affecting the number of layers and neurons in an artificial neural network, including implementing a system for preliminary analysis of the training samples to determine the degree of non-linearity of the problem, determining the influence of the type of transfer function of neurons on the required number of neurons, etc.