On the Computability of Primitive Recursive Functions by Feedforward Artificial Neural Networks

Abstract

We show that, for a primitive recursive function $h(\mathbf{x},t)$, where $\mathbf{x}$ is a n-tuple of natural numbers and t is a natural number, there exists a feedforward artificial neural network $\mathfrak{N}(\mathbf{x},t)$, such that for any n-tuple of natural numbers $\mathbf{z}$ and a positive natural number m, the first $m+1$ terms of the sequence $\left\{h\right(\mathbf{z},t\left)\right\}$ are the same as the terms of the tuple $\left(\mathfrak{N}\right(\mathbf{z},0),\dots ,$$\mathfrak{N}(\mathbf{z},m))$.

1. Introduction

Primitive recursive functions describe, albeit incompletely, the intuitive notion of a number-theoretic algorithm, a deterministic procedure to transform numerical inputs to numerical outputs in finitely many steps. This perception of primitive recursive functions as intuitive counterparts of number-theoretic algorithms may be rooted in the fact that any primitive recursive function can be mechanically constructed from a set of initial functions with finitely many applications of simple, well-defined operations of composition and primitive recursion. These functions and some of their properties have been investigated by Gödel [1], Péter [2,3], Kleene [4], Davis [5], and Rogers [6] in their studies of formal systems, foundations of mathematics, and computability theory. Although the confinement of the construction procedure to two operations may at first seem restrictive, many functions on natural numbers ordinarily encountered in mathematics and computer science are, in fact, primitive recursive (cf., e.g., Ch. 3 in [7]). Primitive recursive functions have been used to investigate the foundations of functional programming. Colson [8] presents a computational model in which a primitive recursive function is viewed as a rewriting system and gives a non-trivial necessary condition for an algorithm to be representable in the system. Paolini et al. [9] define a class of recursive permutations, which they call Reversible Primitive Permutations (RPP), and formalize it as a language that is sufficiently expressive to represent all primitive recursive functions. Petersen [10] uses induction and primitive recursion to develop resource conscious logics where the repeated recycling of assumptions, e.g., repeated applications of the successor function $f\left(n\right)=n+1$ to enumerate natural numbers, has costs.

Feedforward artificial neural networks have their origin in the research by McCulloch and Pitts [11], which describes neural events with propositional logic. McCulloch and Pitts assume that the human nervous system is a finite set of neurons, each of which has an excitation threshold. When a neuron’s threshold is exceeded, the neuron generates an impulse that propagates to other neurons across synapses connecting them to the origin of the impulse. A fundamental insight by McCulloch and Pitts is that if the response of a neuron can be formalized as a logical proposition specifying its stimulus, then behaviors of complex networks of neurons can, in principle, be described with symbolic logic. Artificial neural networks entered mainstream computer science almost half a century after the research by McCulloch and Pitts when Rumelhart, Hinton, and Williams [12] discovered backpropagation, a method for training networks to modify synapse weights by minimizing error between the output and the ground truth. Different types of such networks have been shown to be universal approximators of some classes of functions (e.g., [13,14,15]). Artificial neural networks are increasingly used in embedded artificial intelligence (AI) systems, i.e., systems that run on computational devices with finite amounts of computer memory (e.g., [16]). We will refer to embedded AI as finite AI to emphasize the fact that finite AI systems are realized on computational devices with finite amounts of computational memory.

In this investigation, we seek to relate, in a formal way, primitive recursive functions and feedforward artificial neural networks by investigating whether it is possible, for a given primitive recursive function, to construct a feedforward artificial neural network that arbitrarily computes many values of the function’s co-domain from the corresponding values of the function’s domain. We hope that our investigation contributes to the knowledge of the classes of functions that can be not only approximated, but provably computed by feedforward artificial neural networks. In particular, we formalize feedforward artificial neural networks with recurrence equations, propose a formal definition of the concept of $\mathfrak{N}$-computability, i.e., the property of a function to be computed by a feedforward artificial neural network $\mathfrak{N}$, and prove several lemmas and theorems to show how feedforward artificial neural networks can be constructed to arbitrarily compute many consecutive values of any primitive recursive function. Since these networks consist of finite sets of neurons and are used in some finite AI systems [17,18], our investigation will be of interest to mathematicians and computer scientists interested in the computability theory of finite AI.

The remainder of our article is organized as follows. In Section 2, we review several definitions of primitive recursive functions starting with the original definition by Gödel [1] and proceeding to the later definitions by Kleene [4], Davis [5], Rogers [6], and Davis et al. [7], and Meyer and Ritchie [19]. This section gives the reader a historical bird’s eye view of how the concept of primitive recursive function and its formalization have co-evolved in time. In Section 3, we state the notational conventions and give the definition of a primitive recursive function we use in this article. This section is intended for reference. In Section 4, we offer a formalization of feedforward artificial neural networks in terms of recursive equations. In Section 5, we prove several lemmas and theorems that form the bulk of our theoretical investigation. In Section 6, we present some perspectives on the obtained results and summarize our conclusions in Section 7.

2. Recursive Functions

Gödel [1] (Sec. 2, p. 157) describes the class of number-theoretic functions as the class of functions whose domains are non-negative integers or n-tuples thereof and whose values are non-negative integers. Gödel [1] (Sec. 2, pp. 157–159) states that a number-theoretic function $\varphi (x_1,x_2,\dots ,x_n)$ is recursively defined in terms of the number-theoretic functions $\psi (x_1,x_2,\dots ,x_{n-1})$ and $\mu (x_1,x_2,\dots x_{n+1})$ if $\varphi$ is obtained from $\psi$ and $\mu$ by the following schema:

$$\begin{array}{ccc}\varphi (0,x_2,\dots ,x_n)\hfill & =\hfill & \psi (x_2,\dots ,x_n),\hfill \\ \varphi (k+1,x_2,\dots ,x_n)\hfill & =\hfill & \mu (k,\varphi (k,x_2,\dots ,x_n),x_2,\dots ,x_n),\hfill \end{array}$$

(1)

where the equalities hold for all $k,x_2,\dots ,x_n$. Gödel [1] (Sec. 2, p. 159) defines a number-theoretic function $\varphi$ to be recursive if there exists a finite sequence of number-theoretic functions ${\varphi }_1$, ${\varphi }_2$, …, ${\varphi }_n=\varphi$, where each function ${\varphi }_i$, $1\le i\le n$, is a natural number constant, the successor function $x+1$, or is defined from two preceding functions with Schema (1) or from one preceding function by substitution, i.e., the replacement of the arguments of a preceding function with some other preceding functions.

Kleene [4] (Chap. IX, § 43, p. 219) defines a number-theoretic function to be primitive recursive if it is definable by a finite number of applications of the six schemata in (2), where m and n are positive integers, i is an integer such that $1\le i\le n$, q is a natural number, and $\psi ,{\chi }_1,\dots ,{\chi }_m,$ and $\chi$ are number-theoretic functions with the indicated numbers of arguments.

$$\begin{array}{cccc}\left(I\right)\hfill & \varphi \left(x\right)=x+1;\hfill & \hfill & \\ \left(II\right)\hfill & \varphi (x_1,\dots ,x_n)=q;\hfill & \hfill & \\ \left(III\right)\hfill & \varphi (x_1,\dots ,x_n)=x_i;\hfill & \hfill & \\ \left(IV\right)\hfill & \varphi (x_1,\dots ,x_n)=\psi ({\chi }_1(x_1,\dots ,x_n),\dots ,{\chi }_m(x_1,\dots ,x_n));\hfill & \hfill & \\ \left(Va\right)\hfill & \left\{\begin{array}{cc}\varphi \left(0\right)\hfill & =q,\hfill \\ \varphi (y+1)\hfill & =\chi (y,\varphi (y\left)\right);\hfill \end{array}\right.\hfill & \hfill & \\ \left(Vb\right)\hfill & \left\{\begin{array}{cc}\varphi (0,x_2,\dots ,x_n)\hfill & =\psi (x_2,\dots ,x_n),\hfill \\ \varphi (y+1,x_2,\dots ,x_n)\hfill & =\chi (y,\varphi (y,x_2,\dots ,x_n),x_2,\dots ,x_n).\hfill \end{array}\right.\hfill & \hfill & \end{array}$$

(2)

Schema (I) defines the successor function, Schema (II) defines constant functions, and Schema (III) defines identity functions, which Kleene denotes with the symbol $U_i^n$. Kleene defines the functions satisfying Schemata (I), (II), and (III) in (2) as initial functions. Schema (IV) in (2) obtains $\varphi$ from $\psi$, ${\chi }_1,\dots ,{\chi }_m$ by substitution. Schemata (Va) and (Vb) obtain $\varphi$ from $\chi$ or from $\chi$ and $\psi$, respectively, by primitive recursion. Kleene [4] (Chap. XI, § 55, p. 275) defines a function to be general recursive in functions ${\psi }_1,\dots ,{\psi }_l$ if there is a system E of equations which defines $\varphi$ recursively from ${\psi }_1,\dots ,{\psi }_l$.

Davis [5] (Chap. 2, Sec. 2, p. 36) defines the operation of composition as the operation to obtain the function $h\left({\mathfrak{x}}^{\left(n\right)}\right)$ from the functions $f\left({\mathfrak{y}}^{\left(m\right)}\right)$, $g_1\left({\mathfrak{x}}^{\left(n\right)}\right),\dots ,g_m\left({\mathfrak{x}}^{\left(n\right)}\right)$ with Schema (3).

$$\begin{array}{ccc}h\left({\mathfrak{x}}^{\left(n\right)}\right)\hfill & =\hfill & f(g_1\left({\mathfrak{x}}^{\left(n\right)}\right),\dots ,g_m\left({\mathfrak{x}}^{\left(n\right)}\right)),\hfill \end{array}$$

(3)

where ${\mathfrak{y}}^{\left(m\right)}$ and ${\mathfrak{x}}^{\left(n\right)}$ are tuples of natural numbers with m and n elements, respectively. Davis [5] (Chap. 3, Sec. 4, p. 48) defines the operation of primitive recursion as the operation that uses Schema (4) to construct the function $h\left({\mathfrak{x}}^{(n+1)}\right)$ from the total functions $f\left({\mathfrak{x}}^{\left(n\right)}\right)$ and $g\left({\mathfrak{x}}^{(n+2)}\right)$, where ${\mathfrak{x}}^{\left(n\right)}$, ${\mathfrak{x}}^{(n+1)}$, and ${\mathfrak{x}}^{(n+2)}$ are tuples of natural numbers with n, $n+1$, and $n+2$ elements, respectively.

$$\begin{array}{ccc}h(0,{\mathfrak{x}}^{\left(n\right)})\hfill & =\hfill & f\left({\mathfrak{x}}^{\left(n\right)}\right)\hfill \\ h(z+1,{\mathfrak{x}}^{\left(n\right)})\hfill & =\hfill & g(z,h(z,{\mathfrak{x}}^{\left(n\right)}),{\mathfrak{x}}^{\left(n\right)}).\hfill \end{array}$$

(4)

For a set of natural numbers A, Davis [5] (Chap. 3, Sec. 4, p. 49) defines an A-primitive recursive function or a function primitive recursive in A as a function that can be obtained by a finite number of applications of composition (cf. Schema (3)) and primitive recursion (cf. Schema (4)) from the following j functions:

$$\begin{array}{cccc}\left(1\right)\hfill & C_A\left(x\right);\hfill & \hfill & \\ \left(2\right)\hfill & S\left(x\right)\hfill & =\hfill & x+1;\hfill \\ \left(3\right)\hfill & N\left(x\right)\hfill & =\hfill & 0;\hfill \\ \left(4\right)\hfill & U_i^n(x_1,\dots ,x_n)\hfill & =\hfill & x_i,1\le i\le n,\hfill \end{array}$$

(5)

where $C_A\left(x\right)$ is the characteristic function of A (i.e., $C_A\left(x\right)$ is a total function such that $C_A\left(x\right)=1$ if $x\in A$ and $C_A\left(x\right)=0$ if $x\notin A$), and $S\left(x\right)$ and $U_i^n$ are identical to Kleene’s Schemata (I) and (III) in (2). Davis [5] (Chap. 3, Sec. 4, p. 49) defines a function f to be primitive recursive if it is ∅-primitive recursive, where ∅ denotes the empty set.

Rogers [6] (Chap. 1, § 1.2, p. 6) defines the class C of primitive recursive functions as the smallest class of functions such that

(1)

All constant functions $\lambda x_1{x}_2\cdots x_k\left[m\right]$, are in C, $1\le k$, $0\le m$;

(2)

The successor function $\lambda x[x+1]$ is in C;

(3)

All identity functions $\lambda x_1\cdots x_k\left[x_i\right]$ are in C, $1\le i\le k$;

(4)

If f is a function of k variables in C and $g_1,\dots ,g_k$ are functions in C of m variables each, then the function $\lambda x_1\cdots x_m\left[f(g_1(x_1,\dots ,x_m),\dots ,g_k(x_1,\dots ,x_m))\right]$ is in C, $1\le k,m$;

(5)

If h is a function of $k+1$ variables in C, and g is a function of $k-1$ variables in C, then the unique function f of k variables satisfying Schema (6) is also in C, $1\le k$.

$$\begin{array}{ccc}f(0,x_2,\dots ,x_k)\hfill & =\hfill & g(x_2,\dots ,x_k),\hfill \\ f(y+1,x_2,\dots ,x_k)\hfill & =\hfill & h(y,f(y,x_2,\dots ,x_k),x_2,\dots ,x_k).\hfill \end{array}$$

(6)

Davis et al. [7] (Chap. 3, Sec. 3, p. 42) defines as initial the functions $s\left(x\right)=x+1$, $n\left(x\right)=0$, and $u_i^n(x_1,\dots ,x_n)=x_i$, $1\le i\le n$, and defines a function to be primitive recursive if it can be obtained from the initial functions by a finite number of applications of composition or primitive recursion where primitive recursion is defined by Schema (7) (Chap. 3, Sec. 2, p. 40 in [7]) and Schema (8) (Chap. 3, Sec. 2, p. 41 in [7]). In Schema (7), k is a natural number and g is a total function of two variables. In Schema (8), f and g are total functions of n and $n+2$ variables, respectively.

$$\begin{array}{ccc}h\left(0\right)\hfill & =\hfill & k,\hfill \\ h(t+1)\hfill & =\hfill & g(t,h(t\left)\right)\hfill \end{array}$$

(7)

$$\begin{array}{ccc}h(x_1,\dots ,x_n,0)\hfill & =\hfill & f(x_1,\dots ,x_n),\hfill \\ h(x_1,\dots ,x_n,t+1)\hfill & =\hfill & g(t,h(x_1,\dots ,x_n,t),x_1,\dots ,x_n).\hfill \end{array}$$

(8)

2.1. Computability and Turing Machines

Davis [5] (Chap. 1, Sec. 2, p. 10) gives the following definition of partially computable and computable functions.

Definition 1.

An n-ary function $f(x_1,\dots ,x_n)$ is partially computable if there exists a Turing machine Z such that

$$f(x_1,\dots ,x_n)={\mathrm{\Psi }}_Z^{\left(n\right)}(x_1,\dots ,x_n).$$

In this case, we say that Z computes f. If, in addition, $f(x_1,\dots ,x_n)$ is a total function, then it is called computable.

In subsequent chapters of his monograph (cf. Chap. 2 and Chap. 3 in [5]), Davis separates the notion of computability from Turing machines to make it possible “to demonstrate the computability of quite complex functions without referring back to the original definition of computability in terms of Turing machines” (cf. Ch. 3, Sec. 1, p. 41 in [5]).

Davis et al. [7] (Chap. 2) continue this treatment of computability by designing the programming language $\mathcal{L}$ and then defining partially computable and computable functions in terms of $\mathcal{L}$ programs, viz., finite sequences of $\mathcal{L}$ instructions. In $\mathcal{L}$, the unique variable Y is designated as the output variable to store the output of an $\mathcal{L}$ program $\mathcal{P}$ on a given input. $X_1,X_2,...$ are input variables and $Z_1,Z_2,...$ are internal variables. All variables refer to natural numbers. $\mathcal{L}$ has conditional dispatch instructions, line labels, elementary arithmetic operations, comparisons of natural numbers, and macros.

Davis et al. [7] (Chap. 2, Sec. 3, p. 27) define a computation of an $\mathcal{L}$ program $\mathcal{P}$ on some inputs $x_1,\dots ,x_m$, and $m>0$, as a finite sequence of snapshots $(s_1,\dots ,s_k)$, where each snapshot $s_i$, $1\le i\le k$, $k>0$ specifies the number of the instruction in $\mathcal{P}$ to be executed and the value of each variable in $\mathcal{P}$, and where each subsequent snapshot is uniquely determined by the previous snapshot (Theorem 3.2, Chap. 4, Sec. 3, pp. 74–75 in [7]). The snapshot $s_1$ is the initial snapshot, where the values of all input variables are set to their initial values, the program instruction counter is set to 1, i.e., the number of the first instruction in $\mathcal{P}$, and the values of all the other variables in $\mathcal{P}$ are set to 0. The snapshot $s_k$ in $(s_1,\dots ,s_k)$ is a terminal snapshot, where the instruction counter is set to the number of the instructions in $\mathcal{P}$ plus 1. If some program $\mathcal{P}$ in $\mathcal{L}$ takes m inputs $X_1=x_1$, $X_2=x_2$, …, $X_m=x_m$, then

$${\mathrm{\Psi }}_{\mathcal{P}}^{\left(m\right)}(x_1,x_2,\dots ,x_m)=\left\{\begin{array}{cc}Y\mathrm{in}{s}_k\hfill & \mathrm{if}(s_1,\dots ,s_k)\mathrm{is}\mathrm{a}\mathrm{computation},k\ge 1,\hfill \\ \uparrow \hfill & otherwise.\hfill \end{array}\right.$$

(9)

The definitions of partially computable and computable functions are made by Davis et al. [7] (Chap. 2, Sec. 4, p. 30) in terms of $\mathcal{L}$ programs as follows.

Definition 2.

An n-ary function f is partially computable if f is partial and there is a $\mathcal{L}$ program $\mathcal{P}$ such that Equation (10) holds for all $x_1,\dots ,x_n$.

$$f(x_1,\dots ,x_n)={\mathrm{\Psi }}_{\mathcal{P}}^{\left(n\right)}(x_1,\dots ,x_n).$$

(10)

Definition 3.

A n-ary function f is computable if it is total and partially computable.

Equation (10) in Definition 2 is interpreted so that $f(x_1,\dots ,x_n)$ is defined if and only if ${\mathrm{\Psi }}_{\mathcal{P}}^{\left(n\right)}(x_1,\dots ,x_n)$ is defined. This treatment of computable functions in terms of programs in a formal language is by no means the only one in the literature. For example, as early as 1967, Meyer and Ritchie [19] formalize primitive recursive functions as loop programs consisting of assignment and iteration statements similar to DO statements of the programming language FORTRAN.

2.2. Computability of Primitive Recursive Functions

Davis et al. [7] (Chap. 3, Sec. 3, p. 42) introduce the concept of a primitive recursively closed (PRC) class of functions, which is a class of total functions that contains the initial functions and any functions obtained from the initial functions by a finite number of applications of composition or primitive recursion. Davis et al. [7] (Chap. 3, Sec. 3, pp. 42–43) show that (1) the class of computable functions is PRC; (2) the class of primitive recursive functions is PRC; and (3) a function is primitive recursive if and only if it belongs to every PRC class. A corollary of (3) is that every primitive recursive function is computable.

Péter [2,3] shows it is possible to define functions in terms of recursive equations that are not primitive recursive. In particular, Péter demonstrates that all unary primitive recursive functions are enumerable, i.e., ${\varphi }_0\left(x\right),{\varphi }_1\left(x\right),{\varphi }_2\left(x\right),\dots$ is an enumeration, with repetitions, of all unary primitive recursive functions. By Cantor’s diagonalization (cf., e.g., pp. 6–8 in [4]), the unary function $f\left(x\right)={\varphi }_x\left(x\right)+1$ is not in the enumeration and, hence, not primitive recursive. While f is not primitive recursive, it is computable (cf. Definition 3). Thus, the class of primitive recursive functions is a proper subset of computable functions and, in and of itself, cannot completely capture the intuitive notion of a number-theoretic algorithm. Péter’s argument suffers no loss of generality, insomuch as any n-ary primitive recursive function, $n>1$, can be reduced to an equivalent unary primitive recursive function (cf., Theorems 9.1 and 9.2, Chap. 4, Sec. 9, p. 108 in [7]). Kleene’s separation of recursive functions into general recursive and primitive recursive may have been influenced by Péter’s discovery (cited by Kleene [4] in Chap. XI, § 55, p. 272).

Rogers [6] (Chap. 1, § 1.2, p. 8) defines the Ackermann generalized exponential, a function for which there is no primitive recursive derivation, and formalizes it with the following recursive equations:

$$\begin{array}{ccc}f(0,0,y)\hfill & =\hfill & y,\hfill \\ f(0,x+1,y)\hfill & =\hfill & f(0,x,y)+1,\hfill \\ f(1,0,y)\hfill & =\hfill & 0,\hfill \\ f(z+2,0,y)\hfill & =\hfill & 1,\hfill \\ f(z+1,x+1,y)\hfill & =\hfill & f(z,f(z+1,x,y),y).\hfill \end{array}$$

3. Notational Conventions and Definitions

If f is a function, $dom\left(f\right)$ and $codom\left(f\right)$ are the domain and the co-domain of f. The expression $f:A\mapsto B$ abbreviates the logical conjunction $dom\left(f\right)=A\wedge codom\left(f\right)=B$, for some sets A and B. A function f is partial on A if $dom\left(f\right)$ is a proper subset of A, i.e., $dom\left(f\right)\subset A$. If f is partial on A and $a\in A$, the following statements are equivalent: (1) $a\in dom\left(f\right)$; (2) f is defined on a; (3) $f\left(a\right)$ is defined; (4) $f\left(a\right)\downarrow$. The following statements are also equivalent: (1) $a\notin dom\left(f\right)$; (2) f is undefined on a; (3) $f\left(a\right)$ is undefined; (4) $f\left(a\right)\uparrow$. If $dom\left(f\right)=A$, then f is total on A.

The notation $(a_1,\dots ,a_n)$ is used to denote ordered n-tuples or, simply, n-tuples over some set of numbers A. We will use bold lower-case variables, e.g., $\mathbf{a},\mathbf{x},\mathbf{y}$, to refer to n-tuples. Thus, $\mathbf{a}=(13,17,19)$ is a 3-tuple over the set of natural numbers $\mathbb{N}=\{0,1,2\dots \}$. We will use the symbol ${\mathbb{N}}^{+}$ to denote the set of positive natural numbers. If $\mathbf{x}=(x_1,\dots ,x_n)$ is an n-tuple over A, then $\mathbf{x}\left[j\right]$, $1\le j\le n$, refers to individual elements of $\mathbf{x}$. Thus, if $\mathbf{x}=(2,3,5,7,11)$, then $\mathbf{x}\left[1\right]=2$, $\mathbf{x}\left[2\right]=3$, $\mathbf{x}\left[3\right]=5$, $\mathbf{x}\left[4\right]=7$, $\mathbf{x}\left[5\right]=11$. The individual elements of an n-tuple are not required to be distinct. If $\mathbf{x}$ is an n-tuple, then $dim\left(\mathbf{x}\right)=n$, i.e., the number of elements in $\mathbf{x}$. The 0-tuple is denoted as $\left(\right)$. In calculus, a sequence is an ordered set of numbers in a one-to-one correspondence with $\mathbb{N}$ or ${\mathbb{N}}^{+}$ (cf., e.g., Taylor [20], § 1.62, p. 67). Thus, if $f:\mathbb{N}\to \mathbb{N}$, then $\left\{f\right(n\left)\right\}$ denotes the sequence $f\left(0\right),f\left(1\right),\dots ,f\left(m\right),\dots$ with countably many elements or terms. In computability theory, the term sequence sometimes refers to an n-tuple (cf., e.g., Ch. 3, p. 60 in [7]). Thus, in order to avoid confusion, when we want to emphasize the fact that we are dealing with a finite number of ordered elements, we refer to the collection of these elements as a finite sequence, a tuple, or an m-tuple, where m is the number of the elements.

For $n>0$, $A^n$ is the n-th Cartesian power of A, i.e., $A^n=\left\{(a_1,\dots ,a_n)\right|a_i\in A,1\le i\le n\}$. Thus, if $f:{\mathbb{R}}^2\mapsto \mathbb{N}$, $dom\left(f\right)=\left\{(x_1,x_2)\right|x_1,x_2\in \mathbb{R}\}$, where $\mathbb{R}$ is the set of real numbers. We use statements like $\mathbf{a}\in A^n$ to mean that $\mathbf{a}$ is an n-tuple over A. We do not distinguish between 1-tuples and individual elements, e.g., $a=\left(a\right)$, $a\in A$, and $h\left(a\right)=h\left(\right(a\left)\right)$ for some function h.

In formalizing feedforward artificial neural networks, it is sometimes convenient to treat n-tuples as vectors. Therefore, we occasionally use symbols like $\overrightarrow{x}$, $\overrightarrow{y}$, $\overrightarrow{z}$ to denote n-tuples. If $\overrightarrow{x}\in A^n$, then $dim\left(\overrightarrow{x}\right)=n$ and $\overrightarrow{x}\left[j\right]$, $1\le j\le n$, is the j-th element of $\overrightarrow{x}$. E.g., if $\overrightarrow{x}=(1,1,11)\in {\mathbb{N}}^3$, then $\overrightarrow{x}\left[1\right]=\overrightarrow{x}\left[2\right]=1$ and $\overrightarrow{x}\left[3\right]=11$. If $\mathbf{a}\in A^n$ and $\overrightarrow{a}\in A^n$, and $\mathbf{a}\left[j\right]=\overrightarrow{a}\left[j\right]$, $1\le j\le n$, then $\mathbf{a}=\overrightarrow{a}$. If $f:A^n\mapsto B^m$, $0<n,m$, then $f(x_1,\dots ,x_n)$ = $f\left(\overrightarrow{x}\right)=f\left(\mathbf{x}\right)$ = $f(\overrightarrow{x}\left[1\right],\dots ,\overrightarrow{x}\left[n\right])$ = $f\left(\mathbf{x}\right[1],\dots ,\mathbf{x}[n\left]\right)$ = $\overrightarrow{y}$ = $\mathbf{y}$ = ($\overrightarrow{y}\left[1\right],\dots ,\overrightarrow{y}\left[m\right])$ = $\left(\mathbf{y}\right[1],\dots ,\mathbf{y}[m\left]\right)$. The empty tuple is discarded in function arguments. E.g., if $h:\mathbb{N}\mapsto \mathbb{N}$, then $h\left(\right(),t)=h(t,(\left)\right)=h\left(t\right)$, $t\in \mathbb{N}$. We occasionally separate individual arguments of functions from the remaining arguments combined into tuples. E.g., if $f:{\mathbb{N}}^{n+2}\mapsto \mathbb{N}$, $0<n$, then, for $\mathbf{z}\in {\mathbb{N}}^n$, $x\in \mathbb{N}$, $y\in \mathbb{N}$, $f(\mathbf{z},x,y)=f\left(\mathbf{z}\right[1],\dots ,\mathbf{z}[n],x,y)=f\left(\mathbf{w}\right)$, where $\mathbf{z}\left[i\right]=\mathbf{w}\left[i\right]$, $1\le i\le n$, and $\mathbf{w}[n+1]=x$, $\mathbf{w}[n+2]=y$. If f is a function that maps ${\mathbf{a}}_1\in A_1^{n_1}$, …, ${\mathbf{a}}_k\in A_k^{n_k}$ to $\mathbf{c}\in C^m$, for some sets $A_1,\dots ,A_k$, $0<n_j,m$, $1\le j\le k$, then $f:A_1^{n_1},\dots ,A_k^{n_k}\mapsto C^m$.

A total function $P:A^n\mapsto \{0,1\}$ is a predicate, where 1 arbitrarily designates logical truth and 0 logical falsehood. The symbols ¬, ∧, ∨, → respectively refer to logical not, logical and, logical or, and logical implication. $P\left(\mathbf{x}\right)$ is a shorthand for $P\left(\mathbf{x}\right)=1$, and $\neg P\left(\mathbf{x}\right)$ is a shorthand for $P\left(\mathbf{x}\right)=0$. If P and Q are predicates, then $\neg P\vee Q$ is logically equivalent to $P\to Q$, i.e., $\neg P\vee Q\equiv P\to Q$. The symbols ∃ and ∀ refer to the logical existential (there exists) and universal (for all) quantifiers, respectively. Thus, the statement $(\exists \mathbf{x})P\left(\mathbf{x}\right)$ is logically equivalent to the statement that $P\left(\mathbf{x}\right)$ holds for at least one $\mathbf{x}$ in $dom\left(P\right)$, while the statement $(\forall \mathbf{x})P\left(\mathbf{x}\right)$ is logically equivalent to the statement that $P\left(\mathbf{x}\right)$ holds for every $\mathbf{x}$ in $dom\left(P\right)$.

Let, for $0<k,n$, $f:{\mathbb{N}}^k\mapsto \mathbb{N}$, $g_j:{\mathbb{N}}^n\mapsto \mathbb{N}$, $1\le j\le k$, and $\mathbf{x}\in {\mathbb{N}}^n$. We use the following definitions of composition and primitive recursion in our article. A function of $h:{\mathbb{N}}^n\mapsto \mathbb{N}$ is obtained from $f,g_j$ by composition if h is obtained from $f,g_j$ by Schema (11).

$$h\left(\mathbf{x}\right)=f(g_1\left(\mathbf{x}\right),...,g_k\left(\mathbf{x}\right)).$$

(11)

Let $k\in \mathbb{N}$ and $\varphi :{\mathbb{N}}^2\mapsto \mathbb{N}$ be total. A function $h:\mathbb{N}\mapsto \mathbb{N}$ is obtained from $\varphi$ by primitive recursion if it is obtained from $\varphi$ by Schema (12).

$$\begin{array}{ccc}h\left(0\right)\hfill & =\hfill & k,\hfill \\ h(t+1)\hfill & =\hfill & \varphi (t,h(t\left)\right).\hfill \end{array}$$

(12)

Let $f:{\mathbb{N}}^n\mapsto \mathbb{N}$ and $g:{\mathbb{N}}^{n+2}\mapsto \mathbb{N}$ be total, then $h:{\mathbb{N}}^{n+1}\mapsto \mathbb{N}$ is obtained from f and g by primitive recursion if h is obtained from f and g by Schema (13), where $\mathbf{x}\in {\mathbb{N}}^n$.

$$\begin{array}{ccc}h(\mathbf{x},0)\hfill & =\hfill & f\left(\mathbf{x}\right),\hfill \\ h(\mathbf{x},t+1)\hfill & =\hfill & g(t,h(\mathbf{x},t),\mathbf{x}).\hfill \end{array}$$

(13)

If $\overrightarrow{x}\in {\mathbb{N}}^n$, Schema (13) can be expressed with the vector notation as

$$\begin{array}{ccc}h(\overrightarrow{x},0)\hfill & =\hfill & f\left(\overrightarrow{x}\right),\hfill \\ h(\overrightarrow{x},t+1)\hfill & =\hfill & g(t,h(\overrightarrow{x},t),\overrightarrow{x}).\hfill \end{array}$$

(14)

Let the set of initial functions consist of

$$\begin{array}{ccc}\hfill & s\left(x\right)=x+1,x\in \mathbb{N};\hfill & \\ \hfill & n\left(x\right)=0,x\in \mathbb{N};\hfill & \\ \hfill & u_i^n(x_1,\dots ,x_n)=u_i^n\left(\overrightarrow{x}\right)=\overrightarrow{x}\left[i\right]=u_i^n\left(\mathbf{x}\right)=\mathbf{x}\left[i\right]=x_i,\hfill & 1\le i\le n,\overrightarrow{x}=\mathbf{x}\in \mathbb{N}.\hfill \end{array}$$

(15)

Definition 4.

A function is primitive recursive if it can be obtained from the initial functions by a finite number of applications of Schemata (11)–(13).

A corollary of Definition 4 is that if f is primitive recursive, then there is a sequence of functions ${\varphi }_1,\dots ,{\varphi }_n=f$ such that ${\varphi }_i$, $1\le i\le n$, is either an initial function or is obtained from the previous functions in the sequence by composition or primitive recursion.

4. Feedforward Artificial Neural Networks

A feedforward artificial neural network ${\mathfrak{N}}_z$ is a finite set of neurons, each of which is connected to a finite number of the neurons in the same set through the synapses, i.e., directed weighted edges (cf. Figure 1). The neurons are organized into l layers ${\mathfrak{E}}_1,\dots ,{\mathfrak{E}}_l$, where ${\mathfrak{E}}_1$ is the input layer, ${\mathfrak{E}}_l$ is the output layer, and ${\mathfrak{E}}_e$, $1<e<l$ are the hidden layers. We use the term network synonymously with the term feedforward artificial neural network.

Let ${\mathfrak{z}}_z$ denote the number of layers in ${\mathfrak{N}}_z$ and $n_i^e$ refer to the i-th neuron in layer ${\mathfrak{E}}_e$, $1\le e\le {\mathfrak{z}}_z$. The function $n{n}_z\left(e\right):{\mathbb{N}}^{+}\mapsto {\mathbb{N}}^{+}$ specifies the number of neurons in layer ${\mathfrak{E}}_e$ of ${\mathfrak{N}}_z$. We assume that ${\mathfrak{N}}_z$ is fully connected, i.e., there is a synapse from every neuron in layer ${\mathfrak{E}}_e$ to every neuron in layer ${\mathfrak{E}}_{e+1}$, $1\le e<{\mathfrak{z}}_z$. Each synaptic weight $w_{i,j}^e$ (cf. Figure 1) is a real number. The vector ${\overrightarrow{w}}^e$ is the vector of all synaptic weights in ${\mathfrak{N}}_z$ from ${\mathfrak{E}}_e$ to ${\mathfrak{E}}_{e+1}$. Thus,

$${\overrightarrow{w}}^e=\left(w_{1,1}^e,\dots ,w_{1,n{n}_z(e+1)}^e,\dots ,w_{n{n}_z\left(e\right),1}^e,\dots ,w_{n{n}_z\left(e\right),n{n}_z(e+1)}^e\right).$$

We let ${\overrightarrow{w}}^0=\left(\right)$ and assume, without loss of generality, that, for any synaptic weight $w_{i,j}^e$, $0\le w_{i,j}^e\le 1$, because, if that is not the case, $w_{i,j}^e$ can be so scaled. No loss of generality is introduced with the assumption of full connectivity, because if full connectivity is not required, appropriate synaptic weights are set to zero. If, on the other hand, a given network is not fully connected, synapses with zero weights can be added as needed to make the network fully connected.

Each $n_i^e$, $e>1$ computes an activation function

$${\alpha }_i^e\left({\overrightarrow{a}}^{e-1},{\overrightarrow{w}}^{e-1}\right):{\mathbb{R}}^{dim\left({\overrightarrow{a}}^{e-1}\right)},{\mathbb{R}}^{dim\left({\overrightarrow{w}}^{e-1}\right)}\to \mathbb{R},$$

(16)

where ${\overrightarrow{a}}^{e-1}$ is the vector of the activations of the neurons in layers ${\mathfrak{E}}_{e-1}$, $dim({\overrightarrow{a}}^{e-1})=n{n}_z(e-1)$, and $dim({\overrightarrow{w}}^{e-1})=n{n}_z(e-1)n{n}_z(e)$. If $\overrightarrow{x}$ is the input to ${\mathfrak{N}}_z$, then ${\overrightarrow{a}}^1=\overrightarrow{x}$. For the input layer, we have

$${\alpha }_i^1(\overrightarrow{x},{\overrightarrow{w}}^0)={\alpha }_1^1(\overrightarrow{x},())=\overrightarrow{x}\left[i\right],1\le i\le n{n}_z\left(1\right).$$

(17)

The term feedforward means that the activations of the neurons are computed layer by layer from the input layer to the output layer, because the activation functions of the neurons in the next layer require only the weights of the synapses connecting the current layer with the next one and the activation values, i.e., the outputs of the activation functions of the neurons in the current layer. If $\overrightarrow{x}$ is the input vector, then

$$\begin{array}{ccc}{\overrightarrow{a}}^1\hfill & =\hfill & \left({\alpha }_1^1\left(\overrightarrow{x},\left(\right)\right),\dots {\alpha }_{n{n}_z\left(1\right)}^1\left(\overrightarrow{x},\left(\right)\right)\right)=\overrightarrow{x},\hfill \\ {\overrightarrow{a}}^e\hfill & =\hfill & \left({\alpha }_1^e\left({\overrightarrow{a}}^{e-1},{\overrightarrow{w}}^{e-1}\right),\dots ,{\alpha }_{nn\left(e\right)}^e\left({\overrightarrow{a}}^{e-1},{\overrightarrow{w}}^{e-1}\right)\right),1<e<{\mathfrak{z}}_z.\hfill \end{array}$$

(18)

The feedforward activation function $f_z$ that computes the activations of ${\mathfrak{N}}_z$ layer by layer can be defined as

$$\begin{array}{ccc}{f}_z(\overrightarrow{x},0)\hfill & =\hfill & \overrightarrow{x},\hfill \\ f_z(\overrightarrow{x},e+1)\hfill & =\hfill & \left({\alpha }_1^{e+1}\left(f_z\left(\overrightarrow{x},e\right),{\overrightarrow{w}}^e\right),\dots ,{\alpha }_{nn(e+1)}^{e+1}\left(f_z\left(\overrightarrow{x},e\right),{\overrightarrow{w}}^e\right)\right).\hfill \end{array}$$

(19)

Thus, $f_z(\overrightarrow{x},0)=f_z(\overrightarrow{x},1)={\overrightarrow{a}}^1=\overrightarrow{x}=\mathbf{x}$ and $f_z(\overrightarrow{x},e)={\overrightarrow{a}}^e,1\le e\le {\mathfrak{z}}_z$. If ${\mathfrak{N}}_z$ maps ${\overrightarrow{a}}^1\in A^n$ to ${\overrightarrow{b}}^{{\mathfrak{z}}_z}\in B^m$, for some sets A and B, we define the function ${\zeta }_z:A^n\to B^m$ computed by ${\mathfrak{N}}_z$ as

$${\zeta }_z(\overrightarrow{x})=f_z(\overrightarrow{x},{\mathfrak{z}}_z).$$

(20)

Definition 5.

A function $f:A^n\to B^m$, for some sets A and B, is $\mathfrak{N}$-computable if there is a network ${\mathfrak{N}}_z$ such that, for all $\overrightarrow{x}=\mathbf{x}\in A^n$,

$${\zeta }_z(\overrightarrow{x})=f(\overrightarrow{x},{\mathfrak{z}}_z)=\overrightarrow{y}={\zeta }_z(\mathbf{x})=f(\mathbf{x},{\mathfrak{z}}_z)=\mathbf{y}\in B^m.$$

If ${\mathfrak{N}}_z$ computes f, we refer to ${\mathfrak{N}}_z$ as ${\mathfrak{N}}_{f(·)}$ and use the expression ${\mathfrak{N}}_{f(·)}:A^n\mapsto B^m$ as a shorthand for ${\zeta }_z:A^n\mapsto B^m$. Furthermore, if ${\mathfrak{N}}_z$ computes f, then, for $\overrightarrow{x}=\mathbf{x}\in A^n$, the expressions ${\zeta }_z(\overrightarrow{x})$, ${\zeta }_z(\mathbf{x})$, ${\mathfrak{N}}_z(\overrightarrow{x})$, ${\mathfrak{N}}_z(\mathbf{x})$ are equivalent in that

$${\zeta }_z(\overrightarrow{x})={\mathfrak{N}}_z(\overrightarrow{x})=\overrightarrow{y}={\zeta }_z(\mathbf{x})={\mathfrak{N}}_z(\mathbf{x})=\mathbf{y}\in B^m.$$

(21)

A network ${\mathfrak{N}}_z$ can include other networks. Let ${\mathfrak{N}}_j$ and ${\mathfrak{N}}_k$ be two networks such as ${\zeta }_j:A^m\mapsto B^n$ and ${\zeta }_k:B^n\mapsto C^k$, for some sets A, B, C, and $0<m,n,k$. Then we can construct a new network ${\mathfrak{N}}_l$ by feeding the output of ${\mathfrak{N}}_j$ to ${\mathfrak{N}}_k$ so that ${\zeta }_l:A^m\mapsto C^k$ (cf. Figure 2). We can generalize this case to a network that includes arbitrarily many networks whose outputs are the inputs to another network whose output is the output of the entire complex network (cf. Figure 3). Formally, let ${\mathfrak{N}}_{z_1},\dots ,{\mathfrak{N}}_{z_l}$ be networks such that ${\zeta }_{z_1}:I^{n_{z_1}}\to O^{k_{z_1}},\dots ,{\zeta }_{z_l}:I^{n_{z_l}}\to O^{k_{z_l}}$ for some sets I and O, $0<n_{z_i},k_{z_i}$, and $1\le i\le l$. Let, for some set S, a network ${\mathfrak{N}}_j$ compute the function

$${\zeta }_j:O^{k_{z_1}},O^{k_{z_2}},\dots ,O^{k_{z_l}}\to S^m$$

so that

$${\zeta }_z({\overrightarrow{x}}_{z_1},\dots ,{\overrightarrow{x}}_{z_l})={\zeta }_j({\zeta }_{z_1}({\overrightarrow{x}}_{z_1}),\dots ,{\zeta }_{z_l}({\overrightarrow{x}}_{z_l}))=\overrightarrow{s}\in S^m,{\overrightarrow{x}}_{z_i}\in I^{n_{z_i}},1\le i\le l.$$

Then, for ${\overrightarrow{x}}_{z_i}\in I^{n_{z_i}}$ such that ${\overrightarrow{x}}_{z_i}={\mathbf{x}}_{z_i}$, $1\le i\le l$,

$${\mathfrak{N}}_z({\mathbf{x}}_{z_1},\dots ,{\mathbf{x}}_{z_l})={\mathfrak{N}}_z(\mathbf{y})={\mathfrak{N}}_j({\mathfrak{N}}_{z_1}({\mathbf{x}}_{z_1}),\dots ,{\mathfrak{N}}_{z_l}({\mathbf{x}}_{z_l}))=\mathbf{s}\in S^m,$$

(22)

where $\mathbf{y}=({\mathbf{x}}_{z_1}[1],\dots ,{\mathbf{x}}_{z_1}[n_{z_1}],\dots ,{\mathbf{x}}_{z_l}[1],\dots ,{\mathbf{x}}_{z_l}[n_{z_l}])$, and $\mathbf{s}=\overrightarrow{s}$.

We use the symbol ${\mathfrak{N}}_{id}$ to denote an identity network such that, for $\overrightarrow{a}=\mathbf{a}\in A^n$, $0<n$, ${\zeta }_{id}(\overrightarrow{a})=\overrightarrow{a}={\zeta }_{id}(\mathbf{a})=\mathbf{a}$. One can think of ${\mathfrak{N}}_{id}$ as a single layer network of n neurons, where ${\alpha }_i^1(\overrightarrow{a},())=\overrightarrow{a}\left[i\right]={\alpha }_i^1(\mathbf{a},())=\mathbf{a}\left[i\right]$, $1\le i\le n$.

Our formalization of feedforward artificial neural networks as finite sets of neurons and synapses organized in finitely many layers is in compliance with the original definition by McCulloch and Pitts (Sec. 2, p. 103 in [11]) who state that the neurons of a given network may be assigned designations $c_1$, $c_2$, …, $c_n$. It is also in compliance with the subsequent definition by Rumelhart, Hinton, and Williams [12] as well as with modern treatments of neural networks by Nielsen [17] and Goodfellow, Bengio, and Courville [18] that continue to describe neural networks as finite sets of neurons and synapses.

5. $\mathfrak{N}$-Computability of Primitive Recursive Functions

Lemma 1.

The initial functions are $\mathfrak{N}$-computable.

Proof. 

Let ${\mathfrak{N}}_{n(·)}:\mathbb{N}\mapsto \mathbb{N}$ be a network with a single input node $n_1^1$ and a single output node $n_1^2$ such that $w_{1,1}^1=0$ and ${\alpha }_1^2({\overrightarrow{a}}^1,{\overrightarrow{w}}^1)={\overrightarrow{a}}^1\left[1\right]{\overrightarrow{w}}^1\left[1\right].$ Then, ${\zeta }_{n(·)}\left(x\right)={\alpha }_1^2\left(\left(x\right)\left(0\right)\right)=x·0=0=n\left(x\right)$, $x\in \mathbb{N}$. Let ${\mathfrak{N}}_{s(·)}:\mathbb{N}\mapsto \mathbb{N}$ be a network with a single input node $n_1^1$ and a single output node $n_1^2$ such that $w_{1,1}^1=1$ and ${\alpha }_1^2(\overrightarrow{a},\overrightarrow{w})=\overrightarrow{a}\left[1\right]{\overrightarrow{w}}^1\left[1\right]+1.$ Then, ${\zeta }_{s(·)}\left(x\right)={\alpha }_1^2(\left(x\right),\left(1\right))=x{w}_{1,1}^1+1=s\left(x\right)$, $x\in \mathbb{N}$. Let ${\mathfrak{N}}_{u_i^n(·)}:{\mathbb{N}}^n\mapsto \mathbb{N}$, $1\le i\le n$, $n>0$ be a network with n input nodes $n_1^1,\dots ,n_n^1$ and one output node $n_1^2$. Let $w_i^1=1$, $w_j^1=0$, $i\ne j$, $1\le j\le n$, and

$${\alpha }_1^2({\overrightarrow{a}}^1,{\overrightarrow{w}}^1)=\sum _{j=1}^n{\overrightarrow{a}}^1\left[j\right]{\overrightarrow{w}}^1\left[j\right].$$

Then, if $\overrightarrow{a}=\mathbf{a}\in {\mathbb{N}}^n$,

$${\zeta }_{u_i^n(·)}\left(\overrightarrow{a}\right)={\alpha }_1^2({\overrightarrow{a}}^1,{\overrightarrow{w}}^1)=\overrightarrow{a}\left[i\right]={\alpha }_1^2(\mathbf{a},{\overrightarrow{w}}^1)=\mathbf{a}\left[i\right]=u_i^n(\mathbf{a}\left[1\right],\dots ,\mathbf{a}\left[n\right]).$$

□

We abbreviate ${\mathfrak{N}}_{u_i^n(·)}$ as ${\mathfrak{N}}_{u(·)}$, because n and i are always evident from the context.

Lemma 2.

Let $\mathbf{x}\in {\mathbb{N}}^n$, $n>0$. Let $c_i^n\left(\mathbf{x}\right)$, $1\le i\le n$, be defined as

$$c_i^n\left(\mathbf{x}\right)=\left\{\begin{array}{cc}{u}_1^n\left(\mathbf{x}\right)\hfill & \mathrm{if}i=1,\hfill \\ u_2^n\left(\mathbf{x}\right)\hfill & \mathrm{if}i=2,\hfill \\ \dots \hfill & \\ u_n^n\left(\mathbf{x}\right)\hfill & \mathrm{if}i=n.\hfill \end{array}\right.$$

The $c_i^n$ is $\mathfrak{N}$-computable.

Proof. 

Since $u_i^n$ is primitive recursive, $c_i^n$ is primitive recursive, by the definition by cases theorem and its corollary (cf. Theorem 5.4, Chap. 3, Sec. 5, pp. 50–51 in [7]). Let ${\mathfrak{N}}_{c_i^n(·)}$ be a network with $n+1$ input nodes $n_1^1,\dots ,n_{n+1}^1$, where the first n nodes receive the n corresponding values of $\mathbf{x}\in {\mathbb{N}}^n$, and the last node $n_{n+1}^1$ receives $1\le i\le n$. Let ${\mathfrak{N}}_{c_i^n(·)}$ have one output node $n_1^2$ and let $w_{j,k}^1=1$, $1\le j\le n$. Let the activation function of $n_1^2$ be defined as

$${\alpha }_1^2({\overrightarrow{a}}^1,{\overrightarrow{w}}^1)=\left\{\begin{array}{cc}{\overrightarrow{a}}^1\left[1\right]{\overrightarrow{w}}^1\left[1\right]\hfill & \mathrm{if}{\overrightarrow{a}}^1[n+1]=1,\hfill \\ {\overrightarrow{a}}^1\left[2\right]{\overrightarrow{w}}^1\left[2\right]\hfill & \mathrm{if}{\overrightarrow{a}}^1[n+1]=2,\hfill \\ \dots \hfill & \\ {\overrightarrow{a}}^1\left[n\right]{\overrightarrow{w}}^1\left[n\right]\hfill & \mathrm{if}{\overrightarrow{a}}^1[n+1]=n.\hfill \end{array}\right.$$

Then,

$${\zeta }_{c_i^n}(\mathbf{x},j)=\mathbf{x}\left[j\right]=u_j^n\left(\mathbf{x}\right)=c_j^n\left(\mathbf{x}\right).$$

□

We abbreviate ${\mathfrak{N}}_{c_i^n(·)}$ as ${\mathfrak{N}}_{c(·)}$.

Lemma 3.

Let f be a $\mathfrak{N}$-computable function of k arguments, $k>0$, and $g_1,...,g_k$ be $\mathfrak{N}$-computable functions of n arguments each, $n>0$. Let a function h of n arguments be obtained from $f,g_1,\dots ,g_k$ by Schema (11). Then, h is $\mathfrak{N}$-computable.

Proof. 

Let $f,g_1,\dots ,g_k$ be computable by ${\mathfrak{N}}_{f(·)}:{\mathbb{N}}^k\mapsto \mathbb{N}$, ${\mathfrak{N}}_{g_1(·)}:{\mathbb{N}}^n\mapsto \mathbb{N},\dots ,{\mathfrak{N}}_{g_k(·)}:{\mathbb{N}}^n\mapsto \mathbb{N}$. Then let ${\mathfrak{N}}_j:{\mathbb{N}}^n\mapsto \mathbb{N}$ be a network such that, for $\mathbf{x}\in {\mathbb{N}}^n$,

$${\mathfrak{N}}_j\left(\mathbf{x}\right)={\mathfrak{N}}_{f(·)}({\mathfrak{N}}_{g_1(·)}\left(\mathbf{x}\right),\dots ,{\mathfrak{N}}_{g_k(·)}\left(\mathbf{x}\right)).$$

Then, for $\mathbf{z}\in {\mathbb{N}}^n$, we have

$${\mathfrak{N}}_j\left(\mathbf{z}\right)={\mathfrak{N}}_{f(·)}({\mathfrak{N}}_{g_1(·)}\left(\mathbf{z}\right),\dots ,{\mathfrak{N}}_{g_k(·)}\left(\mathbf{z}\right))=f(g_1\left(\mathbf{z}\right),\dots ,g_k\left(\mathbf{z}\right))=h\left(\mathbf{x}\right),$$

whence

$$\begin{array}{ccc}{\zeta }_j\left(\mathbf{x}\right)\hfill & =\hfill & h\left(\mathbf{x}\right).\hfill \end{array}$$

□

Lemma 4.

Let $k\in \mathbb{N}$. Then k is $\mathfrak{N}$-computable.

Proof. 

Let ${\mathfrak{N}}_{n(·)}$ and $N_{s(·)}$ be as constructed in Lemma 1. Let ${\left\{{\mathfrak{N}}_{s(·)}\right\}}^k$, $k\ge 0$ denote a network that consists of a finite sequence of k networks ${\mathfrak{N}}_{s(·)}$, where the first ${\mathfrak{N}}_{s(·)}$ receives its input from ${\mathfrak{N}}_{n(·)}$ and each subsequent ${\mathfrak{N}}_{s(·)}$ receives its input from the previous ${\mathfrak{N}}_{s(·)}$ (cf. Figure 2). Let ${\left\{{\mathfrak{N}}_{s(·)}\right\}}^0={\mathfrak{N}}_{n(·)}$. Let ${\mathfrak{N}}_{{\mathfrak{J}}_k}\left(0\right)={\left\{{\mathfrak{N}}_{s(·)}\right\}}^k\left({\mathfrak{N}}_{n(·)}\left(0\right)\right)$. Let $s^k\left(x\right)$ denote k compositions of $s\left(x\right)$ with itself, i.e., $s^1\left(x\right)=s\left(x\right)$, $s^2\left(x\right)=s\left(s\left(x\right)\right)$, etc. Then,

$$\begin{array}{ccc}{\mathfrak{N}}_{{\mathfrak{J}}_0}\left(0\right)\hfill & =\hfill & {\left\{{\mathfrak{N}}_{s(·)}\right\}}^0\left({\mathfrak{N}}_{n(·)}\left(0\right)\right)=0;\hfill \\ {\mathfrak{N}}_{{\mathfrak{J}}_1}\left(0\right)\hfill & =\hfill & {\left\{{\mathfrak{N}}_{s(·)}\right\}}^1\left({\mathfrak{N}}_{n(·)}\left(0\right)\right)=s\left(0\right)=1;\hfill \\ {\mathfrak{N}}_{{\mathfrak{J}}_2}\left(0\right)\hfill & =\hfill & {\left\{{\mathfrak{N}}_{s(·)}\right\}}^2\left({\mathfrak{N}}_{n(·)}\left(0\right)\right)=s^2\left(0\right)=2;\hfill \\ \hfill & \cdots \hfill & \\ {\mathfrak{N}}_{{\mathfrak{J}}_k}\left(0\right)\hfill & =\hfill & {\left\{{\mathfrak{N}}_{s(·)}\right\}}^k\left({\mathfrak{N}}_{n(·)}\left(0\right)\right)=s^k\left(0\right)=k.\hfill \end{array}$$

By induction on k, ${\zeta }_{{\mathfrak{J}}_k}\left(0\right)=k$. By construction, ${\zeta }_{{\mathfrak{J}}_k}\left(n\right)=k$, $n\in \mathbb{N}$. □

The next lemma, Lemma 5, is a technical result for Lemma 6. The function $x\stackrel{·}{-}y$ is primitive recursive (cf. Chap. 3, Sec. 4, p. 46 in [7]).

Lemma 5.

Let the function $x\stackrel{·}{-}y$: ${\mathbb{N}}^2\to \mathbb{N}$ be defined as

$$x\stackrel{·}{-}y=\{\begin{array}{cc}x-y\hfill & \mathrm{if}x\ge 0,\hfill \\ 0\hfill & \mathrm{if}x<y.\hfill \end{array}$$

(23)

Then, $x\stackrel{·}{-}y$ is $\mathfrak{N}$-computable.

Proof. 

Let ${\mathfrak{N}}_{\stackrel{·}{-}(·)}$ have two input nodes $n_1^1$, $n_2^1$ and one output node $n_1^2$. Let $w_{1,1}^1$ = $w_{2,1}^1$ = 1 and let

$${\alpha }_1^2({\overrightarrow{a}}^1,{\overrightarrow{w}}^1)=\left\{\begin{array}{cc}{\overrightarrow{a}}^1\left[1\right]{\overrightarrow{w}}^1-{\overrightarrow{a}}^1\left[2\right]{\overrightarrow{w}}^1\left[2\right]\hfill & \mathrm{if}{\overrightarrow{a}}^1\left[1\right]\ge {\overrightarrow{a}}^1\left[2\right],\hfill \\ 0\hfill & \mathrm{if}{\overrightarrow{a}}^1\left[1\right]<{\overrightarrow{a}}^1\left[2\right].\hfill \end{array}\right.$$

Then, for $\overrightarrow{a}=\mathbf{a}\in {\mathbb{N}}^2$, we have

$${\zeta }_{\stackrel{·}{-}(·)}\left(\overrightarrow{a}\right)={\alpha }_1^2({\overrightarrow{a}}^1,{\overrightarrow{w}}^1)={\overrightarrow{a}}^1\left[1\right]\stackrel{·}{-}\overrightarrow{a}\left[2\right]={\alpha }_1^2(\mathbf{a},{\overrightarrow{w}}^1)\mathbf{a}\left[1\right]\stackrel{·}{-}\mathbf{a}\left[2\right].$$

□

Definition 6 confines the notion of $\mathfrak{N}$-computability of some function $f(\mathbf{x},t)$ to the $\mathfrak{N}$-computability of the first k elements of the sequence $\left\{f\right(\mathbf{x},t\left)\right\}$, $t\in \mathbb{N}$.

Definition 6.

A function $f:A^n\times \mathbb{N}\to B^m$, for some sets A and B, is $\mathfrak{N}$-computable elementwise for any $k>0$ if there is a network ${\mathfrak{N}}_z$ such that, for any $\mathbf{z}\in A^n$, the first $k+1$ terms of the sequence

$$\left\{f\right(\mathbf{z},j\left)\right\}=f(\mathbf{z},0),f(\mathbf{z},1),\dots ,f(\mathbf{z},k),\dots$$

are the same as the terms of the tuple

$$\begin{array}{cccc}\left(\mathfrak{N}\right(\mathbf{z},0),\hfill & \mathfrak{N}(\mathbf{z},1),\hfill & \dots ,\hfill & \mathfrak{N}(\mathbf{z},k)),\hfill \end{array}$$

i.e., $f(\mathbf{z},i)=\mathfrak{N}(\mathbf{z},i),0\le i\le k$.

Thus, if a function $f(\mathbf{x},t)$ is $\mathfrak{N}$-computable, it is $\mathfrak{N}$-computable elementwise for any positive k.

Lemma 6.

Let $\varphi :{\mathbb{N}}^2\mapsto \mathbb{N}$ be $\mathfrak{N}$-computable elementwise and $h\left(t\right)$ be a function obtained from ϕ by Schema (12). Then, h is $\mathfrak{N}$-computable elementwise.

Proof. 

Let ϕ be computable elementwise by ${\mathfrak{N}}_{\varphi (·)}$. Let ${\mathfrak{N}}_{{\tilde{h}}_0}\left(0\right)={\mathfrak{N}}_{{\mathfrak{J}}_k}\left(0\right)=k$ as constructed in Lemma 4. In the equations below, we abbreviate ${\mathfrak{N}}_{n(·)}\left(0\right)$ as 0, ${\mathfrak{N}}_{{\mathfrak{J}}_k}\left(0\right)$ as k, ${\mathfrak{N}}_{{\mathfrak{J}}_t}\left(0\right)$ as ${\mathfrak{N}}_{{\mathfrak{J}}_t}$, ${\mathfrak{N}}_{\stackrel{·}{-}(·)}(x,y)$ as $x\stackrel{·}{-}y$ and ${\mathfrak{N}}_{{\tilde{h}}_i}\left(i\right)$ as ${\mathfrak{N}}_{{\tilde{h}}_i}$. Let

$$\begin{array}{ccc}{\mathfrak{N}}_{{\tilde{h}}_0}\hfill & =\hfill & k,\hfill \\ {\mathfrak{N}}_{{\tilde{h}}_{t+1}}\hfill & =\hfill & {\mathfrak{N}}_{\varphi (·)}\left({\mathfrak{N}}_{{\mathfrak{J}}_t},{\mathfrak{N}}_{{\tilde{h}}_t}\right).\hfill \end{array}$$

(24)

By induction on t, $h\left(t\right)={\mathfrak{N}}_{\tilde{h}(·)}\left(t\right)$ (cf. Figure 4). Let

$${\mathfrak{N}}_{h(·)}\left(t\right)={\mathfrak{N}}_{c(·)}\left({\mathfrak{N}}_{{\tilde{h}}_0},\dots ,{\mathfrak{N}}_{{\tilde{h}}_m},t+1\right),0\le t\le m,m>0.$$

(25)

Then, the first $m+1$ terms of the sequence $\left\{h\right(t\left)\right\}$ are the same as the terms of the tuple $({\mathfrak{N}}_{h(·)}\left(0\right),\dots ,{\mathfrak{N}}_{h(·)}\left(m\right))$ (cf. Figure 5 for $m=3$). □

Lemma 7.

Let $f:{\mathbb{N}}^n\mapsto \mathbb{N}$ and $g:{\mathbb{N}}^2\mapsto \mathbb{N}$ be $\mathfrak{N}$-computable elementwise and $h:{\mathbb{N}}^{n+1}\mapsto \mathbb{N}$ be a function obtained from f and g by Schema (13). Then h is $\mathfrak{N}$-computable elementwise.

Proof. 

Let $\mathbf{x}\in {\mathbb{N}}^n$ and $\mathbf{y}\in {\mathbb{N}}^{n+2}$, $n>0$, such that $\mathbf{y}=(y_1,y_2,\mathbf{x}\left[\mathbf{1}\right],\dots ,\mathbf{x}\left[n\right])$. Let f and g be $\mathfrak{N}$-computable elementwise by ${\mathfrak{N}}_{f(·)}$ and ${\mathfrak{N}}_{g(·)}$, respectively. Let us abbreviate ${\mathfrak{N}}_{{\tilde{h}}_{\mathbf{x},t}}(\mathbf{x},t)$ as ${\mathfrak{N}}_{{\tilde{h}}_{\mathbf{x},t}}$ and let

$$\begin{array}{ccc}{\mathfrak{N}}_{{\tilde{h}}_{\mathbf{x},0}}\hfill & =\hfill & {\mathfrak{N}}_{f(·)}\left(\mathbf{x}\right),\hfill \\ {\mathfrak{N}}_{{\tilde{h}}_{\mathbf{x},t+1}}\hfill & =\hfill & {\mathfrak{N}}_{g(·)}\left(t,{\mathfrak{N}}_{{\tilde{h}}_{\mathbf{x},t}},\mathbf{x}\right).\hfill \end{array}$$

(26)

By induction on t, $h(\mathbf{x},t)$ = ${\mathfrak{N}}_{{\tilde{h}}_{\mathbf{x},t}}$. Let

$${\mathfrak{N}}_{h(·)}(\mathbf{x},t)={\mathfrak{N}}_{c(·)}\left({\mathfrak{N}}_{{\tilde{h}}_{\mathbf{x},0}},\dots ,{\mathfrak{N}}_{{\tilde{h}}_{\mathbf{x},m}},t+1\right),0\le t\le m,m>0.$$

(27)

Then the first $m+1$ terms of the sequence $\left\{h\right(\mathbf{x},t\left)\right\}$, i.e., $h(\mathbf{x},0),\dots ,h(\mathbf{x},m)$, agree elementwise with the tuple $({\mathfrak{N}}_{h(·)}(\mathbf{x},0),\dots ,{\mathfrak{N}}_{h(·)}(\mathbf{x},m))$. □

Figure 6 and Figure 7 illustrate sample constructions of Lemma 7. If we treat $h\left(t\right)$ as a shorthand for $h\left(\right(),t)$, then Lemmas 6 and 7 give us the following theorem.

Theorem 1.

Let $h(\mathbf{x},t)$ be a primitive recursive function, $\mathbf{x}\in {\mathbb{N}}^n$, $n\ge 0$. Then $h(\mathbf{x},t)$ is $\mathfrak{N}$-computable elementwise.

We can ask if the elementwise $\mathfrak{N}$-computability of $h(\mathbf{x},t)$ (cf. Definition 6) can be generalized to $\mathfrak{N}$-computability. In other words, is it possible to have the sequences $\left\{h\right(\mathbf{x},t\left)\right\}$ and $\left\{\mathfrak{N}\right(\mathbf{x},n\left)\right\}$ agree term by term, i.e., $h(\mathbf{x},t)=\mathfrak{N}(\mathbf{x},t)$? Since $\mathfrak{N}$ has a finite set of neurons organized into a finite number of layers, $\mathfrak{N}$ can compute, per Lemmas 6 and 7, only the first $m+1$ values of $h(\mathbf{x},t)$, i.e., $h(\mathbf{x},t)$, $0\le t\le m$, although m can be an arbitrarily large natural number. Thus, the answer to this question is negative.

Let us assume that ${\mathfrak{N}}_{h(\mathbf{x},t)}$ in Theorem 1 is allowed to have countably many neurons so that the number of neurons in the hidden layers of ${\mathfrak{N}}_{h(\mathbf{x},t)}$ is countable. Let ${\zeta }_{\mathfrak{N}}(\mathbf{x},t)$ be the function computed by ${\mathfrak{N}}_{h(\mathbf{x},t)}$. Since countably many neurons can be added to ${\mathfrak{N}}_{h(\mathbf{x},t)}$ to compute $h(\mathbf{x},t)$, for any t, we have the sequence $\left\{{\zeta }_{\mathfrak{N}}(\mathbf{x},t)\right\}$ = $\left\{\mathfrak{N}\right(\mathbf{x},t\left)\right\}$, on the one hand, and the sequence $\left\{h\right(\mathbf{x},t\left)\right\}$, on the other hand. Let $f(\mathbf{x},t)=h(\mathbf{x},t)-{\zeta }_{\mathfrak{N}}(\mathbf{x},t)$. Since $h(\mathbf{x},t)={\zeta }_{\mathfrak{N}}(\mathbf{x},t)$, for any $t\in \mathbb{N}$, $\left\{f\right(\mathbf{x},t\left)\right\}$ is vacuously convergent, i.e., ${lim}_{t\to \infty }f(\mathbf{x},t)=0$. Hence, we have the following theorem.

Theorem  2.

Let $h(\mathbf{x},t)$ be a primitive recursive function, $\mathbf{x}\in {\mathbb{N}}^n$, $n\ge 0$. Then there is a network $\mathfrak{N}(\mathbf{x},t)$ with countably many neurons such that for any $\mathbf{z}\in {\mathbb{N}}^n$, the sequences $\left\{h\right(\mathbf{z},t\left)\right\}$ and $\left\{{\zeta }_{\mathfrak{N}}(\mathbf{z},t\}\right)\}$ agree term by term, i.e., $h(\mathbf{z},t)={\zeta }_{\mathfrak{N}}(\mathbf{z},t)$, $t\in \mathbb{N}$.

6. Discussion

As mathematical objects, feedforward artificial neural networks are more computationally powerful than primitive recursive functions inasmuch as the former can compute functions over real numbers whereas the latter, by definition, cannot. E.g., one can define a network that computes the sum of n real numbers, which no primitive recursive function can compute. However, the situation changes when networks cease to be mathematical objects and become computational objects by being realized on finite memory devices. A finite memory device is a computational device with a finite amount of memory available for numerical computation [21]. Such a device is analogous to a human scribe with a pencil and an eraser who is to carry out a numerical computation by writing and erasing symbols from a finite alphabet on a finite number of paper sheets. Finite memory devices are different from finite state automata of classical computability theory (e.g., a deterministic finite state machine (Chap. 2, Sec. 2.2 in [22]), non-deterministic finite state machine (Chap. 2, Sec. 2.3 in [22]), a Mealy or Moore machine (Chap. 2, Sec. 2.7 in [22], a push down automaton (Chap. 5 in [22]), or a Turing machine (Chap. 6 in [7]), because the latter do not put any bounds on the number of cells in their tapes available for computation. A finite state automaton of classical computability becomes a finite memory device only when the number of its tape cells available for computation is bounded by a natural number.

A real number x is signifiable on a finite memory device $D_j$ if and only if the finite amount of memory on $D_j$ can hold its sign, where a sign is a sequence of arbitrary symbols from a finite alphabet [21]. Thus, if the alphabet is { “.”, “0”, “1”, “2”, “3”, “4”, “5”, “6“, “7”, “8”, “9” } and $D_j$ has 8 memory cells to represent a real number, then the real numbers 1.41, 1.414, 1.4142, 1.41421, 1.414213 are signifiable on $D_j$ as “1.41”, “1.414”, “1.4142”, “1.41421”, “1.414213”, respectively, whereas the real numbers 1.4142135, 1.41421356, 1.414213562, 1.4142135623, and 1.41421356237 are not. A consequence of the finite amount of memory is that the set of real numbers signifiable on $D_j$ is finite and, hence, vacuously countable. To put it differently, Cantor’s theorem (§ 2 in [23]) does not apply insomuch as the number of signifiable reals on $D_j$ in any interval $(\alpha \dots \beta )$, $\alpha ,\beta \in \mathbb{R}$, $\alpha <\beta$, is finite. Consequently, all computation of a feedforward artificial neural network ${\mathfrak{N}}_z:{\mathbb{R}}^n\mapsto {\mathbb{R}}^m$, $0<n,m$, realized on $D_j$, can be packed into a unique natural number ${\Omega }_z$ such that there exists a primitive recursive function $\tilde{f}:\mathbb{N}\mapsto \mathbb{N}$ such that ${\zeta }_z\left(\overrightarrow{x}\right)=\overrightarrow{a}$ if and only if $\tilde{f}\left(\tilde{x}\right)=\tilde{a}$, where $\overrightarrow{x}$ uniquely corresponds to $\tilde{x}$ and $\overrightarrow{a}$ to $\tilde{a}$ (cf. Theorem 1, pp. 15–17 in [21]). Theorem 1 is, after a fashion, the converse of Theorem 1 in [21] in the sense that it shows how one can construct a network from a primitive recursive function.

Theorem 2 shows that all values of a primitive recursive function can be computed exactly by a feedforward artificial neural network if the network is allowed to have countably many neurons. This purely theoretical result contributes to the growing collection of universality theorems on feedforward neural networks and various classes of functions (cf. Ch. 4 in [17]). Thus, Hornik et al. [13] show that multilayer feedforward networks with a single hidden layer of neurons with arbitrary squashing activation functions can approximate any Borel measurable function from one dimensional space to another to any desired degree of accuracy so long as the number of the neurons in the hidden layer is unbounded. Gripenberg [14] shows that the general approximation property of feedforward perceptron networks is achievable when the number of perceptrons in each layer is bounded but the number of layers is allowed to grow to infinity and the perceptron activation functions are continuously differentiable and not linear. Guliyev and Ismailov [15] show that single hidden layer feedforward neural networks with the fixed weights of one and two neurons in the hidden layer approximate any continuous function on a compact subset of the real line and proceed to demonstrate that single layer feedforward networks with fixed weights cannot approximate all continuous multivariate functions.

We conclude our discussion with a caveat about universality results of feedforward neural networks with unbounded numbers of neurons. While these results provide valuable theoretical insights, they may not hold much sway with computer scientists interested in computability properties of finite AI, because networks with unbounded numbers of neurons cannot be realized on computational devices with finite amounts of computational memory.

7. Conclusions

We have formalized feedforward artificial neural networks with recurrence equations and proposed a formal definition of the concept of $\mathfrak{N}$-computability, i.e., the property of a function to be computed by a feedforward artificial neural network $\mathfrak{N}$. We have shown that, for a primitive recursive function $h(\mathbf{x},t)$, where $\mathbf{x}$ is an n-tuple of natural numbers and t is a natural number, there exists a feedforward artificial neural network $\mathfrak{N}(\mathbf{x},t)$ such that for any n-tuple of natural numbers $\mathbf{z}$, the first $m+1$ terms of the sequence $\left\{h\right(\mathbf{z},t\left)\right\}$ agree elementwise with the tuple $\left(\mathfrak{N}\right(\mathbf{z},0),\dots ,\mathfrak{N}(\mathbf{z},m\left)\right)$, for any positive natural number m. Our investigation contributes to the knowledge of the classes of functions that can be computed by feedforward artificial neural networks. Since such networks are used in some finite AI systems, our investigation may be of interest to mathematicians and computer scientists interested in the computability theory of finite AI.