A Complete Neural Network-Based Representation of High-Dimension Convolutional Neural Networks

Abstract

Convolutional Neural Networks (CNNs) are a highly used machine learning architecture in various fields. Typical descriptions of CNNs are based on low-dimension and tensor representations in the feature extraction part. In this article, we extend the setting of CNNs to any arbitrary dimension and linearize the whole setting via the typical layers of neurons. In essence, a partial and a full network construct the entire process of a standard CNN, with the partial network being used to linearize the feature extraction. By doing so, we link the tensor-style representation of CNNs with the pure network representation. The outcomes serve two main purposes: to relate CNNs with other machine learning frameworks and to facilitate intuitive representations.

1. Introduction

CNNs, based on the concept of inner product and sliding windows, serve as a vital analyzing tool in data science and artificial intelligence [1,2,3]. They are not only theoretically justified but also practically accepted: radiology [4], image processing [5,6,7], protein analysis [8], computer vision [9], signal processing [10], movement recognition [11], cracked surface detection [12], plant disease detection [13], handwritten character classification [14], etc. Most of the presentation of CNNs is described by graphs [15,16]. This causes two main issues: (1) determining how to unify CNNs with ANNs under the framework—in the literature, they are not unified under the same setting, i.e., neurons and weights, but an informal description of the feature extraction and learning, together with ANNs [17]; (2) how to extend CNNs (typically applied to 2D or 3D objects) to any arbitrary n-dimensional objects. The typical settings in the literature about CNNs are focused on 2D or 3D object learning, such as computer vision [18] and object/image status classification [19,20]. Despite the usual textbooks or applications (2D/3D image recognition/classification, etc.), standard CNNs can also be applied to the high-dimensional data. Indeed, they could also easily be applied via the concepts of high-dimensional tensors for both the input tensor/object and the kernel/filter tensors [21,22]. The main idea of this article is not to show how to apply CNNs to high-dimensional data but to reveal how to convert a typical/standard high-dimensional CNN to a linearized high-dimensional CNN. The main purpose is to yield a coherent network under the same framework. Though these papers mention the need to consider multiple dimensions, or more than three dimensions, they do not provide a theoretical framework to accommodate such a need. To overcome these two limitations, we unify CNNs and ANNs by linearizing the CNN and then combining the systems into one system via partial and full networks, and we extend the CNN to any arbitrary dimensions via tensor power and tensor product. This unified high-dimensional CNN, together with the ANN system, provides both theoretical and practical advantages, since all the objects are treated as one rather than split into various channels. We divide this article into various sections: from a basic introduction of background knowledge, notations, and related definitions to the derivations of theoretical settings and theories and an illustrative example to validate the theory and the practice.

2. Some Basic Notations

Let $\mathbb{N}$ denote the set of all the natural numbers. Let $\mathbb{R}$ denote the set of all the real numbers. Let ${\mathbb{R}}^{\infty }\equiv \mathbb{R}\cup \{+\infty ,-\infty \}$ denote the set of all the real numbers, including infinities. $⌊\alpha ⌋$ and $⌈\alpha ⌉$ are the floor and ceiling functions evaluated at the real value $\alpha \in {\mathbb{R}}^{\infty }$, respectively; in particular, $⌊\infty ⌋=0$. For convenience, let us simply define $⌊{\displaystyle \frac{\alpha }{0}}⌋=0$ for all $\alpha \in {\mathbb{R}}_0^{+}$, the set of all the non-negative real numbers. Let $\overline{n}$ denote the set $\{1,2,\dots ,n\}$ for any arbitrary $n\in \mathbb{N}$. Let $\overrightarrow{n}$ denote the vector of $[1,2,\dots ,n]$. Let the Cartesian product set ${\overline{n}}_1\times {\overline{n}}_2$ denote the set $\{(p,q):p\in {\overline{n}}_1,q\in {\overline{n}}_2\}$ and $\left(\overline{m_1:m_2}\right)\times \left(\overline{m_3:m_4}\right)$ denote the set $\{(p,q):m_1\le p\le m_2,m_3\le q\le m_4\}$ for any $n_1,n_2,m_1,m_2,m_3,m_4\in \mathbb{N}$. Let the n-ary Cartesian product ${\displaystyle \prod _{j=1}^N{\overline{n}}_j}$ denote the set $\{(i_1,i_2,\dots ,i_N):i_j\in {\overline{n}}_j$ for all $1\le j\le N\}$. If S is a set, we use $\left|S\right|$ to denote the size (cardinality) of the set S. Let $\overrightarrow{i}\in {\mathbb{N}}^k$, the k-ary Cartesian product over sets $\mathbb{N}$, be an n-ary column vector. Let $\overrightarrow{i}\left[m\right]$ (or $i_m$) denote the mth element in the vector $\overrightarrow{i}$. Let $|\overrightarrow{i}|$ denote the length of vector $\overrightarrow{i}$. Let $N\in \mathbb{N}$ be arbitrary. Let $n_1,n_2,\dots ,n_N\in \mathbb{N}$ be arbitrary. Suppose A is an arbitrary k-dimensional tensor after all the preliminary processing of the original object (for example, the padding of an input text or image), or A could be identified by a function ${\displaystyle f_A:\prod _{j=1}^k{\overline{n}}_j\to {\mathbb{R}}^D}$ (or ${\displaystyle A\equiv f_A:\prod _{j=1}^k{\overline{n}}_j\to {\mathbb{R}}^D}$), where ${\displaystyle D=\prod _{j=1}^k{n}_j}$. Let ${\overrightarrow{\sigma }}_j=[{\sigma }_{1j},{\sigma }_{2j},\dots ,{\sigma }_{kj}]\in {\mathbb{N}}^k$ denote the stride vector, where ${\sigma }_{ij}$ denotes the number of strides taken in the ith dimension at the jth convolutional layer. Let ${\overrightarrow{f}}_j\equiv [f_{1j},f_{2j},\dots ,f_{kj}]$-sectional tensor $F^{i,j}$ denotes the ith feature/filter in the jth layer. Let $F^j={\left\{F^{i,j}\right\}}_{i=1}^{m_j}$, where $m_j$ denotes the number of filters/features in layer j.

3. Basic Definitions and Properties

Definition 1. 

We use ${\mathcal{T}}_{\overrightarrow{\tau }}$ to denote a tensor $\mathcal{T}$ with sectional/directional vector $\overrightarrow{\tau }$, i.e., ${\displaystyle {\mathcal{T}}_{\overrightarrow{\tau }}:\prod _{k=1}^s\overline{\overrightarrow{\tau }\left[k\right]}\to \mathbb{R}}$, where $s=|\overrightarrow{\tau }|$ is the number of sections (or directions) of $\mathcal{T}$. If $S\subseteq {\prod }_{k=1}^s\overline{\overrightarrow{\tau }\left[k\right]}$, we use ${\mathcal{T}}_{\overrightarrow{\tau }}{\upharpoonright }_S$ to denote its partial function, i.e., ${\mathcal{T}}_{\overrightarrow{\tau }}{\upharpoonright }_S\left(\overrightarrow{\gamma }\right):={\mathcal{T}}_{\overrightarrow{\tau }}\left(\overrightarrow{\gamma }\right)$. Furthermore, ${\mathcal{T}}_{\overrightarrow{\tau }}\left(S\right)=\{{\mathcal{T}}_{\overrightarrow{\tau }}\left(\overrightarrow{\gamma }\right):\overrightarrow{\gamma }\in S\}$.

Remark 1. 

In most of the cases, kernels are representable by a product set (or tensor product). But in some cases, we might need to consider the irregular kernels, i.e., those that could not be represented directly by product sets. If that is the case, in our setting, we could simply pad the undefined cell by 0. For example, a kernel $\kappa :\left\{\right(1,2),(2,1\left)\right\}\to \mathbb{R}$ could be extended to a product set ${\kappa }^{+}:\{(1,1),(1,2),(2,1),(2,2)\}$ in which ${\kappa }^{+}(1,1)={\kappa }^{+}(2,2)=0;{\kappa }^{+}(1,2):=\kappa (1,2);{\kappa }^{+}(2,1):=\kappa (2,1)$. Such a setting could rescue us from redesigning the setting/modeling from scratch. This leads to another question: Why, or under which circumstance, shall we tend to adopt irregular kernels? An obvious one is that even the input data is not representable by a product set or a tensor. If this is the case, we could apply the same technique to pad the undefined cells by 0. Such accommodation suits our setting and is much more consistent with the settings of standard CNNs.

Definition 2. 

Let $\overrightarrow{\sigma }$ denote a stride vector whose elements indicate the strides with respect to each section (or direction).

Definition 3. 

Let $F_{\overrightarrow{f}}$ denote a filter (tensor), or kernel, where $\overrightarrow{f}$ is its sectional vector, i.e., ${\displaystyle F_{\overrightarrow{f}}:\prod _{k=1}^s\overline{\overrightarrow{f}\left[k\right]}\to \mathbb{R}}$. If we need to denote the ith filter in the jth layer, we use the notation $F_{\overrightarrow{f}}^{i,j}$.

Remark 2. 

For each kernel $F_{i,j}$ (the ith kernel in the jth layer), we could associate it with a stride vector ${\overrightarrow{\sigma }}_{i,j}$. This is based on the assumption that a uniform stride is applicable and appropriate for the sliding and inner product operation. In some cases, we might encounter non-uniform strides [23]. A much more generalized setting is where the strides (or the positions where the inner product takes place) are determined by a set of positional vectors (for example, such positions are randomly chosen). Then, this case is beyond our setting, since our setting is based on the standard CNN in which the strides normally are assumed to be uniform. In order not to reformulate our setting, we could add another masking layer after the feature maps [24]. Therefore, all the stride vectors consist of $\overrightarrow{1}\equiv (1,1,\dots ,1)$. Then, each feature map acts on the mask consisting of all values of 1 if the feature map is activated and 0 if the feature map is not activated. After this masking layer, in the pooling layer, one gets rid of all the values of 0 and their associated positions. The remaining values are linearized and fed into the ANN part.

Remark 3. 

Another interesting aspect of feature extraction is the dilated convolution neural network [25,26]. Since normally the dilated parts are padded by 0, the setting is still comparable with our setting regarding kernels. This is because the parameters to be learned lie in the kernels. To accommodate our setting with the dilated kernel is to fix the padded parameters and to learn the other non-padded parts in the kernel.

In order to present the cropped sub-tensors of ${\mathcal{T}}_{\overrightarrow{\tau }}$, given the filter $F_{\overrightarrow{f}}$ and stride vector $\overrightarrow{\sigma }$, we use a power tensor to collect such sub-tensors as follows:

Definition 4 

(power tensor one). Let ${\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}\left({\mathcal{T}}_{\overrightarrow{\tau }}\right)={\left\{T_{\overrightarrow{f}}^{\overrightarrow{\gamma }}\right\}}_{\overrightarrow{\gamma }\in {\prod }_{k=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\tau }\left[k\right]-\overrightarrow{f}\left[k\right]}{\overrightarrow{\sigma }\left[k\right]}}\rfloor +1}}$, where $T_{\overrightarrow{f}}^{\overrightarrow{\gamma }}={\mathcal{T}}_{\overrightarrow{\tau }}\left({\prod }_{k=1}^s\overline{(\overrightarrow{1}+(\overrightarrow{\gamma }-\overrightarrow{1})\odot \overrightarrow{\sigma })\left[k\right]:((\overrightarrow{\gamma }-\overrightarrow{1})\odot \overrightarrow{\sigma }+\overrightarrow{f})\left[k\right]}\right)$, where ⊙ indicates the Hadamard product and $\overrightarrow{1}=[1,1,\dots ,1]$.

Remark 4. 

In the definition, one observes that the floor function might contain a denominator of 0; this is normally defined by extended real numbers. As long as the numerator is non-negative, we could regard the evaluated value of the floor function as 0. As for a non-negative numerator, this is normally true by adding padding into the CNN systems. The same argument goes for the following definitions containing denominator 0.

Definition 5 

(power tensor two). Let ${\mathcal{P}}_{\overrightarrow{p}}^{\overrightarrow{\sigma }}\left({\mathcal{A}}_{\overrightarrow{\gamma }}\right)={\left\{A_{\overrightarrow{p}}^{\overrightarrow{\eta }}\right\}}_{\overrightarrow{\eta }\in {\prod }_{n=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\gamma }\left[n\right]-\overrightarrow{p}\left[n\right]}{\overrightarrow{\sigma }\left[n\right]}}\rfloor +1}}$, where $A_{\overrightarrow{p}}^{\overrightarrow{\eta }}={\mathcal{A}}_{\overrightarrow{\gamma }}\left({\prod }_{n=1}^s\overline{(\overrightarrow{1}+(\overrightarrow{\eta }-\overrightarrow{1})\odot \overrightarrow{\sigma })\left[n\right]:((\overrightarrow{\eta }-\overrightarrow{1})\odot \overrightarrow{\sigma }+\overrightarrow{f})\left[n\right]}\right)$.

Definition 6 

(maximum pooling function). Define

$${\mu }_{\overrightarrow{\gamma }}^{\ast }:\prod _{n=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\gamma }\left[n\right]-\overrightarrow{p}\left[n\right]}{\overrightarrow{\sigma }\left[n\right]}}\rfloor +1}\to \prod _{n=1}^s\overline{\overrightarrow{\gamma }\left[n\right]}$$

by

$${\mu }_{\overrightarrow{\gamma }}^{\ast }\left(\overrightarrow{\theta }\right)=argmax\{{\mathcal{A}}_{\overrightarrow{\gamma }}\left(\overrightarrow{\theta }\right):\overrightarrow{\theta }\in {\prod }_{n=1}^s\overline{(\overrightarrow{1}+(\overrightarrow{\gamma }-\overrightarrow{1})\odot \overrightarrow{\sigma })\left[n\right]:((\overrightarrow{\gamma }-\overrightarrow{1})\odot \overrightarrow{\sigma }+\overrightarrow{p})\left[n\right]}\}.$$

for all $\overrightarrow{\gamma }\in {\prod }_{n=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\tau }\left[n\right]-\overrightarrow{f}\left[n\right]}{\overrightarrow{\sigma }\left[n\right]}}\rfloor +1}$.

For a standard CNN, one could consult Figure 1. The figure shows a complete data set training. Suppose there are B training data points, i.e., $1\le j\le B$. The process consists mainly of three operators: convolution, activation function a, and pooling function p. The convolution operator (or inner product) acts on the input tensor and the filters/features. $F_{\overrightarrow{f}}^n[m;j]$ ($1\le n\le N$ denotes the nth filter at the mth recursive step for the jth input, and $\overrightarrow{f}$ is the dimensional vector of the filter). The activation function is denoted by a, while the pooling function is denoted by p. In the whole process, we insert an auxiliary operator ${\mathcal{P}}_{\overrightarrow{v}}^{\overrightarrow{u}}$ (named a power tensor operator), where $\overrightarrow{u}$ describes the strides assigned for all the dimensions and $\overrightarrow{v}$ is the dimensional vector of a truncated tensor (the tensor used to truncate its preceding tensor). The floor function is used to calculate the dimensional vector of the truncated tensors. Finally, the resulting (column) vector ${}_{j}X_m$ comes from flattening and stacking all the pooled tensors at the mth recursive step for the jth input. In this diagram, we do not directly take padding into consideration. It could be done by a slight tweaking of the setting. In addition, $F_{\overrightarrow{f}}^n[1,j+1]$ is defined to be $F_{\overrightarrow{f}}^n[N,j]$, where (suppose we fix the number) N is the total number of recursive steps for the backward induction. ${\prod }_{k=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\tau }\left[k\right]-\overrightarrow{f}\left[k\right]]}{\overrightarrow{\sigma }\left[k\right]}}\rfloor +1}\equiv {\prod }_{k=1}^s\overline{\overrightarrow{q}\left[k\right]}$, where $\overrightarrow{q}\left[k\right]=\lfloor {\displaystyle \frac{\overrightarrow{\tau }\left[k\right]-\overrightarrow{f}\left[k\right]}{\overrightarrow{\sigma }\left[k\right]}}\rfloor +1$. Moreover, ${\overrightarrow{q}}_j\left[k\right]=\lfloor {\displaystyle \frac{\overrightarrow{q}\left[k\right]-{\overrightarrow{p}}_j\left[k\right]}{{\overrightarrow{\sigma }}_j\left[k\right]}}\rfloor +1$ and ${\prod }_{k=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{q}\left[k\right]-{\overrightarrow{p}}_j\left[k\right]]}{{\overrightarrow{\sigma }}_j\left[k\right]}}\rfloor +1}\equiv {\prod }_{k=1}^s\overline{{\overrightarrow{q}}_j\left[k\right]}$.

A high-dimension tensor ${\mathcal{T}}_{\overrightarrow{\tau }}$ is fed into the CNN system. The cropped sub-tensors

$${\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}\left({\mathcal{T}}_{\overrightarrow{\tau }}\right)={\left\{T_{\overrightarrow{f}}^{\overrightarrow{\gamma }}\right\}}_{\overrightarrow{\gamma }\in {\prod }_{k=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\tau }\left[k\right]-\overrightarrow{f}\left[k\right]}{\overrightarrow{\sigma }\left[k\right]}}\rfloor +1}}$$

interact with the sequence of filters/kernels/tensors ${\left\{F_{\overrightarrow{f}}^n\right\}}_{n=1}^N$ to yield a sequence of feature maps/inner products

$${\left\{\left\{{\mathcal{H}}_{\overrightarrow{\gamma }}^n\equiv \langle T_{\overrightarrow{\tau }}^{\overrightarrow{\gamma }},F_{\overrightarrow{f}}^n\rangle :\overrightarrow{\gamma }\in \prod _{k=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\tau }\left[k\right]-\overrightarrow{f}\left[k\right]}{\overrightarrow{\sigma }\left[k\right]}}\rfloor +1}\right\}\right\}}_{n=1}^N.$$

After applying the activation function a on the feature maps, one has

$${\left\{\left\{{\mathcal{A}}_{\overrightarrow{\gamma }}^n\equiv a\left(\langle T_{\overrightarrow{\tau }}^{\overrightarrow{\gamma }},F_{\overrightarrow{f}}^n\rangle \right):\overrightarrow{\gamma }\in \prod _{k=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\tau }\left[k\right]-\overrightarrow{f}\left[k\right]}{\overrightarrow{\sigma }\left[k\right]}}\rfloor +1}\right\}\right\}}_{n=1}^N.$$

Then, one forms their sub-tensors via a sequence of power tensors ${\left\{{\mathcal{P}}_{{\overrightarrow{p}}_n}^{{\overrightarrow{\sigma }}_n}\right\}}_{n=1}^N$ and yields

$${\left\{\left\{{\mathcal{P}}_{{\overrightarrow{p}}_n}^{{\overrightarrow{\sigma }}_n}\left({\mathcal{A}}_{\overrightarrow{\gamma }}^n\right)\equiv {\mathcal{P}}_{{\overrightarrow{p}}_n}^{{\overrightarrow{\sigma }}_n}\circ a\left(\langle T_{\overrightarrow{\tau }}^{\overrightarrow{\gamma }},F_{\overrightarrow{f}}^n\rangle \right):\overrightarrow{\gamma }\in \prod _{k=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\tau }\left[k\right]-{\overrightarrow{p}}_n\left[k\right]}{{\overrightarrow{\sigma }}_n\left[k\right]}}\rfloor +1}\right\}\right\}}_{n=1}^N$$

$$\equiv {\left\{\left\{\left\{A_{{\overrightarrow{p}}_n}^{\overrightarrow{\gamma },\overrightarrow{\eta }}:\overrightarrow{\eta }\in \prod _{k=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\gamma }\left[k\right]-{\overrightarrow{p}}_n\left[k\right]}{{\overrightarrow{\sigma }}_n\left[k\right]}}\rfloor +1}\right\}:\overrightarrow{\gamma }\in \prod _{k=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\tau }\left[k\right]-\overrightarrow{p}\left[k\right]}{\overrightarrow{\sigma }\left[k\right]}}\rfloor +1}\right\}\right\}}_{n=1}^N.$$

Next, one applies pooling functions, p, on this set to form

$${\left\{\left\{p\left(\left\{A_{{\overrightarrow{p}}_n}^{\overrightarrow{\gamma },\overrightarrow{\eta }}:\overrightarrow{\eta }\in \prod _{k=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\gamma }\left[k\right]-{\overrightarrow{p}}_n\left[k\right]}{{\overrightarrow{\sigma }}_n\left[k\right]}}\rfloor +1}\right\}\right):\overrightarrow{\gamma }\in \prod _{k=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\tau }\left[k\right]-\overrightarrow{p}\left[k\right]}{\overrightarrow{\sigma }\left[k\right]}}\rfloor +1}\right\}\right\}}_{n=1}^N.$$

Lastly, one flattens them into one column vector $\overrightarrow{X}$ with respect to $\overrightarrow{\gamma }$ and n via f and $\tilde{f}$:

$$\tilde{f}({\left\{f\left(\left\{p\left(\left\{A_{{\overrightarrow{p}}_n}^{\overrightarrow{\gamma },\overrightarrow{\eta }}:\overrightarrow{\eta }\in \prod _{k=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\gamma }\left[k\right]-{\overrightarrow{p}}_n\left[k\right]}{{\overrightarrow{\sigma }}_n\left[k\right]}}\rfloor +1}\right\}\right):\overrightarrow{\gamma }\in \prod _{k=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\tau }\left[k\right]-\overrightarrow{p}\left[k\right]}{\overrightarrow{\sigma }\left[k\right]}}\rfloor +1}\right\}\right)\right\}}_{n=1}^N).$$

$$=\tilde{f}\left({\left\{C_{\overrightarrow{\gamma }}^n:\overrightarrow{\gamma }\in \prod _{k=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\tau }\left[k\right]-\overrightarrow{p}\left[k\right]}{\overrightarrow{\sigma }\left[k\right]}}\rfloor +1}\right\}}_{n=1}^N\right)=\overrightarrow{X}.$$

This could also be represented by the diagram presented in Figure 2.

In order to perform the backward propagation/pass/induction, we take the max pooling function p and keep track of all their indices as follows (fix n):

$${\mu }_n\left(\overrightarrow{\gamma }\right):=argmax\left\{A_{{\overrightarrow{p}}_n}^{\overrightarrow{\gamma },\overrightarrow{\eta }}:\overrightarrow{\eta }\in \prod _{k=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\gamma }\left[k\right]-{\overrightarrow{p}}_n\left[k\right]}{{\overrightarrow{\sigma }}_n\left[k\right]}}\rfloor +1}\right\},$$

for all $\overrightarrow{\gamma }\in {\prod }_{k=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\tau }\left[k\right]-\overrightarrow{p}\left[k\right]}{\overrightarrow{\sigma }\left[k\right]}}\rfloor +1}$ and for all $1\le n\le N$.

Example 1. 

Suppose ${\mathcal{T}}_{\overrightarrow{\tau }}={\left[t_{i,j}\right]}_{1\le i\le 6}^{1\le j\le 7}$ is a 6-by-7 matrix (tensor), where $\overrightarrow{\tau }=[6,7]$. Suppose the stride vector $\overrightarrow{\sigma }=[1,2]$ and $\overrightarrow{f}=[2,3]$. Then, $s=2$ and ${\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}\left({\mathcal{T}}_{\overrightarrow{\tau }}\right)={\left\{T_{\overrightarrow{f}}^{\overrightarrow{\gamma }}\right\}}_{\overrightarrow{\gamma }\in \overline{5}\times \overline{3}}=\left[\begin{array}{ccc}{\mathcal{T}}_{\overrightarrow{\tau }}(\overline{1:2}\times \overline{1:3})& {\mathcal{T}}_{\overrightarrow{\tau }}(\overline{1:2}\times \overline{3:5})& {\mathcal{T}}_{\overrightarrow{\tau }}(\overline{1:2}\times \overline{5:7})\\ {\mathcal{T}}_{\overrightarrow{\tau }}(\overline{2:3}\times \overline{1:3})& {\mathcal{T}}_{\overrightarrow{\tau }}(\overline{2:3}\times \overline{3:5})& {\mathcal{T}}_{\overrightarrow{\tau }}(\overline{2:3}\times \overline{5:7})\\ {\mathcal{T}}_{\overrightarrow{\tau }}(\overline{3:4}\times \overline{1:3})& {\mathcal{T}}_{\overrightarrow{\tau }}(\overline{3:4}\times \overline{3:5})& {\mathcal{T}}_{\overrightarrow{\tau }}(\overline{3:4}\times \overline{5:7})\\ {\mathcal{T}}_{\overrightarrow{\tau }}(\overline{4:5}\times \overline{1:3})& {\mathcal{T}}_{\overrightarrow{\tau }}(\overline{4:5}\times \overline{3:5})& {\mathcal{T}}_{\overrightarrow{\tau }}(\overline{4:5}\times \overline{5:7})\\ {\mathcal{T}}_{\overrightarrow{\tau }}(\overline{5:6}\times \overline{1:3})& {\mathcal{T}}_{\overrightarrow{\tau }}(\overline{5:6}\times \overline{3:5})& {\mathcal{T}}_{\overrightarrow{\tau }}(\overline{5:6}\times \overline{5:7})\end{array}\right]$. The convolution between T and some feature matrix is shown in Figure 3.

As for the procedure of linearizing the power tensor, one starts from linearizing each $T{\upharpoonright }_{D_{\overrightarrow{I}}}$ whose size is ${\displaystyle \prod _{p=1}^h{f}_{pj}}$. Hence, the total neurons in this layer should be ${\displaystyle \left(\prod _{p=1}^h{f}_{pj}\right)·\left(\prod _{p=1}^h{q}_{pj}\right)}$, i.e., the power tensor is stacked by the sequence ${\left\{T{\upharpoonright }_{D_{{\mathcal{L}}^{-1}\left(n\right)}}\right\}}_{n=1}^M$, where ${\displaystyle M=\prod _{p=1}^h{q}_{pj}}$ and each $T{\upharpoonright }_{D_{{\mathcal{L}}^{-1}\left(n\right)}}$ is a column vector with the length ${\displaystyle \prod _{p=1}^h{n}_{pj}}$.

4. Theoretical Settings

As for the procedure of linearizing the power tensor, one starts by linearizing each $T{\upharpoonright }_{D_{\overrightarrow{I}}}$ whose size is ${\prod }_{p=1}^h{f}_{pj}$. Hence, the total neurons in this layer should be $\left({\prod }_{p=1}^h{f}_{pj}\right)·\left({\prod }_{p=1}^h{q}_{pj}\right)$, i.e., the power tensor is stacked by the sequence ${\left\{T{\upharpoonright }_{D_{{\mathcal{L}}^{-1}\left(n\right)}}\right\}}_{n=1}^M$, where $M={\prod }_{p=1}^h{q}_{pj}$ and each $T{\upharpoonright }_{D_{{\mathcal{L}}^{-1}\left(n\right)}}$ is a column vector with the length ${\displaystyle \prod _{p=1}^h{n}_{pj}}$.

Definition 7 

($\overrightarrow{v}\widehat{\otimes }\overrightarrow{w}$). For any column vector $\overrightarrow{v}$ and $\overrightarrow{w}$, define $\left[\begin{array}{c}{v}_1\\ v2\\ ⋮\\ v_p\end{array}\right]\widehat{\otimes }\left[\begin{array}{c}{w}_1\\ w2\\ ⋮\\ w_q\end{array}\right]:=$$\left[\begin{array}{cccc}(v_1,w_1)& (v_1,w_2)& \dots & (v_1,w_q)\\ (v_2,w_1)& (v_2,w_2)& \dots & (v_2,w_q)\\ ⋮& ⋮& ⋮& ⋮\\ (v_p,w_1)& (v_p,w_2)& \dots & (v_p,w_q)\end{array}\right]$ and ${\overrightarrow{v}}^1\widehat{\otimes }{\overrightarrow{v}}^2\widehat{\otimes }\dots \widehat{\otimes }{\overrightarrow{v}}^n:={\left\{{\overrightarrow{v}}^1\left[t_1\right]\right\}}_{t_1=1}^{|{\overrightarrow{v}}^1|}\times {\left\{{\overrightarrow{v}}^2\left[t_2\right]\right\}}_{t_2=1}^{|{\overrightarrow{v}}^2|}\times \dots \times {\left\{{\overrightarrow{v}}^n\left[t_n\right]\right\}}_{t_n=1}^{|{\overrightarrow{v}}^n|}$.

Definition 8. 

Define the positional window/vector in the ith dimension by ${\overrightarrow{I}}_i$.

Claim 1 

(dimensional multipliers). If $dim\left(T\right)=[{\tau }_1,{\tau }_2,\dots ,{\tau }_h]$, $dim\left(F^j\right)=[f_{1j},f_{2j},\dots ,f_{h,j}]$ and the stride vector is ${\overrightarrow{\sigma }}_j=[{\sigma }_{1j},{\sigma }_{2j},\dots ,{\sigma }_{h,j}]$, then $q_{ij}=\lfloor {\displaystyle \frac{{\tau }_i-f_{ij}}{{\sigma }_{ij}}}\rfloor +1$, i.e., $Q(T/F^j)=[\lfloor {\displaystyle \frac{{\tau }_1-f_{1j}}{{\sigma }_{1j}}}\rfloor +1,\lfloor {\displaystyle \frac{{\tau }_2-f_{2j}}{{\sigma }_{2j}}}\rfloor +1,\dots ,\lfloor {\displaystyle \frac{{\tau }_h-f_{h,j}}{{\sigma }_{h,j}}}\rfloor +1]$.

Proof. 

Based on the concept of arithmetic sequence and the given conditions, for each $1\le i\le h$, there exists a maximum $e_i\in \mathbb{N}$ such that $1+e_i·{\sigma }_{ij}+(f_{ij}-1)\le {\tau }_i$, i.e., $e_i\le {\displaystyle \frac{{\tau }_i-f_{ij}}{{\sigma }_{ij}}}$. Since $q_{ij}=e_i+1$, the result follows immediately via the representation of the floor function. □

Definition 9. 

Let $\underset{q_{ij}}{\overset{f_{ij}}{\to }}$ denote the column vector $[q_{ij},q_{ij}+1,q_{ij}+2,\dots ,q_{ij}+f_{ij}-1]$.

Definition 10. 

The set of all the positional indices of $T/F^j$ (or $Q(T/F^j)$) is defined by ${\displaystyle \prod _{i=1}^h{\overline{q}}_{i,j}}$

Lemma 1 

($\overrightarrow{I}$-index tensor). The $\overrightarrow{I}$-index tensor of T is $T\left[\overrightarrow{I}\right]=T{\upharpoonright }_{D_{\overrightarrow{I}}}$ (a partial function), where the partial domain $D_{\overrightarrow{I}}=\underset{1+(i_1-1)·{\sigma }_{1j}}{\overset{f_{1j}}{\to }}\widehat{\otimes }\underset{1+(i_2-1)·{\sigma }_{2j}}{\overset{f_{2j}}{\to }}\widehat{\otimes }\dots \widehat{\otimes }\underset{1+(i_h-1)·{\sigma }_{hj}}{\overset{f_{hj}}{\to }}$, where $\overrightarrow{I}=[i_1,i_2,\dots ,i_h]\in {\overline{q}}_{1j}\times {\overline{q}}_{2j}\dots {\overline{q}}_{h,j}$.

Proof. 

Based on Definitions 7–10 and Claim 1, the results follow immediately. □

Claim 2 

(linearity for matrix). The function $l:{\overline{n}}_1\times {\overline{n}}_2\to \overline{n_1·n_2}$ defined by $l(i,j):=i+(j-1)·n_1$ is a bijective function for any given $n_1,n_2\in \mathbb{N}$.

Proof. 

Since $|{\overline{n}}_1\times {\overline{n}}_2|=|\overline{n_1·n_2}|$, it suffices to show that l is injective. Suppose $l(i,j)=l(i^{\prime },j^{\prime })$, or $i+(j-1)·n_1=i^{\prime }+(j^{\prime }-1)·n_1$, i.e., $i-i^{\prime }=(j^{\prime }-j)·n_1$. Since $-n_1<i-i^{\prime }<n_1$, i.e., $-n_1<(j^{\prime }-j)·n_1<n_1$, i.e., $-1<j^{\prime }-j<1$, one has $j=j^{\prime }$ and thus $i=i^{\prime }$. □

Lemma 2. 

(linearity of spatial indices of a tensor) The function ${\displaystyle l:\prod _{j=1}^M{\overline{n}}_j\to \overline{{\prod }_{j=1}^M{n}_j}}$ defined by ${\displaystyle l(i_1,i_2,\dots ,i_M):=i_1+\sum _{k=2}^M[(i_k-1)·\prod _{p=1}^{k-1}{n}_p]}$ is a bijective function.

Proof. 

We show this by mathematical induction. For $M=2$, it is shown to be true in Claim 2. Suppose $M=n$ holds, i.e., for any arbitrary $l(i_1,i_2,\dots ,i_n)=l(i_1^{\prime },i_2^{\prime },\dots ,i_n^{\prime })$, one has $(i_1,i_2,\dots ,i_n)=(i_1^{\prime },i_2^{\prime },\dots ,i_n^{\prime })$. Next, we show $M=n+1$ also holds. Suppose $l(i_1,i_2,\dots ,i_n,i_{n+1})=l(i_1^{\prime },i_2^{\prime },\dots ,i_n^{\prime },i_{n+1}^{\prime })$. Then, ${\displaystyle i_1+\sum _{k=2}^{n+1}[(i_k-1)·\prod _{p=1}^{k-1}{n}_p]=i_1^{\prime }+\sum _{k=2}^{n+1}[(i_k^{\prime }-1)·\prod _{p=1}^{k-1}{n}_p]}$, i.e., ${\displaystyle i_1-i^{\prime }=n_1·\sum _{k=2}^{n+1}[(i_k^{\prime }-i_k)·\prod _{p=1}^{k-1}{n}_p]=0}$. Since $1\le i_1,i_1^{\prime }<n_1$, one has $i_1=i_1^{\prime }$, i.e., ${\displaystyle \sum _{k=2}^{n+1}[(i_k-1)·\prod _{p=1}^{k-1}{n}_p]=\sum _{k=2}^{n+1}[(i_k^{\prime }-1)·\prod _{p=1}^{k-1}{n}_p]}$. Now, set $h=k-1$. Then, ${\displaystyle \sum _{h=1}^n[(i_{h+1}-1)·\prod _{p=1}^h{n}_p]=\sum _{h=1}^n[(i_{h+1}^{\prime }-1)·\prod _{p=1}^h{n}_p]}$, i.e., $l(i_2,\dots ,i_n,i_{n+1})=l(i_2^{\prime },\dots ,i_n^{\prime },i_{n+1}^{\prime })$. Based on the mathematical assumption, one has $i_j=i_j^{\prime }$ for all $2\le j\le n+1$. □

Lemma 2 is vital in converting a tensor-style representation of a CNN into a neuron-style representation of a CNN, as demonstrated in Corollaries 1–6.

Corollary 1 

(vectorization/linearization of the indices of an input tensor ${\mathcal{T}}_{\overrightarrow{\tau }}$). $l_{\mathcal{T}}:{\prod }_{k=1}^s\overline{\overrightarrow{\tau }\left[k\right]}\to \overline{{\prod }_{k=1}^s\overrightarrow{\tau }\left[k\right]}$ is a bijective function if $l_{\mathcal{T}}$ is defined in the same manner as l in Lemma 2.

Corollary 2. 

(vectorization/linearization of the indices of sub-tensors or power tensors of ${\mathcal{T}}_{\overrightarrow{\tau }}$) $l_{\overrightarrow{\tau }}:{\prod }_{j=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\tau }\left[j\right]-\overrightarrow{f}\left[j\right]}{\overrightarrow{\sigma }\left[j\right]}}\rfloor +1}\to \overline{|{\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}\left({\mathcal{T}}_{\overrightarrow{\tau }}\right)|}$ is a bijective function if $l_{\overrightarrow{\tau }}$ is defined in the same manner as l in Lemma 2.

Proof. 

If we take $M=s$ and $n_j=\lfloor {\displaystyle \frac{\overrightarrow{\tau }\left[j\right]-\overrightarrow{f}\left[j\right]}{\overrightarrow{\sigma }\left[j\right]}}\rfloor +1$, then the result follows immediately. □

Corollary 3 

(vectorization/linearization of the number of convolutional operations, or sum of product sop, between ${\mathcal{T}}_{\overrightarrow{\tau }}$ and all the kernels in a given layer). $l_{sop}:\overline{|{\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}\left({\mathcal{T}}_{\overrightarrow{\tau }}\right)|}\times \overline{N_1}\to \overline{|{\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}\left({\mathcal{T}}_{\overrightarrow{\tau }}\right)|\times N_1}$ is a bijective function if $l_{sop}$ is defined in the manner of l in Lemma 2, where $N_1$ is the number of filters (see Figure 4).

Corollary 4 

(vectorization/linearization of the indices of a kernel/filter). $l_F:{\prod }_{k=1}^s\overline{\overrightarrow{f}\left[k\right]}\to \overline{{\prod }_{k=1}^s\overrightarrow{f}\left[k\right]}$ is a bijective function if $l_F$ is defined in the same manner as l in Lemma 2.

Corollary 5. 

$l^{-1}\left(n\right)=(t_1\mathbf{mod}{n}_1,t_2\mathbf{mod}{n}_2,\dots ,t_{N-1}\mathbf{mod}{n}_{N-1},t_N\mathbf{mod}{n}_N)$, where, by applying the floor and ceiling functions,

• $t_N=\lceil {\displaystyle \frac{n}{{\prod }_{p=1}^{N-1}{n}_{pj}}}\rceil$;
• $t_{N-1}=\lceil {\displaystyle \frac{r_N}{{\prod }_{p=1}^{N-2}{n}_{pj}}}\rceil$, where $r_N=n-{\tilde{q}}_N·{\prod }_{p=1}^{N-1}{n}_{pj}$ and ${\tilde{q}}_N=\lfloor {\displaystyle \frac{n}{{\prod }_{p=1}^{N-1}{n}_{pj}}}\rfloor$;
• $t_{N-2}=\lceil {\displaystyle \frac{r_{N-1}}{{\prod }_{p=1}^{N-3}{n}_{pj}}}\rceil$, where $r_{N-1}=r_N-{\tilde{q}}_{N-1}·{\prod }_{p=1}^{N-2}{n}_{pj}$ and ${\tilde{q}}_{N-1}=\lfloor {\displaystyle \frac{r_N}{{\prod }_{p=1}^{N-2}{n}_{pj}}}\rfloor$. Repeat the whole process until
• $t_2=\lceil {\displaystyle \frac{r_3}{{\prod }_{p=1}^1{n}_{pj}}}\rceil$, where $r_3=r_4-{\tilde{q}}_3·{\prod }_{p=1}^2{n}_{pj}$ and ${\tilde{q}}_3=\lfloor {\displaystyle \frac{r_4}{{\prod }_{p=1}^2{n}_{pj}}}\rfloor$;
• $t_1=r_2$, where $r_2=r_3-{\tilde{q}}_2·n_{1j}$ and ${\tilde{q}}_2=\lfloor {\displaystyle \frac{r_3}{{\prod }_{p=1}^1{n}_{pj}}}\rfloor$.

Remark 5. 

Since $l_{\tau },l_{\tau },l_{sop}$ are defined over the sizes of the sections (or sectional vectors), any further expansion, such as zero-padding of T (the input object with sectional vector τ and zero-padding sectional vector ${\tau }^{\prime }$) will not alter its bijectivity, i.e., $l_{{\tau }^{\prime }},l_{{\tau }^{\prime }},l_{so{p}^{\prime }}$ are all bijective (this could be easily shown via Lemma 2).

Corollary 6. 

For the ith filter in the layer (stage) j, or $F^{ij}$, the filter could be linearized by $F^{ij}\left[n\right]$ for all $1\le n\le {\prod }_{p=1}^h{f}_{pj}$, i.e., in the jth layer (stage) related to the feature maps, there are $m_j·{\prod }_{p=1}^h{f}_{pj}$ linearized nodes whose values are decided by the presented results.

Proof. 

For each $1\le i\le m_j$, $F^{ij}\left[{\prod }_{p=1}^k{\overline{f}}_{pj}\right]=F^{ij}\left[l^{-1}\left({\prod }_{p=1}^k{f}_{pj}\right)\right]$, by Lemma 2, the result follows immediately. □

Example 2. 

Suppose T is a 6-by-7 matrix. Suppose the stride vector ${\overrightarrow{\sigma }}_j=[{\sigma }_{1j},{\sigma }_{2j}]=[1,2]$. $Sec\left(F^j\right)\equiv {\overrightarrow{f}}_j=[f_{1j},f_{2j}]=[2,3]$. Then, $h=2$, $Sec\left(T\right)\equiv \overrightarrow{\tau }=[{\tau }_1,{\tau }_2]=[6,7]$. Based on Claim 1, $Q(T/F^j)=[q_{1j},q_{2j}]=[\lfloor {\displaystyle \frac{{\tau }_1-f_{1j}}{{\sigma }_{1j}}}\rfloor +1,\lfloor {\displaystyle \frac{{\tau }_2-f_{2j}}{{\sigma }_{2j}}}\rfloor +1]=[5,3]$, $I(T/F^j)=\overline{5}\times \overline{3}=\{(1,1),(1,2),(1,3),(2,1),(2,2),(2,3),(3,1),(3,2),(3,3),(4,1),(4,2),(4,3),(5,1),(5,2),(5,3)\}$. Suppose $\overrightarrow{I}=[4,2]$. Based on Lemma 1, $D_{\overrightarrow{I}}=\underset{1+(4-1)·{\sigma }_{1j}}{\overset{f_{1j}}{\to }}\widehat{\otimes }\underset{1+(2-1)·{\sigma }_{2j}}{\overset{f_{2j}}{\to }}=\underset{4}{\overset{2}{\to }}\widehat{\otimes }\underset{3}{\overset{3}{\to }}=\left[\begin{array}{c}4\\ 5\end{array}\right]\widehat{\otimes }\left[\begin{array}{c}3\\ 4\\ 5\end{array}\right]=\left[\begin{array}{ccc}(4,3)& (4,4)& (4,5)\\ (5,3)& (5,4)& (5,5)\end{array}\right]$ and thus $T{\upharpoonright }_{D_{\overrightarrow{I}}}=\left[\begin{array}{ccc}T(4,4)& T(4,5)& T(4,6)\\ T(5,4)& T(5,5)& T(5,6)\end{array}\right]$. Hence, ${\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}$ could be represented by ${\mathcal{P}}_{Sec\left(F^j\right)}^T\equiv {\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}={\left[T\left(\underset{1+(i_1-1)·{\sigma }_{1j}}{\overset{2}{\to }}\widehat{\otimes }\underset{1+(i_2-1)·{\sigma }_{2j}}{\overset{3}{\to }}\right)\right]}_{(i_1,i_2)\in \overline{5}\times \overline{3}}={\left[T\left(\underset{i}{\overset{2}{\to }}\widehat{\otimes }\underset{2·i_2-1}{\overset{3}{\to }}\right)\right]}_{(i_1,i_2)\in \overline{5}\times \overline{3}}=\left[\begin{array}{ccc}T\left(\underset{1}{\overset{2}{\to }}\widehat{\otimes }\underset{1}{\overset{3}{\to }}\right)& T\left(\underset{1}{\overset{2}{\to }}\widehat{\otimes }\underset{3}{\overset{3}{\to }}\right)& T\left(\underset{1}{\overset{2}{\to }}\widehat{\otimes }\underset{5}{\overset{3}{\to }}\right)\\ T\left(\underset{2}{\overset{2}{\to }}\widehat{\otimes }\underset{1}{\overset{3}{\to }}\right)& T\left(\underset{2}{\overset{2}{\to }}\widehat{\otimes }\underset{3}{\overset{3}{\to }}\right)& T\left(\underset{2}{\overset{2}{\to }}\widehat{\otimes }\underset{5}{\overset{3}{\to }}\right)\\ T\left(\underset{3}{\overset{2}{\to }}\widehat{\otimes }\underset{1}{\overset{3}{\to }}\right)& T\left(\underset{3}{\overset{2}{\to }}\widehat{\otimes }\underset{3}{\overset{3}{\to }}\right)& T\left(\underset{3}{\overset{2}{\to }}\widehat{\otimes }\underset{5}{\overset{3}{\to }}\right)\\ T\left(\underset{4}{\overset{2}{\to }}\widehat{\otimes }\underset{1}{\overset{3}{\to }}\right)& T\left(\underset{4}{\overset{2}{\to }}\widehat{\otimes }\underset{3}{\overset{3}{\to }}\right)& T\left(\underset{5}{\overset{2}{\to }}\widehat{\otimes }\underset{5}{\overset{3}{\to }}\right)\\ T\left(\underset{5}{\overset{2}{\to }}\widehat{\otimes }\underset{1}{\overset{3}{\to }}\right)& T\left(\underset{5}{\overset{2}{\to }}\widehat{\otimes }\underset{3}{\overset{3}{\to }}\right)& T\left(\underset{5}{\overset{2}{\to }}\widehat{\otimes }\underset{5}{\overset{3}{\to }}\right)\end{array}\right]$.

Example 3. 

Suppose T is a 26-by-37-by-19-by-28 tensor. Suppose the stride vector ${\overrightarrow{\sigma }}_j=[{\sigma }_{1j},{\sigma }_{2j},{\sigma }_{3j},{\sigma }_{4j}]=[6,3,7,5]$. $Sec\left(F^j\right)=[f_{1j},f_{2j},f_{3j},f_{4j}]=[3,4,5,2]$. Then, $h=4$, $Sec\left(T\right)=[{\tau }_1,{\tau }_2,{\tau }_3,{\tau }_4]=[26,37,19,28]$. Based on Claim 1, $Q(T/F^j)=[q_{1j},q_{2j},q_{3j},q_{4j}]=[\lfloor {\displaystyle \frac{{\tau }_1-f_{1j}}{{\sigma }_{1j}}}\rfloor +1,\lfloor {\displaystyle \frac{{\tau }_2-f_{2j}}{{\sigma }_{2j}}}\rfloor +1,\lfloor {\displaystyle \frac{{\tau }_3-f_{3j}}{{\sigma }_{3j}}}\rfloor +1,\lfloor {\displaystyle \frac{{\tau }_4-f_{4j}}{{\sigma }_{4j}}}\rfloor +1]=[4,12,3,6]$, $Q(T/F^j)=\overline{4}\times \overline{12}\times \overline{3}\times \overline{6}$. Suppose $\overrightarrow{I}=[4,7,2,5]$. Based on Lemma 1, $D_{\overrightarrow{I}}=\underset{1+(i_1-1)·{\sigma }_{1j}}{\overset{f_{1j}}{\to }}\widehat{\otimes }\underset{1+(i_2-1)·{\sigma }_{2j}}{\overset{f_{2j}}{\to }}\widehat{\otimes }\underset{1+(i_3-1)·{\sigma }_{3j}}{\overset{f_{3j}}{\to }}\widehat{\otimes }\underset{1+(i_4-1)·{\sigma }_{4j}}{\overset{f_{4j}}{\to }}=\underset{19}{\overset{3}{\to }}\widehat{\otimes }\underset{19}{\overset{4}{\to }}\widehat{\otimes }\underset{8}{\overset{5}{\to }}\widehat{\otimes }\underset{21}{\overset{2}{\to }}=\left[\begin{array}{c}19\\ 20\\ 21\end{array}\right]\widehat{\otimes }\left[\begin{array}{c}19\\ 20\\ 21\\ 22\end{array}\right]\widehat{\otimes }\left[\begin{array}{c}8\\ 9\\ 10\\ 11\\ 12\end{array}\right]\widehat{\otimes }\left[\begin{array}{c}21\\ 22\end{array}\right]$ and thus $T{\upharpoonright }_{D_{\overrightarrow{I}}}=T\left(\underset{19}{\overset{3}{\to }}\widehat{\otimes }\underset{19}{\overset{4}{\to }}\widehat{\otimes }\underset{8}{\overset{5}{\to }}\widehat{\otimes }\underset{21}{\overset{2}{\to }}\right)$. Hence, ${\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}$ could be represented by ${\mathcal{P}}_{Sec\left(F^j\right)}^T\equiv {\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}={\left[T\left(\underset{6·i_1-5}{\overset{3}{\to }}\widehat{\otimes }\underset{3·i_2-2}{\overset{4}{\to }}\widehat{\otimes }\underset{7·i_3-6}{\overset{5}{\to }}\widehat{\otimes }\underset{5·i_4-4}{\overset{2}{\to }}\right)\right]}_{\overrightarrow{I}\in \overline{4}\times \overline{12}\times \overline{3}\times \overline{6}}$, where $\overrightarrow{I}=[i_1,i_2,i_3,i_4]$.

Definition 11. 

$T/F^{ij}={\left[\langle T{\upharpoonright }_{D_{\overrightarrow{I}}},F^{ij}\rangle \right]}_{\overrightarrow{I}\in {\overline{q}}_{1j}\times {\overline{q}}_{2j}\dots {\overline{q}}_{h,j}}$, or a tensor consisting of the standard inner products between input tensors $T{\upharpoonright }_{D_{\overrightarrow{I}}}$ and the feature tensor $F^j$ at stage (layer) j.

Now, each of these inner products is captured by the function sum of products, as we did for the fully connected network. With Definition 11 and Corollary 6, we could reach the conclusion that the total number of neurons in this layer is $\left({\prod }_{p=1}^h{q}_{pj}\right)·m_j$. Since we are going to look into the forward and backward propagations, we partition the indices of the tensors of the previous layer.

5. Theories and Methods

Claim 3. 

The number of neurons in the first layer is${\prod }_{n=1}^s\overrightarrow{\tau }\left[n\right]$under the linearization in Corollary 1, and the value associated with each neuronkis$T\left(l_{\tau }^{-1}\left(k\right)\right)$for all$1\le k\le {\prod }_{n=1}^s\overrightarrow{\tau }\left[n\right]$, where$l_{\tau }$is defined in Corollary 1.

Claim 4. 

The number of neurons in the second layer is$N_2=\left|{\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}\left({\mathcal{T}}_{\overrightarrow{\tau }}\right)\right|·N_1$, where $N_1$is the number of filters/kernels, i.e., the label set for the neurons in the second layer is${\left\{so{p}_j\right\}}_{j=1}^{N_2}={\left\{so{p}_j\right\}}_{j=1}^{K·N_1}$, where$K=|{\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}\left({\mathcal{T}}_{\overrightarrow{\tau }}\right)|$, where$so{p}_j$stands for ‘sum of product’ of thejth node in the referred layer (or in the standard CNN, it is thejth convoluted value).

Lemma 3. 

The indices from the first layer described in Claim 3 that links to the jth neuron in the second layer described in Claim 4 is the set $l_{\mathcal{T}}\left({\prod }_{k=1}^s\overline{(\overrightarrow{1}+(\overrightarrow{\gamma }-\overrightarrow{1})\odot \overrightarrow{\sigma })\left[k\right]:((\overrightarrow{\gamma }-\overrightarrow{1})\odot \overrightarrow{\sigma }+\overrightarrow{f})\left[k\right]}\right)$, where $\overrightarrow{\gamma }=l_{\overrightarrow{\tau }}^{-1}\left({\left(l_{sop}^{-1}\left(j\right)\right)}_1\right)$, and ${\left(l_{sop}^{-1}\left(j\right)\right)}_1=k$, if $l_{sop}^{-1}\left(j\right)=(k,n)$, and where $l_{\mathcal{T}}\left({\prod }_{k=1}^s\overline{(\overrightarrow{1}+(\overrightarrow{\gamma }-\overrightarrow{1})\odot \overrightarrow{\sigma })\left[k\right]:((\overrightarrow{\gamma }-\overrightarrow{1})\odot \overrightarrow{\sigma }+\overrightarrow{f})\left[k\right]}\right)\equiv \left\{l_{\mathcal{T}}\left(\overrightarrow{v}\right):\overrightarrow{v}\in {\prod }_{k=1}^s\overline{(\overrightarrow{1}+(\overrightarrow{\gamma }-\overrightarrow{1})\odot \overrightarrow{\sigma })\left[k\right]:((\overrightarrow{\gamma }-\overrightarrow{1})\odot \overrightarrow{\sigma }+\overrightarrow{f})\left[k\right]}\right\}$.

Proof. 

Given an index j in the second layer, based on Corollary 3: $l_{sop}:\overline{|{\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}\left({\mathcal{T}}_{\overrightarrow{\tau }}\right)|}\times \overline{N_1}\to \overline{|{\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}\left({\mathcal{T}}_{\overrightarrow{\tau }}\right)|\times N_1}$, one has $(k,n)=l_{sop}^{-1}\left(j\right)$, where $k\in |{\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}\left({\mathcal{T}}_{\overrightarrow{\tau }}\right)|$. Then, based on Corrollary 2, $l_{\overrightarrow{\tau }}:{\prod }_{j=1}^s\overline{\lfloor {\displaystyle \frac{\overrightarrow{\tau }\left[j\right]-\overrightarrow{f}\left[j\right]}{\overrightarrow{\sigma }\left[j\right]}}\rfloor +1}\to \overline{|{\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}\left({\mathcal{T}}_{\overrightarrow{\tau }}\right)|}$, one has $l_{\overrightarrow{\tau }}^{-1}\left(k\right)=\overrightarrow{\gamma }$, which, via Definition 4 and Corollary 1, links to the result. □

Theorem 1. 

$so{p}_j=〈{\mathcal{T}}_{\tau }\left(l_{sop}^{-1}{\left(j\right)}_1\right),F^{\left(l_{sop}^{-1}{\left(j\right)}_2\right),1}〉$, where $l_{sop}^{-1}{\left(j\right)}_1\equiv {\left(l_{sop}^{-1}\left(j\right)\right)}_1$ and $l_{sop}^{-1}{\left(j\right)}_2\equiv {\left(l_{sop}^{-1}\left(j\right)\right)}_2$.

Proof. 

Based on Corollary 3, the result follows immediately. □

Theorem 2. 

${\omega }_{ij}=\left\{\begin{array}{c}{F}_{\overrightarrow{f}}^{j_2,1}\left[l_{\overrightarrow{\tau }}^{-1}\left(i\right)\right],\mathrm{if}i\in l_{\mathcal{T}}\left({\prod }_{k=1}^s\overline{(\overrightarrow{1}+(\overrightarrow{\gamma }-\overrightarrow{1})\odot \overrightarrow{\sigma })\left[k\right]:((\overrightarrow{\gamma }-\overrightarrow{1})\odot \overrightarrow{\sigma }+\overrightarrow{f})\left[k\right]}\right),\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.\mathrm{where}\overrightarrow{\gamma }=l_{\overrightarrow{\tau }}^{-1}\left({\left(l_{sop}^{-1}\left(j\right)\right)}_1\right)$, for all $1\le i\le {\prod }_{k=1}^s\overrightarrow{\tau }\left[k\right]$ and $1\le j\le N_2\equiv \left|{\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}\left({\mathcal{T}}_{\overrightarrow{\tau }}\right)\right|\times N_1$; $j_2={\left(l_F^{-1}\left(j\right)\right)}_2$; $F_{\overrightarrow{f}}^{i,j}$ is defined in Definition 3.

Proof. 

Let us define a (weight) function $\omega :\overline{{\prod }_{k=1}^s\overrightarrow{\tau }\left[k\right]}\times \overline{{\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}\left({\mathcal{T}}_{\overrightarrow{\tau }}\right)|\times N_1}\to \mathbb{R}$. Before that, let us define some settings. Since there are $|\overrightarrow{q}|={\prod }_{k=1}^s\overrightarrow{q}\left[k\right]$ sub-tensors, or $|{\mathcal{P}}_{\overrightarrow{f}}^{\overrightarrow{\sigma }}|=|\overrightarrow{q}|$, and the indices for the $\overrightarrow{\gamma }$ th sub-tensor (the size of each sub-tensor is $|\overrightarrow{f}|$), based on the proof in Lemma 3, are ${\prod }_{k=1}^s\overline{(\overrightarrow{1}+(\overrightarrow{\gamma }-\overrightarrow{1})\odot \overrightarrow{\sigma })\left[k\right]:((\overrightarrow{\gamma }-\overrightarrow{1})\odot \overrightarrow{\sigma }+\overrightarrow{f})\left[k\right]}\subseteq {\prod }_{k=1}^s\overline{\overrightarrow{\tau }\left[k\right]}$. Now, we define $w_{ij}:=w(i,j)$. If one observes j, they will find j is decided by a pair of values, say $j=(j_1,j_2)$. Since i could link to all the nodes in the next layer (intermediate output layer, or SOP layer), the value of $\omega (i,j)$ would need to involve j as well. Then, we have to check whether i is activated (convoluted) with respect to j, or more precisely, i is located in the $\overrightarrow{\gamma }$ th sub-tensor, where $\overrightarrow{\gamma }=l_{\overrightarrow{\tau }}^{-1}\left({\left(l_{sop}^{-1}\left(j\right)\right)}_1\right)$. In summary, neuron i is activated (convoluted) if $i\in {\prod }_{k=1}^s\overline{(\overrightarrow{1}+(\overrightarrow{\gamma }-\overrightarrow{1})\odot \overrightarrow{\sigma })\left[k\right]:((\overrightarrow{\gamma }-\overrightarrow{1})\odot \overrightarrow{\sigma }+\overrightarrow{f})\left[k\right]}$ and its activated value is defined to be ${\omega }_{i,j}$, where i will indicate the ith linearized element in the $j_2$ filter/kernel and is decided by $l_{\overrightarrow{\tau }}^{-1}\left(i\right)$. Hence, $w_{ij}$ is defined as the statement. The whole argument could also refer to Figure 4 and Figure 5. □

Remark 6. 

${\left\{{\omega }_{ij}\right\}}_{1\le i\le {\prod }_{k=1}^s\overrightarrow{\tau }\left[k\right]}^{1\le j\le N_2}$ is a set of weights between an input layer, whose input object/tensor is ${\mathcal{T}}_{\overrightarrow{\tau }}$, with ${\prod }_{k=1}^s\overrightarrow{\tau }\left[k\right]$ neurons and an intermediate output layer, whose intermediate output object/values is $sop$ (sum of product), with $N_2$ neurons. ${\omega }_{ij}$ is the weight between the ith neuron in the input layer and the jth neuron in the intermediate output neuron.

Theorem 3. 

The CNN is represented by two networks: a partial network (or parameterized fully connected network) for feature extraction and a fully connected network.

Proof. 

The result follows immediately from Theorems 1 and 2 (see Figure 5). □

6. Illustrative Example

Since the main difference between the standard CNN (sCNN) and this linearized CNN (lCNN) lies in the encoding part, the calculation of the set of weights ${\omega }_{i,j}$ is vital. In this section, we exploit R programming (version 4.5.1) to demonstrate the equivalence between the sCNN and lCNN, i.e., an illustrative example for Theorems 2 and 3 (see https://github.com/raymingchen/linearized-high-dimension-CNN.git, accessed on 1 September 2025). Suppose the input image is a cute puppy (https://unsplash.com/photos/brown-short-coated-dog-on-green-grass-field-dgr0ZDbeOqw, a 3D RGB tensor, accessed on 1 September 2025). To simplify the computation, we extract the R part of the input tensor, or puppyR (a 200-by-200 matrix, or (1,2) cell at Figure 6) as our input tensor. Suppose we consider four kernels (the first two are randomly generated, while the last two are some deterministic features):

$F^{1,1}=$ $\left[\begin{array}{cccccc}40.01257& -71.03048& -90.265515& 64.15175& 0.6114631& -28.811208\\ 91.03395& 22.87645& 13.136873& -15.21936& 79.4672556& 9.257303\\ 92.50072& 90.29318& 9.735757& -86.47676& -41.9774183& -7.608247\\ -70.20737& -35.65345& 51.91615& 95.19638& 52.5290581& -84.474351\\ -32.68287& 40.95929& -28.365139& 44.18469& 83.1533521& -13.688504\\ 38.91981& -91.67589& 14.902744& -97.88255& 74.1104222& 95.176985\end{array}\right]$; $F^{2,1}=$ $\left[\begin{array}{cccc}1.125705& 7.66730341& -2.626473& -2.0697611\\ 3.793908& 5.55425965& 3.695838& -4.7750866\\ -5.391265& 1.76043044& -4.390247& -0.1502723\\ -2.170834& 0.03233433& -3.177388& -6.2366755\end{array}\right]$; $F^{3,1}=\left[\begin{array}{ccccc}1& 0& 0& 0& 0\\ 0& 1& 0& 0& 0\\ 0& 0& 1& 0& 0\\ 0& 0& 0& 1& 0\\ 0& 0& 0& 0& 1\end{array}\right]$ $F^{4,1}=\left[\begin{array}{cccc}1& 1& 1& 1\\ 0& 0& 0& 0\\ 0& 0& 0& 0\\ 1& 1& 1& 1\end{array}\right]$.

Then, we perform the standard CNN (or sCNN) and linearized CNN (or lCNN) for the convolution (this operation is sufficient to show whether the two representations are equivalent). The convoluted results based on sCNN and lCNN are shown in the 4th and 5th rows in Figure 6. The stride vector is fixed at $\overrightarrow{\sigma }=(3,5)$. Based on these, one finds that the number of power tensors (or sub-tensors) with respect to $F^{1,1},F^{2,1},F^{3,1},F^{4,1}$ are $65\ast 39=2535,66\ast 40=2640,66\ast 40=2640,66\ast 40=2640$, respectively. For the linearized CNN, there are $200\ast 200$ = 40,000 nodes in the first/input layer, and there are 2535 + 2640 + 2640 + 2640 = 10,455 nodes in the second layer (intermediate output, or sop layer). Then, we construct (based on Theorem 4) the weights linking layer one and layer two by a 40,000-by-10,455 weight matrix W. Then, the values associated with the nodes of the 2nd layer are computed by a 10,455-by-1 column vector $sop=W^t{P}_l$, where t denotes the transpose operator, and $P_l$ is the linearized 40,000-by-1 column vector of puppyR. In order to show the equivalence of sCNN and lCNN, we have to reshape $sop$ into four matrices: a 65-by-39 matrix $MA1$, and three 66-by-40 $MA2,MA3,MA4$ matrices. The feature map of sCNN and $sop$ of lCNN are identical in the convoluted operation. Since the operations (forward/backward propagation) are also identical regardless of whether it is an sCNN or lCNN, we have demonstrated the equivalence between the two representations.

7. Conclusions

We have reformulated the typical CNN architecture via two sets of networks: (1) a parameterized fully connected network and (2) an (unparameterized) fully connected network (see Figure A1 and Figure A2). This representation will directly link the relation between the CNN and ANN (see Figure 2). Indeed, the CNN could be regarded as a special case of a (parameterized) ANN. In addition, it also offers a very intuitive representation of the transforming process. The main contribution of this work is to incorporate the typical two stages of CNNs, feature extraction and ANN, into one framework. The technique part is the linearization and the high-dimension extension.

Limitations and Future Work

There are some limitations to this setting; for example, we have not conducted any other studies on some newly developed CNNs [16,17] or discussed stacked CNNs [18,19], dense networks [19], or other variants, even though the settings are similar. In the future, we will incorporate high-dimensional CNNs into high-dimensional ANNs and test their efficacy and efficiency if the architecture is unified. We could also study other variants, such as stacked neural networks [27,28,29]. Another drawback for this manuscript is that we do not provide an experiment to implement the framework devised in this article for the following reasons:

• This is purely a theoretical framework for incorporating and expanding existing theories and practices, and its efficacy is guaranteed by the minimized error via forward/backward propagation.
• Our models could be easily downgraded to 1D, 2D, and 3D data or CNN architectures, and those architectures are always proven to be efficient and faithful.
• For higher dimensions, for example, each input data point is a pair of pictures, and this could be represented by a set if two 3D products, or six dimensions. Therefore, theoretically, it is not different from the usual practice, but with significant computational resources—this is beyond our computational capacity. For future research, if a team is capable of implementing the whole algorithm, then it could implement such mechanisms.