Hands-On Fundamentals of 1D Convolutional Neural Networks—A Tutorial for Beginner Users

Abstract

In recent years, deep learning (DL) has garnered significant attention for its successful applications across various domains in solving complex problems. This interest has spurred the development of numerous neural network architectures, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Generative Adversarial Networks (GANs), and the more recently introduced Transformers. The choice of architecture depends on the data characteristics and the specific task at hand. In the 1D domain, one-dimensional CNNs (1D CNNs) are widely used, particularly for tasks involving the classification and recognition of 1D signals. While there are many applications of 1D CNNs in the literature, the technical details of their training are often not thoroughly explained, posing challenges for those developing new libraries in languages other than those supported by available open-source solutions. This paper offers a comprehensive, step-by-step tutorial on deriving feedforward and backpropagation equations for 1D CNNs, applicable to both regression and classification tasks. By linking neural networks with linear algebra, statistics, and optimization, this tutorial aims to clarify concepts related to 1D CNNs, making it a valuable resource for those interested in developing new libraries beyond existing ones.

1. Introduction

Recently, there has been a surge of interest in deep learning (DL), with numerous review papers on the topic published in recent years [1,2,3,4,5,6,7,8,9,10,11,12,13,14]. DL architectures, such as Deep Neural Networks (DNNs), Deep Belief Networks (DBNs), Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and Transformers, have found compelling applications across various fields. These include computer vision, speech recognition [15,16,17,18,19,20,21], natural language processing [22,23,24,25,26,27,28,29], bioinformatics [30,31,32,33,34,35,36,37], medical image analysis [38,39,40,41,42,43,44,45,46,47], climate science [48,49,50,51,52,53,54,55,56,57,58], and material inspection [59,60,61,62,63,64,65,66,67].

Focusing specifically on one-dimensional problems, CNN-based architectures are still the most widely used in the literature compared to other, more recent, and potentially more effective architectures for certain problems. This observation is based on research conducted using prominent databases and search engines (e.g., Scopus, IEEXplore, ScienceDirect, SpringerLink, Web of Science, ACM Digital Library, PubMed). Relevant results have emerged regarding the frequency with which a specific deep learning architecture, associated with one-dimensional problems, appears in paper titles published between 2018 and 2023. Although this research is not exhaustive, it highlights that discussions on 1D CNNs remain highly active in the literature.

However, within the realm of CNNs applications, it is important to note that those focusing on one-dimensional data are less frequent compared to those dealing with multidimensional data (2D and 3D). This discrepancy in research activity can be observed through a simple Google search. For example, a search for terms like “3D CNN”, “2D CNN”, or “1D CNN” reveals a significant drop in the number of results as the dimensionality of the data fed into the network decreases. Although searches include web pages and blog posts, which are generally not peer-reviewed, a similar trend is evident in the peer-reviewed literature as well. The relatively limited number of published papers on 1D CNNs, compared to those on 2D and 3D, suggests an under-explored area that could stimulate further research [68]. One possible outcome of this could be an increase in review papers specifically focusing on 1D CNNs, which are currently quite limited [69,70,71].

1D CNNs offer promising solutions across various sectors. They are particularly well-suited for real-time applications [72,73,74,75] due to their low computational requirements. In fact, continuous online monitoring generates a large quantity of data that needs to be processed and analyzed rapidly and accurately to efficiently detect specific patterns, assets or even diseases. This is the case of Raman spectra detection, in which the use of 1D CNNs has proven effective at identifying in real-time one pure unknown Raman instance from thousands of classes with higher accuracy over traditional multivariate analyses [72]. In the context of real-time applications for 5G communications, the increased bandwidth has posed a significant challenge in linearization of power amplifier over a wide bandwidth, which has been faced implementing 1D CNNs that have proven to be highly effective for real-time digital pre-distortion operation, surpassing traditional methods like Volterra series [73].

The use of 1D CNNs has also been proposed as a computationally efficient solution for fast and accurate diagnostics in the power industry, particularly for monitoring electromagnetic interference. This approach has enabled highly accurate automatic condition monitoring, significantly aiding in the identification of insulation and mechanical faults in high-voltage electrical power assets [74]. Another interesting example the real-time processing capability with significantly lower delay achieved by using 1D CNNs has been presented to classify cardiac acoustic signals for heart anomaly detection. This method proved to be robust and accurate also in reducing the false alarms [75].

The ability of 1D CNNs to learn models from 1D signals such as acoustic signals has also been well demonstrated. Specifically, 1D CNNs are increasingly being used to process raw audio waveforms for environment sound [76], music genre [77], and in general audio [78] classification, along with speech acoustic modeling [79], garnering growing attention in these areas.

Given that 1D CNNs can process any 1D signal, they are also applicable to physiological signals such as electroencephalograms (EEG) and electrocardiograms (ECG). Therefore, fields traditionally dominated by 2D CNNs, such as medicine and biology, could benefit from the application of 1D CNNs. A significant portion of the literature on classification problems involves time series signals as input data, with outputs typically predicting labels related to brain states for tasks such as identifying epileptic episodes [80,81], establishing communication paths between brain processes and external devices [82], and classifying sleep phases [83]. Additionally, there are notable examples of classification of various cardiac disease from ECGs [84,85,86,87], and or other conditions, including Parkinson’s disease [88], COVID-19 [89], and diabetes [90].

Moreover, 1D CNNs have likewise become popular for improving intrusion detection in cyber-physical systems [91,92,93,94,95,96,97,98,99,100,101,102], and, within industrial contexts, particularly for fault and anomaly detection in complex production environments [103,104].

Additionally, 1D CNNs have proven valuable for cognitive tasks, such as classifying emotional states, including speech emotion recognition. They have been shown to be effective for speaker-independent types of emotion recognition [105], offering increased accuracy while maintaining reasonable computational costs [106,107].

Noteworthy applications of 1D CNNs extend to the financial sector, where they can be used to identify line chart patterns from financial time series [108]. In spectroscopy, 1D CNNs have demonstrated intriguing applications, such as predicting Dry Matter Content (DMC) from infrared absorbance spectra of fruits—a crucial indicator of fruit maturity that impacts harvest timing—[109], and classifying corn seed viability [110].

Moreover, in the biomedical field, 1D CNNs have been effectively used to extract lifetime parameters from fluorescence lifetime imaging microscopy signals [111]. In agricultural environmental monitoring, 1D CNNs have been applied to classify soil texture using hyper-spectral data [112,113,114] and to predict various soil properties from visible and infrared spectra [115]. In geology, 1D CNNs have demonstrated their ability to link multivariate geochemical data with known mineral deposit locations, significantly enhancing the precision of exploration areas [116]. They also offer rapid seismic predictions, potentially addressing the limitations of traditional physics-informed systems [117], and can be used to classify seismic signals from different tectonic settings [118].

While this literature review is not exhaustive, it highlights that 1D CNNs continue to be widely utilized within the community. This raises the question of whether their prevalent use, despite the availability of potentially more efficient architectures, is due to a deep understanding of their fundamental mechanisms by researchers or the convenience of using readily available open-source software packages. Although many applications have already been developed and more are on the horizon, there appears to be a gap. Among common themes and recurring issues observed in the literature, we find that many published papers, even if innovative from an application standpoint and highly cited, often lack detailed descriptions of fundamental theoretical aspects. While the use of open-source packages is frequently mentioned, very little attention is generally paid to theoretical aspects such as which definition of convolution has been considered, which loss or activation functions have been used, and whether some kind of padding has been introduced. In those papers where a theoretical section exists, many technical aspects are only briefly outlined. In other words, the success of 1D CNNs partly stems from gaps in knowledge that do not impede progress as long as pre-existing software solutions are utilized. This is evidenced by the ongoing activity within the community on the topic.

However, the situation becomes more challenging if one needs to develop custom neural network libraries in languages other than those supported by available open-source solutions. Implementing a 1D CNN-based solution in new libraries written in different languages presents several potential challenges. A deep understanding of fundamental concepts such as convolutions, activation functions, pooling operations, and backpropagation is essential.

However, this knowledge alone is not enough to tackle the challenges that such a development requires. It is necessary to ensure that the new libraries are accompanied by thorough documentation that explains the theoretical underpinnings of 1D CNNs in detail. Additionally, educational resources, including tutorials, code examples, and step-by-step guides, are necessary to help users grasp the implementation process. Finally, community support is also vital for the successful adoption of these tools, including not only scholarly publications but also forums or user groups where developers can exchange ideas and seek assistance.

On a practical level, once the theoretical foundations are established, attention must be given to various other factors such as performance optimization, compatibility with existing systems, cross-platform support, and fostering user adoption to build a supportive community. Moreover, developing new libraries for 1D CNNs involves potential risks and challenges, including overfitting, generalization issues, data requirements, and computational complexity. While these concerns are important for the developer community, they are beyond the scope of this tutorial.

Although the general theory of CNNs is well-covered in books and papers (especially for 2D CNNs), in the few that specifically refer to 1D CNNs [69,119,120,121,122], there are still theoretical elements that are not sufficiently outlined but that deserve further explanation. This tutorial aims to bridge what is still missing, providing a step-by-step guide that includes all the necessary mathematical expressions for using 1D CNNs. It starts with a clear definition of convolution, which can sometimes be confusing, and progresses through the feedforward and backpropagation equations. The scope is to present these expressions explicitly, helping readers to become more familiar with the numerous neural network libraries available and, over time, aiding in the development of new libraries in languages other than those currently available.

The tutorial covers each aspect of feedforward and backpropagation in 1D CNNs. It particularly addresses limitations noted in our literature review, such as the limited discussion of activation functions (typically only hyperbolic tangent or sigmoid functions are covered) and loss functions (which are often chosen for regression rather than classification tasks). To tackle these issues, we draw on concepts from linear algebra, statistics, and optimization, as schematically depicted in Figure 1.

The tutorial is structured as follows. In Section 2, discrete convolution is defined with an emphasis on its distinction from cross-correlation. In Section 3, the network architecture is described in terms of building blocks. The training process is discussed in Section 4 for the feedforward propagation, and in Section 5 for the backpropagation. Section 6 explores other additional activation functions, such as hyperbolic tangent (tanh), Rectified Linear Unit (ReLU) and logarithmic hyperbolic cosine ($logcosh$), as well as other loss functions for both regression and classification tasks. In Section 7, the conclusions are presented, and Appendix A provides a review of the derivatives used in Section 5 (backpropagation section).

2. Convolution

In general, discrete convolution involves sliding a filter (K) over an input signal (I). The simplest case is when the filter slides over each position of the input.

Assuming that the filter iterates one element at a time (i.e., stride $=1$), this operation can be performed according to the following definitions:

$$\left(I\ast K\right)\left(i\right)=\sum _{u=1}^{s}I(i+u-1)K\left(u\right)$$

(1)

$$\left(I\ast K\right)\left(i\right)=\sum _{u=1}^{s}I(i-u-1)K\left(u\right)$$

(2)

where I is the input of length n and K the filter (kernel) length s.

Equation (1) is known as non-causal convolution and, in the context of digital signal processing, as cross-correlation; (2) is known as causal convolution. According to definition (1), cross-correlation means sliding a filter across an input signal, while convolution (2) means sliding a flipped filter across the input signal [123].

In modern CNN packages (such as PyTorch [124]), the operation commonly referred to as convolution is actually cross-correlation (1).

However, this misnomer does not impact the performance of the network since the network has the ability to adapt accordingly. This is true as long as ready-made packages are used to implement a CNN. If not, to avoid confusion, it is worth specifying which convolution definition is implemented. Here, 1D CNNs are explored using the convolution definition (1), and in the following, we refer to definition (1) as convolution.

Moreover, it should be noted that the convolution is not defined correctly near the boundaries of the input signal.

This can be overcome if the convolution starts at an inner entry of the input signal such that the filter window is completely within the signal. The drawback of this procedure is that the resulting signal is smaller than the original one (Figure 2a). When it is mandatory to maintain the same size, it is possible to artificially add ghost entries to the input signal at the boundaries (padding). In this work, no padding has been considered.

Let us consider an input vector I (size n) and a kernel $K_{1,1}$ (size s). Their convolution is represented by a new vector C (size $n-s+1$), as schematically shown in Figure 2a. It can be computed as follows:

$$C\left(i\right)=\sum _{u=1}^{s}I(i+u-1)K_{1,1}\left(u\right).$$

(3)

The convolution (3) can be extended to the case of a collection of p filters with size s. The result of this multi-filter convolution is represented by a 2D matrix with size $(n-s+1)\times p$, as schematically represented in Figure 2b, where each column $C_j\left(i\right)$ is

$$C_j\left(i\right)=\sum _{u=1}^{s}I(i+u-1)K_{1,j}\left(u\right)$$

(4)

and p represents the number of filters.

Moreover, the convolution definition can be further extended to the case of an input array of p columns, as schematically depicted in Figure 3.

In this case, we can perform the convolution separately on each of the p columns $I_j$ using a multi-filter represented by a third-order tensor (size $s\times p\times q$). A 2D matrix (size $(n-s+1)\times q$) results from a convolution operation. This 2D matrix can be regarded as an array of q columns (Figure 3), in which each column ($C_l\left(i\right)$) is obtained by summing up p convolutions:

$$C_l\left(i\right)=\sum _{j=1}^p{I}_j\ast K_{j,l}\left(u\right).$$

(5)

where $l=1,2,\dots q$.

3. 1D CNN Architecture

CNNs typically perform a series of operations to extract features from an input signal.

For simplicity, this analysis focuses on a model with two convolutional layers $(C{L}_1,C{L}_2)$ followed by a fully connected layer $\left(FCL\right)$, as indicated in the “Layer” column of

Table 1. Description of the considered CNN. The “Layer” column describes the types of layers used sequentially, the “Operation” column indicates the type of operation performed at each layer, the “Filter” column describes the number and size of kernels used in each layer, and the “Output size” specifies the size of the signal obtained at the end of each layer. The illustration of this 1D CNN is schematized as an image in the Supplementary Materials (Figure S1).

Layer	Operation	Filter		Output Size
		N.	Size
$C{L}_1$	Conv + Act	$p=4$	$s^{\left(1\right)}=3$	$C_{\sigma }^{\left(1\right)}=16\times 4$
	Pool	1	$S{S}^{\left(1\right)}=2$	$S^{\left(1\right)}=8\times 4$
$C{L}_2$	Conv + Act	$q=6$	$s^{\left(2\right)}=3$	$C_{\sigma }^{\left(2\right)}=6\times 6$
	Pool	1	$S{S}^{\left(2\right)}=2$	$S^{\left(2\right)}=3\times 6$
$FCL$	Flat	–	–	$f=18\times 1$
	Fully connected + Act	–	–	$2\times 1$

.

In turn, each convolution layer comprises convolution (Conv), activation (Act), and pooling (Pool) operations, as detailed in the “Operation” column of Table 1. The general features of the CNN architecture in terms of sizes are summarised in the “Filter” and “Output size” columns of Table 1. In $C{L}_1$, the convolution is performed using a single-channel input (I) with dimension $n=18$ and $p=4$ filters with dimension $s^{\left(1\right)}=3$. A bias array is then added to the output, and the sigmoid activation function is applied. A new array ($C_{\sigma }^{\left(1\right)}$) of p columns with $n-s^{\left(1\right)}+1=16$ dimensions is obtained. It is then down-sampled (using max or average-pooling) with a factor $S{S}^{\left(1\right)}=2$, resulting in a new array ($S^{\left(1\right)}$) of p columns with size 8.

In $C{L}_2$, the convolution operation is performed using $S^{\left(1\right)}$ as the input signal and $p\times q$ filters with $q=6$ and $s^{\left(2\right)}=3$ as the dimension of each filter.

Similar to $C{L}_1$, a bias array is added here, and an activation function $\left(\sigma \right)$ is applied. This results in an array ($C_{\sigma }^{\left(2\right)}$) of q columns of length 6. It is down-sampled (using max or average-pooling) by a factor of $S{S}^{\left(2\right)}=2$. The resulting $S^{\left(2\right)}$ is then represented by an array of q columns of size 3.

Finally, the output of $C{L}_2$ is flattened, obtaining a new vector $\left(f\right)$ with dimension 18. This latter vector represents the input for the FCL. The output of the 1D CNN is represented by an array with two dimensions. This implies that a weight matrix $2\times 18$ is required in the FCL.

4. Feed Forward Propagation

In $C{L}_1$, the convolution is performed using the input signal $\left(I\right)$ and a multi-filter $\left(K_{1,j}^{\left(1\right)}\right)$, resulting in an array containing p elements:

$$C^{\left(1\right)}=\left(C_1^{\left(1\right)},C_2^{\left(1\right)},\dots ,C_p^{\left(1\right)}\right).$$

(6)

Each element in (6) is then represented by

$$C_j^{\left(1\right)}\left(i\right)=I\left(i\right)\ast K_{1,j}^{\left(1\right)}\left(u\right)=\sum _{u=1}^{s^{\left(1\right)}}I(i+u-1)K_{1,j}^{\left(1\right)}\left(u\right)+b_j^{\left(1\right)}$$

(7)

where the superscript refers to the convolutional layer number, j runs from 1 to the number of filters employed in the layer ($p=4$), u runs from 1 to the filter size $s^{\left(1\right)}$, and $b_j^{\left(1\right)}$ represents the bias. In this specific case, as indicated in Table 1, $p=4$, $s^{\left(1\right)}=3$, and i ranges from 1 to 16.

An activation function $\left(\sigma \right)$ is applied to each element of $C^{\left(1\right)}$ and the resulting output $C_{\sigma }^{\left(1\right)}$ is then pooled.

The primary objective of a pooling operation is to reduce the spatial dimensions of the input for the subsequent convolutional layer. Similar to the convolution operation, pooling uses a sliding window or a specific region that moves in stride across the input, transforming the values into representative ones. This transformation is commonly achieved by taking either the average or the maximum value from the input values within the window. The former is called “average pooling”, while the latter is called “max pooling”.

After pooling, a new array $S^{\left(1\right)}$ with p columns is obtained. Each column $S_j^{\left(1\right)}$ is then represented by

$$\left\{\begin{array}{cc}{S}_j^{\left(1\right)}\left(i\right)=Max{\left\{C_{j,\sigma }^{\left(1\right)}\left(i_s\right)\right\}}_{W=1,2,\dots S{S}^{\left(1\right)}} \hfill & \mathrm{Max}\hfill \\ S_j^{\left(1\right)}\left(i\right)={\displaystyle \frac{1}{S{S}^{\left(1\right)}}}\sum _{W=1}^{S{S}^{\left(1\right)}}{C}_{j,\sigma }^{\left(1\right)}\left(i_s\right) \hfill & \mathrm{Average}\hfill \end{array}\right.$$

(8)

where $i_s=S{S}^{\left(1\right)}i-W+1$. In this specific case, $i_s$ runs from 1 to 16, and with $S{S}^{\left(1\right)}=2$, i runs from 1 to 8. For the sake of notation convenience, henceforth, i will represent the index $i_s$.

In $C{L}_2$, the convolution is performed using $S^{\left(1\right)}$ as the input signal and a multi-filter represented by a third-order tensor. Since this type of tensor refers to a three-dimensional array of numbers, for clarity, the representation of its 2D components is provided in Figure 4.

An array of q columns is then obtained, and each column is represented by

$$C_l^{\left(2\right)}=\sum _{j=1}^p{S}_j^{\left(1\right)}\ast K_{j,l}^{\left(2\right)}+b_l^{\left(2\right)}$$

(9)

In this specific case, $p=4$, l runs from 1 to $q=6$, and i from 1 to $8-3+1=6$.

An activation function is then applied to each element in $C^{\left(2\right)}$, and the resulting output $C_{\sigma }^{\left(2\right)}$ is then pooled. A new array $S^{\left(2\right)}$ with q columns is obtained, in which each column $S_j^{\left(2\right)}$ is then represented by

$$\left\{\begin{array}{cc}{S}_j^{\left(2\right)}\left(i\right)=Max{\left\{C_{j,\sigma }^{\left(2\right)}\left(i_s\right)\right\}}_{w=1,2,\dots S{S}^{\left(2\right)}} \hfill & \mathrm{Max}\hfill \\ \mathrm{or}\hfill \\ S_j^{\left(2\right)}\left(i\right)={\displaystyle \frac{1}{S{S}^{\left(2\right)}}}\sum _{w=1}^{S{S}^{\left(2\right)}}{C}_{j,\sigma }^{\left(2\right)}\left(i_s\right) \hfill & \mathrm{Average}\hfill \end{array}\right.$$

(10)

where $i_s=S{S}^{\left(2\right)}i-w+1$. In $C{L}_2$, $i_s$ runs from 1 to 6, and with $S{S}^{\left(2\right)}=2$i runs from 1 to 3. Again, for convenience of notation, in the following, i will denote the index $i_s$.

In the FCL, the first operation to be performed is represented by a flattening. It involves the 2D output generated at the end of the pooling step in $C{L}_2$ ($S^{\left(2\right)}$) and transforms it into a 1D vector. Generally, this operation can be performed by reshaping in row-major or column-major order. In the case under consideration, $S^{\left(2\right)}$ is a $3\times 6$ matrix. It can be reshaped to form a new 1D vector as follows:

$$\left\{\begin{array}{cc}f(i+m(j-1))=S_j^{\left(2\right)}\left(i\right) \hfill & \text{column-major}\hfill \\ f(j+n(i-1))=S_j^{\left(2\right)}\left(i\right) \hfill & \text{row-major}\hfill \end{array}\right.$$

(11)

where the row index $\left(i\right)$ of $S^{\left(2\right)}$ runs from 1 to $m=3$, and the column index j from 1 to $n=6$. In this specific case, the new vector $\left(f\right)$ has $m\times n=18$.

Once the vector f has been obtained, it can be fed to the fully connected network. Then, each element of the output vector $y_{out}=\sigma (W\times f+b)$ can be written as

$$\begin{array}{cc}\hfill y_{out}\left(i\right)& =\sigma \left(\sum _{j=1}^{18}W(i,j)f\left(j\right)+b\left(i\right)\right)\hfill \end{array}$$

(12)

where $\sigma$ is an activation function $\left(\sigma \right)$, W is the weight matrix $2\times 18$, and b a bias vector $2\times 1$.

5. Backpropagation

The training process of the network comprises both feedforward and backpropagation, as schematically depicted in Figure 5.

Backpropagation aims to minimize a loss function by updating the network’s parameters (e.g., weights and biases) so that the network can correctly map arbitrary inputs to outputs. The choice of loss function depends on the specific task at hand, as different tasks require different loss functions to effectively measure and optimize model performance.

Our literature review indicates that regression and classification are among the most commonly used tasks. Typically, a regression task aims to predict the value of a continuous variable, such as a price or probability. For instance, a 1D CNN can be used in regression tasks to predict the concentration of a specific compound in a sample based on its absorption spectrum. In this example, the input 1D signals are represented by absorbance values recorded at various wavelengths (spectra). Additionally, the same spectral data can be employed for classification tasks by making use of different loss function. In this case, the goal is to categorize different chemical compounds into predefined categories.

While the underlying architecture of 1D CNNs for both regression and classification tasks involves convolutional layers to extract features from sequential data, the key differences lie in the output layer configuration, the choice of activation functions, the loss functions used for training, and the evaluation metrics. For regression, the output is a continuous value and the model is trained to minimize the error between predicted and actual values. For classification, the output is categorical, and the model is trained to correctly classify inputs into one of several classes.

Backpropagation is the process used in training neural networks, where the error (or loss) is propagated backward through the layers of the network, starting from the output layer and moving toward the input layer. Its main purpose is to calculate the gradients (partial derivatives) of the loss function with respect to each parameter (weight) in the network. These gradients indicate how much each parameter needs to change to reduce the loss. The derivatives can be efficiently computed using the chain rule and can be used in the gradient descent algorithm to update the network’s parameters in a way that minimizes the loss function.

In this section, the gradients of each parameter are computed from end to start (e.g., from FCL to $C{L}_1$). In regression problems, the loss function (L) is computed by comparing the actual target $y_t$ with the predicted output values $y_{out}$. One of the most popular loss functions is represented by the mean squared error (MSE):

$$L={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(y_{out}\left(i\right)-y_t\left(i\right)\right)}^2$$

(13)

where i runs from 1 to the dimension of the output, in this case $N=2$.

Let us begin with the FCL. In this case, the parameters to be updated are the weight matrix W and the bias b. The first gradient to be calculated is $\Delta W$, with each entry represented by

$$\begin{array}{cc}\hfill \Delta W(i,j)& ={\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }W(i,j)}}\hfill \\ \hfill & ={\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }{y}_{out}\left(i\right)}}·{\displaystyle \frac{\mathit{\partial }{y}_{out}\left(i\right)}{\mathit{\partial }W(i,j)}}\hfill \\ \hfill & =(y_{out}\left(i\right)-y_t\left(i\right))·{\displaystyle \frac{\mathit{\partial }}{\mathit{\partial }W(i,j)}}\sigma \left(\sum _{j=1}^{18}W(i,j)f\left(j\right)+b\left(i\right)\right)\hfill \end{array}$$

(14)

where $“·”$ refers to the element-wise product. The last derivative in Equation (14) can be calculated once an activation function is provided. Discussing the advantages and disadvantages of using a specific activation function over another is beyond the scope of this work. For this purpose, we refer the curiosity of the reader to the existing literature on this topic [7,125,126,127,128,129].

In this output layer, the sigmoid activation function has been considered [130], while other functions will be discussed in Appendix A.

Defining $\Delta y_{out}$ as a vector $2\times 1$ vector

$$\Delta y_{out}\left(i\right)=(y_{out}\left(i\right)-y_t\left(i\right))·y_{out}\left(i\right)·(1-y_{out}\left(i\right))$$

(15)

the gradient $\Delta W$ can be rewritten as

$$\Delta W=\Delta y_{out}\times f^T.$$

(16)

where $“\times ”$ refers to the matrix product, and $f^T$ represents the transpose of the vector f, which is a $1\times 18$ vector.

The same calculation can be applied for the bias gradient:

$$\begin{array}{cc}\hfill \Delta b\left(i\right)& ={\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }b\left(i\right)}}\hfill \\ \hfill & ={\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }{y}_{out}\left(i\right)}}·{\displaystyle \frac{\mathit{\partial }{y}_{out}\left(i\right)}{\mathit{\partial }b\left(i\right)}}\hfill \\ \hfill & =(y_{out}\left(i\right)-y_t\left(i\right))·{\displaystyle \frac{\mathit{\partial }}{\mathit{\partial }b\left(i\right)}}\sigma \left(\sum _{j=1}^{18}W(i,j)f\left(j\right)+b\left(i\right)\right)\hfill \\ \hfill & =(y_{out}\left(i\right)-y_t\left(i\right))·y_{out}\left(i\right)·(1-y_{out}\left(i\right))\hfill \\ \hfill & =\Delta y_{out}\left(i\right).\hfill \end{array}$$

(17)

Before updating the parameters in $C{L}_2$, it is necessary to calculate the gradient of the vector f. Each entry of $\Delta f$ is then represented by

$$\begin{array}{cc}\hfill \Delta f\left(j\right)& ={\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }f\left(j\right)}}\hfill \\ \hfill & =\sum _{i=1}^2{\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }{y}_{out}\left(i\right)}}·{\displaystyle \frac{\mathit{\partial }{y}_{out}\left(i\right)}{\mathit{\partial }f\left(j\right)}}\hfill \\ \hfill & =\sum _{i=1}^2(y_{out}\left(i\right)-y_t\left(i\right))·y_{out}\left(i\right)·(1-y_{out}\left(i\right))·W(i,j)\hfill \\ \hfill & =\sum _{i=1}^2\Delta y_{out}\left(i\right)W(i,j)\hfill \end{array}$$

(18)

and $\Delta f$ can now be written as

$$\Delta f=W^T\times \Delta y_{out}.$$

(19)

During the feedforward step, the vector f results from $S^{\left(2\right)}$ reshaping. Here, in the backpropagation step, the gradient $\Delta f$ is reshaped back to obtain $\Delta S^{\left(2\right)}$, as schematically depicted in Figure 6.

In the $C{L}_2$, an un-pooling operation should be performed to up-sample the compressed data $\Delta S^{\left(2\right)}$ and obtain $\Delta C_{\sigma }^{\left(2\right)}$. To avoid confusion, great attention is now needed. If average up-sampling was used in feedforward, the backpropagation error is multiplied by $1/S{S}^{\left(2\right)}$ and uniformly assigned to all the elements in the pooling block (Figure 7).

In this case, each column of $\Delta C^{\left(2\right)}$ can be written as

$$\Delta C_{j\sigma }^{\left(2\right)}\left(i_u\right)={\displaystyle \frac{1}{S{S}^{\left(2\right)}}}\Delta S_j^{\left(2\right)}\left(\lceil i_u/S{S}^{\left(2\right)}\rceil \right)$$

(20)

where the index i of $\Delta S_j^{\left(2\right)}$ runs from 1 to $n=3$, $i_u$ runs from 1 to $6=S{S}^{\left(2\right)}n$, and finally $\lceil x\rceil$ represents the ceiling function. As in previous instances, in the following, i will denote the index $i_u$.

On the other hand, if max-pooling was performed during the feedforward step, the error is assigned to the location it originated from, and zero is assigned to all the other elements. This implies that during the feedforward step, the index of the “winning unit” should be stored and reused in the backpropagation step. This aspect may not be evident when using pre-packaged solutions but should be taken into account when developing new CNN libraries.

Once $\Delta C_{\sigma }^{\left(2\right)}$ has been computed, it is possible to update the filter value in $C{L}_2$. This means calculating each element of the third-order tensor $\Delta K^{\left(2\right)}$, with $j=1,\dots p=4$, $l=1,\dots q=6$, and $u=1,\dots 3$, we obtain

$$\begin{array}{cc}\hfill \Delta K_{j,l}^{\left(2\right)}\left(u\right)& ={\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }{K}_{j,l}^{\left(2\right)}\left(u\right)}}\hfill \\ \hfill & =\sum _{i=1}^6{\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }{C}_{l,\sigma }^{\left(2\right)}\left(i\right)}}·{\displaystyle \frac{\mathit{\partial }{C}_{l,\sigma }^{\left(2\right)}\left(i\right)}{\mathit{\partial }{K}_{j,l}^{\left(2\right)}\left(u\right)}}\hfill \\ \hfill & =\sum _{i=1}^6\Delta C_{l,\sigma }^{\left(2\right)}\left(i\right)·{\displaystyle \frac{\mathit{\partial }\sigma \left(C_l^{\left(2\right)}\left(i\right)\right)}{\mathit{\partial }{K}_{j,l}^{\left(2\right)}\left(u\right)}}\hfill \\ \hfill & =\sum _{i=1}^6\Delta C_{l,\sigma }^{\left(2\right)}\left(i\right)·{\displaystyle \frac{\mathit{\partial }\sigma \left(C_l^{\left(2\right)}\left(i\right)\right)}{\mathit{\partial }{C}_l^{\left(2\right)}\left(i\right)}}·{\displaystyle \frac{\mathit{\partial }{C}_l^{\left(2\right)}\left(i\right)}{\mathit{\partial }{K}_{j,l}^{\left(2\right)}\left(u\right)}}\hfill \\ \hfill & =\sum _{i=1}^6\Delta C_{l,\sigma }^{\left(2\right)}\left(i\right)·C_{l,\sigma }^{\left(2\right)}\left(i\right)·(1-C_{l,\sigma }^{\left(2\right)}\left(i\right))·{\displaystyle \frac{\mathit{\partial }{C}_l^{\left(2\right)}\left(i\right)}{\mathit{\partial }{K}_{j,l}^{\left(2\right)}\left(u\right)}}\hfill \\ \hfill & =\sum _{i=1}^6\Delta C_l^{\left(2\right)}\left(i\right)·{\displaystyle \frac{\mathit{\partial }{C}_l^{\left(2\right)}\left(i\right)}{\mathit{\partial }{K}_{j,l}^{\left(2\right)}\left(u\right)}}\hfill \end{array}$$

(21)

where $\Delta C_l^{\left(2\right)}=\Delta C_{l,\sigma }^{\left(2\right)}·C_{l,\sigma }^{\left(2\right)}·(1-C_{l,\sigma }^{\left(2\right)})$ has been obtained, considering sigmoid as the activation function ($\sigma$). In these hidden layers, other activation functions may be used, these cases will be discussed in the Appendix A.

Recalling Equations (3) and (9), the last derivative in (21) can be written as

$$\begin{array}{cc}\hfill {\displaystyle \frac{\mathit{\partial }{C}_l^{\left(2\right)}}{\mathit{\partial }{K}_{j,l}^{\left(2\right)}}}& ={\displaystyle \frac{\mathit{\partial }}{\mathit{\partial }{K}_{j,l}^{\left(2\right)}}}\left(\sum _{j=1}^{n=4}{S}_j^{\left(1\right)}\ast K_{j,l}^{\left(2\right)}+b_l^{\left(2\right)}\right)\hfill \\ \hfill & ={\displaystyle \frac{\mathit{\partial }}{\mathit{\partial }{K}_{j,l}^{\left(2\right)}}}\left(S_1^{\left(1\right)}\ast K_{1,l}^{\left(2\right)}+S_2^{\left(1\right)}\ast K_{2,l}^{\left(2\right)}+\cdots +S_n^{\left(1\right)}\ast K_{n,l}^{\left(2\right)}+b_l^{\left(2\right)}\right)\hfill \\ \hfill & ={\displaystyle \frac{\mathit{\partial }}{\mathit{\partial }{K}_{j,l}^{\left(2\right)}}}\left(S_j^{\left(1\right)}\ast K_{j,l}^{\left(2\right)}\right)\hfill \end{array}$$

(22)

and since only $S_j^{\left(1\right)}·K_{j,l}^{\left(2\right)}$ is relevant for the computation, we obtain

$$\begin{array}{c}\hfill {\displaystyle \frac{\mathit{\partial }{C}_l^{\left(2\right)}\left(i\right)}{\mathit{\partial }{K}_{j,l}^{\left(2\right)}\left(u\right)}}=S_j^{\left(1\right)}(i+u-1).\end{array}$$

(23)

Each element of the error $\Delta K_{j,l}^{\left(2\right)}$ can now be written as

$$\Delta K_{j,l}^{\left(2\right)}\left(u\right)=\sum _{i=1}^6\Delta C_l^{\left(2\right)}\left(i\right)S_j^{\left(1\right)}(i+u-1)$$

(24)

and then the error is

$$\Delta K_{j,l}^{\left(2\right)}=S_j^{\left(1\right)}\ast \Delta C_l^{\left(2\right)}$$

(25)

where $“\ast ”$ represents the convolution operation. In the present case, $S_j^{\left(1\right)}$ and $\Delta C_l^{\left(2\right)}$ are $8\times 1$ and $6\times 1$ 1D columns, respectively. The convolution in Equation (23) yields a $3\times 1$ vector, corresponding to the filter dimension selected in $C{L}_2$$\left(s^{\left(2\right)}\right)$.

Now, great attention should be paid when using Equation (25). In the literature, there are many examples of similar formulae in which the vector $S_j^{\left(1\right)}$ is used in reverse [120], without providing details. The reversing operation is required only when the convolution definition in (2) is used.

Following a similar procedure, it is possible to compute the bias $b_l^{\left(2\right)}$ error. Since $C_l^{\left(2\right)}$ is linear with the bias, ${\displaystyle \frac{\mathit{\partial }{C}_l^{\left(2\right)}}{\mathit{\partial }{b}_l^{\left(2\right)}}}=1$, then the bias error is computed as follows:

$$\begin{array}{cc}\hfill \Delta b_l^{\left(2\right)}& ={\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }{b}_l^{\left(2\right)}}}\hfill \\ \hfill & =\sum _{i=1}^6{\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }{C}_{l,\sigma }^{\left(2\right)}\left(i\right)}}·{\displaystyle \frac{\mathit{\partial }{C}_{l,\sigma }^{\left(2\right)}\left(i\right)}{\mathit{\partial }{b}_l^{\left(2\right)}}}\hfill \\ \hfill & =\sum _{i=1}^6\Delta C_{l,\sigma }^{\left(2\right)}\left(i\right)·{\displaystyle \frac{\mathit{\partial }\sigma \left(C_l^{\left(2\right)}\left(i\right)\right)}{\mathit{\partial }{b}_l^{\left(2\right)}}}\hfill \\ \hfill & =\sum _{i=1}^6\Delta C_{l,\sigma }^{\left(2\right)}\left(i\right)·{\displaystyle \frac{\mathit{\partial }\sigma \left(C_l^{\left(2\right)}\left(i\right)\right)}{\mathit{\partial }{C}_l^{\left(2\right)}\left(i\right)}}·{\displaystyle \frac{\mathit{\partial }{C}_l^{\left(2\right)}\left(i\right)}{\mathit{\partial }{b}_l^{\left(2\right)}}}\hfill \\ \hfill & =\sum _{i=1}^6\Delta C_{l,\sigma }^{\left(2\right)}\left(i\right)·C_{l,\sigma }^{\left(2\right)}\left(i\right)(1-C_{l,\sigma }^{\left(2\right)}\left(i\right))\hfill \\ \hfill & =\sum _{i=1}^6\Delta C_l^{\left(2\right)}\left(i\right).\hfill \end{array}$$

(26)

Let us now enter the $C{L}_1$. The object here is to update filters $K_{1,j}^{\left(1\right)}$ and bias $b_j^{\left(1\right)}$. To achieve this, the error $\Delta S_j^{\left(1\right)}$ must be calculated first.

Using Equations (1) and (9), $C_l^{\left(2\right)}\left(i\right)$ can be expressed as a function of $S_j^{\left(2\right)}(i+u-1)$. Introducing a new index $I=i+u-1$, we find that $C_l^{\left(2\right)}(I-u+1)$ can be represented as a function of $S_j^{\left(2\right)}\left(I\right)$. Moreover, for simplicity, if we denote $I=i$, $C_l^{\left(2\right)}(i-u+1)$ can be written as a function of $S_j^{\left(2\right)}\left(i\right)$. This result is utilized when computing each element of the error $\Delta S_j^{\left(1\right)}$ as follows:

$$\begin{array}{cc}\hfill \Delta S_j^{\left(1\right)}\left(i\right)& ={\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }{S}_j^{\left(1\right)}\left(i\right)}}\hfill \\ \hfill & =\sum _{l=1}^{q=6}\sum _{u=1}^{s^{\left(2\right)}}{\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }{C}_l^{\left(2\right)}(i-u+1)}}·{\displaystyle \frac{\mathit{\partial }{C}_l^{\left(2\right)}(i-u+1)}{\mathit{\partial }{S}_j^{\left(1\right)}\left(i\right)}}\hfill \\ \hfill & =\sum _{l=1}^{q=6}\sum _{u=1}^{s^{\left(2\right)}}\Delta C_l^{\left(2\right)}(i-u+1)·{\displaystyle \frac{\mathit{\partial }{C}_l^{\left(2\right)}(i-u+1)}{\mathit{\partial }{S}_j^{\left(1\right)}\left(i\right)}}\hfill \end{array}$$

(27)

where $s^{\left(2\right)}=3$ is the filter size used in $C{L}_2$.

The partial derivative in Equation (27) can be computed by making use of Equation (9) evaluated in $i-u+1$. Each element of the error $\Delta S_j^{\left(1\right)}$ is then

$$\begin{array}{cc}\hfill \Delta S_j^{\left(1\right)}\left(i\right)& =\sum _{l=1}^{q=6}\sum _{u=1}^{s^{\left(2\right)}}\Delta C_l^{\left(2\right)}(i-u+1)·K_{j,l}^{\left(2\right)}\left(u\right)\hfill \\ \hfill & =\sum _{l=1}^{q=6}\sum _{u=1}^{s^{\left(2\right)}}\Delta C_l^{\left(2\right)}(i+(-u)+1)·K_{j,l}^{\left(2\right)}(-(-u))\hfill \end{array}$$

(28)

and finally, the error can be rewritten as

$$\begin{array}{c}\hfill \Delta S_j^{\left(1\right)}=\sum _{l=1}^{q=6}\Delta C_l^{\left(2\right)}\ast K_{j,lrev}^{\left(2\right)}\end{array}$$

(29)

where $K_{j,lrev}^{\left(2\right)}$ is obtained by reversing the order of the elements in $K_{j,l}^{\left(2\right)}$.

Similar to the approach used in $C{L}_2$ for average pooling, here in $C{L}_1$ we can write

$$\Delta C_{j\sigma }^{\left(1\right)}\left(i_u\right)={\displaystyle \frac{1}{S{S}^{\left(1\right)}}}\Delta S_j^{\left(1\right)}\left(\lceil i_u/S{S}^{\left(1\right)}\rceil \right)$$

(30)

where the index i of $\Delta S_j^{\left(1\right)}$ runs from 1 to $n=8$, $i_u$ runs from 1 to $16=S{S}^{\left(1\right)}n$, and finally, $\lceil x\rceil$ represents the ceiling function. As in previous instances, in the following, i will denote the index $i_u$.

Finally, we can calculate each element of the error $\Delta K_{1,j}^{\left(1\right)}$ as follows:

$$\begin{array}{cc}\hfill \Delta K_{1,j}^{\left(1\right)}\left(u\right)& ={\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }{K}_{1,j}^{\left(1\right)}\left(u\right)}}\hfill \\ \hfill & =\sum _{i=1}^{16}{\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }{C}_{j,\sigma }^{\left(1\right)}\left(i\right)}}{\displaystyle \frac{\mathit{\partial }{C}_{j,\sigma }^{\left(1\right)}\left(i\right)}{\mathit{\partial }{K}_{1,j}^{\left(1\right)}\left(u\right)}}\hfill \\ \hfill & =\sum _{i=1}^{16}\Delta C_{j,\sigma }^{\left(1\right)}\left(i\right)·C_{j,\sigma }^{\left(1\right)}\left(i\right)(1-C_{j,\sigma }^{\left(1\right)}\left(i\right))·I(i+u-1)\hfill \\ \hfill & =\sum _{i=1}^{16}\Delta C_j^{\left(1\right)}\left(i\right)·I(i+u-1)\hfill \end{array}$$

(31)

and then the error is

$$\begin{array}{c}\hfill \Delta K_{1,j}^{\left(1\right)}=I\ast \Delta C_j^{\left(1\right)}\end{array}$$

(32)

where $\Delta C_j^{\left(1\right)}=\Delta C_{j,\sigma }^{\left(1\right)}{C}_{j,\sigma }^{\left(1\right)}(1-C_{j,\sigma }^{\left(1\right)})$ has been obtained considering sigmoid as the activation function ($\sigma$). Again, other activation functions may be used in these hidden layers. These cases will be discussed in the Appendix A.

Analogously, the bias error is

$$\begin{array}{cc}\hfill \Delta b_j^{\left(1\right)}& ={\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }{b}_j^{\left(1\right)}}}\hfill \\ \hfill & =\sum _{i=1}^{16}{\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }{C}_{j,\sigma }^{\left(1\right)}\left(i\right)}}{\displaystyle \frac{\mathit{\partial }{C}_{j,\sigma }^{\left(1\right)}\left(i\right)}{\mathit{\partial }{b}_j^{\left(1\right)}}}\hfill \\ \hfill & =\sum _{i=1}^{16}\Delta C_{j,\sigma }^{\left(1\right)}\left(i\right)·C_{j,\sigma }^{\left(1\right)}\left(i\right)(1-C_{j,\sigma }^{\left(1\right)}\left(i\right))\hfill \\ \hfill & =\sum _{i=1}^{16}\Delta C_j^{\left(1\right)}\left(i\right).\hfill \end{array}$$

(33)

where the sigmoid has been used as the activation function, and the substitution $\Delta C_l^{\left(2\right)}=\Delta C_{l,\sigma }^{\left(2\right)}·C_{l,\sigma }^{\left(2\right)}·(1-C_{l,\sigma }^{\left(2\right)})$ introduced above.

6. Other Activation and Loss Functions

In addition to the sigmoid as an activation function, which was considered in Section 5, here, we aim to extend the calculation to also consider the hyperbolic tangent (tanh) [131,132,133,134], logarithmic hyperbolic cosine (log cosh) [135], and Rectified Linear Unit (ReLU) [136], including some of its variants [129,137,138].

Moreover, along with MSE, other loss functions are explored for both regression and classification problems. To provide the correct expressions for the different cases, in backpropagation, what needs to be modified are only the expressions for $\Delta y_{out}\left(i\right)$, $\Delta C_l^{\left(2\right)}\left(i\right)$, and $\Delta C_j^{\left(1\right)}\left(i\right)$. Once computed, they can be used in Equations (15)–(33). Generally, the choice of the activation function depends on which layer you are dealing with [139].

Although ReLU is commonly preferred in hidden layers (here $C{L}_1$ and $C{L}_2$), there are examples of its use in output layers [140] when the response variable is continuous and greater than zero [141].

In this section, we assume that the ReLU activation function is applied exclusively in hidden layers, while the other activation functions are used in both the output and hidden layers. Let us start by discussing the regression task. The various configurations explored in terms of activation and loss functions for this task are summarized in the

Table 2. Expressions for $\Delta y_{out}$ corresponding to different loss and activation functions in the output layer.

Loss Function	Activation Function in Output Layer	$\mathit{\Delta }{\mathit{y}}_{\mathbf{out}}$
MSE	sigmoid	(15)
MSE	tanh	(34)
log cosh	sigmoid	(36)
log cosh	tanh	(37)

and

Table 3. Expressions for $\Delta C^{\left(n\right)}$ corresponding to different activation functions in hidden layers.

Activation Function in Hidden Layer	$\mathit{\Delta }{\mathit{C}}^{\left(\mathit{n}\right)}$
sigmoid	(20),(30)
tanh	(38)
ReLU	(40)
ELU	(43)
Leaky ReLU	(44)

.

The expression for $\Delta y_{out}\left(i\right)$ depends on both the loss and the activation functions considered in the output layer. When MSE is used with the sigmoid, the valid expression is the one provided in Section 5. Instead, when tanh is used, we have

$$\begin{array}{cc}\hfill \Delta y_{out}\left(i\right)& =\left(y_{out}\left(i\right)-y_t\left(i\right)\right)·\left(1-y_{out}^2\left(i\right)\right).\hfill \end{array}$$

(34)

where the first factor comes from the derivative of MSE, and the second one from the derivative of tanh, which is $1-{tanh}^2$.

Another loss function commonly used in regression problems is represented by the log cosh function

$$L={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left(\log \left(\cosh \left(y_{out}\left(i\right)-y_t\left(i\right)\right)\right)\right)$$

(35)

where i runs from 1 to the dimension of the output (here, $N=2$). With log cosh as loss function, the expression (34) becomes

$$\begin{array}{cc}\hfill \Delta y_{out}\left(i\right)& ={\displaystyle \frac{1}{2}}tanh\left(y_{out}\left(i\right)-y_t\left(i\right)\right)y_{out}\left(i\right)·\left(1-y_{out}\left(i\right)\right)\hfill \end{array}$$

(36)

for sigmoid activation function in the output layer, and

$$\begin{array}{cc}\hfill \Delta y_{out}\left(i\right)& ={\displaystyle \frac{1}{2}}tanh\left(y_{out}\left(i\right)-y_t\left(i\right)\right)·\left(1-y_{out}^2\left(i\right)\right)\hfill \end{array}$$

(37)

for tanh activation in the output layer. As in Equations (34)–(36), the first factor comes from the derivative of the loss function, and the second comes from that of the activation function.

The different activation functions available for hidden layers have an impact on both $\Delta C_l^{\left(2\right)}\left(i\right)$ and $\Delta C_j^{\left(1\right)}\left(i\right)$ expressions. When the sigmoid is used, the valid expressions are those provided in Section 5. Instead, when the tanh activation function is used, we have

$$\begin{array}{cc}\hfill \Delta C_l^{\left(n\right)}\left(i\right)& =\Delta C_{l,\sigma }^{\left(n\right)}\left(i\right)\left(1-{\left(C_{l,\sigma }^{\left(n\right)}\left(i\right)\right)}^2\right)\hfill \end{array}$$

(38)

where $n=1,2$ represents the hidden layer number.

Moreover, with the ReLU activation function in hidden layers defined as follows:

$$\begin{array}{cc}\hfill ReLU\left(x\right)& =\left\{\begin{array}{cc}x\hfill & x\ge 0\hfill \\ 0\hfill & x<0\hfill \end{array}\right.,\hfill \end{array}$$

(39)

Equation (38) can be rewritten:

$$\begin{array}{cc}\hfill \Delta C_l^{\left(n\right)}\left(i\right)& =\Delta C_{l,\sigma }^{\left(n\right)}\left(i\right)\left\{\begin{array}{cc}1\hfill & C_l^{\left(n\right)}\left(i\right)⩾0\hfill \\ 0\hfill & C_l^{\left(n\right)}\left(i\right)<0\hfill \end{array}\right.\hfill \end{array}$$

(40)

where $n=1,2$ represents the hidden layer number.

Among the possible variants of teh ReLU activation function, we can consider ELU (Exponential Linear Unit):

$$\begin{array}{cc}\hfill ELU\left(x\right)& =\left\{\begin{array}{cc}x\hfill & x>0\hfill \\ \alpha (e^x-1)\hfill & x\le 0\hfill \end{array}\right.\hfill \end{array}$$

(41)

and Leaky ReLU

$$\begin{array}{cc}\hfill LeakyReLU\left(x\right)& =\left\{\begin{array}{cc}x\hfill & x\ge 0\hfill \\ \alpha x\hfill & \le 0\hfill \end{array}\right.\hfill \end{array}$$

(42)

with $\alpha$ a parameter that might be considered as trainable.

The expression (38) is now

$$\begin{array}{cc}\hfill \Delta C_l^{\left(n\right)}\left(i\right)& =\Delta C_{l,\sigma }^{\left(n\right)}\left(i\right)\left\{\begin{array}{cc}1\hfill & C_l^{\left(n\right)}\left(i\right)>0\hfill \\ \alpha e^{C_l^{\left(n\right)}\left(i\right)}\hfill & C_l^{\left(n\right)}\left(i\right)\le 0\hfill \end{array}\right.\hfill \end{array}$$

(43)

for ELU, and

$$\begin{array}{cc}\hfill \Delta C_l^{\left(n\right)}\left(i\right)& =\Delta C_{l,\sigma }^{\left(n\right)}\left(i\right)\left\{\begin{array}{cc}1\hfill & C_l^{\left(n\right)}\left(i\right)⩾0\hfill \\ \alpha \hfill & C_l^{\left(n\right)}\left(i\right)<0\hfill \end{array}\right.\hfill \end{array}$$

(44)

for Leaky ReLU.

Let us shift our focus to classification tasks. In these scenarios, the output layer provides probabilities for decision-making when classifying input data. Essentially, the output is presented as a list of probabilities for different potential labels. When calculating the error, the target output $y_t\left(i\right)$ is represented by a one-hot vector that has 1 at one index and 0 at all other indices. Consequently, this implies that ${\sum }_{k=1}^N{y}_t\left(k\right)=1$. Since the output of the output layer may assume both positive and negative real values, the output $y_{out}\left(i\right)$ needs to be transformed into probability-like values. This can be achieved using the Softmax function:

$$\begin{array}{cc}\hfill P_i& ={\displaystyle \frac{e^{y_{out}\left(i\right)}}{{\sum }_{k=1}^N{e}^{y_{out}\left(k\right)}}}\hfill \end{array}$$

(45)

where $0\le P_i\le 1$ and ${\sum }_{i=1}^N{P}_i=1$. The derivative of Equation (43) can be easily calculated as follows:

$$\begin{array}{cc}\hfill {\displaystyle \frac{\mathit{\partial }{P}_k}{\mathit{\partial }{y}_{out}\left(i\right)}}& =\left\{\begin{array}{cc}-P_k{P}_i\hfill & k\ne i\hfill \\ P_k(1-P_i)\hfill & k=i.\hfill \end{array}\right.\hfill \end{array}$$

(46)

When using the Softmax as the output activation function, the loss function that is generally associated is represented by the cross-entropy, given by

$$\begin{array}{cc}\hfill L& =-\sum _{i=1}^N{y}_t\left(i\right)ln\left(P_i\right).\hfill \end{array}$$

(47)

The expression for $\Delta y_{out}\left(i\right)$ can be derived after calculating the derivative of the loss, which is obtained using the derivative of Equation (44):

$$\begin{array}{cc}\hfill {\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }{y}_{out}\left(i\right)}}& =\sum _{k=1}^N{\displaystyle \frac{\mathit{\partial }L}{\mathit{\partial }{P}_k}}{\displaystyle \frac{\mathit{\partial }{P}_k}{\mathit{\partial }{y}_{out}\left(i\right)}}\hfill \\ \hfill & =\sum _{k=1}^N\left({\displaystyle \frac{y_t\left(k\right)}{P_k}}{P}_k{P}_i\right)-{\displaystyle \frac{y_t\left(i\right)}{P_i}}{P}_i\hfill \\ \hfill & =P_i-y_t\left(i\right).\hfill \end{array}$$

(48)

7. Conclusions

This tutorial thoroughly covers the theoretical aspects of feedforward and backpropagation in 1D CNNs, providing a step-by-step guide. It points out existing relationships with linear algebra, statistics, and optimization, addressing gaps identified in our literature review. The provided results consider both regression and classification tasks and include some of the most widely used activation functions. Its goal is to support the development of practical applications using these mathematical concepts. Moreover, the tutorial highlights potential challenges and pitfalls commonly encountered when implementing new library packages for 1D CNN in languages other than the available open-source ones. By bridging theoretical foundations with practical insights, this tutorial serves as a comprehensive guide for both beginners and experienced professionals aiming to deepen their understanding of 1D CNNs. In conclusion, this tutorial not only enhances comprehension of 1D CNNs but also equips beginners with the confidence to apply these techniques effectively. Future research directions might include exploring advanced architectures, integrating novel activation functions, and leveraging emerging optimization algorithms to further improve the performance and applicability of 1D CNNs.