**3. Methodology**

Three types of neural network variants were modeled and tested for accurately forecasting natural gas energy demand. Their implementations differ even though they all belong to the neural network family of algorithms. Each approach has different input variables, however, the general approach remains the same; historical data train a model that is able to produce accurate forecasts of natural gas energy consumption. The approaches, as well as the general process flow, are described in the following sections.

#### *3.1. Artificial Neural Networks (ANN)*

Artificial Neural Networks (ANN) [44] were hypothesized as a method to imitate the human brain and its functions while it performs cognitive tasks, or when it learns. For the mathematical structure of such networks, nodes are used to represent the neurons, and layers are used to represent their interactive synapses in the same fashion as it is within the brain. They received wide acceptance ever since they were conceptualized, however, they started gaining more and more fame ever since the computational power, especially in graphic processing units, has become cheaper and thus easily attainable. Another important reason for their wide acceptance is the vast availability of immense amounts of data that have been collected throughout the past years. This adoption has enhanced the scientific progress of the ANN algorithms, which started from the single-layer perceptron [45], moved to multi-layer perceptron [46], introduced the back-propagation algorithm [47], and led to many new derivatives of ANNs, the most well-known being the deep neural networks that are described in the next paragraph.

#### *3.2. Long Short-Term Memory Networks (LSTM)*

Long Short-Term Memory are also neural networks which are built upon a recurrent fashion by introducing memory cells and their in-between connections, in order to construct a graph directed over a sequence. In general, recurrent networks process sequences by using said memory cells in a fashion that is different than that of the simple ANNs, and even though they are well suited for problems with time dependency, they often face the problem of vanishing gradients, or not being able to "memorize" large portions of data. LSTMs [48] solved this problem because of the specific cell structure they have, which allows the network the ability to variate the amount of retained information. These cell

structures are called gates, and they control which information is stored in the long memory and which is discarded, thus optimizing the memorizing process. Dynamic temporal behavior problems, i.e., time sequences, were suited for such approaches.

#### *3.3. Deep Neural Networks (DNN)*

Deep neural networks [49] are the basis of deep learning, one of the most influential areas of the artificial intelligence for the past decade. Based on the ANN, the DNN is comprised by more layers and nodes in the same sequential fashion. For very deep implementations, the problem of "vanishing" or "exploding" gradients would not allow the network to learn properly, therefore new techniques were introduced, such as "identity shortcut connections" as seen in Resnet [50], as well as others, in order to solve these obstacles. The "deep" approach has been used in all derivatives of neural networks. Deep convolutional neural networks (CNN) have been used for image classification and object detection [51,52], and deep recurrent neural networks (RNN) have been used for word prediction [53] and time series forecasting [54].

#### *3.4. Process Flow*

In order to accurately model the natural gas energy forecast, a specific process flow has been designed. The steps that have been followed, are described below.
