Higher-Order Convolutional Neural Networks for Essential Climate Variables Forecasting

Abstract

Earth observation imaging technologies, particularly multispectral sensors, produce extensive high-dimensional data over time, thus offering a wealth of information on global dynamics. These data encapsulate crucial information in essential climate variables, such as varying levels of soil moisture and temperature. However, current cutting-edge machine learning models, including deep learning ones, often overlook the treasure trove of multidimensional data, thus analyzing each variable in isolation and losing critical interconnected information. In our study, we enhance conventional convolutional neural network models, specifically those based on the embedded temporal convolutional network framework, thus transforming them into models that inherently understand and interpret multidimensional correlations and dependencies. This transformation involves recasting the existing problem as a generalized case of N-dimensional observation analysis, which is followed by deriving essential forward and backward pass equations through tensor decompositions and compounded convolutions. Consequently, we adapt integral components of established embedded temporal convolutional network models, like encoder and decoder networks, thus enabling them to process 4D spatial time series data that encompass all essential climate variables concurrently. Through the rigorous exploration of diverse model architectures and an extensive evaluation of their forecasting prowess against top-tier methods, we utilize two new, long-term essential climate variables datasets with monthly intervals extending over four decades. Our empirical scrutiny, particularly focusing on soil temperature data, unveils that the innovative high-dimensional embedded temporal convolutional network model-centric approaches markedly excel in forecasting, thus surpassing their low-dimensional counterparts, even under the most challenging conditions characterized by a notable paucity of training data.

1. Introduction

Monitoring Earth’s climate change is a significant contemporary challenge, and data collected from different geographical areas over time play a critical role in understanding different physical characteristics and processes. Essential Climate Variables (ECVs) are crucial to this work, as they provide global-scale climate change insight and help assess climate risks and causes. The Global Climate Observing System (GCOS) (https://gcos.wmo.int/en/essential-climate-variables) (accessed on 1 February 2024) currently specifies 54 such variables [1], including soil temperature, soil moisture, above-ground biomass, and sea surface salinity, among others. ECVs are observed according to specific climate monitoring principles (identified by the GCOS), while the typical estimation of ECVs is usually based on numerical simulation models [2].

In the past years, data-driven Machine Learning (ML) methods have also contributed to the field in terms of mitigation and the prediction of climate change [3]. Moreover, the quite prominent performance of Deep Learning (DL) methods in numerous Remote Sensing (RS) tasks [4], such as multispectral [5] and hyperspectral [6] classification, as well as multitemporal land cover classification [7,8], has also paved the way for its involvement in the field [9]. While DL methods have been considered for the instantaneous retrieval of the ECV values, the problem of forecasting has yet to be explored in sufficient detail. Furthermore, existing cutting-edge DL frameworks [9] forecast each ECV independently, thus failing to capitalize on existing correlations.

In RS research, considerable efforts concentrate on DL models for supervised classification tasks, thus spanning land cover classification [10], building detection [11], and scene classification [12], among others. The interest in spatiotemporal data analysis is escalating, thus prompted by its extensive application potential [13]. Urgent global challenges, like uncontrolled deforestation [14,15] and growing environmental pollution [16,17], accentuate the necessity for pioneering research and solutions, thus attracting academia, policymakers, and decision makers.

When the problems in question involve sequential time dependence, DL methods like Recurrent Neural Networks (RNNs) and Long Short Term Memory (LSTM) networks [18] are typically employed. While such methods can capture long-range temporal dependencies, they are not able to simultaneously capture both spatial and temporal observations, which are encoded in 3D spatiotemporal structures. A very successful approach toward that goal is the Conv-LSTM [19], which is a variant of the LSTM network in which the data flow through the cells by keeping the input dimension (3D in that case) instead of being just a 1D vector with features.

Beyond the RNN/LSTM methodologies in sequence modeling tasks, a particular subset of CNN models has emerged as a noteworthy substitute, specifically the Temporal Convolutional Networks (TCNs). The TCNs, which are 1D models, were initially developed for detailed action segmentation tasks [20]. In this pioneering work, two variants of TCNs were presented: the encoder–decoder TCN and the dilated TCN, with the latter drawing inspiration from the Wavenet model [21]. Another model conceived by Google DeepMind [22] leveraged 1D-CNN structures, utilizing dilated convolutions and residual blocks, and it outperformed the capabilities of existing RNN models. This model underscores the effectiveness of 1D-CNN architectures in complex sequence modeling tasks.

Of noticeable interest are works aiming to compensate for multidimensional data with more than three dimensions by adopting 4D architectures. More precisely, 4D-CNNs have been introduced for semantically segmenting cardiac volumetric sequences [23] and classifying multitemporal land cover [7,8], amongst others. Both approaches construct encoder–decoder networks, but while in [23], the upsampling is performed via kNN interpolation, in [7,8], it is executed via fully learnable 4D transpose convolution.

Focusing on spatiotemporal observation analysis, [19] proposed a Conv-LSTM model for the problem of precipitation nowcasting. To predict future radar maps from past radar echo sequences, the method adopted an LSTM architecture that employs convolutions in the input-to-state and state-to-state transitions. Moreover, Conv-LSTM architectures have also been employed in similar tasks, thus dealing with radar [24] and satellite [25] time series data.

In the study conducted in [26], another form of TCN architecture that utilizes residual blocks and dilated causal convolutions was presented. This model was benchmarked against various recurrent architectures, including LSTMs, Gated Recurrent Units (GRUs), and RNNs, in tasks typically employed for RNN evaluations, with results indicating superior performance of the TCN. However, these outcomes are not universal; for example, ref. [27] demonstrated scenarios where LSTM and CNN outperformed TCN, specifically in forecasting day-ahead electricity prices in Spain. This suggests that the effectiveness of these architectures can significantly depend on the context and specific characteristics of the task at hand.

In a noteworthy recent study, Villia et al. introduced an innovative technique for predicting ECV values, which is detailed in their paper [9]. The proposed model, termed ETCN, merges three distinct networks: an encoder, a TCN, and a decoder. The proposed architecture is capable of analyzing spatiotemporal sequences and predicting the next time step. We discuss the method in detail in Section 2.2.

In essence, existing state-of-the-art ECVs’ forecasting architectures comprise lower-order (2D or 3D) models in the quest to accurately forecast ECVs’ values. Retooling these lower-order DL architectures for handling the intrinsically 4D simultaneous ECVs’ forecasting task can lead to inferior performance, as demonstrated in Figure 1.

Simultaneously measuring multiple ECVs such as soil temperature at different depths can provide more reliable estimations by exploiting the inherent correlations, thus mandating the need for models that are capable of performing the regression of high-dimensional structures. In this work, we diverge from traditional methodologies and introduce a sophisticated high-dimensional DL strategy for forecasting ECVs. We accomplish this by expanding the cutting-edge Embedded Temporal Convolutional Network (ETCN) architecture [9] into its more advanced high-dimensional variant, thus enabling the concurrent learning of spatiotemporal features across all ECVs under consideration. This holistic approach not only streamlines the forecasting process, but also leverages the intricate interdependencies present among the various climate variables.

In addition, we present two new ECV forecast datasets for the purpose of training and assessing the efficacy of the suggested model in comparison to leading approaches. We carry out comprehensive tests to measure each model’s performance against various structures and multiple model parameters. The empirical outcomes from actual ECV data underscore the superiority and promise of the proposed advanced feature modeling in all cases under consideration.

The main contributions of this work include the following:

• The theoretical extension of lower-order Convolutional Neural Networks (CNNs) to their N-D analogs by providing a rigorous mathematical foundation for the forward and backward passes;
• The demonstration of issues and workarounds for incorporating this extension into a working DL framework;
• The design and development of a novel 4D-ETCN to tackle the ECVs’ forecasting problem and its evaluation on properly designed datasets for the cause;
• The introduction of two novel datasets encoding satellite-derived geophysical parameters, specifically soil temperature at different levels, obtained on monthly periodicity over 40 years.

The paper is organized as follows: Section 2 introduces the proposed high-dimensional extension of the basic functionality of the convolutional layer for both forward and backward passes, with some parts of the detailed mathematical and conceptual study of this extension being placed in the accompanying appendices of the present work. In addition, we therein sketch the proposed 4D-ETCN architecture for tackling the ECVs’ forecasting problem as an instance of the aforementioned extension for the problem at hand. Section 3 outlines the datasets formulated and the experimental frameworks established, and this is accompanied by the in-depth evaluations performed to determine the effectiveness of the introduced 4D-ETCN. This section also encompasses the consequential conclusions extracted from these investigative procedures. Furthermore, Section 4 furnishes an interpretation of the obtained results, thereby elucidating both the merits and drawbacks of the proposed approach in comparison to its counterparts. Section 5 distills the core conclusions and significant insights gained through this study.

2. Materials and Methods

2.1. Proposed Method: Higher-Order Convolutional Neural Networks

In this section, we demonstrate the case of 4D data analysis by deriving the required equations for generalizing the traditional low-order convolutional layers to their higher-order analogs. For that cause, we introduce two different alternatives, namely tensor decompositions and stacked convolutions.

2.1.1. Preliminary Definitions and Tensor Algebra

A tensor is a multidimensional array denoted by $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N}$. The order of $\mathcal{X}$, N, is determined by the number of its dimensions, which are referred to as modes. Without loss of generality, in this work, we focus on 4-dimensional tensors, where two dimensions are employed for spatial encoding, one for temporal, and one for the specific modality/variable.

A typical operation on a tensor is to reorder the mode n vectors/fibers into a matrix, which is a transformation called “matricization”/unfolding/flattening. Note that this transformation is not unique in the sense that there exist many different ways of stacking up the mode n fibers of a tensor into a matrix. Formally, following the notation proposed in [28], the mode n unfolding of a tensor stacks its mode n fibers as columns to a matrix ${\mathbf{X}}_{\left(n\right)}$, which can be defined as follows:

Definition 1

(Tensor Unfolding). An Nth order tensor $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N}$ can be flattened into a matrix ${\mathbf{X}}_{\left(n\right)}$, where the tensor element $(i_1,i_2,\dots ,i_n)$ is mapped to the matrix element $(i_n,j)$, with

$$\begin{array}{c}\hfill \left\{\begin{array}{c}j=1+\sum _{\begin{array}{c}k=1\\ k\ne n\end{array}}^N(i_k-1)J_k\hfill \\ J_k=\prod _{\begin{array}{c}m=1\\ m\ne n\end{array}}^{k-1}{I}_m\hfill \end{array}\right.\end{array}$$

(1)

An important aspect of tensor algebra is the diverse products it provides. Focusing on the relevant operations, one can define the following products, namely Kronecker product and Mode n product [29,30], as follows:

Definition 2

(Kronecker Product). Given two matrices $\mathbf{X}\in {\mathbb{R}}^{I\times J}$ and $\mathbf{Y}\in {\mathbb{R}}^{K\times L}$, their Kronecker product is a matrix defined as follows:

$$\begin{array}{c}\hfill \mathbf{Z}=\mathbf{X}\otimes \mathbf{Y}=\left[\begin{array}{cccc}{x}_{11}\mathbf{Y}& x_{12}\mathbf{Y}& \cdots & x_{1J}\mathbf{Y}\\ x_{21}\mathbf{Y}& x_{22}\mathbf{Y}& \cdots & x_{2J}\mathbf{Y}\\ ⋮& ⋮& \ddots & ⋮\\ x_{I1}\mathbf{Y}& x_{I2}\mathbf{Y}& \cdots & x_{IJ}\mathbf{Y}\end{array}\right]\in {\mathbb{R}}^{IK\times JL}\end{array}$$

(2)

Definition 3

(Mode n Product). Given a tensor $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N}$ and a matrix $\mathbf{Y}\in {\mathbb{R}}^{K\times I_N}$, their Mode n/Tensor Times Matrix product is a tensor defined as follows:

$$\begin{array}{c}\hfill \mathcal{Z}-=\mathcal{X}{\times }_{\left(n\right)}\mathbf{Y}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_{n-1}\times K\times I_{n+1}\times \dots \times I_N}\end{array}$$

(3)

In essence, each mode n fiber of tensor $\mathcal{X}$ is multiplied by matrix $\mathbf{Y}$. A more eloquent interpretation though is to express the mode n product in terms of unfolded tensors as follows:

$$\begin{array}{c}\hfill \mathcal{Z}-=\mathcal{X}{\times }_{\left(n\right)}\mathbf{Y}\iff {\mathbf{Z}}_{\left(n\right)}=\mathbf{Y}{\mathbf{X}}_{\left(n\right)}.\end{array}$$

(4)

Probably the most prominent advantage of tensors is the diverse decompositions they offer for data of multiple dimensions. Here, we provide a brief overview of the one that is useful in the context of this work (i.e., Tucker), while the other most famous ones (e.g., CP/PARAFAC/CANDECOMP and Tensor-Train) are explained in details in seminal works in the field [28,29,30,31]. The Tucker decomposition has taken its name from the Ledyard R. Tucker [32,33,34] and is widely known as an extension of the classical PCA and SVD methods to higher dimensions [35]. More precisely, we have the following definition.

Definition 4

(Tucker Decomposition). Given a tensor $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N}$ of order N, Tucker decomposition decomposes it into a core tensor $\mathcal{G}\in {\mathbb{R}}^{R_1\times R_2\times \dots \times R_N}$ and a set of factor matrices $\{{\mathbf{U}}^{\left(1\right)},{\mathbf{U}}^{\left(2\right)},\dots ,{\mathbf{U}}^{\left(N\right)}\}$, with ${\mathbf{U}}^{\left(N\right)}=[{\mathbf{u}}_1^{\left(n\right)};{\mathbf{u}}_2^{\left(n\right)};\dots ;{\mathbf{u}}_{R_N}^{\left(n\right)}]\in {\mathbb{R}}^{R_N\times I_N}$ and $n=1,2,\dots ,N$. The core tensor $\mathcal{G}$ is multiplied along each of its modes with the factor matrices $\{{\mathbf{U}}^{\left(1\right)},{\mathbf{U}}^{\left(2\right)},\dots ,{\mathbf{U}}^{\left(N\right)}\}$ in the following way:

$$\begin{array}{cc}\hfill \mathcal{X}& =\mathcal{G}{\times }_{\left(1\right)}{\mathbf{U}}^{\left(1\right)}{\times }_{\left(2\right)}{\mathbf{U}}^{\left(2\right)}{\times }_{\left(3\right)}\dots {\times }_{\left(N\right)}{\mathbf{U}}^{\left(N\right)}\hfill \\ \hfill & =\sum _{r_1=1}^{R_1}\sum _{r_2=1}^{R_2}\dots \sum _{r_n=1}^{R_N}{g}_{r_1{r}_2\dots r_N}({\mathbf{u}}_{r_1}^{\left(1\right)}\circ {\mathbf{u}}_{r_2}^{\left(2\right)}\circ \dots \circ {\mathbf{u}}_{r_N}^{\left(N\right)})\hfill \\ \hfill & =∥\mathcal{G};{\mathbf{U}}^{\left(1\right)},{\mathbf{U}}^{\left(2\right)},\dots ,{\mathbf{U}}^{\left(N\right)}∥\hfill \end{array}$$

(5)

2.1.2. Forward Propagation in ND CNNs

In order to extend CNNs operating on low-dimensional observations to their high-order analogs, we have to generalize the basic concept or convolutions to the N-D case. For that cause, we propose two different ways of computation: one via tensor decompositions and one via stacking of lower-dimensional convolutions.

In the former case, by employing the notations proposed by Cichocki et al. in [36], we initially define the convolution between two tensors along a specific mode, namely Partial (Mode n) Convolution, in the following way:

Definition 5

(Partial (Mode n) Convolution). Given two tensors $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N}$ and $\mathcal{Y}\in {\mathbb{R}}^{J_1\times J_2\times \dots \times J_N}$ of order N, their Partial (Mode n) Convolution derives a tensor $\mathcal{Z}-=\mathcal{X}{⊡}_{\left(n\right)}\mathcal{Y}$ whose subtensors are computed as follows:

$$\begin{array}{cc}\hfill \mathcal{Z}-(k_1,k_2,\dots ,:,\dots ,k_N)& =\mathcal{X}(i_1,i_2,\dots ,:,\dots ,i_N)★\mathcal{Y}(j_1,j_2,\dots ,:,\dots ,j_N)\hfill \end{array}$$

(6)

where ★ stands for the linear convolution operation, and $k_l=\overline{i_l{j}_l}$ for $l=1,2,\dots ,N$. The multi-index $i=\overline{i_1{i}_2\dots i_N}$ is an index which takes all possible combinations of values of indices $i_1{i}_2\dots i_N$ with $i_n=1,2,\dots ,I_n$ and $n=1,2,\dots ,N$, and it is defined in the following way:

$$\begin{array}{cc}\hfill i& =\overline{i_1{i}_2\dots i_N}=i_1+(i_2-1)I_1+(i_3-1)I_1{I}_2+\dots +(i_N-1)I_1\dots I_{N-1}\hfill \end{array}$$

(7)

In the context of this study, we restrict only to a specific case of Equation (6), namely partial (mode 1) convolution of matrices $\mathbf{X}\in {\mathbb{R}}^{I_1\times I_2}$ and $\mathbf{Y}\in {\mathbb{R}}^{J_1\times J_2}$, which is defined as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\mathbf{Z}=\mathbf{X}{⊡}_{\left(1\right)}\mathbf{Y}\hfill \\ \mathbf{Z}(:,k_2)=\mathbf{X}(:,i_2)★\mathbf{Y}(:,j_2)\hfill \end{array}\right.\end{array}$$

(8)

Given the above definitions and notations, the N-D Tucker convolution is then defined [36] as follows:

Definition 6

(N-D Tucker Convolution). Given two tensors $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N}$ and $\mathcal{Y}\in {\mathbb{R}}^{J_1\times J_2\times \dots \times J_N}$ of order N, their N-D Tucker convolution $\mathcal{Z}-=\mathcal{X}★\mathcal{Y}$ is computed in the following way:

1. 

Perform Tucker decomposition on tensor $\mathcal{X}$, and obtain its core tensor ${\mathcal{G}}_{\mathcal{X}}\in {\mathbb{R}}^{R_1\times R_2\times \dots \times R_N}$ and its factor matrices $\{{\mathbf{X}}^{\left(1\right)},{\mathbf{X}}^{\left(2\right)},\dots ,{\mathbf{X}}^{\left(N\right)}\}$:

$$\begin{array}{cc}\hfill \mathcal{X}& ={\mathcal{G}}_{\mathcal{X}}{\times }_{\left(1\right)}{\mathbf{X}}^{\left(1\right)}{\times }_{\left(2\right)}{\mathbf{X}}^{\left(2\right)}{\times }_{\left(3\right)}\dots {\times }_{\left(N\right)}{\mathbf{X}}^{\left(N\right)}=∥{\mathcal{G}}_{\mathcal{X}};{\mathbf{X}}^{\left(1\right)},{\mathbf{X}}^{\left(2\right)},\dots ,{\mathbf{X}}^{\left(N\right)}∥\hfill \end{array}$$

(9)

2. 

Perform Tucker decomposition on tensor $\mathcal{Y}$, and obtain its core tensor ${\mathcal{G}}_{\mathcal{Y}}\in {\mathbb{R}}^{Q_1\times Q_2\times \dots \times Q_N}$ and its factor matrices $\{{\mathbf{Y}}^{\left(1\right)},{\mathbf{Y}}^{\left(2\right)},\dots ,{\mathbf{Y}}^{\left(N\right)}\}$:

$$\begin{array}{cc}\hfill \mathcal{Y}& ={\mathcal{G}}_{\mathcal{Y}}{\times }_{\left(1\right)}{\mathbf{Y}}^{\left(1\right)}{\times }_{\left(2\right)}{\mathbf{Y}}^{\left(2\right)}{\times }_{\left(3\right)}\dots {\times }_{\left(N\right)}{\mathbf{Y}}^{\left(N\right)}=∥{\mathcal{G}}_{\mathcal{Y}};{\mathbf{Y}}^{\left(1\right)},{\mathbf{Y}}^{\left(2\right)},\dots ,{\mathbf{Y}}^{\left(N\right)}∥\hfill \end{array}$$

(10)

3. 

Derive the core tensor of the N-D Tucker convolution, ${\mathcal{G}}_{\mathcal{Z}-}$, by combining the core tensors ${\mathcal{G}}_{\mathcal{X}}$ and ${\mathcal{G}}_{\mathcal{Y}}$ via their Kronocker product:

$$\begin{array}{cc}\hfill {\mathcal{G}}_{\mathcal{Z}-}& ={\mathcal{G}}_{\mathcal{X}}\otimes {\mathcal{G}}_{\mathcal{Y}}\in {\mathbb{R}}^{R_1{Q}_1\times R_2{Q}_2\times \dots \times R_N{Q}_N}\hfill \end{array}$$

(11)

4. 

Derive the factor matrices of the N-D Tucker convolution, $\{{\mathbf{Z}}^{\left(1\right)},{\mathbf{Z}}^{\left(2\right)},\dots ,{\mathbf{Z}}^{\left(N\right)}\}$, by combining the factor matrices $\{{\mathbf{X}}^{\left(1\right)},{\mathbf{X}}^{\left(2\right)},\dots ,{\mathbf{X}}^{\left(N\right)}\}$ and $\{{\mathbf{Y}}^{\left(1\right)},{\mathbf{Y}}^{\left(2\right)},\dots ,{\mathbf{Y}}^{\left(N\right)}\}$ via their partial (mode 1) convolution:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\mathbf{Z}}^{\left(n\right)}={\mathbf{X}}^{\left(n\right)}{⊡}_{\left(1\right)}{\mathbf{Y}}^{\left(n\right)}\hfill \\ {\mathbf{Z}}^{\left(n\right)}(:,s_n)={\mathbf{X}}^{\left(n\right)}(:,r_n)★{\mathbf{Y}}^{\left(n\right)}(:,q_n)\hfill \\ s_n=\overline{r_n{q}_n}=1,2,\dots ,R_n{Q}_n\hfill \end{array}\right.\end{array}$$

(12)

5. 

Derive the output tensor of the N-D Tucker convolution, $\mathcal{Z}-$, by combining the core tensor ${\mathcal{G}}_{\mathcal{Z}-}$ and the factor matrices $\{{\mathbf{Z}}^{\left(1\right)},{\mathbf{Z}}^{\left(2\right)},\dots ,{\mathbf{Z}}^{\left(N\right)}\}$ via their Tucker composition:

$$\begin{array}{cc}\hfill \mathcal{Z}-& ={\mathcal{G}}_{\mathcal{Z}-}{\times }_{\left(1\right)}{\mathbf{Z}}^{\left(1\right)}{\times }_{\left(2\right)}{\mathbf{Z}}^{\left(2\right)}{\times }_{\left(3\right)}\dots {\times }_{\left(N\right)}{\mathbf{Z}}^{\left(N\right)}=∥{\mathcal{G}}_{\mathcal{Z}-};{\mathbf{Z}}^{\left(1\right)},{\mathbf{Z}}^{\left(2\right)},\dots ,{\mathbf{Z}}^{\left(N\right)}∥\hfill \end{array}$$

(13)

The definition of the N-D Tucker convolution [36] between two third-order tensors $\mathcal{X}$ and $\mathcal{Y}$ is illustrated in Figure 2.

2.1.3. Backpropagation in ND CNNs

Based on the aforementioned discussion, the convolution of two ND tensors $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N}$ and $\mathcal{Y}\in {\mathbb{R}}^{J_1\times J_2\times \dots \times J_N}$ can be applied in the context of a CNN by substituting $\mathcal{X}$ with $\mathcal{I}$ as the layer’s input, $\mathcal{Y}$ with $\mathcal{W}$ as the layer’s trainable weights/kernel, and adding the respective bias term $\mathit{b}$. Hence, we can express the basic operation of an ND convolutional layer, l, in the “direct-sum” form as follows:

$$\begin{array}{cc}\hfill {\mathcal{Z}-}_{i_1,i_2,\dots ,i_N}\left(l\right)=& \sum _{j_1}\sum _{j_2}\dots \sum _{j_N}{\mathcal{W}}_{j_1,j_2,\dots ,j_N}\left(l\right){\mathcal{I}}_{i_1-j_1,i_2-j_2,\dots ,i_N-j_N}(l-1)+\mathit{b}\left(l\right)\hfill \\ \hfill =& \mathcal{W}\left(l\right)★{\mathcal{I}}_{i_1,i_2,\dots ,i_N}(l-1)+\mathit{b}\left(l\right)\hfill \end{array}$$

(14)

where $\{j_1,j_2,\dots ,j_N\}$ span the dimensions of the kernel, $\{i_1,i_2,\dots ,i_N\}$ span the dimensions of the input, and $\mathit{b}$ is the bias term. As can be seen from Equation (14), the input of the previous layer $(l-1)$ serves for the computation of the current output layer $\left(l\right)$, and the input of the current layer l can be computed as follows:

$$\begin{array}{c}\hfill {\mathcal{I}}_{i_1,i_2,\dots ,i_N}\left(l\right)=f\left({\mathcal{Z}-}_{i_1,i_2,\dots ,i_N}\left(l\right)\right)\end{array}$$

(15)

where f is a selected activation function (e.g., ReLU, tanh). With Equations (14) and (15), every value in the forward pass of an ND convolutional layer can be computed, so what remains to be defined is the respective equations during the backward pass.

In order to be able to derive the backward pass equations of an ND convolutional layer, we should dive in its functionality depicted in Figure 3. More precisely, given the input $\mathcal{I}$ and the weights $\mathcal{W}$, the layer outputs its N-D convolution during the forward pass. On the contrary, during the backward pass, the los gradient from the next layer, $\frac{\partial \mathcal{L}}{\partial \mathcal{Z}-}$, must be propagated to the previous layers. For that purpose, the local gradients (i.e., $\frac{\partial \mathcal{Z}-}{\partial \mathcal{I}}$, $\frac{\partial \mathcal{Z}-}{\partial \mathcal{W}}$) alongside the chain rule will help us calculate the gradients with respect to the input (i.e., $\frac{\partial \mathcal{L}}{\partial \mathcal{I}}$) and the filters (i.e., $\frac{\partial \mathcal{L}}{\partial \mathcal{W}}$). These gradients need to be updated for the following reasons:

• Since $\mathcal{I}$ is the output of the previous layer, $\frac{\partial \mathcal{L}}{\partial \mathcal{I}}$ becomes the loss gradient for the previous layer.
• $\frac{\partial \mathcal{L}}{\partial \mathcal{W}}$ is used to update the filters of the convolutional layer via the gradient step:

  $$\begin{array}{c}\hfill {\mathcal{W}}_{New}={\mathcal{W}}_{Old}-\mu \frac{\partial \mathcal{L}}{\partial \mathcal{W}}\end{array}$$

  (16)

  where $\mu$ is the step size parameter, which serves as the learning rate of the optimization problem.

More precisely, the aforementioned gradients can be computed via the chain rule, as depicted in Figure 3:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\frac{\partial \mathcal{L}}{\partial \mathcal{I}}=\frac{\partial \mathcal{L}}{\partial \mathcal{Z}-}\ast \frac{\partial \mathcal{Z}-}{\partial \mathcal{I}}\hfill \\ \frac{\partial \mathcal{L}}{\partial \mathcal{W}}=\frac{\partial \mathcal{L}}{\partial \mathcal{Z}-}\ast \frac{\partial \mathcal{Z}-}{\partial \mathcal{W}}\hfill \end{array}\right.\end{array}$$

(17)

Due to space limitations, the precise mathematical computation of Equation (17) in the closed form is provided, for all involved gradients, in Appendix B. In order to set our notation, we summarize the results therein for each one of them as follows:

• Loss gradient from the next layer:

  $$\begin{array}{cc}\hfill \frac{\partial \mathcal{L}}{\partial \mathcal{Z}-}& =f^{\prime }\left({\mathcal{Z}-}_{i_1,i_2,\dots ,i_N}\left(l\right)\right)\left[{\delta }_{i_1,i_2,\dots ,i_N}(l+1)★ro{t}_N\left(\mathcal{W}(l+1)\right)\right]\hfill \end{array}$$

  (18)
• Gradient with respect to the layer’s input:

  $$\begin{array}{c}\hfill \frac{\partial \mathcal{L}}{\partial \mathcal{I}}=conv\left(\frac{\partial \mathcal{L}}{\partial \mathcal{Z}-},\mathcal{W}\left(l\right)\right)\end{array}$$

  (19)
• Gradient with respect to the layer’s weights:

  $$\begin{array}{c}\hfill \frac{\partial \mathcal{L}}{\partial \mathcal{W}}=conv\left(\frac{\partial \mathcal{L}}{\partial \mathcal{Z}-},ro{t}_N\left(\mathcal{I}(l-1)\right)\right)\end{array}$$

  (20)
• Gradient with respect to the layer’s bias:

  $$\begin{array}{c}\hfill \frac{\partial \mathcal{L}}{\partial \mathit{b}}=\sum _{i_1}\sum _{i_2}\dots \sum _{i_N}\left(\frac{\partial \mathcal{L}}{\partial \mathcal{Z}-}\right)\end{array}$$

  (21)

Equations (19) and (20) sketch the general methodology for computing the desired gradients, and they indicate that in reality both the forward and backward passes of an ND convolutional layer are convolutions.

2.1.4. Efficient Implementation: Stacking (N-1)D Convolutions

After successfully expanding the convolution operator from the traditional 2D and 3D scenarios to the multidimensional (ND) context using the Tucker convolution model, we integrated this advanced operation into the DL paradigm. Specifically, we developed a bespoke Tucker convolution layer within the Tensorflow framework [37], thus enabling its application in the creation of higher-dimensional CNN models. Our initial experimentation with this Tensorflow adaptation was conducted on two lower-dimensional benchmark classification datasets, namely MNIST-2D and MNIST-3D. Through this exploration, our objective was to evaluate the computational efficiency of our DL adaptation vis à vis built-in convolutional layers.

To that end, we have constructed two identical model architectures for the two datasets, with only 1 convolutional layer (built-in and Tucker) before the final classification (i.e., fully connected) one. In addition, we have frozen the weights of the fully connected layer in order to measure fairly among the time needed for the respective models to train. To keep the architectures as simple as possible, we used only 4 filters in each convolution case, and we trained the models for up to only 10 epochs. Moreover, we employed only 128 samples for training purposes (i.e., out of the 60,000 available) and accordingly 128 for test purposes (i.e., out of the 10,000 available), while the batch size at each experiment was set to 32.

In

Table 1. Computational time (in seconds) required by the Tucker convolution model and the builtin Tensorflow one, for the MNIST classification problems. When incorporated with deep learning platforms, the Tucker convolution model clearly does not scale well.

Dataset	# Trainable Parameters	Builtin	Tucker
MNIST-2D	40	$10.4867$	$\mathbf{504.861}$
MNIST-3D	112	$9.2061$	$\mathbf{791.394}$

, we report the computational times (in seconds) needed for training each of the aforementioned models for the MNIST classification tasks. As becomes evident, when the Tucker convolution model is incorporated with DL platforms, it faces severe computational problems, although outside this framework, it is at least comparable to its competitors.

By observing the results of Table 1, we notice the nonscalable required computational times when the Tucker convolution model is incorporated into the DL framework. Based on our extensive experimental studies, we concluded that this handicap is mainly attributed to TensorFlow’s fast libraries implementation of convolution, which is performed across every input channel and for every desired output filter in a batch mode [38,39].

On the contrary, our implementation based on tensor decompositions has to make use of specific toolboxes/packages for the respective computations. Towards that end, we have considered the excellent review work of Psarras et al. [40], which surveys the current tensor software landscape. Based on the thorough lists provided therein, we mainly focused our efforts on Python-based Tensorly [41], which provides all tensor operations and decompositions required, as well as GPU computation capabilities. In particular, we designed our Tucker convolution layer either employing as backend Tensorflow (version: 2.3.0) [37], PyTorch (version: 1.9.0) [42], or NumPy (version: 1.18.5) [43]. Unfortunately, neither of our attempts fructified, even when we tried the MATLAB-based Tensor Toolbox package [44].

To circumvent the limitations imposed by GPU constraints, it becomes imperative to leverage Tensorflow’s foundational elements. Specifically, to fully harness the entirety of the data’s informational content, we present an N-dimensional (ND) convolutional layer/module that ingeniously employs the corresponding $(N-1)$D layer/module. Specifically, the computed value for a neuron in the convolved output, situated at coordinates $(i_1,\dots ,i_{N-1},i_N)$, is articulated as follows:

$$\begin{array}{cc}\hfill y_{i_1,\dots ,i_{N-1},i_N}=f(& \sum _c^{C_{in}}\sum _{j_N=0}^{J_N-1}\sum _{j_{N-1}=0}^{J_{N-1}-1}\dots \sum _{j_1=0}^{J_1-1}{w}_{j_1,\dots ,j_{N-1},j_N}{x}_{c,(i_1+j_1)(\dots )(i_{N-1}+j_{N-1})(i_N+j_N)}+b_{j_1,\dots ,j_{N-1},j_N})\hfill \\ \hfill =f(& \sum _{j_N=0}^{J_N-1}\left[\sum _c^{C_{in}}\sum _{j_{N-1}=0}^{J_{N-1}-1}\dots \sum _{j_1=0}^{J_1-1}{w}_{j_1,\dots ,j_{N-1},j_N}{x}_{c,(i_N+j_N)(i_{N-1}+j_{N-1})(\dots )(i_1+j_1)}\right]+b_{j_1,\dots ,j_{N-1},j_N})\hfill \\ \hfill =f(& \sum _{j_N=0}^{J_N-1}{C}_{(N-1)-D}+b_{j_1,\dots ,j_{N-1},j_N})\hfill \end{array}$$

(22)

where $f(.)$ is the activation function, $w_{j_1,\dots ,j_{N-1},j_N}$ stands for the value of the kernel connected to the current feature map at position $(j_1,\dots ,j_{N-1},j_N)$, $x_{c,(i_1+j_1)\dots (i_{N-1}+j_{N-1})(i_N+j_N)}$ represents the value of the input neuron at input channel $c,b_{j_1,\dots ,j_{N-1},j_N}$ is the bias of the computed feature map, $C_{in}$ denotes the number of original channels (i.e., first layer) or the number of feature maps of the previous layer (i.e., intermediate layer), and $j_1,\dots ,j_{N-1},j_N$ are the kernel’s dimensions across each of its modes.

The restructuring of the convolution sums as demonstrated in Equation (22) is attainable because convolution is inherently a linear operation, thus permitting the reordering of the summation processes. This adaptative strategy was previously outlined in sources such as [7,8,23,45], thus serving as a foundational pathway transitioning from 3D CNNs to more complex 4D variations. By embracing this methodology, we engage in the aggregation of numerous sequences of $(N-1)$D convolutions—represented by the term $C_{(N-1)-D}$ in (22)—extended along the final, or Nth, dimension.

From an execution standpoint, additional restructuring of the corresponding for-loop was enacted as per the frameworks suggested in [7,8,23]. This involved the convolution of $(N-1)$D input frames with their respective $(N-1)$D filter frames, which is a crucial adjustment for our custom layer to facilitate genuine (as opposed to separable) ND convolution. An exhaustive description of the respective Stacked convolution algorithm, applicable to tensors of up to 4 dimensions, is comprehensively articulated in Stacked Convolution Algorithm. The repository which contains the Python-scripts involving the implementation of ND CNNs can be found in the “Supplementary Materials” at the back matter heading after Section 5.

2.2. Proposed Simultaneous Spatiotemporal ECVs’ Forecasting Architecture

Within this section, our initial focus involves outlining the state-of-the-art model architecture intended for generalization in addressing the current challenge. Following this, we delineate the specific scenario under scrutiny for the ECVs’ forecasting task, thus elucidating the comprehensive mathematical elements detailing the application of the proposed 4D-ETCNs to address the task at hand. Concluding this section, we furnish implementation particulars and construct the corresponding network architecture.

2.2.1. Embedded Temporal Convolutional Network—ETCN

As previously discussed in Section 1, ETCNs [9] stand out as the leading models for the current task. These models employ a three-part architecture featuring an encoder network, a TCN, and a decoder network. To optimize the processing of the data’s temporal aspect, the authors implemented the Time-Distributed wrapper around 2D-CNNs within the encoder and decoder segments of the model. The encoder segment is structured with convolutional and max pooling layer blocks, whereas the decoder reverses this process by utilizing 2D transpose convolutional and batch normalization layers. The TCN component of the architecture is characterized by residual blocks that include 1D dilated causal convolutional layers, dropout layers, and ReLU activation layers. This particular configuration of causal convolution ensures that the model’s prediction at any specific time is solely based on past inputs.

2.2.2. 4D-ETCN for ECVs’ Forecasting Modeling

In addressing the task of forecasting time series ECVs, a problem inherently involving 4D inputs (comprising 2D spatial, temporal, and ECV modality dimensions), traditional 2D-CNNs fall short, as they inevitably overlook crucial information. Even 3D models can only maintain additional temporal or ECV modality information, not both. Recognizing this limitation, we present a novel 4D model designed to concurrently harness correlations across such diverse dimensions. Our focus is on the ETCN models, and we extend the state-of-the-art architectures initially proposed in [9] to suit the specific requirements of this multidimensional forecasting challenge.

As illustrated in Figure 4, the proposed 4D-ETCN model operates on the principle of efficiency, thus directly ingesting raw time series ECV data—essentially, sequences of 3D ECV data—that are devoid of any prerequisite data manipulation. These raw data are propelled through an intricate cascade of high-dimensional operations within the ETCN framework, including convolution, pooling, and upsampling, with each stage amplifying in sophistication as data traverse further into the network’s layers.

In parallel, the model meticulously conducts spatial downsampling followed by upsampling, which is a strategy that is pivotal for distilling more salient, feature-rich representations from the data, all while assiduously conserving the intricacies of temporal progression and ECV modality specifics up until the culmination point of the network’s regression compartment.

The proposed architecture stands out by virtue of its capacity to exploit the entirety of the high-dimensional data, thus steadfastly performing computations within the very dimensions native to the original data. This approach marks a significant departure from the methodologies employed by lower-order models, which typically impose a preprocessing stage that inevitably simplifies or compresses data ahead of any network training, thus potentially leading to the loss of valuable information.

2.2.3. 4D-ETCN Modules for ECVs’ Feature Learning

In Section 2.1.2, Section 2.1.3 and Section 2.1.4 we provided a detailed pipeline for the transition from conventional lower-order CNNs to the ND ones via tensor decompositions and stacked convolutions. Aiming to incorporate them to the ECV forecasting problem, we have to extend existing lower-order CNN models for that cause to their high-dimensional analogs. Since the vanilla ETCN model [9] consists of the state-of-the-art CNN architecture, we can directly make use of the results derived above in order to construct a higher-order ETCN model. More precisely, the ECV forecasting problem at hand can be considered as a subcase of the ND general case, where the dimensionality of the data is equal to $N=4$ (once the time series ECV data form a 4D tensor). In that sense, we can directly extend all lower-order convolutional layers contained in vanilla ETCN architecture [9] with 4D ones.

Beyond the convolutional layers, it is also necessary to expand the max pooling and transpose convolutional layers/modules referenced in [9] to accommodate four-dimensional functionality. The justification for this enhancement aligns with the underlying logic for the convolutional layer’s expansion, which is comprehensively explained in [7]. By integrating these 4D layers, we equip ourselves with the essential components to elevate the standard lower-dimensional ETCN model to a high-dimensional iteration. Consequently, our advanced 4D-ETCN framework is capable of processing time series ECV data in their original dimensions, thus adeptly navigating the complex demands of the concurrent forecasting challenge presented. This approach not only maintains the integrity of the data’s dimensions but also enhances the model’s capacity to interpret and learn from the high-dimensional nature of the input.

2.2.4. 4D-ETCN Architectures for ECVs’ Forecasting

The primary objective of this study is to evaluate the efficacy of ETCN models in addressing the specified problem and to gain insights into the potential advantages of the proposed 4D structure over its lower-dimensional counterparts. To this end, we use the standard ETCN [9] as a reference model for comparison while developing its 4D equivalent. Our analysis concentrates on ETCN frameworks given their established superiority in forecasting tasks within the ETCN context [9], thus even surpassing the performance of Conv-LSTM structures that are specifically crafted for such challenges.

To explore the potential benefits of our advanced ETCN model compared to the standard version, our architecture retains the fundamental components (namely, the encoder, TCN, and decoder) but only expands those elements amenable to higher-dimensional processing units (i.e., the encoder and decoder). This approach ensures consistency for a fair comparison while allowing us to directly assess the impact and advantages of high-dimensional processing in forecasting tasks.

Each architecture consists of multiple layers, including convolutional, transpose convolutional, max pooling, activation, dropout, and batch normalization layers. These are arranged in “layers-stacks” within the encoder, TCN, and decoder portions of the model, thus adhering to the framework put forth in the conventional ETCN [9]. Specifically, the encoder and the TCN sections are each composed of three stacks, while the decoder section is constructed using two stacks.

Each stack within the encoder includes a convolutional layer, which is directly succeeded by a ReLU activation layer and a max pooling layer, with the exception of the final stack. Concluding the sequence of stacks in the encoder segment involves a reshape layer, which is tasked with reorganizing the data to be compatible with the subsequent 1D-TCN segment of the model. This systematic layering is instrumental in the progressive feature extraction and data transformation essential for the model’s forecasting tasks.

The TCN segment of the model, drawing inspiration from the framework presented in [26], substitutes standard 1D convolutional layers with residual blocks. Each of these blocks is composed of two 1D convolutional layers that maintain uniformity in terms of output filter size and number. Following each convolutional layer are ReLU activation and dropout layers, with weight normalization [46] applied to the convolutional filters. Unchangingly, the input for each residual block within the TCN segment is combined with its output; this sum, after undergoing ReLU activation, produces the final output for the residual block.

Consistent with the vanilla ETCN model’s TCN [9], our model employs causal and dilated 1D convolutions, thus ensuring that the output at any given time is predicated solely on present and preceding inputs. Since the TCN output is a one-dimensional vector, matching in dimensions the number of output filters of the last residual block, we reintroduce a reshape layer that transforms the 1D vectors to 4D tensors, thus rendering them suitable for the decoder segment of the model. This methodical approach facilitates the nuanced processing required for accurate temporal predictions.

Conversely, each stack within the decoder segment is composed of a transpose convolutional layer, which is succeeded by ReLU activation and batch normalization layers. Culminating the architecture, the prediction layer incorporates a single-filter convolutional layer for generating the requisite 3D output. This layer is then followed by a Sigmoid activation layer. This configuration within the decoder segment is crucial for reconstructing the spatial–temporal features from the compressed representations and predicting the continuous values pertinent to the environmental conditions under observation.

Given that our samples inherently possess four dimensions, encompassing two spatial, one temporal, and one ECV modality dimension, designated as $(H,W,T,EC{V}_M)$, their utilization for training within the confines of a vanilla ETCN model necessitates a form of dimensional reduction or “collapsing” for each sample. While the original ETCN model was evaluated on individual ECVs, assertions from [46] suggest its applicability to multichannel images. Leveraging this, we adapted their architecture to handle the “active information” within the ECV time series data (i.e., spatial and temporal dimensions), thus interpreting the ECV modality as distinct channel information. In stark contrast, our innovatively proposed 4D-ETCN model operates directly within the data’s intrinsic dimensions, thereby fully engaging with the complete gamut of information present. This approach eases a more holistic and integrated exploitation of the data’s spatiotemporal dynamics.

The encoder and decoder segments of the devised ETCN model can be construed as U-Nets, which are predominantly employed in segmentation tasks. In conventional U-Net models, both samples and labels typically share the same dimensions, i.e., 2D for image segmentation [47], and 3D for volumetric image segmentation [48]). Unlike existing approaches, the proposed high-dimensional model confronts the challenge of disparate dimensionality between samples (4D) and targets (3D).

To navigate this complexity while retaining maximal information, we adopt a tactic akin to the one outlined in [7]: all operations within the ETCN model are executed in their corresponding high-dimensional spaces up until the final regression layer. At this juncture, we utilize filters with a kernel size of 1 across all dimensions, thus maintaining unit strides across every dimension barring the temporal one slated for reduction (i.e., $(1,1,T,1)$). This method ensures that the supplemental, nonspatial information remains unaltered until the immediate prelude to regression, wherein it is judiciously downsampled during the ultimate step, thereby aligning the network training with the 3D ground-truth targets.

In relation to the foundational topology of the 4D-ETCN, all convolutional layers utilize a modest kernel size of 2 across every dimension combined with “same” padding. Strides are set to unity across all dimensions during all convolutions. Concurrently, the quantity of filters experiences an augmentation as the network delves deeper into the encoder segment of the ETCN: The initial stack of layers possesses a filter count equivalent to a predefined parameter (termed “starting-neurons”), which experiences a doubling for the subsequent stacks (for instance, both the second and third stacks possess filters numbering $2\times$ “starting-neurons”).

In line with our goal to perform spatial downsampling solely on input data samples, we employ max pooling layers with pool size and strides equal to 2 in spatial dimensions and 1 in the temporal and the ECV modality dimensions. “Same” padding is consistently applied across all max pooling layers, and dropout layers are established with a rate of $0.3$.

For the transpose convolutional layers, we utilize compact kernels with a size of 2 across all dimensions, thus consistently applying “same” padding for all versions of the 4D-ETCN models. Due to the unique spatial downsampling in the encoder section, strides for the transpose convolutions are appropriately modified, which are designated as 2 for spatial dimensions (to accommodate the corresponding upsampling) and 1 for the other dimensions.

Regarding the TCN segment of our model, the fundamental topology incorporates 1D filters of size 2, with a filter count of $2\times$ “starting-neurons” at the first and third stack and 64 at the second. The dilation rate of the 1D kernels within the TCN’s stacks adheres to the protocol proposed in the vanilla ETCN setup [9] and is sequentially arranged at $1-2-4$.

And indicative sketch of our 4D-ETCN model architecture can be seen in Figure 5. As described above, the architecture consists of three parts: encoder (Figure 5a), TCN (Figure 5b), and decoder (Figure 5c), which are combined in the way explained earlier in the quest to derive accurate ECV forecasts.

Taking into account that we face a regression task, MSE was used as the loss function. All examined models’ weights were randomly initialized using the Glorot Uniform initializer [49]. Concerning the optimization learning algorithm, we employed the Adam scheme [50] with a constant learning rate of $0.0001$ and exponential decay rates for the first and second moment estimates equal to ${\beta }_1=0.9$ and ${\beta }_2=0.999$, respectively. Training of each below model was performed using a batch size of 4 samples (since the loading of high-dimensional data can become quite memory intensive) for up to 100 epochs.

3. Results

In this section, our first task is to outline the datasets introduced for our imminent experiments, followed by an explanation of the experimental framework implemented. Next, we establish the evaluation criteria used to compare various DL models. Subsequently, we delve into a comprehensive analysis of the results derived from our chosen approach across diverse experimental scenarios, thus utilizing multiple configurations to meticulously assess the impact of each pertinent parameter throughout the entire ECV forecasting process. Concluding, we juxtapose our proposed methodology against prevailing state-of-the-art techniques, thus affirming its superior efficacy.

3.1. Dataset Description

The evaluation of DL strategies and the cultivation of ML models pivot significantly on the accessibility of appropriate training and test sets. This becomes particularly challenging in the realm of high-dimensional data, where relevant datasets are notably limited. In our study, we gauged the effectiveness of the suggested system through its precision in estimating ECVs. In particular, our efforts were concentrated on training both the proposed model and established methodologies using historical observational time series with the objective of learning to forecast forthcoming values of soil temperature at various depths.

We utilized data that are openly accessible from the ERA5-Land Monthly Averaged dataset (https://developers.google.com/earth-engine/datasets/catalog/ECMWF_ERA5_LAND_MONTHLY) (accessed on 1 February 2024) [51]. ERA5-Land is a reanalysis dataset that offers a coherent perspective on the historical progression of terrestrial variables, spanning multiple decades, and does so with improved resolution compared to its ERA5 counterpart. By offering data that extend back over several decades, reanalysis furnishes a detailed account of historical climate conditions. This dataset encompasses all 50 variables available on the Climate Data Store (CDS). The ERA5-Land data, available since 1981, come with precomputed monthly mean averages, thus enhancing the ease and speed of data access for various applications.

For the purposes of our research, we focused on soil temperature variables, specifically levels 1–4. These levels represent the temperature strata within the soil, specifically layer 1 (0–7 cm), layer 2 (7–28 cm), layer 3 (28–100 cm), and layer 4 (100–289 cm) of the ECMWF Integrated Forecasting System. The surface reference point is at 0 cm, and each of these four climatic variables is gauged on the Kelvin scale. The soil temperature is determined at the midpoint of each layer, with heat transfer computations performed at the layers’ interfaces. The model presupposes the absence of heat transfer beneath the base of the deepest layer.

In order to employ the aforementioned dataset, some preprocessing steps had to take place in advance. More precisely, our goal was to create a 4D imagery time series dataset comprising each of the 2D imagery of each climate variable measured at all available sampling times available. Towards that end, we made use of the Google Earth Engine platform [52], with dataset provider Copernicus CDS and dataset availability 1 January 1981–1 December 2020. With this setup, we initially selected our region of interest for downloading our time series imagery at a ≈9 km resolution provided by the platform. Subsequently, we selected the four different climate variables mentioned earlier for downloading, which were accompanied by the desired dimensions of the imagery. At this point, we have to mention that since the high-dimensional models’ memory demands are in direct alignment with their input data size, we chose the spatial dimensions to be patches of size $32\times 32$. In addition, given that the samples were taken on a monthly basis starting from January 1981 up to December 2020, the temporal dimension of our created time series data is equal to $12\times 40=480$. The last dimension of our data is considered the climate variable under investigation, and hence its cardinality is equal to four. Since every climate variable is measured at Kelvin scale, we normalized each one of them separately in the [0–255] scale in order to be interpreted as 8-bit images from the DL models. Before feeding the input to the network, the pixel values were further scaled to the range from 0 to 1 toward enhancing the efficiency of the imminent training process.

To ensure that our study’s findings are not confined to a specific dataset, we evaluated the performance of the proposed methodologies and those for comparison on two distinct datasets. Specifically, the first Region Of Interest (ROI) chosen for our investigation is the province/island of Crete, which is situated in Greece. Concurrently, the second encompasses various territories in Central Italy predominantly concentrated around the Lazio region. Concerning the Crete ROI, its latitude and longitude limits, $[latlim,lonlim]$, are specified as two-element vectors of the form $[latmin,latmax]=[34.7702,35.7035]$ and $[lonmin,lonmax]=[23.385,26.5106]$. As far as the Italy ROI is concerned, the respective limits are formed as $[latmin,latmax]=[41.570252,42.504503]$ and $[lonmin,lonmax]=[12.101440,15.806824]$, respectively. Both selected areas are characterized by a rich diversity in land surfaces, thus featuring an assortment of plains, forests, and mountainous terrains.

3.2. Experimental Setup

In this study, our primary aim is to execute a one-step forward prediction by developing a model that ingests a set quantity of images from a spatiotemporal series of multiple climate variables and forecasts an identical count of images, with each displaced one timestep ahead. To achieve this, it is imperative to pre-establish the number of timesteps employed in this predictive task, thus denoting our data’s temporal dimension. We opted for a timestep of six months, which is a duration sufficient to discern potential disparities among the scrutinized climate variables. By extracting sequential time series segments with overlap, 473 unique samples (with the final month of 2020 excluded), each with dimensions of (32, 32, 6, 4), were produced from the forty-year span of the monthly data available.

Since the problem at hand is a forecasting task, we selected the first 38 years (i.e., 1981–2018) for training purposes, while the 2 most recent years (i.e., 2019–2020) were kept for testing (in a holdout split). Furthermore, we considered a validation set formed by $25\%$ of the training samples, which is a split that was performed uniformly at random to avoid bias in favor/against specific time periods. The data splitting process ended up with a sample distribution summarized in

Table 2. Dataset split among training–validation–test sets. The first 38 years are used for training ($75\%$) and validation ($25\%$) purposes, while the last 2 ones are used for testing.

Total Samples	Training Set	Validation Set	Test Set
473	342	114	17

.

In the forthcoming experiments, we adhered to the dataset splits previously mentioned. Each model underwent training within the designated training set, while its performance underwent validation in the validation set through the exploration of various hyperparameter configurations. Subsequently, the model’s performance was evaluated based on the optimal validation loss achieved throughout training, regardless of whether this occurred at the final epoch. The models exhibiting the best performance were then used in the test set for a single evaluation, thus determining the ultimate performance of the model.

The formulated model structures were crafted within the Python programming environment, thus utilizing the TensorFlow and Keras [53] libraries. This choice was deliberate, given their extensive customization capabilities throughout various aspects of a DL model—an essential feature for our study, given the implementation of diverse custom layers. Additionally, the compatibility of both libraries with GPU acceleration significantly reduced the computational time required for the training process. In our experimental setup, we employed NVIDIA’s Quadro P4000 GPU model, equipped with 8 GB of RAM, to further enhance computational efficiency.

3.3. Evaluation Metrics for Essential Climate Variables Forecasting

Since the problem at hand is essentially a pixel-level regression task, we relied on several ML metrics for the evaluation of the designed models. For that cause, we employed reconstruction error metrics, image quantitative metrics, and correlation metrics. For their definitions, let $x_i$ stand for the pixels of the ground truth image x, while $y_i$ stands for the pixels of the predicted image y (n in total).

Concerning the reconstruction error metrics, the Mean Square Error (MSE), the Mean Absolute Error (MAE), and the Root Mean Square Error (RMSE) between x and y are defined as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}MSE=\frac{1}{n}\sum _{i=1}^n{(x_i-y_i)}^2\hfill \\ MAE=\frac{1}{n}\sum _{i=1}^n\mid x_i-y_i\mid \hfill \\ RMSE=\sqrt{\frac{1}{n}{\sum }_{i=1}^n{(x_i-y_i)}^2}\hfill \end{array}\right.\end{array}$$

(23)

However, if there are biases in either the mean or the amplitude of fluctuations of the predictions, the RMSE could be severely compromised. To alleviate this, the mean bias can be removed from the RMSE, thus leading to the Unbiased Root Mean Square Error (ubRMSE), which is computed as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}Bias=\frac{1}{n}\sum _{i=1}^n(x_i-y_i)\hfill \\ ubRMSE=\sqrt{RMS{E}^2-Bia{s}^2}\hfill \end{array}\right.\end{array}$$

(24)

For the above reconstruction error metrics, values closer to zero indicate a better-performing model.

Since pixel-level regression can be seen as an image reconstruction task, reconstruction error metrics are by no means the only suitable ones to be measured. Indeed, among image fidelity metrics, we initially reported the Peak Signal-to-Noise Ratio (PSNR) metric, which is defined as follows:

$$\begin{array}{c}\hfill PSNR=10{log}_{10}\left(\frac{MA{X}_I^2}{MSE}\right)\end{array}$$

(25)

In Equation (25), $MA{X}_I^2$ is the maximum possible pixel value in the image (e.g., for images scaled in the range $[0,1]$ $MA{X}_I^2=1$, while for 8-bit images, $MA{X}_I^2=255$), while $MSE$ is defined as in Equation (23). Higher PSNR values indicate better image fidelity.

Moreover, the Structural Similarity Index Measure (SSIM) [54] is a perception-based metric that is mainly used for measuring the similarity between two images. The SSIM extracts structure, luminance and contrast from both images x and y, and its values range between 0 and 1 (with higher values desired). Formally, the SSIM is defined as follows:

$$\begin{array}{c}\hfill SSIM=\frac{(2{\mu }_x{\mu }_y+c_1)(2{\sigma }_{xy}+c_2)}{({\mu }_x^2+{\mu }_y^2+c_1)({\sigma }_x^2+{\sigma }_y^2+c_2)}\end{array}$$

(26)

where ${\mu }_x$ stands for the mean of ground truth image x, ${\mu }_y$ stands for the mean of predicted image y, ${\sigma }_x^2$ and ${\sigma }_y^2$ represent the respective variances, and ${\sigma }_{xy}$ is the covariance between them. In addition, $c_1={\left(k_{1}D\right)}^2$ and $c_2={\left(k_{2}D\right)}^2$ are two variables used to stabilize the division with weak denominator, D is the dynamic range of the pixel values (i.e., $D=1$ for images scaled in $[0,1]$, while $D=255$ for 8-bit images), and finally $k_1$ and $k_2$ are by default set as $k_1=0.01$ and $k_2=0.03$.

Treating ground truth and predicted images as two statistical samples, the Pearson Linear Correlation Coefficient (PLCC) is a measure of their linear correlation. The PLCC is essentially a normalized measurement of the covariance such that the result always lies in $[-1,1]$, and, as with covariance itself, the PLCC only reflects a linear correlation of variables. Mathematically, the PLCC is defined as follows:

$$\begin{array}{c}\hfill PLCC=\frac{{\sum }_{i=1}^n(x_i-{\mu }_x)(y_i-{\mu }_y)}{\sqrt{{\sum }_{i=1}^n{(x_i-{\mu }_x)}^2}\sqrt{{\sum }_{i=1}^n{(y_i-{\mu }_y)}^2}}\end{array}$$

(27)

A value of $PLCC=1$ implies that the pixels of the ground truth and predicted images are perfectly positive linearly correlated, while a $PLCC=-1$ implies total negative linear correlation. Values of the PLCC close to 0 indicate that the two images under consideration are not linearly correlated.

Finally, apart from the aforementioned metrics, for each of the trained models presented below, we reported the time for its training and testing/inference as metrics of its computational complexity. The former one refers to the time required for optimizing the model’s weights/parameters throughout the training process, while the latter one quantifies the time required for the model to operate in “production mode” (i.e., derive predictions) several times once the training process has concluded.

3.4. 4D-ETCN Architecture Ablation Study

To obtain a better understanding of different model choices, we explored the impact of different hyperparameters of the proposed 4D-ETCN model in terms of performance. More specifically, we focused on different hyperparameters, including the size and number of filters on 4D convolutional layers for the encoder and the decoder, as well as for the 1D kernels of the TCN. For that cause, we set as a basic architecture one with a filter size at the encoder and decoder equal to $(2,2,2,2)$, a dropout rate equal to $0.3$, and the size of the 1D kernel of TCN was set to 2. With the goal of pinpointing the optimal hyperparameter setup, we undertook a comprehensive ablation study. This involved scrutinizing each hyperparameter in isolation, as well as in various combinations, to guide our selection of the most effective model. In the ensuing experiments, we report the results from the two examined datasets, namely the Crete, Greece and Italy ROIs. Our primary focus was on the model’s MSE loss metric, thus serving as the cornerstone for our ultimate choice of model.

Due to space limitations, the analytical experimental results of this ablation study are provided in Appendix C. As can be seen from the result tables therein, the two datasets requires quite different hyperparameter configurations for achieving optimal results. In general though, adopting architectures with more filters in the encoder part of the network leads to clear performance gains. In addition, wider receptive fields both at the encoder and the decoder parts of the architecture contributed positively to its performance, irrespective of the dataset at hand.

3.5. Convergence Analysis

Figure 6 showcases the convergence trajectory of our proposed 4D-ETCN architecture for both examined datasets. The convergence is illustrated primarily through the regression loss (i.e., MSE) plotted against the number of epochs, thus encompassing both training and validation sets. It is crucial to note that we employed the finest hyperparameters for the suggested 4D models, which were derived from the corresponding tables furnished in Appendix C. We discerned a progressive enhancement in the performance of the 4D-ETCN model coupled with an effective mitigation of overfitting. This was evidenced by the validation curve closely mirroring the training curve, thus signifying robust generalization potential. Another interesting remark is that this behavior was observed for both datasets, thus indicating that the good generalization capabiblity of the proposed approach was dataset-independent.

3.6. Evaluation against State-of-the-Art Methods

Beyond refining the proposed 4D-ETCN architecture delineated in Section 3.4, we undertook a comparative analysis between our advanced high-dimensional model and prevailing cutting-edge methods in the discipline. To ensure an equitable comparison, we selected the relatively novel method advanced by Villia et al. in [9] (previously discussed in Section 1, Section 2.1.1, Section 2.1.2, Section 2.1.3, Section 2.1.4 and Section 2.2.1), thus serving as the foundational architecture that our proposition enhances. More specifically, the authors introduced the concept of ETCN for the problem of forecasting ECVs, thus coping primarily with land surface temperature and soil moisture. In that attempt, they performed an analogous ablation study and derived the best-performing model for the task, which makes use of 2D convolutional kernels both in the encoder and decoder parts. Since on their experimental scenarios were not contained within a temporal window equal to 6 months, we therein considered their best-performing model for 12 months, which mainly consisted of 2D filters of size $(4,4)$, the number of filters in the encoder part equal to 32–64–64, a dropout rate set to $0.3$, and the size of the 1D kernel at the TCN part of three. In addition, since in [9], the input of the time series was imagery of spatial size $28\times 28$ rather than $32\times 32$ in our case, we modified accordingly the number of filters in the decoder part to be 64 rather in 49 and all convolutions to “same” padding mode to match the current experimental setup. Moreover, both the encoder and the decoder parts of the network in [9] were wrapped around by the Time-Distributed Tensorflow’s layer in order to fully exploit the temporal information available in the data. However, in our setup, the ETCN architecture proposed in [9] forecasts all four ECVs jointly at once and not separately in their vanilla setup.

Apart from the aforementioned state-of-the-art method described in [9], we additionally compared our proposed high-dimensional method with a 2D Conv-LSTM [19] architecture also proposed in [9] as a baseline. It consists of two layers with 64 filters per layer and a kernel size equal to $(3,3)$ for both layers. Its output layer is a 3D convolutional layer of kernel size $(3,3,3)$, which maps to the desired output.

Table 3. The proposed higher-dimensional 4D-ETCN model surpasses its counterparts Conv-LSTM and ETCN, both cited in [9] across several evaluated metrics, including reconstruction error, image fidelity, and correlation.

Dataset	Model	MSE	MAE	SSIM	PSNR	PLCC	ubRMSE	MAE (K)
Crete	Conv-LSTM [9]	$0.00154561$	$0.0201477$	$0.928759$	$28.2473$	$0.991675$	$0.0369692$	$0.380813$
	ETCN [9]	$0.00115819$	$0.018624$	$0.952923$	$29.4991$	$0.9939$	$0.0326225$	$\mathbf{0.348874}$
	4D-ETCN-Proposed	$\mathbf{0.00110941}$	$\mathbf{0.0179513}$	$\mathbf{0.959973}$	$\mathbf{30.7597}$	$\mathbf{0.994405}$	$\mathbf{0.0298117}$	$0.377492$
Italy	Conv-LSTM [9]	$0.00287025$	$0.0382445$	$0.840449$	$25.5313$	$0.977507$	$0.0861671$	$1.05962$
	ETCN [9]	$0.00436746$	$0.0527445$	$0.882363$	$23.7782$	$0.987332$	$0.0442294$	$0.940987$
	4D-ETCN-Proposed	$\mathbf{0.00181847}$	$\mathbf{0.031803}$	$\mathbf{0.932413}$	$\mathbf{29.1255}$	$\mathbf{0.990558}$	$\mathbf{0.0329555}$	$\mathbf{0.848434}$

reports the obtained results in the test set for the proposed model architecture as emerged from the ablation study of Section 3.4, as well as its state-of-the-art competing method described above in both under examination datasets. We observe that the proposed high-dimensional model outperformed its competitors in nearly all reported metrics, both in the Crete and in Italy dataset. Moreover, the ETCN model [9] outperformed the Conv-LSTM [9] one in the Crete dataset, while the two models had comparable performance in the Italy dataset. In addition, all methods performed better in the Crete dataset than in the Italy one, but in this toughest case, the proposed 4D-ETCN model widened even further the performance gap from its competitors. Moreover, concerning the unbiased RMSE and MAE measured in nominal Kelvin values (i.e., by converting the predictions back to their prenormalization values) metrics, we observe that once more our proposed higher-order model outperformed its competitors in nearly all cases. Interpreted from a climatological context point of view, our method’s predictions were on average ≈0.37 and ≈0.84 Kelvins away from the ground truth values in the two examined datasets, thus indicating its enhanced performance.

Of course, higher-dimensional models and operations in the respective spaces require more parameters to be learned, thus leading to an increase in the time needed to train the network, as dictated by the reported results in

Table 4. The proposed higher-dimensional 4D-ETCN model’s superior performance comes at the cost of increased computational demands in terms of training, while its inference runtime for deriving predictions is negligible.

Dataset	Model	# Parameters	Time Training (Minutes)	Time Testing/Inference (Seconds)
Crete	Conv-LSTM [9]	459,012	$17.6605$	$\mathbf{0.5}$
	ETCN [9]	$\mathbf{336,874}$	$\mathbf{3.30542}$	$\mathbf{0.5}$
	4D-ETCN-Proposed	34,599,367	$116.909$	1
Italy	Conv-LSTM [9]	459,012	$17.9783$	$\mathbf{0.5}$
	ETCN [9]	$\mathbf{336,874}$	$\mathbf{3.44944}$	$\mathbf{0.5}$
	4D-ETCN-Proposed	47,104,583	$181.733$	1

. Nevertheless, the required training times of ≈2 and ≈3 h for the Crete and Italy datasets, respectively, are surely not prohibitive, especially if more powerful hardware than the employed one is available. The most important thing to note here is that although the training times may vary among the involved models, the respective test/inference ones are nearly identical. This observation is of particular interest, thus taking into account that once the models are trained (a process which is performed only once), they can be employed in “production mode” with nearly the same minimal cost. Consequently, our superior higher-order model can derive enhanced predictions as fast as its competitors in a matter of few seconds.

To provide a visual depiction of the proposed model’s efficacy, Figure 7 presents the actual values of the four distinct ECVs being studied (namely, soil temperature levels 1 through 4) for the specific test sample month of November 2020. This was juxtaposed with predictions derived from our advanced 4D model and the runner-up method, the ETCN [9]. The regression maps featured in Figure 7 indicate that our 4D-ETCN model more accurately discerned the majority of the true values across both datasets, thus resulting in distinctly delineated regions. It is also evident that the ETCN [9] model exhibited substantial “noise effects”, particularly within the Italian dataset (i.e., Figure 7a–e), due to its inability to accurately forecast large segments of each test sample. Conversely, the proposed 4D-ETCN model excelled in mirroring most of the values present in the actual regression maps thanks to its inherent capability to grasp high-dimensional interrelations among the variables, thereby closely reflecting the ground truth maps.

4. Discussion

Within this section, we delve into the outcomes detailed in Section 3. Our discussion unfolds systematically, thus exploring the models’ performance in relation to the extent of accessible training data and aiming to quantify the specific requirements of each model for accurate ECVs’ forecasting. Furthermore, an examination is conducted on the efficacy of the proposed higher-order ETCN alongside its competitors, particularly in scenarios with increased temporal information. This analysis aims to illuminate the respective strengths and limitations inherent in each examined model, thus ensuring a comprehensive and lucid comparison.

4.1. Impact of Training Set Size on Forecasting Accuracy

To assess the efficiency of the proposed 4D-ETCN model concerning its dependency on the volume of training data available, we conducted tests using just $66\%$ and $33\%$ of the original training samples, thereby decreasing them from 342 to 228 and 114, respectively. The number of validation and test samples remained unchanged (114 and 17, respectively). We then proceeded to train the optimally configured models, as identified in Section 3.4, using the reduced training dataset sizes for both scenarios and datasets, alongside their leading contemporary counterparts. The outcomes are consolidated in Figure 8 for each dataset under scrutiny.

We noted that the decrement in training samples marginally impacted the efficacy of the proposed 4D-ETCN model, which managed to maintain similar performance levels even when furnished with merely $33\%$ of the total training samples. Conversely, the performance of the Conv-LSTM [9] and ETCN [9] models was noticeably enhanced across all visual perception metrics when provided with a larger pool of training samples. Overall, the 4D-ETCN model consistently surpassed its rivals across all tested scenarios and datasets, thus underscoring the resilience and reliability of the proposed methodology.

4.2. Effect of Temporal Information on Forecasting Accuracy

In a final set of experiments, we aimed to quantify the effect of the temporal information on the performance of the forecasting models. Throughout the present study, we employed samples containing information at a half-year level (i.e., the temporal size of the data was equal to 6 months). We doubled this temporal window information to 1 year (i.e., the temporal data was now set to 12 months) in an attempt to explore how the models adapted to such an experimental scenario, while the training and validation sets remained the same (in terms of selected samples). On the contrary, as the test set was by assumption/construction focused on the last 2 available years 2019–2020, the number of test samples was reduced from 17 to 11 as a direct consequence of the whole year 2019 being used for forecasting the values of year 2020. With this setup, we performed exactly the same analysis that led to Section 4.1 and quantified the performance of all models in both under investigation dataset for all different numbers of training samples being available to them. The results are depicted in Figure 9, where we observe that once more the proposed 4D architecture outperformed its competing ones in nearly every case. We should notice though that although in the Crete dataset its dominance was clear in every experimental scenario, in the Italy dataset, when $66\%$ of the whole training samples were available, the Conv-LSTM [9] model seemed to question its supremacy. However, its performance throughout all these examined cases indicates that it clearly consists of the best alternative among its competitors.

5. Conclusions

In this study, we introduced a novel DL architecture, the 4D-ETCN, tailored for simultaneous forecasting of multiple ECVs. This methodology expands on the foundations of cutting-edge ETCN models, thus transitioning them into their high-dimensional counterparts by adapting every aspect of their functionalities across all paths. Furthermore, we embarked on the task of creating, experimenting, and assessing diverse network configurations using actual data to pinpoint the most effective structure for the given challenge.

Our empirical investigations, conducted on two meticulously curated datasets, have substantiated that superior forecasting accuracy, relative to traditional and contemporary strategies, can be achieved by executing feature learning within the data’s inherent domain. At the same time, its computational cost for processing data in higher dimensions is quite small during inference stage once the model is trained, while there is also room for further improvement both in terms of hardware and implementation aspects, as dictated in the respective sections. This advancement underscores the potential of our proposed model in enhancing the precision of ECVs’ predictions.