A Two-Level Parallel Incremental Tensor Tucker Decomposition Method with Multi-Mode Growth (TPITTD-MG)

Abstract

With the rapid growth of streaming data, traditional tensor decomposition methods can hardly handle real-time, high-dimensional data of massive amounts in this scenario. In this paper, a two-level parallel incremental tensor Tucker decomposition method with multi-mode growth (TPITTD-MG) is proposed to address the low parallelism issue of the existing Tucker decomposition methods on large-scale, high-dimensional, dynamically growing data. TPITTD-MG involves two mechanisms, i.e., a parallel sub-tensor partitioning mechanism based on the dynamic programming (PSTPA-DP) and a two-level parallel update method for projection matrices and core tensors. The former can count the non-zero elements in a parallel manner and use dynamic programming to partition sub-tensors, which ensures more uniform task allocation. The latter updates the projection matrices or the core tensors by implementing the first level of parallel updates based on the parallel MTTKRP calculation strategy, followed by the second level of parallel updates of different projection matrices or tensors independently based on different classification of sub-tensors. The experimental results show that execution efficiency is improved by nearly 400% and the uniformity of partition results is improved by more than 20% when the data scale reaches an order of magnitude of tens of millions with a parallelism degree of 4, compared with existing algorithms. For third-order tensors, compared with the single-layer update algorithm, execution efficiency is improved by nearly 300%.

1. Introduction

During the past decades, the landscape of computational mathematics has been dramatically changed by data revolution in the era of Big Data, which features 4V’s [1] (i.e., high volume, high velocity, high veracity and high variety) and stimulates the demand for new linear algebra tools to cope with massive amounts of complex data. Among them, a major challenge is how to model high-dimensional data without losing their inherent multi-linear structure. In other words, a provably optimal representation of high-dimensional data is crucial to offer both theoretical and practical foundations for other tasks [2]. In this sense, canonical data representation in the form of vectors or matrices is no longer feasible, because vectorization or matricization (i.e., unfolding of multi-dimensional data into vectors or matrices [3]) for processing may lead to sub-optimal performance due to a loss of relationships across various dimensions [4,5]. However, in areas flooded with high-dimensional data, such as fluid mechanics, electrodynamics and general relativity, etc., concise mathematical frameworks can be successfully constructed for formulating and solving problems by means of tensors.

A tensor refers to an algebraic object that describes a multi-linear correlation between sets of algebraic objects. Mathematically, any tensor of the type $(r,s)$ is represented by a multi-dimensional array with definite numbers of upper indices r and lower indices s to indicate an r-times contravariance and an s-times covariance, respectively. This term has a clear geometrical significance and can be formally defined based on the following general transformation formulas [6,7,8]:

$$T=T_{j_1\cdots i_r}^{i_1\cdots i_r}{g}_{i_1}\cdots g_{i_r}{g}^{j_1}\cdots g^{j_s}=T_{j_1^{\prime }\cdots j_s^{\prime }}^{i_1^{\prime }\cdots i_r^{\prime }}{g}_{i_1^{\prime }}\cdots g_{i_r^{\prime }}{g}^{j_1^{\prime }}\cdots g^{j_s^{\prime }}$$

(1)

where $g_i\left(g^j\right)$ and $g_{i^{\prime }}\left(g^{j^{\prime }}\right)$ represent covariant and contravariant base vectors in coordinate systems $\left\{x^i\right\}$ and $\left\{x^{i^{\prime }}\right\}$, respectively. In order to make this notoriously impenetrable concept easy to be understood, Ballard et al. presented the Chicago crime data (available at www.cityofchicago.org) as an example to be represented by a tensor, whose modes correspond to 365 days, 24 h, 77 communities, and 11 crime types. Entry $\chi (i,j,k,l)$ is the number of times that crime l happened in the neighborhood k during hour j on day i, as shown in Figure 1 [9].

Tensor-based approaches have attracted more and more attention due to their capacity of exploiting multi-linear relationships in multi-way data, among which tensor decomposition (TD) is the art of disassembling multi-dimensional arrays into smaller parts and finds ubiquitous applications in machine learning, neuroscience, quantum computing, signal processing, etc. [9]. Recently, TD has been employed to compress deep convolutional neural networks (DCNN), i.e., reducing the number of parameters and the time of training a model from scratch, with the purpose of accelerating DCNN’s [10,11].

1.1. A Brief Introduction to Tucker Decomposition

In 1963, Tucker [12] proposed the well-known Tucker decomposition, which represents any d-dimensional tensor $\chi \in {\mathcal{R}}^{n_1\times \cdots \times n_d}$ as a contraction between a d-dimensional core tensor $\mathcal{G}\in {\mathcal{R}}^{r_1\times \cdots \times r_d}$ and d projection matrices ${\mathit{U}}_{\left(\mathit{i}\right)}\in {\mathcal{R}}^{n_i\times r_i}(i=1,2,\cdots ,d)$:

$$\chi \approx \mathcal{G}\times \left[{\mathit{U}}_{\mathbf{1}},\cdots ,{\mathit{U}}_{\mathit{d}}\right]=\mathcal{G}{\times }_1{\mathit{U}}_{\left(\mathbf{1}\right)}{\times }_2{\mathit{U}}_{\left(\mathbf{2}\right)}\times \cdots {\times }_d{\mathit{U}}_{\left(\mathit{d}\right)}=\mathcal{G}\times \left\{{\mathit{U}}_{\left(\mathit{n}\right)}\right\}.$$

(2)

Projection matrices are typically orthogonal and serve as the principal component of the corresponding mode. Figure 2 presents a comparison between the principles of the CP decomposition and that of the Tucker decomposition, revealing the correlation between them.

For a hyperspectral image, each projection matrix represents a different feature. For example, ${\mathit{U}}_{\left(\mathbf{1}\right)}$ and ${\mathit{U}}_{\left(\mathbf{2}\right)}$ correspond to spatial features, and ${\mathit{U}}_{\left(\mathbf{3}\right)}$ represents the spectral feature. The corresponding third-order tensor $\chi \in {\mathcal{R}}^{I\times J\times K}$ can be decomposed as follows:

$$\chi \approx \Lambda {\times }_1\mathit{A}{\times }_2\mathit{B}{\times }_3\mathit{C},$$

(3)

where $\Lambda \in {\mathcal{R}}^{R\times R\times R}$ is the core tensor, $\mathit{A}\in {\mathcal{R}}^{I\times R}$, $\mathit{B}\in {\mathcal{R}}^{R\times J}$, $\mathit{C}\in {\mathcal{R}}^{K\times R}$ are orthogonal projection matrices. The mode-k product ($k=1,2,3$) of $\Lambda$ by $\mathit{A}$, $\mathit{B}$ and $\mathit{C}$ is calculated by

$$\begin{array}{c}\hfill (\Lambda {\times }_1\mathit{A})(i_1,j_2,j_3)=\sum _{j_1=1}^{d_1}\Lambda (j_1,j_2,j_3)\mathit{A}(j_1,i_1)\\ \hfill (\Lambda {\times }_2\mathit{B})(j_1,i_2,j_3)=\sum _{j_2=1}^{d_2}\Lambda (j_1,j_2,j_3)\mathit{B}(j_2,i_2)\\ \hfill (\Lambda {\times }_3\mathit{C})(j_1,j_2,i_3)=\sum _{j_3=1}^{d_3}\Lambda (j_1,j_2,j_3)\mathit{C}(j_3,i_3)\end{array}$$

(4)

The decomposition may also be described more directly as

$$\chi (i_1,i_2,i_3)=\sum _{j_1=1}^{d_1}\sum _{j_2=1}^{d_2}\sum _{j_3=1}^{d_3}\Lambda (j_1,j_2,j_3)\mathit{A}(j_1,i_1)\mathit{B}(j_2,i_2)\mathit{C}(j_3,i_3).$$

(5)

Here, $d_i(i=1,2,3)$ are the number of columns in the projection matrices $\mathit{A}$, $\mathit{B}$ and $\mathit{C}$, respectively. If $d_i^{\prime }s$ are smaller than I, J, K, the core tensor $\Lambda$ can be considered a compressed version of $\chi$.

1.2. Challenges Incurred by the Streaming Data

With the advent of streaming data, i.e., data flowing continuously from a source to a sink, the problem becomes extremely challenging because data are usually generated simultaneously at high speed by different sources such as IoT sensors. What is worse, data in real-world applications may continuously arrive at every mode, i.e., data evolve over time at all modes [13]. Take a movie recommendation system as an example, a user–movie–date tensor is constructed, with an element $(i,j,k)$ denoting the rating given by user i to movie j on day k. It is evident that each mode of this tensor is subject to a neverending temporal evolution. Figure 3, cited from the website of the world famous data service vendor Qlik, demonstrates its technical complicatedness and arduousness.

Due to the wide variety of sources, as well as the scale and velocity by which the data are generated, traditional data pipelines cannot keep up for near-real-time or real-time processing because they have to extract, transform, and load data before they can be manipulated. However, in real-world applications (e.g., movie recommendation), data continuously arrive at every mode, i.e., data evolve over time at all modes [13]. Since the existing online TD methods cannot cope with this problem because of high computation and storage costs [14], a desired online tensor decomposition should be able to dynamically update the decomposition of real-time large-scale tensors while preserving its low-rank structure. In this sense, the Tucker decomposition is desirable because of its unique property, i.e., it does not need to decompose at all axes (modes) [15]. Xiao H P et al. [13] proposed an efficient online Tucker decomposition (eOTD) approach to track the tensor decomposition for dynamic large-scale tensors on the fly. In other words, for a given K-th-order tensor ${\chi }^{\left(t\right)}\in {\mathcal{R}}^{{N_1}^{\left(t\right)}\times \cdots \times N_K^{\left(t\right)}}$ at time t, it evolves at all modes, namely,

$$N_k^{(t+1)}\ge N_k^t,\forall k\in \left[K\right]$$

(6)

to generate a tensor stream $\left\{{\chi }^{(t+m)}\in {\mathcal{R}}^{N_1^{(t+m)}\times \cdots \times N_K^{(t+m)}}:m\in \left[M\right]\right\}$. The Tucker decomposition is defined by

$${\chi }^{\left(t\right)}={\mathcal{G}}^{\left(t\right)}\times \left\{{\mathit{U}}_{\mathit{k}}^{\left(\mathit{t}\right)}\right\}$$

(7)

where ${\mathcal{G}}^{\left(t\right)}$ and $U_k^{\left(t\right)}$ denote the core tensor and projection matrices, respectively, at timestamp t. The snapshot tensor at timestamp $t+1$ is ${\chi }^{(t+1)}={\left({\chi }_{i_1\cdots i_K}^{(t+1)}\right)}_{i_k\in \left[2\right]}\in R^{{N_1}^{(t+1)}\times \cdots \times N_K^{(t+1)}}$, where ${\chi }_{1\cdots 1}={\chi }^{\left(t\right)}$ (i.e., ${\chi }^{\left(t\right)}$ is a sub-tensor of ${\chi }^{(t+1)}$, denoted as ${\chi }^{\left(t\right)}\subseteq {\chi }^{(t+1)})$, and ${\left({\chi }_{i_1\cdots i_K}^{(t+1)}\right)}_{(i_1,\cdots ,i_K)\ne (1,\cdots ,1)}$ denotes new coming sub-tensors. ${\chi }^{(t+1)}$ can be decomposed by eOTD. More specifically, eOTD is able to obtain projection matrices $\left\{{\mathit{U}}_{\mathit{k}}^{(\mathit{t}+\mathbf{1})}\right\}$ by updating the $\left\{{\mathit{U}}_{\mathit{k}}^{\left(\mathit{t}\right)}\right\}$ using ${\mathcal{G}}^{\left(t\right)}$ and ${\left({\chi }_{i_1\cdots i_K}^{(t+1)}\right)}_{(i_1,\cdots ,i_K)\ne (1,\cdots ,1)}$, together with some auxiliary matrices obtained at timestamp $t+1$. In a similar vein, the core tensor is updated by a sum of tensors that are calculated by multiplying smaller tensors with matrices.

However, most research on tensor decomposition has been confined to standalone (i.e., single-machine) computational environments, which are not powerful enough for handling large-scale data, e.g., making solar energy affordable requires petaFLOPS of computing resources. This challenge has boosted the involvement of parallel computation [16]. Unlike serial computing, parallel architecture breaks a complex or formidable task into a set of sub-tasks, which are allocated to different computing equipment with the purpose that multiple operations are carried out simultaneously. Although parallel computing has taken a huge leap forward, its engagement with the tensor decomposition is not mature due to several factors, including the nonuniform task assignment and the synchronization among different computing nodes.

In this paper, we take eOTD as the foundation and a starting point and propose a two-level parallel incremental tensor Tucker decomposition method with multi-mode growth (TPITTD-MG) to reduce computational complexity and improve computing efficiency by advocating parallelization for the incremental tensor Tucker decomposition method, particularly for decomposition of tensors evolving over time at multiple modes. Specifically, the main contributions can be summarized as follows:

(1)

Utilizing dynamic programming to achieve efficient and uniform partitioning of sub-tensors, with a tensor partitioning algorithm applied to assign sub-tensors across different task nodes, which is crucial for scaling to tens of millions of data elements. This approach not only accelerates execution by exploiting parallel computation for counting non-zero elements but also ensures more uniform task assignment.

(2)

A two-level parallel update approach for projection matrices and core tensor is designed. The first level conducts updates via a parallel MTTKRP strategy, while the second level independently updates projection matrices based on categorized sub-tensors. This structured update mechanism substantially improves update efficiency.

(3)

The experimental results demonstrate that our method outperforms existing algorithms in terms of a nearly 400% improvement in execution efficiency and a 20% enhancement in uniformity of partition at a parallelism level of 4 for large-scale datasets. Specifically, for third-order tensors, our approach shows a nearly 300% efficiency improvement compared with traditional single-layer update algorithms.

The remainder of this paper is organized as follows. Section 2 reviews the related works, and description of important concepts involved in this paper is the subject of Section 3. In Section 4, we describe the proposed TPITTD-MG framework in detail, whose performance is evaluated. Finally, we conclude the paper in Section 5.

2. Related Works

    The Tucker decomposition can be considered a generalization of the matrix singular value decomposition (SVD), which has led to the development of the higher-order singular value decomposition (HOSVD) [17], a popular approach capable of yielding an orthogonal core tensor [18]. Thenceforth, numerous methods have emerged in the literature to compute the representation of a tensor. Since Tucker decomposition is essentially a best rank-$(R_1,R_2,\cdots ,R_N)$ approximation of tensor $\chi$ [19],

$$\underset{\mathcal{G};U_{\left(1\right)},\cdots ,U_{\left(N\right)}}{minimize}{∥\chi -\left[\mathcal{G};{\mathit{U}}_{\left(\mathbf{1}\right)},\cdots ,{\mathit{U}}_{\left(\mathit{N}\right)}\right]∥}_F^2,$$

(8)

the research in this field focuses on such topics as computation and storage complexity, parallel computing, optimization methods, as well as how to exploit latent features of tensors including sparsity, orthogonality, etc.

Kapteyn and Neudecker [20] extended the capacity of the Tucker decomposition from dealing with third-order tensors to higher-order ones by employing orthogonal projection matrices. De Lathauwer et al. [14] proposed the high-order orthogonal iteration (HOOI) method, which is proven to be more efficient in computing projection matrices, compared with HOSVD. Furthermore, Elden and Savas [21] introduced the Newton–Grassmann algorithm for the Tucker decomposition of third-order tensors, offering a more efficient and less iterative approach by constraining the projection matrix to the Grassmann manifold space. Fang S et al. [22] developed a Bayesian streaming sparse Tucker decomposition method (BASS-Tucker) capable of preventing overfitting and improving the interpretability by automatically selecting meaningful projection interactions.

Thanh et al. [23] provided a contemporary and comprehensive survey on streaming tensor decomposition, with streaming Tucker decomposition algorithms broadly classified into three main classifications, i.e., online tensor dictionary learning, tensor subspace tracking, and multi-aspect streaming Tucker decomposition. Among them, the first two classifications are dedicated for two specific scenarios of single-aspect streaming Tucker decompositions; the last classification is for multi-aspect streaming tensors.

On the other hand, some researchers push the enhancement of Tucker decomposition in direct, randomized, and iterative ways. Vannieuwenhoven et al. [24] presented an alternative strategy, called the sequentially truncated HOSVD (ST-HOSVD), to truncate the higher-order singular value decomposition (T-HOSVD) proposed by De Lathauwer et al, with the purpose to reduce computational complexity and improve the approximation error. Kressner et al. [25] attained a fast algorithm by combining fast matrix-vector products that exploit the structure of Hadamard products with iterative methods, such as the Lanczos method and randomized algorithms, etc. Che et al. [26] took randomized algorithms as powerful tools for scientific computing because such algorithms are usually faster and more robust compared with standard deterministic algorithms. They designed an adaptive randomized algorithm to compute a low multilinear rank approximate of tensor decomposition for tensors with unknown multilinear rank and analyzed its probabilistic error bound under certain assumptions. Chachlakis et al. [27] designed a $L_1$-Tucker, i.e., a reformulation of the standard Tucker decomposition by substitution of the outlier-responsive $L_2$-norm by the sturdier $L_1$-norm, followed by proposal of the $L_1$-norm higher order orthogonal iterations (L1-HOOI) algorithm for the approximate solution to $L_1$-Tucker.

As for the parallel computing, the research focuses on sparse tensors, parallel platforms with multicore nodes, data dependencies, and combination of randomization and parallelization for efficient communication schemes, etc [16,28]. Sejoon Oh et al. [29] developed the parallel Tucker decomposition algorithm P-TUCKER for sparse tensors, emphasizing memory savings and enhanced computational efficiency with scalability. Ballard G and Zhang J et al. [30] explored parallel computing frameworks for tensor Tucker decomposition. The advent of GigaTensor [31] marked a significant advance in tensor decomposition algorithms for large-scale tensors, followed by Park N et al. [32], who integrated these developments into the Bigtensor algorithm library. Acer S et al. [33] focused on tensor partitioning algorithms to further enhance decomposition efficiency. Minster et al. [16] proposed two randomized algorithms based on HOSVD and STHOSVD, respectively, and offered a new parallel implementation for the structured randomized sketch. Their key idea is to perform randomized sketches with Kronecker-structured random matrices in order to reduce computational complexity. Meanwhile, a probabilistic error analysis of the proposed algorithms was provided. As for distributed and parallel algorithms for Tucker decomposition, Yao et al. [34] and Ning et al. [35] proposed implementations based on Apache Spark, which is especially effective in processing high-dimensional tensors with excellent speedups and high robustness.

The aforementioned tensor decomposition algorithms are executed in parallel, which can significantly improve computational efficiency and save memory space. However, these algorithms are all full-scale parallel computation algorithms for tensor decomposition. When streaming data are dealt with, the tensor must be continuously restored and decomposed, resulting in a large amount of redundant computation, which deteriorates computational efficiency. Researchers attempt to address this issue in different ways.

Some researchers focus on incremental tensor decomposition. Nion et al. [36] proposed an adaptive CP decomposition algorithm for third-order tensors. Phan et al. [37] studied incremental tensor decomposition using the idea of block computation. Sarwar et al. [38] investigated the processing of dynamically growing data streams. However, these algorithms can only cope with tensors evolving at a single mode, whereas many real-world applications require tensor decomposition methods for tensors evolving at multiple modes, i.e., multi-aspect streaming tensors. Song et al. [39] proposed the multi-aspect streaming tensor (MAST) algorithm to achieve incremental tensor CP decomposition. Yang et al. [40] proposed the DisMASTD method to realize incremental tensor CP decomposition in a parallel manner. As for Tucker decomposition of the multi-aspect streaming tensors, Xiao et al. [13] proposed the eOTD algorithm, which is designed for running in a standalone mode.

Other researchers adopt the online mechanism. Zhou et al. [41] proposed the online CP algorithm for incremental tensor decomposition. Yu et al. [42] studied the incremental tensor Tucker decomposition and proposed an online low-rank tensor learning algorithm. In 2023, Rontogiannis et al. [43] proposed the online block-term decomposition reweighted least squares (O-BTD-RLS), which employs a sliding (truncated) windowing method with window duration being chosen according to the dynamics of the system under study or be adaptive.

In summary, although incremental tensor decomposition algorithms for processing the streaming data have currently become a research hotspot, the research is rather limited in the literature, with those about tensors of multi-mode growth in parallel particularly insufficient. Therefore, this article focuses on a parallel Tucker decomposition for multi-mode growth incremental tensors as the primary research object.

3. Two-Level Parallel Incremental Tensor Tucker Decomposition Method with Multi-Mode Growth (TPITTD-MG)

    In order to make descriptions clearly, we denote tensors, matrices, vectors and scalars with calligraphic letters (e.g., $\chi$), uppercase bold letters (e.g., $\mathit{U}$), lowercase bold letters (e.g., $\mathit{x}$) and lowercase normal font (e.g., a), respectively. Important notations involved in this paper are listed in

Table 1. Important notations in this paper.

Notation	Meaning
$\chi \in {\mathcal{R}}^{I_1\times I_2\times \cdots \times I_N}$	a N-dimensional tensor
$\mathcal{G}$	core tensor
${\mathit{U}}_{\mathbf{1}},{\mathit{U}}_{\mathbf{2}},\cdots ,{\mathit{U}}_{\mathit{n}}$	projection matrices
${\left(\chi \right)}_{\left(n\right)}$	mode-n unfolding of $\chi$
${\mathit{X}}^T$, ${\mathit{X}}^{-1}$, ${\mathit{X}}^{†}$	transpose, inverse and pseudo-inverse of matrix $\mathit{X}$
$\mathit{X}(i,:),\mathit{X}(:,j)$	i-th row and j-th column of $\mathit{X}$
$\chi {\times }_n\mathit{U}$	mode-n product of $\chi$ with $\mathit{U}$
$\circ ,\odot ,\otimes ,⊛$	outer, Khatri–Rao, Kronecker, and Hadamard product
$\mathcal{X}{\times }_n^1\mathcal{Y}$	mode-$(n,1)$ contracted product of $\mathcal{X}$ with $\mathcal{Y}$
$\left|\right|·{\left|\right|}_F,\left|\right|·{\left|\right|}_p,\left|\right|·{\left|\right|}_{*}$	Euclidean norm, $L_p$ norm and nuclear norm
$\left[{\left\{{\mathit{U}}^{\left(\mathit{n}\right)}\right\}}_{n=1}^N\right]$	${\sum }_{i=1}^r{\mathit{U}}^{\left(\mathbf{1}\right)}(:,i)\circ {\mathit{U}}^{\left(\mathbf{2}\right)}(:,i)\circ \dots \circ {\mathit{U}}^{\left(\mathit{N}\right)}(:,i)$
$\left[\chi ;{\left\{{\mathit{U}}^{\left(\mathit{n}\right)}\right\}}_{n=1}^N\right]$ or $\chi \times \left\{{\mathit{U}}_{\mathit{k}}\right\}$	$\chi {\times }_1{\mathit{U}}^{\left(\mathbf{1}\right)}{\times }_2{\mathit{U}}^{\left(\mathbf{2}\right)}{\times }_3\cdots {\times }_N{\mathit{U}}^{\left(\mathit{N}\right)}$, ${\left(\chi {\times }_{k}U\right)}_{n_1\cdots i_k\cdots n_K}={\sum }_{n_k=1}^N{X}_{n_1\cdots n_k\cdots n_N}{\mathit{U}}_{i_k{n}_k}$
$\mathcal{X}{\times }_{(a,b)}\mathcal{Y}$	The mode-$(a,b)$ product (Tensor Contraction))
$\mathcal{Z}{\times }_{k^{\prime }}{\mathit{U}}_{{\mathit{k}}^{\prime }}^{\mathit{Z}}=\mathcal{X}{\times }_k{\mathit{U}}_{\mathit{k}}^{\mathit{X}}+\mathcal{Y}{\times }_k{\mathit{U}}_{\mathit{k}}^{\mathit{Y}}$	Block Tensor and Matrix Multiplication I (Corollary I proposed by Xiao et al. [13]). In case $k^{\prime }\ne k$ and ${\mathit{U}}_{{\mathit{k}}^{\prime }}^{\mathit{Z}}\in {\mathcal{R}}^{R_{k^{\prime }}\times I_{k^{\prime }}}$, $\mathcal{Z}{\times }_k{\mathit{U}}_{{\mathit{k}}^{\prime }}^{\mathit{Z}}$ is equal to the concatenation of $\mathcal{X}{\times }_{k^{\prime }}{\mathit{U}}_{{\mathit{k}}^{\prime }}^{\mathit{Z}}$ and $\mathcal{Y}{\times }_{k^{\prime }}{\mathit{U}}_{{\mathit{k}}^{\prime }}^{\mathit{Z}}$ at the mode-k.
$\chi \times \left\{{\mathit{U}}_{\mathit{k}}\right\}={\sum }_{(i_1,\cdots ,i_K)\in {\left[2\right]}^K}{\chi }_{i_1{i}_2\cdots i_K}\times \left\{U_{k,i_k}\right\}$	Block Tensor and Matrix Multiplication II (Corollary II proposed by Xiao et al. [13]).

.

3.1. Technical Details of TPITTD-MG

   This subsection deals with the task of incremental Tucker decomposition method for tensors evolving over time at all modes and starts with a brief introduction to an off-the-shelf incremental Tucker decomposition method, i.e., efficient online Tucker decomposition (eOTD) proposed by Xiao et al. [13], followed by our research on performance improvement in terms of computational efficiency by means of parallel computing on the basis of eOTD. When the parallel incremental Tucker decomposition is implemented, it is a prerequisite to allocate the partitioned sub-tensors across different workhorses. However, current tensor partitioning algorithms often encounter such problems as low execution efficiency and nonuniform partition, etc. To address these issues, we propose a parallel sub-tensor allocation method based on dynamic programming, which significantly improves computational efficiency and achieves more uniform sub-tensor allocation, facilitating an optimal balance of computation load among different workhorses. Furthermore, to address the issue of low parallelism in updating projection matrices and the core tensor in current incremental Tucker decomposition methods, a two-level parallel update method is proposed.

3.1.1. A Brief Introduction to the Efficient Online Tucker Decomposition (eOTD) Approach

   Given a tensor ${\chi }^{\left(t\right)}\in {\mathcal{R}}^{N_1^{\left(t\right)}\times \cdots \times N_K^{\left(t\right)}}$ and its Tucker decomposition snapshot, ${\chi }^{\left(t\right)}={\mathcal{G}}^{\left(t\right)}\times \left\{{\mathit{U}}_k^{\left(t\right)}\right\}$ at timestamp t. When the tensor has evolved to ${\chi }^{(t+1)}\in {\mathcal{R}}^{N_1^{(t+1)}\times \cdots \times N_K^{(t+1)}}$ at timestamp $t+1$, the eOTD approach starts with a partition of ${\chi }^{(t+1)}$ into $2^K$ sub-tensors ${\left(x_{i_1\cdots i_K}^{(t+1)}\right)}_{i_k\in \left[2\right]}$ such that ${\chi }_{1\cdots 1}^{(t+1)}={\chi }^{\left(t\right)}$, which will be employed to efficiently implement the Tucker decomposition ${\chi }^{(t+1)}={\mathcal{G}}^{(t+1)}\times \left\{{\mathit{U}}_k^{(t+1)}\right\}$ by updating the core tensor ${\mathcal{G}}^{\left(t\right)}$ and the projection matrices (also called projection matrices) $\left\{{\mathit{U}}_k^{\left(t\right)}\right\}$.

The $2^K$ sub-tensors obtained are classified into K categories according to their geometric positions relative to the ${\chi }_{1\cdots 1}^{(t+1)}$, denoted as ${\left\{{\mathcal{C}}_m\right\}}_{m=1}^K$. Take a third-order tensor ${\chi }^{\left(t\right)}\in {\mathcal{R}}^{N_1^{\left(t\right)}\times N_2^{\left(t\right)}\times N_3^{\left(t\right)}}$ as an example, it evolves to ${\chi }^{(t+1)}$ at timestamp $t+1$, which is split into $2^3$ sub-tensors ${\chi }_{\left(i_1{i}_2{i}_3\right)}^{(t+1)}(i_1,i_2,i_3\in \left[2\right])$, as shown in Figure 4. Among the 8 sub-tensors, ${\chi }_{111}^{(t+1)}$ refers to ${\chi }^{\left(t\right)}$, and the other 7 sub-tensors, i.e., ${\left({\chi }_{i_1{i}_2{i}_3}^{(t+1)}\right)}_{\left(i_1{i}_2{i}_3\right)\ne \left(111\right)}$, correspond to the new coming data, i.e., incremental part of ${\chi }^{(t+1)}$ during time range $[t,t+1]$, which can be categorized into three groups:

$$\begin{array}{cc}\hfill {\mathcal{C}}_1& =\{{\chi }_{211}^{(t+1)},{\chi }_{121}^{(t+1)},{\chi }_{112}^{(t+1)}\}\hfill \\ \hfill {\mathcal{C}}_2& =\{{\chi }_{221}^{(t+1)},{\chi }_{212}^{(t+1)},{\chi }_{122}^{(t+1)}\}\hfill \\ \hfill {\mathcal{C}}_3& =\left\{{\chi }_{222}^{(t+1)}\right\}\hfill \end{array}$$

(9)

according to their geometric position relative to ${\chi }_{111}^{(t+1)}$: Elements in ${\mathcal{C}}_1$ can be considered the rear, right and downstairs neighbors of ${\chi }_{111}^{(t+1)}$, respectively. Namely, each of them shares a face with ${\chi }_{111}^{(t+1)}$. Similarly, elements in ${\mathcal{C}}_2$ and the only element in ${\mathcal{C}}_3$ share an edge or a vertex with ${\chi }_{111}^{(t+1)}$. This geometric understanding will be generalized in higher-order scenarios, which is more useful for tensor analysis, despite worse intuitiveness.

From the perspective of contribution to one or more auxiliary matrices, the three groups defined by Equation (26) can also be understood according to the number of sub-indices that equal to 2, indicating the number of auxiliary matrices that can be updated. For example, ${\chi }_{112}^{(t+1)}$ can be used for updating ${\mathit{U}}_3^{\prime }$, ${\chi }_{122}^{(t+1)}$ can be used for updating both ${\mathit{U}}_2^{\prime }$ and ${\mathit{U}}_3^{\prime }$, while ${\chi }_{222}^{(t+1)}$ corresponds to ${{\mathit{U}}^{\prime }}_1$, ${\mathit{U}}_2^{\prime }$ and ${\mathit{U}}_3^{\prime }$.

3.1.2. Update Methods for Projection Matrices

Every sub-tensor ${\chi }_{i_1\cdots i_K}^{(t+1)}\in {\mathcal{C}}_m$ will be used m times for the update of auxiliary matrix $\left\{{\mathit{U}}^{\prime }\right\}$. According to the Corollary II (i.e., Block Tensor and Matrix Multiplication II defined in Section 3), ${\chi }_{i_1\cdots i_K}^{(t+1)}={\mathcal{G}}^{\left(t\right)}\times \left\{{\mathit{U}}_{\mathit{k}}\right\}$, where

$${\mathit{U}}_{\mathit{k}}=\left\{\begin{array}{cc}{\mathit{U}}_{\mathit{k}}^{\left(\mathit{t}\right)},\hfill & \mathrm{if}{i}_k=1\hfill \\ {\mathit{U}}^{\prime },\hfill & \mathrm{if}{i}_k=2\hfill \end{array}\right.$$

(10)

Let all the sub-indices to be updated form a set $\mathit{S}=\left\{i_{k_1},\cdots ,i_{k_m}\right\}$. For each sub-index $i_{k_j}\in \mathit{S}$, the update rules for auxiliary matrices $\left\{{\mathit{U}}^{\prime }\right\}$ are defined as

$${\mathit{U}}_{i_{k_j}}^{\prime \mathit{new}}\leftarrow \alpha {\mathit{U}}_{i_{k_j}}^{\prime \mathit{old}}+\left(1-\alpha \right){\left({\chi }_{i_1\cdots i_K}^{(t+1)}\right)}_{\left(i_{k_j}\right)}{\left({\mathcal{G}}_{i_{k_j}}\right)}_{\left(i_{k_j}\right)}^{†},$$

(11)

where ${\mathcal{G}}_{i_{k_j}}\triangleq {\mathcal{G}}_{i_1,\cdots i_{k_j},\cdots ,i_K}={\mathcal{G}}^{\left(t\right)}{\times }_{-i_{k_j}}{\mathit{U}}_k^{†}$, and † denotes the pseudo inverse of a matrix.

Subsequently, the $\left\{{\mathit{U}}_k^{\prime }\right\}$ is augmented to $\left\{{\mathit{U}}_k^{\left(t\right)}\right\}=\left[{\left({\mathit{U}}_k^{\left(t\right)}\right)}^T{\left({\mathit{U}}_k^{\prime }\right)}^T\in {\mathcal{R}}^{N_k^{\left(t\right)}\times I_k}\right]$, which are orthogonalized and normalized by means of the modified Gram–Schmidt (MGS) to calculate $\left\{{\mathit{U}}_k^{(t+1)}\right\}$ at timestamp $t+1$ by updating $\left\{{\mathit{U}}_k^{\left(t\right)}\right\}$:

$${\mathit{U}}_k^{(t+1)}=\left\{{\left({\mathit{U}}_{k,1}^{(t+1)}\right)}^T{\mathit{U}}_{k,2}^{(t+1)}\right\}.$$

(12)

Namely, ${\mathit{U}}_k^{(t+1)}$ is also the concatenation of two matrices: ${\mathit{U}}_{k,1}^{(t+1)}\in {\mathcal{R}}^{N^{\left(t\right)}\times I_k}$ and ${\mathit{U}}_{k,2}^{(t+1)}\in {\mathcal{R}}^{\left(N^{(t+1)}-N^{\left(t\right)}\right)\times I_k}$.

Specifically, as for the scenarios of third-order tensors, each of the $2^3$ split sub-tensors ${\chi }_{\left(i_1{i}_2{i}_3\right)}^{(t+1)}\left(i_1,i_2,i_3\in \left[2\right]\right)$ of ${\chi }^{(t+1)}$ can be expressed as ${\chi }_{i_1{i}_2{i}_3}^{(t+1)}={\mathcal{G}}^{\left(t\right)}\times \left\{{\mathit{U}}_k\right\}$ according to the Corollary II, where ${\mathit{U}}_k$ has the same definition as Equation (27). Let

$${\mathcal{G}}_{i_1,i_2,i_3}={\mathcal{G}}^{\left(t\right)}\times \left\{{\mathit{U}}_k\right\},$$

(13)

in case a sub-index is missing, a set of auxiliary tensors can be defined by

$${\mathcal{G}}_{·,1,1}={\mathcal{G}}^{\left(t\right)}{\times }_2{\mathit{U}}_2^{\left(t\right)}{\times }_3{\mathit{U}}_3^{\left(t\right)},{\mathcal{G}}_{1,·,1}={\mathcal{G}}^{\left(t\right)}{\times }_1{\mathit{U}}_1^{\left(t\right)}{\times }_3{\mathit{U}}_3^{\left(t\right)},{\mathcal{G}}_{1,1,·}={\mathcal{G}}^{\left(t\right)}{\times }_1{\mathit{U}}_1^{\left(t\right)}{\times }_2{\mathit{U}}_2^{\left(t\right)},$$

(14)

$${\mathcal{G}}_{·,1,2}={\mathcal{G}}^{\left(t\right)}{\times }_2{\mathit{U}}_2^{\left(t\right)}{\times }_3{\mathit{U}}_3^{\prime \left(t\right)},{\mathcal{G}}_{1,·,2}={\mathcal{G}}^{\left(t\right)}{\times }_1{\mathit{U}}_1^{\left(t\right)}{\times }_3{\mathit{U}}_3^{\prime \left(t\right)},{\mathcal{G}}_{1,2,·}={\mathcal{G}}^{\left(t\right)}{\times }_1{\mathit{U}}_1^{\left(t\right)}{\times }_2{\mathit{U}}_2^{\prime \left(t\right)},$$

(15)

$${\mathcal{G}}_{·,2,1}={\mathcal{G}}^{\left(t\right)}{\times }_2{\mathit{U}}_2^{\prime \left(t\right)}{\times }_3{\mathit{U}}_3^{\left(t\right)},{\mathcal{G}}_{2,·,1}={\mathcal{G}}^{\left(t\right)}{\times }_1{\mathit{U}}_1^{\prime \left(t\right)}{\times }_3{\mathit{U}}_3^{\left(t\right)},{\mathcal{G}}_{2,1,·}={\mathcal{G}}^{\left(t\right)}{\times }_1{\mathit{U}}_1^{\prime \left(t\right)}{\times }_2{\mathit{U}}_2^{\left(t\right)},$$

(16)

$${\mathcal{G}}_{·,2,2}={\mathcal{G}}^{\left(t\right)}{\times }_2{\mathit{U}}_2^{\prime \left(t\right)}{\times }_3{\mathit{U}}_3^{\prime \left(t\right)},{\mathcal{G}}_{2,·,2}={\mathcal{G}}^{\left(t\right)}{\times }_1{\mathit{U}}_1^{\prime \left(t\right)}{\times }_3{\mathit{U}}_3^{\prime \left(t\right)},{\mathcal{G}}_{2,2,·}={\mathcal{G}}^{\left(t\right)}{\times }_1{\mathit{U}}_1^{\prime \left(t\right)}{\times }_2{\mathit{U}}_3^{\prime \left(t\right)},$$

(17)

i.e., they are multiplications of ${\mathcal{G}}^{\left(t\right)}$ with the remaining projection matrices. Equation (31) to Equation (34) lay the theoretic underpinning for defining update rules of auxiliary matrices using every sub-tensor ${\chi }_{\left(i_1{i}_2{i}_3\right)}^{(t+1)}(i_1,i_2,i_3\in \left[2\right])$, category by category, in ${\left\{{\mathcal{C}}_m\right\}}_{m=1}^3$.

A. Update Projection Matrices $\left\{{\mathit{U}}_{\mathit{k}}^{\prime }\right\}$ With Sub-tensors In ${\mathcal{C}}_1$ classification

As described previously, each sub-tensor in ${\mathcal{C}}_1$, i.e., ${\chi }_{211}^{(t+1)}$, ${\chi }_{121}^{(t+1)}$ and ${\chi }_{112}^{(t+1)}$, is used only once for updating the corresponding auxiliary matrix ${\mathit{U}}_1^{\prime }$, ${\mathit{U}}_2^{\prime }$ and ${\mathit{U}}_3^{\prime }$, respectively.

$${\mathit{U}}_1^{\prime }\leftarrow {\left({\chi }_{211}^{(t+1)}\right)}_{\left(1\right)}{\left({\mathcal{G}}_{·,1,1}\right)}_{\left(1\right)}^{†},$$

(18)

$${\mathit{U}}_2^{\prime }\leftarrow {\left({\chi }_{121}^{(t+1)}\right)}_{\left(2\right)}{\left({\mathcal{G}}_{1,·,1}\right)}_{\left(2\right)}^{†},$$

(19)

$${\mathit{U}}_3^{\prime }\leftarrow {\left({\chi }_{112}^{(t+1)}\right)}_{\left(3\right)}{\left({\mathcal{G}}_{1,1,·}\right)}_{\left(3\right)}^{†},$$

(20)

${\mathcal{G}}_{·,1,1}$, ${\mathcal{G}}_{1,·,1}$ and ${\mathcal{G}}_{1,1,·}$ are defined by Equation (31).

When the update of projection matrices $\left\{{\mathit{U}}^{\prime }\right\}$ has been implemented with sub-tensors in ${\mathcal{C}}_1$, they should be further refined by successive updates with sub-tensors in ${\mathcal{C}}_2$ and ${\mathcal{C}}_3$ classifications, because sub-tensors in different classifications have different contributions to one or more auxiliary matrices.

B. Update Projection Matrices $\left\{{\mathit{U}}_k^{\prime }\right\}$ With Sub-tensors In ${\mathcal{C}}_2$ Classification

As defined by Equation (26), each sub-tensor in the ${\mathcal{C}}_2=\left\{{\chi }_{221}^{(t+1)},{\chi }_{212}^{(t+1)},{\chi }_{122}^{(t+1)}\right\}$ has two sub-indices equal to 2, implying it will be used twice to update two corresponding auxiliary matrices. As for ${\chi }_{221}^{(t+1)}$, it is employed to update both ${\mathit{U}}_1^{\prime }$ and ${\mathit{U}}_2^{\prime }$. The update of ${\mathit{U}}_1^{\prime }$ requires the auxiliary matrix ${\mathit{U}}_2^{\prime }$ calculated based on sub-tensors in ${\mathcal{C}}_1$ classification, i.e., ${\mathcal{G}}_{·,2,1}={\mathcal{G}}^{\left(t\right)}{\times }_2{\mathit{U}}_2^{\prime \left(t\right)}{\times }_3{\mathit{U}}_3^{\left(t\right)}$, which can be expressed by

$$\begin{array}{cc}\hfill {\mathit{U}}_1^{\prime \left(211\right)}& \leftarrow \alpha {\mathit{U}}_1^{\prime }+(1-\alpha ){\left({\chi }_{211}^{(t+1)}\right)}_{\left(1\right)}{\left({\mathcal{G}}_{·,2,1}\right)}_{\left(1\right)}^{†},\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill {\mathit{U}}_2^{\prime \left(211\right)}& \leftarrow \alpha {\mathit{U}}_2^{\prime }+(1-\alpha ){\left({\chi }_{211}^{(t+1)}\right)}_{\left(2\right)}{\left({\mathcal{G}}_{2,·,1}\right)}_{\left(2\right)}^{†}.\hfill \end{array}$$

(22)

Here, $\alpha$ is a forgetting factor, indicating how much information should be inherited from the previous step. The same principle goes for ${\chi }_{212}^{(t+1)}$ and ${\chi }_{122}^{(t+1)}$. ${\mathit{U}}_k^{\prime \left(i_1{i}_2{i}_3\right)}$ means the updated auxiliary matrix ${\mathit{U}}_k^{\prime }$ using sub-tensor ${\chi }_{i_1{i}_2{i}_3}^{(t+1)}$.

$$\begin{array}{cc}\hfill {\mathit{U}}_1^{\prime \left(212\right)}& \leftarrow \alpha {\mathit{U}}_1^{\prime }+(1-\alpha ){\left({\chi }_{212}^{(t+1)}\right)}_{\left(1\right)}{\left({\mathcal{G}}_{·,1,2}\right)}_{\left(1\right)}^{†},\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill {\mathit{U}}_3^{\prime \left(212\right)}& \leftarrow \alpha {\mathit{U}}_3^{\prime }+(1-\alpha ){\left({\chi }_{212}^{(t+1)}\right)}_{\left(3\right)}{\left({\mathcal{G}}_{2,1,·}\right)}_{\left(3\right)}^{†},\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill {\mathit{U}}_2^{\prime \left(122\right)}& \leftarrow \alpha {\mathit{U}}_2^{\prime }+(1-\alpha ){\left({\chi }_{122}^{(t+1)}\right)}_{\left(2\right)}{\left({\mathcal{G}}_{1,·,2}\right)}_{\left(2\right)}^{†},\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill {\mathit{U}}_3^{\prime \left(122\right)}& \leftarrow \alpha {\mathit{U}}_3^{\prime }+(1-\alpha ){\left({\chi }_{122}^{(t+1)}\right)}_{\left(3\right)}{\left({\mathcal{G}}_{1,2,·}\right)}_{\left(3\right)}^{†}.\hfill \end{array}$$

(26)

C. Update Projection Matrices $\left\{{\mathit{U}}_k^{\prime }\right\}$ With Sub-tensors In ${\mathcal{C}}_3$ Classification

Since ${\mathcal{C}}_3$ classification contains only one sub-tensor ${\chi }_{222}^{(t+1)}={\mathcal{G}}^{\left(t\right)}\times \left\{{\mathit{U}}_k^{\prime }\right\}$, with three sub-indices equal to 2, ${\chi }_{222}^{(t+1)}$ associates with all auxiliary matrices:

$$\begin{array}{cc}\hfill {\mathit{U}}_1^{\prime \left(222\right)}& \leftarrow \alpha {\mathit{U}}_1^{\prime }+(1-\alpha ){\left({\chi }_{222}^{(t+1)}\right)}_{\left(1\right)}{\left({\mathcal{G}}_{·,2,2}\right)}_{\left(1\right)}^{†},\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill {\mathit{U}}_2^{\prime \left(222\right)}& \leftarrow \alpha {\mathit{U}}_2^{\prime }+(1-\alpha ){\left({\chi }_{222}^{(t+1)}\right)}_{\left(2\right)}{\left({\mathcal{G}}_{2,·,2}\right)}_{\left(2\right)}^{†},\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill {\mathit{U}}_3^{\prime \left(222\right)}& \leftarrow \alpha {\mathit{U}}_3^{\prime }+(1-\alpha ){\left({\chi }_{222}^{(t+1)}\right)}_{\left(3\right)}{\left({\mathcal{G}}_{2,2,·}\right)}_{\left(3\right)}^{†}.\hfill \end{array}$$

(29)

D. Update Projection Matrices ${\mathit{U}}_k^{(t+1)}$

Given projection matrices $\left\{{\mathit{U}}_k^{\left(t\right)}\right\}$ and the auxiliary matrices $\left\{{\mathit{U}}_k^{\prime }\right\}$ obtained at time t, the projection matrices $\left\{{\mathit{U}}_k^{(t+1)}\right\}\left(\forall k\in \left[K\right]\right)$ at timestamp $t+1$ are updated according to the rule:

$${\left({\mathit{V}}_k^{(t+1)}\right)}^T=\left[{\left({\mathit{U}}_k^{\left(t\right)}\right)}^T{\left({\mathit{U}}_k^{\prime }\right)}^T\right]\in {\mathcal{R}}^{N_k^{(t+1)}\times I_k}(\forall k\in \left[K\right]),$$

(30)

which involves a concatenation of ${\mathit{U}}_k^{\left(t\right)}$ and ${\mathit{U}}_k^{\prime }$ at the k-th mode. Since Equation (47) cannot guarantee the generated $\left({\mathit{V}}_k^{(t+1)}\right)$ will be unitary, the modified Gram–Schmidt (MGS) algorithm is performed on ${\mathit{V}}_k^{(t+1)}$ to produce orthonormal projection $\left\{{\mathit{U}}_k^{(t+1)}\right\}$.

3.1.3. Update Method for Core Tensor

   Based on the projection matrices $\left\{{\mathit{U}}_k^{(t+1)}\right\}\in {\mathcal{R}}^{N^{(t+1)}\times I_k}$, the core tensor ${\mathcal{G}}^{\left(t\right)}$ at timestamp t, as well as the new coming data ${\left({\chi }_{i_1\cdots i_k}^{(t+1)}\right)}_{(i_1,\cdots ,i_K)\ne (1,\cdots ,1)}$ at timestamp $t+1$, since $\left\{{\mathit{U}}_k^{(t+1)}\right\}$ are unitary matrices, the core tensor ${\mathcal{G}}^{(t+1)}$ at timestamp $t+1$ can be updated by

$${\mathcal{G}}^{(t+1)}={\chi }^{(t+1)}\times \left\{{\left({\mathit{U}}_k^{(t+1)}\right)}^T\right\}={\mathcal{G}}^{\left(t\right)}\times \left\{{\left({\mathit{U}}_{k,1}^{(t+1)}\right)}^T{\mathit{U}}_k^{\left(t\right)}\right\}+\sum _{(i_1,\cdots ,i_K)\ne (1,\cdots ,1)}{\chi }_{i_1,\cdots ,i_K}^{(t+1)}\times \left\{{\left({\mathit{U}}_{k,i_k}^{(t+1)}\right)}^T\right\},$$

(31)

according to the Second Corollary. The update procedure incorporates splitting of ${\mathit{U}}_k^{(t+1)}$ into two matrices: ${\mathit{U}}_{k,1}^{(t+1)}\in {\mathcal{R}}^{N^{\left(t\right)}\times I_k}$ and ${\mathit{U}}_{k,2}^{(t+1)}\in {\mathcal{R}}^{\left(N^{(t+1)}-N^{\left(t\right)}\right)\times I_k}$.

In summary, eOTD is efficient in dealing with the tensor stream $\left\{{\chi }^{(t+m)}\in {\mathcal{R}}^{N_1^{(t+m)}\times \cdots \times N_K^{(t+m)}}:m\in \left[M\right]\right\}$ defined in Section 1.2 and can be applied to large-scale applications because it merely involves tensor matrix multiplication and matrix pseudo inverse operation, which are cheap compared with the computationally expensive SVD operations. Meanwhile, eOTD is established on a solid theoretical foundation, i.e., two corollaries proposed by Xiao et al. [13], which makes it a powerful tool to track Tucker decomposition of dynamic tensors ${\chi }^{(t+m)}$ with an arbitrary number of modes.

3.2. Parallel Sub-Tensor Partitioning Algorithm

   It can be seen in Section 3.1 that it is an indispensable task to split a tensor into sub-tensors, which act as the fundamental elements for updating the core tensor and the projection matrices. Despite the relatively satisfactory efficiency provided by the eOTD approach, its standalone operation paradigm becomes an obstacle on the path to successful applications in more large-scale scenarios, motivating the involvement of parallel decomposition methods. Generally, for parallel computing, the strategy of allocating the obtained sub-tensors uniformly among different computing resources is a major concern. For ease of representation, we iconically refer to each computing resource in a distributed environment as a workhorse.

In this subsection, we start from the sub-tensors obtained by eOTD and exploit their latent structure to enhance computing efficiency of parallel decomposition and reduce the bottleneck cost incurred by fundamental tensor-related operations, e.g., matricized tensor times Khatri–Rao Product (MTTKRP [44]) in the Tucker decomposition. Since only non-zero elements of tensors contribute to the MTTKRP cost, an appropriate partition criteria for sub-tensors should be that each partitioned group has an equal number of non-zero tensor elements.

For example, given a sub-tensor ${\chi }_{sub}\in {\mathcal{R}}^{3\times 5\times 2}$, We attempt to partition $\chi$ at the mode-2, i.e., the second dimension of $3\times 5\times 2$, producing five slices in total, as shown by Figure 5. Non-zero elements in each slice are $\{4,2,3,4,8,9\},\{5,6,6\},\{71,12,3,43\},\{15,3,8\},\{7,1\}$, respectively. Suppose the task in this scenario is to allocate the five slices among three workhorses (i.e., Workhorse I, Workhorse II and Workhorse III), the optimal strategy can be summarized as follows:

• To allocate the first slice to Workhorse I, the corresponding six non-zero elements form Group I;
• To allocate the second and fourth slices to Workhorse II, the corresponding six non-zero elements form Group II; and
• To allocate the third and fifth slices to Workhorse III, the corresponding six non-zero elements form Group III.

As a result, all the non-zero elements are uniformly partitioned into three groups and each workhorse is allocated six non-zero elements, i.e., an extremely uniform allocation is realized. However, such an ideal situation as has occurred in this simple scenario may not be always true in practice, because splitting a sub-tensor into slices at a mode is analogous to the task of uniformly splitting a set of positive integers, a well-known NP-hard problem due to its complexity and the impracticality of finding an optimal solution [45]. In order to address this issue, Yang et al. [40] introduced a heuristic algorithm, DisMASTD, which counts the number of non-zero elements (NNZ) at each slice at mode-n of the sub-tensor and sums all these counts to yield a total number, $Tota{l}_{NNZ}$. Let $p_n$ represent the desired number of partitions at mode-n; an optimal number of non-zero elements in each partition, named $Target$, is calculated by $Tota{l}_{NNZ}/p_n$. The workflow of DisMASTD is described as follows, which can also be described in pseudo code, as shown by Algorithm 1.

• Initially, DisMASTD traverses $I_n$ slices at the mode-n of a sub-tensor ${\chi }_{\mathrm{sub}}\in {\mathcal{R}}^{I_1\times I_2\times \cdots \times I_N}$ (i.e., ${\chi }_{\mathrm{sub}}$ has N modes in total, and mode-n has $I_n$ slices, $1\le n\le N$), counting non-zero elements at each slice, denoted as ${\left\{re{s}_i^{\left(n\right)}\right\}}_{i=1}^{I_n}$.
• DisMASTD traverses mode-n of ${\chi }_{sub}$ again, slice-by-slice, greedily assigning the traversed slices to the current partition $P_{\left(num\right)}^{\left(n\right)}(num=1,\cdots ,p_n)$, until its total number of non-zero elements (denoted as $sum$) reaches $Target$.
• When the i-th slice is being traversed and the $sum$ of the first partition reaches $Target$, DisMASTD will compare the deviation from $Target$ incurred by two different strategies:
  i. Assigning slice i to the current partition incurs deviation $\left|Target-sum\right|$, and
  ii. Not assigning slice i to the current partition incurs deviation $\left|Target-(sum-a_i^{\left(n\right)}.NNZ)\right|$.
  The strategy with smaller deviation will be accepted, i.e., if $\left|Target-sum\right|<\left|Target-(sum-a_i^{\left(n\right)}.NNZ)\right|$, slice i is assigned to the current partition, i.e., the $num$-th partition $P_{num}^{\left(n\right)}$. Otherwise, go to step (4).
• Generate the next partition $P_{(num+1)}^{\left(n\right)}$ for mode-n, and assign the slice i to it.
• DisMASTD will go ahead to traverse the $(i+1)$-th slice of mode-n and decide whether this slice should be assigned to the partition $P_{(num+1)}^{\left(n\right)}$. The loop will iterate until the $sum$ of partition $P_{p_n-1}^{\left(n\right)}$ reaches $Target$.
• The residual slices at mode-n are assigned to the last partition $P_{p_n}^{\left(n\right)}$.
• In case partitions at all modes are completed, DisMASTD outputs the partition result ${\left\{P_p^{\left(n\right)}\right\}}_{p=1}^{p_n},n=1,2,\cdots ,N$.

Algorithm 1 DisMASTD—Heuristic Tensor Partitioning Algorithm

Input: (1) ${\chi }_{\mathrm{sub}}\in {\mathcal{R}}^{I_1\times I_2\times \cdots \times I_N}$, Sub-tensor to be partitioned;

1:           (2) ${\left\{p_n\right\}}_{i=1}^N$, number of partitions at mode-n;

Output:  ${\left\{P_p^{\left(n\right)}\right\}}_{p=1}^{p_n},n=1,2,\cdots ,N$, mode-wise allocation of sub-tensor ${\chi }_{\mathrm{sub}}$.

2: for $1\le n\le N$ do   3:     $Target=Tota{l}_{NNZ}$/$p_n$ //optimal number of non-zero elements in each partition   4:     ${\left\{re{s}_i^{\left(n\right)}\right\}}_{i=1}^{I_n}$ //count number of non-zero elements at each slice at mode n of ${\chi }_{\mathrm{sub}}$   5:     ${\left\{{\mathit{P}}_p^{\left(n\right)}\right\}}_{n=1}^N\leftarrow \left\{\varnothing \right\}$ //initialize the partition results container   6:     $P\leftarrow \varnothing$ //initialize a temporary container for intermediate partition results   7:     $num\leftarrow 1,sum\leftarrow 0$ //initialize other auxiliary variables.   8:                                  //$sum$ means the total number of non-zero elements allocated to   9:                                  //the current partition, $num$ denotes the current partition, 10:                                  //i.e., $num$-th partition. 11:     for $1\le i\le I_n$ do 12:         $sum+=a_i^{\left(n\right)}.nnz$ 13:         if $sum<Target$ then. 14:            assign slice i to P 15:         else 16:            if $\left|Target-sum\right|<\left|Target-(sum-a_i^{\left(n\right)}.NNZ)\right|$ then 17:                assign slice i to P 18:            end if 19:            if $num<p_n$ then 20:                $P_{num}^{\left(n\right)}=P$;    //generate a new partition $P_{num}^{\left(n\right)}$ to store the content in P 21:                $P\leftarrow \varnothing ,num+=1,sum\leftarrow 0$ 22:            else 23:                Assign the remaining slices to $P_{p_n}^{\left(n\right)}$, break 24:            end if 25:         end if 26:     end for 27:  end for 28:  Output partitioned results ${\left\{P_p^{\left(n\right)}\right\}}_{p=1}^{p_n},n=1,2,\cdots ,N$.

Nevertheless, DisMASTD exhibits a low execution efficiency because the partition of sub-tensors requires traversing all slices of a sub-tensor at each mode multiple times, which executes in a serial manner and incurs unacceptable time consumption.

Table 2. Execution Timetable of Heuristic Tensor Partitioning Algorithm.

Order Number	Data Size	Duration of Counting Non-Zero Element (In Seconds)	Length of Execution of Partitioning Algorithm (In Seconds)	Total Update Time (In Seconds)
1	20∗20∗40	0.52	0.91	2.42
2	200∗20∗40	1.09	1.28	2.82
3	200∗200∗40	2.01	2.17	4.56
4	200∗200∗400	8.02	8.18	16.09

The corresponding hardware/software configuration as well as datasets for experiments are described at length in Section 4.

enumerates the algorithm’s overall time consumption over sub-tensors of various ranks, as well as the specific time spent in counting non-zero elements. For smaller size tensors, counting non-zero elements accounts for over 50% of the total partitioning time. While for larger size ones, the ratio may reach 95% or even higher. In case non-zero elements are nonuniformly distributed, the performance of DisMASTD will go from bad to worse.

To address these issues, we propose a parallel sub-tensor partitioning mechanism based on the dynamic programming (PSTPA-DP), which attempts to realize parallel implementation from both the level of a sub-tensor and the level of a sub-tensor classification:

(1)

From the level of a sub-tensor

If all slices of ${\chi }_{sub}\in {\mathcal{R}}^{I_1\times I_2\times \dots \times I_N}$ at mode-n are divided into $p_n$ parts, numbers of non-zero elements of all $p_n$ parts can be counted in a parallel manner. When all $p_n$ counting tasks are completed, the results are aggregated to obtain the output ${\left\{re{s}_i^{\left(n\right)}\right\}}_{i=1}^{I_n}$, i.e., the number of non-zero elements in each slice of ${\chi }_{sub}$.

(2)

From the level of a sub-tensor classification

Based on the update rules for projection matrices defined in Section 3.1.2, a tensor with K modes is partitioned into $2^K$ sub-tensors, which are further divided into K classifications ${\left\{{\mathcal{C}}_m\right\}}_{m=1}^K$. Because the update processes of auxiliary matrices, which involve $R^{\left(m\right)}$ sub-tensors ${\chi }_{sub,i}^{\left(m\right)}\left(1\le i\le R^{\left(m\right)}\right)$ in classification ${\mathcal{C}}_m$, are independent of each other, it is desirable to partition $R^{\left(m\right)}$ sub-tensors in ${\mathcal{C}}_m$ (i.e., ${\left\{{\chi }_{sub,i}^{\left(m\right)}\right\}}_{i=1}^{R^{\left(m\right)}}$) in a parallel manner. For example, there are three sub-tensors in ${\mathcal{C}}_2=\left\{{\chi }_{221}^{(t+1)},{\chi }_{212}^{(t+1)},{\chi }_{122}^{(t+1)}\right\}$; we wish to partition ${\chi }_{221}^{(t+1)},{\chi }_{212}^{(t+1)},{\chi }_{122}^{(t+1)}$ simultaneously.

PSTPA-DP introduces the dynamic programming mechanism to address the problem of nonuniform distribution of non-zero elements, it first clarifies the task of partition by determining the modes required to be expanded when updating the auxiliary matrices $\left\{{\mathit{U}}^{\prime }\right\}$ using sub-tensors ${\chi }_{sub}^{\left(m\right)}\in {\mathcal{R}}^{I_1\times I_2\times \cdots \times I_N}$ in different classifications ${\mathcal{C}}_m\left(m\in \left[K\right]\right)$. Subsequently, the numbers of non-zero elements, denoted as $re{s}_i^{\left(n\right)}$ at each slice at selected modes, are counted in a parallel manner. Finally, the average value $\overline{AVG}$ of the number of non-zero elements in the remaining slices (i.e., the slices that have not been partitioned) is iteratively refreshed based on the idea of dynamic programming. The algorithm can be described in greater detail by pseudo code in Algorithm 2.

Compared with DisMASTD, PSTPA-DP is advantageous in terms of time complexity. Given a sub-tensor ${\chi }_{sub}\in {\mathcal{R}}^{I\times J\times K}$, the heuristic tensor partitioning algorithm has a time complexity of $\mathcal{O}\left(IJK\right)$, which can be reduced to $\mathcal{O}(IJK/p_n)$ by PSTPA-DP. The significant improvement in performance can be attributed to the division of the sub-tensor partitioning task into $p_n$ sub-tasks, as well as allocation of each sub-task to a different workhorse, scheduled by dynamic programming.

Algorithm 2 Parallel Sub-tensor Partitioning Algorithm Based on Dynamic Programming (PSTPA-DP)

Input: (1) ${\chi }_{sub,i}^{\left(m\right)}\in {\mathcal{R}}^{I_1\times I_2\times \dots \times I_N}\left(1\le i\le R^{\left(m\right)}\right)$: sub-tensors in ${\mathcal{C}}_m(m=1,\cdots ,K)$ classification;

1: (2) $R^{\left(m\right)}$: number of sub-tensors in ${\mathcal{C}}_m$ classification;   2: (3) ${\left\{p_n\right\}}_{n=1}^N$: number of partitions at each mode of ${\chi }_{sub}$.

Output: $\left\{{\left\{P_p^{\left(n\right)}\right\}}_{p=1}^{p_n}\right\}(n=1,2,\cdots ,N)$: mode-wise partition results of each ${\chi }_{sub}$ in ${\mathcal{C}}_m$.

3: for ${\chi }_{sub,i}^{\left(m\right)}$∈${\mathcal{C}}_m$$\left(1\le i\le R^{\left(m\right)}\right)$do   //partition $R^{\left(m\right)}$ sub-tensors ${\chi }_{sub,i}^{\left(m\right)}$∈${\mathcal{C}}_m$ in parallel   4:     divide all slices of ${\chi }_{sub,i}^{\left(m\right)}$ at mode-n into $p_n$ parts;   5:     count non-zero elements at each slice at mode-n of ${\chi }_{sub,i}^{\left(m\right)}$ in a parallel manner, and output ${\left\{re{s}_j^{\left(n\right)}\right\}}_{j=1}^{I_n}$;   6:     sort ${\left\{re{s}_j^{\left(n\right)}\right\}}_{j=1}^{I_n}$ in descending sequence;   7:     calculate the average number of non-zero elements over the $p_n$ parts at mode-n by $\overline{AVG}=\sum \left({\left\{re{s}_j^{\left(n\right)}\right\}}_{j=1}^{I_n}\right)/p_n$;   8:     ${\left\{P_p^{\left(n\right)}\right\}}_{p=1}^{p_n}\leftarrow \varnothing (n=1,2,\cdots ,N)$;       //Initialization of ${\left\{P_p^{\left(n\right)}\right\}}_{p=1}^{p_n}$   9:     for $n\leftarrow 1:N$ do    //mode-wise partition in a parallel manner 10:         for $i\leftarrow 1:I_n$ do    //slice-by-slice assignment at mode-i 11:            if $re{s}_i^{\left(n\right)}>\overline{AVG}$ then 12:                $P_p^{\left(n\right)}\leftarrow re{s}_i^{\left(n\right)}$;    //assign $re{s}_i^{\left(n\right)}$ to $P_p^{\left(n\right)}$; 13:                $I_n=I_n-1$;      //remove $re{s}_i^{\left(n\right)}$ 14:                refresh the average $\overline{AVG}=\sum \left({\left\{re{s}_i^{\left(n\right)}\right\}}_{i=1}^{I_n}\right)/p_n$; 15:            else 16:                $P_p^{\left(n\right)}\leftarrow re{s}_i^{\left(n\right)}$;       //assign $re{s}_i^{\left(n\right)}$ to $P_p^{\left(n\right)}$ and remove $re{s}_i^{\left(n\right)}$; 17:                $\delta =avg-re{s}_i^{\left(n\right)}$; 18:                Traverse other members in $\left({\left\{re{s}_i^{\left(n\right)}\right\}}_{i=1}^{I_n}\right)$: 19:                if there exists a member $M_1\in \left({\left\{re{s}_i^{\left(n\right)}\right\}}_{i=1}^{I_n}\right)$ equal to $\delta$ then 20:                    $P_p^{\left(n\right)}\leftarrow M_1$;    //assign $M_1$ to the current partition $P_p^{\left(n\right)}$; 21:                    exit the traverse; 22:                else if a member $M_2$ satisfies $a>\delta >b$$\left(a,b\in {\left\{re{s}_i^{\left(n\right)}\right\}}_{i=1}^{I_n}\right)$ then 23:                    if $(\delta -b)>(a-\delta )$ then 24:                        $P_p^{\left(n\right)}\leftarrow a$;    //assign a to the current partition $P_p^{\left(n\right)}$ 25:                        exit the traverse; 26:                    else 27:                        $P_p^{\left(n\right)}\leftarrow b$;    assign b to the current partition $P_p^{\left(n\right)}$ 28:                        find $min(\delta -a)$; 29:                        $P_p^{\left(n\right)}\leftarrow a$;    //assign a to the current partition $P_p^{\left(n\right)}$; 30:                        exit the traverse; 31:                    end if 32:                end if 33:            end if 34:         end for 35:     end for 36: end for 37: Output partitioned results ${\left\{P_p^{\left(n\right)}\right\}}_{p=1}^{p_n},n=1,2,\cdots ,N$.

3.3. Parallel Computing Method for the Incremental Tensor Tucker Decomposition

   This subsection deals with the decomposition of the tensor at timestamp $t+1$, i.e., ${\chi }^{(t+1)}$, by the incremental tensor Tucker decomposition on the basis of ${\chi }^{\left(t\right)}={\mathcal{G}}^{\left(t\right)}\times \left\{{\mathit{U}}_k^{\left(t\right)}\right\}$ in a parallel manner, provided the sub-tensors ${\left({\chi }_{i_1\cdots i_k}^{(t+1)}\right)}_{(i_1,\cdots ,i_K)\ne (1,\cdots ,1)}$ have been partitioned by the proposed PSTPA-DP mechanism. We propose a two-level parallel update method for projection matrices and core tensors: the first level updates projection matrices or core tensors based on the parallel MTTKRP calculation strategy, and the second level updates different projection matrices or tensors independently using sub-tensors in different classifications in a parallel manner.

3.3.1. Two-Level Parallel Update Method for Projection Matrices

   In Section 3.1.2, the update Formulas (13)–(30) for projection matrices in a stand-alone setting have been provided, utilizing sub-tensors of different classifications ${\left\{{\mathcal{C}}_m\right\}}_{m=1}^K$ to update the projection matrices. In this subsection, the update method is extended to a parallel manner. Take third-order tensors as an example. Suppose at time t, a third-order tensor ${\chi }^{\left(t\right)}\in {\mathcal{R}}^{I\times J\times K}$ has a Tucker decomposition with the core tensor $\mathcal{G}\in {\mathcal{R}}^{I\times J\times J}$, and projection matrices ${\mathit{U}}_1\in {\mathcal{R}}^{I\times P}$, ${\mathit{U}}_2\in {\mathcal{R}}^{J\times Q}$, ${\mathit{U}}_3\in {\mathcal{R}}^{K\times R}$. At time $t+1$, the projection matrices are updated using sub-tensor ${\chi }_{211}^{(t+1)}$ in ${\mathcal{C}}_1$ classification by Equation (22), i.e., ${\mathit{U}}_1^{\prime }\leftarrow {\left({\chi }_{211}^{(t+1)}\right)}_{\left(1\right)}{\left({\mathcal{G}}_{·,1,1}\right)}_{\left(1\right)}^{†}$, where the core matrix ${\mathcal{G}}_{·,1,1}\in {\mathcal{R}}^{P\times J\times K}$ is expanded at mode-1 to generate ${\left({\mathcal{G}}_{·,1,1}\right)}_{\left(1\right)}\in {\mathcal{R}}^{P\times JK}$. The second dimension probably has a unimaginably big size, e.g., in case $J=K={10}^6$, $J\times K$ may reach the magnitude of ${10}^{12}$, implying the update of projection matrices by a single computing equipment is computationally infeasible. Therefore, we consider implementing the update process in a parallel manner.

A. The First Level of Parallel Update—Parallel MTTKRP Calculation Strategy

According to Equation (34), the projection matrices for high-order tensors can be updated by

$${\mathit{U}}_{\left(k\right)}^{\prime }\leftarrow {\left({\chi }_{i_1\cdots i_k}^{(t+1)}\right)}_{\left(k\right)}{\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}^{†}.$$

(32)

Based on the properties of the pseudo-inverse, the right part of Equation (36) can be expressed as

$$\begin{array}{ccc}\hfill {\left({\chi }_{i_1\cdots i_k}^{(t+1)}\right)}_{\left(k\right)}{\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}^{†}& =& {\left({\chi }_{i_1\cdots i_k}^{(t+1)}\right)}_{\left(k\right)}{\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}^{†}{\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}{\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}^{†}\hfill \\ \hfill & =& {\left({\chi }_{i_1\cdots i_k}^{(t+1)}\right)}_{\left(k\right)}{\left({\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}^{†}{\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}\right)}^T{\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}^{†}\hfill \\ \hfill & =& {\left({\chi }_{i_1\cdots i_k}^{(t+1)}\right)}_{\left(k\right)}{\left({\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}^T{\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}^{†}\right)}^T{\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}^{†}\hfill \\ \hfill & =& {\left({\chi }_{i_1\cdots i_k}^{(t+1)}\right)}_{\left(k\right)}{\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}^T{\left({\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}{\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}^T\right)}^{†}\hfill \end{array}$$

(33)

Considering ${\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}={\mathcal{G}}_{\left(k\right)}{\left({\mathit{U}}_{\left(k\right)}\otimes \cdots \otimes {\mathit{U}}_{(k+1)}\otimes {\mathit{U}}_{(k-1)}\otimes \cdots \otimes {\mathit{U}}_{\left(1\right)}\right)}^T$, we obtain

$$\begin{array}{cc}\hfill & {\left({\chi }_{i_1\cdots i_k}^{(t+1)}\right)}_{\left(k\right)}{\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}^{\top }{\left({\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}^{†}\right)}^{\top }{\left({\mathcal{G}}_{-i_k}\right)}_{\left(k\right)}^{†}\hfill \\ \hfill & ={\left({\chi }_{i_1\cdots i_k}^{(t+1)}\right)}_{\left(k\right)}\left({\mathit{U}}_{\left(K\right)}\otimes \cdots \otimes {\mathit{U}}_{(k+1)}\otimes {\mathit{U}}_{(k-1)}\otimes \cdots \otimes {\mathit{U}}_{\left(1\right)}\right){\mathcal{G}}_{\left(k\right)}^{\top }\hfill \\ \hfill & {\left({\left(\mathcal{G}\right)}_{\left(k\right)}{\left({\mathit{U}}_{\left(K\right)}\otimes \cdots \otimes {\mathit{U}}_{\left(k+1\right)}\otimes {\mathit{U}}_{\left(k-1\right)}\otimes \cdots \otimes {\mathit{U}}_{\left(1\right)}\right)}^T\left({\mathit{U}}_{\left(K\right)}\otimes \cdots \otimes {\mathit{U}}_{\left(k+1\right)}\otimes {\mathit{U}}_{\left(k-1\right)}\otimes \cdots \otimes {\mathit{U}}_{\left(1\right)}\right){\mathcal{G}}_{\left(k\right)}^T\right)}^{†}\hfill \\ \hfill & ={\left({\chi }_{i_1\cdots i_k}^{(t+1)}\right)}_{\left(k\right)}\left({\mathit{U}}_{\left(K\right)}\otimes \cdots \otimes {\mathit{U}}_{(k+1)}\otimes {\mathit{U}}_{(k-1)}\otimes \cdots \otimes {\mathit{U}}_{\left(1\right)}\right){\mathcal{G}}_{\left(k\right)}^{\top }{\left({\mathcal{G}}_{\left(k\right)}{\mathcal{G}}_{\left(k\right)}^{\top })\right)}^{†}\hfill \\ \hfill & ={\left({\chi }_{i_1\cdots i_k}^{(t+1)}\right)}_{\left(k\right)}\left({\mathit{U}}_{\left(K\right)}\otimes \cdots \otimes {\mathit{U}}_{(k+1)}\otimes {\mathit{U}}_{(k-1)}\otimes \cdots \otimes {\mathit{U}}_{\left(1\right)}\right){\mathcal{G}}_{\left(k\right)}^{†}\hfill \end{array}$$

(34)

Consequently, Equation (36) can be expressed as

$$\begin{array}{c}\hfill {\mathit{U}}_k^{\prime }\leftarrow {\left({\chi }_{i_1\cdots i_k}^{(t+1)}\right)}_{\left(k\right)}\left({\mathit{U}}_{\left(K\right)}\otimes \cdots \otimes {\mathit{U}}_{(k+1)}\otimes bm{U}_{(k-1)}\otimes \cdots \otimes {\mathit{U}}_{\left(1\right)}\right){\mathcal{G}}_{\left(k\right)}^{†}\end{array}$$

(35)

For the third-order tensor scenario, Equation (39) can be degenerated to the following formula:

$$\begin{array}{c}\hfill {\mathit{U}}_1^{\prime }\leftarrow {\left({\chi }_{211}^{(t+1)}\right)}_{\left(1\right)}\left({\mathit{U}}_3^{\left(t\right)}\otimes {\mathit{U}}_2^{\left(t\right)}\right){\left({\mathcal{G}}^{\left(t\right)}\right)}_{\left(1\right)}^{†}.\end{array}$$

(36)

Here, ${\mathcal{G}}_{\left(1\right)}^{\left(t\right)}$ represents the expansion of ${\mathcal{G}}^{\left(t\right)}$ at mode-1. The dimensionality of ${\mathcal{G}}^{\left(t\right)}$ is significantly smaller than that of tensor ${\chi }^{\left(t\right)}$.

${\mathit{U}}_2^{\left(t\right)}$ and ${\mathit{U}}_3^{\left(t\right)}$ represent the projection matrices at time t, which are transformed by Equation (40) in order to reduce the risk of explosive computational cost incurred by direct multiplications between the core tensor and the projection matrices. Additionally, the projection matrices are broadcast to each workhorse, facilitating parallel updates of the projection matrices.

In Section 3.2, we propose a mechanism to allocate sub-tensors uniformly among workhorses to realize parallel processing. Take Equation (40) as an example, the update operation for projection matrices can be divided into two independent tasks:

(1)

${\left({\chi }_{211}^{(t+1)}\right)}_{\left(1\right)}\left({\mathit{U}}_3^{\left(t\right)}\otimes {\mathit{U}}_2^{\left(t\right)}\right)$, where ${\left({\chi }_{211}^{(t+1)}\right)}_{\left(1\right)}$ is a matricized tensor, and $\left({\mathit{U}}_3^{\left(t\right)}\otimes {\mathit{U}}_2^{\left(t\right)}\right)$ is the Khatri–Rao product of the projection matrices. This task is essentially the multiplication of a matricized tensor by the Khatri–Rao product of the projection matrices, known as matricized tensor times Khatri–Rao product (MTTKRP) [44]. Given ${\mathit{U}}_2^{\left(t\right)}\in {\mathcal{R}}^{J\times Q}$ and ${\mathit{U}}_3^{\left(t\right)}\in {\mathcal{R}}^{K\times R}$, the Khatri–Rao product ${\mathit{U}}_3^{\left(t\right)}\otimes {\mathit{U}}_2^{\left(t\right)}$ yields a matrix of size ${\mathcal{R}}^{JK\times QR}$.

Considering the situation that slices of the matricized tensor ${\left({\chi }_{211}^{(t+1)}\right)}_{\left(1\right)}$ have been allocated among different workhorses, the overall computation task ${\left({\chi }_{211}^{(t+1)}\right)}_{\left(1\right)}\left({\mathit{U}}_3^{\left(t\right)}\otimes {\mathit{U}}_2^{\left(t\right)}\right)$ can be divided into several sub-tasks according to the allocation results of sub-tensors, which can compute in parallel. Let $M={\left({\chi }_{211}^{(t+1)}\right)}_{\left(1\right)}\left({\mathit{U}}_3^{\left(t\right)}\otimes {\mathit{U}}_2^{\left(t\right)}\right)$, the row-wise computation can be described as follows:

$$\begin{array}{ccc}\hfill M(i,:)& =& \sum _{z=0}^{JK}{\left({\chi }_{211}^{(t+1)}\right)}_{\left(1\right)}(i,z)\left({\mathit{U}}_3^{\left(t\right)}\left({\displaystyle \frac{z}{j}},:\right)\otimes {\mathit{U}}_2^{\left(t\right)}\left(z\%j,:\right)\right)\hfill \end{array}$$

(37)

$$\begin{array}{ccc}\hfill & =& \sum _{k=0}^K\sum _{j=0}^J{\left({\chi }_{211}^{(t+1)}\right)}_{\left(1\right)}(i,j,k)\left({\mathit{U}}_3^{\left(t\right)}(k,:)\otimes {\mathit{U}}_2^{\left(t\right)}(j,:)\right)\hfill \end{array}$$

(38)

which is derived by partitioning $JK$ slices of the matricized tensor ${\left({\chi }_{211}^{(t+1)}\right)}_{\left(1\right)}$ into two parts, with J and K components, respectively. It can be seen that the involved rows in ${\mathit{U}}_2^{\left(t\right)}$ and ${\mathit{U}}_3^{\left(t\right)}$ are determined by indices of ${\left({\chi }_{211}^{(t+1)}\right)}_{\left(1\right)}$. Therefore, each workhorse can determine the indices j and k according to non-zero elements in slices $\left\{P_i^{\left(n\right)}\right\}$ assigned to it and select the required rows in ${\mathit{U}}_2^{\left(t\right)}$ and ${\mathit{U}}_3^{\left(t\right)}$ for computation. At the same time, the selected rows are broadcast to all workhorses and participate in relevant multiplication operations.

(2)

The pseudo-inverse of ${\mathcal{G}}^{\left(t\right)}$, which can be calculated directly, since its dimensionality is much smaller than that of ${\chi }^{\left(t\right)}$.

When each workhorse finishes its tasks, the results are combined to produce the final output. Additionally, by using the proposed parallel dynamic programming-based sub-tensor partitioning mechanism (PSTPA-DP), the uniformity of non-zero element distribution is effectively ensured, leading to a relatively balanced computation assignment among workhorses.

B. The Second Level of Parallel Update—Using Sub-tensors In A Parallel Manner

Now that the parallel computation has been established to update the projection matrices using the sub-tensor ${\chi }_{211}^{(t+1)}$ in ${\mathcal{C}}_1$ classification, we can derive the updating rules for projection matrices using other sub-tensors. Firstly, three auxiliary matrices should be updated in a parallel manner using sub-tensors in three classifications (i.e., ${\mathcal{C}}_1$, ${\mathcal{C}}_2$, ${\mathcal{C}}_3$) in turn.

(1)

Parallel Update of Auxiliary Matrices Using ${\mathcal{C}}_1$ Classification Sub-Tensors

In addition to the sub-tensor ${\chi }_{211}^{(t+1)}$, the ${\mathcal{C}}_1$ classification contains sub-tensors ${\chi }_{121}^{(t+1)}$ and ${\chi }_{112}^{(t+1)}$, which are leveraged to update the auxiliary matrices:

$$\begin{array}{c}\hfill {\mathit{U}}_2^{\prime }\leftarrow {\left({\chi }_{121}^{(t+1)}\right)}_{\left(2\right)}\left({\mathit{U}}_3^{\left(t\right)}\otimes {\mathit{U}}_1^{\left(t\right)}\right){\left({\mathcal{G}}^{\left(t\right)}\right)}_{\left(2\right)}^{†}\end{array}$$

(39)

$$\begin{array}{c}\hfill {\mathit{U}}_3^{\prime }\leftarrow {\left({\chi }_{112}^{(t+1)}\right)}_{\left(3\right)}\left({\mathit{U}}_2^{\left(t\right)}\otimes {\mathit{U}}_1^{\left(t\right)}\right){\left({\mathcal{G}}^{\left(t\right)}\right)}_{\left(3\right)}^{†}\end{array}$$

(40)

The update of each auxiliary matrix using sub-tensors in ${\mathcal{C}}_1$ classification only involves elements generated at timestamp t, which are independent of each other, as shown by Equations (40), (43) and (44), ${\mathit{U}}_1^{\prime }$, ${\mathit{U}}_2^{\prime }$, ${\mathit{U}}_3^{\prime }$ can be updated in a parallel manner.

(2)

Parallel Update of Auxiliary Matrices Using ${\mathcal{C}}_2$ Classification Sub-Tensors

The ${\mathcal{C}}_2$ classification consists of sub-tensors ${\chi }_{221}^{(t+1)}$, ${\chi }_{212}^{(t+1)}$ and ${\chi }_{122}^{(t+1)}$, which are used to update three auxiliary matrices:

$$\begin{array}{cc}\hfill {\mathit{U}}_1^{\prime \left(221\right)}& \leftarrow \alpha {\mathit{U}}_1^{\prime }+(1-\alpha ){\left({\chi }_{221}^{(t+1)}\right)}_{\left(1\right)}\left(\left({\mathit{U}}_3^{\left(t\right)}\right)\otimes \left({\mathit{U}}_2^{\prime }\right)\right){\left({\mathcal{G}}^{\left(t\right)}\right)}_{\left(1\right)}^{†}\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill {\mathit{U}}_2^{\prime \left(221\right)}& \leftarrow \alpha {\mathit{U}}_2^{\prime }+(1-\alpha ){\left({\chi }_{221}^{(t+1)}\right)}_{\left(2\right)}\left(\left({\mathit{U}}_3^{\left(t\right)}\right)\otimes \left({\mathit{U}}_1^{\prime }\right)\right){\left({\mathcal{G}}^{\left(t\right)}\right)}_{\left(2\right)}^{†}\hfill \end{array}$$

(42)

$$\begin{array}{cc}\hfill {\mathit{U}}_1^{\prime \left(212\right)}& \leftarrow \alpha {\mathit{U}}_1^{\prime \left(221\right)}+(1-\alpha ){\left({\chi }_{212}^{(t+1)}\right)}_{\left(1\right)}\left(\left({\mathit{U}}_3^{\prime }\right)\otimes \left({\mathit{U}}_2^{\left(t\right)}\right)\right){\left({\mathcal{G}}^{\left(t\right)}\right)}_{\left(1\right)}^{†}\hfill \end{array}$$

(43)

$$\begin{array}{cc}\hfill {\mathit{U}}_3^{\prime \left(212\right)}& \leftarrow \alpha {\mathit{U}}_3^{\prime }+(1-\alpha ){\left({\chi }_{212}^{(t+1)}\right)}_{\left(3\right)}\left(\left({\mathit{U}}_2^{\left(t\right)}\right)\otimes \left({\mathit{U}}_1^{\prime }\right)\right){\left({\mathcal{G}}^{\left(t\right)}\right)}_{\left(3\right)}^{†}\hfill \end{array}$$

(44)

$$\begin{array}{cc}\hfill {\mathit{U}}_2^{\prime \left(122\right)}& \leftarrow \alpha {\mathit{U}}_2^{\prime \left(221\right)}+(1-\alpha ){\left({\chi }_{122}^{(t+1)}\right)}_{\left(2\right)}\left(\left({\mathit{U}}_3^{\prime }\right)\otimes \left({\mathit{U}}_1^{\left(t\right)}\right)\right){\left({\mathcal{G}}^{\left(t\right)}\right)}_{\left(2\right)}^{†}\hfill \end{array}$$

(45)

$$\begin{array}{cc}\hfill {\mathit{U}}_3^{\prime \left(122\right)}& \leftarrow \alpha {\mathit{U}}_3^{\prime \left(212\right)}+(1-\alpha ){\left({\chi }_{122}^{(t+1)}\right)}_{\left(3\right)}\left(\left({\mathit{U}}_2^{\prime }\right)\otimes \left({\mathit{U}}_1^{\left(t\right)}\right)\right){\left({\mathcal{G}}^{\left(t\right)}\right)}_{\left(3\right)}^{†}\hfill \end{array}$$

(46)

Here, ${\mathit{U}}_k^{\prime \left(i_1{i}_2{i}_3\right)}$ means the updated auxiliary matrix ${\mathit{U}}_k^{\prime }$ using sub-tensor ${\chi }_{i_1{i}_2{i}_3}^{(t+1)}$ in the ${\mathcal{C}}_2$ classification.

The situation is slightly different from that using sub-tensors in ${\mathcal{C}}_1$ classification, i.e., each auxiliary matrix has to be updated twice using different sub-tensors, and there is a dependency between these two updates, e.g., output of Equation (45) provides an element for Equation (47). However, the elements required for updates of different auxiliary matrices are independent of each other. Therefore, Equations (45), (46) and (48) can be calculated in a parallel manner, and so can Equations (47), (49) and (50).

(3)

Parallel Update of Auxiliary Matrices Using ${\mathcal{C}}_3$ Classification Sub-Tensors

The ${\mathcal{C}}_3$ classification contains only one sub-tensor, i.e., ${\chi }_{222}^{(t+1)}$, which is used to update the auxiliary matrices:

$$\begin{array}{cc}\hfill {\mathit{U}}_1^{\prime \left(222\right)}& \leftarrow \alpha {\mathit{U}}_1^{\prime \left(212\right)}+(1-\alpha ){\left({\chi }_{222}^{(t+1)}\right)}_{\left(1\right)}\left(\left({\mathit{U}}_3^{\prime }\right)\otimes \left({\mathit{U}}_2^{\prime }\right)\right){\left({\mathcal{G}}^{\left(t\right)}\right)}_{\left(1\right)}^{†}\hfill \end{array}$$

(47)

$$\begin{array}{cc}\hfill {\mathit{U}}_2^{\prime \left(222\right)}& \leftarrow \alpha {\mathit{U}}_2^{\prime \left(122\right)}+(1-\alpha ){\left({\chi }_{222}^{(t+1)}\right)}_{\left(2\right)}\left(\left({\mathit{U}}_3^{\prime }\right)\otimes \left({\mathit{U}}_1^{\prime }\right)\right){\left({\mathcal{G}}^{\left(t\right)}\right)}_{\left(2\right)}^{†}\hfill \end{array}$$

(48)

$$\begin{array}{cc}\hfill {\mathit{U}}_3^{\prime \left(222\right)}& \leftarrow \alpha {\mathit{U}}_3^{\prime \left(122\right)}+(1-\alpha ){\left({\chi }_{222}^{(t+1)}\right)}_{\left(3\right)}\left(\left({\mathit{U}}_2^{\prime }\right)\otimes \left({\mathit{U}}_1^{\prime }\right)\right){\left({\mathcal{G}}^{\left(t\right)}\right)}_{\left(3\right)}^{†}\hfill \end{array}$$

(49)

The sub-tensor in ${\mathcal{C}}_3$ classification further updates the auxiliary matrices obtained using sub-tensors in ${\mathcal{C}}_2$, and the elements involved in this update process are independent as well. Therefore, Equations (51)–(53) can be calculated in a parallel manner.

So far, the update of the auxiliary matrices $\left\{{\mathit{U}}_k^{\prime }\right\}$ comes to an end. Based on the updated $\left\{{\mathit{U}}_k^{\prime }\right\}$, projection matrices $\left\{{\mathit{U}}_k^{(t+1)}\right\}$ at timestamp $t+1$ can be updated by concatenating $\left\{{\mathit{U}}_k^{\prime }\right\}$ and the projection matrices $\left\{{\mathit{U}}_k^{\left(t\right)}\right\}$, followed by an orthogonalization of the concatenated output ${\left(A_k^{(t+1)}\right)}^T$, as defined by Equation (30).

In order to avoid computing pseudo-inverse ${\left({\mathcal{G}}^{\left(t\right)}\right)}^{†}$ of the matricized core tensor multiple times during the update process, once the ${\left({\mathcal{G}}^{\left(t\right)}\right)}^{†}$ has been computed for the first time (i.e., in the update process of sub-tensors using sub-tensors in ${\mathcal{C}}_1$ classification), it is stored in the Master node, which is accessible for subsequent updates. This approach highlights its capability to avoid redundant computations and save memory space.

In summary, since tensors involved in this subsection are third-order, the degree of parallelism is $3p_n$. If this approach is extended to the tensors with K modes $(K>3)$, the degree of parallelism is increased to $K{p}_n$, which can make more efficient use of the workhorses and improve computational efficiency.

3.3.2. Two-Level Parallel Update Method for Core Tensor

   From Equation (31) in Section 3.1.3, which provides the update rule for core tensors, it can be seen that the update process consists of two computational tasks, i.e., ${\mathcal{G}}^{\left(t\right)}\times \left\{{\left({\mathit{U}}_{k,1}^{(t+1)}\right)}^T{\mathit{U}}_k^{\left(t\right)}\right\}$ and ${\sum }_{(i_1,i_2,i_3)\ne (1,1,1)}{\chi }_{i_1{i}_2{i}_3}^{(t+1)}\times \left\{{\left({\mathit{U}}_{k,i_k}^{(t+1)}\right)}^T\right\}$.

The first task involves only the core tensor ${\mathcal{G}}^{\left(t\right)}$ and the projection matrices $\left\{{\left({\mathit{U}}_{k,1}^{(t+1)}\right)}^T{\mathit{U}}_k^{\left(t\right)}\right\}$, which are both small scale, making it feasible to be performed directly. The second task multiplies the sub-tensors (except for the ${\chi }_{111}^{(t+1)}$) by a series of matrices $\left\{{\left({\mathit{U}}_{k,i_k}^{(t+1)}\right)}^T\right\}$ and then sums all the products, which can be considered a MTTKRP computation as well. By applying the parallel MTTKRP computation method described in Section 3.3.1, we can efficiently update the core tensor in parallel. Take the sub-tensor ${\chi }_{211}^{(t+1)}$ in ${\mathcal{C}}_1$ classification as an example, ${\sum }_{(i_1,i_2,i_3)\ne (1,1,1)}{\chi }_{i_1{i}_2{i}_3}^{(t+1)}\times \left\{{\left({\mathit{U}}_{k,i_k}^{(t+1)}\right)}^T\right\}$ can be transformed as follows:

$${\chi }_{211}^{(t+1)}{\times }_1{\left({\mathit{U}}_{1,2}^{(t+1)}\right)}^T{\times }_2{\left({\mathit{U}}_{2,1}^{(t+1)}\right)}^T{\times }_3{\left({\mathit{U}}_{3,1}^{(t+1)}\right)}^T.$$

(50)

Considering the properties of the Kronecker product and mode multiplication of the tensors, Equation (50) can be expressed as

$${\left({\mathit{U}}_{1,2}^{(t+1)}\right)}^T{\times }_1{\left({\chi }_{211}^{(t+1)}\right)}_{\left(1\right)}\left({\left({\mathit{U}}_{3,1}^{(t+1)}\right)}^T\otimes {\left({\mathit{U}}_{2,1}^{(t+1)}\right)}^T\right).$$

(51)

Here, ${\left({\chi }_{211}^{(t+1)}\right)}_{\left(1\right)}\left({\left({\mathit{U}}_{3,1}^{(t+1)}\right)}^T\otimes {\left({\mathit{U}}_{2,1}^{(t+1)}\right)}^T\right)$ corresponds to the MTTKRP computation. Furthermore, the computation of ${\sum }_{(i_1,i_2,i_3)\ne (1,1,1)}{\chi }_{i_1{i}_2{i}_3}^{(t+1)}\times \left\{{\left({\mathit{U}}_{k,i_k}^{(t+1)}\right)}^T\right\}$ involves summing the products of sub-tensor multiplications. Because there is no dependency between operations on different sub-tensors, such operations can be processed in a parallel manner. In this subsection, we realize the two-level parallel update method for the core tensor through two steps:

(1)

Computation of ${\chi }_{i_1{i}_2{i}_3}^{(t+1)}\times \left\{{\left({\mathit{U}}_{k,i_k}^{(t+1)}\right)}^T\right\}$ related to different sub-tensors can be performed in a parallel manner; and

(2)

Application of the MTTKRP parallel computation method, described in Section 3.3.1, to each sub-tensor.

Assuming the sub-tensor is partitioned into $p_n$ parts at mode-n for processing, the degree of parallelism in the traditional single-level parallel update method is $p_n$. For a sub-tensor with K modes, it can be partitioned into $2^k$ sub-tensors, and the sub-tensors involved in the cumulative computation ${\sum }_{(i_1,i_2,i_3)\ne (1,1,1)}{\chi }_{i_1{i}_2{i}_3}^{(t+1)}\times \left\{{\left({\mathit{U}}_{k,i_k}^{(t+1)}\right)}^T\right\}$ can reach $2^{k-1}$. As a result, the degree of parallelism in this two-level parallel update method can be increased to $\left(2^k-1\right)p_n$.

3.3.3. Parallel Update Method for High-Order Tensors

   The update methods for third-order projection matrices and the core tensor described in the previous subsections should be extended to higher-order tensors. To be specific, we propose a complete parallel incremental Tucker decomposition method for high-order tensors with multi-mode growth, as shown by pseudo code in Algorithm 3.

Algorithm 3 Incremental Tucker Decomposition Method for High-Order Tensors with Multi-Mode Growth

Input: (1) ${\chi }^{\left(t\right)}={\mathcal{G}}^{\left(t\right)}\times \left\{{\mathit{U}}_k^{\left(t\right)}\right\}$: The Tucker decomposition result of ${\chi }^{\left(t\right)}$ at timestamp t, including the core tensor ${\mathcal{G}}^{\left(t\right)}$ and projection matrices $\left\{{\mathit{U}}_k^{\left(t\right)}\right\}$;              (2) ${\chi }_{i_1\cdots i_k}^{(t+1)}$: The incremental sub-tensors at time $t+1$, where $(i_1,i_2,\cdots ,i_K)\ne (1,1,\cdots ,1)$;              (3) $p_n$: Number of partitions for mode-n of a sub-tensors.

Output: ${\chi }^{(t+1)}={\mathcal{G}}^{(t+1)}\times \left\{{\mathit{U}}_k^{(t+1)}\right\}$: The Tucker decomposition result of the sub-tensor at time $t+1$, including the updated core tensor ${\mathcal{G}}^{(t+1)}$ and projection matrices $\left\{{\mathit{U}}_k^{(t+1)}\right\}$.

1:  partition the incremental sub-tensor ${\chi }_{i_1\cdots i_k}^{(t+1)}$ into K classifications ${\left\{{\mathcal{C}}_m\right\}}_{m=1}^K$;   2:  for $1\le m\le K$ do   3:     parallel processing of the sub-tensors ${\chi }_{i_1\cdots i_k}^{(t+1)}$ within ${\mathcal{C}}_m$;   4:     partition the sub-tensors ${\chi }_{i_1\cdots i_k}^{(t+1)}$ according to Algorithm 2;   5:     parallel update of the auxiliary matrices $\left\{{\mathit{U}}_k^{\prime }\right\}$ based on Equation (52);   6:  end for   7:  for $1\le m\le K$ do   8:     concatenate the auxiliary matrices and projection matrices to form a temporary projection matrices $FM$ according to Equation (30);   9:     Perform orthogonalization on the $FM$; 10:  end for 11:  update the core tensor in parallel according to Equation (53); 12:  output the core tensor ${\mathcal{G}}^{(t+1)}$ and projection matrices $\left\{{\mathit{U}}_k^{(t+1)}\right\}$ at time $t+1$.

For a tensor with K modes, the incremental tensor is first partitioned into $2^K-1$ sub-tensors based on their relative position to the original tensor ${\chi }_{1\cdots 1}={\chi }^{\left(t\right)}$. According to the number of digit 2 in indices of the sub-tensor ${\chi }_{i_1\cdots i_k}^{(t+1)}$ in Step (1), these sub-tensors are grouped into K classifications ${\left\{{\mathcal{C}}_m\right\}}_{m=1}^K$. Sub-tensors in each classification are further partitioned by the proposed PSTPA-DP, as described in Step (4). Subsequently, the projection matrices are updated using sub-tensors in ${\mathcal{C}}_1,{\mathcal{C}}_2,\cdots ,{\mathcal{C}}_m$ in turn. Using different sub-tensors in each classification, projection matrices can be updated in a parallel manner, corresponding to the first level of parallel updating. Since the sub-tensors have already been partitioned and the MTTKRP parallel computation method has been involved, the update of each projection matrix ${\mathit{U}}_k$ can also be performed in a parallel manner, i.e., the second level of parallel processing. The update formulas for high-order projection matrices are described as follows:

$${\mathit{U}}_{\left(k\right)}^{\prime new}\leftarrow \alpha {\mathit{U}}_{\left(k\right)}^{\prime old}+(1-\alpha ){\left({\chi }_{i_1\cdots i_k}^{(t+1)}\right)}_{\left(k\right)}\left({\mathit{U}}_{\left(K\right)}\otimes \cdots \otimes {\mathit{U}}_{(k+1)}\otimes {\mathit{U}}_{(k-1)}\otimes \cdots \otimes {\mathit{U}}_{\left(1\right)}\right){\left({\mathcal{G}}^{\left(t\right)}\right)}_{\left(K\right)}^{†}$$

(52)

When $i_k=1,{\mathit{U}}_{\left(K\right)}={\mathit{U}}_{\left(K\right)}^{\left(t\right)}$; when $i_k=2,{\mathit{U}}_{\left(K\right)}={\mathit{U}}_{\left(K\right)}^{\prime }$. This formula corresponds to Step (5), i.e., updating of auxiliary matrices. The updated auxiliary matrices are concatenated with the projection matrices and the concatenated result is orthogonalized, corresponding to Steps (7)–(10). At this point, the projection matrices $\left\{{\mathit{U}}_k^{(t+1)}\right\}$ at time $t+1$ are obtained. Finally, the update of the high-order core tensor is described as follows:

$$\begin{array}{c}\hfill {\mathcal{G}}^{(t+1)}={\mathcal{G}}^{\left(t\right)}\times \left\{{\left({\mathit{U}}_{k,1}^{(t+1)}\right)}^T{\mathit{U}}_k^{\left(t\right)}\right\}+\sum _{(i_1,\dots ,i_k)\ne (1,\dots ,1)}{\chi }_{i_1\cdots i_k}^{(t+1)}\times \left\{{\left({\mathit{U}}_{k,i_k}^{(t+1)}\right)}^T\right\}\end{array}$$

(53)

Equation (53) corresponds to Step (11) of Algorithm 3. So far, the incremental Tucker decomposition method for high-order tensors with multi-mode growth is ready to yield the core tensor ${\mathcal{G}}^{(t+1)}$ and projection matrices $\left\{{\mathit{U}}_k^{(t+1)}\right\}$ at time $t+1$.

4. Experimental Analysis

Experiments in this section are conducted to evaluate performance of the proposed algorithm from the perspective of three tasks: the tensor decomposition and reconstruction, a comparative analysis on the computational efficiency of sub-tensor partitioning algorithms, and an assessment on the parallel update efficiency of projection matrices and core tensors. The experimental environment is comprised of the following hardware and software settings:

(1)

Five DELL workstations, each of them is equipped with an Intel^® Core^TM i7-9700 CPU, 16 GB of RAM, and a 2TB hard drive. One workstation serves as the Master node and the other four as Slave nodes.

(2)

The key algorithms are implemented using Scala 2.12.10 and executed on a Spark 3.0.3 parallel computing cluster.

(3)

The overall system of parallel computation for tensors is developed using the Java programming language 1.8.0_281.

(4)

Simple build tool (sbt) is employed as the build tool to package the project, which is deployed to the Spark cluster. The sbt-assembly is introduced to facilitate the application build process.

(5)

External tools such as scopt and netlib are incorporated for project initialization, because sbt does not inherently package them into the application.

For the comprehensive performance evaluation of the TPITTD-MG algorithm, datasets with uniformly-distributed non-zero elements are generated using Numpy and used as the foundation to construct third-order tensors.

4.1. Experiments on Tensor Decomposition and Reconstruction

In this subsection, a couple of random matrices, $X_1,X_2,\cdots ,X_n(n\ge 3)$, are generated and standardized. By performing operations such as translation and matrix multiplication, etc., these matrices are assembled into three tensors, i.e., the original tensors ${\chi }_1^{\left(t\right)}$, ${\chi }_2^{\left(t\right)}$ and ${\chi }_3^{\left(t\right)}$ at time t, which are decomposed by the Tucker decomposition to obtain a core tensor and a series of projection matrices. At time $t+1$, new increments have been added to each mode of the tensor ${\chi }_i^{\left(t\right)}(i=1,2,3)$, generating the tensor ${\chi }_i^{(t+1)}(i=1,2,3)$. Subsequently, the proposed TPITTD-MG algorithm is applied to yield the core tensor and projection matrices at time $t+1$ by updating their counterparts at time t in a parallel manner, using the new coming data.

The tensors ${\chi }_i^{(t+1)}(i=1,2,3)$ are reconstructed, denoted by ${\widehat{\chi }}_i^{(t+1)}(i=1,2,3)$, on the basis of decomposition results at time $t+1$. An element-wise comparison between ${\widehat{\chi }}_i^{(t+1)}$ and ${\chi }_i^{(t+1)}$ is shown in Figure 6.

In Figure 6, the first row demonstrates the element-wise comparison between ${\widehat{\chi }}_1^{(t+1)}$ and ${\chi }_1^{(t+1)}$ at each mode, represented by blue and red curves, respectively. The second and third rows provide similar comparison for the second tensor and the third one in turn. It is evident that the reconstructed tensor closely fits the original tensor, implying the core tensor and the projection matrices obtained through TPITTD-MG perfectly preserve the information of the original tensor, validating the accuracy of the algorithm.

4.2. Execution Efficiency Comparison of Sub-Tensor Partitioning Algorithms

This subsection verifies time efficiency of the parallel sub-tensor partitioning mechanism based on the dynamic programming (PSTPA-DP) proposed in Section 4.2 and compares it with the DisMASTD method, the heuristic sub-tensor partitioning algorithm shown in Algorithm 1.

Set the parallel degree to be 4; we compare the execution efficiency of the two algorithms over synthetic datasets with uniformly-distributed non-zero elements at different tensor modes. The experimental results are depicted as blue and orange plots, respectively, in Figure 7a.

It can be seen that given a parallel degree of 4, there is no distinguishable difference between the performance of the two algorithms for smaller tensors. However, as the tensor size increases, the advantage of PSTPA-DP over DisMASTD becomes more and more obvious. The same phenomenon is true when the parallel degree is set to 8, which is shown in Figure 7b. Specifically, when the parallel degree increases from 4 to 8, the efficiency of PSTPA-DP is also enhanced.

Specifically, the efficiency of PSTPA-DP is empirically validated in the scenarios characterized by nonuniform distribution of non-zero elements, which might lead to imbalanced partitions by traditional heuristic tensor partition algorithms. To assess the uniformity of partition, we introduce the term “coefficient of variation”, defined as the ratio of standard deviation to mean, $c_v=\sigma /\mu$, as a metric to quantify the uniformity of partition results. For that purpose, the tensor datasets of identical volume but varying $c_v$ are synthesized. $c_v$’s of the partitions generated by both algorithms over datasets with different initial $c_v$’s are computed. Figure 8 illustrates performance of the two tensor partitioning algorithms over datasets with varied initial $c_v$’s.

Figure 8 reveals that for datasets with low $c_v$, two partition algorithms exhibit negligible differences. However, as $c_v$ of the datasets increases, the partition generated by the PSTPA-DP demonstrates significantly higher uniformity compared with those produced by DisMASTD.

4.3. Efficiency Comparison of Parallel Projection Matrix and Core Tensor Updates

This subsection deals with the execution efficiency in updating projection matrices and core tensors, which is conducted over synthetic datasets of different scales, employing both single-level and two-level parallel update methods, as shown in

Table 3. Execution Time of Different Steps for Different Algorithms.

Data Scale	Algorithm Name	Tensor Partition Execution Time (s)	Projection Matrix Update Execution Time (s)	Core Tensor Update Execution Time (s)	Total Update Execution Time (s)
10∗10∗10	Single Layer Update	1.01	1.21	0.34	2.58
10∗10∗10	Two Layer Update	1.19	1.32	0.31	2.87
100∗10∗10	Single Layer Update	1.19	1.11	0.33	2.80
100∗10∗10	Two Layer Update	1.18	1.13	0.42	2.76
100∗100∗10	Single Layer Update	2.49	2.03	0.33	5.03
100∗100∗10	Two Layer Update	1.82	1.54	0.28	4.01
100∗100∗100	Single Layer Update	9.69	7.11	0.38	17.80
100∗100∗100	Two Layer Update	3.47	3.02	0.35	7.78
1000∗100∗100	Single Layer Update	44.32	42.23	1.01	91.87
1000∗100∗100	Two Layer Update	12.53	14.61	0.99	30.94

. As for the single-layer case, the heuristic sub-tensor partitioning algorithm is employed, while the two-level method involves the proposed PSTPA-DP.

Table 3 enumerates the execution time of each step of TPITTD-MG over different datasets with different partition methods. The execution time of the two-level parallel update method is almost the same as the single-level version over small-scale datasets. For medium-scale datasets, the former performs slightly better than the latter. For large-scale datasets, the former can significantly reduce the execution time compared with the latter. Such observations are consistent with the theoretical conclusion, verifying the effectiveness of the two-level parallel update algorithm.

Meanwhile, the effectiveness of TPITTD-MG is verified as well by a comparison with two reference algorithms, i.e., PCSTF and DPHTD, proposed by Yao et al. [34] and Ning et al. [35], respectively, as shown in

Table 4. Comparison of Distributed Tucker Decomposition Methods.

Algorithm	Platform	Parallelism	Data Scale	Key Optimization	Speedup
Yao (PCSTF)	Spark	4 workers	200 × 200 × 93	Distributed core tensor update	3.0
Ning (DPHTD)	OpenStack	4 × 4 workers	128 × 129 × 320	Coarse/fine-grained parallelism	4.1
TPITTD-MG	Spark	4 workers	1000 × 100 × 100	Dynamic subtensor allocation, MTTKRP	4.0

. Although TPITTD-MG operated over 4 workers provides a slightly lower speedup compared with the DPHTD operated over 16 workers, it outperforms PCSTF significantly under the same condition.

5. Conclusions

In this paper, we attempt to reduce the computational complexity associated with the incremental Tucker tensor decomposition by introducing two methods, i.e., a parallel sub-tensor partitioning algorithm based on dynamic programming (PSTPA-DP), and a two-level parallel update method. The first method addresses the inefficiencies encountered in counting non-zero elements during sub-tensor partitioning, leading to an even partition of sub-tensors. The second one aims to accelerate computation by an increasing degree of parallel, it consists of two levels of parallel: the first level of parallel updates using a parallel MTTKRP computation strategy, the second level of parallel updates leveraging the independence property of updating processes of projection matrices using different sub-tensors within the same classification. By integration of these two methods with the Tucker decomposition, we propose a two-level parallel incremental tensor Tucker decomposition method with multi-mode growth (TPITTD-MG) and conduct extensive experiments over synthesized datasets to validate the effectiveness and efficiency of the mechanism involved as well as the algorithm proposed. The experimental results show that in the case of a data scale in the tens of millions and a parallel degree of 4, TPITTD-MG improves execution efficiency by nearly 400% compared to the existing algorithms, with partitioning uniformity improved by over 20%; For third-order tensors, the execution efficiency of this algorithm improves by nearly 300% compared to the single-level update algorithm. It can be concluded that transitioning the decomposition of the initial tensor to a parallel execution is a practical strategy to substantially elevate computational efficiency.