Tensor Completion Based on Triple Tubal Nuclear Norm

Abstract

Many tasks in computer vision suffer from missing values in tensor data, i.e., multi-way data array. The recently proposed tensor tubal nuclear norm (TNN) has shown superiority in imputing missing values in 3D visual data, like color images and videos. However, by interpreting in a circulant way, TNN only exploits tube (often carrying temporal/channel information) redundancy in a circulant way while preserving the row and column (often carrying spatial information) relationship. In this paper, a new tensor norm named the triple tubal nuclear norm (TriTNN) is proposed to simultaneously exploit tube, row and column redundancy in a circulant way by using a weighted sum of three TNNs. Thus, more spatial-temporal information can be mined. Further, a TriTNN-based tensor completion model with an ADMM solver is developed. Experiments on color images, videos and LiDAR datasets show the superiority of the proposed TriTNN against state-of-the-art nuclear norm-based tensor norms.

1. Introduction

In recent decades, the rapid progress in multi-linear algebra [1] has provided a firm theoretical foundation for many applications in computer vision [2], data mining [3], machine learning [4], signal processing [5], and many other areas. Benefiting from its multi-way nature, the tensor has power against the vector and matrix in exploiting multi-way information in multi-modal data, like color images [6], videos [7], hyper-spectral images [8], functional magnetic resonance imaging [9], traffic volume data [10], etc. In many computer vision tasks, the data, like color images or videos, may be moderately redundant, then it can be interpreted by fewer latent factors [11]. The low-rank tensor provides a suitable model for such data [12]. The two most well-known low-rank tensor models are the low-CP-rank model [13], which tries to interpret a tensor in the fewest rank-one tensors [14], and the low-Tucker-rank model [15], which seeks a tensor proxy that is simultaneously low-rank along each mode.

In many computer vision applications, like image or video inpainting, one has to tackle the missing values in the observed data tensor due to many circumstances [2,16], including failure of sensors, errors or loss in communication, occlusions or noise in the environment, etc. However, it is obviously unable to fill in the missing entries perfectly since they can take arbitrary values without other priors taken into consideration. The most adopted prior is the low-rank prior assuming the underlying data tensor has low rank. Low-rank tensor completion [2,17] seeks a low-rank tensor to fit the underlying data tensor. It has been a hot research topic due its wide use [18]. In low-rank tensor recovery, the rank minimization problem (RMP) is often formulated [2]. However, the general rank minimization problem and most tensor problems are NP-hard [19,20]. To obtain polynomial-time algorithms, many different tensor rank surrogates have been proposed [2,7,17,21,22,23,24] to substitute the rank functions in RMP. Surrogates of the tensor CP rank and Tucker rank have been broadly studied [7,17,23,25,26,27,28,29].

Recently, a novel low-rank tensor model called the low-tubal-rank model was proposed [22,30]. The core of it is to model the 3D data as a tensor that has low tubal-rank [31], which is defined through a new tensor singular value decomposition (t-SVD) [1,32]. It has been successfully used in modeling multi-way real-world data, such as color images [6], videos [33], seismic data [34], WiFi fingerprint [35], MRI imaging [22], traffic volume data [36], etc. As pointed out in [37], compared with other tensor models, the low-tubal-rank tensor model is superior in capturing a “spatial-shifting” correlation, which is ubiquitous in real-world data arrays.

This paper focuses on low-tubal-rank models for tensor completion. The  recently-proposed tensor tubal nuclear norm (TNN) [30] based on t-SVD has shown superiority in imputing missing values in 3D visual data, like color images and videos. Its power lies in exploiting tube (often carrying temporal/channel information) redundancy in a circulant way while preserving the row and column (often carrying spatial information) relationship. A simple and successful variant of TNN, dubbed the twist tubal nuclear norm (t-TNN) [16], instead exploits row redundancy in a circulant way while keeping the tube relationship. However, both of them only exploit one kind of redundancy in a circulant way. In this paper, a new tensor norm dubbed the tensor triple tubal nuclear norm (TriTNN) is proposed to simultaneously exploit the row, column and tube redundancy while preserving the relative tube, row and column relationship. Based on the proposed TriTNN, a tensor completion model is studied and optimized by alternating direction multiplier methods (ADMM) [38]. Experimental results on color images, videos and LiDAR datasets demonstrate that the proposed TriTNN has better performances than other state-of-the-art nuclear norm-based tensor norms.

The paper is organized as follows. Some notations and preliminaries are presented in Section 2. The TriTNN is proposed following the introductions of the most related works in Section 3. The problem formulation and the proposed ADMM algorithm are shown in Section 4. Experimental results are reported in Section 5. We conclude this work in Section 6.

2. Notations and Preliminaries

In this section, the notations and the basic definitions are introduced.

2.1. Notations

Vectors are denoted by bold lower case letters, e.g., $\mathit{v}\in {\mathbb{R}}^n$, matrices are denoted by bold upper case letters, e.g., $\mathit{M}\in {\mathbb{R}}^{n_1\times n_2}$, and tensors are denoted by calligraphy letters, e.g., $\mathcal{T}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$. Given a third-order tensor, its fiber is defined as a 1D vector obtained by fixing all indices, but one, and its slice is a 2D matrix defined by fixing all but two indices. Given a 3D tensor $\mathcal{T}$, ${\mathcal{T}}_{ijk}$ denotes the entry with index $(i,j,k)$; ${\mathit{T}}^{\left(k\right)}:=\mathcal{T}(:,:,k)$ denotes the k-th frontal slice. $\tilde{\mathcal{T}}$ denotes the tensor after performing the fast Fourier transformation along the tube fibers of $\mathcal{T}$. Notations $\mathrm{dft}3(·)$ and $\mathrm{idft}3(·)$ are used to represent the discrete Fourier transformation (DFT) and inverse discrete Fourier transformation (IDFT) along the tube fibers of 3D tensors.

Given a matrix $\mathit{M}\in {\mathbb{R}}^{n_1\times n_2}$, its nuclear norm is defined as ${\parallel \mathit{M}\parallel }_{*}:={\sum }_{i=1}^p{\sigma }_i\left(\mathit{M}\right),$ where $p=\min \{n_1,n_2\}$ and ${\sigma }_1\left(\mathit{M}\right)\ge \cdots \ge {\sigma }_p\left(\mathit{M}\right)$ are the singular values of $\mathit{M}$ in non-ascending order. The inner product between two 3D tensors ${\mathcal{T}}_1,{\mathcal{T}}_2\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ is defined as $〈{\mathcal{T}}_1,{\mathcal{T}}_2〉:={\sum }_{ijk}{\mathcal{T}}_1(i,j,k){\mathcal{T}}_2(i,j,k).$ The  Frobenius norm of a tensor $\mathcal{T}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ is defined as ${\parallel \mathcal{T}\parallel }_{\mathrm{F}}:=\sqrt{{\sum }_{ijk}{\mathcal{T}}_{ijk}^2}$. The  $l_{\infty }$-norm of a tensor $\mathcal{T}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ is defined as ${\parallel \mathcal{T}\parallel }_{\infty }:={\max }_{ijk}\left|{\mathcal{T}}_{ijk}\right|$.

2.2. Tensor Singular Value Decomposition

Firstly, five block-based operators, i.e., bvec, bvfold, bdiag, bdfoldand bcirc [1], are introduced. Given a tensor $\mathcal{T}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$, the block vectorizing and its opposite operation are defined as follows:

$$\begin{array}{c}\hfill \mathrm{bvec}\left(\mathcal{T}\right):=\left[\begin{array}{c}{\mathcal{T}}^{\left(1\right)}\\ {\mathcal{T}}^{\left(2\right)}\\ ⋮\\ {\mathcal{T}}^{\left(n_3\right)}\end{array}\right],\mathrm{bvfold}\left(\mathrm{bvec}\left(\mathcal{T}\right)\right)=\mathcal{T},\end{array}$$

the block diag matrix and its opposite operation:

$$\begin{array}{c}\hfill \mathrm{bdiag}\left(\mathcal{T}\right):=\left[\begin{array}{ccc}{\mathcal{T}}^{\left(1\right)}& & \\ & \ddots & \\ & & {\mathcal{T}}^{\left(n_3\right)}\end{array}\right],\mathrm{bdfold}\left(\mathrm{bdiag}\left(\mathcal{T}\right)\right)=\mathcal{T},\end{array}$$

and the block circulant matrix as follows:

$$\begin{array}{c}\hfill \mathrm{bcirc}\left(\mathcal{T}\right):=\left[\begin{array}{cccc}{\mathcal{T}}^{\left(1\right)}& {\mathcal{T}}^{\left(n_3\right)}& \cdots & {\mathcal{T}}^{\left(2\right)}\\ {\mathcal{T}}^{\left(2\right)}& {\mathcal{T}}^{\left(1\right)}& \cdots & {\mathcal{T}}^{\left(3\right)}\\ ⋮& \ddots & \ddots & ⋮\\ {\mathcal{T}}^{\left(n_3\right)}& {\mathcal{T}}^{(n_3-1)}& \cdots & {\mathcal{T}}^{\left(1\right)}\end{array}\right].\end{array}$$

Based on the five operators defined above, we are able to give the definition of the tensor t-product.

Definition 1

 (t-product [1]). Let ${\mathcal{T}}_1\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ and ${\mathcal{T}}_2\in {\mathbb{R}}^{n_2\times n_4\times n_3}$. The t-product of ${\mathcal{T}}_1$ and ${\mathcal{T}}_2$ is a tensor $\mathcal{T}$ of size $n_1\times n_4\times n_3$:

$$\begin{array}{c}\hfill \mathcal{T}={\mathcal{T}}_1\ast {\mathcal{T}}_2=:\mathrm{bvfold}\left\{\mathrm{bcirc}\left({\mathcal{T}}_1\right)\mathrm{bvec}\left({\mathcal{T}}_2\right)\right\}.\end{array}$$

(1)

Viewing a 3D tensor $\mathcal{T}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ as an $n_1\times n_2$ matrix of tubes, the tensor t-product is analogous to the matrix multiplication by replacing scalar multiplication with the vector circular convolution between the tubes, as follows:

$$\begin{array}{c}\hfill \mathcal{T}(i,j,:)=\sum _{k=1}^{n_2}{\mathcal{T}}_1(i,k,:)•{\mathcal{T}}_2(k,j,:),\end{array}$$

(2)

where • denotes the circular convolution [1] between two tube vector $\mathit{x},\mathit{y}\in {\mathbb{R}}^{n_3}$ defined as:

$${(\mathit{x}•\mathit{y})}_j=\sum _{k=1}^{n_3}{\mathit{x}}_k{\mathit{y}}_{1+(j-k)\mathrm{mod}{n}_3}.$$

Due to the relationship between the circular convolution and the DFT, the t-product in the original domain is equivalent to matrix multiplication of the frontal slices in the Fourier domain [1], i.e.,

$$\begin{array}{c}\hfill {\tilde{\mathcal{T}}}^{\left(k\right)}={\widetilde{{\mathcal{T}}_1}}^{\left(k\right)}{\widetilde{{\mathcal{T}}_2}}^{\left(k\right)},k=1,\cdots ,n_3.\end{array}$$

(3)

The tensor transpose, identity tensor, f-diagonal tensor and orthogonal tensor are further defined.

Definition 2

 (Tensor transpose [1]). Let $\mathcal{T}$ be a tensor of size $n_1\times n_2\times n_3$; then ${\mathcal{T}}^{\top }$ is the $n_2\times n_1\times n_3$ tensor obtained by transposing each of the frontal slices and then reversing the order of transposed frontal Slices 2 through $n_3$.

Definition 3

 (Identity tensor [1]). The  identity tensor $\mathcal{I}\in {\mathbb{R}}^{n_1\times n_1\times n_3}$ is a tensor whose first frontal slice is the $n_1\times n_1$ identity matrix, and all other frontal slices are zero.A

Definition 4

 (F-diagonal tensor [1]). A tensor is called f-diagonal if each frontal slice of the tensor is a diagonal matrix.

Definition 5

 (Orthogonal tensor [1]). A tensor $\mathcal{Q}\in {\mathbb{R}}^{n_1\times n_1\times n_3}$ is orthogonal if it satisfies the following relationship:

$${\mathcal{Q}}^{\top }\ast \mathcal{Q}=\mathcal{Q}\ast {\mathcal{Q}}^{\top }=\mathcal{I}.$$

Based on the concepts defined above, the tensor singular value decomposition (t-SVD) and the tensor tubal rank are established as follows.

Definition 6

 (Tensor singular value decomposition and tensor tubal-rank [31]). Any tensor $\mathcal{T}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ can be decomposed as:

$$\begin{array}{c}\hfill \mathcal{T}=\mathcal{U}\ast \mathcal{S}\ast {\mathcal{V}}^{\top },\end{array}$$

(4)

where $\mathcal{U}\in {\mathbb{R}}^{n_1\times n_1\times n_3}$ and $\mathcal{V}\in {\mathbb{R}}^{n_2\times n_2\times n_3}$ are orthogonal tensors and $\mathcal{S}$ is a rectangular f-diagonal tensor of size $n_1\times n_2\times n_3$.

The tensor tubal rank of $\mathcal{T}$ is defined to be the number of non-zero tubes of $\mathcal{S}$ in Equation (4), i.e.,

$$\begin{array}{c}\hfill r_{\mathrm{tubal}}\left(\mathcal{T}\right):=\sum _i\mathbf{1}(\mathcal{S}(i,i,:)\ne \mathbf{0}),\end{array}$$

(5)

where $\mathbf{1}(·)$ is an indicator function whose value is one if the input condition is satisfied, and zero otherwise.

The t-SVD is illustrated in Figure 1. It can be computed efficiently by FFT and IFFT in the Fourier domain according to Equation (3). For more details, see [1].

3. The Triple Tubal Nuclear Norm

In this section, we will define the triple tensor tubal nuclear norm, before which the most related norms, i.e., tubal nuclear norm and twist tubal nuclear norm, are introduced first.

3.1. Tubal Nuclear Norm

Based on the preliminaries introduced in Section 2, the tubal nuclear norm is defined as follows:

Definition 7

 (Tubal nuclear norm [31]). The tubal nuclear norm (TNN) ${\parallel \mathcal{T}\parallel }_{★}$ of a 3D tensor $\mathcal{T}$ is defined as the nuclear norm of the block diagonal matrix of $\tilde{\mathcal{T}}$ (the Fourier domain version of $\mathcal{T}$), i.e.,

$$\begin{array}{c}\hfill {\parallel \mathcal{T}\parallel }_{★}:=\parallel \mathrm{bdiag}(\tilde{\mathcal{T}}){\parallel }_{*}=\sum _{k=1}^{n_3}{\parallel {\tilde{\mathcal{T}}}^{\left(k\right)}\parallel }_{*}.\end{array}$$

(6)

From Definition 7, we can compute the TNN of a tensor efficiently through first conducting FFT along the tube direction and summing the nuclear norms of each frontal slice. Given a tensor $\mathcal{T}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$, the computation cost of ${\parallel \mathcal{T}\parallel }_{★}$ is $O\left(n_1{n}_2{n}_3(n_3+\log n_3)\right)$. Since a block circulant matrix can be block diagonalized through the Fourier transform [1], we obtain:

$$\begin{array}{cc}\hfill {\parallel \mathcal{T}\parallel }_{★}& =\parallel \mathrm{bdiag}(\tilde{\mathcal{T}}){\parallel }_{*}=\parallel ({\mathit{F}}_{n_3}\otimes {\mathit{I}}_{n_1})\mathrm{bcirc}\left(\mathcal{T}\right)({\mathit{F}}_{n_3}^{-1}\otimes {\mathit{I}}_{n_2}){\parallel }_{*}={\parallel \mathrm{bcirc}\left(\mathcal{T}\right)\parallel }_{*},\hfill \end{array}$$

(7)

where ⊗ denotes the Kronecker product [14], ${\mathit{F}}_n$ is the $n\times n$ discrete Fourier transform matrix and ${\mathit{I}}_n$ is an $n\times n$ identity matrix. Note that $({\mathit{F}}_{n_3}\otimes {\mathit{I}}_{n_1})/\sqrt{n_3}$ is a unitary matrix.

The tubal nuclear norm has been used as a convex relaxation of the tensor tubal-rank for tensor completion, tensor robust principle component analysis (TRPCA) and outlier robust tensor principle component analysis (OR-TPCA) [6,30,36,39]. In optimization over TNN, one often needs to compute the proximal operator [40] of TNN defined as:

$$\begin{array}{c}\hfill {\mathfrak{S}}_{\tau }\left({\mathcal{T}}_0\right):=\underset{\mathcal{T}}{\mathrm{argmin}}\frac{1}{2}\parallel {\mathcal{T}}_0-\mathcal{T}{\parallel }_{\mathrm{F}}^2+\tau {\parallel \mathcal{T}\parallel }_{★}.\end{array}$$

(8)

In [3], a closed-form expression of ${\mathfrak{S}}_{\tau }(·)$ is given as follows:

Definition 8

 ([3]). For a 3D tensor $\mathcal{T}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ with reduced t-SVD $\mathcal{T}=\mathcal{U}\ast \mathcal{S}\ast {\mathcal{V}}^{\top }$, where $\mathcal{U}\in {\mathbb{R}}^{n_1\times r\times n_3}$ and $\mathcal{V}\in {\mathbb{R}}^{n_2\times r\times n_3}$ are orthogonal tensors and $\mathcal{S}\in {\mathbb{R}}^{r\times r\times n_3}$ is the f-diagonal tensor of singular tubes, the proximal operator ${\mathfrak{S}}_{\tau }(·)$ at ${\mathcal{T}}_0$ can be computed through the following equation:

$$\begin{array}{c}\hfill {\mathfrak{S}}_{\tau }\left(\mathcal{T}\right):=\mathcal{U}\ast \mathrm{idft}3\left(\max (\mathrm{dft}3\left(\mathcal{S}\right)-n_3\tau ,0)\right)\ast {\mathcal{V}}^{\top }.\end{array}$$

(9)

3.2. Twist Tubal Nuclear Norm

The twist tubal nuclear norm [16] is related to a pair of tensor operations named column twist and column squeeze as follows.

Definition 9

 (Tensor column twist and column squeeze (here, the twist and squeeze operations in [1,16] are renamed as column twist and column squeeze, respectively, since we will define the row twist and row squeeze) [16]). Let $\mathcal{T}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$, then the column twist tensor ${\mathcal{T}}_1=\overrightarrow{\mathcal{T}}$ is a tensor of size $n_1\times n_3\times n_2$ whose lateral slice ${\mathcal{T}}_1(:,k,:)={\mathcal{T}}^{\left(k\right)}$. Correspondingly, the column squeeze tensor of ${\mathcal{T}}_1$, i.e., $\mathcal{T}=\overleftarrow{{\mathcal{T}}_1}$, can be obtained by the reverse process, i.e., ${\mathcal{T}}^{\left(k\right)}=\mathrm{ColSqueeze}\left({\mathcal{T}}_1(:,k,:)\right)$. See Figure 2.

Then, we give the definition of twist nuclear norm as follows:

Definition 10

 (Twist tensor nuclear norm [16]). The twist tensor nuclear norm (t-TNN) based on the t-SVD framework is defined as follows:

$$\begin{array}{c}\hfill {\parallel \mathcal{T}\parallel }_{\overrightarrow{★}}:=\parallel \overrightarrow{\mathcal{T}}{\parallel }_{★}={\parallel \mathrm{bcirc}(\overrightarrow{\mathcal{T}})\parallel }_{*}.\end{array}$$

(10)

From the above definition, t-TNN can be computed efficiently by first twisting the original tensor and then computing the TNN. Given a tensor $\mathcal{T}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$, the computation cost of ${\parallel \mathcal{T}\parallel }_{\overrightarrow{★}}$ is $O\left(n_1{n}_2{n}_3(n_2+\log n_2)\right)$. The t-TNN has attained significant improvement against the TNN in video inpainting [16]. Since the t-TNN is essentially a TNN, the proximal operator of it can be derived from the proximal operator of TNN as follows:

$$\begin{array}{c}\hfill \mathrm{ColSqueeze}({\mathfrak{S}}_{\tau }(\overrightarrow{{\mathcal{T}}_0}))=\underset{\mathcal{T}}{\mathrm{argmin}}\frac{1}{2}\parallel {\mathcal{T}}_0-\mathcal{T}{\parallel }_{\mathrm{F}}^2+\tau {\parallel \mathcal{T}\parallel }_{\overrightarrow{★}}.\end{array}$$

3.3. A Circular Interpretation of TNN and t-TNN

In this subsection, an illustration of TNN and t-TNN in a circular way [16], which motivates the proposal of TriTNN, will be given. For a tensor $\mathcal{T}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$, we define the operation called circulant block matricization of $\mathcal{T}$ [41] in the following equation:

$$\begin{array}{c}\hfill \mathrm{circ}\left(\mathcal{T}\right):=\left[\begin{array}{cccc}\mathrm{circ}\left(\mathcal{T}\right(1,1,:\left)\right)& \mathrm{circ}\left(\mathcal{T}\right(1,2,:\left)\right)& \cdots & \mathrm{circ}\left(\mathcal{T}(1,n_2,:)\right)\\ \mathrm{circ}\left(\mathcal{T}\right(2,1,:\left)\right)& \mathrm{circ}\left(\mathcal{T}\right(2,2,:\left)\right)& \cdots & \mathrm{circ}\left(\mathcal{T}(2,n_2,:)\right)\\ ⋮& ⋮& \ddots & ⋮\\ \mathrm{circ}\left(\mathcal{T}(n_1,1,:)\right)& \mathrm{circ}\left(\mathcal{T}(n_1,2,:)\right)& \cdots & \mathrm{circ}\left(\mathcal{T}(n_1,n_2,:)\right)\end{array}\right]\in {\mathbb{R}}^{n_1{n}_3\times n_2{n}_3},\end{array}$$

(11)

where $\mathrm{circ}\left(\mathit{x}\right)$ denotes an $n_3\times n_3$ matrix with $\mathit{x}\in {\mathbb{R}}^{n_3}$ in the following way:

$$\begin{array}{c}\hfill \mathrm{circ}\left(\mathit{x}\right):=\left[\begin{array}{cccc}\mathrm{circ}\left({\mathit{x}}_1\right)& \mathrm{circ}\left({\mathit{x}}_{n_3}\right)& \cdots & \mathrm{circ}\left({\mathit{x}}_2\right)\\ \mathrm{circ}\left({\mathit{x}}_2\right)& \mathrm{circ}\left({\mathit{x}}_1\right)& \cdots & \mathrm{circ}\left({\mathit{x}}_3\right)\\ ⋮& ⋮& \ddots & ⋮\\ \mathrm{circ}\left({\mathit{x}}_{n_3}\right)& \mathrm{circ}\left({\mathit{x}}_{n_3-1}\right)& \cdots & \mathrm{circ}\left({\mathit{x}}_1\right)\end{array}\right]\in {\mathbb{R}}^{n_3\times n_3}.\end{array}$$

(12)

By the permutation operation, there exist two so-called stride permutation matrices [42] ${\mathit{P}}_1$ and ${\mathit{P}}_2$, such that the following relationship between circ($\mathcal{T}$) and bcirc($\mathcal{T}$) holds:

$$\begin{array}{c}\hfill \mathrm{circ}\left(\mathcal{T}\right)={\mathit{P}}_1\mathrm{bcirc}\left(\mathcal{T}\right){\mathit{P}}_2.\end{array}$$

(13)

Since the matrix nuclear norm is permutation invariant, it holds that [16]:

$$\begin{array}{c}\hfill {\parallel \mathrm{circ}\left(\mathcal{T}\right)\parallel }_{★}={\parallel \mathrm{bcirc}\left(\mathcal{T}\right)\parallel }_{★}.\end{array}$$

(14)

As an example, Figure 3a,b intuitively shows the relationships between the original tensor, the column twist tensor, the block circulant matrix and the circulant block matricization of a tensor $\mathcal{T}\in {\mathbb{R}}^{3\times 3\times 3}$. As illustrated in Subplots (a) and (b) of Figure 3, from the circulant perspective, TNN essentially exploits the tube redundancy in a circulant way while keeping the row and column relationship, and t-TNN essentially exploits the row redundancy in a circulant way while preserving the tube and column relationship [16]. In computer vision applications, the row and column of a data tensor (like color images or videos) often carry spatial information, and the tube often carries temporal or channel information. From the computational perspective, FFT is operated along the tube direction to compute TNN, while t-TNN needs FFT along the row direction.

3.4. The Proposed Row Twist Tubal Nuclear Norm and Triple Tubal Nuclear Norm

As discussed above, TNN and t-TNN need FFT along the tube and row direction, respectively. Note that, for real-word visual data, like color images, the row and column carry homogeneous information, and they are better treated equally. A simple operation similar to the column twist and column squeeze, called row twist and row squeeze, respectively, is defined.

Definition 11

 (Tensor row twist and row squeeze). Let $\mathcal{T}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$, then the row twist tensor ${\mathcal{T}}_1=\underrightarrow{\mathcal{T}}$ is a tensor of size $n_3\times n_2\times n_1$ whose horizontal slice ${\mathcal{T}}_1(k,:,:)={\mathcal{T}}^{\left(k\right)}$. Correspondingly, the row squeeze tensor of ${\mathcal{T}}_1$, i.e., $\mathcal{T}=\underleftarrow{{\mathcal{T}}_1}$, can be obtained by the reverse process, i.e., ${\mathcal{T}}^{\left(k\right)}=\mathrm{RowSqueeze}\left({\mathcal{T}}_1(k,:,:)\right)$. See Figure 4.

Definition 12

(Row twist tubal nuclear norm (rt-TNN)). The row twist tubal nuclear norm (rt-TNN) of a tensor $\mathcal{T}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ is defined as the tubal nuclear norm of its row twisted tensor, i.e.,

$$\begin{array}{c}\hfill {\parallel \mathcal{T}\parallel }_{\underrightarrow{★}}=\parallel \underrightarrow{\mathcal{T}}{\parallel }_{★}={\parallel \mathrm{bcirc}(\underrightarrow{\mathcal{T}})\parallel }_{*}.\end{array}$$

(15)

As illustrated in Subplot (c) of Figure 3, from the circulant perspective, rt-TNN essentially exploits the column redundancy in a circulant way while keeping the row and tube relationship. From the computational perspective, FFT is operated along the column direction to compute rt-TNN. The proximal operator of rt-TNN can be derived from the proximal operator of TNN as follows:

$$\begin{array}{c}\hfill \mathrm{RowSqueeze}({\mathfrak{S}}_{\tau }(\underrightarrow{{\mathcal{T}}_0}))=\underset{\mathcal{T}}{\mathrm{argmin}}\frac{1}{2}\parallel {\mathcal{T}}_0-\mathcal{T}{\parallel }_{\mathrm{F}}^2+\tau {\parallel \mathcal{T}\parallel }_{\underrightarrow{★}}.\end{array}$$

It should be noted that each of TNN, t-TNN and rt-TNN only exploits one type of redundancy, i.e., the tube, row and column redundancies, in a circulant way. Real-world data may have more than one type of redundancy, and it is beneficial to exploit such a property. To simultaneously exploit the tube, row and column redundancy in a circulant way while keeping other relationships, we simply combine the TNN, t-TNN and rt-TNN to get the triple tubal nuclear norm.

Definition 13

 (Triple tubal nuclear norm). The triple tubal nuclear norm (TriTNN) is defined as a weighted sum of its tubal nuclear norm, column twist tubal nuclear norm and its row twist tubal nuclear norm, i.e.,

$$\begin{array}{c}\hfill {\parallel \mathcal{T}\parallel }_{⚹}={\lambda }_1{\parallel \mathcal{T}\parallel }_{★}+{\lambda }_2{\parallel \mathcal{T}\parallel }_{\overrightarrow{★}}+{\lambda }_3{\parallel \mathcal{T}\parallel }_{\underrightarrow{★}},\end{array}$$

(16)

where ${\lambda }_1,{\lambda }_2$ and ${\lambda }_3$ are positive weights satisfying:

$$\begin{array}{c}\hfill {\lambda }_1+{\lambda }_2+{\lambda }_3=1.\end{array}$$

(17)

From the above definition, the computation of ${\parallel \mathcal{T}\parallel }_{⚹}$ can be divided into computations of TNN, t-TNN and rt-TNN, which has the following computational complexity:

$$\begin{array}{c}\hfill O\left(n_1{n}_2{n}_3(n_1+n_2+n_3+\log n_1+\log n_2+\log n_3)\right).\end{array}$$

(18)

Due to the coupling of three tubal nuclear norms, it is very difficult to derive a closed-form expression of the proximal operator of TriTNN.

4. TriTNN-Based Tensor Completion

4.1. Problem Formulation

Let ${\mathcal{L}}^{*}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ be the true, but unknown tensor to be completed. Suppose only a small fraction of its entries are observed and the observations are corrupted by small dense noise. Let $\mathcal{T}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ denote the observed noisy tensor of ${\mathcal{L}}^{*}$. Then, we have the following observation model:

$$\begin{array}{c}\hfill \mathcal{T}=({\mathcal{L}}^{*}+\mathcal{E})\odot \mathcal{O},\end{array}$$

(19)

where $\mathcal{E}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ is the noise tensor with element-wisely i.i.d. Gaussian noise, ⊙ denotes the element-wise multiplication and $\mathcal{O}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ denotes the binary tensor whose entry ${\mathcal{O}}_{ijk}=1$ if the $(i,j,k)$-th entry is observed, otherwise ${\mathcal{O}}_{ijk}=0$. The goal is to estimate ${\mathcal{L}}^{*}$ given noisy observation $\mathcal{T}$ from observation Model (19).

We estimate ${\mathcal{L}}^{*}$ by simultaneously exploiting the tube, row and column redundancy in a circular way through minimizing the proposed triple tubal nuclear norm. Specifically, we come up with the following problem:

$$\begin{array}{cc}\underset{\mathcal{L}}{\min }\hfill & {\parallel \mathcal{L}\parallel }_{⚹}\hfill \\ \mathrm{s}.\mathrm{t}.\hfill & {\parallel \mathcal{O}\odot (\mathcal{T}-\mathcal{L})\parallel }_{\mathrm{F}}\le ϵ,\hfill \end{array}$$

(20)

where parameter $ϵ>0$ denotes the noise level. The motivation is to recover ${\mathcal{L}}^{*}$ by choosing the tensor with the smallest TriTNN from a hyper-ball in ${\mathbb{R}}^{n_1\times n_2\times n_3}$ with radius $ϵ$. It is well known that Problem (20) in the form of convex minimization with a bounded norm constraint is equivalent to the following unconstrained problem [23]:

$$\begin{array}{c}\hfill \underset{\mathcal{L}}{\min }\frac{1}{2}{\parallel \mathcal{O}\odot (\mathcal{T}-\mathcal{L})\parallel }_{\mathrm{F}}^2+\tau {\parallel \mathcal{L}\parallel }_{⚹}.\end{array}$$

(21)

where $\tau >0$ is the regularization parameter.

4.2. An ADMM Solver to Problem (21)

The alternative direction multiplier method (ADMM) [38] has been extensively used in solving composite convex problems like Problem (21). We will solve Problem (21) by using ADMM in this subsection.

Considering the definition of TriTNN, we introduce auxiliary variables $\mathcal{U},\mathcal{V},\mathcal{W}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ and obtain the following constrained problem:

$$\begin{array}{cc}\hfill \underset{\mathcal{L},\mathcal{U},\mathcal{V},\mathcal{W}}{\min }& \frac{1}{2}{\parallel \mathcal{O}\odot (\mathcal{T}-\mathcal{L})\parallel }_{\mathrm{F}}^2+\tau {\lambda }_1{\parallel \mathcal{U}\parallel }_{★}+\tau {\lambda }_2{\parallel \mathcal{V}\parallel }_{★}+\tau {\lambda }_3{\parallel \mathcal{W}\parallel }_{★}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \mathcal{U}=\mathcal{L},\mathcal{V}=\overrightarrow{\mathcal{L}},\mathcal{W}=\underrightarrow{\mathcal{L}}.\hfill \end{array}$$

(22)

First, the augmented Lagrangian of Problem (22) is:

$$\begin{array}{c}{L}_{\rho }(\mathcal{L},\mathcal{U},\mathcal{V},\mathcal{W},{\mathcal{Y}}_1,{\mathcal{Y}}_2,{\mathcal{Y}}_3)\hfill \\ =\frac{1}{2}{\parallel \mathcal{O}\odot (\mathcal{T}-\mathcal{L})\parallel }_{\mathrm{F}}^2+\tau {\lambda }_1{\parallel \mathcal{U}\parallel }_{★}+\tau {\lambda }_2{\parallel \mathcal{V}\parallel }_{★}+\tau {\lambda }_3{\parallel \mathcal{W}\parallel }_{★}+〈{\mathcal{Y}}_1,\mathcal{U}-\mathcal{L}〉+\frac{\rho }{2}{\parallel \mathcal{U}-\mathcal{L}\parallel }_{\mathrm{F}}^2\hfill \\ +〈{\mathcal{Y}}_2,\mathcal{V}-\overrightarrow{\mathcal{L}}〉+\frac{\rho }{2}\parallel \mathcal{V}-\overrightarrow{\mathcal{L}}{\parallel }_{\mathrm{F}}^2+〈{\mathcal{Y}}_3,\mathcal{W}-\underrightarrow{\mathcal{L}}〉+\frac{\rho }{2}{\parallel \mathcal{W}-\underrightarrow{\mathcal{L}}\parallel }_{\mathrm{F}}^2,\hfill \end{array}$$

(23)

where ${\mathcal{Y}}_1,{\mathcal{Y}}_2$ and ${\mathcal{Y}}_3$ are Lagrangian multipliers and $\rho >0$ is the penalty parameter.

Using the framework of ADMM, we update the variables alternatively by fixing others at the $k+1$-th iteration in the following way.

Update $\mathcal{L}$. We update $\mathcal{L}$ by fixing the other variables as follows:

$$\begin{array}{cc}\hfill {\mathcal{L}}^{k+1}=& \underset{\mathcal{L}}{\mathrm{argmin}}{L}_{\rho }(\mathcal{L},{\mathcal{U}}^k,{\mathcal{V}}^k,{\mathcal{W}}^k,{\mathcal{Y}}_1^k,{\mathcal{Y}}_2^k,{\mathcal{Y}}_3^k)\hfill \\ \hfill =& \underset{\mathcal{L}}{\mathrm{argmin}}\frac{1}{2}{\parallel \mathcal{O}\odot (\mathcal{T}-\mathcal{L})\parallel }_{\mathrm{F}}^2+〈{\mathcal{Y}}_1^k,{\mathcal{U}}^k-\mathcal{L}〉+\frac{\rho }{2}{\parallel {\mathcal{U}}^k-\mathcal{L}\parallel }_{\mathrm{F}}^2\hfill \\ & +〈{\mathcal{Y}}_2^k,{\mathcal{V}}^k-\overrightarrow{\mathcal{L}}〉+\frac{\rho }{2}\parallel {\mathcal{V}}^k-\overrightarrow{\mathcal{L}}{\parallel }_{\mathrm{F}}^2+〈{\mathcal{Y}}_3^k,{\mathcal{W}}^k-\overleftarrow{\mathcal{L}}〉+\frac{\rho }{2}{\parallel {\mathcal{W}}^k-\overleftarrow{\mathcal{L}}\parallel }_{\mathrm{F}}^2\hfill \\ \hfill =& (\rho {\mathcal{U}}^k+\rho \overleftarrow{{\mathcal{V}}^k}+\rho \overrightarrow{{\mathcal{W}}^k}+{\mathcal{Y}}_1^k+\overleftarrow{{\mathcal{Y}}_2^k}+\overrightarrow{{\mathcal{Y}}_3^k}+\mathcal{O}\odot \mathcal{T})\oslash (\mathcal{O}+\rho \mathbb{I}),\hfill \end{array}$$

(24)

where ⊘ denotes element-wise division and $\mathbb{I}$ denotes the tensor of all ones.

Update $\mathcal{U}$, $\mathcal{V}$ and $\mathcal{W}$. Tensor $\mathcal{U}$ is updated as follows:

$$\begin{array}{cc}\hfill {\mathcal{U}}^{k+1}=& \underset{\mathcal{U}}{\mathrm{argmin}}{L}_{\rho }({\mathcal{L}}^{k+1},\mathcal{U},{\mathcal{V}}^k,{\mathcal{W}}^k,{\mathcal{Y}}_1^k,{\mathcal{Y}}_2^k,{\mathcal{Y}}_3^k)\hfill \\ \hfill =& \underset{\mathcal{U}}{\mathrm{argmin}}\tau {\lambda }_1{\parallel \mathcal{U}\parallel }_{★}+〈{\mathcal{Y}}_1^k,\mathcal{U}-{\mathcal{L}}^{k+1}〉+\frac{\rho }{2}{\parallel \mathcal{U}-{\mathcal{L}}^{k+1}\parallel }_{\mathrm{F}}^2\hfill \\ \hfill =& {\mathfrak{S}}_{\tau {\lambda }_1/\rho }({\mathcal{L}}^{k+1}-\frac{{\mathcal{Y}}_1^k}{\rho }),\hfill \end{array}$$

(25)

where ${\mathfrak{S}}_{·}(·)$ is the proximal operator of TNN at point ${\mathcal{L}}^{k+1}-{\mathcal{Y}}_1^k/\rho$ with parameter $\tau {\lambda }_1/\rho$ (see Equation (8)).

Tensors $\mathcal{V}$ and $\mathcal{W}$ are updated similarly to $\mathcal{U}$,

$$\begin{array}{cc}\hfill {\mathcal{V}}^{k+1}=& \underset{\mathcal{V}}{\mathrm{argmin}}{L}_{\rho }({\mathcal{L}}^{k+1},{\mathcal{U}}^{k+1},\mathcal{V},{\mathcal{W}}^k,{\mathcal{Y}}_1^k,{\mathcal{Y}}_2^k,{\mathcal{Y}}_3^k)\hfill \\ \hfill =& \mathrm{ColSqueeze}({\mathfrak{S}}_{\tau {\lambda }_2/\rho }(\overrightarrow{{\mathcal{L}}^{k+1}}-\frac{{\mathcal{Y}}_2^k}{\rho })),\hfill \end{array}$$

(26)

and:

$$\begin{array}{cc}\hfill {\mathcal{W}}^{k+1}=& \underset{\mathcal{W}}{\mathrm{argmin}}{L}_{\rho }({\mathcal{L}}^{k+1},{\mathcal{U}}^{k+1},{\mathcal{V}}^{k+1},\mathcal{W},{\mathcal{Y}}_1^k,{\mathcal{Y}}_2^k,{\mathcal{Y}}_3^k)\hfill \\ \hfill =& \mathrm{RowSqueeze}({\mathfrak{S}}_{\tau {\lambda }_3/\rho }(\underrightarrow{{\mathcal{L}}^{k+1}}-\frac{{\mathcal{Y}}_3^k}{\rho })).\hfill \end{array}$$

(27)

Update ${\mathcal{Y}}_1,{\mathcal{Y}}_2$ and ${\mathcal{Y}}_3$. Using dual ascending, we update ${\mathcal{Y}}_1,{\mathcal{Y}}_2$ and ${\mathcal{Y}}_3$ as follows:

$$\begin{array}{cc}\hfill {\mathcal{Y}}_1^{k+1}& ={\mathcal{Y}}_1^k+\rho ({\mathcal{U}}^{k+1}-{\mathcal{L}}^{k+1}),\hfill \\ \hfill {\mathcal{Y}}_2^{k+1}& ={\mathcal{Y}}_2^k+\rho ({\mathcal{V}}^{k+1}-\overrightarrow{{\mathcal{L}}^{k+1}}),\hfill \\ \hfill {\mathcal{Y}}_3^{k+1}& ={\mathcal{Y}}_3^k+\rho ({\mathcal{W}}^{k+1}-\underrightarrow{{\mathcal{L}}^{k+1}}).\hfill \end{array}$$

(28)

We summarize the algorithm in Algorithm 1 and analyze the computational complexity as follows.

Complexity analysis: The main computational cost in each iteration rests in the singular tube thresholding operator, requiring the computation of FFT, IFFT and SVDs. Therefore, the time complexity in each iteration is:

$$\begin{array}{c}\hfill O\left(n_1{n}_2{n}_3(n_1+n_2+n_3+\log \left(n_1{n}_2{n}_3\right))\right),\end{array}$$

(29)

Algorithm 1 Solving Problem (21) using ADMM.

Input: The observed tensor $\mathcal{T}$, the parameters $\tau ,{\lambda }_1,{\lambda }_2,{\lambda }_3,\rho$. 1: while not converged do 2:  Update $\mathcal{L}$ using Equation (24); 3:  Update $\mathcal{U},\mathcal{V}$ and $\mathcal{W}$ using Equations (25), (26) and (27), respectively; 4:  Update ${\mathcal{Y}}_1,{\mathcal{Y}}_2$ and ${\mathcal{Y}}_3$ using Equations (28); 5:  Check the convergence conditions: $$\begin{array}{cc}\parallel {\mathcal{L}}^{k+1}-{\mathcal{L}}^k{\parallel }_{\infty }\le \delta ,\hfill & {\parallel {\mathcal{U}}^{k+1}-{\mathcal{U}}^k\parallel }_{\infty }\le \delta ,\hfill \\ \parallel {\mathcal{V}}^{k+1}-{\mathcal{V}}^k{\parallel }_{\infty }\le \delta ,\hfill & {\parallel {\mathcal{W}}^{k+1}-{\mathcal{W}}^k\parallel }_{\infty }\le \delta .\hfill \end{array}$$ 6: end while

Output:$\widehat{\mathcal{L}}$.

4.3. Convergence of Algorithm 1

As Problem (21) has more than two variables, the convergence property of Algorithm 1 cannot be directly obtained from existing results on the convergence of ADMM [38]. Thus, we prove its convergence in terms of the objective function in the following theorem.

Theorem 1

 (Convergence of Algorithm 1). For any $\rho >0$, if the unaugmented Lagrangian $L_0(\mathcal{L},\mathcal{U},\mathcal{V},\mathcal{W},{\mathcal{Y}}_1,{\mathcal{Y}}_2,{\mathcal{Y}}_3)$ has a saddle point, then the iterations $({\mathcal{L}}^k,{\mathcal{U}}^k,{\mathcal{V}}^k,{\mathcal{W}}^k,{\mathcal{Y}}_1^k,{\mathcal{Y}}_2^k,{\mathcal{Y}}_3^k)$ in Algorithm 1 satisfy the residual convergence, objective convergence and dual variable convergence of Problem (21).

Proof. 

The key idea of the proof is to rewrite Problem (9) into a two-block ADMM problem. Since the RowTwist and ColTwist operations are linear, there exist two matrices ${\mathit{P}}_1,{\mathit{P}}_2\in {\mathbb{R}}^{n_1{n}_2{n}_3\times n_1{n}_2{n}_3}$, such that the constraints $\mathcal{V}=\mathrm{ColTwist}\left(\mathcal{L}\right)$ and $\mathcal{W}=\mathrm{RowTwist}\left(\mathcal{L}\right)$ are equivalent to the vectorization expressions:

$$\begin{array}{c}\hfill \mathrm{vec}\left(\mathcal{V}\right)={\mathit{P}}_1\mathrm{vec}\left(\mathcal{L}\right),\mathrm{and}\mathrm{vec}\left(\mathcal{W}\right)={\mathit{P}}_2\mathrm{vec}\left(\mathcal{L}\right),\end{array}$$

where $\mathrm{vec}(·)$ denotes the operation of tensor vectorization (see [14]).

For notational simplicity, let:

$$\begin{array}{c}\hfill \mathit{x}=\mathrm{vec}\left(\mathcal{L}\right),\mathit{y}=\left[\begin{array}{c}\mathrm{vec}\left({\mathcal{Y}}_1\right)\\ \mathrm{vec}\left({\mathcal{Y}}_2\right)\\ \mathrm{vec}\left({\mathcal{Y}}_3\right)\end{array}\right],\mathit{z}=\left[\begin{array}{c}\mathrm{vec}\left(\mathcal{U}\right)\\ \mathrm{vec}\left(\mathcal{V}\right)\\ \mathrm{vec}\left(\mathcal{W}\right)\end{array}\right],\mathit{A}=\left[\begin{array}{c}\mathit{I}\\ {\mathit{P}}_1\\ {\mathit{P}}_2\end{array}\right],\end{array}$$

and:

$$\begin{array}{c}\hfill f\left(\mathit{x}\right)=\frac{1}{2}{\parallel \mathcal{O}\odot (\mathcal{L}-\mathcal{T})\parallel }_{\mathrm{F}}^2,g\left(\mathit{z}\right)=\tau ({\lambda }_1{\parallel \mathcal{U}\parallel }_{★}+{\lambda }_2{\parallel \mathcal{V}\parallel }_{★}+{\lambda }_3{\parallel \mathcal{W}\parallel }_{★}).\end{array}$$

It is obvious that $f(·)$ and $g(·)$ are closed, proper and convex. Then, Problem (21) can be re-written as follows:

$$\begin{array}{cc}\underset{\mathit{x},\mathit{z}}{\min }\hfill & f\left(\mathit{x}\right)+g\left(\mathit{z}\right)\hfill \\ \mathrm{s}.\mathrm{t}.\hfill & \mathit{A}\mathit{x}-\mathit{z}=\mathbf{0}.\hfill \end{array}$$

According to the convergence analysis in [38], we have:

$$\begin{array}{ccc}\mathrm{objective}\mathrm{convergence}:\hfill & \hfill & \underset{k\to \infty }{\lim }f\left({\mathit{x}}^k\right)+g\left({\mathit{z}}^k\right)=f^{*}+g^{*},\hfill \\ \mathrm{dual}\mathrm{variable}\mathrm{convergence}:\hfill & \hfill & \underset{k\to \infty }{\lim }{\mathit{y}}^k={\mathit{y}}^{*},\hfill \\ \mathrm{constraint}\mathrm{convergence}:\hfill & \hfill & \underset{k\to \infty }{\lim }\mathit{A}{\mathit{x}}^k-{\mathit{z}}^k=\mathbf{0},\hfill \end{array}$$

where $f^{*},g^{*}$ are the optimal values of $f\left(\mathit{x}\right)$, $g\left(\mathit{z}\right)$, respectively. Variable ${\mathit{y}}^{*}$ is a dual optimal point defined as:

$$\begin{array}{c}\hfill {\mathit{y}}^{*}=\left[\begin{array}{c}\mathrm{vec}\left({\mathcal{Y}}_1^{*}\right)\\ \mathrm{vec}\left({\mathcal{Y}}_2^{*}\right)\\ \mathrm{vec}\left({\mathcal{Y}}_3^{*}\right)\end{array}\right],\end{array}$$

where $({\mathcal{Y}}_1^{*},{\mathcal{Y}}_2^{*},{\mathcal{Y}}_3^{*})$ is the dual component of a saddle point $({\mathcal{L}}^{*},{\mathcal{U}}^{*},{\mathcal{V}}^{*},{\mathcal{W}}^{*},{\mathcal{Y}}_1^{*},{\mathcal{Y}}_2^{*},{\mathcal{Y}}_3^{*})$ of the unaugmented Lagrangian $L_0(\mathcal{L},\mathcal{U},\mathcal{V},\mathcal{W},{\mathcal{Y}}_1,{\mathcal{Y}}_2,{\mathcal{Y}}_3)$. □

4.4. Differences from Prior Work

First, we show the difference between the proposed model TriTNN and two mostly related models TNN [30] and t-TNN [16]. Although all of them are based on the tubal nuclear norm, the main difference lies in that TNN and t-TNN only use information of one orientation, whereas TriTNN uses information of three orientations.

Now, we compare the proposed model with Tubal-Alt-Min [37]. It is based on tensor tubal rank and employs the tensor factorization strategy for tensor completion. The differences between TriTNN and Tubal-Alt-Min are: (a) TriTNN preserves the low-rank structure by summing three tubal nuclear norms, whereas Tubal-Alt-Min adopts low-rank tensor factorization to characterize the low-rank property of a tensor. In this way, they are two different kinds of models for tensor completion (since Tubal-Alt-Min and the proposed TriTNN are quite different algorithms and the main goal of this paper is to improve upon TNN, we do not compare Tubal-Alt-Min in the experiment section.) (b) Since TriTNN is based on the tubal nuclear norm, it is formulated as a convex optimization problem (21). Benefiting from convexity, each local minimum of Problem (21) must be a global minimum. However, Tubal-Alt-Min is formulated as a non-convex optimization problem, thus it may produce sub-optimal solutions.

5. Experiments

In this section, extensive experiments will be conducted to explore the effectiveness of the proposed Algorithm 1. All the codes are implemented in MATLAB language, and all experiments are carried out in Windows 10 based on an Intel Core(TM) 2.60-GHz CPU with 12 G RAM.

To explore the effectiveness of the proposed TriTNN-based model, we compare with the following nuclear norm-based tensor completion models:

• The tensor nuclear norm-based model with ADMM solver: high accuracy low-rank tensor completion (HaLRTC, denoted by SNNin this paper) (code available: http://www.cs.rochester.edu/u/jliu/publications.html) [2], The tensor nuclear norm is defined as the weighted sum of nuclear norms of the unfolding matrices along each mode (thus, we denote this model as SNN):

  $$\begin{array}{c}\hfill {\parallel \mathcal{T}\parallel }_{SNN}:={\alpha }_1\parallel {\mathit{T}}_{\left(1\right)}{\parallel }_{*}+{\alpha }_2\parallel {\mathit{T}}_{\left(2\right)}{\parallel }_{*}+{\alpha }_3{\parallel {\mathit{T}}_{\left(3\right)}\parallel }_{*},\end{array}$$

  where ${\alpha }_1,{\alpha }_2$, ${\alpha }_3$ are positive parameters and ${\mathit{T}}_{\left(i\right)}\in {\mathbb{R}}^{n_i\times {\prod }_{j\ne i}{n}_i}$, $i=1,2,3$, is the unfolding matrix of tensor $\mathcal{T}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$ along the i-th mode [2].
• The latent tensor nuclear norm-based model (LatentNN) (code available: https://github.com/ryotat/tensor) [21]. The latent tensor nuclear norm is defined as:

  $$\begin{array}{c}\hfill {\parallel \mathcal{T}\parallel }_{latent}:=\underset{\mathcal{X}+\mathcal{Y}+\mathcal{Z}=\mathcal{T}}{\inf }\parallel {\mathit{X}}_{\left(1\right)}{\parallel }_{*}+\parallel {\mathit{Y}}_{\left(2\right)}{\parallel }_{*}+{\parallel {\mathit{Z}}_{\left(3\right)}\parallel }_{*},\end{array}$$

  where ${\mathit{X}}_{\left(1\right)},{\mathit{Y}}_{\left(2\right)}$ and ${\mathit{Z}}_{\left(3\right)}$ are the first-mode, second-mode and third-mode unfoldings of latent tensors $\mathcal{X},\mathcal{Y}$ and $\mathcal{Z}$, respectively.
• The square nuclear norm-based model (SquareNN) (code available: https://sites.google.com/site/mucun1988/publi) [23]. The square nuclear norm of a tensor is defined as the nuclear norm of the most balanced unfolding of a tensor (see [7,23]).
• The most related tubal nuclear norm-based model (TNN) (code available: https://github.com/jamiezeminzhang/) [30] and the twist tubal nuclear norm-based model (t-TNN) [16].

We conduct tensor completion experiments on color images, videos and a dataset for autonomous vehicle. For an estimation tensor $\widehat{\mathcal{L}}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$, its quality is evaluated by the peak signal-to-noise ratio (PSNR) computed by the definition:

$$\begin{array}{c}\hfill PSNR=10{\log }_{10}(\frac{n_1{n}_2{n}_3{\parallel {\mathcal{L}}^{*}\parallel }_{\infty }^2}{\parallel \widehat{\mathcal{L}}-{\mathcal{L}}^{*}{\parallel }_{\mathrm{F}}^2}),\end{array}$$

where ${\mathcal{L}}^{*}$ is the underlying tensor. The higher the PSNR value is, the better the recovery performance will be.

5.1. Color Image Inpainting

Color images in row × column × channel are naturally expressed in 3D tensor form. Image inpainting aims at reconstructing a color image from a small fraction of its entries. In this experiment, twelve test images of size $256\times 256\times 3$ are used; see Figure 5. Given an image $\mathcal{M}$ of size $d_1\times d_2\times 3$, we randomly sample $30\%$ of its pixels and add i.i.d. Gaussian noise with standard deviation $\sigma =0.1{\sigma }_0$, where ${\sigma }_0={\parallel \mathcal{M}\parallel }_{\mathrm{F}}/\sqrt{3d_1{d}_2}$ is the rescaled magnitude of $\mathcal{M}$.

The weight parameters $\mathit{\alpha }$ of SNN are chosen to satisfy ${\alpha }_1:{\alpha }_2:{\alpha }_3=1:1:0.01$ as suggested in [2]. Parameter $\tau =8e3$ and ${\lambda }_1,{\lambda }_2,{\lambda }_3$ in Problem (22) are chosen to satisfy ${\lambda }_1:{\lambda }_2:{\lambda }_3=1:0.01:0.01$. Parameters of other algorithms are tuned for better performances in most cases. We also employ the structural similarity index measure (SSIM) [43] to measure the quality of inpainted color images. The higher the SSIM value is, the better the inpainting performance will be. Given a color image, we test ten times and report the averaged PSNR and SSIM values.

The inpainting results of five images are shown in Figure 6 for qualitative comparison. We can see that the proposed TriTNN-based model obtains better visual performances. For quantitative comparison, the PSNR and SSIM values on the twelve images of seven algorithms are reported in Figure 7. It can be seen that the proposed TriTNN-based outperforms the competitors in most cases.

5.2. Video Inpainting

The video inpainting task aims at imputing the missing pixels of a video. The performance competition is carried out on five widely-used YUVvideos (They are available from https://sites.google.com/site/subudhibadri/fewhelpfuldownloads): salesman_qcif, silent_qcif, suzie_qcif, tempete_cif and waterfall_cif. Due to the computational limitation, we use the first 30 frames of Y components in each video. This results in three tensors sized $144\times 176\times 32$ and two tensors sized $288\times 352\times 32$. For each video, we uniformly sample 10% of the entries and conduct the video inpainting experiments.

The weight parameters $\mathit{\alpha }$ of SNN are chosen to satisfy ${\alpha }_1:{\alpha }_2:{\alpha }_3=1:1:1$ as suggested in [2]. Parameters $\tau =2,\rho =5e$-5 and ${\lambda }_1,{\lambda }_2,{\lambda }_3$ are chosen to satisfy ${\lambda }_1:{\lambda }_2:{\lambda }_3=1:1:1$ for the proposed model. Parameters of other algorithms are tuned for better performances in most cases. Given a video, we test ten times and report the averaged PSNR value.

The qualitative comparison is shown in Figure 8. The PSNR values are reported in

Table 1. Quantitative evaluation of algorithms in PSNR values for YUV video inpainting: each video is sampled uniformly with ratio $p=0.1$.

Video	SNN [2]	LatentNN [21]	SquareNN [23]	TNN [31]	t-TNN [16]	TriTNN
salesman_qcif	21.10	24.21	20.31	25.56	25.73	26.18
silent_qcif	24.07	23.95	22.98	27.93	28.03	28.47
suzie_qcif	25.79	24.63	24.48	27.92	29.47	29.70
tempete_cif	19.10	18.45	19.23	20.65	21.21	21.46
waterfall_cif	24.10	22.39	24.06	26.71	27.44	28.22

. It can be seen that the TriTNN-based model outperforms the others. The proposed TriTNN has better performances than TNN and t-TNN, since the TriTNN exploits the row, column and tube redundancy simultaneously, whereas TNN or t-TNN only exploit one type of redundancy. The superiority of TriTNN over SNN, LatentNN and SquareNN may be explained by the fact that the circulant block matricization of a tensor coded in TriTNN makes use of more information than directly unfolding along each mode.

5.3. A Dataset for Autonomous Driving

Environment perception for autonomous driving has attracted more and more attention in computer vision. In this subsection, experiments on a dataset collected for autonomous driving are performed.

The dataset (a collection of Frame No. 165–No. 244 in Scenario B and Scenario B; additional sensor data available at http://www.mrt.kit.edu/z/publ/download/velodynetracking/dataset.html) has 80 frames of gray images and LiDAR point cloud data acquired by a Velodyne HDL-64E LiDAR. The image sequence is resized to be a tensor of size $128\times 256\times 80$, and the LiDAR data are resampled, transformed and formatted to be two tensors of size $64\times 436\times 80$ representing the distance data and the intensity data, respectively.

Given a tensor to complete $\mathcal{T}\in {\mathbb{R}}^{n_1\times n_2\times n_3}$, experiments are carried out with seven different observation settings where the sampling ratio p varies from 0.1–0.7. The observed entries are further corrupted by i.i.d. Gaussian noise with standard deviation $\sigma =0.2{\sigma }_0$, where ${\sigma }_0={\parallel \mathcal{T}\parallel }_{\mathrm{F}}/\sqrt{d_1{d}_2{d}_3}$ is the normalized magnitude of $\mathcal{T}$. The weight parameters $\mathit{\alpha }$ of SNN are chosen to satisfy ${\alpha }_1:{\alpha }_2:{\alpha }_3=1:1:1$. Parameters $\tau =2,\rho =5e$-5 and ${\lambda }_1,{\lambda }_2,{\lambda }_3$ are chosen to satisfy ${\lambda }_1:{\lambda }_2:{\lambda }_3=1:1:1$ for the proposed model. Parameters of other algorithm are tuned manually to achieve better performances in most cases. Given a sampling ratio p, we repeat ten trials and report the averaged PSNR value.

The quantitative performance comparison in terms of PSNR is shown in

Table 2. Comparison of the PSNR values for image sequence completion.

Sampling Ratio	SNN [2]	LatentNN [21]	SquareNN [23]	TNN [31]	t-TNN [16]	TriTNN
$p=0.1$	14.58	15.50	17.27	17.23	17.29	17.45
$p=0.2$	17.79	17.27	18.16	18.32	18.59	18.74
$p=0.3$	18.74	18.36	18.80	19.04	19.12	19.53
$p=0.4$	19.29	19.02	19.21	19.50	19.44	20.02
$p=0.5$	19.61	19.43	19.49	19.82	19.64	20.28
$p=0.6$	19.80	19.70	19.69	20.06	19.76	22.01
$p=0.7$	19.90	19.83	19.84	20.23	19.85	22.41

,

Table 3. Comparison of the PSNR values for HDL-64Edistance data completion.

Sampling Ratio	SNN [2]	LatentNN [21]	SquareNN [23]	TNN [31]	t-TNN [16]	TriTNN
$p=0.1$	17.80	16.90	17.64	19.03	18.67	18.91
$p=0.2$	19.26	18.48	18.94	20.04	19.60	20.09
$p=0.3$	20.33	19.71	19.94	20.93	20.41	21.04
$p=0.4$	21.24	20.67	20.89	21.74	21.17	21.92
$p=0.5$	22.05	21.55	21.75	22.47	21.88	22.72
$p=0.6$	22.82	22.39	22.56	23.16	22.61	23.68
$p=0.7$	23.54	23.21	23.34	23.79	23.35	24.51

and

Table 4. Comparison of the PSNR values for HDL-64E intensity data completion.

Sampling Ratio	SNN [2]	LatentNN [21]	SquareNN [23]	TNN [31]	t-TNN [16]	TriTNN
$p=0.1$	17.30	17.63	17.57	18.31	18.62	18.36
$p=0.2$	18.79	19.30	18.92	19.50	19.61	19.63
$p=0.3$	19.85	20.29	19.93	20.38	20.37	20.45
$p=0.4$	20.84	21.11	20.90	21.24	21.08	21.30
$p=0.5$	21.72	21.86	21.76	22.05	21.78	22.18
$p=0.6$	22.51	22.62	22.53	22.79	22.50	22.94
$p=0.7$	23.39	23.40	23.39	23.59	23.27	23.80

for gray image sequence, distance data and intensity data completion, respectively. The highest average PSNR indicating the best recovery performance is highlighted in bold. From Table 2, Table 3 and Table 4, we can see that the proposed TriTNN-based model yields better performances over the other five algorithms for noisy tensor completion.

6. Conclusions

This paper studied the problem of completing a 3D tensor from incomplete observations. It has been used in broad applications of computer vision tasks like image and video inpainting. In this paper, a new tensor norm named the triple tubal nuclear norm (TriTNN) is proposed to simultaneously exploit tube, row and column redundancy in a circulant way by using a weighted sum of three TNNs. A TriTNN-based tensor completion model is presented. It is further solved by an ADMM-based algorithm with convergence guarantee. Experimental results on color images, videos and LiDAR datasets show the superiority of the proposed TriTNN against state-of-the-art nuclear norm-based tensor norms.

For the proposed TriTNN, the authors believe it can outperform many nuclear norm-based tensor completion models because more spatial-temporal information is exploited. However, generally speaking, it has the following two drawbacks:

• Computational inefficiency: Compared to TNN and t-TNN, it is more time-consuming since it involves computing TNN, t-TNN and rt-TNN (see Equation (18)).
• Sample inefficiency: Using the analysis of [23] and [44], to complete an incomplete tensor, TriTNN needs more observations than TNN and t-TNN (limited to the scope of this paper; the authors do not discuss it further).

In the future research, the authors are interested in efficient algorithms like [45] to tackle the problem of computational inefficiency. To decrease the sample complexity of TriTNN, it will be helpful to follow the suggestions in [44] to design new atomic norms like [46]. To get better visual completion performances, the authors would like to consider adding smoothness regularization in the model like [47,48,49] and adopting different tensorization methods like [50]. It is also helpful for studying new tensor completion models using deep neural networks [51]. For potential extensions of TriTNN, the authors would like to explore the p-th order ($p>3$) extension [52] and extensions to other discrete transforms other than DFT like [53].