Low Tensor Rank Constrained Image Inpainting Using a Novel Arrangement Scheme

Abstract

Employing low tensor rank decomposition in image inpainting has attracted increasing attention. This study exploited novel tensor arrangement schemes to transform an image (a low-order tensor) to a higher-order tensor without changing the total number of pixels. The developed arrangement schemes enhanced the low rankness of images under three tensor decomposition methods: matrix SVD, tensor train (TT) decomposition, and tensor singular value decomposition (t-SVD). By exploiting the schemes, we solved the image inpainting problem with three low-rank constrained models that use the matrix rank, TT rank, and tubal rank as constrained priors. The tensor tubal rank and tensor train multi-rank were developed from t-SVD and TT decomposition, respectively. Then, ADMM algorithms were efficiently exploited for solving the three models. Experimental results demonstrate that our methods are effective for image inpainting and superior to numerous close methods.

1. Introduction

Image inpainting refers to the process of completing missing entries or restoring damaged regions of an image. It is a typical ill-posed inverse problem, generally solved by exploiting the image priors [1,2], such as smoothness, sparsity, and low rankness. In recent years, tensor analysis, including tensor low-rank decomposition and tensor completion, has attracted increasing attention [3,4,5,6]. A color image itself is an order-3 tensor, or it can be used to construct a high-order (greater than 3) tensor; then, the image inpainting problem becomes a tensor completion problem. A tensor is more challenging to analyze than a matrix due to the complicated nature of higher-order arrays [7]. Low tensor rank can be constrained to recover the missing pixels. The effectiveness relies on the tensor rank. The lower the tensor rank, the better the recovery results. Thus, finding ways to decrease the tensor rank is essential in the tensor completion problem. Unlike the matrix rank, the definition of tensor rank is not unique and is related to the tensor decomposition scheme.

Low tensor rank completion methods can be categorized according to the tensor decomposition frameworks they use [8]. The traditional tensor decomposition tools include CANDECOMP/PARAFAC (CP) and Tucker decomposition [8,9]. Recently proposed decomposition frameworks include tensor singular value decomposition (t-SVD) [10,11,12], tensor train (TT) decomposition [13,14], tensor tree (TTR) decomposition [6,15], tensor ring (TR) decomposition [16], fully connected tensor network (FCTN) decomposition [17], etc. The CP rank is hard to estimate. Tucker rank is multi-rank, whose elements are the ranks of mode-n matrices that are highly unbalanced. TT rank is also multi-rank, whose elements are the ranks of TT matrices. For a high-order tensor, TT matrices are more balanced than the mode-n matrices. As matrix rank minimization is only efficient when the matrix is balanced, TT decomposition is more suitable for describing global information of high-order tensors than Tucker decomposition. T-SVD defines the tubal rank of the high-order tensor, which can be easily estimated according to a fast Fourier-based method. The tubal rank has been shown to be more efficient than the matrix rank and Tucker multi-rank in video applications [18,19,20]. TTR rank is essentially equivalent to Tucker multi-rank. It has to be mentioned that both TT and TR decompose the Nth-order tensor into a list of third-order core tensors. Among them, TR decomposition is an improvement of TT decomposition, and TR rank is also a type of multi-rank. TR connects multiple core tensors in a circular manner, but TR rank only constrains the factor relationship between two adjacent core tensors. FCTN decomposition can flexibly characterize the correlation of multiple core tensor factors.

Many popular tensor completion methods have applied the traditional CP or Tucker decomposition on color image inpainting. Some recent works have exploited the sparse Tucker core tensor and nonnegative Tucker factor matrices for image restoration [7,21,22]. Some works have constrained the low rankness of the mode-n matrix caused by the decomposition of a color image for inpainting [23,24]. As low tensor rank constraints cannot fully capture the local smooth and global sparsity priors of tensors, some works have combined Tucker and total variation (TV). The SPCTV (smooth PARAFAC tensor completion and total variation) method [25] uses the PD (PARAFAC decomposition, a derivation of Tucker decomposition) framework and constrains the TV (total variation) on every factor matrix of PD. Some works have combined the constraints of the low rankness of every mode-n matrix and the TV regularization on every mode-n matrix for color image inpainting [26,27]. Some works have proposed data restoration methods based on Bayesian tensor completion [28,29,30,31].

The aforementioned methods all take the color image as an order-3 tensor directly and do not deeply explore the potential low-rank decomposition prior to a color image. The RNC-FCTN method [32] proposes a robust tensor completion method based on FCTN, but it is limited to high-dimensional data processing, such as hyperspectral video, color video completion, and video background subtraction. The CFN-RTC method [33] proposes a robust tensor completion method that combines the capped Frobenius norm and TR decomposition and adds dimensions through simple data reshaping for processing image or video data. As TT decomposition is efficient for higher-order tensors, the TMac-TTKA method [34] first uses the ket augmentation (KA) scheme to permute the image to high-order data, then applies the optimal models by enforcing low TT rankness. The KA scheme is proven to be efficient for improving the accuracy of color image/video inpainting and dynamic MR image reconstruction in TT rank-based completion methods [34,35,36,37]. As far as we know, the KA scheme is the only method used to permute data into high-order data.

This study aims to explore the potential low-rank structure of images and find an efficient way to apply SVD, t-SVD, and TT decomposition in image inpainting problems. The contributions of our work are summarized as follows:

• First, we developed a novel rearrangement, named the quarter arrangement (QA) scheme, for permuting the image into three flexible forms of data. The first flexible QA scheme can permute an image into an unfolding matrix (with a low matrix rank structure). The second and the third flexible QA schemes can permute the color image into a balanced 3-order form of data (with low tubal rank structure) and a higher-order form of data (with low TT rank structure), respectively. Because the developed schemes are designed to exploit the internal structure similarity of the original data as much as possible, the rearranged data have the corresponding low-rank structure.
• Second, based on the above QA scheme, we developed three image inpainting models that exploit the unfolding matrix rank, tensor tubal rank, and TT multi-rank of the rearranged data to solve the image inpainting problem.
• Lastly, three efficient ADMM algorithms were developed to solve the above three models. Compared with numerous close image inpainting methods, the experimental results demonstrated the superior performance of our methods.

An abbreviation index is shown in

Table 1. An abbreviation index.

Abbreviations	Full Terms
t-SVD	tensor singular value decomposition
TT	tensor train
TV	total variation
KA	ket augmentation
QA	quarter arrangement

. The remainder of this paper is organized as follows. In Section 2, related work is described. In Section 3, we introduce the proposed methods. Section 4 presents the experimental results and analyses. The conclusion is given in Section 5.

2. Related Work

In this section, a brief introduction to the KA scheme, t-SVD decomposition, and tensor train decomposition is provided. Notations and definitions are summarized in

Table 2. Notations and definitions.

Symbols	Notations and Definitions
fiber	A vector defined by fixing every index but one of a tensor.
slice	A matrix defined by fixing all but two indices of a tensor.
$\mathcal{A} (:, :, k)$	$\mathrm{The} k^{th}$ frontal slice of a 3-order tensor $\mathcal{A}$.
${\mathcal{A}}_{(n)}$	Mode-n matrix, resulting from unfolding tensor $\mathcal{A}$ by reshaping its mode-n fibers to the columns of ${\mathcal{A}}_{(n)}$.
f-diagonal tensor	Order-3 tensor $\mathcal{A}$ is called f-diagonal if each frontal slice $\mathcal{A} (:, :, k)$ is a diagonal matrix [10].
orthogonal tensor	Tensor $\mathcal{A}$ with the size of $n\times n\times n_3$ is called orthogonal tensor if $\mathcal{A}\ast {\mathcal{A}}^H=\mathcal{I},$ where $\mathcal{I}$ stands for identity tensor if the first frontal slice ${\mathcal{I}}^{(1)}$ is the $n\times n$ identity matrix and all other frontal slices ${\mathcal{I}}^{(k)}$$(k=1, 2,\cdots , n_3$) are zero.

.

2.1. Ket Augmentation

The ket augmentation (KA) scheme was originally introduced by Latorre in [38] for casting a grayscale image into the real ket state of a Hilbert space. Bengua et al. [34] used KA to reshape a low-order tensor, e.g., a color image, to a higher-order tensor and proved that KA is efficient in improving the accuracy of the recovered image in TT-based completion.

For example, through the KA scheme, an 8 × 8 matrix can be turned into a 3-order tensor of size 4 × 4 × 4, which is achieved by dividing the matrix into several blocks and rearranging the blocks row by row. In addition, the KA scheme can turn a 3-order tensor size of $x^N\times y^N\times N_3$ into an $\left(N+1\right)$-order tensor with a size of $xy\times xy\times :::\times xy\times N_3$. For more details refer to [37].

2.2. T-SVD Decomposition

Definition 1.

t-product [39]. For $\mathcal{A}\in R^{n_1\times n_2\times n_3}$ and $\mathcal{B}\in R^{n_2\times n_4\times n_3}$, the t-product $\mathcal{A}\ast \mathcal{B}=\mathcal{C}$ is a tensor of size $n_1\times n_4\times n_3$, where $\mathcal{C} (i, j, :)$ is given by ${\displaystyle {\sum }_{k=1}^{n_2}\mathcal{A} (i, k, :)}\circ \mathcal{B} (k, j, :)$, $\circ$ denotes the circular convolution between the two vectors, and $i=1, 2,\cdots , n_1$, $j=1, 2,\cdots , n_4$.

The t-SVD of $\mathcal{A}\in R^{n_1\times n_2\times n_3}$ is given by

$$\mathcal{A}=\mathcal{U}\ast \mathcal{S}\ast {\mathcal{V}}^T$$

where $\mathcal{U}$ and $\mathcal{V}$ are orthogonal tensors of size $n_1\times n_1\times n_3$ and $n_2\times n_2\times n_3$, respectively. $\mathcal{S}$ is a rectangular f-diagonal tensor of size $n_1\times n_2\times n_3$, * denotes t-product [39], and $T$ denotes the tensor transpose defined in [39].

The tensor rank defined in t-SVD is tensor tubal rank, which is the number of nonzero singular tubes in $\mathcal{S}$. Ref. [40] proposed the fast Fourier-based method to calculate the tubal rank and used the tensor nuclear norm (TNN) as the convex relaxation of the tensor tubal rank.

$${‖\mathcal{A}‖}_{TNN}={‖blockdiag(\overline{\mathcal{A}})‖}_{*}$$

where $\overline{\mathcal{A}} = fft (\mathcal{A}, [ ] , 3)$ is the tensor obtained by applying the 1D FFT along the third dimension of $\mathcal{A}$, ${\Vert \Vert }_{*}$ denotes the nuclear norm, and $blockdiag(\overline{\mathcal{A}})=\left[\begin{array}{cccc}{\overline{\mathcal{A}}}^{(1)}& & & \\ & {\overline{\mathcal{A}}}^{(2)}& & \\ & & \ddots & \\ & & & {\overline{\mathcal{A}}}^{(I_3)}\end{array}\right]$.

2.3. Tensor Train Decomposition

$$\mathcal{A}(i_1,i_2,\cdots i_n,\cdots i_N)={\mathcal{U}}_1(:,i_1,:){\mathcal{U}}_2(:,i_2,:)\cdots {\mathcal{U}}_n(:,i_n,:)\cdots {\mathcal{U}}_N(:,i_N,:)$$

Given a tensor $\mathcal{A}\in R^{I_1\times I_2\times \cdots I_N}$, tensor train (TT) decomposition [13,14] can decompose it to $N$ order-3 tensors ${\mathcal{U}}_n\in R^{S_n\times I_n\times \cdots S_{n+1}}$, $n=1,\cdots ,N$. The tensor rank defined in TT decomposition is multi-rank, i.e., $(S_1,S_2,\cdots ,S_{N+1})$, which is combined with the second-dimensional size of each ${\mathcal{U}}_n$.

The widely used way to find TT rank is to estimate the rank of each TT matrix [41] as the element of $(S_1,S_2,\cdots ,S_{N+1})$. The TT matrix ${\mathcal{A}}_{[n]}$ ($n=1,\cdots ,N-1$) with rank $S_n$ is the mode-$\left(1,2,\cdots ,n\right)$ matricization of the tensor with the size of $m\times h$, where $m={\displaystyle \prod _{l=1}^n{I}_l}$, $h={\displaystyle \prod _{l=n+1}^N{I}_l}$.

3. Methods

3.1. Quarter Arrangement

To explore the more efficient low-rank structure of an image, we propose a novel rearrangement scheme, named the quarter arrangement (QA) scheme, to turn a color image into other forms of data. The QA scheme can maintain the internal similarity of the original image in the rearranged data.

The basic QA scheme: For example, as shown in Figure 1a, M is a 2D matrix ($8\times 8$). First, the entries of M are extracted for every other row and column to obtain four smaller matrices. Each smaller matrix has a size of $4\times 4$. Then, these four smaller matrices are placed along the third dimension in a designed order. Lastly, a 3D tensor of size $4\times 4\times 4$ is obtained from the $8\times 8$ matrix M without changing the total number of entries. The entries in the four smaller matrices are labeled as the MATLAB notation $(:,:,1)$, $(:,:,2)$, $(:,:,3)$ and $(:,:,4)$. If M is smooth (most images satisfy), the four smaller matrices are similar in structure due to the adjacent entries.

Applying the basic QA scheme on the single Lena image, the Lena image can be divided into 4 smaller Lena images. As shown in Figure 1b, the four smaller Lena images are similar to each other. In Figure 1c, the pixel values curves of the four smaller images have overlapped into one curve. It can be said that the similarity of the local image structure is mostly maintained by the basic QA scheme.

Under this basic QA scheme, three flexible QA schemes are proposed for permuting the image into three flexible forms of data. The three flexible QA schemes can enhance the low rankness for an image by matrix SVD, tensor train decomposition, and tensor-SVD. Then, by exploiting the flexible QA schemes, three low rank-constrained methods that use the TT rank, tubal rank, and matrix rank as constrained priors are exploited for image inpainting.

The three flexible QA schemes and methods are described in detail in the following three sections. By using the flexible QA scheme, tensor decomposition models based on low rank can be used not only for a color image or video completion but also for grayscale image operations.

3.2. Method 1: The Low Unfolding Matrix Rank-Based Method

The unfolding method is widely used to permute order-3 video or dynamic magnetic resonance images into an unfolding matrix and then exploit the low rankness of this matrix for data reconstruction [23,42]. The unfolding matrix has a low-rank structure because of the similarity of every slightly changed slice along the time dimension.

We try to dig out the potential low unfolding-matrix rankness of a color image by a flexible QA scheme and call this scheme the first flexible QA scheme.

Take a 256 × 256 × 3 Lena image as an example, as shown in Figure 2a. First, the image is permuted into the 3-order tensor size of 32 × 32 × 192 using the basic QA scheme. Then, similar slices of this 3-order are unfolded. Lastly, the balanced (the context of ‘balanced’ is that the size changes from the unbalanced 256 × 256 × 3 to the more balanced size of 1024 × 192) unfolding matrix size of 1024 × 192 is obtained. As the slices (32 × 32) in the 3-order tensor are similar, the unfolding matrix is low rank, as shown in Figure 2b. In practice, the size of the designed unfolding matrix should be balanced so that the minimization of the unfolding matrix rank is efficient.

We exploit the low unfolding matrix rank in image inpainting and give the low unfolding matrix rank-based model as follows:

$$\underset{\mathrm{X}}{\min }\hspace{1em}{‖{\rm M}({\Phi }_{1}X)‖}_{*}\hspace{1em}\mathrm{subject} \mathrm{to}\hspace{1em}X(i, j)=Y(i, j), \forall (i, j)\in \Omega$$

(1)

where $X$ denotes the image to be recovered; ${\Phi }_1$ denotes the operator of permuting the image into a suitable 3-order tensor by multiple basic QA schemes; ${\rm M}$ denotes the operator of the unfolding process, which unfolds every slice along the third dimension of the 3-order tensor ${\Phi }_{1}X$; $\Omega$ is the position without painting; and $Y$ is the painted image with damaged entries at the positions ${\Omega }^{ℂ}$.

To reduce the computational complexity, in model (1), the following SVD-free approach [43,44] is exploited to constrain the low rankness of the unfolding matrix ${\rm M}({\Phi }_{1}X)$ instead of the nuclear norm:

$$\underset{U V^H={\rm M}({\Phi }_{1}X)}{\min }{\displaystyle \frac{1}{2}}\left({‖U‖}_F^2+{‖V‖}_F^2\right)={‖{\rm M}({\Phi }_{1}X)‖}_{*}$$

(2)

As total variation (TV) has been proved as an effective constraint of smooth prior [45,46], model (1) is incorporated with 2D TV to exploit the local smooth priors of visual image data. Then, the image inpainting model (1) turns into the following:

$$\begin{array}{l}\underset{U, V, X}{\min }{\displaystyle \frac{1}{2}}\left({‖U‖}_F^2+{‖V‖}_F^2\right)+{\displaystyle \frac{\beta }{2}}{‖X‖}_{TV}\hspace{1em}\\ \mathrm{subject} \mathrm{to} X(i, j)=Y(i, j), \forall (i, j)\in \Omega , {\rm M}{\Phi }_{1}X=U{V}^H\end{array}$$

(3)

where $\beta$ is the regularization parameter.

The algorithm of alternating direction method of multipliers (ADMM) is used to solve the low unfolding matrix rank and TV-based image inpainting model (3). Firstly, an auxiliary variable $Z=DX$ is introduced, where $D$ is the finite difference operator, and then (3) is rewritten as the unconstrained convex optimization problem (4):

$$\begin{array}{l}\underset{U, V, \Lambda , L, Z, X}{\min }{\tau }_{\Omega }(X)+{\displaystyle \frac{1}{2}}\left({‖U‖}_F^2+{‖V‖}_F^2\right)+{\displaystyle \frac{{\rho }_1}{2}}{‖{\rm M}{\Phi }_{1}X-U{V}^H+\Lambda ‖}_F^2\\ \hspace{1em} +{\displaystyle \frac{\beta }{2}}{‖Z‖}_1+{\displaystyle \frac{\beta {\rho }_2}{2}}{‖DX-Z+L‖}_F^2\end{array}$$

(4)

where ${\tau }_{\Omega }(X)$ denotes the indicator function:

$${\tau }_{\Omega }(X)=\left\{\begin{array}{c}0, X\in \Omega \\ \infty , \mathrm{otherwise} \end{array}\right.’$$

$L$ and $\Lambda$ are the Lagrangian multipliers for variables $Z$ and $U{V}^H$, respectively. The regularization parameter $\beta$ is used to balance the low rankness and sparsity constraints (i.e., TV). The penalty parameters ${\rho }_1>0$ and ${\rho }_2>0$ generally affect the convergence of the algorithm. By applying ADMM, each sub-problem is performed at each iteration t as follows:

$$X^t=\arg \underset{X}{\min }{\tau }_{\Omega }(X)+{\displaystyle \frac{{\rho }_1}{2}}{‖{\rm M}{\Phi }_{1}X-U^{t-1}{V}^{(t-1)H}+{\Lambda }^{t-1}‖}_F^2$$

(5)

$$U^t=\arg \underset{U}{\min }{‖U‖}_F^2+{\displaystyle \frac{{\rho }_1}{2}}{‖{\rm M}{\Phi }_1{X}^t-U{V}^{(t-1)H}+{\Lambda }^{t-1}‖}_F^2$$

(6)

$$V^t=\arg \underset{V}{\min }{‖V‖}_F^2+{\displaystyle \frac{{\rho }_1}{2}}{‖{\rm M}{\Phi }_1{X}^t-U^t{V}^H+{\Lambda }^{t-1}‖}_F^2$$

(7)

$$Z^t=\arg \underset{Z}{\min }{‖Z‖}_1+{\displaystyle \frac{{\rho }_2}{2}}{‖D{X}^t-Z+L^{t-1}‖}_F^2$$

(8)

$${\Lambda }^t={\Lambda }^{t-1}+{\rm M}{\Phi }_1{X}^t-U^t{V}^{(t)H}$$

(9)

$$L^t=L^{t-1}+D{X}^t-Z^t$$

(10)

The initial $U$ and $V$ can be determined by solving the following optimization problem using the LMaFit method [47]:

$$\underset{U,V,X}{\min }{‖U V^H-{\rm M}{\Phi }_{1}X‖}_F^2 subject to\hspace{1em}X(i, j)=Y(i, j), \forall (i, j)\in \Omega$$

(11)

The whole algorithm for solving the model (3) is shown in Algorithm 1.

Algorithm 1. The algorithm for solving the model (3)

Input:$Y, \Omega , {\rho }_1,\beta , {\rho }_2$, maximum number of iteration $t_{\max }$, convergence condition ${\eta }_{tol}$.

Initialization: initial $U^{(0)}$, $V^{(0)}$ by solving the matrix completion problem (11), ${\Lambda }^{(0)}$, $L^{(0)}$,$Z^{(0)}$, t = 0.

While $t<t_{\max }$ and $\eta <{\eta }_{\max }$ do         The first flexible QA scheme: Turn an image into an order-N tensor ${\Phi }_{1}X$, then unfold it.         Solve (5)–(10) for $X^{*}$, where * represents the optimal solution.         Update ${\eta }_{t+1}={\displaystyle \frac{{‖X_n^{t+1}(:)-X_n^t(:)‖}_F}{{‖X_n^t(:)‖}_F}}$, $t=t+1$. End while

Output: $X^{*}$.

3.3. Method 2: The Low Tubal Rank-Based Method

Tensor-SVD decomposition has been efficiently used in video image completion and dynamic MR image reconstruction problems [4,16,48,49]. As the color image is highly unbalanced in the size of three dimensions, which is not suitable for the low tubal rank constraint, the second flexible QA scheme is exploited to dig out the potential low tubal rank prior information.

Considering that tubal rank minimizations are more efficient for the balanced tensor [10], we first turn the unbalanced image into the balanced order-3 data using the second flexible QA scheme.

Take the color image size of 256 × 256 × 3 as an example, as shown in Figure 3a. The order-4 tensor size of 128 × 128 × 4 × 3 is obtained by the basic QA schemes, and by multiplying the basic QA schemes, the order-4 tensor size of 64 × 64 × 4 × 4 × 3 is obtained. Lastly, the order-4 tensor is reshaped into the balanced order-3 tensor size of 64 × 64 × 48. Here, the context of ‘balanced’ is that the size changes from the unbalanced 256 × 256 × 3 to the more balanced size of 64 × 64 × 48. In practice, the size of the designed order-3 tensor should be as balanced as possible. We call the above the second flexible QA scheme.

Figure 3b shows the low tubal rankness of the balanced order-3 data (with the size of $n_1\times n_2\times n_3=64\times 64\times 48$ here) by plotting ${\delta }_j$, which is defined as follows:

$${\delta }_j={\displaystyle \frac{1}{n_3}}{\displaystyle \sum _{i=1}^{n_3}T(i, i, j)},\hspace{1em}i=1, 2, \cdots , \min (n_1, n_2)$$

Then, TNN is used to enforce the tensor tubal rank in the image inpainting model as follows:

$$\underset{X}{\min }{‖{\Phi }_{2}X‖}_{TNN}\hspace{1em}\mathrm{s}ubject to X(i, j)=Y(i, j), \forall (i, j)\in \Omega$$

(12)

where ${\Phi }_2$ denotes the operator of permuting the color image into a more ‘balanced’ order-3 tensor using the second flexible QA scheme. Combining the low tubal rank and sparsity, we introduce auxiliary variables $\mathcal{B}={\Phi }_{2}X$ and $Z=DX$, then rewrite (12) as the following unconstrained convex optimization problem:

$$\begin{array}{l}\underset{X}{\min }{\tau }_{\Omega }(X)+{\displaystyle \sum _i^{I_3}{‖{\overline{\mathcal{B}}}^{(i)}‖}_{*}}+{\displaystyle \frac{\rho }{2}}{‖{\Phi }_{2}X-\mathcal{B}+\Lambda ‖}_F^2\\ \hspace{1em} +{\displaystyle \frac{\beta }{2}}{‖Z‖}_1+{\displaystyle \frac{\beta {\rho }_2}{2}}{‖DX-Z+L‖}_F^2\end{array}$$

(13)

where $I_3$ is the third size of the 3-order tensor ${\Phi }_{2}X$. We apply the algorithm by ADMM to solve model (13), as shown in Algorithm 2.

Algorithm 2. The algorithm for solving the model (13)

Input: $Y, \Omega , {\rho }_1 ,\beta , {\rho }_2$, the maximum number of iteration $t_{\max }$, convergence condition ${\eta }_{tol}$.

Initialization: ${\Lambda }^{(0)}$, $L^{(0)}$, ${\mathcal{B}}^{(0)}$, $Z^{(0)}$, t = 0.

While $t<t_{\max }$ and $\eta <{\eta }_{\max }$ do         QA scheme: Turn an image into the balanced order-3 tensor ${\Phi }_{2}X$.         Update $X^t=\arg \underset{X}{\min }{\tau }_{\Omega }(X)+{\displaystyle \frac{\rho }{2}}{‖{\Phi }_{2}X-{\mathcal{B}}^{t-1}+{\Lambda }^{t-1}‖}_F^2$         Update ${\overline{\mathcal{B}}}^{(i)t}=\arg \underset{{\overline{\mathcal{Z}}}^{(i)}}{\min } {‖{\overline{\mathcal{B}}}^{(i)}‖}_{*}+{\displaystyle \frac{\rho }{2}}{‖{\Phi }_2{X}^t-\mathcal{B}+{\Lambda }^{t-1}‖}_F^2,\hspace{1em}i=1, \cdots , I_3$         Update $Z^t=\arg \underset{{\overline{\mathcal{Z}}}^{(i)}}{\min } {‖Z‖}_1+{\displaystyle \frac{\rho }{2}}{‖D{X}^t-Z+L^{t-1}‖}_F^2$         Update ${\Lambda }^t={\Lambda }^{t-1}+{\Phi }_2{X}^t-{\mathcal{B}}^t$, $L^t=L^{t-1}+D{X}^t-Z^t$         Update ${\eta }_{t+1}={\displaystyle \frac{{‖X_n^{t+1}(:)-X_n^t(:)‖}_F}{{‖X_n^t(:)‖}_F}}$, $t=t+1$. End while

Output: $X^{*}$.

3.4. Method 3: The Low TT Rank-Based Method

TT decomposition works better on higher-order tensors than Tucker decomposition. To fulfill TT decomposition efficiently, the third flexible QA scheme is exploited to permute the 3-order image into a higher-order tensor. Based on the basic QA scheme, high-order tensors can be obtained flexibly.

The third flexible QA scheme is shown below. Take a $16\times 16$ matrix as an example, as shown in Figure 4a. First, a 2-order matrix is turned into a 3D tensor via the basic QA scheme, and the basic QA is repeated to obtain the final 4D tensor with the size of $4\times 4\times 4\times 4$. The entry comes from the i^th smaller matrix of the first basic QA scheme, and the j^th smaller matrix of the second basic QA scheme is labeled as the MATLAB notation $(:,:,i,j)$. By analogy, the third flexible QA scheme can permute a matrix with the size of $4^P\times 4^Q$ to order-$\min \{P,Q\}$ tensor with the size of $\underset{\min \{P,Q\}}{\underbrace{4\times 4\times \cdots \times 4}}$. An RGB image with the size of $4^P\times 4^Q\times 3$ can be permuted into an order-$\min \{P,Q\}+1$ tensor with the size of $\underset{1+\min \{P,Q\}}{\underbrace{4\times 4\times \cdots \times 4\times 3}}$. The third flexible QA scheme should ensure that the designed tensor has a higher order.

We permute the Lena image into a high-order tensor via the third flexible QA scheme, and then obtain the TT matrices of the augmented tensor. We name these TT matrices as QA-TT matrices, and their singular values are shown in Figure 4b, which demonstrates the low TT rankness of the rearranged tensor.

Then, we enforce the low TT rankness to improve the inpainting accuracy. The third model is as follows:

$$\underset{X}{\min }{\displaystyle \sum _{n=1}^N{\alpha }_n{‖{\mathcal{T}}_n{\Phi }_{3}X‖}_{*}}\hspace{1em}\mathrm{s}ubject to X(i, j)=Y(i, j), \forall (i, j)\in \Omega$$

(14)

where ${\Phi }_3$ stands for the third flexible QA used to permute image $X$ into a high-dimensional tensor. We name the tensor obtained by the third flexible QA scheme as a QA tensor. ${\mathcal{T}}_n$ is the operator that converts a tensor into the n^th TT matrix, $n=1, 2, \cdots , N$. The order of QA tensor is $N$. The inverse operators corresponding to $\Phi$ and ${\mathcal{T}}_n$ are ${\Phi }^{-1}$ and ${\mathcal{T}}_n^{-1}$, respectively. The weight ${\alpha }_n$ is as follows:

$${\alpha }_n={\displaystyle \frac{{\theta }_n}{{\displaystyle {\sum }_{n=1}^{N-1}{\theta }_n}}}\mathrm{with} {\theta }_n=\min ({\displaystyle \prod _{l=1}^n{I}_l},{\displaystyle \prod _{l=k+1}^N{I}_l})$$

(15)

where $I_1\times I_2\times \cdots \times I_N$ is the size of the QA tensor.

Combining the low TT rank and sparsity constraints, we introduce auxiliary variables $\mathrm{Z}=DX$ and $U_n{V}_n^H={\mathcal{T}}_n{\Phi }_{3}X$ and rewrite (14) as the following unconstrained convex optimization problem for all $n=1, \cdots ,N-1$:

$$\begin{array}{l}\underset{U_n,V_n,{\Lambda }_n,L,Z,X}{\min }{\tau }_{\Omega }(X)+{\displaystyle \frac{1}{2}}{\displaystyle \sum _{n=1}^{N-1}{\alpha }_n({‖U_n‖}_F^2+{‖V_n‖}_F^2)}+{\displaystyle \frac{\beta }{2}}{‖Z‖}_1\\ +{\displaystyle \frac{{\rho }_1}{2}}{\displaystyle \sum _{n=1}^{N-1}{\alpha }_n{‖{\mathcal{T}}_n{\Phi }_{3}X-U_n{V}_n^H+{\Lambda }_n‖}_F^2)}+{\displaystyle \frac{\beta {\rho }_2}{2}}{‖DX-Z+L‖}_F^2\end{array}$$

(16)

By applying ADMM, each sub-problem is performed at each iteration t. Lastly, we obtain $X$ by $X^{*}={\displaystyle {\sum }_{n=1}^{N-1}{\alpha }_n}{X}_n^{\ast }$, where $X_n^{*}$ represents the optimal solution of the nth subproblem. The whole algorithm for solving the model (16) is shown in Algorithm 3.

Algorithm 3. The algorithm for solving the model (16)

Input:$Y, \Omega , \beta , {\rho }_1 ,{\rho }_2$, the maximum number of iteration $t_{\max }$, convergence condition ${\eta }_{tol}$.

Initialization: $U_n^{(0)}$, $V_n^{(0)}$ by the LMaFit method [47]; ${\Lambda }_n^{(0)}$, $L_n^{(0)}$, $Z^{(0)}$.

For n = 1 to N − 1 do    t = 0.    While $t<t_{\max }$ and $\eta <{\eta }_{\max }$ do         QA scheme: permute image to order-N tensor ${\Phi }_{3}X$.         Update $X_n^t=\arg \underset{X}{\min }{\tau }_{\Omega }(X)+{\displaystyle \frac{{\rho }_1}{2}}{‖{\mathcal{T}}_n{\Phi }_{3}X-U_n^{t-1}{V}_n^{(t-1)H}+{\Lambda }_n^{t-1}‖}_F^2$         Update $U_n^t=\arg \underset{U_n}{\min }{‖U_n‖}_F^2+{\displaystyle \frac{{\rho }_1}{2}}{‖{\mathcal{T}}_n{\Phi }_3{X}^t-U_n{V}_n^{(t-1)H}+{\Lambda }_n^{t-1}‖}_F^2$         Update $V_n^t=\arg \underset{V_n}{\min }{‖V_n‖}_F^2+{\displaystyle \frac{{\rho }_1}{2}}{‖{\mathcal{T}}_n{\Phi }_3{X}^t-U_n^t{V}_n^H+{\Lambda }_n^{t-1}‖}_F^2$         Update $Z^t=\arg \underset{Z}{\min }{‖Z‖}_1+{\displaystyle \frac{{\rho }_2}{2}}{‖D{X}^t-Z+L^{t-1}‖}_F^2$         Update ${\Lambda }_n^t={\Lambda }_n^{t-1}+T_n{\Phi }_3{X}^t-U_n^t{V}_n^{(t)H}$, $L^t=L^{t-1}+D{X}^t-Z^t$         Update ${\eta }_{t+1}={\displaystyle \frac{{‖X_n^{t+1}(:)-X_n^t(:)‖}_F}{{‖X_n^t(:)‖}_F}}$, $t=t+1$.    End while End for

Output: $X^{*}={\displaystyle {\sum }_{n=1}^{N-1}{\alpha }_n{X}_n^{*}}$.

4. Experimental Results and Analyses

We applied the above methods 1–3 to solve image inpainting problems. For simplicity, methods 1–3, which only exploit low unfolding matrix rank, low tensor tubal rank, and low tensor train rankness, are denoted as UfoldingLR, TTLR, and tSVDLR methods, respectively. The methods that enhance the low rank and total variation constraints simultaneously are denoted as UnfoldingLRTV, tSVDLRTV, and TTLRTV methods, respectively. The low matrix rank completion method, which is solved by model (17) and the ADMM algorithm, is denoted as the MatrixLR method.

$$\underset{X}{\min }\hspace{1em}rank(X)\hspace{1em}\mathrm{subject} \mathrm{to}\hspace{1em}X(i, j)=Y(i, j), \forall (i, j)\in \Omega$$

(17)

The method that only exploits sparsity in the gradient domain and is solved by the ADMM algorithm is denoted as the TV method. In addition, the following close methods were applied for comparison; some of their codes are available online.

STDC: This method exploits the images into three-factor matrices and one core tensor for image inpainting [7,21,22] (http://doi.ieeecomputersociety.org/10.1109/TPAMI.2013.164, accessed on 12 December 2019).

HaLRTC: This method constrains the low rankness of the three mode-n matrices caused by the decomposition of a color image for inpainting, which is solved by the ADMM [23,24] (https://www.cs.rochester.edu/u/jliu/publications.html, accessed on 18 December 2019).

SPCTV (https://sites.google.com/site/yokotatsuya/home/software, accessed on 12 January 2020). The smooth PARAFAC tensor completion and total variation method [25] uses the PD (PARAFAC decomposition, a derivation of Tucker decomposition) framework and constrains the TV on every factor matrix of PD.

LRTV: This method combines the constraints of the low rankness of every mode-n matrix and the TV regularization on every mode-n matrix for color image inpainting [26,27] (https://sites.google.com/site/yokotatsuya/home/software/lrtv_pds, accessed on 16 April 2021).

FBCP: Inpainting methods based on Bayesian tensor completion [28,29,30,31] (https://github.com/qbzhao/BCPF, accessed on 10 March 2021).

TTC and TTCTV: The tensor completion method with tensor train and with total variation regularized tensor trains, respectively [50] (https://github.com/IRENEKO/TTC, accessed on 9 March 2021).

CFN-RTC: This robust tensor completion method combines capped Frobenius norm and TR decomposition and adds dimensions through simple data reshaping for processing image [33] (https://github.com/Li-X-P/Code-of-Robust-Tensor-Completion, accessed on 10 March 2021).

RNC-FCTN: This robust tensor completion method is based on FCTN, but it is limited to high-dimensional data processing [32] (https://faculty.swjtu.edu.cn/zhengyubang/zh_CN/zdylm/839966/list/, accessed on 10 March 2021). Therefore, to adapt to their code, we rearranged the images into four-dimensional space.

All simulations were carried out on Windows 10 and MATLAB R2019a running on a PC with an Intel Core i7 CPU 2.8 GHz and 16 GB of memory. For a fair comparison, every method was conducted with its optimal parameters to ensure every method had the best performance. The reconstruction quality was quantified using the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) (http://www.ece.uwaterloo.ca/~z70wang/research/ssim/, accessed on 10 April 2017) [41]. The original color images (from the standard image database) and missing patterns used in the experiments are shown in Figure 5. The images used in this section are from the Berkeley Segmentation dataset (https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/, accessed on 21 April 2024) and USC-SIPI image database (http://sipi.usc.edu/database/database.php, accessed on 17 March 2023).

We set the maximum number of iterations $t_{\max }=100$ and convergence condition ${\eta }_{tol}={10}^{-6}$ in all our methods (UnfoldingLRTV, tSVDLRTV, and TTLRTV). The pixel range of all the images was normalized to 0-1. In UnfoldingLR, tSVDLR, and TTLR methods, we set ${\rho }_1 =$ 0.04, 0.002, and 0.6, respectively. In UnfoldingLRTV, Tsvdlrtv, and TTLRTV methods, the parameter set $({\rho }_1 , \beta , {\rho }_2)$ was (0.4, 0.004, 2), (0.6, 0.07, 0.1), and (0.7, 0.03, 0.1), respectively.

4.1. Analyses of the Three Flexible QA Schemes

Next, we used the first, second, and third flexible QA schemes. The PSNRs (dB)/SSIMs of the UnfoldingLR, tSVDLR, and TTLR methods with and without the QA scheme are shown in

Table 3. PSNR (dB)/SSIM and SSIM of the seven methods without rearrangement and with rearrangement.

Methods		PSNR (dB)/SSIM of Different Color Images Under Different Missing Patterns
		House	Lena	Airplane	Boats
		Random 50%	Lines	Random line	Random 80%
Without rearrangement	MatrixLR	9.38/0.8970	13.34/0.5850	7.118/0.1308	19.18/0.5680
	TTLR	28.61/0.871	13.34/0.585	7.11/0.130	19.25/0.519
	tSVDLR	32.30/0.932	13.34/0.585	7.11/0.130	21.60/0.707
	UnfoldingLR	7.83/0.093	13.34/0.585	7.11/0.130	6.32/0.102
With rearrangement	TTLR	30.21/0.9251	31.79/0.9559	25.77/0.8796	21.44/0.7144
	tSVDLR	29.79/0.8989	31.20/0.9561	18.91/0.8386	21.34/0.6879
	UnfoldingLR	32.58/0.9416	33.45/0.9771	28.75/0.9464	23.46/0.8139

. The red numerical values correspond to the worst results. It can be seen that, without the QA scheme, Lena and Airplane could not be recovered. The UnfoldingLR, tSVDLR, and TTLR methods with the QA scheme had better numerical results than those without the QA scheme. In the low matrix rank completion method (i.e., MatrixLR), no QA scheme was applied, i.e., the color image was dealt with as three-channel matrices directly.

Due to the support of the QA scheme, the low tensor rank-based methods (TTLR, tSVDLR, and UnfoldingLR) with the QA scheme provided better results than the traditional low matrix rank completion method (i.e., MatrixLR method). Therefore, the QA scheme can be successfully used as the first step to explore the low tensor rank prior to an image.

The KA scheme and the third flexible QA scheme can both rearrange an image into a high-order tensor. However, our QA scheme is different from the KA scheme used in [23]. The KA scheme can maintain the local block similarity of the image, while the third flexible QA scheme can use adjacent pixels to maintain the global similarity of the image. We conducted a comparison of KA and the third flexible QA scheme under the corresponding TMac-TTKA [23] and TTLR methods. As shown in Figure 6, the small blocks are obvious in the recovered images obtained by the TMac-TTKA method. The images recovered by the TTLR method preserve more details without the obvious blocks.

4.2. Analyses of the Methods Exploiting Both Low Rankness and Sparsity

We analyzed the recovery results of the methods exploiting low rankness and sparsity. Figure 7, Figure 8 and Figure 9 show visual comparisons of the eleven methods for recovering the House, Lena, and Baboon images, respectively.

Table 4. PSNR (dB)/SSIM and SSIM of the nine methods.

	No.	Methods	PSNR (dB)/SSIM of Different Color Images Under Different Missing Patterns
			House	Peppers	Lena	Airplane	Baboon	Boats
			Random 50%	Text	Lines	Random Line	Blocks	Random 80%
Other methods	1	STDC	32.04/0.9300	33.61/0.9813	28.56/0.8995	23.49/0.7756	27.01/0.9293	21.88/0.7340
	2	HaLRTC	32.07/0.9423	25.84/0.9496	13.34/0.5850	19.94/0.6334	28.04/0.9397	20.56/0.6858
	3	FBCP	26.41/0.8701	NAN	14.56/0.5242	10.25/0.1954	18.71/0.5546	20.91/0.6947
	4	TMac-TTKA	23.18/0.8113	29.47/0.9681	29.93/0.9462	20.82/0.7521	28.04/0.9429	8.83/0.1229
	5	SPCTV	29.56/0.9133	23.38/0.9154	16.02/0.6107	18.58/0.6894	24.21/0.9144	20.98/0.7254
	6	LRTV	30.93/0.9382	36.98/0.9945	34.07/0.9724	26.82/0.9228	27.10/0.9319	21.62/0.7541
Our methods	1	TTLRTV	33.02/0.9579	37.27/0.9945	34.94/0.9823	28.82/0.9561	29.46/0.9559	22.37/0.7487
	2	tSVDLRTV	32.20/0.9550	37.49/0.9950	34.70/0.9818	28.03/0.9507	29.56/0.9574	22.86/0.8021
	3	UnfoldingLRTV	35.61/0.9689	37.72/0.9952	34.87/0.9821	29.55/0.9639	29.59/0.9556	25.43/0.8863

shows the PSNR (dB)/SSIM results of the nine methods for recovering different color images under different missing patterns. Figure 10 depicts the PSNR curves of the inpainting results using the different methods; the missing ratio ranges from 10% to 70% under a random missing pattern.

As shown in Figure 7, Figure 8, Figure 9 and Figure 10 and Table 4, compared to the numerous close STDC, HaLRTC, FBCP, TMac-TTKA, SPCTV, and LRTV methods, the UnfoldingLRTV, tSVDLRTV, and TTLRTV methods had superior performance for both visual and quantity results. The SPCTV and LRTV methods also enhanced the low rankness and sparsity simultaneously, but the results were worse than our methods.

Table 5. PSNR (dB)/SSIM and SSIM of the eight methods.

No.	Methods	PSNR (dB)/SSIM of Different Color Images Under Different Missing Patterns
		House	Peppers	Lena	Airplane	Baboon	Boats
		Random 50%	Text	Lines	Random Line	Blocks	Random 80%
1	MatrixLR	9.38/0.8970	33.23/0.9814	13.34/0.5850	7.118/0.1308	27.62/0.9343	19.18/0.5680
2	TV	29.70/0.8816	34.14/0.9913	29.21/0.9107	22.85/0.8463	23.18/0.9066	20.32/0.6103
3	TTLR	30.21/0.9251	34.86/0.9892	31.79/0.9559	25.77/0.8796	25.42/0.9239	21.44/0.7144
4	tSVDLR	29.79/0.8989	33.86/0.9840	31.20/0.9561	18.91/0.8386	28.03/0.9373	21.34/0.6879
5	UnfoldingLR	32.58/0.9416	36.86/0.9938	33.45/0.9771	28.75/0.9464	22.22/0.9238	23.46/0.8139
6	TTLRTV	33.02/0.9579	37.27/0.9945	34.94/0.9823	28.82/0.9561	29.46/0.9559	22.37/0.7487
7	tSVDLRTV	32.20/0.9550	37.49/0.9950	34.70/0.9818	28.03/0.9507	29.56/0.9574	22.86/0.8021
8	UnfoldingLRTV	35.61/0.9689	37.72/0.9952	34.87/0.9821	29.55/0.9639	29.59/0.9556	25.43/0.8863

shows the PSNR (dB)/SSIM results of the eight methods. The MatrixLR method only constrains the low matrix rank; the TV method only exploits the TV prior; the UnfoldingLR, tSVDLR, and TTLR methods only constrain the low unfolding matrix rank, low tubal rank, and low TT rank, respectively; and the UnfoldingLRTV, tSVDLRTV, and TTLRTV methods combine both sparsity and low tensor rankness. As shown in Table 5, the combination of sparsity and low tensor rankness constraints could yield better inpainting results than enforcing sparsity or low rankness alone. The TTLR method was more efficient than the MatrixLR and TV methods. The results of the tSVDLR method and the TTLR method were comparable. The UnfoldingLR method provided the best results among the TTLR, tSVDLR, TuckerLR, MatrixLR, and TV methods. UnfoldingLRTV, tSVDLRTV, and TTLRTV methods had improved numerical results compared to the corresponding UnfoldingLR, tSVDLR, and TTLR methods, which demonstrates that TV prior is efficient in improving the accuracy of low rank-based inpainting methods.

The visual and numerical PSNR (dB)/SSIM comparisons of our methods for recovering the pepper image under 80% random missing patterns are shown in Figure 11. In the first row of Figure 11, the methods only exploit low-rank constraints. As shown in the color box, there are small blocky errors in the recovered image, these are caused by the QA scheme. This phenomenon can be solved by combining the constraints of low rank and sparsity (TV), as shown in the second row of Figure 11.

All in all, due to the support of the QA scheme and the efficient TV prior, the low tensor rank-based methods (UnfoldingLRTV, tSVDLRTV, and TTLRTV) were superior to other close low tensor rank-based methods. The UnfoldingLRTV method provided the best results among all the methods examined.

4.3. Analyses of TTLR and TTLRTV Methods

The TT-based methods (i.e., TTLR and TTLRTV) were analyzed in detail. As TT rank is multi-rank, how does every TT matrix rank affect the final result? We answer this question with the below experimental results.

We conducted the experiments on recovering House, Lena, and Airplane images with a size of 256 × 256 × 3. The random missing patterns had four missing ratios: 10%, 30%, 50%, and 70%, respectively. We labeled the eight TT matrices as k = 1, 2, …, 8. The PSNR (dB) results of $X_n^{*}$ (the optimal solution of the n^th subproblem, which exploits the n^th TT matrix rank), in the TTLR and TTLRTV methods are shown in Figure 12.

From Figure 12, it can be seen that the PSNR (dB) results of each subproblem is steeply different for the TTLR method, which demonstrates that each TT matrix rank contributes different PSNR results. As there are no rules to find which TT matrix rank meets the best PSNR result, each solution of $X_n^{*}$ should be combined to obtain the final $X^{*}$, i.e., $X^{*}={\displaystyle {\sum }_{n=1}^{N-1}{\alpha }_n{X}_n^{*}}$. Comparing the PSNR curves of TTLR and TTLRTV methods in Figure 12, the PSNR (dB) results of each subproblem is slightly different for the TTLRTV method, which demonstrates that the combination of TT and TV can make the PSNR more balanced among all k.

4.4. Missing Ratio of 90%

To further test the effectiveness of our method under 90% missing entries, we conducted comparative experiments with the state-of-the-art TTC and TCTV methods [50] based on tensor TT decomposition, the CFN-RTC method [33] based on TR decomposition, and the RNC-FCTN method [32] based on FCTN decomposition. For fairness, we directly used the codes and images provided by the TTC and TTCTV methods in [50] and https://github.com/IRENEKO/TTC, where the related parameters are also set by their code. We directly used the codes provided by the CFN-RTC and RNC-FCTN methods, set the optimal parameters such as tensor rank, and arranged the input images according to the size in their codes. The original image is shown in Figure 13. At a random 90% missing entries, the PSNR and SSIM of the observed image were 5.7036 dB and 3.56%, respectively. Comparing ten methods, including TTC, TTCTV, CFN-RTC, RNC-FCTN, TTLR, UnfoldingLR, TTLRTV, tSVDLR, tSVDLRTV, and UnfoldingLRTV, the visual results are shown in Figure 13, and the numerical indicators are shown in

Table 6. Under 90% missing, the PSNR (dB)/SSIM of the ten methods, the observed PSNR (dB)/SSIM was 5.7/0.0356.

Methods	CFN-RTC	TTC	TTLR	tSVDLR	UnfoldingLR
PSNR	19.65	8.70	19.05	20.54	19.44
SSIM	0.4507	0.1745	0.4688	0.4862	0.4455
Methods	RNC-FCTN	TTCTV	TTLRTV	tSVDLRTV	UnfoldingLRTV
PSNR	8.79	8.75	21.46	22.11	21.65
SSIM	0.0929	0.1817	0.5483	0.6007	0.5706

.

Experiments showed that, under high missing rates, there was significant visual pollution in the recovered images of the ten methods. However, compared to the original polluted image, there was a significant improvement, and there were differences in the recovered details among the ten methods. The CFN-RTC, TTC, TTCTV, and tSVDL methods had significant grid-like pollution in repairing images, while the TTLR and UnfoldingLR methods had significant block-like pollution. The RNC-FCTN method had the worst visual effect, but this does not mean that this method has no advantage in processing high-dimensional videos or hyperspectral video (HSV) data. We only forced it to process image data in this experiment, and the results showed that it had a certain effect. The TTLRTV, tSVDLRTV, and UnfoldingLRTV methods had better visual effects than the other methods, but there was significant pulse-like noise in the details.

In terms of numerical indicators, the TTC and TTCTV methods could only improve PSNR and SSIM by about 3 dB and 14%, respectively, which is clearly not advantageous compared to our five methods. Our five methods could improve PSNR and SSIM by more than 13 dB and 40%, respectively. The CFN-RTC method could also achieve close improvements.

The RTC-FCTN method is based on FCTN decomposition, while the CFN-RTC method is based on TR decomposition. FCTN is an improvement of TR decomposition, but the experimental results of the two methods were different. The RNC-FCTN method was directly used to process high-dimensional data, while in the CFN-RTC method experiment, the authors first simply reshaped low-dimensional data into high-dimensional data, which enabled the TR decomposition tool to be applied to low-dimensional data processing.

Analyzing the reasons for the above experimental results, our method may greatly benefit from using more flexible data rearrangement schemes, which make low-rank approximation more sufficient and are based on more effective distributed iterative solving algorithms. Moreover, the TTC and TTCTV methods are not based on the inpainting model established by decomposing tensors into multiple TT matrix forms but rather on directly decomposing tensors into less precise forms of multiple core tensors.

4.5. Runtime and Complexity Analysis

From a high-dimensional curse perspective, converting an image to a higher-order tensor can result in increased complexity, which inevitably leads to a longer runtime. We compared our methods (UnfoldingLRTV, tSVDLRTV, and TTLRTV) with the traditional MatrixLR method and the close STDC, HaLRTC, FBCP, SPCQV, and LRTV methods in terms of running time, as shown in

Table 7. Runtimes(s) of the different methods.

Methods	Runtime (s)
	House	Lena	Airplane	Boats
	Random 50%	Lines	Random Lines	Random 80%
MratrixLR	4.95	0.17	0.16	5.01
STDC	5.43	5.13	5.17	5.16
HaLRTC	8.00	0.88	0.84	6.84
FBCP	188.32	86.45	132.09	219.33
SPCTV	19.25	16.37	16.03	17.69
LRTV	19.08	20.17	21.04	21.05
TTLRTV	145.5	143.2	142.6	142.3
tSVDLRTV	15.23	15.07	15.17	15.14
UnfoldingLRTV	9.49	8.53	8.69	8.72

.

UnfoldingLRTV methods: In the first step, the QA scheme was used to decompose a single image into several small graphs. Because of the similarity of these small graphs, the QA tensor could be reduced to a matrix with a low-rank structure in an unfolding way. Ignoring TV constraints, the unfoldingLRTV method only needed to solve the low-rank matrix completion problem of an unfolding matrix, so the running time was similar to the traditional MatrixLR method, and the accuracy was higher than the traditional MatrixLR method.

The tSVDLRTV methods: As the color image was highly unbalanced in the size of the three dimensions, which is not suitable for the low tubal rank constraint, we used the QA scheme to rearrange an image into a third-order tensor with a more balanced size of every dimension. Then, TNN was used to constrain the low tubal rank of the rearranged tensor. Due to the fast Fourier scheme, it is necessary to perform a low-rank matrix constraint on each frontal slice after the threeh more efficient low t-dimensional Fourier transform. At this time, the SVD decomposition process will increase the time consumption.

TTLRTV methods: TT multi-rank is the combination of the rank of each TT matrix. The TTLRTV method essentially completes the same data amount N-1 times, where N is the order of the QA tensor. Therefore, although the TITRTV method is effective, it is necessarily more computationally expensive than the low matrix rank completion method.

In summary, among the three methods, the UnfoldingLRTV method achieved the best performance both in accuracy and runtime. The TTLRTV method reached better accuracy, but it was time-consuming. The tSVDLRTV method showed moderate performance both in runtime and accuracy.

All in all, the above three methods can exploit the potential low-rank prior of an image and can be successfully used for image inpainting problems, demonstrating that the three flexible QA schemes are perfect for exploring the low-rank prior of an image.

5. Conclusions

To effectively explore the potential of low tensor rank prior to an image, a rearrangement scheme (QA) was first exploited for permuting the color image (3-order) into three flexible rearrangement forms (with a more efficient low tensor rank structure). Based on the scheme, three optimization models exploiting the low unfolding matrix rank, low tensor tubal rank, and low TT multi-rank were developed to improve the accuracy in image inpainting. Combined with TV constraints, efficient ADMM algorithms were developed to solve these three optimization models. The experimental results demonstrate that our low tensor rank-based methods are effective for image inpainting and are superior to the low matrix rank completion method and numerous close methods. The low tensor rank constraint is effective for image inpainting, which is mainly due to the support of the QA scheme.