Research on Unsupervised Feature Point Prediction Algorithm for Multigrid Image Stitching

Abstract

The conventional feature point-based image stitching algorithm exhibits inconsistencies in the quality of feature points across diverse scenes. This may result in the deterioration of the alignment effect or even the inability to align two images. To address this issue, this paper presents an unsupervised multigrid image alignment method that integrates the conventional feature point-based image alignment algorithm with deep learning techniques. The method postulates that the feature points are uniformly distributed in the image and employs a deep learning network to predict their displacements, thereby enhancing the robustness of the feature points. Furthermore, the precision of image alignment is enhanced through the parameterization of APAP (As-projective-as-possible image stitching with moving DLT) multigrid deformation. Ultimately, based on the symmetry exhibited by the homography matrix and its inverse matrix throughout the projection process, image chunking inverse warping is introduced to obtain the stitched images for the multigrid deep learning network. Additionally, the mesh shape-preserving loss is introduced to constrain the shape of the multigrid. The experimental results demonstrate that in the real-world UDIS-D dataset, the method achieves notable improvements in feature point matching and homography estimation tasks, and exhibits superior alignment performance on the traditional image stitching dataset.

1. Introduction

Inthe contemporary digital age, image processing technology is assuming an increasingly pivotal role in a multitude of fields. Image alignment is a crucial issue in the field of computer vision, with applications in image stitching [1], object recognition [2], 3D reconstruction [3], and other areas. Its use in these contexts has the potential to enhance image quality and accuracy, which is of great significance.

Conventional alignment techniques for image stitching are typically founded upon pixel alignment discrepancies and feature point-based image warping. These techniques generally align the images by evaluating the homography of the two images, which is an invertible mapping of a plane in two viewpoints. Pixel alignment error-based methods optimize the homography parameter between two images by iterating [4], but this method generally yields superior results only when the overlap is high. Conventional feature-point-based image alignment methods typically employ disparate feature extractors [5,6,7] and robustness estimation methods [8,9] to identify the feature points following exact matching. These methods then utilize distinct alignment strategies to align the images. Alignment strategies such as APAP [10] employ a grid-based approach to align the image as much as possible. SVA [11] introduces a smoothly varying affine stitching field while retaining the good extrapolation and occlusion processing properties of the parametric transforms. ELA [12] is a robust image stitching method based on the TPS [13] transform for addressing the parallax-tolerance issue in image stitching. TFA [14] employs the technique of triangular facet approximation, which divides the image into small triangular facets and performs locally adaptive image alignment for each triangular region. SPHP [15] employs the semi-projective transform, which lies between affine and projective transforms, to transform the image, thereby preserving its underlying geometry to a certain extent. GSP [16] enhances image stitching by introducing global similarity a priori information, which results in a more natural and coherent stitching outcome. AANAP [17] proposes a novel image stitching method that combines multiple techniques to render the panorama more natural. DHW [18] utilizes two homographies to better align the image. However, feature point-based image alignment methods are subject to certain limitations when applied to low-texture and low-light images. For instance, uneven distribution of feature points can occur when the light is not uniform, which can result in distorted alignment results in regions with fewer features. Additionally, insufficient numbers of matching feature points can be encountered in low-overlap images, which can lead to failure to align due to insufficient feature points.

In comparison to traditional methods, deep learning image stitching methods [19,20,21,22] predict four point displacements of an image and solve for a global homography, a homography that only describes the mapping relationship of points on the same plane in two viewpoints. Consequently, this has limitations when dealing with parallax. Ref. [23] proposed a method of meshing the image and predicting four point displacements for each mesh in order to solve for a multigrid homography. This enables multigrid warping in deep learning, but the problem of not being able to determine the pixel-point mapping relationship occurs when going beyond the mesh space. Consequently, obtaining the final stitched image is difficult. Furthermore, due to the prediction of displacements of mesh points, it is not possible to apply traditional image alignment algorithms to the field of homography.

To address these challenges, this study proposes a predictive feature point method based on deep learning. This method enhances the robustness of deep feature extraction by exploring the matching relationship between deep feature maps. The combination of the traditional APAP (As-projective-as-possible image stitching with moving DLT) multigrid image alignment algorithm with the field of deep learning enables the application of the traditional image alignment algorithm to deep learning, thereby improving the accuracy and robustness of image alignment. In order to obtain a complete stitched image, we introduce a mesh shape-preserving loss and train the model by distorting the target image. After the model is trained, we obtain a complete distorted image by chunking the reference image and reverse distorting it to obtain a complete stitched image by combining it with the target image. Our method employs multi-grid homography estimation to address challenging scenarios, thereby enhancing alignment accuracy in comparison to existing deep learning methods [22] that predict a global homography.

Specifically, our method employs singular value decomposition (SVD) to solve the homography of each grid. This is achieved by first obtaining a weighted map of all feature points in each grid. This is conducted by assuming a uniform distribution of feature points, predicting the displacements of the feature points, and simultaneously calculating the weights of each feature point within each grid. Subsequently, the accurate image alignment is achieved by warping the target image in the grid space. The proposed framework can be readily trained in an unsupervised manner using pixel-level content loss. Additionally, we introduce a shape-preserving loss to constrain the mesh inverse warping, thereby preventing the stitched images from appearing cracked (see Figure 1b). Once training is complete, the reference image is divided into grid form. The invertibility of the homography matrix is then employed to inverse warp the corresponding image blocks, resulting in a warped map of each image block in the target image space. Finally, all the warped image blocks are synthesized into a complete warped map, which is then spliced with the target image.

In the course of our experiments, we assess the efficacy of our approach in the domains of homography estimation, image alignment, and feature point prediction. The experimental results in real scenarios demonstrate the superiority of the method. The principal contributions of this paper are as follows:

1. A deep learning network for predicting feature points is proposed. This network achieves more robust deep feature extraction and ensures that the network outputs the desired number of feature points in any case by custom setting the number of feature points.

2. The APAP multigrid deformation parameterization was implemented for unsupervised multigrid graph alignment.

3. Image chunking inverse warping and mesh shape-preserving loss are proposed. This is achieved by first chunking the reference image, inverting the image chunks using the invertibility of the singular response matrix, and then using the mesh shape-preserving loss to constrain the distances between image chunks in order to obtain a complete spliced image.

2. Related work

2.1. Traditional Image Stitching

The conventional approach to image stitching typically comprises three distinct stages. The initial step is the detection and matching of feature points. The second step is adaptive warping. The final step in the process is image fusion.

Feature point detection and matching: Feature point detection algorithms can extract key points of special significance from an image, which provides an important foundation for applications such as image matching, target tracking, 3D reconstruction, and so forth. The SIFT algorithm is a scale-invariant feature point detection algorithm based on scale invariance. It is capable of detecting the same feature points under different scales, different rotations, and different brightness conditions. The SURF algorithm is a fast and robust feature point detection algorithm. It accelerates the feature point detection and matching process by using techniques such as integral images and fast hash tables. The ORB algorithm combines the advantages of speed, rotation invariance, and scale invariance, rendering it suitable for feature extraction and matching in real-time image processing and in environments where computational resources are limited. The Harris corner point detection algorithm is a classical feature point detection algorithm that searches for significant features in an image by detecting the corner points in the image. To obtain more accurate feature points, RANSAC [24] employs random sampling and iterative optimization to estimate the mathematical model parameters and remove outliers in the data, thereby facilitating the acquisition of more accurate matches. MAGSAC [9] represents an enhancement of RANSAC [24] that utilizes the M-estimator function and adaptive weight updating to enhance resilience to outliers. MAGSAC++ [8] incorporates a preprocessing step, an optimized sampling strategy, and an interior point evaluation method based on MAGSAC [9] to enhance the robustness and efficiency of the algorithm.

Adaptive warping is a common method employed in the alignment of images to be stitched. SVA [11], ELA [12], TFA [14], and AutoStitch [1] employ the extraction of differentiated keypoints to construct a global transform. DHW [18] aligns the foreground and background using two homographies. In contrast, apap [10] grids the image, giving greater weight to nearer feature points and lesser weight to farther feature points. It then seeks a local homography transform for each grid, thereby achieving superior alignment accuracy. Ref. [25] proposed the elimination of feature points with large residuals by means of a local homography transform, followed by culling, in order to obtain a more accurate alignment. The use of SPHP [15], GSP [16], and AANAP [17] was proposed in order to better preserve the shape.

Image fusion is the process of combining two or more images into a single, unified image. Image fusion is the process of combining two images to create a single, complete image. Image fusion is achieved through a process of stitch cutting and image fusion. For instance, stitch cutting [26] identifies the optimal stitch based on pixel value differences, thereby enhancing the visual coherence of the overlapping regions in the image. Additionally, there are gradient differences [27,28], motion and motion-aware differences [29], significant differences [30], and unsupervised learning of sutures [31]. Image fusion, for instance, involves decomposing images using a Gaussian pyramid [32], fusing them across different frequency bands, and finally reconstructing seamless stitched images. Another approach, proposed by Nie et al. [22], involves reconstructing stitched images through unsupervised learning.

2.2. Deep Learning Image Stitching

In the context of deep learning image stitching, the determination of the homography between images is typically achieved through the prediction of the offsets of the image vertices.

DeTone et al. [33] proposed the initial deep homography method, which determines the homography by predicting the displacement of four vertices of an image. Nguyen et al. [19] subsequently proposed an unsupervised deep homography method, which determines the homography by unsupervised loss. Zhang et al. [20] proposed a content-aware unsupervised network, which helps to improve the performance of deep homography. Nie et al. [22] proposed the first end-to-end deep image stitching network, which is divided into an image warping part and an image reconstruction part. The warping part employs the network framework of Nie et al. [21], which connects the feature pyramid and feature correlation in a unified framework. This enables the network to predict homography from coarse to fine and to handle scenes with a relatively large baseline. Nie et al. [23] predicts homography by meshing the image and predicting the displacements of the grid points separately to determine the homography for each grid. In contrast, Nie et al. [31] warps an image by using a network to predict the control points for the TPS [13] algorithm. This is achieved by first predicting a global homography warping of the image to initially distort the control points, and then solving for the final TPS warping matrix while predicting the residual displacements of the control points.

3. Methodological Process

The method comprises two stages: deep multigrid warping and chunked inverse transformation. In the initial stage, as illustrated in Figure 2, our method accepts the reference image $I_r$ and the target image $I_t$ as inputs, generates the displacements of the feature points, and then computes the weight matrix W and multiplies it with the feature point mapping relation matrix A. The distortion matrix of the multigrid is obtained by singular value decomposition of the matrix $WA$. The second stage is depicted in Figure 3. It involves the segmentation of the reference image, the application of an inverse transformation superposition to all image segments, the generation of the reference image in the target image view, and the weighted average fusion of the overlapping regions to obtain the final panorama.

3.1. Multi-Mesh Warping Parameterization

The homography transform is a common method for image alignment. It is an invertible mapping from one picture to another with eight degrees of freedom. Every two degrees of freedom are used for translation, rotation, scaling, and straight lines at infinity. However, the homography transform is only capable of aligning a single plane and is, therefore, inadequate for aligning the real picture. The APAP algorithm, a traditional method, is more effective for this problem. However, the algorithm is based on a traditional feature point extraction method, which may result in difficulties aligning the image when the distribution of image feature points is not uniform or there are not enough feature points. While deep learning methods can effectively address this issue, the current deep learning frameworks are primarily based on the displacement prediction of grid points, which limits their ability to utilize the APAP alignment algorithm effectively. To address this issue, a parameterization of APAP warping is implemented, assuming that the feature points are uniformly distributed on the image in the shape of grid points. Deep learning is employed to predict the displacement of each feature point, calculate the weight of each feature point in each grid, and identify the homography of each grid through singular value decomposition.

The APAP alignment algorithm computes the local homography of each grid by matching two sets of feature points, $X^1=\left[x_1^1,x_2^1\dots x_N^1\right]$ of the reference image and $X^2=\left[x_1^2,x_2^2\dots x_N^2\right]$ of the distorted image. The assumption is that $X^1$ is uniformly distributed in the form of a grid on the reference image, and that $X^2$ is obtained by the deep learning network by predicting the displacements and then summing them up with $X^1$. The i-th pair of feature points is defined as $\left[x_i^1,x_i^2\right]\left(x_i^1\in {\mathbb{R}}^{2\times 1},x_i^2\in {\mathbb{R}}^{2\times 1}\right)$. Let $C=\left[c_1,c_2\dots c_M\right]\left(c_k\in {\mathbb{R}}^{2\times 1},k\in \left[0,1\dots M\right]\right)$ be the set of grid centers and $c_k$ the center of the k-th grid. Based on the distance between the reference image feature points and ck, the weight $w_k^i\left(k\in \left[0,1\dots M\right],i\in \left[0,1\dots N\right]\right)$ of each feature point in the local distortion of the k-th grid can be calculated.

$$\begin{array}{c}\hfill w_k^i=max\left(e^{-{∥c_k-x_i∥}^2/{\sigma }^2},\gamma \right).\end{array}$$

(1)

The scale parameter, denoted by $\sigma$, is used to determine the scale of the feature points, while the parameter $\gamma$ is employed to restrict the minimum weight of the feature points. When $\gamma$ equals one, the distortion becomes a global projection distortion.

Let the projection matrix be $\widehat{h}$.

$$\begin{array}{c}\hfill \widehat{h}=\left[\begin{array}{ccc}{h}_1& h_2& h_3\\ h_4& h_5& h_6\\ h_7& h_8& h_9\end{array}\right]=\left[\begin{array}{c}{h}_{r1}\\ h_{r2}\\ h_{r3}\end{array}\right].\end{array}$$

(2)

The value of $h_{ri}$ represents the i-th row of $\widehat{h}$. The Equation (3) can be derived by utilising a pair of feature points.

$$\begin{array}{c}\hfill \left[\begin{array}{ccccccccc}{x}_i^1& y_i^1& 1& 0& 0& 0& -x_i^1{x}_i^2& -y_i^1{x}_i^2& -x_i^2\\ 0& 0& 0& x_i^1& y_i^1& 1& -x_i^2{y}_i^2& -y_i^2{y}_i^2& -y_i^2\end{array}\right]\\ \hfill \left[\begin{array}{c}{h}_{r1}^T\\ h_{r2}^T\\ h_{r3}^T\end{array}\right]=a_i\widehat{h}=\left[\begin{array}{c}0\\ 0\end{array}\right].\end{array}$$

(3)

By employing all available feature points, the following equation can be derived.

$$\begin{array}{c}\hfill \left[\begin{array}{ccccccccc}{x}_1^1& y_1^1& 1& 0& 0& 0& -x_1^1{x}_1^2& -x_1^1{x}_1^2& -x_1^2\\ 0& 0& 0& x_1^1& y_1^1& 1& -x_1^1{x}_1^2& -y_1^1{y}_1^2& -y_1^2\\ & & & & ⋮& & & & \\ x_N^1& y_N^1& 1& 0& 0& 0& -x_N^1{x}_N^2& -y_N^1{x}_N^2& -x_N^1\\ 0& 0& 0& x_N^1& y_N^1& 1& -x_N^1{y}_N^2& -y_N^1{y}_N^2& -y_N^1\end{array}\right]\\ \hfill \left[\begin{array}{c}{h}_{r1}^T\\ h_{r2}^T\\ h_{r3}^T\end{array}\right]=A\widehat{h}=\left[\begin{array}{c}0\\ 0\\ 0\\ ⋮\\ ⋮\\ 0\\ 0\\ 0\end{array}\right].\end{array}$$

(4)

The minimal effective right singular vector, designated as $\widehat{h}$, can be derived through the singular value decomposition of the matrix comprising all the feature points.

$$\begin{array}{c}\hfill \widehat{h}=\underset{h}{argmin}\sum _{i=1}^N{∥a_{i}h∥}^2=\underset{h}{argmin}{∥Ah∥}^2.\end{array}$$

(5)

The value of $A\in {\mathbb{R}}^{2N\times 9}$ is obtained by vertically stacking all values of $a_i$, while the value of $\widehat{h}$ represents the lowest effective right singular vector. The local deformation matrix, ${\widehat{h}}_k$, for the k-th mesh is computed as follows:

$$\begin{array}{c}\hfill {\widehat{h}}_k=\underset{h_k}{argmin}{∥W_{k}Ah∥}^2,\\ \hfill W_k=diag\left(\left[\begin{array}{ccccc}{w}_k^1& w_k^1& w_k^2& \cdots & w_k^N\end{array}\right]\right),\end{array}$$

(6)

where $W_k^N\in {\mathbb{R}}^{2N\times 9}$. The minimum singular right vector ${\widehat{h}}_k$ is obtained by performing a singular value decomposition of $W_{k}A$.

The objective is to determine the homography matrix H for all meshes.

$$\begin{array}{c}\hfill H=\left({\widehat{h}}_1\cdots {\widehat{h}}_M\right)=\underset{H}{argmin}WA=\left(\underset{h_1}{argmin}{W}_{1}A\dots \underset{h_M}{argmin}{W}_{M}A\right),\end{array}$$

(7)

where $W=\left(W_1\dots W_M\right)$.

In our research on the multigrid twisted transform parameterization, we define feature points to be uniformly distributed over the image as grid points and then predict the displacement of these points. The weights of the feature points within each grid are calculated, and singular value decomposition (SVD) is employed to identify the homography of each grid.

3.2. Network Framework

Figure 2 provides a concise overview of the multigrid deep homography network. Images $I_r$ and $I_t$ are processed to extract semantic features using a ResNet50 [34] model that maps the image to $512\times 512$ resolution feature blocks. This is subsequently mapped into a two-channel $64\times 64$ feature stream using the Contextual Correlation Layer [23], and the input feature stream is then predicted using a regression network with $N\times 2$ displacement parameters for all feature points. The initial feature points and the predicted displacements are incorporated into Equation (4) to yield the matrix A. According to Equation (1), the weights of all the feature points in each grid are obtained. Since the difference between the grids to solve for the local distortion matrix is only the difference in the weights, we can solve for the local distortion matrix of all the grids in parallel, H. After obtaining H, we perform a multigrid distortion of the target image $I_t$ to obtain the distorted target image $I_t^r$. The red dots in $I_{rf}$ represent the initial assumption that the feature points are located on the reference image. The green dots in IA indicate the predicted locations of the feature points on the target image, as determined by the network. The yellow dots in $I_{tf}$ represent the centers of each grid, which are utilized to compute the weight of each feature point within that grid. $I_t^r$ is the distorted image of the target image.

3.3. Chunking and Reverse Distortion

Deep multigrid morphing differs from deep single-grid morphing and image warping using one coordinate transformation formula (e.g., TPS [13]) in that each pixel in deep single-grid morphing and TPS shares the same warping parameter, whereas deep multigrid morphing requires the assignment of a distinct homography to each pixel within the grid following the morphing process. As illustrated in Figure 4a, we only assign a homography to the pixel points within the grid after deformation, which correspond to the shape before distortion, as shown in Figure 4b. In the image, pixel points outside the grid region shown in Figure 4b could not be used to determine the warping parameters for warping. The determination of the precise area of each pixel block in an image and the performance of pixel interpolation operations on a single image are complicated by the uncertainty of the grid shape. Consequently, it is challenging to generate a comprehensive distorted image by distorting the target image.

To address this issue, it was observed that the monoresponse matrix H and its inverse matrix $H^{-1}$ exhibited symmetry during the projection process. This was due to the fact that the transformations applied to the monoresponse matrix and the inverse transformations using its inverse matrix were found to restore the original image to its original state. Moreover, the single response matrix preserves the parallelism of the parallel lines in the image, and the inverse matrix also exhibits this property. Consequently, the inverse of the distortion matrix for each grid can be obtained as $H^{-1}=\left(\begin{array}{ccc}{\widehat{h}}_1^{-1}& \dots & {\widehat{h}}_M^{-1}\end{array}\right)$, which represents the multigrid transformation matrix from the reference image $I_r$ to the target image $I_t$. Thereafter, the reference image is partitioned into corresponding blocks in accordance with the shape of the grid divisions, resulting in $P=\left(\begin{array}{ccc}{p}_1& \dots & p_M\end{array}\right)$. Subsequently, the distorted image blocks of the reference image from the viewpoint of the target image are obtained by applying corresponding distortions to the corresponding blocks. Ultimately, all the distorted blocks are superimposed to obtain the distorted image $I_t^r$.

$$\begin{array}{c}\hfill I_r^t=P{H}^{-1}=\sum _{i=1}^M{p}_i{{\widehat{h}}_i}^{-1}.\end{array}$$

(8)

Once the image $I_r^t$ and the target image $I_t$ have been obtained, the stitched image can be generated through the application of image fusion algorithms, such as the Average Fusion Algorithm and Graphcut Textures [26]. As illustrated in Figure 3.

3.4. Loss Function

In the context of real-world image stitching networks, current losses can be classified into two categories: content loss and shape-preserving loss. Content loss is employed to regulate the degree of overlap between the constituent image parts, with the objective of optimizing the network. However, relying on content alignment loss alone may potentially lead to the emergence of unnatural mesh distortions, such as self-intersections. It is now necessary to impose constraints by utilizing shape-keeping loss. The shape-preserving loss function correlates neighboring meshes with their surrounding environment, thereby ensuring that all meshes maintain a consistent shape. As our network does not predict the displacements of grid points, but rather the displacements of feature points, we can only solve the distortion matrix of different grids using content alignment loss and inversely warp the grids. As a result, the output image blocks obtained from the inverse warping are not connected to each other (see Figure 1a,b). To ensure continuity between the image blocks, we introduce a grid shape-keeping loss function ${\mathcal{L}}_1$.

$$\begin{array}{c}\hfill {\mathcal{L}}_1=\sum _{i=1}^{Y-1}\sum _{j=1}^{X-1}\left|m{\left(i,j\right)}_4-m{\left(i+1,j\right)}_2-{10}^{-14}\right|\\ \hfill +\left|m{\left(i,j\right)}_4-m{\left(i+1,j+1\right)}_1-{10}^{-14}\right|\\ \hfill +\left|m{\left(i+1,j+1\right)}_1-m{\left(i,j+1\right)}_3-{10}^{-14}\right|\\ \hfill +\left|m{\left(i+1,j\right)}_2-m{\left(i,j+1\right)}_3-{10}^{-14}\right|.\end{array}$$

(9)

In the context of inverse warping, $m\left(i,j\right)$ denotes the grid of row i and column j. The subscripts 1, 2, 3, and 4 denote the points on the upper left, upper right, lower left, and lower right of the grid, respectively. Y represents the number of rows of the grid, and X represents the number of columns of the grid. The inclusion of the ${10}^{-14}$ in Equation (9) serves to prevent singular value decomposition errors during training.

In order to guarantee the efficacy of image alignment effects, it is necessary to utilize the content alignment loss ${\mathcal{L}}_2$.

$$\begin{array}{c}\hfill {\mathcal{L}}_2=∥I_r•\phi \left(⊮,H\right)-\phi \left(I_t,H\right)∥\\ \hfill +∥I_t•\phi \left(⊮,H^{-1}\right)-\phi \left(I_r,H^{-1}\right)∥,\end{array}$$

(10)

where $\phi \left(A,B\right)$ represents the distortion operation B applied to image A, • denotes the pixel dot product, and $⊮$ denotes the all-ones matrix. The total network loss is $\mathcal{L}$.

$$\begin{array}{c}\hfill \mathcal{L}={\mathcal{L}}_1+{\mathcal{L}}_2.\end{array}$$

(11)

4. Experimental Results and Analysis

4.1. Data Sets and Experimental Environments

The model was trained on the UDIS-D [22] real-world dataset, which comprises 10,440 image pairs for training and 1106 image pairs for testing. The dataset encompasses a range of overlap rates, degrees of parallax, and variable scenes, including indoor, outdoor, night, dark, snow, and zoom.

The network was trained using the Adam optimizer. The learning rate was exponentially decaying, with an initial decay rate of. The batch size was set to 4, and 400 epochs were trained. The number of feature points was set to (32 + 1) × (32 + 1), the number of meshes was set to 16 × 16, and the $\sigma$, $\gamma$ were set to 1 and 0.0001, respectively. All the implementations were based on PyTorch (https://pytorch.org/), using a single GPU with an NVIDIA RTX 3060 Ti (NVIDIA Corporation, Santa Clara, CA, USA).

4.2. A Comparative Analysis of Homography Estimation Techniques

The network warping results are compared with those obtained using SIFT [5] + RANSAC [24] (a pipeline of AutoStitch [1]), SPW [34], AANAP [17], APAP [10], LCP [35], DAMGDH [23] and UDIS [22]. The image alignment performance is evaluated using the PSNR and SSIM metrics on the real-world UDIS-D dataset. A total of 1106 samples from the test set were utilized for evaluation purposes. Higher metrics indicate superior performance of the image in question. The results of the comparison are presented in

Table 1. Comparison of PSNR and SSIM metrics for various distortion methods on the UDIS-D dataset. The best ones are marked in red, and the second best ones are marked in blue.

	PSNR				SSIM
Method	Easy	Moderate	Hard	Average	Easy	Moderate	Hard	Average
$I_{3\times 3}$	15.87	12.76	10.68	12.86	0.530	0.286	0.146	0.303
AANAP [17]	27.58	23.85	19.70	23.31	0.896	0.823	0.659	0.779
APAP [10]	28.36	24.40	18.39	23.17	0.912	0.837	0.642	0.781
SPW [35]	26.98	22.67	16.77	21.60	0.880	0.758	0.490	0.687
LCP [36]	26.94	22.63	19.31	22.59	0.878	0.764	0.610	0.736
UDIS [22]	27.84	23.95	20.70	23.80	0.902	0.830	0.684	0.793
SIFT [5] + RANSAC [24]	28.75	24.08	18.55	23.27	0.916	0.833	0.636	0.779
DAMGDH [23]	29.52	25.24	21.20	24.89	0.923	0.860	0.708	0.817
Ours	29.56	25.29	21.20	24.92	0.947	0.869	0.698	0.824

. The high and low metrics are used to categorize the metrics into three levels for comparison. The “Easy” category represents the average of the metrics for images with the best splicing performance, the “Moderate” category represents the average of the metrics for images with medium splicing performance, and the “Hard” category represents the average of the metrics for images with poor splicing performance. The “Average” category represents the average of all the image metrics. Furthermore, two images without distortion are employed as a reference. In the process of comparison, some images, due to the number of matching feature points, are not sufficient to allow for the traditional method of alignment to be applied. In such cases, the original image is used instead of the metrics calculation. The experimental results demonstrate that our method has a greater complexity than the PRNR and SSIM metrics, but is superior to other methods in this regard. This indicates that the robustness of our method is superior to that of the other methods. The parameters used in the traditional APAP algorithm and those employed in our method are identical, which suggests that our method of image warping with deep learning to predict feature points is more effective than the traditional method of image warping based on feature extraction.

As illustrated in Figure 5, the blue channel in the reference image and the red channel in the deformed target image were set to zero in order to fuse the reference image and the target image. The non-overlapping region is represented by orange, while the unaligned part in the overlapping region is highlighted with different colors. To facilitate comparison, please zoom in on the red box position in the image. It is evident that the alignment accuracy of our method is superior to that of other methods.

4.3. Comparison Experiment of Spliced Images

4.3.1. A Comparative Analysis of Stitched Images from the UDIS-D Dataset

A comparative analysis of stitched images from the UDIS-D dataset As illustrated in Figure 6, a comparison of our splicing results with those of other methods in the UDIS-D dataset reveals that the traditional methods SPW [34], LCP [36] and APAP [10] exhibit more pronounced artifacts in splicing, while the deep-learning method UDIS [22] still displays some artifacts after reconstruction. In contrast, our method produces both weak texture and large parallax in the case of superior splicing results, thereby demonstrating its superior performance.

4.3.2. A Comparative Analysis of Spliced Images Derived from Conventional Datasets

A comparison is presented of the traditional datasets ANAP [17], APAP [10], DHW [18], and LCP [36]. To obtain the spliced image for our network, we employed a square cropping of the input image, subsequently morphing it to a height and width of 512 pixels. In order to obtain a superior spliced image and enhanced local alignment of the model, the input image was trained for 50 iterations. As illustrated in Figure 7, a comparison is presented with the APAP [10], LCP [36], SPW [34], and UDIS [22] methods, focusing on the overlapping region. The results demonstrate that the splicing effect of our network is comparable to, and in some cases superior to, that of other methods.

4.4. A Comparison of Feature Point Matching in Low-Light Environments

As illustrated in Figure 8, the results of the network’s predicted feature points are presented, with the red dots indicating the hypothesized grid feature points and the green dots indicating the locations of the predicted corresponding feature points. It should be noted that some feature points are located beyond the image space, specifically the green feature points that are situated within the black region. A comparison is made between the traditional feature point matching method SIFT+RANSAC and the proposed approach in low light conditions. Figure 9a,b show a selection of matched feature points. The upper image represents the results of matching by the traditional method, while the lower image shows the results of prediction by the proposed network. It can be observed that the traditional method is prone to matching errors in low light conditions, and these are indicated by the red circles in the figure. The feature points are marked with red circles in the figure. In contrast, our method predicts the feature points inside and outside the image boundary with greater accuracy.

4.5. Ablation Experiment

A series of ablation studies was conducted to assess the efficacy of different loss functions and the potential benefits of incorporating a multigrid deformation module.

A comparison was made between the generated spliced images produced with and without the use of shape-preserving loss. As illustrated in Figure 1b represents the spliced image without the incorporation of shape-preserving loss. This image is comprised of numerous image blocks, and notable black cracks are evident between the blocks. Figure 1d is the resulting image with the introduction of shape-preserving loss. It is evident that the continuity between image blocks is well-maintained after the addition of shape-preserving loss. A series of ablation experiments were conducted to ascertain the impact of including the APAP multigrid deformation module at varying numbers of feature points. As the number of meshes and feature points increases, the video memory required for computation also increases linearly. The mesh is thus divided into a 16 × 16 grid. In the absence of the multigrid deformation module, a global matrix is employed for aligning the feature points, thereby distorting the image. In this study, we use V1 to denote the case without the addition of the multigrid deformation module and V2 to denote the case with the addition of the multigrid deformation module. As demonstrated in

Table 2. A comparison of the average PSNR and SSIM metrics under the UDIS dataset in the presence and absence of the multigrid deformation module for different numbers of feature points is presented.

PSNR
Number of Feature Points	V1	V2
8 × 8	23.69	24.83
16 × 16	23.74	24.85
32 × 32	23.80	24.92
SSIM
Number of feature points	V1	V2
8 × 8	0.767	0.818
16 × 16	0.769	0.822
32 × 32	0.774	0.824

, the PSNR and SSIM metrics exhibit a notable improvement with the incorporation of the multigrid deformation module. Furthermore, the degree of alignment of the overlapping portions of the image increases gradually with the increase in the number of feature points.

5. Discussion

This chapter examines the scalability of the proposed method in terms of alignment performance, processing time, space complexity, the handling of larger datasets, and the ability to accommodate different resolutions. The pseudo-code Algorithm 1 of our network model is also provided at the conclusion of this chapter.

With regard to alignment performance, the method employs a local homography matrix solution based on predicted feature points. While it does enhance grid-based image alignment, it relies on a similar set of feature points to determine the homography, with significant constraints between the local homographies. Consequently, it is unable to distort the image and align in a more flexible manner, as illustrated in Figure 1a demonstrates that even when no grid constraint loss is applied, the image is naturally distorted to a recognizable degree. Consequently, the image content remains misaligned, even in the presence of significant parallax.

In the future, we intend to address this issue in two ways. One approach is to reduce the constraints between the grids, thereby improving their alignment with the image. Conversely, an alternative approach would be to ascertain whether there exists a superior methodology for partitioning the local distortion space of the image. This could entail the division of disparate distorted image blocks according to their respective objects, with the objective of addressing the issue of large parallax.

In the context of larger datasets, the processing time required is necessarily longer. As the training of the model and the acquisition of the spliced images are conducted as two separate processes, we evaluated both in terms of the time required for training and the time required for acquiring the spliced images with the trained model.

Table 3. The time in seconds required to complete a training batch with varying numbers of feature points and grids. The testing was conducted using a NVIDIA RTX 3060 Ti graphics processing unit (GPU).

Running Time		Number of Feature Points
		8 × 8	16 × 16	32 × 32
Number of grids	1 × 1	0.057	0.057	0.063
	2 × 2	0.060	0.061	0.069
	4 × 4	0.067	0.069	0.076
	8 × 8	0.101	0.106	0.128
	16 × 16	0.210	0.235	0.261

illustrates the time required for a single training batch with varying numbers of feature points and grids. It can be observed that the time necessary for a single training session increases gradually with the increase in the number of feature points and grids. Notably, the increase in time associated with an increase in the number of grids is more pronounced.

Table 4. The table illustrates the time (in seconds) necessary to obtain the stitched image by chunking the reverse distortion for varying numbers of meshes. The testing was conducted using a NVIDIA RTX 3060 Ti graphics processing unit (GPU).

Number of grids	1 × 1	2 × 2	4 × 4	8 × 8	16 × 6
running time	0.079	0.094	0.162	0.419	1.214

illustrates the time required to obtain a stitched image through chunked reverse warping using the trained model with varying numbers of grids. It can be observed that due to the implementation of warping for each image chunk, the number of warping operations increases with an increase in the number of grids, resulting in a prolonged processing time.

Accordingly, for larger datasets, a smaller number of grids and a reduced number of features may be selected, thereby enabling the stitching results to be obtained in a more expeditious manner.

In terms of space complexity, this study examines the impact of the number of feature points and the number of meshes on the space complexity of the multigrid twisted parameterization module. Let us assume that the number of feature points is M and the number of grids is N. The computational matrix A requires a space of $O\left(M\right)$. In the weight computation stage, the requisite space for the weights of each grid is $O\left(M\right)$, and the total space required for all N grids is $O\left(NM\right)$. The matrix $WA$ requires a space of $O\left(NM\right)$. The greatest space complexity of the three matrices generated by singular value decomposition SVD is $O\left(N{M}^2\right)$. It is evident that the number of feature points has a more pronounced impact on the computational space resource requirement. Furthermore, an investigation is conducted to ascertain the influence of the number of grids on the space complexity of the chunked inverse warping module. Let us suppose that the image in question has dimensions $H\times W$ and that there are N grids. The requisite space complexity for the implementation of chunked reverse warping is $O\left(NHM\right)$. Given that the dimensions of the stitched image are typically greater than those of the input image, the actual space complexity may be somewhat higher than the theoretical value.

Accordingly, in the multigrid distortion parameterization module, the computation of the distortion matrix of a mesh by reducing the number of feature points can effectively reduce the computational space and resource occupation. In the future, an additional optimization may be achieved by solving the distortion matrix of each mesh independently using a small number of feature points. In the case of the chunked inverse warping module, the generation of the stitched image at a significantly faster rate is possible if the warping matrix of each pixel point can be accurately determined at the time of image generation. This may result in an improvement of up to N-fold in the speed of image generation.

At present, our network is only capable of image stitching at a resolution of $512\times 512$ pixels. However, in practical applications, it is often necessary to handle images of varying resolutions. As our network is based on feature point displacement prediction, we can scale the image to $512\times 512$ pixels, predict the displacement of image feature points, and restore these feature points proportionally. By employing these reduced feature points, the distortion matrix at the original resolution can be calculated, thereby enabling the task of image stitching at different resolutions to be accomplished.

Algorithm 1: networked algorithmic process

6. Conclusions

The article presents an innovative approach to image alignment, proposing a novel method for predicting feature point displacements and combining them with traditional feature point-based warping techniques to achieve a more accurate picture alignment process. By assuming a uniform distribution of feature points and using a deep learning network to predict their displacements, we successfully simulate the position of feature points of one image in the viewpoint of another image to achieve robust feature point extraction. Furthermore, we parameterize the APAP algorithm to achieve deep learning APAP multigrid image stitching, thereby obtaining a more accurate alignment of its results. Furthermore, we introduce a post-processing method of multigrid inverse warping and mesh shape preservation loss for the generation of crack-free panoramic images.

The experimental results demonstrate that the proposed method achieves a significant improvement in the image alignment task, indicating potential applications in real-world applications. The combination of deep learning and traditional algorithms enhances the accuracy of the traditional graphic alignment method, thereby providing substantial support and guidance for the resolution of image processing issues in practical applications.