Image Registration Algorithm for Stamping Process Monitoring Based on Improved Unsupervised Homography Estimation

Featured Application

This paper is applied in the industrial monitoring of the stamping process, where vibrations are common, necessitating the use of image registration and homography estimation methods to align template images with test images. By employing machine-vision and image-processing techniques, the process is monitored in real-time to detect any anomalies, ultimately aiming to protect the stamping molds.

Abstract

Homography estimation is a crucial task in aligning template images with target images in stamping monitoring systems. To enhance the robustness and accuracy of homography estimation against random vibrations and lighting variations in stamping environments, this paper proposes an improved unsupervised homography estimation model. The model takes as input the channel-stacked template and target images and outputs the estimated homography matrix. First, a specialized deformable convolution module and Group Normalization (GN) layer are introduced to expand the receptive field and enhance the model’s ability to learn rotational invariance when processing large, high-resolution images. Next, a multi-scale, multi-stage unsupervised homography estimation network structure is constructed to improve the accuracy of homography estimation by refining the estimation through multiple stages, thereby enhancing the model’s resistance to scale variations. Finally, stamping monitoring image data is incorporated into the training through data fusion, with data augmentation techniques applied to randomly introduce various levels of perturbation, brightness, contrast, and filtering to improve the model’s robustness to complex changes in the stamping environment, making it more suitable for monitoring applications in this specific industrial context. Compared to traditional methods, this approach provides better homography matrix estimation when handling images with low texture, significant lighting variations, or large viewpoint changes. Compared to other deep-learning-based homography estimation methods, it reduces estimation errors and performs better on stamping monitoring images, while also offering broader applicability.

1. Introduction

With the advancement of Industry 4.0, machine-vision technology has replaced traditional manual inspection methods, enabling real-time monitoring of the stamping process to protect molds [1]. By capturing images of the metal sheet before and after the mold closes, the process is analyzed for any abnormalities. If an anomaly is detected, an emergency stop signal is issued to halt the stamping equipment immediately, thereby protecting the molds, reducing economic losses, and enhancing production efficiency. However, the actual stamping process may be affected by factors such as the impact force of the mold opening and closing or unstable feed rates from the feeder, causing random vibrations in the metal sheets. This results in random offsets in the images captured each time, necessitating image registration of the template and real-time detection images. The difference between these two images is then used to determine if there are any abnormalities in the process. Therefore, image registration algorithms are indispensable in the application of monitoring the stamping process.

Image registration refers to the process of transforming different images of the same object from different spatial coordinate systems to the same coordinate system through a certain mapping relationship. In this process, homography estimation plays an important role. Traditional homography estimation methods are divided into direct methods and feature-based matching methods [2]. Direct methods like the Lucas–Kanade algorithm initialize homography parameters and use gradient descent to minimize errors [3,4]. Other methods include optical flow and image gradient-based techniques [5,6], which are sensitive to lighting changes and require good initial values to prevent local optima. The optimization is computationally intensive, complicating their use in high-resolution and complex scenes. The second method involves feature-based matching, which starts with feature detection algorithms like SIFT, SURF, and ORB to extract key points from images [7,8,9]. Each detected key point is assigned a descriptor that characterizes its surrounding image area. These feature descriptors are then compared between two images using matchers like FLANN or Brute-Force to find matching pairs [10]. Finally, robust estimation methods such as RANSAC or LMedS are used to compute the optimal homography matrix [11,12]. While these methods generally perform faster than direct methods, they heavily rely on the detection and matching of key points. In areas with low texture, sufficient key points may not be detectable, and significant changes in perspective or lighting between images can lead to incorrect key point matches, directly impacting the accuracy of homography estimation.

The stamping surveillance environment is influenced by external factors like lighting changes and equipment instability, as well as human factors such as camera angle adjustments. These factors can significantly alter the comparison between real-time detection and template images, impacting the accuracy of feature-point detection and homography estimation. Leveraging the success of deep convolutional neural networks in computer vision, deep-learning-based homography estimation methods can effectively tackle these robustness issues and the reliance on feature points seen in traditional approaches [13]. Detone et al. introduced HomographyNet, a simple convolutional neural network trained end-to-end to directly regress the parameters of the homography matrix [14]. It outperforms traditional methods in complex scenarios such as low-texture environments, but the model is trained on synthetic datasets, leading to poor inference capabilities on real images. Real-world image labeling is also highly costly. Ty Nguyen et al. [15] were the first to propose an unsupervised homography estimation method. Building on a similar network structure as previous approaches, they enhanced the robustness of homography estimation to lighting variations by learning the homography relationship through minimizing the pixel intensity difference between two images. This method eliminates the need for labeled data. However, it performs poorly on images with significant viewpoint differences and lacks scale and rotation invariance. In recent years, there has been increasing research on unsupervised homography estimation [16,17,18,19,20]. Unsupervised learning methods eliminate the complex process of labeling training data, reducing the impact of human bias. These methods are better suited to adapt to complex and changing environments and demonstrate improved generalization ability when handling variations in real-world conditions, such as different lighting conditions, motion blur, shifts, and rotations. Zang J. et al. [16] proposed an unsupervised learning model that learns feature representations for homography estimation and image comparison. This model calculates loss in the feature space rather than in the pixel space, achieving robustness to varying lighting conditions. Koguciuk et al. [17] extended this work by adding perceptual loss computation [21], further enhancing the robustness of deep homography estimation methods to lighting changes. However, both of these new unsupervised homography estimation methods are not applicable to scenes with small image overlapping regions and low similarity. Liu S. et al. [18] proposed a new unsupervised learning framework that improves homography estimation by learning motion bases. This approach can estimate global homography transformations and handle local homography transformations at a finer granularity, thus improving the accuracy of homography estimation. However, due to the complex feature extraction and learning processes involved, this method requires significant computational resources and time, making it unsuitable for industrial applications.

To address the issues mentioned above, this paper proposes an improved unsupervised homography estimation model based on existing models, tailored for the specific industrial context of stamping process monitoring. By incorporating a special deformation convolution module and a GN normalization layer into the backbone network, the model enhances its rotational invariance when handling large-scale, high-resolution images. A multi-scale, multi-stage network architecture is developed, utilizing unsupervised training through minimizing pixel-level photometric loss at each stage, which progressively refines the homography transformations between images from coarse to fine, improving the accuracy of homography estimation and enhancing the model’s learning ability for scale invariance. To make the model suitable for the specific industrial application of the stamping process and improve its robustness against complex environmental changes such as lighting variations and vibration disturbances, the training dataset integrates stamping surveillance image data with random perturbations, lighting, contrast, and blur for data augmentation, addressing the issue of dataset scarcity in specific industrial applications of deep learning.

The remainder of this paper is organized as follows. Section 2 first describes the network structure of the unsupervised single-response estimation model, improves on it by introducing a special deformation convolution module and a GN normalization layer, and constructs a multi-scale, multi-stage overall network structure and establishes a multi-stage coupling relation, minimizing the luminosity loss function. Comparison and ablation experiments are carried out in Section 3 to analyze the effectiveness of the method of this paper, and conclusions are summarized in Section 4.

2. Materials and Methods

2.1. Unsupervised Homography Estimation Model

The unsupervised homography estimation method proposed by Nguyen et al. [15] consists of four main parts: a four-point parameterization homography estimation layer, a tensor-based direct linear transformation layer, a spatial transformation layer, and a loss function computation layer. The overall network structure of this method is illustrated in Figure 1.

2.1.1. Four-Point Parameterization Homography Estimation Layer

This layer is the HomographyNet network model, which serves as the backbone network. It is a regression network similar to the VGG architecture. As shown in Figure 2, the network model includes 8 convolutional layers, 4 max pooling layers, 1 dropout layer, and 2 fully connected layers. The model input consists of two stacked grayscale images. After processing through convolutional layers and fully connected layers, the final output is a 4-point parameterized homography matrix, $H_{4\mathit{points}}$, which is equivalent to a 3 × 3 form of the homography matrix, $H$.

The essence of planar homography estimation is to compute the linear mapping between pixels of two images to provide a non-singular linear relationship between points in the two images. The homography matrix has 8 degrees of freedom, so theoretically, only 4 pairs of matching points are needed to estimate the result. The HomographyNet network model discards the traditional 3 × 3 homography matrix’s rotation, translation, scaling, and other component errors that are difficult to reflect in the loss function, which can affect the training of deep neural network models. Instead, it uses a 4-point parameterized form to represent the homography matrix. The principle is as follows:

Assuming points ${x=(u, v)}^T$ and $x^{\prime }= {\left(u^{\prime }{, v}^{\prime }\right)}^T$ are a pair of 2D coordinates with a mapping relationship in the transformed image, represented in homogeneous coordinates as ${x=(u, v, 1)}^T$ and $x^{\prime }={\left(u^{\prime }{, v}^{\prime }, 1\right)}^T$, the mapping relationship between the two points using the 3 × 3 homography matrix, $H$, is expressed as:

$$\left[\begin{array}{c}{u}^{\prime }\\ v^{\prime }\\ 1\end{array}\right]=\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& h_{33}\end{array}\right]\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]$$

(1)

As long as there are four pairs of matching points, $\left(u_i, v_i\right)\stackrel{\mathit{match}}{\to }\left(u_i^{\prime },v_i^{\prime }\right)$, the unique solution for $H$ can be calculated. Similarly, as long as there are four pairs of $\left(∆u_i, ∆v_i\right)$, the unique solution for $H$ can also be calculated, where $∆u_i=u_i^{\prime } - u_i$ and $∆v_i=v_i^{\prime }-v_i$ represent the offsets of the horizontal and vertical coordinates of these four pairs of matching points. Therefore, the 4-point parameterized matrix, $H_{4\mathit{points}}$, is used to replace the 3 × 3 homography matrix, $H$:

$$H_{4\mathit{points}}=\left[\begin{array}{c}\begin{array}{cc}∆u_1& ∆v_1\\ ∆u_2& ∆v_2\end{array}\\ \begin{array}{cc}∆u_3& ∆v_3\\ ∆u_4& ∆v_4\end{array}\end{array}\right] ~H=\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& h_{33}\end{array}\right]$$

(2)

The mapping relationship between the two points ${x=(u, v, 1)}^T$ and $x^{\prime }={\left(u^{\prime }, v^{\prime }, 1\right)}^T$ is expressed using $H_{4\mathit{points}}$ as:

$$x^{\prime }{=H}_{4\mathit{points}}x$$

(3)

2.1.2. Tensor Direct Linear Transformation Layer

This layer essentially applies the Direct Linear Transformation (DLT) algorithm to the tensors in the deep-learning network. Its function is to convert the 4-point parameterized matrix, $H_{4\mathit{points}}$, obtained in the previous stage into a 3 × 3 homography matrix, $H$, using the DLT algorithm. Additionally, it is differentiable to ensure that backpropagation can occur during training. The input of this layer is the coordinates of the four corner points before and after the transformation, and the output is the 3 × 3 homography matrix, $H$. Suppose $\stackrel{\sim }{H}$ is the homography matrix of the corresponding corner points $x_i\to x_i^{\prime }$. According to the homography relationship, $x_i^{\prime }~\stackrel{\sim }{H}{x}_i$, which is equivalent to $x_i^{\prime }\times \stackrel{\sim }{H}{x}_i=0$. Let $h^{\mathit{jT}}$ represent the j-th row of $\stackrel{\sim }{H}$, and let $h^j$ represent the column vector of $h^{\mathit{jT}}$. Then, $\stackrel{\sim }{H}{x}_i$ can be expressed as:

$$\stackrel{\sim }{H}{x}_i=\left[\begin{array}{c}{h}^{1T}{x}_i\\ h^{2T}{x}_i\\ h^{3T}{x}_i\end{array}\right]=\left[\begin{array}{c}{x_i}^T{h}^1\\ {x_i}^T{h}^2\\ {x_i}^T{h}^3\end{array}\right]$$

(4)

Substituting $x_i^{\prime }={\left(u_i^{\prime },v_i^{\prime },1\right)}^T$ into $x_i^{\prime }\times \stackrel{\sim }{H}{x}_i=0$, we have:

$$\left[\begin{array}{c}{u}_i^{\prime }\\ v_i^{\prime }\\ 1\end{array}\right]\times \left[\begin{array}{c}{x}_i^T{h}^1\\ x_i^T{h}^2\\ x_i^T{h}^3\end{array}\right]=\left[\begin{array}{c}{v}_i^{\prime }{x}_i^T{h}^3-x_i^T{h}^2\\ x_i^T{h}^1-u_i^{\prime }{x}_i^T{h}^3\\ u_i^{\prime }{x}_i^T{h}^2-v_i^{\prime }{x}_i^T{h}^2\end{array}\right]=0$$

(5)

We can arrange this as follows:

$$\left[\begin{array}{ccc}{0}_{3\times 1}^T& -x_i^T& v_i^{\prime }{x}_i^T\\ x_i^T& 0_{3\times 1}^T& -u_i^{\prime }{x}_i^T\\ -v_i^{\prime }{x}_i^T& u_i^{\prime }{x}_i^T& 0_{3\times 1}^T\end{array}\right]\left[\begin{array}{c}{h}^1\\ h^2\\ h^3\end{array}\right]=0$$

(6)

Therefore, given the position changes of the four corner points, substituting them into the above equation allows us to directly solve the system of equations for $h^j (j=1, 2, 3)$, thereby obtaining the 3 × 3 homography matrix.

2.1.3. Spatial Transformation Layer

This layer applies the 3 × 3 homography matrix, $H$, obtained in the previous stage, to each pixel coordinate, $x_i$, of the target image, $I_A$, resulting in the corresponding coordinates, ${H(x}_i)$, after the homographic transformation. These transformed coordinates are necessary for the unsupervised method to compute the pixel intensity loss function between the two images. This layer comprises three parts: the normalization inverse computation of homography estimation, the parametric sampling grid generator, and the differential image sampling. Additionally, it must be differentiable to allow the error gradient to backpropagate for network training.

• Normalized Inverse Computation of Homography Estimation

The purpose of the spatial transformation layer is to align the target image with the reference image using the homography matrix. Let $I_A$ and $I_B$ be a pair of images with a homography relationship, where $I_A$ is the target image to be transformed and $I_B$ is the reference image. First, the pixel coordinates of input images $I_A$ and $I_B$ are normalized to the range [−1, 1]. Then, the inverse homography matrix, ${\stackrel{\sim }{H}}_{\mathit{inv}}$, is calculated.

$${\stackrel{\sim }{H}}_{\mathit{inv}}=M^{-1}{\stackrel{\sim }{H}}^{-1}M$$

(7)

where $\mathrm{M}=\left[\begin{array}{ccc}{\displaystyle \frac{{\mathrm{W}}^{\prime }}{2}}& 0& {\displaystyle \frac{{\mathrm{W}}^{\prime }}{2}}\\ 0& {\displaystyle \frac{{\mathrm{H}}^{\prime }}{2}}& {\displaystyle \frac{{\mathrm{H}}^{\prime }}{2}}\\ 0& 0& 1\end{array}\right]$ and ${\mathrm{W}}^{\prime }$ and ${\mathrm{H}}^{\prime }$ are the width and height of image ${\mathrm{I}}_{\mathrm{B}}$.

• Parametric Sampling Grid Generator

Create a grid, $G=\left\{G_i\right\}$, of the same size as $I_B$ for sampling, where each grid element, $G_i=\left(u_i^{\prime },v_i^{\prime }\right)$, corresponds to the pixel coordinates of image $I_B$. Apply the previously computed normalized inverse homography matrix, ${\stackrel{\sim }{H}}_{\mathit{inv}}$, to the grid, $G$, to obtain the pixel grid of predicted image $I_A$.

$$\left[\begin{array}{c}{\mathrm{u}}_{\mathrm{i}}\\ {\mathrm{v}}_{\mathrm{i}}\\ 1\end{array}\right]={\stackrel{\sim }{\mathrm{H}}}_{\mathrm{inv}}\left({\mathrm{G}}_{\mathrm{i}}\right)={\stackrel{\sim }{\mathrm{H}}}_{\mathrm{inv}}\left[\begin{array}{c}{\mathrm{u}}_{\mathrm{i}}^{\prime }\\ {\mathrm{v}}_{\mathrm{i}}^{\prime }\\ 1\end{array}\right]$$

(8)

• Differential Image Sampling

Based on the sampled points, ${\stackrel{\sim }{H}}_{\mathit{inv}}\left(G_i\right)$, obtained in the previous step, generate a transformed image, $V$ with height $h^{\prime }$, width $w^{\prime }$, and $C$ channels, where $V(x_i)=I_A(\stackrel{\sim }{H}(x_i))$. The image, $V$, is defined as:

$$V_i^C={{\displaystyle \sum }}_n^h{{\displaystyle \sum }}_m^w{I}_{nm}^{c}k\left(u_i-m;{\varnothing }_u\right)k\left(v_i-n;{\varnothing }_v\right) \forall i\in [1\dots h^{\prime }{w}^{\prime }], \forall c\in [1\dots c]$$

(9)

where $h$ and $w$ are the height and width of input image $I_A$, k(·) denotes the sampling kernel, ${\varnothing }_u$ and ${\varnothing }_v$ are parameters of the sampling kernel, $I_{\mathit{nm}}^c$ is the pixel value of the input image at channel $c$ and coordinates $(n, m)$, and $V_i^C$ is the pixel value of the output image at channel $c$ and coordinates (u_i, v_i). Due to the potential non-integer values resulting from image transformation, bilinear interpolation is employed to obtain integer values. Additionally, to ensure that the loss function can be backpropagated during network training, it needs to be differentiable. Therefore, the gradient of the image after bilinear interpolation is defined as:

$${\displaystyle \frac{\partial V_i^C}{\partial I_{\mathrm{nm}}^c}}={\displaystyle \sum }_n^h{\displaystyle \sum }_m^w\mathit{max}(0,1-{|u}_i-m\left|\right)\mathit{max}(0,1-\left|v_i-n\right|)$$

(10)

2.1.4. Loss Calculation Layer

The spatial transformation layer warps the initial image patch, $P_A$, based on the estimated 3 × 3 homography matrix obtained earlier, resulting in the predicted image patch $P_B^{\prime }$. Finally, the photometric loss, $L_{\mathit{PW}}$, is computed pixel-wise between $I_B^{\prime }$ and $I_B$ as the loss parameter. This loss is then backpropagated to HomographyNet, with the assumption that the network’s output, the 4-point parameterized homography matrix estimate $H_{4\mathit{points}}$, should minimize the photometric loss. The formula for $L_{\mathit{PW}}$ is as follows:

$$L_{\mathit{PW}}={\displaystyle \frac{1}{x_i}}{\displaystyle \sum }_{x_i}|I_A\left(\stackrel{\sim }{H}(x_i)\right)-I_B\left(x_i\right)|$$

(11)

2.2. Improved Unsupervised Homography Estimation Model

During the stamping process, the opening and closing of the mold causes random vibrations to the metal sheet, and the complexity of the on-site environment also changes. These factors lead to random shifts in the sheet in the real-time collected images. Therefore, when estimating the homography matrix for image registration, the model is required to have scale and rotation invariance comparable to traditional methods, as well as robustness to lighting changes and noise. The unsupervised homography estimation method mentioned earlier has shown improved performance under lighting changes compared to the supervised method in [14], but it still lacks resistance to deformations and position changes in real environments, making it unsuitable for the scenarios discussed in this paper. Additionally, current supervised and unsupervised homography estimation models are trained on synthetic datasets, which are far removed from real-world conditions.

To address the aforementioned issues, this paper proposes an improved unsupervised homography estimation model that introduces a special deformable convolution module. This module expands the receptive field while enhancing the model’s ability to learn rotational invariance during homography estimation. A GN normalization layer is added after each convolution to improve the model’s stability when training with large-resolution images in small batches. A multi-scale image pyramid is constructed from the input image, allowing for homography estimation at different scales and improving the model’s ability to learn scale invariance. Additionally, a real dataset of stamping process images collected with an industrial camera is created. The dataset is augmented by randomly adding various disturbances, lighting conditions, contrast, and blur levels to enhance the model’s ability to estimate homography for sheet metal shifts in a vibrating stamping environment.

2.2.1. Improved HomographyNet Model for Enhanced Rotational Invariance

The task of unsupervised homography estimation is a pixel-level task, where pixels outside the receptive field do not affect the network output. To ensure that important information is not ignored during decision-making and to better handle images with significant perspective changes, it is generally considered that a larger receptive field is better. Deepening the network can increase the receptive field, but when the network depth reaches a certain level, continuing to add convolutional layers increases computational load and risks network degradation [22,23]. Pooling operations can also increase the receptive field but at the cost of reduced spatial resolution. Dilated convolution is another method to increase the receptive field, as it can expand the receptive field without losing spatial resolution.

This paper introduces a special deformable convolution module [24], which, like dilated convolution, can expand the receptive field without reducing resolution. Additionally, it enhances the network’s ability to extract deformations, thereby improving rotational invariance in homography estimation. As shown in Figure 3, this module differs from regular convolution and dilated convolution, demonstrating its ability to learn rotational characteristics. Introducing the special deformable convolution too early can result in the loss of local information, while introducing it too late makes it difficult for the network to learn rotational invariance. This paper finds that the best approach is to introduce the special deformable convolution module to replace regular convolution right after the first downsampling, enhancing the backbone network’s ability to learn rotational invariance in homography estimation tasks.

In addition, the parameters of each layer in the convolutional neural network change continuously during training. These parameter changes alter the probability distribution of input data for the next layer, necessitating weight initialization and reduced learning rates. Therefore, this paper augments the original network by adding GN normalization layers after each convolutional layer. Unlike Batch Normalization (BN), which depends on batch size, GN normalizes by computing the mean and variance across channels for each sample group, reducing dependency on batch size. This allows the model to maintain stable training, even with smaller batches, making it suitable for training with irregular input sizes and high-resolution images, which are more applicable to the training scenarios in this paper. The improved network with the introduction of the special deformable convolution module and GN normalization is depicted in Figure 4.

2.2.2. Unsupervised Multi-Stage Homography Estimation Model for Improved Scale Invariance

Previous studies have shown that using a multi-stage approach to progressively estimate homography can effectively handle large variations in viewpoint in input images [25,26,27]. Therefore, building on the improved HomographyNet introduced in Section 2.2.1, this paper proposes an unsupervised multi-scale, multi-stage homography estimation method. Initially, the input image pairs are processed into a three-level, down-sampled image pyramid. Subsequently, at different stages of the method, images of varying resolutions are used as inputs, starting with high-resolution images. Local homography transformations are estimated on the high-resolution inputs, while global homography transformations are estimated on the lower-resolution inputs. The coarse-to-fine integration refines the relationships between the homography transformations across progressively scaled images, thereby enhancing the homography estimation network’s ability to learn scale invariance.

The overall network architecture of the proposed method in this paper is illustrated in Figure 5. Template image $I_A$ and target image $I_B$ are inputted in a channel-stacked manner. Image patches of the same size are extracted at corresponding positions for downsampling, constructing a three-level image pyramid with varying resolutions. The resolution of each subsequent level in the pyramid is half of the previous one, with the resolutions of the three pyramid levels being 512 × 512, 256 × 256, and 128 × 128, respectively. These three pyramid levels serve as inputs for different stages of the network. The overall network model consists of three stages, each sharing the same network structure. Each stage includes the Improved HomographyNet layer, the Tensor DLT layer, and the Spatial Transformation layer. The latter two layers draw inspiration from the work described in [15], as detailed in Section 2.1.2 and Section 2.1.3, and therefore will not be elaborated on here.

In the first stage, image patches of size 512 × 512 are inputted into the homography estimation network comprising the Improved HomographyNet layer, the Tensor DLT layer, and the Spatial Transformation layer. This process yields homography matrix $H_1$, which transforms image $P_B^1$ to $P_A^1$, while retaining the offsets ${∆}_1$ corresponding to the coordinates of the four corner points. To minimize the overall network’s training error, and considering the coupling between stages, the input image resolution in the second stage is half of the previous stage. Therefore, the coordinate offsets predicted in the first stage are scaled down by a factor of 2: ${∆}_1^{\prime }={∆}_1/2$. Subsequently, $P_A^2$ is transformed by the inverse homography transformation, $H_{1}S$, to obtain ${\overline{P}}_A^2$, where $S$ is the scaling factor of 1/2. Then, ${\overline{P}}_A^2$ and $P_B^2$ are inputted into the second-stage homography estimation network, following a process similar to the first stage, resulting in $H_2$ and ${∆}_2$. The third stage follows the same procedure to obtain $H_3$ and ${∆}_3$. The final coordinate offsets in the four-point parameterized form corresponding to the entire homography matrix are:

$$∆=\left({\displaystyle \frac{{∆}_1}{2}}+{∆}_2\right)/2+{∆}_3$$

(12)

The homography estimation networks in the first to third stages estimate homography transformations at different scales based on input images of varying resolutions, which helps improve the network’s ability to learn scale invariance.

The unsupervised multi-scale homography estimation network proposed in this paper trains the network in an unsupervised manner by minimizing the pixel differences between images at each stage. The loss function includes three branches:

$$\mathrm{L}={\mathsf{\alpha }}_1\parallel \mathrm{W}\left({\mathrm{P}}_{\mathrm{B}}^1, {\mathrm{H}}_1\right)-{{\mathrm{P}}_{\mathrm{A}}^1\parallel }_1+{\mathsf{\alpha }}_2\parallel \mathrm{W}\left({\mathrm{P}}_{\mathrm{B}}^2, {\mathrm{H}}_2\right)-{{\overline{\mathrm{P}}}_{\mathrm{A}}^2\parallel }_1+{\mathsf{\alpha }}_3\parallel \mathrm{W}\left({\mathrm{P}}_{\mathrm{B}}^3, {\mathrm{H}}_3\right)-{{\overline{\mathrm{P}}}_{\mathrm{A}}^3\parallel }_1$$

(13)

where $H_1$, $H_2$, and $H_3$ represent the homography matrices obtained in the three stages; $W\left(·\right)$ denotes the spatial transformation operation applying the homography matrix to the image; and ${\alpha }_1$, ${\alpha }_2$, and ${\alpha }_3$ are the corresponding weight coefficients.

2.2.3. Data Augmentation Techniques Applied to Stamping Monitoring Images

There are no publicly available datasets containing real-world images for supervised and unsupervised homography estimation tasks. Typically, the COCO dataset is used to generate synthetic data for training. Neural network training requires large datasets, and one major challenge in applying deep learning in industry is the dataset issue. Therefore, this paper uses a portion of the MSCOCO2014 dataset as the original image library and integrates stamping process images captured by industrial cameras for data fusion training. This approach ensures the dataset’s scale while enhancing the model’s generalization ability and robustness in the specific industrial application of stamping monitoring.

Detone et al. [14] first proposed the S-COCO synthetic dataset for homography estimation, using PatchA and PatchB as inputs to the HomographyNet network model and 4 sets of $\left(∆x_i,∆y_i\right)$ as supervision labels. Although it performs poorly for homography estimation on real-world images, it pioneered the task of homography estimation based on deep learning. Nguyen et al. [15] improved robustness to lighting variations by randomly injecting color, brightness, and gamma shifts when training unsupervised deep models. Koguciuk et al. [17] introduced the PDS-COCO dataset, which builds on the S-COCO dataset by using data augmentation techniques, such as randomly adjusting brightness, contrast, saturation, and noise in certain image regions, and randomly swapping image color channels with a 50% probability. This approach addresses the impact of lighting and noise on homography estimation.

So far, datasets used for homography estimation tasks are based on three-channel color images. Considering the specific industrial application of stamping monitoring, where the images captured by industrial cameras are single-channel grayscale images, the COCO dataset used in this paper is first converted to grayscale. Additionally, this paper adopts the data generation algorithm and data augmentation techniques from Detone et al. [14]. To expand the dataset for the specific industrial application scenario of stamping process monitoring, data augmentation is performed by randomly adding different perturbations, lighting variations, and degrees of blurring, simulating the uncertainties in actual stamping environments. This approach makes the model more suitable for the homography estimation of sheet metal displacement in stamping vibration environments, enhancing its robustness. The dataset generation process is shown in Figure 6.

3. Results and Discussion

This method was implemented using Python 3.9.16 and PyTorch 2.0.0+cu118. The software platform was PyCharm 2022.1.4, and the hardware environment was a Windows 11 64-bit operating system with a 12th Gen Intel(R) Core(TM) i9-12900H processor (2.50 GHz) and 16 GB of memory. All training was conducted on a single NVIDIA GeForce RTX 3060 Laptop GPU with a batch size of 64 and 30,000 training iterations. The initial learning rate was set to 0.005, and it was reduced to one-tenth of its original value every 6000 iterations. The Adam optimizer was used with hyperparameters set to ${\beta }_1=0.9$, ${\beta }_2=0.999$, and $\epsilon ={10}^{-8}$. The balance weights for the three branches of the loss function were set to ${\alpha }_1=0.5$, ${\alpha }_2=0.3$, and ${\alpha }_3=0.2$, respectively. The experiments used the MSCOCO2014 dataset and stamping monitoring images as the original image library. The stamping monitoring images were single-channel grayscale images collected by an industrial area-scan monochrome camera with a resolution of 1920 × 1080. The MSCOCO2014 dataset was divided into training, validation, and test sets, generating 90,000 data samples. The stamping monitoring image dataset consisted of 1000 images and generated 10,000 data samples, which were divided into training, validation, and test sets in a 5:3:2 ratio.

3.1. Evaluation Indicators

To measure the accuracy of different methods for homography estimation, this paper uses the Mean Average Corner Error (MACE) as an evaluation metric, which is the average Euclidean distance between the predicted and the true positions of the corner points. Given a pair of images, $(I_s, I_t)$, and their corresponding homography matrix, $H_{\mathit{st}}$, let the coordinates of the four corner points of image $I_s$ be denoted as $p_s^i={\left[x_s^i,y_s^i,1\right]}^T,i\in \left\{1,2,3,4\right\}$. The homography matrix predicted by the homography estimation model is ${\stackrel{\sim }{H}}_{\mathit{st}}$. The true positions, $p_t^i$, and the predicted positions, ${\stackrel{\sim }{P}}_t^i$, of the four corner points in the target image, $I_t$, are calculated respectively.

$$p_t^i=H_{\mathit{st}}{p}_s^i, {\stackrel{\sim }{P}}_t^i={\stackrel{\sim }{H}}_{\mathit{st}}{p}_s^i,i\in \{1,2,3,4\}$$

(14)

The Mean Average Corner Error here is defined as the average of the four corner errors over $M$ pairs of images:

$$\mathrm{MACE}={\displaystyle \frac{1}{4\mathrm{N}}}{{\displaystyle \sum }}_{\mathrm{j}=1}^{\mathrm{M}}{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{N}}{\left({\parallel {\mathrm{p}}_{\mathrm{t}}^{\mathrm{i}}-{\stackrel{\sim }{\mathrm{P}}}_{\mathrm{t}}^{\mathrm{i}}\parallel }_2\right)}_j$$

(15)

3.2. Comparative Experiment Results and Analysis

To evaluate the effectiveness of the proposed method, it will be compared with traditional homography estimation methods and deep-learning-based homography estimation methods. The average L2 distance between the true corner coordinates and the predicted coordinates will be calculated for all four corners of all images in the test set. Specifically, ORB + RANSAC and SIFT + RANSAC will be chosen as traditional feature-based homography estimation methods due to their superior performance over most traditional methods. HomographyNet and PFNet [28] will be selected as supervised learning-based homography estimation methods, where HomographyNet is the first model to estimate homography using deep learning, and PFNet is an excellent supervised homography estimation model that performs well on various datasets. The homography estimation model from Nguyen et al. [15] will be chosen as the unsupervised learning-based homography estimation method, as it is the first to apply unsupervised learning to the homography estimation task and has been used as a basis in subsequent studies. To demonstrate the novelty of the method proposed in this paper, the content-aware unsupervised homography estimation network (C-AHomE) from Zhang J. et al. [16] and the bidirectional implicit loss-based unsupervised homography estimation network (biHomE) from Koguciuk et al. [17] are selected as the latest research methods for comparison.

The comparison results of the mean corner error between the proposed method and other homography estimation methods are shown in Figure 7. It can be seen that the two traditional methods based on image features have significantly higher errors. This is because they overly rely on the quality of feature points during the feature extraction stage and the content of the image itself. Due to the fact that the feature points on the surface of the metal sheet are concentrated on the edges, punching holes, etc., and are not sufficiently abundant or well-distributed, they are prone to errors from lighting changes and perspective changes, resulting in poor homography estimation accuracy. Deep-learning methods, due to their strong feature representation capabilities, extract features of higher quality than traditional algorithms, resulting in lower errors. Among them, PFNet performs better. Unsupervised homography estimation networks do not rely on real labeled data during training and improve robustness to lighting changes by minimizing pixel-level photometric errors, resulting in better accuracy than supervised learning methods. However, this method uses a single-stage network structure. The proposed method achieves the lowest error estimation, reducing the MACE error by 2.66 compared to the previous unsupervised homography estimation network, with a 61.4% improvement in performance. When compared to the latest unsupervised homography estimation networks, C-AHomE and biHomE, the performance is improved by 13.9% and 8.2%, respectively.

Figure 8 presents the visualized results of homography estimation using the aforementioned eight methods across eight different test images, including five sets of stamping monitoring images and three sets of images from the MSCOCO2014 dataset. This selection aims to demonstrate the effectiveness of the proposed method in the specific industrial context of stamping monitoring while also highlighting its general applicability to various types of images. The leftmost column shows the template images of the eight test images, with blue frames representing the initial image patches and corner points. The green frames indicate the predicted positions of the corner points, with different levels of precision based on the homography matrices estimated by each method. The higher the overlap between the blue and green frames, the higher the accuracy of the homography estimation and the lower the error. From Figure 8, it can be seen that the proposed method achieves the highest alignment in the visualization results, indicating that the predicted homography matrix is the most accurate and has the smallest error compared to the actual homography matrix. This performance surpasses that of the other seven methods.

Next, the impact of different offsets on various methods is analyzed. When generating test data, different offset values, $\mathsf{\rho }$, are set, representing the range of random perturbations of the four corner points. The smaller the offset, $\mathsf{\rho }$, the less distortion in the homography-transformed image, and vice versa. In this experiment, the offset, $\mathsf{\rho }$, is chosen from the range [10, 60]. The error results of different methods are shown in

Table 1. Error results of different methods under different offset values.

Method	ORB + RANSAC	SIFT + RANSAC	Homography Net	PFNet	Unsupervised HomographyNet	C-AHomE	biHomE	Ours
ρ = 10	8.34	6.82	3.19	2.72	2.26	0.97	0.89	0.72
ρ = 20	10.82	8.57	6.56	4.85	3.49	1.40	1.31	1.19
ρ = 30	12.10	10.73	9.41	6.36	4.33	1.94	1.82	1.67
ρ = 40	15.79	13.32	15.13	8.91	5.27	3.82	3.51	2.23
ρ = 50	18.35	17.91	19.78	11.28	6.12	5.11	4.67	2.81
ρ = 60	21.27	20.45	24.30	13.14	7.91	6.39	5.75	3.56

. It can be seen that, under the same offset value, the proposed method has the lowest homography estimation error. As the input image offset increases, the error of HomographyNet increases significantly, while PFNet performs relatively better. The two traditional feature-based methods, despite having larger errors, show relatively smaller increases in error changes due to their reliance on feature-point extraction. The error of the unimproved unsupervised homography estimation network is also significantly higher than that of the proposed method. Although the new unsupervised homography estimation methods, C-AHomE and biHomE, exhibit relatively low error estimates, their stability significantly decreases as the perturbation range increases, falling short of the stability demonstrated by the method proposed in this paper. This indicates that the proposed method performs well, even when the input images have a low overlap rate, and demonstrates greater robustness to different scaling, rotation, and other viewpoint changes.

3.3. Ablation Experiment Results and Analysis

To validate the effectiveness of the proposed improvements on homography estimation performance, experimental analysis was conducted on the module selection of the improved HomographyNet, the construction of the scale pyramid, and the number of stages in the overall network.

Firstly, ablation studies were conducted on the special deformation convolution layer and GN normalization layer introduced in the improved HomographyNet, as shown in

Table 2. Ablation study results of on module selection in the improved HomographyNet.

Special Deformation Convolution Layer	Group Normalization Layer	Stage 1	Stage 2	Stage 3
×	×	10.15	6.21	4.32
√	×	4.28	3.16	1.91
×	√	9.93	5.38	3.23
√	√	3.91	2.85	1.67

. The results indicate that the original HomographyNet, even with a three-stage network structure, had the highest error in homography estimation. Introducing either the special deformation convolution layer or the GN normalization layer independently improved network performance to different extents, with the deformation layer having a greater impact. This is because this module expands the receptive field and enhances the learning ability for feature rotational invariance during the feature extraction process, which reduces the error in homography estimation. Meanwhile, the GN normalization layer provides stable training capabilities when dealing with small batches of large sizes, but it does not directly affect the error in homography estimation. The improved HomographyNet model proposed in this paper, which incorporates both the special deformation convolution module and the GN normalization module, shows the smallest error in homography estimation.

Subsequently, the method proposed in this paper was tested by using images of different scales at various stages and images of the same scale at all stages to verify the effectiveness of constructing an image pyramid for multi-scale input in homography estimation. As shown in

Table 3. Ablation study results for same-size image input versus multi-scale image input.

Image Scale	Stage 1	Stage 2	Stage 3
The same 128 × 128	4.57	3.44	2.12
512 × 512, 256 × 256, 128 × 128	3.91	2.85	1.67

, the method with same-scale input exhibited higher homography estimation errors at each stage compared to the method proposed in this article. To demonstrate the effectiveness of the improvements, the same test images from the previous experiment were used for homography estimation at three stages, with both same-scale and different-scale inputs. The visualization results, as shown in Figure 9, include the following: (a) the template image with the initial positions of the four corners marked by red boxes; (b), (c), (d) the estimation results at three stages with same-size image inputs; and (e), (f), (g) the estimation results at three stages with different-scale image inputs, where the yellow boxes indicate the predicted positions of the four corners. The higher the overlap between the two boxes, the smaller the error in the estimated homography matrix.

Finally, an ablation study was conducted on the number of network stages used in the proposed method, with the results tested on the aforementioned eight sets of images, as shown in Figure 10. The number of network stages is a crucial hyperparameter in our method. As indicated in the graph, the homography estimation error significantly decreases as the number of network stages increases from one to three, demonstrating that a multi-stage network structure can progressively refine the homography estimation task, reducing errors and improving accuracy. However, when the number of stages increases to four, the errors begin to rise, and at the fifth stage the errors significantly increase. This is attributed to the reduced image size for input when the network exceeds three stages, leading to unstable network training. Therefore, a three-stage network is optimal for this method.

4. Conclusions

Due to the random vibrations of the metal sheet caused by the opening and closing of the mold during the stamping process, image registration between the template image and the real-time monitoring image becomes crucial, making homography estimation especially important. Addressing the issues of large estimation errors, poor generalization, and limited industrial applications in current unsupervised homography networks under conditions of significant viewpoint changes and high image overlap, this paper proposes a multi-stage, multi-scale unsupervised homography estimation model. First, the HomographyNet model is improved by introducing a specialized deformable convolution module and GN (Group Normalization) layer, enhancing the model’s ability to learn rotation invariance when processing large-scale, high-resolution images. Next, based on the unsupervised homography estimation network model, a multi-scale, multi-stage network structure is constructed. This structure estimates homography at different scales across various stages, minimizing pixel-level photometric loss at each stage for unsupervised training. By progressively refining the homography transformation relationship between images through multiple stages, the accuracy of homography estimation is improved, along with the model’s ability to learn scale invariance. Finally, by integrating training with stamping process image data and the MSCOCO2014 dataset, and applying data augmentation through random perturbations, lighting variations, and different levels of blurriness, the model’s robustness to the complex changes in the stamping monitoring environment is enhanced. This approach not only ensures the data scale required for network training but also addresses the issue of data scarcity in applying deep learning to specific industrial environments. It is worth noting that, due to the uncertainty in the stamping environment, the images of the sheet surface may be significantly occluded. Therefore, improving the robustness of homography estimation under occlusion will be the focus of future research.