1. Introduction
With the advancement of Industry 4.0, machine-vision technology has replaced traditional manual inspection methods, enabling real-time monitoring of the stamping process to protect molds [
1]. By capturing images of the metal sheet before and after the mold closes, the process is analyzed for any abnormalities. If an anomaly is detected, an emergency stop signal is issued to halt the stamping equipment immediately, thereby protecting the molds, reducing economic losses, and enhancing production efficiency. However, the actual stamping process may be affected by factors such as the impact force of the mold opening and closing or unstable feed rates from the feeder, causing random vibrations in the metal sheets. This results in random offsets in the images captured each time, necessitating image registration of the template and real-time detection images. The difference between these two images is then used to determine if there are any abnormalities in the process. Therefore, image registration algorithms are indispensable in the application of monitoring the stamping process.
Image registration refers to the process of transforming different images of the same object from different spatial coordinate systems to the same coordinate system through a certain mapping relationship. In this process, homography estimation plays an important role. Traditional homography estimation methods are divided into direct methods and feature-based matching methods [
2]. Direct methods like the Lucas–Kanade algorithm initialize homography parameters and use gradient descent to minimize errors [
3,
4]. Other methods include optical flow and image gradient-based techniques [
5,
6], which are sensitive to lighting changes and require good initial values to prevent local optima. The optimization is computationally intensive, complicating their use in high-resolution and complex scenes. The second method involves feature-based matching, which starts with feature detection algorithms like SIFT, SURF, and ORB to extract key points from images [
7,
8,
9]. Each detected key point is assigned a descriptor that characterizes its surrounding image area. These feature descriptors are then compared between two images using matchers like FLANN or Brute-Force to find matching pairs [
10]. Finally, robust estimation methods such as RANSAC or LMedS are used to compute the optimal homography matrix [
11,
12]. While these methods generally perform faster than direct methods, they heavily rely on the detection and matching of key points. In areas with low texture, sufficient key points may not be detectable, and significant changes in perspective or lighting between images can lead to incorrect key point matches, directly impacting the accuracy of homography estimation.
The stamping surveillance environment is influenced by external factors like lighting changes and equipment instability, as well as human factors such as camera angle adjustments. These factors can significantly alter the comparison between real-time detection and template images, impacting the accuracy of feature-point detection and homography estimation. Leveraging the success of deep convolutional neural networks in computer vision, deep-learning-based homography estimation methods can effectively tackle these robustness issues and the reliance on feature points seen in traditional approaches [
13]. Detone et al. introduced HomographyNet, a simple convolutional neural network trained end-to-end to directly regress the parameters of the homography matrix [
14]. It outperforms traditional methods in complex scenarios such as low-texture environments, but the model is trained on synthetic datasets, leading to poor inference capabilities on real images. Real-world image labeling is also highly costly. Ty Nguyen et al. [
15] were the first to propose an unsupervised homography estimation method. Building on a similar network structure as previous approaches, they enhanced the robustness of homography estimation to lighting variations by learning the homography relationship through minimizing the pixel intensity difference between two images. This method eliminates the need for labeled data. However, it performs poorly on images with significant viewpoint differences and lacks scale and rotation invariance. In recent years, there has been increasing research on unsupervised homography estimation [
16,
17,
18,
19,
20]. Unsupervised learning methods eliminate the complex process of labeling training data, reducing the impact of human bias. These methods are better suited to adapt to complex and changing environments and demonstrate improved generalization ability when handling variations in real-world conditions, such as different lighting conditions, motion blur, shifts, and rotations. Zang J. et al. [
16] proposed an unsupervised learning model that learns feature representations for homography estimation and image comparison. This model calculates loss in the feature space rather than in the pixel space, achieving robustness to varying lighting conditions. Koguciuk et al. [
17] extended this work by adding perceptual loss computation [
21], further enhancing the robustness of deep homography estimation methods to lighting changes. However, both of these new unsupervised homography estimation methods are not applicable to scenes with small image overlapping regions and low similarity. Liu S. et al. [
18] proposed a new unsupervised learning framework that improves homography estimation by learning motion bases. This approach can estimate global homography transformations and handle local homography transformations at a finer granularity, thus improving the accuracy of homography estimation. However, due to the complex feature extraction and learning processes involved, this method requires significant computational resources and time, making it unsuitable for industrial applications.
To address the issues mentioned above, this paper proposes an improved unsupervised homography estimation model based on existing models, tailored for the specific industrial context of stamping process monitoring. By incorporating a special deformation convolution module and a GN normalization layer into the backbone network, the model enhances its rotational invariance when handling large-scale, high-resolution images. A multi-scale, multi-stage network architecture is developed, utilizing unsupervised training through minimizing pixel-level photometric loss at each stage, which progressively refines the homography transformations between images from coarse to fine, improving the accuracy of homography estimation and enhancing the model’s learning ability for scale invariance. To make the model suitable for the specific industrial application of the stamping process and improve its robustness against complex environmental changes such as lighting variations and vibration disturbances, the training dataset integrates stamping surveillance image data with random perturbations, lighting, contrast, and blur for data augmentation, addressing the issue of dataset scarcity in specific industrial applications of deep learning.
The remainder of this paper is organized as follows.
Section 2 first describes the network structure of the unsupervised single-response estimation model, improves on it by introducing a special deformation convolution module and a GN normalization layer, and constructs a multi-scale, multi-stage overall network structure and establishes a multi-stage coupling relation, minimizing the luminosity loss function. Comparison and ablation experiments are carried out in
Section 3 to analyze the effectiveness of the method of this paper, and conclusions are summarized in
Section 4.
3. Results and Discussion
This method was implemented using Python 3.9.16 and PyTorch 2.0.0+cu118. The software platform was PyCharm 2022.1.4, and the hardware environment was a Windows 11 64-bit operating system with a 12th Gen Intel(R) Core(TM) i9-12900H processor (2.50 GHz) and 16 GB of memory. All training was conducted on a single NVIDIA GeForce RTX 3060 Laptop GPU with a batch size of 64 and 30,000 training iterations. The initial learning rate was set to 0.005, and it was reduced to one-tenth of its original value every 6000 iterations. The Adam optimizer was used with hyperparameters set to , , and . The balance weights for the three branches of the loss function were set to , , and , respectively. The experiments used the MSCOCO2014 dataset and stamping monitoring images as the original image library. The stamping monitoring images were single-channel grayscale images collected by an industrial area-scan monochrome camera with a resolution of 1920 × 1080. The MSCOCO2014 dataset was divided into training, validation, and test sets, generating 90,000 data samples. The stamping monitoring image dataset consisted of 1000 images and generated 10,000 data samples, which were divided into training, validation, and test sets in a 5:3:2 ratio.
3.1. Evaluation Indicators
To measure the accuracy of different methods for homography estimation, this paper uses the Mean Average Corner Error (MACE) as an evaluation metric, which is the average Euclidean distance between the predicted and the true positions of the corner points. Given a pair of images,
, and their corresponding homography matrix,
, let the coordinates of the four corner points of image
be denoted as
. The homography matrix predicted by the homography estimation model is
. The true positions,
, and the predicted positions,
, of the four corner points in the target image,
, are calculated respectively.
The Mean Average Corner Error here is defined as the average of the four corner errors over
pairs of images:
3.2. Comparative Experiment Results and Analysis
To evaluate the effectiveness of the proposed method, it will be compared with traditional homography estimation methods and deep-learning-based homography estimation methods. The average L2 distance between the true corner coordinates and the predicted coordinates will be calculated for all four corners of all images in the test set. Specifically, ORB + RANSAC and SIFT + RANSAC will be chosen as traditional feature-based homography estimation methods due to their superior performance over most traditional methods. HomographyNet and PFNet [
28] will be selected as supervised learning-based homography estimation methods, where HomographyNet is the first model to estimate homography using deep learning, and PFNet is an excellent supervised homography estimation model that performs well on various datasets. The homography estimation model from Nguyen et al. [
15] will be chosen as the unsupervised learning-based homography estimation method, as it is the first to apply unsupervised learning to the homography estimation task and has been used as a basis in subsequent studies. To demonstrate the novelty of the method proposed in this paper, the content-aware unsupervised homography estimation network (C-AHomE) from Zhang J. et al. [
16] and the bidirectional implicit loss-based unsupervised homography estimation network (biHomE) from Koguciuk et al. [
17] are selected as the latest research methods for comparison.
The comparison results of the mean corner error between the proposed method and other homography estimation methods are shown in
Figure 7. It can be seen that the two traditional methods based on image features have significantly higher errors. This is because they overly rely on the quality of feature points during the feature extraction stage and the content of the image itself. Due to the fact that the feature points on the surface of the metal sheet are concentrated on the edges, punching holes, etc., and are not sufficiently abundant or well-distributed, they are prone to errors from lighting changes and perspective changes, resulting in poor homography estimation accuracy. Deep-learning methods, due to their strong feature representation capabilities, extract features of higher quality than traditional algorithms, resulting in lower errors. Among them, PFNet performs better. Unsupervised homography estimation networks do not rely on real labeled data during training and improve robustness to lighting changes by minimizing pixel-level photometric errors, resulting in better accuracy than supervised learning methods. However, this method uses a single-stage network structure. The proposed method achieves the lowest error estimation, reducing the MACE error by 2.66 compared to the previous unsupervised homography estimation network, with a 61.4% improvement in performance. When compared to the latest unsupervised homography estimation networks, C-AHomE and biHomE, the performance is improved by 13.9% and 8.2%, respectively.
Figure 8 presents the visualized results of homography estimation using the aforementioned eight methods across eight different test images, including five sets of stamping monitoring images and three sets of images from the MSCOCO2014 dataset. This selection aims to demonstrate the effectiveness of the proposed method in the specific industrial context of stamping monitoring while also highlighting its general applicability to various types of images. The leftmost column shows the template images of the eight test images, with blue frames representing the initial image patches and corner points. The green frames indicate the predicted positions of the corner points, with different levels of precision based on the homography matrices estimated by each method. The higher the overlap between the blue and green frames, the higher the accuracy of the homography estimation and the lower the error. From
Figure 8, it can be seen that the proposed method achieves the highest alignment in the visualization results, indicating that the predicted homography matrix is the most accurate and has the smallest error compared to the actual homography matrix. This performance surpasses that of the other seven methods.
Next, the impact of different offsets on various methods is analyzed. When generating test data, different offset values,
, are set, representing the range of random perturbations of the four corner points. The smaller the offset,
, the less distortion in the homography-transformed image, and vice versa. In this experiment, the offset,
, is chosen from the range [10, 60]. The error results of different methods are shown in
Table 1. It can be seen that, under the same offset value, the proposed method has the lowest homography estimation error. As the input image offset increases, the error of HomographyNet increases significantly, while PFNet performs relatively better. The two traditional feature-based methods, despite having larger errors, show relatively smaller increases in error changes due to their reliance on feature-point extraction. The error of the unimproved unsupervised homography estimation network is also significantly higher than that of the proposed method. Although the new unsupervised homography estimation methods, C-AHomE and biHomE, exhibit relatively low error estimates, their stability significantly decreases as the perturbation range increases, falling short of the stability demonstrated by the method proposed in this paper. This indicates that the proposed method performs well, even when the input images have a low overlap rate, and demonstrates greater robustness to different scaling, rotation, and other viewpoint changes.
3.3. Ablation Experiment Results and Analysis
To validate the effectiveness of the proposed improvements on homography estimation performance, experimental analysis was conducted on the module selection of the improved HomographyNet, the construction of the scale pyramid, and the number of stages in the overall network.
Firstly, ablation studies were conducted on the special deformation convolution layer and GN normalization layer introduced in the improved HomographyNet, as shown in
Table 2. The results indicate that the original HomographyNet, even with a three-stage network structure, had the highest error in homography estimation. Introducing either the special deformation convolution layer or the GN normalization layer independently improved network performance to different extents, with the deformation layer having a greater impact. This is because this module expands the receptive field and enhances the learning ability for feature rotational invariance during the feature extraction process, which reduces the error in homography estimation. Meanwhile, the GN normalization layer provides stable training capabilities when dealing with small batches of large sizes, but it does not directly affect the error in homography estimation. The improved HomographyNet model proposed in this paper, which incorporates both the special deformation convolution module and the GN normalization module, shows the smallest error in homography estimation.
Subsequently, the method proposed in this paper was tested by using images of different scales at various stages and images of the same scale at all stages to verify the effectiveness of constructing an image pyramid for multi-scale input in homography estimation. As shown in
Table 3, the method with same-scale input exhibited higher homography estimation errors at each stage compared to the method proposed in this article. To demonstrate the effectiveness of the improvements, the same test images from the previous experiment were used for homography estimation at three stages, with both same-scale and different-scale inputs. The visualization results, as shown in
Figure 9, include the following: (a) the template image with the initial positions of the four corners marked by red boxes; (b), (c), (d) the estimation results at three stages with same-size image inputs; and (e), (f), (g) the estimation results at three stages with different-scale image inputs, where the yellow boxes indicate the predicted positions of the four corners. The higher the overlap between the two boxes, the smaller the error in the estimated homography matrix.
Finally, an ablation study was conducted on the number of network stages used in the proposed method, with the results tested on the aforementioned eight sets of images, as shown in
Figure 10. The number of network stages is a crucial hyperparameter in our method. As indicated in the graph, the homography estimation error significantly decreases as the number of network stages increases from one to three, demonstrating that a multi-stage network structure can progressively refine the homography estimation task, reducing errors and improving accuracy. However, when the number of stages increases to four, the errors begin to rise, and at the fifth stage the errors significantly increase. This is attributed to the reduced image size for input when the network exceeds three stages, leading to unstable network training. Therefore, a three-stage network is optimal for this method.
4. Conclusions
Due to the random vibrations of the metal sheet caused by the opening and closing of the mold during the stamping process, image registration between the template image and the real-time monitoring image becomes crucial, making homography estimation especially important. Addressing the issues of large estimation errors, poor generalization, and limited industrial applications in current unsupervised homography networks under conditions of significant viewpoint changes and high image overlap, this paper proposes a multi-stage, multi-scale unsupervised homography estimation model. First, the HomographyNet model is improved by introducing a specialized deformable convolution module and GN (Group Normalization) layer, enhancing the model’s ability to learn rotation invariance when processing large-scale, high-resolution images. Next, based on the unsupervised homography estimation network model, a multi-scale, multi-stage network structure is constructed. This structure estimates homography at different scales across various stages, minimizing pixel-level photometric loss at each stage for unsupervised training. By progressively refining the homography transformation relationship between images through multiple stages, the accuracy of homography estimation is improved, along with the model’s ability to learn scale invariance. Finally, by integrating training with stamping process image data and the MSCOCO2014 dataset, and applying data augmentation through random perturbations, lighting variations, and different levels of blurriness, the model’s robustness to the complex changes in the stamping monitoring environment is enhanced. This approach not only ensures the data scale required for network training but also addresses the issue of data scarcity in applying deep learning to specific industrial environments. It is worth noting that, due to the uncertainty in the stamping environment, the images of the sheet surface may be significantly occluded. Therefore, improving the robustness of homography estimation under occlusion will be the focus of future research.