Adaptive Real-Time Tracking of Molten Metal Using Multi-Scale Features and Weighted Histograms

Abstract

In this study, we study the tracking of the molten metal region in the dross removal process during metal ingot casting, and propose a real-time tracking method based on adaptive feature selection and weighted histogram. This research is highly significant in metal smelting, as efficient molten metal tracking is crucial for effective dross removal and ensuring the quality of metal ingots. Due to the influence of illumination and temperature in the tracking environment, it is difficult to extract suitable features for tracking molten metal during the metal pouring process using industrial cameras. We transform the images captured by the camera into a multi-scale feature space and select the features with the maximum distinction between the molten metal region and its surrounding background for tracking. Furthermore, we introduce a weighted histogram based on the pixel values of the target region into the mean-shift tracking algorithm to improve tracking accuracy. During the tracking process, the target model updates based on changes in the molten metal region across frames. Experimental tests confirm that this tracking method meets practical requirements, effectively addressing key challenges in molten metal tracking and providing reliable support for the dross removal process.

1. Introduction

The metal ingots produced by the non-ferrous metallurgy industry are fundamental materials in various industrial fields, including manufacturing and engineering, construction and infrastructure, and electronics and electrical applications. During the production process of metal ingots, metallic raw materials are melted at high temperatures into a molten metal, which is then cast into fixed molds on the conveyor line via a casting machine. Following cooling and demolding, metal ingots are formed. When the molten metal is poured into the ingot molds, which are not fully sealed, it rapidly reacts chemically with the air upon exposure, leading to the formation of irregularly distributed dross crystals on the top layer of the molten metal. For a high-quality metal ingot, the dross that forms on the surface while the metal is still molten needs to be removed. Currently, this dross removal process is manually executed in most non-ferrous smelting plants, where operators face adverse conditions such as high temperatures, significant dust, and metal spattering. To address these issues, some enterprises have opted to employ industrial robots to replace manual labor for dross removal tasks. However, due to the harsh industrial environment, there are still numerous technical challenges to address as dross removal robots perform their tasks.

The first step in employing dross removal robots for dross removal tasks involves accurately tracking molten metal on the conveyor line. To effectively remove the oxide dross from the surface of the molten metal, the dross removal shovel must be precisely inserted into the molten metal and follow a predetermined trajectory to scoop out the dross. During this process, the shovel must avoid contact with the ingot mold to prevent bending or damage to the shovel and potential harm to the robot itself. Therefore, accurate tracking of the molten metal area is essential to provide the robot with safe insertion guidance.

When tracking the molten metal area, its appearance is influenced by various factors. These include the strong reflectivity of the molten metal surface to light, the varying light intensity on the casting production line between day and night, the irregular shaking of the molten metal in the ingot mold caused by the conveyor line, the formation of irregular oxide dross layers on the molten metal surface due to air exposure, and the rapid solidification of the molten metal in the ingot mold. These factors make the appearance of the molten metal area complex and variable, thus making precise tracking a highly challenging task.

In industrial scenarios, numerous widely applied studies and algorithms for target tracking have been developed, such as [1,2,3,4,5,6]. Some of these methods offer excellent real-time tracking performance but struggle with robustness to target appearance changes, occlusion, and rotation. Others achieve high reliability and tracking accuracy but demand substantial computational resources, resulting in poor real-time performance. To precisely track molten metal, we need to ensure real-time performance while maintaining robustness to appearance changes and occlusion. Mean-shift tracking [7] is a statistical pattern matching method used for target tracking. It makes iterative calculations to find the most probable position of the target. The main steps include target representation, probability density estimation, and the mean-shift algorithm. Due to its non-parametric nature, mean-shift tracking offers strong adaptability, high real-time performance, and robustness to partial occlusion and noise. Although it faces challenges such as local maxima and sensitivity to target scale changes, its simplicity, efficiency, and robustness enable it to produce satisfactory results in many industrial scenarios.

The primary contribution of this paper is the development of a method for the real-time precise tracking of molten metal in harsh industrial environments. Based on raw images from the metal casting production line, we developed a set of candidate features and enhanced the representation of the molten metal area by selecting the most suitable features for real-time tracking. We improved the traditional mean-shift tracking framework by integrating the variance ratio between the molten metal area and its background into the target model, resulting in a new target-weighted histogram. This approach effectively reduces the impact of lighting changes and surface reflections on the appearance of the molten metal area. Additionally, given the complex and variable nature of the molten metal area’s appearance, we introduced a real-time updating mechanism for the molten metal target model to prevent model drift during prolonged tracking periods. We conducted tests on the proposed molten metal region tracking method to evaluate its effectiveness and applicability, identifying further research directions.

The structure of our research is laid out as follows: Section 2 offers a concise overview of related research work. Section 3 outlines the main challenges in tracking molten metal. Section 4 explains the proposed method in detail. Section 5 showcases the results of the experiments. Finally, Section 6 concludes the paper and discusses future research directions.

2. Related Work

Distinguishing the molten metal from its surrounding environment is a key research focus in the field of object tracking. To enhance differentiation between the target and the background, methods that increase their distinctiveness are often employed. In different feature spaces, the same target and background exhibit varying levels of distinction. Therefore, selecting the appropriate feature space, or choosing the right features to represent the target and the background, often results in more effective differentiation. Color features [8,9,10,11] are widely utilized in object tracking due to their robustness against shape changes, scale and rotation invariance, computationally efficiency, and stability under varying lighting conditions. These attributes make them suitable for various objects and scenes. However, color features are also sensitive to lighting variations, have limited discriminative power, lack structural information, and can be easily confused by similar-colored backgrounds and dynamic appearance changes. To address these limitations, advanced tracking methods often integrate color features with other feature types such as texture [12,13,14], contour [15,16,17], and motion [18,19]. Combining these features [20,21,22] allows for a more comprehensive description of the tracked object, enhancing the ability to handle more complex tracking scenarios. Meanwhile, in recent years, many studies have leveraged deep learning [23] and information enhancement techniques to improve the performance of target models. For example, ref. [24] uses residual networks and correlation filters to enhance target models, ref. [25] developed GradNet to handle target variations and background clutter using gradient-guided networks, and ref. [26] improves target localization by utilizing scene information. Although these methods perform well in specific scenarios, they have limitations in computational efficiency, handling complex dynamic environments, and robustness under extreme conditions.

The Histogram of Oriented Gradients (HOG) [27,28,29] is a powerful and efficient feature descriptor for object tracking, capturing edge and gradient structures to reveal local contours and shapes while reducing sensitivity to color variations. However, HOG has limitations: it is sensitive to occlusions, lacks broader contextual information, has high computational complexity, and struggles with smooth or low-contrast objects, reducing its effectiveness. To address these weaknesses, HOG has been combined with Convolutional Neural Networks (CNNs) [30], leveraging edge information and contextual analysis to enhance robustness against occlusions and variations in illumination. Additionally, integral images have been employed to expedite feature extraction, thereby reducing computational complexity for real-time applications [31]. These methods have enhanced the performance of HOG to some extent, but they still have limitations in computational complexity, data and resource requirements, and adaptability to dynamic environments.

Selecting appropriate features for tracking is a key research focus in the field of object tracking. Studies such as [8,21,32,33,34] have proposed various methods for the real-time evaluation of multiple features during tracking. Based on evaluation criteria, these methods select the features with the best tracking performance to improve tracking accuracy. Additionally, these studies have investigated target model update mechanisms and proposed various methods for updating the target model. However, these methods are generally effective only when the target appearance does not change significantly.

In the field of object tracking, non-parametric methods are widely used due to their high flexibility and real-time performance. In [7], the authors compare histograms of a target model and a candidate model using a similarity function, often the Bhattacharyya Coefficient, to evaluate similarity. This method incorporates a background-weighted histogram (BWH) to minimize the influence of background information when building the target model during mean-shift tracking. However, in [35], the authors pointed out that using BWH to reconstruct the color histograms of both the target model and the target candidate region simultaneously reduces the weight of background pixel values, leading to no improvement in tracking performance. Therefore, a corrected BWH (CBWH)-weighted histogram was proposed, which only reconstructs the histogram of the target model, effectively reducing background interference during the tracking process but being less effective in high-density dynamic backgrounds. In [36], a local-background-weighted histogram (LBWH) was further proposed, focusing on local background information within the target model. This made LBWH perform better than CBWH in more complex background scenarios but with a high computational complexity. Additionally, in [37], a Spatiogram Corrected Background-Weighted Histogram was proposed, effectively reducing background information in the target model in complex scenes but with a high implementation complexity and requiring significant computational resources. Overall, these methods have made progress in reducing background interference, but computational complexity and adaptability to dynamic changes still need optimization.

Tracking molten metal exemplifies a practical application of long-term tracking. Recent advancements in long-term visual tracking focus on integrating robust mechanisms and intelligent systems to manage environments with significant uncertainties or clutter. In [38], the authors introduced a novel Meta-Updater using cascaded LSTMs to assess update needs in scenarios with occlusions or target reappearance, common in long-term tracking. This work [39] enhanced tracking accuracy with a sophisticated template update mechanism that uses original templates and updates them based on optical flow in cluttered scenes. Additionally, in [40], a tracking-by-detection framework using structured output prediction eliminates intermediate classification steps, directly addressing practical tracking needs. These methods indicate a shift towards trackers that are not only reactive but also adaptively predictive, adeptly handling the complex dynamics of real-world environments.

The previously discussed research methods perform well in certain scenarios. However, in the challenging industrial settings of metal casting and smelting—characterized by high temperatures, heavy dust, splashing molten metal, intense light reflections from the metal surface, and low contrast between the molten metal and its surroundings—these methods often fail to achieve the real-time tracking of molten metal. The approach proposed in this paper integrates and refines these methods, considering these adverse factors, thereby enabling precise real-time tracking in the demanding environments of metal smelting.

3. Problem Description

The production process of metal ingots can generally be divided into casting, dross skimming, cooling, and demolding, as illustrated in Figure 1a. This paper primarily focuses on the dross skimming process, which involves the pouring of molten metal into ingot molds and the subsequent removal of oxide dross by dross removal robots.

In Figure 1a, the casting machine pours the high-temperature molten metal into the ingot molds, which carry the molten metal forward slowly. When reaching the predetermined dross skimming position of the dross removal robot, the robot utilizes a dross skimming shovel to remove the oxide dross from the molten metal according to a predefined dross skimming trajectory, discarding the dross into an adjacent waste bin. The oxide dross accumulated in the waste bin to a certain volume will be reintroduced into production. Figure 1b demonstrates the specific dross skimming operation, wherein, under the appropriate dross skimming trajectory, the dross removal robot can achieve a dross skimming effect comparable to manual labor.

Following the pouring of the molten metal into the ingot molds, we employ visual sensors to capture images and track the molds awaiting dross skimming. As depicted in Figure 2a, sensor brackets are positioned between the casting machine and the dross removal robot, with visual sensors arranged above the ingot molds to acquire image data of the molds designated for dross skimming. In Figure 2b–d, the challenges associated with tracking the molten metal within the ingot molds are analyzed, and followed by a detailed description.

3.1. Limits of Visual Sensors in Complex High-Temperature Environments

Using visual sensors to track ingot molds essentially involves tracking the molten metal within them. Under ideal conditions, the significant temperature difference between the molten metal and the mold allows for easy identification and tracking of the molten metal using an infrared camera to capture thermal images. Additionally, employing a 3D scanner to obtain point cloud data of the molten metal surface within the mold not only facilitates tracking but also provides detailed information on the distribution of oxide dross on the metal surface. However, in practical applications, high-precision infrared cameras and 3D scanners cannot endure the high-temperature environment of the molten metal surface for extended periods. Moreover, occasional splashing of molten metal onto the sensor lenses can damage the sensors. Therefore, we ultimately opt for more cost-effective and heat-resistant industrial RGB cameras.

3.2. Low Contrast between Target and Background

In general tracking tasks, there is a certain contrast between the target and the background, allowing conventional tracking methods to achieve good results. However, as shown in Figure 2c, in the tracking task discussed in this paper, the metal ingots and the molten metal they contain have a low contrast with the background, increasing the difficulty of tracking the molten metal area.

3.3. Intense Specular Reflections from Liquid Surfaces

When the dross removal robot is performing dross skimming operations, it first needs to track the molten metal region. it is initially necessary to establish a target model. However, the surface of molten metal exhibits strong reflectivity, and the distribution characteristics of the oxide dross within the area cannot be discerned due to this reflection. Additionally, the fluidity of the liquid causes the reflective areas to change constantly, resulting in continuous variations in the appearance of the molten metal area, as shown in Figure 2d. This variability makes it challenging to establish a stable target model for the molten metal region.

4. Method

4.1. Feature Set Construction and Discriminability Evaluation for Tracking

In images captured with industrial RGB cameras, the molten metal, as the target area, shows low contrast with its background. To mitigate this, we transform RGB images to build a feature set for selection. The aim is to identify the most distinctive features between the molten metal and the background, facilitating effective tracking within this feature space.

When tracking molten metal, various features can be considered, such as color, texture, contour, and motion. Additionally, due to the significant temperature difference between the molten metal and the ingot mold, temperature features can also be utilized. Our approach involves selecting a feature that is easy to extract and calculate, then constructing a candidate feature set by adjusting the parameters of this feature space. Finally, we evaluate all sub-features within the candidate feature set to determine the optimal feature space parameters. Based on the specific requirements and characteristics of the molten metal tracking task, we have decided to use color information to establish the candidate feature set.

Using an industrial RGB camera, the raw images obtained are RGB images. One way to construct a candidate feature set is to split these images into their R, G, and B channels and then linearly combine them to convert the images into intensity images. The form of RGB color space candidate feature set is as follows

$$\mathcal{F}\equiv \left\{{\lambda }_{1}R+{\lambda }_{2}G+{\lambda }_{3}B|{\lambda }_{*}\in [-2,-1,0,1,2]\right\}$$

(1)

The coefficients for the R, G, and B channel values are set to integers ranging from −2 to 2. The initial number of candidate features based on these combinations of the three color spaces is $5^3$. However, not all combinations are unique; some are redundant, such as $({{\lambda }_1}^{\prime },{{\lambda }_2}^{\prime },{{\lambda }_3}^{\prime })=k({\lambda }_1,{\lambda }_2,{\lambda }_3)$. We also exclude the trivial case where all coefficients are zero $({\lambda }_1,{\lambda }_2,{\lambda }_3)=(0,0,0)$. After these adjustments, we have 49 distinct features.

The advantages of constructing the candidate feature set in this way are the following: (1) the features involve only simple integer arithmetic, making them computationally efficient; (2) all pixels within the region are processed uniformly, ensuring overall feature consistency; (3) the candidate feature set includes many commonly used features in the color space.

Due to the different combinations of coefficients, the value ranges of the transformed pixels will be inconsistent. To facilitate the evaluation of feature images corresponding to all coefficient combinations, we normalize the pixel value ranges in all feature images to 0 to 255. These are then represented using color histograms with $2^r$ bins, where r is the bit resolution. Typically, we use 6 bits, resulting in histograms with 64 bins, enhancing computational efficiency.

The purpose of establishing the candidate feature set is to select the features most suitable for the tracking task, specifically those that maximize the distinction between the molten metal region and the background within the feature space. We aim to select the most effective features for tracking tasks through rigorous empirical evaluation.

Our tracking initiative specifically targets the molten metal area within the ingot mold, which is geometrically confined to a square region by the boundaries of the mold. To effectively collect pixel data, we adopt a ‘center-surround’ methodology, which facilitates the acquisition of pixel values from both the designated target and the adjacent background areas. Figure 3 clearly delineates the demarcation between the target and background regions and elaborates on the methodology used to evaluate the separability of these areas.

Each feature image f within the candidate feature set displays distinct content. Visually, some feature images show the molten metal to be almost indistinguishable from the surrounding background, making it nearly impossible to accurately identify and track the area of the molten metal. In other feature images, the molten metal is clearly visible with the background information completely absent, facilitating easy identification and tracking of the metal region. Therefore, quantifying the expressive ability of feature images regarding the area of interest, the molten metal, and selecting the most suitable feature images for tracking tasks based on this quantification, rather than visual inspection, is crucial. We choose to define the area of the molten metal and its surrounding background in each feature image, and then quantify the expressive power of each image by calculating the color histograms of these two areas and their logarithmic likelihood ratio and variance ratio. Let $p_{tgt}\left(b\right)$ represent the color histogram of the molten metal region, and $q_{bg}\left(b\right)$ represent the color histogram of the surrounding background region. The value b corresponds to a specific bin in the histogram, with a range of $b\in \left[1,2^r\right]$. The log-likelihood ratio is then calculated using the ratio of $p_{tgt}\left(b\right)$ to $q_{bg}\left(b\right)$, as shown in the following formula

$$Ratio\left(b\right)=log{\displaystyle \frac{max\left\{p_{tgt}\left(b\right),\alpha \right\}}{max\left\{q_{bg}\left(b\right),\alpha \right\}}}$$

(2)

Due to the introduction of logarithmic calculations, it is necessary to avoid situations where the numerator or denominator is zero. Typically, we introduce a value $\alpha$, set to 0.001.

We have estimated the color histograms $p_{tgt}\left(b\right)$ and $q_{bg}\left(b\right)$ for features in the candidate set. To select the most suitable feature images for tracking from the candidate feature set, where the tracking target is most distinctly differentiated from its surrounding background, we calculate the variance of the ratio. Reuse these histograms to compute the variance of $Ratio\left(b\right)$ under the target histogram $p_{tgt}\left(b\right)$ as

$$\begin{array}{cc}\hfill \mathrm{variance}(R;p_{tgt})& =E\left[R^2\left(b\right)\right]-{\left(E\left[R\left(b\right)\right]\right)}^2\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill & =\sum _b{p}_{tgt}\left(b\right)R^2\left(b\right)-{\left[\sum _b{p}_{tgt}\left(b\right)R\left(b\right)\right]}^2\hfill \end{array}$$

(4)

Similarly, the variance of $Ratio\left(b\right)$ under the background histogram $q_{bg}\left(b\right)$ is also calculated in the same way. And the total variance of $Ratio\left(b\right)$ under $p_{tgt}\left(b\right)$ and $q_{bg}\left(b\right)$ is calculated as follows

$$\mathrm{Variance}\mathrm{of}\mathrm{Ratio}(R;p_{tgt},q_{bg})\equiv {\displaystyle \frac{\mathrm{variance}(R;(p_{tgt}+q_{bg})/2)}{\left[\mathrm{variance}(R;p_{tgt})+\mathrm{variance}(R;q_{bg})\right]}}$$

(5)

For a more intuitive understanding of the variance ratio, when the variance ratio value is larger, it indicates that the pixel value distribution within the defined target region and background region is more concentrated. At the same time, the mean difference of pixel values between these two regions is greater. Therefore, a feature image with a larger variance ratio means that this feature image has a stronger representation of the target, making the target more distinguishable from the background.

Figure 4 shows a target region on the metal ingot production line, along with the 49 candidate features derived from this image. We computed the variance ratio for each feature and ranked them accordingly. The feature with the highest variance ratio is in the upper left, and the one with the lowest is in the lower right. This demonstrates that the variance ratio magnitude indicates the discriminative power of the candidate features to differentiate between the target and the background. Selecting features with the highest variance ratio for tracking can result in better tracking performance.

4.2. Mean-Shift Target Tracking with Target-Weighted Histogram

After evaluating the feature images, we select the one with the highest target-background distinction capability to track the molten metal region. First, we briefly review the mean-shift tracking algorithm using background-weighted histograms. Then, we describe the proposed improved target-weighted histogram. Finally, we present the implementation details of our tracking method.

In the initial frame of the video, when the metal casting production line begins operating, it is necessary to manually select the region containing the molten metal, typically a rectangular area. This region is then used to create the molten metal target model. The molten metal region is given by

$$q_b=C{w}_b\sum _{c=1}^{n}k\left({∥x_c∥}^2\right)\delta \left[B\left(x_c\right)-b\right]$$

(6)

In the given molten metal region model, ${\left\{x_c\right\}}_{c=1\dots n}$ represents the pixel coordinates of all the pixel points belonging to the molten metal region in the image, and $B\left(x_c\right)$ is an indexing function that maps input pixel coordinates to their respective bins in the histogram. $k\left(∥x_c∥\right)$ is typically a convex monotonic decreasing kernel weight function. When a pixel is located at the edge of the molten metal region, it assigns a smaller weight to it, while assigning a larger weight to pixels at the center of the molten metal region, thereby emphasizing the pixels located more centrally within the molten metal region. However, depending on the tracking object and task, a kernel function that assigns equal weights to all pixels can also be chosen. $\delta$ represents the Kronecker delta function, and C is a constant, calculated as follows

$$C={\displaystyle \frac{1}{{\sum }_{c=1}^{n}k\left({∥x_c∥}^2\right){\sum }_{b=1}^m{w}_b\delta \left[B\left(b_c\right)-b\right]}}$$

(7)

$w_b$ is a weight value obtained from the background region histogram ${\left\{o_b\right\}}_{b=1\dots m}$ (with ${\sum }_{b=1}^m{o}_b=1$), aimed at reducing background interference in the target model region. As shown in Figure 4a, once the background region is determined, the weight value $w_b$ is calculated using the background region histogram and the smallest nonzero value among all its bins, as follows

$${\left\{w_b=\min ({\displaystyle \frac{o^{*}}{o_b}},1)\right\}}_{b=1\dots m}$$

(8)

The weight values derived from the background histogram suppress background information within the molten metal region, while enhancing the unique pixel value information specific to the molten metal region.

After establishing the molten metal region model in the initial frame, tracking the molten metal region begins in subsequent frames. The molten metal moves slowly along the direction of the production line. Therefore, in the next frame of the video, we re-establish a region slightly larger than the molten metal region, To ensure tracking accuracy and include some background information, we set the length and width of this region to be $1.2$ times that of the target model region defined in the initial frame. The region is centered on the center point $y_{center}$ of the molten metal area in the initial frame, defining it as the molten metal candidate region. In this frame, due to the movement of the molten metal area, we start searching for the molten metal area near the center point $y_{center}$. In this process, the target candidate region is defined by

$$p_b\left(y\right)=C_h{w}_b\sum _{c=1}^{n^h}k({∥{\displaystyle \frac{y-x_c}{h}}∥}^2)\delta \left[B\left(x_c\right)-b\right]$$

(9)

where

$$C_h={\displaystyle \frac{1}{{\sum }_{c=1}^{n^h}{w}_{b}k({∥\frac{y-x_c}{h}∥}^2)}}$$

(10)

is also a constant, aimed at normalizing the model of the molten metal candidate, y is the center of molten metal candidate region, and h is the bandwidth.

At this point, the problem of object tracking becomes finding the optimal y such that the Bhattacharyya coefficient between the molten metal candidate region model at the current position and the target model is maximized. The Bhattacharyya coefficient is a measure of similarity between two histograms and can be expressed as $\beta (p_b\left(y\right),q_b)$. When the Bhattacharyya coefficient is maximized, it indicates the highest similarity between the two histograms.

$$y^{★}=\underset{y}{max}\beta (p_b\left(y\right),q_b)=\underset{y}{max}\sum _{b=1}^m\sqrt{p_b\left(y\right)q_b}$$

(11)

Performing a Taylor expansion on the expression (11) at $y_{initial}$, which is the initial position of target candidate region, we obtain

$$y^{★}\approx {\displaystyle \frac{1}{2}}\sum _{b=1}^m\sqrt{p_b\left(y_{initial}\right)q_b}+{\displaystyle \frac{1}{2}}\sum _{b=1}^m{p}_b\left(y\right)\sqrt{{\displaystyle \frac{q_b}{p_b\left(y_{initial}\right)}}}$$

(12)

Introducing (9), we now obtain

$$y^{★}\approx {\displaystyle \frac{1}{2}}\sum _{b=1}^m\sqrt{p_b\left(y_{initial}\right)q_b}+{\displaystyle \frac{C_h}{2}}\sum _{c=1}^{n_h}{w}_{a}k\left({∥{\displaystyle \frac{y-x_c}{h}}∥}^2\right)$$

(13)

where

$$w_a=\sum _{b=1}^m\delta \left[B\left(x_c\right)-b\right]\sqrt{{\displaystyle \frac{q_b}{p_b\left(y_{initial}\right)}}}$$

(14)

To obtain the maximum value of $y^{★}$, starting from $y_{initial}$, the molten metal target candidate region searches for a nearby point $y_{max}$, such that the value of the second term in (13) reaches a local maximum. At this point, the molten metal target candidate region is considered to have matched the molten metal target region, achieving real-time tracking of the molten metal region. By observing the second term in (13), it is evident that this process essentially involves using kernel density estimation to find the local mode. Then, the process of finding the point $y_{max}$ is as follows

$$y_{max}\equiv {\displaystyle \frac{{\sum }_{c=1}^n{x}_c{w}_{a}g\left({∥\frac{y_{initial}-x_c}{h}∥}^2\right)}{{\sum }_{c=1}^n{w}_{a}g\left({∥\frac{y_{initial}-x_c}{h}∥}^2\right)}}$$

(15)

In the above formula, $g(.)$ is the derivative of $k(.)$. This formula corresponds to the process in mean-shift where the density estimation of all samples within a sample region is iteratively performed to find a new mode.

The background-weighted histogram suppresses pixels within the target region that closely resemble the background. To further enhance the tracking capability of the molten metal region model, we introduce a weight-based method based on the variance ratio to allocate weights to both the molten metal region model histogram and the histogram of the molten metal candidate region. In establishing the molten metal region model, we recalculate the variance of the ratio between the target region and the background by removing all pixels corresponding to histogram interval u within the target region, generating ${\left\{{\tilde{w}}_b\right\}}_{b=1\dots m}$. Here, ${\tilde{w}}_b$ represents the interpolated and normalized result of the variance ratio between the molten metal target region and its surrounding background after removing pixels within interval b, compared to the original variance ratio. The molten metal region model and the molten metal candidate region are now redefined as

$${\tilde{q}}_b=q_b\ast {\tilde{w}}_b$$

(16)

$${\tilde{p}}_b\left(y_{initial}\right)=p_b\left(y_{initial}\right)\ast {\tilde{w}}_b$$

(17)

According to the method for calculating the variance ratio in (5), the variance ratio values between each pixel in the molten metal region and the background are converted into weight coefficients to construct a new weighted histogram. This approach enhances the tracking of pixels representing target features, reduces the influence of background features, and improves tracking accuracy.

4.3. Updating the Target Model

During the tracking process, the visual aspect of the molten metal region undergoes random changes due to the fluidity of the molten metal and the reflection of light on its surface. Therefore, when the molten metal region changes significantly in the current frame, relying on the molten metal target model from previous frames may result in tracking failure. Promptly updating the molten metal target model is essential for achieving accurate tracking of the molten metal region. To address this challenge, we employ a model update method based on the Bhattacharyya coefficient, tracking and updating the target model of the molten metal region in each video frame. The update strategy is as follows

$$q_{updated}=\left\{\begin{array}{cc}(1-\theta )q+\theta q\left(y_t\right)& \mathrm{Bhattacharyya}\mathrm{coefficient}(q,q\left(y_t\right))<\lambda \\ q& \mathrm{Bhattacharyya}\mathrm{coefficient}(q,q\left(y_t\right))>\lambda \end{array}\right.$$

(18)

Here, $y_t$ is the position of the molten metal region in the current frame obtained through the tracking algorithm, and $q\left(y_t\right)$ is the histogram established at this position based on the extent of the molten metal region. In the model update strategy, the coefficient $\theta \in [0,1]$ determines the contribution of $q\left(y_t\right)$ to the new target model, while the parameter $\eta$ determines whether the molten metal region model needs to be updated. The values of $\theta$ and $\lambda$ are empirically determined through extensive experimentation, with $\theta$ set to 0.01 and $\lambda$ set to 0.4. This approach effectively mitigates the risk of model drift by maintaining a robust link to the original target characteristics while allowing for necessary adaptability to changes in the target’s appearance. By balancing historical data with new information, the method achieves more stable and accurate tracking over time. It is particularly beneficial in complex environments where the target undergoes significant appearance changes due to factors like occlusion, lighting, and perspective shifts.

5. Experimental Results

To verify the effectiveness of our proposed method for tracking molten metal, we acquired videos using a stationary camera within the factory and implemented and tested our method. Additionally, we conducted qualitative and quantitative comparative experiments using several state-of-the-art deep learning network methods to further validate our approach. In the video depicting the metal solution moving along the production line, there are 4774 frames, each measuring 2592 × 1944 pixels. However, we need only focus on the movement of the metal solution within the ingot mold, thereby allowing us to reduce the frame size for tracking to 520 × 180 pixels. This reduction avoids a significant amount of unnecessary computation. The target model of the metal solution is initially outlined in the first frame and subsequently used to track the movement of the metal solution in each following frame.

The tracking results are effectively illustrated in Figure 5 and Figure 6. Utilizing mean-shift algorithms, the adaptive target tracker showcased significant robustness in challenging conditions such as low illumination, reflective liquid surfaces, and artificial occlusions. The discriminative features, selected for their optimal efficacy as depicted in the figure, successfully distinguish the metal solution within the ingots from the ambient environment. The raw footage from the industrial camera, capturing the oscillation of the metallic liquid within the casting mold, initially presented substantial interference due to the reflection of the overhead factory lighting. This interference, however, was significantly reduced when the imagery was processed into the feature space, highlighting the effectiveness of the employed method in this specialized metal solution tracking task. This pattern underscores the superior tracking performance and low computational load of our approach, utilizing features with high discriminative power, thereby meeting the requirements for real-time tracking of the metal solution.

In metal casting production line environments, the task of tracking metal solutions inherently differs from standard tracking operations. The primary aim of tracking metal solutions is to facilitate timely dross removal by a robot, which is triggered when the metal solution within an ingot attains a specific position. Consequently, as soon as the current tracking target arrives at the dross removal site, the focus shifts to tracking the metal solution in the subsequent ingot. Our methodology entails establishing a predetermined y-coordinate threshold within the camera’s visual field. As the target region intersects this y-coordinate, a systematic downward search commences along the y-axis to locate the next target zone. Figure 7 illustrates the practical efficacy of this strategic approach.

Qualitative and Quantitative Comparisons

In our comparative experiments, we evaluated our method against current state-of-the-art (SOTA) deep learning-based object tracking algorithms. We selected DiMP [41], KYS [26], and ToMP [42] for comparison, as these methods have demonstrated an excellent performance on general datasets and have made significant contributions to the field of object tracking.

As shown in Figure 8, our method demonstrated superior accuracy and robustness compared to current state-of-the-art (SOTA) deep learning network methods in qualitative comparisons. Tracking the molten metal region presents unique challenges due to its constantly changing appearance, influenced by factors such as lighting conditions, cooling and solidification, liquid surface oscillations, and the formation of surface dross. Additionally, the industrial environment introduces random occlusions and other external factors. While deep learning network methods often perform well on general datasets, they exhibit certain limitations when applied to this specific task. Our method effectively mitigates the impact of these adverse factors, resulting in an exceptional performance in tracking the molten metal region.

Figure 9 illustrates the variation in Intersection over Union (IoU) values for four different object tracking methods over a series of frames in the comparative experiment. Our method (red curve) demonstrates a superior performance with consistently higher IoU values across different frames, indicating better accuracy and robustness in tracking the molten metal region. DiMP (blue curve) shows an excellent performance on general datasets but has some limitations in the challenging industrial environment, with IoU values lower than our method. It also tends to lose the tracking target in frames with significant appearance changes in the molten metal region. KYS (yellow curve) performs better than DiMP but is similarly affected by the unique characteristics of the molten metal region, such as lighting conditions and surface oscillations, which can lead to instances of lost tracking targets. ToMP (green curve) is more lightweight compared to the other two networks, providing higher real-time performance. However, it also faces difficulties in this specific task. Although its peak IoU values are higher than those of DiMP and KYS, its robustness is weaker. The chart clearly highlights the superior tracking capability of our method in handling the dynamic and challenging conditions of the molten metal tracking task.

In a further quantitative analysis, we compared the performance of four methods in tracking molten metal regions, specifically focusing on the following metrics: success rates at overlap thresholds of 0.5 and 0.75, average overlap (AO), and precision. The experimental results are shown in

Table 1. Quantitative comparison of molten metal tracking performance: success rates (SR) at overlap thresholds 0.5 and 0.75, average overlap (AO), and precision.

	DiMP	KYS	ToMP	Ours
${\mathrm{SR}}_{0.50}(\%)$	95.1	93.6	86.2	97.8
${\mathrm{SR}}_{0.75}(\%)$	66.9	68.7	71.9	86.3
$\mathrm{AO}(\%)$	73.2	74.6	71.6	87.7
$\mathrm{Precision}(\%)$	95.4	93.5	89.9	96.2

, with our method’s performance highlighted in red to emphasize its superiority.

Our method achieved the highest success rate at an overlap threshold of 0.5, reaching $97.8\%$. At an overlap threshold of 0.75, it maintained a success rate of $86.3\%$, significantly outperforming the other three methods. This demonstrates its precise tracking capability for the specific target of molten metal. Additionally, our method achieved an average overlap (AO) of $87.7\%$, further proving its excellent performance in accurately tracking molten metal regions. In terms of precision, our method also outperformed the others, with a precision value of $96.2\%$.

In specific industrial applications, deep learning models have certain disadvantages, such as high computational resource demands, large data requirements, model complexity, and limited adaptability. Our research method successfully overcomes these disadvantages by optimizing computational efficiency, reducing data dependency, enhancing model interpretability, and improving environmental adaptability. Therefore, in the task of tracking molten metal regions, our method demonstrates a superior performance.

6. Conclusions

This paper addresses the need for tracking metal solutions within metal ingots during the non-ferrous metallurgical process, proposing an adaptive tracking method based on the online selection of features with optimal discriminative power and the maximization of the Bhattacharyya coefficient. This method enables the real-time tracking of metal solutions under conditions of low illumination, high liquid surface reflection, and random generation of metal oxide dross. It also introduces a target model updating mechanism based on model similarity to mitigate model drift during prolonged tracking, with its effectiveness validated through experiments.

Considering practical application needs, this paper focuses solely on processing captured video images in the color space without selecting other types of features when establishing the feature set for the metal pouring production line. To achieve a superior tracking performance, one can leverage the harsh environment of the metal pouring site and the high-temperature characteristics of the molten metal by selecting infrared images or three-dimensional images for fusion with two-dimensional original images, thereby obtaining more recognizable features of the molten metal. Additionally, after tracking the molten metal region, it is necessary to identify the morphological distribution of oxide dross within the molten metal region. Based on the distribution of the oxide dross, a dross removal strategy can be specified for the dross removal robot, leading to better dross removal results and improved quality of the metal ingot products.

Although current state-of-the-art deep learning models have certain limitations in tracking molten metal, such as high computational resource demands, large data requirements, and model complexity, they possess great potential. Future research should leverage the advantages of deep learning models by optimizing computational efficiency, reducing data dependency, and improving model adaptability to further enhance molten metal tracking. By combining deep learning technology with traditional tracking methods, more accurate and efficient tracking of molten metal can be achieved, thereby improving the overall efficiency and product quality in metal smelting and processing.