*Article* **Self-Sensing Variable Stiffness Actuation of Shape Memory Coil by an Inferential Soft Sensor**

**Bhagoji Bapurao Sul 1, Dhanalakshmi Kaliaperumal 1,\* and Seung-Bok Choi 2,3,\***


**Abstract:** Self-sensing actuation of shape memory alloy (SMA) means to sense both mechanical and thermal properties/variables through the measurement of any internally changing electrical property such as resistance/inductance/capacitance/phase/frequency of an actuating material under actuation. The main contribution of this paper is to obtain the stiffness from the measurement of electrical resistance of a shape memory coil during variable stiffness actuation thereby, simulating its self-sensing characteristics by developing a Support Vector Machine (SVM) regression and nonlinear regression model. Experimental evaluation of the stiffness of a passive biased shape memory coil (SMC) in antagonistic connection, for different electrical (like activation current, excitation frequency, and duty cycle) and mechanical input conditions (for example, the operating condition pre-stress) is done in terms of change in electrical resistance through the measurement of the instantaneous value. The stiffness is then calculated from force and displacement, while by this scheme it is sensed from the electrical resistance. To fulfill the deficiency of a dedicated physical stiffness sensor, self-sensing stiffness by a Soft Sensor (equivalently SVM) is a boon for variable stiffness actuation. A simple and well-proven voltage division method is used for indirect stiffness sensing; wherein, voltages across the shape memory coil and series resistance provide the electrical resistance. The predicted stiffness of SVM matches well with the experimental stiffness and this is validated by evaluating the performances such as root mean squared error (RMSE), the goodness of fit and correlation coefficient. This self-sensing variable stiffness actuation (SSVSA) provides several advantages in applications of SMA: sensor-less systems, miniaturized systems, simplified control systems and possible stiffness feedback control.

**Keywords:** shape memory coil; joule heating effect; self-sensing actuation; variable stiffness actuation; electrical resistance; support vector machine regression model; nonlinear regression model

#### **1. Introduction**

The shape memory coil (SMC) has a larger change in force and controllable stiffness to introduce the structural elastic deformation and to be in tune with a structural load. It provides the actuation to a mechanical structure; actuation with variable load can be sensed by self-sensing the stiffness in the structure via the Shape Memory Alloy (SMA) coil's resistance. This is because of the shape memory effect phenomenon, which is an inherent property present in the nickel-titanium alloy. This inherent property is due to phase transformation from martensite to austenite and vice-versa when subjected to temperature or current. Though the SMA has same chemical composition, atomic weight and mass number, but it is different structure in the austenite and martensite phase.

Until now, none is dedicated to physical sensor or analytical models to sense the stiffness of shape memory alloy or SMA with mechanical structure. Nowadays, the sensing

**Citation:** Sul, B.B.; Kaliaperumal, D.; Choi, S.-B. Self-Sensing Variable Stiffness Actuation of Shape Memory Coil by an Inferential Soft Sensor. *Sensors* **2023**, *23*, 2442. https:// doi.org/10.3390/s23052442

Academic Editor: Simon Tomažiˇc

Received: 4 January 2023 Revised: 7 February 2023 Accepted: 15 February 2023 Published: 22 February 2023

**Copyright:** © 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

function of SMA becomes more important because the high technology for the humanoid robot, industrial automation and medical field is being progressed a lot. It has been found 30 years ago that the electrical resistance of SMA changes during phase transformation of its material [1]. The modelling of SMA resistance to stiffness as compared to the models of temperature to stiffness is very rare in literature. A linear equation only is suggested between stiffness and normalized resistance. Thus, as a need of sensing/measurement and control of stiffness accurately, it must go into details of the topic. In this case, to attenuate the surrounding environmental effect, a robust and simple adaptive control is adopted. The experimental result proved that the stiffness as well as position could be controlled to achieve the desired displacement. The stiffness of the alloy varies depending on its phase. The phase of the alloy can be estimated by measuring its electrical resistance. Electrical resistance is relatively higher in martensite and lower in the austenite phase. Furthermore, a new scheme of stiffness is implemented by considering two feedback inputs- electrical resistance and position [1].

This electrical property is useful for sensing of thermal and mechanical properties like temperature, force and strain of SMA. Since then, many research papers have proved that resistance change is sufficiently linear to measure and control displacement and strain [2]. These works intuitively explained about the relationship between displacements in the SMA wire and its resistance. Another important point is that the SMA actuator exhibits highly nonlinear behavior. Therefore, a neural network is employed to estimate the value of displacement in the SMA from its resistance change. The estimated value of displacement from resistance is used as feedback to control it [2]. The estimation of the state (contraction/displacement) of the SMA wire actuator is done with the help of its resistance. The state of the SMA wire model has been developed by using the concept of unscented Kalman filter (UKF) which uses the measured resistance. The results are compared with the work of the extended Kalman filter and show good accuracy [3]. Accurate self-sensing concept to control the flexures by controlling the SMA wire has been also developed. Then the polynomial model is used to estimate the strain value from measurement of resistance. In addition, the inaccuracies due to the presence of the hysteresis has been overcome by pretension force. By considering standard test signals such as step and sinusoidal signals, performance of the control system has been also tested [4].

The polynomial model accurately enables estimation of the SMA actuator strain by applying an electrical potential across it. The experimental results have shown that the selfsensing model can achieves a small transient error and works effectively. The self-sensing helps to develop to miniaturized devices to perform effectively and efficiently [5] in which a self-sensing concept for control has been well described by adopting an antagonistic SMA wire drive. This drive is tested under different conditions such as pre-strain and duty cycles. Then, the modeling of strain and resistance is derived with the help of a curve-fitted polynomial. It has been also realized that the accurate control of an actuator wire by self-sensing feedback with a hysteresis compensator can be done [6].

From the results of this work, it has been realized that the use of neural networks to characterize the relation between the resistance of SMA wire and its strain is very effective. The great advantage of this concept is that the single SMA wire performs dual tasks as both an actuator and a sensor. This has got great importance when the prime objectives are to reduce the overall weight, size and the cost of the actuator system [7]. Artificial neural network (ANN) is applicable to accurately establish model to develop the relationship between the electrical resistance and manipulator positions. Thus, ANN can estimate the rotary manipulator position accurately. It can be controlled by the variable structure control technique under different conditions. The effect of surrounding temperature on the ability of ANN to predict the manipulator position is thoroughly investigated [8].

It exhibits robust performance with a small tolerance and can operate without being affected by ambient temperature. In the investigation, the authors have suggested an innovative way to calculate the resistance to determine the change in length of a SMA wire. By doing this, it is possible to measure both the voltage across the entire NiTi wire and that

of the fixed-length segment. These two voltages provide direct change in length of the SMA wire. This kind of sensing is used in the feedback control in unknown ambient and loading conditions. This technique is called dual measurement for self-sensing of displacement by resistance change [9]. A self-sensing technique to measure the induced force in SMA wire is developed to control the length. Therefore, it can replace the traditional load cell by the SMA wire which can work for dual purposes as actuator and sensor. A modeling technic of the SMA wire actuator is also investigated for the control of mechanical structures. While designing a controller for the motion of a mechanical structure, a dynamic model of SMA actuator may be needed. So, the relationship between resistance and displacement of SMA that is derived to determine the feasibility of self-sensing in actuator control is investigated in [10]. The stiffness is related to force and displacement linearly as well as nonlinearly according to Hooke's law. This web portal provides information about the basics of stiffness [11]. The physical, mechanical, electrical, and chemical properties are available on the web portal of Dynalloy Inc., and the portal is very helpful for calculations [12].

The sensing of displacement and stiffness without an external sensor is well described in [13]. The works [13] showed little resistance, martensite fraction, and stiffness, but not in depth like the effect of current, frequency, and pre-stress on self-sensing of the stiffness of SMA spring actuator. This enables the sensing of force without a force sensor. The direct stiffness control of the SMA actuator and sensor-less force sensing experiment is conducted successfully. Besides, several benefits of this method include simplicity of mechanism, cleanliness, silent operation, and distributed actuation system i.e., remotability, sensing ability and low driving voltage. The polynomial model of sensing the stiffness of SMA is implemented to see the influence of different activation currents and excitation/switching frequencies. It has been also shown that different activation current and excitation/switching frequencies of power transistors affected to the stiffness resistance characteristics. These experimental modeling and analyses have proved that stiffness and resistance have a sufficiently linear relation which can be easily utilized to control stiffness in a SMA spring actuator and in mechanical structures [14]. The work [14] studied the stiffness and resistance relationship with only two effects: different activation currents and switching frequencies but not the duty cycle and pre-stress. The work presented by [15,16] did not explain about the self-sensing phenomenon/concept of SMA but described modeling of stiffness-temperature and displacement with other parameters like current, temperature, resistance, and force.

A new mathematical function/model of stiffness that shows the hysteresis characteristics between the stiffness and temperature has been developed and its hysteresis characteristics between the stiffness and temperature is verified by experimental data [15]. It has been found that the hysteresis characteristics are affected by different electrical and mechanical parameters such as current, frequency, and pre-stress, respectively. The relation between stiffness and temperature in SMA spring hysteresis is experimentally verified. The hysteresis characteristics' width and height can be controlled by current, frequency and pre-stress.

As the SMA is a highly nonlinear element, its displacement/contraction changes nonlinearly with temperature. It is affected by many electrical and mechanical parameters. Hence, the modeling of the displacement in SMA spring actuator is important. The neural network is the best tool that can easily map one property with others. Therefore, the displacement of the SMA spring can be modeled by ANN and successfully verified by experimental data [16]. In this work, a systematic approach for the implementation of curve fitting models and methods is suggested to achieve an equation that precisely describes the sensor function [17].

It is known that the Support Vector Regression (SVR) technique is normally applied to forecast the tangential displacement of cement concrete dam. Thus, in general, it is tested and verified using Pearson correlation coefficient, mean absolute error (MAE) and mean squared error (MSE) with experimental data [18]. The implementation of SVR is a practical and user-friendly method for creating soft sensors for nonlinear systems. The development of a dynamic non-linear-Auto Regressive (ARX) model-based soft sensor employing SVR is suggested as a heuristic method, in which the ideal delay and order are determined automatically using the input-output data. It is noted here that an Online Support Vector Regressor (OSVR) model is effective to estimate chemical process variable. As mentioned earlier, many works investigated the self-sensing technique to relate position/length/displacement with resistance and its control. The soft sensor developed using SVR in [18–20] achieved excellent performance checked by statistical performance parameters such as MAE, MSE and correlation coefficient. The research work [21] gives the idea about the stiffness of shape memory coil in terms of resistivity and modulus of elasticity. It suggests the resistivity and shear modulus is the best alternative to existing self-sensing methods.

In research article [22], the overall stiffness is adjusted by modifying the shape of the leaf springs. Hence, the geometrical nonlinearity can be used to change global stiffness. The paper [23] method suggested in adapting stiffness in variable stiffness actuator by configuring the fluid circuits, while the humanoid robot is investigated in [24]. It has various interesting features and is a difficult mechatronics structure. Due to the close interdependence of the technological factors, it is challenging to conduct research in a specific direction. A parallel type SMA wire variable stiffness actuator with a synergistically constructed configuration that offers a small size and a wide range of stiffness adjustments in compliant structures is also studied [25]. Additionally, it provides sufficient displacement and force, making it appropriate for applications requiring peculiar soft robotic requirements. The research work [26] proposes a new type of pneumatic variable stiffness actuator (PVSA). It provides expected actuation performance with effective remote stiffness adjustment capability. A novel variable stiffness mechanism is also designed by using specially designed SMA S-spring with different thickness [27]. The actuator stiffness is discretely adjusted by changing the state combination of SMA S-spring with different thickness. The [28] work is self- sensing unique design of sandwich structure comprising active graphene coated glass fabric piezoresistive face sheets bonded to a Nomex™ honeycomb core. The research article [29] explains the design of SMA spring and how to improve the frequency of actuation. The research [30] demonstrates the ability to significantly improve the way a gripper interacts with things that are being handled and offers a path toward developing anthropomorphic grippers. The soft finger's built-in sensor may convey passive proprioceptive feelings of stiffness and curvature. While not altering the mechanics of the robotic movement, it also served as an active jamming element to adjust finger stiffness.

As evident from the literature survey, the self-sensing during actuation of SMA spring is very useful to understand the relationship between the stiffness change and force, displacement, and strain corresponding change in the electrical resistance during the phase transformation. However, so far, a comprehensive study considering several effects such as actuation current, excitation frequencies and duty cycles has not been reported yet. Consequently, the technical novelties of this study are summarized as follows: (a) achievement of the self-sensing behaviour of SMA spring during variable stiffness actuation by experimentation, (b) development of a data driven model of self-sensing variable stiffness actuation based on Support Vector Machine (SVM) regression and nonlinear regression methods, by availing the experimental data, (c) investigation the effect of different excitation/activation currents, switching/excitation frequencies and duty cycle, (d) evaluation of the cycles and pre-stresses on the stiffness-resistance characteristics during the heating cycle of SMA spring by employing SVM algorithm as a soft sensor. From the aspects of the technical novelties, several new characteristics which are significant for the development of self-sensing are found. Some of new findings are given as follows: (i) it is identified that the characteristics of the self-sensing actuation (SSA) are influenced by activation currents, switching/excitation frequencies, duty cycles and pre-stresses, (ii) both SVM regression and nonlinear regression models are acceptable to measure the self-sensing stiffness in SMC actuator during variable stiffness actuation which is experimentally validated with its significance, (iii) it is found that the electrical resistance for all factors is almost in linear

relation with the stiffness of SMC during the heating cycle with a meagre hysteresis, (iv) the resistance of SMC is low under the austenite and high under the martensite phases. It is noted here that compact design of a self-sensing SMA spring/wire actuator device is feasible with the aid of the results achieved in this work.

This paper is organized as follows. After clearly describing the research motivation, literature survey and the technical contributions of this study, an experimental facility and measurement method are presented in Section 2. Section 3 provides the information about SVM regression and nonlinear regression models focusing on the usability, and the basic mathematical relations used to find the resistance and stiffness which are undertaken at various different conditions of activation/excitation current, switching/excitation frequency, duty cycle and pre-stress are given in Section 4. The detailed characteristics on the results regarding to the stiffness response, resistance response and stiffness-resistance characteristic is fully discussed in Section 5 with the brief information about application of nonlinear regression modeling, followed by conclusion in Section 6 where some of benefits achieved form this work such as robust model, cost effectiveness and reliability for compact design of self-sensing actuator.

#### **2. Experimental Set Up**

#### *2.1. Facility*

The study to self-sense the stiffness of the SMA spring during variable stiffness actuation from resistance measurement is highly significant with respect to the quality of device (in terms of accuracy, precision, sensitivity and linearity etc., compactness and cost effectiveness). This experimental study is used to validate Support Vector Machine Regression and Nonlinear Regression model and realized by MATLAB 2020b software. A set-up to run the tests is designed and fabricated and shown in Figure 1 with the help of the photograph. It has following sections: (a) Mechanical Actuation System—It has two guide rods, with two linear bearings on them to actuate the SMA spring biased with an antagonistic tensile passive steel spring. This helps obstacle free movement which is measured with the help of a flap placed between two springs and Keyence—made contactless laser displacement sensor. A force sensor is also connected between the fixed frame and the SMA spring. The complete assembly is fixed in an acrylic frame. (b) Electronic Actuation System—This system consists of on Metal-Oxide-Semiconductor Field-Effect Transistor (MOSFET, TIP-122 ST Microelectronics) with gate resistance (1.5 kΩ and <sup>1</sup> ⁄4 watt) to limit the base current; a source with a rheostat (4 Ω and 8.5 A) and drain is connected to the ground. The SMA spring is connected between the rheostat and the source of power transistor. The current sensor is connected between the ammeter and the rheostat so that it can display the current flowing through the SMA spring. (c) Power Supply— Different power supply systems are required for working of different auxiliary devices and the complete actuation systems. (i) DC regulated power supply—this is required for relay circuit, excitations for current sensor, transistor circuit, and as an input signal to op amp circuits. (ii) Dual Power Supply—Dual power supply provides +/− 15 V and 0.5 A current. This is important for op amp-based amplification circuits to boost physical signals like temperature signal and displacement signal. (iii) AC Power supply is required for bigger auxiliary devices. (d) Instrumentation and Data Acquisition (DAQ) System—These includes current sensor, temperature sensor, miniature force sensor, laser displacement sensor, voltages across SMA spring and rheostat.

Voltage signals from different sensors are converted into proper level (0 to 10 V) through signal conditioning, so that it is compatible with the data acquisition system and stored in the personal computer's memory. Figure 1 also shows the DAQ card used for data acquisition and forwarding it to the computer memory. (e) Shape memory alloy spring and passive steel spring—The SMA spring is manufactured by Dynalloy Inc. (1562 Reynolds Avenue (949) 502-8548 office, Irvine, CA 92614 USA) and their technical specifications are given in Table 1. Other details of the passive steel spring are shown in Table 2. During activation/heating of the SMA spring, the biased tensile spring expands and stores the

mechanical energy. During deactivation/cooling, the bias spring will use the stored energy to pull the SMA spring back to its pre-stress/deform state.

**Figure 1.** Experimental setup.

**Table 1.** Technical specifications of the passive steel spring.


#### *2.2. Experimentation*

The experimental modeling and analysis of self-sensing of the stiffness in the Shape Memory Coil during variable stiffness actuation is performed in the following ways. In the first set of experiments, activation currents (0.8 A, 1.0 A and 1.2 A) are varied by D.C. regulated power supply and by keeping voltage constant. Then different sensor voltages, the voltage across rheostat and voltage across the Shape Memory Coil are recorded in the memory of personal computer via DAQ card through repetitive cycle of switching of the power transistor. This procedure is repeated for aforementioned activation currents by keeping the switching frequency, duty cycle and pre-stress constant. The recorded information is used to predict stiffness by nonlinear regression modeling and validation. It is presented in detail in Section 4.

The recorded instantaneous value is used to determine the resistance and stiffness properties of SMC by use of Equations (6) and (7) for the aforementioned activation currents. In the second set of experiments, the switching frequency of the power transistor is changed e.g., 10 mHz, 20 mHz and 30 mHz by ensuring that all other parameters such as activation current, pre-stress on the SMC and duty cycle of the switching frequency are constant. The instantaneous values of the SMA spring's properties in terms of voltages is continuously recorded in the memory of personal computer via DAQ card. These recorded voltages are converted into proper units of the properties of SMC such as force, displacement, temperature, resistance, and stiffness.

Also, the experimental modeling and validations are explained in detail in Section 4. In the third set of experiments, the pre-stress (100 g, 150 g and 200 g) on SMC is varied by applying more tension with the help of a tensile passive steel spring. All other parameters are kept constant mentioned in the earlier set of experimentation. Similarly, the properties of SMC in terms of voltages are recorded by different sensors and described in detail in Section 4. Some of the properties of SMC e.g., stiffness and resistance are derived with help of Equations (6) and (7), respectively. In the fourth set of experiments, the duty cycle (40%, 50%, and 60%) of switching frequency is varied by adjusting the knob of the function generator such that activation and deactivation of SMC occur smoothly with

help of passive bias tensile spring and explained in detail in Section 4. Basically, SMA works in three different modes of operations as actuators: (i) Free recovery mode means a constant force and variable contraction of SMC. (ii) Constraint recovery mode means a variable force and fixed contraction of SMC. (iii) Work production mode means both force and contraction of SMC changing. The work production mode is the most popular and applicable to practical engineering applications. It is a controllable actuator where both force and displacement vary. The first mode of operation is free recovery where force is constant, and stiffness is a function of displacement and varies. In the second mode, displacement is constant in which stiffness is a function of force and varies [15]. The inferior vena cava filter and eyeglass frame are designed in the free recovery mode of SMA operation. Fasteners, connectors, and hydraulic coupling uses constrained recovery mode in the SMA operation. Circuit breakers, heat engine and actuators are a few applications of the work production mode in the SMA operation [15]. In the experiment, the variable stiffness actuation of the SMA coil is controlled by using currents of 0.8 A, 1.0 A, and 1.2 A, and the corresponding forces, displacements, currents, and voltages are monitored. The data are displayed to investigate the system characteristics after preprocessing.


**Table 2.** Specifications of the SMC.

#### **3. Facility and Experimentation**

#### *3.1. Principle of Self Sensing of Variable Stiffness Actuation by Support Vector Machine Regression*

The regression is statistical tool to model and analyze the relation between one or more than one independent variable and a dependent variable to produce a particular outcome. In other words, the basic idea is to approximate the functional relation between set of independent variable and dependent variable by minimizing risk function which will use prediction error. Kernel: A lower dimensional data set is mapped into a higher dimensional data set using the Kernel function. In essence, a hyperplane is a line that will enable us to estimate a continuous value or aim. Boundary Line: In SVM, a margin is created by two lines other than the hyperplane. The Support Vectors may be on or outside the boundary lines. The experimental data recorded for resistance and stiffness of shape memory coil are denoted as *Xi* and *Yi*.

$$\{ (X\_1, \mathbf{Y}\_1), (X\_2, \mathbf{Y}\_2), (X\_3, \mathbf{Y}\_3), \dots, \dots, \dots, (X\_n, \mathbf{Y}\_n) \} \in \mathbb{R}^N \times \mathbb{R} \tag{1}$$

where, *i* is varying from 0 to *n* and *n* is number of training data points. The goal is to find the function which map the relation between *Xi* and *Yi*. The Support Vector Regressor algorithm approximate the function as,

$$f(\mathbf{x}) =  +b \text{ and } w \in \mathbb{R}^N, b \in \mathbb{R} \tag{2}$$

The one dimensional and multidimensional SVR problem is defined as

$$f(\mathbf{x}) = \sum\_{j=1}^{M} w\_j \ast \mathbf{x}\_j + b \text{ and } w \in \mathbb{R}^M, \ b \in \mathbb{R} \tag{3}$$

$$f(\mathbf{x}) = \binom{w}{b} \binom{\mathbf{x}}{1} = w^T \mathbf{x} + b; \; \mathbf{x}, w \in R^{M+1} \tag{4}$$

where, *w* is weight vector, *b* is bias and *ϕ(x)* is high dimensional data space. The weight vector and bias can be determined from risk function and as,

$$R(\mathbb{C}) = \frac{1}{2} \mid w \mid \: ^2 + \mathbb{C} \frac{1}{2} \sum\_{i=1}^{n} L\_{\varepsilon}(f(X\_i), \: \: \: \: \: \_i) \tag{5}$$

The <sup>1</sup> <sup>2</sup> | *w* | <sup>2</sup> control the function capacity and *C*<sup>1</sup> <sup>2</sup> <sup>∑</sup>*<sup>n</sup> <sup>i</sup>*=<sup>1</sup> *Lε*(*f*(*Xi*), *Yi*) is the error. *C* is regularization constant. The insensitive loss function is defined as,

$$L\_{\mathfrak{c}}(f(X\_i), \, Y\_i) = \begin{cases} \, \mid \, f(X\_i), \, -Y\_i \mid \, -\varepsilon. & \text{when} \quad \vert \, f(X\_i), \, -Y\_i \mid \, \ge \varepsilon \\\, \mid \, 0 \,, & \text{and otherwise} \end{cases} \tag{6}$$

where, *ε* is the boundary line [18].

$$f(X\_i)\_{\prime} = q(x\_i)^T w + b \tag{7}$$

*ϕ(x)* is high dimensional data space.

#### *3.2. Principle of Self-Sensing of Variable Stiffness Actuation by Nonlinear Regression*

Most self-sensing actuation model literature have related SMA in terms of the displacement/strain with self-sensed electrical resistance. To represent the stiffness of SMA in terms of its electrical resistance, there is a need to establish an appropriate and reliable stiffness-resistance model. To find the appropriate mathematical model that expresses the relationship between dependent variable (stiffness) and the independent variable (resistance), a data driven model is used; it is a parallel to mathematical model with self-sensing characteristics i.e., sensing under actuation. So, the data driven model is a function of

the independent variable involving one or more coefficients. The nonlinear regression model with continuous one-to-one mapping is efficient to describe the relation between the dependent variable (stiffness) and independent variable (resistance). The Nonlinear Regression model is preferred as the modeling of the self-sensing actuation phenomenon (the relation of change in electrical resistance to the change in stiffness during the nonlinear thermo-mechanical phase transformation) is not as complex as that of modeling the basic phenomenon of SMA, the thermo-mechanical phase transformation. Also, the Nonlinear Regression model can be used to approximate complex nonlinear phenomena and then the relationship is curvilinear. The *j*th order polynomial model in one variable is given by,

$$k = \beta\_0 + \beta\_1 R\_{\text{suma}} + \beta\_2 R\_{\text{suma}}^2 + \dots + \beta\_j R\_{\text{suma}}^j + \varepsilon \tag{8}$$

where, *Rsma* is resistance of SMA spring, *β*0, *β*1, *β*2,..., *βi*, and *Rsma Rsma*2 *Rsma*3 ... *Rsma* ... *Rsmai*, *i* = 1, 2, 3, . . . , *j* are the effect parameters and ε an error.

The nonlinear regression of sufficiently high degree can always be found that provides a good fit for data. A good strategy should be used to choose the order of an approximate polynomial; keep the order increasing until *t*-test for the highest order term is non-significant. It is called the forward selection procedure to fit the model with experimental data. Also, goodness of fit statistics is used to find the best polynomial. MATLAB function "polyfit" is used to obtain coefficients of the polynomials [3]. Data from the measurement of the force, displacement and voltage sensors is saved in an Excel.csv file and used whenever required for training and testing the model. In the first step, characteristics must be represented as predicted data, response data, and weights. In the second step, nature i.e., shape and specificity of the self-sensing characteristic of the appropriate parameter polynomial model is selected. After fitting the data into a model, its goodness of fit is determined by adopting any of the following two ways: (i) Graphical (ii) Numerical. The plotting residuals and prediction bound aid visual interpretation. Graphical measures allow viewing the entire data set at once, and they easily display a wide range of relationships between the model and data. Numerical measures are more narrowly focused on a particular aspect of the data and often try to compress that information into a single number. In practice, to find the best fit of the sensor's characteristics, the above-mentioned methods are used on extensive experimental data and analyzed [17]. Figure 2 shows the trial-and-error procedure to find the correct polynomial model in comparison with the experimental sensor's characteristics; it can be seen that the third-order and above models match the experimental data.

**Figure 2.** Polynomial model and experimental self-sensing variable stiffness actuation characteristics.

#### **4. Result and Discussion of Self-Sensing Variable Stiffness Actuation**

The influence of different factors such as activation current, excitation frequency, duty cycle, and pre-stress on self-sensing characteristics are observed and presented in this section. The influence of different factors such as activation current, excitation frequency, duty cycle, and pre-stress on self-sensing characteristics are observed and presented in this section. The SMA's electrical resistance is sensitive to compositions, transformation

path and heat treatment [1]. During phase transformation, the crystallographic structure of SMA changes due to heating and cooling. As a result, SMA's electrical resistance changes. This resistance change is useful to measure SMA's stiffness without any physical sensor. Furthermore, it is possible to measure stiffness of the SMA based structure. Table 3 has useful information about SMA material. It is provided by the manufacturer and that the resistance of the SMA wire depends on its length and diameter. It also says that resistance per meter decreases with an increase in wire diameter. The basic relation used to calculate resistance [3,5] and stiffness of the SMC actuator are as follows.

$$R\_{SMA} = \frac{V\_{SMA}}{Vs - V\_{SMA}} \ast R \tag{9}$$

$$k = \frac{d^4G}{8 \ n \ D^5} \tag{10}$$

where, *RSMA* is the resistance of the SMA spring (Ω), *VSMA* is the voltage of the SMA spring (V), *R* is the known resistance (Ω), *VS* is the bias voltage of the MOSFET (V),

*k* is the instantaneous stiffness of the SMA spring (N/m), *G* is the instantaneous shear modulus of the SMC (N/m2), *d* is the wire diameter of the SMC (m), and *D* is the coil diameter of the SMC (m). The instantaneous value of *G* is calculated from force and displacement measurements using transducers.

The first step in the design process is to choose the smallest wire diameter, or "*d*". The force-displacement relationship and cooling performance associated to the actuation frequency are most sensitively influenced by the wire diameter "*d*", which is a dominant design parameter of the SMA coil spring actuator. The SMA coil spring actuator has the smallest material mass and the quickest cooling time when the wire diameter is the smallest.

Iterative calculations are used to determine the coil diameter "*D*". When the shear strain reaches the predetermined value, which is adjusted to be slightly greater than the wire diameter "*d*", the force is calculated. If "*D*" is tiny, the shear stain does not reach its maximum value before the force surpasses the intended value. If so, the calculation is redone with a larger "*D*" The iteration ends and the "*D*" value at the last step is the maximum coil diameter if the force reaches the required value at the maximum shear strain while "*D*" is growing. With the desired stroke and the single coil stroke at the necessary loading condition, the coil number "*n*" is calculated. The displacement gap between the martensite and austenite models is used to determine the single coil stroke. The desired actuation stroke value divided by the single coil stroke yields the coil number "*n*".

**Table 3.** General properties of SMA [1].


So, the initial value of displacement of SMC is assumed to be zero and it is set to zero in the transducers and recorded in the personal computer. This modeling is aimed at the self-sensing phenomenon when the SMA is under variable stiffness actuation i.e., not particularly on the modeling of the basic actuation or shape memory effect or phase transformation of SMA. Moreover, the study is based on the SMA being activated by joule heating, whereby it is controlled by different electrical parameters such as current, frequency and duty cycle and, pre-stress. The change in resistance corresponding to the change in stiffness is determined during the heating phase, and the relation is extracted as a nonlinear regression model and validated by different metrics through experimentation. The model is valid for the joule heating current of 0.7 A to 1.2 A. Figure 3 shows the stiffness characteristics at 0.8 A, 1.0 A and 1.2 A which is when the phase change starts before which the SMA does not display any linear response. But then after 1.5 A, the response is more

linear in characteristic as seen in Figure 3 and also at a current 1.0 A and 1.2 A, respectively. The model and experimental response are compared in the heating cycle when work is completed, specifically between 0.7 A and 1.2 A also, wherein a large change in stiffness and resistance is revealed.

**Figure 3.** Comparison of models of stiffness sensing characteristics at different currents.

#### *4.1. The Effect of Different Activation Currents*

In the first set of experiments, the SMA spring is electrically heated by a 1/100 Hz square wave signal with a constant duty cycle and constant pre-stress. The current of the heating signal is varied, and the data recorded. The current/electrical power affects both the stiffness and resistance of the SMA spring actuator. The effect of changing current is clearly seen in Figure 3. Stiffness-resistance heating characteristics are modeled by the SVR and Nonlinear regression. Figure 3 reveal, that as electrical current increases, the slope of the curve increases, and the experimental characteristics overlap the modeled characteristics. The implementation of both models is done in MATLAB by "polyfit", "polyval" and other built-in functions.

The mathematical Nonlinear regression model between stiffness and resistance during the austenite phase is estimated from the experimental data.

$$k = -1.6446 \ast R\_{SMA}{}^3 + 2.7072 \ast R\_{SMA}{}^2 - 2.0349 \ast R\_{SMA} + 0.9655 \tag{11}$$

$$k = -0.5627 \ast R\_{SMA}^{-3} - 0.0684 \ast R\_{SMA}^{-2} - 0.3944 \ast R\_{SMA} + 1.0032 \tag{12}$$

$$k = 1.1301 \ast R\_{SMA}^{-3} - 0.0543 \ast R\_{SMA}^{-2} - 1.9872 \ast R\_{SMA} + 1.0067 \tag{13}$$

where, *k* is the instantaneous stiffness in N/m and "*RSMA*" is the resistance in ohm of the SMA spring actuator. Three cases are considered to present the data uniformly corresponding to each effect in parameter variations like current, frequency, duty cycle and pre-stress. The Nonlinear Regression model is used to represent self-sensing actuation in particular to relate stiffness with electrical resistance. The big and unusual coefficient of the nonlinear regression model can be reduced by normalizing stiffness and resistance data. The modeled and experimental self-sensing of stiffness of the Shape Memory Coil during variable stiffness actuation agree in terms of quality as Figure 3 has performance metrics such as goodness factor, mean squared error, correlation matrix and root mean square error within the specified range. The root mean square error (RMSE) should be less than 0.80. The goodness factor is out of range in Figure 3 as nonlinearity is present for the phase conversion which has not yet started.

The values of metrics of the model comparison for Figure 3 is mentioned in Table 4; there is large difference in stiffness for the three (maximum values are 75, 145 and 1500 N/m) cases. Table 5 gives the Correlation matrix which displays the correlation coefficients for matching the stiffness (model and experimental) at different independent variables, and the activation current. The matrix depicts the correlation between the possible pairs of values in the table; this tool helps to summarize the dataset, to identify and visualize the match of patterns in the data. From the matrix tables it is observed that, the characteristics at 0.8 A do not match as they are more non-linear than that for the other two higher activation currents.

**Table 4.** Metrics of inferential models at different currents.


**Table 5.** Correlation Coefficient between observed and predicted stiffness at different currents.


#### *4.2. The Effect of Different Excitation Frequencies*

In the second set of experiments, the Shape Memory Coil is electrically heated over a fixed current of 1.2 A of different frequencies (10 mHz, 20 mHz, and 30 mHz) of square wave signal with fixed duty cycle and pre-stress (pre-tension). The self-sensing of stiffness is modeled during the heating cycle only. Both the curves almost agree with each other (modeled and experimental). Figure 4 show the characteristics modeled and experimental plots at different frequencies. The effect of frequencies on stiffness is inversely proportional i.e., stiffness decreases when resistance increase with an increase in frequency. The quadratic mathematical models of stiffness at different frequencies are as follows:

$$k = 1.1074 \ast R\_{SMA}^{\ast} - 1.9230 \ast R\_{SMA} + 0.8835 \tag{14}$$

$$k = 1.5465 \ast R\_{SMA}{}^2 - 2.4596 \ast R\_{SMA} + 0.9701\tag{15}$$

$$k = -0.5693 \ast R\_{SMA}{}^2 - 0.2908 \ast R\_{SMA} + 0.9574\tag{16}$$

The experimental results validated the self-sensing of stiffness of the SMC actuator by measurement of resistance during the heating cycle (austenite phase). Figure 4 reveals the linear relationship between these two properties of the SMA. Also, the resistance change of the SMA spring actuator is higher at a lower frequency and lower at higher frequency over 0 to 1.2 A of electrical power.

**Figure 4.** Comparison of models of self-sensing stiffness characteristics with experimental result at different excitation frequencies.

The modeled and experimental self-sensing of stiffness of the SMC during variable stiffness actuation agree in terms of quality because Figure 4 has the performance metrics such as goodness factor, standard deviation, correlation matrix and root mean square error within the specified range, as seen from Table 6. Also, Table 7 provides the correlation matrix which validates the match for the range of frequency (10 mHz to 30 mHz) though the study is conducted until 0.6 Hz.


**Table 6.** Metrics of inferential models at different frequencies.

**Table 7.** Correlation Coefficient between observed and predicted stiffness at different frequencies.


#### *4.3. The Effect of Different Duty Cycles*

An SVR and Nonlinear regression model is developed to understand the effect of duty cycle on stiffness-resistance characteristics. The comparison of experimental and curve-fitted model is simulated by the MATLAB® program, the characteristics for different duty cycles (40%, 50% and 60%), at a constant current of 1.5 A and constant frequency of 20 mHz showed that they almost agree with each other. The nonlinear regression model of stiffness at different duty cycles with resistance change as independent variable are as follows:

$$k = 0.5778 \ast R\_{SMA}^{\ast} - 1.5332 \ast R\_{SMA} + 0.9982\tag{17}$$

$$k = 0.3808 \ast R\_{SMA}^{-2} - 1.3471 \ast R\_{SMA} + 1.0264 \tag{18}$$

$$k = -0.2699 \ast R\_{SMA}{}^2 - 0.7399 \ast R\_{SMA} + 1.0239 \tag{19}$$

The effect of duty cycle on stiffness-resistance characteristics are linear and useful in controlling stiffness effectively. The comparison between the experimental and modeled characteristics at different duty cycles are presented in Figure 5 and mathematically represented by quadratic (second-order polynomial) Equations (17)–(19). The modeled and the experimental self-sensing of stiffness of the SMC during variable stiffness actuation agree in terms of quality as Figure 5 contains performance metrics such as goodness factor, mean squared error, correlation matrix and root mean square error within the specified range, as seen in Table 8. Table 9 also gives the correlation matrix which validates the match for a range of duty cycle (40% to 60%), though the study is conducted from 20% to 80%.

**Figure 5.** Comparison of models of self-sensing stiffness characteristics with experimental results at different duty cycles.




**Table 9.** Correlation coefficient at different duty cycles.

#### *4.4. The Effect of Different Pre-Stresses (Pre-Tension)*

The effect of pre-stress on stiffness-resistance characteristics is developed as mathematical models for different pre-stresses (100 g, 150 g and 200 g) at a constant current of 1.2 A and constant frequency of 10 mHz. The nonlinear regression model of stiffness at different duty cycles with resistance change as independent variable are as follows:

$$k = -2.9194 \ast R\_{SMA}^{-3} + 5.3974 \ast R\_{SMA}^{-2} - 3.3947 \ast R\_{SMA} + 0.8517\tag{20}$$

$$k = -2.4860 \ast R\_{SMA}{}^3 + 4.6766 \ast R\_{SMA}{}^2 - 3.0950 \ast R\_{SMA} + 0.8585 \tag{21}$$

$$k = -0.9385 \ast R\_{SMA}^{-3} + 1.5918 \ast R\_{SMA}^{-2} - 1.5918 \ast R\_{SMA} + 1.0084\tag{22}$$

The comparison between experimental and modeled characteristics at different prestresses found in Figure 6 and mathematically represented by third order polynomial Equations (20)–(22). The effect of pre-stress on stiffness-resistance characteristics is highly nonlinear and difficult to model in comparison with those on the effect of current, frequency and duty cycle. Figure 6 reveals this and it is found from the correlation matrix that stiffness and resistance of the SMA spring at different stresses do not have strong statistical correlation. Table 10 validates the two models, compares the two models with each other and gives the information about the accuracy of prediction by using experimental and predicted data with the help of different metrics. There is a perfect correlation of each variable with itself as seen from Table 11.

**Figure 6.** Comparison of Models of Self-sensing Stiffness characteristics with experimental results at different pre-stresses.

**Table 10.** Metrics of inferential models at different pre-stresses.


**Table 11.** Correlation coefficient at different pre-stresses.


#### **5. Investigation of Stiffness Characteristics of the SMC Actuator**

*5.1. Effect of Current on Stiffness-Resistance Characteristics*

With the help of four sets of experiments, the stiffness-resistance characteristics of the SMC actuator are analyzed to explore its self-sensing capability. Data recorded from the first set of experiments are used to plot and analyze. Resistance response to the heating cycle is plotted for 50 s; it found that as activation current increased, resistance decreased and that the change of resistance decreased over a period of time as shown in Figure 7. Stiffness response to different activation currents found that at 1.2 A, stiffness increased very rapidly in comparison to the other two activation currents. Excitation frequency is chosen as 10 mHz, as enough time is available to relax/deform the SMA spring and avoid residual strain. As excitation frequency increased, the time to complete the cycle is reduced e.g., two heating-cooling cycles occurred at 10 mHz and 6 cycles at 50 mHz. Data of force, displacement, the voltage across fixed resistance and the SMA spring are recorded for 3 min and are also saved in computer memory via 1408FS plus DAQ card with a sampling frequency of 2 Hz. At 0.8 A stiffness is less in value, compared to the other two activation currents (1.0 A and 1.2 A) as the SMA spring does not completely transform from martensite phase to the austenite phase. Corresponding resistance (Ω) is recorded in terms of its voltage and plotted in Figure 7; at 0.8 A. It is observed that resistance change is larger than that at 1.2 A. Responses for the three different currents are plotted and shown in Figures 8 and 9. Figure 8 shows stiffness variations at different currents (0.8 A, 1.0 A and 1.2 A). Figure 9 shows a linear relation between stiffness and resistance. The minimum and maximum values of resistance at these activation currents and their respective stiffness values are shown in Table 12.

**Figure 7.** Resistance of SMA spring actuator at different current.

**Figure 8.** Stiffness of the SMA spring actuator at different currents.

**Figure 9.** Stiffness − Resistance characteristics of the SMA spring actuator at different currents.

**Table 12.** Resistance and Stiffness at different currents.


#### *5.2. Effect of Frequency on Stiffness—Resistance Characteristics*

Data recorded from the second set of experiments are used to plot and analyze. Self-sensing actuation characteristics of the SMA spring actuator is obtained for varied frequencies from 10 mHz to 30 mHz keeping activation current constant at 1.2 A; Stiffness is

determined and plotted as shown in Figures 10–12. When the excitation frequency of PWM signal is increased beyond 50 mHz, the number of cycles is reduced to less than one, also the heating and cooling cycle frequency (mechanical cycle) did not match the excitation frequency (electrical cycle), subsequently, the SMA spring would not completely contract or deform. Some significant observations are arrived at from these plots: (i) The excitation frequency has a significant effect on stiffness and resistance of Shape memory Spring: At a higher frequency, resistance change is higher and at a lower frequency, the resistance change is lower. (ii) The effect of frequency on stiffness change of the SMA spring actuator is converse to resistance change. (iii) Overall linear relationship exists between resistance and stiffness. (iv) The Resistance and Stiffness change from minimum to maximum values at different frequencies is in Table 13.

**Figure 11.** Effect of frequency on Stiffness of the Shape memory spring.

**Figure 12.** Effect of different frequency on Stiffness − Resistance Characteristics of the Shape Memory spring actuator.


**Table 13.** Stiffness and resistance at different frequencies.

#### *5.3. Effect of Pre-Stress on Stiffness—Resistance Characteristics*

At a constant current (1.2 A) passing through the SMA spring and constant frequency (10 mHz) of excitation current, stiffness is determined for pre-stress which is varied from 100 g to 200 g. An increase in pre-stress beyond 200 g would not allow complete contraction (bias force is higher) and would not completely deform below 100 g for the requirement of restraining/pulling force [15]. It is learnt from Figures 13 and 14 that stiffness increased, and resistance decreased with an increase in pre-stress. Figure 15 provides information about the stiffness sensing characteristics at different pre-stresses, which reveal a large change in stiffness at a lower value of resistance of the SMA spring actuator.

**Figure 13.** Resistance response at different Pre-Stresses.

**Figure 14.** Stiffness response at different Pre−Stress.

**Figure 15.** Stiffness-Resistance characteristics at different Pre-Stresses.

#### *5.4. Effect of Duty Cycle on Stiffness—Resistance Characteristics*

Similarly, for different duty cycles (40%, 50%, and 60%) at 1.5 A and 20 mHz, the resistance response, stiffness response and stiffness—Resistance characteristics of the SMA spring actuator are obtained and presented in Figures 16–18 respectively. Resistance values are smaller in comparison to the effect of current, frequency and pre-stress. The change in resistance is higher for higher duty cycles and lower for lower-duty cycles. Change in stiffness is also higher for a higher duty cycle and lower for a lower duty cycle; stiffness is higher in comparison with a lower duty cycle due to the availability of minimal time to completely heat the SMA spring. Figure 16 presents the resistance variation due to changes in duty cycles and Figure 17 corresponds to the stiffness variations. As the duty cycle increased, stiffness also increased some extent. Figure 18 shows the characteristics at different duty cycles, indicating a large change in stiffness with a large change in the resistance of the SMA spring actuator at a higher duty cycle. It also, proved that at higher stiffness, resistance is also higher but within the specified limit of duty cycle. Table 14 summarizes stiffness and resistances at different pre-stresses and their effect. Table 15 depicts the effect of duty cycle on stiffness and resistance, which is more on stiffness and less on resistance.

**Figure 16.** Resistance response at different duty cycles.

**Figure 17.** Stiffness response at different duty cycles.

**Figure 18.** Effect of duty cycles on the Stiffness—Resistance characteristics.





Experimental data is collected for repeated cycles (3 consecutive cycles), and the responses are similar/not with much deviation. One set of data is used for the plotting and analysis of the stiffness-resistance characteristics, the usable form of the data is through the use of an average function and normalization; the other set of data are used for the validation. The highest change in resistance corresponding to the change of stiffness with regard to the effect of the influencing factors like activation current, excitation frequency, pre-stress and duty cycle are presented in Table 16. It is observed that a change in resistance of the SMA spring in the configuration switches is the highest (0.3516 Ω) during the change in pre-stress and stayed the lowest (0.0160 Ω) during the change in duty cycle. The purpose

of this study is to use the suggested technique during control of actuation in the SMA spring by using an appropriate controller. The polynomial models are appropriate to the relevance to the factor of consideration and are able to accurately predict the stiffness equivalent to that obtained through experimentation by the measurement of change in electrical resistance. The level of predictability is high for the factor's activation current, excitation frequency and duty cycle but low for pre-stress and low value of current (0.8 A) due to nonlinear characteristics of self-sensing of the SMA spring; the statistical analysis is presented in Tables 4–11. Table 17 attests that the polynomial model, predicted accurately at different activation currents, excitation frequencies and duty cycle but at the pre-stress.

**Table 16.** Resistance and stiffness values at different Influencing factors.


**Table 17.** Performance of SVR and nonlinear regression model prediction.


#### **6. Conclusions**

In this work, an experimental facility is developed to determine the electrical resistance of the Shape Memory Coil (SMC) actuator that is biased by a tensile steel spring under self-sensing variable stiffness actuation. The SVM regression model is constructed based on experimental data (Expert Knowledge) and provided excellent performances. The performance of the SVM regressor model is verified by a correlation coefficient, mean square error (MSE), root mean square error (RMSE) and goodness of fit (R2). The developed SVM model showed an excellent result of prediction in comparison with the nonlinear regression model and experimental data. The experimental analysis has proved that the stiffness of the SMA is sensed from its resistance change. While the stiffness is changed due to different activation currents/joule heating, excitation frequencies, pre-stresses and duty cycles, the stiffness of the SMC is successfully determined as the variable stiffness actuator. Among many new findings from this work, the most interesting result is that the stiffness of the SMA spring can be measured without knowing the activation current and initial geometry or configurations of the SMC. This is possible from the realization of both SVR and nonlinear regression models of the stiffness using the electrical resistance of the SMC during austenite phase transformation. The responses achieved from two models are compared to the experimental response showing both models would harness the self-sensing capability of the SMC actuator. In addition, it has found from this work that the effect of frequency and duty cycle is more linear when compared to the other two parameters of current and pre-stresses. It has been concluded from this work that from the practical view of point, the self-sensing of stiffness of SMC can reduce the number of sensors for making application systems associated with shape memory alloys. For example, one stiffness sensor can use for two sensors of force and displacement. Therefore, it is self-explanatory justifying that the self-sensing technique gives birth to sensor-less control systems which are relatively cost effective, and hence the overall system becomes compact in comparison with traditional control systems having dedicated many sensors. Therefore, it is expected that the proposed self-sensing variable stiffness actuation method can be applicable to many control systems including the grasping force of robot grippers, surgical SMA wire of biomedical sciences, vibration and dynamic motion control of flexible structures in aeronautical fields such as morphing control and health monitoring control

system using SMA wires Associated magnetic coils. It is finally remarked that some benefits achieved from this work will be demonstrated by applying to robot gripper systems in near future.

**Author Contributions:** B.B.S. and D.K. contributed equally to carry out the research work under the title "Self-Sensing of variable stiffness actuation of Shape Memory Coil by an Inferential Soft Sensor". They conceived and designed the analysis of self-sensing variable stiffness actuation by support vector regression and nonlinear regression method. S.-B.C. contributed to clearly addressing the technical novelty of this work, and he carefully checked and revised all equations and figures to improve the article quality. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The raw/processed data required to reproduce these findings cannot be shared at this time as the data also form part of an ongoing study. In the future, however, the raw data required to reproduce these findings will be available from the corresponding authors.

**Conflicts of Interest:** The authors declare no conflict of interest. The authors also declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

#### **References**


**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

**Yuxuan Li 1,2, Weihao Jiang 1, Zhihui Shi <sup>1</sup> and Chunjie Yang 2,\***


**Abstract:** In complex industrial processes such as sintering, key quality variables are difficult to measure online and it takes a long time to obtain quality variables through offline testing. Moreover, due to the limitations of testing frequency, quality variable data are too scarce. To solve this problem, this paper proposes a sintering quality prediction model based on multi-source data fusion and introduces video data collected by industrial cameras. Firstly, video information of the end of the sintering machine is obtained via the keyframe extraction method based on the feature height. Secondly, using the shallow layer feature construction method based on sinter stratification and the deep layer feature extraction method based on ResNet, the feature information of the image is extracted at multi-scale of the deep layer and the shallow layer. Then, combining industrial time series data, a sintering quality soft sensor model based on multi-source data fusion is proposed, which makes full use of multi-source data from various sources. The experimental results show that the method effectively improves the accuracy of the sinter quality prediction model.

**Keywords:** multi-source data fusion; sintering quality prediction; image feature extraction; keyframe extraction

#### **1. Introduction**

In the process industry such as the sintering process, process monitoring is mainly based on conventional sensors which measure the temperature, pressure, flow and other data of the process. The measurement of quality indicators is generally offline testing and it is difficult to achieve online analysis. At present, some online sensors that meet the needs have been developed and applied to some processes with relatively simple reaction mechanisms.

The most important quality index of the sinter is the FeO content of the sinter, which can reflect the reducibility and strength of the sinter. The reducibility of iron ore raw material is an important index for the later blast furnace ironmaking, which determines the furnace conditions and the adjustment of various parameters.

With the increase in FeO content, the output of molten iron in the blast furnace will decrease. However, we cannot simply pursue low FeO content, because FeO content also determines the strength of the sinter. The low content will reduce the strength of the sinter, damage the morphology and increase the proportion of powder. This will hinder the rise of gas flow in the furnace body of the blast furnace, affect the permeability of the material column, make it difficult for the furnace to run smoothly and reduce the output [1].

The common detection method is offline laboratory tests. The operator takes the sintered sample on the conveyor belt and sends it to the chemical composition analysis room. Potassium dichromate titration is often used for offline laboratories and the operation is cumbersome. This method necessitates grinding the sample. If the sample is difficult to dissolve, the analysis results will be poor. However, in general, the method has high accuracy and is the common chemical analysis method for FeO of sinter in iron and steel

**Citation:** Li, Y.; Jiang, W.; Shi, Z.; Yang, C. A Soft Sensor Model of Sintering Process Quality Index Based on Multi-Source Data Fusion. *Sensors* **2023**, *23*, 4954. https:// doi.org/10.3390/s23104954

Academic Editor: Simon Tomažiˇc

Received: 19 April 2023 Revised: 16 May 2023 Accepted: 16 May 2023 Published: 21 May 2023

**Copyright:** © 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

enterprises [2]. Another test method is X-ray diffraction, which is still an offline detection method. The ore sample to be measured and the supporting internal standard reagents need to be dried in an oven at 105 ◦C for 2 h and cooled to room temperature before they can be taken. The powder must then be carefully ground to a uniform and consistent level; otherwise, errors may be introduced [3].

The above offline tests are relatively accurate; however, they are time-consuming, often taking three or four hours to produce a result, and the labor and material costs of the tests are high. Whereas the process of sintering ore from batching to completion of sintering takes about one hour, the test results obtained after three or four hours have a significant lag. If the test results do not meet the process requirements, only the current working condition parameter settings can be adjusted to compensate. This is a great challenge for line operators who want to obtain the control objective of consistently good sinter quality.

Therefore, it is urgent to develop online measurement means of sintering quality indicators. Moreover, sophisticated online measurement means are the basis for building an automatic control system for sinter production. At present, online measurement means mainly include magnetic analyzers and online measurement devices. Shougang Group introduced the magnetic analyzer of the Belgium company to detect via cutting the magnetic induction line when the iron ore passes and causing the change of magnetic field current [4]. However, this method has strict requirements for samples and it also needs regular calibration and calibration by the instrument side. Shougang Group reported that the measurement accuracy of the magnetic induction coil was poor because the particles of the measured sinter samples were too small and there were many powder particles and the problem could not be solved by the equipment manufacturer's personnel again.

As more and more industrial data can be recorded with the use of automated systems in factories, data-driven soft measurement methods are emerging as viable solutions for industrial measurements. Li et al. used LSTM neural networks with self-encoders to construct a prediction model for FeO content [5,6]. Yan et al. proposed a denoised selfencoder framework for predicting sintering endpoints [7]. Yang et al. used a hidden variable model to model sinter quality index [8].

With the development of industrial cameras and image algorithms in recent years, image-based online soft sensors in the field of industrial inspection have gradually gained the attention of academia and industry [9]. Usantiaga proposed a temperature measurement system for the sinter cooling process based on infrared thermal imaging technology [10]. Jiang combined the mechanism with the image characteristics acquired via the infrared imager and proposed a method for measuring the polymorph of FeO content in sinter based on the heterogeneous characteristics of infrared thermal images [11]. However, this method uses the fuzzy classification labels obtained via mechanism analysis and the label accuracy required via regression analysis is insufficient. In addition, the above study used BP neural networks as regressors, which made it difficult to obtain time-series information. And the shallow feature extractor used in this study, cannot obtain deep feature information of the image. In recent years, deep learning methods have been increasingly applied in the image field, such as deep convolutional network models such as ResNet [12,13], achieving better results than shallow networks. This is also one of the motivations for the research in this paper.

This paper combines the low-resolution images of a thermal imaging camera with the high-resolution images of an industrial camera to achieve more accurate access to critical information. Because there are too few assay labels, it is difficult to detect in real time, and more quality variable labels need to be predicted. The work in this paper relies on experts to calibrate a batch of label data and obtains a better database. On this basis, a sintering quality prediction model based on multi-source data fusion is proposed and video data collected via industrial cameras and thermal imagers are introduced. Firstly, the keyframe information of the tail video of the sintering machine is obtained via the keyframe extraction method based on the feature height. Secondly, using the shallow layer feature construction method based on sinter layering and the deep layer feature extraction

method based on ResNet, the feature information of the image is extracted from both the deep layer and the shallow layer. Then, the industrial time series data information is fused and the sintering quality prediction model based on multi-source data fusion is designed, which fully extracts the heterogeneous data information from multiple sources. Finally, the method is applied in a practical industrial case. The experimental results show that the model can effectively improve the accuracy of the sinter quality prediction model.

The main contributions of this paper are as follows:


The remainder of this paper is organized as follows. In Section 2, the characteristics of the sintering process and quality variables are analyzed. The method and model proposed in this paper are introduced in Section 3. Then, the proposed method is verified via the actual production process data in Section 4. Section 5 summarizes the full text and puts forward the new prospect and future work direction.

#### **2. Characteristics of Multi-Source Data in Sintering**

#### *2.1. Description of the Sintering Process and Test Data*

A sintering process is shown in Figure 1. Before the sintered ore is fed into the blast furnace, it is divided into several processes: proportioning, mixing, sintering, crushing, screening and cooling. There are many different types of raw materials used for sinter production. There are more than 10 different bins, consisting mainly of iron ore fines, fuels, fluxes and some additional raw materials. A reasonable material ratio should be developed based on the different compositions of the raw materials, the quality requirements of the sinter ore and the quality requirements of the blast furnace ironmaking. The sintering ore mixing process requires full mixing of the components to obtain a mixture with a uniform and stable chemical composition, while adding water to obtain a good granularity and the necessary material temperature and to improve permeability. The mixture is fed into the belt sintering machine for production and after the sintering is completed, crushing, screening and cooling are required. The finished sintered ore obtained is sent to the ironmaking plant as raw material for blast furnace ironmaking. The sintering data collected contain operational variables, condition variables and quality variables. The quality variables depend on manual testing.

**Figure 1.** Schematic diagram of image acquisition system of sintering process.

#### *2.2. Sintering Image Data Acquisition*

Due to the scarcity of laboratory data, the sintering process information is not perfect. To better obtain information on the sintering process, an industrial camera and a thermal

camera were set up at the observation port at the end of the sintering machine. A schematic diagram of the sintering machine image acquisition is shown in Figure 1. The industrial visible light camera has a resolution of 1920 × 1080 and the captured images are shown in Figure 2. The thermal imager has a resolution of 640 × 480 and the captured image is shown in Figure 3.

The captured information is uploaded to the database for storage via the industrial fieldbus. Live video information is displayed on a display screen in the central control room for the controller to view or can be downloaded to a cloud server via a remote connection.

**Figure 2.** The visible light image acquired via the sintering system.

**Figure 3.** Thermal image acquired via the sintering system.

#### *2.3. Analysis of Sintering Image Data*

At present, most of the quality indexes of sinter are obtained via laboratory analysis. In the process of sinter production, due to the lag of laboratory analysis, the actual working conditions often depend on expert experience to judge. During the sintering process, the internal state of the sinter is not visible and only the topmost surface is exposed. However, the topmost part is the part that is ignited and sintered first. Therefore, on the sintering machine with a length of 90 m, most of the length is in a sintered and finished state. Only the flatness information about the ore laying can be observed from the top and the key sintering state information cannot be obtained. Fortunately, once sintering is complete, the sintered ore reaches the end of the trolley and falls naturally into the collection bin. After the last piece of sintered ore has fallen and before the next piece of sintered ore falls, we can observe the flame information in the sinter section. There is a flame viewing port at the end of the sintering machine. Experts with rich production experience can analyze the range of the FeO quality index of the sinter at the moment by observing the flame red layer data at the end of the sintering machine. Through such manual judgment, it is possible to rely on expert experience to help judge and control the process in the absence of sufficient test indicators. However, there are still significant limitations to this method, with the subjective errors of manual judgement being high and subject to the harsh conditions of the industrial site. In harsh conditions such as high temperatures, loud noises, vibrations and dust, it is difficult for workers to carry out observations for long periods of time, making it difficult to replace conventional observation with fire observation as an aid. Therefore, in the actual production process, it is necessary to develop stable measurement means to replace expert manual observation. Experts can make judgements by observing flame images. Therefore, in this experiment a camera is set up to capture the images and learn information and it is feasible to gauge experts' levels of experience and knowledge through soft measurements.

Due to the different sintering states of raw materials in different directions and different temperatures, the flame colors presented by them are also different. The surface of the black part is sintered and the temperature is reduced to below 300 ◦C, resulting in dim color and unclear observation. The red fire layer is the burning part of the sinter. It is generally located in the lower part of the sinter, mainly in red, and the temperature is about 600 ◦C. The stomatal layer is the brightest part of the image. Because the temperature exceeds 800 ◦C, it appears yellow and white in the image. The sinter will fall periodically at the tail of the sintering machine. When a batch of sintered ore reaches the last wind box at the end of the sintering machine, it will break and fall with the rest of the ore still on the trolley due to the loss of the support of the trolley. At the moment of falling, the flame information of the fault is very clear. After a short time, the falling of ore will raise a large amount of dust and cause vibration at the same time, making the image captured via the camera blurred and accompanied by shaking. It is difficult to obtain accurate flame information from the image at this time. Therefore, the system needs to screen out the clear images at the moment of falling as the analysis sample set. Experienced workers can judge the FeO content of the sinter at this time by observing the section.

#### **3. The Soft Sensor Model Based on Multi-Source Data Fusion**

*3.1. Keyframe Extraction from Video Image of Sintering Machine Tail*

In this experiment, the monitoring video is obtained from the sintering monitoring system for preprocessing. Since the original color image is three-channel RGB data with a data dimension of 16.77 million, the processing is relatively complicated and the calculation pressure is relatively large, so it is grayed first. The grayscale calculation formula is as follows:

$$Gray = Blue \times 0.114 + Green \times 0.587 + Red \times 0.299 \tag{1}$$

where *red*, *green* and *blue*, respectively, represent the values of red, green and blue channels and gray is the gray value.

According to the monitoring video, the falling of sinter is a periodic process. When a batch of ore arrives at the tail, the front part breaks and falls, exposing the red fire layer of the section. The image at this time is the keyframe required for the experiment. After that, this batch of ore will continue to fall and its shape will change in the air and at the same time it will raise smoke and dust, which is unfavorable for observing its sintering characteristics. Therefore, we need to obtain these keyframes in the video as our input. The inter-frame difference method is often used for the algorithm of acquiring the keyframe of the target motion when video monitoring the moving target [14]. The algorithm performs differential processing on continuous images in continuous time. The difference in the grayscale is calculated by subtracting the pixel values of different frames. Then, the threshold value of the gray difference is set. When the value exceeds the threshold value, this frame is selected as the keyframe, indicating that the monitoring target has changed greatly from the previous time at this frame time. When the inter-frame difference method is used to monitor the red layer at the tail of the sintering machine, the brightness difference between different frames can be used as the threshold control to select the keyframe. Since the falling of sinter is a periodic process, when the current ore falls to the bottom and the new batch of ore has not yet broken down, the brightness of the image is the lowest. After a new batch of ore appears, the image information will increase rapidly. Through this characteristic, the sinter keyframe extracted based on the inter-frame difference intensity is the required cross-sectional image. For each frame, the difference is made with the previous frame to obtain the different strengths between the frames and the drawing is as shown in Figure 4. After smoothing, each extreme point can be extracted as our keyframe and the result is shown in Figure 5.

However, the accuracy of the inter-frame differencing algorithm depends on the choice of threshold values. In addition, there are some abnormal states during the falling process of the sinter. The sinter disintegrates in the air due to collision and other reasons, resulting in sudden changes in image intensity. As shown in Figure 6, abnormal frames are extracted and need to be manually removed during calculation.

**Figure 4.** Graph of raw inter-frame difference intensity.

**Figure 5.** Graph of smmoothed inter-frame difference intensity.

**Figure 6.** Abnormal frames extracted via the inter-frame difference algorithm.

Because of the limitation of the algorithm based on inter-frame difference, a more intuitive keyframe extraction algorithm based on feature height is proposed in this experiment.

First, mask segmentation based on the gray threshold is performed on the image that has been converted to gray level by mask and the segmented image is the red fire layer area to be monitored. Next, the contour of the red fire layer is obtained by using the findcontours algorithm. According to the contour, the circumscribed rectangle is fitted to represent the current sintering red layer. The center of gravity of this rectangle is the feature height. The feature height of the video collected in this experiment is shown in Figure 7 and the periodic feature height fluctuation diagram obtained after smoothing is shown in Figure 8.

**Figure 7.** Plot of feature height.

**Figure 8.** Plot of smoothing feature height.

Because the sinter falls periodically, the characteristic height of the red layer of the sinter section is at the highest point when the red fire layer we need has just appeared. Therefore, according to the peaks extremum algorithm, the maximum value of the feature height is obtained and its abscissa is the index of the required keyframe, that is, the image keyframes when the red fire layer has just appeared. The keyframe is extracted via the algorithm based on the characteristic height of the red layer of the sinter and then saved, as shown in Figure 9. The inter-frame intensity difference method of keyframe extraction is disturbed by anomalous image intensity variations, as shown in Figure 6. The feature height method, designed in this paper, is a combination of specific processes where the intensity changes abruptly, but the feature height does not change abruptly, so keyframes are extracted more accurately.

**Figure 9.** Keyframes extracted via the characteristic height algorithm.

#### *3.2. Construction and Extraction of Image Shallow Features*

Using the video keyframe extraction algorithm for the red layer at the end of the sinter in the previous section, several image keyframes were obtained for this experiment. For each keyframe image, the sintered red layer appears at the same position at the top. This location is our region of interest (ROI). First, we fix the region and calculate the area of the region as a fixed observation window. Again, the keyframe is segmented by using a mask based on a grey threshold. Based on practical industrial experience and mechanisms, sintered ore has different sintering states and corresponding temperatures. Assisted by an infrared thermal imager for temperature measurement, the layered temperature is converted into an image threshold. Four different thresholds are chosen, as shown in Table 1.

**Table 1.** Image threshold selection table.


According to the threshold segmentation, four regions are obtained and then their contours are obtained respectively and the area around each contour is calculated as a feature. To better extract the feature information of the image, the area ratio of the area surrounded by each contour to the ROI is further calculated. The proportion information represents the information of the red fire layer and reflects the distribution characteristics and temperature characteristics of the sintering fault, to calculate the shallow layer characteristic information of the ROI area. The shallow layer features and time series data features of the red layer image at the tail of the sintering machine are extracted and the correlation is shown in Figure 10. Features 0, 1, 2 and 3 in the figure are shallow features, while the remaining features are features of industrial time series data. The shallow features are concatenated with the time series data features for fusion.

**Figure 10.** Correlation between shallow features of sintering machine tail image and time series data features.

#### *3.3. Deep Feature Extraction Based on ResNet*

Convolutional neural network (CNN) is a neural network modeled on human visual mechanisms. When humans first saw an image, they could not judge the whole image. Instead, different nerve cell receptive fields transformed the image into different features for recognition. Similar to the receptive field in human visual cortex cells, CNN extracts the features of images through convolution kernels. The convolution kernel successively crosses the picture with the set convolution step in order and calculates the convolution results of the elements of the picture in the receptive field. As a computer, the ability to recognize images is not as good as human beings. After an image is simply rotated, the matrix information stored in the computer is completely different. Therefore, the convolution kernel is designed to extract features for computer image recognition by imitating the human visual perception field. After convolution, multi-level features such as color and contour can be extracted from the original image to help identify the image [15]. For images that cannot be recognized via single-layer convolution, multi-layer convolution needs to be stacked to extract deep features. However, after convolution, the dimension of the original image will be increased. Simply stacking convolution layers will cause the dimension to be too high and it is difficult to train the depth network. Therefore, it is necessary to perform a pooling operation after convolution. The structure is shown in Figure 11. By calculating the mean value or the maximum value of the rectangular window of the specified size to replace the original rectangle, dimension reduction can be completed and the function of nonlinear mapping can be achieved. The stacking of convolution and pooling layers has a good effect within 20 layers. However, after increasing the number of layers of the network, the performance of the model does not improve, but will decline. when the network depth is increased, the network performance will neither improve nor even deteriorate [16].

**Figure 11.** Schematic diagram of CNN structure for identifying numbers.

To solve this problem, He et al. proposed a deep residual network (ResNet), which sets a residual connection between every two layers [12]. Similar to the circuit short-circuit mechanism, the residual connection method constructs an identity map, which enables the network to have the ability of layer hopping transmission. When the number of stacked layers is too deep, the weight of training can select layer hopping connection, as shown in Figure 12. The original function mapping is expressed as H(*x*). Here, we build another map F(*x*) = H(*x*) − *x* that is used to fit stacked networks. At this time, the original mapping is transformed into H(*x*) = F(*x*) + *x*, which can be achieved by a short circuit connection. Such processing does not increase the number of additional parameters, nor does it increase the computational complexity, and it is still easy to train.

**Figure 12.** Schematic diagram of residual connection structure.

The network achieved good results in Imagenet image recognition. For the deep stacked network, the residual connection block set between every two layers can be expressed as:

$$\mathbf{y} = \mathcal{F}(\mathbf{x}, \,\{\mathcal{W}\_i\}) + \mathbf{x}. \tag{2}$$

*x*,*y* are the input vector and output vector of the residual layer, respectively. F(*x*, {*Wi*}) is a representation of the residual map that needs to be learned.

The trained network classifier ResNet-50 is used in this experiment and the 1000 output of its classification is used as the deep feature fixed extractor of the experiment. A total of 1000 deep-seated features were obtained through the ResNet model and their correlation was analyzed, as shown in Figure 13. The features were selected with the correlation coefficient greater than 0.25 as the deep features of the red layer image at the tail of the sintering machine, as shown in Figure 14.

**Figure 13.** Schematic diagram of deep feature correlation extracted via ResNet model.

)HDWXUHV

**Figure 14.** Schematic diagram of selected deep feature correlation.

#### *3.4. Prediction Method of Sintering Multi-Source Data Fusion*

Because the data information from many different sources can not be directly extracted and utilized, we propose a prediction method of the sintering process based on multisource data fusion. The shallow features obtained via keyframe extraction, threshold segmentation, edge detection and feature construction and the deep features obtained via ResNet model are connected to obtain our deep and shallow fused image features. The time series data information from the process is processed using the encoder–decoder dynamic time features expanding and extracting method (DTFEE) based on the LSTM network [5,6]. The characteristics of the image, together with the characteristics obtained from the time series data information of the process, are used as the input of the encoder–decoder model to train the predicted output of the quality variable. The block diagram of the multi-source data fusion prediction method combined with deep and shallow image features is shown in Figure 15.

**Figure 15.** Diagram of Sintering prediction model based on multi-source data fusion.

#### **4. Case Study on Sintering Process**

#### *4.1. Introduction to the Multi-Source Dataset of Sintering Process and Settings*

The images used for the soft sensor of the sinter are collected via the thermal imager and industrial camera, respectively. Video data were collected from 8 December 2021 to 10 December 2021. The acquisition resolution of the industrial camera is 1920 × 1080 and that of the thermal imager is 640 × 480. Using the feature height method, the keyframes are obtained from the video for expert calibration. The extracted deep and shallow features and process time series data features jointly construct a dataset, with a total of 1319 samples. Each sample has 35 features, including 17 time-series features, 4 shallow features and 14 deep features. There are 1100 sets of samples in the training set and 219 sets of samples in the test set. The model consists of two layers of LSTM, with 50 hidden layer units and an input length of 50. The optimizer uses Adam and the early stop step is set to 20.

#### *4.2. Comparison of Results and Analysis*

The effect of shallow feature extraction is shown in Figure 16. The online acquisition of characteristic height, the thickness of the red flame layer, the distribution area and the proportion of four flame layers are presented.

**Figure 16.** Video feature extraction diagram of the red layer at the end of the sintering machine.

In this experiment, the accuracy of the prediction model was evaluated by mean square error (MSE), mean absolute error (MAE) and hit rate (HR), where *y* and *y*ˆ are the real value and prediction, respectively, and *n* is the number of test samples. See Table 2 for the average evaluation indexes of the model obtained from the 10 experiments.

$$MSE = \frac{1}{n} \sum\_{i=1}^{n} (y\_i - \hat{y}\_i)^2 \tag{3}$$

$$MAE = \frac{1}{n} \sum\_{i=1}^{n} |y\_i - \mathcal{Y}\_i| \tag{4}$$

$$H\_i = \begin{cases} \ 1, & |y\_i - \hat{y}\_i| / y\_i < = 1.5\% \\ 0, & |y\_i - \hat{y}\_i| / y\_i > 1.5\% \end{cases} \tag{5}$$

$$HR = \frac{1}{n} \sum\_{i=1}^{n} H\_i \tag{6}$$

As a control group, the model used only time series data from the industrial process and did not use image information fusion. The prediction results are shown in Figure 17 and the decline in the loss function during network training is shown in Figure 18. The second model added shallow image information fused with time series data. The prediction results are shown in Figure 19 and the loss function decline diagram during network training is shown in Figure 20. The prediction results of the model fused with the deep and shallow image information are shown in Figure 21 and the decline in the loss function when training the network is shown in Figure 22.

It can be seen from Table 2 that the prediction model of multi-source data fusion proposed in this paper achieved good results and fast convergence speed. Compared with the method using only time series data and shallow features, the model fusing time series data, deep and shallow features obtains more process information and rich features at different scales, and the prediction effect is improved to some extent.

Figures 10 and 14 show that the correlation of the extracted deep and shallow features is similar to that of the time series data, which has some significance for the prediction model.

Compared with the industrial process time series data model without introducing images, the MSE of the deep and shallow image information model proposed in this paper decreases by about 29% and the hit rate increases from 86.5% to 93.1%. Compared with the temporal data fusion shallow feature model, the MSE of the fusion deep and shallow image information model proposed in this paper decreases by about 24% and the hit rate increases from 89.8% to 93.1%. The actual industrial application verifies the effectiveness of the multi-source data fusion method proposed in this paper.

**Methods MSE MAE HR** Time series data 0.0082 0.072 0.865 Time series data + shallow feature fusion 0.0076 0.069 0.898 Time series data + Deep and shallow feature fusion (ours) 0.0058 0.061 0.931

**Table 2.** Evaluation index of prediction value of different methods.

**Figure 17.** Plot of prediction results using time series data for industrial processes only.

**Figure 18.** Plot of the loss function of the time series data model during network training.

**Figure 19.** Plot of prediction results using time series data and shallow feature fusion model for industrial processes.

**Figure 20.** Plot of the loss function of time series data and shallow feature fusion model during network training.

**Figure 21.** Plot of prediction results using time series data, shallow feature and deep feature fusion model for industrial processes.

**Figure 22.** Plot of the loss function of time series data, shallow feature and deep feature fusion model during network training.

#### **5. Conclusions**

This paper presents a method to detect FeO content in sinter based on multi-source information fusion. The method first collects video data of the red layer at the end of the sintering machine through an industrial camera. Secondly, the keyframe extraction algorithm based on feature height and the shallow feature construction method based on sinter layering are designed according to the actual process. Then, deep features of sinter tail red layer images are extracted from keyframes by ResNet model. Finally, combined with the process parameters of the production process, an online real-time prediction model of FeO content in sinter is established through the LSTM network.

The model solves the problems of poor time efficiency and high cost of existing technologies by extracting multi-scale information from industrial camera video data and integrating the process parameters. It has practical significance for the guidance of the sinter production process and provides technical support for energy conservation, emission reduction, and quality and efficiency improvement of iron and steel enterprises. There are also quality variables in the sintering process that are relevant to the images of the faults, such as the total iron content, tumbler index, etc. This method can be extended to other variable predictions as long as suitable image labeled data are available. However, the new system requires labeled data before deployment, high quality image labeled data have a direct impact on system accuracy. To reduce manual effort and improve deployment efficiency, a future direction that could be considered is self-supervised learning of images, thus reducing the workload of expert labeling.

**Author Contributions:** Software, Y.L.; Supervision, W.J.; Writing—original draft, Y.L.; Writing review and editing, C.Y. and Z.S. All authors have read and agreed to the published version of the manuscript. All authors have read and agreed to the published version of the manuscript.

**Funding:** Funding was received from the National Natural Science Foundation of China (No. 61933015) and the Fundamental Research Funds for the Central Universities (Zhejiang University NGICS platform).

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** The authors would like to thank the editors and reviewers for their contribution to the improvement and publication of this article. And we also want to thank Liuzhou Steel Group for its strong support and cooperation.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
