Analysis and Design Multiple Layer Adaptive Kalman Filter Applied to Electro-Optical Infrared Payload Vision System

Abstract

This study designs a trigger to determine the number of layers of a multi-layer adaptive Kalman filter and applies it to optoelectronic infrared payload system vision. This feature reduces the number of mechanically stabilized motors, equipment weight, CPU resources, and power for an electro-optical infrared payload system. The goal is to reduce the traditional use of multiple gyroscopes to perform calibration measurements on different gimbal frames by this design. In this study, mathematical modeling was carried out for the three-axis, three-frame camera stabilizer system, and the system foundation without motor and gimbal frame was established to achieve aperture-type camera mode. The exposure of the drone’s payload structure outside the aircraft can be reduced. This study provides the adaptive Kalman filter with the offset parameters of the camera image Minimum Output Sum of Squared Error and the three-axis degrees of freedom vector and angle data on the gyroscope. By using the image processing unit, the offset was corrected at each frame per second. The experimental results show that under the same hardware, failure limit and camera field of view constraints. The processing time by this method was compared to the traditional frame correction and full image stabilization methods. The results show that the proposed method can shorten 6 microseconds under the traditional method and can be used to provide lower power consumption, lower image delay, and a larger viewing angle range.

1. Introduction

Globally, in response to the technological advancement of drone micro-electromechanical systems, drone costs have fallen sharply. Applications of optoelectronic payloads and stabilizers become a standard component. Optoelectronic payloads are detection devices installed on servo-stabilized systems on moving carriers such as drones and track ground or air targets based on boresight stabilization technology. With the varying attitude angle of the carrier, the gyroscope’s photoelectric payload can be moved without interference from the carrier, stabilizing the picture and ensuring the detection equipment on the photoelectric payload. It can be found that optoelectronic payloads for drones have the advantage of key components in the drone market. Therefore, from 2016 to 2018, many solutions were obtained due to the microchip processing speed and data amount increased. It causes many different applications of optoelectronic payloads used in various military and civilian fields. As the speed of the processor increases and the amount of data processing is larger. The ability to perform calculations on images and signals simultaneously in one processor emerges is required. The traditional advantages of mechanical processing are gradually reduced, and instead, the image stabilization processing is performed by the image and the mechanical gimbal frame at the same time. Image processing research is the basis of this method. It can greatly increase the processing efficiency and reduce the use of the hardware gimbal frame. It further reduces the weight and resource consumption of image stabilization and can further utilize one processor to complete multiple tasks in the traditional method. The performance evaluation index of the photoelectric payload is the reliability of the data captured by the optical sensor. Therefore, the reliability of the data captured by the sensor is a key issue. Ref. [1] proposed an adaptive unscented Kalman filter for target tracking with time-varying noise covariance based on multi-sensor information fusion. An innovative optimal information fusion methodology is proposed based on adaptive and robust unscented Kalman filter (UKF) for multi-sensor nonlinear stochastic systems. Based on the linear minimum variance criterion, this multi-sensor information fusion method has a two-layer architecture: at the first layer, a new adaptive UKF scheme for the time-varying noise covariance was developed and served as a local filter to improve the adaptability together with the estimated measurement noise covariance by applying the redundant measurement noise covariance estimation, which is isolated from the state estimation; the second layer is the fusion structure to calculate the optimal matrix weights and gives the final optimal state estimations. Based on the hypothesis testing theory with the Mahalanobis distance, the new adaptive UKF scheme utilizes both the innovation and the residual sequences to adapt the process noise covariance timely. The results of the target tracking simulations indicate that the proposed method is effective under the condition of time-varying process-error and measurement noise covariance. Ref. [2] proposed the design of a stability control system for pan, tilt, and roll. The traditional gyro stabilization control method uses the angle information obtained by the gyro to extend the Kalman Filter and then corrects the position of the three-degree-of-freedom velocity integration in the original space [3]. The value is provided to calculate the parameters of the controller, and the angle information can be converted into a unit pulse (Pulse Width Modulation, PWM) signal to the MOSFET to generate a higher voltage to the motor. Ref. [4] uses a fixed-wing UAV to implement an automatic tracking system design. Reference [5] analyzed the various error sources of the three-axis gimbal, used the accuracy of the gyro error model to calibrate the accuracy, and established the photoelectric payload and the gyro model error equation. However, the actual internal error of the controller includes: gyro accuracy error, acceleration caused by gyro angle error, motor response time delay, payload vibration, and image scanning frequency resonance [6,7]; the above four items cause the final imaging image to be unclear or fuzzy. Key factors proposed a fuzzy PID control algorithm to control the operational stability of the photoelectric payload. However, updating the accuracy of the gyroscope causes a serious burden on the private sector. Acceleration caused by the gyro angle error can be corrected in real-time by setting the spatial position of one axis in the middle of the three-axis degrees of freedom as a constant. References [3,8] reposed a PTZ controller design method for UAV’s own motion compensation. By analyzing the effects of translation and rotation movement of UAV on the imaging of the target in the image plane, the relationship between the angular velocity of the body and the angular velocity of the PTZ attitude. In order to compensate for the change in the camera’s boresight caused by the UAV’s own motion, the position of the target in the image plane is used to correct the camera tracking error. In this paper, applied to the electro-optical infrared payload system’s vision offset function, the multiple layer Kalman Filter characteristic [9] was used to analyze the filter inertial measurement unit and design a trigger to set up the layer number of Kalman Filter [10]. This function can decrease the electro-optical infrared payload system’s number of stabilized motors, the weight of the device, central processing unit resource, and power. In this research, an illustrated example of a three-axis, three-gimbal system was analyzed by mathematical and simulation inertial measurement unit vibration. Three-axis degrees of freedom vector and gimbal motion data obtained by Kalman filter were provided to the graphics processing unit for offsetting each frame [10,11]. However, none of the studies mentioned above can be optimized for the efficiency of image stabilization. Additionally, the constraints of the main research are based on the design of the existing hardware. The fundamental problem is not substantially solved, and the response of the motor of the ring frame is lower due to the physical limit. Less response performance means that a powerful motor and lower energy consumption are required. The stabilization of the image only assists the system. Therefore, based on the literature [9,10,11], this study designs the input of the adaptive Kalman filter and combines the image tracking technology of [7] to improve the image quality. The goal was to use the proposed method to reduce the dependence of the optoelectronic payload driven by the motor. Ref. [12] proposed a dual-optimized adaptive Kalman filtering algorithm based on bp neural network and variance compensation for laser absorption spectroscopy. The analysis and design of a dual-optimized adaptive Kalman filtering (DO-AKF) algorithm based on back propagation (BP) neural network and variance compensation was developed for high-sensitivity trace gas detection in laser spectroscopy [13,14]. The BP neural network was used to optimize the Kalman filter (KF) parameters. Variance compensation was introduced to track the state of the system and to eliminate the variations in the parameters of dynamic systems. The proposed DO-AKF algorithm showed the best performance compared with the traditional multi-signal average, extended KF, unscented KF, KF optimized by BP neural network (BP-KF), and KF optimized by variance compensation (VC-KF) [15]. The optimized DO-AKF algorithm was applied to a QCL-based gas sensor system for an exhaled CO analysis [16]. The experimental results revealed a sensitivity enhancement factor of 23. The proposed algorithm can be widely used in the fields of environmental pollutant monitoring, industrial process control, and breath gas diagnosis.

The above mainly describes the design improvement and analysis of the EKF. The improvement of this algorithm was also used on UAVs, such as in reference [17]. A UAV swarm navigation using dynamic adaptive Kalman filter and network navigation was proposed. Ref. [18] also proposed a multi-sensor fusion based on adaptive Kalman filtering. The optimal performance of the conventional Kalman filters is not guaranteed when there is uncertainty in the process and measurement noise covariances. In this paper, in order to reduce the effect of noise covariance uncertainty, the Fuzzy Adaptive Iterated Extended Kalman Filter (FAIEKF) and Fuzzy Adaptive Unscented Kalman Filter (FAUKF) are proposed to overcome this drawback. The Proposed FAIEKF and FAUKF were applied to fuse signals from Global Positioning System (GPS) and Inertial Navigation Systems (INS) for the autonomous vehicles’ navigation. In order to validate the accuracy and convergence of the proposed approaches, results obtained by FAUKF and FAIEKF were compared to the Fuzzy Adaptive Extended Kalman Filter (FAEKF), Extended Kalman Filter (EKF), Unscented Kalman Filter (UKF), and Iterated Extended Kalman Filter (IEKF). The simulation results illustrate superior performance. It can be seen that the design and improvement of EKF are very common on UAVs. This research mainly focuses on the integration and optimization of image stabilization and UAV photoelectric stabilization systems. Ref. [10] showed a modified unscented Kalman filter using modified filter gain and variance scale factor for highly maneuvering target tracking. It improved the low tracking precision caused by lagged filter gain or imprecise state noise when the target highly maneuvers; therefore, a modified unscented Kalman filter algorithm based on the improved filter gain and adaptive scale factor of state noise was presented. In every filter process, the estimated scale factor is used to update the state noise covariance, and the improved filter gain is obtained in the filter process of unscented Kalman filter (UKF) via predicted variance, which is similar to the standard Kalman filter. Simulation results show that the proposed algorithm provides better accuracy and ability to adapt to the highly maneuvering target compared with the standard UKF. Ref. [8] proposed a dynamic Siamese network with an adaptive Kalman filter for object tracking in complex scenes. It solved the problem that the tracking accuracy of a fully-convolutional Siamese network (SiamFC) in complex scenes such as similar object interference, fast-moving, and appearance change is not good. A new object tracker based on a dynamic template updating strategy and the re-location mechanism based on the adaptive Kalman is proposed. In order to suppress the object interference and overcome the instability of fast-moving object tracking, an adaptive Kalman filter method was designed to change the selection method of search region and select the bounding box of the object closest to the predicted position. For the adaptation of appearance change, the high-confidence tracking results are fused with the initial template to dynamic update the template. Compared with the traditional Kalman filter, the expectation of residual error for the adaptive Kalman filter method can be controlled in a low range by the adjustment of the gain online. The introduction of the adaptive Kalman-based re-location mechanism improves the discriminative ability of SiamFC in the interference scene. With the dynamic template updating strategy, the tracker obtains strong generalization capability to adapt to the appearance change of the tracking target. It was demonstrated that the proposed method performs real-time object tracking at the speed of 43 fps and achieves competitive performance on OTB, VOT, and TC128 datasets compared with other state-of-the-art trackers.

The above research shows that the trend of the current technology is gradually changing from separate motor and image stabilization to pure gyro image stabilization. However, the above literature also shows that it is very unwise to use the gyroscope for image stabilization directly, and the noise on the gyroscope still has a large deviation even after traditional filtering. Therefore, how to use the gyroscope on the drone to perform image stabilization more effectively is a key issue.

This study shows the completion of the coordinate analysis of photoelectric gimbal payload and the preliminary description of coordinates, and the analysis was carried out under the assumption that variables such as material deformation are not considered. Mainly, the Minimum Output Sum of Squared Error (MOSSE) and Fully-Convolutional Siamese Networks (SiamFC) extraction methods of images were firstly established for theoretical establishment. The tracked target was then extracted relative to the error variable in the image. The theory of adaptive extended Kalman filter was established and described. Then substitute the variables were extracted by MOSSE and SiamFC. The actual hardware was verified first and then verified on different devices. The image error should be lower than one pixel, and the steady-state error of the gyroscopes in the experimental should be lower than 100 LSB/°/s. The key experimental data related to this study were extracted and shown in the following sections of the result and conclusion.

Table 1. Definition of Symbols.

Symbols	Definitions
$g$, $G$	Response of error least square sum filter method
$f$	Input samples and outer product of $g$
$h$	Filtering model
$F_i$	Input samples and outer product of $G$
$T$	Initial target template for domain tracking
$X_n$	Search area in the nth frame
$\phi$	Feature extractor in the SiamFC
$Corr(•)$	Correlation operation
$G(T,X_n)$	Represented as the response map
${\mu }_t$	Best quality control amount with time t
$x_{k|k-1}$	Prediction of the previous state
$C_k$	State transition matrix
$B_k$	System parameter
$U_k$	Control quantity of the process at the time.
$G_k$	Kalman gained in reference time in the filter
$Z_k$	Observation value in reference time in the filter
$I$	Identity matrix
$k$	Reference time in the filter
$x$	State of the Kalman filter
$H$	Observation matrix in reference time in the filter
$p_x\left|p_y\right.$	Target position formed the state of the Kalman filter
$v_x\left|v_y\right.$	Target velocity formed the state of the Kalman filter
${\lambda }_k$	Solution gain to adjust the Kalman filter.
$\rho$	Fading factor.

shows the relevant abbreviations and symbols used in the following.

2. Material and Method

2.1. Coordinate Analysis of Photoelectric Gimbal Payload

In this paper, it is assumed that the platform and the carrier are rigidly connected, that is, regardless of the damping effect between the platform base and the carrier. Therefore, the base coordinate system is considered to be the carrier coordinate system in the following analysis. As shown in Figure 1, the three-axis, three-frame gimbal stabilizer can be divided into vibration reduction system (defined as coordinate system 0), yaw frame (defined as coordinate system 1), pitch frame (defined as coordinate system 2), and roll frame (defined as the coordinate system 3) and other four parts, including the rotation angle of the yaw axis, the rotation angle of the tilt axis, the rotation angle of the roll axis, and the distance from the center point of the pitch frame to the motor of the roll frame. The distance is the vertical distance from the center point of the pitch frame to the position where the sensor is placed in the roll frame and the horizontal distance from the left end of the lower damping plate to the coordinate system (0). The structure is connected by three rotating joints, and each joint is driven by a motor; the yaw axis motion, pitch axis motion, and roll axis motion are controlled separately. When the carrier is vibrated or tilted by the external environment, the inertial measurement unit (IMU) installed on the rolling frame obtains the frame deviation. After calculation by the control algorithm, it outputs a compensation signal to drive the motor to generate a balanced torque so that the stabilizer eliminates deviation and maintains the horizontal stability of the boresight.

The mathematical modeling of the spherical gimbal in Figure 1 is derived in the following. The mechanism of the gimbal is composed of an anti-vibration plate, i.e., yaw frame, pitch frame, and roll frame, and each frame motion is driven by a motor. In order to analyze the dynamics of the gimbal, components of the gimbal are named as a sub-coordinate system, where the anti-vibration plate is denoted as coordinate system 0, and the frames of yaw, pitch, and roll are denoted as coordinate system 1, 2, and 3, respectively. As shown in Figure 2, ${\theta }_1$ is rotation angular of yaw axis, ${\theta }_2$ is rotation angular of pitch axis, ${\theta }_3$ is rotation angular of roll axis, $l_1$ is the minimum distance between yaw axis and motor of roll axis, $h_1$ is the minimum distance between anti-vibration plate and pitch axis, $h_3$ is the minimum distance between pitch axis and gyroscope sensor, and $b_2$ is the radius of anti-vibration plate. The motion of the spherical gimbal is measured by an inertial measurement unit (IMU) and is controlled by three motors to stabilize the aerial photography. The relative motion of each coordinate system can be obtained. Cartesian coordinates of systems 0, 1, 2, and 3 are defined as $\left(x_0,y_0,z_0\right)$, $\left(x_1,y_1,z_1\right)$, $\left(x_2,y_2,z_2\right)$, and $\left(x_3,y_3,z_3\right)$, respectively. The relative geometric relationship of coordinate system 0 and coordinate system 3 can be given as (1).

$${\mathbf{T}}_3^0=\left[\begin{array}{cccc}{c}_1{c}_2& \left(\begin{array}{l}-s_1{c}_3\\ +c_1{s}_2{s}_3\end{array}\right)& \left(\begin{array}{l}{s}_1{s}_2\\ +c_1{s}_2{s}_3\end{array}\right)& \left(\begin{array}{l}{l}_1{c}_1{c}_2-s_1{h}_3{c}_3+c_1{s}_2{h}_3{s}_3\\ -l_1{s}_2{c}_1-b_2{s}_1-b_2{c}_1\end{array}\right)\\ s_1{c}_2& \left(\begin{array}{l}{c}_1{c}_3\\ +s_1{s}_2{s}_3\end{array}\right)& \left(\begin{array}{l}-c_1{s}_3\\ +s_1{s}_2{c}_3\end{array}\right)& \left(\begin{array}{l}{l}_1{s}_1{c}_2+c_1{h}_3{c}_3+s_1{s}_2{h}_3{s}_3\\ -l_1{s}_2{s}_1+b_2{c}_1-b_2{s}_1\end{array}\right)\\ -s_2& c_2{s}_3& c_2{c}_3& \left(-l_1{s}_2+c_2{h}_3{s}_3-l_1{c}_2+h_1\right)\\ 0& 0& 0& 1\end{array}\right]$$

(1)

where $c_i=\cos {\theta }_i$ and $s_i=\sin {\theta }_i$ for $i=1,2,3$ are defined. Therefore, we have $\left[\begin{array}{c}{x}_3\\ y_3\\ z_3\\ 1\end{array}\right]=T_3^0\left[\begin{array}{c}{x}_0\\ y_0\\ z_0\\ 1\end{array}\right]$. Here, based on (1), the required command can be calculated, i.e., ${\theta }_1$, ${\theta }_2$, and ${\theta }_3$, by an appropriate control system.

2.2. MOSSE and SiamFC

In 2010, David S. B. proposed visual object tracking using adaptive correlation filters, which mentioned the error least square sum filter method [7]. The response $g$ represents the outer product of $f$ and the filtering model $h$ as given in (2).

$$g=f\otimes h$$

(2)

Therefore, for the fastest response, the filtering model $h$ should be designed. Based on the Fast Fourier Transform (FFT) proposed by David S. B. [7], (2) can be written as (3).

$$G=F\left(g\right)=F\left(f\otimes h\right)=F\left(f\right)\cdot F{\left(h\right)}^{\ast }=F\cdot H^{\ast }$$

(3)

Then the model $H^{\ast }=\raisebox{1ex}{$G$}\!\left/ \!\raisebox{-1ex}{$F$}\right.$ can be obtained. However, the actual target locking and tracking process must take into account the deformation of the target’s appearance or the appearance of the image due to the rotation of the target. Therefore, multiple images of the target need to be accessed as a reference to improve the robustness of the filter model. Thus, the above formula (3) can be transformed into (4) and (5).

$$mi{n}_{(H_{wv}^{*})}{\displaystyle {\sum }_{i=1}^m|H_{wv}^{*}{F}_{wvi}-G_{wvi}{|}^2}$$

(4)

$$H=\frac{{\displaystyle {\sum }_i^{}{F}_{imv}{G}_t^{*}}}{{\displaystyle {\sum }_i^{}{F}_t{F}_t^{*}}}$$

(5)

The input samples $F_i$ and the input features are generated through the Gaussian distribution. The $H$ value of the filter can be calculated after obtaining the training samples. Note that the sampling pixels of $F$, $G$, and $H$ are given the same. In order to improve the robustness of the filter in the environment of target type change or contrast and brightness change interference, based on [7], we have $H=A_i/B_i$, and $A_i$ and $B_i$ are given as (6) and (7), respectively.

$$A_t=\eta F_t·G_t^{*}+(1-\eta )A_t-1$$

(6)

$$B_t=\eta F_t·F_t^{*}+(1-\eta )B_{t-1}$$

(7)

where $\eta$ are update parameters, including the gray-scale gain ${\displaystyle {\sum }_i^{}{F}_{imv}{G}_t^{*}}$ and ${\displaystyle {\sum }_i^{}{F}_t{F}_t^{*}}$ when filtering are the above, respectively, and the updated variables $\eta$ are added to the loop when the image is mutated. The MOSSE image is tracked after real-time identification, and the feature image can be scanned in real-time without the need to build a model for the target. Because the traditional color binarization feature process is fast, it is easy to cause the recognition feature to fail at night or when the sensor does not invent a visible target. For other processes, the output range with naive filtering is too large to identify the feature position. For UMACE filtering, the output feature result has the highest accuracy, but it takes too much time to cause excessive frame loss. Such that the frames cannot be connected smoothly. The number of images that can be provided per unit time is the least. The MOSSE filtering results are similar, but the visible and thermal image fuzzy recognition is similar to the UMACE results. Using too much memory causes a loss of frame numbers.

In this study, the image captured by the optical payload is used to perform the MOSSE filtering on the pixels between 100 × 100 pixels at the position of four points in three-quarters of a pixel, as shown in Figure 3. The four point positions averaged the ideal sight center coordinates of the previous frame that was displaced due to the shock and averaged the position information of this shock and the feed-forward value in the motion control card to the position information of the next frame. The ability to approach real-time control is very high, so MOSSE is used for tracking and identification.

In Figure 4, $T$ is the initial target template for domain tracking and $X_n$ is the search area in the nth frame. An end-to-end Siamese network model is trained using fully connected layers and predefined learning methods in SiamFC. The image $T$ on the current frame and the search area $X_n$ are the inputs to SiamFC. The features of $T$ and $X_n$ in the feature space are extracted and make correlation connections. Note that the symbol of $\ast$ is represented as the product operator.

$$G(T,X_n)=Corr(\phi (T),\phi (X_n))+b$$

(8)

The parameter of $b$ denotes a signal that takes value in all locations. The parameter $\phi$ is the feature extractor in the SiamFC. The function of $Corr(•)$ is the correlation operation and $G(T,X_n)$ is represented as the response map. The score means the similarity between the image template and the searching region. The position with maximum value in $G(T,X_n)$ corresponds to the location of the target. In order to realize real-time tracking, the feature extractor is usually used based on a convolutional neural network such as AlexNet [10].

It is well known that Kalman filtering belongs to an autoregressive filter. The largest flaw originally designed must be applied to systems that conform to the Gaussian distribution, and extended Kalman filtering (EKF) can be used in nonlinear dynamic systems and discrete data collection. The key five types of EKF can be shown in (9)–(13).

$${\mu }_t=g({\mu }_t,{\mu }_{t-1})$$

(9)

$${\displaystyle {\sum }_t=}{G}_t{\displaystyle {\sum }_{t-1}{G}_t^T+R_t}$$

(10)

$$K_t={\displaystyle {\sum }_t{H}_t^T{(H_t{\displaystyle {\sum }_t{H}_t^T+H_t})}^{-1}}$$

(11)

$${\mu }_t={\mu }_t+K_t(z_t-h({\mu }_t))$$

(12)

$${\displaystyle {\sum }_t=}(I-K_t{H}_t){\displaystyle {\sum }_t}$$

(13)

where ${\mu }_t=g({\mu }_t,{\mu }_{t-1})$ is given. The best quality control amount is based on the state of the top. At present, only the average value is calculated. When the Gaussian noise is 0, the second term can be set to 0. At the same time, the Kalman gain can be calculated by the squared difference of the predicted value, and the predicted value and the measured value are simultaneously calculated. The final Kalman gain is calculated by the squared difference of the proportionality factor, and the predicted value and the squared difference of the state ${\mu }_t={\mu }_t+K_t(z_t-h({\mu }_t))$ are also obtained. At the same time, the measured value of the state is also obtained. At this time, the estimated state of the state can be calculated through the Kalman gain. The higher the reliability of the measured value of the system. Single-layer Kalman filtering was used to perform triaxial image rotation and vertical and horizontal movement.

2.3. Object Tracking with Adaptive EKF

Object tracking is a process involving time and space dimensions. The position contains the time trajectory information of the target. Therefore, the Kalman filter is used to integrate the temporal trajectory information of the target with the tracker, but it has obvious flaws when tracking. The Kalman filter tracking stabilization based on the mathematical model is precisely sufficient. Incorrect models and external disturbances can lead to loss of accuracy and even filter divergence. Kalman gain tends to stabilize when the Kalman filter reaches a steady state. If the target state changes suddenly, the Kalman filter will not be able to adjust the gain quickly, and the filter will diverge. The extended Kalman filter is used especially in the condition of only IMU output. Unlike using a Kalman filter in a simulation. The noise caused by the actual IMU output or the lack of proper high-frequency vibration isolation can easily cause the IMU to generate serious noise. This easily leads to a serious failure of the Kalman filter, and even the gain approaches zero. Based on this problem, this study provides a new method to suppress interference then overcome non-stabilization when the object moves too fast, or vibration isolation is incomplete. By using the adjustment factor, it is ensured that the filter gain matrix can be adjusted in time and the residual signals at different times are close to quadrature. Thus, the robustness of model tracking and the ability to adapt to target state mutation is obtained. The traditional Kalman filter has two update processes, time update and observation update. Time updates are mainly used for system predictions such as state and covariance predictions. The observation update includes the calculation of Kalman gain, state update, and covariance update, which is called the correction phase. The state is observed and used to correct the predicted value during the system prediction process. The state prediction can be expressed as (14)

$$x_{k|k-1}=C_k{x}_{k-1|k-1}+B_k{U}_k$$

(14)

where $x_{k|k-1}$ is a prediction of the previous state, $C_k$ is the state transition matrix, $B_k$ is the system parameter, $U_k$ is the control quantity of the process at the time $k$. The state of the Kalman filter $x$ was formed by target positions $p_x\left|p_y\right.$ and velocities $v_x\left|v_y\right.$. The residual at time $k$ can be expressed as (15). $Z_k$ is observation at time $k$. The parameter $H$ is the observation matrix. The covariance prediction process is used to predict the state as (16)

$$V_k=Z_k-H{x}_{k|k-1}$$

(15)

$$P_{k|k-1}={\lambda }_k{C}_k{P}_k{P}_{k-1|k-1}{C}_k^T+Q_k$$

(16)

where $P_{k|k-1}$ is covariance prediction of state $x_{k|k-1}$, and $Q_k$ is the covariance matrix of the system process. The parameter ${\lambda }_k$ is the adjustment factor for $P_{k-1|k-1}$ when the Kalman filter ${\lambda }_k=1$ is given. First, the Kalman gain of state update and covariance update is calculated to obtain the current state observation value. State updates can be achieved through current optimal state estimates of predicted values, observed values, and gains. The Kalman gain can be defined as (17). $G_k$ is Kalman gain and $R_k$ is the covariance matrix of the measurement noise. The status update process can be shown as Equation (18). The parameter $x_{k|k}$ is the optimal state estimation and $Z_k$ is observation value, and covariance status update process can be shown as (19).

$$G_k=P_{k|k-1}{H}^T{(H{P}_{k|k-1}{H}^T+R_k)}^{-1}$$

(17)

$$x_{k|k}=x_{k|k-1}+G_k(Z_k-H{x}_{k|k-1})$$

(18)

$$P_{k|k}=(I-G_{k}H)P_{k|k-1}{(I-G_{k}H)}^T+G_k{R}_k{G}_k^T$$

(19)

where $I$ is the identity matrix. The main difference between adaptive Kalman filter and Kalman filter is the design of $R_k$, $Q_k$, and $k$. In the traditional Kalman filter, $R_k$ and $Q_k$ are constants. The effectiveness of the Kalman filter depends on the initial value of $R_k$ and $Q_k$. Thus, this study allows the update process of the Sage–Husa adaptive filter to be introduced into the Kalman filter, as shown in (20) and (21).

$$P_k=\mu R_{k-1}+(1-\mu )\left[V_k{V}_k^T-H{P}_{k|k}{H}^T\right]$$

(20)

$$Q_k=\mu Q_{k-1}+(1-\mu )\left[G_k{V}_k{V}_k^T{G}_k^T+P_{k|k}-C{P}_{k-1|k-1}{C}^T\right]$$

(21)

where $\mu$ is the fading factor of $R_k$ and $Q_k$. Experimental results show that taking large values can lead to severe drift of the filter. Therefore, it should satisfy the condition of $0\le \mu \le 0.1$. A suitable adaptive Kalman filter is designed to satisfy (21). Ideally, the residual of the Kalman filter is noise satisfied by a Gaussian distribution (21). However, it can be seen from the residuals that if the target has drifted, the state estimation of the Kalman filter will be incorrect. The solution of ${\lambda }_k$ can be shown in (22), which can be derived from the properties of the Kalman filter. According to (16) and (22), Equation (23) can be derived, then simplified to (24). Equations (15) and (24) are combined and simplified to (25). Then it can make ${\lambda }_k$ rewritten as (26).

$$E\left[V_{k+j}{V}_k^T\right]=0$$

(22)

$$E\left[V_{k+j}{V}_k^T\right]=0\Rightarrow P_{k|k-1}{H}^T-G_k{V}_k=0$$

(23)

$$P_{k|k-1}{H}^T\left[I-{(H{P}_{k|k-1}{H}^T+R)}^{-1}{V}_k\right]=0$$

(24)

$$H{P}_{k|k-1}{H}^T=V_k-R$$

(25)

$${\lambda }_{k}HC{P}_{k-1|k-1}{C}^T{H}^T=V_k-HQ{H}^T-R$$

(26)

$${\lambda }_k=\frac{tr(V_k-HQ{H}^T-R)}{tr(H{C}_k{P}_{k-1|k-1}{C}_K^{T}H)}$$

(27)

Although ${\lambda }_k$ in (27) can quickly adjust the filter, it will cause a serious response delay of the filter. There is always noise or delay in the tracking process. We found that adjusting ${\lambda }_k$ too quickly can cause a shift in the filter response. Therefore, the parameter ${\lambda }_k$ of the filter is re-designed on this basis, as shown in (28) and (29), and the square difference ${\lambda }_{0,k}$ can be defined as (30).

$${\lambda }_k^{\ast }=\left\{\begin{array}{c}{\lambda }_0,{\lambda }_0\ge 1\\ 1,{\lambda }_0<1\end{array}\right.$$

(28)

$${\lambda }_0=\frac{tr[V_{0,k}-HQ{H}^T-R]}{tr[H{C}_k{P}_{k-1|k-1}{C}_k^{T}H]}$$

(29)

$$V_{0,k}=\left\{\begin{array}{c}{V}_1{V}_1^T,k=1\\ \rho V_{0,k-1}+(1-\rho )V_k{V}_k^T,k>1\end{array}\right.$$

(30)

$$\rho =0.45+0.4\ast sigmoid(k-10),sigmoid(t)=\frac{1}{1+e^{-k}}$$

(31)

where $\rho$ is the fading factor in $V_{0,k}$ which is designed as a $sigmoid$ function as shown in (31). The value of $\rho$ will be small at the beginning of the trace. $k$ is mainly affected by the covariance of residual error $V_k{V}_k^T$. As time $k$ increases, the effect of $k$ becomes smaller, so the filter converges quickly.

3. Experimental Results and Analysis

In this research, the inertial measurement units are used in the experiment, where the angular scale range is ±2000°/s, the angular of sensitivity is 16.4 LSB/°/s, the angular noise rate is 0.005 mdps/rtHz, the acceleration of scale range is ±16 gravity, and the acceleration of sensitivity is 2048 LSB/g. Two electric optical payloads are on the same platform and use the same inertial measurement units. The inertial measurement units used in the electric optical payload are with angular scale range ±1000°/s, angular of sensitivity 32.8 LSB/°/s, angular noise rate 0.005 mdps/rtHz, acceleration of scale range ±8 gravity, and acceleration of sensitivity 4096 LSB/g.

In this study, a constant with a radix of 4 was given as an input, plus a variable of random $\pm 1$ intervals to simulate the noise, and a single-layer extended Kalman filter was performed for the experiment. As shown in Figure 5, the input source is marked blue line. The value under extended Kalman filtering is given as an orange line. Figure 5b shows the simulation noise with random $\pm 2$ interval variables under extended Kalman filtering.

As shown in Figure 5a, with a single layer of extended Kalman filtering, the original base number constant 4 is shown in Figure 5b. The output result is different from the original base number 4, so it still causes errors from the original signal that must be compensated in the controller design. The gyro stabilization system designed by this institute is equipped with a three-gyro system on a drone and sacrifices one axis as a calibration axis for each unit of inertial measurement. The three inertial measurement outputs are complementary to the other two. Furthermore, the stability accuracy of drones and optoelectronic payloads are improved. The angular acceleration and acceleration output by the inertial measurement unit is obtained through an integrator, and the speed directly enters the adaptive extended Kalman filter loop. The angular velocity then outputs the angle data through the integrator and into the adaptive extended Kalman filter loop. The angular velocity and velocity output is integrated through the angular acceleration and acceleration output from the inertial measurement unit, and the angular velocity is output as an angle once integrated. The system designs a low-pass filter at the angular acceleration to filter and output to the trigger unit, where the unit decides to extend the Kalman filter filtering times. Through the final outputs of angle and speed to compare whether its Kalman gain value is much low, and raise a correction warning.

In the system design layout, the low-pass filter is generally limited to the average value of the input signal. The current average value of the input signal is obtained through the average filter, and upper and lower limits are given for the average value of the input signal. The process of Kalman filtering mainly avoids triggering more layers of extended Kalman filtering, so the limits for this low-pass filtering are shown in

Table 2. Design logic of upper-limit modulation of low-pass filter before trigger.

EKF Gain	LPF Processing
EKF Gain < 1	LPF = LPF + 0.5 (Unit: m/s2)
EKF Gain = 1	LPF = LPF + 0.2 (Unit: m/s2)
EKF Gain > 1	LPF = LPF (Unit: m/s2)

.

When the difference is larger, a multi-layer Kalman filter is used. Due to the infinite increase in the simulation, the experimental hardware is physically limited in the actual environment. There must be an upper limit and a warning to provide a gyroscope for recalibration and a warning that the Kalman filter gain is too low and needs to be recalibrated. Among them, judgment is given in the experiment.

In the actual flight process of the strong wind environment or high-speed flight, the acceleration has a chance to exceed 6 m/s², so only the alarm reminder is provided; when the square difference exceeds 8 m/s², the damage to the aircraft is lost. Therefore, during the process of warning and calibration of the gyroscope, it is detected synchronously whether the inertial gauge is malfunctioning or other sensing element is malfunctioning.

Figure 6 shows the result with the analog input of the sine wave and with one layer EKF. The input signal source is blue, the output signal after filtering through EKF is orange, and the output image at the same time is 54 fps. The average power consumption of the body system is 5.233 W.

Figure 7 shows the result with the analog input of the sine wave and with the EKF of three layers. The input signal source is marked by a blue line, and the output signal with filtering through EKF is as an orange line. The frame rate of the output image is 41 fps. The average power consumption of the body system is 7.006 W.

Figure 8 shows the result with the analog input of a sine wave and with an EKF of four layers. The input signal source is a blue line, and the output signal is an orange line. The output frame rate is 39 fps. The average power consumption of the body system is 7.853 W.

Figure 9 shows the result with the analog input of a straight-up and EKF of one layer. The input signal source is a blue line, and the output signal with EKF is an orange line. The output image is 54 fps per frame, while the average power consumption of the body system is 4.321 W.

Figure 10 shows the result with three layers of EKF. The input signal source is a blue line, and the output signal is an orange line. The output image at the same time is 43 fps, and the average power consumption of the body system is 6.337 W.

Figure 11 shows the result with the analog input of a straight-up and EKF of four layers. The input signal source is a marked blue line, and the output signal is a marked orange line. The output frame rate is 40 fps, and the average power consumption of the body system can be calculated as 7.100 W.

After adaptive multi-layer extended Kalman filtering, the output information can be provided to the image compensation for the stabilization effect that cannot be achieved by mechanical stabilization. As shown in Figure 12, the image vibration compensation is provided after filtering through the gyroscope to fit the image stabilization requirements of the drone or photoelectric payload

The angle of view of the roll effect when the drone is placed horizontally is shown in Figure 13a, and the angle of view of the payload tilted by 45 degrees is shown in Figure 13b. The image can be compensated by the gyroscope to achieve the effect of non-mechanical stabilization.

Figure 14 shows the result of a single-layer EKF actually swinging within $\pm 40$ degrees. The input signal is blue, and the output signal is orange. The output image is 56 fps, airborne. The average power consumption of the ROS hardware system is 4.511 W.

Figure 15 shows the result of two layers of EKF flying in the range of $\pm 40$ degrees. The input signal is blue, the output signal is orange, and the output image is 49 fps. The average power consumption of the ROS hardware system is 5.277 W.

Figure 16 shows the result of three layers EKF under actually swinging within $\pm 40$ degrees. The input signal is blue, the output signal is orange, and the output image is 43 fps. The average power consumption is 6.866 W.

Figure 17 shows the result of four layers of EKF with actually swinging at $\pm 40$ degrees. The input signal is blue, and the output signal is orange. The output image is 38 fps. The average power consumption of the ROS hardware system is 7.122 W.

Figure 18 shows the result of a single-layer EKF actually swinging within $\pm 40$ degrees. The input signal is blue, and the output signal is orange. The output image is 51 fps. The average power consumption of the ROS hardware system is 3.998 W.

Figure 19 shows the result of a single-layer EKF actually swinging within $\pm 40$ degrees. The input signal is blue, and the output signal is orange. The output image at the same time is 38 fps. The average power consumption of the ROS hardware system is 5.512 W.

In the above verification data, it can be found that the method designed in this study is feasible. Despite the constraints of the hardware, the simultaneous processing of image and gyro information by a single processor results in a reduction in frames per second and cannot maintain the original image output of 60 fps.

Table 3. Comparison table between the traditional method without motor drive and the method used in this study.

	Traditional Method	This Research Method
Frame per second (Number)	50	31~56
Maximum response delay (Micro-second)	10	4
Motor power consumption (Watt)	0	0
GPU power consumption (Watt)	9.6	6.3
Maximum total power consumption (Watt)	9.6	6.3
Maximum horizontal angle (Degree)	±6	±32
Maximum vertical angle (Degree)	±5	±30

shows the comparison between the proposed method and the traditional method that separates frame stabilization and image stabilization. Note that the gimbal holder is locked and can be measured during the study. It can be seen that using the traditional method, because the camera is separated from the gimbal frame, the frame rate of the output image is corrected and stabilized by the fixed global image of each frame, so it is kept at 50 fps. However, the proposed method in this study uses a single GPU for calculation, provides partial sampling and performs adaptive Kalman filtering for image stabilization, and cannot provide stable output frames per second, which is between 31 and 56 fps. In the power consumption comparison of GPU, the average power consumption of the traditional method is 9.6 watts, and the power consumption of the method designed in this study is 6.3 watts. In the comparison of the horizontal and vertical angles with the maximum allowable and no steady-state error, because the traditional method uses global image sampling, the cropping area is small, and the sampling cannot exceed the existing steady-state error, so the horizontal angle cannot be shifted by more than $\pm 6$ degrees, and the vertical cannot exceed $\pm 5$ degrees. The method used in this study is local sampling, so as long as it is within the camera’s viewing angle, it can be sampled and corrected at the same time with the gyroscope. Therefore, the horizontal angle cannot be offset by more than $\pm 32$ degrees, and the vertical angle cannot exceed $\pm 30$ degrees.

However, in terms of practical application scenarios, the ring frame design of the traditional optoelectronic payload will activate the motor for hardware stabilization, thus increasing the maximum horizontal and vertical allowable angles, as shown in

Table 4. Comparison table between the traditional method with motor drive and the method used in this study.

	Traditional Method	This Research Method
Frame per second (Number)	50	31~56
Maximum response delay (Micro-second)	10	4
Motor power consumption (Watt)	22.5	0
GPU power consumption (Watt)	9.6	6.3
Maximum total power consumption (Watt)	32.1	6.3
Maximum horizontal angle (Degree)	360 continue	±32
Maximum vertical angle (Degree)	360 continue	±30

. Horizontal and vertical angles can be rotated without restriction. However, in this way, the power consumption is greatly increased to 32.1 watts.

4. Conclusions

According to the experimental results, the processing time of the proposed method in the same hardware, fault limit, and camera field of view was compared with traditional frame correction and full image stabilization methods. The design method of this study is more effective than the traditional method in the efficient application of image resources. However, according to the comparison, the field of view in this research method is limited by the viewing angle of the camera. The special feature of this method is that it does not require the use of a motor for stabilization. Therefore, as drones shrink in size, there is less need for energy usage. For future applications, multiple aperture cameras will be used for panoramic shooting without dead angle, and image stabilization will be performed at the same time. This has actually been used in a few advanced anti-radar manned aircraft in the world at present, and the purpose is to reduce the radar reflection area of the external pod.

Therefore, based on this research, when more aperture cameras are used for image stabilization, resources can be more effectively used for calculation, and the image delay and power consumption can be reduced.