Continuous-Time Fast Motion of Explosion Fragments Estimated by Bundle Adjustment and Spline Representation Using HFR Cameras

Abstract

This paper introduces a continuous-time fast motion estimation framework using high frame-rate cameras. To recover the high-speed motions trajectory, we inherent the bundle adjustment using a different frame-rate strategy. Based on the optimized trajectory, a cubic B-spline representation was proposed to parameter the continuous-time position, velocity and acceleration during this fast motion. We designed a high-speed visual system consisting of the high frame-rate cameras and infrared cameras, which can capture the fast scattered motion of explosion fragments and evaluate our method. The experiments show that bundle adjustment can greatly improve the accuracy and stability of the trajectory estimation, and the B-spline representation of the high frame-rate can estimate the velocity, acceleration, momentum and force of each fragments at any given time during its motion. The related estimated result can achieve under 1% error.

1. Introduction

A continuous-time motion analysis has great engineering support for the damage assessment and cause analysis of an explosion [1]. Reconstructing a continuous-time trajectory of scattered fragments has great value. In this paper, we present a method of fast motion estimation of the explosion fragments. The difficulty of the fragment’s motion estimation is randomly scattered. The fragments created by an explosion follow these four stages:

• Radial expansion of the shell;
• A crack appears somewhere in the shell;
• Material leakage, resulting in pressure difference between the inside and outside of the shell;
• The shell is broken randomly, forming irregular fragments which are scattered with a related high initial velocity.

To capture this fast, diffuse motion, the high frame-rate (HFR) cameras are deployed in this research. We used two HFR cameras to build a high-speed vision system capturing the high-speed motion.

The HFR camera has overcome the restrictions posed by conventional video signals. Many offline HFR cameras can be operated at over 1000 FPS (Frame Per Second) during a very short time [2,3,4]. HFR cameras can help us recognize the high-speed phenomena in the real world, which cannot be captured by conventional cameras (e.g., 30 FPS). Many phenomena have been analyed by the HFR camera, such as crash testing, sports motion estimation, industrial robot control, chemical reaction, etc. For 3D measurement, 3D shape inspection using HFR cameras, cooperated with the related projected and coded pattern, can reconstruct the dynamic phenomena of the shape [5,6,7,8]. Deformation and vibration can also be analyzed by only the HFR system or a similar camera array [9,10]. For motion estimation, unlike shape inspection and vibration analysis, the measured object is not a continuous plane or continuous rigid body. A high frequency projector does not work well. A pure vision-based system is widely deployed for activity sensing. To capture the detail of the motion, plenty of HFR vision systems have been developed [11,12,13]. Vision-based motion capture systems are either using passive markers [14,15] or marker-less systems where motion estimation employs object kinematic models. Image analyzing methods, such as pixel level temporal frequency analyzing, are also presented [16,17]. Our high-speed vision system uses an infrared camera which senses the heating information which can control the HFR cameras and evaluate our method.

HFR cameras can only capture some samples at a certain time stamp at the moment of the explosion. The triangulation measurement of the stereo vision has accuracy drift problems, etc. To avoid this problem, we inherent the bundle adjustment (BA) method from a static object’s 3D reconstruction. Bundle adjustment has played an essential role in the 3D reconstruction problem. It estimates six degrees of freedom (DOF), (three for position and three for orientation) of the camera’s moving trajectory and three DOF of point clouds (e.g., 3D map) [18,19,20,21,22].

We explore the bundle adjustment to reconstruct the motion along the time axis, i.e., the moving trajectory of each fragment created by the explosion is a 3D structure along continuous time. Furthermore, the derivation of these 3D positions have their own physical meaning, i.e., velocity and acceleration. We give the analytical result of the continuous-time position, velocity and acceleration estimation by using the cubic B-spline representation. The key idea of this study is shown in Figure 1.

2. Bundle Adjustment for Motion Estimation Using HFR Cameras

2.1. Bundle Adjustment Reconstructs Motion’s Trajectory

The key idea of BA is that there exist bundles of light rays from the camera center and a projected position in images at each moment, and all parameters adjust simultaneously and iteratively together to obtain the optimal bundles of all times. Reprojection errors in bundle adjustment come from all deviations between the observed position in the projected images and the reprojected position at the same image at each time. The motion and the trajectory are estimated along with the image sequences whereas the cameras are static to the world coordinate. We define the reproject error $F_{ij}$ as the following equation:

$$F_{ij}=c_{ij}-b_{ij}=\left[\begin{array}{c}{c}_{ij,u}\\ c_{ij,v}\end{array}\right]-\left[\begin{array}{c}{b}_{ij,u}\\ b_{ij,v}\end{array}\right]$$

(1)

where $b_{ij}$ is the observed position, $c_{ij}$ is the reprojected position, $i$ is the identification of the recording camera, $j$ is the time stamp.

The reprojected function can be expressed in short as:

$$c_{ij}=\Pi \left(P{W}_j,{\varphi }_{iw},t_i^{iw},y_i\right)$$

(2)

where $P{W}_j$ is the position of the world coordinate at the time stamp $j$; ${\varphi }_{iw}$ is the rotation vector presented in lie algebra $\mathfrak{s}\mathfrak{o}$(3) from the world coordinate to camera $i$ coordinate; $t_i^{iw}$ is the translation vector from the world to camera $i$, presented in camera i coordinate; $y_i$ is the intrinsic parameter of camera $y_i={\left[f_i,k_{1i},k_{2i}\right]}^T$ whereas $f_i$ is the focal length of the camera i; $k_{1i}$ and $k_{2i}$ are the distortion parameters of the camera. Equation (2) follows the pinhole camera model with the distortion expression. It can be detailed by the following equations:

$$\begin{array}{l}P{C}_{ij}=\exp ({\widehat{\varphi }}_{iw})P{W}_j+t_i^{iw}\\ { \mathrm{c}}_{ij}^{\prime }=\left[\begin{array}{c}\frac{P{C}_{ij,x}}{P{C}_{ij,z}}\\ \frac{P{C}_{ij,y}}{P{C}_{ij,z}}\end{array}\right]\\ c_{ij}=f_i(1+k_{1i}|\left|c_{ij}^{\prime }\right|{|}^2+k_{2i}\left|\right|c_{ij}^{\prime }|{|}^4)c_{ij}^{\prime }\end{array}$$

(3)

The optimized parameters can be expressed as, with m recording cameras and n captured positions, $\phi ={\left[\cdots ,{\varphi }_{iw},t_i^{iw},y_i,\cdots ,P{W}_j,\cdots \right]}_{mn}^T$. All single reprojection errors $F_{ij}$ form a $mn\times 1$ reprojection vector $F={\left[F_{11}^T,F_{21}^T,\dots ,F_{m1}^T,\dots F_{ij}^T,\dots ,F_{mn}^T\right]}^T$. The optimization process for the bundle adjustment is finding the minimum of all reprojection errors. In this paper, we use the sum of the squares constructed by a large number of nonlinear functions as the cost function to form a nonlinear least-square problem.

$$\underset{(\phi )}{\arg \min }\frac{1}{2}{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^n{F}_{ij}^T}}{F}_{ij}$$

(4)

To solve this unconstrained optimization Equation (4), basing on Taylor’s theorem, a combination of linear search and trust region method Levenberg–Marquardt (LM) algorithm is deployed in this paper. For each iteration of the LM algorithm, the increment of optimized i can be calculated by the following equation:

$$\left[{\mathcal{J}}^T\mathcal{J}+\lambda D\right]\Delta \phi =-{\mathcal{J}}^{T}F$$

(5)

where $\mathcal{J}$ is the Jacobian matrix, calculated analytically by the partial derivatives of the reprojection error; damping factor $\lambda$ controls the LM algorithm search for the result following the linear search or trust region; D is the damping matrix. For the original LM algorithm, D is set as the identity matrix. In this paper, we set D as a diagonal matrix; each element at the diagonal is the eigenvalue of the matrix ${\mathcal{J}}^{\mathcal{T}}\mathcal{J}$. The benefit of this design is that it can balance the magnitude gap between the position, orientation and intrinsic parameters. $\left[{\mathcal{J}}^{\mathcal{T}}\mathcal{J}+\lambda D\right]$ is also denoted as the Hessian matrix.

To solve the bundle adjustment, the key is the Jacobian matrix. From Equation (3), we give the analytically partial derivatives of the reprojection error to each component of $\phi$ by using rotation lie algebra $\mathfrak{s}\mathfrak{o}$(3) properties and the chain rule of derivatives. For ${\varphi }_{iw}$, the partial derivative can be expressed as:

$$\begin{array}{cc}\hfill \frac{\partial F_{ij}}{\partial {\varphi }_{iw}}& =\frac{\partial F_{ij}}{\partial c_{ij}}\frac{\partial c_{ij}}{\partial c_{ij}}\frac{\partial c_{ij}}{\partial P{C}_{ij}}\frac{\partial P{C}_{ij}}{\partial {\varphi }_{iw}}\hfill \\ & =\frac{\partial c_{ij}}{\partial c_{ij}^{\prime }}\frac{\partial c_{ij}^{\prime }}{\partial P{C}_{ij}}\left(-\widehat{\left(\exp (\widehat{{\varphi }_{iw}})P{W}_j\right)}{J}_l({\varphi }_{iw})\right)\hfill \end{array}$$

(6)

where the equation in that means the skew matrix of the related vector; the other two partial derivatives can be easily calculated by using Equation (3). $J_l\left(\varphi \right)$ is the left Jacobian for the BCH formula, with

$$\mathrm{s}{J}_l(\varphi )=\frac{sin\Vert \varphi \Vert }{\Vert \varphi \Vert }{I}_3+\left(1-\frac{sin\Vert \varphi \Vert }{\Vert \varphi \Vert }\right)\frac{\varphi {\varphi }^T}{\Vert \varphi {\Vert }^2}+\left(\frac{1-cos\Vert \varphi \Vert }{\Vert \varphi \Vert }\right)\frac{\widehat{\varphi }}{\Vert \varphi \Vert }$$

For translation vector $t_i^{iw}$, the partial derivate is expressed in Equation (7). Meanwhile, Equation (8) is for intrinsic $y_i$, and Equation (9) is for position $P{W}_j$ at time stamp j,

$$\frac{\partial F_{ij}}{\partial t_i^{iw}}=\frac{\partial F_{ij}}{\partial c_{ij}}\frac{\partial c_{ij}}{\partial c_{ij}^{\prime }}\frac{\partial c_{ij}^{\prime }}{\partial P{C}_{ij}}\frac{\partial P{C}_{ij}}{\partial t_i^{iw}}=\frac{\partial c_{ij}}{\partial c_{ij}^1}\frac{\partial c_{ij}^{\prime }}{\partial P{C}_{ij}}$$

(7)

$$\frac{\partial F_{ij}}{\partial y_i}=\frac{\partial F_{ij}}{\partial c_{ij}}\frac{\partial c_{ij}}{\partial y_i}= \left[\begin{array}{c}dis\cdot c_{ij,x}^1\\ dis\cdot c_{ij,y}^1\end{array}\begin{array}{c}{f}_i\Vert c_{ij}^{\prime }{\Vert }^2{c}_{ij,x}^{\prime }\\ f_i\Vert c_{ij}^{\prime }{\Vert }^2{c}_{ij,y}^{\prime }\end{array}\begin{array}{c}{f}_i\Vert c_{ij}^{\prime }{\Vert }^4{c}_{ij,x}^{\prime }\\ f_i\Vert c_{ij}^{\prime }{\Vert }^4{c}_{ij,y}^{\prime }\end{array}\right]$$

(8)

$$\frac{\partial F_{ij}}{\partial P{W}_j}=\frac{\partial F_{ij}}{\partial c_{ij}}\frac{\partial c_{ij}}{\partial c_{ij}^{\prime }}\frac{\partial c_{ij}^{\prime }}{\partial P{C}_{ij}}\frac{\partial P{C}_{ij}}{\partial P{W}_j}=\frac{\partial c_{ij}}{\partial c_{ij}^{\prime }}\frac{\partial c_{ij}^{\prime }}{\partial P{C}_{ij}}\exp \left({\widehat{\varphi }}_{iw}\right)$$

(9)

with $dis=\left(1+k_{1i}{\Vert c_{ij}^1\Vert }^2+k_{2i}{\Vert c_{ij}^1\Vert }^4\right)$. Equations (6)–(9) form the Jacobian matrix. We denote ${\mathcal{J}}_c^{ij}={\left[\frac{\partial F_{ij}}{\partial {\varphi }_{iw}},\frac{\partial F_{ij}}{\partial t_i^{iw}},\frac{\partial F_{ij}}{\partial y_i}\right]}^T$ formed by Equations (6)–(8) as the camera’s Jacobian for camera i at time stamp j, with ${\mathcal{J}}_{𝓅𝓌}^{𝒾𝒿}$ formed by Equation (9) as the position’s Jacobian observed by camera i at time stamp j. The Jacobian matrix will be detailed by the subsequent section which combines the HFR camera’s properties and deploying strategy.

The optimization procedure of the bundle adjustment for motion estimation are summarized by pseudocode Algorithm 1.

Algorithm 1: Bundle adjustment for motion estimation
Input: Series of observed point $b_{ij}$ from image sequences of all cameras
Output: Cameras parameters and all captured position $\mathsf{\phi }$
1:	V$:= 2; k:= 0; \mathrm{initial} \mathrm{variable} {\mathsf{\phi }}_0$;
2:	$\mathrm{Calculate} \mathrm{Jacobian} \mathrm{matrix} \mathcal{J}={\left[{\mathcal{J}}_{𝒸},{\mathcal{J}}_{𝓅𝓌}\right]}^T \mathrm{using} \mathrm{Equations} \left(6\right)–\left(9\right)$
3:	$\mathcal{A}={\mathcal{J}}^{\mathcal{T}}\mathcal{J};$$F_{ij}$=$c_{ij}\left(\mathsf{\phi }\right)-b_{ij}$; g :=$\mathcal{J}F$;
4:	found := $|g{|}_{\infty }\le {\mathsf{\epsilon }}_1$; $\mathsf{\lambda } := \mathsf{\tau }\times max\left(\mathcal{A}\right)$
5:	while not found and $k<k_{max}$ do
6:	k = k + 1;
7:	Solve Equation (5)
8:	$\mathbf{i}\mathbf{f} \left|\mathsf{\Delta }\mathsf{\phi }\right|\le {\mathsf{\epsilon }}_2\left|\mathsf{\phi }\right|+{\mathsf{\epsilon }}_2$
9:	found := true;
10:	else
11:	$\mathsf{\rho }:=\frac{\left|\mathrm{F}\left(\mathsf{\phi }\right)|-|\mathrm{F}\left(\mathsf{\phi }+\mathsf{\Delta }\mathsf{\phi }\right)\right|}{{\mathsf{\Delta }\mathsf{\phi }}^T\left(\mathsf{\lambda }\mathsf{\Delta }\mathsf{\phi }-g\right)}$;
12:	$\mathbf{if} \mathsf{\rho }< 0$ then
13:	$\mathsf{\lambda } := \mathsf{\lambda }·\mathrm{V}$; $V := 2V$;
14:	else
15:	$\mathrm{Update} \mathrm{all} \mathrm{parameters} \mathrm{by} \mathsf{\Delta }\mathsf{\phi }$ using Equations (1)–(9)
16:	$\mathcal{A}:=\mathcal{J}{\left(\varphi +\mathsf{\Delta }\varphi \right)}^{\mathcal{T}}\mathcal{J}\left(\varphi +\mathsf{\Delta }\varphi \right)$; g :=$\mathcal{J}\left(\mathsf{\varphi }+\mathsf{\Delta }\mathsf{\varphi }\right)F\left(\varphi +\mathsf{\Delta }\varphi \right)$
17:	$\mathrm{found} := |g{|}_{\infty }\le {\mathsf{\epsilon }}_1$
18:	$\mathrm{V} := 2; \mathsf{\lambda } := \mathsf{\lambda }\cdot max\left\{\frac{1}{3},1-{\left(2\mathsf{\rho }-1\right)}^3\right\}$
19:	end if
20:	end if
21:	end while

2.2. Jacobian Matrix for Stereo HFR Cameras

Stereo vision is widely used for non-contact measurement, which is suitable for the physical evaluation of an explosion. In this section, we use the synchronized stereo HFR camera’s captured motion. For each fragment captured by these stereo HFR cameras, during its movement, the formation of the Jacobian matrix is shown in Figure 2. Figure 2 presents the HFR cameras with 540 FPS of the first 20 samples (e.g., for the first 37 ms). In Figure 2, each 4 × 18 matrix block of yellow and purple presents the Jacobian matrix of both cameras at time stamp j, and the matrix block in yellow are the ${\mathcal{J}}_{𝒸}^{𝒾𝒿}$. The matrix block in light blue is ${\left[{\mathcal{J}}_{pw}^{1j},{\mathcal{J}}_{pw}^{2j}\right]}^T$ at time stamp j. The matrix block in purple from Figure 2 are zeros. Both the Jacobian matrix and Hessian Matrix are sparse matrices. Therefore, we use the Shur Complement to solve Equation (5). Equation (5) can be specified as:

$$\begin{array}{r}\left[\begin{array}{cc}{\mathcal{J}}_C^T{\mathcal{J}}_C+\lambda D_C& {\mathcal{J}}_C^T{\mathcal{J}}_{PW}\\ {\mathcal{J}}_{PW}^T{\mathcal{J}}_C& {\mathcal{J}}_{PW}^T{\mathcal{J}}_{PW}+\lambda D_{PW}\end{array}\right]\left[\begin{array}{c}\Delta {\phi }_C\\ \Delta {\phi }_{PW}\end{array}\right]=-\left[\begin{array}{c}{\mathcal{J}}_C^{T}F\\ {\mathcal{J}}_{PW}^{T}F\end{array}\right]\\ \left[\begin{array}{cc}C& E\\ E^T& P\end{array}\right]\left[\begin{array}{c}\Delta {\phi }_C\\ \Delta {\phi }_{PW}\end{array}\right] =-\left[\begin{array}{c}{g}_C\\ g_{PW}\end{array}\right] \end{array}$$

(10)

Since the dimension of the cameras is related less to the position, ${\mathcal{J}}_{𝓅𝓌}$ is the diagonal block matrix which makes the ${\mathcal{J}}_{𝓅𝓌}^{\mathcal{T}}{\mathcal{J}}_{𝓅𝓌}+\mathsf{\lambda }{D}_{pw}$ full rank. We use the Shur complement in the following:

$$(C-E{P}^{-1}{E}^T)\Delta {\phi }_C=-g_C+E{P}^{-1}{g}_{PW}$$

(11)

$$\Delta {\phi }_{PW}=P^{-1}\left(-g_{PW}-E^T\Delta {\phi }_C\right)$$

(12)

Equation (11) efficiently reduces the complexity of the calculation. From Figure 2b we can see that since matrix P is a block diagonal matrix, its inversion is relatively simple. The original Equation (5) can be easily solved by Equations (11) and (12), especially since the number of captured images is large.

2.3. Jacobian Matrix for HFR Cameras with Frame-Rate Multiplication

For HFR cameras, the captured frame-rate is highly reliant on the resolution of the frame, the exposure time, the bandwidth of the hardware, etc. In general, the resolution affects the frame-rate the most. the resolution influences the monitoring field of view (FOV). For fragments produced by the explosion, we want to use high frame-rate (over 500 FPS) to capture the moment of the explosion, and we also want to record the scattered fragments as long as possible.

To solve this problem, we use two HFR cameras with frame-rate multiplication, e.g., one HFR camera under 540 FPS, and another HFR camera under 1080 FPS. The frame-rate of the second HFR camera is twice that of the first camera. This is the same as Figure 2 for the first 37 ms, i.e., 20 frames for the first camera and 40 frames for the second camera. The Jacobian matrix of these two HFR cameras are presented in Figure 3. Each 6 × 18 matrix block of yellow and purple presents the Jacobian matrix of both cameras at time stamp j and j + 1, with 4 × 18 for both cameras at time j, 2 × 18 for camera two at time j + 1. The matrix block in yellow is the ${\mathcal{J}}_{𝒸}^{𝒾𝒿}$. The matrix block in light blue is ${\left[{\mathcal{J}}_{pw}^{1j},{\mathcal{J}}_{pw}^{2j}\right]}^T$ at time stamp j and ${\mathcal{J}}_{𝓅𝓌}^{2𝒿}$ for time j + 1. The Jacobian matrix and Hessian matrix are also the sparse matrices, and with Equations (11) and (12), the iteration step can be solved efficiently.

The interval between captured frames and the synchronization of our HFR cameras is strictly controlled by hardware instruments. This gives us the benefits that: (1) the Jacobian matrix is formatted strictly along the time axis with the same interval; (2) the Jacobian matrix presented in this paper is blocked diagonally, which reduces the complexity of the iteration. These are clearly demonstrated in the examples of Figure 2 and Figure 3.

To improve the accuracy and reduce the iteration step, the initialization of i is important. For synchronization stereo HFR cameras, we can initial i by a stereo calculation at each time stamp. In this section, for a certain time stamp, there is only one camera observing the fragment. To solve this problem, we use pose averaging to initialize the position which is observed by only one camera, i.e.,

$$P{W}_j=\frac{1}{2}\left(P{W}_{j-k}+P{W}_{j+k^{\prime }}\right)$$

(13)

where k is the time interval to the last time stamp where at least two cameras observed, and $k^{\prime }$ is the time interval to the follow up time stamp where two cameras captured the fragment. Based on our initialization strategy, the iteration step can be reduced to less than 300 steps for both frame synchronization stereo cameras and the frame-rate multiplication where we set ${ϵ}_1={10}^{-6}$, and the norm of all reprojection errors is less than ${10}^{-2}$ pixels.

3. Continuous-Time Spline Representation for Fast Motion Estimation

With the bundle adjustment under HFR cameras, each fragment’s movement can be optimized to a trajectory sampled by the high frame-rate. For continuous-time motion estimation, the velocity and the acceleration cannot be calculated preciously by the difference in the bundle adjustment’s results directly. To solve this problem, we use B-spline for continuous-time motion estimation, i.e., position, velocity and acceleration estimation based on bundle adjustment optimized results.

3.1. B-Spline Interpolation

Splines are an excellent choice for representing a continuous trajectory because their derivatives can be computed analytically. B-splines are chosen in this paper as they are a well-known representation for trajectories in R³ [22,23,24]. B-splines use a series of control points (also denoted as knots) to define a linear combination of continuous function which is constrained by the knots. The interpolation between a series of 3D positions (knots) presented in the B-spline is as follows:

$$p(t)={\displaystyle \sum _{i=0}^n{p}_i}Bi,k(t)$$

(14)

where $B_{i,k}\left(t\right)$ is the De Boor-Cox recursive formula, the fractional coefficient for each given control point $p_i$. The formula of $B_{i,k}\left(t\right)$ is:

$$B_{i,k}(t)=\frac{t-t_i}{t_{i+k}-t_i}{B}_{i,k-1}(t)+\frac{t_{i+k+1}-t}{t_{i+k+1}-t_{i+1}}{B}_{i+1,k-1}(t)$$

(15)

3.2. Cubic B-Spline Representation for Continuous-Time Motion Estimation

Estimated velocity and acceleration need a C² continuity, a good approximation of minimal torque trajectories and a parameterization of rigid-body motion devoid of singularities [25,26]. A cubic B-spline representation can satisfy these conditions. We update the 3D position to SE(3), i.e.,

$$T_j=\left[\begin{array}{cc}{R}_w^j& P{W}_j\\ 0& 1\end{array}\right]T_j\in SE(3),R_w^j\in SO(3)$$

(16)

where $R_w^j$ is the rotation between the world coordinate and the fragment coordinate at time stamp j. When we treat the fragment as a single point, $R_w^j=I_3$. For cubic B-spline, Equation (14) can be updated by using Equation (16), i.e.,

$$T(t)=exp\left(B_{0,4}(t)log(T_{j-1})\right){\displaystyle \prod _{k=1}^{3}e}xp\left(B_{k,4}(t)log(T_{k-1}^{-1}{T}_k)\right)$$

(17)

We define time fraction as: $u(t)=\frac{t-t_j}{t_{j+1}-t_j},u\in [0,1)$. By using lie algebra of SE(3) and the related propriety, we give the analytical result for motion estimation of the position (Equation (18)), the velocity (Equation (19)), and the acceleration (Equation (20)) at time u(t) during time $t_j$ and $t_{j+1}$.

$$T(u(t))=T_{j-1}{A}_0{A}_1{A}_2$$

(18)

$$\dot{T}(u(t))=T_{j-1}({\dot{A}}_0{A}_1{A}_2+A_0{\dot{A}}_1{A}_2+A_0{A}_1{\dot{A}}_2)$$

(19)

$$\begin{array}{cc}\hfill \ddot{T}(u(t))=& T_{j-1}({\ddot{A}}_0{A}_1{A}_2+A_0{\ddot{A}}_1{A}_2+A_0{A}_1{\ddot{A}}_2\\ & +2{\dot{A}}_0{\dot{A}}_1{A}_2+2A_0{\dot{A}}_1{\dot{A}}_2+2{\dot{A}}_0{A}_1{\dot{A}}_2\hfill \end{array}$$

(20)

We define ${\Omega }_k^{k-1}=log(T_{k-1}^{-1}{T}_k)$, the parameters $A_k$ are under these formulas:

$$A_k=exp(B_k(u(t)){\Omega }_{i+k}^{i-1+k})$$

(21)

$${\dot{A}}_k={\dot{B}}_k(u(t)){\widehat{\Omega }}_{i+k}^{i-1+k}{A}_k$$

(22)

$${\ddot{A}}_k={\dot{B}}_k(u(t)){\widehat{\Omega }}_{i+k}^{i-1+k}{\dot{A}}_k+{\ddot{B}}_k(u(t)){\widehat{\Omega }}_{i+k}^{i-1+k}{A}_k$$

(23)

3.3. Accuracy Evaluation of B-Spline Representation

Using Equations (18)–(20), the continuous-time’s position, velocity and acceleration of a moving object can be estimated. The sampling frequency of knots (i.e., the frame-rate of cameras) influences the accuracy of the estimated result. We use HFR cameras with a different frame-rate to capture the known motion of a moving object. Using cubic B-spline motion estimation, the distribution of the position error, velocity error and acceleration error in three axes of the world coordinate are shown in Figure 4.

With the initial velocity at around 30 m/s, with a captured frame-rate of over 500 FPS, the velocity error of the B-splines estimation can be controlled under 16 cm/s. The percentage error of the position, velocity and acceleration are under 1%.

4. Experiments and Evaluation

4.1. Experiments Setup of Explosion Simulation

We used two balloons with coffee beans installed between them to simulate the explosion fragments. The reason why we installed coffee beans between two balloons is because the broken shell’s initial velocity is slow. With around 0.1 g of coffee beans, a high speed scattered motion can be created. Similar to the regular explosion presented in the introduction, our explosion followed these four similar stages:

• The balloons are inflated in the inner shell and the inflation behavior is mainly radial;
• The elastic limit causes breakage of the balloons;
• Because of the elastic contraction of the balloon, the pressure difference between the inside and outside is constructed at the same time;
• The shells of the balloons are randomly broken, and the coffee beans are distributed randomly during the inflation. After the balloons are broken, the coffee beans between these two balloons are scattered with a high initial velocity due to the elasticity and pressure difference.

We used two Emergent Vision Technology (EVT) HFR cameras (10 GigE, HT-2000). When using these two cameras for experiments, the frame-rate can be set from 1080 FPS to 91 FPS, where the resolution of these cameras are from 512 × 320 to 2048 × 1088, respectively. HFR cameras captured the scattered motion of the explosion. We also deployed a monitoring module composed of two long-wavelength infrared (LWIR) cameras and one large FOV visible camera under the regular frame-rate (60 FPS). The covering FOV of this module is 120 degrees in both the visible band and LWIR band. The monitoring module helped us (1) monitorthe explosion’s environment for safety concerns, (2) control the stop time of the HFR cameras to save storage space, and (3) capture the possible landing position of each fragment for further research. Our instruments and constructed balloons are shown in Figure 5.

4.2. Evaluation

The procedure of the continuous-time fast motion estimation is detailed as follows, shown in Figure 6, where:

(a)

Explosion fragments’ motion are captured by the HFR cameras with a related frame-rate setting strategy (synchronization or multiplication); the large FOV LWIR cameras monitor the explosion (shown in Figure 6a)

(b)

With recording images from both HFR cameras, we use the dense optical flow: the Farneback method [27] is used to extract the motion area of the images at each time stamp j. The pixel motion of different fragments are analyzed (shown in Figure 6b). Then we combine the HOG feature extraction and the connected component at every extracted motion area from the optical flow estimation. The image position (area) of fragments at each captured frame are extracted, i.e., the dataset of each camera’s observed position $\left\{b_{ij}^k\right\}$ of fragment k is established;

(c)

The bundle adjustment is optimized based on the HFR cameras’ working strategy, which has been detailed in the previous section. Figure 6c demonstrates the variation of three key parameters (damping factor, norm of reprojection error $\Vert F\Vert$, and ${\Vert g\Vert }_{\infty }$, from top to bottom) of the BA along the iteration step. From the BA optimization, the 3D positions $\left\{P{W}_j^k\right\}$ of fragment $k$ at each captured time stamp j are established.

(d)

With optimized 3D positions, the continuous-time motion (position, velocity, acceleration) of each fragment is estimated by cubic B-spline representation.

The accuracy of our BA optimization with different frame-rate strategy is presented in this section. We compare synchronization frame-rate bundle adjustment (denoted as SBA) and bundle adjustment with frame-rate magnitude (denoted as BAM) to stereo calculation directly. Euclidean distance is deployed to evaluate the position error, while the norm of the reprojection error from each camera evaluates the accuracy of the proposed methods. We use boxplot to demonstrate the distribution of the position error, shown in Figure 7a, and the reprojection error as shown in Figure 7b (Where -Ci means reprojection in camera i).

The related data of the mean error of position (MEP), mean square error of position (MSEP), mean error of reprojection error (MERE) and mean square error of reprojection error (MESRE) in pixel are shown in

Table 1. Accuracy evaluation of BA.

Method	Stereo	SBA	BAM
Duration	538 ms	538 ms	883 ms
MSP (mm)	33.29	20.94	25.88
MESP (mm)	31.44	12.03	12.57
MERE-C1	0.040	0.00012	0.00053
MSERE-C1	0.39	0.00014	0.00072
MERE-C2	0.041	0.00011	0.00079
MSERE-C2	0.41	0.00014	0.0015

.

Figure 7 and Table 1 demonstrate that by using bundle adjustment optimization, the accuracy and stability of the trajectory estimation is highly improved, especially using the frame synchronization strategy (SBA). BAM can efficiently increase the motion’s recording time. Comparing to SBA, the lack of observed motion from camera one at certain time stamps for BAM causes roughly 5 mm of accuracy degradation of the position, 0.001 pixel in MESRE. This is why the outlier of BAM-C2 is significantly high. When captured motion is around the camera’s center area (around 45 pixels), the distortion affection is small, and the position calculated by stereo directly can achieve related high accuracy. Bundle adjustment optimization updates the intrinsic parameters instead of the position around these areas. This is why the minimum position error of stereo calculation directly is lower than BA’s optimization. This is the only drawback of trajectory estimation using bundle adjustment.

A representative of one fragment’s motion estimated by SBA and BAM with uncertainty bounds (3$\mathsf{\sigma }$) is shown in Figure 8. We plot the whole 419 ms from the explosion to being scattered out of the cameras’ FOV. The sample frame-rate for SBA of both HFR cameras is 540 fps, with 1024 × 480 resolution. The frame-rate for BAM is: one camera under 540 fps, with 1024 × 480 resolution, another under 1080 fps, with 512 × 320 resolution.

The sampling data calculated by B-spline representation is six times the higher frame-rate, e.g., 6480 FPS, which exceeds the limits of our HFR cameras. The uncertainty data of five fragments scattered in this explosion are shown in

Table 2. The uncertainty data of five fragments using synchronization frame-rate bundle adjustment (SBA).

ID	$3\mathsf{\sigma }$ of PWx (mm)	$3\mathsf{\sigma }$ of PWy (mm)	$3\mathsf{\sigma }$ of PWy (mm)	Momentum (kg m/s) × 10−4	Force (N)
1	63.2	22.5	29.1	8.04	0.0019
2	31.4	31.5	43.3	6.19	0.0015
3	59.5	43.2	24.3	10.2	0.0024
4	32.1	27.0	32.8	6.62	0.0016
5	48.0	19.2	51.2	7.39	0.0018

and

Table 3. The uncertainty data of five fragments using bundle adjustment with frame-rate magnitude (BAM).

ID	$3\mathsf{\sigma }$ of PWx (mm)	$3\mathsf{\sigma }$ of PWy (mm)	$3\mathsf{\sigma }$ of PWy (mm)	Momentum (kg m/s) × 10−4	Force (N)
1	79.8	33.6	42.8	8.04	0.0019
2	22.8	29.2	30.6	6.20	0.0015
3	69.8	43.2	18.0	10.1	0.0024
4	61.8	27.0	32.8	6.63	0.0016
5	13.9	69.2	34.2	7.38	0.0018

. All these supplementary data are calculated by B-spline representation. Based on estimated velocity and acceleration, the momentum and force of each fragment can also be provided. These physical parameters are very important for damage assessment, etc.

5. Discussion and Conclusions

In this paper, a continuous-time explosion fragments motion estimation method is proposed. We balanced the FOV and the frame-rate of the HFR camera, designed a high-speed vision system, and captured the moment of the explosion and the motion of the high-speed fragments. Basing on our frame-rate strategy, an adjusted bundle adjustment, i.e., SBA and BMA are proposed for trajectory estimation of high speed. With the optimized trajectory, a cubic B-spline representation of the continuous-time position, velocity and acceleration is proposed for this fast motion. Furthermore, the momentum and force can also be estimated at any given time. With these quantization parameters, the damage assessment of fragments can be evaluated.

The experiments show that our adjusted bundle adjustment cooperated with B-spline representation, high accuracy and robust position, that the velocity and acceleration can be estimated at any given time, and that the related error of the estimated motion can be controlled under 1% by using high frame-rate cameras.