A Spatial-Motion-Segmentation Algorithm by Fusing EDPA and Motion Compensation

Abstract

Motion segmentation is one of the fundamental steps for detection, tracking, and recognition, and it can separate moving objects from the background. In this paper, we propose a spatial-motion-segmentation algorithm by fusing the events-dimensionality-preprocessing algorithm (EDPA) and the volume of warped events (VWE). The EDPA consists of depth estimation, linear interpolation, and coordinate normalization to obtain an extra dimension (Z) of events. The VWE is conducted by accumulating the warped events (i.e., motion compensation), and the iterative-clustering algorithm is introduced to maximize the contrast (i.e., variance) in the VWE. We established our datasets by utilizing the event-camera simulator (ESIM), which can simulate high-frame-rate videos that are decomposed into frames to generate a large amount of reliable events data. Exterior and interior scenes were segmented in the first part of the experiments. We present the sparrow search algorithm-based gradient ascent (SSA-Gradient Ascent). The SSA-Gradient Ascent, gradient ascent, and particle swarm optimization (PSO) were evaluated in the second part. In Motion Flow 1, the SSA-Gradient Ascent was 0.402% higher than the basic variance value, and 52.941% faster than the basic convergence rate. In Motion Flow 2, the SSA-Gradient Ascent still performed better than the others. The experimental results validate the feasibility of the proposed algorithm.

1. Introduction

As a new type of sensing imaging device, event cameras, such as the dynamic vision sensor (DVS) or dynamic and active-pixel vision sensor (DAVIS) [1,2], are different from traditional cameras that shoot scenes or objects at a fixed frame rate [3]. The moving objects and light-intensity changes are focused on by event cameras. In general, the intensity change of each pixel is transmitted by a stream of asynchronous events. The information carried by an event includes the pixel coordinates $\left(x,y\right)$, trigger time ($t$), and positive- or negative-event polarity ($p$). Compared with traditional cameras, event cameras have many advantages, for example, latency in microseconds, low motion blur, and high dynamic range (HDR) [4]. In computer vision, event cameras are often used in many fields, such as detection, tracking, recognition, and simultaneous localization and mapping (SLAM) [5,6].

In motion segmentation, independent motion-related pixels can be separated from the video sequence. Based on different motion types, the foreground objects are divided from the background by clustering these pixels [3,7]. Traditional vision-based processing algorithms usually assume that the video is shot under ideal circumstances [8,9,10]. Many factors are not taken into account by traditional vision tasks (e.g., moving speed, stability, and lighting conditions). In practical-application scenarios, the performance of motion-segmentation algorithms will be severely affected by fast-moving devices, such as drones and self-driving vehicles. The problem is well handled by motion segmentation based on event cameras [11]. The task of spatial-motion segmentation aims to separate the tracked feature points according to the respective rigid three-dimensional (3D) motion [12].

In this paper, we propose a framework by fusing the EDPA and 3D-motion-compensation approach to establish the spatial-motion-segmentation algorithm. The advantages of 3D information and motion segmentation are well combined by the proposed algorithm, which can not only satisfy the accuracy and stability requirements of vision works, but also separate the objects from the redundant background. The datasets of the experiments derive from the ESIM, which is an event-camera simulator [13]. After collecting the datasets, the pseudo-depth of the per-event is obtained by the EDPA in each frame of a pixel. The VWE is conducted by accumulating the warped events, and the iterative-clustering algorithm is presented to maximize the contrast in the VWE. The proposed algorithm has great segmentation effectiveness in low-speed- and high-speed-motion scenes, and in low-light and low-exposure scenes (e.g., high noon and night scenes). The main contributions of this paper are as follows:

(1)

In order to inexpensively and conveniently extend 3D (2D plane ($x,y$) and time ($t$)) to 4D (3D space ($x,y,z$) and time ($t$)) for events without an RGB-depth-map (RGB-D) camera [14], or based on the light detection and ranging system (LiDAR), the EDPA is proposed for application in a variety of scenarios in this paper;

(2)

Compared with the traditional frame-based and plane-based methods [15,16,17], the proposed algorithm combines the EDPA and 3D-motion compensation to accomplish the spatial-motion segmentation. The proposed algorithm provides a more refined segmentation model, and the segmentation accuracy can be greatly improved;

(3)

The SSA-Gradient Ascent is presented to maximize the contrast in the VWE in this paper, which blends the advantages of SSA [18] and gradient ascent to obtain a better fitness value and faster convergence rate than other algorithms, such as PSO (see Section 4.2).

The remainder of this paper is organized as follows. In Section 2, we review some related works on depth estimation and motion segmentation. The mathematical model and main methods are presented in Section 3. Section 4 presents the process of the experiments. The conclusions are shown in Section 5.

2. Related Work

Depth estimation is one of the most important components of the EDPA. In [19], the authors introduced some of the high-accuracy and high-speed structured-light 3D-reconstruction methods. Deep learning enables us to accomplish visual tasks that are difficult to achieve by traditional geometry-based algorithms. Muhammad, K. et al. proposed a bidirectional long short-term memory (BiLSTM)-based attention mechanism with a dilated convolutional neural network (DCNN) to recognize the different human actions in videos [20]. A primary-prioritized recurrent deep reinforcement learning algorithm for dynamic-spectrum access based on cognitive-radio (CR) technology was proposed in [21]. In [22], a game-theory strategy was proposed to improve the energy-consumption performance of the MEC system. David et al. presented the first monocular depth estimation based on a convolutional neural network (CNN) [23]. Ibraheem et al. presented supervised learning without coupled geometric information [24]. To overcome the disadvantages of the RGB information-only-based method, an improved moving-object-detection method was proposed in [25]. Depth estimation based on event cameras has been gradually developed in recent years. In [26], the authors introduced the event-based multiview stereo (EMVS) to estimate semidense 3D structures with known trajectories. A novel 3D-reconstruction method was researched by Kim et al., which can perform real-time 3D reconstruction from a single hand-held event camera with no additional sensing [27]. In [28], a unifying framework was studied to solve several computer-vision problems with event cameras: motion, depth, and optical-flow estimation.

A variety of segmentation algorithms have been proposed by researchers to recognize and separate the objects from the background in the scene. Wang et al. proposed the EvDistill to learn a student network on the unlabeled and unpaired event data (target modality) via knowledge distillation (KD) [29]. In [30], the authors introduced a framework for the segmentation of human-motion sequences based on sensor networks. Li et al. presented a novel approach, which is the motion segmentation of 3D-active-contour registration [31]. Event cameras are also widely made use of in motion segmentation. Stoffregen et al. presented a motion-compensation-based approach to establish the image of warped events (IWE) for segmentation [10]. The dataset was proposed by Mitrokhin et al. to perform the motion segmentation of event cameras [32]. The authors developed a technique for generalized motion segmentation based on spatial statistics across timeframes [33]. In [34], Zhou et al. introduced an event-based motion-segmentation method with spatiotemporal graph cuts, which cast the problem as an energy minimization one involving the fitting of multiple motion models.

3. Main Methods

The objectives of our research are to extend 4D events and realize spatial-motion segmentation. We assume the prior knowledge of the scene that we cannot know. Because the information carried by the per-event is limited, the events in the set $\epsilon ={\left\{e_k\right\}}_{k=1}^{N_e}$ are processed so that we can obtain enough information. The depth maps of the images are obtained by depth estimation. The initial motion parameter $\mathit{\theta }={\left[{\theta }_x,{\theta }_y,{\theta }_z\right]}^T$ can be obtained by motion-flow estimation [35]. The contrast maximization is a framework that provides state-of-the-art results on several event-based computer-vision tasks [36]. A coherent motion is represented by a cluster. The problems we need to address are to classify the events into distinct clusters and maximize the contrast in the VWE.

3.1. Events from 3D to 4D by EDPA

The target of EDPA is to obtain an extra dimension ($Z$) for events. Firstly, the pre-pixel depths of $Fram{e}_1$ and $Fram{e}_2$ are derived by depth estimation as $Z_{fram{e}_1}$ and $Z_{fram{e}_2}$, respectively. As shown in Figure 1a, we calculate the depth of the per-event by Equation (1). At the same time, the depth ($Z$) and events coordinates $\left(x,y\right)$ are not in the same coordinate system. The coordinates need to be normalized by transformation. Then, we transform the pixel coordinates and depth ($Z$) into the same coordinate system through a global transformation, as shown in Equations (2) and (3) and Figure 1b. Finally, we have an extra dimension, as shown in Equation (4):

$$Z_k=\frac{t_k-T_{fram{e}_1}}{T_{fram{e}_2}-T_{fram{e}_1}}\left|Z_{fram{e}_2}-Z_{fram{e}_1}\right|$$

(1)

where $t_k$ is the time when the event ($k$) is triggered; $T_{fram{e}_j}$ is the time of the frame ($j$) in a video; $Z_{fram{e}_j}$ is the pixel depth on the frame ($j$), and $Z_k$ is the depth of the event ($k$);

$$\frac{x_k}{X_k}=\frac{y_k}{Y_k}=\frac{f}{Z_k}$$

(2)

where $\left(x_k,y_k\right)$ is the raw coordinate of the event ($k$); $\left(X_k,Y_k,Z_k\right)$ are the global coordinates of the event ($k$); $f$ is the scale factor;

Writing Equation (2) in matrix form, we have:

$$Z_k\left[\begin{array}{c}{x}_k\\ y_k\\ 1\end{array}\right]=\left[\begin{array}{ccc}f& 0& 0\\ 0& f& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}_k\\ Y_k\\ Z_k\end{array}\right]$$

(3)

$${\left\{e_k=\left(x_k,y_k,t_k\right)\right\}}_{k=1}^{N_e}\to {\left\{e_k=\left(X_k,Y_k,Z_k,t_k\right)\right\}}_{k=1}^{N_e}$$

(4)

where $e_k$ is the event ($k$) in the set, and $N_e$ is the total number of events.

3.2. Motio-Flow Estimation

A method of voxel motion in space over time is described by the motion flow. The grayscale-invariance assumption is shown in Figure 2.

$${\mathit{I}}_{t_1}(x_1,y_1,z_1,t_1)={\mathit{I}}_{t_2}(x_2,y_2,z_2,t_2)$$

(5)

where ${\mathit{I}}_{t_k}$ is the grayscale value at the time ($t_k$), and $\left(x,y,z\right)$ are the coordinates of the voxel in space.

As shown in Figure 2, according to the grayscale-invariance assumption, the movement of a voxel from $t$ to $t+dt$ is shown in Equation (6):

$$\mathit{I}\left(x+dx,y+dy,z+dz,t+dt\right)=\mathit{I}\left(x,y,z,t\right)$$

(6)

When Taylor expansion is performed on the left side of the above equation and retains the first-order term, we will obtain:

$$\mathit{I}\left(x+dx,y+dy,z+dz,t+dt\right)\approx \mathit{I}\left(x,y,z,t\right)+\frac{\partial \mathit{I}}{\partial x}dx+\frac{\partial \mathit{I}}{\partial y}dy+\frac{\partial \mathit{I}}{\partial z}dz+\frac{\partial \mathit{I}}{\partial t}dt$$

(7)

Comparing the right sides of Equations (6) and (7), we have:

$$\frac{\partial \mathit{I}}{\partial x}dx+\frac{\partial \mathit{I}}{\partial y}dy+\frac{\partial \mathit{I}}{\partial z}dz+\frac{\partial \mathit{I}}{\partial t}dt=0$$

(8)

Both sides of the above equation are divided by $dt$, and $\frac{\partial \mathit{I}}{\partial t}$ is moved to the right side of the equation:

$$\frac{\partial \mathit{I}}{\partial x}{\theta }_x+\frac{\partial \mathit{I}}{\partial y}{\theta }_y+\frac{\partial \mathit{I}}{\partial z}{\theta }_z=-\frac{\partial \mathit{I}}{\partial t}$$

(9)

where ${\theta }_x,{\theta }_y,{\theta }_z$ are the velocities in the $x,y,z$ directions, respectively.

To obtain enough equations to estimate the $\mathit{\theta }$, three planes ($S_1,S_2,S_3$) are created as $\left(x,y,t\right),\left(y,z,t\right),\left(z,x,t\right)$, respectively, and their grayscale values are ${\mathit{I}}_1,{\mathit{I}}_2,{\mathit{I}}_3$, respectively, which are shown in Figure 3. Equation (9) can be written as:

$$\frac{\partial {\mathit{I}}_{\mathit{k}}}{\partial i}{\theta }_i+\frac{\partial {\mathit{I}}_{\mathit{k}}}{\partial j}{\theta }_j=-\frac{\partial {\mathit{I}}_{\mathit{k}}}{\partial t}$$

(10)

where $\left(i,j\right)$ is the element that belongs to $\left\{\left(x,y\right),\left(y,z\right),\left(z,x\right)\right\}$, and $k$ is the index of the set that belongs to $\left\{1,2,3\right\}$. For example, when $k$ = 1, $\left(i,j\right)=\left(x,y\right)$, Equation (10) can be written as:

$$\frac{\partial {\mathit{I}}_1}{\partial x}{\theta }_x+\frac{\partial {\mathit{I}}_1}{\partial y}{\theta }_y=-\frac{\partial {\mathit{I}}_1}{\partial t}$$

(11)

Equation (10) can be written as a matrix form:

$$\left(\begin{array}{ccc}{I}_{1,x}& I_{1,y}& 0\\ 0& I_{2,y}& I_{2,z}\\ I_{3,x}& 0& I_{3,z}\end{array}\right)\left[\begin{array}{c}{\theta }_x\\ {\theta }_y\\ {\theta }_z\end{array}\right]=-\left(\begin{array}{c}\raisebox{1ex}{$\partial I_1$}\!\left/ \!\raisebox{-1ex}{$\partial t$}\right.\\ \raisebox{1ex}{$\partial I_2$}\!\left/ \!\raisebox{-1ex}{$\partial t$}\right.\\ \raisebox{1ex}{$\partial I_3$}\!\left/ \!\raisebox{-1ex}{$\partial t$}\right.\end{array}\right)$$

(12)

where $I_{1,x}$ is $k$ = 1 and the partial derivative of $I_1$ to $x$.

$I_{k,\left(i,j\right)}$ are recorded as $a_k$ and $b_k$, and the above equation can be written as:

$$\left(\begin{array}{ccc}{a}_1& b_1& 0\\ 0& a_2& b_2\\ b_3& 0& a_3\end{array}\right)\left[\begin{array}{c}{\theta }_x\\ {\theta }_y\\ {\theta }_z\end{array}\right]=-\left(\begin{array}{c}{c}_1\\ c_2\\ c_3\end{array}\right)$$

(13)

By calculating the optical flow, ${\theta }_x$ can be obtained such that:

$$\mathit{\theta }=\left(\begin{array}{c}{\theta }_x\\ \raisebox{1ex}{$-a_1{\theta }_x+c_1$}\!\left/ \!\raisebox{-1ex}{$b_1$}\right.\\ \raisebox{1ex}{$-b_3{\theta }_x+c_3$}\!\left/ \!\raisebox{-1ex}{$a_3$}\right.\end{array}\right)=\left(\begin{array}{c}{\theta }_x\\ {\theta }_y\\ {\theta }_z\end{array}\right)$$

(14)

3.3. 3D-Motion-Compensation and the Iterative-Clustering Algorithm

As shown in Figure 4, the objects are warped in the direction of the optical flow, and the image becomes sharp; in the contrary case, the image becomes a blur. The warped events are given by:

$${\mathbf{x}}_k={\left(x_k,y_k,z_k\right)}^T$$

(15)

$${\mathbf{x}}_k^{\prime }={\mathbf{x}}_k-\left(t_k-t_{ref}\right)\mathit{\theta }={\mathbf{x}}_k-\Delta t_k\mathit{\theta }$$

(16)

where $t_k$ is the trigger time of the event ($k$), and ${\mathbf{x}}_k^{\prime }$ is the position where the event (${\mathbf{x}}_k$) is warped.

Similar to the IWEs, we accumulate the warped events according to the weight values. The definition of the VWE is:

$$V_j\left(\mathbf{x}\right)={\displaystyle \sum _{k=1}^{N_e}{p}_{kj}}\delta \left(\mathbf{x}-{\mathbf{x}}_{kj}^{\prime }\right)$$

(17)

where $p_{kj}$ represents the probability that the event ($k$) belongs to the optical flow ($j$), and $\delta =\{\begin{array}{c}1,\mathbf{x}-{\mathbf{x}}_{kj}^{\prime }=0\\ 0,otherwise\end{array}$ represents the Dirac function. ${\mathbf{x}}_{kj}^{\prime }$ is the location, where the event ($k$) is warped along the optical flow ($j$).

Variance is employed to evaluate the contrast in this paper. To make the VWE sharper, the contrast is maximized as:

$$\mathrm{Var}\left(V_j\right)=\frac{1}{\left|\mathrm{\Omega }\right|}{{\displaystyle {\int }_{\mathrm{\Omega }}\left(V_j\left(\mathbf{x}\right)-{\mu }_{V_j}\right)}}^{2}d\mathbf{x}$$

(18)

where $\mathrm{\Omega }$ is the image volume, and ${\mu }_{V_j}$ denotes the mean of the VWE over the image volume ($\mathrm{\Omega }$).

For the discrete VWE, the above equation can be written as:

$$\mathrm{Var}\left(V_j\right)={{\displaystyle \sum _{a,b,c\in \mathrm{\Omega }}\left(V_{abc}-{\mu }_{V_j}\right)}}^2$$

(19)

where $a,b,c$ is the index of the voxel in spacetime.

Now, the $\mathit{\theta }$ and $\mathbf{P}$ are needed to initialize and iterate. The elements of the $\mathbf{P}$ are the event-cluster membership probabilities, and the elements of the $\mathit{\theta }$ are the motion parameters. Because our clustering type is motion flow, the k-means algorithm was used to find the cluster center to initialize the $\mathit{\theta }$. The loss function of the k-means algorithm is defined as follows:

$$J\left(c,\lambda \right)=\min {\displaystyle \sum _{i=1}^M{\Vert x_i-{\lambda }_{c_i}\Vert }^2}$$

(20)

where $x_i$ represents the sample point ($i$), $c$ is the cluster to which $x_i$ belongs, ${\lambda }_{c_i}$ represents the center point corresponding to the cluster, and $M$ is the total number of samples.

To make the per-event have the same probability of being classified into the cluster, we initialize the $\mathbf{P}$:

$$\mathbf{P}=\frac{1}{N_l}$$

(21)

where $N_l$ is the clusters number.

The above process can be summarized as the need to optimize the $\mathit{\theta }$ and $\mathbf{P}$ that maximize the variance, and the $\mathit{\theta }$ and $\mathbf{P}$ are updated by the iterative-clustering algorithm. Each iteration of the iterative-clustering algorithm has two steps: a fixed $\mathbf{P}$ to update the $\mathit{\theta }$, and a fixed $\mathit{\theta }$ to update the $\mathbf{P}$:

$$\left({\mathit{\theta }}^{\ast },{\mathbf{P}}^{\ast }\right)=\underset{\left(\mathit{\theta },\mathbf{P}\right)}{\arg \max }{\displaystyle \sum _{j=1}^{N_l}\mathrm{Var}\left(V_j\right)}$$

(22)

The iteration of the $\mathit{\theta }$ makes use of the SSA-Gradient Ascent. In SSA, the producers’ location update is given by:

$${\theta }_{i,j}^{t+1}=\{\begin{array}{cc}{\theta }_{i,j}^t\cdot e^{\left(\frac{-i}{\alpha \cdot ite{r}_{\max }}\right)}\hfill & if R_2<ST\hfill \\ {\theta }_{i,j}^t+q\cdot l\hfill & if R_2\ge ST\hfill \end{array}$$

(23)

where $t$ is the current iteration number, and ${\theta }_{i,j}^t$ represents the value of the dimension ($j$) of the sparrow ($i$) at iteration ($t$). The largest number of iterations is represented by $ite{r}_{\max }$.$\alpha \in \left(0,1\right]$, and $q$ is a random number obeying a normal distribution. $R_2\in \left[0,1\right]$ represents the alarm value, and $ST\in \left[0.5,1\right]$ represents the safety threshold ($l\in {ℝ}^{1\times d}$).

The scroungers are updated by:

$${\theta }_{i,j}^{t+1}=\{\begin{array}{cc}q\cdot e^{\left(\frac{{\theta }_{worst}^t-{\theta }_{i,j}^t}{i^2}\right)}& if \mathrm{i}>\frac{n}{2}\hfill \\ {\theta }_p^{t+1}+\left|{\theta }_{i,j}^t-{\theta }_p^{t+1}\right|\cdot n^{+}\cdot l& otherwise\hfill \end{array}$$

(24)

where ${\theta }_p$ is the optimal position currently occupied by the producers; ${\theta }_{worst}$ represents the present global worst position; $n\in {ℝ}^{1\times d}$, and $n^{+}=n^T{\left(n{n}^T\right)}^{-1}$.

The sparrows that are aware of the danger are given by:

$${\theta }_{i,j}^{t+1}=\{\begin{array}{cc}{\theta }_{best}^t+\beta \cdot \left|{\theta }_{i,j}^t-{\theta }_{best}^t\right|& if f_i>f_g\hfill \\ {\theta }_{i,j}^t+K\cdot \left(\frac{\left|{\theta }_{i,j}^t-{\theta }_{worst}^t\right|}{\left(f_i-f_w\right)+\epsilon }\right)& if f_i=f_g\hfill \end{array}$$

(25)

where ${\theta }_{best}$ represents the present global best position; $\beta$ is the step-size-controlled parameter; $K\in \left[-1,1\right]$; $f_g$ and $f_W$ are the current global fitness values, which represent the best and worst fitness values, respectively; $\epsilon$ is a constant that avoids zero in the denominator.

The SSA is used to find the better $\mathit{\theta }$ in the global area. Gradient ascent is used to obtain the best in the local area, as shown in (25).

$$\mathit{\theta }\leftarrow \mathit{\theta }+\mu {\nabla }_{\mathit{\theta }}\left({\displaystyle \sum _{j=1}^{N_l}\mathrm{Var}\left(V_j\right)}\right)$$

(26)

where $\mu$ is the iteration step size.

The iteration of the $\mathbf{P}$ is given by:

$$p_{kj}=\frac{c_j({\mathbf{x}}_k^{\prime }({\mathit{\theta }}_j))}{{\displaystyle \sum _{i=1}^{N_l}{c}_i({\mathbf{x}}_k^{\prime }({\mathit{\theta }}_i))}}$$

(27)

where $c_i$ is the VWE of the $i$-th; ${\mathbf{x}}_k^{\prime }({\mathit{\theta }}_j)$ is the position where the event ($k$) is warped along the 3D-motion flow ($j$); $p_{kj}$ is the probability that the event ($k$) belongs to the $j$-th.

The algorithm flowchart is shown in Figure 5. The Algorithm 1 and Figure 5 show the overall process of our method. Variance convergence can usually be judged by the following conditions:

(1)

The variance curves tend to be horizontal, or the fluctuation changes a little;

(2)

In the variance values, more than a certain constant can be considered variance convergence;

(3)

The iterations will be interrupted when the number of iterations reaches a certain value.

Algorithm 1. Spatial-Motion Segmentation

Start

Input: Raw events, set ${\epsilon }_1={\left\{e_k=\left(x_k,y_k,t_k,p_k\right)\right\}}_{k=1}^{N_e}$; original images; the number of clusters ($N_l$).

Output: Event-cluster membership probabilities ($\mathbf{P}$), and motion parameters ($\mathit{\theta }$).

Procedure:

1:For each image, do depth estimation.

End for;

2: For$k\leftarrow 1$ to $N_e$, do:

$Z_k=\frac{t_k-T_{fram{e}_1}}{T_{fram{e}_2}-T_{fram{e}_1}}\left|Z_{fram{e}_2}-Z_{fram{e}_1}\right|$

$Z_k\left[\begin{array}{c}{x}_k\\ y_k\\ 1\end{array}\right]=\left[\begin{array}{ccc}f& 0& 0\\ 0& f& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}_k\\ Y_k\\ Z_k\end{array}\right]$

End for:

${\epsilon }_2={\left\{e_k=\left(X_k,Y_k,Z_k,t_k,p_k\right)\right\}}_{k=1}^{N_e}$

3:For$e_k$, do:

computing ${\mathit{\theta }}_k$ according to $\mathit{\theta }={\left({\theta }_x,\frac{-a_1{\theta }_x+c_1}{b_1},\frac{-b_3{\theta }_x+c_3}{a_3}\right)}^T$

End for:

4: For$k\leftarrow 1$ to $N_e$, do:

warping $e_k$ according to ${\mathbf{x}}_k^{\prime }={\mathbf{x}}_k-\left(t_k-t_{ref}\right)\mathit{\theta }={\mathbf{x}}_k-\Delta t_k\mathit{\theta }$

End for:

5: For each warped $e_k$, do weighted accumulate:

$V_j\left(\mathbf{x}\right)={\displaystyle \sum _{k=1}^{N_e}{p}_{kj}}\delta \left(\mathbf{x}-{\mathbf{x}}_{kj}^{\prime }\right)$

6: For$i\leftarrow 1$ to $\infty$, do:

If the variance is not converged, then:

update the $\mathbf{P}$ using $p_{kj}=\frac{c_j({\mathbf{x}}_k^{\prime }({\mathit{\theta }}_j))}{{\displaystyle \sum _{i=1}^{N_l}{c}_i({\mathbf{x}}_k^{\prime }({\mathit{\theta }}_i))}}$

update the $\mathit{\theta }$ according to SSA-Gradient Ascent:

computing $Var\left(V_j\right)={{\displaystyle \sum _{a,b,c\in \mathrm{\Omega }}\left(V_{abc}-{\mu }_{V_j}\right)}}^2$

End if:

$i\leftarrow i+1$

End for:

End

4. Experiments

The performance of the proposed algorithm was evaluated by two experiments. The following experiments were conducted to verify the feasibility of the proposed algorithm. We utilized our datasets, which were derived from the event-camera simulator (ESIM). Our experimental process consisted of two parts: in the first part, the spatial-motion segmentation was implemented in the exterior scene and interior scene, and in the second part, the influence of the optimization-algorithm performance on the variance in the VWEs was estimated by SSA-Gradient Ascent, PSO, and gradient ascent.

4.1. Experiment 1: Spatial-Motion Segmentation of Exterior and Interior Scenes

The Monodepth2 [38] depth-estimation network was utilized to predict the exterior scene in this paper, which was in a situation where the camera was stationary and only had moving objects. The predicted result is shown in Figure 6a, which displays the depth map of the first keyframe. The EDPA needs to utilize two images to expand the dimension of the events. After we obtained the depth map of the first key frame, the second key frame was selected for the depth estimation based on the trigger time of the 15,000th event. The spatial-motion segmentation was performed as shown in Figure 6b,c. Figure 6b is warped in the wrong direction, which is not the motion-flow direction of the objects. Figure 6c is the segmentation result by warping in the correct direction. When the algorithm iterates to around the 10th round, the variance has been maximized, and we can consider that the variance has converged. The iterative plots of variance are shown in Figure 6d. The changes in the $\mathit{\theta }$ are shown in Figure 6e–g. Six curves represent lateral-velocity vectors (${\mathit{\theta }}_{\mathit{x}}$), column-velocity vectors (${\mathit{\theta }}_{\mathit{y}}$) and velocity vectors on the z-axis (${\mathit{\theta }}_{\mathit{z}}$).

The fully convolutional residual network (FCRN) [39] was applied to predict the interior scene, as shown in Figure 7a. In Figure 7b,c, two carts along the upper-left and lower-right corners of the table with different directions and speeds were successfully separated into different classes in 3D space, and little redundant background information was segmented into the foreground. Experimenting with the objects moving in different directions demonstrated the feasibility of the proposed algorithm in the interior scene.

4.2. Experiment 2: Comparison of the Effects of Different Optimization Algorithms

The SSA has a preferable ability in global optimization, while it is weak in local searches. The gradient ascent has an excellent performance in local optimization, but it easily falls into the local extreme value. The SSA-Gradient Ascent combines the advantages of SSA and gradient ascent. The updates of the $\mathit{\theta }$ by the SSA-Gradient Ascent, PSO, and gradient ascent were evaluated in this part, as shown in Figure 8. The variance-convergence rate of the algorithms and the value of the variance after convergence are listed in

Table 1. Comparison of algorithm performances.

		Motion Flow 1		Motion Flow 2
Variance value	SSA-Gradient Ascent (ours)	0.00200104	+0.800%	0.00189911	+0.819%
	PSO	0.00198522	0%	0.00188369	0%
	Gradient Ascent	0.00197723	−0.402%	0.00189746	+0.731%
Convergence speed (iteration no.)	SSA-Gradient Ascent (ours)	8	+52.941%	7	+46.154%
	PSO	17	0%	13	0%
	Gradient Ascent	None		None

. As presented by the data in Table 1 and Figure 8, we chose the PSO method as the benchmark value. In Motion Flow 1, the proposed algorithm outperforms the basic value, with the gradient ascent 0.402% lower than the baseline. In Motion Flow 2, the gradient ascent is 0.731% higher than the basic value, while the proposed algorithm is 0.819% higher than the basic value. The same initial value of the $\mathit{\theta }$ is used by the above algorithms, and the maximum-variance value of the SSA-Gradient Ascent indicates that the proposed algorithm is less dependent on initialization. The gradient-ascent method does not converge until 19 iterative rounds. Compared with PSO, the proposed algorithm’s convergence rate was 52.941% higher in Motion Flow 1, and 46.154% higher in Motion Flow 2. The better performance of the proposed algorithm was confirmed according to this experiment.

5. Conclusions

A spatial-motion-segmentation algorithm is proposed in this paper, which has fused the EDPA and 3D-motion-compensation approach. The accuracy for tasks such as feature detection in complex environments is addressed by the proposed algorithm, while the advantages of 3D information and motion segmentation are well combined. The pseudo-depth of the per-event was obtained by the EDPA in each frame of a pixel. The VWE was conducted by accumulating the warped events, and a spatial-motion-segmentation algorithm is presented to maximize the contrast in the VWE. Interior and exterior scenes were segmented in the first part of the experiments. In the second part, the effect on the variance in the VWEs was estimated by SSA-Gradient Ascent, PSO, and gradient ascent. In Motion Flow 1, the SSA-Gradient Ascent was 0.402% higher than the basic variance value, and 52.941% faster than the basic convergence rate. In Motion Flow 2, the gradient ascent was 0.731% higher than the basic value, and 46.154% faster than the basic convergence rate. As a result, the experimental results validate the feasibility of the proposed algorithm. For future research, more complex scenes and conspicuous-spatial-motion segmentation will be studied by us.