Compressive Online Video Background–Foreground Separation Using Multiple Prior Information and Optical Flow

Abstract

In the context of video background–foreground separation, we propose a compressive online Robust Principal Component Analysis (RPCA) with optical flow that separates recursively a sequence of video frames into foreground (sparse) and background (low-rank) components. This separation method operates on a small set of measurements taken per frame, in contrast to conventional batch-based RPCA, which processes the full data. The proposed method also leverages multiple prior information by incorporating previously separated background and foreground frames in an n-${\ell }_1$ minimization problem. Moreover, optical flow is utilized to estimate motions between the previous foreground frames and then compensate the motions to achieve higher quality prior foregrounds for improving the separation. Our method is tested on several video sequences in different scenarios for online background–foreground separation given compressive measurements. The visual and quantitative results show that the proposed method outperforms other existing methods.

1. Introduction

Emerging applications in surveillance and autonomous driving are challenging the existing visual systems to detect and understand objects from visual observations. Video background–foreground separation is one of most important components for object detection, identification, and tracking. In video separation, a video sequence can be separated into a slowly changing background (modeled by $\mathit{L}$ as a low-rank component) and the foreground (modeled by $\mathit{S}$, which is a sparse component). RPCA [1,2] was shown to be a robust method for separating the low-rank and sparse components. RPCA decomposes a data matrix $\mathit{M}$ into the sum of unknown sparse $\mathit{S}$ and low-rank $\mathit{L}$ by solving the Principal Component Pursuit (PCP) [1] problem:

$$\underset{\mathit{L},\mathit{S}}{min}{\parallel \mathit{L}\parallel }_{*}+\lambda {\parallel \mathit{S}\parallel }_1\mathrm{subject}\mathrm{to}\mathit{M}=\mathit{L}+\mathit{S},$$

(1)

where ${\parallel ·\parallel }_{*}$ is the matrix nuclear norm (sum of singular values) and ${\parallel ·\parallel }_1$ is the ${\ell }_1$-norm (sum of absolute values). Many applications of RPCA can be found in computer vision, web data analysis, and recommender systems. However, batch RPCA processes all data samples, e.g., all frames in a video, which demands high computational and memory requirements.

Moreover, the video separation can be improved by taking into account the correlation between consecutive frames. The correlations can be obtained in the form of motions, which manifest as change in information from one frame to the other. Detecting motion is an integral part of the human visual system. A popular and convenient method for estimating motion in computer vision is by using optical flow [3,4,5] by variational methods. The optical flow estimates motion vectors of all pixels in a given frame due to the relative motions between frames. In particular, the motion vectors at each pixel can be estimated by minimizing a gradient-based matching of pixel gray value, combined with a smoothness criteria [3]. Thereafter, the computed motion vectors in the horizontal and vertical directions [4] are used to compensate and predict information in the next frame. For producing highly accurate motions and accurate correspondences between frames, a large displacement optical flow [6], that combines a coarse-to-fine optimization with descriptor matching can be used to estimate the motions from previously separated frames and subsequently use them to support the separation of the current frame.

In order to deal with the video separation in an online manner, we consider an online RPCA algorithm that recursively processes a sequence of frames (a.k.a., the column-vectors in $\mathit{M}$) per time instance. Additionally, we aim at recovering the foreground and background from a small set of measurements rather than a full frame data, leveraging information from a set of previously separated frames. In particular, at time instance t, we wish to separate ${\mathit{M}}_t$ into ${\mathit{S}}_t=[{\mathit{x}}_1{\mathit{x}}_2\dots {\mathit{x}}_t]$ and ${\mathit{L}}_t=[{\mathit{v}}_1{\mathit{v}}_2\dots {\mathit{v}}_t]$, where $[·]$ denotes a matrix and ${\mathit{x}}_t,{\mathit{v}}_t\in {\mathbb{R}}^n$ are column-vectors in ${\mathit{S}}_t$ and ${\mathit{L}}_t$, respectively. We assume that ${\mathit{S}}_{t-1}=[{\mathit{x}}_1{\mathit{x}}_2\dots {\mathit{x}}_{t-1}]$ and ${\mathit{L}}_{t-1}=[{\mathit{v}}_1{\mathit{v}}_2\dots {\mathit{v}}_{t-1}]$ have been recovered at time instance $t-1$. In the next instance t, we have access to compressive measurements of the full frame, a.k.a., vector ${\mathit{x}}_t+{\mathit{v}}_t$, that is, we observe ${\mathit{y}}_t=\mathbf{\Phi }({\mathit{x}}_t+{\mathit{v}}_t)$, where $\mathbf{\Phi }\in {\mathbb{R}}^{m\times n}(m<n)$ is a random projection, ${\mathit{x}}_t$ is the sparse component (foreground) and ${\mathit{v}}_t$ is the low-rank component (background) at time instance t. We proceed with the assumption that video can be seperated into dynamic, sparse foreground components and a static or slowly changing, low-rank background components. The recovery problem at time instance t is thus written [7] as

$$\underset{{\mathit{x}}_t,{\mathit{v}}_t}{min}\parallel \left[{\mathit{L}}_{t-1}{\mathit{v}}_t\right]{\parallel }_{*}+\lambda {\parallel {\mathit{x}}_t\parallel }_1\mathrm{subject}\mathrm{to}{\mathit{y}}_t=\mathbf{\Phi }({\mathit{x}}_t+{\mathit{v}}_t),$$

(2)

where ${\mathit{L}}_{t-1}$ and ${\mathit{S}}_{t-1}$ are given.

1.1. Related Work

Many methods [8,9,10,11,12,13,14] have been proposed to solve the separation problem by advancing RPCA [1]. Incremental PCP [8] processes each column-vector in $\mathit{M}$ at a time, but it needs access to complete data (e.g., full frames) rather than compressive data. A counterpart of batch RPCA that operates on compressive measurements known as Compressive PCP can be found in [15]. The studies in [11,12,13,14,16] aim at solving the problem of online estimation of low-dimensional subspaces from randomly subsampled data for modeling the background. An algorithm to recover the sparse component ${\mathit{x}}_t$ in (2) has been proposed in [17], however, the low-rank component ${\mathit{v}}_t$ in (2) is not recovered per time instance from a small number of measurements.

The alternative method in [18,19] estimates the number of compressive measurements required to recover foreground ${\mathit{x}}_t$ per time instance via assuming the background ${\mathit{v}}_t$ not-varying. This assumption is invalid in realistic scenarios due to variations in illumination or dynamic backgrounds. The online method in [16] works on compressive measurements without taking the prior information into account.

Separating a video sequence or a set of frames using prior information brings about significant improvements in the context of online RPCA [17,20,21]. Some studies on recursive recovery from low-dimensional measurements have been proposed to leverage prior information [17,18,20,22]. The study in [22] provided a comprehensive overview of the domain, reviewing a class of recursive algorithms. A review of all the recent problem formulations is made in [23] with a unified view with Decomposition into Low-rank plus Additive Matrices (DLAM).

The studies in [17,20] used modified-CS [24] to leverage prior knowledge under the condition of slowly varying support and signal values. However, this method as well as the methods in [11,12,14] do not explore the correlations between the current frame and multiple previously separated frames. The recent work in [7] leverages correlations across the previously separated foreground frames. However, displacements between the previous foreground frames and the current frame are not taken into account. These displacements can incur the degradation of the separation performance. An interesting method proposed in [25] makes use of the low-rank and sparse components of a video frame in addition to a dense noise component ($\mathit{G}$), i.e., $\mathit{M}=\mathit{L}+\mathit{S}+\mathit{G}$ for exoplanet detection.

There are other alternative approaches such as, double-constrained RPCA, namely shape and confidence map based RPCA (SCM-RPCA) in [26] that combines saliency map along with RPCA for automated maritime surveillance. Recently a method that incorporates the spatial and temporal sparse subspace clustering into the RPCA framework was developed in [27] and earlier in [28]. Recent studies in [29] proposes a non-parametric approach for background subtraction.

Background subtraction is a widely used technique to separate foreground and background. A detailed review of commonly used techniques in performing background subtraction has been made in  [30]. A survey of all the existing methods in background subtraction has been thoroughly carried out in [31] and another survey [32] is focused on detecting stationary foreground objects. Evaluation of 29 background subtraction methods using the BMC dataset [33] has been performed in [34]. An interesting technique by making use of genetic programming can be found in [35]. This is a powerful method that uses a repertoire of different background subtraction algorithms and chooses a method or a combination of methods appropriately, based on the video sequence in order to obtain better results. Semantics based approach is proposed in [36]. This method improves a background separation algorithm by making use of semantics to identify objects as foregound or background. It is worth noting that, in this method, semantics are used as prior information. In [37], a method is proposed to analyze dynamic background region and reduce the false positives by checking the false positives again. If foreground is detected in the dynamic background region, it is removed by re-checking false positives from the dynamic background samples. Ref. [38] makes use of a different approach to perform background subtraction. This method evaluates the importance of each background sample in an online manner based on the recurrence among local all the local observations. A persistence based word dictionary is used, which addresses both short term and long term adaptation at pixel level and frame level. A good set of videos to test any background separation algorithm are provided in  [33,39]. Ref. [33] is called the BMC (Background Models Challenge) and presents a benchmark dataset and evaluation process built from both synthetic and real videos. It is focused on outdoor sequences and with varying weather conditions. Additionally, an evaluation criterion is also provided with an associated software. CDnet dataset in  [33,39] focuses on evaluation of change and motion detection approaches. It contains challenging scenarios to test background subtraction and motion detection.

Another closely related field is background generation. Having a background image free of any foreground objects is important in most applications. Generating a stationary background image is challenging when the background is not fully visible. And this method needs a finite data volume but it is specific for each scene. A detailed survey focused on model initialization for background is carried out in [40]. It provides a basis for easy comparison of existing and new methods using a common set of ground truth sequences. Another survey and benchmarking of scene background initialization methods has been extensively carried out in [41] using several evaluation metrics on a large video dataset called SBMNet [42]. Background generation using motion detection is proposed in [43]. It uses a temporal median filter and a patch selection mechanism based on motion detection performed by a background subtraction algorithm. An enhanced version of this method is made in [44] by using optical flow for motion detection. It leverages memoryless dense optical flow algorithms to compute velocity vector between two frames for each pixel. There are methods that make use of neural networks. One such method is described in [45], which performs background estimation using weightless neural networks. This algorithm could be used as a preliminary step in video foreground detection, as detecting foreground or moving objects is not in the scope of this method.

1.2. Contributions

In this paper, we propose a compressive online robust PCA with optical flow (CORPCA-OF) method (Our work has been reported in [46]), which is based on our previous work in [7]. We leverage information from previously separated foreground frames via optical flow [6]. The novelty of CORPCA-OF over CORPCA [7] is that we make use of optical flow to estimate and compensate motions between the foreground frames, in order to generate new prior foreground frames. These new prior frames have high correlation with the current frame and thus improve the separation. We also exploit the slowly changing characteristics of backgrounds known as low-rank components via an incremental $\mathrm{SVD}$[47] method. The compressive separation problem in (2) is solved in an online manner by minimizing not only an n-${\ell }_1$-norm cost function [48] for the sparse foreground but also the rank of a matrix for the low-rank backgrounds. Thereafter, the new separated foreground and background frames are used to update the prior knowledge for the next processing instance. This method makes video separation better with higher accuracy and more efficient implementation. The algorithm has been implemented in C++ using OpenCV and also tested on Matlab. Various functions/methods that are needed in performing video separation or general compressed sensing have been developed in C++ and can be used as stand-alone libraries.

The paper is organized as follows. We provide a brief introduction to the CORPCA algorithm [7], on which our proposed CORPCA-OF is built upon. Based on this, we formulate the problem statement in Section 2.1, followed by the proposed CORPCA-OF algorithm in Section 2.2.2. The visual and quantitative results obtained by testing our method on real video sequences are presented and discussed in Section 3.

2. Compressive Online Robust PCA Using Multiple Prior Information and Optical Flow

In this section, we firstly review the CORPCA [7] algorithm for online compressive video separation and state our problem. Thereafter, we propose our CORPCA-OF method, which is summarized in the CORPCA-OF algorithm.

2.1. Compressive Online Robust PCA (CORPCA) for Video Separation

The CORPCA algorithm [7] for video separation is based on Reconstruction Algorithm with Multiple Side Information using Adaptive weights (RAMSIA) [48], that solves an n-${\ell }_1$ minimization problem with adaptive weights to recover a sparse signal $\mathit{x}$ from low-dimensional random measurements $\mathit{y}=\mathbf{\Phi }\mathit{x}$ with the aid of multiple prior information or side information ${\mathit{z}}_j$, $j\in \{0,1,\cdots ,J\}$, with ${\mathit{z}}_0=\mathbf{0}$. The objective function of RAMSIA [48] is given by

$$\underset{\mathit{x}}{min}\left\{H\left(\mathit{x}\right)=\frac{1}{2}{\parallel \mathbf{\Phi }\mathit{x}-\mathit{y}\parallel }_2^2+\lambda \sum _{j=0}^J{\beta }_j{\parallel {\mathbf{W}}_j(\mathit{x}-{\mathit{z}}_j)\parallel }_1\right\},$$

(3)

where $\lambda >0$ is a regularization parameter and ${\beta }_j>0$ are weights across the prior information, and ${\mathbf{W}}_j$ is a diagonal matrix with weights for each element in the prior information signal ${\mathit{z}}_j$; namely, ${\mathbf{W}}_j=\mathrm{diag}(w_{j1},w_{j2},\dots ,w_{jn})$ with $w_{ji}>0$ being the weight for the i-th element in the ${\mathit{z}}_j$ vector.

The CORPCA algorithm processes one data vector per time instance by leveraging prior information for both its sparse and low-rank components. At time instance t, we observe ${\mathit{y}}_t=\mathbf{\Phi }({\mathit{x}}_t+{\mathit{v}}_t)$ with ${\mathit{y}}_t\in {\mathbb{R}}^m$. Let ${\mathit{Z}}_{t-1}:=\{{\mathit{z}}_1,\dots ,{\mathit{z}}_J\}$, a set of ${\mathit{z}}_j\in {\mathbb{R}}^n$, and ${\mathit{B}}_{t-1}\in {\mathbb{R}}^{n\times d}$ denote prior information for ${\mathit{x}}_t$ and ${\mathit{v}}_t$, respectively. The prior information ${\mathit{Z}}_{t-1}$ and ${\mathit{B}}_{t-1}$ are formed by using the already reconstructed set of vectors $\{{\widehat{\mathit{x}}}_1,\dots ,{\widehat{\mathit{x}}}_{t-1}\}$ and $\{{\widehat{\mathit{v}}}_1,\dots ,{\widehat{\mathit{v}}}_{t-1}\}$.

The objective function of CORPCA is to solve Problem (2) and can be formulated by

$$\begin{array}{cc}\hfill \underset{{\mathit{x}}_t,{\mathit{v}}_t}{min}& \{H({\mathit{x}}_t,{\mathit{v}}_t|{\mathit{y}}_t,{\mathit{Z}}_{t-1},{\mathit{B}}_{t-1})=\frac{1}{2}{\parallel \mathbf{\Phi }({\mathit{x}}_t+{\mathit{v}}_t)-{\mathit{y}}_t\parallel }_2^2\hfill \\ \hfill & +\lambda \mu \sum _{j=0}^J{\beta }_j{\parallel {\mathbf{W}}_j({\mathit{x}}_t-{\mathit{z}}_j)\parallel }_1+\mu \Vert \left[{\mathit{B}}_{t-1}{\mathit{v}}_t\right]{\Vert }_{*}\},\hfill \end{array}$$

(4)

where $\mu >0$ is a relaxation parameter. It can be seen that if ${\mathit{v}}_t$ is static (not changing), Problem (4) reduces to Problem (3). Furthermore, when ${\mathit{x}}_t$ and ${\mathit{v}}_t$ are batch variables and we do not take the prior information, ${\mathit{Z}}_{t-1}$ and ${\mathit{B}}_{t-1}$, and the projection matrix $\mathbf{\Phi }$ into account, Problem (4) reduces to Problem (1).

The CORPCA algorithm (The source code, the test sequences, and the corresponding outcomes of CORPCA are available at [49]) solves Problem (4) given that ${\mathit{Z}}_{t-1}$ and ${\mathit{B}}_{t-1}$ are known (they are obtained from the time instance or recursion). Thereafter, we update ${\mathit{Z}}_t$ and ${\mathit{B}}_t$, which are used in the subsequent time instance.

Let us denote $f({\mathit{v}}_t,{\mathit{x}}_t)=(1/2){\parallel \mathbf{\Phi }({\mathit{x}}_t+{\mathit{v}}_t)-{\mathit{y}}_t\parallel }_2^2$, $g\left({\mathit{x}}_t\right)=\lambda {\sum }_{j=0}^J{\beta }_j{\parallel {\mathbf{W}}_j({\mathit{x}}_t-{\mathit{z}}_j)\parallel }_1$, and $h\left({\mathit{v}}_t\right)={\parallel \left[{\mathit{B}}_{t-1}{\mathit{v}}_t\right]\parallel }_{*}$, where $f(.)$ is a function of both background and foreground, $g(.)$ is a function of the foreground (sparse) component and $h(.)$ is a function of the background (low rank) component. The components ${\mathit{x}}_t^{(k+1)}$ and ${\mathit{v}}_t^{(k+1)}$ are solved iteratively at iteration $k+1$ via the soft thresholding operator [50] for ${\mathit{x}}_t$ and the single value thresholding operator [51] for ${\mathit{v}}_t$:

$${\mathit{v}}_t^{(k+1)}=\underset{{\mathit{v}}_t}{arg\; min}\left\{\mu h\left({\mathit{v}}_t\right)+\Vert {\mathit{v}}_t-\left({\mathit{v}}_t^{\left(k\right)}-\frac{1}{2}{\nabla }_{{\mathit{v}}_t}f({\mathit{v}}_t^{\left(k\right)},{\mathit{x}}_t^{\left(k\right)})\right){\Vert }_2^2\right\},$$

(5)

$${\mathit{x}}_t^{(k+1)}=\underset{{\mathit{x}}_t}{arg\; min}\left\{\mu g\left({\mathit{x}}_t\right)+\Vert {\mathit{x}}_t-\left({\mathit{x}}_t^{\left(k\right)}-\frac{1}{2}{\nabla }_{{\mathit{x}}_t}f({\mathit{v}}_t^{\left(k\right)},{\mathit{x}}_t^{\left(k\right)})\right){\Vert }_2^2\right\}.$$

(6)

2.2. Video Foreground and Background Separation Using CORPCA-OF

Problem statement: Using the prior information in CORPCA [7] has provided significant improvement of the current frame separation. However, there can be displacements between the consecutive frames that can deteriorate the separation performance. Figure 1 illustrates an example of three previous foreground frames, ${\mathit{x}}_{t-3},{\mathit{x}}_{t-2}$, and ${\mathit{x}}_{t-1}$. These frames can be used directly as prior information to recover foreground ${\mathit{x}}_t$ and background ${\mathit{v}}_t$ due to temporal correlations between ${\mathit{x}}_t$ and ${\mathit{x}}_{t-3},{\mathit{x}}_{t-2},{\mathit{x}}_{t-1}$, as seen in CORPCA. In the last row of the prior foreground frames in Figure 1, it can be seen that motions exist between frames. By estimating motion using optical flow [6], we can obtain motions between the previous foreground frames as in Figure 1, which are visualized using color codes based on the magnitude and direction of motion vectors [6]. These motions can be compensated to generate better quality prior frames (compare compensated frames ${\mathit{x}}_{t-3}^{\prime },{\mathit{x}}_{t-2}^{\prime }$ with ${\mathit{x}}_{t-3},{\mathit{x}}_{t-2},{\mathit{x}}_{t-1}$), and it is better correlated to ${\mathit{x}}_t$. In this work, we discuss a new algorithm—CORPCA with Optical Flow (CORPCA-OF), whose work flow is shown in Figure 1. Optical flow [6] is used to improve prior foreground frames.

2.2.1. Compressive Separation Model with CORPCA-OF

A compressive separation model using the CORPCA-OF method is shown in Figure 2. At a time instance t, the inputs consist of compressive measurements ${\mathit{y}}_t=\mathbf{\Phi }({\mathit{x}}_t+{\mathit{v}}_t)$ and prior information from time instance $t-1$, ${\mathit{Z}}_{t-1}$ (foreground) and ${\mathit{B}}_{t-1}$ (background). The model outputs foreground and background information ${\mathit{x}}_t$ and ${\mathit{v}}_t$ by solving the CORPCA minimization problem in (4). Finally, the outputs ${\mathit{x}}_t$ and ${\mathit{v}}_t$ are used to generate better prior foreground information via a prior generation using optical flow and update ${\mathit{Z}}_{t-1}$ and ${\mathit{B}}_{t-1}$ for the next instance via a prior update. The novel block of CORPCA-OF compared with CORPCA [7] is the Prior Generation using Optical Flow, where prior foreground information is improved by exploiting the correlation between frames using large displacement optical flow [6]. The method is further described in Algorithm 1.

2.2.2. Prior Generation using Optical Flow

The main idea of CORPCA-OF is to improve the foreground prior frames using the correlation between frames, which is done by estimating motion between frames via optical flow. In Algorithm 1, the prior frames are initialized with ${\mathit{x}}_{t-1}$, ${\mathit{x}}_{t-2}$ and ${\mathit{x}}_{t-3}$. Optical flow is used to compute the motions between frames ${\mathit{x}}_{t-1}$ and ${\mathit{x}}_{t-2}$ (also ${\mathit{x}}_{t-1}$ and ${\mathit{x}}_{t-3}$) to obtain flow vectors for these two frames. This can be seen in Figure 1 from the color coded representation of optical flow fields [6]. The function $f_{ME}(·)$ in Lines 2 and 3 (see Algorithm 1) computes the motions between prior foreground frames. It involves computing the optical flow vectors consisting of horizontal ($\mathsf{x}$) and vertical ($\mathsf{y}$) components, and is denoted by ${\mathbf{v}}_{1\mathsf{x}},{\mathbf{v}}_{2\mathsf{x}}$ and ${\mathbf{v}}_{1\mathsf{y}},{\mathbf{v}}_{2\mathsf{y}}\in {\mathbb{R}}^n$, respectively. The estimated motions in the form of optical flow vectors, $({\mathbf{v}}_{1\mathsf{x}},{\mathbf{v}}_{1\mathsf{y}})$ and $({\mathbf{v}}_{2\mathsf{x}},{\mathbf{v}}_{2\mathsf{y}})$, are then used to predict the following frames by compensating for the forward motions on ${\mathit{x}}_{t-1}$. The prior frames, ${\mathit{x}}_{t-2}^{\prime }$ and ${\mathit{x}}_{t-3}^{\prime }$ are generated by performing motion compensation. This is indicated by the function $f_{MC}(·)$ as shown by Lines 5 and 6 in Algorithm 1.

Algorithm 1: The proposed CORPCA-OF algorithm.

Considering a point or a pixel i in the given frame, the horizontal and vertical components ${\mathsf{v}}_{1\mathsf{x}i}$ and ${\mathsf{v}}_{1\mathsf{y}i}$ of corresponding horizontal and vertical flow vectors ${\mathit{v}}_{1\mathsf{x}}$ and ${\mathit{v}}_{1\mathsf{y}}$ are obtained, as mentioned in [52], by solving:

$$I_{1\mathsf{x}}·{\mathsf{v}}_{1\mathsf{x}i}+I_{1\mathsf{y}}·{\mathsf{v}}_{1\mathsf{y}i}+I_{1t}=0,$$

(7)

where $I_{1\mathsf{x}}=\partial I_1/\partial \mathsf{x}$ and $I_{1\mathsf{y}}=\partial I_1/\partial \mathsf{y}$ are the intensity changes in the horizontal ($\mathsf{x}$) and vertical ($\mathsf{y}$) directions, respectively, constituting the spatial gradients of the intensity level $I_1$; $I_{1t}=\partial I_1/\partial t$ is the time gradient, which is a measure of temporal change in the intensity level at point i. There are various methods [3,4,5,6] to determine ${\mathsf{v}}_{1\mathsf{x}i}$ and ${\mathsf{v}}_{1\mathsf{y}i}$. Our solution is based on large displacement optical flow [6], that is a combination of global and local approaches to estimate all kinds of motion. It involves optimization and minimization of error by using descriptor matching and continuation method, which utilizes feature matching along with conventional optical flow estimation to obtain the flow field. We combine the optical flow components of each point i in the image into two vectors $({\mathbf{v}}_{1\mathsf{x}},{\mathbf{v}}_{1\mathsf{y}})$, i.e., the horizontal and the vertical components of the optical flow vector. Similarly, we obtain $({\mathbf{v}}_{2\mathsf{x}},{\mathbf{v}}_{2\mathsf{y}})$.

The estimated motions in the form of optical flow vectors are used along with the frame ${\mathit{x}}_{t-1}$ to produce new prior frames that form the updated prior information. Linear interpolation is used to generate new frames via column interpolation and row interpolation. This is represented as $f_{MC}(·)$ in Lines 5 and 6 in the Algorithm 1. The obtained frame is the result of using the flow fields $({\mathbf{v}}_{1\mathsf{x}},{\mathbf{v}}_{1\mathsf{y}})$ and $(\frac{1}{2}{\mathbf{v}}_{2\mathsf{x}},\frac{1}{2}{\mathbf{v}}_{2\mathsf{y}})$ to predict motions in the next frame and compensate them on the frame ${\mathit{x}}_{t-1}$ to obtain ${\mathit{x}}_{t-2}^{\prime }$ and ${\mathit{x}}_{t-3}^{\prime }$ respectively. It should be noted that ${\mathit{x}}_{t-3}^{\prime }$ is obtained by compensating for half the motions, i.e., $(\frac{1}{2}{\mathbf{v}}_{2\mathsf{x}},\frac{1}{2}{\mathbf{v}}_{2\mathsf{y}})$, between ${\mathit{x}}_{t-1}$ and ${\mathit{x}}_{t-3}$. These improved frames ${\mathit{x}}_{t-2}^{\prime }$, ${\mathit{x}}_{t-3}^{\prime }$ are more correlated to the current frame ${\mathit{x}}_t$ than ${\mathit{x}}_{t-2}$, ${\mathit{x}}_{t-3}$, i.e., without motion estimation and compensation. We also keep the most recent frame ${\mathit{x}}_{t-1}^{\prime }={\mathit{x}}_{t-1}$ (in Line 4) as one of the prior frames.

Thereafter, ${\mathit{v}}_t^{(k+1)}$ and ${\mathit{x}}_t^{(k+1)}$ are iteratively computed as in Lines 14 and 15 in Algorithm 1. It can be noted that the proximal operator ${\mathbf{\Gamma }}_{\tau g_1}(·)$ in Line 13 of Algorithm 1 is defined [7] as:

$${\mathbf{\Gamma }}_{\tau g_1}\left(\mathit{X}\right)=\underset{\mathit{V}}{arg\; min}\left\{\tau g_1\left(\mathit{V}\right)+\frac{1}{2}\left|\right|\mathit{V}-{\mathit{X}\left|\right|}_2^2\right\},$$

(8)

where $g_1(·)={\parallel ·\parallel }_1$ (${\ell }_1$-norm). The weights ${\mathbf{W}}_j$ and ${\beta }_j$ are updated per iteration of the algorithm (see Lines 16 and 17). As suggested in [2], the convergence of Algorithm 1 in Line 8 is determined by evaluating the criterion $\parallel \partial H({\mathit{x}}_t,{\mathit{v}}_t){|}_{{\mathit{x}}_t^{(k+1)},{\mathit{v}}_t^{(k+1)}}{\parallel }_2^2<2\times {10}^{-7}{\parallel ({\mathit{x}}_t^{(k+1)},{\mathit{v}}_t^{(k+1)})\parallel }_2^2.$ In the next step, we perform an update of the priors ${\mathit{Z}}_t$ and ${\mathit{B}}_t$.

2.2.3. Prior Update

The update of ${\mathit{Z}}_t$ and ${\mathit{B}}_t$ [7] is carried out after each time instance (see Lines 21 and 22, Algorithm 1). Due to the correlation between subsequent frames, we update the prior information ${\mathit{Z}}_t$ by using the J latest recovered sparse components, which is given by, ${\mathit{Z}}_t:={\{{\mathit{z}}_j={\mathit{x}}_{t-J+j}\}}_{j=1}^J$. For ${\mathit{B}}_t\in {\mathbb{R}}^{n\times d}$, we consider an adaptive update, which operates on a fixed or constant number d of the columns of ${\mathit{B}}_t$. To this end, the incremental singular value decomposition [47] method ($\mathrm{incSVD}(·)$ in Line 12, Algorithm 1) is used. It is worth noting that the update ${\mathit{B}}_t={\mathit{U}}_t{\mathbf{\Gamma }}_{\frac{{\mu }_k}{2}{g}_1}\left({\mathbf{\Sigma }}_t\right){\mathit{V}}_t^{\mathrm{T}}$, causes the dimension of ${\mathit{B}}_t$ to increase as ${\mathit{B}}_t\in {\mathbb{R}}^{n\times (d+1)}$ after each instance. However, in order to maintain a reasonable number of d, we take ${\mathit{B}}_t={\mathit{U}}_t(:,1:d){\mathbf{\Gamma }}_{\frac{{\mu }_k}{2}{g}_1}\left({\mathbf{\Sigma }}_t\right)(1:d,1:d){\mathit{V}}_t{(:,1:d)}^{\mathrm{T}}$. The computational cost of $\mathrm{incSVD}(·)$ is lower than conventional SVD [8,47] since we only compute the full $\mathrm{SVD}$ of the middle matrix with size $(d+1)\times (d+1)$, where $d\ll n$, instead of $n\times (d+1)$. The computation of $\mathrm{incSVD}(·)$ is presented as follows: The goal is to compute $\mathrm{incSVD}\left[{\mathit{B}}_{t-1}{\mathit{v}}_t\right]$, i.e., $\left[{\mathit{B}}_{t-1}{\mathit{v}}_t\right]={\mathit{U}}_t{\mathbf{\Sigma }}_t{\mathit{V}}_t^{\mathrm{T}}$. By taking the SVD of ${\mathit{B}}_{t-1}\in {\mathbb{R}}^{n\times d}$ to obtain ${\mathit{B}}_{t-1}={\mathit{U}}_{t-1}{\mathbf{\Sigma }}_{t-1}{\mathit{V}}_{t-1}^{\mathrm{T}}$. Therefore, we can derive $({\mathit{U}}_t,{\mathbf{\Sigma }}_t,{\mathit{V}}_t)$ via $({\mathit{U}}_{t-1},{\mathbf{\Sigma }}_{t-1},{\mathit{V}}_{t-1})$ and ${\mathit{v}}_t$. We write the matrix $\left[{\mathit{B}}_{t-1}{\mathit{v}}_t\right]$ as

$$\left[{\mathit{B}}_{t-1}{\mathit{v}}_t\right]=\left[{\mathit{U}}_{t-1}{\displaystyle \frac{{\delta }_t}{\parallel {\delta }_t{\parallel }_2}}\right]·\left[\begin{array}{cc}{\mathbf{\Sigma }}_{t-1}& {\mathit{e}}_t\\ {\mathbf{0}}^{\mathrm{T}}& \parallel {\delta }_t{\parallel }_2\end{array}\right]·\left[\begin{array}{cc}{\mathit{V}}_{t-1}^{\mathrm{T}}& \mathbf{0}\\ {\mathbf{0}}^{\mathrm{T}}& 1\end{array}\right],$$

(9)

where ${\mathit{e}}_t={\mathit{U}}_{t-1}^{\mathrm{T}}{\mathit{v}}_t$ and ${\delta }_t={\mathit{v}}_t-{\mathit{U}}_{t-1}{\mathit{e}}_t$. By computing the $\mathrm{SVD}$ of the central term of (9), we obtain $\left[\begin{array}{cc}{\mathbf{\Sigma }}_{t-1}& {\mathit{e}}_t\\ {\mathbf{0}}^{\mathrm{T}}& \parallel {\delta }_t{\parallel }_2\end{array}\right]=\tilde{\mathit{U}}\tilde{\mathbf{\Sigma }}{\tilde{\mathit{V}}}^{\mathrm{T}}$. Eventually, we obtain ${\mathit{U}}_t=\left[{\mathit{U}}_{t-1}{\displaystyle \frac{{\delta }_t}{\parallel {\delta }_t{\parallel }_2}}\right]·\tilde{\mathit{U}}$, ${\mathbf{\Sigma }}_t=\tilde{\mathbf{\Sigma }}$, and also, ${\mathit{V}}_t=\left[\begin{array}{cc}{\mathit{V}}_{t-1}^{\mathrm{T}}& \mathbf{0}\\ {\mathbf{0}}^{\mathrm{T}}& 1\end{array}\right]·\tilde{\mathit{V}}$.

3. Experimental Results

In this section we present and discuss the experimental results obtained by applying our method CORPCA-OF on real video data. We also evaluate the performance of our algorithm and compare the results with other existing methods.

Experimental setup: The experiments were carried out on two computers. The Matlab implementation was carried out and tested on a desktop PC (Linux) with Intel i5 3.5 GHz CPU (4 cores) and 12GB RAM. The C++ implementation was carried out and tested on a computer (Windows) with i7-4510U 2.0 GHz CPU (2 cores) and 8 GB memory. By doing that, the cross platform functionality can also be verified with these two machines.

For the experimental evaluation with the various existing methods, mainly two sequences [53], Bootstrap $(80\times 60)$ and Curtain $(80\times 64)$ were used. The Bootstrap sequence consists of 3055 frames and has a static background and a complex foreground. The Curtain sequence contains 2964 frames with a dynamic background and simple foreground motion. For separating each of these sequences, 100 frames are randomly selected and used for initialization of prior information. The prior information is later updated by selecting three most recent frames as seen in Section 2.2.3.

We evaluate the performance of the proposed CORPCA-OF in Algorithm 1 and compare it against the existing methods, Robust Principal Component Analysis (RPCA) [1], Grassmannian Robust Adaptive Subspace Tracking Algorithm(GRASTA) [11], and Recursive Projected Compressive Sensing(ReProCS) [17,22]. RPCA [1] is a batch-based method assuming full access to the data, while GRASTA [11] and ReProCS [17] are online methods that can recover either the (low-rank) background component (GRASTA) or the (sparse) foreground component (ReProCS) from compressive measurements.

3.1. Prior Information Evaluation

We evaluate the prior information of CORPCA-OF compared with that of CORPCA [7] using the previously separated foreground frames directly. For CORPCA-OF, we generate the prior information by estimating and compensating motions among the previous foreground frames. Figure 3 shows a few examples of the prior information generated for the sequences Bootstrap and Curtain. In Figure 3a, it can be observed that frames #2210’, #2211’ and #2212’ (of CORPCA-OF) are better than corresponding #2210, #2211 and #2212 (of CORPCA) for the current frame #2213, similarly in Figure 3b–d, . Especially in Figure 3c, the generated frames #448’ and #449’ have significantly improved due to dense motion compensations. In Figure 3d, it is clear that the movements of the person is well compensated in #2771’ and #2772’ by CORPCA-OF compared to #2771 and #2772 respectively, of CORPCA, leading to better correlations with the foreground of current frame #2774. Replace one of the curtain and bootstrap sequences by an elevator and a fountain sequence.

3.2. Compressive Video Foreground and Background Separation

We assess our CORPCA-OF method in the application of compressive video separation and compare it against the existing methods, CORPCA [7], RPCA [1], GRASTA [11], and ReProCS [17]. We run all methods on the test video sequences. In this experiment, we use $d=100$ frames as training vectors for the proposed CORPCA-OF, CORPCA [7] as well as for GRASTA [11] and ReProCS [17]. Three latest previous foregrounds are used as the foreground prior for CORPCA. Meanwhile, CORPCA-OF refines them using optical flow [6].

3.2.1. Visual Evaluation

We first consider background and foreground separation with full access to the video data; the visual results of the various methods are illustrated in Figure 4. It is evident that, for both the video sequences, CORPCA-OF delivers superior visual results than the other methods, which suffer from less-details in the foreground and noisy background images. We can also observe improvements over CORPCA.

Additionally, we also compare the visual results of CORPCA-OF, CORPCA and ReProCS for the frames Bootstrap #2213 (in Figure 5) and for Curtain #2866 (in Figure 6) with compressed rates. They present the results under various rates on the number of measurements m over the dimension n of the data (the size of the vectorized frame) with rates: $m/n=\{0.8;0.6;0.4;0.2\}$. Comparing CORPCA-OF with CORPCA, we can observe in Figure 5 and Figure 6 that CORPCA-OF gives the foregrounds that are less noisy and the background frames of higher visual quality. On comparison with ReProCS, our algorithm outperforms it significantly. At low rates, for instance with $m/n=0.6$ (in Figure 5a) or $m/n=0.4$ (in Figure 6a), the extracted foreground frames of CORPCA-OF are better than those of CORPCA and ReProCS. Even at a high rate of $m/n=0.8$ the sparse components or the foreground frames using ReProCS are noisy and of poor visual quality. The Bootstrap sequence requires more measurements than Curtain due to the more complex foreground information. It is evident from Figure 5 and Figure 6 that the visual results obtained with CORPCA-OF are of superior quality compared to ReProCS and have significant improvements over CORPCA.

3.2.2. Quantitative Results

We evaluate quantitatively the separation performance via the receiver operating curve (ROC) metric [54]. The metrics True positives and False positives are defined in [54] as:

$$Truepositives=\frac{|\left\{\mathrm{Foreground}\right\}\cap \left\{\mathrm{Groundtruth}\mathrm{Foreground}\right\}|}{|\left\{\mathrm{Groundtruth}\mathrm{Foreground}\right\}|}$$

(10)

$$Falsepositives=\frac{|\left\{\mathrm{Foreground}\right\}\cap \left\{\mathrm{Groundtruth}\mathrm{Background}\right\}|}{|\left\{\mathrm{Groundtruth}\mathrm{Background}\right\}|}$$

(11)

For plotting the ROC curve, a set of foreground frames corresponding to the ground truth frames are selected. Then we threshold the foreground frames at various levels and compute true positives and false positives by comparing with the ground truth frames for each threshold level. We then plot true positives against false positives.

Figure 7 illustrates the ROC results when assuming full data access, i.e., $m/n=1$, of CORPCA-OF, CORPCA, RPCA, GRASTA, and ReProCS. The results show that CORPCA-OF delivers higher performance than the other methods.

Furthermore, we compare the foreground recovery performance of CORPCA-OF against CORPCA and ReProCS for different compressive measurement rates: $m/n=\{0.8;0.6;0.4;0.2\}$. The ROC results in Figure 8 and Figure 9 show that CORPCA-OF can achieve higher performance in comparison to ReProCS and CORPCA. In particular, with a small number of measurements, CORPCA-OF produces better curves than those of CORPCA. This is evident for Bootstrap at $m/n=\{0.2;0.4;0.6\}$ (see Figure 8a). For the Curtain sequence, which has a dynamic background and less complex foreground, the measurements at $m/n=\{0.2;0.4\}$ (see Figure 9a) are clearly better. The ROC results for ReProCS are quickly degraded even with a high compressive measurement rate $m/n=0.8$ (see Figure 9c).

3.3. Additional Results

3.3.1. Escalator and Fountain sequences

The CORPCA-OF method was also compared with CORPCA against Escalator and Fountain sequences for compressive measurements. From Figure 10a,b, it is clear that CORPCA-OF performs slightly better than CORPCA. In Figure 11a,b we can see that for the Fountain sequence, which is similar to the Curtain sequence in terms of complexity of foreground motions, the results are better for CORPCA-OF compared to CORPCA at rate $m/n=\left\{0.2\right\}$ and almost the same for higher rates.

3.3.2. Visual Comparison of CORPCA-OF and CORPCA for Full Resolution

The visual comparison of CORPCA and CORPCA-OF for frame #2213 in full resolution ($160\times 120$) of the Bootstrap sequence can be seen in Figure 12. In Figure 13, we compare the full resolution ($160\times 128$) frame #2866 of the Curtain sequence. It can be seen that the background and foreground frames of CORPCA-OF for both Bootstrap and Curtain are much smoother and the contents have better structure compared to that of CORPCA. The improvements in foreground can be observed significantly at rates $m/n=\{0.6;0.4;0.2\}$ for the Bootstrap sequence for CORPCA-OF in Figure 12a over CORPCA in Figure 12b. But in case of Curtain sequence, at low rates $m/n=\{0.4;0.2\}$, there is significant improvement of foreground in Figure 13a for CORPCA-OF over CORPCA in Figure 13b.

3.3.3. Separation Results with Various Datasets

CDnet dataset: We tested our algorithm with some sequences from the CDnet dataset [39]. tramCrossroad1 is a sequence captured at low frame rate. The foreground and backgrouns separation result for frame #655 is shown in Figure 14a. A thermal imaging sequence corridor was seperated using CORPCA-OF and the result for frame #686 is shown in Figure 14b. The separation results for a sequence with camera jitter, badminton is shown in Figure 14c. canoe is a challenging sequence with background motion or in other words, with dynamic background. The results for this sequence is shown in Figure 14d for frame #145. We can observe some artifacts of the canoe from foreground in the background image. This is because of the complex motion of water. The sequences badminton and canoe are also part of the SBMNet dataset.

SBMNet dataset: The results for sequences from SBMNet dataset [42] are shown in Figure 15. MPEG4_40 is an animated sequence of cars at a traffic signal. It can be seen from Figure 15a that our algorithm works for synthetic images as well. A cluttered sequence IndianTraffic3 was tested and the result for frame #622 can be seen from Figure 15b. We also tested our algorithm on an underwater sequence Hybrid and the result is shown in Figure 15c.

These results demonstrate the versatility of our algorithm.

HighwayI: This sequence is from the ATON dataset [55]. It is a sequence with fast motion of cars along a highway, with strong shadows and small camera jitter. The clear foreground and background separation for frames #285 to #293 can be seen from Figure 16.

Laboratory: This sequence is from the ATON dataset [55]. It contains an indoor sequence with two persons appearing for certain intervals. Original frames #142 to #158 are shown in Figure 17 along with the separated background and foreground frames.

4. Conclusions

We have considered online video foreground–background separation using RPCA approach from compressive measurements. A compressive online robust PCA algorithm with optical flow (CORPCA-OF) that can process one frame per time instance from compressive measurements was proposed. CORPCA-OF efficiently incorporates multiple prior frames based on the n-${\ell }_1$ minimization problem. Furthermore, the proposed method can exploit motion estimation and compensation using optical flow to refine the prior information and obtain better quality separation. The proposed CORPCA-OF was tested on compressive online video separation application using numerous video sequences. The visual and quantitative results have shown the improvements on the prior generation and the superior performance offered by CORPCA-OF compared to the existing methods including the CORPCA baseline.

It is worth mentioning that CORPCA-OF can work in a general manner, with any number of prior frames. In this work, for the video separation, three prior frames are used as a compromise between performance and computational load. The C++ implementation (see Appendix A) of CORPCA-OF (The code for the CORPCA-OF algorithm is available at [56]) has been carried out, where the methods can be used as libraries for video separation. For separating a frame of resolution $80\times 60$ and full data (m/n = 1), on an average, the Matlab implementation takes about 8 s per frame, out of which about 0.4 s is for the computation of optical flow. The same algorithm on C++ takes about 28 s per frame, with about 0.15 s for the optical flow computation. Building on this, the next step would be to make the algorithm more robust, adaptive, dynamic, and real-time by using Graphics Processing Units (GPUs) so that the algorithm can handle larger images.