Detection of Abnormal Events via Optical Flow Feature Analysis

Abstract

In this paper, a novel algorithm is proposed to detect abnormal events in video streams. The algorithm is based on the histogram of the optical flow orientation descriptor and the classification method. The details of the histogram of the optical flow orientation descriptor are illustrated for describing movement information of the global video frame or foreground frame. By combining one-class support vector machine and kernel principal component analysis methods, the abnormal events in the current frame can be detected after a learning period characterizing normal behaviors. The difference abnormal detection results are analyzed and explained. The proposed detection method is tested on benchmark datasets, then the experimental results show the effectiveness of the algorithm.

1. Introduction

With the development of human society, the security challenges in public scenes are gradually increased. In the last several decades, the cost for camera and network communication has been significantly reduced. Furthermore, video camera sensors are used wildly in many areas of human life. However, the traditional way of visual surveillance is a labor-intensive non-stop human attention work, and the efficiency is low. Thus, tackling visual surveillance problems automatically by adopting video processing technique plays a paramount role in the computer vision research area. The scientific challenges in this area include developing strategies to ensure public safety and detecting the abnormal behavior of an individual or a group.

The methods modeling behavior by adopting the Bayesian network were introduced in [1–4]. In [5], delta-dual hierarchical Dirichlet processes (dDHDP) were used to detect abnormal activity patterns in the field of visual features. By analyzing the statistical property, abnormal events were detected. Successful results were obtained on several scenes, but the prediction was based on the complicated probability model.

Some researchers took notice of spatio-temporal features. In [6], the movement was represented by a co-occurrence matrix and modeled by a Markov random field model. Abnormal activities, which were the significant changes in the scene, were detected. The work was similar to the foreground subtraction method in a non-stable background scene.

Low-level motion features have also gained attention for detecting abnormal events. In [7], an algorithm was proposed to detect the action of a single individual, such as hand-waving, boxing, etc. In [8], bionics technology was applied to model the superior colliculus (SC) to discover abnormalities in the panoramic image. These methods were based on partial information, such as contained in small observation windows of the image. In other words, they did not employ global information within the frame.

Based on the feature representation and the pattern classification, an abnormal detection method is proposed in this paper. The datasets used in our work are Performance Evaluation of Tracking and Surveillance (PETS2009) [9] and University of Minnesota (UMN) dataset [10], as shown in Figure 1. A normal scene means that the individuals are promenading in different directions. In the abnormal scenes of the PETS dataset, people are moving (walking or running) in the same direction, while the UMN abnormal scene means that the individuals are running. The proposed algorithm is composed of two parts. Firstly, the visual features are extracted without object tracking. Secondly, abnormal events are detected by classifying the extracted features. In fact, one-class support vector machine (SVM) and principal component analysis (PCA) are used in this paper. By learning the normal behaviors, the classifiers detect the abnormal ones. The rest of the paper is organized as follows. In Section 2, the optical flow-based feature is proposed. In Section 3, a one-class SVM classification method and a kernel PCA for novelty detection method are presented, and thus, the corresponding abnormal detection framework is described. In Section 4, the experimental results and the analysis are given. Finally, the paper is concluded with future works in Section 5.

2. Feature Selection for Abnormal Detection

Because optical flow can represent the movement information of actions, we choose the Horn-Schunck (HS) [11] method to compute it. The HS method formulates the optical flow as a global energy functional for the gray image sequence:

$$E=\int \int [{(I_{x}u+I_{y}v+I_t)}^2+\alpha ({\Vert \nabla u\Vert }^2+{\Vert \nabla v\Vert }^2)]dxdy$$

(1)

where I_x,I_y and I_t are the derivatives of the image intensity values along the horizontal direction x, vertical direction y and time t dimension, respectively, u,v are the horizontal and vertical components of the optical flow, α is a regularization constant.

In [12], the abnormal global frame detection was proposed, and the frame covariance matrix descriptor was constructed based on the optical flow. In this paper, we analyze the details of the histogram of the optical flow orientation (HOFO) with different parameters. The optical flow orientation features of an image are extracted at fixed resolution and then gathered into a high dimensional feature vector. A 2 × 2 rectangular cell HOFO descriptor of the original image or the foreground image is shown in Figure 2. By a trigonometric function, the orientation is computed from horizontal and vertical optical flow. The orientation is voted into n bins in 0°–360° (noted as signed angle) or 0°–180° (noted as unsigned angle). Nine bins are chosen in this paper. The optical flow magnitude of a pixel is considered as a weight coefficient in the voting process. A block contains h_b × w_b cells; it is set as 2 × 2 in this paper to present the spatial information of the HOFO. The HOFO dimension of one block is 36 (9×2×2). The HOFO feature describes the global movement information of one frame (or foreground frame) by gathering the histogram of the optical flow orientation in the sub-frame (block). Because the movement of an abnormal image usually has a bigger value of the optical flow strength and more directions, the element in the HOFO vector of an abnormal image is generally higher than a normal one. Four normalization schemes are chosen when HOFO is calculated:

$$L1-\mathit{\text{norm}}:\mathbf{\text{v}}\to \frac{\mathbf{\text{v}}}{{\Vert \mathbf{\text{v}}\Vert }_1+\mathit{\epsilon }}$$

(2)

$$L1-\mathit{sqrt}:\mathbf{\text{v}}\to \sqrt{\frac{\mathbf{\text{v}}}{{\Vert \mathbf{\text{v}}\Vert }_1+\mathit{\epsilon }}}$$

(3)

$$L2-\mathit{\text{norm}}:\mathbf{\text{v}}\to \sqrt{\frac{\mathbf{\text{v}}}{{\Vert \mathbf{\text{v}}\Vert }_2^2+\mathit{\epsilon }}}$$

(4)

$$L2-\mathit{\text{Hys}}:\mathbf{\text{v}}\to \sqrt{\frac{\mathbf{\text{v}}}{{\Vert \mathbf{\text{v}}\Vert }_2^2+\mathit{\epsilon }}}$$

(5)

$$\{\begin{array}{cc}if& {\Vert \mathbf{\text{v}}\Vert }_2^2>0.3,{\Vert \mathbf{\text{v}}\Vert }_2^2=0.3\\ if& {\Vert \mathbf{\text{v}}\Vert }_2^2>0.4,{\Vert \mathbf{\text{v}}\Vert }_2^2=0.4\end{array}$$

(6)

where v is the HOFO descriptor vector before being normalized and ε is a small constant to make the calculation reasonable.

3. Abnormal Detection Method Based on Optical Flow Analysis

The objective of the abnormal event detection problem is to find the samples that are different from the training ones. Thus, two classification methods, the one-class support vector machine (one-class SVM) and kernel principal component analysis (KPCA) for novelty detection, suit this application. In this section, we firstly introduce these two methods and then propose the abnormal detection algorithm in video sequences.

3.1. One-Class Support Vector Machine

Vapnik and Lerner initially proposed the support vector machine for classification or regression based on statistical learning theory [13]. Later, by adopting the kernel methods, the support vector machine was extended to deal with non-linear problems [14–16]. Thus, the non-linear one-class support vector machine is one development of the basic SVM theory to find out an appropriate region containing most of the data drawn from an unknown probability distribution. The problem of the non-linear one-class support vector machine can be presented as [17,18]:

$$\begin{array}{ll}\underset{\mathrm{\omega },\mathrm{\xi },\mathrm{\rho }}{min}\frac{1}{2}{\Vert \mathit{w}\Vert }^2+\frac{1}{\nu n}\text{∑}_{i=1}^n{\xi }_i-\mathrm{\rho }\hfill & \hfill \\ \text{subject to}\langle \mathit{w},\mathrm{\Phi }\left({\mathit{x}}_i\right)\rangle \ge \mathrm{\rho }-{\mathrm{\xi }}_i,\hfill & {\mathrm{\xi }}_i\ge 0\end{array}$$

(7)

where x_i ∈ χ, i ∈ [1 … n] are n training samples in the original data space χ. ξ_i is the slack variable for penalizing the outliers. The hyperparameter ν ∈ (0,1; is the weight for the controlled slack variable. Φ is a map from the non-empty set of the original input data χ to a feature space . For computing dot products in , the kernel function is defined as κ(x,x′) = 〈Φ(x) · Φ(x′)〉. The decision function is defined as:

$$f\left(\mathit{x}\right)=\mathbf{sgn}(\sum _{i=1}^n{\alpha }_i\kappa \left({\mathit{x}}_{\mathit{i}},\mathit{x}\right)-\mathrm{\rho })$$

(8)

where x is a vector in the input data space χ and κ is the kernel function. The Gaussian kernel is used to deal with the non-linear problem in this paper.

3.2. Kernel Principal Component Analysis

Kernel principal component analysis [19,20] extends the standard PC A to non-linear data distributions. Before performing PCA, map the n datum points x_i ∈ ℝ^d to a higher-dimensional feature space where standard PCA is performed:

$${\mathit{x}}_i\to \mathrm{\Phi }\left({\mathit{x}}_i\right)$$

(9)

In kernel PCA, an eigenvector V of the covariance matrix in is a linear combination of Φ(x_i):

$$V=\sum _{i=1}^n{\alpha }_i\tilde{\mathrm{\Phi }}\left({\mathit{x}}_i\right)$$

(10)

$$\tilde{\mathrm{\Phi }}\left({\mathit{x}}_i\right)=\mathrm{\Phi }\left({\mathit{x}}_i\right)-\frac{1}{n}\sum _{r=1}^n\mathrm{\Phi }\left({\mathit{x}}_r\right)$$

(11)

where α_i is the component of a vector α. This vector is an eigenvector of the matrix k̃ (x_i,x_j) = 〈 (Φ̃(x_i) · Φ̃(x_j)〉.

For novelty detection [21], the reconstruction error p(Φ̃); can be defined as:

$$p\left(\mathit{x}\right)=p_S\left(\mathit{x}\right)-\sum _{l=1}^q{f}_l{\left(\mathit{x}\right)}^2$$

(12)

subject to:

$$p_S\left(\mathit{x}\right)={\Vert \mathbf{\Phi }\left(\mathit{x}\right)-\frac{1}{n}\sum _{r=1}^n\mathrm{\Phi }\left({\mathit{x}}_r\right)\Vert }^2$$

(13)

$$\begin{array}{ll}{f}_l\left(\mathit{x}\right)\hfill & =\langle \tilde{\mathbf{\Phi }}\left(\mathit{x}\right)\cdot {\mathit{V}}^l\rangle \hfill \\ \hfill & =\text{∑}_{i=1}^n{\alpha }_i^l[k\left(\mathit{x},{\mathit{x}}_i\right)-\frac{1}{n}\text{∑}_{r=1}^{n}k\left({\mathit{x}}_i,{\mathit{x}}_r\right)-\hfill \\ \hfill & \frac{1}{n}\text{∑}_{r=1}^{n}k\left(\mathit{x},{\mathit{x}}_r\right)+\frac{1}{n^2}\text{∑}_{r,s=1}^{n}k\left({\mathit{x}}_r,{\mathit{x}}_s\right)]\hfill \end{array}$$

where f_l(x) is the projection of Φ̃(x) on the eigenvector V^l, and the index l denotes the l-th eigenvector, with l = 1 for the eigenvector with the largest eigenvalue.

3.3. Abnormal Detection Algorithm Based on Optical Flow Feature Classification

By adopting the histogram of the optical flow orientation feature descriptor and these two novel detection methods, the abnormal event detection method in video streams is summarized in Algorithm 1. [I₁ … I_m] is the set of normal training frames where the individuals are walking in all directions. The abnormal samples are the frames where the individuals are moving toward one direction or running. This definition of an abnormal event indicates that the individuals are attracted by some particular event or escaping from a dangerous zone.

Algorithm 1 Abnormal detection algorithm.
Require:
	Image I.
1:	Compute the optical flow of training frame [I1,…, Im] via the HS method.
	$$\left[I_1,I_2,\dots ,I_m\right]\to \left[O_1,O_2,\dots ,O_m\right]$$
2:	Compute the histogram of the optical flow orientation of the original image or foreground image.
	$$\left[O_1,O_2,\dots ,O_m\right]\to \left[H_1,\dots ,H_m\right]$$
3:	(1) SVM method: training data are learned by the one-class SVM method to obtain the support vectors. $$\left[H_1,\dots ,H_m\right]\underrightarrow{\mathit{\text{SVM}}}\text{support vector}\left[S_1,\dots ,S_{m_1}\right]$$ (2) PCA method: compute the principal components by the KPCA method and measure the squared distance. $$\left[H_1,\dots ,H_m\right]\underrightarrow{\mathit{\text{PCA}}}\text{principal component}\left[P_1,\dots ,P_{m_2}\right]$$
4:	(1) SVM method: Each incoming frame Hn,…,q is classified by one-class SVM. The abnormal event or normal event is detected in the current image. (2) PCA method: each incoming frame Hn,…,q is classified by KPCA.
5:	The detection results are filtered by state transition restriction.

Step 1

Compute the optical flow of each frame via the Horn–Schunck (HS) optical flow method in the gray scale.

Step 2

Calculate the histogram of the optical flow orientation (HOFO) of each frame. The sketch image for choosing the HOFO feature in the original image or in the foreground image is shown in Figure 2. If the HOFO descriptor is computed on the foreground image, the optical flow in the background is zero. Thus, the background area is not considered, and then, computing time is saved.

Step 3

The one-class support vector machine or kernel principal component analysis method is used to classify feature samples of the incoming video frames. The flowchart of our method is shown in Figure 3.

SVM method

The training feature samples are extracted from the normal images, which include HOFO in the original images or in the foreground images. The HOFO feature of the k-th frame is labeled as H_k. The training samples H_1,…,m, m > 1 are gathered, and then, the support vectors are obtained in the SVM training step. Based on the support vectors, the incoming feature samples H_n,…,q are classified.

PCA method

The normal training feature samples for KPCA are mapped into a high-dimensional feature space. In this space, PCA extracts the principal components of the data distribution. Then, the squared distance of each testing sample to the corresponding principal subspace is measured for novelty detection [21].

Step 4

If a normal event or an abnormal event is observed, it means that the video clip holds one state in several consecutive frames. Thus, we use a state transition restriction method by presetting a threshold N to filter the short fluctuation clip. If the number of the predicted abnormal frames after a normal video clip is larger than N, the state of the abnormal detection system is changed from “normal” to “abnormal”.

4. Experimental Results and Analyses

This section presents the experimental results and the analyses of the proposed abnormal detection method. The datasets PETS [9] and UMN [10] are used.

4.1. PETS Dataset

The detection accuracies of PETS dataset under different features and classification methods are shown in Figure 4. The HOFO features are obtained under different conditions, which include original image, foreground image, signed angle, unsigned angle, L1-norm, L1-sqrt, L2-norm, L2hys-0.3, L2hys-0.4 and none normalization. The KPCA novelty detection method obtains less accuracy than the one-class SVM. The best accuracy of the KPCA results is 89.5% under the condition of the original image, signed angle and L1-norm normalization. The best accuracy of the one-class SVM is 96.8% under the condition of the original image, singed angle and L2hys-0.4 normalization. Examples of this high dimension feature space are illustrated in Figure 5 by using the projection of the three largest principal components. The training normal data (labeled as a blue cross) are confused with the testing normal data (labeled as a cyan diamond) and the testing abnormal data (labeled as a red rectangle). In a word, the training data are mixed with the test-abnormal data. One-class SVM has a slack variable, which tunes the number of acceptable outliers of the training data. This soft margin strategy makes the one-class SVM obtain more precision.

The results adjusted by restriction of the state transition are shown in Figure 6. As shown in the figure, the fluctuations between the “abnormal” and “normal” state are reduced. The detection results of the PETS scene are shown in Figure 7.

4.2. UMN Dataset

The HOFO descriptor can represent not only the information of optical flow orientation, but also the optical flow magnitude. The results of the benchmark dataset UMN are shown in Figure 8. The HOFO descriptor can deal with the abnormal scene in which people are running in all directions.

For the lawn scene, the detection accuracies of different conditions are shown in Figure 9. One-class SVM and KPCA classification methods can get great accuracy without the state transition restriction strategy.

For the indoor scene and the plaza scene, the detected accuracy of different conditions are shown in Figures 10 and 11, respectively The restriction of the state transition improves the accuracy In summary, the KPCA method is generally better than one-class SVM for abnormal detection in these experiments. Furthermore, the data distributions need to be considered.

The performance summary of the UMN dataset compared with the state-of-the-art methods is shown in

Table 1. The comparison of our proposed method with the state-of-the-art methods for global abnormal event detection in the UMN dataset. TPR, true positive rate; FPR, false positive rate. NN, nearest neighbor. SRC, sparse reconstruction cost. STCOG, spatial-temporal co-occurrence Gaussian mixture models.

Method	Area under ROC
	Lawn	Indoor	Plaza
Social Force [22]		0.96
Optical Flow [22]		0.84
NN [23]		0.93
SRC [23]	0.995	0.975	0.964
STCOG [24]	0.9362	0.7759	0.9661
HOFO SVM (Ours)	0.9845	0.9037	0.9815
HOFO PCA (Ours)	0.9992	0.9880	0.9989

. The results in the table are not post-processed by the state transition restriction strategy. Our method obtains great accuracy for all three scenes in the UMN dataset.

5. Conclusions

We propose an abnormal detection method by analyzing the optical flow feature. The method is based on two components, computing the histogram of the optical flow orientation (HOFO) and applying one-class support vector machine and kernel principal component analysis for classification. The HOFO feature is computed in the original frame or foreground image. Moreover, the details of the parameters are analyzed. The algorithm has been tested on several video sequences, and the experimental results show the effectiveness of the algorithm. From the experimental results, we can see that the normalization schemes, none and L2hys-0.4, generally get the best performance. The detection results under the signed angle and original image condition is broadly acceptable. In general, the KPCA novelty detection method is as good as one-class SVM, but under a certain distribution of the data, the one-class SVM can obtain more accurate performance.

Future work will aim at reducing the false alarms and training the samples online. Two solutions are under consideration: capturing more efficient features based on the optical flow or replacing the optical flow by other approaches that can represent the information of the events. Online learning is also urgent. Due to the large amount of normal examples, it is hard to learn the training samples as one batch. Moreover, our method focuses on detecting global abnormal events, but detecting local abnormal events is also important. Improving the method to detect the global and local abnormal events jointly is also necessary.