Deep Learning Approach for Human Action Recognition Using a Time Saliency Map Based on Motion Features Considering Camera Movement and Shot in Video Image Sequences

Abstract

In this article, a hierarchical method for action recognition based on temporal and spatial features is proposed. In current HAR methods, camera movement, sensor movement, sudden scene changes, and scene movement can increase motion feature errors and decrease accuracy. Another important aspect to take into account in a HAR method is the required computational cost. The proposed method provides a preprocessing step to address these challenges. As a preprocessing step, the method uses optical flow to detect camera movements and shots in input video image sequences. In the temporal processing block, the optical flow technique is combined with the absolute value of frame differences to obtain a time saliency map. The detection of shots, cancellation of camera movement, and the building of a time saliency map minimise movement detection errors. The time saliency map is then passed to the spatial processing block to segment the moving persons and/or objects in the scene. Because the search region for spatial processing is limited based on the temporal processing results, the computations in the spatial domain are drastically reduced. In the spatial processing block, the scene foreground is extracted in three steps: silhouette extraction, active contour segmentation, and colour segmentation. Key points are selected at the borders of the segmented foreground. The last used features are the intensity and angle of the optical flow of detected key points. Using key point features for action detection reduces the computational cost of the classification step and the required training time. Finally, the features are submitted to a Recurrent Neural Network (RNN) to recognise the involved action. The proposed method was tested using four well-known action datasets: KTH, Weizmann, HMDB51, and UCF101 datasets and its efficiency was evaluated. Since the proposed approach segments salient objects based on motion, edges, and colour features, it can be added as a preprocessing step to most current HAR systems to improve performance.

1. Introduction

Actions are physical behaviours through which humans can, for example, interact with their environment or communicate with each other. The development of artificial intelligence, machine vision, and robotics has significantly expanded Human Action Recognition (HAR) systems, which are now used in several applications, such as robotics and security. For instance, smart surveillance systems are employed in security facilities and smart cities to detect inappropriate and dangerous actions. Typically, visual data is the main input of a HAR system, along with other inputs such as sound or infrared data. A HAR system must be trained based on the user’s actions, or it should include a pre-trained algorithm for action recognition. Machine learning methods in action recognition are usually categorised as knowledge- and data-based, with most current methods belonging to the latter group. In terms of processing steps, HAR systems typically include three distinct steps: image preprocessing, feature extraction, and action recognition. Preprocessing is usually designed according to the application, mainly to remove distracting and unnecessary input elements, which increases the system’s efficiency and reduces the error and processing time. When a HAR system distinguishes people from the scene and focuses on them in the input video image sequence, it can achieve better and faster results than when it tries to process the entire scene.

HAR also has challenges, such as variable ambient light, noise, and camera and/or scene movement, which increase the complexity of the problem. In this article, a HAR method for urban scenes is proposed that uses camera movement cancellation and motion detection to identify the regions of interest. Active contour and colour segmentation are then used to segment the human and important parts of the scene. Finally, the method uses the motion vector of key points for HAR; therefore, by eliminating useless points and keeping just the key points, the computational cost and memory requirements are minimised, and the system’s accuracy is maintained.

2. State of the Art

Most HAR systems emphasise motion features because HAR is a motion-dependent process. On the other hand, in most HAR research, optical flow is the most common technique used to determine the direction and magnitude of the motion. For example, Caetano et al. [1] used a combination of optical flow and Co-occurrence matrices to build features for event detection. Their method considers both the direction and intensity of the motion. The classifier used in their study was a Support Vector Machine (SVM) with a Radial Basis Function (RBF) kernel. Gupta and Balan [2] directly used optical flow for feature extraction. In their study, the output of the Farneback dense optical flow algorithm was converted into colour images and then used to train a Convolutional Neural Network (CNN). Only the frames where an action was present were used in the training and testing phases. Kumar and John [3] used optical flow to build HAR features. They first extracted the foreground of the images, then applied optical flow and used the results to extract features. In this study, the authors only considered frames that contained motion and ignored static image frames, and the used classifier was an SVM with an RBF kernel.

In their HAR system, Rashwan et al. [4,5] used Histograms of Optical flow Co-occurrence (HOF-CO) to build features and CNN and Long Short-Term Memory (LSTM) models as classifiers. Xu et al. [6] used optical flow to extract dense trajectories. Their method used warped optical flow, instead of normal optical flow, to compute a dense trajectory to eliminate camera movement and remove the scene’s background. The method also removed redundant features to reduce the memory required for clustering. The authors considered various methods for classification, such as Artificial Neural Networks (ANN), K-nearest neighbours (KNN), and SVM, and the best result was obtained using the SVM classifier.

Liu et al. [7] combined two different systems for action recognition. The first system used a Motion History Image (MHI), whereas the second one used a Kalman filter, and both systems used a VGG-16 CNN for classification. The final result was achieved by combining the results obtained from the two systems. Kumar et al. [8] proposed a method based on a trajectory matrix for feature extraction along with a probabilistic Kalman filter. The extracted features were applied to a Gated Recurrent Unit (GRU) for the final classification. This method used a Gaussian mixture model, graph concepts, and edge features to increase the system’s accuracy. Abdelbaky and Aly [9] combined spatial and temporal features with two parallel networks and combined their results. Both parallel networks were Principal Component Analysis Networks (PCANet). In this approach, motion features were extracted using a Short-Time Motion Energy Image (ST-MEI) that calculates the difference between consecutive image frames, and an SVM classifier was used. Guha et al. [10] processed image frames with background subtraction, detected the motion region, and extracted four Histogram of Oriented Gradient (HOG), GLCM, Speeded Up Robust Features (SURF), and GIST features from the detected region of each image frame. Then, redundant features were removed using a genetic algorithm, and a Multilayer Perceptron (MLP) classifier was used. Dash et al. [11] used a 3D CNN for action recognition. The three-dimensional networks were trained using Scale-Invariant Feature Transform (SIFT) to build the spatial features and the difference of frames to build the motion features, and the key points were identified using edge detection operators. Khan et al. [12] suggested a 26-layer CNN and motion and temporal features for action recognition. The main innovation of their study was the reduction in the number of used features by using a Poisson distribution along with univariate measures (PDaUM).

Jaouedi et al. [13] proposed a HAR method using a Gaussian mixture model to detect human motion against a scene background, and the results are then processed using an RNN. Zheng et al. [14] used video sketch as an auxiliary feature to enhance the efficiency of HAR systems. The authors used a ranking-based method to find effective action sketches and extract the pattern of each action. Ramya and Rajeswari [15] extracted silhouette images from video image sequences using correlation coefficient and image background removal and then determined the type of action based on distance and entropy features and using an ANN as a classifier. Haddad et al. [16] used a combination of Gunner Farneback’s dense optical flow (GF-OF), Gaussian mixture model, and information divergence to represent human actions accurately. Kullback–Leibler divergence was used as the classification criterion. Snoun et al. [17] extracted the human skeleton for action detection, and dynamic skeleton, skeleton superposition, and body articulation methods were used to increase the accuracy achieved by a Deep Neural Network (DNN) classifier. Abdelbaky and Aly [18] used three deep CNNs for HAR and combined the results using an SVM. The first network was trained based on the original image; the second one was trained to model horizontal changes, and the last one to model vertical changes. Instead of a single 3D CNN, they used three 2D CNNs. Xia and Ma [19] obtained the motion in the input video image frames using optical flow to determine the type of action. The authors calculated the Joint Action relevance and physical characteristics, including Divergence, Curl, and Gradient as features, and used an SVM classifier.

Carrillo et al. [20] used the Riemannian manifold to obtain action patterns. The covariance matrix and a set of improved dense motion trajectories were used to form the Riemannian manifold. Guo and Wang [21] proposed a deep and sparse spatiotemporal Gabor neural network for action detection. Aghaei et al. [22] used Residual Network (ResNet), Conv-Attention- LSTM, and Bidirectional LSTM (BiLSTM) models for the same purpose, using optical flow to extract the motion. Zebhi et al. [23] used Gait History Image (GHI) and gradient to extract spatiotemporal features and the time-sliced averaged gradient boundary to characterise motion, using a VGG-16 classifier. Wang et al. [24] extracted spatial features using gradient and extracted motion features using optical flow, which were then classified using two different 3D CNNs. Khan et al. [25] used DenseNet201 and InceptionV3 to extract features and a Kurtosis-controlled Weighted KNN as the final classifier. Xu et al. [26] proposed a HAR system based on a scene image and human skeleton, which was robust to optical flow changes. The proposed system was a dual-stream model with sparse sampling combined with a scene image classification scheme. Wu et al. [27] used a descriptor-level Improved Dense Trajectory (IDT) and optical flow for feature extraction. El-Assal et al. [28] used the difference between two consecutive image frames and the rate of change of these differences for action detection.

Boualia and Amara [29] submitted video image frames to a 3D CNN to describe the action involved without preprocessing or feature extraction. Mishra et al. [30] identified regions of interest in the input video image frames, then used MHI and motion energy images to extract features. Finally, they performed action classification using an SVM classifier. Ha et al. [31] used image frame difference for motion detection. In their study, three DNNs were trained using motion and spatial features, and the results were combined in a fusion step. Gharahbagh et al. [32] used a combination of temporal and spatial features, such as frame differences and gradient, for the best frame selection in a HAR system to improve the training efficiency. Hajihashemi and Pakizeh [33] proposed a HAR system based on the gradient in both temporal and spatial domains. The authors clustered extracted features and formed a bag of video words representing each action. Deshpande and Warhade [34] used the HOG for feature extraction and stacked HOG features for motion detection. Ma et al. [35] used a pre-trained ResNet18 for feature extraction and a CNN with the Class Incremental Learning (CIL) method as a classifier. Shekokar and Kale [36] performed various preprocessing steps on the image frames, such as brightness and contrast correction and Z-score normalisation, and submitted the results to a DNN structure. Sawanglok and Songmuang [37] extracted the global branch, fine-grain branch, Temporal Shift Module (TSM), and TSM fine-grain branch for their HAR system, which included an ANN classifier.

Shi and Jung [38] used a slow-fast network structure for action detection, which allowed the system to respond differently to slow and fast motions. Gao et al. [39] extracted the silhouette region of the image according to the background and then specified the skeleton in different image frames using an LSTM. Wang et al. [40] used a dense optical flow field to detect motion in video image sequences and submitted the extracted features to a DNN. Nasir et al. [41] segmented the human body region after normalising the input video image frames and performing background removal. The authors extracted temporal and spatial features, such as the 3D Cartesian plane, Joints MOCAP, and n-point trajectory from the segmented human region. Sowmyayani and Rani [42] proposed a DNN for action detection in video retrieval systems. The image frame difference was used to detect motion and remove static frames. Singh et al. [43] used optical flow and HOG features to specify motion regions. Finally, the extracted features were used to create a sparse-coded composite descriptor to model the action. Mithsara [44] used a Yolo v5 network to segment humans and then used skeleton extraction to determine the action type.

Megalingam et al. [45] used SIFT to extract spatial features and optical flow to extract motion features for their HAR system. In another work, Nair and Megalingam [46] reviewed HAR methods. Bayoudh et al. [47] proposed a method based on a hybrid 2D/3D CNN, an LSTM network, and a visual attention mechanism for HAR. Liang et al. [48] used the DCT of image frames as input to a CNN. Khater et al. [49] suggested a residual inception ConvLSTM network for feature extraction in a HAR system. Motion information can also be extracted from non-video data such as skeleton data or 3D (RGB-D) videos. Using different modalities increases the accuracy of human activity recognition (HAR) systems. Momin et al. [50], Sun et al. [51], and Wu and Du [52] investigated HAR systems based on different data modalities. Ahn et al. [53], Vaitesswar and Yeo [54], and Lee et al. [55] used skeleton data for HAR. Wu et al. [56], Radulescu et al. [57], Yan et al. [58], and Liao et al. [59] are among other studies that used depth as a data method in combination with video for HAR. The main classifier in all the aforementioned methods is deep learning. Most multimodal methods use graph-based convolutional neural networks (GCNN) or graph neural networks (GNN) because they provide a deep learning method for irregular domains in the machine learning community.

Based on our review, HAR methods based on deep learning consist of two main steps: feature extraction before deep neural networks and feature extraction with DNN. In both steps, spatial and motion features have been used. However, camera movement, sensor movement, and scene movement can increase motion feature errors and decrease the HAR systems’ accuracy. Another challenge in video-based HAR is the unclear location of the person in the scene. Some studies have used background removal and image extraction in video data as preprocessing steps to solve the above challenges. These methods generally locate a person, isolate the person from the scene’s background, and identify the motion region.

The proposed method focuses on preprocessing to compensate for camera movement, sensor movement, and scene movement before DNN. Some steps of the proposed method are based on modifications of previous studies. These parts are shot detection to detect sudden changes in the scene during action detection, camera movement cancellation to separate the motion of the objects of interest from the movement of the camera, optical flow computation, silhouette image extraction, and active contour and colour-based image segmentation.

Various methods have been proposed for detecting scene changes in video image sequences. For example, Bi et al. [60] used coherent spatio-temporal patterns, Lu and Shi [61] performed candidate segment selection and Singular Value Decomposition (SVD), Mishra [62] employed a complex dual-tree wavelet with Walsh–Hadamard transform. Rashmi and Nagendraswamy [63] used a block-based cumulative approach. However, all of these methods are slow when used for action detection. For camera movement cancellation, Hu et al. [64] used spatial feature matching within image frames to distinguish object motion from camera movement. After this step, the scene background and foreground were separated, and the moving objects were extracted. Some approaches for removing camera movement, which are also employed in creating panoramic images, include image frame registration and stitching methods [65]. Moore et al. [65] proposed a matching approach that uses spatial patterns and a Principal Component Analysis (PCA)-based cost function to remove camera movement. Zhang et al. [66] proposed an optical flow method to separate camera movement from object motion in a scene. They used the difference between the angles of the flow vector in static and moving objects. Once the camera movement is determined, their method corrects the optical flow. The aforementioned camera movement cancellation methods were proposed for other applications and must be modified for a HAR system.

The extraction of silhouette images has been used as one of the useful tools in action detection. For example, Ahammed et al. [67] used silhouette images to detect human walking. They extracted silhouettes from video foregrounds. Lam et al. [68] separated background from foreground in video image sequences by filtering and extracting silhouettes. Jawed et al. [69] used silhouette images to develop a human gait recognition system. In this study, silhouette images are obtained using background and frame difference, using statistical features such as variance and covariance of scene brightness to extract the background. Maity et al. [70] proposed a HAR system using silhouette images, built by making a background history from consecutive image frames and subtracting it from the image frames. Vishwakarma and Dhiman [71] also used silhouette images for action detection based on Gaussian filters, video motion energy, and image texture for the background and foreground segmentation. The proposed method includes active contour and colour-based segmentation. Active contour methods are generally divided into two categories [72]: parametric and geometric active models. In parametric models, the edge of an object is extracted based on its shape and energy features of the image. The traditional models of this method include the original snake model [73], balloon active contour [74], gradient vector flow [75], and vector field convolution [76]. Parametric models offer high processing speed but are sensitive to the initial point and object geometry. On the other hand, geometric models have lower sensitivity to the initial point and object geometry but have higher computational costs. Examples of geometric models include the Mumford–Shah model [77], geodesic active contour [78], the Chan–Vese model [79], and LBF model [80]. Some studies have also employed image co-segmentation methods for human segmentation that require training [81,82]. Merdassi et al. [83] reviewed several image co-segmentation methods and highlighted their strengths and weaknesses.

In the field of colour-based image segmentation, Anitha et al. [84] adjusted colour thresholds of different RGB channels to maximise Otsu’s criterion and Kapur’s entropy. Jing et al. [85] used a Fully Convolutional Neural Network (FCN) for image segmentation. They selected the architecture of this DNN by combining different models [86,87,88,89,90]. Kabilan et al. [91] used the fastmap method for colour-based image segmentation, where the results of fastmap and PCA were combined and optimised in an iterative process. Their method incorporated the Sobel gradient, median filter, and Gaussian smoothing. Abualigah et al. [92] analyzed different image segmentation criteria using heuristic methods and compared the results. In addition to Otsu’s class variance criterion and Kapur’s entropy criterion, the authors also studied fuzzy entropy, Tsallis entropy, and Renyi’s entropy criteria [92]. Sathya et al. [93] used three criteria for image segmentation: Kapur, Otsu, and Minimum Cross Entropy, and combined the results of these methods to increase the segmentation accuracy.

None of the above studies have been used in HAR systems and should be modified to be used. From the above, the innovations of the proposed method are the following:

• The cancellation of the camera movement based on optical flow;
• The use of optical flow and image frame difference to determine motion regions and processing only the selected regions in the later steps, which reduces about 90% of the required computation;
• The use of active contour and colour-based image segmentation in the motion regions for more accurate human segmentation from the scene background;
• The use of previously computed optical flow vectors at selected key points as final features to save computation time.

3. Materials and Methods

In this study, a new hierarchical method for HAR is proposed. This section describes different parts of the proposed method, which is depicted in Figure 1. Some of the shown steps may be discarded, i.e., not considered, depending on the input video image sequences. For example, the first step, which detects shots in the video and identifies scene changes, can be discarded in videos with static scenes. After this step, the temporal processing block detects moving regions in the input video image frame. This block includes camera movement cancellation, optical flow, and image frame difference. Finding regions of movement is significant to reduce the computation required in the following steps. The spatial processing block improves the segmentation of humans and other moving objects based on the results of temporal features. The final features are extracted from key points of the segmented region. The final step is the classification block based on an LSTM.

3.1. Shot Detection

In some videos, sudden changes in the scene during action detection can cause discontinuities and motion detection errors. A HAR system typically relies on segmenting moving regions in the input image frames, which means that scene changes must be taken into account during the feature extraction to avoid errors. The proposed method aims to detect shots and address these errors during the feature extraction step.

In a HAR system, finding a video shot is typically not very complex and can be achieved using simpler methods. In this study, image optical flow and frame difference are used to detect scene changes. When the scene suddenly changes, the image optical flow and frame difference increase suddenly. Between the optical flow and frame difference, the optical flow is more suitable for shot detection because it is more stable than the frame difference. When a shot occurs, the optical flow values in several consecutive frames are higher than normal. When a shot is detected, the proposed method excludes some frames before and after the shot from the feature extraction and action detection processes.

3.2. Camera Movement Cancellation

The objective of the temporal processing block is to separate moving objects from the static components of the input video image sequences. There are two types of motions in videos with actions: the motion of objects and the movement of the camera. Usually, in this domain, the motion of objects is associated with action, and camera movement is uninteresting. Consequently, camera movement should be removed to ensure that the feature extraction is not affected; therefore, the temporal block consists of camera movement cancellation, optical flow, and frame difference sub-blocks. Regarding the priority, first, camera movement should be cancelled, and then, optical flow and frame difference should be calculated.

In our method, static and moving objects were separated by comparing the flow angles. Static objects, as the main part of an image frame, have similar flow angles due to camera movement, and moving objects have different flow angles. A cluster containing many similar or denser components is considered to contain camera movement, while the remaining clusters represent object motion. The camera movement in the horizontal and vertical directions is determined according to the angle and intensity of the optical flow in the centre of the denser cluster. Subsequently, all flow vectors are adjusted according to the camera movement. In this way, the static components of the scene that previously had a motion vector due to the camera movement become static. Moreover, it is not necessary to recalculate the optical flow, and the flow results can be corrected by subtracting the centre of the denser cluster from all flows. In the next step, the frame difference is calculated based on the corrected frames. After the camera movement cancellation, the image optical flow and frame difference provide a more accurate representation of the object’s motion. In cases where the dataset is acquired with a static camera, this step is unnecessary and can be discarded.

3.3. Optical Flow

The proposed method uses two different approaches for motion detection to enhance its efficiency and reduce errors. The first approach is optical flow. One limitation of this method is its inability to distinguish between the movements caused by the camera and the actual motion in the scene. This limitation has been addressed through the camera movement cancellation step. In addition, the optical flow method has a relatively high computational cost because of its spatial and temporal derivatives. Among the various current optical flow algorithms, the Lucas-Kanade and Horn-Shunck algorithms have been mainly used in HAR systems. The difference between these two methods lies in how they compute the derivative and some initial assumptions. Both methods use a similar approach to estimate the motion vector. In a two-dimensional (2D) grey video image sequence, it is assumed that the motion in $x,y$ directions is $\delta x$, $\delta y$ at time $\delta t$, and the image pixel values of the first image frame are $E\left(x,y,t\right)$, and of the second is $E(x+\delta x,y+\delta y,t+\delta t)$, thus:

$$E(x+\delta x,y+\delta y,t+\delta t)=E(x,y,t)+\frac{\partial E}{\partial x}\delta x+\frac{\partial E}{\partial y}\delta y+\frac{\partial E}{\partial t}\delta t+\epsilon ,$$

(1)

where $\epsilon$ represents the error that occurs due to the nonlinear form of the motion. The motion in a short time between two image frames can be assumed to be linear:

$$\epsilon \approx 0.$$

(2)

The brightness of objects usually does not change significantly between two consecutive image frames, so each object has approximately the same brightness; hence:

$$E(x+\delta x,y+\delta y,t+\delta t)=E(x,y,t).$$

(3)

According to Equations (2) and (3), one can obtain:

$$\frac{\partial E}{\partial x}\delta x+\frac{\partial E}{\partial y}\delta y+\frac{\partial E}{\partial t}\delta t=0.$$

(4)

After dividing by $\partial t$ and simplifying the previous equation, one obtains:

$$\begin{array}{c}\frac{\partial E}{\partial x}\frac{\delta x}{\delta t}+\frac{\partial E}{\partial y}\frac{\delta y}{\delta t}=-\frac{\partial E}{\partial t},\hfill \\ \frac{\partial E}{\partial x}{V}_x+\frac{\partial E}{\partial y}{V}_y=-\frac{\partial E}{\partial t},\hfill \end{array}$$

(5)

which can be rewritten in the form of:

$$\begin{array}{c}{\nabla }_{y}E.V_x+{\nabla }_{y}E.V_y=-\frac{\partial E}{\partial t},\hfill \\ \nabla E.\left[\begin{array}{c}{V}_x\hfill \\ V_y\hfill \end{array}\right]=-\frac{\partial E}{\partial t},\hfill \end{array}$$

(6)

where $V_x$ and $V_y$ are the velocities of the object in the horizontal and vertical directions, which are unknown, and ∇ is the gradient of motion in the spatial domain. Because Equation (6) involves two unknowns, some constraints must be introduced to solve it. Assuming that the motion of the objects between two image frames is smooth in all directions, one obtains:

$$\frac{\partial V_x}{\partial x},\frac{\partial V_y}{\partial y},\frac{\partial V_x}{\partial y},\frac{\partial V_y}{\partial x}\to 0,$$

(7)

or

$$\begin{array}{c}{\left(\frac{\partial V_x}{\partial x}\right)}^2+{\left(\frac{\partial V_x}{\partial y}\right)}^2\to 0,\hfill \\ {\left(\frac{\partial V_y}{\partial y}\right)}^2+{\left(\frac{\partial V_y}{\partial x}\right)}^2\to 0.\hfill \end{array}$$

(8)

Solution methods such as Farneback, Horn–Schunck, Lucas–Kanade, and Lucas–Kanade derivative of Gaussian have been suggested to determine $Vx$ and $Vy$ while minimising both terms in Equation (8) and satisfying Equation (6). The Lucas–Kanade algorithm, implemented in parallel processing mode [94], is used for optical flow calculation in the present study.

3.4. Time Analysis Block

In summary, in the first step of the temporal block, camera movement is removed using optical flow data obtained in the shot selection step. The image frame difference is calculated after the camera movement cancellation. The motion vector obtained using optical flow is adjusted according to the camera movement vector and then used as a saliency map in combination with the result of the frame difference. The relationship used in the combination of optical flow and frame difference is:

$$SaliencyMap={\left(\mathrm{optical}\mathrm{flow}\right)}^{\alpha }{\left(Framedifference\right)}^{\beta },$$

(9)

where values of $\alpha$ and $\beta$ are positive numbers greater than 1 (one) and, here, were calculated in a simulation step (see Section 5.1). Therefore, because the optical flow achieves a better motion pattern than the frame difference, it has a greater influence than the frame difference in Equation (9). The spatial processing block has three stages: silhouette image extraction, active contour segmentation, and colour-based image segmentation.

Because moving parts usually occupy a small region in the image frame, spatial feature extraction can be performed quickly with relatively few calculations.

3.5. Silhouette Image Extraction

In the current study, silhouette extraction is performed by assuming that the background contains motionless regions or is far from the motion region. First, the gradient of the image frame is calculated using the Sobel operator. It is assumed that the silhouette region is rectangular and contains motion, as well as parts that overlap with the motion region. This includes the entire body, i.e., incorporating both moving and non-moving parts, to differentiate between human actions accurately. The maximum dimension of the rectangular window for the silhouette image is assumed to be $1.5$ times the salient region specified in the temporal processing block. As the motion region has already been identified in the preceding steps, the silhouette extraction is more accurate and faster than the general methods.

3.6. Active Contour Segmentation

Silhouette extraction approximates the boundaries of an object; therefore, accurate segmentation methods must then be used to detect the entire object of interest. Active contour is one of the powerful image segmentation methods based on spatial features. Because pre-trained models are not acceptable and geometric methods are more efficient, the proposed solution includes a geometric method. Among the geometric methods, [79,80] showed a lower error and good performance; thus, the model suggested in [80] is used in the proposed solution.

3.7. Colour-Based Image Segmentation

The objective of this block is to extract temporal features based on colour information, which, if the location of the person or object is known, can accurately distinguish the person or object from the scene background. Here, the Sathya et al. [93] method based only on Minimum Cross Entropy criteria is used. In the proposed method, colour-based segmentation is just applied to the silhouette region, and the final segmentation is limited to the silhouette region dilated $1.5$ times. The outcome of this step, which is an accurate segmentation of humans and moving objects, is then transferred to the final feature extraction step.

3.8. Final Feature Extraction

After segmenting the motion region, which includes humans and other foreground objects, appropriate features must be extracted. The final feature is a vector, including x and y as the location and motion vector of some selected points of a segmented human. The main challenge in this step is that despite all the steps taken to extract spatial and temporal features, there are still segmentation errors. Moreover, the angle between the person and the camera plays a crucial role in feature extraction. In a first attempt, a 15-point model was employed for feature extraction, which can model simple motion if the view is assumed from the front, Figure 2.

In the 15-point model, the front view is assumed to be a complete view. However, a challenge arises when a body part is missing during segmentation because it leads to incomplete feature extraction. To address this issue, several assumptions were made:

• A 15-point model was used in the dataset, including the whole body view. In cases where any of these 15 points are missing, the corresponding position is considered empty.
• In cases where a full body view is unavailable, the feature vector is created by selecting 65 points along the extracted contour, with equal distances between them, as key points.
• In the full-body model, the extracted feature includes the location of the points relative to point 9 as an approximation to the body centroid and the intensity and direction of the motion vector in the key points. If the 15-point model is not available, the point with the centre of the segments is considered the coordinate origin.
• In general, 65 points are selected, each with four features, including x and y locations and motion vector.

It should be noted that the input of the DNN classifier is a vector with a dimension of $65\times 4\times \mathrm{Nf}$ when full body view is not available, and $15\times 4\times \mathrm{Nf}$ when full body view is available, where $\mathrm{Nf}$ is the number of image frames. When an action is repeated in a video image sequence, these time intervals are used as two samples of the same action in the training step. In this step, the calculated optical flow at selected points is used as features, and there is no need to extract new features.

3.9. Long Short-Term Memory

In the last step, an RNN is used as a classifier. Among DNNs, RNNs are the most suitable structures for processing time series data, such as actions, where each sample depends on previous samples. In a basic RNN, due to its simple structure, memory limitations in the hidden layers, and weight update algorithm that uses gradient-based methods, the system cannot be properly trained when the pattern in the data, such as action data, spans a long period. In gradient-based updating, the influence of the data in the more distant samples decreases compared to the closer samples until it eventually becomes 0 (zero). LSTM is a modified RNN used for learning long-term patterns. Figure 3 shows the structure of an LSTM.

The LSTM equations are as follows:

$$\begin{array}{c}{f}_t={\sigma }_f\left(W_{xf}{x}_t+W_{hf}{h}_{t-1}+b_f\right),\hfill \\ g_t={\sigma }_g\left(W_{xc}{x}_t+W_{hc}{h}_{t-1}+b_c\right),\hfill \\ i_t={\sigma }_f\left(W_{xi}{x}_t+W_{hi}{h}_{t-1}+b_i\right),\hfill \\ c_t=f_t\odot c_{t-1}\odot g_t\odot i_t,\hfill \\ o_t={\sigma }_f\left(W_{xo}{x}_t+W_{ho}{h}_{t-1}+b_o\right),\hfill \\ h_t=o_t\odot c_t,\hfill \end{array}$$

(10)

where $x_t$ is the input, $h_{t-1}$ the previous states, $c_t$ the previous states within the forget gate, W the weight and b the bias of each inner block of the LSTM, $\sigma$ is the activation function, Hedmard multiplication $\left(\odot \right)$ corresponds to pointwise multiplication, and the final output is $h_t$. The forget gate determines which inputs and previous states affect the output and which should be discarded. This enables the LSTM to learn long-term patterns.

4. Experimental Setting

4.1. Datasets

Several action datasets with different conditions are available for evaluating HAR methods. Five datasets were used in this study: BVSD, KTH, Weizmann, HMDB51, and UCF101 datasets, Figure 4. Details of these datasets are given in the following sections.

4.1.1. Berkeley Video Segmentation Dataset

The Berkeley Video Segmentation Dataset (BVSD) was the first dataset used to determine the powers in the saliency map function (Equation (9)). This dataset consists of high-resolution video image frames labelled for evaluating video segmentation methods [95], and was later extended to evaluate motion detection methods [96].

Although this dataset contains some errors and incorrect segmentations in several frames, it is a useful benchmark for evaluating methods that separate moving objects from a scene. Consequently, it has been widely used in many studies [97,98,99].

4.1.2. KTH Dataset

The KTH Dataset contains 600 video image sequences of six different actions [100]. There are a total of 2391 frames in the videos, all acquired in black and white at a resolution of $120\times 160$, and only one person is seen in each video. The actions included in this dataset are walking, jogging, running, boxing, hand waving, and hand clapping. The videos were acquired from a fixed distance, and the person’s body is fully visible in them, and the angle and direction of the scene are fixed. However, in some cases, the human scale changes. The clothing of the people in the scene also changes. Because the camera angle and scene are fixed, and there are no sudden changes in the scene or background, and therefore no shot detection or camera movement cancellation is required for this dataset. It has been considered a good benchmark in many studies [101,102,103,104]. The low resolution of the videos also reduces the required computations.

4.1.3. Weizmann Dataset

The Weizmann dataset [105] was acquired in a static scene with simple backgrounds as a HAR dataset for outdoor environments. There is only one person in each action, and there are 10 actions involved, including Running, Walking, Skipping, Jack, Jump, Pjump, Side, Wave2, Wave1, and Bending, so it includes more actions than the KTH dataset. The number of videos in this dataset is 90, and the resolution is $144\times 180$, which is higher than that of the KTH dataset. Since there are no sudden scene changes or camera movement in this dataset, shot detection and camera movement cancellation are not required. The subject is fully positioned in the video, and therefore, the proposed 15-point model can be used for feature extraction. This dataset has been extensively used to evaluate HAR methods [106,107,108]. In this study, the Weizmann dataset was used as the second dataset to calculate the powers of the saliency map function (Equation (9)). Because the saliency map function contained only two variables, it was unnecessary to use optimisation methods to calculate the optimal values, and the selection was performed simply by assigning two different intervals for the powers and checking all the possible values.

4.1.4. HMDB51 Dataset

The HMDB51 dataset [109] was acquired under real conditions and contains all the challenging factors a HAR system may encounter, including camera movement, complete absence of humans in many videos, i.e., considerably long periods without action or people, sudden scene changes, different video qualities, and fluctuating lighting conditions. According to these challenging factors, all parts of the proposed algorithm are required to process this dataset, which contains 51 different actions tagged in about 7000 different clips. The videos have varying lengths, with a total duration of 101 min. The actions of this dataset are classified into the following five groups:

• General facial actions;
• Facial actions with object manipulation;
• General body movements;
• Body movements with object interaction;
• Body movements with human interaction.

Additionally, it should be noted that each category includes different actions.

4.1.5. UCF101 Dataset

The UCF101 dataset, similar to the HMDB51 dataset, was acquired under real-world conditions and includes camera movement, scene changes, and lighting changes [110].

This dataset contains 101 different actions, such as Baseball Pitch, Basketball, Shooting, and Bench Press. The videos in this dataset are all in colour, with varying duration, and higher resolution than the Weizmann and KTH datasets.

Table 1. Characteristics of the UCF101 dataset.

Actions	101
Clips	13,320
Groups per Action	25
Clips per Group	4–7
Mean Clip Length	7.21 s
Total Duration	1600 min
Min Clip Length	1.06 s
Max Clip Length	71.04 s
Image Frame Rate	25 fps
Image Resolution	320 × 240
Audio	Yes (51 actions)

presents details on this dataset.

4.2. Evaluation Metrics

In a HAR system, if the system is binary, i.e., it only distinguishes between the presence or absence of an action, the following criteria are used to evaluate the system:

• True positive: Action exists and is correctly detected;
• True negative: Absence of action is correctly detected;
• False positive: Action does not exist, but the system mistakenly indicates its existence;
• False negative: Action is present but incorrectly detected as absent.

In multi-class mode, i.e., where the number of actions is more than two, the confusion matrix is used to evaluate the system. In this matrix, each row or column represents a class, the main diagonal elements show the number of correctly recognised samples of that class, and the other elements indicate the following:

• If the element is interpreted based on the action associated with its row, it indicates the number of samples of row classes that were mistakenly assigned to other classes;
• If the element is interpreted based on the action associated with its column, it indicates the number of samples of other classes that were mistakenly assigned to the column class.

The sum of the main diagonal elements divided by the total number of samples indicates the system’s accuracy. Confusion matrices can be used to evaluate the performance of HAR systems in each class separately. The percentage reported at the end of each row of the confusion matrix is the true positive rate for the row class. The percentage reported at the end of each column of the confusion matrix is the precision of the column class. Precision is the number of true positives divided by the total number of elements assigned to that class by the system.

5. Experimental Results and Discussion

A series of simulations were performed to determine the parameters of the different parts of the proposed method and evaluate its competence. In this section, details of the results are presented and discussed.

5.1. Time Features

Firstly, the values of $\alpha$ and $\beta$ of Equation (9) should be calculated. An optimisation approach using the GA method was used to determine these values. The $\alpha$ and $\beta$ were chosen as limited values between 1 (one) and 10, and the aim was to obtain the minimum value of these variables to find the moving objects between frames. All Weizmann and BVSD samples are input for the GA cost function. The default values of the MATLAB optimisation toolbox were used to implement the GA. Finally, the values of $\alpha$ and $\beta$ were obtained as equal to 1.32 and 1, respectively. The results of the optical flow and frame difference blocks before and after cancelling the camera movement are shown in Figure 5. In the selected frames, the person moves slightly, and the camera moves simultaneously.

As can be seen, the optical flow and frame difference without the camera movement cancellation have relatively high errors. In these two cases, parts of the door and borders are erroneously selected as moving objects. After the cancellation, the optical flow and frame difference have much better results; however, in both cases, some noise can be seen, and the static objects are very faintly visible. The result of the optical flow is better than that of the frame difference with and without the camera movement cancellation. After combining the two results using the proposed function, the border around the image disappeared completely, and the person was clearly separated from the background. The reduction of noise and the absence of borders in the temporal processing step are very important because, in this step, the region of interest was extracted and transferred to the next steps. In Figure 6, there is another example where the camera moves simultaneously with the person under study. Before the camera movement cancellation, the error was significantly high, as can be perceived from the results.

After the camera movement cancellation, the frame difference and optical flow results were much better, although the left margin and some discontinuities are still visible. In the lower part of Figure 6, it can be observed that after applying the proposed approach, the errors disappeared completely, and the part containing the person was completely separated from the rest of the scene. The resulting output was then applied to the spatial processing block as a time saliency map to segment important parts of the frames using spatial features. Figure 7 shows the active contour segmentation result for a specified silhouette region. However, due to the presence of static object edges within the silhouette region that overlaps with the boundaries of the moving object, the active contour did not yield optimal results. The active contour performs better in cases where only the edges of the moving object are inside the silhouette region without overlapping with other objects.

Figure 8 shows the result of colour-based image segmentation within a silhouette region in green. In the proposed method, the colours of the outer part of the silhouette region and the result of the active contour are chosen as background colours, and the colours of the inner part of the moving region are chosen as the foreground. Since the moving region was accurately separated, the moving person was accurately segmented based on its colour difference from the background. The most important parameter in colour-based image segmentation is the colour difference between moving objects and other objects in the silhouette region. The resulting segmentation is combined with the active contour segmentation method by intersection and then used for feature extraction. Figure 9 shows the final segmented region, which confirms the proposed method provides a very good approximation of the moving person.

5.2. Effect of the Different Steps of the Proposed Method

In this section, different parts of the proposed method are added to a HAR system, and their impact on the system’s efficiency is evaluated. KTH, Weizmann, and HMB51 datasets were used to evaluate the proposed method. According to the preprocessing steps’ structure, the training and testing data selection was done only in the DNN training (Section 5.2), and the rest of the steps were done using the whole data. In the DNN training, for each dataset, only its data was used for training and testing. The selected percentages for training and testing at this step were 70 and 30, respectively.

In the first evaluation, the proposed time saliency map (Equation (9)) was compared with the conventional optical flow in KTH and Weizmann datasets. The two implementations were performed under similar conditions regarding the training and testing data, classifier structure, and DNN parameters. The only difference between the two implementations was the time saliency map, with one being based on the optical flow and the other on the proposed method.

Table 2. Confusion matrix obtained for the HAR system using optical flow on the KTH dataset.

	Walking	Jogging	Running	Boxing	Hand Waving	Hand Clapping	Accuracy (%)
Walking	27	3	0	0	0	0	90
Jogging	1	27	2	0	0	0	90
Running	0	3	27	0	0	0	90
Boxing	0	0	0	28	0	2	93
Hand waving	0	0	0	1	27	2	90
Hand clapping	0	0	0	0	0	29	100
Accuracy (%)	96	82	93	97	100	88

,

Table 3. Confusion matrix obtained for the HAR system using the proposed method on the KTH dataset.

	Walking	Jogging	Running	Boxing	Hand Waving	Hand Clapping	Accuracy (%)
Walking	30	0	0	0	0	0	100
Jogging	0	30	0	0	0	0	100
Running	0	1	29	0	0	0	96.67
Boxing	0	0	0	30	0	0	100
Hand waving	0	0	0	0	30	0	100
Hand clapping	0	0	0	0	0	30	100
Accuracy (%)	100	96.77	100	100	100	100

,

Table 4. Confusion matrix obtained for the HAR system using optical flow on the Weizmann dataset.

Activity	Running	Walking	Skipping	Jack	Jump	Pjump	Side	Wave2	Wave1	Bending	Accuracy (%)
Running	3	0	0	0	0	0	0	0	0	0	100
Walking	1	2	0	0	0	0	0	0	0	0	66.67
Skipping	0	0	3	0	0	0	0	0	0	0	100
Jack	0	0	0	2	0	0	0	1	0	0	66.67
Jump	0	0	0	0	3	0	0	0	0	0	100
Pjump	0	0	0	0	0	2	1	0	0	0	66.67
Side	0	1	0	0	0	0	2	0	0	0	66.67
Wave2	0	0	0	0	0	0	0	2	1	0	66.67
Wave1	0	0	0	0	0	0	0	0	3	0	100
Bending	0	0	0	0	0	0	0	0	0	3	100
Accuracy (%)	75	66.67	100	100	100	100	66.67	66.67	75	100

and

Table 5. Confusion matrix obtained for the proposed HAR system on the Weizmann dataset.

Activity	Running	Walking	Skipping	Jack	Jump	Pjump	Side	Wave2	Wave1	Bending	Accuracy (%)
Running	3	0	0	0	0	0	0	0	0	0	100
Walking	1	2	0	0	0	0	0	0	0	0	66.67
Skipping	0	0	3	0	0	0	0	0	0	0	100
Jack	0	0	0	3	0	0	0	0	0	0	100
Jump	0	0	0	0	3	0	0	0	0	0	100
Pjump	0	0	0	0	0	3	0	0	0	0	100
Side	0	0	0	0	0	0	3	0	0	0	100
Wave2	0	1	0	0	0	0	0	2	0	0	66.67
Wave1	0	0	0	0	0	0	0	0	3	0	100
Bending	0	0	0	0	0	0	0	0	0	3	100
Accuracy (%)	75	66.67	100	100	100	100	100	100	100	100

show the results obtained for KTH and Weizmann datasets. Considering that all comparison conditions were the same, it can be concluded that using the proposed temporal saliency function, the performance was enhanced even when there was no movement in the scene.

In general, the accuracy of the conventional optical flow was 92.18%, which was increased to 99% by using the proposed method. The confusion matrices for the test data are shown in Table 2 and Table 3. The KTH dataset contained 30 samples per class in the test data. In the Weizmann dataset, there are 9 or 10 different samples for each action. Here, 3 samples were used for testing and the rest for training. In this dataset, the scene is static, and the camera does not move; therefore, it was possible to accurately evaluate the feature extraction part of the proposed method without the effect of the camera movement cancellation. Table 4 and Table 5 show the results obtained by the conventional and proposed methods, respectively. All the other simulation conditions remained unchanged.

The results demonstrate that the proposed method achieved an accuracy of 92%, while the conventional method achieved an accuracy of 83%. The final evaluation was performed using the HMDB51 dataset.

Table 6. Total number of samples considered for the HMDB51 dataset for “Training” and “Testing” in terms of the action involved.

	Total	Train	Test
Clap	130	91	39
Wave	68	76	32
Climb	151	66	45
Jump	232	162	70
Run	548	384	164
Walk	64	73	31

shows the number of samples and actions considered using this dataset.

The results reported here refer to the test data. As shown in

Table 7. Confusion matrix obtained for the HAR system using optical flow without camera movement cancellation on the HMDB51 dataset.

Activity	Clap	Wave	Climb	Jump	Run	Walk	Accuracy (%)
Clap	22	10	3	2	1	1	56.41
Wave	7	20	1	3	1	0	62.50
Climb	0	6	22	10	5	2	48.89
Jump	8	10	12	32	3	5	45.71
Run	8	8	12	15	82	39	50.00
Walk	0	2	0	3	9	17	54.84
Accuracy (%)	48.89	35.71	44.00	49.23	81.19	26.56

, the overall accuracy of a HAR system using optical flow without camera movement cancellation on the HMDB51 dataset was $195/381\cong 51\%$, which was calculated as the sum of the main diagonal elements divided by the total number of samples (see Section 4.2).

Table 8. Confusion matrix obtained for the proposed HAR system on the HMDB51 dataset.

Activity	Clap	Wave	Climb	Jump	Run	Walk	Accuracy (%)
Clap	27	9	0	2	1	0	69.23
Wave	4	22	0	1	4	1	68.75
Climb	3	0	35	7	0	0	77.78
Jump	0	2	8	54	3	3	77.14
Run	3	6	4	9	108	34	65.85
Walk	0	0	4	0	2	25	80.65
Accuracy (%)	72.97	56.41	68.63	73.97	91.53	39.68

indicates that the system’s overall accuracy was $271/381\cong 71\%$. Therefore, by adding camera movement cancellation as a preprocessing step at the input of the used HAR system, the feature extraction efficiency was increased, and the system error decreased.

5.3. Comparison with Other Methods

Considering that the proposed method focuses on the cancellation of camera movement and accurate moving object extraction, it was tested using a simple DNN structure. The reported percentages were obtained based on the test set in all comparisons.

Table 9. Comparison among state-of-the-art and proposed methods on the KTH dataset.

Ref	Accuracy (%)
Sargano et al. [113]	89.86
Dasari et al. [112]	87
El-Henawy et al. [114]	95
Jain et al. [115]	95
Shao et al. [116]	95
Yang et al. [117]	96
Cheng et al. [118]	97
Liu et al. [111]	94
Sharif et al. [119]	99
Elharrouss et al. [120]	99.82
Shao et al. [121]	97.50
Shi et al. [122]	96.80
Aslan L et al. [123]	96.16
Afza et al. [124]	100
Proposed	99.44

and Figure 10 show the accuracy of recent state-of-the-art methods on the KTH dataset. It can be seen that the proposed method was slightly weaker than the methods proposed in [111,112], but it should be noted that these two methods are more complex than the proposed method.

Table 10. Comparison among state-of-the-art and proposed methods on the Weizmann dataset.

Ref	Accuracy (%)
Jiang et al. [125]	95
Zhang et al. [126]	98
Kaminski et al. [127]	81
Sharif et al. [119]	95
Elharrouss et al. [120]	99.85
Simonyan and Zisserman [128]	92.80
Tran et al. [129]	95.69
Tran et al. [130]	94.03
Karpathy et al. [131]	91.11
Li et al. [132]	97.90
Proposed	96.67

and Figure 11 show the results for the Weizmann dataset, which confirm that the proposed method was more efficient than most existing methods.

The final comparison was conducted on the HMDB51 and UCF101 datasets. Due to the large amount of data in these datasets, researchers often use different assumptions to report their results. In this study, the proposed method was applied to some actions in the two datasets and compared with the most recent relevant methods. The confusion matrix of the test data for the UCF101 dataset is presented in

Table 11. Confusion matrix for the proposed HAR system on the UCF101 dataset.

Activity	Basketball	Biking	Billiards	Diving	Haircut	HorseRace	Skiing	Walking with Dog	Accuracy (%)
Basketball	36	0	0	3	0	0	0	1	90
Biking	0	34	0	0	0	2	3	1	85
Billiards	1	0	41	0	0	0	0	3	91.11
Diving	3	0	0	40	0	2	0	0	88.89
Haircut	1	0	2	0	36	0	0	1	90
HorseRace	0	1	0	0	0	35	1	0	94.59
Skiing	0	3	0	0	0	0	37	0	92.5
Walking with Dog	1	0	1	0	0	0	0	35	94.59
Accuracy (%)	85.71	89.47	93.18	93.02	100	89.74	90.24	85.37

, and the results of the proposed method are compared with the ones of related works in

Table 12. Comparison among state-of-the-art and proposed methods on the HMDB51 and UCF101 datasets.

Method	HMDB51 (%)	UCF101 (%)
Zhang et al. [133]	78.82	97.27
Carreira and Zisserman [134]	80.7	98
Wang et al. [135]	68.5	94
He et al. [136]	——	93.5
Jiang et al. [137]	72.2	96.2
Li et al. [132]	73.3	96.9
Proposed	71.13	90.74

and Figure 12. Despite the significantly lower complexity of the proposed system, the achieved accuracy was competitive with most of the state-of-the-art methods in this field.

In other words, the accuracy improves significantly when the HAR methods minimise the influence of static background objects and camera movement in the preprocessing step and perform proper initial segmentations. In this study, all simulations were performed on MATLAB R2020b and PyCharm IDE Professional Edition 2020 using a personal computer with an Intel Core i7-9700 CPU (8 cores, 12M Cache, up to 4.70 GHz), 16GB of RAM and a GeForce RTX 2070 SUPER 8GB GPU. Among the different parts of the proposed method, the camera motion cancellation in the temporal feature extraction had the longest execution time. In the spatial feature extraction phase, all three parts had relatively equal execution time, but colour-based image segmentation took the longest. In total, the processing of every 10-frame set took about 1.9 s. Finally, considering deep neural network response time, action detection in a 10-s video of the UCF 101 dataset (Table 1), with a frame rate of 25 frames per second, took less than 60 s.

6. Conclusions

In this study, an accurate method for a HAR system was proposed. Generally, a deep learning HAR system consists of two steps: preprocessing, which may include feature extraction, and deep learning-based classification. Camera movement and shot changes in the input video and the unknown location of the person under study in the scene are some of the most important factors that can cause errors in HAR systems. The proposed method improves the efficiency of existing HAR systems by adding preprocessing steps, including camera movement cancellation and shot detection, and specifying the human’s location as accurately as possible. To reuse the features extracted in the preprocessing step, the proposed method was designed so that extracted features can be used with a deep learning classifier. Therefore, it can be added to existing HAR systems with minimal computational overhead.

The proposed method was added to a deep learning-based HAR system, and the effectiveness of each step on system accuracy was validated. The results indicated that the accuracy achieved by the proposed method was comparable to that of state-of-the-art methods across all tested scenarios. Because the proposed unsupervised approach separates the important regions of each frame based on motion and then texture- and colour-based features, it can be added as a preprocessing block to HAR methods and enhance their overall performance.

One of the main areas of future work is to investigate the sensitivity of existing HAR methods to camera movement and shot in the input videos. Another possibility would be to investigate the impact of camera movement and shot changes on different optical flow extraction methods. The findings will help to develop more robust HAR methods that are less sensitive to camera movement and shot in videos.