Combining Keyframes and Image Classification for Violent Behavior Recognition

Abstract

Surveillance cameras are increasingly prevalent in public places, and security services urgently need to monitor violence in real time. However, the current violent-behavior-recognition models focus on spatiotemporal feature extraction, which has high hardware resource requirements and can be affected by numerous interference factors, such as background information and camera movement. Our experiments have found that violent and non-violent video frames can be classified by deep-learning models. Therefore, this paper proposes a keyframe-based violent-behavior-recognition scheme. Our scheme considers video frames as independent events and judges violent events based on whether the number of keyframes exceeds a given threshold, which reduces hardware requirements. Moreover, to overcome interference factors, we propose a new training method in which the background-removed and original image pair facilitates feature extraction of deep-learning models and does not add any complexity to the networks. Comprehensive experiments demonstrate that our scheme achieves state-of-the-art performance for the RLVS, Violent Flow, and Hockey Fights datasets, outperforming existing methods.

1. Introduction

Surveillance cameras are increasingly prevalent in public places, and are used to collect evidence and deter potential criminals. Security services urgently need to monitor violence in real time. The main violence detection models include “handcrafted” feature methods [1,2], CNN + LSTM [3,4], 3D convolutional networks [5,6], two-stream convolutional networks [7,8], and human pose estimation models [9,10], all of which mainly use CNN, 3DCNN, and LSTM to extract spatiotemporal features. However, these methods have high hardware resource requirements and can be impacted by various interference factors, such as background information, camera movement, and occlusion.

To overcome these problems, S3-Net [11] locates and segments target sub-scenes while extracting structured time-series semantic features, which are used as inputs by an LSTM-based spatiotemporal model. Motivated by S3-Net, we propose that segmentation of the human body can enable the model to learn better features and develop a stronger generalization ability. Semantic segmentation [12,13,14,15] can remove the background from video frames. We use Deeplab-V3plus [14] to obtain 100,000 background-removed images (segmented images). However, on the test set of the original image, the recognition rate of a model trained on segmented images is much lower than one trained on the original images; on the test set of the segmented image, the recognition rate of a model trained on original images is much lower than one trained on the segmented images. This means that the deep-learning model cannot learn the essential image features, because the mapping from inputs to outputs performed by deep nets quickly ceases to make sense if new inputs differ even slightly from those observed during training. This phenomenon is common in automatic speech recognition, since datasets without noise are readily available. The models trained with clean data were difficult to apply to noisy scenes. Therefore, they add noise to the data to increase the robustness of the model. For example, the wav2vec-Switch [16] encodes noise robustness into contextualized representations of speech via contrastive learning, which feeds original–noisy speech pairs simultaneously into the network. Therefore, we use pairs of segmented and original images to train the model, forcing the network to produce consistent predictions for the original and segmented image. Specifically, if the model wants to minimize the loss, its best choice will be to match the prediction of the segmented image with the original image. This enables the development of a robust model that predicts the original image in line with the segmented image.

However, current violent behavior recognition models have high hardware resource requirements, making them difficult to deploy on a large scale. Ismael et al. [17] propose a hybrid “handcrafted/learned” feature framework, which provides better accuracy than previous feature-learning methods. Inspired by this, we combine keyframes and deep learning to develop an approach that extracts the keyframes of the video and judges the violent behavior with reference to the keyframe recognition results. Specifically, the proposed method treats video frames as independent events and judges whether a violent event is occurring based on whether the number of keyframes exceeds a given threshold. The number of extracted keyframes can control the recognition efficiency of the system, while the judgment threshold can control the system’s sensitivity.

In summary, we propose a new keyframe-based violent-behavior-recognition scheme with low hardware requirements and a high recognition rate. Furthermore, we build a dataset of violent images with the backgrounds removed and propose a new training method.

2. Related Works

2.1. Datasets

Existing violent behavior datasets include Real Life Violence Situations Dataset (RLVS) [18], Hockey Fights Dataset [19], RWF-2000 [20], CCTV Fights Dataset [21], and Violent-Flow [22]. The RLVS contains 2000 short videos divided into 1000 violence videos and 1000 non-violence videos. The RWF-2000 contains 1000 violent videos and 1000 non-violent videos captured by surveillance cameras in real-world scenes. The CCTV dataset contains 1000 videos of real fights, with more than 8 h of annotated CCTV footage. The Hockey Fights dataset introduces a video database containing 1000 sequences divided in two groups: fights and non-fights. Existing semantic segmentation datasets include PASCAL Context [23], Cityscapes [24], Microsoft Common Objects in Context (MS COCO) [25], and VSPW [26], all of which contain human annotations. Human poses are normal in the semantic segmentation, while human poses are abnormal in the violence datasets. In addition, semantic segmentation datasets mostly consist of high-quality images (for example, the pictures in the COCO dataset are high resolution and taken with professional equipment; the latest VSPW dataset is the first large-scale dataset for video scene parsing in the wild, but more than 96% of its video frames are high-quality and lack abnormal behavior), while images in violent behavior datasets usually are collected by security cameras and mobile phones, meaning that they are usually low resolution and unevenly illuminated. Therefore, models trained with semantic segmentation datasets will experience widespread feature loss on violent behavior datasets. To overcome these problems, we will need to annotate video frames containing violence to supplement the semantic segmentation dataset, enabling the model to better adapt to violent scenes.

2.2. Semantic Segmentation Algorithms

Traditional Gaussian [27] and KNN [28] approaches are suitable for static surveillance cameras, but not suitable for video with moving angles. Compared with traditional methods, deep-learning-based semantic segmentation models are better able to adapt to complex scenes. SegNet [13] aims to solve image semantic segmentation for autonomous driving or robotics, and is suitable for road scenes. Deeplab-V3plus [14] can capture multiscale information and recover the edge information of objects, producing an excellent semantic segmentation effect when the targets in the image are of different sizes. Background Matting V2 [29] proposes a real-time, high-resolution background-replacement technique that operates at 30 fps in 4K resolution. Therefore, we need to choose an image segmentation algorithm that is suitable for violent scenes.

2.3. Keyframe Extraction Methods

The detection of keyframes can reduce the number of redundant frames processed by the system, thereby improving the system’s efficiency. Keyframe extraction methods include traditional methods [30,31,32] and deep-learning methods [33,34,35]. Traditional methods are fast and require no training, while deep-learning methods can more accurately extract keyframes. Compared with traditional methods, deep-learning methods have high hardware requirements and need the support of large amounts of data. Because the judgment of violent events does not depend on specific features, traditional methods can be used to extract keyframes in our approach.

2.4. Violence-Detection Models

Depending on the deep-learning-network structure employed, the main violence-detection models include CNN + LSTM, VGG + LSTM, 3D convolutional networks (3DCNN), two-stream convolutional networks, and human-pose estimation. These models mainly use CNN, 3DCNN, and LSTM to extract spatiotemporal features based on image or optical flow data. Therefore, they can be impacted by various interference factors, such as camera movement, occlusion, complex scenes, etc. Specifically, CNN-based models will struggle with interference from the background and occlusion. Moreover, the background and camera movement will greatly impact 3DCNN-based and LSTM-based models, making the currently available datasets unsuitable for model training. Although the two-stream convolutional network adds optical flow data to make the model pay attention to the moving human body, it will be impacted by the interference of camera movement and illumination variation. The recognition rate of human-motion estimation will decrease when the number of people increases and the body posture is blocked. Finally, adding attention, motion detection, and other mechanisms to the model will cause the model to pay attention to behavioral features, but will reduce the model’s speed.

3. Proposed Method

We propose a violence-detection system that combines keyframe extraction, image classification, and probability. In addition, we use segmented–original image pairs to improve the recognition rate of the deep-learning models.

3.1. The Overall Framework

Our model incorporates keyframe extraction, image classification, and violent-behavior recognition (Figure 1). The keyframe extraction algorithm uses the frame difference method to calculate the local maximum value of the sliding window (continuous N frames) and selects the local maximum frame as the keyframe in every continuous K frames. Compared with the current mainstream violence recognition models, this design reduces the number of processed frames to 1/K of the total video frames. ResNet18 is used as a keyframe-feature-recognition network, which outputs the probability that a given keyframe contains violent behavior. Compared with 3DCNN-based and VGG-based sorts, our model parameters are smaller. This approach reduces the hardware requirements of the system. Furthermore, our model is easier to train compared to 3DCNN-based and LSTM-based sorts. The violent-behavior-recognition module determines the video category with reference to the number of keyframes belonging to the category of frames containing violence. We control the recognition efficiency of the system by setting the number of keyframes and control the recognition sensitivity by setting the threshold. Therefore, our model is robust and can be flexibly adjusted. In applications, the threshold and processing efficiency can be selected according to the requirements of the actual situation.

3.2. Dataset

We use the frame difference method to extract the keyframes of the RLVS, RWF-2000, and Hockey Fights datasets, and use Labelme to annotate 5000 human poses in order to train a semantic segmentation model (Figure 2). In our experiments, we found that the labeling of the data can significantly impact the model. If the fight participants are accurately labeled, the model will split the human figures; moreover, if a dense group of people is labeled as a single person, the model will not effectively segment the crowd. Therefore, we label the individual fight participants as a human pose, label the dense group of people as a human body, and also label them separately. These annotated data are combined with the MS COCO dataset to train a semantic segmentation model. We set up a dataset to test whether the model is suitable for scenes of violence. Based on the segmentation effect (Figure 3), we select the Deeplab-V3plus as the segmentation model. The establishment of our dataset is shown in Figure 4. The segmented images need to establish a mutual index relationship with the original images so that the background-removed and original pair can be loaded during training (Figure 5), and the final set of images includes 100,000 pictures with backgrounds removed.

3.3. Keyframe Extraction

While deep-learning methods can accurately extract keyframes, they require high-performance hardware and reduce the system’s processing speed. The characteristics of violent video frames are apparent; thus, traditional methods can be used to extract keyframes. The available traditional methods have roughly equivalent probabilities of extracting frames containing violence, with all of them able to reach or exceed a success rate of 98.90%. In practical applications, the frame difference method can better control the number of extracted keyframes. Therefore, we propose to use this method in our approach, as it can reduce the number of false-positive (non-violent) frames extracted by controlling the length of the window. For example, some nonviolent behaviors (such as hugging, shaking hands, etc.) may briefly appear similar to violent behaviors, and an appropriate window length can reduce the chances of these being flagged as violent. Moreover, because violent events are uncertain in the real world, keyframes can also be extracted at fixed intervals.

3.4. Our Training Method

Current violence-detection systems mainly extract the spatiotemporal features of the video, leading to the accumulation of errors in consecutive frames. For example, the 3DCNN-based method extracts 16 consecutive frames as input, and this method will bring many uncertainties. The movement of the camera and changes in illumination will generate more errors. In addition, these structures will increase the structural complexity of the model and increase the hardware requirements. The human pose in violent video frames is significantly different from that of normal video frames, and can thus be classified by the deep-learning model. This reduces the structural complexity of the model. We tested the recognition effect of ResNet18, ResNet34, ResNet101, DesNet121, VGG19, and GoogleNet on frames containing violence and proved that this is feasible. These models are trained by segmented–original image pairs. The segmented and original images guide each other, forcing the network to make consistent predictions for the original and segmented images. Specifically, if the model is robust to noise, the representation of an original image should also predict the target concerning its segmented version, and vice versa. This is equivalent to creating a feature space between the segmented image and the original image; theoretically, such a robust feature space can be found. We provide an interpretable theoretical proof below.

Proof.

$A_{n\times m}$ is the perfect feature space of the segmented images. $B_{n\times m}$ is the perfect feature space of the original images. $n$ is the number of images, and $m$ is the dimension of the image feature vector. $a_i$ is the feature representation of $segment\_imag{e}_i$, $b_i$ is the feature representation of $original\_imag{e}_i$, $l_i^{\prime }$ is the model prediction feature of $segment\_imag{e}_i$, and $l_i^{″}$ is the model prediction feature of $original\_imag{e}_i$:

$$A_{n\times m}={\left[a_1,a_2,a_3\cdots a_n\right]}^T,a_i=\left({\alpha }_{i1},{\alpha }_{i2,\cdots }{\alpha }_{im}\right),l_i^{\prime }=\left(x_{i1},x_{i2,\cdots }{x}_{im}\right)$$

$$B_{n\times m}={\left[b_1,b_2,b_3\cdots b_n\right]}^T,b_i=\left({\beta }_{i1},{\beta }_{i2,\cdots }{\beta }_{im}\right), l_i^{″}=\left(y_{i1},y_{i2,\cdots }{y}_{im}\right)$$

$$Loss={\displaystyle \sum }_{i=1}^n\Vert l_i^{\prime }-a_i{\Vert }^2+\Vert l_i^{″}-b_i{\Vert }^2$$

$$={\displaystyle \sum }_{i=1}^n{\displaystyle \sum }_{j=1}^m\left[{\left(x_{ij}-{\alpha }_{ij}\right)}^2+{\left(y_{ij}-{\beta }_{ij}\right)}^2\right]$$

When $\left[{\left(x_{ij}-{\alpha }_{ij}\right)}^2+{\left(y_{ij}-{\beta }_{ij}\right)}^2\right]$ is at its smallest, $Loss$ is also at its smallest. □

$$f\left(x_{ij},y_{ij}\right)={\left(x_{ij}-{\alpha }_{ij}\right)}^2+{\left(y_{ij}-{\beta }_{ij}\right)}^2$$

(1)

We define the feature representation difference between the original image and its background-removed image as the distance. The segmented image is the foreground information of the original image, so the distance between them is small. Therefore, we add Function (2). Constraint (3) is obtained from Function (2).

$$l_i^{\prime }-l_i^{″}=\Delta l_i,\Delta l_i=\left({\Delta }_{i1},{\Delta }_{i2,}{\Delta }_{i3,\cdots }{\Delta }_{im}\right), x_{ij}-y_{ij}={\Delta }_{ij}$$

(2)

$$g\left(x_{ij},y_{ij}\right)=y_{ij}-x_{ij}+{\Delta }_{ij}=0$$

(3)

We aim to find the minimum value of $f\left(x_{ij},y_{ij}\right)$ under Constraint (3). Then use the Lagrange multiplier to solve $L\left({\mathrm{x}}_{ij},{\mathrm{y}}_{ij}, \lambda \right)=f\left(x_{ij},y_{ij}\right)+\lambda g\left({\mathrm{x}}_{ij},{\mathrm{y}}_{ij}\right)$ to obtain the minimum point. Solve $L_x=\frac{\partial L}{\partial x_{ij}}=2x_{ij}-2{\alpha }_{ij}-\lambda =0, L_y=\frac{\partial L}{\partial y_{ij}}=2y_{ij}-2b_{ij}+\lambda =0, L_{\lambda }=\frac{\partial L}{\partial \lambda }={\mathrm{y}}_{ij}-{\mathrm{x}}_{ij}+{\Delta }_{ij}=0$ to obtain $x_{ij}={\alpha }_{ij}+\frac{1}{2}\left({\beta }_{ij}-{\alpha }_{ij}+{\Delta }_{ij}\right), y_{ij}=x_{ij}-{\Delta }_{ij}$. $L_{xx}=2, L_{yy}=2, L_{xy}=0$ proves that $x_{ij}={\alpha }_{ij}+\frac{1}{2}\left({\beta }_{ij}-{\alpha }_{ij}+{\Delta }_{ij}\right), y_{ij}=x_{ij}-{\Delta }_{ij}$ as the minimum value.

Segmented pictures are the foreground information of the original pictures, and they express the same information. When the model converges, the distance between $l_i^{\prime }$ and $l_i^{″}$ tends towards $0$.

$$\underset{{\Delta }_{ij}\to 0}{\lim }{x}_{ij}=\underset{{\Delta }_{ij}\to 0}{\lim }{y}_{ij}={\alpha }_{ij}+\frac{1}{2}\left({\beta }_{ij}-{\alpha }_{ij}\right)$$

The feature space learned by the model is $C_{n\times m}$. The features predicted by the model are between $A_{n\times m}$ and $B_{n\times m}$, which is equivalent to $C_{n\times m}=A_{n\times m}+\frac{1}{2}\left(B_{n\times m}-A_{n\times m}\right)$. In classification models, the loss consists of three parts (4): the loss of the segmented image and label ($loss\_seg$), the loss of the original image and label ($loss\_\mathrm{un}seg$), and the distance between the fully connected layers of the two images ($D$), while $\beta$ is a hyperparameter. As shown in Figure 1, in training, the loss of segmented–original images and labels is reduced, which is equivalent to finding the minimum value of Formula (1). At the same time, the model’s fully connected layer of the segmented–original image gradually approaches, which is equivalent to reducing the value of $\Delta l_i$ in Formula (2). The optimization problem of Formulas (1) and (2) can be effectively solved by means of a stochastic gradient descent and backpropagation. The segmented image and original image make the model create a feature space between the segmented image and the original image; theoretically, such a robust feature space can be found, thereby reducing the interference of background information.

$$Loss=los\_seg+loss\_unseg+\beta \ast D$$

(4)

3.5. Judgment of Violence

In social dynamics, a simple approach consists in substituting the continuous distribution over the micro-state by a discrete one so that each node of the discrete micro-scale variable represents the number of particles in a certain domain of the space of the microscopic states [36]. Therefore, we extract the key frames of the video as the basis for judging the behavior. We use the proportion of violent frames in keyframes to measure whether violence is present in the video, which reduces the error accumulation of the spatiotemporal features of consecutive frames. Here, $P$ is the recognition rate of the image classification model, $Keyframe\_no$ is the number of keyframes extracted from a violent video, $Threshold\_no$ is the threshold for judging that the video contains violent behavior, and $P_{Recognition}$ is the probability that the model recognizes violent behavior. We define the evaluation function of $P_{Recog}$. The smallest $\frac{Keyframe\_no}{Totalframe}$ means that the model processes the fewest frames, which can improve the model’s processing speed; the largest $\frac{Threshold\_no}{Keyframe\_no}$ means that the model has the best stability when identifying violent behaviors. Therefore, we can adjust the system according to the needs of the actual situation. Specifically, the extraction of keyframes can control the number of frames processed by the model. According to the evaluation Function (6), choosing $min\left(\frac{Keyframe\_no}{Totalframe}\right)$ means that the smallest number of frames can be handled. When the number of keyframes is determined (Figure 6 shows the recognition with 16 keyframes), we can adjust the system’s sensitivity by adjusting the threshold. When p = 0.9, if $P_{Recognition}$ exceeds 95%, we can select 8, 9, 10, 11, or 12 as the $Threshold\_no$; according to $\frac{Threshold\_no}{Keyframe\_no}$, $Threshol{d}_{no}=12$ produces the most reliable judgment.

$$P_{Recognition}={\displaystyle \sum }_{k>Threshold\_no}^{Keyframe\_no}{C}_{Keyframe\_no}^k{P}^k{\left(1-P\right)}^{Keyframe\_no-k}$$

$$Threshold\_no\in [\lceil \frac{Keyframe\_no}{2}\rceil ,keyframe\_no]$$

(5)

$$Evaluate\left(P_{Recog}\right)=\{\min \left(\frac{Keyframe\_no}{Totalframe}\right),\max \left(\frac{Threshold\_no}{Keyframe\_no}\right)\}$$

(6)

4. Experiments

4.1. Deeplab-V3plus

We combine the annotated violent frames with the MS COCO dataset to train Deeplab-V3plus, which includes 12,000 human poses. Hyper-parameter settings are shown in

Table 1. Hyperparameters for Deeplab-V3plus.

Hyperparameter
backbone	xception
downsample	16 or 8
input_shape	224 × 224
Freeze_Epoch	10
Freeze_lr	5.00 × 10−4
UnFreeze_Epoch	80
Unfreeze_lr	5.00 × 10−5

, and these settings enable better adaptation to violent scenes. The backbone network uses the ‘xception’ to adapt to the changes in human poses and complex backgrounds in frames containing violence. The downsampling value is 8, which enables better segmentation of small objects. The input image size is set to 224 × 224 to facilitate better handling of low-quality frames containing violence. The training of the model consists of a freezing stage and an unfreezing stage. In the freezing stage, the backbone network is frozen, and the feature extraction network does not change. In the unfreezing phase, the backbone network is not frozen, and the feature extraction network changes, which will take up a lot of video memory. We recommend training for 60 epochs (we used a pretrained model; see Figure 7). The model achieves performance of 90.1% and 86.20% on the training and testing sets, respectively.

4.2. Facilitation of Model Learning by Segmented–Original Images

To explore the effect of segmentation on the distribution of the dataset features, we visualized the dimensionality-reduced segmented dataset and the original dataset. We set up two control groups. The first group contains 10,000 violent pictures and 10,000 non-violent pictures randomly selected from each original dataset. The second set contains 10,000 violent images and 10,000 non-violent images, which are the background-removed images of the first set. We used t-SNE to reduce the dimensionality of these pictures and then visualized them. In the violent behavior dataset, as shown in Figure 8b,d, the two sets of dimensionality-reduction results prove that segmenting the human pose can make the feature space of the dataset more aggregated.

We designed six comparative experiments (

Table 2. Comparative experiment grouping.

No.	Training Dataset	Test Dataset
1	Original Image	Original Image
2	Original Image	Segmented Image
3	Segmented Image	Segmented Image
4	Segmented Image	Original Image
5	Segmented + Original Image	Original Image
6	Segmented + Original Image	Segmented Image

) to verify the promotion effect of the segmented–original image pair on the model. We performed experiments on VGG19, GoogleNet, ResNet18, and DesNet121. On ResNet18 and DesNet121, we adopt two methods: let the fully connected layer features of the segmented–original image pair be close to each other (the hyperparameter$\beta$ is $0.001$); directly use the segmented–original image pair for training. Experiments show that both methods are effective. For VGG19 and GoogleNet, we adopt the second methods. Figure 9 shows the change in the recognition rate for each group during the training process. The significant difference between the recognition rates of the first and second groups indicates that the model trained on the original image has not learned the essential features, which is mainly due to the interference of background information. The significant difference between the recognition rates of the third and fourth groups indicates that the model trained on the segmented images has not learned the essential features, which is mainly because the model trained on pure data is not robust. Moreover, the recognition rate of the third group is higher than that of the fourth group, indicating that segmented images can reduce the influence of the background on the model. The fifth and sixth groups have the highest recognition rates for both original and segmented images, indicating that the segmented–original pairs promote the model’s learning of image features without increasing the model complexity. In summary, the segmented–original image pair aids the model in learning essential features, thereby improving the recognition rate (

Table 3. The effect of our method on different models.

	RLVS		Hockey Fights
	Original Image	Segmented Image	Original Image	Segmented Image
VGG19	85.70%	70.79%	97.62%	90.38%
Ours (VGG19)	88.34%	89.34%	97.03%	96.52%
ResNet18	86.59%	81.98%	98.59%	91.81%
Ours (ResNet18)	91.39%	91.78%	98.27%	96.35%
DesNet121	87.96%	83.41%	98.60%	91.13%
Ours (DesNet121)	93.01%	92.34%	98.40%	97.58%
GoogLeNet	90.49%	84.42%	98.48%	91.23%
Ours (GoogLeNet)	92.80%	93.45%	98.24%	97.02%

).

We design an encoder–decoder network to visualize the interaction between the segmented image and the original image (

Table 4. Implementation of the model.

# Define the reconstruction loss function def recon_loss(recon_x,x): loss = torch.sum((recon_seg—recon_unseg)**2).mean()   return loss # Define segmented-original image pair hidden layer distance def Distance(laten_seg,laten_unseg): D = torch.sqrt((laten_seg—laten_unseg)**2).mean()   Return D #model training for seg, unseg in emulate (trainloader): laten_seg = Encoder(seg) recon_seg = Decoder(laten_seg) #compute the segmented image reconstruction loss recon_loss_seg = recon_loss(seg, recon_seg)    laten_unseg = Encoder(unseg)    recon_unseg = Decoder(laten_unseg)    #compute the original image reconstruction loss recon_loss_unseg = recon_loss(unseg, recon_unseg)    #compute the distance of segmented-original image pair in the hidden layer Distance = Distance(laten_seg, laten_unseg) Loss = recon_loss_seg + recon_unseg_loss+ 0.001*Distance    Loss.backward()    optimizer()

). The loss consists of three parts (7): the reconstruction loss of the segmented image, the reconstruction loss of the original image, and the distance between the hidden layers of the two images, while $\beta$ is a hyperparameter. The reconstruction loss of the segmented and original image is reduced, which is equivalent to finding the minimum value of Formula (1). At the same time, the distance between the hidden layers of the two images gradually approaches convergence, which is equivalent to reducing the value of $\Delta l_i$ in Formula (2). The optimization problem of Formulas (1) and (2) can be effectively solved by means of stochastic gradient descent and backpropagation. Although this design increases the reconstruction error, it also forces the model to pay attention to the foreground information, thereby decoupling the reconstruction from the background information (Figure 10). Our experiments show that the segmented image and original image make the model create a feature space between the segmented image and the original image; theoretically, such a robust feature space can be found, thereby reducing the interference of background information.

$$Loss=recon\_loss\_seg+recon\_loss\_\mathrm{un}seg+\beta \times D$$

(7)

4.3. Keyframes

We counted the data distribution of frames in violent videos and found that the number of video frames varies greatly between datasets (

Table 5. Statistics on the number of video frames in the violence dataset.

Datasets	Average	Median	Var	Std	Min	Max
HockeyFights	41.056	41	0.4528	0.6733	40	49
RLVS	143.69	132	84,490	290.74	29	11,272
RWF-2000	150	150	0	0	150	150
CCTV	8797.55	1295	1,301,301,216.97	36,091.60	95	472,304

). Moreover, the data distributions of different violent videos are different (Figure 11); as a result, the important violent frames (such as Figure 11a) cannot be extracted from consecutive frames. Therefore, using a fixed number of frames is an unreliable approach to identifying violent behavior. We instead recommend dynamically adjusting the number of frames to be identified based on the total number of frames in the video. We select keyframes in a given window length that are most different from other frames. This allows our system to control the number of keyframes. Therefore, the number of keyframes can be used to control the recognition efficiency and robustness of the system. Specifically, a sliding window is established containing $N$ frames, and the local maximum frame in every continuous $K$ frames is selected as the keyframe. In this way, $N/K$ keyframes for judgment can be obtained. The setting of$N$ can prevent short-duration behaviors that are superficially similar to violent behaviors (such as friendly play-fighting hugs, and handshakes) from causing false alarms. Moreover, $K$ can make uniform sampling, which can prevent camera movement and lighting changes from causing many redundant frames being extracted. In summary, extracting the keyframe can more effectively accumulate the recognition results of a single frame and improve the overall recognition accuracy.

4.4. Our Violent-Behavior-Recognition System

Our proposed training method enables ResNet18, ResNet34, ResNet101, DesNet121, and GoogleNet to recognize frames containing violence over 90%. This shows that our method is applicable to almost all convolutional networks. We choose ResNet18 as the recognition model because of its low hardware requirements. The training and test set ratios on the RLVS and Hockey Fights datasets are 0.8 and 0.2, respectively. On the Violent Flow dataset, the ratios of the training set and test set are 0.9 and 0.1, respectively. On Violent Flow dataset, it is difficult for the semantic segmentation algorithm to obtain enough segmented images, so we only use the original images for training. ResNet18 achieved a recognition rate of more than 91.39% (

Table 6. Video frame recognition rate.

	Average	Violence	Nonviolence
RLVS	91.39%	87.88%	94.90%
Hockey Fights	98.59%	99.13%	98.05%
Violent Flow	92.06%	87.12%	98.73%

) on three datasets and can theoretically achieve a recognition rate of more than 95% for violent behaviors (Figure 6). Setting the value of $K$ can control the number of violent frames extracted. When $K$ is fixed, the threshold setting of the system can affect the recognition rate (

Table 7. The relationship between recognition rate and threshold.

	Threshold = 0.5			Threshold = 0.7
	A	V	N	A	V	N
RLVS	94.6%	93.2%	96.00%	91.50%	89.50%	93.5%
Hockey	98.5%	98.0%	99.0%	97.5%	96.0%	99.0%
Violent Flow	95.0%	90.0%	100%	93.75%	87.50%	100%

A: average, V: violence, N: nonviolence, Window Length = 5.

). Changes to $K$ and the threshold will affect the recognition rate of the system (

Table 8. The relationship between window length and recognition rate.

	$\mathit{K}$	RLVS	Hockey Fights	Violent Flow
Threshold = 0.5	5	94.6%	98.50%	95.0%
	10	96.75%	97.5%	95.0%
	15	96.25%	94.0%	95.0%
	20	96.0%	92.0%	95.0%
Threshold = 0.7	5	91.50%	91.5%	91.5%
	10	93.0%	98.0%	93.5%
	15	93.75%	95.5%	95.0%
	20	92.5%	91.5%	93.5%

). The recognition rate of our system reaches state-of-the-art levels on RLVS and Violent Flow and is close to the state of the art on Hockey Fights (

Table 9. Comparisons between the proposed method and others on previous datasets.

Method	Hockey Fights	RLVS	Violent Flow
FightNet [10]	97.00%	–	–
3D CNN [37]	99.62%	–	94.30%
CNN + LSTM [3]	97.10%	–	94.57%
C3D [6]	96.50%	–	84.44%
I3D [38]	97.50%	–	86.89%
VGG + LSTM [17]	95.10%	88.20%	90.01%
Ours	98.50%	94.60%	95.00%

). Compared with other models that extract spatiotemporal features, our approach has a more straightforward structure, making it easier to train and reducing the hardware performance requirements.

5. Conclusions

We combine keyframes and image classification to implement a robust and adjustable violent behavior recognition system. The number of keyframes can control the recognition efficiency of the model, while probabilistic thresholds can control the robustness and sensitivity of the system. In addition, we propose a new training method to improve the model’s recognition rate. Our system performance can surpass or approach that of current violent behavior recognition systems without increasing the complexity of the model structure.