(2) Magnitude Reactivity Map Generation: *R*mag

The frame was considered that it contains suspicious behavior when a frame with a standard deviation of the motion vector at the time (t) is greater than 1.2, which was calculated experimentally. The region where the magnitude of the vector is calculated by optical flow at the point (x, y) is larger than the summation of the average, and the variance of the whole image was considered to be the suspicious behavior region. Since the movement is slightly different for each person even when performing the same ordinary behavior, the region was detected based on the summation of the average and the variation. Based on these facts, the magnitude reactivity map (*R*mag) is calculated as follows.

$$\begin{aligned} R\_{\text{mag}} &= F\_{\text{mag}}(\mathbf{x}, y, t) \times \sigma(F\_{\text{mag}}(\mathbf{x}, y, t)) \\ \text{where, } \sigma(F\_{\text{mag}}(\mathbf{x}, y, t)) &> 1.2 \text{ and } F\_{\text{mag}}(\mathbf{x}, y, t) > \mu \{ F\_{\text{mag}}(\mathbf{x}, y, t) \} + \sigma(F\_{\text{mag}}(\mathbf{x}, y, t)) \end{aligned} \tag{2}$$

Figure 5 shows the magnitude reactivity map generated by the proposed method. Figure 5a shows the video of people gathered at the center of the park moving in various directions simultaneously with any signal. In this video, the suspicious behavior is the sudden movement of people. Figure 5b shows the magnitude feature map in polar coordinates, and with this feature map, we can see the moving area of people. Figure 5c shows the generated final magnitude reactivity map. We can see that the area in which people are running away has been properly detected. Figure 5d shows the detection of anomalous regions based on the magnitude reactivity map. In the scene where several people are running away, we can see that the motion vector is greatly increased, and all of the suspicious behavior is detected.

**Figure 5.** Example result of calculating *R*mag. (**a**) original frame; (**b**) *F*mag; (**c**) *R*mag; (**d**) result of the system.

(3) Gradient Reactivity Map Generation: *R*grad

Unlike a group of people who are regularly moving in the same direction, if there is an object moving in the opposite direction, this can be considered suspicious behavior. To detect this behavior, a reactivity map for the gradient feature of the motion vector was generated. First, the motion vector calculated for each pixel is divided into object units to prevent the movement direction of the object and the part included in the object from being different directions. For example, when a man moves to the left with his arms and legs shaking up and down, the main movement direction may be misjudged because of the movement direction of the arms and legs, although the main direction of the man's movement is left direction. Based on these, the gradient reactivity map (*R*grad) is calculated as follows.

$$\begin{aligned} R\_{\text{grnd}} &= \text{grad}(\mathbf{x}, \mathbf{y}, t) + \frac{\text{grad}(\mathbf{x}, \mathbf{y}, t)}{180} \\ \text{where, if } \left| \mu \{ F\_{\text{grnd}}(\mathbf{x}, \mathbf{y}, t) \} - F\_{\text{grnd}}(\mathbf{x}, \mathbf{y}, t) \right| &\leq 180 \\ \text{grad}(\mathbf{x}, \mathbf{y}, t) &= \left| \mu \{ F\_{\text{grnd}}(\mathbf{x}, \mathbf{y}, t) \right\} - F\_{\text{grnd}}(\mathbf{x}, \mathbf{y}, t) \right| \\ \text{clse} \\ \text{grad}(\mathbf{x}, \mathbf{y}, t) &= \left| \mu \{ F\_{\text{grnd}}(\mathbf{x}, \mathbf{y}, t) \} - F\_{\text{grnd}}(\mathbf{x}, \mathbf{y}, t) \right| - 180 \end{aligned} \tag{3}$$

The region of the object that moves differently from the average direction of movement becomes a component of the gradient reactivity map.

Figure 6 shows the gradient reactivity map generated by the proposed method. Figure 6a shows the original video of the people walking around the park, and Figure 6b shows the gradient feature map of the motion vector. The circle in the middle represents the average of the angles in the entire image and is displayed on the screen to show the angle of 181◦. The rectangle drawn on the right is the area detected by applying Equation (3). Figure 6c is the final generated gradient reactivity map. In the reactivity map, we can see that when the average direction of the people is to the left, weights are added to the object moving in the opposite direction, and the system has responded to it very strongly.

**Figure 6.** Example result of calculating *R*grad. (**a**) original frame; (**b**) *F*grad; (**c**) *R*grad.

The two reactivity maps described above are incorporated into the temporal saliency map through weighted combinations. Among the feature values constituting the temporal saliency map, a region having a high value is a region that includes noticeable suspicious behavior. Therefore, the presence or absence of suspicious behavior can be determined through the temporal saliency map. The two reactivity maps are combined as follows.

$$S(t) = \alpha \times R\_{\text{mag}} + (1 - \alpha) \times R\_{\text{grad}} \tag{4}$$

The weight value α is applied proportionally to the maximum value of the magnitude. In general, the anomalous behaviors to detect are always increased in the magnitude of the motion, but the moving direction is partially applied, so the weight value is calculated proportionally to the magnitude value. In the final generated temporal saliency map, neighboring pixels are clustered, and the area of 30 pixels or more is displayed as the final detection area.

Figure 7 shows the final detected suspicious behavior region using the temporal saliency map. Figure 7a shows the experimental result on a video in which a man is walking to the right and a child is jumping to the left. Figure 7c,d show the reactivity map for the magnitude and gradient of the motion vector, respectively, and Figure 7e shows the temporal saliency map finally generated through weighted combination. At the moment the child jumps, the motion increases greatly, and the reactivity to the magnitude of the motion increases significantly. Additionally, the reactivity to the direction also

increased because everyone else moves to the right while the child jumps to the left. Figure 7b shows the final result of the proposed method. Two reactivity maps were combined and finally, a temporal saliency map was generated to detect the suspicious behavior regions.

**Figure 7.** Example result of calculating temporal saliency map. (**a**) original frame; (**b**) result of the system; (**c**) *R*mag; (**d**) *R*grad; (**e**) temporal saliency map (*TS*).

#### **3. Results and Discussion**

In order to verify that the proposed method detects suspicious behavior region correctly, the experiments were conducted on 10 different types of video sequences mentioned in Section 2.1. In addition, to carry out a quantitative evaluation, the proposed method was compared with the state-of-art methods with the experiments, which were conducted on two different publicly available datasets, namely UMN and Avenue. Although some of the compared methods perform evaluations on videos that are gathered from the Internet, these videos are not available online for comparison. Therefore, comparison evaluations were conducted on the UMN and Avenue datasets, which are publicly available.
