Pedestrian Localization in a Video Sequence Using Motion Detection and Active Shape Models

Abstract

There is increasing interest in video object detection for many situations, such as industrial processes, surveillance systems, and nature exploration. In this work, we were concerned with the detection of pedestrians in video sequences. The aim was to deal with issues associated with the background, scale, contrast, or resolution of the video frames, which cause inaccurate detection of pedestrians. The proposed method was based on the combination of two techniques: motion detection by background subtraction (MDBS) and active shape models (ASM). The MDBS technique aids in the identification of a moving region of interest in the video sequence, which potentially includes a pedestrian; then, the ASM algorithm actively finds and adjusts the silhouette of the pedestrian. We tested the proposed MDBS + ASM method with video sequences from open repositories, and the results were favorable in scenes where pedestrians were in a well-illuminated environment. The mean fit error was up to 4.5 pixels. In contrast, in scenes where reflections, occlusions, or pronounced movement are present, the identification was slightly affected; the mean fit error was 8.3 pixels in the worst case. The main contribution of this work was exploring the potential of the combination of MDBS and ASM for performance improvements in the contour-based detection of a moving pedestrian walking in a controlled environment. We present a straightforward method based on classical algorithms which have been proven effective for pedestrian detection. In addition, since we were looking for a practical process that could work in real-time applications (for example, closed-circuit television video or surveillance systems), we established our approach with simple techniques.

1. Introduction

Pedestrian detection is an essential field of study for video surveillance or similar applications [1]. It consists of detecting mobile subjects in a 2D digital image or video sequence. Some factors, such as variability in the appearance, clothing, occlusions, lighting levels, background scene, and weather conditions, represent challenges for accurate detection [2]. Furthermore, the detection of pedestrians with video surveillance cameras requires methods to capture their different walking postures, which vary throughout the video sequence [3]. Currently, pedestrian detection has been carried out through different algorithms—for example, deformable techniques such as statistical shape models (SSM) active contour models (ACM), and active appearance models (AAM) [4]. A drawback of these methods is the capturing of the texture, in addition to the variations in the shape [5].

In this work, pedestrian detection is investigated by combining two techniques: motion detection using background subtraction (MDBS), which identifies the region of interest where the possible pedestrian is located, and active shape models (ASM), which is a deformable technique that detects the contour of the pedestrian. MDBS has been successfully tested in the detection of vehicles [6]; it is also characterized by its low computational cost, which makes it appropriate in this context. ASM is a natural continuation of the research reported in reference [7]. Our approach seeks to detect the contour of a person who appears on the scene despite being seen in different backgrounds. The detection process begins with the automatic calculation of the necessary translation parameters obtained with the MDBS technique; later, ASM adjusts the silhouette of the pedestrian.

The main contribution of this work is combining MDBS and ASM to explore the potential improvement provided in detecting the contour of a moving pedestrian walking in a controlled environment. In summary, the MDBS technique finds the movement region through a rectangle that automatically provides the translation parameters; later, these parameters are calculated as coordinates used by one of the stages of the ASM algorithm to find and fit the contour. In this sense, this work shows a different technique concerning the state-of-the-art on how to get the parameters that ASM uses to locate the contour. Moreover, the results obtained in this work are similar to those obtained in references [8,9]; however, MDBS has not been used together with the ASM algorithm, especially in the adaptive form of adjustment in each frame $t+1$ through the updated coordinates provided by MDBS.

2. Related Work

Background subtraction is one of the main methods to address the problem of motion detection [10]. It provides separation between the background and the objects of interest, but it is sensitive to dynamic changes due to lighting and occlusions. There are different techniques to perform motion detection by background subtraction (basic background modeling, statistical background modeling, among others [11]). Khalifa et al. [12] proposed a method that looks for a dynamic background region to reduce false positives in the video obtained by a CCTV (closed-circuit television) camera. This method was evaluated with the CDnet 2012/2014 datasets (CDNET, http://jacarini.dinf.usherbrooke.ca, accessed on 8 January 2020). Average precisions of 86.50% and 76.68% were obtained on the CDnet2012 and CDnet2014 databases, respectively. Camplani and Salgado [13] used a combination of classifiers that improves background subtraction, reducing false detections. Although the image had variability in color, shadows, and lighting, an average precision of 70% was obtained in their work. Ramya and R. Rajeswar [14] presented a new technique that improves the frame difference method by classifying at the pixel level the first frames as background and their planes with a correlation coefficient. They compared their results with different original images. The best precision was 0.9984 in a camouflaged image. Sehairi et al. [15] presented 12 techniques based on motion detection to find the object of interest. They used the CDnet2014 dataset, obtaining the best result with the GMM (Gaussian mixture model) technique with a specificity of 0.99593.

About ASM, a wide variety of literature related to the adjustment of forms in organs and parts of the human body, such as hands or faces, is cited [16,17,18]. Esfadianrkhani and Hossein [19] mention the construction of a point distribution model (PDM) where the silhouette of the human and its anatomical joints can be combined to build the ASM model, and subsequently be able to segment the modeled body as new images. The use of deformable methods in pedestrian detection is limited because ASM is considered to require great time consumption due to the computational load [20]. For this reason, most of the investigations are based on the detection of static people (standing pose and standing facing the camera). Although it should be mentioned how walking people are detected with ASM, this is done with gait recognition techniques [21].

Baumberg and Hogg [22] made a detector that works for poses and views, even with two or three rigid pedestrians. Pedestrian detection is performed by means of an ellipsoidal silhouette trained with 40 characteristic points of the image (landmarks) using the Kalman filter. Koshan et al. [23] actively applied a model to detect pedestrians in a video sequence using a different number of landmarks (10, 14, 21, and 42) and received three lineups of silhouettes of a single person. The contour detection was performed, and the normalized error between the actual ground landmarks and the estimated landmarks was calculated: the mean error in the distance between such points was 2.74 pixels with 21 landmarks, and it was 2.42 pixels with 42 landmarks.

J. Changhyuk and J. Keechul [24] proposed a model based on an exoskeleton to detect human poses. They used background subtraction and a pattern matching algorithm in the fitting process. The accuracy in three poses resulted in 97.32%, 93.3%, and 80.62%. Kim et al. [25] identified the way pedestrians walk. Pedestrians were detected with ASM, using 32 landmarks for the silhouette of a human from images of 720 × 480 pixels, reaching up to 90% accuracy. Lee and Choi [26] detected pedestrians using active models using infrared (IR) images and visible-light cameras that allow pedestrians to be followed in degraded and low-light environments. Ma and Ren [27], proposed a model-based method to recognize pedestrians; however, they did not specify the number of landmarks with which to train the model. Images with a resolution of 320 × 240 pixels were obtained from an omnidirectional RGB (red-green-blue) camera. A 94% in success rate in detecting pedestrians was achieved. Ide [28] proposed a technique to segment pedestrians using a model trained with silhouettes identified with the grab-cut tool. This method compares segmentation results of grab-cut against the results of the model, and error rates of 2.32% and 2.15% with the foreground (FG) and background (BG) models, respectively, were reported.

3. Experimental Set-Up

This section presents the experimental methodology used in this work. The process is summarized in four stages: (a) Input video, from which the video sequences to be studied were selected. (b) Detection of movement. In this step, the proposed model (MDBS) for motion detection by background subtraction was applied. (c) Adjustment of the model (using ASM). Finally, (d) evaluation of the effectiveness of the proposal.

3.1. Experimental Data

In the first stage of the experimental process, two datasets of video sequences commonly used in pedestrian detection were selected: CDnet2014 [29] and CASIA Gait dataset [30,31]. CDnet2014 contains six recording categories: baseline, dynamic background, camera jitter, shadow, intermittent object motion, and thermal. CASIA Gait is a database used to investigate the walking patterns of various pedestrians.

The CDnet2014 and CASIA Gait datasets contain scenes where videotaped subjects appear. They present three critical problems: angle of vision, clothing, and changes in conditions when walking; thus, their use is appropriate for the objectives of this study because they involve situations where there are underlying problems and allow the evaluation of the effectiveness of the proposed model. In this sense, only scenes with situations where pedestrians present problems with background variations were selected.

Three video sequences with pedestrians were chosen from the CDnet2014 database: office, PETS2006, and sofa. The first two video recordings belong to the baseline category, and the third belongs to the intermittent object motion category (see Figure 1). The three selected video sequences (Figure 1), contain changes, mainly lighting and background, were recorded a speed of 0.17 frames per second, and are 360 × 240 pixels in the cases of office scenes and sofa, whereas the PETS2006 scene has a resolution of 720 × 576 pixels. The CASIA Gait dataset contains video sequences where encoded videos are stored (320 × 240 pixels). They have information on 124 pedestrians walking in different directions. For experimentation, four scenes of pedestrians walking in four directions (0${}^{\circ }$, 36${}^{\circ }$, 54${}^{\circ }$, and 90${}^{\circ }$) were selected (see Figure 2).

It is essential to notice that our approach converts color frames from the selected video sequences into grayscale images to perform a unique solution to process color frames. Thus, it allows the processing of not only color frames but also frames from closed circuit television video (CCTV) or video surveillance systems.

3.2. Motion Detection by Background Subtraction (MDBS)

In this stage, the challenge is to detect regions of movement. The motion detection with background subtraction technique is proposed to separate the object of interest (foreground) from the background. Algorithm 1 shows the main steps for the detection of motion regions. The detailed sequence to perform the detection of the movement is as follows:

Algorithm 1 The basic sequence of the detection of motion with the MDBS technique.

1: Convert the first frame I to grayscale and set it as the background F 2: while remaining frames in the video sequence do 3:     Read next frame I and convert it to grayscale 4:     $B(x,y)\leftarrow I(x,y)-F(x,y)$ 5:     Binarize B given the threshold $Th$ 6:     Erode B 7:     Eliminate residuals in B with 4 neighbors 8:     Fill holes in B 9:     Calculate regression line $y^{\prime }=a_0+a_{1}x$                        ▹ See Equations (3)–(5) 10:     Calculate Principal Components                                    ▹ See Equations (8) and (10) 11:     Delimit the region of movement 12: end while

• Convert the first frame (I) of the video sequence to grayscale and set it as the background frame (F).
• Read the next frame in the video sequence $\left(I\right)$ and convert it to grayscale.
• Subtract the background F from the image with Equation (1) for every pixel $(x,y)$ in the image.

  $$B(x,y)=I(x,y)-F(x,y)$$

  (1)
• Binarize B, considering a threshold $Th=61$. Values greater than $Th$ are normalized to 1, and values less than $Th$ are normalized to 0; consequently, the possible pedestrian contour is segmented. $Th$ is computed by Otsu’s method [32]. This considered one of the best and most stable methods for choosing threshold values automatically [33]. The simplicity of procedure characterizes Otsu’s method, and it is a non-parametric, valuable technique because of its unsupervised nature for threshold selection. It minimizes the intraclass variance of black and white pixels to obtain a optimal threshold ($Th$), following a sequential search and using the simple cumulative quantities as follows:
  Consider that $G=[0,L-1]$ is the range of the grayscale of an image $f(x,y)$ and $P_i$ is the probability of i grayscale level. The threshold value t splits the image into two classes $C_0=[0,t]$ and $C_1[t+1,L-1]$, with probabilities of ${\alpha }_0={\sum }_{i=0}^t{P}_i$ and ${\alpha }_1=1-{\alpha }_0$, respectively. The means $\left(\mu \right)$ of the gray values to $C_0$ and $C_1$ are ${\mu }_0={\sum }_{i=0}^{t}i{P}_i/{\alpha }_0={\mu }_t/{\alpha }_0$ and ${\mu }_1={\sum }_{i=t+1}^{L-1}i{P}_i/{\alpha }_1=(\mu -{\mu }_t)/(1-{\alpha }_1)$, respectively, where $\mu ={\sum }_{i=0}^{L-1}i{P}_i$ and ${\mu }_t={\sum }_{i=0}^{t}i{P}_i$. Thus, the criterion function is defined in Equation (2) as the variance between two classes ($C_0$ and $C_1$), where the optimal threshold $Th$ is the t value that maximizes $\eta$.

  $${\eta }^2\left(t\right)={\alpha }_0{({\mu }_0-\mu )}^2+{\alpha }_1{({\mu }_1-\mu )}^2={\alpha }_0{\alpha }_1{({\mu }_0-{\mu }_1)}^2.$$

  (2)
  In this case, the calculation of $Th$ was carried out under controlled conditions in environments with artificial light.
• Erode B, using a circular structured element E of radius $R=5$, which helps to round the contour. R = 5 was obtained through a trial-and-error strategy (a greedy solution). This result is a tread-off between the frame resolution and the person’s scale into the frame.
• Eliminate residuals in B applying the 4-neighbor technique, using the set of isolated pixels of the person’s silhouette as the residual.
• Fill gaps in B, which contain 0 values inside the outline of the person’s silhouette.
• Calculate the equation of the regression line, which is used to delimit the region of interest where the moving object was detected (Equation (3)).

  $$y^{\prime }=a_0+a_{1}x,$$

  (3)

  where $y^{\prime }$ is the estimated ordinate of the n points $(x,y)$, which correspond to the pixels in B whose values are 1; $a_0$ and $a_1$ are constants that get the values of the normal equations of the least-squares line, which in essence form a system of two equations whose solutions are defined by Equations (4) and (5).

  $$\sum _{i=1}^n{y}_i=n{a}_0+\sum _{i=1}^n{a}_1{x}_i,$$

  (4)

  $$\sum _{i=1}^n{x}_i{y}_i=a_0\sum _{i=1}^n{x}_i+\sum _{i=1}^n{a}_1{x}_i^2,$$

  (5)
• Calculate the principal components considering $\mathbf{x}$ (of dimension n), which represents the M points with coordinates $(x,y)$ corresponding to the pixels in B, whose value is 1. The mean value ${\mathbf{m}}_{\mathbf{x}}$ is given by Equation (6).

  $${\mathbf{m}}_{\mathbf{x}}=\frac{1}{M}\sum _{i=1}^M{\mathbf{x}}_i.$$

  (6)
  Similarly, the covariance matrix ${\mathbf{C}}_{\mathbf{x}}$, of size $n\times n$, can be approximated by Equation (7).

  $${\mathbf{C}}_{\mathbf{x}}=(\mathbf{x}-{\mathbf{m}}_{\mathbf{x}}){(\mathbf{x}-{\mathbf{m}}_{\mathbf{x}})}^T.$$

  (7)
  The principal component transform (also called Hotelling transform [34]), is obtained by Equation (8).

  $$\mathbf{y}=\mathbf{A}(\mathbf{x}-{\mathbf{m}}_{\mathbf{x}}),$$

  (8)

  where A is the transformation matrix that applies $\mathbf{x}$ on $\mathbf{y}$, and the mean of $\mathbf{y}$ is ${\mathbf{m}}_{\mathbf{y}}=0$. Rows in the matrix $\mathbf{A}$ are eigenvectors of ${\mathbf{C}}_{\mathbf{x}}$, sorted from largest to smallest eigenvalues. The covariance matrix of $\mathbf{y}$ is obtained by $\mathbf{A}$ and ${\mathbf{C}}_{\mathbf{x}}$ as follows:

  $${\mathbf{C}}_{\mathbf{y}}=\mathbf{A}{\mathbf{C}}_{\mathbf{x}}{\mathbf{A}}^T.$$

  (9)
  Therefore, the values of $\mathbf{x}$ can be obtained through the inverse transformation in Equation (10).

  $$\mathbf{x}={\mathbf{A}}^T(\mathbf{y}+{\mathbf{m}}_{\mathbf{x}})$$

  (10)
• Finally, to delimit the region of movement (see Figure 3), estimation of the extreme points is performed: $P1(x_1,y_1)$, $P2(x_2,y_2)$, $P3(x_3,y_3)$, and $P4(x_4,y_4)$, of the lines $L_1$ and $L_2$, whose direction vectors are the eigenvectors $[{\overrightarrow{v}}_1,{\overrightarrow{v}}_2]$. The orthogonal lines intersect at the midpoint $O({\overline{x}}_i,{\overline{y}}_i)$ of the pixels in B whose value is 1. Therefore, the region of movement is bounded by the points $E1(x_4,y_1)$, $E2(x_2,y_1)$, $E3(x_2,y_3)$, and $E4(x_4,y_3)$, as shown in Figure 3. The points that form the line $L_1$ can also be calculated using the regression line of Equation (3).

3.3. Active Shape Model (ASM)

ASM is a deformable model, which is generated from a statistical model containing variations created from a training set of labeled images. The proposed ASM is detailed in Figure 4.

ASM consists of two main stages: (a) training or generation of the point distribution model (PDM) and (b) an adjustment stage. The PDM is trained on a set of annotated images; a perpendicular gray-level image model or a gradient profile is extracted for each PDM landmark, usually trained with the same set of annotated images called the training set. In each image of the training set, several reference points (landmarks) are annotated (manually) along the contour of the object of interest, as illustrated in Figure 5, where an image with 50 landmarks can be observed.

Afterward, a non-aligned set of images of the pedestrian marked with 50 landmarks in each one of the frames is obtained, as exemplified in Figure 6.

The set of annotated contours is aligned to remove differences in the pose (translation at $X_t$ and $Y_t$, scaling s, and rotation $\theta$). Then, principal component analysis is performed to identify the main modes of shape variation in the aligned set of annotated contours. Next, image models are built, obtaining the gray or gradient levels for each point of the PDM by sampling the image profile perpendicular to the contour of the object at each landmark (l) for each of the training images, as is illustrated in Figure 7. The mean image profile ${\overline{g}}_i$ and the corresponding profile covariance ${\sum }_i$ are calculated for each point l of the PDM. In our example, it is necessary to calculate 50 average profiles perpendicularly and thus obtain the covariance matrices of each image [35].

In summary, the ASM-based algorithm consists of the following steps: (1) assignment of reference points or landmarks, (2) application of principal components analysis, (3) adjustment of the PDM, and (4) modeling of the local structure. Figure 8 shows a person with 50 manually selected landmarks on the contour of a training set. The necessary transformations for the alignments are determined in an iterative process.

3.3.1. Landmarks

When a video frame is captured, suitable reference points must be assigned to the object’s outline. The landmarks must be located in the contour of the pedestrian and must be updated from one image to another. In a two-dimensional image, they are represented as landmarks by a two-dimensional vector given by Equation (11). The configuration of this system consists of 50 manually assigned reference points ($L=50$). The main feature of reference points or landmarks is to control the shape of the model contours. Specifically, the initially assigned landmarks are updated by minimizing the deviation from the original profile, which is normal to the boundary at each landmark.

$$\mathbf{x}={[x_1,\dots ,x_L,y_1,\dots ,y_L]}^T$$

(11)

3.3.2. Principal Component Analysis (PCA)

A set of L reference points (landmarks) represents the shape of the object. They are used to train the ASM model, which is the human contour detection method. Figure 8 shows a set of M misaligned shapes of the same image which belongs to the training set. Although each shape m in the training set is in two-dimensional space, the shape can be modeled with a reduced number of parameters using the PCA technique. Suppose we have M shapes in the training set that are represented by ${\mathbf{X}}_m$, for $m=1,\dots ,M$. The PCA algorithm works as follows:

• Calculates the mean of the M sample shapes in the training set (Equation (12)).

  $$\overline{x}=\frac{1}{M}\sum _{m=1}^M{\mathbf{x}}_m$$

  (12)
• Computes the covariance matrix $\left(S\right)$ of the training set (Equation (13)).

  $$S=\frac{1}{M}\sum _{i=1}^M({\mathbf{x}}_m-\overline{x}){({\mathbf{x}}_m-\overline{x})}^T$$

  (13)
• Build the matrix of eigenvectors (Equation (14)).

  $$\Phi =\frac{1}{M}\sum _{m=1}^M[{\varphi }_1\mid {\varphi }_2\mid \dots \mid {\varphi }_q]$$

  (14)

  where ${\Phi }_j$, $j=1,\dots ,q$ represents the eigenvectors of S corresponding to the largest q eigenvalues.
• Given $\Phi$ and $\overline{x}$, each form can be updated as described by Equation (15).

  $${\mathbf{x}}_m\approx \overline{x}+\Phi b_m$$

  (15)

  where:

  $$b_m={\Phi }^T({\mathbf{x}}_m-\overline{x})$$

  (16)
• Calculate q such that the sum of the largest eigenvalues is greater than 98% of the sum of all eigenvalues.

To generate fair shapes, the distribution of b is evaluated. To constrain acceptable values, hard conditions can be applied to each element $b_m$ or a constraint to b, which can be reflected in a hyper-ellipsoid.

The PCA algorithm reduces the amount of data by simplifying the initial image to a set of eigenvalues and eigenvector while preserving enough information to perform the training stage. The matrix of eigenvectors $\mathrm{\Phi }$ (Equation (14)) contains the principal components of the training subset, which are used to update the model parameters during the fitting stage explained in the following section.

3.3.3. Fitting the Model

To perform the adjustment of the model, the best shape and pose parameters to match a shape in the model coordinate frame, x and y, with a new shape in the image coordinate frame are located, while minimizing the error function as shown in Equation (17).

$$E={(y-\mathbf{T}\mathbf{x})}^T\mathbf{W}(y-\mathbf{T}\mathbf{x})$$

(17)

where W is a diagonal matrix whose elements are weight factors for each landmark, and T represents the geometric transformation of rotation, translation, and scaling. The weight factors are set relative to the offset between the computed positions of the old and new landmarks in the profile. If the offset is large, then the corresponding weight factor in the array is set to low, and if the offset is small, the weight is set to high. Given a single point denoted by ${[x_0,y_0]}^T$, the geometric transformation matrix is defined by Equation (18).

$$\mathbf{M}=\left[\begin{array}{c}{x}_0\\ y_0\end{array}\right]=s\left[\begin{array}{cc}cos\left(\theta \right)& sin\left(\theta \right)\\ -sin\left(\theta \right)& cos\left(\theta \right)\end{array}\right]\left[\begin{array}{c}{x}_0\\ y_0\end{array}\right]+\left[\begin{array}{c}{x}_t\\ y_t\end{array}\right]$$

(18)

After computing the set of pose parameters ($\theta$, rotation; t, translation; and s, scaling), the projection of y onto the model’s coordinate frame is obtained from Equation (19).

$${\mathbf{x}}_m={\mathbf{M}}^{-1}y$$

(19)

Finally, the model parameters are updated as depicted in Equation (20).

$$b_m={\Phi }^T({\mathbf{x}}_m-\overline{x})$$

(20)

The optimal displacement of a landmark is obtained from the search procedure through the detected contours. The combination of optimally updated landmarks generates a new shape in the image coordinate frame. This new form is now used to find the closest form using Equation (17). After computing the best pose, denoted by M, this new shape is projected onto $\mathrm{\Phi }$, which contains the principal components of the given training set. This process updates the $\mathbf{b}$ parameter of the model. As a result, only a similar variation corresponding to the principal components can affect the model parameters. After computing the model parameters, the new form, denoted by ${\mathbf{x}}_m$, can be generated by Equation (15), and this new form is used for subsequent iterations, as in Equation (17). After a suitable number of iterations, the final form obtained is ${\mathbf{x}}_{new}$.

In this work, experimental parameters M = 50, L = 50, and the segment length of 40 pixels were set according to state of the art and the frame resolution [9,36,37]; additionally, they are based on the scale of the person’s contour. A programming function could be used to analyze the relationship of those parameters and estimate an optimal value; nevertheless, we considered that out of this study’s scope, which was focused on the analysis of MDBS and ASM methods for pedestrian localization in a video sequence.

4. Experiments

This section shows the results of the moving pedestrian detection process with the combination of MDBS and ASM (MDBS + ASM). MDBS identifies motion regions through the background subtraction motion detection method (Section 3.2); then, ASM (Section 3.3) adjusts the silhouette of the pedestrian.

All video sequences have different numbers of frames, but we standardized them to 50 frames of each video, only using those that contained moving pedestrians in the scene. For example, the office video scene has 2700 frames, but only frames from 620 to 700 (50 in total) were chosen because those are the frames is which the pedestrian appears to be walking. Then, the PDM (see Section 3.3) used 50 landmarks in the pedestrian contour, considering some anthropometric points of the body silhouette. In Figure 5, an example of the initial marking (which is done manually) is shown. Subsequently, the training set was aligned, originating the average form seen in Section 3.3. In capturing gray profiles in the CDnet2014 and CASIA Gait scenes, a 40 pixel (20 pixels up and 20 pixels down) line perpendicular to each landmark was drawn (see Figure 7).

4.1. Qualitative Results

The results obtained with the CDnet2014 dataset are shown in Figure 9 and Figure 10. They exhibit the original marking (GT, ground truth, in green) used for training and the estimated contour (red points calculated by our method). These points mark the silhouettes of the pedestrians detected with the ASM algorithm. As can be seen in the scene exhibited in Figure 9, the algorithm fits the contour of the pedestrian very closely in the office scene because the lighting, and especially the background, allow precise detection. However, there are many problems in the PETS2006 scene (Figure 10a): the reduced scale of the pedestrian, occlusion from the train, and reflections from the floor (background problems); these problems cause ASM not to fit correctly. In the sofa scene (Figure 10b), there are also occlusion problems, especially the reflections from the floor, which create contrast problems, making it more difficult for ASM to perform a close fit. Despite the mentioned troubles, the pedestrian’s silhouette was correctly identified in all cases.

The results from the CASIA Gait dataset, where the pedestrian is walking in different orientations with respect to the camera, are shown in Figure 11. The pedestrian is shown walking at 0 degrees, i.e., directly to the camera, in Figure 11a. The red points (approximate result) are very close to the landmarks (green points), which implies a good performance of the combination of MDBS + ASM. These were the best results of all orientations because there was not as much variation in the pedestrian’s arms and legs. However, when the pedestrian is walking at 36 degrees (Figure 11b), the red points in the arms (approximate contour) are moderately far from the true landmarks (green dots). In this scene, the pedestrian has variations in the movement of his arms, the opening of his legs, and his angle of walking (movement factors), which means that the ASM adjustment does not match the reference landmarks. Next, Figure 11c exhibits the adjustment results of a pedestrian walking at 54 degrees. The approximate points tend to be slightly off the landmarks in the legs, but the contour is more precise where the arms are closer to the body. Finally, the results of the adjustment of a pedestrian walking at 90 degrees are presented in Figure 11d, where the pedestrian walks side-on to the camera. The fit results shown by the (approximate) red dots tend to be far from the landmarks on the legs. The adjustment by ASM in this scene beings the dots close to the contour of the pedestrian at all points except for the gap between the legs due to the movement.

Good approximation of pedestrian detection is shown in Figure 9, Figure 10 and Figure 11 due to the combination of MDBS and ASM from a qualitative point of view. The contour adjustment of a moving pedestrian is better when background problems (occlusion or illumination) are not evident and the factor of movement associated with the pedestrian is not too pronounced (Figure 9). In contrast, in situations with background problems (Figure 10) or too many movement factors (Figure 11), the detection with MDBS and the adjustment to the contour with ASM showed limited (but acceptable) performance; i.e., the closeness between the real landmarks and those obtained by the approximation is not as good as expected, but the pedestrian was correctly identified, and the contour is well defined.

4.2. Quantitative Results

A quantitative analysis of the results obtained by the combination of MDBS + ASM was performed to deepen the study of this proposal. Leave-one-out cross-validation [38] was selected to verify the accuracy of the fitness achieved by the model. The ASM model was trained with $M-1$ frames, and the values obtained (or approximated) were evaluated with respect to the frame that was left out of the training. This process was repeated for M frames. The leave-one-out cross-validation is very useful to strengthen the validity of the experimental results when the available dataset is relatively small [39]—in this study, $M=50$. The fit error for each frame m was obtained by Equation (21), which uses the Euclidean distances between the landmarks $(x_l,y_l)$ and the points approximated by ASM $(x_l^{\prime },y_l^{\prime })$ for the total of landmarks L.

$$e_m=\frac{1}{L}\sum _{l=1}^L\sqrt{{(x_l-x_l^{\prime })}^2+{(y_l-y_l^{\prime })}^2}$$

(21)

Despite its simplicity, the Euclidian distance was used in this work because it has proved its effectiveness as part of the ASM procedure. Even recent state-of-art research continues applying the Euclidian distance for this reason (references [16,40]). Moreover, we consider using this parameter appropriate for the aim of this work because it calculates the distances between the landmarks and the approximate points obtained by the ASM during the contour detection.

Then, the mean fit error of the ASM is given by Equation (22), where $M=50$ and corresponds to the number of cross-validation iterations (number of frames) in the training set.

$$\overline{e}=\frac{1}{M}\sum _{m=1}^M{e}_m$$

(22)

In this paper, we use the box-plot data analysis method to explain the behavior of experimental results obtained by the MDBS + ASM approach. This tool (also called the box and whisker plot method) easily allows the comparison of different datasets. It uses the median, the approximate quartiles (Q${}_f$, where $f=\{1,2,3\}$), and the lowest (Min) and highest (Max) data points to convey the level, spread, and symmetry of a distribution of data values. It can also be easily refined to identify outlier values [41]. Figure 12 and Figure 13, visually summarize and compare groups of data corresponding to the studied scenes. The data correspond to the accuracy of the fitness achieved by the model, and they are represented in terms of the mean fit error of pixels (Equation (22)). Details of data shown in Figure 12 and Figure 13 are presented in

Table 1. Summary of the main characteristics of data shown in the box-plot diagrams of Figure 12 and Figure 13.

Scene	Min	Max	Median	Range	IQR	Q1	Q2	Q3
Office	3.0	6.5	4.5	3.5	1.5	3.8	4.5	5.3
PETS 2006	3.7	10	5.9	6.2	3.2	4.8	5.9	8.0
Sofa	2.8	8.9	6.1	6.1	3.4	3.9	6.1	7.3
Pedestrian 0 degrees	3.0	6.5	4.5	3.5	1.4	3.8	4.5	5.2
Pedestrian 36 degrees	3.7	10.1	5.9	6.4	3.4	4.6	5.9	8.0
Pedestrian 54 degrees	3.0	9.9	6.2	6.9	3.3	3.9	6.2	7.2
Pedestrian 90 degrees	2.9	11	5.1	8.0	3.9	3.3	5.1	7.2

, where important information for each scene, such as maximum and minimum values, ranges, IQR (interquartile range), and the average of pixels by quartiles is presented.

The box plot in Figure 12 shows the mean fit error for the CDnet2014 dataset. Each frame in this dataset has a resolution of 360 × 240 pixels for the office and sofa scenes and 720 × 576 pixels for the PETS2006 scene. The number of evaluated frames was set to $M=50$, the number of landmarks was $L=50$, and the size of the landmarks in the gray-scale profiles was 40 pixels. The analysis was done twice. The office scene resulted in a mean fit error of 7.2 pixels, which is a satisfactory result considering the level of occlusion in the lower extremities of the pedestrian in the scene. In the PETS2006 scene, a mean fit error of 8.3 pixels was obtained, despite the challenges of floor reflections, shadows, and occlusion. Similarly, in the sofa scene, a mean fit error of 7.1 pixels was achieved. The best performance of the MDBS + ASM proposal was observed for the sofa and office scenes and was slightly worse for PETS2006, because of the reduced size of the captured pedestrian, but also due to the background problems and occlusion with the train in the scene. In all cases, the error is small in comparison to the size of the original frames.

From a general point of view, Figure 12 shows that the proposed approach had a better performance in the Office scene than in PETS2006 or Sofa scenes (in this order, respectively). This is explained by the Office scene having less movement and low illumination variations (only when the pedestrian enters the office and is near a piece of furniture or when the pedestrian is reading a book). In the PETS2006 scene, the MDBS + ASM method presents the most significant errors when the camera is far from the objective (the pedestrian is at the extreme of the stage); also, the camera makes some adjustments due to the brightness of the floor or the occlusion that exists with background objects (such as a train that passes behind the pedestrian). Similarly, in the Sofa scene, the natural and artificial lighting affects the brightness of the floor, and it causes occlusion when the pedestrian passes near the sofa. Furthermore, opening and closing their feet and hands (movement variations) contributes to worsening the algorithm’s performance (compared to the other scenes).

For the evaluation of scenes in the CASIA Gait dataset, where pedestrians walk at different orientations relative to the camera, the results are shown in Figure 13. In this collection of video scenes, all frames have a resolution of 320 × 240 pixels. The number of evaluated frames was set to $M=50$, $L=50$ landmarks were used, the size of each one of the landmarks was 40 pixels, and the analysis was done twice, as it was for the CDnet2014. In the scene where the pedestrian walk at zero degrees, a mean fit error of 4.5 pixels was calculated (the overall best result in our experiments). Tegarding the pedestrian walking at 36 degrees, the mean fit error was 6.4 pixels; in the case of the scene at 54 degrees, the error was 6.1 pixels; lastly, in the scene with a pedestrian walking at 90 degrees, the error was 5.6 pixels. Higher error was noted when the pedestrian opened his legs, or variations in the movement of the arms were detected. However, the excess or lack of illumination or occlusions (background problems) also affect the performance of the MDBS + ASM technique because they avoid correctly detecting the pedestrian silhouette.

The best performance observed in Figure 13 was found for scenes where the position of the camera showed less movement variation. In the scene of the pedestrian walking at zero degrees, the shot is from the front, and the movement of hands and feet is not perceptible by the camera, so there was reduced error at the time of detection. However, in the scene with a pedestrian walking at 36 degrees, the camera angle is a factor when it focuses on the pedestrian moving his arms and legs when walking. In the scene of a pedestrian walking at 54 degrees, like the previous one, the camera’s position influences the detection, although there is less variation in the movement of the arms and legs. Finally, in the scene of the pedestrian walking at 90 degrees, it is observed walking sideways and walking from left to right, and some errors occurred at the time of detection between the legs. Moreover, the pedestrian’s clothing also affects the detection process, especially when pants are occluded by the floor squares (which can be almost the same as the pants, especially in the grayscale). To summarize, the complexity intrinsic to changes in the camera position causes varied performance of the MDBS + ASM model. For the above reasons, we consider it necessary to deeply study the method to improve its performance in complex scenarios.

To analyze if the mean fit error evaluated in the selected datasets has a normal distribution, the Jarque–Bera test was performed. It is a goodness-of-fit test of departure from normality based on the sample skewness and kurtosis. If a data distribution has a normal distribution, then the skewness (W) will be close to zero, and the kurtosis will be $K=3$, ideally. From a collection of N observations, the statistic test $JB$ is defined as follows [42]:

$$JB=\frac{N}{6}(W^2+\frac{{(K-3)}^2}{4})$$

(23)

In our experiments, the number of observations (groups) could be assumed as the total of frames used for each scene, being $N=50$. Then, the total of landmarks by frame $L=50$ led to 2500 data points per scene, analyzed to estimate the $JB$ significance value. The null hypothesis $H_0:W=0$ and $K=3$ is rejected if high values of $JB$ are obtained; i.e., the observed data do not follow a normal distribution. Moreover, the probability (p-value) of the null hypothesis assumes the fact that $JB$ is asymptotically distributed as a chi-Square function with two degrees of freedom [43], and the p-value could be estimated from classical probability tables [44] or standard spreadsheet software. Considering the right tail of the chi-square distribution, the null hypothesis can be rejected if the p-value < 0.05 [42]. In

Table 2. Results of the Jarque–Bera test for the selected datasets.

Scene	Mean Error	Skewness	Kurtosis	JB	p-Value
Office	4.5	1.220	2.375	13.212	0.00135
PETS 2006	8.3	0.769	0.928	13.868	0.00097
Sofa	7.2	0.916	0.498	20.031	0.00004
Pedestrian 0 degrees	4.5	1.268	2.492	13.940	0.00094
Pedestrian 36 degrees	6.4	1.260	3.200	13.486	0.00118
Pedestrian 54 degrees	6.2	1.390	2.680	16.39	0.00028
Pedestrian 90 degrees	5.6	1.090	1.170	17.03	0.00020

, the summary of the results of the Jarque–Bera normality test is exhibited.

From the normality test, it can be assumed that the fit error in all datasets does not follow a normal distribution, which in this case implies that each landmark in each frame has the same probability of error, and there was not a trend or bias in the experiments. This can be visually seen in the graphs in Figure 14, which have the mean fit error by frame for the Office, PETS2006, and Sofa. Similarly, Figure 15 exhibits the behavior of the mean error by frame for the CAISA Gait collection for each pedestrian walking orientation. In all cases, the error varies from one frame to another, in the intervals specified in Table 1. The variations in the error are explained because, in each frame, different conditions could be encountered: lighting, occlusions, pose, and so on. However, the error size is always relatively low (in the range of 2.8 to 11 pixels) compared to the size of the frame (hundreds of pixels).

5. Conclusions

The performance of a combination of MDBS with ASM for detecting moving pedestrians was studied in this work. In previous works, ASM has been reported as an appropriate technique for adjusting people’s silhouettes in static poses [9]. For this reason, it is relevant to explore its possibilities in the detection of people in motion. Our approach uses MDBS to locate pedestrians in motion; then, the silhouette of the located pedestrian is fitted with ASM. Background subtraction allows the detection of a moving region in a video sequence; then, image transformations, namely, translation, rotation, scaling, and especially translation coordinates, are necessary to adjust the silhouette of the pedestrian via the active shape model.

The primary contribution of this work was exploring the improvement potential of the combination of MDBS and ASM in detecting the contour of a moving pedestrian walking in a controlled environment. MDBS + ASM is a straightforward method based on classical algorithms which have shown effectiveness in pedestrian detection. We are looking for a practical process that could work in real applications, which is why we established our approach based on simple techniques.

Experimental results showed that the combination of MDBS and ASM achieves good performance in detecting moving pedestrians; however, it is slightly affected in situations where evident background problems exist (occlusion or lighting) or when pronounced movements of either arms or legs are present. The future of this work is to search for new mechanisms to improve the capacity of the MDBS + ASM proposal for the detection of moving pedestrians under challenging scenarios, such as those with background problems or movement factors, leading to its application in diverse environments.