Research on Luggage Package Extraction of X-ray Images Based on Edge Sensitive Multi-Channel Background Difference Algorithm

Abstract

Many security detectors do not have the ability to output individual luggage package images and are not compatible with deep learning algorithms. In this paper, a luggage package extraction of X-ray images based on the ES-MBD (Edge Sensitive Multi-channel Background Difference Algorithm) method is proposed, which is aiming at the problem that background difference binarization is insensitive to texture features and edge detection binarization is insensitive to smooth areas. In this method, X-ray luggage package images from complex original video images are used as a key target, the RGB three-channel background difference is calculated from the original X-ray image, the edge detection of the grayscale map is performed using the Sobel operator optimized by local gradient enhancement, and the morphological expansion process is performed on the combined results to obtain the complete wrapping target. The Suzuki algorithm is used to detect the outline of the binarized package image, match the package frame area and determine the key target. The ES-MBD method solves the problem of information loss in the traditional binarization method, and retains the information of insensitive regions while reducing noise. Through experimental comparison, the accuracy of ES-MBD binarization method reaches 97.3%, the recall rate reaches 96.5%, and ES-MBD method has obvious advantages in key target extraction of X-ray images.

1. Introduction

Deep learning models for specific target detection has been applied in a wide range especially in the transportation industry [1,2]. In the security inspection scenario, there is also a need to use deep learning models to assist or even replace labor. In fact, there are already many new security machines that integrate contraband detection. Since the target to be detected is relatively fixed, the deep learning model actually performs quite well in this scenario.

However, the use of deep learning models to assist security checks has not been widely used. The main reason is that the input required by the deep learning object detection model is a single image. Although this function is not difficult to implement, the old security detectors of various customs and express delivery stations do not include the function of transmitting a single package image to the deep learning algorithm server at the beginning of the design, so if there is a need to use the deep learning model for contraband detection, the new security detector can only be replaced, resulting in a waste of resources.

At present, there are two main solutions to this problem:

• The video image of the security detector is input to the algorithm server in a frame-by-frame or frame-skipping manner;
• Through the combination of target extraction, target tracking and keyframe detection, the package image in the keyframe is input to the algorithm server.

In the first method, because the same package appears in multiple frames of the video, it leads to repeated detection of packages, resulting in a large amount of GPU overhead, higher detection costs, and incomplete packages in some frame images, which is prone to false detection.

The second method can ensure that each package is only detected once as much as possible, but the current commonly used grayscale-based target extraction method is not accurate, easy to cause incomplete package images.

In this scenario, there are generally two methods to achieve package extraction: one is the object detection algorithm based on deep learning, and the other is the unsupervised detection algorithm based on binarization. The advantage of deep learning-based target detection algorithm is that the detection results are accurate, but the disadvantage is that the shape of the target to be detected is highly uniform. Since it is a supervised model, when the shape of the target to be detected is different and not fixed, a convergent model cannot be trained, so the target cannot be detected. Meanwhile, the deep learning model has higher requirements on hardware computing power. With the second method, the package can be distinguished from the background by binarization of the image, and the location of the package area can be obtained by contour detection. Compared with deep learning object detection algorithm, this method has the advantages of less computation and no fixed object shape to be extracted. It is a lightweight unsupervised detection algorithm. It is widely used in the scenarios of pipeline and road foreign body detection. The limitation of the algorithm is that the background features need to be fixed, and the algorithm cannot be used when the background changes. The common binarization method has the disadvantage that the extraction target may not be complete. In this paper, some improvements are made to the method to improve extraction precision.

By binarizing the X-ray image and using the set threshold to remove the background information, the wrapped image is finally extracted using contour detection [3,4]. The contour feature extraction method depends on the relationship between the background gray level and the package gray level, which has certain limitations. When the background gray level is between the maximum and minimum gray level of the package, the method cannot work effectively because there is no threshold to separate the background and the package in this scene. Mei [5] used the edge features to extract the contours of moving objects. Wu et al. [6] used the edge detection operator to extract the golden region of the image. Their work indicated that image feature extraction by the edge detection operator is more sensitive to the texture region in the image. However, the features extracted from the image region with weak texture cannot form closure, which leads to errors in the subsequent contour detection, thus affecting the effect of target extraction. Tian and Liu [7] used the binarization method of the LoG edge detection operator; when the image color of cardboard boxes, backpacks and other items is similar to the background color, the effect is better. This is because the binarization method based on the edge detection operator is more sensitive to the gradient change in the image, and it can accurately detect the intersection of the edge and background. But for large areas of the image with the same color and less texture, the edge detection operator cannot offer a better detection effect.

Threshold method is a classical target extraction algorithm which has been widely used in many fields [8,9,10,11]. The method optimizes a threshold through some algorithm, the final threshold value as the dividing line, greater than the threshold value for the change class and less than the threshold value for the unchanged class [12]. The threshold method deals with determination of the threshold value. The maximum expected value algorithm is the EM algorithm. Based on the segmentation method of automatic selection of multiple thresholds, Li [13] used the watershed algorithm based on markers to extract the image histogram and obtain multiple thresholds. On the basis of threshold segmentation, the watershed algorithm is used to segment the image, so as to extract the region of interest in the X-ray image. Bruzzone [14] applied the maximum expected value algorithm of Gaussian model to the analysis of difference graph. The difference graph was modeled. The varying pixels and invariant pixels obey the Gaussian distribution, respectively, and the threshold value is obtained through multiple iterations. The parameters of the model are obtained through multiple iterations of EM algorithm. Bazi et al. [15] applied the KI (Kittler–Illingworth) threshold method to the difference graph analysis algorithm. The KI threshold method is based on the Bayes theory of minimum error. Later, the researchers improved the model to produce a generalized KI threshold algorithm [16]. The threshold method is simple and fast, but it cannot use spatial information effectively.

This paper focuses on the accurate extraction of the luggage package target. An ES-MBD (Edge Sensitive Multi-channel Background Difference Algorithm) method based on edge-sensitive multi-channel background difference is proposed to achieve more accurate image binarization. In addition, the Suzuki contour detection algorithm is adopted to detect the contour of the binary image obtained by the ES-MBD method to subsequently match the area of the package box and then make a judgment. The detection result is output as the key target of the X-ray image.

The main contributions of this paper are as follows:

(1)

In this paper, the ES-MBD method is proposed. It solves the problems of low detection efficiency and high GPU overhead due to the unfixed shape of luggage packages, stacking, and occlusion in the video. The ES-MBD method is able to improve the detection efficiency and extract a more complete image of luggage packages.

(2)

The ES-MBD method combines the background difference binarization and edge detection operator binarization, which solves the problem of the binarization method of background difference being insensitive to texture features, while the binarization method based on edge detection operator is insensitive to smooth regions. Through experimental comparison, the precision rate of the ES-MBD binarization method reaches 97.3% and the recall rate reaches 96.5%, and the ES-MBD method has obvious advantages in the luggage package target extraction of X-ray images.

(3)

The Sobel operator optimized by local gradient enhancement performs edge detection on the grayscale image. The loss of local information can be reduced, and a better detection effect can be obtained. Using the Suzuki algorithm, the binary image contour obtained by the ES-MBD method can be detected. The proposed ES-MBD method can solve the problem of information loss in the traditional binarization method and preserves the information of the insensitive area while reducing the noise.

The rest of the paper is organized as follows: Section 2 summarizes the object extraction method of the X-ray image. Section 3 analyzes and compares different binary methods. Section 4 proposes an optimized Sobel operator with local gradient enhancement and presents the ES-MBD method. Section 5 introduces the key target tracking and extraction process. Section 6 provides experimental results and analysis. Section 7 concludes the paper.

2. Object Extraction Method of X-ray Image

In the field of image detection, the region of concern is called the region of interest (ROI). In the detection and tracking algorithm, the region where the target is detected is the region of interest, and other regions are the background. As shown in Figure 1, the car in the red box is the region of interest. In contrast to the background, the area of interest is the object that needs to be extracted. The area where the luggage package is located in the X-ray security check image is the object, and the area outside the package is the background. The first step of detection tracking algorithm is object extraction. The deep learning model is used to extract a specific object in complex scenes, such as in the detection of vehicles and pedestrians on the road as shown in Figure 2. The advantages of this method are that it has high detection accuracy for relatively homogeneous targets (such as people and vehicles), has good robustness, and is not susceptible to interference from complex environments. However, it has high requirements on hardware and has a poor detection effect on targets with unfixed features.

The specific method of X-ray package object extraction is shown in Figure 3. Binarization is carried out on the input image; then, the contour detection method is used to extract the object area; luggage package target tracking and key frame judgment are carried out; and finally package target extraction is carried out [17]. As shown in Figure 4, in the X-ray security inspection image, the shape features of the luggage package are different, while the background is often a pure color. The interference mainly comes from the noise in the image and the image of the small stains on the conveyor belt of the security inspection machine. The deep learning detection algorithm is not applicable to this scene. Due to the relatively simple background and interference in the X-ray image, this paper uses the image processing method to extract the X-ray image of the luggage package. The area where the package is located can be determined.

3. Comparative Analysis of Different Binarization Methods

Binarization is the process of removing background and noise information from the image, and by binarizing the image and performing morphological processing (opening and closing operations) noise reduction on the binary image, an image containing only the wrapping information can be obtained. At present, the commonly used binarization method is a grayscale-based binarization method, which belongs to a color-sensitive binarization method. In this paper, a multi-channel background difference binarization method is proposed, which has a more accurate effect than the grayscale-based binarization method.

At the same time, a texture-sensitive binarization method based on the Sobel operator is proposed, and a method with higher accuracy (the ES-MBD method) is obtained by combining this method with the color-sensitive binarization method.

3.1. Color-Sensitive Binarization Methods

3.1.1. Binarization Method Based on Gray Level

Converting a color image to a grayscale image is a common method in image processing, as shown in the common grayscale transformation Formula (1):

$$Gray=0.299\times Red+0.587\times Green+0.114\times Blue,$$

(1)

where $Gray$ represents the calculated gray value, $Red$, $Green$ and$Blue,$ respectively, represent the RGB three-channel values of the pixel point.

As shown in Figure 5, the background of the X-ray security image is white or close to white. After converting it into a gray image [18], the background color is brighter than the object color. A threshold is specified such that pixel points with a gray value greater than the specified threshold are zero, and points with a gray value less than the specified threshold are one, resulting in a binarized X-ray security image.

As shown in Figure 6, the binary image is operated to eliminate the noise in the image. However, there are also some problems: when the gray value of the background is within the range of the gray value of the object (that is, the gray value of part of the object is greater than that of the background, while the gray value of the other part is less than that of the background), it is impossible to find a suitable threshold to distinguish the object and the background. This method is more sensitive to background noise. When the background noise is complex, it cannot find an appropriate threshold value for binarization. When the threshold is too large, as shown in Figure 7a, the background noise cannot be removed; when the threshold is too small, as shown in Figure 7b, the object information is lost.

3.1.2. Binarization of Background Differences

Due to the simple structure, less image information and sensitive background noise, the traditional gray binarization method cannot completely extract the object outline of the luggage package in the X-ray image. To solve this problem, this paper proposes a binarization method based on background difference, which introduces more background information in the X-ray image into the algorithm to solve the problem of background noise sensitivity. The difference map is calculated from the grayscale image. The value of each pixel in the difference map is shown in Formula (2):

$$Gra{y}_{diff}=\left|Gray-Gra{y}_{bg}\right|,$$

(2)

where Gray is the gray of the pixel point and $Gra{y}_{bg}$ is the gray of the background. Figure 8 shows the gray difference map and the binary gray difference map. As can be seen from Figure 8, although the wrapping contour of the binary gray difference map is clearer than that of the gray difference map, it cannot cover the entire wrapping area.

In order to exclude the case that different RGB values may correspond to the same gray value, Formula (3) is used to calculate the difference map:

$$Gra{y}_{diff}=\frac{r\times \left|Red-Re{d}_{bg}\right|+g\times \left|Green-Gree{n}_{bg}\right|+b\times \left|Blue-Blu{e}_{bg}\right|}{r+g+b},$$

(3)

where $Red$, $Green$ and $Blue$ are the RGB values of the pixel, $Re{d}_{bg}$, $Gree{n}_{bg}$ and $Blu{e}_{bg}$ are the RGB values of the background, and $r$, $g$ and $b$ are the preset weight parameters.

It can be seen from Figure 9a,b, in the gray difference map calculated with RGB values that the difference between background and object is more obvious, which can basically cover the whole luggage package area, and it is easier to find the appropriate threshold for binarization processing. However, background noise is also obvious in the figure. The open operation of the RGB difference map after binarization can eliminate the noise in the image, and the detection effect of the luggage package contour is better.

3.2. Texture-Sensitive Binarization Methods

Binarization Method Based on Edge Detection Operator

In addition to grayscale and background difference methods, edge detection operators are also widely used for binarization of X-ray images. Edge refers to the position where the gray value changes sharply, and during the image formation process, due to the difference in physical factors such as brightness, texture, color, shadow, etc., the gray value of the image is abruptly changed, and then the edge is formed. Edges are quantified by examining each pixel’s neighborhood and quantifying its grayscale change, which is equivalent to the difference between the square derivative or discrete series of continuous functions in calculus.

Edge detection is mostly based on the convolution method based on directional derivative mask [19] (gradient directional derivative), and the convolution operators to calculate the change in gray level are based on the convolution method based on the directional differential convolution kernel. When the edge detection operator consisting of two or more convolution kernels is used, it is assumed that there are n convolution kernels; there are usually multiple ways to measure the edge intensity of the final output. The sum of the absolute values of corresponding positions is selected as the edge intensity of the final output.

In this paper, the Sobel operator [20,21,22,23] is used, which is an improvement on the Prewitt operator [24,25,26,27], and a weight of two is used on the central coefficient, as shown in Formulas (4)–(7):

$$g_x=\frac{\partial f}{\partial x}=\left(z_7+2z_8+z_9\right)-\left(z_1+2z_2+z_3\right),$$

(4)

$$g_y=\frac{\partial f}{\partial y}=\left(z_3+2z_6+z_9\right)-\left(z_1+2z_4+z_7\right),$$

(5)

$$g_x{}^{\prime }=\frac{\partial f}{\partial x}=\left(z_2+2z_3+z_6\right)-\left(z_4+2z_7+z_8\right),$$

(6)

$$g_y{}^{\prime }=\frac{\partial f}{\partial x}=\left(z_6+2z_8+z_9\right)-\left(z_1+2z_2+z_4\right).$$

(7)

The corresponding operator is

$$g_x=\left[\begin{array}{ccc}-1& -2& -1\\ 0& 0& 0\\ 1& 2& 1\end{array}\right], g_y=\left[\begin{array}{ccc}-1& 0& 1\\ -2& 0& 2\\ -1& 0& 1\end{array}\right],$$

$$g_x{}^{\prime }=\left[\begin{array}{ccc}0& 1& 2\\ -1& 0& 1\\ -2& -1& 0\end{array}\right], g_y{}^{\prime }=\left[\begin{array}{ccc}-2& -1& 0\\ -1& 0& 1\\ 0& 1& 2\end{array}\right].$$

Since the Sobel operator can carry more information about the edge direction and has a better suppression effect on noise, the Sobel operator in all directions is used to operate the gray image and sum the absolute value of the final result for binarization. The binarization image obtained in this way is more sensitive to gradient changes (i.e., texture features). The results of binarization based on the edge detection operator are shown in Figure 10. Compared with the above two methods, the image texture features extracted by the binarization method based on edge detection operators are more obvious.

3.3. Comparison of Different Binarization Methods

As shown in

Table 1. Advantages and disadvantages of different binarization methods.

Method	Advantage	Disadvantage
Gray binarization method	The implementation is simple and requires no additional hyperparameters	Object whose gray is smaller than background and larger than background cannot be detected simultaneously
Background difference binarization method	Sensitive to color differences	The color parameter of the background needs to be known, and it is insensitive to areas similar to the background color
Sobel operator binarization method	Texture-sensitive	It is not sensitive to a large area of the same color, and information is often lost when the operator is used alone

, the comparative analysis of different binarization methods shows that the method of binarization through the gray level is simple to achieve without additional hyperparameters. However, due to its fixed threshold, this method cannot simultaneously detect the regions where the gray is smaller than the background and the gray is larger than the background. The setting of two threshold values at the same time has the same effect as the background difference binarization method. Background difference binarization uses color difference as input, as long as the area with a different color from the background can be detected. Its disadvantage is that it needs to know the background color in advance, and as for the area similar to the background color, the color difference is too insignificant to detect. The Sobel operator is extremely sensitive to texture, so even the region with a similar color to the background can be detected by the Sobel operator as long as there are color fluctuations in or near the region. Its disadvantage is also obvious. For a large area with the same color, the result obtained by the Sobel operator is zero; that is, it cannot be detected. Therefore, many unrecognized regions appear when the Sobel operator is used as the binarization method alone.

4. ES-MBD Binarization Method

4.1. Optimization of Sobel Operator with Local Gradient Enhancement

The traditional Sobel operator binarization method uses horizontal and vertical directions for gradient operation [28,29,30]. As shown in Figure 11, due to the diverse angle of the X-ray image, the Sobel operator using only two directions tends to lose part of the gradient information of the image.

The Sobel operator is added to four directions of a $\pm {45}^{°}$ diagonal line, as shown in Figure 12. The gradient features are calculated and fused, respectively. Compared with traditional methods, the gradient features of the image can be extracted more completely.

Since the threshold is fixed in the binarization process, the parts with a weak local gradient in the image are filtered out by binarization, resulting in information loss. In order to solve this problem, the method of local gradient enhancement is adopted in this paper, and the gradient features are locally enhanced before binarization, as shown in Formula (8). For pixel point P in the local region, the gradient $G_P$ after enhancement is

$$G_p=\left\{\begin{array}{c}T+1, G_P^0>mea{n}_C\\ 0, G_P^0<mea{n}_C\end{array},\right.$$

(8)

where

• T—The binarized threshold, generally using the average value of the global gradient;
• $G_P^0$—The gradient of the pixels before the enhancement;
• $mea{n}_C$—The average gradient of a local area.

That is, if the gradient of a point is greater than the average gradient of the area, its gradient is enhanced to above the threshold. In this way, the loss of local information can be reduced, and a better detection effect can be obtained.

Since the result obtained by the edge detection operator is smaller, smaller open operation structure elements need to be used to prevent the object from disappearing completely during the corrosion operation.

For some items (such as cartons, backpacks), the imaging color under the X-ray security detector may be close to the background color, the use of background difference binarization method cannot better extract the complete target area. On the contrary, binarization method based on edge detection operator is more sensitive to the gradient changes in the image and can accurately detect the intersection of the edge and the background. For a large area of the image with the same color and less texture (such as a book, a knife, etc.), the edge detection operator cannot obtain better results.

4.2. Morphological Processing

The morphological image processing [31,32] method was proposed by J. Seerra in 1964. Its basic idea is to use a special structural element to measure or extract the corresponding shape or feature in the input image for further image analysis and target recognition. The input of morphological method is a binary image, where one (the white area in the image) is the object and zero is the background. After image binarization processing, background noise as shown in Figure 13 cannot be completely filtered out through binarization, so the morphometric method is used to filter the noise to prevent its interference to contour detection.

Structuring Elements (SE) [33] can be of any shape. Common SE structuring elements include rectangle and cross. The value can be zero or one. The structure element has an anchor point, which is generally defined as the center of the structure element (the location can be optionally defined). Figure 14 shows several structuring elements of different shapes, with the purple area as anchor points.

We let f be the original image and s be the structure element.

Erosion can be denoted by symbol $\mathrm{f}\ominus \mathrm{s}$. The operation method is to slide the structuring element on the image and set the image pixel value of the anchor point of the structure element to the minimum value of the image area pixel corresponding to the area of the structure element value of one. As shown in Figure 15a, the object of the image is visually “corrupted”.

Dilation can be denoted by symbol $\mathrm{f}\oplus \mathrm{s}$. The operation method is to slide the structure element on the image and set the image pixel value of the anchor point of the structure element to the maximum value of the image area pixel corresponding to the area of the structure element value of one. As shown in Figure 15b, the object of the image is visually “bloated”.

Etching and then expanding the image with the same structure element is called the Opening operation, denoted by symbol $\left(\mathrm{f}\ominus \mathrm{s}\right)\oplus \mathrm{s}$. As shown in Figure 16a, objects that are originally connected are visually “separated”. The first expansion and then the etching of the image with the same structure element is called the Closing operation, which is denoted by symbol $\left(\mathrm{f}\oplus \mathrm{s}\right)\ominus \mathrm{s}$. As shown in Figure 16b, this visually “closes” the previously separated parts. The noise of Color 1 rather than object in the binary image can be removed by the morphological method.

4.3. ES-MBD Binarization Method

The binarization method based on background difference is insensitive to the package close to the background in the image, while the edge detection operator is sensitive to texture but insensitive to large areas of the same color. This paper proposes an ES-MBD (Edge Sensitive Multi-channel Background Difference Algorithm) method, the process of which is shown in Figure 17.

• Step 1: the RGB three-channel background difference is calculated for the original image. At the same time, the original image is converted to grayscale and processed using the Sobel edge detection operator;
• Step 2: The two results obtained in Step 1 are binarized using different threshold values, respectively;
• Step 3: The results obtained in Step 2 are denoised by morphological processing, respectively, to obtain $F_d$ and $F_s$, as shown in Formula (9);
• Step 4: The union set of $F_d$ and $F_s$ is $F_{ds}$, that is,

$$\left\{\begin{array}{c}{F}_{ds}=0, F_d=0, F_s=0 \\ F_{ds}=1, else , \end{array}\right.$$

(9)

where $F_d$ denotes the image obtained after processing by the multi-channel background difference method followed by binarization and morphological noise reduction. $F_s$ denotes the image obtained after processing by the Sobel edge detection operator followed by binarization and morphological noise reduction.

The combined $F_{ds}$ is expanded morphologically to fill the missing area and finally output $F_{ds}$.

Whether the background difference binarization method or the binarization method based on edge detection operator, morphological noise reduction is accompanied by the loss of information in the insensitive region. The advantage of ES-MBD method is that it can achieve noise reduction and ensure that the information in the insensitive region is not lost. In addition, by expanding the binarization result after merging, the detection region can be better obtained without worrying about excessive noise.

Figure 18a shows the binary result of RGB background differences, and Figure 18b shows the binary result of the Sobel edge detection operator. It can be seen from the figure that there are missing areas for wrapping, only wrapping lines can be extracted, and the whole contour cannot be covered. Figure 18c shows the binary result of the ES-MBD method, which shows that compared with the background difference and edge operator, the package scope is more complete, and the target package is easier to extract.

5. X-ray Image Key Target Tracking and Extraction

5.1. Suzuki Contour Detection

Profile detection is a method of acquiring connected areas in an image through binarization processing, and the Suzuki contour detection algorithm proposed by Satoshi Suzuki is commonly used [34]. The four boundaries of the image are called the frame of the image, and for an image with width w and height h, it is regarded as a matrix of order h×w composed of pixels; then, Rows 1, h, 1, w of the matrix constitute the frame of the image. A pixel with a gray value of zero is called a zero pixel, and a pixel with a gray value of one is called a one pixel. In this algorithm, the frames of the binarized image are assumed to be zero pixels, and if the frame of the input image has one pixel, it is changed to zero pixels.

Figure 19 is an example figure of Suzuki’s algorithm, in which pixels with the same absolute value belong to the same boundary, and the relationship between each boundary is recorded on the right side of the figure, where ob is the outer border, hb is the hole border, and the parent border means that the outer layer is the parent of the inner layer.

For the pre-extraction of X-ray images, the foreground that needs to be extracted is actually the outer boundary of the frame in all parent boundaries in the contour inspection results.

5.2. X-ray Image Key Target Tracking

(1)

Target tracking algorithm

The binarization method can extract the target in each frame picture. In order to find out the corresponding relationship between each target in the adjacent frame, it is necessary to use the target tracking algorithm to track the package box extracted based on the object as input. The steps are as follows:

• Step 1: The frame in which packages appear for the first time is taken as the initial frame. Each package $B_k$ detected in this frame is assigned a unique number $f\left(B_k\right)$ and stored in the cache;
• Step 2: A new frame is detected. Each package $B_i$ detected by a new frame with all the packages in the cache is compared, the most matched cache package $B^{-1}{}_j$ is selected, they are considered to be the same package, $f\left(B_i\right)=f\left(B^{-1}{}_j\right)$ is set, and the cache of the package is updated;
• Step 3: If package $B_m$ is detected in the new frame of Step 2 and does not match the cache, it is considered as a new package, and a new unique number $f\left(B_m\right)$ is assigned to it, and it is included in the cache;
• Step 4: If the cache package $B^{-1}{}_n$ in Step 2 does not match the cache package of the new frame, the cache of the package is deleted.

(2)

The selection of loss function

The function used to evaluate the degree of package matching is called the loss function used by the tracking algorithm. The overlapping degree of package boxes is used as the loss function to evaluate the degree of match, and the intersection ratio (IoU) and its variants are adopted.

IoU is the most commonly used index in target detection, and its definition is shown in Formula (10):

$$\mathrm{IoU}=\frac{\left|A\cap B\right|}{\left|A\cup B\right|}.$$

(10)

IoU can reflect the overlap degree of two targets, and it has scaled invariance. However, there are some disadvantages of IoU.

If two objects do not intersect, according to definition, IoU = 0. Then, IoU cannot accurately reflect the degree of overlap. As shown in Figure 20, IoUs are equal in all three cases, but the degree of coincidence is not the same. The graph on the right has the best regression and the graph on the left has the worst regression.

GIoU’s thought is put forward by Rezatofighi H et al. in CVPR2019 [35]. Its definition is shown in Formula (11):

$$\mathrm{GIoU}=\mathrm{IoU}-\frac{\left|{\mathrm{A}}_{\mathrm{c}}-\mathrm{U}\right|}{\left|{\mathrm{A}}_{\mathrm{c}}\right|}.$$

(11)

First, the minimum closure area of the two boxes is calculated to determine the IoU; then, the proportion of the closure area that does not belong to the two boxes to the closure area is computed, and finally this proportion is subtracted from IoU to obtain GIoU.

Like IoU, GIoU is a distance measure, and it is insensitive to scale. At the same time, GIoU is the lower bound of IoU; in the case that the two boxes coincide infinitely, IoU = GIoU. On the other hand, IoU takes the value of [0, 1], but GIoU has a symmetric interval, taking the value range of [–1, 1]. The maximum value 1 is taken when the two coincide, and the minimum value −1 is taken when they have no intersection and are infinitely far away, so GIoU is a very good distance measure. Different from IoU, which only focuses on the overlap area, GIoU focuses on not only the overlap area, but also other non-overlap areas, which can better reflect the overlap degree of the two.

In order to make the target frame regression more stable, for the scene of target frame regression, the distance, overlap rate and scale between the target and anchor are taken into account to obtain DIoU [36], which does not have the divergence problem in the training process like IoU and GIoU. Its definition is shown in Formula (12):

$$\mathrm{DIoU}=\mathrm{IoU}-\frac{{\mathsf{\rho }}^2\left(\mathrm{b},{\mathrm{b}}^{\mathrm{gt}}\right)}{{\mathrm{c}}^2},$$

(12)

where

• b—the center point of the prediction box;
• ${\mathrm{b}}^{\mathrm{gt}}$—the center point of the real box;
• $\mathsf{\rho }$—the Euclidean distance between two center points;
• $\mathrm{c}$—the diagonal distance of the minimum closure region that can contain both the predicted box and the real box.

Similar to GIoU, DIoU can provide direction of movement for the bounding box when it does not overlap with the target box. DIoU can directly minimize the distance between the two target boxes, which converges much faster than GIoU. As for the case involving two boxes in both horizontal and vertical directions, the DIoU can make the regression very fast, while the GIoU almost degenerates into the IoU. DIoU can also replace the normal IoU evaluation strategy and be applied to the NMS to make the results of the NMS more reasonable and effective.

The aspect ratio of the three elements of bbox regression (bounding box regression) is not considered in the calculation, and CIoU is proposed on the basis of DIoU. Its definition is shown in Formula (13):

$$\mathrm{CIoU}=\mathrm{IoU}-\frac{{\mathsf{\rho }}^2\left(\mathrm{b},{\mathrm{b}}^{\mathrm{gt}}\right)}{{\mathrm{c}}^2}-\alpha v,$$

(13)

where $v$ is used to measure the similarity of the aspect ratio, as shown in Formula (14):

$$v=\frac{4}{{\pi }^2}{\left(\arctan \frac{w^{gt}}{h^{gt}}-\arctan \frac{w}{h}\right)}^2.$$

(14)

The above four loss functions are used to judge the degree of coincidence of two targets in target detection and related fields. Since IoU has many shortcomings, DIoU and CIoU loss functions are introduced to improve the target frame regression. In the case of target tracking, GIoU is more suitable for relatively simple use.

5.3. Extraction of Key Targets in X-ray Images

The detection and tracking algorithm can accurately locate the location of each package in each frame image. To intercept each package completely and without repeating, it is necessary to extract the key target of the video [37,38,39]. That is, in the process of a package from entering the picture to leaving the picture, a frame is selected in which the package is completely displayed as the key target of the package, and the package area of the frame is cut out as the image of the package. Since the X-ray machine usually moves the package in a fixed direction, assuming that the package moves from right to left, the extraction method is as described below.

A trigger line is set at a distance to the right of the image. For any package $\mathrm{B}$ and frame $F_k$ containing package $\mathrm{B}$, $Flag\left(F_k{}^B\right)$ represents the position relationship between the right edge of the package frame and the trigger line. $Flag\left(F_k{}^B\right)=0$ is when the right edge of the package frame is on the right side of the trigger line; otherwise, $Flag\left(F_k{}^B\right)=1$.

We let $F_0,F_1,\dots ,F_n$ be all the video frames of a package appearing in the picture, and sort them in chronological order, that is, the next frame of $F_i$ is $F_{i+1}$.

Since the package enters from the right at the beginning, before the package passes through the trigger line, as shown in Figure 21a, the right boundary of the package frame must be on the right side of the trigger line at the beginning, and obviously there is $0<i<n$ that makes

$$Flag\left(F_k{}^B\right)=0, k<i.$$

(15)

As the package moves to the left, as shown in Figure 21b, after the package passes through the trigger line, there must be $i<j<n$ that makes

$$Flag\left(F_k{}^B\right)=0, k<j,$$

(16)

$$Flag\left(F_j{}^B\right)=1,$$

(17)

where $F_j$ is the frame that passes through the trigger line for the first time on the right side of the package frame, and $F_j$ is the key frame of the package used to obtain the complete image of the package.

X-ray package detection tracks the extraction process, as shown in Figure 22.

• Step 1: The ES-MBD method is used to process the input video frame, calculate its multichannel background difference, and use the Sobel operator to process the gray image. The two are binarized and denoised respectively, and then merged. The combined results are expanded by morphology to obtain the binarized image.
• Step 2: The Suzuki algorithm is used to detect the contour of the binary image. The outer boundary whose parent boundary is the frame in the result is selected as the detected package box area.
• Step 3: The package box region obtained in Step 2 is matched with the package in the cache, and the cache is updated. At the same time, the key frame judgment is carried out for the successfully matched package.
• Step 4: If there is a package with successful key frame judgment in Step 3, the package box area is intercepted as the image output of the package.

6. Experimental Results and Analysis

Several X-ray machine security videos are selected, and different binarization methods are used to compare the extraction of key targets. There are 113 packages in the video. Figure 23 shows some package binarization images obtained by each method.

As can be seen in Figure 23, there are many incomplete packages in the method based on gray binarization. Some packages are missing or truncated in the method based on background difference and the method based on Sobel operator binarization, while the method based on ES-MBD binarization still maintains the integrity of packages after removing noise.

Comparing the intercepted package image in the key target with the actual package image, if the intercepted package image is complete, the package is considered to be successfully detected. In addition, if the adjacent package is detected as a whole package by the algorithm, detection fails. The detection results of different methods are shown in

Table 2. Test results of different methods.

Method	Detection Number (TP)	False Detection Number (FP)	Total True Packages
Gray binarization method	55	60	113
Background difference binarization method	89	30	113
Sobel operator binarization method	70	65	113
ES-MBD binarization method	109	3	113

.

The package detection problem in this paper is not a binary classification problem. Binary classification is a deterministic quantity problem for both P (positive) and N (negative). Packages are treated as positive class. Package detection does not set a fixed value for N (background is negative class), so it is an atypical binary classification problem. The evaluation of this algorithm is performed with the help of evaluation metrics (precision, recall, ${\mathrm{F}}_1-\mathrm{Score}$, accuracy).

In this paper, the algorithm detection results are evaluated using the confusion matrix which includes the following four values:

True Positive (TP): Positive samples are detected as positive samples, i.e., packages are detected as packages, indicating the number of correctly detected packages.

False Positive (FP): Negative samples are detected as positive samples, i.e., the background is detected as packages, indicating the number of misdetected packages.

False Negative (FN): Positive samples are detected as negative samples, i.e., packages are detected as background, indicating the number of missed packages.

True Negative (TN): Negative samples are detected as negative samples, i.e., the background is detected as a background, and since the algorithm only detects packages, the term is constant to zero.

The confusion matrices for the four methods are shown in

Table 3. Confusion matrix for the Gray binarization method.

	True	False
Positive	55	60
Negative	58	0

,

Table 4. Confusion matrix for the Background difference binarization method.

	True	False
Positive	89	30
Negative	24	0

,

Table 5. Confusion matrix for the Sobel operator binarization method.

	True	False
Positive	70	65
Negative	43	0

and

Table 6. Confusion matrix for the ES-MBD binarization method.

	True	False
Positive	109	3
Negative	4	0

.

The confusion matrix allows us calculation of the following evaluation metrics:

Precision represents the proportion of positive samples that are detected correctly:

$$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}.$$

(18)

Recall represents the proportion of all input positive samples that are detected:

$$\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}.$$

(19)

${\mathrm{F}}_1-\mathrm{Score}$ is a harmonic average of precision and recall used to avoid a situation where one is high and the other is low. The higher the F1-Socre, the better the algorithm works:

$${\mathrm{F}}_1-\mathrm{Score}=\frac{2\times \mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}.$$

(20)

Accuracy represents the proportion of positive and negative samples that the algorithm detects correctly overall. As this algorithm only focuses on the detection effect of positive samples, this metric is only for reference:

$$\mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FP}+\mathrm{TN}+\mathrm{FN}}.$$

(21)

The evaluation of the detection effect of each method is shown in

Table 7. Evaluation metrics of different binarization methods.

Method	Precision Rate	Recall Rate	${\mathbf{F}}_1-\mathbf{Score}$	$\mathbf{Accuracy}$ Rate
Gray binarization method	47.8%	48.7%	0.482	31.8%
Background difference binarization method	74.8%	78.8%	0.767	62.2%
Sobel operator binarization method	51.9%	61.9%	0.565	39.3%
ES-MBD binarization method	97.3%	96.5%	0.969	94.0%

.

As can be seen in Table 7, the precision rate of the gray binarization method is 47.8%, the recall rate is 48.7%, and the ${\mathrm{F}}_1-\mathrm{Score}$ is 0.482. Accuracy is a global metric and is related to both positive and negative samples. Because luggage package detection does not focus on negative samples in this paper, the accuracy rate is not discussed in the result analysis. Among the four binarization methods, the overall effect is the worst. Due to the simple structure, less image information and sensitive background noise, the traditional gray binarization method cannot completely extract the object outline of the luggage package in the X-ray image. The precision rate of the background difference binarization method is 74.8%, the recall rate is 78.8%, and the ${\mathrm{F}}_1-\mathrm{Score}$ is 0.767. The main problem of the background difference binarization method is that it cannot detect some edge areas of the package whose color is not obvious, resulting in incomplete package. The precision rate of the Sobel operator binarization method is 51.9%, the recall rate is 61.9%, and the ${\mathrm{F}}_1-\mathrm{Score}$ is 0.565. The Sobel operator binarization method is insensitive to large package regions with fewer homochromatic textures, which can lead to many packages being split into multiple parts with a very high number of false detections. The ES-MBD method combines background difference binarization with Sobel operator binarization, which can avoid the shortcomings of both. The precision rate reaches 97.3% and the recall rate reaches 96.5%. The results prove that the ES-MBD method has an obvious detection effect.

7. Conclusions

The ES-MBD method proposed in this paper combines the Sobel operator-based binarization method with the background difference binarization method, which effectively addresses the limitations of both methods and reduces the false detection rate. This method can significantly improve the final detection accuracy and the extraction effect of key targets in X-ray images.

(1)

In the ES-MBD method, the Sobel edge detection operator is optimized. Compared with traditional methods, it can extract image gradient features more completely, reduce the loss of local information, and obtain a better detection effect. This method is used in parallel with the background difference binarization method, which can effectively improve the integrity of image extraction and greatly reduce image noise.

(2)

The ES-MBD method can solve the problem of general methods only being sensitive to strong background differences and insensitive to large areas of the same color.

(3)

Through the experimental comparison of various binarization methods of X-ray images, the ES-MBD method has the best integrity and the least image noise, reaching a 97.3% accuracy rate and a 96.5% recall rate, while other methods only have the highest effect of a 74.8% accuracy rate and a 78.8% recall rate.