A Smart and Robust Automatic Inspection of Printed Labels Using an Image Hashing Technique

Abstract

This work is focused on the development of a smart and automatic inspection system for printed labels. This is a challenging problem to solve since the collected labels are typically subjected to a variety of geometric and non-geometric distortions. Even though these distortions do not affect the content of a label, they have a substantial impact on the pixel value of the label image. Second, the faulty area may be extremely small as compared to the overall size of the labelling system. A further necessity is the ability to locate and isolate faults. To overcome this issue, a robust image hashing approach for the detection of erroneous labels has been developed. Image hashing techniques are generally used in image authentication, social event detection and image copy detection. Most of the image hashing methods are computationally extensive and also misjudge the images processed through the geometric transformation. In this paper, we present a novel idea to detect the faults in labels by incorporating image hashing along with the traditional computer vision algorithms to reduce the processing time. It is possible to apply Speeded Up Robust Features (SURF) to acquire alignment parameters so that the scheme is resistant to geometric and other distortions. The statistical mean is employed to generate the hash value. Even though this feature is quite simple, it has been found to be extremely effective in terms of computing complexity and the precision with which faults are detected, as proven by the experimental findings. Experimental results show that the proposed technique achieved an accuracy of $90.12\%$.

1. Introduction

Labels printed on products are usually made up of paper, metal, plastic film, cloth, etc. A label contains information about a product and hence it is necessary that products contain clearly printed labels. Inspection of labels ensures that labels are printed correctly and placed at the proper position. Inspection can be performed manually or automatically. Manual inspection requires a person to check each label, which slows down the process. This method is rarely used today because the results of an inspection vary from person to person. To overcome this problem, sophisticated computer vision algorithms are used. Label inspection is performed by mounting single or multiple cameras on an assembly line as shown in Figure 1. The camera captures images containing labels one at a time and sends them to a computer, where a computer vision algorithm checks each label for any possible fault. If the label is tampered due to a printing defect, then a feedback signal is sent to the assembly line to reject the label. Despite the fact that automatic inspection is accurate and faster than manual inspection, there are many problems that make it difficult to design an automatic label inspection algorithm. For example, the captured image may undergo distortions due to changes in appearance, illumination, colour, shape, rigid transformation and non-rigid transformations. Moreover, it is a challenging task to recognize objects in images with complex backgrounds depending upon the viewpoint [1].

In this paper, a new technique is proposed using image hashing to detect printing faults in product labels. The hash of an image gives a compact and fixed-length numeric representation of image content and has been widely used for integrity verification [2,3,4,5,6]. The advantage of using image hashing is its robustness to content preserving operations, like compression, contrast enhancement, geometric transformations, etc. On the other hand, the image hash should be sensitive in detecting tampering and can further locate tampered areas. It is imperative to note that traditional cryptographic hash functions such as MD5, SHA1, SHA256, etc., are not suitable for label inspection application because they are sensitive to a single bit change in pixel value, and hence are not robust to content preserving operations [7,8]. To design an image hashing algorithm, both robustness and discrimination are the main factors. There is a trade-off between these two properties, which means that if robustness is increased then discrimination is reduced and vice versa [9,10].

For label inspection, a hashing algorithm should be fast and highly robust to geometric transformations and sensitive to detect minute level tampering. It is difficult to find all these properties in a single hashing scheme. In this paper, a new image hashing scheme is proposed to verify the integrity of printed labels. The proposed scheme is resilient to content preserving operations like geometric transformations and is able to detect minute level tampering. Therefore, any small area which has a printing fault can be easily detected. Secondly, due to the robustness property of image hashing, there is a very low false rejection of correctly printed labels. Generally, the false rejection occurs due to alignment, lightning and scaling issues. Most of the image hashing techniques presented so far mainly focus on multimedia security and image authentication purposes. In our work, we extended the application of image hashing and used it for the label inspection work. Label inspection requires that the algorithm be fast and detect faults at high accuracy. To make the inspection process fast we used a simple yet effective approach to detect tampering in labels. Experimental results show that the proposed scheme can detect faults in labels at high speed with high accuracy. In Section 2, a review of a number of image hashing schemes proposed in the literature is presented. The hashing methodology adopted in this paper for label inspection is illustrated in Section 3. The proposed hashing scheme for label inspection is presented in Section 4. Experimental results are given in Section 5 and Section 6 concludes the paper.

2. Related Work

Conventionally, perceptual image hashing is used in multimedia security, content authentication and image forensics. In recent years, many researchers have used image hashing in various applications. For example, to search an image in large databases [11], facial recognition [12] and image authentication [13]. In the last decade, many perceptual image hashing techniques have been developed to detect tampering in images. In early techniques, Discrete Cosine Transform (DCT) and Discrete Wavelet Transform (DWT) were used to create image hash. Fridrich et al. [14] used DCT coefficients to extract image content and proposed a hashing scheme that is used in the application of digital watermarking. Venkatesan et al. [15] constructed image hash by using statistics of DWT coefficients. This approach makes the scheme robust against geometric distortions and compression. Karsh et al. [16] presented a hashing scheme which uses Discrete Wavelet Transform-Singular Value Decomposition (DWT-SVD) and the spectral residual method for hash formulation. From an image, DWT-SVD extract global features and the spectral residual model extract the saliency regions. In another work, Karsh et al. [17] used global and local features of an image obtained by using ring partitions and salient regions for hash construction. Qin et al. [18] utilized the Dual Cross Pattern (DCP) and salient structural features. In this scheme, a Gaussian low pass filter and Singular Value Decomposition (SVD) operation were applied on the image. Texture features were extracted from DCP and all the salient features were combined to compress the textual features to generate a robust image hash. Experiments show that the proposed scheme gives better performance in anti-collision attacks.

Tang et al. [19] proposed a hashing scheme based on Colour Vector Angles (CVA). For the calculation of CVA, the input image was first normalized by interpolation and low pass filtering. A histogram was extracted within an inscribed circle of the normalized image. Finally, a hash was formed by compressing the histogram. Tang et al. [20] extended his work and applied DCT, DWT and Discrete Fourier Transform (DFT) on the histogram to obtain a hash of small size. In both these techniques, the calculation of CVA takes all the pixels into account which makes the scheme resistant to rotation. In a recent work, Tang et al. [21] proposed rotation invariant image hashing. From the normalized image, the rotation invariant feature matrix was extracted through Log Polar Transform (LPT) and DFT. By using Multi-Dimensional Scaling (MDS), the algorithm learns a discriminative and compact representation of the feature matrix.

Ouyang et al. [22] presented a new hashing scheme that uses combined Scale-Invariant Feature Transform (SIFT) and quaternion Zernike moments to extract features. Only the top k features are selected to construct image hash and repeated key points were removed during feature extraction. This method can locate tampering and also classifies the type of tampering in an image. The hash produced in this scheme is short and the hashing is robust to large angle rotation. Vadlamudi et al. [23] used the n distinct SIFT feature points from the Lightness (L) component of $L^{*}{a}^{*}{b}^{*}$ (L. Lightness, a. red-green and b. yellow-blue) colour image space to extract the image content. DWT is subsequently applied to the extracted content to obtain approximation coefficients. Binary hash is formed by normalizing the approximation coefficients thus making the proposed method invariant to many distortions, for example, compression, scaling, filtering, contrast adjustment and brightness.

Gharde et al. [9] introduced a dual perceptual hash function to generate a hash using a fuzzy colour histogram. The $L^{*}{a}^{*}{b}^{*}$ colour space was used in this scheme. An unbiased fuzzy colour histogram was generated through tuning factor, which improves the system robustness and discriminative capability. Ng et al. [24] utilized Convolutional Neural Networks (CNN)s to develop the scheme named Multi-Level Supervised Hashing (MLSH) with deep features. The multiple-hash-table mechanism was integrated into the mechanism to extract the features from deep convolutional layers of the network so that both the semantic and structural information can be preserved. Moreover, the recall and precision rate of the scheme illustrates the better performance of the algorithm than existing techniques. Hosny et al. [7] introduced a hashing scheme which is based on Quaternion Polar Complex Exponential Transform (QPCET). In this scheme, the input image is first normalized using bicubic interpolation and the Gaussian low pass filter is then applied. This is followed by hash construction from features that are extracted through QPCET moments. Pun et al. [25] proposed a feature extraction method that combines local features of both structure and colour information. The hash is constructed through Horizontal Location-Context Hashing (HLCH) and Vertical Location-Context Hashing (VLCH) methods. Image authentication and tamper localization methods are also proposed. The results in [25] show robustness against different signal processing and geometric attacks. Fei et al. [26] proposed a hash-based template matching algorithm for video sequences. To improve tracking, Laplace-based Hash (LHash) and Laplace-based Difference Hash (LDHash) are proposed. In this scheme, video objects are tracked efficiently as compared to conventional tracking algorithms such as mean-shift or Compressive Tracking (CT) with low-computational cost. Ji et al. [27] proposed an image segmentation algorithm for Synthetic Aperture Radar (SAR) images which is based on perceptual image hashing. First, segmentation is done with multi-thresholding followed by removal of speckle noise and transforming the image in the DCT domain. The DCT coefficients are then compressed by using Principle Component Analysis. Hash values of the image regions are obtained by applying the Different Hashing Algorithm [28]. To achieve better results, the uneven background is removed with morphological methods.

The image hashing techniques reviewed above, and, generally speaking, nearly all the hashing techniques proposed in the literature are robust to a limited type of non-malicious distortion. It is difficult to find a single image hashing scheme that is resilient to all types of non-malicious distortions. Secondly, some of the techniques are slow and cannot be applied in real time. In image hashing, the type of features selected has a significant impact on the performance of the hashing scheme. The fundamental requirement of fault detection in printed labels is that the hashing scheme should be fast and resilient to geometric transformations and distortions due to illumination and noise. To address these issues, the statistical mean of non-overlapping image blocks is used as a feature to generate the image hash. This feature, although very simple, requires less computational time and has been found to be robust. To undo geometric distortions, SURF features are used for the alignment of query and template images. This makes the proposed scheme invariant to geometric and other distortions.

3. Methodology

The general block diagram of the proposed hashing algorithm used for label inspection is shown in Figure 2. The whole process consists of two parts; in the first part, hash and SURF features of the template image is obtained. In the second part, before calculating the hash of the query image, the Region of Interest (ROI) containing the label under consideration is extracted. This is done so that one-to-one matching between the template label image and the query label image becomes possible. The process requires SURF features of both template and the query image. To find tampered areas in the query image, it is matched with the hash of the template image.

To list all possible faults, a database of actual labels supplied by Sprint Digital Sdn Bhd and SKOP Malaysia is used in this paper. Images in the database were collected from an assembly line by installing a camera to capture the printed labels. In this database, some images are faulty and some are without any fault. At first, possible sets of faults are identified manually by observing the labels. The proposed hashing algorithm then identifies the labels as correct or faulty. From the database, one image is selected as the template image in which there is no fault, as shown in Figure 3. All the images are compared with the template image using perceptual image hashing to identify any possible tampering.

There are two major issues that need to be resolved before calculating the hash of the label under consideration. If these two problems are not addressed, then a correct label could be classified as incorrect. The first problem is the partial presence of other labels in the captured frame as shown in Figure 4a–g. Interestingly, these seven images also have printing faults that are encircled. The hashing algorithm should ignore the partial presence of other labels and only detect fault(s) in the ROI under consideration which is shown by the bounding box. The second problem is due to geometric transformation between the template image and the query image labels as shown in Figure 5. The hashing algorithm should be robust or invariant to geometric transformations. In the proposed hashing algorithm, SURF features are used to undo geometric distortions.

To match a template label image with the given query image label, there should be a correspondence between them. Many algorithms have been proposed to find correspondence between images. These algorithms have been used in a wide a range of applications like object tracking [29], image fusion [30], image registration [31], and object detection and recognition [32,33]. To extract the ROI containing the label under consideration, the ROI of the template image is required. The goal is to detect the template ROI in the query image. To solve this problem, a reliable and fast ROI extraction method is required. Various approaches have been proposed to recognize features in a digital image, for example, colour histograms [34], receptive field histograms [35], eigenspace matching [36], etc. When an image is subjected to affine transformation, rotation, scaling and or illumination, most of these detection algorithms may not be robust against all types of distortions. Moreover, they are comparatively slow and cannot be used in real-time applications which require algorithms to be fast. In 1999, Lowe proposed SIFT [37] to detect features that are invariant to a number of distortions. SIFT, however, is not fast enough for real-time applications. To ensure high-speed feature detection, the Speeded-Up Robust Feature (SURF) detector was introduced in 2006 [38]. The SURF algorithm is able to detect features in an image similar to SIFT, but at a higher speed because of low dimensionality feature descriptors.

Feature extraction produces a “descriptor” which is a vector with numerical values. In the feature extraction stage of SURF, descriptors are computed around the interest points, also called key points. Hence, a descriptor is defined as a numerical feature obtained around a key point. Traditionally, these descriptors are matched by using Euclidean distance. This method, however, produces a lot of outliers, thus significantly reducing the accuracy of matching. If matching is inaccurate, then the geometric transformation between template and query images will not be correct. This will directly affect the ROI extraction. To filter out these outliers and correctly estimate geometric transformation between the template and label images, SURF features are used along with Maximum Likelihood Estimation Sample Consensus (MLESAC) [39] algorithm to match images. SURF features are chosen because experimental results discussed in Section 5.2 show that it gives robust and fast results when compared with other known feature extraction algorithms like KAZE [40], MSER [41] and BRISK [42]. By using MLESAC, maximization of log-likelihood estimates is transformed into minimization of the cost function [39]. This method effectively removes mismatched feature points and easily extract the ROI which contains the label under consideration.

4. The Proposed Label Inspection Algorithm

In this section, a new scheme is proposed to automatically inspect printed labels using SURF features for alignment of images and perceptual image hashing for image matching. Figure 6 shows the detailed block diagram of the proposed scheme along with the different types of algorithms used in each step. The entire process consists of three parts as explained below.

• Template image hash: The image which is selected as a template should be error-free because the hash of other images is compared with this image. First, the template image is converted into grey scale to reduce processing time. To calculate the hash of the template image, it is divided into $16\times 16$ non-overlapping blocks and the mean value of each block is calculated. Mean values of all the blocks are finally concatenated to generate the hash of the template image. Both SURF features and the image hash are stored in a database. SURF features are used for ROI alignment of the query label image in the image matching module.
• Query image hash: To calculate the hash of the query image, it is first aligned with the template image because images captured from the camera are generally misaligned as shown in Figure 5. To properly align query and template images, SURF features of both template and query images are used by the MLESAC algorithm to extract ROI from the query image. By finding the ROI, it is ensured that the query image does not contain extra patches of proceeding or preceding labels. The extracted ROI is divided into $16\times 16$ non-overlapping blocks and the mean of all the blocks are concatenated to obtain the hash of the query image.
• Hash matching: To compare the hash of both template and query images, the mean value of each block of the template image is matched with the corresponding block mean of the query image. The sum of the absolute difference between template and query image hash is compared with the chosen threshold to decide whether the corresponding area is tampered or not. In case of a printing fault in the label, the difference between the mean value of the corresponding spatial area will be greater than the threshold. This enables not only the detection of the faulty area but also localizing its exact spatial location where the printing error has occurred in the label.

Following are the main steps of the proposed hashing algorithm.

• Image pre-processing.
• Image alignment.
• ROI extraction.
• Hash generation.
• Hash matching.

These steps are summarized in Algorithm 1 and details of each step are presented in the following sections.

Algorithm 1: Hash generation steps.
	Template image hash
	1. Read the template image.
	2. Convert the image from RGB to grey scale.
	3. Extract SURF feature vector for the template image, $F_s$.
	4. Divide the image into $16\times 16$ blocks.
	5. Find mean of the image blocks, $M_T$.
	6. $F_s$ and $M_T$ are stored in the database.
	Hash of query image
	1. Read the query image.
	2. Convert the image from RGB to grey scale.
	3. Extract SURF feature vector of the query image $F_q$.
	4. Outliers rejection and image alignment using MLESAC algorithm using $F_s$ and $F_q$
	5. Divide the aligned image into $16\times 16$ blocks.
	6. Find the mean of the image blocks $M_Q$.
	7. $M_Q$ is stored in the database.
	Hash matching
	1. $M_{diff}$ = $M_T$ − $M_Q$   if $M_{diff}$ > $th$           area of the image is tampered   elseif $M_{diff}$ ≤ $th$           area of the image is not tampered

4.1. Image Pre-Processing

In this step, the RGB label image is converted to grey scale by using National Television Standards Committee (NTSC) technique [1] as follows:

$$\begin{array}{c}\hfill g(x,y)=0.2989\ast I(x,y,R)+0.5870\ast I(x,y,G)+0.1140\ast I(x,y,B)\end{array}$$

(1)

In Equation (1), $g(x,y)$ is the grey scale image and $I(x,y,R)$, $I(x,y,G)$ and $I(x,y,B)$ are the red, green and blue pixel values of the RGB image, respectively. Grey scale conversion is performed to reduce the processing time of the hashing algorithm. Figure 7 shows the grey scale version of both template and query images.

4.2. Image Alignment

Image alignment is the most important part of the proposed scheme. The captured query image may undergo various geometric transformations like translation, rotation, scaling or their combination. Hence it is necessary that before calculating the hash of the query image, the query image should first be aligned with the template image. The alignment parameters are obtained by matching the SURF features of the query image and the template image. Once the query image is aligned with the template image, the ROI from the query image is extracted by cropping with respect to the coordinates of the template image. The process of alignment consists of the following steps:

4.2.1. Feature Detection

The amount of feature detection varies from image to image because each image is captured differently. Figure 8 shows SURF [38] feature detection for template and query images.

4.2.2. Feature Matching

Feature matching finds correspondence between the template and query images. The process of matching is based on the sum of absolute differences. Some factors like image size and image quality affect the time and reliability of feature matching. For a visual representation, the matched features points of both images detected in the previous step are joined by lines as shown in Figure 9.

4.2.3. Outliers Rejection and Region of Interest (ROI) Alignment

Outliers are those features that are generated due to false interest points. In Figure 9, it can be clearly seen that there are a lot of false points which are indicated by red lines, while inliers are shown by yellow lines. The outliers are mainly due to camera orientation because whenever a camera captures an image, it usually contains background along with the ROI. For example, portions of other labels marked by a square can be seen in Figure 10.

Due to possible geometric transformation, the ROI of the query image may not be properly aligned with the template, as shown in Figure 8 and Figure 9. In addition, portions of proceeding and preceding labels are also captured in the frame. To correctly match template and query images, the ROI of the query image should be properly aligned so that the exact portion of the misprinted area could be located. To address this problem, the image captured by the camera is modelled as a rotated, scaled and translated version of the template image. The transformation matrix of such an image is given as follows [1]:

$$\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ 1\end{array}\right]=\left[\begin{array}{c}T\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right].$$

(2)

$$\begin{array}{c}\hfill X^{\prime }=TX.\end{array}$$

(3)

Let $X^{\prime }$ be the resultant image after applying transformation matrix T on the image X.

$$X^{\prime }=\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ 1\end{array}\right],X=\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$$

where T is given by

$$\begin{array}{c}\hfill T=\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\\ a_{31}& a_{32}& a_{33}\end{array}\right].\end{array}$$

(4)

The matrix T can be any transformation matrix such as translation ($T_r$), scaling (S) or rotation (R). These transformations can be expressed as [1]:

$$\begin{array}{c}\hfill T_r=\left[\begin{array}{ccc}1& 0& t_1\\ 0& 1& t_2\\ 0& 0& 1\end{array}\right].\end{array}$$

(5)

$$\begin{array}{c}\hfill R=\left[\begin{array}{ccc}cos\theta & -sin\theta & 0\\ sin\theta & cos\theta & 0\\ 0& 0& 1\end{array}\right].\end{array}$$

(6)

$$\begin{array}{c}\hfill S=\left[\begin{array}{ccc}{S}_1& 0& 0\\ 0& S_2& 0\\ 0& 0& 1\end{array}\right].\end{array}$$

(7)

By combining the transformations given by Equations (5)–(7), new transformations can be obtained as shown below.

$$E=\left[\begin{array}{cc}R& T_r\end{array}\right]S_i=\left[\begin{array}{cc}SR& T_r\end{array}\right]A_f=(S,T_r,S_i)$$

For example, the Euclidean (E) transformation is a combination of translation and rotation. The similarity ($S_i$) transformation is a combination of scaling, translation and rotation, while the affine ($A_f$) transformation is a combination of scaling, rotation, translation and similarity transformations.

To overcome the problems due to outliers rejection and image alignment, the RANSAC algorithm can be used but the Maximum Likelihood Estimation Sample and Consensus (MLESAC) algorithm gives better performance than RANSAC [39]. Both RANSAC and MLESAC are used when input data is contaminated with outliers. When the MLESAC algorithm is applied, the data is divided into inliers and outliers. It uses feature vectors from both the template and query images and approximate the transformation matrix to project the query image on the template image and eliminate all the outliers points [39].

In Figure 11a,b, feature matching results on sample images indicate that all the outliers are not rejected, due to which the remaining outliers interfere with the results of matching. Figure 11c,d are the results obtained after applying the MLESAC algorithm.

4.3. ROI Extraction

The MLESAC algorithm gives the output transformation matrix, $T_q$ which is used to extract the ROI from the query image.

$$\begin{array}{c}\hfill X_q=T_q{X}_{qq}\end{array}$$

(8)

In Equation (8), $X_q$ is the aligned query image with portions of proceeding and preceding labels being cropped and $X_{qq}$ is the actual query image. Therefore, the image $X_q$ contains only the portion of the query image (ROI) which is to be matched with the template image. Results obtained after applying the MLESAC algorithm are shown in Figure 12. There is a large number of SURF feature points, however, a limited number of feature points are shown in Figure 11 for ease of illustration.

4.4. Hash Generation

The aligned images shown in Figure 12a is divided into $16\times 16$ non-overlapping blocks. Let $B_1,B_2,\cdots ,B_n$ be the blocks of image and $M_1,M_2,\cdots ,M_n$ be the mean of each block. Let $B_1$ be the first block of the query image and $a_{1,1},a_{1,2},\cdots ,a_{16,16}$ be the raw pixel values of $B_1$ as shown in Equation (9).

$$\begin{array}{c}\hfill B_1=\left[\begin{array}{cccc}{a}_{1,1}& a_{1,2}& \cdots & a_{1,16}\\ a_{2,1}& a_{2,2}& \cdots & a_{2,16}\\ ⋮& ⋮& \ddots & ⋮\\ a_{16,1}& a_{16,2}& \cdots & a_{16,16}\end{array}\right],\end{array}$$

(9)

The mean of $B_1$, represented as $M_1$ is calculated by using Equation (10).

$$\begin{array}{c}\hfill M_1=\frac{{\displaystyle \sum _{i=1}^{16}}{a}_{1,i}/16+{\displaystyle \sum _{i=1}^{16}}{a}_{2,i}/16+\cdots +{\displaystyle \sum _{i=1}^{16}}{a}_{16,i}/16}{16}.\end{array}$$

(10)

In a similar way, mean of all the blocks are calculated. The hash of the template image ($H_i^T$) is formed by concatenating all the mean value of the blocks as shown by Equation (11).

$$\begin{array}{c}\hfill {H_i}^T=[{M_1}^T,{M_2}^T,\cdots ,{M_n}^T]\end{array}$$

(11)

Similarly, the hash of the query image ($H_i^Q$) is expressed as:

$$\begin{array}{c}\hfill {H_i}^Q=[{M_1}^Q,{M_2}^Q,\cdots ,{M_n}^Q]\end{array}$$

(12)

where $[{M_1}^T,{M_2}^T,\cdots ,{M_n}^T]$ and $[{M_1}^Q,{M_2}^Q,\cdots ,{M_n}^Q]$ in Equations (11) and (12) represents mean of all blocks in template and query image.

4.5. Hash Matching

The mean obtained in Section 4.4 is used to detect misprinted or tampered areas in the query image using hashing. To calculate the hash of the template image, the steps discussed in Section 4.1 are applied and the hash are generated by using Equation (10). The steps discussed in Section 4.2.1 is also applied to the template image to obtain SURF feature vectors, which are stored in a database. SURF features are used for the ROI alignment of the query image. To calculate the hash of the query image, all the steps discussed from Section 4.1, Section 4.2, Section 4.3 and Section 4.4 are applied to the query image and this process also uses the stored SURF feature vectors of the template image which is already present in the database. The integrity of the query image is checked by calculating the hash distance between the template and the query image.

$$\begin{array}{c}\hfill M_{di}=|{H_i}^T-{H_i}^Q|\end{array}$$

(13)

In Equation (13), $M_{di}$ is a matrix containing the sum of the absolute difference between the hashes of each block. The decision of which area is tampered is based on $M_{di}$. Ideally, if the area is not tampered, then its absolute difference will be zero, However, due to distortion like scaling, rotation etc., the value of $M_{di}$ is slightly greater than zero. This problem can be solved by selecting a suitable threshold ($Th$); if $M_{di}$ is greater than the predefined threshold it means that the block is tampered, otherwise not.

$$\begin{array}{c}\hfill Decision=\left\{\begin{array}{cc}Imageblockiistampered\hfill & :M_{di}>Th\hfill \\ Imageblockiisnottampered\hfill & :M_{di}\le Th\hfill \end{array}\right.\end{array}$$

(14)

The threshold ($Th$) value should be selected in such a way that it should not be very large nor too small. If the selected threshold is too large, the hashing algorithm will not detect any tampered points even if they are present. Similarly, if the selected threshold is too small, then outliers will be detected and the algorithm may falsely detect them as tampered regions.

5. Experimental Results

In this section, the adopted methodology to experimentally evaluate parameters of the proposed scheme such as threshold selection, ROC curve and implementation time is presented.

5.1. Threshold Value Selection

To select a suitable threshold, the proposed algorithm is tested on different threshold values to obtain variation between threshold values and the accuracy of the system. The accuracy of the system is defined as:

$$\begin{array}{c}\hfill Accuracy=\frac{Correctlyidentifiedimages}{Totalnumberofimages}\end{array}$$

(15)

To experimentally estimate the accuracy of the proposed hashing scheme, a labelled dataset of 81 images captured from a real assembly line are used from which 41 images are tampered. The results of applying the proposed hashing scheme to all the images are shown in Figure 13. It is observed that by increasing the threshold, the accuracy of the system increases at the start but when the threshold value reaches around 65, the accuracy starts to decrease because the algorithm identifies all the 81 images as correct. After doing a number of experiments, the threshold value of 65 has been found to give good results with an accuracy of $90.12\%$.

To identify the spatial position area in which tampering has been detected, a binary mask of the same size as that of the query image is generated. If a particular block is tampered, then it is displayed in white and all other blocks are displayed as black. Figure 14a,c,e,g,i,k,m shows labels that are captured from the assembly line and misprinted areas are highlighted by a square. Figure 14b,d,f,h,j,l,n shows the output of the proposed label inspection algorithm which detects the misprinted regions displayed as white. The black portion indicates that the area is not tampered. For example, Figure 15b is fully black, which indicates that the image is not tampered. Although the algorithm works on grey scale images, it can also detect printing of wrong colours. This happens because the change in colour changes the grey scale value which affects the mean of the corresponding block. For example, in Figure 14e, there are two faults in the image; the letter ‘S ’is missing and the colours are misprinted, both of these faults were successfully detected by the proposed algorithm.

5.2. Implementation Time

In the proposed scheme, SURF feature extraction and alignment consumes most of the computation time. There are many other algorithms, for example, KAZE [40], MSER [41] and BRISK [42] which give similar performance as SURF but they are slow compared to SURF. To compare the processing time of SURF with other algorithms, an experiment was performed on a laptop with an Intel Core i7 (4500U) processor, 8 GB RAM and MATLAB 2018a.

Table 1. Time for different features.

Feature	Computational Time (Seconds)
$MSER$	$52.24$ s
$KAZE$	$19.97$ s
$BRISK$	$10.32$ s
$SURF$	$5.07$ s

gives an idea about the timing of the proposed algorithm when SURF features are replaced with other features. The timing of SURF comes out to be the fastest, i.e., 5.07 s among all the other algorithms.

A high processing time of 5.07 s is probably due to the presence of high-resolution images. Although, algorithm has the capability to process images of any given size but most of the images have sizes of around 3391 × 1017 × 3. However, this much time is not acceptable for real-time applications. To overcome this problem, the proposed algorithm was also implemented in the Python programming language using the Open-CV library which helped to reduce the implementation time from 5.07 s to 0.4 s.

5.3. Receiver Operating Characteristics (ROC) Curves

To evaluate the performance of the proposed scheme and its performance between discrimination and robustness, parameters such False Acceptance Rate ($FAR$) and False Rejection Rate ($FRR$) are computed. $FAR$ is defined as the number of times tampered images are detected as authentic. $FRR$ is defined as the number of times authentic images are detected as tampered. $FAR$ and $FRR$ are defined as [13].

$$\begin{array}{c}\hfill FAR=\frac{n_1}{N_1}\end{array}$$

(16)

$$\begin{array}{c}\hfill FRR=\frac{n_2}{N_2}\end{array}$$

(17)

In Equation (16), $n_1$ is the number of tampered images that are misclassified as genuine and $N_1$ is the total number of images in the dataset. Similarly, in Equation (17), $n_2$ is the number of genuine images which are misclassified as tampered and $N_2$ is the total number of genuine images in the dataset.

To estimate $FAR$ and $FRR$, all the images in the dataset are used. One image is selected as the template and the hash of all the 81 images are compared with it. The threshold is varied from 1 to 75 to obtain $FAR$ and $FRR$ values. Both $FAR$ and $FRR$, as shown in Figure 16 and Figure 17, are constant in the start due to the fact that the algorithm parameters are not optimized to work on less threshold. When the threshold value starts increasing, the $FAR$ starts increasing and $FRR$ starts decreasing. After reaching the threshold value of 65, both $FAR$ and $FRR$ become constant and therefore 65 is selected as a suitable threshold.

In a ROC curve, false acceptance rate and false rejection rate are plotted on the x-axis and y-axis, respectively. Both $FRR$ and $FAR$ are inversely proportional and there is a trade-off between these quantities. When $FRR$ increases, then $FAR$ decreases and vice versa. The ROC curve of the proposed scheme is plotted in Figure 18. When the $FAR$ is very small, the value of $FRR$ is $0.6$ which is very high and unacceptable. When the threshold value is increased, the $FRR$ start to decrease but at the same time, the value of $FAR$ starts increasing. When the threshold value reaches 65, the $FAR$ start increasing but at the same time, the $FRR$ becomes constant. A value of 65 is therefore selected as a threshold for this dataset. The values of $FRR$ and $FAR$ at threshold 65 are listed in

Table 2. Values of $FAR$ and $FRR$.

Metric	Value
$FAR$	$0.08641$
$FRR$	$0.025$

. The result is promising as it suggests that at a low value of $FRR$, the $FAR$ of the system is also small.

6. Conclusions

In this paper, a label inspection scheme is proposed using the idea of image hashing. The proposed scheme can find misprinted areas in labels of different products. Open SURF features of both template and query images are used which makes the tamper detection robust against rotation, scaling and other geometric transformations. The MLESAC algorithm removes outliers and features which are not robust. The mean of the blocks is computed to generate the image hash. If the mean of a block is greater than the chosen threshold, then the area corresponding to that block is considered tampered. Experimental results have demonstrated that the mean value of a block which is used as the feature to generate hash is effective to find faults in printed errors in labels. Since the calculation of the mean value does not require any complex mathematical operation, therefore, the response time of the system when implemented in Python is 400 ms. Other feature extraction methods were also tested but they make the computational time slow. From the ROC curve and results shown in Table 2, it is evident that the proposed hashing scheme for label inspection is robust to geometric distortions and at the same time sensitive to detect tampering with fault localization. Although the proposed scheme is designed for label inspection, it can also be applied for authentication of natural and synthetic images.