3.3.1. Grouping Triplets of Finder Patterns

In the previous steps, Finder Patterns in the image were identified and now such triplets (from the list of all Finder Patterns) must be selected, which can represent 3 corners of a QR Code. Matrix of the distances between the centroid of all Finder Patterns is build and all 3-element combinations of all Finder Patterns are examined. For each triplet it is checked whether it is possible to construct a right-angled triangle from it so that the following conditions are met:


In this way, a list of QR Code candidates (defined by a triplet FP1, FP2, FP3) is built. However, such a candidate for a QR Code is selected only on the basis of the mutual position of the 3 FP candidates. As is shown in Figure 9 not all QR Code candidates are valid (dotted red FP3-FP3"-FP2" is false positive). These false positive QR Code candidates will be eliminated in the next steps.

**Figure 9.** QR Code candidates.

Finally, Bottom-Left and Top-Right Finder Pattern from the triplet (FP1, FP2, FP3) is determined by using formula:

If (FP3.x − FP2.x)(FP1.y − FP2.y) − (FP3.y − FP2.y)(FP1.x − FP2.x) < 0 then Bottom-Left is FP3 and Top-Right is FP1 else vice versa.

#### 3.3.2. Grouping Pairs of Finder Patterns

If any QR Code has one of the 3 Finder Patterns significantly damaged, then this Finder Pattern might not be identified and there remains two Finder Patterns in the Finder Patterns list, that were not selected (as the vertices of a right-angled triangle) in the previous step (Figure 10a). The goal of this step is to identify these pairs and determine the position of the third missing Finder Pattern. A square shape of the QR Code is assumed.

**Figure 10.** QR Code with one damaged Finder Pattern: (**a**) two known Finder Patterns; (**b**) two possible regions of QR Code; (**c**) two possible positions of 3rd Finder Pattern.

All two element combinations of remaining Finder Patterns, whose distance is in a predefined interval, are evaluated. A pair of Finder Patterns can represent Finder Patterns that are adjacent corners of a QR Code square (Figure 10a) or are in the opposite corners. If they are adjacent corners, then there are two possible positions of the QR Code (in Figure 10b depicted as a red square and green square). If they are opposite corners, then there are other two possible positions of the QR Code.

All four possible positions of the potential QR Code are evaluated against the following criteria:


Density of edge points is computed as the ratio of the number of edge points to area\*2/*MW*.

Region of a QR Code is expected to have relative balanced density of white and black points and relative balanced ratio of edges to area.

A square region that meets all the above conditions is considered a candidate for a QR Code. There are two possible corners in the QR Code bounding square where 3rd Finder Pattern can be located (Figure 10c). For both possible corners Finder Pattern match score is computed and one with better score is selected (in other words question, "In which corner is the structure that more closely resembles the ideal Finder Pattern?" must be answered). Match score is computed as

$$MS = \operatorname\*{argmin}(OS + \min(BS, WS))\tag{7}$$

where *MS* is match score (lower is better), *OS* is overall pattern match score, *BS* is black module match score and *WS* is white module match score. *BS* stores matches only for expected black points and *WS* stores matches only for expected white points between the mask and the image. *BS* and *WS* was introduced to handle situations when over the area of Finder Pattern is placed black or white spot, which would cause a low match score if only a simple pattern matching technique were used.

The match score is computed for several Finder Pattern mask positions by moving the mask in a spiral from its initial position up to a radius of *MW* with a step of *MW*/2 (for the case of small geometric deformations of the QR Code).

#### *3.4. Verification of Quiet Zone*

According to ISO standard a Quiet Zone is defined as "a region 4X wide which shall be free of all other markings, surrounding the symbol on all four sides". So, it must be checked if there are only white points in the image in the rectangular areas wide 1*MW* which is parallel to line segments defined by FP1–FP2 and FP2–FP3 (Figure 11). For fast scanning of the rectangle points Bresenham's line algorithm is utilized [20].

**Figure 11.** Quiet Zones.

QR Code candidates which do not have quiet zones around are rejected. Rejected are also QR Code candidates whose outer bounding box (larger) contains outer bounding box (smaller) of another QR Code candidate.

#### *3.5. QR Code Bounding Box*

Centroids of the 3 Finder Patterns (which represent the QR Code candidate) are the vertices of the triangle FP1–FP2–FP3. This inner triangle must be expanded to outer triangle P1–P2–P3, where the arms of the triangle pass through boundary modules of the QR Code (Figure 12). For instance, the shift of FP3 to P3 may be expressed as

$$\mathbf{P}\_3 = \mathbf{F}\mathbf{P}\_3 + \frac{\mathbf{F}\mathbf{P}\_3 - \mathbf{F}\mathbf{P}\_1}{|\mathbf{F}\mathbf{P}\_3\mathbf{\angle}\mathbf{F}\mathbf{P}\_1|}\mathbf{M}\mathbf{W}\sqrt{\mathbf{1}\mathbf{8}}\tag{8}$$

where *MW* is module width (Equation (6))

**Figure 12.** Bounding Box.

#### *3.6. Perspective Distortion*

For perspective undistorted QR Codes (only shifted, scaled, rotated, or sheared) it is sufficient to have only 3 points to set-up the affine transformation from a square to destination parallelogram. However, for perspective (projective) distorted QR Codes 4 points are required to set-up perspective transformation from a square to destination quadrilateral [21].

Some authors (for example [7,22]) search for Alignment Pattern to obtain 4th point. However, version 1 QR Codes does not have Alignment Pattern, so we have decided not to rely on Alignment Patterns.

*Appl. Sci.* **2020**, *10*, 7814

Instead of that we use an iterative approach to find the opposite sides to P2–P1 and P2–P3, which aligns to QR Code borders.

	- a. If L3 was whole in the QR Code and the shifted L3 is out of the QR Code, then initial estimation of P4 is good and we end.
	- b. If L3 was whole in the QR Code and 3rd third of the shifted L3 is again in the QR Code, then we continue by step 5.
	- c. If 3rd third of L3 was in quiet zone and 2nd and 3rd third of the shifted L3 is in the quiet zone or if 2nd third of L3 was in the quiet zone and 1st and 2nd third of the shifted L3 is in the quiet zone then we continue by step 6.

**Figure 13.** Perspective distortion: (**a**) initial estimate of the point P4 and lines L1, L3; (**b**) first shift of the line L3; (**c**) second shift of the line L3.

Once the position of 4th point, P4, is obtained, perspective transformation from the source square representing the ideal QR Code to destination quadrilateral representing the real QR Code in the image can be set-up (Figure 14).

**Figure 14.** Perspective transformation.

Using equations [21]:

*<sup>u</sup>* = *ax*+*by*+*<sup>c</sup> gx*+*hy*+<sup>1</sup> , *<sup>v</sup>* <sup>=</sup> *dx*+*ey*<sup>+</sup> *<sup>f</sup> gx*+*hy*+<sup>1</sup> , where the transformation coefficients can be calculated from the coordinates of the points P1(*u*1, *v*1), P2(*u*2, *v*2), P3(*u*3, *v*3), P4(*u*4, *v*4) as

$$\begin{aligned} a &= (\mathfrak{u}\_3 - \mathfrak{u}\_2)/A + \mathfrak{g}\mathfrak{u}\_3, \ b = (\mathfrak{u}\_1 - \mathfrak{u}\_2)/A + \mathfrak{h}\mathfrak{u}\_1, \ c = \mathfrak{u}\_2 \\ d &= (\mathfrak{v}\_3 - \mathfrak{v}\_2)/A + \mathfrak{g}\mathfrak{v}\_3, \ c = (\mathfrak{v}\_1 - \mathfrak{v}\_2)/A + \mathfrak{h}\mathfrak{v}\_1, \ f = \mathfrak{v}\_2 \\ \hline \quad d\mathfrak{u}\_3 &\quad d\mathfrak{u}\_2 &\quad \left| \begin{array}{c} d\mathfrak{u}\_1 & \mbox{d}\mathfrak{u}\_3 \\ d\mathfrak{v}\_1 & d\mathfrak{v}\_2 \\ d\mathfrak{v}\_1 & d\mathfrak{v}\_2 \\ \hline \end{array} \right|}{\operatorname\*{d}\mathfrak{u}\_1 \quad d\mathfrak{v}\_2}, \ h = \left| \begin{array}{c} d\mathfrak{v}\_1 & \mbox{d}\mathfrak{v}\_3 \\ d\mathfrak{u}\_1 & d\mathfrak{v}\_2 \\ d\mathfrak{v}\_1 & d\mathfrak{v}\_2 \\ \hline \end{array} \right|} \\ \hline \end{aligned}$$

$$\begin{aligned} d\mathfrak{u}\_1 &= (\mathfrak{u}\_3 - \mathfrak{u}\_2)A, \ d\mathfrak{u}\_2 = (\mathfrak{u}\_1 - \mathfrak{u}\_4)A, \ d\mathfrak{u}\_3 = \mathfrak{u}\_2 - \mathfrak{u}\_3 + \mathfrak{u}\_4 - \mathfrak{u}\_1 \\ d\mathfrak{v}\_1 &= (\mathfrak{v}\_3 - \mathfrak{v}\_2)A, \ d\mathfrak{v}\_2 = (\mathfrak{v}\_1 - \mathfrak{v}\_4)A, \ d\mathfrak{v}\_3 = \mathfrak{v}\_2 - \mathfrak{v}\_3 + \mathfrak{v}\_4 - 1 \end{aligned}$$

It sometimes happens, that the estimate of P4 position is not quite accurate so we move P4 in spiral from its initial position (obtained in previous step) and we calculate match score of bottom-right Alignment Pattern (Alignment Pattern exists only in QR codes version 2 and above). For each shift we recalculate coefficients for the perspective transformation, and we recalculate also match score. For the given version of the QR Code we know the expected position and size of bottom-right Alignment Pattern so we can calculate match between expected and real state. The final position P4 is the one with the highest match score.

Another alternative method how to handle perspective distorted QR Codes and how to determine position of the P4 point is based on edge directions and edge projections analysis [23].

#### *3.7. Decoding of a QR Code*

A QR Code is 2D square matrix in which the dark and light squares (modules) represent bits 1 and 0. In fact, each such a module on the pixel level is usually made up of cluster of adjacent pixels. In a QR Code decoding process, we have to build a 2D matrix that has elements with a value of 1 or 0 (Figure 15).

**Figure 15.** Transformation of the QR Code into the binary grid.

The pixels, where the brightness is lower than the threshold, are declared as 1 and the others are declared as 0. When analyzing such a cluster of pixels (module), the central pixel of the module plays a decisive role. If the calculated position of the central pixel does not align to integer coordinates in the image, the brightness is determined by bilinear interpolation.

Once the binary matrix of 1 and 0 is created, the open-source ZBar library [24] can be used to start the final decoding of the binary matrix and to receive the original text encoded by the QR Code.

#### **4. Results**

We used a test dataset of 595 QR Code samples to verify the method described in this paper. The testing dataset contained 25 artificial QR codes of different sizes and rotations, 90 QR Codes from Internet images, and 480 QR Codes from a specific industrial process. Several examples of the testing samples are in Figure 16.

**Figure 16.** Testing samples: (**a**) artificial, (**b**) Internet, (**c**) industrial.

In Table 1 and Figure 17 there are our results compared to competing QR Code decoding solutions (commercial and also open-source). In the table are the numbers of correctly decoded QR Codes from the total number of 595 QR Codes.


**Table 1.** Comparison of competing solutions with our proposed method.

**Figure 17.** Comparison of the recognition rate of our method against competing solutions.

Our method successfully detected all QR Codes but failed to decode three samples. Two samples were QR Codes placed on a bottle, where perspective and cylindrical distortions were combined. How to deal with this type of combined distortion is a challenge for future research.

As the commercial solutions have closed source code, we performed the testing using the black-box method. We have compiled our own QR Code test dataset (published together with this article under "Supplementary Material") to evaluate and compare our method, as a standardized universal dataset is not publicly available.

In Table 2 the computational complexity of our algorithm is compared to competing open-source solutions (commercial solutions were tested online). Our algorithm was implemented in Free Pascal and tests were run on an i5-4590 3.3GHz CPU (Intel Corporation, Santa Clara, CA, USA).

**Table 2.** Dependence of computational complexity on image resolution and number of QR codes in an image.


We see the main contribution of our method in the way the broken Finder Pattern is dealt with (Section 3.3.2) and how the QR Code bounding box is determined (Section 3.6), especially for perspective distorted QR Codes. The presented method can still locate a QR Code if one of the three Finder Patterns is significantly damaged. Consecutive tests in the real manufacturing process showed that this situation occurs much more often than a situation where two or three opposite Finder Patterns are damaged. In order to detect a QR Code with multiple damage Finder Patterns, it will be necessary to combine Finder Pattern based localization with the region based localization.

#### **5. Conclusions**

We have designed and tested a computationally efficient method for precise location of 2D QR Codes in arbitrary images under various illumination conditions. The proposed method is suitable for low-resolution images as well as for real time processing. The designed Finder Pattern based localization method uses three typical patterns of QR Codes to identify three corners of QR Codes in an image. If one of the three Finder Patterns is so destroyed that it cannot be localized, we have suggested a way to deal with it. The input image is binarized and scanned horizontally to localize the Finder Pattern candidates, which are subsequently verified in order to localize the raw QR Code region. For distorted QR Codes, the perspective transformation is set-up by gradually approaching the boundary of the QR Code.

This method was validated on the testing dataset consisting of a wide variety of samples (synthetic, real world, and specific industrial samples) and it was compared to competing software. The experimental results show that our method has a high detection rate.

The application of QR Codes and their optical recognition has wide use in identification, tracing or monitoring of items in production [30], storing and distribution processes, to aid visually impaired and blind people, to let autonomous robots [31] to acquire context-relevant information, to support authorization during log-in process, to support electronic payments, to increase industrial production surety factor [32], etc.

In cases where is required to place the 2D matrix codes on a very small area it may be preferable to use Data Matrix codes [33].

**Supplementary Materials:** The following are available online at http://www.mdpi.com/2076-3417/10/21/7814/s1, one ZIP file contains images of QR Codes testing dataset used to evaluate the competing solutions.

**Author Contributions:** Conceptualization, methodology, software, writing—original draft preparation, L.K.; validation, writing—review and editing, visualization, E.P.; supervision, project administration, funding acquisition, P.B.; All authors have read and agreed to the published version of the manuscript.

**Funding:** The article is funded of the research project KEGA 013TUKE-4/2019 "Modern educational tools and methods for shaping creativity and increasing the practical skills and habits of graduates of technical departments of universities".

**Acknowledgments:** The contribution is sponsored by the project.

**Conflicts of Interest:** The authors declare no conflict of interest.

## **References**


**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
