Object/Scene Recognition Based on a Directional Pixel Voting Descriptor

Abstract

Detecting objects in images is crucial for several applications, including surveillance, autonomous navigation, augmented reality, and so on. Although AI-based approaches such as Convolutional Neural Networks (CNNs) have proven highly effective in object detection, in scenarios where the objects being recognized are unknow, it is difficult to generalize an AI model for such tasks. In another trend, feature-based approaches like SIFT, SURF, and ORB offer the capability to search any object but have limitations under complex visual variations. In this work, we introduce a novel edge-based object/scene recognition method. We propose that utilizing feature edges, instead of feature points, offers high performance under complex visual variations. Our primary contribution is a directional pixel voting descriptor based on image segments. Experimental results are promising; compared to previous approaches, ours demonstrates superior performance under complex visual variations and high processing speed.

1. Introduction

Detecting objects in images is crucial in multiple applications, including surveillance, autonomous navigation, augmented reality, and so on [1,2,3,4]. While AI-based approaches such as Convolutional Neural Networks (CNNs) have demonstrated high performance in object detection [5,6], in scenarios where object/scene being recognized is unknown it is difficult to achieve accurate recognition. i.e., without prior training data on that specific object, traditional AI methods reach poor performance.

On the other hand, feature-based approaches like SIFT [7], SURF [8], and ORB [9] offer capability to search any template but have limitations under complex visual variations. These methods extract distinctive features from an image and use them to detect objects [10,11]. By carefully selecting and combining these feature-based techniques, developers can achieve efficient object detection solutions without compromising performance.

Another approach to object detection involves high-level abstraction descriptors such as those based on edges and shapes. These methods combine the advantages of both CNNs and feature-based techniques [1,12]. They aim to be more robust than feature-based approaches, akin to CNNs, and at the same time they can maintain relatively simple complexity allowing to search any template, akin to feature-based methods.

In this work, we present an edge-based object/scene recognition method. We propose that the utilization of shapes obtained from edge points should achieve high performance given any type of template. Our contribution lies in the introduction of a novel directional pixel voting descriptor using image segments. The experimental results demonstrated high performance under complex visual variations. Furthermore, our methodology enables both recognition of both objects and scenes. This dual capability opens up promising avenues for future exploration and practical real-world applications.

1.1. The Object Recognition Problem

The object recognition problem is a key challenge in computer vision, with applications in several domains, such as robotics, surveillance, and augmented reality. Object recognition involves identifying and categorizing objects within digital images or video frames. This task is particularly challenging due to the variability in object appearance, including differences in viewpoint, scale, lighting conditions, occlusions, and so on.

Addressing the object recognition problem typically involves developing algorithms and techniques capable of extracting discriminative features from visual data, followed by classification or matching processes to determine the identity or category of detected objects. Traditional approaches, such as SIFT, SURF, or ORB, were designed to capture distinctive characteristics of objects, as shown in Figure 1. However, these methods offer poor accuracy under complex visual variations and required extensive tuning for optimal performance across different scenarios [13,14].

1.2. The Scene Recognition Problem

Scene recognition is a key issue in several computer vision applications such as robotics and autonomous driving. In recent years, deep learning-based approaches have demonstrated high performance for scene recognition. In particular, convolutional Neural Networks (CNNs) demonstrated remarkable capabilities in learning representations of scenes directly from raw pixel data, enabling accurate scene categorization across diverse datasets and scenarios [15,16,17,18].

Unlike object recognition, which focuses on identifying individual objects within images, scene recognition involves understanding and categorizing entire scenes or environments depicted in images or video frames. This task presents its own set of challenges, primarily regarding the interaction on numerous objects, textures, structures, and spatial arrangements within a scene, see Figure 2.

Scene recognition algorithms must consider a wide range of factors, including variations in scene composition, lighting conditions, viewpoints, occlusions, and clutter. In addition, scenes often exhibit semantic context and spatial relationships between objects, further complicating the recognition process. Overcoming these challenges involves the development of complex computational models capable of extracting robust visual features from scene representations and effectively capturing the contextual information suitable for accurate scene categorization.

In the current literature, scene recognition approaches consist of techniques such as bag-of-visual-words, spatial pyramid matching, and deep learning architectures like Convolutional Neural Networks (CNNs). These methods aim to encode both local and global spatial information, enabling robust recognition across diverse scene categories. However, achieving reliable scene recognition performance is not possible, with opportunities for innovation in feature representation, model design, and training strategies [19,20,21,22].

2. Related Works

Several works which addressed both feature-based and CNN-based approaches for object and scene recognition have been presented. Researchers have explored several feature extraction techniques, such as SIFT, SURF, ORB, and their variants, to capture discriminative information from visual data. Furthermore, multiple works have focused on the utilization of deep learning architectures, particularly CNNs. These works have contributed to advancing the state-of-the-art in object and scene recognition, offering insights into effective feature representation, model architecture design, and optimization strategies.

2.1. State of the Art for Object Recognition

In current state of the art in object recognition, deep-learning-based approaches, particularly Convolutional Neural Networks (CNNs), have demonstrated high performance and flexibility in terms of application domain. Models like ResNet, EfficientNet, and DenseNet have laid down the boundaries of object recognition accuracy and efficiency. In [23], the authors combine both object detection and image dehazing methods for real-time applications such as remote sensing, video surveillance, driverless automatic vehicles, etc. They presented an effective and efficient image dehazing method using transfer learning, which helps recognize objects in real time with more clarity and can automatically detect objects with a high recognition rate and lower probability of error. The trained model AOD-net + YOLO v3 consistently outperforms non-joint and naive YOLO v3 techniques. Ref. [24] proposed a recurrent CNN (RCNN) for object recognition by incorporating recurrent connections into each convolutional layer. Though the input is static, the activities of RCNN units evolve over time so that the activity of each unit is modulated by the activities of its neighboring units. This property enhances the model’s ability to integrate context information, which is important for object recognition. Like other recurrent neural networks, unfolding the RCNN through time can result in an arbitrarily deep network with a fixed number of parameters. Although both approaches allow high robustness in terms of detection, both algorithms were based on machine learning, and the accuracy under unknown objects, different than those in the training, limits the scope and performance.

Furthermore, transfer learning techniques demonstrated high performance with pre-trained CNN models on large-scale datasets such as ImageNet for downstream object recognition tasks with limited labeled data. This approach allows for rapid development and deployment of object recognition systems in various domains. In [25], the use of transfer learning, via deep Convolutional Neural Networks (CNN), for the image classification problem within the context of X-ray baggage security screening was presented. The proposed algorithm achieved 98.92% detection accuracy, outperforming previous work, and further extended their evaluation to a multiple object classification task within this context. Additionally, in [26], multiple CNN architectures were proposed to improve the system performance. In this context, AlexNet, VGG16, and VGG19 are popular CNNs which allow transfer learning to fine-tune the pre-trained network parameters (VGG19) for image classification. Performance analysis demonstrated that the fine-tuned VGG19 architecture outperforms other CNN architectures and hybrid learning approaches for image classification tasks.

While deep learning models achieved high performance in several domains, feature-based approaches like SIFT and SURF still find applications in scenarios with limited computational resources or when there are specific challenges such as small object detection or texture-rich environments. Feature-based methods such as Scale-Invariant Feature Transform (SIFT) and Speeded-Up Robust Features (SURF) are still relevant in object recognition tasks. These feature-based approaches offer distinct advantages, including robustness to variations in scale, rotation, and illumination. In [27], a new tactile-SIFT descriptor is proposed to extract features based on gradients in tactile images to represent objects, allowing the features to be invariant to object translation and rotation. The tactile-SIFT segments a tactile image into overlapping subpatches, each represented using a dn-dimensional gradient vector, similar to the classic SIFT descriptor. Tactile-SIFT descriptors obtained from multiple touches form a dictionary of k words, and the bag-of-words method is then used to identify objects. The proposed method was validated by classifying 18 real objects with data from an off-the-shelf tactile sensor, achieving a recognition rate of 91.33% with a dictionary size of 50 clusters using only 15 touches. Additionally, in [28], the authors have compared the performance of the Shi-Tomasi corner detector with SIFT and SURF feature descriptors and evaluated the performance of Shi-Tomasi in combination with SIFT and SURF feature descriptors. In both cases, relatively high accuracy in terms of detection was achieved, motivating us to continue this trend with an edge-based approach which increases the robustness of traditional approaches (SIFT/SURF/ORB).

Finally, hybrid approaches that combine deep learning with traditional feature-based methods have gained traction, aiming to maintain the strengths of both paradigms. For instance, deep learning models can be augmented with features extracted using SIFT or SURF, providing complementary information and improving overall recognition accuracy. For example, in [29], a technique was proposed using deep convolutional neural network (DCNN) and scale invariant features transform (SIFT). First, an improved saliency method is implemented, and the features points are extracted. Then, DCNN features are extracted from two deep CNN models like VGG and AlexNet. The proposed method is evaluated on three public datasets including Caltech101, Barkley 3D, and Pascal 3D and reached classification accuracy of 93.8%, 99%, and 88.6%—it demonstrated high performance compared to previous works.

2.2. State of the Art for Scene Recognition

For scene recognition, deep learning-based approaches remain at the forefront of the current state of the art, offering significant advancements in accuracy and robustness. Convolutional Neural Networks (CNNs) have emerged as the backbone of several recognition systems, capable of automatically learning visual features from images. One notable trend in scene recognition is the integration of contextual information and spatial relationships between objects within a scene [30,31].

Finally, the fusion of multi-modal data sources, such as images, textual descriptions, and semantic annotations, have demonstrated high performance for scene recognition. Techniques such as multi-modal fusion networks and cross-modal retrieval provide complementary information from different modalities in order to improve the scene understanding [32].

3. The Proposed Algorithm

In Figure 3, an overview of our algorithm is shown. Our approach consists of an image preprocessing step, in which input images are filtered and some primitive information such as image gradients $(g_x,g_y,d)$ are extracted. Then, in a second step, relevant patterns such as edge segments s are extracted and described using our directional pixel voting descriptor. Finally, the input image is compared with a target (template), which can be an object or a scene, to determine whether it can be recognized or not.

3.1. Image Preprocessing

Given a gray scale image $I(i,j)$, a Gaussian filtering is applied on I image on order to reduce noise and to remove fine-scale structures that affect the performance of the feature extraction step. The Gaussian filter is defined by Equation (1) as follows:

$$G(m,n)={\displaystyle \frac{1}{2\pi {\sigma }^2}}{e}^{-{\displaystyle \frac{m^2+n^2}{2{\sigma }^2}}}$$

(1)

where $G(m,n)$ is the Gaussian function and $\sigma$ is the standard deviation, which controls the amount of smoothing applied to the image.

To perform Gaussian filtering on the image I, a convolution operation is applied between the image I and a Gaussian kernel G. The Gaussian kernel is usually represented as a odd size 2D matrix or mask. For a generic $5\times 5$ ($N\times N$) Gaussian mask, it can be represented as:  

$${\displaystyle \frac{1}{273}}\left(\begin{array}{ccccc}1& 4& 7& 4& 1\\ 4& 16& 26& 16& 4\\ 7& 26& 41& 26& 7\\ 4& 16& 26& 16& 4\\ 1& 4& 7& 4& 1\end{array}\right)$$

The above matrix represents the coefficients of the Gaussian mask, which are computed based on the Gaussian function with an appropriate standard deviation. This mask is then applied to the image using convolution, resulting in a smoothed version of the original image.

$$I_s(i,j)=\sum _{m=-{\displaystyle \frac{N-1}{2}}}^{{\displaystyle \frac{N-1}{2}}}\sum _{n=-{\displaystyle \frac{N-1}{2}}}^{{\displaystyle \frac{N-1}{2}}}I(i+m,j+n)·G(m+{\displaystyle \frac{N-1}{2}},n+{\displaystyle \frac{N-1}{2}})$$

Given the smoothed image $I_s(i,j)$, the gradients $g_x$ and $g_y$ are calculated via Prewitt operator [33] as follows:

$$g_x(i,j)=(-I_s(i-1,j-1)-I_s(i,j-1)-I_s(i+1,j-1)+I_s(i-1,j+1)+I_s(i,j+1)+I_s(i+1,j+1))$$

$$g_y(i,j)=(-I_s(i-1,j-1)-I_s(i-1,j)-I_s(i-1,j+1)+I_s(i+1,j-1)+I_s(i+1,j)+I_s(i+1,j+1))$$

Finally, the direction $d(i,j)$ is determined as follows:

$$d(i,j)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}(g_x(i,j)>g_y(i,j))\&(g_x(i,j)+g_y(i,j))\ge \mathrm{th}\hfill \\ 2\hfill & \mathrm{if}(g_x(i,j)\le g_y(i,j))\&(g_x(i,j)+g_y(i,j))\ge \mathrm{th}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(2)

where $\mathrm{th}$ is a threshold for a “valid” gradient value. We define the possible values of $\mathrm{th}$ inspired by the image bit-depth standard format, i.e., 8, 16, 32, 64, and so on. We propose $\mathrm{th}=32$ as a threshold because, even after the smoothing process using the Gaussian filter, certain levels of noise and fine details may persist in the image. These artifacts can negatively affect the performance of subsequent steps, such as feature extraction or object recognition. By setting a threshold equal to 32, we aim to identify and suppress gradients that are not significant enough to represent relevant features in the image. Gradient values below this threshold may be considered as unwanted noise or fine details that we aim to eliminate. Note that the threshold value $\mathrm{th}$ does not explicitly depend on the bit depth of the image. However, given that the gradients $g_x$ and $g_x$ are computed from the absolute values of the image, which inherently depend on its bit depth, it is reasonable to argue that the sum $g_{xy}$ should be compared to a threshold $\mathrm{th}$ that scales with the maximum gray level of the image. Specifically, a fixed threshold value of 32 might be appropriate for images with 8-bit depth (gray values ranging from 0 to 255), but could be inadequate for images with higher bit depths, such as 16-bit images with gray values ranging from 0 to 65,535.

3.2. Feature Extraction and Description

Our main contributions, on the one hand, consist of introducing a novel approach for feature (segments) extraction and, on the other hand, presenting a robust description criterion.

3.2.1. Feature Extraction

As mentioned before, our approach is based on edge primitives. Although there are existing algorithms, such as Sobel and Canny, for that purpose, they are sensitive to noise or difficult to segment properly. To address those issues, we propose a new extraction method as follows, where the edge map $e(i,j)$ (see Figure 4) is defined as follows:

$$e(i,j)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\left(({\mathrm{dif}}_1\ge \mathrm{th}\&{\mathrm{dif}}_2\ge \mathrm{th})\right)\&d(i,j)==1\hfill \\ 1\hfill & \mathrm{if}\left(({\mathrm{dif}}_3\ge \mathrm{th}\&{\mathrm{dif}}_4\ge \mathrm{th})\right)\&d(i,j)==2\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(3)

where ${\mathrm{dif}}_1$, ${\mathrm{dif}}_2$, ${\mathrm{dif}}_3$, ${\mathrm{dif}}_4$, and $g_{xy}(i,j)$ are defined as follows:   $g_{xy}(i,j)=g_x(i,j)+g_y(i,j)$

• ${\mathrm{dif}}_1=g_{xy}(i,j)-g_{xy}(i,j-1)$,
• ${\mathrm{dif}}_2=g_{xy}(i,j)-g_{xy}(i,j+1)$,
• ${\mathrm{dif}}_3=g_{xy}(i,j)-g_{xy}(i-1,j)$,
• ${\mathrm{dif}}_4=g_{xy}(i,j)-g_{xy}(i+1,j)$.  

Figure 4. Let (a) be the input image template an edge map is computed via Equation (3) given the resulting image presented in (b).

Figure 4. Let (a) be the input image template an edge map is computed via Equation (3) given the resulting image presented in (b).

Then, the set of anchor points $f\left(k\right)$ is computed via Algorithm 1.

Algorithm 1: Constructing anchor points vector $\mathit{f}\left(\mathit{k}\right)$

Initialize vector f as empty. Initialize $k=0$. For each pair of coordinates $(i,j)$ in the edge image $e(i,j)$: (a)  If $e(i,j)==1$: i.  $k\leftarrow k+1$ ii.  Add coordinate $\{i,j\}$ to vector f at position k. Return vector f.

3.2.2. Segment Generator (Feature Extraction)

To generate “robust” edge segments, we propose an approach inspired on the canonical seed-segmentation method [34,35]. In this method, a seed is selected to initiate the segmentation process. This seed serves as the starting point for segment expansion. The segmentation algorithm iteratively grows the segment by incorporating neighboring pixels that satisfy certain criteria, such as similarity in color, texture, or intensity. This process continues until the segment reaches a predefined stopping criterion, ensuring that it encompasses a coherent region of interest. By leveraging the canonical seed-segmentation approach, we aim to achieve accurate and reliable edge segments that capture meaningful structures in the image data. This approach offers robustness against noise, variations in lighting conditions, and other sources of interference, making it well-suited for a wide range of image segmentation tasks.

Let $g_{xy}(i,j)$ be the gradient map ($g_{xy}(i,j)=g_x(i,j)+g_y(i,j)$), $d(i,j)$ be the direction map computed via Equation (2), as shown in Figure 5b, and $f\left(k\right)$ denote an anchor point $(i,j)$; these anchor points are grown horizontally or vertically depending on the direction image at anchor point $(i,j)$, as illustrated in Figure 6, where for each anchor $\left(f\right(k\left)\right)$, there are 5 $\left(n\right)$ possible growth directions. Each direction (marked by a red square) is evaluated using the gradient map $\left(g_{xy}(i,j)\right)$ as an edge criterion, where $\mathrm{Crl}\left(n\right)$ defines the correlation between the anchor point and the $n\text{-}\mathrm{th}$ growth direction. In practice, the growth direction could change from vertical to horizontal in the same segment, but this is supported by the diagonal element of the grown pattern, $\{p_2,p_4\},\{p_4,p_2\}$ Figure 6a. Therefore, when a change in direction occurs, it is addressed by the current growth pattern, and when the direction changes (in the directions image), the growth pattern is changed. All possible directions are ordered clockwise. This facilitates the software implementation by implementing shift vectors. Notice that all evaluated directions are disabled by resetting their corresponding value to zero. Finally, the segment grows in the direction with the highest edge correlation compared to the anchor, and only if this correlation is always greater than a threshold $\mathrm{th}$. The whole process is lay down in Algorithm 2. For practical purposes, we recommend a threshold value of 32. The whole process is illustrated in Figure 7; when the current pixel does not allow growth, the algorithm returns to the original anchor and starts to grow in the opposite direction. Finally, in Figure 8, the extracted features, ‘edge segments’, are shown.

Algorithm 2: Determining the segment image $\mathit{s}(\mathit{i},\mathit{j})$ and segment vectors ${\mathit{s}}_{\mathit{k}}$

Initialize $s(i,j)=0$ for all $(i,j)$. For each pair of coordinates $(i,j)$ in the image: (a)  If $s\left(f_k\right)==0$: i.  Initialize vector $s_k$ as empty. ii.  Set $s(i,j)=1$, add tuple $(i,j)$ to vector $s_k$. (b)  Else if $s\left(p_n\right)==0$: i.  While segment can grow:  If $s\left(p_n\right)==0$ (where n is the grown direction being evaluated):  Find $n={\arg \min }_n\mathrm{Crl}\left(n\right)$.  If $\mathrm{Crl}\left(n\right)\ge \mathrm{th}$:  Set $s(i,j)=1$, add grown point $(i,j)$ to vector $s_k$.  Else:  Break the loop. Return the segment image $s(i,j)$ and the segment vectors $s_k\{i,j\}$.

3.2.3. The Directional Pixel Voting Descriptor (Feature Description)

Let $p(i,j)$ represent a pixel inside a segment (it could be an anchor $f_k$ or any pixel added after the growing process, see Section 3.2.2). Each added pixel involves a global image direction, as illustrated in Figure 9a. There are eight possible image directions. Here, it is not possible to maintain our clockwise order, since at the global level the objective is to describe the shape of the segment. However, the global directions order can be superposed on the growth patterns presented in the previous section, as shown in Figure 9b. Let $d_1,d_2,\dots ,d_8$ represent the global image directions. A segment descriptor k (for the k-th segment) is defined as follows: $S_k\{d_1,d_2,\dots ,d_8,g_1,g_2,\dots ,g_8\}$, where $d_n$ represents the count of pixels inside the segment grown in the i-th direction and is computed via Equations (4)–(6) and is initialized to zero, $p_n$ is the global grown direction of the i-th pixel $s_k\{p{(i,j)}_1,p{(i,j)}_2,\cdots p{(i,j)}_i\}$ inside the k segment, see Algorithm 2, and N is the size of the segment vector. Moreover, $g_1,g_2,\dots ,g_8$ represent the normalized mean of the gradient values at each global grown direction. For this, we define Equations (7)–(9), where $g_{xy}$ is the gradient value of the i-th pixel $p(i,j)$ inside the segment, $g_{xy}(i,j)=g_x(i,j)+g_y(i,j)$, see Section 3.2.1, ${\displaystyle \frac{1}{\left|\mathrm{Max}\right(g_k\left)\right|}}$ is the normalization factor, and $\mathrm{M}$ is the operator for the mean value. This process is illustrated in Algorithm 3.

$$\begin{array}{c}\hfill S_k\left\{d_1\right\}=\sum _{i=1}^N{d}_1=d_1+1\mathrm{if}{s}_k\left(i\right)\mathrm{has}{p}_1\mathrm{grown}\mathrm{direction}\end{array}$$

(4)

$$\begin{array}{c}\hfill S_k\left\{d_2\right\}=\sum _{i=1}^N{d}_2=d_2+1\mathrm{if}{s}_k\left(i\right)\mathrm{has}{p}_2\mathrm{grown}\mathrm{direction}\\ \hfill ⋮\end{array}$$

(5)

$$\begin{array}{c}\hfill S_k\left\{d_8\right\}=\sum _{i=1}^N{d}_8=d_8+1\mathrm{if}{s}_k\left(i\right)\mathrm{has}{p}_3\mathrm{grown}\mathrm{direction}\end{array}$$

(6)

$$\begin{array}{cc}\hfill S_k\left\{g_1\right\}& ={\displaystyle \frac{1}{\left|\mathrm{Max}\right(g\left)\right|}}\times {\mathrm{M}}_{\mathrm{g}}\mathrm{k}>0\left(g\left(i\right)=\sum _{i=1}^N\left|g_{xy}\left(s_k\left(i\right)\right)\right|\mathrm{if}{s}_k\left(i\right)\mathrm{has}{p}_1\mathrm{grown}\mathrm{direction}0\mathrm{otherwise}\right)\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill S_k\left\{g_2\right\}& ={\displaystyle \frac{1}{\left|\mathrm{Max}\right(g\left)\right|}}\times {\mathrm{M}}_{\mathrm{g}}\mathrm{k}>0\left(g\left(i\right)=\sum _{i=1}^N\left|g_{xy}\left(s_k\left(i\right)\right)\right|\mathrm{if}{s}_k\left(i\right)\mathrm{has}{p}_2\mathrm{grown}\mathrm{direction}0\mathrm{otherwise}\right)\hfill \\ \hfill & ⋮\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill S_k\left\{g_8\right\}& ={\displaystyle \frac{1}{\left|\mathrm{Max}\right(g\left)\right|}}\times {\mathrm{M}}_{\mathrm{g}}\mathrm{k}>0\left(g\left(i\right)=\sum _{i=1}^N\left|g_{xy}\left(s_k\left(i\right)\right)\right|\mathrm{if}{s}_k\left(i\right)\mathrm{has}{p}_8\mathrm{grown}\mathrm{direction}0\mathrm{otherwise}\right)\hfill \end{array}$$

(9)

Algorithm 3: Creation of the directional pixel voting descriptor

Initialize $g\left(i\right)$ as empty for $k=1$ to 8. For each tuple $(i,j)$ in $s_k$: (a)  For $n=1$ to 8 (where n are the grown directions, see Figure 9a): i.  If $s_k(i,j)$ has $p_n$ grown direction:  $g\left(i\right)\leftarrow g\left(i\right)+|g_{xy}\left(s_k(i,j)\right)|$. ii.  Else:  $g\left(i\right)\leftarrow g\left(i\right)+0$. For $n=1$ to 8 (where n are the grown directions, see Figure 9a): (a)  Compute $S_k\left\{g_n\right\}={\displaystyle \frac{1}{\left|\mathrm{Max}\right(g\left)\right|}}\times \mathrm{Mean} \mathrm{of}g\left(i\right)\mathrm{where}g\left(i\right)>0$. Return $S_k\left\{g_n\right\}$ for $n=1$ to 8 (where n are the grown directions, see Figure 9a).

3.3. Recognition

Let $S_k$ be the descriptors for the template while $S_q$ be the descriptors on the query (the real-world scene in which the object/place is being looked at). Comparisons between the $S_k$ segment from template and the $S_q$ segment in query are carried out by the Hamming distances of the descriptor components as follows:

$$\mathrm{hammingDistance}=\sum _{i=1}^n|S_k-S_q|$$

(10)

where n is the number of components in the descriptor.

Of course, an all vs. all approach could be inconvenient in terms of processing speed and resources usage; to address this problem, we propose a look-up table approach based on the segment length, where each block represents a range of segment lengths. Thus, let $S_k$ be the segment being matched and $S_k$ inside the block 2; this segment is compared with each segment on the query table for the same block size. Of course, smaller length partitions will allow fast searches, since there will be only a few candidates for each size, but this could limit proper matching, since lengths between template and query could be a little different. On the other hand, huge length partitions will allow a proper match even with strong changes between template and query, but the search process will require high computational resources since a lot of candidates could be available. In practice, we recommend a partition as follows: $t{h}_1=8,t{h}_2=16,t{h}_3=32,\cdots ,t{h}_{n-1}=128$. The whole process is shown in Algorithm 4.

$$\mathrm{Crl}\left(i\right)=\sum _{i=1}^{i=n}\mathrm{hammingDistance}(S_k,S_i)$$

(11)

$$\mathrm{match}\left(i\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}{\mathrm{argMin}}_i\mathrm{Crl}\left(i\right)\le {\mathrm{th}}_{\mathrm{match}}\hfill \\ 0\hfill & \mathrm{if}\mathrm{otherwise}\hfill \end{array}\right.$$

(12)

where $\mathrm{match}\left(i\right)$ represents the match status, being zero if the segment i was not found in the query or 1 otherwise. ${\mathrm{th}}_{\mathrm{match}}$ is a similarity threshold which defines if a valid matching was found. We recommend values between 0.7 and 0.8. Finally, an object or scene is recognized only if a proper number of the i segments which define the object or scene were found. A proper number of matches could be greater than 70%. Of course, greater values could guarantee less recognition accuracy, but detection could not allow important changes (in terms of illumination, rotation, or scale) between template and query. On the other hand, lower values will allow easy detection with high changes between template and query but can generate false positives, i.e., wrong object or scene detections.

Algorithm 4: Object recognition

Initialize $\mathrm{hammingDistance}=0$. Group all segments $S_q$ into blocks based on their length: (a)  For each segment $S_q$: i.  If $S_{\mathrm{length}}\left(S_q\right)\le {\mathrm{th}}_1$:  Assign $S_q$ to block 1. ii.  Else if $S_{\mathrm{length}}\left(S_q\right)\le {\mathrm{th}}_2$:  Assign $S_q$ to block 2. iii.  Else if ⋯:  ⋯ iv.  Else:  Assign $S_q$ to block M. For each segment $S_k$: (a)  Identify block b to which $S_k$ belongs. (b)  For each segment $S_q$ in block b: i.  $\mathrm{hammingDistance}\leftarrow \mathrm{hammingDistance}+|S_k-S_q|$. Return hammingDistance.

4. Results

In this section, we present experimental results for the proposed algorithm compared with previous approaches. Since the use of standard datasets has become common in the computer vision area (for learning-based approaches), we looked for some proper datasets. However, since there is a lack of standardized datasets for image-based algorithms, previous works such as [13,14,27] used their own images, and this is the approach that we will follow here. Following this idea, we compare our algorithms with other canonical algorithms in the current literature, which have served as the basis for multiple works. These are SIFT [7], SURF [36], FAST [37], and ORB [9]. Multiple tests for object/scene detection and processing speed were carried out. For practical purposes, only some examples will be presented in this manuscript. In all cases, MatLab 2023b and a PC with an Intel(R) Core(TM) i9-9900K CPU 3.60 GHz, 32 GB RAM DDR3 were used (Intel, Santa Clara, CA, USA).

In Figure 10, results for object detections are shown. In all cases, accuracy is defined as the percentage of template features successfully detected. Since accuracy is higher than 70%, the object being recognized is found in all cases. Different rotations and scale variations between the template and query were tested. In all cases, accurate detection can be reached. In Figure 10a, two mismatches (outliers) were generated: in one case, the texture on the wall matched with a small pattern on the bottle. On the other hand, a line in the template was matched with a line in the query that is different from the line in the template. There is no global spatial information, which could be expected, since, following our descriptor, both lines’ matches present a similar orientation, length, and gradient (as both present similar contrasts between background and object, of which the feature owns). Furthermore, we can see that the “U” character was found two times. This is because we have not applied any non-maximal matched, and therefore, one feature in the template can be matched multiple times in the query. In this case, each “U” character in the template can be matched with each “U” character in the query, as expected. In Figure 10b,c, only one outlier was generated each. Finally, no outliers are presented in Figure 10d. In all cases, the square/mesh pattern on the bottle generated mismatches, but again, given our algorithm undertakes feature matching based on shape and color contrast against the features (gradients), it is expected that repetitive patterns can generate wrong matches. These problems can be addressed by adding global information so that each feature can be the same in terms of gradient or shape but different, for example, with respect to the Euclidean distance to the origin. Of course, this is only a hypothesis and will be explored in future work. In any case, these results demonstrate, at a high level, relatively accurate detection for object recognition. Regarding performance under complex visual variations in In Figure 11, results for object detection are shown. In all cases, accuracy is defined as the percentage of template features that are successfully detected. Accuracy higher than 70% is reached, and therefore, accurate/robust detection is reached for objects. In Figure 11a, changes in rotation and objects with similar shapes were tested. In all cases, accurate detection can be reached. Even with rotations such as the one presented in the black cylinder and the wood box, our descriptor can properly match these objects. This demonstrates the rotational invariance of our descriptor. In Figure 11b, the complex texture scenario was tested, which demonstrated the robustness of the proposed method; shapes such as circles, cuboids, and so on and complex segments such as a book cover were correctly detected. Again, rotational invariance is demonstrated by the algorithm’s ability to properly match segments inside the book cover.

In Figure 12, the performance for other methods in the literature, in particular, the original SIFT [7] with non-maxima suppression (i.e., one feature on the template (left side), which can have multiple matches in the query (right side)) and original SURF [36], is shown. As can be seen, by comparing only feature points or blobs, low performance is achieved. In both cases, several mismatches or multiple matches for the same feature are generated. This demonstrates the low performance that point information can generate. On the other hand, our approach in Figure 10 and Figure 11, which is based on shape and gradient, involves higher robustness for the matching process. This is shown in

Table 1. Mean accuracy for the proposed algorithm compared with previous approaches in the literature.

Method	Accuracy
SIFT [7]	43%
SURF [36]	69%
FAST [37]	60%
ORB [9]	67%
This work	81%

, in which average accuracy values by considering multiple test for several templates are presented.

In Figure 13 and Figure 14, results for scene recognition are shown. Again, when compared to our approach (Figure 13) with the original SIFT (Figure 14), our approach clearly outperforms previous work. In our work, most important mismatches are generated for some cloud patterns in the template which match the tree patterns in the query. Again, this is as expected, since matched patterns have the same shape and gradient. On the other hand, SIFT generated several mismatches, and in this case, it could be difficult to determine if the scene was recognized or not.

Finally, in

Table 2. Processing speed for the proposed algorithm compared with previous approaches in the literature. Input image resolution 2160 × 1080 (Full HD).

Method	Processing Time (fps)
SIFT [7]	1.0044
SURF [36]	3.6597
FAST [37]	3.9117
ORB [9]	0.3820
This work	0.5726

and

Table 3. Processing speed for the proposed algorithm compared with previous approaches in the literature. Input image resolution 1080 × 540 (HD).

Method	Processing Time (fps)
SIFT [7]	3.3245
SURF [36]	11.4322
FAST [37]	11.4230
ORB [9]	1.0328
This work	2.0012

, processing speed comparisons are presented. Our approach, which consists of complex features (compared with points which are the features used by previous approaches), requires more processing time. However, even with this complexity, our algorithms outperform the original ORB. Considering that algorithms like ORB have been successfully used in real-time applications such as SLAM [38], robotics [39], and so on, we consider our approach can be successfully used. Note that our algorithm was implemented in MATLAB, without hardware acceleration or advanced programming patterns, while the other algorithms have been highly improved in terms of implementation in order to reach optimal processing speed. We believe our approach can be improved in terms of implementation, and this should increase the processing speed. This will be addressed in future work.

Limitations: Since the direction in the proposed Directional Pixel Voting Descriptor is defined in the 2D image pixel coordinate, rotations which deforms or occlude primitive shapes in template are not supported, as shown in Figure 15a. However, even classical methods such as SIFT or SURF suffer from this in the case of textureless objects Figure 15b.

Asa final discussion, experimental results demonstrate the effectiveness and robustness of our proposed approach for object and scene recognition tasks. We have demonstrated that our method outperforms previous approaches, particularly in scenarios with variations in rotation, scale, and texture. Despite the increased computational complexity of our method compared to traditional point-based approaches, our algorithms achieve promising performance, laying down the basis for future improvements and applications in real-world applications. Additionally, while our implementation in MATLAB provides a solid implementation, future optimizations and hardware accelerations are expected to further improve processing speed and efficiency. We believe that our work allows for improvements such as additional constraints for the descriptor previously laid down, non-maximal suppression methods (to avoid multiple matches for some features in the template and query), and so on.

5. Conclusions

The importance of object detection under several applications, including surveillance, autonomous navigation, and augmented reality, cannot be overstated. While AI-based approaches like Convolutional Neural Networks (CNNs) have demonstrated high performance in object detection, there are limitations for unknown objects being detected. Feature-based methods such as SIFT, SURF, and ORB support unknown object detection but have limitations under visual variations. In response, this work presented a novel edge-based object/scene recognition technique. Our proposed directional pixel voting descriptor, based on image segments, represents a significant advancement in this domain. Experimental results demonstrated high performance compared with previous works in the current literature. In future work, improvements such as additional constraints for the descriptor previously laid down, non-maximal suppression methods (to avoid multiple matches for some features in the template and query), optimizations, and hardware accelerations will be implemented in order to improve processing speed and efficiency.