DTFS-eHarris: A High Accuracy Asynchronous Corner Detector for Event Cameras in Complex Scenes

Abstract

The event camera, a new bio-inspired vision sensor with low latency and high temporal resolution, has brought great potential and demonstrated a promising application in machine vision and artificial intelligence. Corner detection is a key step of object motion estimation and tracking. However, most existing event-based corner detectors, such as G-eHarris and Arc*, lead to a huge number of redundant or wrong corners, and cannot strike a balance between the accuracy and real-time performance, especially in complex scenes with high texture that require higher computational costs. To address these issues, we propose an asynchronous corner detection method: a double threshold filter with Sigmoid eHarris (DTFS-eHarris) and an asynchronous corner tracker. The main contributions are that a double threshold filter is designed to reduce the redundant events and the improved Sigmoid function is utilized to represent the Surface of Active Events ($Sigmoi{d}^{*}$-SAE). We selected four scenes—shapes, dynamic, poster and boxes—from the public event camera dataset DAVIS240C to compare with the existing state-of-the-art hybrid method; our method has shown more than a 10% reduction in false positive rate and a 5% and 20% improvement in accuracy and throughput, respectively. The evaluations indicate that DTFS-eHarris shows a significant improvement, especially in complex scenes. Thus, it is anticipated to enhance the real-time performance and feature detection accuracy for future robotic applications.

1. Introduction

Recently, a new bio-inspired vision sensor, the event camera [1], has emerged and broke the limitation of the standard camera working principle and frame rate. Different from frame of traditional cameras, the event camera could respond to pixel-wise brightness changes in the scene asynchronously and independently.

Due to the low temporal resolution, high latency and highly redundant data, the frame-based camera performs poorly in complex scenes such as high-speed motion scenes and suffers from missed and incorrect corner detection tasks in complex scenes due to the motion blur and the low dynamic range of its output.

With the unique imaging mechanism, event cameras are low latency (in the order of microseconds), have low power consumption and have a very high dynamic range (120 dB) and very high temporal resolution. As such, event cameras can capture the scene changing rapidly without motion blur and effectively filter the background, which has brought about a new approach to machine vision and artificial intelligence.

Event cameras have been widely applied to various visual tasks such as optical flow estimation [2], simultaneous localization and mapping (SLAM) [3] and feature detection [4]. Feature detection is a basic and core component in machine vision tasks such as robot vision servoing [5] and autonomous driving [6]. As a special feature, corners show drastic grey changes in many directions, which makes them easily distinguishable from the neighbouring pixels. The shortcoming of traditional cameras is that they often result in missed and false detections of corners, introducing errors to higher-level vision tasks. Event cameras tend to show better performance in complex senses due to their characteristics. Therefore, how to extract corners from asynchronous event streams quickly and accurately is still an open problem. The main existing event-based corner detection algorithms are template-based methods [7], gradient-based methods [8] and hybrid methods [9].

High-resolution event cameras are helpful to obtain more detailed features in these tasks, but the existing higher-resolution event cameras, such as CeleX-V (1280 × 800) [10] and Samsung’s DVS Gen4 (1280 × 960) [11], can generate more than 10 million events per second, dozens of times more events than DAVIS240C which used in event camera datasets [12]; therefore, realizing real-time processing is still a problem.

Therefore, to obtain higher performance in complex scenes, it is necessary to explore a more effective method to remove redundant and noisy events as much as possible to reduce computational cost and save computing resources. To address the aforementioned problem, we propose an improved event-based corner detector and design a novel filter and a new local Surface of Active Events (SAE) representation method. The filter uses a double-threshold to remove redundant and noisy events, which is able to significantly improve throughput while maintaining performance. The proposed local SAE representation is a modification of the original Sigmoid function, which is computationally efficient and better reflects the spatio-temporal distribution of events than existing representations. We evaluate our method on public datasets and achieve better performance.

The main contributions of this paper are the following:

• A simple and effective event-based double threshold filter that reduces redundant and noisy events from the events stream effectively which has better performance than the previous filter and is suitable for the existing event-based corner detector;
• An efficient local SAE representation method called $Sigmoi{d}^{*}$-SAE which has lower computation and clearer spatio-temporal distribution;
• An asynchronous corner tracker which verifies the effectiveness of our detector.

The remainder of the paper is organized as follows. In Section 2, we present the related work about event-based feature extraction. Section 3 describes our proposed detector in detail. In Section 4, we describe the experimental design and evaluation results. In Section 5, we present a summary and look forward to future work.

2. Related Work

Since the development of event cameras, several works have been dedicated to extracting features from event streams and detecting and tracking features such as lines [13,14] and circles [15]. As a special feature, corner detection has also attracted many researchers.

2.1. Event Representation

For event-based corner detection tasks, SAE is a spatial-temporal representation of event-based data, which is used to distinguish new events from old events. SAE can be represented as a mapping function ${\mathbf{M}}_{\mathbf{e}}$ that maps pixels to its corresponding timestamps of the latest event in $\mathrm{x}={[x,y]}^T$, shown as Equation (1).

$$\begin{array}{cc}\hfill {\mathbf{M}}_{\mathbf{e}}& \hfill :{\mathbb{R}}^2\stackrel{}{\to }[0,\infty ],\hfill \\ \hfill \mathrm{x}& \hfill \mapsto {\mathbf{M}}_{\mathbf{e}}\left(\mathrm{x}\right)=t.\hfill \end{array}$$

(1)

The mapping of the whole scene is global SAE, which can be used for global perception. The mapping of local areas is called local SAE, which is used to extract local features. A typical SAE is shown in Figure 1, where positive polarity events are updated in SAE+ and negative polarity events are updated in SAE−.

Ideally, to obtain clearer gradient information, there should be a significant gradient between the timestamps of the new and old events on the local SAE. To better distinguish between new and old events, Li et al. [9] and Mohamed et al. [16] took a binary SAE and the latest $\mathcal{N}$ timestamps were assigned to 1 and the rest to 0. In addition, there are other SAE constructions such as min-max normalization [17] and exponential decay [18].

2.2. Frame-Based Corner Detector

Currently, frame-based corner detectors have been widely used in various fields. For example, Harris [19] detects corners by detecting grey-level changes around pixels. The core idea is that a small movement of the corner in any direction will cause great changes in the direction of the gradient in the selected area. That is, the pixel is a corner if the gradient of the grey image is a local maximum. As one of the most popular frame-based corner detectors, it has been widely used in chessboard calibration, image registration and others. Shi et al. [20] proposed the Shi-Tomasi detector, which modified the score function of the Harris algorithm and obtained a better result.

The Smallest Univalue Segment Assimilating Nucleus (SUSAN) [21] detector uses a circular sliding window template in which the grey value of each pixel within the sliding window is compared with the central pixel. If the difference in grey value between two pixels is less than a threshold, the pixel is identified as having a similar grey value to the central pixel. The area of pixels that meet this condition is called the Univalue Segment Assimilating Nucleus (USAN). The value of the USAN is then calculated and compared to a given threshold, and if the pixel’s USAN value is less than the threshold, it is identified as a SUSAN corner. This method has a lower number of parameters and is therefore more computationally efficient and more robust to noise.

Feature from Accelerated Segment Test (FAST) [22] draws a circle with a radius of 3 pixels around the selected centre pixel, and there are 16 pixels in the circle. The differences between these pixels and the selected centre pixel are calculated; if there are consecutive n (generally n = 12) pixels and the absolute value of the difference between the grey value of the centre pixel is greater than the threshold, the centre pixel is identified as the FAST corner feature.

2.3. Event-Based Corner Detector

The first event-based asynchronous corner detector uses plane fitting to detect corners and event characteristics [23]. Most of the subsequent corner detectors are based on Surface of Active Events (SAE) [24], which can effectively reflect the spatio-temporal information around the events. There are two main types of event-based corner detection algorithms, which are template-based (e.g., eFAST [7] and Arc* [25]) and gradient-based (e.g., eHarris [8]). They have their own strengths and weaknesses. In contrast, the template-based detector can detect corners faster but with low accuracy, while the gradient-based detector is more accurate but very time-consuming and difficult to detect in real-time. Therefore, the most popular method is to combine the two types of algorithms and take a trade-off.

2.3.1. Template-Based Corner Detector

The most representative template-based corner detector is eFAST [7], which detects the corners by finding continuous arcs on two circles, respectively. If the pixels which meet the conditions in each circle form a continuous arc, and the length of the arc is within the given range, the centre pixel is considered a corner. However, the corner must be less than 180 degrees to prevent incorrect detection results.

Alzugaray proposed an Arc* detector [25] which overcomes the defect of eFAST that fails to detect angles greater than 180 degrees. This was the first time a filter was used on Arc* to remove redundant events from the event stream in corner detection tasks. In their approach, the timestamps were stored in two global SAEs—the latest SAE and the filtered SAE. This filter removes a large number of redundant events and improves the throughput. This corner detector can detect corners more than 4× faster than eFAST and 50× faster than eHarris.

2.3.2. Gradient-Based Corner Detector

eHarris [8] is the first event gradient-based corner detector which generates local SAE for each pixel in the image plane and stores timestamps with a size of 9 × 9. The normalized timestamps in the local SAE are used to calculate the local gradient. However, once a new event arrives, each local SAE containing the pixels corresponding to this event needs to be updated, which is time-consuming. Li et al. [9] proposed G-eHarris, in which the timestamps are stored on a global SAE and then extracted to the local SAE from the global SAE when needed, which greatly improves the efficiency of eHarris.

2.3.3. Hybrid Corner Detector

In contrast, a template-based detector can detect corners faster, but the accuracy is low, while a gradient-based detector has higher accuracy, but it is very time-consuming and difficult to realize real-time detection. To achieve a better trade-off between throughput and accuracy, researchers combined two types of methods. Li et al. [9] proposed the first hybrid event-based corner detector called FA-Harris in which after processing the original event stream with a template-based detector, events that are completely impossible to be corners are discarded. Then, a gradient-based method was used to accurately locate the position. Similarly, the same idea is also applied in [16,26] to improve corner detection quality through multi-layer structure; firstly, redundant events are removed by the filter, then the template-based method is used to detect quickly, and finally a gradient-based method is used to refine the detection.

3. Materials and Methods

Inspired by [9,16], to achieve a better trade-off between throughput and accuracy, we propose an improved hybrid method. First, we use a double-threshold filter to remove redundant and noisy events, then we use a template-based method to locate corners faster and a modified gradient-based method with the proposed $Sigmoi{d}^{*}$-SAE to provide better corner detection results. By combining the two types of methods, a trade-off can be achieved between throughput and accuracy. The pipeline is shown in Algorithm 1, and a more intuitive flow diagram is shown in Figure 2.

Algorithm 1 The DTFS-eHarris Algorithm.

3.1. Event Generation Model

Each pixel of the event camera senses changes in brightness in the scene independently and asynchronously [27]. If a pixel senses a change in brightness that exceeds a predetermined threshold, an event is generated for that pixel. The event is output in the Address Event Representation (AER) format and can be expressed as $e={(x,y,t,p)}^T$, where x and y represent the coordinate position of the pixel generating the event, t is the time stamp of the event, and p is the polarity of the event (i.e., is either 1 or −1).

The mathematical model for event generation is shown in Equation (2), where ${\mathbf{L}}_{xy}$ is the latent image, representing the instantaneous brightness of the pixel at coordinates $(x,y)$ at the current moment, and $t_{ref}$ is the instantaneous brightness at the pixel location when the previous event was triggered [28]. The instantaneous brightness is calculated by taking the logarithm of the instantaneous brightness values at both moments. If the intensity of the increase in brightness exceeds the threshold, a positive event is generated and the value of p is 1. Conversely, if the intensity of the decrease in brightness exceeds the threshold, a negative event is generated and the value of p is −1, as shown in Equation (3), where c denotes a predetermined threshold value and d denotes the brightness change value.

$$\sigma =\mathcal{T}\left(log\left(\frac{{\mathbf{L}}_{xy}\left(t\right)}{{\mathbf{L}}_{xy}\left(t_{ref}\right)}\right),c\right)$$

(2)

$$p=\mathcal{T}(d,c)=\left\{\begin{array}{cc}+1,\hfill & d\ge c\hfill \\ 0,\hfill & d\in (-c,c)\hfill \\ -1,\hfill & d\le -c\hfill \end{array}\right.$$

(3)

When a new event is triggered at that pixel, ${\mathbf{L}}_{xy}\left(t_{ref}\right)$ should be updated to the instantaneous brightness at the time of the latest triggered event.

3.2. Double Threshold Filter

The main purpose of the double-threshold filter is to reduce redundant events and noise. This step can significantly reduce the number of events to be detected, greatly improve the real-time performance of the algorithm and improve the quality of corner detection.

A sudden and drastic brightness change would generate multiple events in one pixel almost instantaneously according to Equation (4). Due to hardware limitations, however, there exists a minimum amount of time between consecutive events triggered at the same pixel. However, such time-intensive events are not necessary for corner detection tasks.

$$\Delta I(x,y)=I(x,y,t+\Delta t)-I(x,y,t)=pc$$

(4)

The time filter proposed in [25] removes redundant events in corner detection tasks. Events are divided into two parts by a single threshold. Only events whose timestamps over the threshold will be detected, other events will be treated as redundant events.

However, this method can only remove redundant events through a single threshold and can not remove filter noisy events. Therefore, we propose a double threshold filter to remove redundant and noisy events. Timestamps are stored in two global SAE, $S$ and $S^{*}$, the SAE $S$ is defined as $S:(x,y)\in {\mathbb{R}}^2\mapsto t_l\in \mathbb{R}$, where $t_l$ is the timestamp of the latest event triggered at the pixel location $(x,y)$, $S^{*}$ is defined as $S^{*}:(x,y)\in {\mathbb{R}}^2\mapsto (t_r,t_l)\in {\mathbb{R}}^2$, and $t_r$ is the reference time. $S$ will be updated for every event and $S^{*}$ will be updated only if the previous event in the same pixel was not triggered within the time window [$t_r$, $t_l$], as shown in Equations (5) and (6).

$$\Delta t=S{(x,y)}_i-S{(x,y)}_{i-1}$$

(5)

$$\left\{\begin{array}{cc}\mathrm{update}S\mathrm{and}{S}^{*}\hfill & \Delta t\in \left[t_r,t_l\right]\hfill \\ \mathrm{update}S\hfill & \mathrm{others}\hfill \end{array}\right.$$

(6)

Only the events in $S^{*}$ are considered in subsequent processing, the rest of the events are treated as redundant events or noise.

3.3. Corner Pre-Detection

To ensure real-time processing, a template-based method [25] is used to select corners first. This algorithm is currently the largest throughput corner detection algorithm which is used to quickly filter corners. For each coming event which has been updated in $S^{*}$, a local SAE (denote as $S_L$) can be extracted centred on the event with a size 9 × 9. Then, it focuses only on the pixels on two concentric circles of radius 3 (blue) and 4 (green), as shown in Figure 3.

The location of a new event is used as the centre of a circular mask of a predefined radius. This circular mask defines a set of locations, the value of which we retrieve in SAE $S^{*}$, using the same polarity as the new event, and create a circular set of elements C. We initialize the arc of the newest elements $\mathcal{A}$ with the newest element in C and a pair of supporting elements in the adjacent clockwise (CW) and counter clockwise (CCW) positions; $E_{CW}$ and $E_{CCW}$ and ${\mathcal{A}}^{\prime }$ are its complementary arcs, as shown in Figure 3. The arc $\mathcal{A}$ starts from the newest timestamp until all values in $\mathcal{A}$ are greater than others. The length thresholds of continuous arcs $\mathcal{A}$ and ${\mathcal{A}}^{\prime }$ are [3, 6] for the inner circle and [4, 8] for the outer circle, respectively. The centre pixel of the local SAE will be considered as a corner if the length of any arc is within the threshold range.

After corner pre-detection, events that can not be a corner at all are discarded. The selected events will be further detected.

3.4. Corner Detection Refinement

In this step, the local SAE will be extracted from the global SAE $S^{*}$ and then be normalized. For corner detection tasks, local SAE is usually expected to have significant gradient information between new and old events. Li et al. [9] and Mohamed et al. [16] sorted the timestamps on the local SAE, assigning the latest 25 timestamps to 1 and the rest to 0. In addition, there are other SAE constructions, such as min-max normalization [17] and exponential decay [18]. However, those methods completely ignore the time interval among events, which can not well reflect the temporal and spatial distribution of local SAE. Therefore, we propose a new normalization method that uses the Sigmoid function to normalize the timestamp information.

Firstly, the timestamps stored in global SAE are transformed from the range of (0, $+\infty$) to a value between [0, 1]. To obtain clearer gradient information, the timestamp of the centre pixel on $S_L$ (the latest event) is used as the reference time and calculates relative time with the rest of the events $S_L$, as shown in Equation (7). For the normalization of local SAE, we do not expect the data to be distributed evenly. The newer the event, the closer we expect the value to be 1 and for the older event, we expect it to be close to 0. That is, for the time difference stored in local SAE (as shown in Figure 4a), the closer to 0, the closer the normalized value should be to 1; however, the value is close to 0.

$$\Delta t=t_{ref}-t(x,y)$$

(7)

where $\Delta t$ is the relative time, $t_{ref}$ indicates the timestamp of the middle pixel and $t(x,y)$ denotes the timestamp at the position $(x,y)$. Inspired by the activation function of machine learning, we improved the Sigmoid function to obtain the nonlinear distribution. The proposed normalized method is shown in Equation (8) and Figure 5, named $Sigmoi{d}^{*}$.

$$Sigmoi{d}^{*}=1-\frac{\alpha }{\alpha +e^{-\beta \Delta t}}$$

(8)

where two hyper-parameters $\alpha$ and $\beta$ are introduced into the original Sigmoid function, which is fine-tuned to adjust normalized results and can be changed based on the complexity of scenes. It has lower computation and clearer spatio-temporal distribution than binary normalization [9], min-max normalization [17] and exponential decay [18].

Figure 6 presents the trends of $Sigmoi{d}^{*}$ with multi-parameters; in a word, smaller alpha and bigger beta are preferred for time-dense events, and vice versa. For this experiment, we set $\alpha$ = 0.005 and $\beta$ = 300 manually. It is obvious that when the relative time is close to 0, the function value is close to 1; with the increase in relative time, the function value has a sharp decline. Figure 4b is the result of $Sigmoi{d}^{*}$, and it is obvious that the results of Figure 4b,d are the best and our method has lower computation.

Then, the Harris [19] detector is used on $S_L$ which is normalized by the $Sigmoi{d}^{*}$, and the score is calculated by Equations (9) and (11)

$$M=\sum _{x,y}w(x,y)\left[\begin{array}{cc}{I}_x^2& I_x{I}_y\\ I_x{I}_y& I_y^2\end{array}\right]=\left[\begin{array}{cc}a& b\\ b& c\end{array}\right]$$

(9)

where $I_x$ and $I_y$ are the gradients of the patch in the x and y directions, respectively, and $w(x,y)$ is a Gaussian filter with $\sigma =1$, shown as Equation (10).

$$w(x,y)=\frac{1}{2\pi {\sigma }^2}{e}^{\frac{-(x^2+y^2)}{2{\sigma }^2}}$$

(10)

If both eigenvalues ${\lambda }_1$ and ${\lambda }_2$ in matrix M are large, a corner event is probable. The Harris score is calculated by Equation (11).

$$\begin{array}{cc}\hfill Score& =det\left(M\right)-k·trace{\left(M\right)}^2\hfill \\ \hfill & ={\lambda }_1{\lambda }_2-k·{({\lambda }_1+{\lambda }_2)}^2\hfill \\ \hfill & =(ac-b^2)-k·{(a+c)}^2\hfill \end{array}$$

(11)

where k is an empirical constant and is commonly set at 0.04. If the Harris score is greater than the preset threshold, the detected event will be classified as a corner event.

3.5. Event Based Corner Tracker

Our method associates the corner events based on proximity under the assumption that the corner detection is accurate. The proposed tracker split the events into the parent node and leaf node, denoted as $N_p$ and $N_l$. A leaf node plane (denote as ${\mathcal{P}}_l$) is constructed to store $N_l$ spatio-temporally for every existing trajectory. A coming event will be divided into an existing feature or a new feature and ${\mathcal{P}}_l$ will be updated. Then, a spatio-temporal window can be created for each trajectory by distance threshold $d_{thresh}$ and time threshold $t_{thresh}$.

If the event is not in any existing spatio-temporal window, it will be treated as the first event of a new trajectory. The pipeline is shown in Algorithm 2.

Algorithm 2 The Tracker Algorithm.

4. Experiments and Results

Our method is evaluated on the public Event Camera Datasets [12], which provides asynchronous event stream and its corresponding intensity images (frame-based) captured on the same chip at the same time. The scenarios we used to evaluate are shown in Figure 7.

The method proposed in this paper only works on event stream, and intensity images are used to extract the ground truth of corners. Our method is evaluated on shapes, dynamic, poster and boxes scenes, which cover high, medium and low textures and each scene contains three forms of movement.

We compare the performance of our work with eFAST, eHarris, Arc* and FA-Harris, using the original implementation provided by the authors. We evaluate both G-eHarris and eFAST with the proposed double threshold filter, denoted as G-eHarris† and eFAST†, respectively. To check the performance of our proposed double threshold filter, we modify the single threshold filter of Arc* and FA-Harris into a double threshold filter; the new version is called Arc*† and FA-Harris†.

For the event-based corner detector, there are four important evaluation metrics, which are Throughput, Reduction Rate, False Positive Rate and Accuracy; we use the same definition in [9,25].

All experiments were run on a laptop equipped with an Intel i5-5200U CPU with 2.20 GHz and 12GB of RAM and implemented using single-threaded C++.

4.1. Ground-Truth Collection

Similar to the method in [7,25], a Harris detector [19] and LK tracker [29] are used to collect the ground truth on the intensity image. For two adjacent intensity images, we use the original Harris detector to detect corners on the first image with the shortest distance of 5, and then take the LK tracker to track them on the second image. Considering that new corners may appear in the second image, after tracking, we continue to detect new corners on the second image. If the distance between detected new corners and tracked corners is less than 5 pixels in the second image, the two corners are considered the same corner. Then all corners will be tracked continuously.

The frame rate of the intensity image is about 25 Hz. We approximately consider that the corners move in a uniform linear motion between two adjacent images. Then, based on the segment generated by the motion of this corner (denote as l), as shown in Figure 8, we can construct two concentric cylinders and calculate the distance from the event corners to $\mathit{l}$. If the distance is less than or equal to 3.5 pixels, we consider it as true event corner and label it as the ground truth. For an event whose distance is greater than 3.5 but less than 5, we consider it to be a wrong corner and label it as a false event corner.

4.2. Qualitative Analysis

In

Table 1. Throughput and Reduction Rate performance. Two hybrid methods are listed separately. The best result of the hybrid method is made in bold.

	Time per Event [μs/Event]	Max. Event Rate [Meps]	Red. [%]
eFAST†	0.862	1.150	95.69
G-eHarris†	3.558	0.280	95.06
Arc*	0.260	3.830	90.62
blackFA-Harris	0.975	1.025	97.28
Ours	0.781	1.278	98.80

, we report the Reduction Rate (Red.) of our detector with other SOTA detectors as a percentage. The Reduction Rate indicates the ability to reduce the event stream to a feature stream [7]. Our work shows the best performance.

Figure 9 shows the detection quality comparison on poster scene, which is the most complex scene in this experiment. We show the results between two adjacent frames. Our method outperforms all other methods and has lower noisy detection. As shown in Table 1, our work reduces the most events than other detectors. The main issue of the template-based detector in Figure 9e,f is that many false corners are mistakenly detected, e.g., the bottom-right of the scene.

4.3. Real-Time Performance

As event cameras are often used in highly dynamic scenes, detection efficiency is important for event-based corner detection methods and data throughput is an important real-time performance evaluation metric. Table 1 shows the real-time performance of different corner detection algorithms, incorporating the average time required to process a single event (μs) and the maximum number of events that can be processed per second in millions of events per second (Meps). The first three rows in the table are for the normal detection method and the last two rows are for the hybrid detection method. Our method improves by about 20% faster than FA-Harris and 4× faster than eHarris†. However, Arc* is 3× faster than ours since we apply the gradient-based method to refine corners for higher accuracy.

4.4. Accuracy Performance

The False Positive Rate (FPR) and Accuracy are reported in

Table 2. False Positive Rate (%) of event-based corner detectors. The best results are in bold.

	Shapes	Dynamic	Poster	Boxes
eFAST†	11.08	12.12	27.85	28.22
G-eHarris†	7.91	16.9	42.50	45.21
Arc*	19.70	34.00	58.11	56.32
FA-Harris	7.05	7.67	19.27	18.00
Ours	4.77	3.45	5.31	5.15

and

Table 3. Accuracy (%) of event-based corner detectors. Two hybrid methods are listed separately. The best results are in bold.

	Shapes	Dynamic	Poster	Boxes
eFAST†	76.79	68.61	60.67	60.53
G-eHarris†	77.12	68.18	59.78	59.30
Arc*	76.58	63.87	59.00	58.33
FA-Harris	79.74	72.34	61.88	62.36
Ours	78.82	75.4	64.95	67.38

, respectively, using the same definition as [9,25]. For the ground-truth corners, the trajectories of the real corners have been obtained using the Harris and LK algorithms in Section 4.1. The confusion matrix is shown in

Table 4. Confusion matrix for corner detection.

Distance Range	Label as Corners	Label as Non-Corners
[0, 3.5]	TP	FN
(3.5, 5]	FP	TN

.

The TPR is calculated as Equation (12), the FPR is calculated as Equation (13) and the Accuracy is calculated as Equation (14).

$$TPR=TP/(TP+FN)$$

(12)

$$FPR=FP/(FP+TN)$$

(13)

$$ACC=TP/(TP+FP)$$

(14)

The proposed method shows the best performance in FPR and better performance in Accuracy. With the increase in scene complexity, the advantage gradually expands, which means that our detector is more robust for complex scenes (poster and boxes).

4.5. Double Threshold Filter Performance

Finally, we compare the proposed double threshold filter separately. We change the single threshold filter of Arc* and FA-Harris into the double threshold filter, denoted as Arc*† and FA-Harris†, which are used to compare with the original Arc* and FA-Harris. A weighted average was made for different movements in each scene, and the experimental results are shown in

Table 5. Experimental results of double threshold filter.

		Time per Event [μs]	ACC	TPR	FPR
shapes	Arc*†	15.35%	−1.54%	−5.60%	2.24%
	FA-Harris†	20.18%	−1.74%	−2.77%	0.58%
dynamic	Arc*†	10.76%	−0.33%	−0.12%	−0.39%
	FA-Harris†	6.38%	3.00%	−2.21%	1.19%
poster	Arc*†	7.72%	−0.17%	−0.09%	−0.30%
	FA-Harris†	4.46%	−0.28%	−0.11%	−0.14%
boxes	Arc*†	9.90%	−0.05%	−0.37%	0.17%
	FA-Harris†	10.8%	−0.07%	−0.12%	0.04%

. We evaluated its throughput, Accuracy, TPR and FPR, and the data indicate the comparison of the method using double-threshold filtering with the original method, with larger percentages indicating better results. It can be seen that the proposed double threshold filter can effectively improve the throughput when other metrics do not change significantly.

4.6. Asynchronous Corner Tracker Performance

Figure 10 shows the tracker quality comparison to verify the effectiveness of our detector; we synthesize events within 100 ms on the intensity images for visualization. We use fewer corners and achieve similar results with less noise. This will bring great benefits for the following tasks.

5. Conclusions

This paper presents a simple and effective event-based asynchronous corner detector (DTFS-eHarris) and an asynchronous event-based corner tracker. For the corner detector, a double threshold filter is proposed to reduce the amount of events and it is easy to combine with the existing corner detection methods. Moreover, the proposed $Sigmoi{d}^{*}$ function is a novel SAE normalization method which can normalize the time difference rapidly and reflect its spatio-temporal information. Compared with the existing detectors, the evaluations indicate that DTFS-eHarris has a significant improvement, especially in complex scenes, which has shown more than a 10% reduction in false positive rate and a 5% and 20% improvement in accuracy and throughput, respectively, compared to the existing SOTA hybrid detector. For the corner tracker, an asynchronous event-based tracker is proposed and further verifies the effectiveness of our detector. It is believed that the proposed method could enhance the real-time performance and accuracy of future robotic applications. Regarding the limitations of our method, the filter thresholds and $Sigmoi{d}^{*}$ parameters could not update automatically for optimal performance in different scenarios. In future work, we will try to track the detected corners asynchronously and explore dynamic threshold setting methods instead of setting fixed thresholds manually.