1. Introduction
Currently, multi-camera localization is used in many fields, e.g., the currently hot field of autonomous driving [
1,
2]. But, in many real-world scenarios such as swarm formations and mobile robots, multiple cameras often constitute an unsynchronized multi-camera localization system (UMCLS) without additional synchronization processing [
3]. The main challenge posed by unsynchronized cameras is that each scene point would be captured at different instants by each camera. This would induce triangulation errors because the two image rays would either intersect at an incorrect location or simply not intersect at all [
4]. Therefore, unsynchronization would bring large errors to traditional multi-camera calibration and localization algorithms, which are designed on the basis of synchronized cameras. This paper addresses the challenge posed by unsynchronized cameras by using timestamp matching and pixel fitting.
For a UMCLS, the first step is to perform multi-camera calibration, which aims to accurately compute the intrinsic parameters (principal point, lens distortion, etc.) and the extrinsic parameters (rotation matrix and translation vector between the camera coordinate system and the reference coordinate system) of each camera. Multi-camera calibration is the basis of 3D localization since the calibration results would be subsequently used during the process of 3D localization. The localization accuracy of a UMCLS will be determined by the calibration accuracy of multiple cameras directly, so the process of multi-camera calibration is very important. According to the dimension of the calibration object, existing multi-camera calibration methods can be roughly divided into five kinds: methods based on 3D calibration objects [
5,
6], methods based on 2D calibration objects [
7,
8], methods based on 1D calibration objects [
9], methods based on point objects [
10], and self-calibration methods [
11,
12,
13,
14,
15,
16]. Note that methods based on 1D calibration objects can quickly and easily complete the calibration of multiple cameras without being affected by occlusion [
17], so this paper chooses to use the 1D calibration method.
Until now, most of the multi-camera calibration methods are developed for synchronized cameras, and synchronization is controlled by a hardware trigger. But, in many cases, the camera frame rates are different, the cameras work asynchronously, and the acquired image sequences are naturally not matched. Therefore, image synchronization and camera calibration are usually carried out simultaneously for multiple unsynchronized cameras. From the perspective of the calibration object, existing calibration methods for unsynchronized cameras could be generally classified into two kinds: methods based on point objects [
18,
19,
20] and methods based on 3D calibration objects [
21]. As mentioned above, 1D calibration methods are quite suitable for calibrating multiple cameras, and a practical 1D calibration method needs to be designed for a UMCLS.
On the other hand, 3D localization is a crucial function for unsynchronized multiple cameras, and there are some related research works. For example, Benrhaiem et al. [
22] proposed a temporal offset-invariant 3D reconstruction method to solve the problem of camera unsynchronization. Their approach only deals with stereo cameras based on the epipolar geometry. Piao and Sato [
23] proposed a method to achieve the synchronization of multiple cameras and compute the epipolar geometry from uncalibrated and unsynchronized cameras. In particular, they used the affine invariance on the frame numbers of camera images to find the synchronization. Considering that 3D localization needs to meet the real-time requirement in practice, the acquired unsynchronized multi-camera images should be quickly processed. Therefore, a fast and high-precision feature point localization method needs to be designed for a UMCLS.
In this paper, in order to solve the problem of unsynchronization, we propose a time synchronization scheme and a wand-based calibration method to complete the calibration of multiple perspective/fish-eye cameras. Then, we propose an EKF-based unsynchronized 3D localization method. Finally, we built a real UMCLS and verified its performance by using a flapping-wing micro air vehicle (FWAV) to accomplish fixed-height experiments. The main contributions of this study are as follows:
(1) For unsynchronized perspective or fish-eye cameras, a wand-based multi-camera calibration method is proposed by using timestamp matching and pixel fitting, and the calibration result was verified by real experiments.
(2) An EKF-based 3D localization algorithm is proposed for unsynchronized perspective or fish-eye cameras.
(3) An actual UMCLS was built, and the performance of the system was evaluated through the feature point reconstruction experiments and fixed-height experiments of an FWAV.
The remainder of this paper is organized as follows. The problem formulation, designed calibration algorithm, and designed 3D localization algorithm are introduced in
Section 2.
Section 3 presents the results and a discussion of the calibration and fixed-height experiments.
Section 4 is the conclusion of this paper.