^{*}

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).

This paper presents a novel way to address the extrinsic calibration problem for a system composed of a 3D LIDAR and a camera. The relative transformation between the two sensors is calibrated via a nonlinear least squares (NLS) problem, which is formulated in terms of the geometric constraints associated with a trihedral object. Precise initial estimates of NLS are obtained by dividing it into two sub-problems that are solved individually. With the precise initializations, the calibration parameters are further refined by iteratively optimizing the NLS problem. The algorithm is validated on both simulated and real data, as well as a 3D reconstruction application. Moreover, since the trihedral target used for calibration can be either orthogonal or not, it is very often present in structured environments, making the calibration convenient.

Multi-sensors are commonly equipped on mobile robots for navigation tasks. Currently, for instance, ranging sensors such as high-speed 3D LIDARs are often used in conjunction with cameras for a robot to detect objects [

A variety of methods have been developed to address the LIDAR-camera extrinsic calibration problem. Among them, early interest focuses on systems consisting of a 2D LIDAR and a camera [

In recent years, with the development of 3D laser ranging techniques, several methods were proposed to calibrate 3D LIDAR-camera systems [

In this work, we propose a novel way to conduct the extrinsic calibration between a 3D LIDAR and a camera. In contrast to most of the published techniques, our method distinguishes itself in two aspects:

It takes advantage of a trihedron—which may or may not be orthogonal—for calibration. Such trihedral targets are ubiquitous in both indoor and outdoor structured environments, such as two adjacent walls of a building together with the floor. Hence, it is quite convenient for a robot to collect data for calibration. Compared to the aforementioned calibration rigs, the trihedral configuration is less likely to be perturbed even under severe weather conditions, and is easier to be captured.

In contrast to these calibration-rig-based methods that require a user to specify both the region of a plane in 3D LIDAR and the corners in images, our method requires fewer user inputs. Only the region of each plane of the trihedron in the sensors' data is needed. Moreover, the precision of the manual inputs does not make much of a difference.

To present the proposed method, we organize the remainder of this paper as follows. In Section 2, we first describe the extrinsic calibration problem via taking advantage of an trihedral calibration rig, and introduce the associated geometric and motion constraints. Section 3 presents the entire calibration procedure. Experiments conducted on both simulations and real data are exhibited in Section 4, followed with conclusions in Section 5.

Let us formally define the problem of 3D LIDAR-camera extrinsic calibration. We are given a camera and a 3D LIDAR that are rigidly mounted with respect to each other. Both sensors are assumed to be pre-calibrated, meaning that their intrinsic parameters are known. A trihedron is observed synchronously by them. Our objective is to determine the relative transformation between the two sensors, by taking advantage of the constraints associated with the trihedron.

For the sake of clarity, in the remaining of this section, we introduce the related definitions and notations, together with the geometric and motion constraints established between the measurements of the two sensors.

_{i}^{th}

_{i}

_{3}'s normal vector, and

Once the frames are defined, we represent the relative rotation and translation from one frame _{AB}_{AB}_{i}_{i}_{A}_{B}_{B}_{AB}_{A}_{AB}_{LiCi}_{LiCi}_{LC}_{LC}

In addition, we know that a plane in a frame is specified by ^{T}^{th}^{th}

The proposed method makes use of a trihedron as a calibration rig. Hence, in order to address the extrinsic calibration problem, several constraints are taken into consideration. They are summarized as follows.

_{WA}_{WA}_{WA}_{WA}_{WA}_{WA}_{WA}_{WA}_{WA}

_{B}_{B}

^{−1} (_{1} and _{2}. In the first frame, the plane on which all the features lie is defined by {_{C1}, _{C1}}. Then, pair-wise corresponding image features _{C1} and _{C2} satisfy

In order to estimate the relative transformation between a 3D LIDAR and a camera, we capture

_{LC}

_{LC}

^{th}

^{th}

_{WLi}

_{WLi}

^{th}

_{C1Ci},

_{C1Ci}}; Estimate

_{WC1},

_{WC1},

_{WCi},

_{WCi}};

_{LC}

_{LC}

_{LC}

_{LC}

Given the ^{th}_{WLi}_{WLi}^{th}^{th}^{th}_{WLi}_{WLi}

When more than one observation is available, we can further refine the results by using the planarity constraints established between each pair of the LIDAR frames. Thus, we get
_{LiL1}, _{LiL1}, _{L1Li}, and _{L1Li} are the functions of the planes' parameters,

Let
_{0} and iteratively updates the parameters via
_{t}

This step is to determine the transformations, {_{WCi}_{WCi}

Given two LIDAR-camera observations, we first estimate _{C1C2} and _{C1C2} between the two camera frames. Once a user delimits the regions of the planes on two images, a set of point features are detected by SIFT [_{C1C2} and _{C1C2} from _{C1C2} is of unit norm. We hence use the motion constraint defined in

Once the relative motion between two views is determined, we are able to determine the planes by taking advantage of the planarity constraint established between two images, as defined in _{C1C2}, _{C1C2} are also refined. It is a nonlinear least squares problem solved by LM. The estimates of _{Ci}_{Ci}

With the above-estimated parameters, we now formulate the LIDAR-camera extrinsic calibration task as a nonlinear least squares problem. In terms of the planarity constraints established between the LIDAR and the camera frames, we get the form

We implement the proposed method in MATLAB. The running time of our algorithm is coarsely measured on a laptop with an Intel Core2Duo 2.26 GHz processor and 3 GB memory. Except for the manual input procedure, it takes about 20 seconds in average to perform the entire calibration when 9 LIDAR-camera observations are considered. Each contains 5,000 LIDAR points and 100 registered image points. In order to evaluate the proposed method, a series of experiments have been carried out. The algorithm is first tested on simulated data to validate its correctness and explore its sensitivity with respect to noise. Then, it is used to calibrate a real system composed of a 3D LIDAR and a camera. The calibration results are subsequently used for 3D reconstruction.

The first experiment validates the correctness and numerical stability of our algorithm. We hereby generate sets of data to simulate multiple observations of a trihedron obtained by a 3D LIDAR-camera system. The system is of the following properties. The rotation and translation from the LIDAR to the camera are set, respectively, as _{LC}^{T}_{LC}^{T}_{LC}_{LC}

The extrinsic calibration can be conducted with two or more LIDAR-camera observations. In this experiment, we investigate the impact of the observation number on the calibration performance. Nine LIDAR-camera observation pairs are generated, in which Gaussian noise with zero mean and _{LC}_{LC}

Real ranging sensors produce noisy measurements. Hence, this experiment explores the sensitivity with respect to noise on LIDAR points. We conduct the experiment on the first two simulated observations. Zero mean Gaussian noise is added to points of the LIDAR observations, with

The feature detection and matching algorithm we use in this work is SIFT [

To further evaluate the proposed algorithm, we employ it to calibrate a real system and use the calibration results to reconstruct 3D scenes. The system is composed of a 3D Velodyne HDL-64E LIDAR [

In the experiments, we collect two LIDAR-camera measurements of a scenario containing a trihedral object. The trihedron consists of two adjacent walls of a building, together with the ground plane, as shown in

In order to validate the calibration results, the determined extrinsic parameters are further used for 3D reconstruction. With the calibrated _{LC}_{LC}

In this paper, we have presented a new method of conducting the extrinsic calibration for a 3D LIDAR-camera system. Specifically, instead of using planar checkerboard patterns, we take advantage of arbitrary trihedral objects, which might be either orthogonal or not, for calibration. This kind of configuration is ubiquitous in structured environments, so that it is very convenient for a mobile robot to collect data. We have validated the algorithm on both simulated and real scenarios. Although the experimental results are presented from 3D LIDAR and omnidirectional camera systems, the algorithm is applicable to systems composed of any kind of 3D LIDARs and cameras. Our method is interesting for both indoor or outdoor mobile robots equipped with such sensors. The calibration results can be further used for data fusion applications.

This research work is partially supported by the National Natural Science Foundation of China via grants 61001171, 60534070, 90820306, and Chinese Universities Scientific Fund.

A typical calibration configuration. (

Errors ^{T}

Errors ^{T}

Errors ^{T}

The robotic platform and the sensors in our experiment. (

The first LIDAR-camera view used for calibration. (

The second LIDAR-camera view used for calibration. (

A portion of matched features on the trihedron. The matched feature pairs on the three planes are marked with lines in different styles.

Calibration results of a real 3D LIDAR-camera system.

_{X} |
T_{Y} |
_{Z} |
||||
---|---|---|---|---|---|---|

The proposed method | 0.257 | 0.007 | −0.323 | −1.788 | 1.446 | −88.542 |

The method in [ |
0.203 | 0.036 | −0.285 | −1.358 | 1.799 | −88.996 |