Assessment of the Spatial Co-registration of Multitemporal Imagery from Large Format Digital Cameras in the Context of Detailed Change Detection

Abstract

Large format digital camera (LFDC) systems are becoming more broadly available and regularly collect image data over large areas. Spectral and radiometric attributes of imagery from LFDC systems make this type of image data appropriate for semi-automated change detection. However, achieving accurate spatial co-registration between multitemporal image sets is necessary for semi-automated change detection. This study investigates the accuracy of co-registration between multitemporal image sets acquired using the Leica Geosystems ADS40, Intergraph Z/I Imaging® DMC, and Vexcel UltraCam-D sensors in areas of gentle, moderate, and extreme terrain relief. Custom image sets were collected and orthorectified by imagery vendors, with guidance from the authors. Results indicate that imagery acquired by vendors operating LFDC systems may be co- registered with pixel or sub-pixel level accuracy, even for environments with high terrain relief. Specific image acquisition and processing procedures facilitating this level of co- registration are discussed.

1. Introduction

Commercial large format digital camera (LFDC) systems have been utilized for airborne collection of multispectral imagery since 2001. These systems include the Leica Geosystems ADS40, the Intergraph Z/I Imaging® DMC, and the Vexcel UltraCam. Large format digital camera systems collect imagery with detail comparable to large format film and are defined by [1] as those capable of acquiring an image frame of at least 36 megapixels. The ADS40, DMC, and UltraCam have been on the market longest and are the most commonly available LFDC systems.

LFDC systems are able to efficiently collect and process wide swaths of high spatial resolution imagery by direct digital capture. These systems have been utilized to collect high spatial resolution imagery for engineering applications, city/county geographic information systems, programs such as the United State Department of Agriculture Farm Service Agency (USDA-FSA) National Agriculture Imagery Program (NAIP) and to generate products such as United States Geological Survey (USGS) Digital Orthophoto Quadrangles (DOQ) [2].

The spectral-radiometric properties of imagery from LFDC systems also make this type of imagery appropriate for semi-automated change detection. That is, LFDC systems collect multispectral (visible and near-infrared) imagery by directly sampling electromagnetic radiation reflected from the ground surface and by quantifying this reflected radiation with high radiometric precision. However, a critical requirement of semi-automated change detection is that multitemporal images must be co-registered accurately. This study investigates the accuracy with which multitemporal image products generated by vendors of LFDC systems can be co-registered for detailed (sub-meter) change detection analysis.

2. Background

2.1. Image Co-registration

Achieving accurate spatial co-registration between multitemporal image sets can be challenging. Studies utilizing Landsat satellite imagery for change detection have considered a spatial co- registration error of 0.5 to 1.0 pixels to be adequate for reducing misregistration artifacts in change detection products [3-6]. However, achieving this level of spatial co-registration with airborne imagery is complicated by the wide view angles utilized by airborne systems and the resulting image parallax [7]. Misregistration between multitemporal airborne image sets is often on the order of several pixels in areas of high relief [8].

Coulter et al. (2003) [7] presented a technique for acquiring and precisely registering airborne frame imagery. This technique, referred to as frame center (FC) matching [9], is based upon repeating the camera station positions in terms of horizontal position and altitude between multitemporal acquisitions. When camera stations are matched between multitemporal image sets, terrain related distortions are replicated between images, image parallax is minimized, and images can be accurately co-registered using simple (low order polynomial) warping algorithms. The FC approach was found to be highly effective with small format digital imagery acquired using an Airborne Data Acquisition and Registration (ADAR) 5500 digital camera system. Four specific tools are required for operational FC matching using GPS data [7]:

1)

GPS for logging and digitally archiving flight line and FC coordinates for each image acquisition;

2)

Flight planning software integrated with digital coordinates of flight line and frame coordinates from previous image dates;

3)

In-flight, heads-up display enabling a pilot to maintain flight line course and altitude (based on GPS coordinates); and

4)

Automatic triggering of image frames (based on digitally archived coordinates and in-flight GPS).

2.2. Change Detection Application

The context of this study is monitoring of the United States border region with high spatial resolution imagery. The United States Border Patrol (USBP) is interested in utilizing high resolution imagery acquired on a regular basis for monitoring changes along the border resulting from immigrant and law enforcement activities. Multitemporal image products with accurate spatial co-registration will facilitate manual or semi-automated detection of changes. LFDC systems may be preferable to large format film cameras for imaging large extents of the border because no film purchase, processing, or scanning is required in the creation of the digital images. In addition, a completely digital workflow may result in faster and more efficient product generation.

Researchers at the Department of Geography at San Diego State University (SDSU) are working with personnel from the San Diego Sector of the USBP to assess the utility of commercially available imagery from LFDC systems for border monitoring and change detection. As part of this assessment, SDSU researchers surveyed companies operating ADS40, DMC, and UltraCam LFDC systems within the United States and asked these companies if they would be willing to collect and provide demonstration data sets for the evaluation. Six companies responded to the survey, and four of these companies provided custom imagery or data processing services.

2.3. Large Format Digital Camera Systems

The ADS40 system is a line array system that employs the three-line scanning technique, in which multiple linear charge coupled device (CCD) arrays are situated on the focal plane within a single sensor head and view in the forward, nadir, and backward direction. The ADS40 uses large (12,000 pixel) linear CCD arrays which yield wide cross-track image coverage per flight line [10]. Each line of imagery is acquired over a unique time period and is associated with a unique aircraft position and orientation. Therefore, the ADS40 requires integrated global positioning system (GPS) and inertial measurement unit (IMU) data for direct orientation support [11].

The DMC and UltraCam systems utilize frame (area) CCD arrays, and collect imagery in a manner similar to film-based cameras with unique camera stations per frame. These digital cameras use multiple CCD arrays to create individual large format image frames [11-13]. Multispectral data are acquired at a coarser spatial resolution than the panchromatic, and pan-sharpening is employed to create the final high resolution multispectral images [11,14].

2. Study Areas

Custom demonstration data sets were acquired at two locations. A study area near Tecate, CA was selected as the primary area for the evaluation. Tecate is a small town along the U.S./Mexico border that is located 40 km east of the Pacific Ocean. DMC and UltraCam-D imagery was acquired for this location. The 6.1 km by 1.9 km Tecate study area is characterized by terrain varying from gentle to extreme, with Mediterranean-type vegetation including natural grass, coastal sage scrub, and chaparral.

ADS40 imagery was not collected at the Tecate, CA area due to logistical constraints. Therefore, custom ADS40 imagery was acquired for an area in Big Bend, TX which had terrain and vegetation characteristics similar to those of the Tecate, CA site. The Big Bend area is 12.8 km by 1.5 km and is located in the western portion of Big Bend National Park, TX. The climate of the Big Bend area is more arid that of the Tecate area, and the desert vegetation is more sparse.

3. Image Data Collection

Multitemporal image sets were acquired with each sensor based upon the principals of the FC matching technique at the request of the authors. Multitemporal DMC and UltraCam frame-based image sets were each acquired from the same camera stations in the sky (same horizontal and vertical position) by navigating the plane along pre-planned flight lines and using GPS-assisted camera triggering to fire the cameras at precisely the same camera stations on each pass. DMC and UltraCam image collections were performed independently and did not utilize the same camera stations. The ADS40 line array system continually collects imagery as it flies. Therefore, it was only practical to use GPS-based navigation to guide the plane down the same flight line two times for the multitemporal collection. Each imaging pass of the ADS40 sensor was conducted using the same east to west heading and the same altitude along the flight line.

Multitemporal (repeat pass within 10-15 minutes) ADS40 imagery was acquired on 05 January 2007 and processed by Kucera International Inc. (Willoughby, Ohio). The Kucera International ADS40 utilized the SH40 sensor head containing 10 individual CCD line arrays. These line arrays included two arrays collecting panchromatic imagery at a 28 degree forward viewing angle, two arrays collecting panchromatic imagery at a 14 degree backward viewing angle, three arrays collecting blue, green, and red (color imagery or RGB) at nadir, and three arrays collecting green, red, and near- infrared (color infrared, or CIR) at 16, 14, and 18 degrees forward, respectively. Imagery was collected with a ground resolution element of 0.15 m (six inch) from an nominal altitude of 1500 m above ground level (AGL).

Multitemporal (multidate) DMC image sets were acquired and processed by Digital Mapping, Inc. (Huntington Beach, CA). The image sets were collected on 19 May 2006 and 19 June 2006 from a nominal altitude of 1500 m AGL. The ground resolution element of the DMC imagery was 0.15 m. Separate three-band image products were created for RGB and CIR display from the collected data sets.

Multitemporal (repeat pass within 10 minutes) UltraCam imagery was acquired on 10 July 2006 by Keystone Aerial Surveys, Inc (Philadelphia, PA) using an UltraCam-D, sensor and was processed by Infotech Enterprises America, Inc. (Sterling, VA). UltraCam-D imagery was collected with a ground resolution element of 0.15 m from a nominal altitude of 1950 m AGL. Four band, registered multitemporal image sets were delivered by Infotech.

4. Methods

4.1. Image Data Preprocessing

Imagery vendors were provided with specific image processing parameters in order to 1) ensure that the image co-registration techniques were comparable between the LFDC systems and 2) evaluate the co-registration that may be achieved using (presumedly) lower cost orthorectified products based on USGS digital elevation models (DEMs) and not incorporating surveyed ground control collection, custom digital terrain model (DTM) creation using stereo image products, nor purchase of high resolution terrain data (e.g., light detection and ranging (LIDAR) or interferometric synthetic aperture radar (IFSAR) products). Vendors were asked to use GPS/IMU data and/or freely available USGS DEMs and DOQQs for horizontal and vertical control of aerotriangulation. The vendors were also asked to orthorectify the imagery using USGS DEM data. It was stressed that the vendors should make every effort to co-register the imagery with high accuracy. Absolute image positional requirements were less stringent, and vendors were asked (if practical given the previous constraints) to deliver products meeting a 1:4800 scale national map accuracy standard (NMAS), having a circular error probability of 90% of all points (CE90) being correctly located within 4.1 m of their true location.

Orthorectification of the multitemporal ADS40 image sets was conducted using a USGS National Elevation Dataset (NED) 10 m DEM. Kucera International, Inc. also collected LIDAR data and provided a second set of image products that were orthorectified using a LIDAR derived DTM. This additional product allowed assessment of co-registration accuracy between images processed using a higher accuracy and higher resolution terrain product. The LIDAR data were collected from approximately 2250 m AGL using a Leica ALS50 83kHz LiDAR system. The average LIDAR post spacing was 2.42 m and the average illuminated footprint diameter was 0.78 m.

Multitemporal UltraCam-D image sets were orthorectified by Infotech Enterprises America, Inc. using a USGS DOQ image product for horizontal ground control and a USGS NED 30 m DEM for vertical ground control. Each image set was orthorectified independently using aero-triangulation techniques and the 30 m DEM. While the imagery was collected with matched frame centers, processing procedures did not follow those outlined in [7], where corresponding multitemporal image frames are co-registered on a frame-by-frame basis. Therefore, an alternate procedure was also performed by the authors, such that frames were co-registered on a frame-by-frame basis using FC matching procedures in order to evaluate differences between the two approaches. As this was conducted for evaluation purposes only, the frame-to-frame co-registration was performed on raw (non-georeferenced/orthorectified) image frames.

AutoSync software by Leica Geosystems was used for the FC co-registration of UltraCam-D imagery. Three manual tie points were located within corresponding UltraCam-D image frames, and then an automated tie point generation process was used to obtain a large number of tie points between the corresponding frames. Automatically generated tie points were visually reviewed, then second- order polynomial warping routines were utilized to warp pass 2 image frames to match corresponding pass 1 image frames. This procedure was performed for individual sets of frame center matched image frames. Between 1170 and 1194 tie points were used for registering the individual image frame sets.

Multitemporal DMC imagery was not orthorectified using USGS DEMs. Instead Digital Mapping, Inc. created a DTM using the stereo DMC imagery, and orthorectified the imagery using this DTM in combination with the airborne GPS/IMU data collected during the flight. The DTM was generated through autocorrelation of the triangulated images, and no breaklines or manual editing of mass points was performed. The cell size of the DTM was 12.2 m (40 ft).

4.2. Co-registration Error Assessment

Three 500 m by 500 m sites representing gentle, moderate, and extreme (GME) terrain conditions were selected from each study area to assess the spatial co-registration error of multitemporal image products from LFDC systems. Site selection followed a procedure designed to ensure that the gentle, moderate, and extreme sites were comparable between the Tecate and Big Bend study areas. First, arbitrary GME sites for each study area were selected based on visual review of a USGS NED 10 m DEM, and three terrain related attributes were summarized for each site to characterize the level of terrain complexity. The three attributes were elevation range, elevation variation (indicated by the elevation standard deviation), and average slope.

The terrain attribute values were reviewed for each site and study area, and value ranges representing gentle, moderate, and extreme terrain at both the Tecate and Big Bend study areas were identified (

Table 1. Terrain characteristic criteria for selecting suitable 500 m by 500 m sites for co-registration assessment.

Terrain Type/Site	Elevation Range (m)	Elevation Standard Deviation (m)	Average Slope (degrees)
Gentle	45-55	10	7
Moderate	95-105	10-20	10-20
Extreme	145-155	25-35	20-30

). A GIS modeling routine incorporating the DEMs was used to locate 500 m by 500 m areas having attribute values characteristic of gentle, moderate, or extreme terrain within the Tecate and Big Bend study areas. The resulting thematic maps provided a number of candidate sites within which GME terrain conditions were similar between the two study areas. The candidate sites were reviewed and gentle, moderate, or extreme terrain sites were selected for each study area (Figure 1). The terrain attribute values for the sites selected for the analysis are listed in

Table 2. Terrain characteristics at 500 m by 500 m sites for which co-registration accuracy was assessed.

Terrain Type/Site	Elevation Range (m)	Elevation Standard Deviation (m)	Average Slope (degrees)
Gentle - Tecate	52	10	7
Gentle - Big Bend	55	10	7
Moderate - Tecate	104	20	10
Moderate - Big Bend	97	20	17
Extreme - Tecate	155	35	23
Extreme - Big Bend	155	30	23

.

The multitemporal imagery was then subset to the exact boundary of each of these sites and the co- registration accuracy was assessed for each of the three terrain conditions (GME). Assessment of co- registration accuracy was based on the red waveband of each image set, as this waveband exhibits the greatest contrast in southern California Mediterranean-type ecosystems and likely the greatest contrast in the arid environment of Big Bend, TX as well [15]. The red wavebands of the ADS40 and DMC image sets were extracted from the CIR image products (not RGB), since the USBP is also interested in using high resolution image products for vegetation analysis.

Co-registration error was quantified by locating corresponding test points between subsets (with GME terrain) of multitemporal image sets and comparing the file (pixel) coordinates of the test points. The root mean square error (RMSE) of file coordinates was calculated using Equation 1 to quantify the co-registration error. Between 361 and 602 automatically generated but visually verified test points were used to assess co-registration error for each data set per 500 m by 500 m GME site.

$$\text{RMSE}=\sqrt{\frac{1}{n}\sum _{i=1}^n(\Delta X_i^2+\Delta Y_i^2)}$$

(1)

Where:

• n = the number of test points
• i = test point (TP) number
• ΔX_i = the X misregistration distance for TP_i
• ΔY_i = the Y misregistration distance for TP_i

5. Results

Root mean square error values of image-to-image co-registration for the various LFDC system image products are provided by terrain type in Figure 2. Excluding the UltraCam-D product provided by the vendor (UCD), the co-registration RMSE between like image products was less than 2.1 pixels (0.32 m) regardless of terrain type. The RMSE for UCD-FC in gentle and moderate terrain and the ADS40 LIDAR in moderate terrain was 0.5 pixels (0.075 m) or less. These results suggest that LFDC systems can achieve near-pixel level spatial-temporal co-registration for semi-automated change detection. An examination of the image preprocessing techniques utilized explains differences in the results per sensor.

The higher co-registration error of the UltraCam-D product provided by the vendor likely resulted from the image preprocessing guidelines requested by the authors. The vendor was asked to orthorectify and co-register the multitemporal image products using USGS DOQQ and DEM data for horizontal and vertical control, but was not provided guidance on the technique to use. The UltraCam imagery was processed using standard aerotriangulation techniques with each multitemporal image set separately. Therefore, positional errors resulting from the use of relatively low quality horizontal and vertical control (USGS DOQ and 30 m DEM) were present in each data set and were dissimilar (possibly compounded) between multitemporal image sets.

When the UltraCam-D imagery (collected with matched frame centers) was co-registered on a frame-by-frame basis using FC matching techniques (UCD-FC), co-registration errors were substantially lower and were comparable to error levels found in the other image products. The RMSE for the extreme terrain site dropped from 16.2 pixels (for UCD) to 2.1 pixels (for UCD-FC) when the FC matching technique was used for co-registration. The FC matching approach yielded low co- registration errors in gentle and moderate terrain (0.5 and 0.4 pixels, respectively).

Image co-registration errors for DMC imagery were generally consistent for each of the three terrain types, and varied between 1.4 and 1.6 pixel RMSE. The DMC imagery was collected using matched frame centers, but was not processed on a frame-by-frame basis using FC matching techniques. The multitemporal DMC image sets were triangulated and orthorectified using high quality GPS/IMU data and the same DTM derived from stereo imagery. This procedure yielded low co- registration error.

Orthorectification of ADS40 imagery with a LIDAR DTM (ADS40 LIDAR) resulted in multitemporal image products with low co-registration error. High quality GPS/IMU data and a high quality LIDAR DTM were utilized for orthorectification of both image sets resulting in accurate co- registration. Co-registration errors for the ADS40 image set orthorectified with a LIDAR DTM ranged from 0.4 to 0.9 pixels.

Orthorectification of ADS40 imagery with a USGS NED 10 m DEM (ADS40 NED) also resulted in imagery with low co-registration error (0.6 to 2.0 pixels). This result suggests that use of a highly accurate DTM during image orthorectification may not be required for accurate co-registration, but rather use of a consistent DTM with both multitemporal image sets may be sufficient. Absolute positioning of the ADS40 NED based image products was not as accurate as the ADS40 LIDAR based products, as the former exhibited high magnitude and variable shifts relative to the latter. However, for change detection purposes, high absolute positional accuracy may not be required.

6. Discussion

High resolution, multitemporal image sets acquired using LFDC systems may be co-registered with near-pixel level or sub-pixel level accuracy for the purpose of semi-automated change detection. While sub-pixel co-registration is generally considered sufficient for image-based change detection, the influence of misregistration on change detection accuracy will depend on the scale of land cover change features of interest and the heterogeneity of the scene. Townshend et al. (1992) [16] demonstrated that a spatial co-registration error of 0.2 pixels or better must be obtained to reduce NDVI change magnitude errors to less than 10% with moderate to coarse spatial resolution imagery in heterogeneous areas. If the combination of spatial co-registration error and land cover heterogeneity is such that misregistration is likely to affect change detection products created using LFDC systems, then low pass filtering or spatial aggregation of registered data sets could be used to minimize these effects while preserving land surface change “signals.” Further, spatial aggregation of two to three pixels with 0.15 m resolution imagery still provides sub-meter resolution for change detection.

The accuracy of co-registration between LFDC system image sets will depend upon the technique used to orthorectify the imagery. Of the vendor provided products, those utilizing GPS and IMU data for horizontal and vertical control of aerotriangulation and high quality DTM data for image orthorectification yielded the lowest co-registration error. DTM products were generated from stereo imagery (in the case of DMC imagery) or derived from LIDAR data (in the case of the ADS40 with LIDAR product).

Multitemporal frame array image sets were acquired with matched frame centers, and multitemporal ADS40 line array image sets were acquired by repeating acquisition along the same flight line. To achieve accurate spatial co-registration, imagery from LFDC systems should be acquired in this manner to reduce varying distortions between image sets that are caused by relief displacement. This will minimize misregistration errors that may not be corrected by the orthorectification procedures. Further, in the absence of high quality horizontal and vertical control, frame imagery acquired with matched frame centers may be processed on a frame-by-frame basis using the techniques of [7] to obtain accurate spatial co-registration.

Differences in co-registration error between the ADS40 LIDAR and ADS40 NED products were minimal for sites with gentle and moderate terrain (0.7 vs. 0.8 pixels and 0.4 vs. 0.6 pixels for gentle and moderate, respectively). However, for the extreme terrain site, co-registration error for the ADS40 NED image set was more than twice that of the ADS40 LIDAR image set (2.0 vs. 0.9 pixels, respectively). Visual review of the co-registration between ADS40 image sets for the extreme site indicates that low magnitude co-registration errors for the ADS40 LIDAR data are largely systematic and associated with errors inherent to GPS/IMU data and/or image resampling. Conversely, co- registration errors for the ADS40 NED image set are localized with variable magnitude and direction, with the highest errors in areas with the most extreme relief changes. These differences in co- registration error for the extreme terrain site are attributed to terrain distortions resulting from different view angles per pass that are not sufficiently corrected when using the NED DEM, but that are were well corrected when using the LIDAR DEM. This co-registration error can be expected to increase as terrain variability increases and/or when differences in viewing geometry between multitemporal passes increase. Therefore, high quality terrain data should be used with ADS40 imagery when accurate spatial co-registration between multitemporal image sets is desired.

7. Conclusions

Multitemporal imagery acquired and processed by vendors operating large format digital camera systems may be co-registered with near-pixel level or sub-pixel level accuracy, even in environments with high terrain relief. Once accurate co-registration is achieved, such high spatial resolution, multitemporal image sets can be utilized for semi-automated change analysis and detection of fine scale changes.

Co-registration accuracy will depend upon image acquisition and/or image preprocessing techniques. For this study, imagery from frame array sensors (i.e., DMC and UltraCam) was acquired with matched frame centers, and imagery from a line array sensor (i.e., ADS40) was acquired by repeating the same flight line (and same altitude) on multitemporal imaging passes. These methods of image data collection are recommended when accurate co-registration of multitemporal image sets is desired.

Results from this study suggest that achieving pixel-level co-registration with frame center matched frame imagery or flight line matched line array imagery requires at least: 1) accurate horizontal and vertical control for aerotriangulation and a consistent digital terrain model for orthorectification of each image set or 2) frame-to-frame co-registration of FC matched frame imagery. Incorporation of high quality digital terrain data for image orthorectification will provide the best results, but can be a more expensive solution. Accurate horizontal and vertical control may be generated with GPS/IMU or ground control points.