Automated Cart with VIS/NIR Hyperspectral Reflectance and Fluorescence Imaging Capabilities

Abstract

A system to take high-resolution Visible/Near Infra-Red (VIS/NIR) hyperspectral reflectance and fluorescence images in outdoor fields using ambient lighting or a pulsed laser (355 nm), respectively, for illumination purposes was designed, built, and tested. Components of the system include a semi-autonomous cart, a gated-intensified camera, a spectral adapter, a frequency-triple Nd:YAG (Neodymium-doped Yttrium Aluminium Garnet) laser, and optics to convert the Gaussian laser beam into a line-illumination source. The front wheels of the cart are independently powered by stepper motors that support stepping or continuous motion. When stepping, a spreadsheet is used to program parameters of image sets to be acquired at each step. For example, the spreadsheet can be used to set delays before the start of image acquisitions, acquisition times, and laser attenuation. One possible use of this functionality would be to establish acquisition parameters to facilitate the measurement of fluorescence decay-curve characteristics. The laser and camera are mounted on an aluminum plate that allows the optics to be calibrated in a laboratory setting and then moved to the cart. The system was validated by acquiring images of fluorescence responses of spinach leaves and dairy manure.

1. Introduction

Hyperspectral imaging has its origins in remote sensing [1] and has evolved into high-resolution imaging systems used in laboratory settings [2]. The goal of the project was to transition laboratory instrumentation back to field use to allow the acquisition of high resolution hyperspectral fluorescence images irrespective of solar lighting conditions. A specific laboratory objective was to address food safety risks associated with fecal contamination in produce fields. Our laboratory developed laser-induced fluorescence imaging technologies that can be used to detect fecal materials [3,4,5,6,7,8,9,10]. The cart-mounted system described herein was developed to be able to detect fecal materials in produce fields using these technologies.

Food-borne illness is an ever-present problem throughout the world. According to the Centers of Disease Control and Prevention (CDC), food-borne illness is responsible for the hospitalization of 128,000 Americans and 3000 American deaths each year [11]. Of the 9.6 million annual incidents of food-borne illness in the United States, 2.2 million (22%) are attributed to leafy-green produce [12]. Leafy green produce can be contaminated prior to harvest, and one of the most common sources of contamination is fecal material [13]. To reduce the risk of food-borne illness, the current practice is to have produce fields inspected by workers for fecal material and signs of animal intrusion before harvest [14]. The objective of this project is to develop an autonomous system to augment this current practice.

CALGMA (California Leafy Green Products Handler Marketing Agreement) recommends that both a pre-planting and a pre-harvest assessment be performed to check for fecal material or signs of animal intrusion. Signs of animal intrusion include damaged produce, evidence of feeding, fur, feathers, animal tracks, and other signs of animals in sufficient quantities. Problem sites are marked and nothing within a designated distance of the site is harvested [14].

Our laboratory has developed methods that use fluorescence responses to detect fecal materials [3,4,5,6,7,8,9,10]. One method utilizes laser-induced fluorescence and time-resolved hyperspectral line-scan imaging [5,6,7,8,9,10]. To use this technology in produce fields, a field deployable mounting system and a motorized cart are needed. Therefore, the goals of this study were to modify existing imaging technologies for field use, to develop a motorized cart, and to design and build a semi-autonomous system to facilitate the acquisition of hyperspectral data in fields.

This paper is organized as follows. In Section 2, the materials and methods used to develop the cart with camera and laser mount will be discussed. The cart design and field results are discussed in Section 3. Validation of the results and implications to future work are also provided.

2. Materials and Methods

2.1. Imaging

Images are acquired using a gated, intensified, camera (iStar DH334T-18U-73; Andor Technologies, Belfast, UK). Camera specifications include a USB interface, a minimum gate width of less than two ns, 1024 × 1024 pixels, 16-bit resolution, and responsivity in the visible wavelengths through the very near infrared. The camera is thermoelectrically cooled to −20 °C and incorporates a digital delay generator that is used to control image acquisitions. After enabling an image acquisition, an image is acquired after a programmed delay following input of a pre-lase trigger signal. A spectral adapter (ImSpector V9; Specium, Oulu, Finland) inserted between the camera and lens allows the acquisition of hyperspectral, line-scan, images. Lens selection is based on desired operational specifications such as the area to be imaged; tests were conducted using a 16 mm lens (M1614-MP2 2/3″ 16 mm F1.4; Computar, Tokyo, Japan). Images are acquired using a program written in Visual Basic Version 6 (Microsoft, Seattle, WA, USA) and an SDK (Andor Technologies, Belfast, UK), are corrected for dark current, and are saved in either 16-bit TIF or ENVI formats [10].

Illumination for reflectance images is provided by ambient solar radiation. Illumination for fluorescence excitation is provided by a frequency-tripled Nd:YAG pulsed laser (~6 ns pulse width; Ultra 100 THG WS MVAT, Quantel USA, Bozeman, MT, USA). The laser power supply (ICE450) generates a TTL signal pulse 100 ns prior to the Q-switch trigger. The 355-nm laser has a maximum pulse rate of 20 Hz and a maximum energy of ~31 mJ per pulse, and includes an attenuation module programmed using a serial interface. A Powell lens with a fan angle of 30° (Powell, 1987; 8.9 mm lens dia. for 4 mm beam dia., BK7 glass, Laserline Optics Canada, Osoyoos, BC, Canada) is used to convert the Gaussian energy profile of the laser beam into a line illumination source more appropriate for use with line-scan imaging. To address spillover from the 532-nm doublet wavelength and possible back-reflection from the expansion optics, a 4 mm iris and a 355-nm laser window tilted 5° from the optical axis (Techspec 1/20λ high-power laser-line window, Edmund Optics Inc., Barrington, NJ, USA) are inserted between the laser and the Powell lens. The expansion optics are mounted in standard 1″ lens tubes that are connected to the laser using a fabricated mounting plate with SM1 threading. The Powell lens is mounted in a rotation mount (CRM05, Thorlabs, NJ, USA) after first inserting the lens into a hole drilled into the center of a threaded plug (SM05PL, Thorlabs). The rotation mount allows the angle of the illumination profile to be adjusted to match the field of view of the hyperspectral adapter. Offsets are addressed by adjusting the position of the laser relative to the camera.

2.2. Cart

The cart frame consists of 40 mm by 40 mm black anodized aluminum extrusions that are connected using standard fastening components (Parker Hannifin Corporation, Cleveland, OH, USA, 2005). A stepper motor (AR98MK-PS36-3, Oriental Motor U.S.A. Corp., Braintree, MA, USA) powers each of the two front wheels. Both motors are controlled using a single controller (EMP Series Programmable Motion Controller, Oriental Motor U.S.A Corp.) using a serial interface and ASCII commands. Motor mounts (SPL-9SB, IndustrialeMart, Mundelein, IL, USA) are used to attach the motors to the cart frame. The 6.25″ dia. wheels (6″ Traction Tires with VersaHub, VEX Robotics, Greenville, TX, USA) are attached to the frame using PPB8 pillow block housings and 0.5″ steel shafts. The front wheels are connected to the motors using chains (ANSI 35 roller chains) and sprockets (22 and 15 tooth, respectively). With this configuration, a single pulse to the motor corresponds to a horizontal movement of about 0.0094 mm. A 24 VDC 5 Amp Power Supply (PSB24-120S, AutomationDirect, Cumming, GA, USA) is used to power the motors and the control electronics. All electronics are mounted on a DIN rail in a waterproof enclosure (Attabox Polycarbonate Enclosure Clear Cover, 18″ × 16″ × 10″, Attabox, Jacksonville, TX, USA). Cart movement is controlled using a laptop computer and the data acquisition program [5]. The cart is generally powered using a long extension cable. It can also be run for about 30 min using an onboard Uninterruptible Power Supply (UPS). As most of the electrical components can use 24 VDC, longer duration un-tethered runs would require the use of a 24 VDC rechargeable battery and an inverter for the few components that can only use 120 VAC.

2.3. Camera and Laser Mount

The camera and laser are mounted on a 0.25″ aluminum plate to allow alignments to be preserved when the optics are moved from the laboratory to the cart and when, during field use, the equipment is subject to normal transport vibrations. Determining the appropriate offset and angular placement of the laser relative to the camera required modeling the expected illumination field relative to the imaging field. The model also considered the height of the optics above the ground, the angle of the aluminum plate relative to fully vertical, the width of the desired imaging field, and angles relative to the optics selected for use. Calculations were done for 1° arc intervals using the tangent function. To align the plane of the laser to the plane of the camera, holes were drilled in the aluminum plate so that the bolts attached to the laser could be used to adjust the distance of the laser from the aluminum plate. Rotational alignment of the Powell lens and offset adjustments of the laser requires very precise movements. To facilitate such movements, cable ties were added as appropriate and a micrometer mounted on a magnetic platform is used to push against the cable ties.

Precise alignment of the illumination field with the imaging field is a multistep process. As the 355 nm laser illumination is not within the spectral response range of the camera, fluorescence responses of glossy photo paper are used where appropriate for alignment image acquisitions. First, the spectral adapter has to be rotated so that the imaging field is parallel to the plane of the aluminum plate. Second, the Powell lens has to be rotated until the illumination field is also parallel to the plane of the aluminum plate. Rotational alignments are facilitated by a program written in-house that calculated the least squares fit of a line to an image within a selected region of interest. The program displays the angle and intercept of the calculated line. To determine the column of pixels that corresponded to the center of the imaging line of the spectral adapter, a picture with a white vertical stripe on a black background is moved across the field of view using a translation table, with a halogen lamp used for illumination. The physical location that maximizes the spectral response is noted, and the translation stage is moved to that position. The spectral adapter is then removed and to allow acquisition of full-field images. The pixel column that corresponds with the center of the white stripe in the full-field images is considered to be the center of the effective imaging field of the spectral adapter. The offset of the laser from the aluminum plate is adjusted until full-field images of the illumination profile are centered on this column. To make sure that the illumination plane is parallel to the imaging plane, full-field images of the laser illumination need to be acquired with the target area at multiple distances from the camera.

2.4. Control Program

A computer program written using Visual Basic 6 to integrate image acquisition, control of the laser, and stepping of a translation table was modified to also allow control of the cart. Information on each aspect of control (including: laser, camera, table, and cart) is displayed in respective frames on a summary screen. Towards the center of the summary screen is a frame that displays the spreadsheet parameters that will be used to acquire sets of images at each step of the translation table or cart. Clicking on any of the individual frames brings up a form where appropriate frame values can be updated.

2.5. Cart Performance Testing

To verify cart movement, videos were acquired from the front and side of the cart with the cart moving at different programmed velocities. The accuracy of step intervals was estimated by measuring the distance the cart moved following a programmed number of steps.

2.6. System Validation

Dairy manure was acquired from the United States Department of Agriculture (USDA) Beltsville Dairy Facilities. Bagged baby spinach leaves were acquired from a local grocery store. Dollops of manure and leaves were placed on a non-reflective plastic surface which was then placed on an asphalt road surface. The cart was programmed to take images with a step size of 2.5 mm. The pixel resolution was about 1.5 mm/pixel. Reflectance line-scan images were acquired using an exposure time of 0.1 ms and a camera gain of 3000. Fluorescence images were acquired using a laser attenuation of 150 and a camera gain of 3000 and two different exposure settings: 5 ns delay with 25 ns gate width and 20 ns delay with 10 ns gate width.

3. Results and Discussion

3.1. Cart Design

Figure 1 shows the cart and a schematic view indicating measurements used to model the effects of camera and laser placement. Figure 2 shows the stepper motor configuration in more detail. The choice of motor and gearing was governed by the need for sufficient torque to allow the cart to move from a dead stop up a slight incline on loose surface material (Figure 3). The tradeoff was the top speed of the cart, which is about 2.2 km/h. After testing, it was decided that the design constraints were overly restrictive and, if needed, sprocket configuration could be altered to allow the top speed to be doubled, which would also double the distance moved in response to a single pulse. Alternatively, a gearbox could be added or, if only continuous motion was desired, the stepper motors could be replaced by servo motors.

Videos of the cart movement on pavement indicated that the motion was uniform by step. The only quirk was the finding that one motor started and stopped a fraction of a second before the other motor. It is assumed that this is due to the sequencing of events by the controller. Alternating motor starts on a step by step basis was considered, but was not implemented as there did not appear to be a resulting directional bias in movement over a number of steps. The goal is to be able to manually position the cart at the start of a crop row and then have the cart move down the row. It is not clear if active steering using differential control of the individual motors will be necessary to keep the cart on course. In case there is a problem, a prototype control algorithm was developed that uses the imaging data to alter motor commands to follow the edges of a crop row. Testing will require field deployment of the cart and optics system. Another problem that needs to be addressed for actual use in produce fields is the variety of widths of crop rows across different fields and crops. The width of the cart can be changed by altering the length of the cross bars.

The cart was designed with the potential to be fully autonomous. The front section with the motors could be replaced with a single axle that allows steering control. Alternatively, the rear wheels are mounted so as to facilitate the addition of two additional stepper motors to allow independent four-wheel drive. Observations in actual produce fields suggest that full autonomy would be difficult to achieve due to area restrictions at the end of some rows and occasional instances of rough terrain (e.g., large deep grooves made by tractors when the ground was muddy).

3.2. Camera and Laser Mount Design

There are a number of considerations for mounting the camera and laser given that these components do not share a common aperture and are being used in a line-scan mode. First, the laser and camera can be mounted so that the illumination and imaging fields are on intersecting planes or on a single plane. For both configurations, the illumination field and the imaging field will only fully overlap for a limited range of focal distances (Figure 4). One advantage of single plane configuration is that the imaging field will always be at the center of the illumination field, so changes in the focal distance will not alter the effective illumination intensity given a uniform intensity along the center axis of the illumination profile. In contrast, when using the intersecting planes configuration, illumination intensity is a function of the focal distance. Another advantage of the single plane configuration is that the usable range of focal distances can be expanded by increasing the length of the illumination profile relative to the imaging profile.

A second consideration is whether to mount the components in parallel, or with one or both components at an angle. This consideration is complicated by the limited choices in the selection of Powell and camera lenses. Given the constraint that the desired imaging field of view should be 1.5 m and the height of the components above the ground should be less than 3 m, the best choices are a 30° Powell lens and a 16 mm camera lens. With this selection, if the components were to be mounted in parallel, the illumination field would slightly smaller than 1.5 m. It was decided to mount the camera perpendicular to the ground and to angle the laser so as to stretch the illumination field. Figure 5 shows this mounting arrangement. Figure 6 demonstrates the equations used to calculate the offset and angle of the laser that would allow the illumination field to be 1.5 m. The final design called for the components to be separated by 105.1 mm at an angle of 2.0°. This positioning produces overlapping 1.5 m fields when the components are mounted 2.74 m above the ground at an angle of 12.2° relative to fully vertical. The components were mounted at an angle relative to fully vertical for stability, i.e., so that the center of gravity remained within the wheelbase if the cart were to transverse a small incline.

Angling the laser relative to the camera produces a non-uniform illumination intensity profile along the major axis of the illumination. Figure 7 shows the calculated relative distortion in illumination. While it is possible to mathematically correct images for this distortion, this is probably not warranted as the variability introduced if this distortion if ignored is small compared to the response differentials that would be expected for fecal contaminates.

Figure 8 shows the camera and laser mounted on the cart. Use of the aluminum mounting plate allows calibrations to be done in the laboratory and then transfer of the plate to the cart without the need for further calibration. Figure 9 shows a screenshot of the calibration program used to align and test the uniformity of illumination in the laboratory. As can be seen, uniformity along the major axis is very good.

3.3. Validation

Reflectance and two sets of fluorescence hyperspectral data cubes were acquired of a non-reflective board that had three baby spinach leaves and three dollops of dairy manure on it. Figure 10 shows a photograph of the target. Figure 11, Figure 12 and Figure 13 show an image at a selected wavelength and representative spectra for the leaves and for the manure. The banding in the reflectance images is due to changes in ambient illumination over time. For one fluorescence acquisition (Figure 12), the gate delay and width were set to capture the full fluorescence response of both the leaves and the manure. For the other fluorescence acquisition (Figure 13), the delay was extended to allow the response from the leaves, which has a faster decay rate, to die out while still capturing the tail of the response from the manure. This result suggests that detection of fecal materials in fields could be accomplished using a single wavelength by taking into account relative fluorescence decay characteristics of fecal materials and intact plant materials. The ability to differentiate the fluorescence responses from intact plant materials and from manure using differences in fluorescence decay rates has been previously documented in laboratory settings [7,9,10].

4. Conclusions

The primary impetus for developing this laser-induced VIS/NIR fluorescence imaging system was to allow the acquisition of high resolution hyperspectral fluorescence images in fields, irrespective of solar lighting conditions. The validation of the system described herein represents the successful transition of high resolution laser-induced fluorescence imaging for use in fields. The system is unique and allows examination of problems such as the spread of fecal materials in produce fields over time. A particular advantage of the system is the ability to progress in small, precise, steps and to acquire sets of images with different acquisition parameters at each step. This ability allows the system to be used to model the operating characteristics of different optical systems and to contrast detection efficacy using identical, real world, targets. In particular, this ability could be used to address potential cost and detection sensitivity tradeoffs among different camera systems. For commercial interests, this would allow the selection of the least expensive camera and optics that would likely detect the level of contamination of concern. Another use of the ability to acquire sets of images would be to examine fluorescence decay characteristics by acquiring fluorescence response data at appropriate delay times and intervals.

For food safety, detection of fecal materials in fields is a recognized concern and validation results demonstrate the ability to detect fecal materials outside during daylight hours and to differentiate fecal responses from the responses of intact plant leaves. For a commercial system to be used to detect fecal materials in fields, validation results suggest that detection could be accomplished using a single wavelength by taking advantage of the faster fluorescence decay of intact plant materials compared to the slower fluorescence decay of fecal materials. While issues related to food safety were the driving factors for development of this system, the system is a powerful research tool that has the potential to address a wide variety of research concerns.