Image Correction and In Situ Spectral Calibration for Low-Cost, Smartphone Hyperspectral Imaging

Abstract

Developments in the portability of low-cost hyperspectral imaging instruments translate to significant benefits to agricultural industries and environmental monitoring applications. These advances can be further explicated by removing the need for complex post-processing and calibration. We propose a method for substantially increasing the utility of portable hyperspectral imaging. Vertical and horizontal spatial distortions introduced into images by ‘operator shake’ are corrected by an in-scene reference card with two spatial references. In situ light-source-independent spectral calibration is performed. This is achieved by a comparison of the ground-truth spectral reflectance of an in-scene red–green–blue target to the uncalibrated output of the hyperspectral data. Finally, bias introduced into the hyperspectral images due to the non-flat spectral output of the illumination is removed. This allows for low-skilled operation of a truly handheld, low-cost hyperspectral imager for agriculture, environmental monitoring, or other visible hyperspectral imaging applications.

1. Introduction

Hyperspectral imaging has risen to prominence in recent years due to its growing role in sensing applications in agriculture and environmental [1,2,3,4] monitoring, civil engineering [5,6] and medical applications such as cancer and Alzheimer’s detection [7,8]. It is suited to these applications because it comprises additional information that is contained within a hyperspectral image. A specific example used in this work to demonstrate the advantage of defect detection and quality assurance in fruit afforded by hyperspectral imaging over conventional monitoring techniques is well documented in the literature. It is shown not as a novel application but to illustrate the utility of the technique we propose in this work. In digital photography, the pixel brightness of an image, measured in digital levels (DL) corresponds to wavelength-integrated radiance, either across the spectral range of the sensor for monochromatic cameras or within the spectral bands of the Bayer filter in colour imaging. A hyperspectral image is, in contrast, a multitude of images all captured contemporaneously. Each image corresponds to a narrow band of wavelengths (typically 1–10 nm full width half maximum (FWHM) [9,10]) and each pixel value is the wavelength-integrated radiance across that band. This allows features to be detected in a reflection spectrum that are otherwise lost in traditional imaging techniques. For instance, this is applied to aerial monitoring of large ground areas, disease detection in plants and to aid the segmentation of cells in microscopy [11,12,13].

Hyperspectral imaging can be prohibitively expensive and so different approaches have been taken to produce hyperspectral images at reduced cost [14]. Coded aperture snapshot spectral imaging (CASSI)-based hyperspectral imaging systems use a coded aperture to encode spectral information and produce hyperspectral images. This technique can produce hyperspectral images at high frame rates but requires intensive post-processing [15]. Scanning systems offer the advantage of producing images that can be easily extracted from raw data captured where high-speed acquisition is not required, for example, in agricultural measurements. Rapid data collection is often not necessary because vegetation changes over time scales of tens of minutes or hours [16]. Such systems are often attached to drones to produce hyperspectral images of large areas such as fields to assess characteristics of large numbers of plants simultaneously [17]. Lab-based systems are also used to assess individual plant properties, such as damage and/or deterioration in fruits and other fresh produce for non-destructive quality control and assessment [9,16]. Wider availability of low-cost, easy to operate systems that require minimal post-processing would enable better individual plant monitoring during growth and improved quality control in food distribution.

The maturity of silicon focal plane array (FPA) camera sensors, coupled with advances in smartphone technology affords significant developments in handheld scientific instrumentation [18]. Smartphones can be easily adopted as platforms for prototype and commercial instruments in a wide variety of scientific fields due to the fast on-board computing power and built-in sensors [9,19,20,21,22,23,24,25,26,27]. In a previous publication, a smartphone-based hyperspectral imaging device was reported and demonstrated as a viable, low-cost alternative to more expensive lab-based systems [9]. This device was very simple in its design but demonstrated the potential of smartphone technology in a side-by-side comparison with a non-portable lab setup.

Push-broom scanning systems, such as the one described above, are limited in two ways. Firstly, when operated as a handheld scanner, they suffer serious image distortion due to ‘operator shake’; the inability of users of hand-held devices to translate the instrument across a scene with perfectly linear motion and constant velocity [9]. This results in distorted images with regions of spatial compressions and expansion. This can be avoided using a motorised translation stage either to translate the imaging system [9] or the target of the imaging [28,29]. Alternatively, a three-axis gimbal has been shown to reduce the effect of operator shake [30]. However, these systems require bulky or expensive equipment and somewhat reduce the utility and cost-effectiveness of a handheld system.

We address the aforementioned deficiencies of push-broom hyperspectral imaging systems in this work utilising two in-scene spatial refences; one to correct horizontal distortion and the other for vertical distortion. Both references leave measurable traces in the hyperspectral images that can be used to correct the images for spatial distortion. This is an ultra-low-cost solution because the in-scene reference is simply printed on a sheet of white paper, with the added novel complexity hidden within a software algorithm. Slight rotation of the system during a scan does produce distortions in the final image but are negligible compared to the horizontal and vertical distortions.

The second limitation is that of calibration drift. Calibration of such systems is generally performed using known spectral sources such as a xenon, mercury or sodium lamp [31,32]. Spectral features can be identified from instrument output, and these are used to establish the range and resolution of the instrument. For push-broom design imagers [9,30], where a vertical slice of the scene is captured while the system scans in the other spatial dimension, any change in alignment between the slit, grating and imaging optics of the system causes the spectral calibration to drift. This is because the spectrum of the scene alters and is no longer projected onto the same region of the FPA. The probability of this is greater for low-cost systems, where mechanical rigidity is traded for ease of production and reduction in price; especially where the system is being deployed for long periods of time in a field environment.

In situ calibration is ideal for any instrument because it reduces the probability of drift in the instrument between the measurement and the calibration. However, it is not always practical or even possible to perform an in situ calibration due to the need for extra equipment and the subsequent increase in measurement duration and post-processing.

Our solution to this problem is a ground-truth in-scene reference target. This is imaged simultaneously with the object of the imaging providing a known spectral reflection. The target is three colours, red–green–blue, each of which have had their reflection spectrum measured beforehand with a spectrometer (Thorlabs CCS200). A comparison of the ground-truth reflection spectrum to the measured, uncalibrated spectrum from the instrument allows for the hyperspectral image to be calibrated. Specific features in the measured spectrum relate to those of the ground-truth spectrum and provide reference points for spectral calibration.

There are many sources of systematic error in a hyperspectral image. For example, the spectral response of silicon is temperature dependent [33]. More importantly, the light source that illuminates the scene is not spectrally flat, which introduces a spectral bias and can vary significantly with time in the case of natural light [34]. The in-scene reference determines, and removes, the spectral bias inherent in raw hyperspectral images. The removal of the spectral biases on the measured reflection spectrum is shown in our results to be robust to different sources of illumination. This is demonstrated in this work for natural light, LED illumination and incandescent light.

In this work, we utilised the ultra-low-cost system described by Stuart et al. [9] to highlight the existing limitations of handheld scanning techniques before presenting an innovative solution to these problems and demonstrating the potential of the instrument as a handheld hyperspectral imager in a range of environmental monitoring applications. An in-scene reference target and an algorithm coded in MATLAB for vertical and horizontal spatial correction are used alongside an in situ light-source-independent spectral calibration. The result is a spatially accurate hyperspectral image which is spectrally calibrated, captured with a low-cost, fully handheld, and portable smartphone hyperspectral imaging system. This provides a novel, low-cost alternative to existing hyperspectral imaging techniques and can be applied to a wide range of handheld imaging applications that have been hitherto limited by spectral and spatial distortions.

Figure 1 shows a brief overview of the stages detailed in this work in the form of a workflow diagram. This describes the steps sequentially as they were performed during the work, but not necessarily the order they are covered in the body of the work.

2. Materials and Methods

2.1. Image Acquisition

A smartphone-based hyperspectral imaging system was used throughout this work. The system has a 14 nm full width half maximum spectral resolution and a spectral range of 400–700 nm. The field of view of this instrument is determined by the slit height and scan duration. The system was the same system characterised and presented by Stuart et al. [9] and a more detailed breakdown of this instrument and its components is available in that work. The hyperspectral imaging system used in our developments was a push-broom style scanning system, utilising the two spatial dimensions within the FPA of the smartphone camera to capture a vertical ‘slice’ of a scene in the spectral plane. A series of such slices are acquired at 30 frames per second by the FPA and combined to produce a hyperspectral image by way of a translational scan of the full scene. Each frame captured by the camera corresponds to the visible spectrum of one of these vertical slices of the image, as shown in Figure 2. A typical scan duration was between 6 and 12 s for an object at a working distance of 40 cm.

Once the full scene had been scanned, each column of pixels in the captured spectrum was assembled into a series of images. Each of these corresponded to a different band of the visible spectrum. This formed a data cube; a stack of images where the x–y axes relate to spatial dimensions and the third axis relates to the spectral plane. This idea is illustrated in Figure 3. A colour image could be created as a convenient reference by selecting three images from the red, green, and blue parts of the spectrum and using these as colour channels.

To obtain a hyperspectral dataset using our handheld setup, the target object was placed within the “target object location” section of a calibration card, as shown in Figure 4. The vertical correction reference line, located at the bottom of the calibration card is a straight line that runs parallel to the scan direction. This was used to determine the amount of vertical distortion within a captured scene. The line will deviate in the output images proportionally to the amount that the scanner deviates from the ideal path. A correction algorithm written in MATLAB was then used to carry out a correction. The calibration card was printed on a sheet of A4 paper for this work.

The bottom section of the image was thresholded according to the minimum brightness value of the column in question. This allowed the distance in pixels that the line had deviated from straight (the y-deviation value) to be measured for each x-value of the line-mask. Each column of pixels was then shifted down by the corresponding y-deviation value. Figure 5 show this process visually in stages.

The horizontal correction scale bar was used to determine the compression or expansion of the image due to the varying velocity of the scan along the scan direction. It was comprised of alternating black and white squares and the widths of these squares varied with scan speed. The top region of the images was thresholded to produce a mask which allowed the widths of the squares to be measured for each square. A line profile in MATLAB allowed the widths of the black and white thresholded regions to be quantified in pixels. The threshold value was half the height of the top hat function produced by taking a line profile of the horizontal correction scale bar and is expressed in Equation (1):

$$\mathrm{Threshold}={\mathrm{mask}}_{\min }+\frac{{\mathrm{mask}}_{\max }-{\mathrm{mask}}_{\min }}{2}$$

(1)

where ${\mathrm{mask}}_{\min }$ and ${\mathrm{mask}}_{\max }$ were the minimum and maximum values of the selected region, respectively. The image was then sliced into vertical segments of widths corresponding to the widths of the squares of the horizontal correction scale bar. Each of these slices was then automatically resized using the imresize function in MATLAB to have width equal to that of the height of the horizontal correction squares. The resized image slices were then recombined to produce the corrected images. This corrected for the horizontal distortion induced by non-constant scanning speed. The steps of this process are shown visually in Figure 6.

2.2. Bias Correction

The images of the data cube needed to be sensor-bias corrected before they could be spectrally calibrated. The dark signal (signal present due to the offset voltage of the sensor, thermal noise, etc.) was subtracted RGB channel-wise to dark correct the images.

The resulting measured signal in each image was dependent on many influences as shown below in Equation (2) below:

$${\mathrm{S}}_{\mathrm{image}}\left(\mathsf{\lambda },\mathrm{x},\mathrm{y}\right)=\mathrm{QE}\left(\mathsf{\lambda }\right)\times {\mathrm{T}}_{\mathrm{Optics}}\left(\mathsf{\lambda }\right)\times {\mathrm{T}}_{\mathrm{Bayer}}\left(\mathsf{\lambda }\right)\times {\mathrm{Q}}_{\mathrm{e}}\left(\mathsf{\lambda }\right)\times {\mathrm{R}}_{\mathrm{object}}\left(\mathsf{\lambda },\mathrm{x},\mathrm{y}\right)\times {\mathrm{G}}_{\mathrm{eff}}\left(\mathsf{\lambda }\right)$$

(2)

where ${\mathrm{S}}_{\mathrm{image}}$ is the measured signal in an image, $\mathrm{QE}$ is the quantum efficiency of the silicon sensor, ${\mathrm{T}}_{\mathrm{Optics}}$ is the spectral transmission of the optical system, ${\mathrm{T}}_{\mathrm{Bayer}}$ is the spectral transmission of the Bayer filter on the sensor, ${\mathrm{Q}}_{\mathrm{e}}$ is the spectral radiant energy from the light source, ${\mathrm{R}}_{\mathrm{object}}$ is the spectral reflectivity of each point in the scene and ${\mathrm{G}}_{\mathrm{eff}}$ is the grating efficiency. Ideally, hyperspectral images are a measure of the spectral reflectivity only. A ratio of the measured signal at each point in the image to the signal at a point in the scene, specifically a white point on the paper test card, was taken to achieve this. This result is expressed in Equation (3) below:

$${\mathrm{S}}_{\mathrm{calibrated}}\left(\mathsf{\lambda },\mathrm{x},\mathrm{y}\right)=\frac{{\mathrm{S}}_{\mathrm{image}}\left(\mathsf{\lambda },\mathrm{x},\mathrm{y}\right)}{{\mathrm{S}}_{\mathrm{paper}}\left(\mathsf{\lambda }, \mathrm{x}, \mathrm{y}\right)}\times {\mathrm{R}}_{\mathrm{paper}}\left(\mathsf{\lambda }\right)$$

(3)

where the ratio must be multiplied by the reflection spectrum of the paper, ${\mathrm{R}}_{\mathrm{paper}}$, because its inverse was introduced when the ratio was taken. This was achieved after the spectral calibration (Section 2.3) had been performed but is shown here because it was the last stage of the bias correction. All the other terms cancelled because they were present and equal in both signals and only the reflection spectrum of the object at a point ($\mathrm{x},\mathrm{y}$) in the image remained. This is illustrated in Figure 7.

2.3. Spectral Calibration

The methodology described above produced an uncalibrated hyperspectral image, corrected for light-source bias and sensor bias. The images were then calibrated for wavelength, which was achieved through the use of an in-scene reference.

The spectral calibration reference target shown in Figure 4 was used to calibrate the images after spatial correction and bias correction with the reflection spectrum measured using a spectrometer (Thorlabs CCS200). The target was illuminated with a broad-spectrum white LED and the reflection spectrum of the red, green, and blue sections of the target were measured. The LED light source emission spectrum was then measured and the reflection spectra were corrected as per Equation (4).

$${\mathrm{R}}_{\mathrm{corr}}=\frac{{\mathrm{S}}_{\mathrm{RGB}}}{{\mathrm{S}}_{\mathrm{LED}}}$$

(4)

where ${\mathrm{R}}_{\mathrm{corr}}$ is the true reflection spectrum of the spectral calibration reference target section, ${\mathrm{S}}_{\mathrm{RGB}}$ is the measured reflection spectrum of the test card section and ${\mathrm{S}}_{\mathrm{LED}}$ is the measured signal of the emission spectrum of the light source. ${\mathrm{R}}_{\mathrm{corr}}$ was then normalised to unity for each colour section of the spectral calibration reference target. After the data cube was assembled, the normalised, ground-truth intensity for the spectral calibration reference target could be compared with the calibrated spectra of the spectral calibration reference target. The crossover points between blue and green and green and red were recorded from the calibrated spectra as 495 nm and 596 nm, respectively. These features could be identified in the uncalibrated spectra and gave known points to interpolate between, and extrapolate from, to provide quantitative wavelength values for each image.

The light source correction could not be fully applied until the spectral calibration was complete; the paper’s reflection spectrum could not be multiplied by the data until there was a 1:1 wavelength correspondence. However, the ratio of ${\mathrm{S}}_{\mathrm{image}}$ and ${\mathrm{S}}_{\mathrm{paper}}$ was still taken, and the reflection spectrum of the paper was introduced to the ground-truth reflectance spectrum of the spectral calibration reference target. Both the ground-truth and smartphone-measured reflectance spectra of the spectral calibration reference target had the influence of the inverse reflection spectrum of the paper. Figure 8 shows a comparison of the ${\mathrm{S}}_{\mathrm{image}}$/${\mathrm{S}}_{\mathrm{paper}}$ ratio signal and the lab-measured spectral reflection of the spectral calibration reference target with the ©nverse of the paper’s reflection spectrum introduced.

The widths of Δλ’ and Δλ were used to calculate the wavelength increment between images in nanometres by taking the ratio of Δλ to Δλ’. This increment was then applied to each image from the starting point of the known point at 495 nm and allowed a wavelength value to be assigned to each image in the data cube, thus spectrally calibrating the data.

3. Results

The robustness of the spatial correction to operator shake is shown in Figure 9. The image from the data cube is significantly less recognisable before the spatial correction was applied. The corrected image shows how effective the spatial correction algorithm can be, even when supplied with images heavily affected by operator shake. The improvement is sufficient as to provide context to the spectral data. This is critical for field use where capturing images by hand in potentially inhospitable environments means operator shake is a serious problem. The spectral data is not affected by the spatial correction, being able to relate spectral information to a spatial reference within the image is key to extracting the relevant information from the hyperspectral image.

Although additional optics in the imaging system could produce better spectral and spatial resolution, they significantly increase the cost of the system, therefore reducing the low-cost accessibility of the instrument. Figure 9 presents the utility of the spatial correction. The scan time is short and so the horizontal resolution is limited. This demonstrates that spatial correction significantly improves the utility of the handheld system by providing better context for the spectral information allowing for more convenient retrieval of spectral artifacts of interest within the scene.

The instrument’s light-source-independent spectral calibration is demonstrated in Figure 10. The reflection spectra of a sample of lapis lazuli are shown, measured with illumination from three different light sources. ©e reflection spectrum as measured with the Thorlabs CCD spectrometer as a reference as the blue dotted line. All three of the measured spectra from the smartphone show agreement on the trend of a peak in the blue at 485 nm with a tail off towards the red before showing the indication of an increase towards the near infrared.

The data shown in Figure 10 were captured with a single scan for each light source. The natural sunlight plot shows the closest resemblance to the lab-measured spectrum with a root mean squared error (RMSE) of 0.014 and this is to be expected because it was a more intense light source and provided strong illumination at all wavelengths. The spectrum measured under incandescent illumination shows a closer resemblance to the lab-measured spectrum than does the LED illuminated measurement below 525 nm. However, the LED data have a marginally lower RMSE compared to the incandescent data with a RMSE of 0.0030 and 0.0033, respectively. There is more noise present in the LED measurement indicative of low signal. However, the LED spectrum in Figure 10 shows there are lower levels of illumination across the spectrum when compared to the natural and incandescent illumination spectra. The LED spectrum also contains a strong localised peak at 450 nm; however, this presents in the measured spectrum of the sample only as a slight increase which is within the variation due to noise. All of this indicates a resilience to uneven spectral illumination in our approach to in situ calibration. This demonstrates the instrument’s ability to work effectively in a range of illumination conditions, allowing for accurate spectral calibrations to be achieved in both outdoor and indoor settings. This further demonstrates the utility of this instrument as a whole, as a fully portable handheld hyperspectral imaging device, capable of accurate and robust hyperspectral analysis.

4. Discussion

Example Applications

Hyperspectral imaging is an invaluable tool, benefitting a wide range of environmental monitoring applications [18]. In vegetation monitoring and precision agriculture applications in particular, the need for early and accurate diagnosis of plant stress is critical to the mitigation of crop losses [35,36,37,38]. Furthermore, the accurate and early detection of poor-quality produce can significantly reduce losses due to spoiling products destined for customers and further food production [14,39]. To date, a range of hyperspectral imaging systems have been utilised as effective, non-destructive means of vegetation health monitoring and quality assessment in food products [1,40,41,42].

Hyperspectral imaging provides a non-invasive, rapid means of determining plant quality and health, delivering significant benefits over traditional monitoring methods. The handheld hyperspectral smartphone system further benefits these applications by the user-friendly nature of the instrument and combined with the benefits of the spectral and spatial calibration techniques discussed within this article, this makes it a highly effective accessible hyperspectral imaging system and a powerful tool for accurate quality assessment.

The instrument’s utility as a hyperspectral imaging system in quality assessment applications is shown in Figure 11, where a red Gala apple with bruising damage is used as the target object. The apple has a bruise in its centre which is more apparent at certain wavelengths than others, e.g., 670 nm. This demonstrates the ability of this instrument because it is capable of detecting damage and defects before they become apparent to the naked eye, providing valuable time to prevent the loss of further products within a large batch [43,44]. Figure 11 shows bruising is much more apparent withing the red portion of the spectrum due to the increased reflectivity of healthy tissues at these wavelengths. This can be compared to the reduced visibility, particularly within the blue portion of the spectrum due to the lower reflectance of fruit tissue at these wavelengths [14]. This demonstrates the utility of hyperspectral imaging for the purpose of quality control because the imaging allows for enhanced contrast of certain features that present at specific wavelengths that are lost when imaged using Bayer-filter photography that integrates over wider bands of wavelengths. This is particularly pertinent in fruits with darker pigmentation because the damage can remain undetectable by traditional methods for extended periods, increasing the potential of further losses [45,46].

5. Conclusions

A method for spatially correcting for operator shake and performing in situ spectral calibration in handheld hyperspectral imaging has been reported alongside a demonstration of its application withing the field of fruit quality assessment. This technique dramatically improves the utility of a low-cost, smartphone-based hyperspectral imaging system using a printed in-scene reference card enabling accurate measurements to be taken in the field without need for translation stages. This work broadens the application of low-cost hyperspectral imaging to industries and scientific investigations for which it would otherwise be insufficiently robust to long measurement sessions in the field. The in situ light-source-independent calibration allows for extended field operation while minimising the influence of calibration drift and removing the need for calibration after the imaging has been completed. The foundations have been laid for agricultural and environmental monitoring studies that require non-destructive testing where the availably of data is afforded by the low cost of the systems.