Design of an Airborne Low-Light Imaging System Based on Multichannel Optical Butting

Abstract

For the purpose of achieving long-range, high-resolution, and ultra-wide-swath airborne earth imaging at extremely low-light levels (0.01 Lux), a low-light imaging system built on multi-detector optical butting was researched. Having decomposed the system’s specifications and verified its low-light imaging capability, we proposed to employ an optical system with a large relative aperture and low distortion and achieve imaging through the field-of-view (FOV) butting facilitated by eight 1080P high-sensitivity scientific complementary metal-oxide semiconductor (SCMOS) detectors. This paper elaborates on the design concept of the mechanical configuration of the imaging system; studies the calculation method of the structural parameters of the reflection prism; provides mathematical expressions for geometric parameters, such as the length and width of the splicing prism; and designs in detail the splicing structure of six reflection prisms for eight-channel beam splitting. Based on the design and computational results, a high-resolution, wide-swath imaging system for an ambient illuminance of 0.01 Lux was developed. Exhibiting a ground sampling distance (GSD) of 0.5 m (at a flight height of 5 km), this low-light imaging system keeps the FOV overlap ratio between adjacent detectors below 3% and boasts an effective image resolution of 4222 × 3782. The results from flight testing revealed that the proposed imaging system is capable of generating wide-swath, high-contrast resolution imagery under airborne and low-light conditions. As such, the way the system is prepared can serve as a reference point for the development of airborne low-light imaging devices.

1. Introduction

In a natural environment without any artificial light sources, the ground illuminance on a clear night with a full moon is approximately 0.1 Lux, while that on a moonless night yet with faint starlight nears 0.01 Lux [1,2]. In low-light environments, conventional visible-light cameras struggle to capture the exceedingly feeble energy reflected by targets, whose critical details are overshadowed by the background, thus compromising the discernibility of targets and restricting the working distance of systems [3,4]. Low-light imaging, therefore, represents a technology dedicated to graphically visualizing targets or scenes in dimly lit areas.

As modern warfare continues to evolve, the timely and effective acquisition of battlefield information has become increasingly important. And all-time, high-resolution, and efficient battlefield situational awareness is crucial for the success of military operations. Compared to infrared imaging, low-light imaging technology has garnered greater attention in recent years due to its ability to achieve near-daylight imaging effects in nighttime environments while offering superior resolution and color rendition. Currently, countries such as the United States and Israel have equipped a large number of low-light imaging systems, especially in the aviation airborne field. However, in China, low-light night-vision systems are limited by the development of core imaging components and generally have a short operating range. They are mainly used for individual anti-counterfeiting and security monitoring and cannot achieve fast imaging capabilities over long distances and large widths, which limits their application in the field of aviation reconnaissance [5,6,7,8].

To enhance the low-light imaging system’s performance in dimly lit environments, a feasible way is to increase the relative aperture of the optical lens, thereby amplifying the amount of target energy captured by the system. For the system to possess long-range imaging capabilities under airborne conditions, an increase in the focal length and magnification of the lens is necessitated. A large relative aperture is, however, in contradiction to an extended focal length, as it engenders larger dimensions and greater weight for the lens. The fabrication of sizable optical lenses presents difficulties with diminished production yields, while the assembly and adjustment of such lenses become considerably more challenging.

At the same time, enhancing the information acquisition of the low-light imaging system and ensuring its high resolution and wide coverage for a single-frame image require detectors with an ultra-large pixel scale. The three most commonly used low-light detectors in China are the intensified charge-coupled device (ICCD), electron multiplying charge-coupled device (EMCCD), and high-sensitivity scientific complementary metal-oxide semiconductor (SCMOS). The ICCD couples with an image intensifier to enhance detection capabilities and exhibits a high incident photon response. Yet, it suffers from a limited pixel scale and poor image contrast. The EMCCD typically multiplies the photoelectron for a greater signal electron count. Its high levels of dark current noise result in a lower signal-to-noise ratio (SNR), thus necessitating cooling for proper operation. Built on the complementary metal-oxide semiconductor (CMOS), the SCMOS has come a long way in recent years for its elevated integration levels and diminished power consumption and readout noise. Beyond that, it can achieve high photon detection sensitivity and image fidelity with low-noise hardware circuits and image-processing software. As the pixel scale of China’s technically mature SCMOS fails to meet the requirements of high-resolution, wide-swath imaging systems, detector butting is, therefore, required to increase the field-of-view (FOV). The common methods of detector butting include mechanical butting and optical butting [9,10]. Mechanical butting involves arranging detectors across the entire focal plane of the optical system, which cannot be performed seamlessly as it needs high hardware precision. Optical butting, on the other hand, utilizes optical prisms or reflectors to divide the FOV into multiple image planes, and then multiple detector arrays are spatially installed for butting [11,12,13]. This approach demands a meticulous design of optical reflective components and the assembly and adjustment of detector arrays. As part of a special aviation project, this paper investigated the development of a long-range, high-resolution, and ultra-wide-swath earth imaging system suitable for low-light environments. After analyzing system specifications, we proposed a technical approach to optical butting and imaging based on eight detectors and devised an assembly of six reflective prisms for eight-channel beam splitting, followed by a detailed exposition of its working mechanism and mechanical configuration. Through rigorous validation, the low-light imaging system exhibited remarkable performance, meeting all specified parameters and delivering exceptional imaging outcomes under low-light conditions. These results underscore the system’s capability to fulfill the requirements of low-light imaging in airborne scenarios.

2. System Design Principles

2.1. Major Technical Specifications of the System

For practical application, the major technical specifications of the airborne low-light imaging system are as follows:

• Minimum illuminance: 0.01 Lux;
• Ground sampling distance: ≤0.5 m (at a flight height of 5 km);
• Area covered by pixels within an image: ≥4 km² (at a flight height of 5 km);
• Spectral range: visible light, RGB;
• Weight: ≤10 kg.

Beyond indicating the inclusion of an aircraft platform and the requirement for true-color image generation, the specifications describe the system’s capability to operate in daylight conditions. However, the use of high-speed aircraft platforms imposes constraints on the integration time of the imaging system, whereas the low-light performance of color CMOS detectors tends to be inferior to that of their monochrome equivalents [14,15]. That means the bar for the overall performance of the low-light imaging system will be raised if it is to boast both daylight and low-light operational capabilities and an extensive dynamic range.

2.2. Optical System

Based on the input of the system’s technical specifications, the specifications were calculated and decomposed. And taking into account the parameters of the selected high-sensitivity SCMOS detector, we determined the optical system’s design parameters, as shown in

Table 1. Parameters of the optical system.

Project	Index
Focus	131 mm
FOV angle	24.6° × 21.8°
F-number	1.6~20
Operating waveband	435 nm~656 nm
MTF@40 lp/mm	≥0.40
Distortion	≤2.2%
Back focal length	105 mm

.

With the abovementioned parameters, the design of the optical system is displayed in Figure 1.

For the low-light imaging system to capture scenes at a wide range of light levels, from 0.01 Lux to 10,000 Lux, while also having a wide-swath imaging capability, the optical system needs to feature such attributes as a large aperture, a wide dynamic range, a wide FOV, and low distortion. The optical system adopts an enhanced dual-Gaussian spherical surface design to meet the requirements of large aperture optical systems. In addition, the proposed optical system incorporates an iris diaphragm, enabling continuous adjustment of the aperture from F1.6 to F20 and resulting in a dynamic range of 43.8 dB. A focusing lens group is also set up to solve defocusing problems caused by different working distances and flight heights [16,17,18]. Furthermore, to enhance optical transmittance and minimize the impact of stray light, dielectric anti-reflective coatings with high transmittance are applied to the surface of all optical components, keeping the surface reflectivity under 0.5% within the spectral range from 435 nm to 656 nm. The FOV butting technique is adopted to achieve a wide swath and high resolution, and the optical system is designed with a sufficient back working distance to meet the splitting conditions of the reflective prism.

An analysis was performed on the imaging quality of the optical system, and the modulation transfer function (MTF) was calculated up to a spatial frequency of 40 lp/mm based on the CMOS pixel size. From the system’s MTF curves illustrated in Figure 2, it can be observed that, when the f-number in the optical system stands at 1.6 or expressed as F1.6, the MTF at the central FOV is greater than 0.68, and the MTF at 0.7 FOV exceeds 0.6. Likewise, when the aperture is set at F20, the MTF at the central FOV is above 0.39, and the MTF at the 0.7 FOV surpasses 0.37. These findings indicate that the optical system exhibits excellent imaging performance.

2.3. Imaging Performance Assessment

The proposed low-light imaging system is equipped with a high-sensitivity SCMOS sensor with a pixel size of 13 μm × 13 μm. Its major performance parameters are presented in

Table 2. Parameters of the SCMOS sensor.

Parameter	Value
Active array size	1920 × 1080
Pixel size	13 × 13 μm
Full well capacity	45 Ke
Peak QE	55%
Read noise	1.5e (RMS)
Dark current	5e/pixel/s@20 °C
Dynamic range	65 dB

.

2.3.1. Imaging Distance Estimation

In accordance with the principles of geometric optics, the imaging system’s ground sampling distance (GSD) is as follows:

$$GSD=\frac{a\cdot H}{f\cdot \cos \theta },$$

(1)

where a is pixel size; H represents flight height; θ denotes the declination angle of sight axis; and f indicates the optical system’s focal length. As illustrated in Figure 3, inspired by the Johnson Criteria chart, the resolution of a typical 4 m × 6 m target at different distances under vertical downward viewing was calculated and is shown in

Table 3. Statistics related to the working distance.

Flight Height	Imaging Pixel Number	Imaging Line Pair Number	Surveillance Level	Probability
5 km	8 × 12	4 × 6	Recognition	≥50%
10 km	4 × 6	2 × 3	Detection	≥90%
15 km	3 × 4	1 × 2	Detection	≥50%

. The results affirmed the system’s capacity for high-resolution imaging at extended distances.

2.3.2. Exposure Time Estimation

Given the low-light imaging system deployed on an aircraft platform, prolonged exposure time can result in forward image motion caused by the flight. As required by the imaging system, the forward image motion needs to be less than half the pixel size to ensure clear imaging of the ground. The forward image motion is primarily determined by factors such as imaging resolution, velocity–height ratio, and exposure time, and its analytical model is illustrated in Figure 4.

The exposure time can thus be expressed as follows:

$$t_{\max }\le \frac{a\cdot H}{2f\cdot \cos {\theta }^2\cdot v},$$

(2)

where v is the typical flight velocity. For safety considerations, the nighttime flight speed is set to 200 km/h. The maximum exposure time for the low-light imaging system at typical flying altitudes is shown in

Table 4. Statistics of maximum exposure time for the system.

Flight Height	Flight Velocity	Viewing Method	Maximum Exposure Time
5 km	200 km/h	Vertical downward view	4.5 ms
2 km	200 km/h	Oblique viewing of a target at a distance of 5 km	11.2 ms
1 km	200 km/h	Oblique viewing of a target at a distance of 5 km	22.5 ms

.

Therefore, at low illumination conditions during nighttime, the system achieved a maximum exposure time of 4.5 ms for vertical downward viewing. By adopting an oblique viewing approach, the exposure time could be further increased in a way that ensures superior image quality.

2.3.3. Imaging SNR Estimation

The imaging SNR effectively reflects the detection sensitivity of low-light imaging systems. It is influenced by the imaging signal strength of the target being measured and the inherent noise of the detector. The target’s imaging strength is primarily related to the ambient lighting, target reflectivity, and atmospheric transmittance [19,20,21,22].

The calculation model for the imaging system’s SNR can be expressed as follows:

$$SNR=20\lg \left(N_{sig}/N_{noi}\right),$$

(3)

where N_sig is the number of signal electrons received by the detector and N_noi represents the number of noise electrons generated by the detector.

The target’s luminance L can be calculated using the following:

$$L=E\cdot \rho \cdot {\tau }_a/\pi ,$$

(4)

where E is the ambient lighting of the target, set at 0.01 Lux; $\rho$ denotes the target’s reflectivity, set at 0.3 given the existence of dark-colored tanks; and ${\tau }_a$ is atmospheric transmittance, set at 0.7 in light of a mid-latitude region on Earth, at an altitude of 5 km, and from a vertical downward viewing perspective.

The detector’s illuminance of the image plane can be calculated using the following:

$$E^{\prime }=\frac{1}{4F^2}\pi \cdot L\cdot {\tau }_o$$

(5)

The illuminance unit Lux is a photometric unit that needs to be converted to a radiometry unit. The conversion relationship is the visual function V(λ) of wavelength. In the visual function V(λ), we can know that ${1\mathrm{W}/\mathrm{m}}^2=683\mathrm{Lux}(@\mathsf{\lambda }=555\mathrm{nm})$ and ${1\mathrm{W}/\mathrm{m}}^2=180\mathrm{Lux}(@\mathsf{\lambda }=360~680\mathrm{nm})$ within the response band of the detector. ${\tau }_o$ is the transmittance of the optical system, set at 0.75; and F represents the f-number of the optical system, set at 1.6, given that the aperture maximizes in low-light environments.

In accordance with the photoelectric conversion theory, the number of signal electrons received by the detector during the integration time is as follows:

$$N_{sig}=sum\frac{E^{\prime }\left(\lambda \right)\cdot t_m\cdot A_s\cdot \eta \left(\lambda \right)\cdot d\lambda }{h\cdot c}=\frac{E^{\prime }\cdot t_m\cdot A_s\cdot \overline{\eta }(\lambda )\cdot \overline{\lambda }}{h\cdot c},$$

(6)

where hc/λ is the energy carried by a single photon; $\eta (\lambda )$ represents the quantum efficiency of the detector, with its average $\overline{\eta }(\lambda )=0.55$ according to the relevant chip datasheet; the mean wavelength $\overline{\lambda }=550\mathrm{nm}$; t_m suggests the longest exposure time for the imaging system, set at 22.5 ms; and A_s is the area of a single pixel. Then, it is calculated that

$$N_{sig}=4.94e.$$

In low-light environments, we need to set a 4X maximum gain mode. Then, the equivalent number of electrons is ${N^{\prime }}_{sig}=19.8e$. The noise in the imaging system originates from various sources, and the impact of each noise component on the SNR is considered equivalent. Thus, the total noise can be represented as the square root of the sum of the squares of individual noise components. With readout noise, photon shot noise, dark current noise, and quantization noise taken into account, the number of noise electrons in the imaging system can be expressed as follows:

$$N_{noise}=\sqrt{{\sigma }_{read}^2+{\sigma }_{shot}^2+{\sigma }_{dark}^2+{\sigma }_{AD}^2}$$

(7)

Under the relevant chip datasheet, the SCMOS detector’s readout noise is ${\sigma }_{read}=1.5e$. Its dark current noise during an exposure time of 22.5 ms is 0.11e. The standard deviation of photon shot noise is approximately equal to the square root of the number of signal electrons, which is ${\sigma }_{shot}^2=N_{sig}$. The A/D quantization noise is related to the full well capacity of the detector, expressed as N_FW, and the number of quantization levels (12 bits), denoted as B_bit, and its standard deviation can be expressed as follows:

$${\sigma }_{AD}=\frac{N_{FW}}{2^{\left(1+B_{bit}\right)}\cdot \sqrt{3}}=3.17e$$

Then, $N_{noise}=4.15e$.

By Equation (3), the SNR of the imaging system is obtained, which is 13.6 dB. Under an illuminance of 0.01 Lux, the calculation results indicate that the system has good adaptability to low-light environments.

3. Multichannel Optical Butting System Configuration

3.1. Optical Butting Reflectors

Imaging sensors with a large pixel scale are required for the imaging system to have wide-swath, large-coverage, and high-resolution capabilities. Taking into account the low-light imaging system’s GSD, at least eight SCMOS detectors are needed to simultaneously form images to meet the system’s requirements for a single-frame image coverage [23,24,25]. In our research, each SCMOS detector had a pixel scale of 1920 × 1080. By arranging eight CMOS detectors in a 4 × 2 layout and with the FOV overlap ratio between adjacent detectors at 3%, the resulting butted image resolution reached 4222 × 3782. This can be visualized in the schematic diagram of the equivalent imaging detector array, as shown in Figure 5.

Seamlessly bonding individual detector chips is challenging, if not impossible. That means a more feasible approach is to arrange the eight detectors in three-dimensional space, with adjacent detectors in the vertical and horizontal directions located in separate regions. In doing so, consideration must also be given to the issue of self-occlusion caused by the reflectors [26,27,28]. Through multiple optimizations of the spatial layout of these detectors, an optical path-splitting strategy was developed, as presented in Figure 6.

The abovementioned optical path-splitting strategy was developed by utilizing six reflective prisms to divide into eight beams of light. Each prism had a reflective surface inclined at a 45° angle with respect to the optical axis. Through the prism assembly, the eight light beams were projected onto eight sets of highly sensitive detector components in five different directions in three-dimensional space. To avoid obstruction between the prisms, the reflective surfaces were positioned at different locations in front of the image plane. Given the challenges associated with prism fabrication and bonding, the projection surfaces of the prisms were designed as rectangular shapes. The reflective surfaces exhibited high reflectivity, whereas the non-reflective surfaces were coated with a light-absorbing material to minimize stray light. The operational mechanism of the prism assembly is depicted in Figure 7.

3.2. Calculation of Reflective Prism Parameters

The size and height of the reflective prisms vary depending on their positions in the optical path. As is known in geometrical optics, the VIII image plane is considered comparatively intricate. In this case, the relationship between reflector size and focal plane size is illustrated in Figure 8.

The computational model concerning the marginal ray from the FOV in the Y direction and the reflector size is depicted in Figure 9.

R is the exit pupil radius; D_y represents the dimensions of a single detector in the Y direction (with a height of 1920 pixels); L denotes the back focal length; and L₁ suggests the distance between the reflector’s location and the detector. Calculated using the geometrical relationship, the length of the reflector in the Y direction is as follows:

$$F_1{F}_2=\sqrt{2}{L}_y=\sqrt{2}\frac{L_1\cdot R+\left(L-L_1\right)\cdot D_y}{L}$$

(8)

The reflector dimensions in the Y direction determine the relative position of the reflector’s F₁F₄ and F₂F₃ edges in front of the detector. Different positions would lead to a variation in both edges’ widths in the X direction. The computational model concerning the marginal ray from the FOV in the X direction and the reflector dimensions in the X direction is illustrated in Figure 10.

D_x is the dimension of a single detector in the X direction (with a width of 1080 pixels). Calculated using the geometrical relationship, the length of the reflector in the X direction is as follows:

$$F_1{F}_4=\frac{L_1\cdot (R-2D_x)}{L}+D_x$$

(9)

$$F_2{F}_3=\frac{(L_1+L_y)\cdot (R-2D_x)}{L}+D_x$$

(10)

Following the preceding deductive process, the parameters for the six reflectors were individually computed. To ensure engineering feasibility, the projection surfaces of all reflective facets along the optical axis were configured as rectangular shapes, with their dimensions encompassing the maximum theoretical size calculated. Furthermore, a 3% FOV overlap ratio was taken into account in designing the reflectors.

3.3. Mechanical Configuration of the System

Upon an in-depth investigation into the working mechanism of the system, we configured the low-light imaging system. Mechanically, the system was comprised of optical lenses and an imaging assembly of multichannel butting detectors, as shown in Figure 11. The optical lens assembly encompassed various crucial elements, including the baffle; iris diaphragm; and front, rear, and focusing lens groups. Notably, the iris diaphragm component could alter the lens’s f-number within a continuous range from 1.6 to 20, with a maximum aperture diameter of 100 mm. To tackle potential defocusing issues arising from variations in working distances, the focusing lens group was ingeniously driven by a cam mechanism, allowing for a precise focusing range of ±2 mm. Furthermore, the optical reflecting prism assembly, adhered within the prism mount, was positioned at the rear end of the optical lens arrangement. Strategically arranged in five orientations in the frame, the eight detectors were individually positioned on the imaging circuit board in a biasing manner to circumvent any potential circuitry interference. The frame with light holes, though, was technically an enclosed structure designed to eliminate stray light. The fixed structure of detectors was fortified with adjustable elements and locking pins that facilitate assembly, adjustment, and post-installation positioning.

4. Test Results of the System

The final design of the actual optical reflector assembly is presented in Figure 12, and the complete system was processed, assembled, adjusted, and produced. The alignment and calibration of the FOV butting were accomplished using a collimator and a gradient target, as illustrated in Figure 13, which showcases the scene of the calibration site. The overlap ratio of each butted FOV was controlled to be within 3%, and image butting software was employed to guarantee a three-pixel butting accuracy for adjacent FOVs.

To verify the imaging capability of the proposed system in low-light environments and assess its dynamic range, rigorous tests were conducted under varying target illuminance conditions, as depicted in Figure 14. Within a controlled darkroom setting, a high-dynamic-range integration technique was employed to simulate target light levels, and the average gray values of the images captured by the imaging system were recorded.

Table 5. Results of the imaging dynamic range test.

Illuminance (Lux)	SCMOS No.	I	II	III	IV	V	VI	VII	VIII
0.01	Gray value	113.3	95.1	101.8	124.4	117.0	103.3	92.8	128.8
10,000	Gray value	786.2	520.5	613.3	811.1	659.6	580.8	533.2	850.5

presents the test data for target illuminance at 0.01 Lux and 10,000 Lux. For low-illumination conditions with an illuminance of 0.01 Lux, a longer exposure time and greater gain were adopted. For high-illumination environments, the exposure time and gain were reduced. It can be observed that, when the target illuminance was 0.01 Lux, the low-light imaging system gathered images with high gray values, indicating a high SNR in dimly lit environments. Furthermore, even at an illuminance of 10,000 Lux, the images did not exhibit saturation, highlighting the system’s wide dynamic range [29].

The imaging performance of the proposed low-light imaging system was assessed through a field experiment and an aerial dynamic test. The field experiment was conducted under overcast and moonless conditions, characterized by dim and distant lighting, to validate the imaging capability of the system in low-light environments. Having been measured by a high-precision illuminometer, the ground illuminance was determined to be 0.0099 Lux. A comparative analysis of the imaging outcomes between a conventional camera and the multichannel optical butting low-light imaging system was conducted, with the results presented in Figure 15. The aerial dynamic test took place at 5:30 a.m. on December 2021 at Neifu Airport in Shaanxi Province. The flight was performed at an altitude of 2000 m and a speed of 200 km/h. The ground illuminance during image capture ranged from 1 Lux to 10 Lux. Figure 16 showcases a single-frame image captured during the aerial test, while Figure 17 provides a zoom-in view of the aerial image. The experimental results demonstrated that the low-light imaging system is proficient in generating high-definition, RGB color images with remarkable contrast, even in an environment with a low illuminance of 0.01 Lux. Furthermore, the aerial images exhibit exceptional precision in stitching multiple FOVs, encompassing a broad coverage area and providing clear visibility of buildings, farmland, and trees with well-defined edges. Upon calculation, the GSD reached 0.2 m (at a flight height of 2 km), underscoring the system’s exceptional long-range and wide-swath imaging capabilities. And the stability and reliability of the airborne imaging system were also corroborated through the flight test.

5. Discussion

This study is dedicated to the research and development of an airborne wide-swath low-light imaging system capable of capturing high-resolution images from long distances. To this end, we proposed to employ an optical system with a large relative aperture and low distortion and generate images through FOV butting achieved by eight 1080P high-sensitivity SCMOS detectors. Built on the selected SCMOS parameters, a meticulous verification of the imaging system’s working distance, integration time, and low-light imaging capabilities was conducted. Moreover, we presented an optical path-splitting strategy using six reflective prisms to divide into eight beams of light and examined the mathematical expressions for the geometric parameters of the prism assembly. With these research results, an airborne low-light imaging system based on multichannel optical butting was developed. The system can produce high-contrast true-color images in an environment at a light level of 0.01 Lux while maintaining a GSD of 0.5 m (at a flight height of 5 km) and an imaging FOV of 24.6° × 21.8°. Through a sequence of field and flight tests, all performance indicators of this low-light imaging system meet the expected requirements. The findings and design methodology presented in this paper can provide a reference point for the development of airborne low-light imaging equipment.