Color Sensing and Image Reconstruction Using Intelligent Machine Learning Algorithm with PINIP Radial Junction Imager

Abstract

The development of a filterless imager has been eagerly awaited to overcome the diffraction limit when pixel sizes decrease to subwavelength scales. We propose an architecture for a filterless imager based on a symmetric inversely stacked radial junction (RJ) PINIP photodetector over silicon nanowires (SiNWs), whereby the diameter of which is less than 500 nm, which preliminarily displays the capability of bias-selected and tunable spectrum responses to the R, G, and B color bands. Assisted via suitably trained deep learning algorithms, the imager can provide more accurate color discrimination and imaging capabilities. Here, we used KNN (k-nearest neighbor) and convolution neural network (CNN) methods to retrieve the RGB ratios from the measured photocurrent value based on the pre-trained bias-tuned spectrum responses and reconstructed the images with high accuracy. Further, we demonstrated the capability of restoring sub-sampling pictures via CNN with a U-net architecture, and satisfactory reconstruction was obtained even with a sampling ratio as low as 20%. Our imaging scheme cannot only be used for high-resolution imaging but can also pave the way for application in single-pixel imaging and compressive sensing.

1. Introduction

Image sensing is of great importance in our modern information age. It is widely used in a variety of electronic devices for surveillance and monitoring, machine vision, and for medical and biological testing. An image sensor is one of the key components of the digital imaging system, usually consisting of an image sensor, optical system, and digital processors [1,2]. Nowadays, image sensors usually comprise an array of pixels on silicon-based charge-coupled devices (CCDs) or complementary metal-oxide semiconductors (CMOSs), which convert light to electrical signals. To obtain color images, an array of color filters are employed and deposited on top of the image sensor, typically a two dimensional symmetric mosaic Beyer filter array with the unit pattern of RGGB (two Gs for green, one R for red, and one B for blue) [3,4,5,6]. Dramatic progress has been achieved for this kind of pixelated image sensor. Conventional color image sensors already have record pixel densities above 100 megapixels per chip, where the color sensor pixel area features dimensions of about 2 × 2 μm² [7]. However, it is very difficult to increase the pixel density for conventional pixelated color image sensors in order to achieve finer resolution imaging due to the diffraction limit. Moreover, the traditional approach of one pixel with four filters largely reduces the incoming light flux, resulting in low sensitivity and limiting further miniaturization to shrink the unit size. To break this limit, novel imaging schemes or even filterless imaging architectures have been proposed and demonstrated over the past decade [8]. For example, in metal nano-disk imagers, metal nano-disks are used to confine light on the nanoscale through the plasmon effect and to distinguish color according to the dispersion of plasmon resonance peaks related to the size of the nano-disk [9,10]. Silicon nanowire PN junction imagers consist of vertical silicon nanowire PN junction arrays with different diameters, which cannot only discriminate color, but also convert light into electrical signals by themselves. The color is detected/selected through the size-dependent cavity mode resonance effect [11,12,13,14]. There are also semiconductor nanoparticle imagers [15], where colloidal quantum dots (CQDs) of different sizes act as multiband filters because of the quantum confinement effect.

Besides the conventional imaging scheme, computational imaging (CI) using algorithms empowers the versatilities of imaging, which began with the restoration of imperfect images in medical diagnosis. With the aid of the machine deep learning (DL) approach, high-quality image restoration has been achieved in terms of accuracy, high resolution, and fast restoration speed [16]. DL is currently widely used in many fields and research areas, especially for use in image or spectral recognition tools [17]. First, the DL approach has the ability to exploit information from data that may be indiscernible using traditional methods. Second, its flexibility in design makes it compatible with nanophotonic platforms. Third, DL algorithms can be applicable to various functions, such as spectral reconstruction, high-resolution imaging, classification, and noise suppression. DL also boosts single-pixel imaging (SPI) where signals are collected by a single detector in a raster scanning way or by interrogating the scene using a series of patterns (the Hadamard pattern, for example); based on these data/patterns, the original image is reconstructed via the machine DL algorithm [18].

In recent work, we demonstrated a filterless imager based on a symmetric inversely stacked radial junction (RJ) PINIP (here, P, I, and N stand for p-type, intrinsic, and n-type amorphous Si film) photodetector fabricated around silicon nanowires (SiNWs) [19], where different incident wavelengths are absorbed with different penetration depths in stacked PIN junctions (as shown in the inset of Figure 1a). Such a 3D structure can confine light within the nanowire and redistribute the light field into different absorber layers [20], where the inner layer can absorb long wavelengths while the outer layer can absorb short wavelengths. Thus, this RJ PINIP sensor has a preliminary capability of bias-selected and tunable spectrum responses to the R, G, and B color bands, and it may be the smallest multiplex color sensor on the nanoscale [21,22]. By distinguishing the dispersion of the photocurrent under different bias voltages, this imager can deduce the RGB ratio from the measured photocurrent values. However, different RGB combinations or light with different power can give the same photocurrent value; thus, how to precisely discriminate the RGB color based on photocurrent values under different external bias voltages is the key problem. To address these issues, we employ the DL approach, which is widely used in image restoration and in the miniaturization of modern technologies [23,24,25,26].

Therefore, in this work, we use DL information process models such as k-nearest neighbors (KNNs) and convolutional neural networks (CNNs) to determine the RGB ratios from the measured photocurrent value based on the pre-trained bias-tuned spectrum responses, as well as conduct image restoration with higher accuracy. Further, using CNN to restore under-sampled pictures, a satisfactory image is reconstructed with only 20% sampling. These results highlight that our imaging paradigm combining unique RJ PINIP architecture and the DL approach can efficiently discriminate color and construct color images with high accuracy, laying the building blocks using an RJ PINIP unit with a diameter of around 500 nm for highly dense pixel color imaging as well as paving the way for applications in single-pixel imaging and compressive sensing.

The remainder of this paper is organized as follows. Section 2 contains the main body of our work, first introducing the architecture and bias-tuning color discrimination mechanism of our imager; then, using our imager aided by KNN and CNN methods, we demonstrate how to carry out the color recognition and image restoration, as well as make a comparison between the two methods. Finally, we restore the under-sampled image using the CNN. Section 3 gives a brief discussion and summary of the paper.

2. Experiments and Results

Figure 1a shows the scanning electron microscopy (SEM) image of the final fabricated RJ PINIP imagers, measuring ~500 nm in diameter and 1 $\mathsf{\mu }\mathrm{m}$ in length. The inset schematically draws the radially stacked PINIP multilayer structure. The simplified fabrication process is briefly described as follows. First, we fabricated the center boron (B)-doped p-type SiNWs upon (aluminum-doped ZnO) AZO glass substrates via a vapor–liquid–solid (VLS) method using the low-melting-point metal of tin (Sn) as a catalyst in a plasma-enhanced chemical vapor deposition (PECVD) system. As AZO is conductive, it acted as the bottom electrode. Next, the other layers of I/N/I/P were used for the a-Si:H layers with corresponding doping types; the inner and the outer intrinsic layers were both 50 nm and the n-type layer was 5 nm. Lastly, a 50 nm transparent and conductive ITO layer was coated by shadow masks in a magnetron sputtering system to define the top electrodes. Fabrication details can be found in Reference [13]. Figure 1b presents the external quantum efficiency (EQE) responses under different bias voltages [27]. It was found that the photocurrent response spectra could be continuously tuned by the bias voltage, indicating that the absorption center peak of the wavelength could be continuously modulated by the external bias (note that positive and negative photocurrents just mean opposite current directions). The photocurrent has a dispersion with the external bias voltage mainly due to the bias-selected absorption layer, with the inner or outer layer corresponding to the long or short wavelength, whereby each of which has unique responsivity with rich color information. This is the key idea for the RJ PINIP color discrimination imager.

2.1. Bias-Tunable Color Response of the Imager

We pre-trained our imager to determine the relationship between the photocurrent value under different biases and the known RGB ratios. Figure 2a shows the pre-trained setup. We used a computer-controlled projector to supply light with different RGB ratios. The RJ imager is placed in a fixed position in front of the projector, the bias is supplied by a source meter, and the data are collected automatically by the computer. Figure 2b shows the imager’s photocurrent response curves, which can be interpreted as the photocurrent value vs. bias voltage for a given color at specific RGB ratios. There is a slight distortion of the curves due to unknown vibrations in the environment. For each color from 0 to 255, increasing by 51 steps, we used 125 combinations of the RGB color. For each combination, we chose a bias voltage from 0 to 0.5 V in steps of 0.1 V and recorded the photocurrent value accordingly. We examined the photocurrent distribution and correlation, as shown in Figure 3. The diagonal spike pictures show the distribution of the current data at several voltages. The abscissa is the current. The ordinate is the number. The higher the number is, the more times this current appears. It can be seen that the current distribution pattern is different under different voltages. The other graphs of the scatter plot in Figure 3 outside the diagonal line are the photocurrent data under different voltages (the abscissa voltage A and the ordinate voltage B indicate that the scatter points on this graph are based on the photocurrent values under voltage A, as well as the abscissa and voltage B). The color of each point is the real color of the light illuminating the sample. It can be seen that these scattered points are relatively dispersive and not all concentrated on the diagonal. This means that these currents are not linearly related and can be used as independent, characteristic data for training.

2.2. Color Recognition and Image Restoration via KNN Method

The k-nearest neighbor (KNN) algorithm is an important tool in the field of machine learning [28,29]. The principle of KNN is to extract each specific parameter from an n-dimensional dataset and map it to an n-dimensional dataset, thereby achieving effective analysis of the data. When predicting a new value x, it determines which category x belongs to based on the categories of the k-nearest points, as illustrated in Figure 4. In addition, KNN can also be used for model optimization and data reconstruction to improve the accuracy and reliability of the model.

A photo of ‘Lena’ was pixelated and divided into 128 × 128 pixels for simplification (see Figure 5a). The pixel RGB color information was automatically scanned and projected onto the RJ-PINIP imager, as shown in Figure 2a, where the photocurrent signals I_bias were collected under different bias voltages. Then, these photocurrent signals were put together as a vector of I_vec, = (I₀, I_0.1, I_0.2, I_0.3, ……), where the subscript is the bias voltage and the number of I_vec depends on the number of biases used. First, we adopted the KNN classification method to guess the color (the R, G, and B values) from each value of vector I_vec with K = 2, based on the pre-trained knowledge shown in Figure 3 where photocurrents of each known RGB combination under different biased voltages are given. The RGB value of every pixel was obtained and, thus, the image was reconstructed, as shown in Figure 5b–d, with the photocurrent under 0–0.1 V, 0–0.2 V, and 0–0.3 V, respectively, and the step = 0.1 V. We can see that the image can be reconstructed with only two sets of photocurrent values between 0 and 0.1 V, but obviously, there is some false color which comes from sub-sampling, resulting in a larger L₂ distance [30]; as a matter of fact, the more sets of photocurrent values, the better the restoration, but if the bias is above 0.4 V, there will be more false color due to the fact that there will be too many of the same photocurrent values for one RGB combination under different types of bias. The K value also has a great influence on the restoration effect [31]. We used the loss function to evaluate the recovery accuracy as a percentage, and as shown in Figure 5e, four sets of photocurrents and K = 2 form the optimal solution. In fact, as shown in Figure 3, most of the photocurrents under 0.4 V and 0.5 V are linear with the other photocurrents under different biases, indicating that more similar photocurrents occur, which makes it difficult for the KNN methods to obtain an accurate RGB value. A proper photocurrent number and proper K value can result in better image restoration. It is worth noting that the restored image is purplish, mainly due to the color tablet of the projector being rich in purple.

2.3. Color Recognition and Image Restoration Using CNN Method

We used the KNN algorithm to carry out the color recognition and restore the image, and the restored image is closer to the original. However, there are still many false colors and the restored image is not as smooth as it could be. In addition, this method is highly dependent on the training dataset, as the categories distinguished by the KNN algorithm are limited to existing pixel values in the training set [32]. In the field of image recognition and target detection, the CNN algorithm performs well [24]. It can learn hierarchical features, including high-level features, and is a key mechanism in feature extraction. For the CNN model, taking the current value corresponding to the input voltage of 0 to 0.3 V as an example, as sketched in Figure 6a, the corresponding input features are 0, I₁, 0.1, I₂, 0.2, I₃, 0.3, and I₄, and the number of channels of the input data is eight. The output features are R, G, and B, and the number of output channels is three. Since the input feature and output feature structures of the training data are relatively simple, in order to reduce the computational cost, we adopted a relatively simple CNN model structure of two convolutional layers and two fully connected layers. First, the first convolutional layer uses eight input feature channels and 12 filters, and performs convolution operations with three continuous input features (convolution kernel size). The stride is one, the padding is one, and the output size is 12 feature channels. Then, the output is processed nonlinearly through the ReLU (Rectified Linear Unit) activation function layer, which can set negative values to zero. Next, the second convolutional layer receives the output of the previous layer, with twelve input feature channels and six filters, using three consecutive input features for convolution operations. The stride is one, the padding is one, and the output size is six feature channels. Subsequently, the first fully connected layer maps these six features to a 256-dimensional feature space, performs linear transformation, and processes the output through the ReLU activation function. Finally, the second fully connected layer maps the 256-dimensional features to a 3-dimensional (corresponding to three RGB eigenvalues) feature space, performs linear transformation, and processes the output through the ReLU activation function. A simplified module is illustrated in Figure 6b.

The reconstructed images are given in Figure 7a–e using two to six sets of photocurrent data. The loss value of these two methods is given in

Table 1. Comparisons of loss values between KNN and CNN algorithms regarding the restoration of the image with data obtained under different biases.

Loss Value	0~0.1 V	0~0.2 V	0~0.3 V	0~0.4 V	0~0.5 V
KNN	30.17	22.21	24.58	22.32	22.67
CNN	15.58	13.40	12.13	12.74	13.10

according to Formula (1).

$$\mathrm{Loss}=\frac{{\sum }_{\mathrm{i}=1}^{\mathrm{n}}{\left|\mathrm{expected} \mathrm{RGB} \mathrm{Value}-\mathrm{real} \mathrm{RGB} \mathrm{value}\right|}^2}{\mathrm{n}}$$

(1)

These images restored using the CNN are obviously better than those restored using the KNN method, due to the algorithm’s success in feature extraction and the activation function layer for rectification [33,34].

2.4. Sub-Sampling Image Restoration Using CNN Method

Our RJ-PINIP unit could be used as a single-pixel imager [35] and applied in compressive sensing [36]. In these scenarios, the raster scan strategy of sub-sampling is an efficient means of imaging, supported by a deep machine learning image reconstruction algorithm, resulting in a fast and super-resolution image. Using the CNN algorithm, we attempted to reconstruct Lena’s visage using 70% to 10% pixels. We built our CNN network in a conventional U-net structure based on the Keras deep learning model [37]. The U-net structure is a classic convolutional neural network structure [38] consisting of a set of symmetrical encoders (down-sampling paths) and decoders (up-sampling paths), connected through skip connections. The main advantage of this structure is that it can handle local and global features of images, and the introduction of skip connections helps to better recover details and mitigate information loss. This structure has been widely used and achieved good results in the fields of medical image segmentation [39], semantic segmentation [40,41], and image reconstruction [42]. The encoder part consists of multiple convolutional layers and pooling layers, which gradually extract the features of the image and reduce the size of the feature map. In the decoder part, the size of the feature map is restored to the size of the input image through an up-sampling operation, and skip connections are used to connect the output of the encoder with the input of the decoder to fuse feature information of different scales. The U-net structure designed here contains five down-sampling stages and four up-sampling stages. Each stage contains two convolutional layers, and the number of output channels of the last up-sampling stage is three, that is, the image is divided into RGB channels. Finally, the Adam optimizer is used to optimize the model, and the mean squared error (MSE) function is used as the objective function of training. We can observe the loss function to obtain the optimized image. First, we pre-trained this model with hundreds of known perfect white female face images and their corresponding damaged images (with 30–90% pixel loss, respectively). All of the original images had the same size of 128 × 128 pixels. Then, we obtained Lena’s distorted images by setting 40–90% of the pixel values to zero. To optimize the accuracy of the restoration, we first carried out the iteration test on an image with 30% pixel loss, as given in Figure 8; when the epoch = 100, the image is almost the same as its original with a small-enough MSE value.

Following this, we restored the damaged images to different degrees (40–90% pixel loss) with epoch = 100, and the results are shown in Figure 9. The images can be restored even with 80% pixel loss, and when the pixel loss is 90%, there are distorted features in the restored image, as well as false color. These results indicate that the PINIP radial junction imager only needs to randomly collect 20% of the pixel information to restore the overall facial structure. Our tiny RJ PINIP imager is expected to overcome the diffraction limit to realize super-fine imaging, and can also be used as a single-pixel imager application for holography [1], phase imaging [42], ophthalmic imaging [43], and ultrafast imaging [18]. This algorithm could greatly improve the single-pixel imaging efficiency of the PINIP radial junction imager.

3. Conclusions

Beyer-filter-based color imaging pixels reach a critical size in terms of subwavelength. In order to break the diffraction limit, we proposed a novel imaging scheme as well as the smallest color imaging unit in the world using an inversely stacked PINIP radial junction, whereby the spectrum response of which can be continuously tuned via the bias voltage. The responsivity under each bias is unique and rich in features, laying the basis for image reconstruction using the proper algorithm. Supported by deep machine learning algorithms, this imager cannot only distinguish color but can also sense and reconstruct the color image. In the KNN color classification method, the proper bias (0–0.3 V) and K value (K = 2) help to achieve higher color retrieval accuracy, while the CNN algorithm achieves better color recognition and image restoration. A satisfactory color image was reconstructed using only 20% of the pixel information, laying the foundation for single-pixel imaging and compressive sensing.