1. Introduction
Performance of atmospheric electro-optical (EO) systems, such as free-space laser communication, remote sensing, active imaging, directed energy, and optical surveillance can be significantly degraded by atmospheric effects (e.g., optical turbulence, refractivity and absorption) [
1,
2,
3,
4,
5]. Atmospheric turbulence causes the most detrimental impact on laser-beam and image characteristics, especially in the deep turbulence conditions typical for slant and/or extended-range propagation scenarios [
6]. In contrast with refractivity and absorption, atmospheric turbulence strength, as characterized by the refractive index structure parameter
, can strongly fluctuate during only a few seconds for a stationary target [
7,
8] and by an order of magnitude for high-velocity targets when the line-of-site rapidly sweeps across a large volume of turbulence.
To evaluate and mitigate the negative impact of atmospheric effects on the performance of EO systems, it is necessary for these effects to be accurately characterized and potentially forecast along the line-of-site to the target (including moving targets) at a temporal resolution that is significantly higher (in situ) than in today’s available atmospheric turbulence characterization EO sensors. In situ turbulence strength characterization can be applied for real-time parameter adjustment in wavefront sensing, beam control and adaptive optics systems [
5,
9,
10], for turbulence effects mitigation in atmospheric imaging [
11], and to reduce the bit error rate in laser communication systems [
12,
13,
14]. Conventional electro-optics (EO) sensors currently used for
measurements are based on estimation of the statistical characteristics of the received optical field, e.g., the laser beam scintillation index, focal spot wander and/or widening, image sharpness, etc. [
6,
15,
16]. Time-averaging is used to calculate these statistical characteristics based on sensing data collected over a relatively long time (typically a few minutes), resulting in low temporal resolution for
evaluation.
A novel approach for path-integrated
parameter estimation at high temporal resolution was introduced in [
8], where deep neural network (DNN)-based signal processing models (Cn^2Net DNN models) were developed for
value prediction based on the processing of short-exposure laser beam intensity scintillation patterns. Preliminary results that describe the hardware implementation of this approach using EO sensor configurations (TurbNet sensors) with coherent and incoherent laser beacons and deep-machine-learning-based signal processing have been reported in conference presentations [
17,
18]. In this study we further extend these early results.
The application of deep-machine-learning approaches to turbulence strength characterization is a rapidly growing research area. The major emphasis in recent publications in this field has been on the development of DNN architectures for
prediction by utilizing pre-recorded or simulated input data obtained using meteorological sensors [
19], numerical weather prediction simulations [
12], or wave-optics numerical simulations of turbulence-degraded imagery [
11,
20]. The goal of the research presented here was to experimentally demonstrate and compare the performance of DNN-based TurbNet sensors over a 7 km distance for path-integrated
parameter evaluation at a high temporal rate (approximately 1.5 s. per
measurement) under a wide range of turbulence, environmental and meteorological conditions.
This paper provides an in-depth description of TurbNet sensors, including the EO hardware (
Section 2), the pre-processing of scintillation images (
Section 3), the generation of datasets during the set of atmospheric measurement trials (
Section 4), and DNN model training, validation, and testing (
Section 5). The performance evaluation of TurbNet sensors in atmospheric inference experiments for
value prediction using pre-trained DNN models under diverse atmospheric turbulence and weather conditions is described in
Section 6. The results of TurbNet sensor development and evaluation are summarized in
Section 7.
2. TurbNet Sensor: Hardware Implementation and Experimental Setting
A TurbNet sensor is comprised of optical receiver modules, a laser beacon located at opposite ends of an atmospheric propagation path, and DNN signal-processing hardware (
Figure 1). In the experiments described here, we used two laser-beacon types: coherent, based on a single-mode laser source, and incoherent, utilizing a laser-emitting diode (LED). The corresponding TurbNet sensor configurations are referred to here as TurbNet-LB and TurbNet-LED for convenience.
The laser-beacon module of the TurbNet-LB sensor in
Figure 2 (top right) is based on a single-mode fiber-collimator with an aperture diameter of 50 mm emitting a collimated Gaussian beam of 30 mm width at 1064 nm wavelength (about 5 mW power). The fiber collimator was mounted on a gimbal platform used for angular alignment of the beacon beam, directing it towards the optical receiver module located at the opposite side of the 7 km propagation path at the UD atmospheric test range, as illustrated in
Figure 1. The characteristic laser-beam footprint at the receiver plane was of the order of 60 cm.
Due to the impact of environmental factors (e.g., weather conditions, position of the sun and clouds) and atmospheric refractivity, the laser-beacon footprint slowly drifted with respect to the optical receiver during the experimental measurement trials. The characteristic range of these drifts was of the order of the beam footprint size over a 60 to 90 min timescale. To exclude the influence of laser-beam footprint centroid drift on measurements, the laser beacon was frequently (approximately every 10–15 min) re-aligned. This circumstance precluded uninterrupted multi-hour data collection, which is desirable for the generation of large datasets of data instances (scintillation images and
values) that are representative of a wide range of turbulence and weather conditions. These datasets were used for DNN model training, validation, and testing, as described below (see
Section 5 and
Section 6).
Note that laser-beam divergence of the TurbNet-LB beacon can, in principle, be intentionally increased to enlarge the laser-beam footprint at the optical receiver plane, thus mitigating the impact of laser-beacon misalignment on measurements. Nevertheless, to maintain high contrast in the scintillation images captured by the CCD camera, increasing the divergence would require a corresponding increase in laser power, which was undesirable for eye-safety reasons.
The laser-beam footprint drift issue was addressed in the TurbNet-LED sensor by utilizing an LED (940 nm center wavelength, 1.0 W output power) as a light source. To reduce the emitted light divergence, the LED was placed in the focal plane of a collimating Fresnel lens with a 30 cm aperture and 30 cm focal length, as shown in
Figure 2 (bottom right). The laser-beam footprint diameter at the optical receiver plane for the LED-based beacon was about 20 m. Significantly enlarging the beam footprint dramatically reduced the impact of environmental factors, thus allowing continuous measurements over 24 h without the need for laser-beacon re-alignment.
The optical receivers of both TurbNet sensors were comprised of a telescope lens (
L1) and imaging lens (
L2) that were utilized for re-imaging the telescope pupil into a CCD camera, as illustrated in
Figure 1. A diaphragm (
D1) in the focal plane of the telescope was used to reduce the impact of ambient light.
The TurbNet-LB receiver module in
Figure 2 (top left) was implemented based on a refractive telescope of aperture diameter
D = 11 cm and focal distance
F = 77 cm. The light-gathering power of the telescope was not sufficient to obtain high-contrast scintillation images with the LED beacon. For this reason, in the TurbNet-LED sensor receiver module in
Figure 2 (bottom left), the refractive telescope was replaced by a reflective Schmidt–Cassegrain type telescope having an approximately 2.8-times larger aperture diameter (
D = 30.48 cm,
F = 3.048 m). Note that the LED-based sensor necessitated a more expensive and bulkier receiver telescope in comparison with the receiver telescope of the TurbNet-LB sensor.
Both TurbNet sensors utilized an identical CCD camera (12-bit Allied Vision sensor with 808 × 608 pixel resolution) to capture short-exposure laser-light-intensity distributions (scintillation frames). The camera integration time (exposure time) τCCD was adjusted based on the received laser-light-intensity level: shorter (τCCD = 0.1 ms) for the TurbNet-LB sensor with the coherent laser beacon, and significantly longer (τCCD = 3 ms) for the TurbNet-LED sensor due to the relatively low received laser-light intensity.
A characteristic example of scintillation frames captured by the TurbNet sensors under different atmospheric turbulence conditions is illustrated in
Figure 3. From visual assessment, the spatial structures of the scintillation images are noticeably different for various turbulence strengths.
3. Pre-Processing of Scintillation Images and Dataset Generation
Pre-processing of the input scintillation images for assestment by the TurbNet sensors was performed using an embedded-edge AI computer (Jetson Xavier Nx) that was synchronized with the scintillometer. The values sequentially measured by the scintillometer were considered as “ground truth” for DNN model optimization, training, testing, and performance evaluation in the inference experiments. The scintillometer was pre-set to provide a single measurement per minute—the shortest measurement rate available with the BLS 2000 instrument. During each = 60 s time interval between sequential measurements, the CCD camera captured 300 frames, corresponding to a frame rate of 5 frames/s.
The GStreamer multimedia framework was used to create a pipeline from the CCD camera to the OpenCV application of the AI computer, as illustrated in
Figure 4. The pipeline was used to pre-set the camera frame rate and exposure time, convert the camera raw video stream into AVI format, and downscale from a 10-bit to 8-bit grayscale video stream. Image pre-processing steps were carried out for each frame before saving it to the dataset or inputting to the DNN models and included resizing frames to 128 × 128 pixel resolution, masking the region of interest (ROI), excluding low-contrast frames, and performing intensity normalization on the maximum pixel value within each frame. Corresponding to the receiver telescope aperture, the ROI was defined as an inscribed circle for the TurbNet-LB sensor and as two inscribed concentric circles for the TurbNet-LED (see
Figure 3). Pre-processed frames were tagged with the corresponding “true”
values to create a dataset composed of instances (normalized
values and the corresponding 300 scintillation frames) obtained during the data-collection trials. The same media pipeline and preprocessing steps were applied to real-time inference experiments. A description of software and driver versions used for implementation of the GStreamer framework and AI processor can be found in [
18].
4. Data Collection Experimental Trials
The experimental setup described in
Section 2 (see
Figure 1) was utilized for synchronous recording of short-exposure scintillation images and corresponding “true”
values under diverse turbulence and environmental conditions using both the TurbNet-LB and TurbNet-LED sensors. Summaries of the data collection atmospheric trials are presented in
Table 1 and
Table 2, which include information about date and local time (EDT) of the measurement trials, the number of recorded scintillation frames, and weather conditions. During these trials a total of approximately 0.25·10
6 (TurbNet-LB dataset) and 1.3·10
6 (TurbNet-LED dataset) data instances were acquired with the TurbNet-LB and TurbNet-LED sensors, respectively. Note that the 300 scintillation frames captured between sequential scintillometer measurements were associated with a single
value (see
Section 3).
Data collection with the TurbNet-LB sensor (
Table 1) was performed during 12 relatively short (lasting less than six hours) measurement trials conducted between 15 July 2021 and 2 September 2021. During each trial the laser beacon required frequent re-alignment due to laser-beacon footprint drift (see
Section 2), which made it difficult to pursue data collection representing day and night turbulence variability. Enduring data-collection trials were conducted using the TurbNet-LED sensor, which did not require LED beacon realignment and, thus, allowed continuous measurements over durations exceeding 24 h. Correspondingly, with the TurbNet-LED sensor, a large dataset (5.2-times larger than with the TurbNet-LB sensor) was obtained from only five measurement trials (lasting many hours), as summarized in
Table 2.
The distributions of scintillation frames obtained under different turbulence conditions in the TurbNet datasets are illustrated in
Figure 5 by histograms showing the number of scintillation frames within the interval
(histogram bin width). Note that the first bin acounts for all scintillation frames aquired under weak turbulence conditions corresponding to
.
It is apparent that the distribution of scintillation frames in the TurbNet-LB dataset in
Figure 5 (left) is more balanced (more uniform across
values) than the corresponding TurbNet-LED dataset (
Figure 5 (right)), which shows that a predominantly large number of frames were recorded under medium-strength turbulence
. This was not a surprise, as data collection with the TurbNet-LED sensor was performed continuously for multiple days under wide-ranging turbulence conditions. This non-uniformity in the distribution of instances in the TurbNet-LED dataset is not desirable for DNN model training, as it leads to higher error in
prediction for both low- and strong-turbulence conditions due to the obvious bias in training data volume towards medium-strength turbulence. In the case of the TurbNet-LED dataset containing a large number of instances, the desired uniformity in data distribution (dataset balancing) was achieved by selective partial removal of data instances belonging to the medium-strength turbulence range. The modified (balanced) TurbNet-LED dataset, comprised of approximately an equal number of instances (0.25·10
6) as in the TurbNet-LB dataset, was used for DNN model training as described in
Section 5. In general, removal of instances from a dataset is not the optimal method to balance it. Nevertheless, from a practical viewpoint, it was convenient to conduct a single “unsupervised” atmospheric data collection trial for 24 h and to further “balance” this dataset by selective and random removal of data instances recorded under medium-strength turbulence. An alternative was to conduct many additional “supervised” atmospheric data-collection trials by selecting days and times when
was either low or high.
For data collection using the TurbNet-LB sensor, uniformity of data distribution across the span was achieved without data removal by intentionally increasing the number of “supervised” measurement trials performed under both low- and high-turbulence conditions.
5. DNN Model Training, Validation, and Testing
Processing of scintillation images in the TurbNet sensors was performed using the DNN (Cn^2Net) model architecture illustrated in
Figure 6 [
8]. A set of 16 pre-processed consecutive scintillation frames from the CCD camera were used as the input for
MFEB = 16 feature-extraction blocks (FEBs) with identical topology and trainable weights. The input frames entered the DNN models with a sliding window shift of
mshift = 8 frames. The parameter
mshift (1≤
mshift ≤16) controls the frequency of each frame reprocessing within the 16-input frame sequence. This parameter (DNN hyper-parameter) was used for
prediction accuracy optimization and stabilization of the DNN training process. Input frames were simultaneously processed by the FEBs using three convolutional and max-pooling layers, followed by perceptron and fully connected layers. Additional details of the Cn^2Net DNN model can be found in [
8]. The Cn^2Net software code was used to generate several (about 5–15) pretrained DNN models obtained using different realizations of the initial random weights.
Both the TurbNet-LP and modified (balanced) TurbNet-LED datasets were subdivided into three subsets: training (70%), validation (20%), and testing (10%). The DNN models were trained with identical training and validation subsets at a learning rate of 0.0001 using 100 epochs and applying an early stop to the validation phase using 10% of the number of epochs as the validation patience (number of epochs allowed to improve performance without error decrease).
The mean squared error (MSE) in
prediction was selected as the cost function for performance evaluation of the DNN model-training process. The DNN models that predicted
with less than 0.1 MSE in the training process were selected to evaluate the “never seen” data from the corresponding validation subsets. The three DNN models that demonstrated the best performance for each TurbNet-LB and TurbNet-LED data subsets were saved and loaded to the AI processor for real-time
prediction in the inference experiments. The outputs of these models were averaged to increase prediction accuracy. A characteristic example of Cn^2Net model optimization is presented in
Figure 7 by scatter plots that compare
values predicted by a DNN model to the “ground truth” measurements obtained for the training, validation, and testing subsets of the TurbNet-LED dataset. As shown in
Figure 7, the highest prediction errors were observed for data instances corresponding to strong-turbulence conditions. Note that most of the prediction errors were in the
20% range for both TurbNet sensors.
6. Performance Evaluation of TurbNet Sensors in Real-time Inference Experiments
Performance evaluation (inference) trials of the TurbNet sensors were conducted using the experimental setup shown in
Figure 1. The sensors’ CCD cameras and AI computers were synchronized with the scintillometer that provided “ground-truth”
values at a
second time rate. The AI computer received scintillation frames at a time interval of
. Each scintillation frame was pre-processed and added to the image buffer to create an input sequence containing 16 sequentially captured and pre-processed scintillation frames. Each of the
Nmodel pre-trained DNN models (
Nmodel varied from one to five) loaded into the AI computers received identical input sequences of 16 pre-processed scintillation frames in sequential order. For a given scintillation frame input sequence, prediction of a
value by a single DNN model required about
= 100 ms. The overall processing time
depended on the number of DNN models
used—a parameter that is contigent on such factors as the characteristics of the training dataset (e.g., volume, distribution of data instances, etc.), the DNN training approach, the selection of DNN models for inference experiments, and a compromise between the processing time
and
prediction accuracy. In the atmospheric inference trials conducted, the
prediction rate was varied from approximately
= 1.4 s (
Nmodel = 1) to
1.8 s (
Nmodel = 5), which was from 33 to 43 times faster than the corresponding
sensing rate of the commercial scintillometer used to obtain “ground-truth”
values.
The output data provided by the AI computer consisted of (j = 1,…, Nmodel) predictions independently computed by the DNN models, their average (model-average) value , and the moving-average (also referred to as the rolling-average) prediction , which was obtained based on ten sequential model average outputs. Here is the timestamp with time interval .
TurbNet sensor performance was evaluated in a set of atmospheric interference trials repeated several weeks (from one to 24) after the last day of the training dataset collection trials to ensure that performance was not affected by environmental factors, including diurnal and seasonal changes in the weather. Exemplary results of the atmospheric inference trials are presented in
Figure 8 and
Figure 9, where the “true”
values measured by the scintillometer are compared to both model-average
and moving-average
TurbNet sensor outputs for the entire range of turbulence conditions observed during the inference trials, which lasted approximately two hours (
Figure 8 top and
Figure 9) and 24 h (
Figure 8 bottom). Note that the last inference trial (see
Figure 9) was performed in the middle of a sunny, warm day in early May 2022, six months after recording the corresponding DNN training dataset during several days of relatively cold weather in November 2021 (see
Table 2).
The plots in
Figure 8 and
Figure 9 show a close match between the sequence of datapoints [
and
] obtained with the TurbNet sensors and the corresponding measurements [
] obtained with the scintillometer. The corresponding values of the correlation coefficient
γ computed for the moving-average DNN output and scintillometer datapoints were
γ = 0.7 for the inference trial in
Figure 8 (top),
γ = 0.77 for the trials in
Figure 8 (bottom), and
γ = 0.9 for the inference trial in
Figure 9. Note that a similar span of the correlation coefficient
γ was observed in several more recent field experiments.
Because of the high
sensing rate, the TurbNet sensors can be used to monitor rapid changes in atmospheric turbulence dynamics. This unique capability is illustrated in
Figure 10. The measured vs. DNN-predicted
time evolution plots in this figure represent exemplary segments of the corresponding dependences in
Figure 8. The span of model-average
data provided by the TurbNet-LB and TurbNet-LED sensors clearly indicates the significant changes that occurred in turbulence strength during the 60 s time interval between sequential
measurements by the scintillometer. The DNN-predicted
values show about a two-fold change in
Figure 10 (left) and nearly a five-fold change in
Figure 10 (right).
The selected scintillation images shown in
Figure 10 (bottom) correspond to the highest and lowest
values (points A through D in
Figure 10 (top)). The scintillation pattern spatial features in these images corroborate the TurbNet sensor measurements in
Figure 10 (top).
These results show that the TurbNet sensors enabled detection of changes in turbulence strength at high temporal resolution (in real-time), which cannot be achieved with conventional EO instruments based on collection and time-averaging of sensing data.
7. Summary and Concluding Remarks
In this paper, we discuss the implementation and field evaluation of novel EO sensors (TurbNet sensors) that enable atmospheric turbulence strength characterization at high temporal resolution via DNN-based processing of scintillation patterns originating from a remote laser or LED beacon, using an embedded-edge AI computer (Jetson Xavier Nx). The DNN models were developed and trained with datasets composed of a large number of instances (scintillation images and their corresponding values) collected during several atmospheric measurement trials over a 7 km propagation path. TurbNet sensor performance was validated in a set of atmospheric inference trials performed side-by-side with a conventional scintillometer. The results of the inference trials demonstrated good correspondence (correlation coefficient γ ranging from 0.7 to 0.9) between the scintillometer measurements and turbulence strength assessment with the TurbNet sensors. It was shown that, due to the high temporal rate (between 1.4 and 1.8 s per measurement, depending on the number of DNN models used), the TurbNet sensors enabled real-time evaluation of rapid changes in atmospheric turbulence dynamics.
For practical applications of EO sensors utilizing DNN-based signal processing for
evaluation, one can potentially avoid atmospheric field trials used for the collection of DNN training datasets, which require side-by-side installation of both a DNN-based EO sensor (TurbNet) and a reference scintillometer, as described in
Section 3. The training dataset can be generated using wave-optics numerical simulations that provide accurate modeling of laser-beam propagation over any selected turbulence characterization site and, thus, can be utilized for the generation of sufficiently large datasets (SIM datasets) for DNN training. A DNN that is trained with this dataset can then be further used for real-time processing of scintillation images and
prediction at the selected site, as described in [
8]. In the case of DNN training using a simulated dataset, a scintillometer for collection of the training data in the field is not required. Furthermore, the EO sensor with a DNN model trained using a SIM dataset could be utilized at other propagation sites simply by computing a new SIM dataset specific for that site, with corresponding DNN retraining [
8].
In conclusion, it should be noted that since DNN models of TurbNet sensors are “trained” using a dataset (training dataset) obtained for specific system parameters and propagation lengths, deviation from these pre-set conditions results in a corresponding decline in
prediction accuracy. As recently shown, this problem can be addressed via scaling of the TurbNet sensor output
values [
23]. The required
scaling factors can be obtained separately for each system parameter or propagation length using either an analytical expression derived from the classical Kolmogorov turbulence theory (theory-based scaling), or through wave-optics numerical modeling and simulations (M&S-based scaling) mimicking sensor operation. For the TurbNet sensor described in this paper, the theory-based scaling factor
α(L) that minimizes
prediction error for the propagation distance
L is given by the simple expression
α(L) = L0 ⁄
L, where
L0 is the propagation length (
L0 = 7 km) used for generation of the training dataset. A similar scaling (adjustment) of the EO sensor output signal to address changes in path length or wavelength is used in conventional
sensing systems. Typically, these adjustments are also based on either theory (available analytical expressions), simulations, or experimental measurements performed during sensor calibration [
15,
21].
Furthermore, conventional EO atmospheric turbulence sensing systems (e.g., scintillometers [
15], differential image motion monitor (DIMM) sensors [
16], etc.) are based on analytical expressions derived from Kolmogorov turbulence theory, and, for this reason, are restrained by the theoretical assumptions and simplifications made in the derivations of these formulas. This significantly limits the utilization of these sensing systems outside the provisions prescribed by the theory, which are specific for each sensor type, e.g., weak scintillations, quasi-monochromatic, spatially coherent laser-beacon beam, point-like incoherent light source, etc. Violation of these prescribed requirements can lead to low accuracy or even incorrect
sensing data. This limitation of conventional
sensors can be overcome in the DNN-based sensing approach described here, which can be exploited for various EO sensing modalities (e.g., turbulence strength characterization using a partially coherent optical wave generated by the LED beacon of the TurbNet-LED sensor). The only requirement is that instantaneously measured output sensing data in such systems distinctly change in response to variations in atmospheric turbulence conditions.