Best Scanline Determination of Pushbroom Images for a Direct Object to Image Space Transformation Using Multilayer Perceptron

Abstract

Working with pushbroom imagery in photogrammetry and remote sensing presents a fundamental challenge in object-to-image space transformation. For this transformation, accurate estimation of Exterior Orientation Parameters (EOPs) for each scanline is required. To tackle this challenge, Best Scanline Search or Determination (BSS/BSD) methods have been developed. However, the current BSS/BSD methods are not efficient for real-time applications due to their complex procedures and interpolations. This paper introduces a new non-iterative BSD method specifically designed for line-type pushbroom images. The method involves simulating a pair of sets of points, Simulated Control Points (SCOPs), and Simulated Check Points (SCPs), to train and test a Multilayer Perceptron (MLP) model. The model establishes a strong relationship between object and image spaces, enabling a direct transformation and determination of best scanlines. This proposed method does not rely on the Collinearity Equation (CE) or iterative search. After training, the MLP model is applied to the SCPs for accuracy assessment. The proposed method is tested on ten images with diverse landscapes captured by eight sensors, exploiting five million SCPs per image for statistical assessments. The Root Mean Square Error (RMSE) values range between 0.001 and 0.015 pixels across ten images, demonstrating the capability of achieving the desired sub-pixel accuracy within a few seconds. The proposed method is compared with conventional and state-of-the-art BSS/BSD methods, indicating its higher applicability regarding accuracy and computational efficiency. These results position the proposed BSD method as a practical solution for transforming object-to-image space, especially for real-time applications.

1. Introduction

Linear pushbroom imaging sensors have become the most widely used optical satellite sensors due to modern high-resolution imaging technology [1,2]. These sensors capture high-resolution images with high revisit frequency and large coverage area, making them suitable for different photogrammetric and remote sensing tasks [3,4]. As a result, they are extensively used for Earth’s surface imaging in diverse applications, such as land cover classification, three-dimensional (3D) reconstruction, and environmental monitoring [5,6].

Linear pushbroom images have a more complex geometry and acquisition characteristics compared to frame-type images, such as aerial photographs [7]. In particular, each scanline of a linear pushbroom image has its own Exterior Orientation Parameters (EOPs) because it is captured at a specific exposure time [8]. This is different from frame-type images, which have a single set of EOPs [9,10,11]. As a result, the perspective center position and rotation angles of pushbroom images change scanline by scanline [7,12], making the object-to-image space transformation a complicated process [13,14]. When using a rigorous sensor model such as the Collinearity Equation (CE), physical parameters of a pushbroom image, including Interior Orientation Parameters (IOPs) and EOPs, are required to describe the imaging geometry on a line-by-line basis [15,16]. This means that the problem of the Best Scanline Search/Determination (BSS/BSD), or equivalently determining the exact time of exposure [17] must be solved to obtain the EOPs for each scanline. The process of transforming a particular point on the object/ground to its corresponding point in an image, known as object-to-image space transformation, is essential in photogrammetric applications like stereoscopic measurements, rectifying pushbroom images, epipolar resampling, and orthoimage generation [18,19]. Achieving sub-pixel accuracy is crucial for this transformation to ensure the accuracy of the resulting products [20].

Various BSS/BSD methods have been developed to apply object-to-image transformation for linear pushbroom images. The Sequential Search (SS) approach involves iteratively searching for the specific scanline of a ground point through the CE, which is inefficient due to the high computational cost of multiple transformations between object and image spaces per line through the CE. The Bisecting Window Search (BWS) method, introduced by [21], reduces the SS search space by repeatedly halving the image space and using the CE for object-to-image space transformation. However, the BWS method still requires an iterative procedure and is not efficient for practical applications. The Newton Raphson (NR) method proposed by [22] is another iterative BSS method that uses the mathematical Newton Raphson root-finding approach to account for pushbroom sensors’ characteristics. The Central Perspective Plane (CPP) approach proposed by [23] has proved superior applicability in aerial pushbroom images; although, the associated assumptions are violated when working with satellite images with distortion, and the efficiency is notably reduced [24]. This is because the CPP of the scanline becomes a 3D curved surface instead of a 3D plane. Therefore, it is necessary to divide the curved shape linear array into numerous short line segments. This segmentation enables the precise identification of the accurate CPP corresponding to the ground point [25]. The iterative process of establishing the relationship between each ground point and the CPP, as well as the computation of the best scanline through CE during the refinement stage make the CPP-based method relatively challenging to implement. In the General Distance Prediction (GDP) approach proposed by [26], the ground point is back-projected to image space using EOPs of the first scanline, then the GDP, the distance between the projected image point and the first line of the image, is calculated and used for refinement of specific equations. This iterative procedure will continue until the stopping criterion is met and the best scanline corresponding to the ground point is determined by linear interpolation. This method performs in an iterative procedure with several interpolations and thus requires moderate computation time. However, both the iterative process and the use of the CE to search for the best scanline result in a heavy computational burden for all these methods.

More recently, Ahooei Nezhad et al. [13] proposed two non-iterative three-stage BSD methods called Optimal Global Polynomial (OGP) and Artificial Neural Network (ANN). These methods use the CE and geometric calculations to obtain subpixel accuracies within a short time. Simulated points generated by the CE are used to model the errors between the exact and estimated scanline of these points through geometric interpolation. The OGP model later utilized a fifth-degree polynomial and Genetic algorithm to compress the model and improve the accuracy of the estimated scanline following the geometric interpolation of any given ground point. On the other hand, the ANN model used a different structure based on image characteristics for the refinement stage. While these methods do not involve iteration, they are relatively complex due to the three-stage procedure and use of geometric calculations.

This paper introduces a new approach for transforming objects to image space using a Multilayer Perceptron (MLP) Machine Learning (ML) algorithm to address the BSD problem. This method uses the CE to accurately simulate points across pushbroom images. A portion of these points is used to train the MLP to establish a relationship between object-to-image space, which can then be used to directly determine the best scanline for any arbitrary ground points. The proposed method was tested using ten images with diverse characteristics, and a comparison with previous methods was conducted to assess its accuracy and computational efficiency. This paper is divided into five sections. The Section 1 gives an overview of the BSS/BSD field and its importance in photogrammetry. The Section 2 begins with a presentation of the used datasets. Afterwards, the mathematical models that were dealt with are outlined. At the end of this section, the proposed method is described in detail, along with the prerequisite concepts of MLP. The experimental results of the proposed method are given in the Section 3. A discussion is included in a later section. Some conclusions are drawn in the final section.

2. Materials and Methods

A diagram of the proposed method is illustrated in Figure 1. In addition, the inputs and outputs of each step of the proposed method are provided in

Table 1. The inputs and outputs of the proposed method in each step.

		Inputs	Outputs
Preprocessing and data preparation	Space resection using the MPC model (Equation (2))	IOPs Image and object/ground coordinates of GCPs	EOPs of the whole image for each scanline
	SCOPs and SCPs generation using CE (Equation (1))	IOPs EOPs Mean height of the study area	Image and ground coordinates of SCOPs Image and ground coordinates of SCPs
Processing steps of BSD	Training phase of the MLP model (Equation (3))	Image and ground coordinates of SCOPs	Fitted MLP model for object-to-image transformation
	Prediction (testing) phase of the MLP (Equations (3)–(5))	Ground coordinates of SCPs	Image coordinates of SCPs estimated using the fitted MLP model Accuracy assessment

. The proposed method consists of two general steps. In the first step, the MLP network is trained using Simulated Control Points (SCOPs) to establish the relation between the object/ground and image spaces. In the second step, the model is evaluated using Simulated Check Points (SCPs), as arbitrary ground points, through the transformation from the ground space to the image space and determination the corresponding best scanlines. More details of the proposed method will be provided in Section 2.4.

2.1. Dataset

Ten satellite images acquired by eight different sensors, whose specifications are provided in

Table 2. The characteristics of the used images.

Image.	Sensor	Region	Dimension (Pixel)	Resolution (m)	Central Latitude	Central Longitude
ISB	IKONOS	Sao Paulo, Brazil	8300 × 8600	0.8	−23.54	−46.63
PMA	Pleiades 1A	Melbourne, Australia	6000 × 7000	0.5	−37.77	144.86
PAU	Pleiades 1B	Annapolis, USA	6057 × 5636	0.5	38.98	−76.49
QJI	QuickBird	Jaipur, India	6000 × 6000	0.6	26.92	75.78
SJB	SPOT 6	Jaicos, Brazil	6200 × 6600	1.5	−7.26	−41.27
SAN	SPOT 7	Amsterdam, Netherland	5824 × 6616	1.5	52.37	4.91
SCB	SPOT 7	Curitiba, Brazil	2597 × 1463	6	−24.63	−49.69
WBU	Worldview 1	Boulder, USA	6000 × 6000	0.5	40.02	−105.28
WSA	Worldview 2	Sydney, Australia	6000 × 6000	0.5	−33.84	151.20
WSU	Worldview 2	SanDiego, USA	3996 × 4015	0.5	32.72	−117.16

, were employed to evaluate the proposed BSD method. The experiments were carried out on images captured by various linear array pushbroom sensors, including IKONOS, Pleiades 1A, Pleiades 1B, QuickBird, SPOT 6, SPOT 7, Worldview 1, and Worldview 2. These images covered several parts of the world, including Sao Paulo (Brazil), Melbourne (Australia), Annapolis (USA), Jaipur (India), Jaicos (Brazil), Amsterdam (Netherlands), Curitiba (Brazil), Boulder (USA), Sydney (Australia), and San Diego (USA). Ten images with diverse sensor characteristics (e.g., dimension and spatial resolution) and distinct landscapes and topographic conditions (i.e., urban areas, flat areas, agricultural areas, and a mixture of all) were considered to conduct a robust assessment (see Figure 2). In particular, the spatial resolution of images varied between 0.5 m and 6 m. Note that all the images used in this study were accompanied by the Rational Polynomial Coefficients (RPC) files; the required Ground Control Points (GCPs) were obtained by these auxiliary files. Additionally, elevation values of the GCPs and the mean height of the study areas were extracted from the available Digital Elevation Model (DEM) sources.

2.2. Mathematical Models

To determine the dynamic geometry of satellite images, mathematical models were expanded, which relates the two-dimensional (2D) image space and 3D object/ground space [27,28,29]. The well-known CE is a commonly used mathematical model for linear array pushbroom images in photogrammetry, which can be expressed by Equation (1) [30].

$$\begin{gathered} \mathrm{x}=-\mathrm{f}{\displaystyle \frac{{\mathrm{r}}_{11}^{\mathrm{i}}\left(\mathrm{X}-{\mathrm{X}}_{\mathrm{S}}^{\mathrm{i}}\right)+{\mathrm{r}}_{12}^{\mathrm{i}}\left(\mathrm{Y}-{\mathrm{Y}}_{\mathrm{S}}^{\mathrm{i}}\right)+{\mathrm{r}}_{13}^{\mathrm{i}}\left(\mathrm{Z}-{\mathrm{Z}}_{\mathrm{S}}^{\mathrm{i}}\right)}{{\mathrm{r}}_{31}^{\mathrm{i}}\left(\mathrm{X}-{\mathrm{X}}_{\mathrm{S}}^{\mathrm{i}}\right)+{\mathrm{r}}_{32}^{\mathrm{i}}\left(\mathrm{Y}-{\mathrm{Y}}_{\mathrm{S}}^{\mathrm{i}}\right)+{\mathrm{r}}_{33}^{\mathrm{i}}\left(\mathrm{Z}-{\mathrm{Z}}_{\mathrm{S}}^{\mathrm{i}}\right)}}=0 \\ \mathrm{y}=-\mathrm{f}{\displaystyle \frac{{\mathrm{r}}_{21}^{\mathrm{i}}\left(\mathrm{X}-{\mathrm{X}}_{\mathrm{S}}^{\mathrm{i}}\right)+{\mathrm{r}}_{22}^{\mathrm{i}}\left(\mathrm{Y}-{\mathrm{Y}}_{\mathrm{S}}^{\mathrm{i}}\right)+{\mathrm{r}}_{23}^{\mathrm{i}}\left(\mathrm{Z}-{\mathrm{Z}}_{\mathrm{S}}^{\mathrm{i}}\right)}{{\mathrm{r}}_{31}^{\mathrm{i}}\left(\mathrm{X}-{\mathrm{X}}_{\mathrm{S}}^{\mathrm{i}}\right)+{\mathrm{r}}_{32}^{\mathrm{i}}\left(\mathrm{Y}-{\mathrm{Y}}_{\mathrm{S}}^{\mathrm{i}}\right)+{\mathrm{r}}_{33}^{\mathrm{i}}\left(\mathrm{Z}-{\mathrm{Z}}_{\mathrm{S}}^{\mathrm{i}}\right)}} \end{gathered}$$

(1)

where ($x$,$y$) are 2D image point coordinates, $f$ is the focal length, ($X$,$Y$,$Z$) are 3D ground point coordinates, i represents the scanline number, ($X_S^i$,$Y_S^i$,$Z_S^i$) are the object space coordinates of the perspective center, and ($r_{11}^i$,…,$r_{33}^i$) are the rotation matrix elements, including rotational angles $\omega$, $\phi$, and $\kappa$ of EOPs. It is worth noting that the x is theoretically equal to zero according to linear array pushbroom imagery characteristics.

In the practical processing of pushbroom images, when using the CE, the reliable EOPs of each scanline must be obtained during the space resection [30]. In this paper, the Multiple Projection Center (MPC) model was employed for EOPs’ calculation, the equations of which are given in Equation (2) [7,31].

$$\begin{gathered} {X_s}^i\left(t\right)=X_0+X_1\left(t_i\right)+X_2\left(t_i^2\right) \\ {Y_s}^i\left(t\right)=Y_0+Y_1\left(t_i\right)+Y_2\left(t_i^2\right) \\ {Z_s}^i\left(t\right)=Z_0+Z_1\left(t_i\right)+Z_2\left(t_i^2\right) \\ {w_s}^i\left(t\right)=w_0+w_1\left(t_i\right)+w_2\left(t_i^2\right) \\ {{\phi }_s}^i\left(t\right)={\phi }_0+{\phi }_1\left(t_i\right)+{\phi }_2\left(t_i^2\right) \\ {k_s}^i\left(t\right)=k_0+k_1\left(t_i\right)+k_2\left(t_i^2\right) \end{gathered}$$

(2)

where (${X_s}^i\left(t\right)$, ${Y_s}^i\left(t\right)$, …, and ${k_s}^i\left(t\right)$) are the ith scanline’s EOPs, ($X_0$, $X_1$, …, and $k_2$) are the achieved reference scanline’s EOPs, and $t_i$ indicates the ith scanline’s exposure time, which can be used as a substitute for the along-track coordinate of the satellite or the scanline number [17]. The availability of EOPs of each scanline and other specifications of the pushbroom sensor enables us to use CE to transfer any ground point to the corresponding image point in image space. At the stage of space resection based on the MPC model, appropriate control points with known image and ground coordinates must be available.

2.3. Simulated Points Generation

Two categories of simulated points, SCOPs and SCPs, were generated. The SCOPs and SCPs were simulated as a regular grid in the image space to comply with suitable distribution across images. Thereupon, these image points were mapped to the object space by employing the CE and the mean height of the study area. This projection step required the EOPs of each scanline, which were computed through the MPC model (Equation (2)). Hence, the image and object/ground coordinates and the exact scanline number (r) of SCOPs and SCPs were available. The SCOPs were used in the training step of the MLP, and the SCPs were used to evaluate the proposed BSD approach.

2.4. MLP Model

MLP models are widely recognized and used more often than other types of neural networks in various problem domains [32]. The MLP algorithm is a type of supervised machine learning that consists of interconnected elements called neurons, arranged in three distinct layers: input, hidden, and output [33,34]. Information is transmitted from the input layer to the output layer through the intermediary hidden layer. Each neuron within a layer forms complete connections with neurons in adjacent layers, and these connections are represented as weights during the computational process [32]. Each neuron maps multiple inputs to an output [35,36] using an activation function [37]. The number of independent variables in the model determines the number of neurons allocated to the input layer, while the number of neurons in the output layer corresponds to the count of dependent variables. The output layer can consist of a single neuron or multiple neurons. The number of neurons in the hidden layer of the MLP model depends on the network’s ability to model nonlinear functions [38]. MLP networks can effectively model and approximate both linear and nonlinear functions [39]. The MLP model establishes a connection between inputs and outputs by adjusting the weighted connections between neurons through the error back-propagation technique during the training process to minimize discrepancies between the anticipated target values and those generated by the model [32,33]. If the errors exceed a predefined threshold, weight adjustments are made to reduce these discrepancies. To define an MLP structure, important parameters such as the number of neurons, hidden layers, learning algorithm, and activation function must be considered [33]. In this study, several MLP topologies were tested to find the most accurate one for transforming object-to-image space. Only the Levenberg–Marquardt algorithm and the sigmoid function were used as the back-propagation learning algorithm and activation function, respectively, due to their efficiency and ease of implementation [13,40]. The Levenberg–Marquardt algorithm has been identified as a suitable learning algorithm for similar fields [41].

The proposed BSD method involves using SCOPs to train MLP models. The inputs are ground coordinates, and the output is the corresponding exact scanline to establish a relationship between object and image spaces (Equation (3)). The trained MLP models can be used to determine the scanline number of any ground points without the CE and iterative procedure. The trained MLP models were applied to SCPs, and the estimated scanline numbers were then compared with the exact scanline numbers to compute statistical assessment criteria.

$$Scanline number \left(r\right)=f\left(Ground coordinates\left(X, Y\right)\right)$$

(3)

In Equation (3), $\left(X, Y\right)$ are ground points’ coordinates, f is the MLP model and r is row/scanline number. It is worth noting that the relationship for object-to-image transformation involves the coordinates (X, Y, Z) and (r, c), which connect 3D ground space to 2D image space. However, Equation (3) lacks the third ground component due to the use of the average height of the study area, which remains constant for all ground points to simplify computation. Also, the equation does not include the second component of the image (c). The primary challenge in object-to-image transformation is calculating the scanline number.

2.5. Accuracy Assessment

As noted earlier, the SCPs were used to evaluate the performance of the proposed method. In this regard, the estimated scanline number using the MLP models was compared with the exact scanline number known from the simulation phase. Two statistical measures of Root Mean Square Error (RMSE) and dr_max (the maximum error in the scanline number determination) were computed according to Equations (4) and (5). Additionally, the computational time of the proposed method was measured to specify its time efficiency for real-time applications. Finally, the obtained results were compared with the ones computed by several other BSS/BSD methods it investigate the superiority of the proposed method.

$$RMSE=\sqrt{{\displaystyle \frac{d_r^T\times d_r}{n\left(d_r\right)}}}$$

(4)

$$d_r={|r}_{exact}-r_{estimated}|$$

(5)

In Equations (4) and (5), $r_{estimated}$ is the calculated scanline value obtained from the MLP model, $r_{exact}$ is the exact scanline value obtained from the simulation phase, $d_r$ is the difference between these two values, and $n\left(d_r\right)$ is the total number of scanlines in each image.

The SCOPs and SCPs were created separately in regular grids across each image as outlined in Section 2.3. In the evaluation stage, five million SCPs were considered. During the training phase, five different groups of SCOPs, each comprising 10, 20, 30, 50, and 100 points, were taken into account. The SCPs worked as check points for measuring criteria such as RMSE and dr_max. The SCOPs were randomly divided into two groups: 70% for training and 30% for validation. They were then used in the training phase of the MLP algorithm.

3. Results

During the training phase of MLP models, the structure of the model and the number of SCOPs used have a significant impact on the performance. To identify the most suitable MLP structure, a grid search analysis was conducted with the number of layers and neurons ranging between one and five (with an interval of one) and 10 and 50 (with an interval of ten), respectively. Increasing the number of layers and neurons in the model results in a more complex MLP model with a higher number of unknown parameters (ranging between 41 and 10,401), which requires more SCOPs. The study employed several sets of SCOPs with different sizes, ranging from 10 to 1000 points (in increments of 10, 20, 30, 50, 100, 300, 500, and 1000) to explore the impact of SCOPs size. Although this led to 1000 cases for each image, only feasible cases (i.e., achieving sub pixel accuracy) were considered to ensure convergence concerning the number of SCOPs and unknown parameters.

Table 3. The results of ISB image analysis using different MLPs.

Number of SCOPs	Number of Layers	Number of Neurons	RMSE (Pixel)	Computation Time (s)	drmax (pixel)	Number of MLP Parameters
50	1	10	0.06	3.27	0.15	41
100	1	10	0.01	3.40	0.02	41
	1	20	0.05	4.20	0.17	81
300	1	10	0.02	3.90	0.07	41
	1	20	0.02	4.84	0.06	81
	1	30	0.02	5.62	0.06	121
	1	40	0.02	6.36	0.08	161
	1	50	0.09	7.75	0.03	201
	2	10	0.06	5.70	0.37	151
	2	20	0.14	19.63	0.69	501
500	1	10	0.03	3.95	0.09	41
	1	20	0.02	4.69	0.07	81
	1	30	0.001	5.95	0.003	121
	1	40	0.001	6.64	0.005	161
	1	50	0.002	7.76	0.01	201
	2	10	0.06	5.80	0.30	151
	2	20	0.05	21.48	0.15	501
	3	10	0.02	8.47	0.20	261
1000	1	10	0.04	4.04	0.09	41
	1	20	0.06	5.17	0.15	81
	1	30	0.05	6.81	0.16	121
	1	40	0.10	8.04	0.24	161
	1	50	0.02	9.72	0.08	201
	2	10	0.03	6.86	0.20	151
	2	20	0.01	26.40	0.09	501
	3	10	0.14	10.23	0.90	261
	4	10	0.02	17.83	0.93	371
	5	10	0.01	26.53	0.09	481

provides a summary of the statistical results (RMSE and dr_max) and computation times (using an Intel Core i7-7500U 2.90 GHz processor) for the ISB image based on MLP models. The evaluations were carried out with five million SCPs, and only the feasible cases that achieved the desired sub-pixel accuracies are displayed in Table 3. The RMSE and dr_max values were almost the same (differing by just hundredths and thousandths of a pixel) for all cases, with variations primarily in computation times. Larger MLP models led to longer processing times and more SCOPs, resulting in higher complexity. Consequently, MLP models with fewer layers (one), neurons (five to ten), and SCOPs were chosen to minimize unnecessary complexity and reduce the risk of overfitting in subsequent steps [42].

The detailed results for the ISB image are presented here as the outcomes for other images were found to be similar.

The results of the MLP models with one layer and five-to-ten neurons for all the images are shown in Figure 3, Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8. Generally, the results indicated that the number of neurons had no significant effect on the RMSE and dr_max values, considering the cases that achieved sub-pixel accuracies. To evaluate the performance of the BSD method, five SCOPs sets with 10, 20, 30, 50, and 100 simulated points and five million SCPs per image were used. However, the SCOPs sets with 10 and 20 simulated points failed to establish a transformation between object and image spaces with sub-pixel accuracy, so their results were excluded. This suggests that a limited number of SCOPs cannot be used to relate object and image spaces, even with shallow MLP models. According to the results, even 30 SCOPs were not enough for object-to-image transformation with the desired accuracy. For some images, the obtained RMSE and dr_max values were higher than one pixel, which were mapped to one for better representation. When using an MLP with a 5-neuron structure and 30 SCOPs, the ISB, SJB, and SAN images had RMSE values higher than one pixel. This may be due to the lower spatial resolution of these images compared to others. However, the land cover in these images varies, including urban, flat, and mixed areas, so the accuracy is not solely dependent on the land cover. Increasing the number of SCOPs to 100 resolved this issue, providing sub-pixel accuracy in all images. In the case of a 6-neuron structure with 30 or 50 SCOPs, the dr_max values of ISB, PMA, PAU, and SJB images were more than one pixel, but using 100 SCOPs achieved the desired sub-pixel accuracy.

Similarly, using a 7-neuron MLP structure resulted in RMSE values of more than one pixel for SAN, SJB, and ISB images due to their low resolution. The complexity of an 8-neuron MLP network required more SCOPs, and using 30 SCOPs still resulted in RMSE and dr_max values of more than one pixel for ISB, PMA, SJB, SAN, and WSU, indicating the insufficiency of this number of SCOPs. Finally, increasing the number of neurons led to convergence issues or very low accuracy due to the increase in unknown parameters and deficient SCOPs to estimate them. However, the MLP models obtained reasonable RMSE values for the SCB image even when the number of neurons reached ten, possibly due to the smaller dimensions of the image or its coarser spatial resolution. In general, the cases without sub-pixel accuracy increased as the number of neurons increased, which is due to the fact that MLP models with a higher number of neurons have more unknown parameters, and 30 SCOPs were insufficient. The issues were solved when the number of SCOPs reached 50, and almost all MLP models converged with sub-pixel accuracy, except for a few cases in the ISB and PMA images. This might be due to the dimension of the ISB and PMA images, in which 50 SCOPs were not enough to cover the images appropriately and enable a sub-pixel transformation. The existence of dense buildings in the urban landscape of the ISB image might introduce more complexity to the MLP models. The results imply that the desired RMSE and dr_max in all images were attained when 100 SCOPs were used in the training phase of the MLP models. In other words, 100 SCOPs comprehensively supported achieving sub-pixel accuracy regardless of image dimension, spatial resolution, and landscape characteristics. It should be noted that the MLP models could also attain sub-pixel accuracies when a higher number of SCOPs are used; however, such numbers incur computation time and increase the risk of overfitting. In general, the proposed MLP algorithm can determine the best scanline with very high accuracy on different data with different characteristics by choosing the right network structure and the optimal number of SCOPs.

In this study, the proposed method for BSS/BSD was compared to previous state-of-the-art methods. The comparison was based on several quantitative measures: RMSE, computation time, dr_max, and the number of SCOPs (

Table 4. Obtained results of the proposed approaches and other well-known methods for best scanline determination.

Datasets	Quantitative Measurements	Methods’ Name
		Newton Raphson (NR) [22]	Bisecting Window Search (BWS) [21]	ANN BSD [13]	OGP BSD [13]	Proposed Method (MLP)
ISB	RMSE (pixel)	5.840 × 10−10	0.57	0.29	0.29	0.015
	Computation time (second)	511.919	1490.505	3.295	6.812	3.29
	${dr}_{max}$ (pixel)	1.727 × 10−9	1	0.61	0.57	0.043
	Number of SCOPs	-	-	400	400	100
	Number of neurons	-	-	10	-	9
PMA	RMSE (pixel)	1.057 × 10−9	0.58	0.30	0.30	0.003
	Computation time (second)	591.588	1413.097	3.427	7.941	3.31
	${dr}_{max}$ (pixel)	2.616 × 10−9	1	0.67	0.67	0.007
	Number of SCOPs	-	-	500	500	100
	Number of neurons	-	-	10	-	10
PAU	RMSE (pixel)	9.561 × 10−10	0.58	0.30	0.30	0.003
	Computation time (second)	520.065	1294.618	3.398	7.640	2.92
	${dr}_{max}$ (pixel)	1.203 × 10−9	1	0.67	0.77	0.010
	Number of SCOPs	-	-	500	500	50
	Number of neurons	-	-	10	-	10
QJI	RMSE (pixel)	4.401 × 10−10	0.58	0.30	0.30	0.002
	Computation time (second)	423.808	1320.126	3.815	7.839	3.23
	${dr}_{max}$ (pixel)	1.162 × 10−9	1	0.72	0.69	0.006
	Number of SCOPs	-	-	500	500	100
	Number of neurons	-	-	10	-	10
SJB	RMSE (pixel)	6.182 × 10−10	0.58	0.30	0.30	0.002
	Computation time (second)	480.380	1335.430	3.776	9.959	3.01
	${dr}_{max}$ (pixel)	2.184 × 10−9	1	0.73	0.63	0.005
	Number of SCOPs	-	-	500	1000	100
	Number of neurons	-	-	10	-	7
SAN	RMSE (pixel)	2.503 × 10−10	0.57	0.29	0.31	0.002
	Computation time (second)	471.247	1259.437	4.452	8.305	2.96
	${dr}_{max}$ (pixel)	6.207 × 10−10	1	0.58	0.81	0.004
	Number of SCOPs	-	-	700	500	100
	Number of neurons	-	-	10	-	5
SCB	RMSE (pixel)	6.593 × 10−10	0.58	0.29	0.29	0.001
	Computation time (second)	412.186	883.381	3.387	7.070	3.13
	${dr}_{max}$ (pixel)	2.379 × 10−9	1	0.61	0.55	0.003
	Number of SCOPs	-	-	400	200	100
	Number of neurons	-	-	10	-	9
WBU	RMSE (pixel)	1.181 × 10−9	0.57	0.29	0.28	0.002
	Computation time (second)	474.471	1376.620	3.576	7.967	2.96
	${dr}_{max}$ (pixel)	2.756 × 10−9	1	0.52	0.52	0.005
	Number of SCOPs	-	-	400	400	100
	Number of neurons	-	-		-	7
WSA	RMSE (pixel)	9.931 × 10−10	0.57	0.32	0.32	0.003
	Computation time (second)	470.185	1245.763	3.310	7.803	2.09
	${dr}_{max}$ (pixel)	2.461 × 10−9	1	0.72	0.71	0.006
	Number of SCOPs	-	-	500	500	100
	Number of neurons	-	-	10	-	5
WSU	RMSE (pixel)	3.860 × 10−10	0.57	0.30	0.30	0.001
	Computation time (second)	483.562	1281.063	3.714	8.796	3.02
	${dr}_{max}$ (pixel)	9.327 × 10−10	1	0.57	0.58	0.003
	Number of SCOPs	-	-	500	500	100
	Number of neurons	-	-	10	-	6

). The NR method achieved sub-pixel accuracy but had a long computation time due to its iterative procedure of root-finding using the CE. The BWS method, despite being an improved version of SS, still required considerable time to determine the best scanline. Additionally, the BWS method could not achieve sub-pixel accuracy according to dr_max, although the RMSE values were below one pixel. In contrast, the proposed method achieved sub-pixel accuracy in both RMSE and dr_max and required significantly less computation time. The ANN and OGP are two state-of-the-art BSD methods that require SCOPs. The required number of SCOPs varied between 200 and 700, with 500 being sufficient for most images. The proposed method achieved sub-pixel accuracy with only 100 SCOPs, while the ANN and OGP methods required trial and error attempts to identify the optimal number of SCOPs. Moreover, the proposed method required nearly half of the computation time compared to the OGP. The ANN and the proposed method (both with one hidden layer) obtained similar results, although the proposed method was more efficient, requiring fewer SCOPs and achieving better RMSE and dr_max values.

4. Discussion

Efficiency and accuracy are two crucial criteria when evaluating a BSS/BSD method in object-to-image space transformation in linear array pushbroom imagery. Efficiency refers to computation time, while accuracy pertains to the difference between estimated and exact scanline numbers. Sub-pixel accuracy is desired for a BSS/BSD method to derive suitable products in photogrammetric tasks. The proposed BSD method showed significant performance compared to previously well-known and state-of-the-art methods, as shown in Figure 3, Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8 and Table 4. Ten images with diverse sensor characteristics (e.g., dimension and spatial resolution) and distinct landscapes and topographic conditions (i.e., urban areas, flat areas, agricultural areas, and a mixture of all) were considered to conduct a robust assessment. We believe that due to this variety and the achieved promising results, the proposed method is likely to exhibit broad applicability and generality across other datasets.

The research used the MLP model for the BSD and also tested other popular ML algorithms such as Random Forest (RF) [43] and Support Vector Machines (SVM) methods [44] with various numbers of SCOPs and five million SCPs. However, these methods were unable to accurately establish the relationship between object and image space with sub-pixel precision. The minimum RMSE obtained from these methods reached one hundred pixels, which is unacceptable. The evaluation demonstrates that using the MLP method is more effective than other ML methods.

The main goal of this paper was to utilize the MPC model in space resection to compute the EOPs of each scanline for all images. These EOPs were then employed to simulate points. If the EOPs are provided with the images as an auxiliary file, they can be directly used in the BSD method, and the space resection step can be bypassed. The results demonstrated that the number and distribution of SCOPs are crucial in obtaining the desired accuracy. The number of SCOPs should be the least required, and their distribution across the image should be in a regular grid with an appropriate number. The MLP model’s non-compliance with image and region characteristics and an inappropriate number of SCOPs are among the factors that produce undesired values for RMSE and dr_max.

It is worth noting that in pushbroom imagery, the sensor captures the image perpendicular to its flight direction. According to theory, the first component of the CE for this kind of sensor should be zero (Equation (1)). However, in practice, it is possible to obtain a small x value, around 10⁻⁶ or smaller. In such cases, this difference can be ignored as it will not affect the algorithm process.

The ANN and OGP BSD methods developed by [13] performed closely to the proposed BSD method. However, the proposed method directly determines the best scanline number, making it simpler than ANN and OGP methods, both of which were three-step algorithms. In particular, the ANN model was used for the refinement of scanline numbers after estimating the initial row and column values of each ground point through an affine transformation and computing its approximate time of exposure using a specific equation. The ANN model then establishes the relation between the initial row and column values of each ground point and difference between calculated approximate time and exact time, which is available from SCOPs. The ANN method obtained sub-pixel accuracies, but its computation time was 22% higher than the proposed BSD method on average. The OGP method replaced the ANN with an optimized polynomial expression. The optimization task was conducted by the genetic algorithm, which increases the processing time. Although the ANN and MLP methods may seem very similar, the ANN method is a part of a three-step BSD algorithm for error modeling, while the MLP method is based on a single-step neural network, the purpose of which is to calculate the best scanline directly. Both models have a similar three-layer structure, but due to different goals, the number of neurons in the hidden layer and the number of SCOPs are different from each other. Owing to the distinction between these two methods, we named the first method as the ANN method and the proposed method as MLP method. Moreover, both ANN and OGP methods required a higher number of SCOPs to successfully relate object and image spaces with sub-pixel accuracy, while 100 SCOPs were sufficient for the MLP models regardless of image and landscape characteristics. Furthermore, the proposed BSD method has no specific assumption and can be employed for linear array pushbroom sensors with varying exposure times.

5. Conclusions

A robust and accurate ground-to-image transformation method is crucial in the geometric processing workflows of linear pushbroom images. Due to the different imaging geometry in these sensors, the transformation of coordinates from the ground space to the image space is more complicated than frame images. For this reason, the BSS/BSD methods in this field are proposed. This study proposes a new algorithm for object-to-image space transformations in linear array pushbroom images. The algorithm is a single-stage, non-iterative approach based on MLP models, resulting in sub-pixel accuracy. The study shows that using MLP models with lower complexity and 100 SCOPs can ensure achieving sub-pixel accuracy regardless of using diverse sensor (e.g., dimension and spatial resolution) and different landscapes and topographic conditions (urban areas, flat areas, agricultural areas, and a mixture of all). The proposed method is compared with well-known BSS/BSD methods such as NR, BWS, ANN, and OGP. The results reveal that the MLP model has low processing complexity, demands less computation time, and delivers significantly improved accuracies compared to previous methods. The proposed method has no specific assumption without the need to acquire information about the structure of detectors and is applicable for all linear array pushbroom images and subsequent photogrammetric tasks. The obtained accuracies and computation time suggest the capability of the proposed method for near real-time applications. In future studies, the utilization and assessment of the proposed method will be conducted based on its efficacy and precision, particularly in the realm of photogrammetric tasks like ortho-photos generation.