Neural Network-Aided Optical Navigation for Precise Lunar Descent Operations

Abstract

Advanced navigation capabilities are essential for precise landing operations, enabling access to critical lunar sites and supporting future lunar infrastructure. To achieve accurate positioning, innovative navigation methods leveraging neural network frameworks are being developed to detect distinctive lunar surface features, such as craters, from imaging data. By matching detected features with known landmarks stored in an onboard reference database, key navigation measurements are retrieved to refine the spacecraft trajectory, enabling real-time planning for hazard avoidance. This work presents a crater-based navigation system for planetary descent operations, which leverages a robust machine learning approach for crater detection in optical images. A thorough analysis of the attainable detection accuracies was performed by evaluating the network performance on diverse sets of synthetic images rendered at different illumination conditions through a custom Blender-based pipeline. Simulation campaigns, based on the JAXA Smart Lander for Investigating Moon mission, were then carried out to demonstrate the system’s performance, achieving final position errors consistent with 3 − $\sigma$ uncertainties lower than 100 m on the horizontal plane at altitudes as low as 10 km. This level of accuracy is key to achieving enhanced control during the approach and vertical descent phases, thereby ensuring operational safety and facilitating precise landing.

1. Introduction

Optical measurements have supported spacecraft operations in deep space since the Voyager era, complementing Earth-based navigation techniques that rely on processing radio tracking data during the approach phase to planets, moons, and asteroids [1]. Images captured by spacecraft cameras are transmitted to Earth, where the navigation team analyzes them to identify known celestial objects. The image coordinates (sample and line) of these identified landmarks are then used to refine the spacecraft orbit determination, allowing for the precise computation of trajectory correction maneuvers.

Technological advancements over the last decades enabled one of the first successful implementations of an autonomous onboard optical navigation (OpNav) system with the Deep Space-1 mission [2]. This achievement paved the way for advanced image-based navigation technologies, which have subsequently supported real-time operations during crucial mission phases, including close proximity operations at small bodies [3,4], touch-and-go maneuvers for terrain sampling [5], and pinpoint landing [6,7].

Depending on the distance between the target body and the spacecraft, different techniques are used to extract navigation information from imaging data [8,9]. Center-finding and limb-scanning methods, for instance, are employed during the cruise and approach phases. As the spacecraft approaches the surface of the celestial body, the onboard systems carry out landmark-based navigation. This approach is based on the detection and tracking of surface features, such as specific morphological elements (e.g., craters) or distinct terrain patches (e.g., maplets [10]).

In the context of terrain sampling maneuvers, the OSIRIS-REx mission utilized an autonomous maplet-based navigation system to achieve precise targeting of the asteroid Bennu’s surface [5,11]. This system correlated camera images with the expected appearance of landmark patches, enabling accurate corrections of the spacecraft positioning. Similarly, the Perseverance rover employed a feature-based navigation system for precise landing at the Jezero crater [6]. By matching point-like features (e.g., Harris interest points) between descent images and onboard reference maps of the landing area, the onboard flight system obtained real-time, accurate estimations of its state during descent, leading to unprecedented landing precision. Cross-correlation techniques were used to align keypoints between the descent images and the onboard reference map, ensuring a close match in appearance [12]. The process of generating and validating these reference maps is both complex and time-consuming. As a result, while this navigation approach holds promise for lunar missions, alternative navigation methods that leverage known surface features, such as craters, are being explored.

Craters are indeed utilized as fiducials to enhance the spacecraft localization, as demonstrated by the JAXA Smart Lander for Investigating Moon (SLIM) mission. A crater-based navigation approach was adopted to achieve landing accuracies better than 100 m [7]. By matching craters detected in images from the navigation camera with those on onboard crater maps, the system provided accurate estimates of the spacecraft position during descent [13]. Crater detection was carried out through computer vision techniques (i.e., Haar-like features), which identify craters based on their characteristic appearance, including illuminated interiors and shaded neighborhoods [14]. To address challenges posed by non-nominal illumination conditions, neural network-based approaches are gaining traction for their ability to handle environmental uncertainties effectively. A careful training dataset augmentation can enhance the network’s adaptability to diverse operational scenarios, thereby improving navigation accuracies [15].

Although the literature extensively covers machine learning techniques for crater detection, including various neural network architectures designed to identify crater locations from diverse input data (e.g., optical images, radar images, topographic maps), the practical challenges associated with integrating artificial intelligence (AI) frameworks into onboard navigation systems remain largely unaddressed. A critical limitation in space applications is the deployment of computationally demanding AI-driven algorithms on resource-constrained, space-qualified electronic hardware. However, future missions are expected to leverage commercial-off-the-shelf (COTS) computing platforms [16,17], facilitating the onboard execution of AI-enabled methods. The performance and reliability of these algorithms must be rigorously assessed to ensure their suitability for autonomous navigation and decision-making in space environments. Evaluations of AI-based crater detectors typically focus on the metrics such as precision and recall [18], but a crucial aspect that is often unexplored is the accuracy of the estimated crater centroid coordinates, which serve as critical measurements for OpNav. This accuracy directly influences the performance of multi-sensor navigation systems [19], as the precision of the data provided by each instrument significantly impacts the overall accuracy of the spacecraft’s reconstructed trajectory. These key performance factors are investigated in this study through experiments carried out in a high-fidelity synthetic lunar environment. A quantitative estimate of the attainable crater detection accuracies is retrieved by analyzing the crater detection errors under different observation conditions. Data augmentation is incorporated to enhance the network’s robustness to the illumination conditions, ensuring unbiased crater detections independent of solar direction.

In this paper, we present a crater-based navigation pipeline that leverages a state-of-the-art object detection model to enhance crater extraction accuracy. The performance of the navigation system is rigorously evaluated through extensive open-loop numerical simulations conducted within a custom high-fidelity digital planetary environment. The paper is structured as follows. Section 2 provides a detailed description of the software tools used to generate synthetic images of planetary surfaces and offers an in-depth discussion of the crater detection and matching process. In Section 3, we analyze the robustness of the crater detection and matching system under varying operational conditions. Section 4 presents the simulation setup used to assess the system performance, which is modeled after the JAXA SLIM mission. In Section 5, the attainable navigation accuracies are analyzed, including a Monte Carlo simulation, to assess the system’s robustness. Conclusions and future work are presented in Section 6.

2. Data and Methods

A high-level picture of the feature-based optical navigation pipeline is shown in Figure 1. At each new image acquisition (Section 2.1), craters are extracted by using an AI-enabled crater detection algorithm (Section 2.3) and then matched with pre-mapped craters from an onboard database (Section 2.2). If a sufficient number of crater matches is established (Section 2.4), the information is processed through a sequential estimation filter and the spacecraft’s trajectory is adjusted accordingly (Section 4).

2.1. Generation of Synthetic Images

To support the design of novel vision-based navigation techniques, testing these systems in a realistic synthetic environment is essential. A range of software tools, both commercial (e.g., PANGU [20], SurRender [21]) and open-source (e.g., SISPO [22], CORTO [23]) have been developed to create high-fidelity synthetic scenarios. Open-source tools often utilize rendering platforms, including Blender [24] and Unreal Engine [25]. Advanced techniques for generating photorealistic images of synthetic scenes employ physically based path tracing, which models light as rays and applies physical principles to simulate its interaction with both the camera and scene objects. Blender, with its state-of-the-art path-tracing engine (i.e., Blender Cycles), has become widely adopted in the scientific community for producing realistic images of planetary surfaces [22,23,26,27]. Furthermore, Blender’s scripting environment allows for the implementation of custom reflectance functions, enabling precise modeling of the complex photometric behavior of the planetary surface regolith [28]. Leveraging these capabilities, we developed a custom Blender-based pipeline to generate high-fidelity images of planetary surfaces.

The image rendering process entails an accurate definition of the scene, including cameras, light sources, and a 3D model of the celestial body. The first step involves deriving a mesh representing the observed portion of the body’s surface from a Digital Terrain Model (DTM), which is a 2D array of elevation values h referenced to a planetary shape, such as a sphere with radius $R_{ref}$. Each cell in the DTM corresponds to a specific point on the planet’s surface. After defining the mesh, it is imported into Blender for rendering. Although Blender plugins facilitate DTM-to-mesh conversion (e.g., Blender-Hirise-DTM-Importer (https://github.com/phaseIV/Blender-Hirise-DTM-Importer, accessed on 1 January 2025), Blender-GIS (https://github.com/domlysz/BlenderGIS, accessed on 1 January 2025), they often encounter limitations such as non-georeferenced points (i.e., the DTM is imported as a displacement texture) and a time-consuming mesh creation process (e.g., accessing elements of the 2D array using nested loops). To address these issues, we implemented a cost-effective pipeline for generating georeferenced meshes from planetary DTMs.

First, a 3D point cloud is generated from the DTM by computing the 3D coordinates $(\rho ,lon,lat)$ for each surface point corresponding to a DTM cell (see Appendix C). These coordinates are obtained from the map coordinates of the corresponding cell $(x,y)$ and the associated elevation value h (i.e., $\rho =R_{ref}+h$). The computed 3D coordinates $(\rho ,lon,lat)$ are then transformed into Cartesian coordinates $(X,Y,Z)$ with respect to the body-fixed frame. Each point, representing a vertex of the final mesh, is assigned a unique identifier (ID) to streamline the process of defining the mesh faces.

As an example, a triangular face in the mesh is defined by a 3D vector, where each element represents the ID of a vertex. To efficiently assign these vertex IDs, a numbering scheme that leverages the DTM’s rasterized structure was adopted. First, an index matrix M is created, matching the size of the DTM array, which assigns a unique ID to each cell. The cells are numbered sequentially within each column, starting from the top and moving downwards (with an index of 1 assigned to the top-left cell). Upon reaching the bottom of a column, the numbering continues, proceeding until the entire grid is covered, resulting in the following matrix:

$$M=\left[\begin{array}{ccc}1& \cdots & n_R(n_C-1)+1\\ ⋮& \ddots & ⋮\\ n_R& \cdots & n_R{n}_C\end{array}\right].$$

(1)

Alternatively, the matrix can be expressed in compact form as

$$M_{ij}=i+(j-1)n_R,$$

(2)

where $i=1,\cdots ,n_R$ and $j=1,\cdots ,n_C$ are the row and column indices of the DTM cells, respectively, with $\left(n_R\right)$ representing the number of rows and $\left(n_C\right)$ the number of columns in the DTM array.

The computation of the index matrix enables the straightforward definition of the face matrix F, which has dimensions $(n_F\times 3)$, where $n_F$ represents the number of the faces in the mesh. Each row of the face matrix corresponds to a face in the final mesh and contains the IDs of the three vertices defining that face, listed in counter-clockwise order as seen from the outward normal vector. From each set of four neighboring vertices, two triangular faces are extracted, i.e., an upper triangular face and a lower triangular face. First, two auxiliary matrices of dimension $((n_F/2)\times 3)$ are computed, which collect the vertex IDs of all the upper triangular faces ($F_{UT}$) and the lower triangular faces ($F_{LT}$). The final face matrix (F) is then defined by accounting for the contribution of these two matrices as:

$$F=\left[\begin{array}{c}{F}_{UT}\\ F_{LT}\end{array}\right],$$

(3)

where

$$\begin{array}{cc}\hfill F_{UT}& =\left[\begin{array}{ccc}{F}_2& F_3& F_4\end{array}\right]\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill F_{LT}& =\left[\begin{array}{ccc}{F}_1& F_2& F_4\end{array}\right].\hfill \end{array}$$

(5)

Here, $F_1$ is a column vector derived from the matrix $\overline{M}$ by traversing it in row-major order, where $\overline{M}$ is a reduced-size matrix obtained from M by removing its first row and column. The remaining face matrix elements are computed as follows: $F_2=F_1-1$; $F_3=F_2-n_R=F_1-(n_R+1)$; and $F_4=F_1-n_R$. After computing the mesh vertices and faces, the 3D model can be generated and imported into Blender. Since the coordinates of each vertex are defined in the body-fixed frame, Blender’s coordinate system will align with the body-fixed frame’s axes. Furthermore, because the pipeline relies solely on matrix operations, it ensures a significant reduction in the time required for mesh generation.

Because the time required to convert a DTM into a mesh increases with its size, processing a full-size DTM, such as a global map, becomes impractical. Smaller DTMs are then extracted from the full-size model based on the camera footprint, which defines the visible surface area in the image. The extraction procedure involves several key steps. First, intersection points are computed between a set of 3D vectors sampled from the boundaries of the camera’s field-of-view (i.e., viewing frustum) and the surface of the celestial body, which is assumed to be spherical with radius R. This step yields a set of 3D points $(X_i,Y_i,Z_i)$, whose coordinates are referred to the body-fixed frame. These points are converted into spherical coordinates $({\rho }_i,lo{n}_i,la{t}_i)$, and then projected onto a 2D map to obtain map-projected points $(x_i,y_i)$ [29]. The boundaries $(x_{\min },x_{\max },y_{\min },y_{\max })$ of the smaller-sized DTM are computed from these map-projected points. Subsequently, the full-size DTM is cropped according to these boundaries and the resulting smaller DTM is converted into a mesh, which is then imported into Blender.

The choice of full-size DTMs for the generation of local DTMs depends on the latitude of the observed area. To simulate images collected at equatorial and mid-latitude regions, we used full-size DTMs in equirectangular map projection, whereas to render images at high-latitude and polar regions we employed full-size DTMs in a stereographic projection. This ensures that the local maps accurately reflect the characteristics of the terrain based on its latitude.

Figure 2 shows some examples of meshes representing a $(5^{°}\times 5^{°})$ area of lunar surface centered at $(lo{n}_c,la{t}_c)=({48}^{°},-{13}^{°})$. The meshes are derived from smaller-sized DTMs extracted from global Moon DTMs at resolutions of 128 px/deg (left), 16 px/deg (center), and 4 px/deg (right), respectively (https://pds-geosciences.wustl.edu/lro/lro-l-lola-3-rdr-v1/lrolol_1xxx/data/lola_gdr/cylindrical/img/, accessed on 1 January 2025). A wireframe view of the mesh is also provided on the right panel, highlighting the vertices and the faces that define the mesh.

A preliminary validation campaign was carried out to assess the Blender-based pipeline for generating synthetic images. The mission cases accounted for in these tests are based on observation campaigns in the context of the exploration of the Moon, planets, and small bodies. Synthetic images were produced to match real spacecraft imagery in terms of camera properties, acquisition geometries, and illumination conditions. The test cases included images from Kaguya’s High-Definition Television (HDTV) camera and the framing cameras on the Rosetta and MESSENGER spacecraft.

To accurately position the camera in the scene, the camera attitude matrix with respect to the planet’s body-fixed frame was extracted from the mission attitude kernels. The quaternion associated with the pointing matrix ${\mathbf{R}}_B^C$ is computed and used to define the camera orientation within the Blender scene (Blender’s camera frame convention differs from that used in the computer vision community, with the z-axis pointing opposite to the boresight direction, and the x- and y-axes pointing to the right side and to the top of the image, respectively. The camera attitude matrix retrieved from the mission attitude kernels should be adjusted accordingly). The camera position is determined from the mission position kernels at epoch $t=t_0+t_{exp}/2$, where $t_0$ is the initial image acquisition time, and $t_{ext}$ is the image exposure time, both of which are provided in the image metadata.

Figure 3 and Figure 4 show a real (left) and a rendered image (right) of the Moon, as seen from Kaguya’s HDTV telephotographic (TELE) and wide-angle (WIDE) cameras, respectively [30]. The real and rendered images are consistent with surface features aligning closely and appearing similarly in size and in terms of illuminated and shadowed regions. Nevertheless, the HDTV-TELE rendered image appears to be slightly “pitched up”. This discrepancy may be attributed to minor errors in the reconstructed camera attitude, as derived from the mission SPICE kernels. The impact of these mismodeling is amplified by the spacecraft observation geometry, with the camera oriented in the forward direction. Such inconsistencies preserve information on orbital corrections that may be derived through skyline-based image residuals.

In addition to lunar missions, the image generation pipeline is also adaptable to other mission scenarios, accommodating cameras with differing fields of view and image resolutions. Additionally, the pipeline enables the seamless extraction and integration of camera position and attitude data from the mission SPICE kernels.

2.2. Optical Data Modeling

The optical measurements supporting spacecraft navigation are based on a landmark-based approach. In this method, the spacecraft’s state is refined by minimizing the 2D discrepancies between observed crater centroids and the projected centroids of pre-cataloged craters. This comparison allows for precise adjustments to the spacecraft position and velocity, ensuring accurate navigation relative to the observed celestial body.

A perspective distortion-free camera model is used to project 3D points onto a 2D image plane (Figure 5). For a given 3D point in the camera frame C, ${}^C\mathit{P}={[X,Y,Z]}^{\mathrm{T}}$, its 2D projection onto the image plane, $\mathit{p}={[x,y]}^{\mathrm{T}}$, is computed as follows [8]:

$$\left[\begin{array}{c}\mathit{p}\\ 1\end{array}\right]=\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]={\displaystyle \frac{1}{Z}}\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right],$$

(6)

where $f_x$ and $f_y$ are the focal length scaled by the pixel size along horizontal and vertical directions, respectively (for square pixels $f_x=f_y=f$), and $(c_x,c_y)$ is the principal point (i.e., the image point where the optical axis intersects the image plane, assumed to be orthogonal to the image plane). The 3D point ${}^C\mathit{P}$ is derived from

$${}^C\mathit{P}={\mathbf{R}}_B^C\left[{}^B\mathit{r}_{\mathit{C}}-{}^B\mathit{r}_{\mathit{SC}}\right],$$

(7)

where ${\mathbf{R}}_B^C$ is the rotation matrix from the Moon’s body-fixed frame to the camera frame, while ${}^B\mathit{r}_C$ and ${}^B\mathit{r}_{\mathit{SC}}$ are the position vectors of the crater centroid and the spacecraft, relative to the Moon’s center, both expressed in the Moon’s body-fixed frame B.

The Cartesian coordinates of the i-th cataloged crater in the Moon’s body-fixed frame are derived from its spherical coordinates $({\rho }_i,la{t}_i,lo{n}_i)$. Accurate modeling of the crater’s distance from the Moon’s center (${\rho }_i$) is key to correctly project its location onto the image plane. This distance is expressed as

$$\rho =\overline{\rho }+\Delta h,$$

(8)

where $\overline{\rho }$ represents the radial distance from the Moon’s center to the crater floor, and $\Delta h$ is the crater’s depth.

An estimate of the crater’s depth is carried out from Wang’s crater catalog [31], which provides the crater’s center and diameter, and auxiliary morphometric data, including depth. The value of $\overline{\rho }$ for each crater is estimated using a global LOLA-based DTM of the Moon with a resolution of 128 px/deg [32], corresponding to ∼237 m/px at the equator, applying a nearest-neighbor interpolation at the coordinates $(lo{n}_i,la{t}_i)$. Once the spherical coordinates of the crater are computed, its Cartesian coordinates are determined as

$$B_{{\mathit{r}}_{C,i}}={\rho }_i{\left[\begin{array}{c}cos\left(lat\right)cos\left(lon\right)\\ cos\left(lat\right)sin\left(lon\right)\\ sin\left(lat\right)\end{array}\right]}_i.$$

(9)

Accounting for the crater’s depth is crucial for obtaining an accurate projection of the crater centroid from the catalog, as the crater rim is offset from the Moon’s surface (Figure 6).

The effect of incorporating a crater’s depth into the projection process is illustrated in Figure 7. The left panel shows the reprojected crater rims onto a synthetic image of the lunar surface, captured by a wide-angle nadir-looking camera from an altitude of ∼100 km, without accounting for the crater’s depth. This results in a misalignment between the projected and observed crater rims. Conversely, when the depth is included (right panel of Figure 7), the discrepancies are mitigated, leading to projected crater rims that are fully consistent with the observed craters.

2.3. Crater Detection

Craters are key features for spacecraft navigation across challenging mission scenarios, including pinpoint landing [13,33,34], long-range surface exploration [35], and precise orbit determination of orbiting platforms [36,37]. A robust crater detection algorithm, capable of handling varying illumination conditions and viewing geometries, is essential for a feature-based navigation system. Several techniques have been proposed to perform crater detection on optical images, ranging from well-established computer vision methods (e.g, edge detection [38], image segmentation [39,40,41], Haar-like features [14]) to advanced artificial intelligence approaches (e.g, [34,42,43,44,45]). Among existing Convolutional Neural Network (CNN) architectures, single-stage detectors, such as SSD [46] and YOLO [47], are emerging as state-of-the-art methods for real-time object detection applications. Compared to single-stage detectors, two-stage models, such as Faster R-CNN [48], achieve more accurate bounding box regression by refining the Region-of-Interest (ROI) proposals and incorporating a feature enhancement step in the second stage. However, the improved detection performance comes at the cost of significantly higher computational complexity, resulting in slower inference times than one-stage architectures. A trade-off analysis was, thus, conducted between multiple techniques in terms of detection accuracy, computational efficiency, and memory usage.

During the preliminary stages of this study, in addition to neural network-based crater detectors, detection methods based on solid computer vision techniques, such as image segmentation with k-means clustering [39], were accounted for. After segmenting the input image, candidate craters are retrieved by fitting an ellipse to pairs of dark and bright patches of similar size, which are aligned to the Sun’s direction (Figure S1). Preliminary results indicate that the image segmentation-based approach enables crater detection across varying solar illumination angles while achieving faster processing times compared to network-based approaches. However, this approach yields fewer detections with higher noise levels, leading to increased uncertainties in crater centroid estimation (Figure S2). Even with optimized parameter tuning (e.g., the number of image clusters), AI-enabled approaches are expected to achieve superior detection accuracy. The ongoing development of space-graded high-performance computing platforms [49] and the implementation of strategies to mitigate radiation-induced errors on COTS instruments [17] further support the adoption of AI-aided crater detection, which has been successfully applied in previous studies for geological and morphological features mapping [43,50].

To better assess the performance of the proposed navigation toolchain, testing and validation campaigns were conducted to evaluate alternative object detection frameworks, including SSD (with a VGG16 backbone), Faster R-CNN (with a ResNet50 backbone), YOLOv5-extra-large, and YOLOv5-nano (https://github.com/ultralytics/yolov5 (accessed on 1 January 2025) networks. Compared with the smaller YOLOv5-nano model, the other networks demonstrated comparable or superior precision and recall, aligning with the findings of Parracino et al. [51] who evaluated SSD (with a MobileNetV2 backbone) and YOLOv7. However, on average, the YOLOv5-nano model exhibited the shortest inference times. Given these trade-offs, YOLOv5-nano was selected for this study and extensively analyzed for detection accuracy and robustness under different illumination conditions (Section 3).

Training

A supervised transfer learning approach was employed to fine-tune a pre-trained YOLOv5-nano model for crater detection. By leveraging the low-level features learned by the pre-trained network, transfer learning enables high detection accuracies even with a limited dataset while also accelerating the training process. Given an input image, the fine-tuned network extracts multiple crater instances as bounding boxes centered on candidate locations, each assigned a confidence score ranging between 0 and 1. Higher confidence scores indicate a greater likelihood that the detected instance corresponds to a crater.

In a supervised framework, the network is trained using a ground truth (GT) dataset of true positive detections. We, thus, created a GT dataset consisting of images and corresponding crater labels. This dataset includes over 1500 images, obtained by extracting small-sized image patches (i.e., 512 × 512 px) from a global mosaic of the lunar surface (i.e., 128 px/deg, equivalent to approximately ∼237 m/px at the equator) [32]. Due to the distortion effects of the mosaic’s map projection, which are more pronounced at higher latitudes, we restricted the extraction of image patches to mid-latitudes.

For each image in the dataset, we defined GT labels corresponding to the craters within the image patch (Figure 8). Each label is represented by a rectangle defined by a four-dimensional vector $(x_c,y_c,w,h)$, where $(x_c,y_c)$ is the rectangle’s center, and w and h are its width and height, respectively. The GT crater labels were derived from state-of-the-art lunar crater databases (e.g., Robbins’ [52] and Wang’s [31] catalogs). By extracting the crater’s diameter and center location $(lo{n}_c,la{t}_c)$ from these catalogs, the edges of the craters are represented as 2D ellipses in the map-projected image. Bounding boxes were then computed for each crater as the smallest rectangle that circumscribes the ellipse (see Appendix A); note that the sides of each rectangle are parallel to the image boundaries. The box parameters were normalized to a range between zero and one by dividing $x_c$ and w by the image width (in pixel), and $y_c$ and h by the image height (in pixel).

A key requirement for the crater detection subsystem is its robustness to varying observation geometries encountered during spacecraft operations. Since the patches extracted from the mosaic are characterized by nearly uniform illumination conditions (e.g., Sun-from-west; see Figure 8 and Figure 9A), we augmented the dataset by applying rotations to the image tiles to simulate different solar azimuth angles (Figure 9D). While open-source packages offer a range of data-augmentation techniques, including rotation transformations, directly applying these methods can lead to labeling issues; because the sides of the rectangles should be parallel to the image boundaries, bounding boxes in rotated images are defined as rectangles that pass through the vertices of the original bounding boxes, rather than being tangent to the crater rims (Figure 9F). This misalignment becomes critical when rotation angles approach $\pm {45}^{°}$. To address this issue, we implemented a custom data rotation algorithm designed to determine bounding boxes that are tangent to the crater edges (Figure 9E). This algorithm operates by first rotating the ellipses (and the image) rigidly and then computing the bounding boxes as rectangles that are tangent to the rotated elliptical contours, as illustrated in Figure 9.

To train the model, the augmented crater dataset was partitioned into training and validation sets by adopting a 75/25 split ratio, and the stochastic gradient descent (SGD) optimizer was used, with a batch size of 64 and a learning rate of $1\times {10}^{-2}$. After training, the fine-tuned YOLO-nano network achieved a precision of 0.64 and a recall of 0.54 on the validation set. Although enhanced detection performance can be achieved by adopting models with more parameters, our results indicate that the fine-tuned network is well suited to fully support navigation operations, demonstrating high robustness to the illumination conditions (see Section 3.1). Furthermore, its limited requirements in terms of memory consumption could make it suited for deployment on future COTS computers for space applications [16]. Nevertheless, in case improved detection metrics are required, as in the case where a more comprehensive global crater database has to be created, larger models can be adopted, such as the YOLO-extra-large model (for comparison purposes, the YOLO-extra-large and YOLO-nano models have more than 85 million and ∼2 million parameters, respectively) [43]. The enhanced detection performance of these models, however, comes at the cost of increased resource and memory consumption, which is not compatible with the constrained computational capabilities of onboard systems. To mitigate this limitation, the YOLO-nano model was selected minimizing the computational burden associated with crater detection, facilitating its deployment on future robotic platforms expected to feature expanded processing capabilities. Nevertheless, this network exhibits high efficiency and fast inference even on CPU-based platforms with limited computational resources [53].

Given an image as input to the trained network, bounding boxes are generated for each detected crater, with the center of the box assumed to correspond to the crater’s center. The crater diameter (in pixels) is then estimated as the average of the bounding box’s width and height, calculated as $d=(w+h)/2$. This parameter plays a critical role in the crater matching process (Section 2.4). Despite being trained on real lunar images only, the network is also capable of detecting craters in synthetic lunar images generated through the Blender-based pipeline (Section 3).

By improving the image contrast, enhanced crater detection performance is achieved by the network especially in poorly textured images. In our experiments, the CLAHE (Contrast Limited Adaptive Histogram Equalization) algorithm was, thus, applied to pre-process the images before feeding them into the YOLO-based network for crater extraction.

2.4. Crater Matching

Crater matching (or crater identification) is a fundamental task in defining crater-based navigation measurements. This process involves establishing correspondences between two distinct sets of craters: those detected in an image (i.e., “observed” craters) and those listed in an onboard database (i.e., “cataloged” craters). Crater matching is extremely challenging because the two sets may differ significantly. For instance, the detector might identify craters not present in the catalog or fail to detect known craters. Different approaches have been proposed in the literature to address this challenge, including methods based on mathematical invariants of multi-crater patterns (e.g., [41,54,55]), stochastic and RANSAC-based approaches (e.g., [39,40]), and machine learning-based techniques (e.g., [44]).

To address the crater matching task, an initial approach based on nearest-neighbor matching was considered. However, this method is prone to generate erroneous associations in dense crater fields, where a high number of detected/catalog-projected craters increased the likelihood of a noisy match, reducing the effectiveness of statistical filtering techniques. To improve robustness against incorrect data associations, a method based on the Joint Compatibility Branch and Bound (JCBB) technique was explored, as it iteratively identifies a mutually consistent set of correspondences [56]. However, its computational complexity limited its applicability to large datasets.

Given these challenges, the crater matching strategy was then implemented by using a triad-based approach (Figure 10), building upon previous work on feature-based OpNav techniques [55]. Originally conceived for spacecraft state initialization in a lost-in-low-lunar-orbit scenarios, the crater matching cost function was adapted to incorporate a priori spacecraft state information. A  statistical filtering step was introduced to perform a preliminary rejection of outlier matches. The complete crater matching pipeline follows a structured sequence of steps, which are outlined in Algorithm 1 and detailed in the following sections.

Algorithm 1 Crater Matching Algorithm

1: Input: Centroids and diameters of observed craters ($\mathit{z}$, $D^{obs}$), centroids and diameters of projected catalog craters ($\mathit{h}$, $D^{cat}$), positive values ${ϵ}_1$, ${ϵ}_2$, ${ϵ}_{px}$, ${ϵ}_{perc}$, ${\overline{\chi }}_{\alpha ,2}^2$   2: Output: Optical residuals, ${\mathit{ϵ}}_C$   3: $d^{obs}$ ← DefineTriads($\mathit{z}$, $D^{obs}$)                                                                            ▹$N^{obs}$ triads   4: ($d_{all}^{cat}$, $k_{vec}$) ← DefineTriads($\mathit{h}$, $D^{cat}$)                                            ▹$N^{cat}$ triads and k-vector   5: Create empty triad correspondence matrix $C_T$   6: for $i=1$ to $N^{obs}$ do                                                                     ▹ for each observed triad   7:       ${\alpha }_{S,i}$ ← $d_i^{obs}$                                                                     ▹ triad’s smallest internal angle   8:       $d_{sub}^{cat}$ ← SelectTriads($d_{all}^{cat}$, $k_{vec}$, ${\alpha }_{S,i}$, ${ϵ}_1$, ${ϵ}_2$)                        ▹ downselect catalog triads   9:       if $d_{sub}^{cat}$ is not empty then 10:             $E_i$, $\overline{j}$ ← FindBestTriadMatch($d_i^{obs},d_{sub}^{cat}$)                       ▹$E_i=E(i,\overline{j})$, with $\overline{j}\le N^{cat}$ 11:             Append $(i,\overline{j},E_i)$ to $C_T$ 12:       end if 13: end for 14: if $C_T$ is not empty then 15:       Initialize crater correspondence matrix C ← $C_T$ 16:       C ← SolveOneToMany(C, $C_T$) 17:       C ← DiameterFiltering(C, $D^{obs}$, $D^{cat}$, ${ϵ}_{px}$, ${ϵ}_{perc}$) 18:       C ← ChiSquareFiltering(C, $\mathit{z}$, $\mathit{h}$, ${\overline{\chi }}_{\alpha ,2}^2$) 19:       ${\mathit{ϵ}}_C$ ← ComputeResiduals(C, $\mathit{z}$, $\mathit{h}$) 20:       return ${\mathit{ϵ}}_C$ 21: end if

Correspondences between different triads are established by comparing their invariants, a set of scalar quantities that uniquely characterize a crater triad and can be used to evaluate the similarity between two triads. First, two sets of crater triads are defined: observed triads, based on craters detected by the network, and database triads based on the projected craters from the catalog (derived from the estimated spacecraft’s pose). A preliminary estimate of the spacecraft state is assumed to be available from the navigation filter, enabling a downselection of the onboard crater catalog. This filtering step excludes craters outside the field of view (FOV), reducing the computational load associated with generating database triads. However, due to the combinatorial nature of the problem, processing large datasets of detected/catalog projected crater can still require significant computational time.

For each observed and database triad, a 6-dimensional descriptor is computed by accounting for geometrical properties (Lines 3 and 4). This descriptor includes the cosines of the smallest and largest internal angles of the triangle, the normalized diameters of the three craters (i.e., normalized by the longest side of the triangle), and a parameter $\delta$ that represents the triad’s orientation [55]. The orientation of the triad is determined by moving across its vertices in order from the smallest angle to the intermediate angle, and then to the largest angle (i.e., $\delta =1$ if the triad has a counterclockwise orientation; $\delta =-1$ if it has a clockwise orientation). The invariant parameters related to the l-th triad are collected in the invariant array $d_l$, defined as

$$d_l=\begin{array}{cccccc}[cos({\alpha }_S),& cos\left({\alpha }_L\right),& {\overline{D}}_i,& {\overline{D}}_j,& {\overline{D}}_k,& \delta \end{array}],$$

(10)

where i, j, and k refer to the indices of the craters in the triad (Figure 10), ${\overline{D}}_i$, ${\overline{D}}_j$, and ${\overline{D}}_k$ are their normalized diameters, and ${\alpha }_S$ and ${\alpha }_L$ are the smallest and largest internal angles, respectively. A detailed discussion on the steps required to compute the descriptor for a given triad is provided in Appendix B.

For each observed triad, we identify the corresponding database triad that minimizes the cost function E, which is defined as

$$E(m,\overline{n})=\mathrm{SAD}(d_m^{obs},d_{\overline{n}}^{cat})+\epsilon {\left(\sum _{i=1}^3\left[{(x_{m,i}-x_{\overline{n},i})}^2+{(y_{m,i}-y_{\overline{n},i})}^2\right]\right)}^{1/2},$$

(11)

where SAD is the sum of absolute differences; $d_m^{obs}$ and $d_{\overline{n}}^{cat}$ are the descriptors of the m-th observed triad and the $\overline{n}$-th database triad, respectively; and $(x_{m,i},y_{m,i})$ and $(x_{\overline{n},i},y_{\overline{n},i})$ are the centroids of the i-th crater (i = 1, 2, 3) in the observed and candidate database triads, respectively. The second term in the cost function accounts for minimizing the relative distance between corresponding craters. However, this term is scaled by a small positive coefficient $\epsilon$ (e.g., $\epsilon =0.003$), giving greater weight to the similarity of the triad invariants in the overall matching metric.

Calculating the cost for each database triad for a given observed triad is computationally expensive. To speed up this process, a downselection of the database triads is performed based on the smallest internal angle of the observed triad (${\alpha }_S$), retaining only triads whose smallest internal angle (${\tilde{\alpha }}_S$) is sufficiently close to ${\alpha }_S$ using the following criterion:

$$c_{min}<cos\left({\tilde{\alpha }}_S\right)<c_{max}$$

(12)

where $c_{min}=cos\left({\alpha }_S\right)-{ϵ}_1$ and $c_{max}=cos\left({\alpha }_S\right)+{ϵ}_2$, with ${ϵ}_1,{ϵ}_2>0$. The downselection is optimized using the k-vector range searching technique [57] (Line 8), which sorts the database triads by their smallest internal angle for faster processing.

The output of the crater matching step is a set of candidate correspondences between craters detected by the network and those projected from the catalog onto the image plane. Ambiguous matches are filtered out based on their triad matching score (Line 16). For example, if a catalog crater in the $\overline{i}$-th database triad is paired with three observed craters from different observed triads $l,m,n$ with matching costs $E(l,\overline{i})>E(m,\overline{i})>E(n,\overline{i})$, the catalog crater will be paired with the crater from triad n.

After completing this step, a set of one-to-one crater correspondences is obtained, but mismatches may still occur. A two-steps filtering approach is then applied to filter them out. First, crater pairs with diameter differences greater than ${ϵ}_{perc}$ = 25% (or an absolute difference exceeding ${ϵ}_{px}$ = 5 px) are eliminated (Line 17). Next, statistical filtering is carried out based on the distribution of 2D optical residuals (Line 18). Given a crater match i, the corresponding residual ${\mathit{ϵ}}_i$ is computed as the difference between the image location of the i-th detected crater ${\mathit{z}}_i={[x_i,y_i]}^{\mathrm{T}}$ and the corresponding catalog-projected crater ${\mathit{h}}_i={\mathit{p}}_i={[{\overline{x}}_i,{\overline{y}}_i]}^{\mathrm{T}}$ (see Equation (6)). The Mahalanobis distance is computed between the residual point (${\mathit{ϵ}}_{C,i}={\mathit{z}}_i-{\mathit{h}}_i$) and the median of the distribution ($\mathit{\mu }$), as

$${({\mathit{ϵ}}_{C,i}-\mathit{\mu })}^T{\mathbf{\Sigma }}^{-1}({\mathit{ϵ}}_{C,i}-\mathit{\mu })$$

(13)

where $\mathbf{\Sigma }$ is the covariance matrix of the 2D residuals. If the computed distance exceeds a preset threshold value ${\overline{\chi }}_{\alpha ,2}^2$ based on the chi-square distribution with two degrees of freedom (DOF) (e.g., ${\overline{\chi }}_{0.9,2}^2\approx 4.605$, corresponding to a $90\%$ confidence interval for a chi-square distribution with two DOF; note that two is the number of DOF of the measurement model), the crater pair is eliminated as an outlier. The median is used instead of the mean to reduce the influence of outliers. The filtering process may be repeated until no more outliers are identified.

Figure 11 shows the statistical filtering process applied to crater matches in an example image captured by the navigation camera. Matched crater pairs are reported in the left panel, with catalog-projected craters represented as ellipses and detected craters as boxes. The right panel presents the 2D distribution of optical residuals, where crater pairs failing the Mahalanobis-distance test are marked in red and rejected, while those passing the test (in blue) are incorporated into the navigation filter (see Section 4).

3. Software Pipeline Testing Campaign

The computation of the optical data is affected by different error sources, primarily stemming from the detection performance of the neural network. Additional errors arise from the crater catalog itself, including inaccuracies in crater depth estimation. In a multi-sensor navigation system, each data type must be weighted according to its expected measurement accuracy [19]. Thus, a thorough characterization of the crater detection accuracy is key for fine-tuning the onboard navigation filter.

3.1. Robustness Analysis of Machine Vision Techniques

To evaluate the robustness of crater detection and matching algorithms under varying conditions (e.g., solar azimuth and elevation angles), a series of thorough testing campaigns were conducted using synthetic datasets generated through our custom Blender-based toolbox.

Multiple sets of rendered images were created by using consistent camera poses simulating a spacecraft in a low-altitude lunar orbit equipped with a nadir-looking wide-angle camera (WAC). Each dataset featured different solar illumination conditions, adjusted by modifying the direction of solar rays in the simulated environment (Figure 12). The crater detection and matching algorithms were then independently applied to each dataset, allowing for the calculation of the optical residuals (i.e., pixel discrepancies between the centroids of matched craters). Ideally, a robust crater detector should remain unaffected by changes in image illumination, leading to optical residuals following a zero-mean Gaussian distribution.

An initial analysis was carried out using a preliminary version of the crater detector, trained on a ground truth dataset including only horizontally flipped tiles (i.e., Sun-from-west and Sun-from-east illumination conditions) and tiles rotated by ${90}^{°}$ clockwise or counterclockwise (i.e., Sun-from-south and Sun-from-north illumination conditions). Craters were first extracted from images whose illumination matched with the training dataset (e.g., left panel in Figure 12). A zero-transfer experiment was conducted to evaluate the network’s generalization capability under “unknown” illumination conditions not included in the training dataset. This involved extracting craters from synthetic datasets with solar azimuth angles outside the training set (i.e., Sun-from-southwest, Sun-from-southeast, Sun-from-northeast, and Sun-from-northwest azimuth angles).

Figure 13 presents the statistical distribution of optical residuals for synthetic datasets that account for a fixed solar elevation angle of ${15}^{°}$ (top row of Figure 12) under various assumptions regarding the solar azimuth. As expected, the optical residuals for datasets with known illumination conditions (e.g., Sun-from-west, Figure 13(left)) exhibited an unbiased distribution. However, for images with “unknown” illumination conditions (center and right panels in Figure 13), crater matching performance significantly decreased. The optical residual distributions revealed biases of approximately half a pixel, with a strong correlation between horizontal and vertical directions. These results underscore the preliminary detector’s limitations in accurately determining crater bounding boxes due to variations in crater shadow orientation and shape that were not accounted for during training.

To compensate for the crater detector’s limitations under previously unseen illumination conditions, the network was enhanced through fine-tuning of the pre-trained YOLO model. This improvement was achieved by training on an augmented ground truth dataset that included a finer sampling of solar azimuth and elevation angles. Specifically, the dataset incorporated rotated images with a more detailed discretization of rotation angles (see Section 2.3). By expanding the range of training data, the re-trained network demonstrated significantly improved performance, particularly in its robustness to varying illumination conditions. For instance, Figure 14 and Figure 15 show the 2D distribution of the optical residuals for three example datasets with different solar azimuth angles (i.e., from left to right, Sun-from-west, Sub-from-southwest, and Sun-from-southeast) and a ${15}^{°}$ (Figure 14) and $22.5^{°}$ (Figure 15) elevation angle, respectively (see Figure 12). The updated network achieved smaller biases and reduced correlations between residuals in the horizontal and vertical directions for each dataset, with 1 − $\sigma$ formal uncertainties of about two pixels in both dimensions. This improvement highlights the detector’s enhanced ability to accurately extract crater locations from images regardless of illumination conditions.

This refined training process enabled more accurate detection of crater locations, reducing biases and correlations in the optical residuals. Consequently, the updated detector was used in numerical simulations of the landing mission scenario (Section 4).

3.2. Impact of Surface Landmark Mismodeling on Crater Matching Performance

An accurate estimation of the crater depth is key to ensure that the projected crater centroids align with those detected by the network. To better understand the effects of mismodeled crater coordinates on the landmark matching algorithm, we conducted experiments using the crater matching pipeline without accounting for the crater depth information (see Equation (8) and Figure 6). Figure 16 shows the 2D distribution of the optical residuals for the same datasets as in Figure 15, but without considering crater depth in the onboard database definition. A direct comparison of the plots between the two figures reveals that, although the distribution median and the number of matched craters remain largely unchanged, the residuals exhibit a significantly greater spread. This suggests that accounting for the crater depth is essential to avoid projection errors of crater-based landmarks onto the image plane (e.g., Figure 7). Accurate depth modeling enables precise computation of measurement residuals and improves the spacecraft trajectory estimation through the navigation filter.

These analyses provided a quantitative assessment of the effects of crater mismodeling on the matching process. However, specific values for the optical discrepancies vary depending on several factors, including camera characteristics (e.g., field-of-view, spatial resolution, calibration) and spacecraft pose (e.g., higher altitudes with respect to the surface reduce the impact of crater mismodeling). Additional error sources include the methods used for determining crater center locations in the catalog (e.g., manual labeling or AI-based techniques) and the quality of input data sources (e.g., surface mosaic, 3D topography models).

An accurate and comprehensive onboard crater database is crucial for supporting navigation operations. However, existing catalogs of craters smaller than 1 km are limited to specific regions only, such as the lunar South Pole [58]. Since onboard cameras during landing observe small-scale craters not included in global catalogs, the SLIM mission navigation team created and stored onboard a comprehensive database of craters expected to be visible to the spacecraft. Similarly, we created a custom dataset of small-scale craters to support optical navigation in simulated mission scenarios, leveraging open-source data and tools.

The first step involved importing the LRO WAC global lunar mosaic [32] into QGIS [59]. Using the OpenCraterTool plugin [60], we visualized the crater footprints based on Robbins’ data (Figure 17, light blue) and manually labeled small-scale craters not included in this catalog (Figure 17, orange). Our analysis focused on the predicted ground track of the SLIM spacecraft (i.e., ${24}^{°}\le lon\le {36}^{°}$, $-{70}^{°}\le lat\le -{11}^{°}$), resulting in an auxiliary dataset of more than 500 craters.

The creation of a custom catalog of small craters from the LRO WAC global mosaic introduced a significant challenge in ensuring consistency with the high-resolution DTM used to generate high-quality synthetic images for a landing scenario. For these synthetic images, we employed the SLDEM2013 model, derived from Kaguya’s Terrain Camera (TC) data, which are archived as $1^{°}\times 1^{°}$ tiles in the SELENE Data Archive (https://data.darts.isas.jaxa.jp/pub/pds3/sln-l-tc-5-sldem2013-v1.0, accessed on 1 January 2025). When projecting these newly cataloged craters onto synthetic images generated based on the simulated trajectory of the SLIM spacecraft, substantial discrepancies emerged between the manually labeled craters and the network’s detections.

The SLDEM2013 model was georeferenced using Lunar Orbiter Laser Altimeter (LOLA) data processed with LRO trajectories reconstructed prior to the significant improvements in lunar gravity modeling provided by the Gravity Recovery and Interior Laboratory (GRAIL) mission [61]. In contrast, the LRO WAC global mosaic was produced using more accurate LRO orbits, determined with a higher-resolution lunar gravity field.

These georeferencing inconsistencies between the LRO WAC mosaic and the SLDEM2013 model were quantified through the analysis of multiple synthetic images. Figure 18 shows that the average discrepancies between crater centers from the LRO WAC mosaic and the SLDEM2013 model were measured to be approximately $0.{004}^{°}$ in longitude and $-0.{003}^{°}$ in latitude (Figure 19). These offsets necessitated adjustments to the coordinates of the manually labeled craters in our catalog to align them with the SLDEM2013 model and improve the accuracy of the crater matching process in the simulated landing scenario.

3.3. Inference Times

A primary challenge constraining the widespread adoption of AI-enabled techniques in space applications is the limited computational capacity of radiation-certified processors (e.g., BAE RAD750). Accurately estimating the network’s inference times requires hardware-in-the-loop (HITL) testing on state-of-the-art resource-constrained computing platforms. However, preliminary assessments can be performed on higher-performance computers by artificially restricting computational resources through software-based limitations.

To facilitate these campaigns, the Ubuntu scheduler was configured to enforce execution of the inference task on a single 2.5 GHz core of a modern laptop CPU. Additionally, CPU usage was restricted to 8% of the total processing time, effectively simulating a 200 MHz clock. Under these conditions, approximately 350 images were processed, yielding an average inferring time of 360 ms per image. This test provides a preliminary but representative estimate of the inference time, suggesting that the algorithm could support real-time navigation operations. In-depth analyses are planned for the development and implementation of a semi-autonomous Guidance, Navigation, and Control (GNC) subsystem for orbiting and surface space probes.

4. Numerical Simulations on a Landing Test Case

To validate and test the optical navigation pipeline, a landing test case was considered that is based on the JAXA SLIM mission. Landed in January 2024 close to the Shioli crater, the mission demonstrated the successful use of a vision-based pipeline to perform a pinpoint landing on the Moon, achieving an unprecedented accuracy of 100 m with respect to the targeted landing site [13]. A multi-crater pattern matching technique was used to establish crater correspondences between a pre-labeled reference crater map and the craters detected in the camera images by using a dedicated crater extraction pipeline [14], enabling an accurate localization of the lander throughout the descent phase.

A similar approach was used in this work to retrieve sequential adjustments to the spacecraft’s state by minimizing the optical residuals, which are based on correspondences between the craters detected in the image by a neural network (Section 2.3) and those projected from an onboard catalog on the image plane according to the estimated spacecraft’s pose (Section 2.2).

As a first step, synthetic images were generated by using a reference spacecraft’s trajectory to simulate the images taken by the onboard nadir-looking navigation camera during the descent phase. The characteristics of our navigation camera (NavCam) are based on SLIM’s NavCam (i.e., the detector size, shutter type, and image acquisition rate) [62], but with a wider FOV of ${45}^{°}\times {45}^{°}$ compared to SLIM’s NavCam (${30}^{°}\times {30}^{°}$). While initial tests were conducted with a narrower FOV, a wider FOV was ultimately selected to increase the number of cataloged craters visible at lower altitudes, thereby enhancing the navigation system’s performance.

A reference trajectory for the SLIM spacecraft was retrieved from the NASA Horizons web application (https://ssd.jpl.nasa.gov/horizons/, accessed on 1 January 2025) in the form of tabulated spacecraft’s position and velocity in the Inertial Celestial Reference Frame (ICRF). In our numerical simulations, a mission time span of about 25 min was considered, during which the spacecraft’s altitude decreased from ∼60 to ∼10 km (

Table 1. Initial and final mission epochs, with corresponding spacecraft coordinates and altitude above the lunar surface.

Parameter	Initial Epoch	Final Epoch
Epoch (UTC)	19 January 2024 14:50:00	19 January 2024 15:15:00
Longitude	25.4°	25.3°
Latitude	−74.4°	−13.4°
Altitude	61.0 km	9.5 km

).

The analysis of the reference trajectory for the SLIM spacecraft revealed inconsistencies between the tabulated spacecraft velocity v and the velocity computed by numerically differentiating the tabulated spacecraft position ($\tilde{\mathit{v}}=\Delta \mathit{r}/\Delta t$). An updated position profile was then retrieved by integrating the spacecraft kinematic equations backward in time according to the tabulated velocity profile.

The updated trajectory provides the ground truth for simulating data collected by the onboard sensors. Synthetic images were generated based on the integrated trajectory by assuming a nadir-pointed camera with the z-axis pointing towards the lunar surface. The orientation of the camera frame with respect to the inertial frame was retrieved according to:

$$\begin{array}{c}{\mathbf{R}}_I^C\left(t\right)=\left[\begin{array}{c}-\widehat{\mathit{n}}\\ -\widehat{\mathit{t}}\\ -\widehat{\mathit{r}}\end{array}\right]\end{array}$$

(14)

where $\widehat{\mathit{r}}=\mathit{r}/\left|\right|\mathit{r}\left|\right|$, $\widehat{\mathit{n}}=(\mathit{r}\times \mathit{v})/\left|\right|(\mathit{r}\times \mathit{v})\left|\right|$, and $\widehat{\mathit{t}}=\widehat{\mathit{n}}\times \widehat{\mathit{r}}$ are the radial, normal, and transverse unit vectors, respectively, which are computed according to the inertial spacecraft’s state.

An accurate modeling of the spacecraft dynamics is crucial to propagate the spacecraft’s state between successive image acquisitions. Because the reference SLIM’s trajectory is based on a time-varying thrust acceleration profile, its contribution should be modeled in the dynamical equations. The thrust acceleration is included by simulating the readings of an onboard accelerometer that is part of an inertial measurement unit (IMU). A true acceleration profile (in the inertial frame) was first computed by numerical differentiating the spacecraft’s velocity. After removing the contribution due to conservative forces from the computed acceleration profile, the instrument readings were simulated by corrupting the nominal acceleration data with zero-mean Gaussian noise ($n_{VRW}$) and a time-varying bias (${\mathit{b}}_a$) modeled as a random walk process (

Table 2. Onboard sensor characteristics.

Instrument	Parameter	Value
Camera	Field of view	${45}^{°}\times {45}^{°}$
	Detector size	$512\times 512$ px
	Ground sampling distance	∼16 m (10 km height)
	Acquisition rate	0.01 Hz (10 s)
Accelerometer	Velocity Random Walk (${\sigma }_{VRW}$)	$4.9\times {10}^{-4}$ m/s2/$\sqrt{\mathrm{Hz}}$
	Acceleration Random Walk (${\sigma }_{AccRW}$)	$4.9\times {10}^{-5}$ m/s3/$\sqrt{\mathrm{Hz}}$
	Acquisition rate	100 Hz (0.01 s)

) [63]. As a final step, a rotation was applied to express the computed acceleration measurements in the IMU frame, which was assumed to coincide with the camera frame.

The propagation of the state equations was carried out according to:

$$\dot{\mathit{x}}=\left[\begin{array}{c}\dot{\mathit{r}}\\ \dot{\mathit{v}}\\ {\dot{\mathit{b}}}_a\end{array}\right]=\left[\begin{array}{c}\mathit{v}\\ {\mathit{a}}_G+{\left({\mathbf{R}}_I^{SB}\right)}^T{\mathit{a}}_{IMU}\\ 0_{3\times 1}\end{array}\right]$$

(15)

where ${\mathit{a}}_G$ is the acceleration due to gravitational forces (in the inertial frame), ${\mathit{a}}_{IMU}$ is the thrust acceleration measured by the accelerometer, assuming that the non-conservative forces, including solar radiation pressure, are negligible, and ${\mathbf{R}}_I^{SB}$ is the spacecraft attitude matrix (i.e., rotation matrix from the inertial frame to the spacecraft body frame, which is assumed to coincide with the IMU and camera frames). The state equations are integrated with a constant time step of $\Delta t=0.01$ s by using a first-order numerical model, which propagates the estimated state $\widehat{\mathit{x}}$ from time step k to time step $k+1$. In this work, the spacecraft’s attitude is not adjusted in the navigation filter that only accounts for the spacecraft’s position and velocity. Attitude determination errors are accounted for in the integration of the state equations by perturbing the true spacecraft attitude based on the expected performances of the onboard attitude determination system (ADS). A perturbed attitude matrix ${\overline{\mathbf{R}}}_I^{SB}$ is then computed as:

$${\overline{\mathbf{R}}}_I^{SB}={\mathbf{R}}_{err}{\mathbf{R}}_I^{SB}$$

(16)

where ${\mathbf{R}}_{err}={\mathbf{R}}_x\left(\delta {\theta }_x\right){\mathbf{R}}_y\left(\delta {\theta }_y\right){\mathbf{R}}_z\left(\delta {\theta }_z\right)$ is a sequence of three elementary rotations, with $\delta {\theta }_x$, $\delta {\theta }_y$ and $\delta {\theta }_z$ representing angular errors on the pitch, roll, and yaw axes, respectively, which are computed as zero-mean Gaussian variables with variance ${\sigma }_{\theta }=0.{05}^{°}$.

The state covariance matrix ${\widehat{\mathbf{P}}}_k$ is propagated according to:

$${\overline{\mathbf{P}}}_{k+1}=\mathbf{\Phi }(t_{k+1},t_k){\widehat{\mathbf{P}}}_k{\mathbf{\Phi }}^T(t_{k+1},t_k)+{\mathbf{Q}}_d$$

(17)

where $\mathbf{\Phi }(t_{k+1},t_k)$ is the state transition matrix (STM) and ${\mathbf{Q}}_d$ is the discrete-time process noise covariance matrix. A first-order Taylor series expansion is used to compute the state transition matrix $\mathbf{\Phi }(t_{k+1},t_k)$ from the $(9\times 9)$ discrete-time system matrix ${\mathbf{F}}_k={\left[\partial \dot{\mathit{x}}/\partial \mathit{x}\right]}_k$ [64]. A first-order model is also used to compute the matrix ${\mathbf{Q}}_d$ as

$${\mathbf{Q}}_d\approx \mathbf{G}({\mathbf{Q}}_c\Delta t){\mathbf{G}}^T$$

(18)

where $\mathbf{G}={\left[\partial \dot{\mathit{x}}/\partial {\mathit{n}}_w\right]}_k$ is the $(9\times 6)$ discrete-time noise mapping matrix [64]; and ${\mathbf{Q}}_c$ is the $(6\times 6)$ continuous-time system noise covariance matrix defined as

$${\mathbf{Q}}_c=\left[\begin{array}{cc}{\sigma }_{VRW}^2{\mathbf{I}}_{3\times 3}& {\mathbf{0}}_{3\times 3}\\ {\mathbf{0}}_{3\times 3}& {\sigma }_{AccRW}^2{\mathbf{I}}_{3\times 3}\end{array}\right].$$

(19)

At each new image acquisition, an attempt is made to adjust the spacecraft’s state based on the crater matches. The statistical filtering process based on the 2D distribution of the optical residuals enables a preliminary rejection of outlier crater pairs (Section 2.3). However, the effectiveness of the filtering process depends on the number of available crater correspondences. The greater the number of crater matches, the more effective the outlier filtering process, as it enables a more robust estimation of the distribution’s covariance. For instance, when crater correspondences are limited and include outliers, the filtering process may fail to reject any crater pairs, potentially allowing some outliers to remain unfiltered. An additional Mahalanobis distance-based test (i.e., the Normalized Innovation Squared (NIS) test) is, thus, conducted for each crater pair, rejecting matches that do not satisfy the following condition [6,64,65]:

$${({\mathit{z}}_i-{\mathit{h}}_i)}^T{({\mathbf{H}}_i{\overline{\mathbf{P}}}_{k+1}{\mathbf{H}}_i^T+{\mathbf{R}}_i)}^{-1}({\mathit{z}}_i-{\mathit{h}}_i)<{\overline{\chi }}_{\alpha ,2}^2$$

(20)

where ${\mathbf{H}}_i$ is the Jacobian matrix of the i-th optical data, which collects the partial derivatives of the observable ${\mathit{h}}_i$ with respect to the state variables; ${\mathbf{R}}_i={\sigma }_{px}^2{\mathbf{I}}_{2\times 2}$ is the measurement noise matrix (note that, because crater mismodeling effects tend to increase at lower altitudes, an inflated uncertainty ${\sigma }_{px}=3\mathrm{px}$ was considered in the implementation of the navigation filter); and ${\overline{\chi }}_{\alpha ,2}^2$ is a preset threshold based on the probability density function of the chi-square distribution with two DOF (e.g., $\alpha =0.9$).

The filtering process leads to a subset of L crater correspondences that are accounted for in the state estimation filter. The (2L × 1) optical residual vector (${\mathit{ϵ}}_C$), the (2L × 9) Jacobian matrix (H) and the (2L × 2L) block-diagonal measurement noise matrix (R) are computed by incorporating the contribution of each inlier crater pair. The propagated state vector (${\overline{\mathit{x}}}_{k+1}$) and covariance matrix (${\overline{\mathbf{P}}}_{k+1}$) are then updated according to the Extended Kalman Filter (EFK) equations, as:

$${\widehat{\mathit{x}}}_{k+1}={\overline{\mathit{x}}}_{k+1}+\mathbf{K}{\mathit{ϵ}}_C$$

(21)

$${\widehat{\mathbf{P}}}_{k+1}=({\mathbf{I}}_{9\times 9}-\mathbf{K}\mathbf{H}){\overline{\mathbf{P}}}_{k+1}{({\mathbf{I}}_{9\times 9}-\mathbf{K}\mathbf{H})}^T+\mathbf{K}\mathbf{R}{\mathbf{K}}^T$$

(22)

where $\mathbf{K}={\overline{\mathbf{P}}}_{k+1}{\mathbf{H}}^T{(\mathbf{H}{\overline{\mathbf{P}}}_{k+1}{\mathbf{H}}^T+\mathbf{R})}^{-1}$ is the Kalman gain matrix.

An a priori estimate of the spacecraft’s state is assumed to be available that is based on a ground-based orbit determination (OD) solution. To simulate errors in the OD process, the initial spacecraft’s state is perturbed by adding a zero-mean Gaussian noise to each component of the position (${\sigma }_P$ = 300 m) and velocity (${\sigma }_V$ = 3 m/s) vectors. The initial covariance matrix ${\mathbf{P}}_0$ is defined accordingly.

5. Discussion

In this section, we discuss the performance of the crater-based navigation algorithm through numerical simulations based on the landing profile of the SLIM mission. A Monte Carlo analysis was also conducted to investigate the robustness of the navigation system to the initial state estimation errors.

Figure 20 and Figure 21 show the position and velocity errors and uncertainties along the cross-track (left), along-track (center), and radial (right) directions for one of the Monte Carlo runs. The dots represent image acquisitions by the onboard camera. Red dots indicate trajectory adjustments based on crater matches, which require at least three pairs of matched craters. Blue dots indicate cases where fewer than three crater correspondences were established; in these instances, the information was discarded to avoid large and erroneous position adjustments. The position and velocity estimation errors are fully consistent with the 3 − $\sigma$ formal uncertainties (orange). As expected, a refined reconstruction of the spacecraft trajectory was retrieved along the cross- and along-track directions, which define the horizontal plane, compared to the radial direction that is aligned with the camera boresight.

A more uniform distribution of matched craters across the image plane would improve constraints on the spacecraft’s position along the radial direction. At lower altitudes, however, crater matches tended to cluster in specific areas of the image, resulting in optical residuals that were primarily minimized by adjusting the spacecraft’s position in the horizontal plane. The numerical simulations indicate that the implemented crater-based navigation pipeline supports the accurate reconstruction of the spacecraft’s state at altitudes as low as 10 km. Below this altitude, no crater matches were detected, resulting in state propagation relying solely on inertial data. The system achieved final position errors consistent with 3 − $\sigma$ uncertainties, with ∼85 m on the horizontal plane and 250 m along the radial direction. Consistent results were obtained for the spacecraft’s velocity, with final errors aligning with the 3 − $\sigma$ formal uncertainties, ∼0.8 m/s on the horizontal plane and 1.0 m/s in the radial direction.

The prolonged lack of matched craters occurring towards the end of the simulated descend phase led to no optical-based corrections by the navigation filter. As a result, the spacecraft’s state and covariance were propagated only based on inertial measurements, leading to increasing state uncertainties and errors over time. When new crater correspondences were identified, state adjustments were computed, allowing the estimated trajectory to converge toward the true trajectory. The lack of crater matches occurs because the spacecraft flies over areas that are nearly crater-free or contain cataloged craters that do not align with the network’s detections. For instance, Figure 22 shows the predicted craters from the Robbins’ catalog for an image in which no craters were detected by the network. Although multiple craters were expected, most were either very shallow or partially buried (e.g., Figure 8b in [60]), preventing robust feature detection and identification for navigation. The CNN in the proposed framework was trained on labeled craters with diameters above a certain threshold, meaning the small craters present in the image fell below the detection capability of the network. This limitation underscores the need for an enhanced database of surface features, including accurate custom crater catalogs, to support reliable navigation throughout the descent and landing phases. A dedicated crater mapping survey using high-resolution orbital images would allow for detailed mapping of small-scale craters [66], supporting continuous crater-based trajectory updates down to altitudes of a few hundred meters. An alternative optical navigation approach could then be adopted until touchdown based on mapped point-like features (e.g., keypoints [6,64]) or visual odometry-based techniques [67].

To assess the robustness of the crater-based navigation system against initial state estimation errors, a 100-run Monte Carlo analysis was performed, with position and velocity errors initialized using zero-mean Gaussian noise. Figure 23 and Figure 24 show the evolution of position and velocity estimation errors across the three axes, along with the average 3 − $\sigma$ formal uncertainties (in red). The results demonstrate that the vision-based navigation system remained robust to initial state estimation errors, achieving final RMS dispersions, at an altitude of ∼10 km: below 65 m in horizontal position, 110 m in radial position, and 0.6 m/s in velocity.

Though the final approach and landing stages are not directly addressed in this work, our results demonstrate that the crater-based navigation system achieved trajectory accuracies well suited to support critical navigation operations throughout the descent phase, providing a reliable foundation for precise landing.

6. Conclusions

A crater-based navigation pipeline was presented in this work that leverages a joint processing of inertial and imaging data through a sequential filter to retrieve an accurate reconstruction of the descent trajectory of a planetary landing platform. Adjustments to the spacecraft’s state are computed by establishing matches between the craters listed in an onboard reference database and those detected in the camera images through a neural network devoted to the crater detection task.

A detailed description of a custom image generation tool used to retrieve high-fidelity images of planetary surfaces was provided, including an efficient approach to convert planetary DTMs into 3D meshes that can be imported in open-source rendering software. A thorough discussion of the training operations of the crater detector was then provided, including data-augmentation and labeling techniques to retrieve an extensive ground truth dataset. Comprehensive analyses were conducted to assess the robustness of the crater detector under varying illumination conditions, such as the number of detected craters and bounding box location, and to investigate the impact of catalog errors on the crater matching pipeline. By analyzing the distribution of the optical residuals, we retrieved a quantitative estimation of the optical data accuracies, which is a key parameter in the framework of a multi-sensor navigation system.

To assess the performances of the proposed crater-based optical navigation (OpNav) system in a representative mission scenario, we conducted numerical simulations by accounting for a lunar lander based on the JAXA SLIM mission. Simulated camera acquisitions from an onboard down-looking camera were retrieved by using the Blender-based image generation tool, and then processed through the implemented navigation pipeline to estimate the spacecraft’s trajectory during the descent operations. A Monte Carlo analysis was conducted to thoroughly assess the performances of the navigation system, revealing its robustness to the initial state estimation errors. As expected, a refined reconstruction of the spacecraft’s orbital state was achieved in the cross- and along-track directions, which define the image plane parallel to the horizontal plane, compared to the radial direction aligned with the camera boresight. At an altitude of approximately 10 km, the 3 − $\sigma$ uncertainties were around 85 m and 250 m for horizontal and radial positions, respectively, and approximately 0.8 m/s and 1.0 m/s for horizontal and radial velocities, respectively.

The achieved state estimation accuracies indicate that the proposed navigation scheme, based on existing crater catalogs, is well suited to support navigation operations during the descent phase down to altitudes of approximately 10 km. At lower altitudes, the absence of cataloged craters caused the crater matching step to fail. By adopting a more comprehensive crater catalog, which also accounts for sub-kilometer-scale craters, robust navigation performances could also be achieved during the final approach and landing phases.

Future developments will focus on incorporating attitude estimation into the navigation filter by processing gyroscope and star tracker data, as well as exploring alternative network architectures, such as instance segmentation networks, to enable crater detection under off-nadir observation geometries. Quantization, pruning, and knowledge distillation techniques will be also explored to optimize the crater detection networks. Auxiliary frame-to-frame optical measurements will be incorporated into the navigation filter to improve trajectory propagation when no image-to-database crater correspondences are available. To reduce computational costs when detecting a large number of craters, various matching techniques will be explored, including RANSAC-like methods, multi-crater patterns such as crater polygons, and graph-based techniques. Additionally, a simulation environment will be developed using the Robot Operating System (ROS) to prepare for the real-time implementation of the navigation filter on space-qualified hardware.