Spacecraft and Asteroid Thermal Image Generation for Proximity Navigation and Detection Scenarios

Abstract

On-orbit autonomous relative navigation performance strongly depends on both sensor suite and state reconstruction selection. Whenever that suite relies on image-based sensors working in the visible spectral band, the illumination conditions strongly affect the accuracy and robustness of the state reconstruction outputs. To cope with that limitation, we investigate the effectiveness of exploiting image sensors active in the IR spectral band, not limited by the lighting conditions. To run effective and comprehensive testing and validation campaigns on navigation algorithms, a large dataset of images is required, either available or easy to obtain in the visible band, not trivial and not accessible for the thermal band. The paper presents an open-source tool that exploits accurate finite volume thermal models of celestial objects and artificial satellites to create thermal images based on the camera dynamic. The thermal model relies on open CFD code (OpenFOAM), pushed to catch the finest details of the terrain or of the target geometries, and then the temperature field is processed to compute the view factors between the camera and each face of the mesh; thus, the radiative flux emitted by each face is extracted. Such data feed the rendering engine (Blender) that, together with the camera position and attitude, outputs the thermal image. The complete pipeline, fed by the orbiting target and the imaging sensor kinematic, outputs a proper synthetic thermal image dataset, exploitable either by a relative navigation block or any other scope of research. Furthermore, in the same framework, the article proposes two different thermal sensor models but any sensor model can be applied, providing full customization of the output. The tool performance is critically discussed and applied for two typical proximity scenarios, asteroid and artificial satellite; for both cases, the challenges and capabilities of the implemented tool for synthetic thermal images are highlighted. In the end, the tool is applied in a phase B mission design sponsored by ESA and in related research works; for such cases, the results are reported in the article.

1. Introduction

Thermal images have an advantage over optical images since no light sources are needed. This has a tremendous impact on the navigation reliability of a space mission, enabling the spacecraft to see the dark regions of planets and asteroids or detect other spacecraft in the dark.

As an example, operating around an asteroid during proximity phases requires continuous pointing and navigation even during night phases. Thermal cameras can guarantee the detectability of the shape of the asteroid in all conditions. Furthermore, the biggest advantage of thermal images over optical ones is that they enable the detection of objects in the deep sky more easily despite the lower instrument resolution [1,2]. Indeed, optical cameras have more pixels which translate into bigger spatial resolution and lower detection limits for the size of the objects, but optical images capture all the star lights with a resulting image full of bright spots other than the target. By looking at real thermal images for the detection of asteroids and planets, the target is the only bright spot in the image, making the detection much easier for navigation purposes when the images are processed in a real-time closed loop [3].

These are just the two biggest advantages of thermal images; many other applications are made possible thanks to thermal infrared (TIR) images such as site selection in dark regions, deployable satellite tracking and navigation, and target marking [1]. Indeed, there is novel literature on the application of thermal infrared images for navigation in adverse illumination conditions in proximity scenarios, for example, for active debris removal [4,5,6]. In the same context, an interesting application of TIR images is their combination with visible (VIS) images using fusion algorithms to retain the advantages of both sensors (i.e., high definition for VIS and robustness in low illumination conditions for TIR); on this matter the interested reader can refer to [2,7,8].

Despite the unquestionable advantages of TIR images, the literature on how to create synthetic thermal images is still scarce, but at the same time, the possibility to create thermal images is fundamental for any mission involving thermal cameras for the following reasons:

• Thermal image datasets can be created to train and test navigation algorithms in all mission phases.
• Test software and hardware in the loop electronics, such as image processing boards.
• Simulate the scientific output of the mission.

Given the importance of the above points, the paper describes a method to create thermal images starting from high geometrical accuracy thermal simulation results and reports the application of the method in two different scenarios: the first one where the target is a spacecraft, and the second one where the target is an asteroid. The proposed method uses a 3D finite volume method to compute the temperature field of the object, accounting for all the thermal properties of the different materials of the target. The generation of the thermal image is then performed using a rendering engine, and finally, two thermal camera models are proposed as future starting points to simulate the actual output of the instrument.

The structure of the work is as follows: In Section 2, we report the modeling of the infrared radiation and the infrared camera models. In Section 3, the method to generate the infrared images is presented and applied to a spacecraft and to an asteroid scenario. In Section 4, the method is applied to real case scenarios to show the actual capability of the method and its importance in a real space mission design. In the end, Section 5 gathers the conclusions of the presented work.

2. Thermal Modeling

This section reports the major steps for the creation of the thermal image starting from the finite volume thermal model up to the thermal camera model output.

2.1. Finite Volume Thermal Model

The temperature field of the object is computed using the finite volume open source code OpenFOAM. The code takes as input the geometry of each component of the target in the .stl file format. The actual computational mesh is created using such files and for each of the components, the material properties are assigned as well as the thermal contact resistances. The solar radiation direction and intensity are provided as input, and starting from a guessed initial temperature field for each component, the finite volume method is applied to solve for the temperature field. The code offers the geometrical accuracy required to create realistic thermal images and the flexibility needed to interface with the rendering engine. For a detailed description of the code and the numerical testing, the reader can refer to [8,9]. The code has been compared to real thermal vacuum test data in a CubeSat scenario proving to be reliable in computing the temperature field; moreover, preliminary mono-material cases using such code for thermal image generation are presented in [2,10,11].

For the multi-region case presented in the article, different properties are assigned to the different parts of the object. From the resulting temperature field, each mesh face temperature is available together with the face orientation, which is fundamental for the view factor computation as reported in the next sections.

2.2. Radiation Modeling

There are two types of thermal cameras: photon counters and microbolometers [12]. The presented research work focuses on the latter one, being more versatile and more robust with respect to the former ones. Furthermore, they have a wide space heritage. For more details, the reader can refer to [13,14,15]. Microbolometer pixels are sensitive to the heat flux that hits the pixel; thus, the goal is to compute the heat flux per unit area emitted by each mesh face in the frequency spectrum of the thermal camera. For microbolometers, it is the Long Wavelength Infrared (LWIR) spectrum from 8 $\mu \mathrm{m}$ to 14 $\mu \mathrm{m}$; therefore, the heat flux emitted in this range by each face over a hemisphere [16], i.e., emissive power [$\mathrm{W} {\mathrm{m}}^{-2}$], is given by [13]:

$$q_{out}\left(T\right)=F\left(T\right)=\pi {\int }_0^{\infty }\epsilon \left(\lambda \right)B(\lambda ,T)R\left(\lambda \right)d\lambda ,$$

(1)

where $\epsilon \left(\lambda \right)$ is the emissivity, $B(\lambda ,T)$ is the Max Planck equation, and $R\left(\lambda \right)$ is a function limited between 0 and 1. It represents the detection efficiency of the bolometer in the LWIR range, the transmittance of the band-pass filter, and the transmittance of the germanium lens. An example of the shape of this function is reported in [13].

In order to unify the notation with ([16], Section 10.1), the emissive power can be rewritten in terms of intensity of radiation, i [$\mathrm{W} {\mathrm{m}}^{-2} {\mathrm{sr}}^{-1}$]:

$$q_{out}\left(T\right)=\pi {\int }_0^{\infty }\epsilon \left(\lambda \right)B(\lambda ,T)R\left(\lambda \right)d\lambda =\pi i.$$

(2)

The next step is to compute the portion of heat flux emitted by an infinitesimal part of a generic mesh face and intercepted by an infinitesimal part of a generic camera pixel, hence introducing the view factor between the mesh face and the pixel [16].

An infinitesimal area of a mesh face $d{A}_f$ emits heat flux in all directions, distributing it over a hemisphere with a total amount described by Equation (2). An infinitesimal area of the pixel of a camera $d{A}_p$ intercepts a portion of that hemispheric heat flux, and based on the solid angle $d{\omega }_p$, it subtends with respect to $d{A}_f$. Indeed, the heat flux that leaves $d{A}_f$ within $d{\omega }_p$ stays within $d{\omega }_p$ as it travels towards $d{A}_p$. If $d{A}_p$ is placed at a distance $r=|{\mathbf{r}}_{fp}|$ with an angle ${\beta }_{fp}$ with respect to the normal of $d{A}_f$ and it is normal to the radial direction ${\widehat{\mathbf{r}}}_{fp}={\mathbf{r}}_{fp}/\left|{\mathbf{r}}_{fp}\right|$, then $d{A}_p$ will see $d{A}_f$ as having an area of $cos{\beta }_{fp}d{A}_f$. At the same time, if $d{A}_p$ is not normal to the radial direction ${\widehat{\mathbf{r}}}_{fp}$, the actual area $d{A}_f$ will see $cos{\beta }_{pf}d{A}_p$, where ${\beta }_{pf}$ is the angle between the radial direction ${\widehat{\mathbf{r}}}_{fp}$ and the normal of $d{A}_p$, hence the approximated projection (assuming $d{A}_p$ infinitesimal) of $d{A}_p$ onto the hemisphere of radius r. A visualization of the angles is reported in Figure 1.

Using such a description, the heat flux emitted by $d{A}_f$ and intercepted by $d{A}_p$ can be written as:

$$d{Q}_{fp}=id{\omega }_{p}cos{\beta }_{fp}d{A}_f.$$

(3)

Using the definition of a solid angle, $d{\omega }_p$ can be written as the portion of the area of the sphere over the radius of the sphere squared. The portion of the area of the sphere to be used is the projection of $d{A}_p$ onto the sphere, and hence $cos{\beta }_{pf}d{A}_p$, pictured in Figure 2. Thus, Equation (3) can be written as:

$$d{Q}_{fp}=i{\displaystyle \frac{cos{\beta }_{pf}d{A}_p}{|{\mathbf{r}}_{fp}{|}^2}}cos{\beta }_{fp}d{A}_f,$$

(4)

where ${\mathbf{r}}_{fp}$ is the position vector that goes from $d{A}_f$ towards $d{A}_p$, hence the position of $d{A}_p$ with respect to $d{A}_f$. Introducing the normal versor of $d{A}_f$ and $d{A}_p$ as ${\widehat{\mathbf{n}}}_f$ and ${\widehat{\mathbf{n}}}_p$ respectively, then $cos{\beta }_{fp}$ and $cos{\beta }_{pf}$ can be written as a simple scalar product between the normal versor and the position vector, normalized, paying attention to invert the position vector for $d{A}_p$ (i.e., ${\widehat{\mathbf{r}}}_{pf}$ instead of ${\widehat{\mathbf{r}}}_{fp}$). A visualization of the formulation is reported in Figure 3. Thus, the equation can be written as:

$$d{Q}_{fp}=i{\displaystyle \frac{({\widehat{\mathbf{n}}}_f·{\mathbf{r}}_{fp})({\widehat{\mathbf{n}}}_p·{\mathbf{r}}_{pf})}{|{\mathbf{r}}_{fp}{|}^4}}d{A}_{f}d{A}_p.$$

(5)

The intensity of radiation can be rewritten inverting Equation (2); therefore, the final expression for the heat flux emitted by $d{A}_f$ and intercepted by $d{A}_p$ is the following:

$$d{Q}_{fp}={\int }_0^{\infty }\epsilon \left(\lambda \right)B(\lambda ,T)R\left(\lambda \right)d\lambda {\displaystyle \frac{({\widehat{\mathbf{n}}}_f·{\mathbf{r}}_{fp})({\widehat{\mathbf{n}}}_p·{\mathbf{r}}_{pf})}{|{\mathbf{r}}_{fp}{|}^4}}d{A}_{f}d{A}_p.$$

(6)

Given the small size of the pixels and their close positions, it is assumed that the view factors between one mesh face and all the camera pixels are equal; thus, it is sufficient to compute the view factor between the mesh face and the full camera array. Thus, the formulation becomes:

$$d{Q}_{fc}={\int }_0^{\infty }\epsilon \left(\lambda \right)B(\lambda ,T)R\left(\lambda \right)d\lambda {\displaystyle \frac{({\widehat{\mathbf{n}}}_f·{\mathbf{r}}_{fc})({\widehat{\mathbf{n}}}_c·{\mathbf{r}}_{cf})}{|{\mathbf{r}}_{fc}{|}^4}}d{A}_{f}d{A}_c.$$

(7)

The formula needs to be integrated over the camera area and over the mesh face. This step is discretized, assuming the mesh face is small enough to write the expression as:

$$\Delta Q_{fc}={\int }_0^{\infty }\epsilon \left(\lambda \right)B(\lambda ,T)R\left(\lambda \right)d\lambda {\displaystyle \frac{({\widehat{\mathbf{n}}}_f·{\mathbf{r}}_{fc})({\widehat{\mathbf{n}}}_c·{\mathbf{r}}_{cf})}{|{\mathbf{r}}_{fc}{|}^4}}\Delta A_f\Delta A_c.$$

(8)

In the expression above, the approximation is in the fact that the mesh face normal vector is assumed constant over the small area of the real object it is representing, and therefore, it is a good approximation only when the mesh face is small compared to the size of the object. Indeed, when the object geometry is meshed, it is a multitude of small flat faces. For what concerns the camera, it is assumed to be a flat surface; hence, it can be approximated as one flat face with a given area $\Delta A_c$ and normal vector ${\widehat{\mathbf{n}}}_c$. To be more specific, the just mentioned approximations are introduced in the step between Equation (7) and Equation (8), where the part of the equation regarding the view factor is approximated using the following formula:

$${\int }_{A_c}{\int }_{A_f}{\displaystyle \frac{({\widehat{\mathbf{n}}}_f·{\mathbf{r}}_{fc})({\widehat{\mathbf{n}}}_c·{\mathbf{r}}_{cf})}{|{\mathbf{r}}_{fc}{|}^4}}d{A}_{f}d{A}_c\approx {\displaystyle \frac{({\widehat{\mathbf{n}}}_f·{\mathbf{r}}_{fc})({\widehat{\mathbf{n}}}_c·{\mathbf{r}}_{cf})}{|{\mathbf{r}}_{fc}{|}^4}}\Delta A_f\Delta A_c.$$

(9)

As verified in [9], such an approximation is able to catch the correct values of the view factors. Dividing by the mesh face and considering a single diffuse value for the emissivity, the equation provides the heat flux per unit area emitted by each face and intercepted by the camera:

$$q_{fc}={\displaystyle \frac{\Delta Q_{fc}}{\Delta A_f}}=\epsilon {\int }_0^{\infty }B(\lambda ,T)R\left(\lambda \right)d\lambda {\displaystyle \frac{({\widehat{\mathbf{n}}}_f·{\mathbf{r}}_{fc})({\widehat{\mathbf{n}}}_c·{\mathbf{r}}_{cf})}{|{\mathbf{r}}_{fc}{|}^4}}\Delta A_c.$$

(10)

To compute the heat flux, the camera pose and attitude (i.e., ${\widehat{\mathbf{r}}}_{fp}$ and ${\widehat{\mathbf{n}}}_c$) must be provided to the algorithm as well as the different emissivities of each part of the mesh. The final result is a field of the heat flux emitted by each mesh face and intercepted by the camera. In this way, the image accounts for the view factor effects of the scene. The field is then passed to the render engine (Blender v2.93, https://www.blender.org/, accessed on 21 March 2024) for the actual image generation. As a general recap of the method workflow, the pipeline is summarized in Figure 4, where it is possible to see that the method starts from the geometry of the object, the thermal simulation then produces a mesh and a temperature field of the object, and they are both used to compute the view factors given the camera attitude and position. Once the view factors are computed, it is possible to compute the actual heat flux received by the camera using Equation (10) and generate the image. More details on the actual pipeline implementation and the respective results are presented in the next sections.

2.3. Thermal Camera Model

Microbolometer pixels produce an output voltage that is proportional to the heat flux received. The voltage is then amplified with a gain and converted to a digital bit string. The conversion is performed by the Analog Digital Unit (ADU) of the camera, and the number of discretization intervals depends on the bit architecture of the ADU; for example, for a 16 bit architecture, there will be $2^{16}$ intervals also called Digital Number (DN). The pixel output then is a DN whose value is proportional to the heat flux received.

The approach for modeling the camera is to start from the image produced by the rendering software (Blender v2.93, https://www.blender.org/, accessed on 21 March 2024). By knowing the bit depth of the image (in Blender either 8 or 16 bit) and the scale values, it is possible to associate the pixel DN of the image produced by Blender with the corresponding heat flux value. Then, using the sensor response curve, i.e., how the DN is proportional to the incoming heat flux, it is possible to compute the actual DN produced by the camera. From there, the noise can be added, and the output, in DN, is the actual image produced by the camera. The details of the workflow are described in Section 2.4.

At this point, two scenarios are possible based on the thermal camera manufacturing, and that is detection and radiometry. For the radiometry case, as described in the next section, the goal is to retrieve the temperature field of the target, so once the actual DN is obtained, the sensor response function can be inverted again, and the corresponding noisy heat flux is obtained, and the radiative temperature recovered.

For the detection case, the goal is to create an image with high contrast with no interest at all in recovering the temperature field. This is achieved by rendering an image with a scale proportional to the DN computed with the proposed method and tuning the scale to reach such a goal. The interested reader can refer to the FLIR-Tau2 user guide (Teledyne FLIR, FLIR Systems, Inc., Wilsonville, OR, USA, https://www.flir.com/support/products/tau-2/?pn=Tau+2&vn=46336100H#Documents, accessed on 21 March 2024) in the Automated Gain Control algorithm section for more information on this point. In the presented case, the $q_{fc}$ is converted into a DN field, and the scale is properly set.

2.4. Radiometry

To accurately recompute the temperature field, the distance from the object must be provided in order to be able to use the calibration curve of the thermal camera, usually given at a very short distance from the target [14,15].

The sensor function can be modeled in various ways. From the literature, here we report two sensor functions and their respective proposed implementations.

2.4.1. DN–Temperature Sensor Function

The first approach links DN to temperature with the following relation [15]:

$$DN=a·T^{\alpha }+b.$$

(11)

To compute a, b, and $\alpha$, a calibration is performed with a black body at different temperatures. The coefficient values are reported in [15]. To use such a model, the heat flux is recovered for each pixel using the image scale values, and then it is converted into the corresponding radiative temperature assuming the object as a black body and accounting for the object distance.

The effect of the distance is taken into account, as reported in Equation (12), in the computation of $T_{rad}$. If the distance is not compensated, then the equivalent $T_{rad}$ values would be extremely low. Then, the value is fed into the sensor function to compute the corresponding DN. Such values are modified according to the noise models, and the noisy temperature is recovered, inverting the sensor function (Equation (11)).

The scale of the image can then be adjusted accordingly to the selected temperature range. The steps are summarized in Equation (12):

$$\begin{array}{cc}\hfill q_{fc}\to T_{rad}={\left({\displaystyle \frac{q_{fc}\pi {\left|{\mathbf{r}}_{fc}\right|}^2}{\sigma \Delta A_c}}\right)}^{1/4}\to DN=a·T_{rad}^{\alpha }+b\to & \\ \hfill \to D{N}_{noisy}=DN+Noise\to T_{rad+noise}={\left({\displaystyle \frac{D{N}_{noisy}-b}{a}}\right)}^{1/\alpha }.\end{array}$$

(12)

It must be noted that in using such an approach, the computation of the received heat flux must be coherent with the thermal camera response function, meaning that we need to write $q_{fc}$ using the same input that is taken by the function that provides the DN. For example, in this case, the function $DN=a·T^{\alpha }+b$, uses T as the input; thus, we cannot use Equation (10) because (as explained in the next paragraph) if we invert the equation to recover the emitted heat flux, we obtain $F\left(T\right)$, when instead we need to recover T. Thus, a slightly different formulation of $q_{fc}$ that allows obtaining T is needed. A workaround, in this case, is given by the following equation, which has been used in Equation (10):

$$q_{fc}={\displaystyle \frac{\epsilon \sigma T_{rad}^4}{\pi }}{\displaystyle \frac{({\widehat{\mathbf{n}}}_f·{\mathbf{r}}_{fc})({\widehat{\mathbf{n}}}_c·{\mathbf{r}}_{cf})}{|{\mathbf{r}}_{fc}{|}^4}}\Delta A_c.$$

(13)

In using this approach, the sensors attenuation caused by the optical lenses and the sensor responsivity (i.e., the parameter $R\left(\lambda \right)$ in Equation (10)) is contained directly in the formula $DN=a·T^{\alpha }+b$ in the exponent $\alpha$ as stated in [15].

Given the fact that the camera response is directly proportional to the heat flux received, it is preferred to use the thermal camera model presented in the next paragraph, where the third calibration parameter $\alpha$ can be eliminated by simply calibrating the camera with the total emissive power of the emitter.

2.4.2. DN-Heat Flux Sensor Function

The second approach follows the same steps as the previous one but using the following formula for the pixel output as [13]:

$$DN=a·F\left(T\right)+b,$$

(14)

where $F\left(T\right)$ is used to keep the same notation of [13], and it is the emissive power:

$$F\left(T\right)=q_{out}\left(T\right)=\pi {\int }_0^{\infty }\epsilon \left(\lambda \right)B(\lambda ,T)R\left(\lambda \right)d\lambda .$$

As for the previous case, a and b coefficients are computed by calibration using black bodies at different temperatures. For the exact coefficient values, please refer to [13]. The steps are summarized as follows:

$$\begin{array}{cc}\hfill q_{fc}\to F\left(T\right)={\displaystyle \frac{q_{fc}\pi {\left|{\mathbf{r}}_{fc}\right|}^2}{\Delta A_c}}\to DN=a·F\left(T_{rad}\right)+b\to & \\ \hfill \to D{N}_{noisy}=DN+Noise\to F{\left(T\right)}_{noisy}={\displaystyle \frac{D{N}_{noisy}-b}{a}}.\end{array}$$

(15)

With the above formulation, the radiative temperature can be recovered using the sensor calibration curve [13], where each $F_{noisy}\left(T_{rad}\right)$ value corresponds to a radiative temperature value. As a recap, the workflow is pictured in Figure 5, where the pipeline starts from the Blender rendered image, which is in 16 bit, so each pixel has a DN value that goes from 0 to $2^{16}$. Since we know the maximum heat flux value of the scene and the minimum one, the pixel with the maximum DN value can be associated with the maximum heat flux value, and the same goes for the minimum one. In this way, any DN value of a pixel of the image can be associated with a $q_{fc}$ value. In other words, the scale of the image is converted from DN values to $q_{fc}$ values; therefore, every pixel now corresponds to a $q_{fc}$ value. With such values, it is possible to compute $F\left(T\right)$ for each pixel, which is the input variable for the camera model described in Equation (14); therefore, by knowing the gain and the offset of the thermal sensor function, i.e., a and b, it is possible to compute the exact DN value that the given thermal camera will produce. In conclusion, the DN value of each pixel is converted in the actual DN values produced by the given thermal camera by using Equation (14). With such an image, the noise of the given thermal camera can be added to produce the realist output of the thermal sensor. For more details on the noise model and characterization of thermal sensors, please refer to [17]. The final step is now possible, and that is to reconvert the DN just computed for each pixel and with the noise added in $F{\left(T\right)}_{noisy}$ values so that by knowing the dependency with the temperature [13], it is possible, for each pixel, to obtain the radiative temperature values.

In both approaches, in the inversion of Equation (10), the part relative to the faces–camera orientation is left out in order to retain the view factor effects in the final output, and it can be observed in the results in the next section. Indeed, it is not impossible to know the exact shape of the object a priori and compensate for the view factors effects caused by the orientation of each face.

The temperature field can always be corrected with the real emissivity of the material, if known, thus recovering the kinetic temperature. Starting from the radiative temperature, the kinetic one can be computed as:

$$T_{kin}={\displaystyle \frac{T_{rad}}{{\epsilon }^{1/4}}}.$$

(16)

Figure 5. Radiometry camera model workflow.

Figure 5. Radiometry camera model workflow.

2.5. Detection

For the detection case, the camera does not need to be calibrated; thus, the output DN is taken as directly proportional to the heat flux received by the camera $q_{fc}$. Therefore, the formulation is the following:

$$q_{fc}\to DN=a·q_{fc}+b\to D{N}_{noisy}=DN+Noise.$$

(17)

As stated before, the goal is just to obtain the actual DN produced by the camera. Using this method, the $q_{fc}$ can be translated into just a DN field, and the scale of the image can be set in order to have the best contrast between the target and background. Commercially available thermal cameras such as FLIR-Tau2 (https://www.flir.it/products/tau-2/, accessed on 21 March 2024) are sold, indeed, also without calibration, and they have image processing algorithms that can set the scale in non-uniform ways around the image to increase the target contrast. The interested reader can refer to the user guide of FLIR-Tau2 (https://www.flir.com/support/products/tau-2/?pn=Tau+2&vn=46336100H#Documents, accessed on 21 March 2024) for more details on such algorithms. As a recap, the workflow of the method is presented in Figure 6, where the first part of the pipeline is identical to the one presented in Figure 5, and that is to associate a $q_{fc}$ value to each pixel of the image. Starting from the image produced in Blender, since the maximum and minimum heat flux values $q_{fc}$ are known, they are associated with the maximum and minimum DN values of the pixels in the image; in this way, the scale of the image is converted from DN values into $q_{fc}$ values, and each pixel now corresponds to a $q_{fc}$ value. With such values, each pixel $q_{fc}$ value can be converted into the actual DN value produced by the given thermal camera based on the sensor model of Equation (17). In this conversion, the respective noise model can be added [17], and therefore the actual $D{N}_{noisy}$ value produced by the given thermal camera is computed for each pixel, reproducing the actual output of the thermal camera.

2.6. Method Implementation

The method is implemented thanks to the flexibility of the opensource code OpenFOAM, as well as the one of Blender. Indeed, the temperature field is exported in a VTK format (https://vtk.org/, accessed on 21 March 2024), a popular format for scientific visualization of the results. On the other side, Blender, thanks to a free add-on named VTKNodes (https://github.com/tkeskita/BVtkNodes, accessed on 21 March 2024), is able to import such a file into its node architecture for image rendering. In particular, a Python v3.9 code takes the temperature VTK file, within which the mesh is stored, as well as all the camera data, i.e., pose, pixel size, and sensor size. With such information, the radiative temperature field that is emitted by the target and received by the camera $q_{fc}$ is computed and stored in a different VTK file. This latter file is the one imported in Blender and used for the actual rendering. The overall pipeline is illustrated in Figure 7.

Once $q_{fc}$ is computed, it is possible to use it for the application of any thermal camera model. Indeed, the results presented in the section focus on such a quantity, as everything in the final thermal image is proportional to it. Furthermore, since $q_{fc}$ is scaled by the emissivity value $\epsilon$ of the material, it is clear that $\epsilon$ has a big impact on the final result as reported in the following subsections. The presented images are noiseless and for thermal images, the black and white scale is set from the minimum to the maximum heat flux values of the scene, producing a realistic thermal image, upon which, given the presented thermal camera models, the actual output of the instrument can be produced. For such considerations, since the goal is to test the pipeline, the computation of $q_{fc}$ is the one reported in Equation (18), where the optics attenuation is not considered:

$$q_{fc}={\displaystyle \frac{\epsilon \sigma T_{rad}^4}{\pi }}{\displaystyle \frac{({\widehat{\mathbf{n}}}_f·{\mathbf{r}}_{fc})({\widehat{\mathbf{n}}}_c·{\mathbf{r}}_{cf})}{|{\mathbf{r}}_{fc}{|}^4}}\Delta A_c.$$

(18)

It is a simplification because, with the limited wavelength range of the bolometer, the measured heat flux is not necessarily proportional to $T^4$, but since the image can always be scaled to the maximum heat flux observed in the scene, the final image is not affected. Furthermore, it does not affect the pipeline structure or the details of the final image, thus demonstrating the flexibility of the method and its ability to produce relevant data without a specific camera model. Indeed, the full-camera characterization (calibration and noises) is not always available, and the pipeline can cope with that. Having said so, this simplification shall be used just to test the pipeline and not to produce thermal images for real-case scenarios. The pipeline is currently applied with the complete formulation that considers the optics attenuation and the noise of the camera for the thermal camera data available from Hayabusa [1,3,13,14]. The results using such a thermal camera model are reported in Section 4.3. The proposed method has been thoroughly tested under multiple environmental conditions with different heat flux ranges, different combinations of material properties, and background disturbances. The interested reader can refer to [17] for more details on this point and for more information on the application of the pipeline using the complete formulation with optics attenuation and camera noise.

3. Results

The section presents the results of the application of the method reported in Section 2. The goal of the application is to test the ability of the pipeline to catch all the geometrical details of the scene and the ability to reproduce the effects of the different materials emissivities as well as of the view factors. The pipeline can be tuned with any camera model and noise model; thus, given a specific thermal camera, it allows the correlation of the generated synthetic thermal images, but this is left as a future step.

3.1. Spacecraft Case

The TANGO satellite is used as test case because the future goal is to create a thermal image dataset equivalent to the ones already available in the optical spectrum [18] so that, given a mission with both visible and thermal cameras, the detection and navigation algorithm can be compared.

3.1.1. Geometry and Materials

The exact TANGO spacecraft topology is not accessible; nonetheless, a representative geometry made of multiple parts is created, and each part is assigned with an emissivity value. It is reported in Figure 8, and it is made of 205 solar cells, 5 panels, 5 antennas, 7 patch antennas, and 1 coil with 2 supports for a total of 225 parts. The emissivities of each part are reported in

Table 1. Emissivity values.

Part Name	Emissivity
Antennas	0.86
Patch antennas	0.90
Coil	0.86
Support coil	0.86
Panels	0.40
Solar cell	0.60

.

3.1.2. Results

All the thermal images presented are produced using a circular orbit at 3 $\mathrm{m}$ from the center of TANGO with the camera in a center pointing attitude. The first thing to highlight in the results is the effect of the different components’ emissivities. As is clearly visible for a given camera pose reported in Figure 9, the solar cells are isothermal with the support panel, making them invisible in the temperature field image (left image in Figure 9), but due to the difference between the emissivity of the solar cells and of the panels, respectively 0.86 and 0.40, the heat flux received by the camera is completely different, thus creating a sharp contrast in the final thermal image (right image in Figure 9), where the solar cells shapes are indeed visible.

The same goes for the patch antenna, completely invisible as isothermal with the lateral panel (left image in Figure 9); it stands out clearly in the thermal image (right image in Figure 9). Furthermore, due to the different orientations of the mesh faces of the patch antenna with respect to the camera, given its curved shape, it is possible to appreciate the effects of the view factors on the amount of heat flux intercepted by the camera. As the face normal is aligned closer to the camera normal, the color is brighter, and vice versa. The same consideration goes for other curved shapes, such as the cylindrical antennas of TANGO (top right, right image in Figure 9).

Figure 9. Tango temperature field (left) and radiance field intercepted by the camera $q_{fc}$ (right).

Figure 9. Tango temperature field (left) and radiance field intercepted by the camera $q_{fc}$ (right).

The effects of the emissivities and view factors are even more visible for an arbitrary sequence of images taken from an orbit around the spacecraft pointing its center (Figure 10). As the solar panel normal gets closer to being perpendicular to the camera normal, the intercepted heat flux lowers down as reported in the second and fourth rows of Figure 10. To show the different orientations of the panels, the frames are not taken at equal intervals; thus, the second and fourth rows of the frames have different angles with respect to the camera, even though they look symmetric. This also highlights how small-angle differences in the camera–object system can affect the results. To better clarify the last point, two different orientations are reported in Figure 11, where we can see the drop in the received heat flux due to the different orientation of the solar panel; the same considerations go for all the panels of the spacecraft. In conclusion, the actual output of the pipeline is a black and white image that is sent to the navigation chain, and thus, the final grayscale image is reported in Figure 12.

3.1.3. Spacecraft Case Conclusions

The goal of the method was to create a dataset of thermal infrared images of a spacecraft in order to be able to test navigation algorithms. The reported results show that all the details of the geometry of the target, the effects of the multiple materials emissivities, and the effects of the different view factors between the camera and the object geometries are retained in the final thermal image. In this way, the method can produce a realistic thermal images dataset to start testing the performance of navigation algorithms in the infrared spectrum. The interested reader can refer to [10,17] to see the usage of such images for navigation purposes. In particular, the ASTRA Polimi research team is now focused on testing the use of such an image in combination with VIS images in order to cover the phase of proximity operations in which there are low or adverse illumination conditions as reported in [10] and also reported in Section 4.1.

It must be highlighted that the method as it is presented is still not validated; therefore, the accuracy of the thermal images is not guaranteed. Nonetheless, given the details of the output and the qualitative correctness, it can be used right away as a starting point.

Figure 10. Temperature field (left) radiance field $q_{fc}$ (right) for orbit at 5$\mathrm{m}$ around TANGO.

Figure 10. Temperature field (left) radiance field $q_{fc}$ (right) for orbit at 5$\mathrm{m}$ around TANGO.

Figure 11. In the last row, the received radiative heat flux reduces due to the view factors when surface normals are almost parallel with the camera normal. Temperature field (left) and radiance field intercepted by the camera $q_{fc}$ (right).

Figure 11. In the last row, the received radiative heat flux reduces due to the view factors when surface normals are almost parallel with the camera normal. Temperature field (left) and radiance field intercepted by the camera $q_{fc}$ (right).

Figure 12. Radiance field $q_{fc}$ (left) infrared image black and white image (right).

Figure 12. Radiance field $q_{fc}$ (left) infrared image black and white image (right).

The validation of the method requires the calibration of the thermal camera and the correlation of real thermal images taken in a controlled environment with the ones produced by the method. The whole method is built to be fully tuned; thus, the camera calibration curve, i.e., the function that provides the DN, can be easily modified inside the pipeline. The ASTRA Polimi team is now working on such topics; the interested reader can refer to the ESA e.Inspector project for future developments (more on this in Section 4).

For what concerns the computational time for the generation of thermal images, it must be highlighted that it strictly depends on the number of faces of the mesh, as all the view factors for each camera pose must be computed in order to calculate the actual infrared heat flux received by the camera. Once the heat flux is computed, the rendering of the image through Blender depends on the performance of the GPU mounted on the PC. As an order of magnitude, for the TANGO spacecraft case, the mesh has six million faces and it takes six hours in total to compute the heat flux for two hundred camera poses and then render such images.

3.2. Asteroid Case

For the asteroid case, Ryugu has been selected, as thermal images from the Hayabusa mission are available, thus enabling the future validation of the results. Furthermore, Ryugu geometry is freely available; on the other hand though, all the files have corrupted geometry that must be cleaned before the thermal simulation.

3.2.1. Geometry and Materials

Two geometry models are considered: the 800K faces model and the 3M faces model. Both models have corrupted geometry, where for corrupted geometry, we consider all unrealistic sudden changes in face normals and face deformations; some examples of corrupted geometry are reported in Figure 13. Thus, starting from the 800K model, the first step is to clean the geometry to perform a smooth thermal simulation.

A fast approach to cleaning the geometry is to apply a smoothing filter to the whole asteroid using Blender sculpting modifiers. Although the result is perfect for the mesh generation for the thermal analysis, there is an evident loss in the terrain detail as reported in Figure 14; therefore, this solution is retained just to check and test the numerical setup of the thermal simulation, but it is discarded for the final output, as the goal is to retain all the terrain details to produce an image close to the real one as much as possible.

Analyzing the 3M model, no differences can be spotted in Figure 15 but given the smaller faces, the 3M better approximate the terrain curves and details, easing the meshing phase of the simulation. On the other hand, as for the 800K model, corrupted geometry is still present, and it is actually worse, as the smaller faces increase details also in the errors of the geometry as shown in Figure 16.

Since the goal is to produce a realistic image, it is decided to use the 3M model, as it provides a more accurate mesh. To clean the geometry without losing too much detail, a weighted paint smoothing filter is applied just on the corrupted regions. This method is a Blender sculpting technique, and consists in painting the region that must be smoothed with a color that is proportional to the smoothing strength, so the hottest color corresponds to a heavy smoothing and the cold colors to a lighter one. All the regions that have been healed are reported in Figure 17, where it is possible to see all the regions with corrupted geometry.

The final result is reported in Figure 18; from the picture, it is possible to verify that the realism of the geometry is retained, and at the same time, the geometry is clean enough to be meshed for the thermal analysis.

Figure 17. Smoothing filter weights on the Ryugu 3M model. Red corresponds to the maximum smoothing available, down to green for the minimum, and violet for zero smoothing applied.

Figure 17. Smoothing filter weights on the Ryugu 3M model. Red corresponds to the maximum smoothing available, down to green for the minimum, and violet for zero smoothing applied.

The number of cells on the Ryugu exterior face is increased in the snapping parameters to overcome radiation illness of the linear system produced in the finite volume discretization process. Due to the low conductivity of the asteroid regolith, at the surface of Ryugu, where radiation and diffusion are combined, radiation overcomes diffusion for each cell at that surface. This creates instability due to the big oscillations caused by the errors in the temperature field that are elevated to the power of four in the code. To better clarify the problem, the code is benchmarked with a cube of 12 $\mathrm{m}$ and with different mesh discretizations. The analysis of such a problem is out of the scope of the research work, but in conclusion, reducing the cell size and using relaxation is enough to stabilize the algorithm. Given the problem though, different mesh resolutions were tested to verify the results were correct at least from a numerical point of view.

The final simulation is single material, and it is run in steady-state mode; moreover, to avoid losing convergence in the first steps, the simulation is started with a high conductivity value at 270 $\mathrm{W}$ m⁻¹ K⁻¹. In this way, the diffusion contribution in the finite volume discretization is larger than the radiation term and the simulation can start in a smoother way. Indeed, once convergence is reached, it is lowered to 1 $\mathrm{W}$ m⁻¹ K⁻¹ (whilst keeping the infrared emissivity at 0.97 and the visible absorptivity at 0.95 [19,20]). Using a high conductivity before the very low one of the asteroid ensures a starting temperature field closer to the final one, hence reducing the initial temperature oscillations given by the guessed starting temperature field. All the mesh sizes tested to prove the numerical convergence of the grid are reported in

Table 2. Ryugu simulation stabilizes at 6M faces on the asteroid surface.

N Faces	Face Size [m]	Min [K]	Max [K]	Note
3.7M	0.4	−340	1280	Oscillation
4.3M	0.36	77	800	Oscillation
5.7M	0.33	40	418	Stable
6.4M	0.31	40	416	Stable
8.9M	0.26	42	419	Stable

.

3.2.2. Results

All the thermal images presented are produced using a circular orbit at 3 $\mathrm{k}$$\mathrm{m}$ from the center of Ryugu with the camera in a center pointing attitude. The final temperature field is reported in Figure 19. The level of detail that can be reached from the tool is very high, but of course in this case, the computational cost is much higher with respect to the spacecraft case.

As for the spacecraft case, the view factors play a big role. To better clarify this point, a sequence of frames is taken using a circular orbit around the asteroid with the camera facing the center. From Figure 20, it is possible to see how the rocks and faces parallel to the camera create bright contrasts, producing a realistic thermal image. In Figure 21, it is clear how the curved shape of the object lowers the view factors, moving from the equator region towards the polar region; as a consequence, the heat flux intercepted by the camera is higher when looking at the equator. In conclusion, an example of the final output in black and white scale is reported in Figure 22.

3.2.3. Asteroid Case Conclusions

As for the spacecraft case, the goal of the method was to produce a thermal image retaining all the details of the terrain for an asteroid geometry. The reported results show that the pipeline is able to create images with a very high level of detail, producing realistic thermal images of complex terrain geometries. The pipeline therefore enables us to start testing navigation algorithms in such scenarios. As for the spacecraft case, it must be pointed out that the method is still not validated; thus, the accuracy of the generated images is not guaranteed. Nonetheless, given the ability to deal with such a level of geometry complexity and the qualitative correctness of the results, this is a good starting point. In particular, as for the spacecraft case, the ASTRA Polimi team is focused on the application of such images in combination with VIS images for covering the proximity phases, in which VIS images have low or adverse illumination conditions.

In order to validate the thermal images of Ryugu, the thermal model must be tuned and run in transient mode to recreate the exact temperature field, and the thermal images must be recreated with the exact thermal camera parameters and noise levels used in the Hayabusa mission. The pipeline is perfectly able to do that, but it is out of the scope of this article which focuses entirely on the method.

Figure 20. Temperature field (left), radiance field $q_{fc}$ (right).

Figure 20. Temperature field (left), radiance field $q_{fc}$ (right).

Figure 21. Ryugu temperature field (left), and radiance field $q_{f}c$ (right).

Figure 21. Ryugu temperature field (left), and radiance field $q_{f}c$ (right).

Figure 22. Radiance field $q_{fc}$(left), infrared black and white image (right).

Figure 22. Radiance field $q_{fc}$(left), infrared black and white image (right).

4. Applications

This section presents the application of the method in real research projects. Again, the tool has proved to be fundamental to verify the analyses of the different research projects. Three different applications are presented: the first one is on the fusion of TIR images with visible ones to increase the visibility of the target. The second one focuses on a far range detection in the context of an ESA phase B space mission on debris detection and approach. The very last case shows how the method can be applied also for the generation of the Earth background.

4.1. Image Fusion

The preliminary results of the tool presented in Section 2 were applied in [2,10,11] for image fusion with the goal to enhance optical image visibility using thermal infrared camera data. The context of such research work was focused on facing problems like the one presented in Figure 23c, where in a proximity scenario, the target object, in this case, the TANGO satellite, enters a low visibility condition such as the eclipse phase, where the object is in complete shadow. This is exactly one of the problems described in the introduction in Section 1.

In this scenario, the thermal image generation pipeline is able to provide the requested thermal images with the same camera attitude of the visible one, thus enabling the fusion of the two. As it is visible from the results reported in Figure 24, the visible image has parts that are in complete shadow and cannot be used by the algorithms to fully detect the shape of the object, thus missing features that can be used as key points for the navigation algorithms. Such regions are completely visible in the thermal infrared image, so fusing the thermal image with the visible one provides a third image that retains the detail of the visible spectrum image (thermal cameras usually have smaller pixel arrays, thus fewer details) but without shadowed regions. In Figure 24, the best fusion methods are reported; for deeper analysis and comparison of different fusion methods, the reader can refer to [2].

4.2. Detection

The tool is used also for far-range detection, where the advantages of the thermal image are clearer. The object analyzed in this case is the VESPA (VEga Secondary Payload Adapter) debris at 100 $\mathrm{m}$ distance. The object is pictured in Figure 25, and the interested reader can refer to [11,18] for more information.

The context of the research work in which the method is applied is a Phase B mission funded by ESA, named e.Inspector, whose main goal is to perform proximity orbits around the non-cooperative object to demonstrate the navigation capability of dedicated algorithms.

Figure 25. VESPA adapter (left) and thermal model (right).

Figure 25. VESPA adapter (left) and thermal model (right).

The generated synthetic thermal image is compared with the respective visual one in Figure 26 (picture sizes are different as the visible camera and TIR camera have different pixel array sizes). As stated in the introduction, the visible image captures all the star lights, making it harder to detect the target compared to the thermal infrared image, where it is a bright light point, which means the image processing algorithms might take more iterations, and thus more time, to detect and isolate the object in the image. The same goes for long-exposure images in Figure 27, (credits to M.Bechini for the images in the visible spectrum [18]). Even though it is possible to remove the stars from the images by knowing the camera pose and using star catalogs, it is a time-consuming operation and it increases the overhead of the image processing in a real-time navigation scenario, thus leaving the thermal image the faster way to detect the target for this case.

Of course, in a real scenario as reported in [3], the camera noise level and response function must be modeled as presented in Section 2 to clearly asses the detection range of the object. The tool is able to take these data as input together with the noise models or noise maps once the full camera characterization is complete. Nonetheless, if the response function and noise data are not available, the tool is able to simulate and provide the team with a realistic input that can be used to test on ground the navigation and detection chain, something that is fundamental to access the next phases of the mission.

Figure 26. Visible image with stars (left) and thermal image (right).

Figure 26. Visible image with stars (left) and thermal image (right).

Figure 27. Inertial pointing visible image with stars (left) and thermal image (right).

Figure 27. Inertial pointing visible image with stars (left) and thermal image (right).

4.3. Earth Background

The method is also applied to generate the thermal image of the Earth as well as the one of the target as shown in Figure 28. This demonstrates again the flexibility of the tool and its applicability to any scenario. Furthermore, thanks to this application, the detection algorithms can be tested to check if they are able to find the target when Earth is in the background Figure 29. The research work on this topic is still ongoing, and it will be presented in a dedicated research outcome. For more information, the interested reader can refer to [17].

5. Conclusions

The presented tool proposes a methodology to create synthetic thermal infrared images, enabling the simulation of the infrared camera output given the camera position and attitude. It is demonstrated how the tool creates realistic thermal images in both spacecraft and asteroid proximity scenarios and both in single and multi-material cases, proving to be suited for multiple mission cases. Indeed, the tool has already been used in a real space mission design phase B, proving to be an essential tool for testing navigation algorithm pipelines. Furthermore, two camera models are presented, and they can be used as starting points for the production of realistic instrument output. The presented work is a step in the existing literature, as no methodologies to produce synthetic infrared images from thermal simulations have ever been presented, up to the authors’ knowledge. The approach can be repeated and implemented by any other research team since all the software used are open source and free of license, thus making the method available to anyone interested in the research.