Geometric Calibration of Thermal Infrared Cameras: A Comparative Analysis for Photogrammetric Data Fusion

Abstract

The determination of precise and reliable interior (IO) and relative (RO) orientation parameters for thermal infrared (TIR) cameras is critical for their subsequent use in photogrammetric processes. Although 2D calibration boards have become the predominant approach for TIR geometric calibration, these targets are susceptible to projective coupling and often introduce error through manual construction methods, necessitating the development of 3D targets tailored to TIR geometric calibration. Therefore, this paper evaluates TIR geometric calibration results obtained from 2D board and 3D field calibration approaches, documenting the construction, observation, and calculation of IO and RO parameters. This includes a comparative analysis of values derived from three popular commercial software packages commonly used for geometric calibration: MathWorks’ MATLAB, Agisoft Metashape, and Photometrix’s Australis. Furthermore, to assess the validity of derived parameters, two InfraRed Thermography 3D-Data Fusion (IRT-3DDF) methods are developed to model historic building façades and medieval frescoes. The results demonstrate the success of the proposed 3D field calibration targets for the calculation of both IO and RO parameters tailored to photogrammetric data fusion. Additionally, a novel combined TIR-RGB bundle block adjustment approach demonstrates the success of applying ‘out-of-the-box’ deep-learning neural networks for multi-modal image matching and thermal modelling. Considerations for the development of TIR geometric calibration approaches and the evolution of proposed IRT-3DDF methods are provided for future work.

1. Introduction

The determination of precise and reliable intrinsics for thermal infrared (TIR) cameras is critical for their subsequent application in photogrammetric processes. Whilst geometric calibration represents a largely routine procedure for cameras sensing in the visible spectrum (RGB) [1], where 2D boards or 3D coded test fields are commonly used, the process presents several challenges when undertaken for TIR cameras. Notably, these comprise the following: (1) the inherent low resolution of commercial TIR sensors (typically 180 × 120–640 × 480 pix), (2) the construction of accurate calibration targets with sufficient difference between material emissivities ($\epsilon$), and (3) the imaging of control points with high contrast and low noise [2]. Consequently, the adoption of 2D calibration boards from the computer vision (CV) community has become the predominant approach for TIR geometric calibration, made using low-emissivity materials ($\epsilon <0.5$; e.g., aluminum foil, copper, and stainless steel) contrasted with high-emissivity materials ($\epsilon >0.8$; e.g., rubber, wood, and velvet) [3]. Such methods have been utilised to calibrate infrared sensors across different spectral wavelengths [4,5,6], for passive and active thermography [7,8,9], and for indoor and outdoor applications [10,11,12]. However, the overall accuracy of existing 2D calibration boards, often constructed through manual methods to generate calibration patterns, has required new methods for geometric calibration tailored to TIR cameras.

In a bid to improve the quality of calibration boards, several works have looked to reduce human intervention by employing superior manufacturing processes for accurate board construction. Usamentiaga et al.’s [2] critical study demonstrates the use of DiBond^®, an aluminium composite material, and UV-printing to produce a cost-efficient, easy-to-manufacture, and accurate 2D calibration board. Similarly, ElSheikh et al.’s [13] comprehensive literature review analysed the best materials, heating techniques, and associated errors of existing geometric calibration methods for TIR cameras, using a stainless-steel (SS304) base plate with UV-printing and laser-engraving for calibration boards. Zhang et al. [14] demonstrated the suitability of high-tolerance 3D printing to improve accuracy, constructing 3D-printed elements (PLA and aluminium plate) configurable into a 2D checker-board pattern. Roshan et al. [15] introduced high-tolerance aluminium, ArUco, [16] components into a traditional checker-board, mitigating the established issues of partial pattern observation and extraction for 2D calibration boards. Although these approaches provide considerable improvements in target construction, 2D calibration boards remain prone to the projective coupling of interior (IO) and exterior (EO) orientation parameters [7], with higher correlations between parameters being a likely by-product of poorer network geometry typically imposed by low resolution TIR cameras.

Conversely, effective 3D calibration approaches reduce the propensity for correlations between (and across) IO and EO parameters, leading to greater overall accuracy for subsequent photogrammetric applications [1,7]. Much like 2D boards, 3D calibration fields have been constructed using various materials and pattern configurations for TIR geometric calibration, with Westfeld et al. [17] creating a 3D test field using velour and silver foil and Yastikli and Guler [18] using an iron calibration rig with 77 plastic markers of contrasting depths. Rizzi et al. [19] constructed a 3D test field from custom markers to calibrate a FLIR P640 thermal camera (640 × 480 pix), calculating material emissivity values to validate material choice. In a bid to calibrate thermal stereo set-ups, Rangel et al. [20] and Adán et al. [21] used reflective markers for the calibration of thermal cameras fixed with a depth camera and laser scanner, respectively. Both Luhmann et al. [22] (2D board and 3D field) and Dlesk et al. [23] (2D board, and 2-level and 3-level 3D fields) compared multiple calibration approaches, determining that 3D fields are superior for subsequent 3D thermal modelling. However, opportunities are still present to determine the performance of new 3D test fields for the following: the calculation of IO parameters, their comparison against existing 2D board approaches, and their application for InfraRed Thermography 3D-Data Fusion (IRT-3DDF) [24].

In parallel, the estimation of the relative orientation (RO) between sensors has played a crucial role in the development of data fusion methods using TIR cameras. By establishing a fixed RO between independent cameras [25], or exploiting tandem sensors within the same camera housing [26], the mapping of TIR-RGB image pairs can be realised through effective calibration. This relationship is of particular importance when there are no means of determining the poses of TIR images independently, often due to low contrast or noisy images providing few discernible features for photogrammetric image matching. Furthermore, if TIR and RGB images are to be captured concurrently, this fixed relationship can help mitigate sub-optimal conditions for each image modality during image acquisition. By assuming a fixed RO between multiple cameras or sensors, such information can be used as an a priori constraint for sensor fusion [27], to determine a universal transformation key between image pairs [23], or as a means of additional sensor integration [28]. Therefore, the determination of reliable approaches for relative pose estimations is critical for the use of multi-modal imagery within IRT-3DDF.

The exploitation of fixed stereo systems within IRT-3DDF has enabled calibration approaches to determine the RO between sensors and cameras for data fusion applications. For sensors in fixed relative orientation within the same camera housing, Lecomte et al. [29] and Macher and Landes [30] used MATLAB’s Stereo Camera Calibrator tool to estimate the relative pose of TIR and RGB sensors, the latter comparing results to known camera dimensions. Fixing several cameras on a stereo rig, Lerma et al. [31] present a distance-constrained self-calibration approach, obtaining more precise IO and EO parameters for a FLIR B4 thermal camera when integrating known baseline constraints. Ursine et al. [32] assessed the accuracy of RO parameters using a 2D board, determining on-site calibration as a suitable approach under favourable conditions. Notably, the development of reliable RO calibration methods and the application of derived parameters for novel IRT-3DDF methods remains overlooked and warrants further investigation.

Whilst the standard error of image coordinate residuals [13] and individual calibration parameters [7] have become common precision metrics for estimating the performance of CV geometric calibration, these cannot be used as a proxy for accuracy within subsequent photogrammetric applications [33]. Therefore, the metric evaluation of derived IO and RO parameters must be undertaken both during and after calibration for meaningful comparisons to be made regarding the quality of different TIR geometric calibration processes [22,34]. Furthermore, the appraisal of calibration methods requires emphasis to be placed on the use of IO and RO parameters for IRT-3DDF, particularly when specific challenges and novel applications rely on precise geometric calibration [23]. Finally, as a wealth of photogrammetric CV software capable of performing geometric calibration now exists, a comparative analysis of popular commercial software benchmarked against dedicated metrology software may indicate the level of precision, control, and information being sacrificed in order to achieve greater procedural automation [1,35].

1.1. Research Aims

This paper aims to comparatively evaluate TIR geometric calibration approaches for the determination of IO and RO parameters. Moreover, the performance of calibration approaches applied to two IRT-3DDF methods is assessed to determine the efficacy of calibration processes for modelling architectural and cultural heritage. The research reported in this paper will accomplish the following:

1.

Compare 2D board and 3D field calibration processes, detailing production, observation, and calculation of IO and RO parameters using three commercial software packages: MathWorks’ MATLAB, Agisoft Metashape, and Photometrix’s Australis;

2.

Assess the accuracy of derived IO parameters applied to IRT-3DDF using a combined TIR-RGB bundle block adjustment for a historic building façade (Method 1);

3.

Assess the accuracy of derived RO parameters applied to IRT-3DDF using a relative pose implementation on medieval frescoes (Method 2).

1.2. Paper Structure

This paper is structured as follows: Section 2 outlines the methodology for 2D board and 3D field calibration targets, comparing the generated IO and RO results from the specified software packages; Section 3 applies these parameters to two IRT-3DDF methods, evaluating their performance using a combined TIR-RGB bundle block adjustment and relative pose implementation; and Section 4 reviews the performance of both 2D board and 3D field calibration processes, providing recommendations for future adoption, refinement, and applications.

2. Geometric Calibration

The following section outlines the processes involved in the geometric calibration of all cameras used in subsequent IRT-3DDF methods. This includes the following: (1) the construction of both the 2D board and 3D field calibration targets; (2) the calculation of the IO parameters for a combined TIR-RGB bundle block adjustment (Method 1); and (3) the determination of RO parameters for a relative pose implementation (Method 2). Importantly, the TIR and RGB cameras are not fixed in RO for Method 1, meaning imagery for the calibration processes and IRT-3DDF method were acquired asynchronously. For Method 2, independent cameras were fixed on a stereo rig to determine the IO and RO parameters for the eventual IRT-3DDF methods.

2.1. Cameras

For the capture of TIR images in both methods, a Workswell^® WIRIS Pro UAV camera (Prague, Czech Republic), featuring both thermal (640 × 512 pix) and visible (1920 × 1080 pix) spectrum sensors within the same camera housing, was configured to be used terrestrially (Figure 1). This camera provides thermal–visible image pairs captured simultaneously during each imaging survey. Although the Workswell WIRIS Pro features both an infrared and visible sensor, WWP is used as the designation for the infrared sensor within the camera. To differentiate between visible spectrum images derived from the sensor housed within the Workswell WIRIS Pro, as opposed to images captured from independent digital cameras, the term VIS is adopted for the visible spectrum sensor housed within the Workswell WIRIS Pro thermal camera, whilst the term RGB is used to denote separate digital cameras. For RGB images, two different digital cameras were used for each data fusion method listed in Section 1.1, including a Sony (Tokyo, Tokyo) α 7R II (7952 × 5304 pix) (RGB1) and a Nikon D750 (Tokyo, Tokyo) (6016 × 4016 pix) (RGB2). Specifications for each camera are listed in

Table 1. Camera specifications for the Workswell WIRIS Pro (WWP and VIS), Sony α 7R II (RGB1), and Nikon D750 (RGB2) cameras.

Workswell WIRIS Pro Infrared Sensor (WWP)
Resolution	640× 512 pix
Sensor Size (FPA)	10.88 × 8.71 mm
Pixel Size	17.00 µm
Nominal Focal Length	13.00 mm
Spectral Range (LWIR)	7.5–13.5 µm
Temp. Sensitivity	0.05 °C
Temp. Accuracy	±2 °C
Workswell WIRIS Pro Visible Sensor (VIS)
Resolution	1920 × 1080 pix
Sensor Size (CMOS)	5.23 × 2.94 mm
Pixel Size	2.72 µm
Nominal Focal Length	3.50 mm
Sony α 7R II (RGB1)
Resolution	7952× 5304 pix
Sensor Size (CMOS)	35.90× 24.00 mm
Pixel Size	4.50 µm
Nominal Focal Length	35.00 mm
Nikon D750 (RGB2)
Resolution	6016× 4016 pix
Sensor Size (CMOS)	35.90 × 24.00 mm
Pixel Size	5.95 µm
Nominal Focal Length	50.00 mm

.

2.2. Calibration Targets

Where calibration methods for IRT-3DDF differ from other TIR approaches is that targets must be observable in both infrared and visible spectral ranges, allowing common points to be identified across image modalities for the estimation of RO parameters. Therefore, construction and observation processes must appreciate the duality of the required geometric calibration, creating targets capable of calibrating multiple sensors precisely and simultaneously. Although 2D calibration boards have become the established standard for IRT-3DDF, forming the comparative basis within this study, this work aims to introduce a novel, low-cost, and multi-purpose 3D field that is precise in manufacturing and construction, easy to assemble and configure, and provides clear markers in both image modalities.

For both calibration targets, the ASTM standard ‘Practice for Measuring and Compensating for Emissivity Using Infrared Imaging Radiometers’ (ASTM E1933-14) was used to determine emissivity values of each target material [36]. Here, ASTM E1933-14’s non-contact thermometer method (NTM) was modified to utilise ThermoLab (Version 0.9.4), Workswell’s freely available post-processing software allowing emissivity values to be derived retroactively. For each calibration target, eight perpendicular images were captured at varying temperatures, with each image constituting four points where a surface-modifying material was adhered (TESTO SE & Co., Alton, UK, emission tape) ($\epsilon =0.95$). The emissivity values for both 2D board and 3D field targets can be seen in

Table 2. Calculation of material emissivity ($\epsilon$), standard deviation ($\sigma$), and difference in emissivity of materials ($\epsilon \Delta$) used for both 2D board and 3D field calibration targets.

	Material	$\mathit{\epsilon }$	$\mathit{\sigma }$	$\Delta \mathit{\epsilon }$
2D Board	DiBond®	0.62	0.04	0.20
	UV-printing	0.82	0.04
3D Field	Aluminium	0.20	0.03	0.65
	Rubber	0.85	0.05

. For all 2D and 3D approaches, a temperature-controlled water bath was utilised to submerge the 2D board and 3D field markers to excite thermal contrast. Components were submerged in the water bath for 1 min 30 s at 40 °C prior to positioning, constituting a passive thermography approach once image capture was undertaken [2,13].

2.2.1. 2D Board

For the 2D calibration target, a checker-board was designed using DiBond^®, an aluminium composite material, and an industrial signage UV printer [2] (Figure 2). DiBond, comprising two aluminium sheets sandwiching a polymer core, provides several benefits over previously listed materials as it is (1) perfectly planar; (2) a good heat conductor; (3) inexpensive; (4) available in several material finishes; (5) able to be cut to any size; (6) appropriate for high-resolution UV printing (i.e., <600 dpi). An A3 piece of silver, brushed-finish DiBond was utilised, printed on with black ink using two passes of a SwissQ, city: Kriessern, country: Switzerland Nyala 4 UV printer (1350 dpi, 53.1 dots/mm). As both checker-board [13] and circle-board [2] patterns have been deemed suitable for high-precision calibration targets, a $5\times 8$ ($35\times 35$ mm) checker-board pattern was printed.

2.2.2. 3D Field

For the 3D calibration targets, two separate multi-level test fields were configured using custom markers. Importantly, the novelty of the 3D test fields, designed to be low-cost, easy-to-manufacture, and highly configurable, is that the multi-purpose markers solve several open challenges for IRT-3DDF, notably (1) their use as configurable calibration markers for lab- and field-based geometric calibration procedures, (2) their use as survey markers for photogrammetric scaling, referencing, and validation, and (3) their use as ground truth for radiometric calibration, correction, and verification. Each test field featured markers of 60 mm × 60 mm constructed from milled aluminium and rubber squares fixed to a polymer baseplate. These provide a crosshair generated by materials with different colours and emissivity values (Table 2), clearly identifiable in both image modalities. A total of 16 markers provided a suitable configuration that covered both image formats and could be positioned after active heating. Figure 3 shows the two 3D calibration targets developed for the determination of IO and RO parameters in Method 1 and Method 2, respectively. For the metric evaluation of the IO values in Method 1, each marker was measured using a Leica TS10 total station (angular accuracy: 0.1 mgon; distance accuracy (non-prism): 2 mm + 2 ppm), although 3D world coordinates were not included within the self-calibrating bundle adjustment. For Method 2, a series of scale bars were measured between markers to provide scaling and reference for the relative pose estimation.

2.3. Interior Orientation

For the geometric calibration of TIR cameras, the WWP can be defined as a standard photogrammetric camera following the Brown model [37], merely observing a different spectral range (7.5–13.5 µm). However, as the configuration of the WWP’s germanium lens and solid-state focal plane array (FPA) sensor is optimised for radiometry as opposed to geometry [7], the determination of accurate distortion parameters is paramount. Therefore, to test the importance of the distortion coefficients, parameters were determined for each calibration target with either two or three radial distortion coefficients ($k_1$, $k_2$, $k_3$), in addition to tangential distortion coefficients ($p_1$, $p_2$). After initial investigations, the FPA sensor was deemed perpendicular, meaning no differential scaling ($b_1$) or non-orthogonality ($b_2$) values were included as required calibration parameters [33]. Finally, the optics of the WWP were fixed during manufacturing, with the camera focussed at infinity and unchanged during each calibration procedure [38]. As an appropriate camera network geometry is fundamental for effective calibration using both 2D and 3D calibration targets [39], images were taken with high convergence, orthogonal roll (+/−90°), varying depths along the principal axis, and the entirety of the target within the image format [1] (Figure 4).

For the calculation of IO parameters, three separate commercial software packages were employed for geometric calibration. Firstly, MathWorks’ MATLAB, R2024b (v24.2.0.2773142) a popular software for CV camera calibration, was used for the geometric calibration using the 2D board. Secondly, for the 3D field calibration target, Agisoft Metashape Pro (v2.1.3) an industry-standard software widely adopted for Structure-from-Motion (SfM) and Multi-View-Stereo (MVS) workflows, was used to assess the feasibility of the proposed methods to work with commonly employed photogrammetric software. Finally, to assess the validity of proposed 3D field methods for high-precision metrological applications, Photometrix’s Australis (v8.555) was also used to undertake calibration processes. Although both 3D field software are capable of undertaking calibration using marker-less feature-based self-calibration [40], only 2D marker coordinates were used within the calibration pipelines. For both software, the same 2D image coordinates, 3D world coordinates, and accompanying constraints were used for implementation. The 2D image coordinates of the test field markers were user-defined in Agisoft Metashape and weighted at 0.33 pix. Similarly, 3D marker coordinates, used solely for metric evaluation, were measured with a Leica, Heerbrugg, Switzerland TS10 total station, with a weight of 2 mm applied to account for propagated error values.

2.3.1. 2D Board (MATLAB)

For the 2D board, MATLAB’s Camera Calibrator app (2024b) was used for the calibration of the WWP, using Zhang’s algorithm [41]. This method involves the capture of a 2D planar target from which detected points can be compared to known metric pattern dimensions. Geometric calibration is achieved through a closed-form solution, followed by a non-linear refinement based on the maximum likelihood criterion. Of the 42 images captured for calibration, 2 were discarded due to partial pattern recognition not meeting a nominal minimum threshold (MinCornerMetric = 0.3), with the remaining 40 being used to calculate IO parameters and accompanying errors. The 2D board distortion profiles (Figure 5) and distortion models (Figure 6) can be seen below, with the resulting IO parameters compared in

Table 3. IO parameters for WWP obtained with the 2D board using MATLAB, 3D Field using Agisoft Metashape, and 3D Field using Australis.

Coefficient	Intrinsics	2D Board (MATLAB)		3D Field (Agisoft Metashape		3D Field (Australis)
		Value	$\mathbf{\sigma }$	Value	$\mathbf{\sigma }$	Value	$\mathbf{\sigma }$
$k_2$	f (mm)	13.021	0.032	13.038	0.015	13.034	0.014
	$c_x$ (pix)	311.219	1.821	312.585	0.626	312.926	0.740
	$c_y$ (pix)	255.558	1.928	257.755	0.614	257.921	0.721
	$k_1$	0.044	0.009	−0.043	0.004	−0.039	0.003
	$k_2$	0.005	0.054	0.368	0.014	0.347	0.013
	$p_1$	−0.000	0.001	−0.001	0.000	0.000	0.000
	$p_2$	0.001	0.001	0.000	0.000	−0.000	0.000
	MRE (pix)	0.50		0.16		0.12
	${\mathit{S}}_{\mathit{o}}$ (mm)	3.20		3.25		2.06
$k_3$	f (mm)	13.017	0.032	13.033	0.015	13.030	0.014
	$c_x$ (pix)	311.232	1.823	312.541	0.623	312.908	0.730
	$c_y$ (pix)	255.482	1.926	257.741	0.612	257.940	0.711
	$k_1$	0.055	0.017	−0.031	0.008	−0.031	0.008
	$k_2$	−0.177	0.247	0.247	0.078	0.270	0.073
	$k_3$	0.791	1.047	0.350	0.221	0.214	0.204
	$p_1$	−0.000	0.001	−0.001	0.000	0.000	0.000
	$p_2$	0.001	0.001	0.000	0.000	−0.000	0.000
	MRE (pix)	0.50		0.16		0.12
	${\mathit{S}}_{\mathit{o}}$ (mm)	3.65		3.25		2.08

.

2.3.2. 3D Field (Agisoft Metashape)

As the Metashape GUI does not allow for purely marker-based image orientation, several procedures were implemented to undertake a self-calibrating bundle adjustment prioritising marker 2D coordinates for the 3D field. Firstly, nominal values for the focal length and pixel size were provided, with redundant calibration parameters fixed at zero (i.e., $b_1$, $b_2$, and extraneous k coefficients). Secondly, images were oriented in a standard photogrammetric workflow, including weighted 2D marker coordinates as part of the initial bundle adjustment. Once all images had been oriented, all tie points were removed (apart from one to allow for optimisation) and cameras were optimised in an additional self-calibrating bundle adjustment. This allowed for the markers to be the predominant information for the bundle adjustment as opposed to ambiguous tie points generated from local features. The 3D field distortion profiles (Figure 5) and distortion models (Figure 6) can be seen below, with the resulting IO parameters compared in Table 3.

2.3.3. 3D Field (Australis)

Finally, Photometrix’s Australis was included to provide a robust metrological solution for calculating the IO parameters of the WWP. Australis’ workflow was implemented with a series of spatial resections and bundle adjustments undertaken to orient all images using the weighted 2D marker coordinates. Once completed, an additional free network self-calibrating bundle adjustment was undertaken to refine the initial IO parameters. Of the 37 images used for calibration, all featured a minimum of eight intersecting rays per marker, with the convergence limit and number of bundle adjustment iterations set at 0.005 and 12, respectively. The results generated by Australis for the 3D field, including distortion profiles (Figure 5), distortion models (Figure 6), and IO values (Table 3) can be seen below.

To assess the performance of each calibration approach, the standard error of IO parameters ($\sigma$), Mean Reprojection Error (MRE) (pix) in 2D image space, and standard error ($S_O$) (mm) of camera poses in 3D object space are used [7]. The results in Table 3 show notable differences between the 2D board and 3D field calibration approaches. Firstly, it must be noted that the scale of the 2D board (A3-sized) and resulting camera networks are significantly smaller then the 3D calibration targets (∼4×), with the derived MRE and $S_O$ being heavily scale-dependent. MATLAB calculates a focal length ∼13 µm smaller than the 3D field approaches, which show comparable focal lengths across all four distortion coefficient tests (all within 8 µm). This is likely a product of insufficient camera-to-object distance variation in the 2D board network geometry, with the planar target providing less depth in feature points and individual camera positions [33]. In addition, the higher intrinsic standard errors are largely due to the ambiguous extraction of checker-board corners (MRE = 0.50 pix) propagating into greater IO imprecision. Most significantly, the results generated by the 2D board show significantly different distortion models from that of the 3D field calibrations (Figure 6), the culmination of higher IO standard errors, poor feature point extraction, and the difficulty in filling the image format effectively whilst maintaining preferable network geometry.

In contrast, the observable similarities in the IO values, distortion models, and MRE from both Agisoft Metashape and Australis are expected, utilising the same 2D image coordinates for their respective bundle adjustments and obtaining marginally different radial and tangential profiles (Figure 5). Both software packages determined similar IO values across calibration tests, capturing the convex germanium lens geometry equating to a distortion of ∼0.25 pix on the right of the image format. Australis obtains the best MRE and $S_O$ results across both the $k_2$ and $k_3$ coefficients. Notably, Australis provides greater control over the self-calibrating bundle adjustment (inc. iterations, convergence limit, and minimum number of rays per marker), which, when coupled with native access to the covariance matrix, allowed for results to be refined for greater IO precision and metric accuracy. However, the validity of these metrics must not be viewed in isolation for the assessment of calibration processes, but taken into account when intrinsics are applied independently in subsequent photogrammetric applications.

2.4. Relative Orientation

The estimation of the relative pose between fixed stereo cameras determines the translation and rotation vectors necessary for mapping one image onto another. In the case of IRT-3DDF, this allows accurate geometries generated from high-quality RGB images to be textured or assigned temperature information from proximal TIR images. For the comparison of 2D board and 3D field calibration targets, two separate baselines were assessed. Firstly, as the Workswell WIRIS Pro thermal camera features both a infrared sensor and a visible sensor within the same camera housing, the baseline between both internal sensors was determined. However, as the resolution of the VIS sensor is sub-optimal for detailed modelling (1920 × 1080 pix), a stereo rig fixing the WWP and RGB2 was assembled to acquire additional data for Method 2. For each calibration set-up, as with the calculation of the IO parameters, baselines were determined with two or three radial distortion coefficients to assess their importance in determining precise and reliable RO parameters for IRT-3DDF (Figure 7).

2.4.1. 2D Board (MATLAB)

MATLAB’s Stereo Camera Calibrator App (2024a) was used to compute the 2D board calibration. Firstly, IO values for the Workswell WIRIS Pro sensors (WWP and VIS) and RGB2 were re-generated using the same processes outlined in Section 2.3.1. These values are subsequently used as fixed values for relative pose estimation. MATLAB’s Stereo Camera Calibrator is limited by its inability to calibrate two cameras of different resolution, meaning that imagery and associated IO values (i.e., $f_x$, $f_y$, $c_x$, and $c_y$) for VIS and RGB2 had to be resampled to that of the WWP to accommodate stereo calibration (ultimately rescaled upon completion). For the VIS-WWP camera and RGB2-WWP stereo rig, 37 and 20 image pairs were processed, respectively. The estimated relative translation and rotation values for each stereo pair can be seen in

Table 4. RO parameters for WWP, VIS, and RGB2 using the 2D board and 3D field calibration targets.

Coefficient	Baseline	Parameters	2D Board (MATLAB)		3D Field (Agisoft Metashape)		3D Field (Australis)
			Value	$\mathbf{\sigma }$	Value	$\mathbf{\sigma }$	Value	$\mathbf{\sigma }$
$k_2$	VIS-WWP	X (mm)	−41.530	0.191	−39.738	0.396	−40.271	2.691
		Y (mm)	−1.992	0.149	0.409	0.363	−0.148	2.845
		Z (mm)	0.309	0.114	1.142	0.219	6.124	1.456
		$\omega$ (°)	−0.080	0.013	−0.092	0.008	−0.175	0.073
		$\varphi$ (°)	0.579	0.016	0.717	0.009	0.701	0.071
		$\kappa$ (°)	−0.080	0.010	0.001	0.005	0.000	0.021
		MRE (pix)	0.76		0.55		0.22
		${\mathit{S}}_{\mathit{o}}$ (mm)	4.48		3.20		2.32
	RGB2-WWP	X (mm)	−184.464	0.331	−183.659	0.629	−183.084	1.493
		Y (mm)	−0.216	0.241	1.501	0.578	1.242	2.023
		Z (mm)	6.925	0.252	9.737	0.501	11.154	1.276
		$\omega$ (°)	0.917	0.015	0.586	0.015	0.424	0.080
		$\varphi$ (°)	3.644	0.020	3.243	0.015	3.208	0.066
		$\kappa$ (°)	−0.046	0.015	−0.147	0.014	−0.175	0.296
		MRE (pix)	0.87		1.10		0.14
		${\mathit{S}}_{\mathit{o}}$ (mm)	5.26		2.17		0.77
$k_3$	VIS-WWP	X (mm)	−41.503	0.191	−39.721	0.395	−40.540	2.889
		Y (mm)	−1.990	0.149	0.406	0.362	−0.614	2.895
		Z (mm)	0.371	0.114	1.272	0.219	−1.338	1.094
		$\omega$ (°)	−0.080	0.013	−0.092	0.008	0.025	0.075
		$\varphi$ (°)	0.579	0.016	0.713	0.009	0.715	0.072
		$\kappa$ (°)	−0.080	0.010	0.001	0.005	−0.007	0.019
		MRE (pix)	0.76		0.55		0.29
		${\mathit{S}}_{\mathit{o}}$ (mm)	4.48		3.20		3.54
	RGB2-WWP	X (mm)	−184.446	0.331	−183.644	0.629	−183.143	1.495
		Y (mm)	−0.215	0.241	1.490	0.578	1.064	2.143
		Z (mm)	7.017	0.252	9.340	0.502	11.132	1.263
		$\omega$ (°)	0.917	0.014	0.565	0.015	0.418	0.085
		$\varphi$ (°)	3.644	0.020	3.267	0.015	3.213	0.068
		$\kappa$ (°)	−0.046	0.015	−0.146	0.014	−0.154	0.311
		MRE (pix)	0.87		1.10		0.14
		${\mathit{S}}_{\mathit{o}}$ (mm)	5.26		2.11		0.76

, showing the pose of the Workswell WIRIS Pro thermal sensor with respect to the VIS/RGB2 sensor.

2.4.2. 3D Field (Agisoft Metashape)

For the relative pose estimation using Agisoft Metashape, using a different 3D field than that used for the calculation of the initial camera parameters (Figure 3), IO values for all sensors and cameras were re-calculated using the approach outlined in Section 2.3.2. Derived IO parameters were subsequently used as fixed values for Agisoft Metashape’s Multi-Camera System for the VIS-WWP sensors and RGB2-WWP stereo rig. Much like the IO method, ambiguous tie points were removed after initial orientation, with an additional self-calibrating bundle adjustment undertaken to refine RO values. Results were then scaled using a collection of scale bars allowing error metrics to be derived. For the VIS-WWP camera and RGB2-WWP stereo rig, 37 and 31 image pairs were processed, respectively. Results for both baseline tests can be seen in Table 4, similarly showing the mapping of the WWP onto VIS/RGB2.

2.4.3. 3D Field (Australis)

For the calculation of RO parameters in Australis, the initial IO values for all sensors were obtained through the processes outlined in Section 2.3.3. Subsequently, image pairs were imported into Australis for stereo calibration with the same 2D marker coordinates and constraints as those used in Agisoft Metashape. A self-calibrating bundle adjustment was then undertaken to obtain the extrinsic parameters of the cameras, with scale bars used to scale the scene, derive the RO parameters, and calculate error metrics. Results for both baseline tests in Australis can be seen in Table 4.

From the results in Table 4, an indication of the overall performance of the derived RO parameters can be determined, with the translation (X, Y, Z) and rotation ($\omega$, $\varphi$, $\kappa$) of the TIR sensor with respect to the reference sensor (VIS or RGB2). Here, $XYZ$ are aligned with the image coordinate system (x, y), where X is aligned with x pointing right, Y is aligned with y pointing down, and Z points forwards from the perspective centre. To gain some understanding of the performance of relative pose estimation, the baselines (horizontal displacement X in camera space) between the fixed sensors was utilised. According to Workswell s.r.o, the baseline between the centre of both lenses within the Workswell WIRIS Pro is 40.7 mm. Similarly, the measured baseline between the lenses of the RGB2 and WWP stereo rig was 184 mm, with a depth offset of 10 mm. All results achieve a reasonable estimation for the baseline (X) between the VIS-WWP and RGB2-WWP sensors, with MRE mostly <1 pix and 3D $S_O$ < 6 mm for all tests. Expectedly, a greater proportional range is present within the translation values and $S_O$ for the VIS-WWP across each software package and distortion coefficients than that of the RGB2-WWP. This is largely due to the shorter baseline and poorer resolution of the VIS sensor with respect to RGB2. However, the RO results for both 3D field tests show markedly similar results for the stereo rig baseline, although the $\sigma$ values for the Australis captured within the RO results are notably higher than both MATLAB and Agisoft Metashape across all results. Whilst the derived RO parameters and error metrics suggest adequate performance for all calibration processes, their application for IRT-3DDF is needed to determine the value of the calibration processes for specific data fusion and heritage-specific applications.

3. InfraRed Thermography 3D Data Fusion (IRT-3DDF)

The following section outlines the application of the derived IO and RO parameters for the documentation of architectural and cultural heritage using IRT-3DDF. Method 1 executes a combined TIR-RGB bundle block adjustment using the IO parameters obtained in Section 2.3 as fixed camera intrinsics, utilising multi-modal image matching to generate correspondence across image modalities. Method 2, a relative pose implementation, applies the RO parameters from Section 2.4 to derive a textured 3D thermal model of medieval wall paintings. Furthermore, insights into the radiometric calibration, correction, and verification of thermal infrared cameras are highlighted, although the focus of this section remains on geometric calibration and resulting network geometry.

3.1. Method 1: Combined TIR-RGB Bundle Block Adjustment

For the evaluation of the IO parameters obtained in Section 2.3, Lenton Lodge Gatehouse, a 19th century Grade ${\mathrm{II}}^{*}$ listed property in Nottingham, UK, was used as a case study (Figure 8). Built by Jeffry Wyatville and largely comprising Mansfield sandstone, the structure represents one of the finest examples of Elizabethan Revival architecture in England. Currently, Lenton Lodge Gatehouse shows visible signs of material degradation, notably around the parapet, decorative traceried panels, and turrets. Facing East-South-East, thermal images were acquired in January 2024 between 10:30 and 11:00 a.m. using passive thermography, maximising the baking of the structure from solar radiation and minimising shadows on the façade. A total of 20 WWP images were captured in a single strip with a Ground Sample Distance (GSD) of 9.81 mm, whilst 25 RGB1 images were captured in two strips with GSDs of 1.03 mm and 1.29 mm. Importantly, imagery from the WWP and RGB1 were captured independently with no fixed relative orientation, meaning no consistent baseline between the cameras can be assumed.

Additionally, a network of control points, utilising the custom markers from the 3D calibration test fields and additional natural points, were surveyed using a Leica, Heerbrugg, country: Switzerland TS10 total station to provide four ground control points (GCPs) and four check points (CPs) for subsequent referencing, scaling, and metric evaluation. The WWP was radiometrically calibrated by Workswell s.r.o prior to the execution of the imaging survey, in line with recommendations for routine servicing provided by the manufacturer. Preliminary lab experiments determined a 45 min acclimatisation time was needed for the IR sensor and CPU to generate stable temperature readings, in line with similar findings using the WWP [42]. Furthermore, non-uniformity corrections (NUCs) were undertaken at 3 min intervals during the survey to maintain consistent temperature values [43]. Finally, thermometer readings on the custom markers were taken as ground truth for quantitative temperature verification, comparing known and derived 3D thermal model temperature values.

With the proliferation of deep-learning approaches for both mono- [44] and multi-modal [45] image matching, the possibility of using ‘out-the-box’ neural networks for photogrammetric thermal modelling is becoming a reality for IRT-3DDF [46]. Therefore, to explore multi-modal image matching between TIR and RGB images, Deep-Image-Matching (DIM), an open-source Python library, Deep-Image-Matching (DIM) (v1.0.0) designed for robust multi-view image matching, was utilised [47]. DIM provides a modular workflow to configure and customise both hand-crafted and deep-learning feature matchers, allowing correspondence to be exported for seamless integration into various photogrammetric software. To facilitate effective multi-modal matching, WWP images underwent a histogram normalisation and UnSharp Masking to enhance detectable features. Additionally, RGB images were downsampled from their original resolution (7952 × 5304 pix) to ${\displaystyle \frac{1}{8}}$th size (994 × 663 pix) to more closely mirror the resolution of the WWP images. For all WWP and RGB images, 8000 keypoints were detected using a SuperPoint [48] feature extractor and LightGlue feature matcher [49] (Sp + Lg) (Figure 9). Utilising DIM, all 45 images were matched exhaustively, resulting in a possible 990 image pairs across the dataset.

For the combined TIR-RGB bundle block adjustment, the generated database from DIM was imported into COLMAP for image orientation [50]. Here, each block of WWP and RGB1 images were assigned a distinct camera model, using the IO parameters from the TIR geometric calibration in Section 2.3. To keep the focus of the experiments on the performance of the TIR geometric calibration (independent variable), intrinsics for RGB1 were calculated through self-calibration using the 25 images of the Lenton Lodge Gatehouse and held constant for each calibration subset. Importantly, IO values for the RGB1 focal length and principal point were divided by eight to match the resolution of the downsampled images provided to DIM. For each calibration set-up, an independent bundle adjustment was undertaken, fixing the IO values for both the WWP and RGB1 to assess the accuracy of the geometric calibration. Finally, to determine the possibility of providing no initial IO values for a combined TIR-RGB bundle block adjustment, a self-calibration approach using two radial distortion coefficients was undertaken wherein camera parameters were refined during the bundle adjustment. For all cases, derived sparse 3D tie points and camera poses were imported into Agisoft Metashape for metric evaluation (Figure 10).

Results for the combined TIR-RGB bundle block adjustment are listed in

Table 5. Results of the combined TIR-RGB bundle block adjustment for the Lenton Lodge Gatehouse, separating results based on the software and distortion coefficients utilised.

	Parameters	RMSERGB1 (mm)	MREWWP (pix)	RMSEWWP (mm)
				X	Y	Z	TOT
Self-Calibration	$k_2$	2.86	0.32	4.02	16.31	9.03	19.07
MATLAB	$k_2$	2.59	0.40	3.88	14.11	10.55	18.04
	$k_3$	2.60	0.31	3.98	12.39	8.63	15.62
Agisoft Metashape	$k_2$	2.44	0.27	2.72	5.74	6.12	8.82
	$k_3$	2.47	0.28	2.45	6.91	5.69	9.28
Australis	$k_2$	2.32	0.29	3.05	5.44	6.35	8.89
	$k_3$	2.19	0.29	3.06	5.60	6.28	8.95

, separating RGB1 and WWP image blocks into their respective modalities for metric evaluation. For the WWP blocks, Root-Mean Squared Error (RMSE) on the CPs is separated into horizontal (X), depth (Y), and vertical (Z) components, with the RMSE_WWP TOT representing the total RMSE of results in 3D space. Additionally, the RMSE of the RGB blocks (RMSE_RGB1) and the MRE of the WWP blocks (MRE_WWP) are included for a comprehensive assessment of the combined TIR-RGB bundle block adjustment and camera intrinsics. Firstly, the results of the WWP image blocks show notable differences between calibration approaches, with the RMSE_WWP TOT obtained from both 2D board calibration tests roughly twice that of the 3D field results. Calculating a focal length ranging from 13 to 16 µm less than that obtained from the 3D field methods (Table 3), coupled with an inability to capture the unique distortion of the WWP’s germanium lens (Figure 6), the 2D board shows significant error in the depth (Y) component of the RMSE_WWP. Consequently, the results obtained from the the 2D board calibration targets are similar to that of COLMAP where no prior calibration is undertaken and nominal IO values are corrected via ‘on-the-job’ self-calibration. The poorer MRE_WWP and RMSE_WWP of the 2D board using MATLAB is likely due to the scale of the calibration network geometry being insufficient for upscaling to the 3D scene and the imprecision within the calibration rendering IO values that are heavily scene-dependent [22].

In contrast, the results of the 3D fields using both Agisoft Metashape and Australis show positive results, obtaining similar MRE_WWP values for the combined TIR-RGB bundle block adjustment, and all RMSE_WWP TOT values are in line with the GSD of the WWP block (GSD = 9.81 mm). Agisoft Metashape achieved the best RMSE_WWP TOT results with two radial distortion coefficients, with Australis achieving comparable results with the best RMSE_RBG1 when three radial distortion coefficients were used. Interestingly, the MRE_WWP and RMSE_WWP results obtained using either two or three radial distortion coefficients show little discernible difference, which, when combined with the similar distortion models and profiles (Figure 5 and Figure 6), suggests that for germanium lenses where no complex distortion is present, either two or three distortion coefficients could be used for effective TIR geometric calibration.

Furthermore, as each combined TIR-RGB bundle block adjustment featured the same RGB1 IO values and multi-modal correspondence, all RGB blocks present similar RMSE_RGB1 values of <3 mm, with mono-modal matches (those between RGB images) from Sp + Lg allowing for successful orientation of the downsampled RGB block. However, a strong correlation is also apparent between the WWP (RMSE_WWP TOT) and RGB1 (RMSE_RGB1) image blocks (r = 0.86), suggesting the influence of the WWP (by virtue of the fixed IO values) on multi-modal image orientation. Similarly, the MRE_WWP across all tests shows notable correlation, closely mirroring the RMSE_RGB1 and RMSE_WWP values. This reiterates the suitability for image and object space metrics to be utilised in tandem for the assessment of the combined TIR-RGB bundle block adjustment and camera intrinsics [51].

The results obtained with COLMAP, undertaking simultaneous IO and EO via ‘on-the-job’ self-calibration, proves the benefit that can be provided by effective geometric calibration for sensor fusion pipelines [34]. Given the nominal focal length and image centre as starting IO values, the results of the combined TIR-RGB bundle block adjustment perform worse than if successful pre-calibration is undertaken, suggesting the benefit that can be gained from calibration using multi-modal correspondence. Notably, RMSE_WWP TOT were the same for when nominal values ($f_x$, $f_y$, $u_o$, and $v_o$) or full IO values from the 3D field ($f_x$, $f_y$, $c_x$, $c_x$, $k_1$, $k_2$, $p_1$, and $p_2$) were refined, with the bundle adjustment deriving the same IO values for both the WWP and RGB1 regardless of the initial starting point. The resulting 3D models, generated from the RGB1 images and textured with WWP thermal images, can be seen in Figure 11.

3.2. Method 2: Relative Pose Implementation

For the evaluation of the RO parameters, Castello Di Arco, located in the Sarca Valley of Trentino, northern Italy, was chosen as a case study. The 14th century medieval frescoes, depicting aristocracy and court life, have undergone several phases of investigation and restoration, largely due to water ingress and plaster delamination caused by the castle’s rock wall foundations (Figure 12). In a bid to reduce this phenomena, a space heater has been utilised to actively heat the façades after heavy rain and reduce the impact of the moisture on the frescoes. Whilst active IRT provides established non-destructive testing (NDT) techniques for cultural heritage wall paintings [52], to determine the extent of the deterioration and assess the current conservation practices, IRT-3DDF was employed to localise visible signs of delamination and discolouration with quantitative temperature differences. Therefore, to assess the impact of current conservation practices, the imaging survey was undertaken after active thermography, with the space heater removed just prior to data acquisition. The survey was undertaken in October 2024 after successive days of dry weather (avg. temp. 13 °C). The fresco corner, facing North-East, was subject to solar radiation from 06:52 (sunrise) until the commencement of the survey (10:45–11:30 a.m.).

To utilise the findings generated in Section 2.4, stereo results were used from both the in-built thermal camera and purpose-built stereo rig baselines. For both baselines (VIS-WWP and RGB2-WWP), 40 stereo pairs were captured of the fresco corner and adjoining walls (Figure 12). Captured at a distance ∼2.5 m away from the frescoes, the GSD for the WWP, VIS, and RGB2 sensors were 3.3 mm, 2.0 mm, and 0.3 mm, respectively. As previously stated, the benefit of utilising known RO parameters is essential in cases where no discernible thermal contrast can be identified or generated through active thermography. For Castello Di Arco, presenting a temperature range of ∼3 °C, the derived thermal images presented low contrast and high noise, meaning few suitable correspondences could be used for feature-based matching (Figure 13). Initial tests proved the inability for WWP images to be oriented mono-modally, necessitating RO parameters with a reference camera/sensor for effective modelling. To mitigate these challenges, a network of the 3D field custom markers, actively heated and positioned in front of the walls (Figure 12), provided manual tie points to aid in the orientation of the WWP block and generate multi-modal correspondence [10]. To generate control for the resulting models, measured scale bars were taken between positioned markers, with two used as control and two reserved for validation. To undertake the modelling, Agisoft Metashape’s Multi-Camera System was utilised, with VIS/RGB2 set as the ‘primary’ camera (geometry) and the WWP set as the ‘secondary’ camera (texture). IO values were fixed from the initial calibration process (Section 2.4), with the X, Y, and Z translations and $\omega$, $\varphi$, and $\kappa$ rotations input as initial constraints for the bundle adjustment. In addition, 2D image coordinates (custom markers and tie points) were both weighted at 1.0 pix and 1.5 pix for the RGB2-WWP and VIS-WWP baselines, respectively.

Table 6. RO results for Castello Di Arco’s medieval frescoes, with the measured scale bar RMSE (mm) for each baseline (VIS-WWP or WWP-NGB2) separated into each image modality.

	Parameters	RMSE VIS-WWP (mm)		RMSE RGB2-WWP (mm)
		VIS	WWP	RGB2	WWP
Self-Calibration	$k_2$	2.43	4.30	1.46	1.42
MATLAB	$k_2$	1.49	5.04	2.20	8.72
	$k_3$	1.51	5.01	2.17	8.82
Agisoft Metashape	$k_2$	2.72	4.12	1.26	1.87
	$k_3$	2.74	4.24	1.26	1.89
Australis	$k_2$	2.25	3.83	1.26	1.58
	$k_3$	2.28	3.95	1.26	1.65

shows the results from each baseline calibration test specifying the distortion coefficients used for the WWP during calibration ($k_2$ or $k_3$), with the generated textured frescoes visible in Figure 14. The RMSE on the measured scale bars are separated for each modality to provide an assessment of the image orientation in each imaging domain. Additionally, as with the results reported in Section 3.1, a self-calibration process has been undertaken where no initial IO or RO parameters were provided, with IO, RO, and EO parameters obtained simultaneously via a self-calibrating bundle adjustment. For the VIS-WWP baseline, the success of the obtained RO parameters is evident, with all approaches able to orient the 40 stereo pairs successfully. For the sensors within the Workswell WIRIS Pro, calibration results are comparable across all calibration targets, obtaining similar RMSE_VIS-WWP results for the reference VIS sensor ranging from 1.5 to 2.7 mm. Interestingly, the 2D board obtained the best RMSE_VIS-WWP value amongst VIS blocks, showing the suitability of 2D boards for standard CV geometric calibration. Importantly, all results (apart from MATLAB) provide a better RMSE_VIS-WWP for the WWP image blocks than that of pure self-calibration, suggesting the superiority of pre-calibration for sensor systems with short baselines.

For the stereo rig (RGB2-WWP), all approaches achieve a similar RMSE_RGB2-WWP for the RGB2 image blocks, showing the suitability of each calibration approach (across different distortion coefficients) to obtain camera intrinsics for traditional digital cameras. However, notable differences for the 2D board calibration target using MATLAB are identifiable, with the IO and RO parameters leading to a markedly higher RMSE_RGB2-WWP for the WWP block of ∼9 mm. This is likely due to the different Field-of-View (FOV) between the fixed cameras during RO calibration, with the different focal lengths of the sensors (50 mm vs. 13 mm) making it more difficult to fill both image formats with the 2D board during simultaneous image capture [32]. However, we see comparable results from both 3D field calibration approaches, obtaining an RMSE_RGB2-WWP of <2 mm for the WWP image block. Significantly, Australis was able to obtain the lowest MRE, $S_O$, and RMSE for the WWP blocks in RO calibration methods across both distortion coefficient subsets. Much like the findings of Section 3.1, the results from both 3D calibration methods demonstrate the superiority of the 3D field (and accompanying software) to obtain precise and reliable IO values that are image invariant, scene-independent, and metrically accurate for photogrammetric reconstruction.

As self-calibration was used as a benchmark to determine the ability for IO, RO, and EO parameters to be obtained simultaneously in the same bundle adjustment, it is important to understand the performance of these results when compared to the RO pre-calibration processes. Firstly, the block RMSE values from the VIS-WWP baseline were favourable, establishing intrinsics in line with pre-calibrated values and an estimated baseline (X) of −41.2 mm (∼0.5 mm from the reference value). However, the results of the RGB2-WWP baseline, obtaining a better RMSE_RGB2-WWP for the WWP block than any calibration approach, generates a markedly different distortion model for the WWP (notably $c_x$, $c_y$ and $p_1$, $p_2$) and translation values (X = −187.8 mm; Z = −0.8 mm). This suggests the susceptibility for ‘on-the-job’ self-calibration to find any scene-dependent solution for IO and RO parameters that fit with the extracted tie points and included markers, especially when fewer than 200 tie points exist in the WWP block. This reiterates the importance of known IO and RO parameters for scenes where self-calibration is more impressionable, with parameters (and logical constraints) a necessary feature when insufficient correspondence is present or when modelled sensors present unique distortion profiles.

4. Discussion

4.1. TIR Geometric Calibration

4.1.1. Calibration Targets

As historical methods of TIR geometric calibration have been overly reliant on the manual construction of 2D boards, the ambition of this work was to determine a novel 3D test field design, preferable for photogrammetric camera calibration, which could be compared against established contemporary approaches. Critically, the success of all TIR geometric calibration approaches is in the ability for control points to be extracted with high contrast and low noise, with the greatest contributing factor being the emissivity of the materials used. As seen in Table 2, the difference in emissivity between the DiBond and UV-printing ink was notably small ($\Delta \epsilon =0.20$). This made it difficult to excite thermal contrast and extract accurate checker-board corners, resulting in uncertainty propagating into imprecise IO values and larger error values (i.e., IO $\sigma$, MRE, and $S_O$) (Table 3). Although scale, network geometry, and calibration set-up still play a role, material emissivity is likely the largest contributing factor for the poor calibration and IRT-3DDF results obtained with the 2D board, demonstrating the importance of material choice and emissivity calculation as part of effective TIR geometric calibration.

In contrast, the emissivity difference for the 3D test field provided well-defined crosshairs with high contrast throughout the entirety of the imaging surveys ($\Delta \epsilon =0.65$). Highly-configurable in terms of quantity, geometry, scale, and setting, the novelty of the proposed custom markers is characterised by their suitability as (1) crosshair targets for 3D test fields; (2) additional tie points for photogrammetric multi-modal image matching; (3) GCPs and CPs for photogrammetric evaluation; and (4) known points for quantitative temperature verification. However, the proposed method is still limited by the manual identification of markers within the imaging surveys as opposed to fully automatic feature detection. Roshan et al.’s [15] 2D ChArUco checker-board provides a logical progression of the current thermal markers, with milled aluminium coded markers as a novel approach for both future calibration and survey applications.

However, although coded markers [15]; distinctive shapes [53]; speckles [54]; actively heated elements [9]; and mono- or multi-modal local features [40] may provide avenues for future exploration, the ability for these to be observed by commercial thermal cameras with the inherently low resolution (typically 180 × 120–640 × 480 pix), common for three-dimensional thermal imaging systems (TTISs) incorporating multiple image- and range-based sensors [55], remains an open challenge. Finally, whilst the use of active thermography provided the necessary contrast for effective control point identification in both calibration targets, the considerable limitations of existing and alternate active thermography approaches include the following: the difficulty in constructing electrical targets, the inaccuracy of control point extraction, the inability to take devices into the field, and the need for repeated heating cycles [14,54].

4.1.2. Interior Orientation

The derived IO parameters, obtained by comparing calibration targets, the number of distortion coefficients, and software packages, point to several considerations for TIR geometric calibration. As previously highlighted, the quality of the 2D board drastically affects the derived intrinsics, incapable of capturing the distortion of the germanium lens and producing significantly larger IO $\sigma$, MRE, and $S_O$ values (Table 3). In addition, whilst an initial set of 20 images were captured for interior orientation using MATLAB, an additional 20 (totalling 40 images) were needed to refine IO values within expected values (as seen in Dlesk et al. [23]). Although MRE and $S_O$ errors obtained during calibration would indicate agreeable performance of the 2D board, these metrics cannot be viewed in isolation, with the comparison and application of IO values in Section 3.1 demonstrating the inability for the 2D board to achieve precise TIR camera intrinsics for scene-independent photogrammetric use. As a result, even though 2D boards are established as a method for TIR geometric calibration, the susceptibility of projective coupling from sub-optimal calibration network geometry, the tendency for derived IO values to be image-variant, and the observed scene-dependency of calibration processes means that the use of 2D boards should be limited for metrically accurate IRT-3DDF.

In contrast, the 3D field, presenting a strong configuration of multi-level points, was able to capture the unique distortion of the WWP optics with either two or three radial distortion coefficients. The acknowledged benefits of a 3D calibration field, including the ability for the image format to be properly filled, the greater scale and depth information present, and the high convergence between point observations, allowed for more precise and reliable estimates of IO values. The comparable performance of both Agisoft Metashape and Australis software packages, both in terms of distortion profiles (Figure 5) and distortion models (Figure 6), is likely due to the identical camera network geometry, 2D marker coordinates, and associated constraints. However, for these results to be achieved, the inability for the Agisoft Metashape GUI to run a purely marker-based bundle adjustment required the removal of ambiguous tie points in the ‘textureless’ calibration target after initial alignment and for an additional optimising bundle adjustment to be undertaken. Therefore, when deciding between software packages, considerations between automation and accuracy must be made in accordance with the camera network for TIR geometric calibration, with Australis allowing for greater access to the self-calibrating bundle adjustment and accompanying parameter correlations (Australis was the only software to provide access to both IO and IO/EO covariance matrices for the assessment of intra- and inter-image projective coupling). Finally, regarding the choice of distortion coefficients or software package, there is little in the IO or RO calibration performance metrics or IRT-3DDF results that would suggest a preference for two or three radial distortion coefficients; however, this will largely be TIR camera- and calibration dependent.

For the self-calibration results in Table 5, the influence that imprecise intrinsics can have for multi-modal image matching is apparent. The obtained principal point for the self-calibration presented a difference from calibrated IO values in $c_x$ and $c_y$ of ∼10 pix and 4 pix, respectively; which, even though capable of determining a similar distortion profile to the 3D field approaches, resulted in expectedly higher RMSE_TIR TOT [56]. As discussed in Section 3.1, the resulting IO values from the combined TIR-RGB bundle block adjustment self-calibration were practically the same when either nominal- (simple pinhole camera model), full- (OpenCV camera model), or pre-calibrated IO values were refined [50]. This demonstrates a considerable opportunity to understand the influence of bundle iterations and initial intrinsics on multi-modal image orientation. Therefore, the exploration of on-site self-calibration [57], utilising varying initial camera intrinsics and constraints, is a logical progression for Section 3.1 [34].

Finally, the difference in TIR intrinsics obtained for Section 2.3 and Section 2.4 must be noted, stressing the need for TIR cameras to be regularly calibrated prior to photogrammetric use (IO and RO calibration experiments were undertaken six months apart). For example, an observable offset in the principal point and tangential distortion coefficients was present for WWP IO values used in the Section 3.1 and Section 3.2. This phenomena could be caused by mechanical stress or component instability within the Workswell WIRIS Pro, a possible side effect of rotational movements during calibration or camera handling and transportation [7]. Furthermore, the influence of sensor temperature and ambient conditions during geometric calibration can be seen to have an influence on the derived TIR images, causing an observable increase in temperature values within the obtained calibration datasets (Figure 3). As for the sensor, camera housing and environmental temperatures have an established impact on resulting camera intrinsics [58] and radiometric accuracy [42]; the analysis of this effect on FPA sensors (detecting the very radiation that will be causing instability) warrants future investigation.

4.1.3. Relative Orientation

The results of the relative pose estimations in Section 2.4 and Section 3.2 demonstrate the success of the proposed calibration approaches and the importance of pre-calibrated IO and RO parameters for IRT-3DDF. Firstly, the results for the 2D board provided an agreeable estimation of the relative pose for both baselines, notably for the VIS-WWP baseline where the focal lengths and FOVs of the VIS-WWP image pairs made filling the image format more achievable. However, the results from both 3D field calibration approaches demonstrate the superiority of a strong camera network and multi-level calibration target for TIR relative pose estimation, with both Agisoft Metashape and Australis achieved successful results when IO and RO parameters were applied (notably for the RGB2-WWP stereo rig <2 mm for all image blocks). Importantly, both Section 2.4 and Section 3.2 show the capability for RO parameters to be generated across varying baselines and sensors, critical for implementation within IRT-3DDF where a vast array of resolutions and TTISs is present.

The importance of pre-calibration in Section 3.2 is evident, where self-calibration was highly unsuccessful in obtaining simultaneous IO, RO, and EO parameters in a low-contrast setting. Even though IO values for VIS and RGB2 were in line with pre-calibrated values, both VIS-WWP and RGB2-WWP results failed to provide reliable IO values for the WWP, with VIS-WWP results determining a focal length ∼100 µm from pre-calibrated values and RGB2-WWP results achieving a principal point (282, 243) significantly off from the image centre (320, 256). Clearly, Agisoft Metashape’s Multi-Camera System calculates IO values for the ‘primary’ camera as a priority, fitting the ‘secondary’ camera within the remaining constraints. In addition, altering the weights given to WWP tie points and user-defined coordinates of the custom markers, imperative for generating multi-modal correspondence, provided notable differences in obtained IO and RO parameters, likely due to the extremely low number of tie points in the WWP image block (No._WWP < 200). Therefore, additional investigations assessing the stability of relative pose estimations for IRT-3DDF via ‘on-the-job’ self-calibration are needed, notably the inclusion of the following: (1) initial estimates and constraints for IO parameters; (2) initial estimates and constraints for RO parameters (most likely including horizontal displacement in X) [31]; and (3) greater TIR mono- and TIR-RGB multi-modal tie points (in a similar vein to Section 3.1).

4.2. InfraRed Thermography 3D-Data Fusion (IRT-3DDF)

Whilst the use of IRT as part of a broader catalogue of NDT techniques has long been established for architectural heritage [59], the presented examples of IRT-3DDF reveal opportunities where 2D imaging techniques can embrace synergies with 3D modelling practices [55]. The ability to generate rapid, real-time, and repeatable thermal models creates opportunities for the assessment of architectural heritage through the comparison of 3D thermal models over time, the localisation of thermal anomalies, and the quantification of material degradation. Similarly, the results from Castello di Arco, with known and observable discrepancies in the temperature of the medieval frescoes, point to possible interventions and areas for investigation. Firstly, it is evident that the use of a space heater to alleviate the effects of the water seeping through the wall has been insufficient to address a pressing issue in the upper corner, with a temperature difference of ∼1.5 °C from the fresco average still apparent. The use of IRT-3DDF helps demonstrate the extent of the temperature difference and the proximity of affected areas in relation to the space heater and identifies additional areas on the wall where conservation work should be considered. However, whilst the focus of this work has been on geometric calibration, for additional insights to be extracted, future emphasis on radiometric calibration, correction, and validation is needed for the implementation of effective IRT-3DDF [60].

The results from Section 3.1 also demonstrate the success of a novel deep-learning-based multi-modal image matching method for thermal modelling, representing a real opportunity for the development of IRT-3DDF by exploiting ‘out-of-the-box’ neural networks not trained on TIR images. This demonstrates the following: (1) the ability for feature correspondence to be determined across image modalities, (2) the strength of these points for simultaneous image orientation as part of a combined TIR-RGB bundle block adjustment, and (3) the increased automation of IRT-3DDF without a reliance on known survey points or manual registration. Similarly, the results in Section 3.2 represent a critical method for IRT-3DDF where insufficient temperature differences means little thermal contrast is visible within a given scene. The ability for scenes to exploit proximal sensors and reliable calibration procedures represents a critical opportunity for indoor thermal modelling, where the generation of contrast is a likely obstacle. However, the manoeuvrability and construction of the RGB2-WWP stereo rig provides considerable limitations in terms of data capture and synchronisation, likely exacerbated when the scene increases in scale and complexity. The findings from Section 3 demonstrate the benefit of using a high-resolution digital camera to generate the ‘base’ geometry for IRT-3DDF, with most commercial thermal cameras (and accompanying visible sensors) developed for building inspection as opposed to photogrammetry. Therefore, the logical evolution of this research should focus on combining the complementary methods of Section 3 for the establishment of TIR geometric calibration as a fundamental component of IRT-3DDF. This would further develop the novelty of multi-modal correspondence for orienting TIR and RGB images whilst utilising the proximal sensors within thermal cameras, helping mitigate environments where both mono- and multi-modal TIR correspondence may not be determined.

5. Conclusions

The determination of reliable IO and RO parameters for TIR cameras is critical for their subsequent use in photogrammetric processes. The performance of calibration procedures cannot be used as a proxy for their success in eventual 3D reconstruction, notably for data fusion methods utilising sensors of varying resolutions and modality. This work has comparatively evaluated existing 2D- and novel 3D calibration targets, detailing construction, calibration, and applications for IRT-3DDF. Results suggest the superiority of a 3D field for the estimation of TIR intrinsics and provide insights regarding appropriate camera parameters, software packages and and future refinements. In addition, the estimation of the relative pose between two baselines was determined to assess the accuracy of image orientation utilising in-built sensors from a singular thermal camera or a constructed stereo rig. The application of IO and RO parameters to two IRT-3DDF methods demonstrates the opportunities to be gained for NDT within architectural heritage exploiting thermal modelling. The first method, proposing a novel combined TIR-RGB bundle block adjustment utilising deep-learning multi-modal correspondence, demonstrates the capability of using neural networks generalised outside of their training domain. This presents an automatic data fusion method where simultaneous image orientation can be achieved without a reliance on GCPs or manual registration. The second method, utilising a fixed stereo rig for the generation of a textured 3D thermal model, provides a workflow demonstrating the importance of proximal sensors and effective RO parameters when low thermal contrast and few TIR correspondence can be determined. Future work will look to combine developed methods to exploit the resolution and flexibility of commercial digital cameras whilst utilising the fixed sensors within a thermal camera for instances when TIR images require additional support for successful image orientation.