Photometric Stereo Techniques for the 3D Reconstruction of Paintings and Drawings Through the Measurement of Custom-Built Repro Stands

Abstract

In the digital 3D reconstruction of the shapes and surface reflectance of ancient paintings and drawings using Photometric Stereo (PS) techniques, normal integration is a key step. However, difficulties in locating light sources, non-Lambertian surfaces, and shadows make the results of this step inaccurate for such artworks. This paper presents a solution for PS to overcome this problem based on some enhancement of the normal integration process and the accurate measurement of Points of Interest (PoIs). The mutual positions of the LED lights, the camera sensor, and the acquisition plane in two custom-designed stands, are measured in laboratory as a system calibration of the 3D acquisition workflow. After an introduction to the requirements and critical issues arising from the practical application of PS techniques to artworks, and a description of the newly developed PS solution, the measurement process is explained in detail. Finally, results are presented showing how the normal maps and 3D meshes generated using the measured PoIs’ positions, and further minimized using image processing techniques, which significantly limits outliers and improves the visual fidelity of digitized artworks.

1. Introduction

Since 2010, our research team has developed a solution for the digitization of, visualization of, and interaction with ancient drawings and paintings, addressed to various stakeholders who engage with knowledge, preservation, conservation, and communication activities regarding paintings and ancient drawings: museum visitors, art historians, restorers, professional operators, and other figures. While the initial research was focused on ancient drawings (mainly by Leonardo da Vinci), resulting in the successful application ISLe—InSight Leonardo [1], the approach was later extended to manuscripts as well as ancient paintings with the applications AnnunciatiOn App [2] and the recent GigaGuercino App. These outcomes follow a different trajectory from the usual answer given by the scientific community to the related issues, based on 2D outputs following two paths:

• Digital representations based on images with a very high density of spatial content (i.e., the so-called gigapixel images). This solution is well illustrated by the Rijksmuseum’s 2019 gigantic Operation Night Watch project, which was able to reproduce Rembrandt’s painting with a resolution of 5 µm using 717 billion pixels [3].
• Images from Dome Photography (DP) that can be used in three ways: (1) visualization of the surface behavior of the artwork through the interactive movement of a virtual light source over the enclosing hemisphere, i.e., the Reflectance Transformation Images (RTI) [4]; (2) a 3D reconstruction of the object surface; (3) the modeling of the specular highlights from the surface and hence a realistic rendering.

These representations of paintings or drawings are generally accurate in resolution but limited to the simple reproduction of the apparent color and only able to show the artwork from a single, predefined point of view, missing its three-dimensionality and reflectance properties. Paintings and drawings, instead, represent complex artistic creations; producing a faithful copy implies the reproduction of the thickness and reflectance of brushstrokes, pens, and pencils (which provide insights into the techniques employed by painters), the subtle nuances of their surfaces, the presence of craquelures (valuable for determining the preservation state), and the optical properties of the papers and painting materials [5].

To meet these requirements, the solution developed by our team is 3D-based and rendered in real-time, allowing for the following:

• The visualization of the artwork in a digital context that simulates the three-dimensional environment in which it is placed;
• The free exploration of the painting or drawing, allowing users to zoom in on details, to observe surface behaviors under changing lighting conditions and at different angles, and to manipulate the artifact in real-time ‘as in your hands’ [6];
• The reproduction of the shape and the optical properties of the materials that make up the artwork, i.e., their total appearance [7].

In practice, the artwork is represented as a 3D model in a virtual space, and the optical behavior of its surface is modeled by tracing it back to phenomena belonging to three different scales: the microscopic scale, which can be summarized for artwork as its color (diffuse albedo), brilliance, and transparency/translucency; the mesoscale, which describes the roughness of the surface (what can be called its 3D texture or topography); and finally the macroscopic scale, which can be described as the whole shape of the artifact [8].

Based on this modeling ranking, some software and techniques have been developed to return each scale correctly: nLights (version 0.0.9.1), a software based on Photometric Stereo (PS) [9], handles the mesostructure components; an analytically approximated Bidirectional Reflectance Distribution Function (BRDF) derived from the Cook–Torrance physical model and implemented via a shader allows for the microstructure to be reproduced [10,11]; and the SHAFT (SAT and HUE Adaptive Fine Tuning, version 2.2.6) program enables a faithful replication of the color [12]. Finally, its macrostructure can be obtained from time to time using different techniques. In some cases, it was assimilated to a simple plane [13]; in others, a PS-based solution was used, exploiting computer graphics techniques to correct outliers [14]; for the paintings, photogrammetric techniques were used as an efficient solution, as the literature confirms [15].

Recently, we planned to merge all these solutions to obtain an all-inclusive software, allowing for a workflow that is simple, accessible, and economically viable for institutions of varying sizes and resources, which is accurate and can be used not only by expert researchers, but also by professionals in the Cultural Heritage (CH) sector for the mass digitization of artworks [16]. A main goal of the new hardware/software system, with the aim of minimizing the complexity of the process and the negative effects of prolonged exposure of the artwork to the light, is the removal of double data collection techniques to reproduce surfaces and their optical behavior, such as photogrammetry techniques for the shape, and PS to extract optical reflectance properties [17]. Despite the well-known limitations of the PS techniques in shape reproduction, we based the whole process on these methods due to their superior ability to accurately reproduce surface reflections. Custom-developed software and independently designed and manufactured horizontal and vertical repro stands [18] allow for accurate results to be obtained and ensure a quick progression from acquisition to visualization (Figure 1 and Figure 2).

This paper delves into the solution developed for 3D reconstruction using PS techniques only, explaining the problems and fixes, mainly focusing on resolving the critical issues in the quantitative determination of surface normals. The assumption that the whole object is lit from the same illumination angle with the same illumination intensity across the entire field of view and the correct localization of the light sources, is a requirement rarely reached for a mismatch between the lighting model and real-world experimental conditions. When the surface normals are integrated, these inconsistencies result in incorrect surface normal estimations, where the shape of a plane becomes a so-called “potato-chip” shape [19,20,21]. In practice, as the literature shows [22], calibrating PS with n ≥ 3 images is a well-studied problem that can be resolved without resorting to integrability, but an error in the evaluation of the intensity of one light source is enough to cause bias [23] and outliers may appear in shadow regions, thus providing normal fields that can be highly nonintegrable. To remove dependence on this far light assumption, various techniques were developed but, in our opinion, these are not accurate enough for art conservation, where an accurate quantitative determination of the surface’s normals is crucial.

Determining the mutual position of components can be considered a system calibration operation and must be performed only once. In the following, the entire PS process is illustrated, as well as the refinements used to eliminate noise due to non-Lambertian surfaces, and shadows, and an approximate evaluation of light intensity and its attenuation. Mainly, the measurement process of the two different repro stands designed for capturing drawings and paintings (positioned horizontally and vertically) is presented.

The paper is organized in five main sections. After the Section 1, Section 2 begins with a state-of-the-art review of relevant techniques in PS, followed by a description of the developed PS software. The specifications and features of the developed stands conclude the paragraph. In Section 3, the metrological context is introduced, and the measurement approach for the ‘as built’ stands is illustrated. Section 4 presents the results and Section 5 sums up the key points and findings of the research and examines possible future works.

2. The Photometric Stereo Framework

2.1. State of the Art

Originally introduced by Woodham [9] in the context of computer-based image interpretation, PS is a well-known computer vision approach that is frequently used to recover the surface shape from the image intensity. According to the principle that the intensity of the reflected light depends on the angle of incidence of the light on the surface, the technique estimates the surface orientation at any point on the object’s surface as a normal vector. The original formulation assumed that lights were infinitely far away, the camera was orthographic, and the object surface was Lambertian and convex (i.e., no shadows or inter-reflections). With a perfect Lambertian surface and in the absence of noise, three intensity values from non-coplanar light sources are sufficient to solve for both normal direction and surface albedo. In practice, for noisy image data, better results are obtained by taking the median of the results for many triplets of light sources [24]. Traditional approaches [25,26,27] extract geometry from surface normals using gradient fields and three out of four lighting directions, where surfaces appear more Lambertian [28,29,30].

From the original introduction of PS, several researchers have attempted to generalize the technique to more realistic cameras, surfaces, and lighting models [31]. Belhumeur et al. [32] found that with an orthographic camera model and uncalibrated lighting, the surface of the object could be uniquely determined to within a bas-relief ambiguity. Papadhimitri and Favaro et al. [33] later suggested that this ambiguity is resolved using the perspective camera model. New techniques have been introduced based on non-Lambertian reflectance models [34,35,36,37] or sophisticated statistical methods to automatically filter out non-Lambertian effects [38,39]. The ‘bounded regression’ method [40], and the lowest median of squares for the regression joint with Gaussian radial basis functions are used to handle effects such as specular highlights and shadows [41].

Although often overlooked, issues relating to illumination are almost always the most important aspect to be considered when designing PS solutions. A 1% uncertainty in the intensity estimation will produce a 0.5–3.5-degree deviation in the surface normal orientation calculation [42]. Assumptions about parallel light and orthogonal projection often cause global shape deformation [43,44]. This deviation varies with surface properties and object dimensions. Several researchers [45,46] investigated removing the far-light assumption to improve the accuracy of PS. Others [47] considered further non-isotropic illuminations. Another direction of research on PS is in the study of more realistic lighting models to simplify the acquisition of data., e.g., methods have been developed to handle images acquired under nearby point light illumination [48], which finds a natural application in LED-based PS [49]. PS can be ill-posed when lighting is unknown (uncalibrated PS). The problem must be reformulated globally, and the integrability constraint must be imposed. But even then, a low-frequency ambiguity known as the generalized bas-relief ambiguity remains [32]: it is necessary to introduce additional priors, a problem for which various solutions have been proposed [43].

While PS can capture fine details even on non-collinear objects [50,51,52], a common objection to PS is that it is prone to a low-frequency bias that can distort the global geometry [53]. Such a bias usually results from a contradiction between the assumptions behind the image formation model and the actual experiments, e.g., assuming a directional light source instead of a nearby point source. From a practical point of view, it is easier to remove the low-frequency bias by coupling PS with another 3D reconstruction method, such as shape-from-silhouette [54], multi-view stereo [26], or depth sensing [55]. In such work, PS provides fine-scale geometric details that are combined with the coarse geometry provided by the alternative technique. A complete review of the literature, coupled with a clear illustration of the PS framework, is provided in [56].

2.2. The Adopted PS Solution

The software solution developed for PS is a customization of the MATLAB PSBox by Ying Xiong (Release v0.3) [57], designed to be coupled with the two custom stands built for the image capture phase. While the initial version of nLights (version 0.0.9.1) was based on four intensity values, the most recent version accommodates eight different light directions (N, S, E, W, at 45° and 15° relative to the acquisition plane), maintaining a fixed camera position perpendicular to the artwork’s surface. Redundant conditions are used to refine the results by progressively discarding the closest values. nLights produces the following as outcomes:

• Albedo map;
• Normal map;
• Depth map, by integration of estimated normal vector field;
• Reflection map generated as the difference in the apparent color with the albedo;
• Mesh with a resolution of a vertex for each pixel (i.e., 40 μm) exploiting the MATLAB functions surfaceMesh and meshgrid. In practice, for each pixel, a vertex is generated with coordinates x, y. The z depth is derived from the depth map, and finally a Delaunay triangulation generates the mesh. The mesh spatial density parameters can be adjusted through quadric decimation.

PSBox implements PS through a defined sequence of steps based on two series of images captured under the same sequence of illumination: the first set of images is a picture of a sphere; the second is the object to be digitized. The sequence is as follows:

• Circle fitting from manually selected points on a chrome sphere image;
• Light direction determination using the chrome sphere image;
• Light strength estimation and lighting matrix refinement through nonlinear least squares optimization;
• PS computation to generate albedo and normal maps;
• Depth map reconstruction through the integration of the estimated normal vector field.

In detail, PSBox calculates light source directions by analyzing specular highlights on a chrome sphere under the same light conditions as the artwork to be replicated, assuming an orthographic camera model. The function requires three key inputs: an image I of a specular chrome sphere, circle parameters defined by a 3 × 1 vector containing the sphere’s center coordinates and radius, and a threshold value that identifies specular highlights (e.g., 250 for 8-bit images). The algorithm processes these inputs to output a unit vector (3 × 1) representing the light source direction. This approach to the estimation of the light direction is the major drawback of the software. The parallel and uniform condition is violated using commonly available light sources, because it is impossible to have a distant lighting setup that implicitly requires a large space for the whole system. Moreover, since the radiance from the source at the surface falls off in accordance with the inverse square law, a longer working distance will tend to cause the radiance to drop rapidly, correspondingly decreasing the signal/noise ratio of the whole system. Secondly, the evaluation of the light position and direction using the sphere is usually inaccurate (2–3-degree errors are common). All these lacks determine the configuration for the representation of a flat surface resembling a “potato chip”.

Light strength in PSBox is estimated by solving a nonlinear least squares problem from M images that are indexed by i, and each image has N pixels that are indexed by j. The intensity of j-th pixel in i-th image is therefore denoted as I_ij. At j-th pixel, n_j denotes the surface normal, j denotes the albedo, and b_j = ρ_jn_j the scaled normal. Since each of the M images comes from different lighting directions l_i, the rendering function considering directional light plus the ambient component α can be expressed as follows:

$$I_{i,j}={\rho }_j\left(l_i^T{n}_j+{\alpha }_i\right)=l_i^T{b}_j+{\alpha }_i‖b_j‖$$

Therefore, the estimation of the whole scene property is an optimization problem that can be solved using the following:

$$\begin{array}{c}min\\ b_j\end{array}\sum _{i=1}^M{\left(I_{i,j}-l_i^T{b}_j-{\alpha }_i‖b_j‖\right)}^2$$

The PS technique is integrated into PSBox by grouping pixels with similar shadow patterns in different images, then solving a least squares system for each group to find scaled normal vectors, and finally separating the results into albedo (surface reflectivity) and unit normal vectors. The code handles shadows through the mask input, with numerical stability achieved when at least three valid measurements per pixel are found.

Normal integration from gradient fields exploits the Frankot and Chellappa method [58] based on Fourier transforms to regularize (i.e., to enforce the integrability of) the gradients in the frequency domain. PSBox then presents all the problems with this solution:

• Lack of precision at the border of rectangular domains, if the boundaries are not constrained;
• Inaccuracies for very low frequencies, although the photometric gradients provide a good representation of the spatial frequencies in the surface, right up to the Nyquist frequency. Errors can result in ‘curl’ or ‘heave’ in the base plane [59].

Moreover, in the PSBox pipeline, the surface normals are estimated at first and then integrated into a height map. This strategy is, however, suboptimal, since any error in the normal estimation step will propagate during the subsequent normal integration.

To solve the problems with PSBox’s implementation, the following improvements have been introduced:

A.

A nearby light source model is used, so they can be modeled as a distant point light (this is possible when the working distance from an illuminator to an object surface is more than five times the maximum dimension of the light-emitting area) [60]. The position and direction of the light are found through measurement of the mutual position of the camera, lights, and acquisition plane. This geometric constraint provides a robust and deterministic approach, as the spatial relationships between points are predetermined by the physical setup rather than relying on potentially error-prone manual fitting operations. We evaluated the required accuracy of the measurement of the components’ mutual position through a series of tests aiming to evaluate the maximum possible error. At the end of the PS process, the maximum errors need to be as follows:

• No more than 0.1 pixels in the final normal map (maximum angular difference of 0.5° in the evaluation of the direction of the normal);
• No more than 1 mm in the mesh.

In practice, we measured the distance between a synthetic plane and a plane as obtained from the PS solution, developed by virtually changing the position of the LED lights. Figure 3 demonstrates that the maximum error allowed in the measurement is 5 mm.

B.

Frankot and Chellappa’s method for normal integration failures is corrected following a series of observations. As noted in [22], the accuracy of Frankot and Chellappa’s method ‘relies on a good input scale’ and a big improvement could be achieved through exploiting solutions that are able to run on non-periodic surfaces (“The fact that the solution [of Frankot and Chellappa] is constrained to be periodic leads to a systematic bias in the solution” [61]) and to manage a non-rectangular domain. The latter condition is negligible in our case because paintings and drawings usually have a rectangular domain or—if not—can easily be inscribed into a rectangle anyway. We improved the other conditions, exploiting the solution suggested by Simchony et al. [62], which consists of solving the discrete approximation of the Poisson equation using discrete Fourier transform instead of discretizing the solution of the Poisson.

C.

The most common solution to the problem of the wrong representation of the surface at low frequencies is to replace the inaccurate low frequencies of the photometric normal with the more accurate low frequencies of a surface constructed from a few known heights measured with a laser scanner, or a probe, or a photogrammetric process [20,63]. We developed a different process, like that proposed by [21], which also allowed us to minimize problems caused by other factors, such as shadows, irregularity in the light sources and their position, different brightnesses for each light source, and a lack of perfect parallelism among the light beams. We use the distribution of light irradiance sampled from a flat reference surface. The non-uniformity of the radiance distribution is compensated using the reference images. In practice, a flat surface is measured that covers the whole light field and the normal field is calculated. Different normal values are qualified as systematic distortions and their value is subtracted from the normal field of the represented object. With this solution, there is no additional significant time cost required to solve the PS problem, as the procedure remains a linear problem. Finally, a surface deformation correction is applied by a 3 × 3 three-dimensional parabolic fitting algorithm, exploiting the MATLAB function fit and minimizing the error at the least squares at all the points of the surface [64].

2.3. The Hardware Solutions

The hardware part of the solution consists of two distinct repro stands, each one optimized for different scenarios and artwork positions: a horizontal repro stand with a mobile acquisition plane for movable artworks and a vertical robotized one for the automatic acquisition of paintings and drawings that are hanging from walls or not movable.

2.3.1. The Horizontal Stand

The horizontal repro stand is sized 1470 × 1470 × 1992 mm and it features an acquisition plane of 700 × 880 mm (500 × 375 mm on the acquisition plane, per single shot). Its structure includes two main elements (Figure 4), each independently transportable and assembled:

• A lower frame with a capture surface (Figure 5a), consisting of a sliding base equipped with rails for translation along both the axes of the acquisition plane. This frame weighs 14 kg;
• A vertical frame system (Figure 5b), designed to house 32 Relio² LED lights and camera, composed of four uprights made from square aluminum profiles, held in place by components manufactured through 3D rapid prototyping. This frame weighs 6 kg.

Figure 4. The horizontal repro stand.

Figure 4. The horizontal repro stand.

The total weight of the horizontal stand is about 20 kg, without the camera. The support frames for the surface feature four ground-contact points, whose height can be adjusted using screws with the aim of precisely leveling the acquisition plan. The vertical frame consists of a profile system topped by a truncated pyramidal box element, inside which the camera is positioned on a constrained slide.

This pyramidal system is designed to stand on four inclined vertical uprights, which also provide housing for the 32 LED lights manufactured by Relio Labs (Brescia, Italy) [65]. These lights feature a 4000 K Correlated Color Temperature (CCT) and an illuminance of 40,000 lux at 0.25 m, do not generate any potentially harmful Ultra-Violet (UV) or Infra-Red (IR) emissions, and adopt a TIR (Total Internal Reflection) lens type mounted on the photoemitting diode with an emission angle of 25°. With this lens, the luminance drops to less than 50% of the maximum value.

The medium format digital camera mounted on the repro stand is a Hasselblad X2D-100C equipped with a Hasselblad XCD 3.5/120 Macro lens system. The specifications of this system are provided in

Table 1. Hasselblad X2D-100C camera.

Technology	Focus	Resolution	Sensor Size	ISO Sensibility	Noise Level	Color Depth
100 Megapixel BSI CMOS Sensor	Phase Detection Autofocus PDAF (97% coverage)	100 megapixel (pixel pitch 3.78 μm)	11,656 (W) × 8742 (H) pixel	64–25,600	0.4 mm a 10 m	16 bit

and

Table 2. Hasselblad XCD 3,5/120 macro lens system.

Focal Length	Equivalent Focal Length	Aperture Range	Angle of View diag/hor/vert	Minimum Distance Object to Image Plane
120.0 mm	95 mm	3.5–45	26°/21°/16°	430 mm

.

2.3.2. The Vertical Stand

The vertical repro stand is sized 2625 × 1572 × 2447 mm and consists of three elements (Figure 6):

• A lower frame (1800 × 1100 × 400 mm), consisting of a raisable base equipped with a rail for translation along the horizontal axis of the entire structure. The raisable base comprises a lifting frame that can be disassembled into individual arms (300 or 600 mm long) (Figure 7a). This frame weighs 35 kg (including the lifting frame and its ballasts);
• A vertical frame system composed of four carbon fiber uprights held in place by two lightweight aluminum cross-braces (Figure 7b). This frame weighs 3 kg;
• A trapezoidal frame (850 × 850 × 1200 mm) to which 32 Relio² LED lights and the mounting system for the camera are secured (Figure 7c). This frame weighs 10 kg. The capture area is 500 × 375 mm on the acquisition plane, per single shot.

Figure 6. The robotized vertical repro stand and its components.

Figure 6. The robotized vertical repro stand and its components.

The total weight of the vertical stand when fully assembled is about 48 kg. The lower frame is constructed from aluminum profiles, assembled through joints secured by bolt connections. Through an electric stepper motor, the lower frame hosts two linear actuators that are able to translate the upper structure horizontally. The upper vertical frame assembly is made of four carbon tubes placed vertically into their circular seats in both the top and bottom aluminum custom cross-braces. A darkening system consisting of a black jersey fabric cover, shaped around the trapezoidal frame, completes the stand (Figure 8).

3. The Measurement Methodology

3.1. Metrological Context and Approach

A key step of our workflow is the measurement of the relative positions of the following: the camera, considered as central point of its sensor plane; the capture plane, where drawings and paintings are placed; the light sources, corresponding to the weighted centroids of the different groups of four LED lamps. This includes not only their 3D digital representation according to the measurements [66], but also the processing algorithm capable of transforming the dataset into calibrated coordinates that can be visualized and analyzed and the metrological characteristics of the instrument. The metrological aspects directly influence the ‘raw’ data produced by spatial measurements and their evaluation is important because of their role in defining and optimizing the entire measurement process [67].

The term metrics quality [68], which is extensively documented in the research (e.g., [69,70,71,72,73,74]), is used to quantify how much aspects related to a measurement deviate from a predefined dimension. Metrics quality is typically evaluated using quantities such as a priori knowledge of the 3D imaging device used for surface measurements (calibration and characterization), the uncertainty (i.e., the superposition of trueness or accuracy (the mean of the measurement), and precision of the measurement (i.e., the standard deviation of the measurement)), and, finally, the traceability of the measurement process (i.e., “A property of a measurement result whereby the result can be related to a reference through a documented unbroken chain of calibrations, each of which contributes to the measurement uncertainty” [75]), taking into account the object’s material and local surface features.

In detail, the measured coordinates produced by a 3D imaging system must then be accompanied by a quantitative statement of their uncertainty. Existing standards and geometric features drive this evaluation. Figure 9, after [76,77] summarizes the main factors that affect the uncertainty in a 3D imaging system.

In this context, according to the International Vocabulary of metrology—Basic and general concepts and associated terms (VIM), resolution is “the smallest change in a measured quantity that causes a noticeable change in the corresponding display”, i.e., for 3D imaging systems, the minimum geometric detail that the rangefinder is capable of capturing. Obviously, this value represents the maximum resolution allowed by the 3D sensor. It can be divided into two components: the axial resolution, along the optical axis of the device (usually specified as z), and the lateral resolution, on the xy plane [78].

For 3D sensors, accuracy needs then to be evaluated in both the axial and lateral directions. In general, the depth accuracy is the most important. In the literature, setting the resolution level of the range camera is not yet extensively codified. Usually, this parameter is qualitatively adjusted to generate a 3D model that visually contains details of interest. This general visual rule is a geometric transposition of Nyquist’s sampling theorem, according to which an analog signal can be reconstructed exactly from its sampled version if the sampling frequency is at least double the signal’s variation frequency [79]. In the geometric case, it is therefore assumed that detail is correctly acquired with a sampling step of at least half the size of the minimum detail of interest. As [80] points out, this criterion provides a “rule of thumb” for estimating a minimum geometric sampling step, below which it is certain that the smaller geometric details will be lost.

While accuracy is affected by systematic errors, precision is mostly affected by random errors, resulting in a certain degree of unpredictability in the measured value. In the case of laser-based devices, the main source is the laser speckle effect [81]. In the photogrammetric process, precision defines the statistical noise of an adjustment, i.e., it models the internal correctness of a system [82]. As the Structure-from-Motion (SfM) methods [83] used for camera localization and orientation provide only limited information about the internal quality of the Bundle Adjustment (BA) process [84] (i.e., only the final reprojection error), this can be improved starting from the orientation results obtained in the free-network approach, by adding constraints linked to a set of targets measured with more than five times higher uncertainty. After a similarity transformation, these 3D coordinates could be compared with those previously measured with laser scanners. The residuals and corresponding statistics can then be derived [73].

For an active 3D sensor, uncertainty estimation can be achieved by acquiring the range map of a target whose shape is known in advance, such as a plane, and evaluating the standard deviation of each 3D point with respect to the ideal shape [85]. Since a range map can easily be generated from millions of points, statistical significance is implicit. For modeling applications, the uncertainty of the range sensor should not exceed a fraction of the resolution step to avoid topological anomalies in the final mesh [86]. A good guideline is to avoid a resolution level smaller than the measurement uncertainty of the instrument.

For the photogrammetry, usually, the uncertainty assessment is performed by comparing the achieved results to a ground truth, which should theoretically be two or three times more accurate than the expected results. Although this general approach may be seen as reasonable, to achieve better metrological traceability, a geometric artifact with a known form and size is frequently used and its measurement is compared with another made with another instrument, e.g., a laser scanner.

In practice, in our case, the measurement process follows the workflow shown in Figure 10. The metrics quality was evaluated starting from a reduced version of the German guidelines DIN VDI/VDE 2634 [87] and the VDI/VDE 2617 6.1 [88]. After a calibration/characterization step, the measurement of the lights/camera/plane system was made through a photogrammetric process with a Hasselblad X2D-100C camera. This measurement was compared with a laser scanning capture with a Leica RTC360 Time-of-Flight (ToF) Terrestrial Laser Scanner (TLS) system to assess the metric quality of the 3D model. The inter-comparison of the laser scanner data used as a reference with dense stereo generated 3D data is a well-consolidated approach. Several papers illustrate problems and solutions, methods, and best practices [89,90]. As metric reference to scale the photogrammetry data from a Scantech iReal M3 laser scanner ensuring measurement provides results that are more than five times accurate than the expected results as per the ISO 14253 [76] are used.

3.2. The Instruments Used for Measurements

3.2.1. Scantech iReal M3 Laser Scanner

The Scantech iReal M3 laser scanner (Scantech Co. Ltd., Hangzhou, China) is a 3D handheld system that uses triangulation technology based on a dual infrared laser light source, one of which is a Vertical Cavity Surface Emitting Laser (VCSEL). At its core, the handheld scanner projects seven parallel infrared laser beams onto the scanned object, whose reflected beam is analyzed and localized by two sets of calibrated industrial cameras. Operatively, the handheld system works with a positioning mechanism based on retroreflective targets, similar to stickers, which can be easily applied to various surfaces and create a Positioning Model (PM). This PM serves to orient the scanner within 3D space and establishes a coordinate system for reporting measurements. From the PM, a second scan with VCSEL light reconstructs the surface. The two cameras simultaneously detect both retroreflective targets using invisible infrared backlight, which can improve marker recognition and adaptability to black materials, and VCSEL-projected laser lines. The corresponding spatial coordinates (X, Y, Z) of points on the object detected by the laser beams can be calculated based on the parallax of the image obtained from the cameras. Measuring uncertainty essentially depends on the size of the distance between the emitted laser and sensor, the diffraction of the light source, and the speckle effect. Technical specifications are provided in

Table 3. Technical specifications of the Scantech iReal M3 laser scanner.

Technology	Framed Range	Accuracy	Lateral Resolution
7 parallel infrared laser lines + VCSEL infrared structured light	580 × 550 mm (DOF 720 mm with an optimal scanning distance of 400 mm)	0.1 mm	0.01 mm

.

3.2.2. Laser Scanner Leica RTC360 Tof TLS System

The Leica RTC360 TLS system (Leica Geosystems AG, Heerbrugg, Switzerland) for the measurement exploits the Wave Form Digitizer (WFD) Technology. WFD combines the ToF technique (which measures the round-trip flight time of a laser beam emitted by a light source relative to a point on the object to be measured) and the phase shift (where the distance is calculated based on the time interval between a start and stop pulse, which is digitized from the received signal) (Figure 11). Compared to a pure ToF measurement system, WFD technology enables a better overall measurement performance thanks to the rapid distance measurements, reduced laser spot size, and increased measurement accuracy. The measurement uncertainty depends on the signal-to-noise ratio and the pulse rise time. Technical specifications are provided in

Table 4. Technical specifications of the Leica RTC360 ToF TLS system.

Technology	Framed Range	Accuracy	Resolution	Precision
High dynamic ToF with Wave Form Digitizer Technology (WFD)	360° (H)–300° (V)	1.9 mm at 10 m	3 mm at 10 m	0.4 mm at 10 m

.

3.2.3. Hasselblad X2D-100C Camera

The photogrammetric measurement of both repro stands was carried out using a Hasselblad X2D-100C camera (Hasselblad, Göteborg, Sweden), whose technical specifications are in Table 1 of Section 2.3.1. A Hasselblad XCD 38 mm f/2.5 V lens was used, whose technical specifications are provided in

Table 5. Hasselblad XCD 38 mm f/2.5 V lens system.

Focal Length	Equivalent Focal Length	Aperture Range	Angle of View diag/hor/vert	Minimum Distance Object to Image Plane
38.0 mm	30 mm	2.5–32	70°/59°/46°	300 mm

.

3.3. Calibration and Characterization of Measurement Instruments

System calibration/characterization plays a key role in ensuring metrics quality (i.e., for calibrated devices, precision and uncertainty coincide), and the system measurement accuracy is largely dependent on the calibration accuracy. The instrument calibration/characterization was therefore the first step of our measurement process.

For the two laser scanners used (see Section 3.2.1 and Section 3.2.2), the calibration certification of the manufacturer was used as the first reference, but then the instruments were characterized through a planar artifact. As demonstrated by Russo et al. [92], the 3D analysis of a reference plane allows for the simultaneous estimation of uncertainty and accuracy. The best-fitting plane is used as an approximation of the actual position of the physical plane, while the point-to-point distance of each 3D point belonging to the range map can be measured and characterized. Since the plane is constructed to be the best fit with the original data, the average distance is automatically zero, and therefore no absolute precision error can be evaluated. The standard deviation provides an estimate of the distribution of values around the ideal plane and allows for the evaluation of measurement uncertainty. Furthermore, even though the mean is zero, the chromatic map of the deviation between the point and plane allows for the identification any non-random pattern due to the accumulation of error in specific areas and any errors arising from systematic causes (precision).

For photogrammetry, the self-calibration process was used [93,94]. The solution of a self-calibrating BA leads to the estimation of all internal parameters and Additional Parameters (APs), beginning with a set of manually measured image correspondences (tie points). The overall network geometry, particularly the configuration of the camera stations, is critical to the accuracy of the process.

All measurements were carried out in an environmentally controlled laboratory with a temperature of 20° ± 0.1° ∆C and a relative humidity of ~50%. During measurement activities, access to the laboratory was strictly controlled and limited to authorized personnel to ensure the reliability of the measurements (the floor was prevented against undesired movements). Before proceeding with the measurement operations, careful attention to the operator’s manual guidelines, combined with extensive experimentation, was paid. A diffused and controlled ambient light (LED) provides illumination without casting shadows.

3.3.1. Calibration and Characterization of the Scantech iReal M3 Laser Scanner

At first, the system was calibrated using a plate provided by the manufacturer in combination with the iReal 3D software (version r2023) that assists the user with the correct procedure. The scanner was then characterized through the acquisition of a reference test field consisting of a laminated glass panel (dimensions: 600 × 700 mm; thickness: 12 mm; planarity guaranteed to be within 10 µm across the entire surface) coated with a matte white PolyVinyl Chloride (PVC) film (Figure 12). The acquisition of the plane was performed maintaining a constant distance of 400 mm from the plane and a sampling resolution of 0.10 mm, i.e., an accuracy more than five times the resolution of the photogrammetry. The results are presented in

Table 6. Measured precision and accuracy of the Scantech iReal M3.

Captured Area	295 × 440 mm
Sampled points	6,130,559
Average distance between a fitted plane and point cloud	0.000441619 mm
Standard deviation	0.0172472 mm

and Figure 13.

3.3.2. Characterization of the Leica RTC360 ToF TLS System

The characterization of the Leica RTC360 ToF TLS system was carried out through the acquisition of a reference test field consisting of the laminated glass panel described in Section 3.3.1. In this case, the acquisition was performed maintaining a constant distance of 790 mm from the plane and a sampling resolution the same as that of the photogrammetric measurement of the stands (0.40 mm). The results are provided in

Table 7. Measured precision and accuracy of the Leica RTC360.

Captured Area	250 × 500 mm
Sampled points	664,675
Average distance between a fitted plane and point cloud	0.374145 mm
Standard deviation	0.313806 mm

and Figure 14, which show that the distribution of measured values is well within the expected threshold.

3.3.3. Camera Calibration

The geometric calibration of a camera is defined as the determination of deviations of the physical reality from a geometrically ideal imaging system based on the collinearity principle: the pinhole camera. The measurement of parameters for the acquisition system (see Section 3.2.3) was carried out following guidelines from experimental studies [95], through the self-calibration process using Colmap software (version 3.8) [96]. Brown’s formula including 10 parameters was used as a camera model [97]:

• Focal length (f): expressed in pixels.
• Principal point coordinates (Cx, Cy): defined as the coordinates of the intersection point of the optical axis with the sensor plane, expressed in pixels.
• Affinity and non-orthogonality coefficients (b1, b2): expressed in pixels.
• Radial distortion coefficients (k1, k2, k3): dimensionless.
• Tangential distortion coefficients (p1, p2): dimensionless.

Figure 15 illustrates the image residuals (the average vector of the reprojection error for the pixels in the corresponding cells), and

Table 8. Calibration coefficients.

	Value	Error	f	Cx	Cy	b1	b2	k1	k2	k3	p1	p2
f	10,228.5	0.64	1.00	−0.07	0.05	−0.90	0.04	−0.18	0.19	−0.18	−0.11	−0.09
Cx	8.87514	0.44	-	1.00	−0.07	0.10	0.14	0.01	−0.01	0.00	0.93	−0.07
Cy	−20.5956	0.53	-	-	1.00	−0.21	0.08	0.03	−0.03	0.03	−0.09	0.72
b1	−10.4546	0.59	-	-	-	1.00	−0.01	0.01	−0.03	0.04	0.14	0.00
b2	−6.16297	0.23	-	-	-	-	1.00	−0.01	0.01	−0.00	0.05	0.04
k1	−0.015391	0.00023	-	-	-	-	-	1.00	−0.97	0.93	0.02	0.03
k2	0.0460376	0.0017	-	-	-	-	-	-	1.00	−0.99	−0.02	−0.03
k3	−0.114143	0.0048	-	-	-	-	-	-	-	1.00	0.02	0.03
p1	0.000138551	0.000014	-	-	-	-	-	-	-	-	1.00	−0.07
p2	0.000219159	0.000011	-	-	-	-	-	-	-	-	-	1.00

reports the calibration coefficients and the correlation matrix.

3.4. Description of the Measurement Processes

The measurement process covered the following aspects:

• The acquisition of a series of coded RAD targets using the Scantech iReal M3 3D laser scanner to provide a metric reference to scale the model in the photogrammetric process (Section 3.4.1);
• The acquisition of the stands by the Leica RTC360 ToF TLS system (Section 3.4.2);
• The acquisition of the stands by photogrammetry (Section 3.4.3);
• The comparison of the photogrammetric data with the Leica RTC360 ToF TLS system data (Section 3.4.4).

The measurement of the uncertainty of the mutual positions among the camera image sensor plane, the acquisition plane, and the light sources that is required to correctly apply the PS process is 2 mm (see Section 2.2), but we decided to set double this value, i.e., 1 mm, as our goal, and a reference uncertainty of 0.1 mm.

3.4.1. Target Acquisition Through Scantech iReal M3 3D Laser Scanner

Sixteen reference markers consisting of coded Ringed Automatically Detected (RAD) targets, each 5 mm in diameter, were placed across one plane positioned horizontally for the measurement of the horizontal stand and vertically for the vertical one (Figure 16). Both the arrangement and number of targets followed the scanner manufacturer’s recommendations. Specifically, the targets were positioned to maintain distances between 20 and 100 mm from each neighboring target. The measurements’ lateral resolution was 0.1 mm. The measurement process begins by creating the PM. During this phase, the scanner must acquire all the targets on a planar surface to achieve a stable target array position. Once the PM is established, the VCSEL light is activated, and surface data collection is performed at a constant distance of approximately 400 mm over the plane, maintaining a perpendicular orientation to the planar surface, even if the measurement angle could be modified to allow for non-perpendicular scanning without compromising the accuracy of the distances, since the reference PM was already defined [69]. The acquired dataset was subsequently processed using the iReal 3D software (version r2023) and exported in .E57 format, preserving the RGB chromatic information collected by the device that allows for to the target centroid position to be visually checked.

3.4.2. Stand Acquisition with Leica RTC360 ToF TLS System

A point cloud was captured with a single station for both the horizontal and vertical stands, maintaining a medium distance of 790 mm from the stands and a sampling resolution the same as that of the photogrammetric measurement of the stands (1 mm). This resolution was reached through setting the instrument to the maximum level of lateral resolution and checking that the output distance between the points was the established distance (i.e., 1 mm).

3.4.3. Stand Acquisition with Photogrammetry

To perform the measurement on both repro stands, the conventional automatic photogrammetric pipeline was followed: image acquisition, camera calibration (see Section 3.3.3), image orientation, and dense image matching [98] to extract dense cloud points. Taking the maximum uncertainty in the measurement required by the PS algorithm for the Points of Interest (PoI) into account, the Ground Sample Distance (GSD) was 0.20 mm (i.e., using the Nyquist sample theorem 0.4 mm), corresponding to a camera-to-object distance of ≈1000 mm as shown in the following:

$$D={\displaystyle \frac{F_r·imW·GSD}{S_w}}$$

where:

• D is the distance in mm from the acquisition plane;
• S_w is the camera sensor width expressed in mm (equal to 43.8 mm for the Hasselblad X2D-100C);
• imW is the image width expressed in pixels (equal to 11,656 pixels for the Hasselblad X2D-100C output);
• F_r is the focal length of the adopted lens expressed in mm (equal to 38 mm for the Hasselblad XCD 38 mm f/2.5 V lens).

A second camera network parameter that greatly influences the accuracy of the process is the distance/baseline ratio. As demonstrated by Guidi et al. [99], for SfM-based processes the best results are achieved within a range of 4.7–6.2. We adopted a medium ratio of 5.5 to fulfill this requirement. The image acquisition does not use rigidly connected cameras on a stable structure, but all the images were captured using a tripod that remotely controls the camera through Hasselblad Phocus to avoid direct physical interaction and motion blur effects. All photos were shot with the same photographic parameter setup, i.e., focus fixed at infinity, aperture f16, shutter 1 s, and ISO 400. The shots were stored in Hasselblad’s raw .FFF format and processed in the Hasselblad Phocus (version 3.8.4) software environment to obtain 8-bit measurements. The TIFF format rendered images in the sRGB color space for later use in photogrammetric alignment. Unsharp mask and denoise filters were applied according to the manufacturer’s specifications for that combination of lens, camera, exposure, ISO, and aperture.

To fulfill the needs of the vertical repro stand, all the positions of the PoI were measured considering two distinct configurations, without the assembled occlusion system and with the assembled occlusion system, to evaluate displacements due to its installation.

The photogrammetric measurement was performed through multi-image capture of both systems with different camera positions; images were acquired following a convergent camera network surrounding the stand structures and their reference planes (Figure 17). The number of captures is as follows:

• n. 132 for the horizontal acquisition stand;
• n. 133 for the vertical robotic stand without darkening occlusion;
• n. 102 for the vertical robotic stand with darkening occlusion.

Figure 17. Radial camera network for the horizontal (a) and vertical (b) repro stands.

Figure 17. Radial camera network for the horizontal (a) and vertical (b) repro stands.

Each point of the stands is visible using a minimum of eight cameras (Figure 18) and a constant overlap of 60% between successive shots was maintained, with a maximum angular deviation not exceeding 20 degrees. The selected camera network configuration, characterized by convergent shots capable of closing the capture ring, ensures robustness in camera positioning and alignment compared to a parallel camera arrangement, as documented in various works published in the scientific literature [100,101,102].

As in [99], we used the tie points’ reprojection error to check the quality of the calibration and orientation steps. The processing workflow includes the following steps:

• Run the alignment procedure on the full set of captured images;
• Check the reprojection error on the resulting tie points. If below 0.5 pixels, stop here; otherwise, proceed with the next step;
• Delete about the 10% of the tie points providing the higher reprojection error;
• Rerun the BA step on the cleaned set of tie points and go back to step 2.

The results of the camera’s orientation and calibration were then imported to the Agisoft Metashape Professional software (version 2.2.0) [103] to scale the camera positions for the 16 coded RAD targets present in the scenes. Finally, the dense point clouds were generated at Ultra-High quality, meaning that the original images were processed at their full resolution without any preliminary downscaling.

3.4.4. Comparison of the Photogrammetric and TLS Data

The last step of the process concerns the comparison of the data captured by photogrammetry and laser scanner in the open-source software CloudCompare version 2.13.2) [104]. The photogrammetric models were aligned with the laser scanner point clouds using CloudCompare’s implementation of the Iterated Closest Point (ICP) algorithm [105,106]. After subsampling the photogrammetric point clouds to obtain models with the same GSD as the ToF TLS models (1 mm), the residual deviation of the 3D coordinates gathered with photogrammetry from the reference point cloud was statistically analyzed to calculate the Root Mean Square (RMS) error, mean error, and histogram error. We made sure that the orientation step was iterated until the mean value was less than 0.5 mm for each dense cloud. This allowed us to confirm that the alignment process was performed correctly, influencing the random error estimation with a systematic factor.

4. Results

The following section presents the results concerning the measurements of the stands, the comparison between the photogrammetric and the laser scanner measurements, and the improvement in the performances of the nLights software (version 0.0.9.1), exploiting measured localizations of the lights and the algorithmic refinements described in Section 2.2.

4.1. As-Built Measurement of the Horizontal Repro Stand

4.1.1. Measurement Using Scantech iReal M3 Laser Scanner

The Scantech iReal M3 laser scanner was employed to measure the position of the spatial coordinates of the centroids the 16 coded RAD targets used to scale the photogrammetric dense point cloud. These coordinates were exported in .CSV format. In

Table 9. Positions of the coded targets (origin at the center of the plane, mm).

ID	X	Y	Z
1	−112.1833981	−265.6449903	1.1604001
2	−356.9368841	27.3719204	1.1928803
3	218.4486223	167.6946044	0.7668501
4	−83.5303045	335.2168791	0.8040630
5	−356.2072197	−267.7837971	1.4174284
6	422.8374003	36.3053955	0.8180338
7	418.5192553	−267.5176537	0.7987372
8	172.0460260	−268.9918738	1.0840074
9	−166.6299110	−121.8379844	1.2866903
10	20.7659104	−72.1971757	0.9835656
11	219.4242678	−120.2348847	0.8334230
12	21.4847097	100.0491205	0.8094964
13	175.3704988	336.8665062	0.7985485
14	421.7712423	327.9472855	1.4888459
15	−357.4699748	331.4955047	0.9538767
16	−163.2943198	169.2829513	0.7963140

, the positions of the coded targets are presented.

4.1.2. Measurement Using Leica RTC360 ToF TLS System

The Leica RTC360 ToF TLS system was employed to obtain a measurement to compare with the photogrammetric measurement. A scan was acquired using the Cyclone Register 360+ software (version 2024.0.2.r26474). Then, to facilitate further processing, the raw data were exported in .E57 format. The data exceeding the stand shape were erased using CloudCompare. The TLS yielded a dataset made of 88,732,868 points which, after cleaning the elements outside the stand, contained 10,294,639 points.

4.1.3. Measurement with Photogrammetry

The photogrammetric measurement was employed to obtain a dense point cloud of the stand, scaled through the 16 coded RAD targets measured with the Scantech iReal M3 laser scanner. Data exceeding the stand’s shape were cleaned within CloudCompare. The results are presented in

Table 10. Photogrammetry point cloud outcomes.

	Agisoft Metashape Professional	Colmap
Number of registered images	-	132
Number of tie points	-	27,951
Mean observations per image	-	859,106
Number of points in the dense cloud	10,367,336	-
RMS reprojection error	-	0.485 px

.

4.1.4. Comparison Between ToF TLS and Photogrammetry

To compare the two datasets, the point cloud from photogrammetry was resampled to obtain an average image GSD of ~1 mm. The comparison between the point cloud obtained from the ToF TLS and the dense cloud obtained through photogrammetry (Figure 19) produced the following results (

Table 11. Comparison between ToF TLS and photogrammetry.

Average distance of points	0.5214 mm
Standard deviation	0.77006 mm

).

4.1.5. Measurement of Points of Interest (PoIs) for the Horizontal Repro Stand

The coordinates of the Points of Interest (PoIs), representing the camera sensor plane, the capture plane, and the light sources, were extracted as 3D coordinates from the dense cloud generated by Agisoft Metashape Professional (Figure 20), through vector graphic interpolation. These values were exported to AutoCAD version 2024 in .DXF interchange format to describe all the points in a new reference system and the sensor position along a straight directrix perpendicular to the acquisition plane and with its trace coinciding with the origin of the Cartesian axis.

In AutoCAD, numerical tolerances were set to the fifth decimal place to avoid missing data, with the origin positioned at the center of the framed rectangle on the stand base (500 × 375 mm). The extracted and transformed coordinate values of PoI are shown in

Table 12. PoIs’ extracted and transformed coordinate values (mm).

PoI	X	Y	Z
Origin	0	0	0
Relio_1	−646.79	3.0601	171.98
Relio_2	−3.6600	643.86	166.24
Relio_3	641.34	0.3800	163.33
Relio_4	−5.1200	−645.04	165.76
Relio_5	−474.21	2.6700	471.77
Relio_6	0.1300	476.28	459.22
Relio_7	466.87	0.0700	467.11
Relio_8	−5.3300	−485.74	442.38
Camera	0.0600	0.2602	1542.48

.

4.2. As-Built Measurement of the Robotic Vertical Repro Stand

4.2.1. Measurement Using Scantech iReal M3 Laser Scanner

The Scantech iReal M3 laser scanner was employed to measure the position of the spatial coordinates of the centroids of the 16 coded RAD targets used to scale the photogrammetric dense point cloud. These coordinates were exported in .CSV format. In

Table 13. Positions of coded targets (origin at the center of the plane, mm).

ID	X	Y	Z
1	−91.7384979	−254.3196531	0.9266232
2	−318.6724500	98.8737499	0.9983411
3	161.4589829	252.5450786	1.0497326
4	319.6547597	−319.3383336	1.0325548
5	−324.2391213	−105.5458775	0.8632963
6	102.4372316	−79.8087193	0.8986234
7	−170.2707334	252.3647593	1.4474652
8	−319.6547597	319.3383336	1.3325548
9	95.2218163	96.9329011	1.1688642
10	146.1604507	−261.1907211	0.9117996
11	−168.9669076	−16.3564388	1.2615517
12	322.6046628	126.2068312	0.9887015
13	−8.7779598	248.4658266	1.2592235
14	−313.2096315	−327.6061740	1.1325548
15	319.9531910	319.3383336	1.1325548
16	317.9581155	−126.1577396	0.8518509

, the positions of coded targets are provided.

4.2.2. Measurement Using Leica RTC360 ToF TLS System

The Leica RTC360 ToF TLS system was employed to obtain a measurement to compare with the photogrammetric one. A scan was acquired using the Cyclone Register 360+ software (version 2024.0.2.r26474). Then, to facilitate further processing, the raw data were exported in .E57 format. The data exceeding the stand shape were erased within CloudCompare. The TLS yielded a dataset made of 165,536,379 points which, after cleaning the elements outside the stand, contained 11,378,129 points for the repro stand with darkening fabric occlusion, and a dataset made of 166,253,743 points which, after cleaning the elements outside the stand, contained 11,419,578 points for the repro stand without darkening fabric occlusion.

4.2.3. Measurement with Photogrammetry

The photogrammetric measurement was employed to obtain a dense point cloud of the stand, scaled through the 16 coded RAD targets measured with the Scantech iReal M3 laser scanner. Data exceeding the stand’s shape were cleaned within CloudCompare. The results are shown in

Table 14. Point cloud outcomes from photogrammetry (with darkening fabric occlusion).

	Agisoft Metashape Professional	Colmap
Number of registered images		102
Number of tie points		70.221
Mean observations per image	-	1099.99
Number of points in the dense cloud	11,373,875	-
RMS reprojection error	-	0.465 px

and

Table 15. Point cloud outcomes from photogrammetry (without darkening fabric occlusion).

	Agisoft Metashape Professional	Colmap
Number of registered images	-	133
Number of tie points	118,213	-
Mean observations per image	-	2990.75
Number of points in the dense cloud	12,054,708	-
RMS reprojection error	-	0.499 px

.

4.2.4. Comparison Between ToF TLS and Photogrammetry

To compare the two datasets, the point cloud from photogrammetry were resampled to provide an average image GSD of ~1 mm. The comparison between the point cloud obtained from the ToF TLS and the dense cloud obtained through photogrammetry (Figure 21 and Figure 22) produced the following results (

Table 16. Comparison between ToF TLS and photogrammetry (with darkening fabric occlusion).

Average distance of points	0.5218 mm
Standard deviation	0.79912 mm

and

Table 17. Comparison between ToF TLS and photogrammetry (without darkening fabric occlusion).

Average distance of points	0.4924 mm
Standard deviation	0.69464 mm

):

4.2.5. Measurement of Points of Interest (PoIs) for the Vertical Repro Stand

The coordinates of the Points of Interest (PoIs), representing the camera sensor plane, the capture plane, and the light sources, were extracted as 3D coordinates from the dense cloud generated by Agisoft Metashape Professional (Figure 23) through vector graphic interpolation. These values were exported to AutoCAD version 2024 in .DXF interchange format to describe all the points in a new reference system and the sensor position along a straight directrix perpendicular to the acquisition plane whose trace coincides with the origin of the Cartesian axis. In AutoCAD, numerical tolerances were set to the fifth decimal place to avoid missing data, with the origin positioned at the center of the framed rectangle on the stand base (500 × 375 mm). The extracted and transformed coordinate values are provided in

Table 18. PoIs’ extracted and transformed coordinate values (mm—with darkening fabric occlusion).

PoI	X	Y	Z
Origin	0	0	0
Relio_1	10.7631	−225.7121	601.7119
Relio_2	605.4825	−226.7232	−4.3642
Relio_3	7.4924	−239.9226	−609.6511
Relio_4	−603.3234	−230.8313	1.2631
Relio_5	0.3228	−537.4174	472.5922
Relio_6	461.0301	−537.3921	8.2132
Relio_7	1.5820	−541.6323	−456.7912
Relio_8	−461.62	−537.2876	8.1521
Camera	0.0323	−1543.8149	0.0101

and

Table 19. PoIs’ extracted and transformed coordinate values (mm—without darkening fabric occlusion).

PoI	X	Y	Z
Origin	0	0	0
Relio_1	−1.8714	−235.8112	611.5131
Relio_2	607.6222	−225.0312	−0.5913
Relio_3	11.6712	−238.3611	−624.8112
Relio_4	−589.1021	−223.7463	−2.9221
Relio_5	−2.9265	−543.3825	469.5811
Relio_6	460.6141	−538.2921	8.1423
Relio_7	4.6122	−539.5241	−463.3921
Relio_8	−458.0721	−533.0126	8.1811
Camera	0.04	−1543.3821	0.1712

, while the PoI variations obtained after assembling the darkening fabric occlusion are shown in

Table 20. PoI variation (after assembling the darkening fabric occlusion).

PoI	X	Y	Z	Euclidean Distance
Origin	0	0	0	0
Relio_1	−12.6345	−10.0991	9.8012	18.9125
Relio_2	2.1397	1.6920	3.7729	4.6557
Relio_3	4.1788	1.5615	−15.1601	15.8023
Relio_4	14.2213	7.0850	−4.1852	16.4304
Relio_5	−3.2493	−5.9651	−3.0111	7.4301
Relio_6	−0.4160	−0.9000	−0.0709	0.9940
Relio_7	3.0302	2.1082	−6.6009	7.5629
Relio_8	3.5479	4.2750	0.0290	5.5555
Camera	0.0077	0.4328	0.1611	0.4618

.

4.3. Results Following the Performance Optimization of the PS Techniques

The acquired measurements were integrated into the nLights software to generate albedo, specular, and normal maps. Compared with approaches that determine light source directions based on reflections from a sphere, the developed method produced more detailed results, as illustrated in Figure 24 and Figure 25. The real-time rendering of the XVIII Century drawing shown in Figure 1 shows a significant visual enhancement (Figure 26).

Finally, the results of the solution developed to minimize problems caused by shadows, the irregularity of light sources and their position, the different brightnesses of each light source, and the lack of perfect parallelism of light beams are presented. In Figure 27, the normal maps and the meshes of a plane without and with correction are shown. Normal colors are exaggerated for better visibility. The meshes are fitted against a reference plane. In Figure 28, the results of the developed solution for the drawing in Figure 1 are shown, with both PoIs measured and the residuals of the inaccurate low frequencies of the photometric normal minimized, as illustrated in Section 2.2, as follows: normal maps with and without errors (a), meshes with and without outliers (b), and a comparison between meshes with and without residuals (c). These results demonstrate strong improvements in the global shape representation.

5. Discussion

The PS technique often introduces criticalities, which we address in this section along with the solutions we propose.

• An incorrect normal estimation often leads to warped surface reconstructions: one of our goals was to minimize distortions in the production of 3D meshes inferred from normal map integration. This proved to be achievable through the rigorous measurement of positions for lights, camera, and the acquisition plane. In the Section 4, we demonstrated that distortions in specific of paintings and drawings obtained using our measured solutions are mostly negligible.
• The accuracy of PS is heavily dependent on precise light positioning. Any misalignment in light direction or intensity estimation introduces errors in the 3D model: for this reason, we decided to measure the stands and elements’ positions to obtain an accurate layout that can be used more than once with the same level of accuracy.
• PS assumes surfaces reflect light evenly (Lambertian), but paintings and drawings have specular highlights, shadows, and non-uniform textures, and this affects normal estimation, leading to inaccuracies. We minimized the error in normal estimation using the following two strategies: 1. the non-Lambertian effect manifests more prominently at the edges of the shots, but we do not consider these due to the stitching, especially in wide paintings, which introduces a strong overlap between the captures; 2. the calibration process on a plane tends to minimize the effects of non-Lambertian surfaces. Certainly, in cases with extremely glossy surfaces, the problem remains, but it can be eliminated through the use of polarization techniques, which remain a future possibility for our solution.
• The horizontal and vertical repro stands are designed for medium-sized paintings and drawings. Larger artworks require stitching multiple images, which can introduce misalignment errors: our stands can deal with larger artworks, especially the vertical one, which has a robotized movement and can be shifted and/or lifted. The system is carefully leveled using a spherical level, with laser distance meters that ensure parallelism with the acquisition plane during the translation of the stand. In this way, misalignment errors are already minimized at the time of shooting.
• While stable, the repro lacks full automation, requiring manual adjustments. Vibrations, misalignment, or uneven placement of the artwork can introduce minor distortions: our stands, particularly the robotized vertical one, are equipped with vibration sensors that can check and avoid possible blurring in the captured images.
• The PSBox software that was initially used has limitations in accurately estimating surface normals. Frankot and Chellappa’s normal integration method fails to estimate object edges, causing inaccuracies in the reconstruction of detailed textures: we considered these criticalities, but we did not detail them as they are already well described in the literature [22]. In our solution, we adopted the natural boundary condition formulated by Neumann to overcome this issue, but details on its implementation are beyond the innovations presented in this paper.

The mesh generation process requires significant computational power, limiting real-time processing capabilities: our solution is meant to work in a modular way, so real-time processing for mesh generation was not our priority, even if future improvements will likely include parallel calculations to keep processing times reasonable.

6. Conclusions

This paper presents a PS technique that is able to overcome the difficulties in normal integration, i.e., mainly issues in locating light sources, non-Lambertian surfaces, and shadows, to properly reconstruct 3D surfaces of artworks such as paintings and ancient drawings.

The solution, based on two key features (i.e., the use of image-processing techniques to minimize residuals, and the measurement of the mutual positions of light sources, camera position and acquisition plane), proved to be successful in managing the mentioned criticalities.

In more detail, the description of the complete processes of the calibration, characterization and measurement of the two stands to find the PoIs allows us to substantiate the procedure and explain its efficiency.

In fact, despite its complexity, since the stands remain unchanged throughout their lifetime and are built with extremely low-deformation materials, once the entire measurement process is completed, the end user is exempt from solving its problems by performing complex measurements, which are difficult to understand for most professional users working in the art world, for whom this solution is meant. Future works may include the simplification of the measurement process to foster higher flexibility, allowing for quick but accurate exchanges of parts in the hardware system (i.e., the lights, the camera and the stand). Emerging trends in Artificial Intelligence (AI) include the following:

• The integration of Deep Learning-based PS methods for better shadow handling and normal estimation;
• The use of AI-driven light calibration to dynamically adjust illumination parameters;
• The implementation of multi-spectral imaging to better differentiate material properties;
• Hybrid reconstruction using Machine Learning to improve shape recovery, even for highly textured or reflective surfaces;
• The addition of fully automated scanning systems with AI-based positioning and lighting control and the care to copyrights from a digital perspective [108];

These are applications of novel technologies that will likely influence the outcomes of future progress in the field. Furthermore, more accurate techniques for the elimination of residuals for each specific stand will allow for the easy use of the whole technical solution for the 3D acquisition and visualization of paintings and ancient drawings, which will enable the greater involvement of professional operators working in the art reference field.