Measurement Studies Utilizing Similarity Evaluation between 3D Surface Topography Measurements

Abstract

In the realm of quality assurance, the significance of statistical measurement studies cannot be overstated, particularly when it comes to quantifying the diverse sources of variation in measurement processes. However, the complexity intensifies when addressing 3D topography data. This research introduces an intuitive similarity-based framework tailored for conducting measurement studies on 3D topography data, aiming to precisely quantify distinct sources of variation through the astute application of similarity evaluation techniques. In the proposed framework, we investigate the mean and variance of the similarity between 3D surface topography measurements to reveal the uniformity of the surface topography measurements and statistical reproducibility of the similarity evaluation procedure, respectively. The efficacy of our framework is vividly demonstrated through its application to measurements derived from additive-fabricated specimens. We considered four metal specimens with 20 segmented windows in total. The topography measurements were obtained by three operators using two scanning systems. We find that the repeatability variation of the topography measurements and the reproducibility variation in the measurements induced by operators are relatively smaller compared with the variation in the measurements induced by optical scanners. We also notice that the variation in the surface geometry of different surfaces is much larger in magnitude compared with the repeatability variation in the topography measurements. Our findings are consistent with the physical intuition and previous research. The ensuing experimental studies yield compelling evidence, affirming that our devised methods are adept at providing profound insights into the multifaceted sources of variation inherent in processes utilizing 3D surface topography data. This innovative framework not only showcases its applicability but also underlines its potential to significantly contribute to the field of quality assurance. By offering a systematic approach to measuring and comprehending variation in 3D topography data, it stands poised to become an indispensable tool in diverse quality assurance contexts.

1. Introduction

Surface topography data is a type of data that portrays the surface geometry of an entity such as a piece of land, ocean, a planet, a metal product, etc. This type of data provides critical information regarding a surface’s geometric, dimensional and tolerance features as well as the surface roughness, and is very useful in many fields including digital cartography, Geographic Information System (GIS), remote sensing, additive manufacturing (AM), and forensic science. Digital cartography is the study and practice involving the creation, examination, management, and display of maps using digital topography data. It encompasses a wide range of activities such as city planning, environmental management, and conservation, map navigation, agriculture monitoring, etc. [1]. In digital cartography studies, 3D GIS and remote sensing analysis are very useful tools to analyze surface topography data [2]. In AM (or 3D-printing, a promising technology that enables the fabricating process to create intricate geometries based on layer-by-layer addition of materials), surface topography measurements which can be acquired during AM processes are very useful for both in situ process monitoring and post-process quality inspection [3]. In forensic science, surface topography data provide crucial information for the identification of forensic evidence such as ballistic tool marks [4], firearm identification [5], fingerprints [6], and illicit tablets identification [7]. For example, Imani et al. [3] quantified the pores in the laser powder bed fusion AM process and investigated the cause of porosity using optical images obtained by an X-ray computed tomography system. Jones et al. [6] collected the surface morphology and structures to analyze the latent fingerprint deposition with iron oxide powder suspension.

Similarity evaluation of surface topography is useful in certain fields such as uniformity inspection in AM [8], ballistic evidence comparison in forensic [9], friend/foe detection system in military [10], structure classification in molecular biology [11], and disease diagnosis [12]. For example, in AM, the similarity between topography measurements reflects the uniformity in the surface topography of the printed layers which is pivotal to the micro-scale quality of AM products [8]. In forensic science, the degree of similarity between the surface topography of bullets or cartridge cases helps to determine whether they are fired from the same firearm to provide evidence in criminal justice systems. It is a common practice and straightforward way to use a similarity score to quantify the similarity between topography measurements in both AM and forensic science. That is, given a pair of topography measurements, a similarity score is calculated following a sequence of data analysis steps to quantify the similarity.

The overall objective of this research is to establish a framework to conduct measurement studies on the pairwise similarity scores of surface topography measurements, using additive-fabricated products for illustration. In a general quality assurance context, examining the influence of measurement errors over quality is the main goal of measurement studies. These studies enable quantifying, dissecting, and probing for physical sources of variation/low quality and are of paramount importance because a valid, precise, and accurate measurement is a necessity in any sensible quality assurance [13]. Notice that the quality in our context is the quality of conformance, that is, a small discrepancy between what is designed and intended.

This research is mainly motivated by the observation that poor repeatability and reproducibility of manufacturing processes, and consequently poor quality in products, are still critical issues in AM although AM technology has advanced rapidly in recent years. Attributable to the design flexibility, wide application, and great potential to reduce cost, the AM market size has exploded in recent years [14], with the global AM market annual revenue reaching USD 9.3 billion in 2018, and keeps increasing greatly [15]. However, compared to traditional manufacturing, AM costs much more in post-processing due to support removal, surface treatment, stress relief, etc. [16]. Thus, improving the quality and repeatability of additively manufactured parts by quality assurance is essential to fulfill the needs and requirements in various sectors (e.g., aerospace applications, healthcare).

To monitor the stability of an AM process as well as to inspect the product quality after the manufacturing is finished, high-density (HD) data (e.g., surface topography data, thermal imaging stream, and X-ray computed tomography image data) generated by advanced measurement technologies are a necessity. This is because these technologies have a significant advantage over traditional measurements [17]. The HD data depict both the micro-scale and macro-scale information of intricate products or the spatial–temporal thermal history during the fabrication, which can reveal potential defects or process instability that may otherwise fail to be detected by simple metrics such as the surface roughness parameter, product dimension, etc. [17]. For example, Everton et al. [18] conducted a review of in situ process monitoring and metrology for metal 3D printing; Imani et al. [3] used optical imaging to conduct in-process monitoring of porosity in laser powder bed fusion. However, conducting quality assurance using HD data compared with ordinary univariate measurements is challenging due to the complexity and high volume of HD data.

For ordinary univariate measurements such as length and weight, there is a large body of literature on measurement studies. To name a few, Vardeman and Jobe [13] provide a comprehensive guideline for different cases of measurement studies for ordinary univariate measurements (e.g., length, weight) with corresponding data collection plans and statistical approaches. Montgomery and Runger [19] establish an ANOVA (analysis of variance)-based estimation while Wang and Chien [20] provide range-based estimation for a common measurement study called gauge R&R. However, measurement studies using HD data are, in general, lacking.

It is possible to conduct measurement studies using HD data by simply extracting a few features from the data, e.g., the mean temperature of the regions of interest, and surface roughness parameter [21]. Unfortunately, such practice only uses partial information of the original data and may miss important features of the measurand. Another approach is to utilize advanced modeling methods, e.g., Dastoorian and Wells [22] conduct a gauge R&R study regarding the root mean square error of fitted Gaussian process models of point-cloud data (a type of HD data) as pseudo univariate measurements. Nevertheless, such complex modeling may introduce new modeling errors in gauge capability studies and is computationally expensive. Considering the limitations of existing work, it is necessary to develop new frameworks to leverage HD measurements for measurement studies.

Another observation that motivates this research is that the uniformity of the surface topography of different layers in AM is of paramount importance for the quality of AM products. This is because an AM process propagates the discrepancy in consecutive layers due to its layer-by-layer printing nature. Lack of uniformity of surface topography, or larger variation in other words, indicates possible defects generated during the manufacturing of specific layers, which consequently costs more in post-processing [23].

As we mentioned at the beginning of this article, similarity evaluation can be leveraged to examine the uniformity of different surfaces. Our rationale is that the discrepancy in surface topography data leads to the dissimilarity between measurements. The larger the discrepancy between two surfaces, the more dissimilar the two measurements are. For example, we expect a high similarity score on average when we measure the same specimen’s surface by the same device multiple times if these repeated measurements are consistent. On the contrary, for multiple measurements from distinctly different surfaces, the average pairwise similarity scores would be relatively low due to the discrepancy between surfaces. Since it is a common practice to use a pairwise similarity metric to quantify the similarity, it is sensible and useful to develop approaches to leverage similarity scores to facilitate uniformity inspection which provides insights on the quality of additive-manufactured products.

We propose to conduct measurement studies using the pairwise similarity scores between topography measurements as pseudo univariate measurements. This practice facilitates the adoption of established measurement studies to process HD data while eliminating the need for complex modeling. In the field of ballistic forensics, Rice [24] adapts the Gauge R&R study to quantify different sources of variability in the similarity scores of bullet land-engraved areas (i.e., areas that contact the barrel of a gun closely). Other types of measurement studies leveraging similarity scores are also of interest and can be useful in practice. For example, analyzing the similarity scores of measurements obtained from the same surface by different devices will reveal the sensitivity of the similarity evaluation to devices. To the best of our best knowledge, there lacks a comprehensive study to conduct measurement studies on similarity evaluation under different scenarios.

To fill the above-mentioned research gaps, as shown in Figure 1, we create an intuitive framework to conduct measurement studies on the pairwise similarity scores of 3D surface topography data. We elaborate the details of this framework in Section 3. On the one hand, the average of the similarity scores in a measurement study sheds light on the uniformity or consistency of surface topography measurements. Hence, the mean similarity scores can be leveraged further for uniformity inspection in surface topography, as illustrated in our case studies in Section 4. On the other hand, the variation in the similarity scores reveals the statistical reproducibility of a similarity evaluation procedure, which indicates the sensitivity of the similarity scores to the changes in an analysis pipeline induced by factors such as operators, optical metrologies, and the evaluation procedure [24].

To the best of our knowledge, this study represents a pioneering effort in conducting comprehensive and systematic measurement studies focused on pairwise similarity scores derived from surface topography data. In a departure from conventional approaches, our investigation encompasses both the mean (Aim 1) and variation (Aim 2) in similarity, a distinctive feature that sets our research apart. This departure from conventionality is especially noteworthy since the mean is typically of limited practical significance in traditional measurement studies.

This research significantly contributes to the literature on measurement studies by specifically addressing high-density data, enriching our understanding of similarity assessments in this context. The proposed framework holds promise for applications in surface quality inspection, informed surface polishing decisions, and comparisons within optical sensing systems. For instance, by scanning discrete areas on a product’s surface, we can assess its smoothness and consistency in topography, thereby identifying specific regions that may require additional polishing to achieve the desired roughness. As a result, our work plays a pivotal role in enhancing the quality assurance cycle for additive manufacturing. Moreover, the versatility of our framework extends beyond surface topography data and additive manufacturing. It can seamlessly be applied to other forms of HD data, such as thermal imaging streams and X-ray computed tomography (XCT) image data, as well as diverse manufacturing processes, thereby facilitating effective quality assurance measures across various domains.

The rest of this article is organized as follows. Section 2 recaps the related literature and summarizes the existing challenges and gaps in measurement studies. Section 3 introduces the statistical modeling of variation quantification in seven scenarios using surface topography data. Section 4 shows the experiment study and analysis results and we also discuss some insights in Section 5. The article concludes in Section 6.

2. Related Literature Review

In this section, we introduce 3D optical sensing technologies in Section 2.1, review measurement studies in Section 2.2, and summarize similarity evaluation in manufacturing in Section 2.3.

2.1. Three-Dimensional Optical Scanning Technologies

In recent decades, 3D optical scanning technologies have been widely applied in the field of AM [3], forensics [24], reverse engineering [25], medicine [12], astronomy [26], etc. For example, such technologies enable the in situ monitoring of an AM process [18], and contactless scanning of a burnt victim in forensic science [27].

Many in situ optical sensing techniques have matured and become commercialized in recent decades. To name a few, there are structured light scanning (SLS) [28], and focus variation microscopy (FVM) [29,30], confocal microscopy [31,32], coherence scanning interferometry [33], and laser scanning [34]. These optical metrologies can rapidly generate millions of data points, i.e., point clouds, to portray the geometry of a part. These high-density (HD) data provide useful information, such as the texture, roughness, and fine structures, which are crucial for characterizing materials, assessing quality, and understanding physical properties.

However, significant discrepancy between measurements exists among 3D optical scanning technologies due to the intrinsic optical principles, sensor noise, or sampling approaches [35]. Associated manufacturers or users usually have different scanning instruments, the disagreements of which may cause confusion or miscommunication in practice, such as in a cyber-environment where surface data are shared by all members of the manufacturing streamline. In addition, the study of quality and adequacy of a measurement system is a prerequisite to implementing any statistical quality assurance, which is not well-established for 3D scanners.

This research facilitates the study of the adequacy of 3D scanners by using the pairwise similarity scores of surface topography data, as illustrated in Section 3 and Section 4 (Case 3). In this research, the data for the experimental study in Section 4 were collected using two widely used optical measurement systems including SLS and FVM. SLS has a much larger field view and lower resolution compared with FVM [36]. For SLS, the resolution is approximately 3100 pixels per inch while FVM has a resolution of 27,000 pixels per inch. FVM collects the points in grids while SLS does not.

2.2. Measurement Studies

The primary goal of measurement studies is to examine the validity, precision, and accuracy of a measurement procedure. Measurement studies are critical in a variety of fields since random error and discrepancy naturally exist in a process while adequate measurement is the prerequisite of any sensible quality assurance. Some common applications of measurement studies include the comparison between ground truth and the analytic results from measurements, device calibration, tolerance determination, and specification [37].

There are mainly three possible sources of discrepancy in a measurement context: variation in the measurands (e.g., different surfaces), variation in repeated measurements (e.g., repeated measurements of the same surface), and variation induced by different devices (e.g., using different optical sensors, different operators). The word “device” describes a particular combination of equipment, people, protocol, and procedures in a measurement procedure.

For univariate measurements such as length and weight, these different sources of variation can be easily quantified given properly collected data using traditional measurement studies. Vardeman and Jobe [13] present different data collection plans for measurement studies, introduce some basic one- and two-sample statistical measurement analysis methods on these data, and illustrate how physical variation interacts with a data collection plan as well as what can be learned from these data according to the corresponding data collection plan. For example, repeated measurements on the same measurand by a fixed device (one sample) help to conduct inference on the size of the device measurement noise, and comparing measurands from two stable processes with a fixed device (two samples) helps to examine the manufacturing variation in different fabricating processes. Gauge repeatability and reproducibility (R&R) study is another well-known measurement system analysis approach that aims to evaluate a gauge’s adequacy by quantifying two main variation components: repeatability and reproducibility [13,19]. The variation components can be estimated by a random effect Analysis of Variance (ANOVA) [19], or X-bar and range chart-based method [20,38]. ANOVA is a statistical method used to analyze the differences among group means. The core idea behind ANOVA is to compare the variance (or variability) between groups to the variance within groups. If the variance between groups is significantly larger than the variance within the groups, then it indicates a significant effect of the variable or treatment being tested.

To handle multivariate measurements, Wang and Yang [39] transfer related multiple quality characteristics to uncorrelated components by principal component analysis. Alternative Gauge R&R methods based on spatial statistics have been developed in the works by Takatsuji et al. [40], Niemczewska-Wójcik et al. [41], Piratelli-Filho and Di Giacomo [42] to model uncertainty in low-density data obtained from modern measurement systems such as coordinate measurement machines (CMMs). Gauge R&R based on advanced modeling for HD data also exists. For example, Dastoorian and Wells [22] examine the repeatability and reproducibility of the root mean square error from a Gaussian process model on the HD data portraying a flat whiteboard acquired by 3D laser scanners. However, the Gaussian process-based modeling is relatively complex, computationally expensive, and may introduce new variation in modeling which is not neglectable. There is a need to develop new approaches for measurement studies using HD data.

Another interesting application of measurement studies is to analyze the reproducibility of a data analysis procedure (e.g., similarity evaluation procedure). The statistical reproducibility of a data analysis procedure is defined as the sensitivity of the quantitative results to the changes in an analysis framework [24]. Rice [24] proposes a holistic process to quantify different sources of variability in the similarity scores of bullet land-engraved areas by extending the traditional Gauge R&R study. They extract 2D signatures from the 3D microscopic topography of bullets following a series of processing steps and use the random forest score as a similarity score. Then, a linear mixed model is utilized to quantify the repeatability and reproducibility of the similarity score. Rice [24] focuses on Gauge R&R study in the similarity scores between bullet land-engraved areas. Other types of measurement studies to analyze the reproducibility of a procedure can also be of interest and useful in practice. For example, we can assess the reproducibility of the similarity evaluation between measurements on the same surface when using different optic scanning machines. However, to the best of our best knowledge, a comprehensive study quantifying the variation in similarity evaluation across diverse scenarios is currently lacking in the existing literature.

2.3. Similarity Evaluation in Manufacturing

As we summarized in Section 1, similarity evaluation can be used to inspect surface uniformity and quantify the discrepancy (i.e., random errors) in surface topography measurements induced by repeated measurements, different operators, ambient noises, and different systems. In general, the surface’s topographical information should demonstrate similarity when input parameters of the process (e.g., laser power, powder feed rate, and toolpath control) are consistent.

Investigating the variation in the similarity scores is also valuable to reveal the statistical reproducibility of a similarity evaluation procedure. This is because the variation in similarity scores indicates the sensitivity of the similarity evaluation procedure to the random errors in an analysis pipeline induced by factors such as operators, optical metrologies, and the evaluation procedure. For example, studying how environmental noise affects the reproducibility of the similarity evaluation can provide guidance to eliminate these noises. To make it more concrete, when we make repeated measurements on a surface, if one of the measurements is disturbed by the unintentional movement of the surface during the measurement process, a precise and accurate optic scanner can capture the disturbance. Consequently, similarity scores between this problematic measurement and each of the other four measurements decrease, leading to higher variation in the pairwise similarity scores. Thus, the variation in the similarity scores in this example indicates the sensitivity of similarity evaluation to random errors.

There are mainly two types of similarity quantification methods of surface topography measurements generated by optic metrologies: the first type of approaches characterizes the topography discrepancy by mean and maximum height differences [43,44] or surface roughness comparison [45]. However, these methods oversimplify the spatial relation and geometric features of the surface topography measurements, which consequently leads to imprecise quantification. The second type of work uses a similarity metric to quantify the similarity in the surface geometry. Zhang et al. [46], Zheng et al. [47] use the Pearson correlation coefficient (PCC) to quantify the similarity of surface topography from AM products scanned by two optical sensors. Both Wang et al. [8], Jiang et al. [48] establish a thorough procedure of surface topography data similarity evaluation and generate a scaled similarity score for each pair of point-clouds data, while they evaluate the similarity in spatial domain and frequency domain, respectively.

We choose the methodology of similarity evaluation in the work by Wang et al. [8] to calculate the similarity score in Section 4 as a demonstration because it is thorough and flexible. Similarity metrics calculated by other methods such as in Jiang et al. [48] can also be used, which reflect the generalizability of our framework.

The intuition of the similarity evaluation method in the work by Wang et al. [8] is to compare the corresponding depth values at the shared lateral positions. If two point clouds portray the same surface, the depth values should be highly correlated. A point cloud of the surface topography $D\in R^{n\times k}$ is a collection of enormous points $D_i=(x,y,z(x,y))\in R^k$, where k represents the dimension of the point clouds (in our context, $k=3$, i.e., three-dimensional points), n is the number of points collected by the measurement system, and i is the point index. For example, $D_i^y$ is the y value of ith point in the point cloud. We may also treat x and y as the lateral location while z is the depth of a point on the surface. z can be treated as a function of x and y.

Wang et al. [8] include preprocessing steps to ensure that the two point clouds share the same coordinates, as detailed in Appendices Appendix A and Appendix B. Then, a similarity score is calculated by comparing the depth of each pair of points in pre-processed point clouds $\widetilde{D_1}$ and $\widetilde{D_2}$. The Pearson’s correlation coefficient (PCC), a commonly used similarity metric ranging from −1 to 1, is calculated by

$$PCC\left(\widetilde{D_1},\widetilde{D_2}\right)=\frac{{\sum }_{i=1}^n\left(\widetilde{D_{1l}^Z}-\overline{D_1^Z}\right)\left(\widetilde{D_{2l}^Z}-\overline{D_2^Z}\right)}{\sqrt{{\sum }_{i=1}^n{\left(\widetilde{D_{1l}^Z}-\overline{D_1^Z}\right)}^2}\sqrt{{\sum }_{i=1}^n{\left(\widetilde{D_{2l}^Z}-\overline{D_2^Z}\right)}^2}},$$

where $\widetilde{D_{1i}^z}$, $\widetilde{D_{2i}^z}$ are the depth values (i.e., z values) of the ith point in $\widetilde{D_1}$ and $\widetilde{D_2}$, and $\overline{D_1^Z}$ and $\overline{D_2^Z}$ are the mean depth values for the two point clouds $\widetilde{D_1}$ and $\widetilde{D_2}$, respectively. Larger PCC values indicate higher similarity while a negative PCC value suggests a negative correlation between two point clouds.

According to Hemphill [49], a PCC value larger than 0.3 indicates a strong correlation or higher similarity in our context. Notice that the criterion of highly similar can be different based on the application and data. Wang et al. [8] find thresholds for the similarity scores to determine whether two measurements are portraying the same surface, and the optimal threshold varies from 0.33 to 0.60 in different scenarios.

3. Measurement Studies Based on Similarity Evaluation

In this section, we illustrate how to conduct measurement studies using HD data by leveraging the similarity evaluation between 3D surface topography measurements. We exemplify the whole framework with seven typical case studies because the knowledge learned from a dataset, in other words, how measurement error is conveyed in the data, is affected largely by the interaction between physical variation and the corresponding data collection plan.

Notice that our premise is that the similarity evaluation of the surface topography is accurate in quantifying the similarity so the investigation in this article is sensible and practically useful. Another point is that traditional measurement studies have primarily focused on the variance because the mean usually does not have any practical meaning. In our study, both the mean and variance of the pseudo measurements are informative. This is because the mean indicates the discrepancy between topography measurements while the variance indicates the sensitivity of the similarity evaluation to different factors (e.g., operators, optical scanners, etc.).

This section is organized as follows: Section 3.1 illustrates the overall schematic of the framework. Section 3.2 introduces a generic model to achieve the two aims and adapts the model in seven cases. Section 3.3 elaborate the cases and statistical inference on the parameters.

3.1. Description of the Framework

The flowchart in Figure 1 in Section 1 demonstrates the schematic of the framework and we elaborate on each step in more detail as follows:

(1) First, we identify suitable data collection plans based on the needs. For example, if we are interested in the repeatability of a fabricating process, we require the same operator to scan the same specimen repeatedly using the same optical sensor to collect the surface topography data. For other possible data collection plans, we illustrate them in detail in Section 3.3.

(2) Second, we acquire the topography measurements using optical scanners from desired specimens.

(3) In the third step, the pairwise similarity scores of interests are calculated.

(4) In the fourth step, we build corresponding statistical models for measurement studies.

(5) In the fifth step, we examine either one or both the mean and the variation in the distribution of the similarity scores in a scenario of interests. For Aim 1, we quantify the discrepancy in surface topography data by examining the mean similarity score. We make inferences on the mean difference in the similarity scores when it comes to two-sample comparisons. For Aim 2, we quantify the sensitivity of the similarity evaluation procedure to various factors (i.e., operators, scanners, etc.) by conducting statistical inference on the variance of the similarity scores.

3.2. A Generic Model

After calculating pairwise similarity scores from surface topography measurements, we adapt traditional statistical concepts and methods for measurement studies. In this section, we describe a generic model for measurement analysis based on similarity evaluation. All the seven cases in Section 3.3 are special cases of this generic model.

Statistical Inference of the Generic Model

We denote $\tilde{S}$ to be the true similarity score between two surfaces, S to be the calculated similarity score from topography measurements D and $D^{\prime }$, and $\epsilon$ to be the random error produced during the measuring procedure and similarity evaluation. Notice that the true similarity score is only determined by the surface topography, excluding the influence of measurements and similarity evaluation errors. We also assume that the scanning systems in this research are linear. Then, a generic model is expressed as

$$S=\tilde{S}+\epsilon ,$$

(1)

where $\tilde{S}\sim Normal\left({\mu }_{\phi },{\sigma }_{\phi }^2\right),\epsilon \sim Normal\left(\delta ,{\sigma }^2\right)$, and they are independent. Then, $S\sim Normal\left({\mu }_S,{\sigma }_S^2\right)$, where ${\mu }_S={\mu }_{\phi }+\delta$, and ${\sigma }_S^2={\sigma }_{\phi }^2+{\sigma }^2$. The calculated similarity score S consists of two components: the true similarity score of surfaces $\tilde{S}$ and the error induced by factors such as the optical scanners, operators, and the similarity evaluation algorithm. ${\mu }_{\phi }$ stands for the mean value of the true similarity score of the surfaces, which reflects the discrepancy in the surface topography. ${\sigma }_{\phi }^2$ represents the variance in the true pairwise similarity scores, which can be used to examine the stability of a process producing many surfaces. $\epsilon$ is the error induced by the measuring process and similarity evaluation. The parameter $\delta$ is interpreted as the bias in the similarity score induced by factors like the optic sensors and operators during the measuring process, and the similarity evaluation procedure.

Notice that the evaluation algorithm in the work by Wang et al. [8] yields the same similarity score given the same pair of point clouds. Hence, the bias induced by the similarity evaluation is constant. Therefore, ${\sigma }^2$, in this case, is the variation in the similarity scores that are only induced by measuring process errors, which is further decomposed into two components ${\sigma }_{repeatablity}^2$ and ${\sigma }_{reproducibility}^2$.

For traditional measurement studies, the repeatability variation is the error in the measurements induced by repeated measurements while the reproducibility variation is the error in the measurements induced by operators. To distinguish from traditional studies, in our study, we define ${\sigma }_{repeatablity}^2$ to be the error in the similarity score evaluation instead of the measurements induced by repeated measurements, which indicates the sensitivity of the similarity score to factors such as environmental noises in the repeated measuring process. Similarly, we define ${\sigma }_{reproducibility}^2$ as the error in the similarity score evaluation induced by devices, which reflects the sensitivity of the similarity score to different optical scanners or different operators.

For Aim 1, we examine the mean value of the true similarity score ${\mu }_S$ and the bias $\delta$ to discuss the variation in the topography measurements. For Aim 2, we estimate the variation in the true similarity score ${\sigma }_{\phi }^2$ and the random error ${\sigma }^2$ to quantify the sensitivity of the similarity evaluation procedure to the measuring process errors.

3.3. Measurement Studies under Different Scenarios

In this section, we exemplify how to conduct measurement studies by leveraging the similarity evaluation between surface topography measurements with seven cases.

Table 1. Summary of the data collection plan and parameter estimation including mean (Aim 1) and variance (Aim 2) for each case.

Case	Data Collection Plan	Parameters	Estimation: Point Estimate, CI	Statistical Inference
1	Multiple measurements scanned using the same device from a single surface	${\mu }_S=\tilde{S}+\delta$	Equation (2), Equation (3)	One-sample
		${\sigma }_S^2={\sigma }^2$	Equation (6), Equation (7)
2	Single measurement on different surfaces from a stable process	${\mu }_S={\mu }_{\phi }+\delta$	Equation (2), Equation (3)
		${\sigma }_S^2={\sigma }_{\phi }^2+{\sigma }^2$	Equation (6), Equation (7)
3	Measurements on each surface using two devices	${\mu }_{S_i}=\tilde{S}+\delta$	Equation (2), Equation (3)
		${\sigma }_{S_i}^2={\sigma }^2$	Equation (6), Equation (7)
4	Comparing two sets of repeated measurements from two surfaces using a single device	${\mu }_{S_i}=\tilde{S}+\delta$	Equation (2), Equation (4)	Two-sample
		${\sigma }_{S_i}^2={\sigma }^2$	Equation (6), Equation (8)
5	Comparing two sets of surfaces using the same device	${\mu }_{S_i}={\mu }_{\phi }+\delta$	Equation (2), Equation (4)
		${\sigma }_{S_i}^2={\sigma }_{\phi }^2+{\sigma }^2$	Equation (6), Equation (8)
6	One possible data collection plan for estimating process variation by comparing two samples	${\mu }_{S_1}={\mu }_{\phi }+\delta$ ${\mu }_{S_2}=\tilde{S}+\delta$	Equation (2), Equation (4)
		${\sigma }_{S_1}^2={\sigma }_{\phi }^2+{\sigma }^2$ ${\sigma }_{S_2}^2={\sigma }^2$	Equation (6), Equation (8)
7	One-way random-effects model for a single measurement system	${\mu }_S={\mu }_{\phi }$	ANOVA	ANOVA
		${\sigma }_S^2={\sigma }_{\phi }^2+{\sigma }^2$

provides a summary of the data collection plan and parameter estimation including mean (Aim 1) and variance (Aim 2) for each case. For Cases 1 to 3, one-sample statistical methods are applied while two-sample statistical comparisons are used in Cases 4 to 6. In Case 7, a one-way random effect model is used. Other types of studies can be conducted similarly, and the models can be adjusted based on the application context. We extend the generic model in each case and summarize the interpretation of parameters for Cases 1–3 in

Table 2. Interpretation of the parameters in the generic model in Equation (1) for Cases 1–3.

Case	1	$\tilde{\mathit{S}}$	${\mathit{\mu }}_{\mathit{\phi }}$	$\mathit{\epsilon }$	$\mathit{\delta }$	${\mathit{\sigma }}^2$
1	True similarity score, constant value	$\tilde{S}$	0	Random error in the similarity score	Bias in the similarity score induced by repeated measurements and the similarity evaluation process	Variation in the similarity scores induced by repeated measurements, i.e., ${\sigma }_{repeatablity}^2$
2	True similarity score, random value	Mean value of the true similarity scores of surfaces	Variance in the true pairwise similarity scores due to different surfaces	Same as Case 1	Same as Case 1	Same as Case 1
3	Same as Case 1	$\tilde{S}$	0	Same as Case 1	Same as Case 1	Variation in the similarity scores due to the combined influence of two devices, i.e., ${\sigma }_{reproducibility}^2$

. For Cases 4–7, the interpretations are similar except that the parameters are for the differences between the two samples. We present the details of parameter estimation in the following paragraphs.

For Aim 1 of Step 5, in one-sample Cases 1–3, we estimate the mean by maximum likelihood estimation (MLE) following the equation:

$$\widehat{{\mu }_S}=\frac{1}{N}\sum _{i=1}^N{S}_i,$$

(2)

where N is the number of total calculated pairwise similarity scores and i is the similarity score index. The corresponding confidence interval (CI) of ${\mu }_S$ is

$$\left(\widehat{{\mu }_S}+t_{\frac{\alpha }{2},v}\sqrt{\frac{\widehat{{\sigma }_S^2}}{\mathrm{N}}},\widehat{{\mu }_S}+t_{1-\frac{\alpha }{2},v}\sqrt{\frac{\widehat{{\sigma }_S^2}}{\mathrm{N}}}\right),$$

(3)

where $\widehat{{\sigma }_S^2}$ is estimated by Equation (4), $t_{\frac{\alpha }{2},v}$ is the lower $\frac{\alpha }{2}$ percentile of the t-distribution, with the degree of freedom to be $v=N-1$.

For Aim 1 of Step 5, in two sample comparisons in Cases 4–6, the difference of two sets of surfaces in similarity scores is estimated as $\widehat{{\mu }_{S_1}}-\widehat{{\mu }_{S_2}}$, where the $\widehat{{\mu }_{S_1}}$ and $\widehat{{\mu }_{S_2}}$ are the estimated mean similarity scores for two sets of surfaces, respectively. $N_1$ and $N_2$ are the corresponding sample sizes. $\widehat{{\sigma }_{S_1}^2}$ and $\widehat{{\sigma }_{S_2}^2}$ are the estimated variance in the similarity score of the two samples. The CI for the mean difference is

$$\left({\widehat{\mu }}_1-{\widehat{\mu }}_2+t_{\frac{\alpha }{2},\widehat{v}}\sqrt{\frac{\widehat{{\sigma }_{S_1}^2}}{N_1}+\frac{\widehat{{\sigma }_{S_2}^2}}{N_2}},{\widehat{\mu }}_1-{\widehat{\mu }}_2+t_{1-\frac{\alpha }{2},\widehat{v}}\sqrt{\frac{\widehat{{\sigma }_{S_1}^2}}{N_1}+\frac{\widehat{{\sigma }_{S_2}^2}}{N_2}}\right).$$

(4)

The approximate Satterthwaite degree of freedom [50] is

$$\widehat{v}=\frac{{\left(\frac{\widehat{{\sigma }_{S_1}^2}}{N_1}+\frac{\widehat{{\sigma }_{S_2}^2}}{N_2}\right)}^2}{\frac{{\widehat{{\sigma }_{S_1}^2}}^2}{\left(N_1-1\right)N_1^2}+\frac{{\widehat{{\sigma }_{S_2}^2}}^2}{\left(N_2-1\right)N_2^2}}.$$

(5)

For Aim 2 of Step 5, in one-sample Cases 1–3, the MLE estimator of the variation in similarity is

$$\widehat{{\sigma }_S^2}=\frac{1}{N-1}\sum _{i=1}^N{\left(S_i-\widehat{{\mu }_S}\right)}^2.$$

(6)

The CI for ${\sigma }_S^2$ is

$$\left(\widehat{{\sigma }_S^2}\frac{v}{x_{1-\frac{\alpha }{2},v}^2},\widehat{{\sigma }_S^2}\frac{v}{x_{\frac{\alpha }{2},v}^2}\right),$$

(7)

where $x_{\frac{\alpha }{2},v}^2$ is the lower $\frac{\alpha }{2}$ percentile of the $x^2$ distribution whose degree of freedom is $v=N-1$. Notice that the confidence interval limits of the ${\sigma }_S^2$ must be non-negative.

For Aim 2 of Step 5, in two sample comparisons in Cases 4–6, the ratio of the variation in similarity scores calculated from two sets of surfaces is estimated as

$$\left(\frac{\widehat{{\sigma }_{S_1}}}{\widehat{{\sigma }_{S_2}}}\frac{1}{\sqrt{F_{1-\frac{\alpha }{2},\left(N_1-1\right),\left(N_2-1\right)}}},\frac{\widehat{{\sigma }_{S_1}^2}}{\widehat{{\sigma }_{S_2}^2}}\frac{1}{\sqrt{F_{\frac{\alpha }{2},\left(N_1-1\right),\left(N_2-1\right)}}}\right),$$

(8)

where $F_{\frac{\alpha }{2},\left(N_1-1\right),\left(N_2-1\right)}$ is the lower $\frac{\alpha }{2}$ percentile of the F-distribution whose two degree of freedoms are $N_1$ and $N_2$. The variance in the similarity score indicates the sensitivity of the similarity evaluation to the uncertainty in a measurement process, as elaborated in Section 3.3.2.

In Case 7, a one-way random effects model is used, and we elaborate it in Section 3.3.3. The parameters can be estimated by well-established REML (restricted maximum likelihood) [51,52] of the one-way Analysis of Variance (ANOVA) model. ANOVA is a statistical method used to analyze the differences among group means in a sample. In the context of measurement studies, ANOVA is particularly useful for determining whether there are statistically significant differences between the mean measurements of three or more independent (unrelated) groups. ANOVA analysis is available in many statistical packages (e.g., JMP, R, and SAS).

3.3.1. One-Sample Measurement Studies

Case 1: Multiple measurements are scanned using the same device from a single surface. As Figure 2 shows, one makes multiple measurements (m times) on the same surface by the same device to quantify the variation in the measurements and the similarity scores induced by repeated measurements. Again, the word “device” describes a particular combination of equipment, operators, procedures, etc. For example, different operators using the same scanner and following the same measuring procedure are counted as different devices. Since the measurements are obtained from the same surface, $\tilde{S}$ in Equation (1) is no longer a random variable. The true similarity scores $\tilde{S}$ if one of the two repeated measurements are exactly the same. The random error $\epsilon$ is induced by the measuring and similarity evaluation procedures, and it satisfies the same assumption as in the generic model in Equation (1).

For Aim 1, we estimate ${\mu }_S=\tilde{S}+\delta$ and 95% CI of ${\mu }_S$ by Equation (2) and Equation (3), respectively. ${\mu }_S$ reflects the variation in the measurements induced by repeated measurements. A high similarity score ${\mu }_S$ indicates that the repeated surface topography measurements are rather similar to each other so the repeatability variation in the measurements of the optical scanners is relatively small, which suggests that the device works accurately. On the contrary, a low value of ${\mu }_S$ suggests that the optical scanner is incapable of measuring the surface accurately, which can be possibly eliminated by proper calibration.

For Aim 2, we estimate ${\sigma }_S^2={\sigma }^2$ and the 95% CI of ${\sigma }_S^2$ by Equation (6) and Equation (7), respectively. ${\sigma }_S^2$ reflects the sensitivity of the similarity score evaluation to the error induced by repeated measurements, i.e., ${\sigma }_{repetability}^2$. If a valid, precise, and accurate optical scanner is used in Case 1, we obtain very similar repeated measurements, and thus the similarity scores show a narrow spread with a high mean value (i.e., high ${\mu }_S$ and low ${\sigma }_S^2$). On the other hand, if an optical scanner is not properly calibrated, the variation ${\sigma }_S^2$ in the similarity scores induced by repeated measurements would be bigger. In this circumstance, the average similarity score is still close to one while the distribution of the similarity scores tends to be left-skewed. Consequently, the mean similarity scores ${\mu }_S$ are expected to decrease slightly, and the variation ${\sigma }_S^2$ in similarity increases dramatically.

Case 2: Single measurements on different surfaces from a stable manufacturing process. Though humans may play a part in the variances of the same type of products from the same process, the focus of this research is on the measurement processes instead of the manufacturing processes. We assume that the manufacturing process is stable. As Figure 3 shows, single measurements are made on m different surfaces from a stable process by the same device to quantify the reproducibility variation in the measurements and similarity scores induced by different surfaces. For the generic model in Equation (1) for Case 2, $\tilde{S}$ stands for the true similarity score between surfaces printed by a stable AM process while $\epsilon$ represents the random error induced during the measurement and similarity evaluation procedures.

For Aim 1, the mean similarity score ${\mu }_S$ consists of two components: the true similarity score ${\mu }_{\phi }$ between surfaces, and the bias $\delta$ in the similarity score. ${\mu }_{\phi }$ reflects the discrepancy between surface topography. $\delta$ is interpreted as the bias in the similarity scores induced by the measuring process and similarity evaluation algorithm. If surfaces fabricated by a stable process are similar to each other, the estimated mean similarity score ${\mu }_S$ is comparatively high. Compared with Case 1, the mean similarity scores ${\mu }_S$ are expected to be lower since the measurements are collected from different surfaces. In other words, we expect a smaller true similarity score ${\mu }_{\phi }$ between surfaces than in Case 1.

For Aim 2, there are two possible sources of variation in the similarity scores: ${\sigma }_{\phi }^2$, the variation in the true similarity scores induced by the fabricating process, and ${\sigma }^2$, the variation in the similarity induced by the measuring process. ${\sigma }_{\phi }^2$ is closely related to the fabrication process stability. To make it more concrete, the noises during the fabricating process, such as an unstable power supply to the printing machine, lead to a few defective products. Consequently, the variation ${\sigma }_{\phi }^2$ in surfaces from the fabricating process increases dramatically. ${\sigma }^2$ only contains the repeatability variation ${\sigma }_{repeatability}^2$ in the similarity score. If the similarity score evaluation is sensitive to the uncertainty during the measurement procedure, the value of ${\sigma }^2$ increases. Ultimately, compared with Case 1, we expect a higher variation ${\sigma }_S^2$ in the similarity score since there is a new source of variation ${\sigma }_{\phi }^2$.

Case 3: Measurements on each surface using two devices. As Figure 4 shows, a pair of single measurements are made on different surfaces from a stable process by two different devices to examine the variation in the measurements and similarity scores induced by different devices. Again, the device is a combination of equipment, people, procedures, etc. Therefore, the two devices can be two different scanners with the same operator, or two operators with the same scanner. The statistical model of Case 3 is the same as the one in Case 1 but is different in interpretation. Similar to Case 1, $\tilde{S}$ stands for the true similarity score between two surfaces. Since we are comparing the same surface, $\tilde{S}$ is a constant value of 1. $\epsilon$ represents the combined influence of the random error produced during the measurement procedure by two devices and the error in the similarity evaluation.

The method described by Case 3 can be extended to multiple devices, similar to the extension from two-sample comparisons to multiple comparisons. Then, test errors need to be adjusted to accommodate the inflated test errors induced by multiple comparisons [53].

For Aim 1, we estimate the mean value of the similarity score $\left({\mu }_S=\tilde{S}+\delta \right)$ by Equation (2) and 95% CI of ${\mu }_S$ by Equation (3). ${\mu }_S$ reflects the reproducibility variation in surface measurements induced by different devices. A high similarity score ${\mu }_S$ indicates that the surface topography measurements obtained by the two devices are rather similar to each other. Furthermore, the mean similarity score ${\mu }_S$ is close to the value in Case 1 if the intrinsic difference between devices is neglectable.

For Aim 2, we estimate the variation in the similarity score $\left({\sigma }_S^2={\sigma }^2\right)$ by Equation (6) and the CI of ${\sigma }_S^2$ by Equation (7). ${\sigma }_S^2$ in Case 3 reflects the combined influence of different devices on the sensitivity of the similarity evaluation to the uncertainty during the measurement process. Compared with Case 1, ${\sigma }^2$ represents the sensitivity of the similarity score to the noises in the measuring process induced by different devices, i.e., ${\sigma }_{reproducibility}^2$.

3.3.2. Two-Sample Measurement Studies

Case 4: Comparing two sets of repeated measurements on two surfaces using a single device. As Figure 5 shows, we conduct two-sample comparisons on two surfaces repeatedly measured by the same device to examine the discrepancy in the measurements and similarity scores induced by repeated measurements between different surfaces. Each set of samples follows the same model in Case 1. We use the subscription l to denote the similarity in the sample set l. The statistical model is as follows:

$$S_l=\tilde{S}+\epsilon ,l=1,2.$$

For Aim 1, the difference of the mean similarity score ${\mu }_{S_1}-{\mu }_{S_2}$ and the corresponding CI are estimated by Equation (2) and Equation (4), respectively. On the one hand, the true similarity score $\tilde{S}$ in each sample should be one because the measurements are from the same surface. On the other hand, the bias $\delta$ induced by the measuring process and the similarity evaluation should be the same for the two samples since measurements are obtained by the same device. Thus, it is obvious that the difference value of the similarity score ${\mu }_{S_1}-{\mu }_{S_2}$ is zero no matter whether the two surfaces are fabricated by the same process or not.

For Aim 2, the ratio of the variation in the similarity score $\frac{{\sigma }_{S_1}^2}{{\sigma }_{S_2}^2}$ and the corresponding CI are estimated by Equation (6) and Equation (8), respectively. Clearly, the ratio of the variation in the similarity score is one because the measurements are obtained by the same device.

Case 5: Comparing two sets of surfaces using the same device. As Figure 6 shows, we compare two sets of surfaces from two processes to examine the difference in the variation in the measurements and similarity scores induced by two fabricating processes. In each sample set, the measurements are obtained from different surfaces fabricated by a stable process. Notice that we can also compare different batches of surfaces fabricated by the same process, which is a special scenario of Case 5. Comparing two sets of measurements from the same process provides insight into the stability of the process. Similar to Case 4, we use the subscription l to denote the similarity in a sample set l. The model for each sample set is the same as the model in Case 2.

For Aim 1, the difference of the mean similarity score ${\mu }_{S_1}-{\mu }_{S_2}$ and the corresponding 95%CI are estimated following Equation (2) and Equation (4), respectively. The difference value compares the consistency in the surface topography of the products fabricated by two processes. A positive value of ${\mu }_{S_1}-{\mu }_{S_2}$ indicates a smaller discrepancy in the surface topography of the products fabricated by the first process. If we compare different batches of the surfaces fabricated by the same process, similar to Case 4, the difference value is expected to be zero. A large magnitude in the difference value indicates that one batch of the products has a more uniform topography than the other one.

For Aim 2, the ratio of the variation in the similarity score $\frac{{\sigma }_{S_1}^2}{{\sigma }_{S_2}^2}$ and the corresponding 95% CI are estimated following Equation (6) and Equation (8), respectively. This ratio compares the stability of two processes. Both sets of samples share the same variation in the similarity score (${\sigma }^2$) induced by the measuring process since the measurements are obtained by the same device. The stability of two processes mainly influences the ratio $\frac{{\sigma }_{S_1}^2}{{\sigma }_{S_2}^2}$. The lower confidence limit is larger than one if the first fabricating process is less stable.

Case 6: One possible data collection plan for estimating the variation in the similarity scores by comparing two samples. As Figure 7 shows, in the first sample, the similarity scores are calculated using the single measurements on multiple surfaces obtained by the same device. In the second sample, the similarity scores are calculated using repeated measurements on a surface obtained by the same device as the first sample. Notice that the surface in the second sample is different from the surfaces in the first sample. The first sample follows the same model as Case 2 while the second sample can be described by the model in Case 1.

For Aim 1, the difference of the mean similarity score ${\mu }_{S_1}-{\mu }_{S_2}$ and the corresponding CI are estimated following Equation (2) and Equation (4), respectively. We have discussed in Case 1 that $\tilde{S}$ is one and $\tilde{S}\le 1$ in Case 2. Therefore, the difference value in the similarity score of two samples in Case 6 is nonpositive. A larger magnitude in difference indicates that the surfaces fabricated from the process are not very similar to each other in topography. Thus, there exists significant dissimilarity between measurements of the surfaces in the same process.

For Aim 2, the ratio of the variation in the similarity score $\frac{{\sigma }_{S_1}^2}{{\sigma }_{S_2}^2}=\frac{\sigma {\phi }^2+{\sigma }^2}{{\sigma }^2}$ and the corresponding 95% CI are estimated following Equation (6) and Equation (8), respectively. It is obvious that the ratio of the variation is larger than one. A ratio much larger than one indicates that the variation in the similarity score induced by the measuring process and similarity evaluation contributes less than the variation in the fabricating process.

3.3.3. One-Way ANOVA

Case 7: One-way random-effects model for a single scanning system. A typical measurement study consists of repeated measurements on several surfaces from a stable process using a fixed device, as shown in Figure 8. It is standard to model these measurements as independent random draws from some distributions and use the one-way random-effects model to make inferences. Therefore, we use random effects for surfaces.

Define J as the total number of surfaces, $m_j$ as the total number of measurements for Surface j. For Surface j, we calculate the pairwise similarity score $S_{l{l}^{\prime }}^j$ for surfaces l and $l^{\prime }$ and there are $\frac{m_j(m_j-1)}{2}$ of them. The corresponding one-way random-effects model is

$$S_{l{l}^{\prime }}^j=\widetilde{S_j}+{\epsilon }_{l{l}^{\prime }}^j,l,l^{\prime }=1,2,\cdots ,m_j,l\ne l^{\prime },j=1,2,\cdots ,J,$$

where $\widetilde{S_j}\sim N({\mu }_{\phi },{\sigma }_{\phi }^2)$, ${\epsilon }_{l{l}^{\prime }}^j\sim N(0,{\sigma }^2)$, and they are all mutually independent. $\widetilde{S_j}$ is the mean similarity score of repeated measurements for Surface j drawn at random from a normal distribution with a mean of ${\mu }_{\phi }$ and variance of ${\sigma }_{\phi }^2$. ${\mu }_{\phi }$ reflects the repeatability variation in the surface measurements while ${\sigma }_{\phi }^2$ is interpreted as the between-group variance in the similarity score attributed to different surfaces. ${\epsilon }_{l{l}^{\prime }}^j$ is the random error induced by the measuring process when calculating the similarity score of surfaces l and $l^{\prime }$. ${\sigma }^2$ is the within-group variance in the similarity score attributed by repeated measurements of the same surface, which represents the sensitivity of the similarity score evaluation to the uncertainty in repeated measurements. These parameters are estimated by well-established methods of inference for random-effects models, such as the restricted maximum likelihood (REML) approach [51], which are available in many statistical packages, e.g., JMP, R, and SAS.

We also transfer the concept of a popular quality-of-measurement metric to assess the performance of the measurement system in Case 7. The ratio $\frac{{\sigma }^2}{{\sigma }^2+{\sigma }_{\phi }^2}$ is the fraction of repeatability variance in the similarity score attributable to the total variance in the similarity score. It reflects the proportion of the variance in the similarity score explained by the measurement system out of the total variance in the similarity score.

4. Experimental Studies Using Measurements from Additive-Fabricated Specimens

In this section, to demonstrate the effectiveness of our holistic framework, we apply the seven case studies in Section 3 to the data and similarity scores provided in the work by Wang et al. [8]. We describe the data in Section 4.1 and present the results of the seven cases in Section 4.2. The details for the similarity evaluation method in Wang et al. [8] are summarized in Section 2 and Appendix A. The data we used are openly available in Harvard Dataverse at https://doi.org/10.7910/DVN/0NLH95 (accessed on 1 October 2020).

4.1. Data Description

For the data collection, four AM single-layered specimens with tool paths in four different directions shown in Figure 9a were manufactured by a direct energy deposition (DED) additive printer with 316L steel as the powder metal material. DED is a collection of AM processes that feeds materials to a substrate while melting the powder or wire simultaneously by either an electron beam or plasma arc [54]. The size of each specimen is $15\times 15$ ${\mathrm{cm}}^2$. To obtain more data, handmade tinfoil masks were placed to segment five small windows on each specimen, as shown in Figure 9b, and the size of each window is $3\times 3$ ${\mathrm{cm}}^2$. Figure 9c shows a 3D surface point cloud of a window on Specimen B scanned by SLS, where the color represents the depth of the points. There are several obvious gullies, which are in the same direction as the tool paths during the printing process.

As shown in Figure 10, the four metal specimens shown in Figure 9 were scanned by three operators using two scanning systems (SLS and FVM). For SLS, the field of view covers the whole specimen with five windows. Operators I and II scanned each surface once while operator III repeatedly scanned each surface five times. The SLS scan of each specimen was cropped into five point clouds, each of which covers most areas of one window. Overall, for SLS, one operator scanned each window five times and the other two operators scanned each window once. For FVM, each operator scanned each window once. Due to limited resources, the repeated measurements for FVM were not collected. In total, there are 60 pieces of point-cloud data from FVM and 140 pieces of point-cloud data from SLS. There are 200 point clouds in total.

4.2. Case Study Results

In this section, we present the numeric results and interpret the corresponding findings of each case. The seven cases in this section correspond to the seven cases in Section 3.3. Notice that we only used a subset of the original data for each case as an illustration.

4.2.1. One-Sample Measurement Studies

Case 1: Multiple measurements were scanned using the same device from a single surface. In Case 1, to examine the variation induced by repeated measurements, we randomly selected four windows from two of the specimens for demonstration. The similarity scores are calculated between surface topography measurements collected by one particular operator (Operator III) using SLS on the same window from a specimen. Each window on a specimen is measured repeatedly five times, yielding 10 pairwise similarity scores.

Table 3. Case 1: Parameter estimation on the mean and standard deviation of the similarity scores of four randomly selected windows scanned by the same operator using SLS.

Specimen	Window	${\mathit{\mu }}_{\mathit{S}}$	${\mathit{\sigma }}_{\mathit{S}}$	95% CI of ${\mathit{\mu }}_{\mathit{S}}$	95% CI of ${\mathit{\sigma }}_{\mathit{S}}$
B	3	$0.620$	$0.082$	$(0.561,0.678)$	$(0.056,0.150)$
B	4	$0.664$	$0.069$	$(0.615,0.714)$	$(0.047,0.125)$
C	1	$0.737$	$0.063$	$(0.692,0.782)$	$(0.043,0.114)$
C	3	$0.655$	$0.094$	$(0.587,0.722)$	$(0.065,0.172)$

shows the parameter estimation on the mean and standard deviation of the similarity scores, where each row shows the estimation of parameters in Equation (6) using 10 similarity scores from one window. ${\mu }_S$ and ${\sigma }_S$ stand for the mean value and standard deviation of the similarity scores, respectively. Since ${\mu }_S$ is the mean of the similar scores from repeated measurements of the same surface, ${\mu }_S$ reflects the variation in the topography measurements induced by repeated measurements. ${\sigma }_S$ is the standard deviation of the similar scores and reflects the sensitivity of the similarity evaluation procedure to the measurement error induced by repeated measurements. The last two columns in Table 3 show the corresponding 95% CIs of ${\mu }_S$ and ${\sigma }_S$, correspondingly. For each interval, we are 95% confident that the interval includes the true parameter value.

For Aim 1 of Step 5 in Figure 1, Table 1 shows the mean similarity score ${\mu }_S$ of some randomly selected windows on specimens, which are all much larger than 0.3. In addition, the average ${\mu }_S$ of all the 20 windows (five windows per specimen) is 0.673, which is far higher than 0.3. The lower bound of 95% CI of ${\mu }_S$ is also above 0.3, which indicates pronounced similarity. Such a high mean value of the similarity score indicates a negative value of $\delta$ with a small magnitude, which further suggests a small discrepancy in surface measurements induced by repeated measurements using SLS. Such results confirm our expectation since SLS is a mature optic scanning device and is relatively stable in collecting surface topography information accurately under suitable conditions and operation. It is reasonable that various noises during the measuring process (e.g., improper calibration, changes in ambient light or temperature, smudges, etc.) will increase the discrepancy between surface measurements. Therefore, we cannot obtain the same surface measurements and the similarity score is always smaller than 1.

For Aim 2 of Step 5 in Figure 1, all the ${\sigma }_S$ values are smaller than 0.1. The average value of the estimated standard deviation in the similarity score ${\sigma }_S$ of all the 20 windows is only 0.09, which is a small variation compared to the variation in the similarity scores in other cases. A small variation in the similarity score shows that the similarity evaluation is not sensitive to the error induced by the repeated measurements.

To summarize, these results show that both the repeatability variation in the topography measurements and the degree of sensitivity of the similarity evaluation to the error induced by repeated measurements are small in Case 1. Under proper calibration and operation, SLS is capable of collecting rather precise surface measurements and the similarity evaluation process is not sensitive to the error induced by the repeated measurements.

Case 2: Single measurement on different surfaces from a stable process. In Case 2, to quantify the variation attributed to different surfaces, a single measurement is made on different surfaces from a stable process using the same device. Notice that there are five small windows on each specimen for the data we used. Then, we consider two scenarios in Case 2. For Scenario 1, we considered the surface topography measurements collected from all five windows on the same specimen using the same device. For each window, if there are repeated measurements, i.e., each window was measured five times by Operator III using SLS, we randomly selected one measurement. For Scenario 2, we randomly selected one window from each specimen and considered a single measurement using the same device per window. Scenario 1 examines the variation in different surfaces on the same specimen while Scenario 2 compares the surface variation on different specimens.

For Case 2 Scenario 1, to examine the influence of different scanners on the similarity score, Figure 11 shows the distribution of the pairwise similarity scores from the four specimens where each similarity score is calculated from a pair of measurements from different windows on the same specimen obtained by the same operator using the same scanner. For the five repeated measurements taken by Operator III using SLS, we only considered the first measurement. There are four specimens, three operators, and five windows per specimen. On the same specimen, there are 10 similarity scores. Thus, we have 120 $(10\times 3\times 4)$ similarity scores in each histogram. Figure 11a,b show the distribution of the similarity scores obtained by SLS and FVM, respectively, and (c) is the boxplot of these scores categorized by scanners. Most of the similarity scores by SLS fall between 0.1 and 0.3, and the scores look normally distributed. On the contrary, the mean similarities score by FVM is around 0.7 and the distribution is skewed to the left. The average similarity score obtained by FVM is much higher than the one obtained by SLS. Moreover, there are some obvious outliers of the similarity scores obtained by FVM, which are also marked in the boxplot in Figure 11c. We can see that, although the higher resolution of FVM results in higher similarity scores, FVM is more sensitive to uncertainty during a measurement procedure.

For Scenario 1,

Table 4. Case 2: Parameter estimation on the mean and variation of the similarity scores calculated from single measurements on different surfaces from a stable process.

Scenario	Scanner	Operator, Specimen	${\mathit{\mu }}_{\mathit{S}}$	${\mathit{\sigma }}_{\mathit{S}}$	95% CI of ${\mathit{\mu }}_{\mathit{S}}$	95% CI of ${\mathit{\sigma }}_{\mathit{S}}$
1: Surfaces on the same specimen	SLS	I, B	0.445	0.058	$(0.403,0.486)$	$(0.040,0.106)$
	SLS	I, D	0.319	0.068	$(0.270,0.368)$	$(0.047,0.124)$
	SLS	II, B	0.335	0.048	$(0.301,0.370)$	$(0.033,0.088)$
	SLS	II, D	0.240	0.102	$(0.167,0.313)$	$(0.070,0.187)$
	FVM	I, B	0.701	0.052	$(0.663,0.738)$	$(0.036,0.095)$
	FVM	I, D	0.698	0.091	$(0.633,0.764)$	$(0.063,0.167)$
	FVM	II, B	0.736	0.091	$(0.671,0.801)$	$(0.063,0.167)$
	FVM	II, D	0.652	0.139	$(0.553,0.752)$	$(0.096,0.254)$
2: Surfaces from different specimens	SLS	I, A–D	0.076	0.042	$(0.031,0.121)$	$(0.026,0.104)$
	SLS	I, A–D	0.035	0.073	$(-0.042,0.112)$	$(0.046,0.179)$
	SLS	III, A–D	0.084	0.045	$(0.037,0.131)$	$(0.028,0.110)$
	FVM	I, A–D	0.197	0.158	$(0.031,0.363)$	$(0.099,0.388)$
	FVM	II, A–D	0.159	0.094	$(0.060,0.258)$	$(0.059,0.231)$
	FVM	III, A–D	0.200	0.155	$(0.037,0.363)$	$(0.097,0.381)$

shows the parameter estimation on the mean and variation in the similarity scores, where each row shows the estimation of parameters in Equation (6) using 10 similarity scores of the windows on the same specimen. We only show part of the results in the upper half of Table 4. (1) For Aim 1, we notice that there is a pronounced difference in ${\mu }_S$ when using SLS and FVM under the same measurement condition. The average similarity score in Table 4 of using FVM is above 0.3 while the average value is below 0.3 using SLS. The average similarity score of all specimens and operators using FVM is 0.27 higher than using SLS, which indicates that SLS has a smaller bias value $\delta$ in the similarity score than FVM according to Equation (1). Although we did not have the repeated measurements using FVM, we expect FVM to have a higher similarity score than SLS in Case 1. Compared with Case 1, the mean similarity score when using SLS is rather small, because the discrepancy between windows is naturally larger than repeated measurements. (2) For Aim 2, the variation in the similarity ${\sigma }_S$ is higher when the measurements are obtained using FVM. The average ${\sigma }_S$ of using SLS is 0.083 while ${\sigma }_S$ of using FVM is 0.157. Large variation in the similarity score ${\sigma }_S$ suggests that the similarity score evaluation is more sensitive to the error induced by measuring process using FVM, which is the same conclusion as we draw from Figure 10.

In Scenario 2, as Table 4 shows, we randomly select one window from each specimen for each row. Then, the results per row are calculated from six similarity scores. (1) For Aim 1, the mean similarity scores in Table 4 are lower than 0.3, which indicates that the two topography measurements portray different surfaces. The mean similarity score ${\mu }_S$ decreases compared with Scenario 1 because the surface discrepancy across specimens is more significant than the one within a specimen, i.e., smaller ${\mu }_{\phi }$. We also notice that FVM demonstrates a more dramatic drop in the similarity scores than SLS. Compared with Scenario 1, the average value of the similarity score ${\mu }_S$ of all operators and windows decreases from 0.524 to 0.185 using FVM and decreases from 0.245 to 0.065 using SLS. Notice that the mean similarity score in Scenario 2 when using SLS is close to zero, which means that the two surfaces are rather different. We do not expect a further decrease and this explains why the similarity score does not decrease much in Scenario 2 when using SLS. (2) For Aim 2, the variation in the similarity scores is still higher when using FVM, which is the same phenomenon in Scenario 1. The variation in the similarity scores does not change much compared with Scenario 1 when using the same device, which suggests that the stability of the fabricating process does not change when printing these specimens.

In both scenarios, compared with using SLS, the mean similarity score ${\mu }_S$ and the variation in the similarity score ${\sigma }_S$ are higher using FVM because the measurement error is smaller when using FVM and the similarity score evaluation is more sensitive to the error. Compared with Case 1, the mean similarity score ${\mu }_s$ decreases since the surface topography on different specimens is different.

Case 3: Measurements on each surface using two devices. In Case 3, to examine the variation in the measurements and similarity scores induced by different devices, a pair of single measurements are obtained on different surfaces from a stable process by two devices. The device represents a particular combination of optical scanners and operators. Therefore, we can either compare different operators or scanners. For Scenario 1, we calculated the similarity scores of topography measurements on the same window scanned by two operators. For Scenario 2, we calculated the similarity scores of surface topography measurements of the same window taken by two optical scanners. Scenario 1 focuses on the variation induced by the operators while Scenario 2 focuses on the variation induced by different scanners. In both scenarios, we included the measurements from all of the 20 windows.

To examine the influence of different operators on the similarity scores, Figure 12 shows a side-by-side boxplot of the similarity scores by the operator. Each similarity score was calculated from two topography measurements on the same window scanned by the same operator using two different scanners. The median value in Figure 12 shows little difference among different operators. Meanwhile, the 25th and 75th percentiles are similar among all operators, respectively. Unlike the obvious difference between scanners in Figure 11, there is not much difference in the mean and variance of the similarity scores among the three operators. The easy-operated scanning process and well-trained operators help to alleviate the discrepancy between measurements.

In Scenario 1, as

Table 5. Case 3: Parameter estimation on the mean and variation of similarity scores on the same surface measured by different devices.

Scenario	Scanner	Operator	${\mathit{\mu }}_{\mathit{S}}$	${\mathit{\sigma }}_{\mathit{S}}$	95% CI of ${\mathit{\mu }}_{\mathit{S}}$	95% CI of ${\mathit{\sigma }}_{\mathit{S}}$
1: Same scanner, different operators	SLS	I, II	0.626	0.108	(0.575, 0.677)	(0.082, 0.158)
	SLS	I, III	0.614	0.086	$(0.573,0.654)$	$(0.066,0.126)$
	SLS	II, III	0.568	0.124	$(0.510,0.626)$	$(0.094,0.181)$
	FVM	I, II	0.680	0.233	$(0.571,0.790)$	$(0.178,0.341)$
	FVM	I, III	0.740	0.171	$(0.661,0.820)$	$(0.130,0.249)$
	FVM	II, III	0.751	0.157	$(0.678,0.824)$	(0.119, 0.229)
2: Different scanners, same operator	SLS, FVM	I	0.544	0.165	$(0.466,0.621)$	$(0.126,0.241)$
	SLS, FVM	II	0.504	0.150	$(0.434,0.574)$	$(0.114,0.219)$
	SLS, FVM	III	0.568	0.151	$(0.498,0.639)$	$(0.114,0.220)$

shows, each row demonstrates the parameter estimation when calculating the similarity scores between measurements of the same window scanned by different operators using the same scanner. Each row consists of 20 similarity scores from all the 20 windows. (1) For Aim 1, the average similarity score for FVM reaches 0.724, which indicates that the pair of surface measurements are almost the same. Similar to the finding in Case 2, the mean similarity score ${\mu }_S$ is comparatively high when using FVM. The gap is not large since the similarity score is close to one and there is not much space for rising. We also notice that ${\mu }_S$ only decreases by 0.07 compared with Case 1, which means that the reproducibility variation in the measurements induced by the operators is similar to the repeatability variation. (2) For Aim 2, the average reproducibility variation in the similarity score when using SLS is above 0.1, which is higher than the one in Case 1. This increase suggests that the similarity evaluation is more sensitive to reproducibility variation than repeatability variation in the measuring process. The ${\sigma }_S$ is higher when using FVM, which suggests that the similarity score evaluation is more sensitive to the error induced by different operators when using FVM.

In Scenario 2, as Table 5 shows, each row demonstrates the parameter estimation when calculating the similarity score between measurements of the same surface scanned by the same operator using both FVM and SLS. Each row consists of 20 similarity scores from all 20 windows. (1) For Aim 1, the mean similarity scores ${\mu }_S$ are all above 0.5, which indicates that the surface measurements are rather similar to each other compared with Scenario 1, which means the reproducibility variation in the measurements induced by the operator is smaller than the one induced by the optical scanner. (2) For Aim 2, ${\sigma }_S$ is still higher than the variation in Case 1, which means that the reproducibility variation induced by different optical scanners is larger than the repeatability variation. Compared with Scenario 1, ${\sigma }_S$ slightly increases, which indicates that the similarity score evaluation is more sensitive to the error induced by different optical scanners than by operators.

In Case 3, we notice that the reproducibility variation in the measurements induced by the operators is as small as the repeatability variation in Case 1. For Aim 2, the similarity evaluation is more sensitive to reproducibility variation than repeatability variation in the measuring process. Furthermore, the similarity score evaluation is more sensitive to the error induced by different optical scanners than by operators.

4.2.2. Two-Sample Measurement Studies

Case 4: Comparing two sets of repeated measurements on two surfaces using a single device. In Case 4, to examine the discrepancy of the variation in the measurements and similarity scores induced by repeated measurements between different surfaces, we conducted two-sample comparisons on two sets of surfaces from a stable process. Each set of surfaces is repeatedly scanned five times using SLS by the same operator. Each set of samples consists of 10 similarity scores, and we also consider two scenarios in Case 4. For Scenario 1, we considered that the measurements in both sets are collected from surfaces on the same specimen but different windows. For Scenario 2, we randomly selected two windows from different specimens and compared repeated measurements using the same device per window. Scenario 1 examines the difference in the similarity score on the same specimen while Scenario 2 compares the similarity score on different specimens.

For Scenario 1,

Table 6. Case 4: Parameter estimation on the mean difference and variation ratio of similarity scores on the difference of two randomly selected surfaces measured by the same device.

Scenario	Specimen(s)	Windows on the Specimen(s)	${\mathit{\mu }}_{{\mathit{S}}_1}-{\mathit{\mu }}_{{\mathit{S}}_2}$	$\frac{{\mathit{\sigma }}_{{\mathit{S}}_1}}{{\mathit{\sigma }}_{{\mathit{S}}_2}}$	95% CI of ${\mathit{\mu }}_{{\mathit{S}}_1}-{\mathit{\mu }}_{{\mathit{S}}_2}$	95% CI of $\frac{{\mathit{\sigma }}_{{\mathit{S}}_1}}{{\mathit{\sigma }}_{{\mathit{S}}_2}}$
1: The same specimen	A	1, 2	$-0.181$	$1.175$	$(-0.250,-0.111)$	$(0.739,1.867)$
	B	1, 3	$-0.015$	$0.578$	$(-0.057,0.027)$	$(0.364,0.918)$
	C	2, 3	$-0.016$	$1.357$	$(-0.087,0.054)$	$(0.854,2.156)$
	D	2, 5	$-0.021$	$0.651$	$(-0.083,0.041)$	$(0.409,1.034)$
2: Different specimens	A, B	2, 2	$0.158$	$1.782$	$(0.106,0.210)$	$(1.121,2.832)$
	A, C	3, 3	$-0.071$	$1.024$	$(-0.130,-0.011)$	$(0.644,1.628)$
	B, C	4, 4	$0.042$	$0.459$	$(-0.032,0.115)$	$(0.289,0.729)$
	D, C	5, 5	$0.006$	$0.720$	$(-0.058,0.070)$	$(0.453,1.145)$

shows the parameter estimation on the mean and variation in the similarity scores when calculating two topography measurements scanned by Operator III using SLS on surfaces from the same specimen. (1) For Aim 1, the 95% CI of ${\mu }_{S_1}-{\mu }_{S_2}$ in the first row does not include zero, indicating a pronounced difference in the mean similarity of the two windows. Ideally, the difference in the mean similarity scores between the two sets is trivial since the two surfaces are fabricated by the same stable process and measured by the same device. The possible reason for the significant difference is that the environmental noises during fabrication or measurement procedures introduce non-neglectable bias. After further analysis, we find that the average absolute difference in the similarity score $|{\mu }_{S_1}-{\mu }_{S_2}|$ for the four specimens are 0.143, 0.034, 0.059, and 0.052, correspondingly. Specimen A shows an unusually higher value than other specimens, which may be due to the noise caused by improperly segmenting the specimen with tinfoil masks or the roughness of the surface geometry. (2) For Aim 2, Table 6 tells that the CIs of $\frac{{\sigma }_{S_1}}{{\sigma }_{S_2}}$ in most rows include one, which is sensible in that the ratio of the standard deviation is close to one since both sets share the same variation components.

For Scenario 2, the similarity scores are calculated on surfaces from different specimens under the same conditions as Scenario 1. For Aim 1, similar to Scenario 1, the 95% CIs of ${\mu }_{S_1}-{\mu }_{S_2}$ do not include zero in some rows. Compared with Scenario 1, the average $|{\mu }_{S_1}-{\mu }_{S_2}|$ is slightly larger, which means that the discrepancy between surfaces on different specimens magnifies the differences in the similarity scores. For Aim 2, we derive the same conclusion as Scenario 1.

In summary, the results do not meet our expectations perfectly. Topography measurements from some surfaces show more discrepancy than others, which may be caused by the uncertainty during the measurement procedures, or noises during the window segmentation of the specimen. The standard deviation ratio is also influenced by the above-mentioned uncertainty.

Case 5: Comparing two sets of surfaces using the same device. To compare the variation in the measurements and similarity scores induced by different specimens, we conducted two-sample comparisons on two sets of surfaces. Notice that for our data, the two sets come from the same stable process, which is a special situation in Case 5. Each set of measurements was collected using the same scanner by the same operator from all five windows on the same specimen. Each set of measurements results in 10 pairwise similarity scores.

For Aim 1, as

Table 7. Case 5: Parameter estimation on the mean difference and variation ratio of the similarity scores of surfaces from two randomly selected specimens.

Specimens	Scanner	Operator	${\mathit{\mu }}_{{\mathit{S}}_1}-{\mathit{\mu }}_{{\mathit{S}}_2}$	$\frac{{\mathit{\sigma }}_{{\mathit{S}}_1}}{{\mathit{\sigma }}_{{\mathit{S}}_2}}$	95% CI of ${\mathit{\mu }}_{{\mathit{S}}_1}-{\mathit{\mu }}_{{\mathit{S}}_2}$	95% CI of $\frac{{\mathit{\sigma }}_{{\mathit{S}}_1}}{{\mathit{\sigma }}_{{\mathit{S}}_2}}$
A, D	SLS	I	$-0.219$	$1.497$	$(-0.302,-0.137)$	$(0.746,3.003)$
B, C	SLS	II	$0.119$	$0.602$	$(0.056,0.183)$	$(0.300,1.209)$
A, C	SLS	III	$-0.083$	$0.664$	$(-0.207,0.040)$	$(0.331,1.333)$
A, D	FVM	I	$-0.266$	$0.980$	$(-0.351,-0.181)$	$(0.488,1.966)$
B, C	FVM	II	$0.386$	$0.289$	$(0.156,0.616)$	$(0.144,0.581)$
A, C	FVM	III	$0.150$	$0.428$	$(-0.022,0.321)$	$(0.213,0.859)$

shows, the difference in the average similarity score between specimens is significant since the 95% CI of ${\mu }_{S_1}-{\mu }_{S_2}$ does not include zero, which indicates that a pronounced discrepancy exists between these two sets of surfaces from two specimens. We also notice that the magnitude of the difference in the mean similarity score between specimens is larger when using FVM, which means the similarity score shows larger variability when using FVM. Such a phenomenon is consistent with the conclusion in Case 2. Compared with Case 4, the magnitude of the difference is much larger since the measurements are from different surfaces instead of repeated measurements from the same surface.

For Aim 2, four out of six rows of 95% CI of $\frac{{\sigma }_{S_1}}{{\sigma }_{S_2}}$ include one, indicating a non-significant difference in the repeatability variation in the similarity scores. This phenomenon is consistent with the model of Case 5 since the sources of variation are the same $\left({\sigma }_{S_1}={\sigma }_{S_2}=\sqrt{{\sigma }_{\phi }^2+{\sigma }^2}\right)$, because the underlying process and device are the same in both sets. As for the significant CIs on the fifth and sixth rows in Table 7, it is possible that the large random error leads to a significant discrepancy between measurements.

For Case 5, FVM shows a larger magnitude of the difference in the similarity score $|{\mu }_{S_1}-{\mu }_{S_2}|$ due to higher sensitivity of the similarity score evaluation to the error in measuring process when using FVM, as also confirmed in Case 2. The ratio of the standard deviation $\frac{{\sigma }_{S_1}}{{\sigma }_{S_2}}$ is usually insignificantly different from one, which is consistent with the generic model.

Case 6: To examine the variation in the true similarity score $\left({\sigma }_{\phi }^2\right)$, we collected two samples scanned by Operator III using SLS. For the first sample, we calculated similarity scores using single measurements made on all five windows on a specific specimen. Since each window was repeatedly measured by Operator III five times, we randomly selected one trial from each window. Each row results in six similarity scores in the first sample. For the second sample, we used the five repeated measurements of a randomly chosen window from another specimen. Each row results in 10 similarity scores in the second sample.

Table 8. Case 6: Parameter estimation on the mean difference and variation ratio of similarity scores from two samples scanned by Operator III using SLS.

Specimens (First Sample, Second Sample)	Window # of the 2nd Sample	${\mathit{\mu }}_{{\mathit{S}}_1}-{\mathit{\mu }}_{{\mathit{S}}_2}$	$\frac{{\mathit{\sigma }}_{{\mathit{S}}_1}}{{\mathit{\sigma }}_{{\mathit{S}}_2}}$	95% CI of ${\mathit{\mu }}_{{\mathit{S}}_1}-{\mathit{\mu }}_{{\mathit{S}}_2}$	95% CI of $\frac{{\mathit{\sigma }}_{{\mathit{S}}_1}}{{\mathit{\sigma }}_{{\mathit{S}}_2}}$
A, B	2	$-0.384$	$0.615$	$(-0.491,-0.277)$	$(0.238,1.302)$
A, D	3	$-0.513$	$0.490$	$(-0.619,-0.407)$	$(0.189,1.037)$
B, C	5	$-0.411$	$2.958$	$(-0.526,-0.296)$	$(1.144,6.264)$
B, D	3	$-0.364$	$0.516$	$(-0.464,-0.263)$	$(0.200,1.094)$
C, D	5	$-0.426$	$0.574$	$(-0.517,-0.335)$	$(0.222,1.215)$

shows the parameter estimation on the difference and variation ratio of the similarity scores when comparing two samples in Case 6.

For Aim 1, as shown in Table 8, the differences in the similarity scores are rather large, and none of the 95% CIs of the ${\mu }_{S_1}-{\mu }_{S_2}$ include zero. The similarity scores in the second set are significantly larger than the first set in that the different surfaces show larger variations in the surface measurements than the repeated measurements on the same surface, which is reasonable and consistent with the conclusion from Cases 1 and 2.

For Aim 2, according to the generic model, the ratio of the standard deviation $\frac{{\sigma }_{S_1}}{{\sigma }_{S_2}}$ is represented as $\frac{{\sigma }_{\phi }^2+{\sigma }^2}{{\sigma }^2}$, which is larger than one. Based on the 95% CI of $\frac{{\sigma }_{S_1}}{{\sigma }_{S_2}}$ in Table 8, the ratios are not significantly larger than one, which indicates that ${\sigma }_{\phi }^2$ is much smaller than ${\sigma }^2$. We made a further analysis to separate different components of the variation in Case 7.

For Case 6, the similarity score of the first sample is significantly smaller than the second one, which has been discussed in Cases 1 and 2. The ratio of the standard deviation $\frac{{\sigma }_{S_1}}{{\sigma }_{S_2}}$ is insignificantly different from 1, which seems conflicting with the generic model. We guess that the ${\sigma }^2$ dominates in the total variance.

4.2.3. One-Way ANOVA

Case 7: One-way random-effects model for a single scanning system. To quantify the variance in the surface measurements and similarity scores within and between specimens, we used the repeated measurements of all the twenty windows from the four specimens by the same operator using SLS to fit the model in Section 3.3.3. Each window was scanned five times, resulting in 10 similarity scores per surface. Therefore, there are 200 similarity scores in total, where each similarity score is calculated between the repeated measurements on the same surface. The results show that the variations in similarity scores are reliable indicators of the variation in the measurements.

To examine the distribution of the similarity score obtained by the same device, Figure 13a shows the histogram of all the 200 similarity scores by the same operator using SLS, where each similarity score is calculated between the repeated measurements on the same surface. The distribution indicates normality; (b) represents the side-by-side boxplot of the similarity scores for four randomly chosen windows from four different specimens; and (c) shows the side-by-side boxplot of the similarity scores for the five windows on Specimen A. The boxplots show variation in both repeated measurements (between-specimen) and in specimens (within-specimen). As shown in the two boxplots, the variation in the similarity scores seems to be close within or between specimens.

For Scenario 1 shown in Figure 13b, the estimated parameters are: $\widehat{{\mu }_{\phi }}=0.717$, the 95% CI for ${\mu }_{\phi }$ is (0.606, 0.828); $\widehat{{\sigma }_{\phi }}=0.099$, $\widehat{\sigma }=0.064$, the 95% confidence interval for ${\sigma }_{\phi }$ is (0.045, 0.213) and (0.052, 0.082) for $\sigma$. For Aim 1, the average similarity score is comparatively high, which is close to the value in Case 1. For Aim 2, we can see that the variation in the similarity score attributed by different specimens ${\sigma }_{\phi }^2$ (between-group) is significantly larger than the repeatability variation in the similarity score $\sigma$ (within-group) because the confidence intervals have no overlap. The variation in the similarity score caused by repeated measurements $\left({\sigma }^2\right)$ is 29.6% of the total variation, which is much smaller than the variation among specimens $\left({\sigma }_{\phi }^2\right)$. Notice that the magnitude of the repeatability variation and that of the specimen variation are in the same order, which is possibly due to the errors induced by cropping SLS data. If the SLS data are not cropped, the pre-processing steps in calculating the similarity score can be simplified, which will result in a much smaller error and thus a bigger difference between repeatability variation and specimen variation.

For Scenario 2 shown in Figure 13c, we expect that the standard deviation attributed by different specimens ${\sigma }_{\phi }$ will be much closer to the standard deviation $\sigma$, because these windows on the same specimen are very similar. For example, if using Specimen A, $\widehat{{\mu }_{\phi }}=0.693$, the 95% CI for ${\mu }_{\phi }$ is (0.582, 0.805); $\widehat{{\sigma }_{\phi }}=0.111$, $\widehat{\sigma }=0.106$, the 95% confidence interval for ${\sigma }_{\phi }$ is (0, 0.176) and (0.088, 0.133) for $\sigma$. For Aim 1, the average similarity score is still high but smaller than the one in Scenario 1. We think the uniformity of the surfaces in Specimen A leads to such a result. For Aim 2, the variation caused by repeated measurements is 47.5% of the total variation, which is much closer to the variation among specimens compared with Scenario 1.

For Scenario 2, we also examined other specimens, as shown in

Table 9. The one-way random-effect model results of Scenario 2 for each specimen. $\sigma$ stands for the repeatability standard deviation in the similarity scores while ${\sigma }_{\phi }$ stands for the standard deviation attributed by different specimens. The last column shows the portion of repeatability variation vs. the total variation.

Specimen	$\widehat{{\mathit{\mu }}_{\mathit{\phi }}}$	95% CI of ${\mathit{\mu }}_{\mathit{\phi }}$	$\widehat{\mathit{\sigma }}$	$\widehat{{\mathit{\sigma }}_{\mathit{\phi }}}$	95% CI of $\mathit{\sigma }$	95% CI of ${\mathit{\sigma }}_{\mathit{\phi }}$	$\frac{\widehat{{\mathit{\sigma }}^2}}{\widehat{{\mathit{\sigma }}^2}+\widehat{{\mathit{\sigma }}_{\mathit{\phi }}^2}}$ $\times 100\%$
A	$0.693$	$(0.582,0.805)$	$0.106$	$0.111$	$(0.087,0.132)$	$(0.052,0.224)$	$47.5\%$
B	$0.623$	$(0.596,0.649)$	$0.064$	$0.019$	$(0.053,0.080)$	$(0.000,0.050)$	$92.1\%$
C	$0.672$	$(0.626,0.719)$	$0.108$	$0.034$	$(0.089,0.135)$	$(0.000,0.088)$	$90.9\%$
D	$0.703$	$(0.663,0.743)$	$0.105$	$0.025$	$(0.087,0.131)$	$(0.000,0.074)$	$94.7\%$

. We find that other specimens show completely different results compared with Specimen 1. For Aim 2, ${\sigma }_{\phi }$ is much smaller than the value in Specimen A. ${\sigma }^2$ almost contributes to all the variation, which indicates that windows on Specimens B, C, and D have relatively consistent surface topography. It is likely that the poor stability of the AM printing process leads to the variation in the surface topography of Specimen A, or there is information loss during the process of cropping the specimens with handmade tinfoil masks.

In summary, repeatability variation in the similarity scores (${\sigma }^2$) contributes less to total variance when surfaces are from windows on different specimens. This is reasonable because the variation in the similarity score induced by different $\widehat{{\sigma }_{\phi }^2}$ (between-group) is relatively small when windows are chosen from the same specimen.

5. Discussion

Considering Aim 1 when examining the mean similarity in our experimental studies, as can be seen from Cases 1 and 3, we find that the repeatability variation of the topography measurements and the reproducibility variation in the measurements induced by operators are relatively smaller compared with the variation in the measurements induced by optical scanners (SLS and FVM in our study), which is consistent with the fact that there is a pronounced intrinsic difference between SLS and FVM. Therefore, it is critical to choose a proper optical scanner. We also notice that the variation in the surface geometry of different surfaces is much larger in magnitude compared with the repeatability variation in the topography measurements (Cases 1 and 2). Nevertheless, these two devices are capable of obtaining precise measurements with small repeatability and reproducibility variation as illustrated in Cases 1 and 2. Our findings conform to existing studies. For example, Wang et al. [8], Jiang et al. [48] classify whether two surface measurements are from the same surface or not using the same source data as our research. They find that the variation caused by different specimens is larger than the variation caused by different optical scanners.

Considering Aim 2 when examining the variation in the similarity in our experimental studies, the variation in the similarity scores induced by different surfaces is relatively smaller than the repeatability variation in the similarity scores (Cases 1 and 2). In addition, the similarity evaluation is more sensitive to the error induced by the device than repeated measurements (Cases 1 and 3). Therefore, it seems for the data and similarity evaluation method we used, the variation in the similarity evaluation is mainly caused by repeated measurements and differences between devices (Scanner or operator), whereas the discrepancy in surface topography has relatively smaller effects. This phenomenon conforms to the fact that the similarity evaluation in Wang et al. [8] has high accuracy. Moreover, the similarity evaluation using FVM is more sensitive to the uncertainty during measuring processes than SLS (Case 3). Interestingly, the higher resolution of FVM helps to obtain more accurate measurements but yields higher sensitivity in similarity evaluation. Overall, the insights from our experimental studies are sensible, which confirms the effectiveness of our proposed framework.

Besides the major findings, it is interesting that the discrepancy in the surface geometry contributes the most to the total topography measurement variation (Aim 1 in Cases 1, 2, and 3) but the least to the variation in the similarity scores (Aim 2 in Cases 1, 2 and 3). On the one hand, highly inconsistent surface measurements yield similarity scores close to zero. Consequently, the size of the variation in the similarity scores due to uncertainty in the measurement process is relatively smaller (Case 1 Aim 2). On the other hand, very similar surface geometry results in high similarity scores close to one; accordingly, the size of the variation in the similarity scores is relatively higher (Case 2 Aim 2). Therefore, it is misleading to compare the variation in the similarity scores of two sets of surfaces without considering the degree of discrepancy in the surface geometry. This awareness helps to interpret some unexpected significant CIs for the ratio of two sample variations in the similarity scores in Cases 4 and 5. More rigorous investigation would help to compare the variation in the similarity scores of two sets of surfaces for future work.

We would also like to mention the following points. (1) Notice that we treat the two measurement systems (SLS and FVM) as fixed effects because we are specifically interested in these two scanners. The measurement systems can also be treated as random effects in some other context such as the work in Rice [24] for which ballistic evidence is usually examined by multiple agencies with distinct microscopes and operators. In this case, the interest is more on the variation in the device population rather than a specific device. (2) We need more evidence to examine the hypothesis that a higher resolution of the scanner results in a higher similarity score under the same circumstance. Usually, higher resolution is quite helpful in capturing the subtle discrepancy between surface geometry. However, the intrinsic difference between the mechanisms of scanners is another reason to consider when making decisions. FVM has a higher resolution but might be more sensitive to uncertainty during the measurement process, whereas SLS requires proper calibration and postprocessing on the distorted pattern which affects the accuracy of the measurements. (3) The insights from our experimental studies might be specific to the data and the similarity evaluation methods we used. Cautions need to be taken when generalizing the findings to other experimental settings.

6. Conclusions

This investigation establishes a comprehensive framework for the systematic conduct of measurement studies, aimed at quantifying diverse sources of variation including surfaces, operators, measurement systems, and repeated measurements. The framework leverages advanced similarity evaluation techniques applied to topography data.

The articulation of methods unfolds across seven distinct cases, each serving as a unique context for the application of the framework. Within each case, meticulous examination ensues, encompassing the assessment of both the average similarity scores, enabling scrutiny of the uniformity in surface topography measurements, and the variance in similarity scores, facilitating an in-depth exploration of the statistical reproducibility inherent in the similarity evaluation procedure.

We exemplify the efficacy of this framework through its application to measurements derived from specimens produced through additive fabrication. Empirical studies underscore the framework’s capacity to judiciously quantify sources of uncertainty intrinsic to the measurement process, particularly when dealing with 3D surface topography data. These experiments substantiate a fundamental principle. That is, the interplay between physical uncertainty and the intricacies of a data collection plan significantly influences the insights attainable from a dataset. Consequently, the strategic selection of an appropriate data collection plan emerges as a pivotal consideration aligned with practical imperatives.

The proposed framework, poised at the nexus of scientific rigor and practical utility, holds promise for diverse applications within quality assurance contexts. By providing a structured and robust methodology for the analysis of 3D topography data, this research contributes not only to the advancement in scientific inquiry but also to the enhancement of precision and reliability in addressing the multifaceted challenges encountered in real-world quality assurance scenarios.

For future work, a more rigorous examination of our independence assumption in the similarity scores is helpful. It may be appropriate to assume that the surface measurements are independent. However, the presumption of independence in the similarity scores may raise concerns. In addition, it is difficult to justify independence rigorously. Nevertheless, our framework provides sensible insights from the experimental studies. Still, it would be interesting to investigate in future work what remedies are available when the independence assumption in the similarity scores is violated.