Synthesis of Valid Camera Poses for the Inspection of Triangular Facets in a 3D Mesh

Abstract

Automation of visual quality inspection tasks in manufacturing with machine vision is beginning to be the de facto standard for quality inspection as manufacturers realize that machines produce more reliable, consistent and repeatable analyses much quicker than a human operator ever could. These methods generally rely on the installation of cameras to inspect and capture images of parts; however, there is yet to be a method proposed for the deployment of cameras which can rigorously quantify and certify the performance of the system when inspecting a given part. Furthermore, current methods in the field yield unrealizable exact solutions, making them impractical or impossible to actually install in a factory setting. This work proposes a set-based method of synthesizing continuous pose intervals for the deployment of cameras that certifiably satisfy constraint-based performance criteria within the continuous interval.

1. Introduction

While the implementation of quality inspection procedures on manufacturing lines is a well-established practice with a long history, the use of automated methods and machine vision techniques is a relatively new area of study. Central to this new area is the effective deployment of cameras for image capture; however, this is often left to the experience of engineers or technicians to pick what they feel is the best deployment for the task. In most cases, the hardware is purchased beforehand, based on some initial assumptions that later prove to be incorrect. While good deployment results are often possible with this approach, they lack mathematical rigour and quantifiable parameters that could be used to analyze the deployment and objectively assess its effectiveness at inspecting the desired object.

Methods have been devised for the synthesis of optimal deployments; however, these are often extremely limited in their scope and are thus theoretical exercises with no ability to be practically implemented in a real factory setting. This research aims to address this gap in the literature by proposing a set-based method of synthesizing continuous pose intervals for camera deployments, which can be certified to completely inspect any section of a part’s surface for any pose contained within the prescribed interval. Representative results can be seen in Figure 1. This not only allows camera deployment solutions to be quantified, but also allows for their actual implementation, as they allow for realistic integration errors during installation.

The problem of computing complete and realistic inspection spaces for large areas of a part’s surface is complex and multi-faceted, and, as such, the research contained herein imposes the following conditions to limit the scope of this preliminary exploration: the part will be represented as a triangular mesh, the valid inspection poses will be solved for a single triangle from this mesh and idealized models will be used for the inspection camera.

2. Problem Definition and Terminology

It is first useful to establish some definitions of terms that appear frequently in the literature and use these to frame the problem.

The first, and most important, term is the inspection task. The inspection task is defined in [1] as a “measurement, or set thereof, to be performed by a given sensor on some features of an object, for which a geometric model is known”. Likewise, in this work, the inspection task is defined as the surfaces of the part that need to be acceptably imaged to perform a suitable inspection. The method for planning the inspection task will be, as stated in [1], based on a known geometric model of the object and a set of inspection constraints that define valid inspection regions for the given inspection camera.

Next, it is important to define what is meant by the model. The model is the 3D data representation of the part’s geometric structure. While early attempts to solve this problem made use of parametric expressions of part geometry [1,2], it has since become the standard in the literature to use tessellated model formats [2,3,4,5]. This means that instead of representing a part as a set of parameterized surfaces and edges, it is represented by a mesh, or tessellation, of small polygons. The format most commonly used in these formulations is a triangular mesh. A triangular mesh is a representation of a given part geometry as a triangular mesh with edges, faces, and vertices defining the 3D structure of the part surfaces [6]. As a visual representation of this concept, Figure 1 shows a part’s triangular mesh representation. Each vertex in the mesh is described as a set of 3D coordinates, and these coordinates are compiled in a list of vertices. Each triangular facet is then described as a set of three vertices (edges are described by a set of two vertices). Additionally, each vertex and face has an outward-facing normal vector assigned to it [2,3,4,5].

Once the inspection task and the model have been defined, it is important to treat what will be referred to in later sections as the inspection space. Using set-based methods, which will be described in more detail in later sections, the proposed methods generate a set of valid inspection poses for each facet of the part’s mesh. The set of these poses defines a region in space wherein any camera position or orientation will yield a valid solution to the inspection task, i.e., the given facet will be properly inspected. This will be called the inspection space. The cloud of yellow boxes in Figure 1 shows what this inspection space would look like for a given facet of interest. Intersecting these inspection spaces for each facet in future work would give the final deployment solutions to inspect the entire part; however, this is beyond the scope of this paper.

For the work contained herein, the inspection task and space will be defined for a single “facet of interest”, or foi. This is simply the single facet of the part’s triangular mesh that the camera must inspect, and for which inspection poses will be defined according to inspection constraints. While this paper focuses on methods for a single facet of interest, future work will expand on using this methodology to derive inspection spaces for multi-facet inspection tasks.

The last term that must be treated is the sensor. The sensor, in the broadest terms, is the imaging device used to capture data about the part during the inspection task. In this study, it will be considered as a single camera. Other sensors could be treated in future work using the same methodology, but because this research focuses on visual inspection, only cameras will be considered herein. As such, the terms “camera” and “sensor” will be used interchangeably.

Thus, the overall problem can be defined as solving the complete inspection space for a particular model and inspection task pair according to the constraints that will be imposed by the characteristics of the sensor that will be used to carry out the inspection task.

3. Existing Literature

While the problem of camera pose generation for observation of an object has been around since the mid-1980s, the first work which bears any significant resemblance to the problem in its current state began to appear in the mid-1990s [1,7]. These methods were primarily concerned with one of three tasks: object recognition, scene reconstruction, and feature detection [7]. Of these, feature detection and inspection is the area from which the sensor deployment problem would arise.

These early methods also quickly established the importance of several aspects of the problem which need to be considered in order to adequately address the inspection problem, particularly the identification and accurate representation of the sensor and object models [7]. However, the most important by far is proper modelling of the sensor, and proper consideration of the constraints these sensor characteristics impose on the relative positions of the sensor and the object in order to properly satisfy the inspection task, i.e., modelling the constraints on the camera pose for task satisfaction.

3.1. Sensor Modelling

Modelling of the sensor is accomplished most simply in the literature through the use of a pinhole camera model [1,4,5,8,9], which is analogous to the very earliest camera obscura in which light simply passes through a very small hole onto the image capture plane. This model is most often chosen for its simplicity, as its most basic form only requires two parameters: the image principal point, p, and the focal length, f. This model is often considered accurate enough for most applications [8], but for this research, it is insufficient. Thus, this model will be abstracted to the thin lens case, which assumes an infinitesimally thin lens with a given aperture diameter [10]. This requires only one additional parameter, the aperture diameter. This additional parameter allows for the modelling of real effects such as image blur [5]. The thin lens model is illustrated in Figure 2. While other more complex models such as the thick lens model [2,7,10] and the affine sensor model [11] exist, their complexity is beyond the scope of this paper.

3.2. Constraint Modelling

From very early work in the field, it was agreed that there are three main criteria which must be met in order to have an image of a part, or a feature thereof, properly satisfying the inspection task requirements [2,3,4,5,7]. They are:

• The part or feature must be visible within the image;
• The part or feature must be in focus;
• The part or feature must be imaged with a suitable resolution.

The visibility constraint implies exactly that; the part feature in question must be visible from the viewpoint of the sensor. This means that it must not be occluded in any way by other part features or external geometries, and generally, that the angle between its normal and the negative of the sensor viewing axis must be less than 90 degrees [3], i.e., backface culling. This also includes a field of view constraint, which is related to a given sensor’s maximum field of view angle. The focus constraint dictates that the part or feature must be within a given distance from the sensor such that the image is sufficiently sharp (this is usually expressed in terms of blur circle radius, which will be discussed in more detail in later sections) [4]. Finally, the resolution criteria is based on the pixel resolution of the imaging sensor. It requires that the projection of the part or feature in the image have a minimal size (in pixels) such that it can be examined with a sufficient level of detail for the inspection task [7,12], and is accounted for by the distance and viewing angle constraints in this work [4].

How researchers attempted to satisfy these constraints has continuously evolved over the years. Some of the earliest methods used minimum radius geodesic domes [1,7,13] constructed around the object in question to ensure the sensor was always at an appropriate distance to satisfy the focus constraint. This dome was also constructed such that the part would always be at its centre, fully within the field of view, and imaged at an acceptable level of resolution. Finally, they used ray tracing from proposed viewpoints on the dome surface in order to ensure visibility [1], although newer methods suggest using sampled point clouds and various statistical filtering techniques to determine the presence of occluding geometry [14]. These methods were also implemented on parametric models of the target part. Additionally, these early methods tended to fall under what is referred to in the literature as “generate and test” strategies, in which sensor configurations are generated via less formal heuristics and gradually improved [7,15].

However, once the use of tesselated part models became the standard, methods began to evolve in order to suit this new representation. The most notable improvement was the shift from heuristic based “generate and test” methods towards more mathematically rigorous “synthesis” methods. These methods, as in [2,3,4,5,16,17,18,19,20,21] attempt to frame the constraints as mathematical functions relating camera position and feature characteristics so as to have a more quantifiable relationship between the two. This includes using triangle normals to help determine camera rotations and visibility [2,3,4,5], as well as spatial triangle coordinates to determine focus, field of view, and resolution. Framing the constraints as algebraic expressions also allows for them to be used as constraints in more computationally modern optimization routines, which is how most modern approaches attempt to solve the problem. For instance, Ref. [2] makes use of recursive solvers, Ref. [3] uses hierarchical genetic algorithms, Refs. [16,17,18] use artificial neural networks and fuzzy inference systems, Ref. [19] makes use of Parisian algorithms, and Refs. [4,5,22] use convex optimization methods such as genetic algorithms and particle swarm optimization. While these methods have all shown promising results, they are all limited by their generation of discrete point solutions with no ability to account for uncertainty in modelling or deployment, which is the gap in the literature which this research addresses.

4. Camera Models

Given that the problem is primarily concerned with how well a camera can inspect a particular facet in a triangular mesh, one must first establish the camera model that will be used in the formulation of the problem. This is done to understand how the camera will capture an image and how this will inform the constraints that define whether or not an object is suitably inspected from a given pose.

While a large portion of the literature on optimal sensor placement assumes simple pinhole camera models, these models are overly simplistic, and thus not realistic enough for the deployment of real cameras. As such, the camera model assumed in this research is the thin lens approximation model. This model assumes an aperture with a finite diameter, along with an infinitesimally thin ideal lens [10].

The first basic aspect of the model that must be defined is the optical axis. The optical axis is the presumed axis that passes through the centre of the lens and the image centre [23]. Next, the principal plane of the camera model is defined as the plane normal to the camera axis which intersects it at the lens [10,24]. The final basic aspect of the model is the focal point, which is defined in [10] as the point along the optical axis which has the property of any rays passing through it into the lens will be refracted parallel to the optical axis after passing through the lens. These three elements are illustrated in Figure 2.

The distance between the camera’s principal plane and the focal point is referred to as the focal length, f. The focal point is defined as the point behind the lens at which all rays passing through the lens parallel to the optical axis converge [10]. This, coupled with the distance between an object and the lens, l, and the distance from the lens to the image plane, $l^{\prime }$, form the basis for the basic equation describing image formation [24], as presented in Equation (1).

$$\frac{1}{l}+\frac{1}{l^{\prime }}=\frac{1}{f}$$

(1)

This can be rearranged to represent the distance at which the projection of a given object will converge behind the lens as

$$l^{\prime }=\frac{fl}{l-f}.$$

(2)

The next key parameter in the model that is derived from lens characteristics is the $f_{stop}$, which is the ratio of focal length to aperture diameter, $a_{diam}$, expressed as

$$f_{stop}=\frac{f}{a_{diam}}.$$

(3)

The aperture diameter is the diameter of the circular opening at the front of the lens assembly which controls the amount of light let through the lens [10]. Expanding on Equation (2), one can determine the relationship between the locations of the focal plane and the image plane (the focal plane is the plane in front of the lens in which an object will be projected perfectly onto the image plane behind the lens [10], see Equation (2) to be

$$d_{focus}^{\prime }=\frac{f{d}_{focus}}{d_{focus}-f}.$$

(4)

The final sensor model concept that will be used in constraint generation is the blur circle, or circle of confusion [5]. This phenomenon is the circular blurring that can be seen in an image around an object when it is not perfectly in focus. The blur circle is the result of the projection of the object in question being either in front of or behind the image plane, which results in it being projected as a circle as opposed to a point. The diameter of the blur circle is expressed as

$$c_{diam}=a_{diam}\frac{l^{\prime }-d_{focus}^{\prime }}{l^{\prime }}.$$

(5)

Blur can also be expressed as the blur angle (denoted as ${\theta }_{blur}$), which is expressed in [10] as

$$tan\frac{{\theta }_{blur}}{2}=\frac{c_{diam}/2}{d_{focus}^{\prime }}.$$

(6)

By leveraging the small angle identity $tan\frac{{\theta }_{blur}}{2}\approx \frac{{\theta }_{blur}}{2}$ and substituting Equation (6) into Equation (5), along with some rearranging, Ref. [10] showed that ${\theta }_{blur}$ can then be expressed as

$${\theta }_{blur}=a_{diam}\left(\frac{1}{d_{focus}^{\prime }}-\frac{1}{l^{\prime }}\right).$$

(7)

These blur quantities are useful, as they will allow the formulation of upper and lower limits on the distance a given object can be from the focal plane while also remaining sufficiently in focus in the final image to allow for adequate inspection.

We must also consider the projection of points in front of the camera onto the camera’s image plane. The image plane is the available surface of the camera’s sensor and is bound in pixel space by the sensor’s height h and width w. We must also consider the pixel aspect ratio $\delta$, sensor skew s, and camera focal length f. Altogether, these let us define the camera intrinsic matrix $\mathbf{K}$ as

$$\mathbf{K}=\left(\begin{array}{ccc}f& s& c_x\\ 0& \delta f& c_y\\ 0& 0& 1\end{array}\right).$$

(8)

In $\mathbf{K}$, $(c_x,c_y)$ is the ordered pair defining the image projection centre in pixel space. With an ideal lens, this would mean $(c_x,c_y)=(\frac{w}{2},\frac{h}{2})$, but with real lens aberrations, these values may be slightly different from their ideal values, and these offsets are usually derived via camera calibration algorithms. Additionally, the aspect ratio $\delta$ is simply the ratio of pixel height to pixel width, and the sensor skew s describes the degree of misalignment of the camera sensor and image plane.

Thus, a point ${\mathbf{o}}_W$ in world space

$${\mathbf{o}}_W=\left(\begin{array}{c}x\\ y\\ z\end{array}\right)$$

(9)

will be projected into the camera’s pixel space on the image plane as ${\mathbf{o}}_C$

$$\left(\begin{array}{c}{\mathbf{o}}_C\\ 1\end{array}\right)=\mathbf{K}\left(\mathbf{R}\right|\mathbf{T})\left(\begin{array}{c}{\mathbf{o}}_W\\ 1\end{array}\right)$$

(10)

where $\left(\mathbf{R}\right|\mathbf{T})$ is the homogeneous transformation matrix defining the point relative to the world and camera frames.

5. Set-Based Methods

This section aims to present some of the fundamental concepts that will frame the novel set-based approach to the optimal inspection problem. It will cover the basics of interval analysis theory, the basics of set-based pose representations, and some simple examples of interval analysis being used to solve constraint problems.

5.1. Interval Analysis Methods

This research takes a novel approach to solve the optimal sensor locations by applying interval analysis methods. Originally developed in the 1960s to address the inability of computers to exactly represent most numbers [25], these methods find extensions to standard point number mathematical operations using interval values instead of discrete exact values. By treating numbers as intervals, one can account for rounding and measurement errors in calculations and produce ranges of solutions that are guaranteed to contain the true solution to the given problem. Intervals in this system are represented as

$$\left[x\right]=[\underline{x},\overline{x}]=\left\{x\in \mathbb{R}\mid \underline{x}\le x\le \overline{x}\right\}$$

(11)

where $\underline{x}$ and $\overline{x}$ are the lower and upper bounds of the interval, respectively.

Other useful components of intervals are their midpoint,

$$\mathrm{mid}\left(\left[x\right]\right)=(\overline{x}+\underline{x})/2$$

(12)

and their width,

$$\mathrm{wid}\left(\left[x\right]\right)=\overline{x}-\underline{x}.$$

(13)

Other fundamental properties of intervals are discussed in detail in [25].

It is also useful to characterize the interactions between multiple intervals. The two key operations for doing so are the intersection of two intervals,

$$\left[x\right]\cap \left[y\right]=[\max (\underline{x},\underline{y}),\min (\overline{x},\overline{y})]$$

(14)

and the hull, or interval union, of two intervals

$$\left[x\right]\cup \left[y\right]=[\min (\underline{x},\underline{y}),\max (\overline{x},\overline{y})].$$

(15)

A detailed discussion of interval extensions to standard arithmetic operations can also be found in [25]. Interval extensions of functions typically require that the function be monotone, although there are interval extensions of non-monotonic functions [25].

This leads to the fundamental theory of interval analysis, which states that “the interval extension of a monotonic function $f\left(\right[x\left]\right)$ yields the inclusion function $\left[f\right]$, such that $f\left(\right[x\left]\right)$ is contained inside of $\left[f\right]$ [26],”

$$f\left(\right[x\left]\right)=\left\{f\right(x)\mid x\in [x\left]\right\}\subseteq \left[f\right].$$

(16)

These interval methods can also be extended in order to describe vectors and matrices of intervals. An interval vector represents an ordered n-tuple of intervals [25]

$$\left[\mathbf{x}\right]=[\left[x_1\right],\left[x_2\right],\dots ,\left[x_n\right]]=[[\underline{x_1},\overline{x_1}],[\underline{x_2},\overline{x_2}],\dots ,[\underline{x_n},\overline{x_n}]].$$

(17)

By extension, an interval matrix is represented as

$$\left[\mathbf{A}\right]=\left(\begin{array}{cc}[\underline{a_{11}},\overline{a_{11}}]& [\underline{a_{12}},\overline{a_{12}}]\\ [\underline{a_{21}},\overline{a_{21}}]& [\underline{a_{22}},\overline{a_{22}}]\end{array}\right).$$

(18)

These interval methods are applied to the inspection constraints in order to allow for a continuous evaluation of the space around a given facet in order to certifiably synthesize its entire valid inspection space.

5.2. Interval Analysis for Constraint Satisfaction

Interval analysis methods for constraint satisfaction in this research will work based on two principal method classes: simplification, and bisection [25]. Given a system of constraints $\mathcal{C}\left(\right[\mathbf{u}\left]\right)$, where $\left[\mathbf{u}\right]$ represents the constraint variables, the two methods are applied consecutively through an iterative branch and bound process. For other in-depth analyses of applications of interval methods for the solving of constraint satisfaction problems, see [25,26,27,28,29,30]. The interval-based constraint satisfaction solver algorithm used here is shown in Algorithm 1.

Algorithm 1 Interval-based constraint satisfaction solver algorithm

1: Initialize system variables [u] 2: Initialize constraints $\mathcal{C}\left(\right[\mathbf{u}\left]\right)$ 3: Initialize empty list of unclassified boxes ${\mathcal{L}}_{\mathbf{u}}$ 4: Initialize empty list of valid boxes ${\mathcal{L}}_{\mathbf{v}}$ 5: Initialize search space by contracting $\left[\mathbf{u}\right]$ according to $\mathcal{C}\left(\right[\mathbf{u}\left]\right)$ 6: Add contracted $\left[\mathbf{u}\right]$ to end of list ${\mathcal{L}}_{\mathbf{u}}$ 7: if ${\mathcal{L}}_{\mathbf{u}}=Ø$ then 8:     Terminate solver and return ${\mathcal{L}}_{\mathbf{v}}$ 9: else 10:     Pop ${\left[\mathbf{u}\right]}_i$ from ${\mathcal{L}}_{\mathbf{u}}$ and evaluate $\mathcal{C}$(${\left[\mathbf{u}\right]}_i$) 11:     if Constraints are fully satisfied then 12:         Classify ${\left[\mathbf{u}\right]}_i$ as VALID 13:         Add ${\left[\mathbf{u}\right]}_i$ to back of ${\mathcal{L}}_{\mathbf{v}}$ 14:         go to 6 15:     else if Constraints are partially satisfied then 16:         if Widest interval in ${\left[\mathbf{u}\right]}_i$ is narrower than $ϵ$ then 17:            Classify ${\left[\mathbf{u}\right]}_i$ as BOUNDARY 18:            Add ${\left[\mathbf{u}\right]}_i$ to back of ${\mathcal{L}}_{\mathbf{v}}$ 19:            go to 6 20:         else 21:            Bisect ${\left[\mathbf{u}\right]}_i$ ${\left[\mathbf{u}\right]}_{i_1}$ and ${\left[\mathbf{u}\right]}_{i_2}$ 22:            Add ${\left[\mathbf{u}\right]}_{i_1}$ and ${\left[\mathbf{u}\right]}_{i_2}$ to end of ${\mathcal{L}}_{\mathbf{u}}$ 23:            go to 6 24:         end if 25:     else 26:         go to 6 27:     end if 28: end if

5.2.1. Simplification Methods

Simplification methods are heuristic methods whose goal is to reduce any excess width of $\left[\mathbf{u}\right]$ in $\mathcal{C}\left(\right[\mathbf{u}\left]\right)$ [25], such as HC4, ACID, 2B and 3B filtering, and Newton methods [25,26,27,28]. This work uses HC4 and ACID methods in order to simplify initial variable search spaces (represented by interval vectors) according to constraints as much as possible prior to the application of bisection methods to further refine the solutions.

These methods work by iteratively applying interval arithmetic to the constraint functions in order to narrow the domains of the variables as much as possible. For instance, HC4 works by applying consecutive iterations of forward arithmetic and backward arithmetic [25] to a tree representation of a system to successively narrow the domain of its variables [28]. For example, in the equation ${(\left[x\right]-\left[y\right])}^2-\left[z\right]=0$, with $\left[x\right]\in [0,10]$, $\left[y\right]\in [0,4]$, and $\left[z\right]\in [9,16])$, the forward step yields a result of $[-16,91]$ at the root. Setting the root to $[0,0]$ for the backwards step and refining for $\left[x\right]$, we find $\left[x\right]$ simplified to $\left[x\right]\in [0,8]$. Application of these steps continues over each variable until a given stopping criteria (usually variable width $ϵ$) is met [28].

5.2.2. Bisection Methods

Bisection methods split the interval $\left[u\right]$ along the dimension i into $\left[u_1\right]$ and $\left[u_2\right]$, as long as the width of $\left[u\right]$ is greater than a given threshold $ϵ$ [25]. The union of these sub-intervals is equal to the original interval and, as such, they still represent a continuous evaluation of the solution space. The bisection strategy used herein is known as largest-first [27], in which an interval vector $\left[\mathbf{u}\right]$ is bisected at the midpoint of its widest component interval, and all other components of the interval vector remain unchained in the resultant child interval vectors. These bisections continue until the widths of all components of $\left[\mathbf{u}\right]$ are below a given threshold, or $\left[\mathbf{u}\right]$ is found to either fully satisfy constraints or not represent a valid solution. Bisected intervals are added to the list ${\mathcal{L}}_u$ [27].

6. Set-Based Extensions of Pose and Inspection Constraints

A camera position and orientation are described using $x,y,z$ coordinates and ZXZ Euler angles ($\phi ,\gamma ,\beta$), respectively. Together the position and orientation define the camera pose.

The goal of the set-based constraint formulation is to derive the sets of camera poses such that all points on a given facet are visible from the camera, and the camera specification and other inspection constraints are satisfied. That is, the set of poses $\mathcal{P}$ that ensure the facet can be properly inspected is given by

$$\begin{array}{c}\hfill \mathcal{P}=\left\{(x,y,z,\phi ,\gamma ,\beta )\right|{\mathcal{C}}_k(x,y,z,\phi ,\gamma ,\beta )\mathrm{is}\mathrm{valid},\\ \hfill x,y,z\in \mathbb{R},\phi ,\gamma ,\beta \in [-2\pi ,2\pi ],k=1..n\}\end{array}$$

(19)

where ${\mathcal{C}}_k(x,y,z,\phi ,\gamma ,\beta )$ is one of n inspection constraints.

Let a camera pose interval be given by $\left[\mathbf{p}\right]=(\left[x\right],\left[y\right],\left[z\right],\left[\phi \right],\left[\gamma \right],{\left[\beta \right]}_i)$ for a given facet. The pose solution guarantees that the entire facet satisfies the considered constraints $\forall \mathbf{p}\in \left[\mathbf{p}\right],\left[\mathbf{p}\right]\in \mathcal{P}$.

6.1. Set-Based Representation of Poses

The two elements of pose which must be addressed are position and orientation. The position of a point in the world frame ${\mathbf{o}}_W$, and the set-based extension of this vector, are based on imposing upper and lower uncertainty limits [31] on each element of the vector such that each is transformed to be

$$\begin{array}{cc}\hfill \left[x\right]& =[\underline{x},\overline{x}]\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill \left[y\right]& =[\underline{y},\overline{y}]\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill \left[z\right]& =[\underline{z},\overline{z}].\hfill \end{array}$$

(22)

Thus, ${\mathbf{o}}_W$ becomes $\left[{\mathbf{o}}_W\right]$, where

$$\left[{\mathbf{o}}_W\right]=\left(\begin{array}{c}\left[x\right]\\ \left[y\right]\\ \left[z\right]\end{array}\right)$$

(23)

where $\left[{\mathbf{o}}_W\right]$ now represents all of the points contained inside a 3D box in position space, as opposed to the discrete point represented by ${\mathbf{o}}_W$.

This research will express orientation as a vector of ZXZ Euler angles ($\phi$, $\gamma$, $\beta$) [32] as

$$\mathbf{e}=\left(\begin{array}{c}\phi \\ \gamma \\ \beta \end{array}\right)$$

(24)

This representation is chosen due to its ease of bisection, and its compatibility with orientation constraints. These rotations are applied successively such that the rotation matrix of cumulative rotation operation can be expressed as

$${\mathbf{R}}_{\mathbf{e}}={\mathbf{R}}_z\left(\phi \right){\mathbf{R}}_{x^{\prime }}\left(\gamma \right){\mathbf{R}}_{z^{″}}\left(\beta \right)$$

(25)

and is demonstrated in Figure 3.

This ZXZ representation can easily be extended to a set-based context using intervals much like the position was in Equation (23) such that $\mathbf{e}$ becomes $\left[\mathbf{e}\right]$,

$$\left[\mathbf{e}\right]=\left(\begin{array}{c}\left[\phi \right]\\ \left[\gamma \right]\\ \left[\beta \right]\end{array}\right)=\left(\begin{array}{c}[\underline{\phi },\overline{\phi }]\\ [\underline{\gamma },\overline{\gamma }]\\ \underline{\left[\beta \right]},\overline{\beta ]}]\end{array}\right).$$

(26)

Thus, $\left[\mathbf{e}\right]$ now represents a 3D box in rotation space as opposed to the single finite point represented by $\mathbf{e}$. Together, $\left[{\mathbf{o}}_W\right]$ and $\left[\mathbf{e}\right]$ represent a 6D box in pose space that describes a continuous set of poses.

The position and orientation intervals are solved as two separate constraint systems in this methodology, along with a third system for determining if any derived poses result in the camera’s view of the foi being occluded by any objects within the scene. Since the occlusion test is orientation-independent, it is grouped with position constraints but is solved by a separate system of constraints within them.

6.2. Position Constraints

The constraints that must be solved for a given pose box $\left[\mathbf{p}\right]$ in the main position constraint system ${\mathcal{C}}_p$ for a facet are:

• Does $\left[\mathbf{p}\right]$ intersect the facet?
• Is $\left[\mathbf{p}\right]$ an appropriate distance from the facet?
• Is $\left[\mathbf{p}\right]$ in front of the facet?
• Does $\left[\mathbf{p}\right]$ inspect the facet from a suitable angle?

First, to test if $\left[\mathbf{p}\right]$ intersects the facet, we consider the set of all points on the surface of the facet as the region bounded by the set of plane inequality constraints ${\mathcal{C}}_f$. ${\mathcal{C}}_f$ is defined by the 3D plane that contains the facet, and the three planes perpendicular to it, which each contain one of the edges of the facet. We can then say

$${\mathcal{C}}_f\cap \left[\mathbf{p}\right]\ne Ø,\left[\mathbf{p}\right]\mathrm{is}\mathrm{not}\mathrm{valid}$$

(27)

$${\mathcal{C}}_f\cap \left[\mathbf{p}\right]=Ø,\left[\mathbf{p}\right]\mathrm{satisfies}\mathrm{constraint}1,\mathrm{continue}.$$

(28)

To solve for the valid set of $\left[x\right],\left[y\right],\left[z\right]$ positions in $\left[\mathbf{p}\right]$ for the facet bounded by ${\mathcal{C}}_f$, the following distance constraint is required:

$${\mathcal{C}}_{distance}={\left(\begin{array}{c}\left[x\right]\\ \left[y\right]\\ \left[z\right]\end{array}\right)}^2-{\left[\mathbf{c}\right]}_f^2\subseteq [d_{min}^2,d_{max}^2]$$

(29)

where $d_{min}$ and $d_{max}$ are constants defining the minimum and maximum depth of field values for the image of the facet to be suitably in focus for a given inspection camera, and ${\left[\mathbf{c}\right]}_f$ is the interval vector containing the valid solutions to ${\mathcal{C}}_f$. The $d_{min}$ and $d_{max}$ parameters are derived according to lens intrinsic parameters. They determine how far away from the facet a camera can be while still satisfying inspection requirements.

The constraint for testing whether a box $\left[\mathbf{p}\right]$ is in front of a facet is called the backface constraint, and is evaluated by creating a half-space constraint defined by the plane containing the facet. Using the facet’s normal, $\mathbf{n}={[n_x,n_y,n_z]}^T$, and one vertex, $\mathbf{v}={[v_x,v_y,v_z]}^T$, we can define the constraint as

$$\left[\mathbf{p}\right]=\left\{\begin{array}{cc}\mathrm{in}\mathrm{front}:\hfill & \mathrm{if}\left(\right[\mathbf{p}]-\mathbf{v})·\mathbf{n}>0\hfill \\ \mathrm{intersecting}:\hfill & \mathrm{if}0\in \left(\right[\mathbf{p}]-\mathbf{v})·\mathbf{n}\hfill \\ \mathrm{behind}:\hfill & \mathrm{if}\left(\right[\mathbf{p}]-\mathbf{v})·\mathbf{n}<0.\hfill \end{array}\right.$$

(30)

Finally, to determine if the viewing angle is sufficiently large for the foi to be inspectable, we define the constant ${\theta }_v$ as the minimum viewing angle, and the facet’s geometric centre as ${\mathbf{f}}_{j_c}$. The constraint is then

$$arccos\left(({\left[\mathbf{p}\right]}_i-{\mathbf{f}}_{j_c})\right)·\mathbf{n}\le (\frac{\pi }{2}-{\theta }_v).$$

(31)

Once box $\left[\mathbf{p}\right]$ has been shown to satisfy these position constraints, it must be tested to ensure that no position in it represents one whose view of $f_j$ would be occluded by any other geometry within the scene.

6.3. Occlusion Testing

6.3.1. Underlying Methodology

Once the three-dimensional position box $\left[\mathbf{p}\right]$ has been solved for the facet of interest, it must then be checked to verify that it does not contain positions for which the view of the facet is occluded by any other geometries (either internal, i.e., other facets belonging to the part, or external, i.e., by other objects/geometries present in the inspection space). Note that only the $\left[x\right],\left[y\right],\left[z\right]$ components of $\left[\mathbf{p}\right]$ are considered in this test, as it is primarily concerned with determining if there is any straight continuous path between any point on the facet and any point in the position box which intersects other geometries. The occlusion testing process will be demonstrated herein in 2D, but the techniques in question are easily extrapolated into 3D.

The first step in the occlusion testing algorithm is to build the convex hull containing both $\left[\mathbf{p}\right]$ and the facet of interest (this will be called the camera mesh) as shown in Figure 4a. This convex hull is constructed, and further operations are conducted, using exact predicates and geometric operations with CGAL such that it can be certified that any results are an exact computation of any further mesh boolean operations.

The scene mesh is then subtracted from this camera mesh, and the visibility of the facet of interest from $\left[\mathbf{p}\right]$ can then be quantified based on the result of this boolean difference operation.

The first test case will be for a set of camera positions for which the facet will be fully visible. Figure 4b shows the camera mesh and the part mesh before the differencing operation, as well as the resultant mesh. As the box represents positions from which the facet is fully visible, the camera mesh remains unchanged, as one would expect.

Next, a box in which some, but not all, positions within the set have another geometry occluding the facet is presented. Figure 4c shows the camera mesh and the part mesh, along with the resultant mesh, before and after the differencing operation.

From these, it is plain to see the effect that the differencing operation has had on the camera mesh, as the section of the part mesh that intersected the camera mesh has been subtracted from the initial camera mesh. The resultant void causes the differenced mesh to have more facets than the original camera mesh, which tells the algorithm that at least some degree of occlusion is present. One can say that the occlusion is partial rather than full, because while the differenced mesh is discontinuous, the vertices corresponding to the camera position box and those corresponding to the facet of interest are still part of the same continuous sub-mesh, and there are still edges remaining which connect at least one box vertex to at least one facet vertex.

Finally, a box for whom all inner positions represent an occluded view is presented. Figure 4d shows the camera mesh, the part mesh, and the resultant mesh.

Because the mesh subtracting has resulted in two separate meshes, which separately contain the original box vertices and the original facet vertices, the algorithm will classify this result as a case in which the particular facet of interest is fully occluded from any viewpoint within the original pose box. While this is the most common case for a full occlusion, there is a second case to consider: the case in which the subtraction still results in one continuous mesh, but one which does not contain any of the original facet vertices. This would be the case for a box that was behind the plane of the facet of interest, which, while possible, would be filtered out as a possible valid set of poses by the previously described backface culling condition.

6.3.2. Facet Occlusion Classification

The algorithm for classifying the visibility of the facet of interest from $\left[\mathbf{p}\right]$ using the above methods is described in Algorithm 2.

The results of this algorithm for classifying the visibility of facets on the sample part for the box described in Equation (32) as

$$\left[\mathbf{p}\right]=\left(\begin{array}{c}[-0.1, 0.1]m\\ [-0.1, 0.1]m\\ [0.25, 0.3]m\end{array}\right)$$

(32)

are presented in Figure 5b,c, with the box and part in Figure 5a. In these figures, green represents a fully visible facet, blue represents a partially visible facet, and red represents a fully occluded facet.

Algorithm 2 Algorithm for determining if facet is occluded

1: Generate convex hull containing all facet and $\left[\mathbf{p}\right]$ vertices (camera mesh) 2: Generate convex hull containing just $\left[\mathbf{p}\right]$ vertices (box mesh) 3: Subtract part mesh from camera mesh 4: Compare differenced camera mesh to initial camera mesh 5: if Identical then 6:     facet is fully visible 7: else 8:     go to 10 9: end if 10: Check if camera mesh contains all original facet/box vertices 11: if Differenced camera mesh contains no original box vertices, subtract part mesh from initial box mesh then 12:     if Result is empty then 13:         $f_j$ is not visible 14:     else 15:         go to 20 16:     end if 17: else if Differenced camera mesh does not contain all facet vertices then 18:     facet is not visible 19: end if 20: Check if facet vertices and remaining box vertices (or vertices of differenced box mesh if no original box vertices are present) are connected in differenced camera mesh 21: if yes then 22:     facet is partially visible 23: else 24:     facet is not visible 25: end if

6.4. Orientation Constraints

The orientation of the camera from any given position such that the facet is within its field of view is primarily determined by the camera’s field of view (FOV) half angles. Let the FOV half angles be ${\alpha }_v$ and ${\alpha }_h$ for the vertical and horizontal axes, respectively. The camera axis $\mathbf{c}$ is given by

$$\left[\mathbf{c}\right]=\left[\mathbf{R}(\left[\phi \right],\left[\gamma \right],\left[\beta \right])\right]\widehat{\mathbf{z}}$$

(33)

where

$$\widehat{\mathbf{z}}=\left(\begin{array}{c}0\\ 0\\ 1\end{array}\right).$$

(34)

A FOV constraint ${\mathcal{C}}_{FOV}(x,y,z,\phi ,\gamma ,\beta )$ is formulated to ensure that the entire facet is visible from the camera pose.

Before constraint systems can be generated and solved to define complete orientation intervals, a few transformations must be made in order to simplify the problem and allow for a more efficient solving. The formulation of the orientation limits for a given box corresponding to a given facet begins by considering the facet vertices and the 3D box defining a set of solutions for camera position. The facet vertices are presented in a matrix of discrete values, ${\mathbf{F}}_v$, as

$${\mathbf{F}}_v=\left(\begin{array}{ccc}{v}_{1_x}& v_{2_x}& v_{3_x}\\ v_{1_y}& v_{2_y}& v_{3_y}\\ v_{1_z}& v_{2_z}& v_{3_z}\end{array}\right).$$

(35)

The solver also requires the camera position box, $\left[\mathbf{p}\right]$, and facet bounding box ${\left[\mathbf{o}\right]}_f$.

The difference equations in Equations (36) and (37) are then applied in order to transform the facet and vertex boxes such that instead of attempting to determine the relationship between two boxes, we can examine the positions of the boxes relative to a common discrete origin. The difference of the facet and camera boxes is referred to as ${\left[\mathbf{d}\right]}_f$ and that of the vertices and the camera is ${\left[\mathbf{D}\right]}_v$.

$${\left[\mathbf{d}\right]}_f={\left[\mathbf{o}\right]}_f-\left[\mathbf{p}\right]$$

(36)

$${\left[\mathbf{D}\right]}_v={\mathbf{F}}_v-(\left[\mathbf{p}\right],\left[\mathbf{p}\right],\left[\mathbf{p}\right])$$

(37)

By then taking the midpoint of ${\left[\mathbf{d}\right]}_f$, one can solve for a nominal camera vector ${\mathbf{c}}_{nom}$ originating at the centre of ${\left[\mathbf{p}\right]}_i$ and passing through the facet geometric centre. If we consider this as the z-axis of a nominal camera frame, we can use simple transformations to solve for the first two components of a ZXZ Euler angle rotation sequence, which results in a nominal camera frame oriented towards the facet from $\left[\mathbf{p}\right]$. We call these values ${\phi }_{nom}$ and ${\gamma }_{nom}$, and we set ${\beta }_{nom}=0$.

Finally, we apply the ZXZ Euler rotation sequence defined by $\mathbf{R}({\phi }_{nom},{\gamma }_{nom},{\beta }_{nom})$ to the columns of ${\left[\mathbf{D}\right]}_v$,

$${\left[\mathbf{D}\right]}_{v_{1:3,i}}^{\prime }=\mathbf{R}({\phi }_{nom},{\gamma }_{nom},{\beta }_{nom}){\left[\mathbf{D}\right]}_{v_{1:3,i}},i=1\dots 3$$

(38)

in order to place the vertices of the differenced box into camera space. We call this transformed matrix ${\left[\mathbf{D}\right]}_v^{\prime }$. From here, we can define two constraint systems which will solve the orientation components. It should be noted that the first constraint system is solving for the allowable offsets of $\left[\phi \right]$ and $\left[\gamma \right]$ (${\left[\phi \right]}_{offset}$ and ${\left[\gamma \right]}_{offset}$) about their nominal values (${\phi }_{nom}$, ${\gamma }_{nom}$), and the final $\left[\phi \right]$ and $\left[\gamma \right]$ intervals will be the sum of the offset intervals and the nominal values. However, the second constraint system solves for $\left[\beta \right]$ directly.

6.4.1. Allowable ${\left[\phi \right]}_{offset}$ and ${\left[\gamma \right]}_{offset}$ Intervals

If we look at each rotation component separately and consider the corresponding FOV half angles, we can create a set of four hyperplanes passing through the origin which bound the points in ${\left[\mathbf{D}\right]}_v^{\prime }$ from Equation (38). These four hyperplanes are split into two pairs, with one pair parallel to the camera frame’s XZ-plane, and the other parallel to the camera frame’s YZ-plane. By rotating each in opposite directions about the camera’s X- or Y-axis, respectively, we create an artificial “frustum” which bounds the vertices of ${\left[\mathbf{D}\right]}_v^{\prime }$. By examining the angles between each hyperplane and its original plane in the camera frame, we can determine the allowable offsets about ${\phi }_{nom}$ and ${\gamma }_{nom}$. The variables involved in this constraint system are the four hyperplane angles,

• $\left[{\phi }_{left}\right]\in [0,\pi ]$
• $\left[{\phi }_{right}\right]\in [-\pi ,0]$
• $\left[{\gamma }_{up}\right]\in [-\pi ,0]$
• $\left[{\gamma }_{down}\right]\in [0,pi]$

along with the constants

• ${\alpha }_h$
• ${\alpha }_v$
• ${\left[\mathbf{D}\right]}_v$.

This, in turn, leads to the constraints

$${\left[\mathbf{D}\right]}_{v_{i_x}}cos\left[{\phi }_{left}\right]+{\left[\mathbf{D}\right]}_{v_{i_z}}sin\left[{\phi }_{left}\right]\le 0$$

(39)

$${\left[\mathbf{D}\right]}_{v_{i_x}}cos\left[{\phi }_{right}\right]+{\left[\mathbf{D}\right]}_{v_{i_z}}sin\left[{\phi }_{right}\right]\ge 0$$

(40)

$${\left[\mathbf{D}\right]}_{v_{i_y}}cos\left[{\gamma }_{up}\right]+{\left[\mathbf{D}\right]}_{v_{i_z}}sin\left[{\gamma }_{up}\right]\le 0$$

(41)

$${\left[\mathbf{D}\right]}_{v_{i_y}}cos\left[{\gamma }_{down}\right]+{\left[\mathbf{D}\right]}_{v_{i_z}}sin\left[{\gamma }_{down}\right]\ge 0$$

(42)

$$|\left[{\phi }_{right}\right]-\left[{\phi }_{left}\right]|\le {\alpha }_h$$

(43)

$$|\left[{\gamma }_{up}\right]-\left[{\gamma }_{down}\right]|\le {\alpha }_v.$$

(44)

By then using HC4 and ACID contractors to contract the interval variables, we can solve for their domains,

$${\left[\phi \right]}_{offset}=[-{\alpha }_h+\underline{{\phi }_{left}},{\alpha }_h+\overline{{\phi }_{right}}]$$

(45)

and

$${\left[\gamma \right]}_{offset}=[-{\alpha }_v+\underline{{\gamma }_{down}},{\alpha }_v+\overline{{\gamma }_{up}}].$$

(46)

Having solved for the domains, we can then define the full allowable $\phi$ and $\gamma$ intervals for the given pose box as

$$\left[\phi \right]={\phi }_{nom}+{\left[\phi \right]}_{offset}$$

(47)

$$\left[\gamma \right]={\gamma }_{nom}+{\left[\gamma \right]}_{offset}.$$

(48)

6.4.2. Allowable $\beta$ Interval

Once allowable $\left[\phi \right]$ and $\left[\gamma \right]$ intervals have been calculated for a given pose box, it is possible to identify the allowable $\left[\beta \right]$ angle interval for a given set of camera poses. The process starts with ${\left[\mathbf{D}\right]}_v^{\prime }$. We can then solve for the projection of each of the vertices (the columns of ${\left[\mathbf{D}\right]}_v^{\prime }$) onto the camera’s image plane. For each column ${\left[\mathbf{d}\right]}_{v_i}^{\prime }$, $i=1\dots 3$ in ${\left[\mathbf{D}\right]}_v^{\prime }$, we can use the matrices $\mathbf{K}$ and $\left[\mathbf{R}\right(\left[\phi \right],\left[\gamma \right],\left[\beta \right]\left)\right]$ to get the projection ${\left[\mathbf{d}\right]}_{p_i}$,

$${\left[\mathbf{d}\right]}_{p_i}=\mathbf{K}\left[\mathbf{R}(\left[\phi \right],\left[\gamma \right],\left[\beta \right])\right]{\left[\mathbf{d}\right]}_{v_i}^{\prime },i=1\dots 3.$$

(49)

Because $\mathbf{K}$ and ${\left[\mathbf{d}\right]}_{v_i}^{\prime }$ are constant matrices, and we have already solved for $\left[\phi \right]$ and $\left[\gamma \right]$, we can then define the constraints

$$\mathbf{K}\left[\mathbf{R}(\left[\phi \right],\left[\gamma \right],\left[\beta \right])\right]{\left[\mathbf{d}\right]}_{v_i}^{\prime }>{\left(\mathbf{0}\right)}_{3\times 1}$$

(50)

$${\left(\mathbf{K}\left[\mathbf{R}(\left[\phi \right],\left[\gamma \right],\left[\beta \right])\right]{\left[\mathbf{d}\right]}_{v_i}^{\prime }\right)}_{1:2}<\left(\begin{array}{c}w\\ h\end{array}\right).$$

(51)

then subsequently apply a combination of HC4 and ACID contractors to contract the domain of $\left[\beta \right]$, such that the constraints are satisfied.

7. Overall Pose Solver Algorithm

Using all of the concepts established herein, it is now possible to formulate an overall pose solver algorithm for a single facet in an arbitrary 3D mesh representing a real part geometry. First, the initial constraints are formulated for determining whether or not a given set of poses represents a valid solution to the inspection pose problem. This includes distance constraints, orientation/field of view constraints, and visibility and occlusion constraints. Next, using interval-based system contraction methods, the search space around the part is contracted to find the initial position search space, which is the axis-aligned bounding box that most tightly bounds all possible solutions. From there, the position constraints are evaluated over this box, and it is iteratively subdivided and analyzed in order to determine the full set of valid, boundary, and invalid position boxes within the initial box. Next, the boxes are tested for any possible occlusion of a camera’s view of the facet from that box by any other geometries present in the scene. Finally, these position boxes have their respective orientation intervals evaluated as per the method in Section 6.4, and are subsequently refined/subdivided should they be too wide for the camera’s field of view. Once all pose intervals (position and orientation) for each box have been solved and all boxes have been classified, the full tree structure containing all boxes is returned as the final synthesized solution set of all possible valid inspection poses for the facet of interest. This process is detailed in Algorithm 3.

Algorithm 3 Overall algorithm for determining a complete set of valid inspection poses for single facet

1: Initialize empty list of unclassified position boxes ${\mathcal{L}}_{\mathbf{u}}$ 2: Initialize empty list of valid position boxes ${\mathcal{L}}_{\mathbf{v}}$ 3: Initialize empty list of solution 6D pose boxes ${\mathcal{L}}_{\mathbf{s}}$ 4: Initialize search space by contracting position variables to get the initial box $\left[\mathbf{u}\right]$ 5: Add box to end of list ${\mathcal{L}}_{\mathbf{u}}$ 6: if ${\mathcal{L}}_{\mathbf{u}}\ne Ø$ then 7:     Evaluate position and occlusion constraints 8:     if Constraints are fully satisfied then 9:           Classify box ${\left[\mathbf{k}\right]}_i$ as VALID 10:         Add ${\left[\mathbf{k}\right]}_i$ to back of ${\mathcal{L}}_{\mathbf{v}}$ 11:         go to 6 12:     else if Constraints are partially satisfied then 13:         if Widest interval in ${\left[\mathbf{u}\right]}_i$ is narrower than minimum resolution then 14:            Classify box ${\left[\mathbf{u}\right]}_i$ as BOUNDARY 15:            Add box ${\left[\mathbf{k}\right]}_i$ to back of ${\mathcal{L}}_{\mathbf{v}}$ 16:            go to 6 17:         else 18:            Bisect ${\left[\mathbf{u}\right]}_i$ into ${\left[\mathbf{u}\right]}_{i_1}$ and ${\left[\mathbf{u}\right]}_{i_2}$ 19:            Add ${\left[\mathbf{u}\right]}_{i_1}$ and ${\left[\mathbf{u}\right]}_{i_2}$ back onto back of ${\mathcal{L}}_{\mathbf{u}}$ 20:            go to 6 21:         end if 22:     else 23:         Remove ${\left[\mathbf{u}\right]}_i$ from ${\mathcal{L}}_{\mathbf{u}}$ 24:         go to 6 25:     end if 26: else 27:     Terminate position solver and return ${\mathcal{L}}_{\mathbf{v}}$ 28:     while ${\mathcal{L}}_{\mathbf{v}}\ne Ø$ do 29:         Pop ${\left[\mathbf{k}\right]}_i$ from ${\mathcal{L}}_v$ and solve orientation intervals according to Section 6.4 30:         if $({\left[\phi \right]}_{k_i}>{\alpha }_h)|({\left[\gamma \right]}_{k_i}>{\alpha }_v)$ then 31:            Bisect ${\left[\mathbf{k}\right]}_i$ ${\left[\mathbf{k}\right]}_{i_1}$ and ${\left[\mathbf{k}\right]}_{i_2}$ 32:            Add ${\left[\mathbf{k}\right]}_{i_1}$ and ${\left[\mathbf{k}\right]}_{i_2}$ back onto back of ${\mathcal{L}}_{\mathbf{v}}$ 33:         else 34:            Add completely resolved ${\left[\mathbf{k}\right]}_i$ to end of ${\mathcal{L}}_{\mathbf{s}}$ 35:         end if 36:     end while 37:     Return ${\mathcal{L}}_{\mathbf{s}}$ as a complete set of camera deployment poses for facet of interest 38: end if

8. Single Facet Case Study

Combining all previous elements, we can now solve for a given facet’s entire set of valid poses. This was tested for the facet highlighted as the facet of interest in Figure 6, as it has obvious occlusions due to the protrusion on the top surface of the part, and as such is a good representative sample for the effectiveness of the visibility test in removing invalid boxes.

The valid position boxes for this facet are presented from various positions in Figure 6b–f show a sample selected pose box with nominal camera orientation so as to effectively demonstrate the full 3D solution space.

The simulation was carried out on a laptop computer with an Intel^® Core^TM i5-6440HQ CPU @ 2.60 GHz × 4 processor (Intel Corporation, Santa Clara, CA, USA) and 8 GB of RAM. The sample part has a total of 1145 vertices and 2298 faces, with a single face selected as the facet of interest. The test was conducted with a position bisection stopping threshold of 0.1 m, and an orientation bisection stopping threshold of 0.38 rad.

9. Discussion of Results and Future Work

The end result of this simulation produced a total of 1614 6D pose boxes. On average, this test took approximately 53 s. It is clearly visible in Figure 6b that the pose solutions obey the maximum distance and viewing angle constraints as expected, while Figure 6d shows the underside of the solution set obeying minimum distance requirements as well. Furthermore, Figure 6c clearly shows the solution set obeying the occlusion constraint, as no boxes are present behind the planes containing the adjacent facets of the protrusion on the top surface which would occlude the view of the facet of interest. Figure 6e,f also shows that the camera is correctly oriented to have the field in its field of view.

It is clear from these results that the algorithm was able to effectively synthesize the complete continuous set of valid camera poses (for a given resolution) for the example facet on the part given the constraints defined in previous sections.

However, the 53-s computational time for a single facet’s complete set of pose solutions does imply that the computational complexity of the algorithm may increase dramatically with the inclusion of more facets and the use of more complex parts. As such, computational optimizations will be a significant focus for future work, but they are beyond the scope of this paper.

10. Conclusions

The methodology presented herein establishes the fundamental concepts for synthesizing complete inspection spaces for a given part geometry. This is undertaken by first establishing the methods for synthesizing the continuous inspection space for a single facet from a part’s triangular mesh representation. The set-based constraints which were formulated were able to effectively solve for the complete valid inspection space for the example facet in question, and are a promising preliminary result towards the solving of inspection spaces for a part’s entire geometry. Future work will focus on solving the intersections of individual facet inspection spaces for complete solution synthesis and further refinement of constraints to account for real camera intrinsic parameters, which are considered idealized in this scenario, as well as emphasizing computational efficiency. To the best of the authors’ knowledge, the derived methodology is new and can be readily adopted and scalable in various manufacturing facilities and applications.