Energy-Aware Camera Location Search Algorithm for Increasing Precision of Observation in Automated Manufacturing

Abstract

Visual servoing technology is well developed and applied in many automated manufacturing tasks, especially in tools’ pose alignment. To access a full global view of tools, most applications adopt an eye-to-hand configuration or an eye-to-hand/eye-in-hand cooperation configuration in an automated manufacturing environment. Most research papers mainly put efforts into developing control and observation architectures in various scenarios, but few have discussed the importance of the camera’s location in the eye-to-hand configuration. In a manufacturing environment, the quality of camera estimations may vary significantly from one observation location to another, as the combined effects of environmental conditions result in different noise levels of a single image shot in different locations. In this paper, we propose an algorithm for the camera’s moving policy so that it explores the camera workspace and searches for the optimal location where the image’s noise level is minimized. Also, this algorithm ensures the camera ends up at a suboptimal (if the optimal one is unreachable) location among the locations already searched with the limited energy available for moving the camera. Unlike a simple brute-force approach, the algorithm enables the camera to explore space more efficiently by adapting the search policy by learning the environment. With the aid of an image-averaging technique, this algorithm, in the use of a solo camera, achieves observation accuracy in eye-to-hand configurations to a desirable extent without filtering out high-frequency information in the original image. An automated manufacturing application was simulated, and the results show the success of this algorithm’s improvement in observation precision with limited energy.

1. Introduction

In the automated industry, automatic alignment plays a critical role in many different applications, such as micromanipulation, autonomous welding, and industrial assembly [1,2,3]. Visual servoing is the technique commonly used in tasks of alignment as it can guide robotic manipulators to their desired poses (positions and orientations) [4,5,6]. Visual servoing is the method of controlling a robot’s motion using real-time feedback from visual sensors that continuously extract image features [7].

According to the relative positions between cameras and the tool manipulators, visual servoing can be generally divided into two categories: eye-in-hand (EIH) configuration and eye-to-hand (ETH) configuration [8,9]. In EIH, a camera is mounted directly on a robot manipulator, in which case the motion of the robot induces the motion of the camera, while in ETH, the camera is fixed in the workspace and observes the motion of the robot from a stationary configuration. Both configurations have their own merits and drawbacks regarding the field-of-view limit and precision that oppose each other. EIH has a partial but precise view of the scene, whereas ETH has a global but less precise view of it [10]. For complex tasks in automated manufacturing, one configuration of visual servoing is inadequate as these tasks require the camera to provide global views and be maneuverable enough to explore the scene [10]. To take advantage of both stationary and robot-mounted sensors, two configurations can be employed in a cooperative scheme, which can be seen in some studies [11,12,13].

In recent years, many works have been published to show visual servoing paradigms being applied in industrial environments. For example, Lippiello et al. [14] presented a position-based visual servoing approach for a multi-arm robotic cell equipped with a hybrid eye-in-hand/eye-to-hand multicamera system. Zhu et al. [15] discussed Abbe errors in a 2D vision system for robotic drilling. Four laser displacement sensors were used to improve the accuracy of the vision-based measurement system. Liu et al. [16,17] proposed a visual servoing method for the automative assembly of aircraft components. With the measurements from two CCD cameras and four distance sensors, the proposed method can accurately align the ball-head, which is fixed on aircraft structures in a finite time. Sensor accuracy is a key element in many studies as it is not only a requirement for some delicate applications but also affects the robustness of visual servoing techniques [18].

However, gaps exist between real-world applications and the open-space experiments proposed in the above studies. Real-world products are usually manufactured in a confined and complicated environment where complex factors may significantly degrade the quality of observed signals given from even the most precise sensors. For example, illumination changes in space [18] can be a serious issue for camera estimation in real manufacturing processes, which is not normally addressed in research experiments where illumination is usually sufficient and equally distributed.

In this paper, we would like to address the importance of camera location in ETH or ETH/EIH cooperative configurations and discuss the benefits of developing algorithms that place the camera in an optimal location. To observe a single point in the workspace, there are an infinite number of poses that a fixed camera can be placed in the space. Among all possible poses of the camera that keep the same target within the field-of-view limit, the image noise at different locations may play a great role in the precision of observations. Many environmental conditions (such as illumination, temperature, etc.) affect the noise level of a single image [19]. In a manufacturing environment, the quality of estimations from a camera may vary significantly from one observation location to another. For instance, the precision of an estimation improves greatly if the camera moves from a location where it is placed within the shade of machines to a location that has better illumination. Also, places near machines may be surrounded by strong electrical signals that introduce extra noise into the camera sensors.

Many papers in visual servoing consider applying filters to reduce image noise and, therefore, increase the precision of observations [20,21,22,23]. Gaussian white noise can be dealt with using spatial filters, e.g., the Gaussian filter, the Mean filter, and the Wiener filter [24]. In addition, wavelet methods [25,26] have the benefit of keeping more useful details but at the expense of computational complexity. However, filters that operate in the wavelet domain still inevitably filter out (or blur) some important high-frequency useful information of the original image, even though they succeed in preserving more edges compared with spatial filters. Especially at locations where the signal-to-noise ratio (SNR) in an image is low, no matter what filters are applied, it is difficult to safeguard the edges of the noise-free image as well as reduce the noise to a desirable level. Thus, developing an algorithm that searches for locations of the camera where the SNR is high is beneficial.

We can approach the denoising problem with multiple noisy images taken from the same perspective. This method is called signal averaging (or image averaging in the application of image processing) [27]. Assume images taken from the same perspective but at different times have the same noise levels. Then, averaging multiple images at the same perspective will reduce the unwanted noise as well as retain all the original image details. We can also assume the robot’s end-effector holds the camera rigidly so that any shaking and drift are negligible when shooting pictures. Moreover, the number of images required for averaging has a quadratic dependence on the ratio of an image’s original noise level to the reduced level that is desirable for observations. Therefore, the number of images required is a measurement of the noise level in a single image. Furthermore, in the denoising process, precise estimations require that the original image details are retained as much as possible. Based on previous statements, we decided to choose image averaging over all other denoising techniques in this work.

Frame-based cameras, which detect, track, and match visual features by processing images at consecutive frames, cause a fundamental problem of delays in image processing and the consequent robot action [28]. By averaging more images for an observation, fewer noises are maintained in the averaged image, but more processing time is required for acquiring visual features; it is a typical tradeoff between high-rate and high-resolution in cameras [18]. The control loop frequency is limited in many cases due to the low rate of imaging, which undermines the capability/usage of visual servoing techniques in high-speed operations. Therefore, it is rewarding for an eye-in-hand configuration to find the location of the camera where the original image noise is lowest; thus, the least number of images is needed in order to facilitate the manufacturing speed.

Brute-force search [29] is a simple and straightforward approach to looking for the optimal location of the camera. It systematically searches all possible locations in space and gives the best solution. This method is only feasible when the size of the search space is small enough. Otherwise, brute force, which checks every possibility, is usually inefficient, especially for large datasets or complex problems, where the number of possibilities can grow exponentially. An intelligent algorithm is necessarily developed for the sake of energy efficiency and time consumption.

In this paper, we propose an algorithm that efficiently searches the camera’s workspace to find an optimal location (if its orientation is fixed) of the camera so that a single image taken at this location has the smallest noise level among images taken at all locations in the space. With limited energy for moving the camera, this algorithm also ensures the camera ends up at a suboptimal (if the optimal pose is unreachable) location among the locations already searched.

2. Denoising Technique Comparison

In this section, we further discuss various existing image-denoising techniques and conduct a comparison study of some techniques based on their noise-reduction rate and high-frequency data loss.

Various denoising techniques have been proposed so far. A good noise removal algorithm ought to remove as much noise as possible while safeguarding edges. Gaussian white noise has been dealt with using spatial filters, e.g., the Gaussian filter, the Mean filter, and the Wiener filter 25]. Images can be viewed as a matrix with variant intensity values at each pixel. The spatial filter is a square matrix that performs convolutional production with the image matrix to produce filtered (noise-reduced) images, i.e.,

$$g\left(i,j\right)=k\ast v(i,j)$$

(1)

where $v(i,j)$ is the original (noisy) image matrix entry at the coordinate $(i,j)$. $k$ is the spatial filter (or kernel), and * accounts for the convolution. $g\left(i,j\right)$ is the filtered (noisy-reduced) image matrix entry at the coordinate $(i,j)$. All the spatial filters have the benefit of being computationally efficient but the shortcoming of blurring edges [25].

Also, image Gaussian noise reduction can be approached in the wavelet domain. In the conventional Fourier Transform, only frequency information is provided, while temporal information is lost in the transformation process. However, wavelet transformation will keep both the frequency and temporal information of the image [25]. The most widely used wavelet noise-decreasing method is the wavelet threshold method, as it can not only obtain an approximate optimal estimation of the original image but also calculate speedily and adapt widely [25]. The method can be divided into three steps. The first step is to decompose the noise-polluted image by wavelet transformation. Then, wavelets of useful signals will be retained, while most wavelets of noise will be set to zero according to the set threshold. The last step is to synthesize the new noise-reduced image by inverse wavelet transformation of the cleaned wavelets in the previous step [25,26]. The wavelet transformation method has the benefit of keeping more useful details but is more computationally complex than spatial filters. The threshold affects the performance of the filter. Soft thresholding provides smoother results, while a hard threshold provides better edge preservation [25]. However, whichever threshold is selected, filters that operate in the wavelet domain still filter out (or blur) some important high-frequency useful details in the original image, even though more edges are preserved than with spatial filters.

All the aforementioned methods present ways to reduce noise in image processing starting from a noisy image. We can approach this problem with multiple noisy images taken from the same perspective. Assume the same perspective ensures the same environmental conditions (illumination, temperature, etc.), which affect the image noise level. Given the same conditions, an image noise level taken at a particular time should be very similar to another image taken at a different time. This redundancy can be used for the purpose of improving image precision estimation in the presence of noise. The method that uses this redundancy to reduce noise is called signal averaging (or image averaging in the application of image processing) [27]. Image averaging has the natural advantage of retaining all the image details as well as reducing unwanted noises, given that all the images for the averaging technique are taken from the same perspective.

The image-averaging technique is illustrated in Figure 1. Assume a random, unbiased noise signal and that this noise signal is completely uncorrelated with the image signal itself. As noisy images are averaged, the original true image is kept the same, and the magnitude of the noise signal is compressed, thus improving the signal-to-noise ratio. In Figure 1, we generated two random signals with the same standard deviation, and they are, respectively, represented by blue and red lines. The black line is the average of the two signals, whose magnitude is significantly decreased compared to each of the original signals. In general, we can come up with a mathematical relationship between the noise level reduction and the sample size for averaging. Assume we have N numbers of Gaussian white noise samples with the standard deviation $\sigma$. Each sample is denoted as $z_i$, where i represents the $i^{th}$ sample signal. Therefore, we can acquire:

$$var\left(z_i\right)=E\left({z_i}^2\right)={\sigma }^2$$

(2)

where $var\left(·\right)$ is the variance, $E\left(·\right)$ is the expectation of the signal, and $\sigma$ is the standard deviation.

By averaging N Gaussian white noise signals, we can write:

$$var(z_{\mathrm{a}\mathrm{v}\mathrm{g}}) var({\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{z}_i) {\displaystyle \frac{1}{N^2}}N{\sigma }^2={\displaystyle \frac{1}{N}} {\sigma }^2={({\displaystyle \frac{1}{\sqrt{N}}}\sigma )}^2$$

(3)

where $z_{\mathrm{a}\mathrm{v}\mathrm{g}}$ is the average of N noise signals. Equation (3) demonstrates that a total number of N samples is required to reduce the signal noise level by $\sqrt{N}$. Since our goal is to reduce the image noise within a fixed threshold (represented by a constant standard deviation), a smaller deviation in the original image requires much fewer samples to make an equivalent noise-reduction estimation. Thus, it is worthwhile for the camera to move around, rather than being stationary, to find the best locations where the image noise level estimation is small. In general, we can reduce the noise level as much as needed by taking more samples.

A comparison of the above denoising methods is presented in the following: In Figure 2, a default picture from MATLAB R2023b is polluted with Gaussian-distributed white noise with an SNR equating to 5 dB. A spatial filter (Gaussian filter), a wavelet filter (the wavelet threshold method), and the image-averaging technique (1000 average sample) were applied to denoise the noisy image. Figure 1 shows all techniques succeed in reducing the noise to some extent, while the wavelet filter and the image-averaging technique perform better than the spatial filter regarding denoising. The difference between the wavelet filter and the image-averaging technique is not obvious in Figure 2 but becomes more significant in the frequency domain.

Figure 3 compares the performance in the frequency domain of image-averaging methods with different samples, and Figure 4 shows the difference in the frequency domain after applying the above three denoising methods. The power intensity is separated from low to high, as shown in those two figures. By comparing the original image and the noisy image, it can be observed that noise intensity is larger in the higher frequency domain. Figure 3 shows that by averaging more samples, the noise level decreases in all frequencies. In particular, the averaged image is almost the same as the original one when 1000 samples are averaged.

Figure 4 shows the image generated from the Gaussian filter has a lower power intensity than that of the original image with high frequency, which indicates that a lot of high-frequency data in the original image has been filtered out as well. Although it performs better than a Gaussian filter, the wavelet filter still loses some details of the original image. Only the averaging method can reduce the noise level and retain all the information.

3. Algorithm Flowchart

The proposed algorithm aims to search and move the camera to the location where the least number of pictures are needed to reduce the noise level of the averaged image to a degree ${\sigma }_{noise\_reduced}$. A flowchart describing the algorithm is shown in Figure 5.

A camera, mounted on a 6-joint robot manipulator, moves freely in 6 degrees of freedom (6 DoFs) in space. Assuming the camera is fixed in orientation towards the face of the object of interest, only the position (3 DoFs) of the camera can vary from the movement of the robot arm. The camera can only move to the locations within the maximum reach of the robot manipulator, and the object can only be detectable within the view angle of the camera. These constraints create a camera operational space (Figure 6a), which is then gridded with a certain resolution of distance. All the intersections of grids come up with a set of nodes ($S_{total}$), which are all possible candidates of locations that the camera may move and search utilizing the algorithm. Each node is indexed and denoted as ${Node}_{index}$.

The algorithm iteratively commands the camera to take a picture at one location, generate the next target location, and move to that location until it finds the optimal location that requires the least number of pictures. In one iteration, the camera takes a photo at the current location $P^C,$ and after image processing, which transforms a picture into a pixel’s intensity matrix, it produces a noisy image with an intensity matrix $I_a.$ A previously developed algorithm [30] estimates the noise level ${\sigma }_a$ across the image. Then, at the current location of the camera, the number of pictures $N^C$ needed to reduce the noise level to ${\sigma }_{noise\_reduced}$ is calculated by the equation: $N^C={({\displaystyle \frac{{\sigma }_a}{{\sigma }_{noise\_reduced}}})}^2$. Based on this information ($N^C$ at $P^C$), the algorithm then generates the next target location of the camera, $P^{Next}$. The decision-making process of generating $P^{Next}$ is illustrated in the dashed circle of the below figure. In general, this algorithm first determines, from the current location $P^C$, the set of grid nodes that can be explorable ($S_{explored}$) in the next iteration based on the available moving energy E. Then, a hash map regarding the estimated number of pictures required for each explorable node ($M_{explorable}$) is evaluated from an estimation function that utilizes data from the hash map of all explored nodes ($M_{explored}$). A stochastic process further modifies the hash map ${\widehat{M}}_{explorable}$ to add fidelity to the estimation model. In the end, the next node’s location $P^{Next}$ is selected so that it has the smallest number of images required among all nodes in ${\widehat{M}}_{explorable}.$ If the next generated target position $P^{Next}$ is the same as the current position $P^C$, $P^C$ is the optimal location, and the camera stays and observes the object at this location. Otherwise, the robot joint controller moves the robot joint angles to the targets $\overline{q}$ (calculated from forward kinematics [31] (Appendix B, Equations (A1)–(A5))) so that the camera will move to the next target position, $P^{Next}$.

4. Core Algorithm

The core of this algorithm is to generate the target locations to be explored in the camera’s operational space. The details of this algorithm are shown in the area circled with dashed lines in Figure 5. In this section, we put effort into the development of the picture number estimation function, the explorable nodes set, the stochastic modified evaluation function, the next location selection process, and the estimated energy cost function.

4.1. Picture Number Estimation Function

In each iteration of the algorithm, the camera moves to a new node with location $P^C$, and the algorithm evaluates the number of pictures $N^C$ needed at this location. The node $\left({Node}_{index}\right)$ is then set to be explored and saved in a set $S_{explored}.$ Each node that has been explored in iterations is one-on-one paired with the values of the location and picture numbers of that node. All pairs that are expressed as ${Node}_{index}^{explored}$: ($P_{index}^{explored},N_{index}^{explored}$) are saved in a hash map $M_{explored}$. Then, $M_{explored}$ is used to develop an estimation function that can estimate the number of pictures needed for the rest of the unexplored nodes. The estimated picture number calculated for all nodes in space is denoted as $S_{\tilde{N}}$. Expressions of any element in $S_{\tilde{N}}$ are:

For any node index $x$ in the Total Node Set: $S_{total}=\left\{{Node}_x\right|x\in \left(\mathrm{1,2},\dots m\right)\}$, where $m$ is the total number of nodes.

$${\tilde{N}}_x=\left\{\begin{array}{c}{\sum }_{i\in S_{index-exp}}{\displaystyle \frac{(\overline{ED}\left(P_i^{exp},P_x\right)^(-K_{est}))}{{\sum }_{j\in S_{index-exp}}((\overline{ED}\left(P_j^{exp},P_x\right)^(-K_{est}))}}{N}_i^{explored}\\ N_x^{explored}, if {Node}_x \in S_{explored} \end{array}\right., if {Node}_x\notin S_{explored}$$

(4)

$$\mathrm{Then}: S_{\tilde{N}}=\left\{{\tilde{N}}_x\right|x\in \left(\mathrm{1,2},\dots m\right)\}$$

(5)

where $K_{est}$ is a positive constant parameter.

$$S_{index-exp}=\left\{index\right|{Node}_{index}\in S_{explored}\},$$

(6)

$$\begin{gathered} \overline{ED}\left(P_a,P_b\right)=\sqrt{{(P_{ax}-P_{bx})}^2+{(P_{ay}-P_{by})}^2+{(P_{az}-P_{bz})}^2}, \\ \mathrm{with} P_a,P_b \mathrm{are} 3\mathrm{D} \mathrm{locations} \mathrm{of} {Node}_a and {Node}_b: \\ {P}_a=\left(P_{ax},P_{ay},P_{az}\right) and P_b=(P_{bx},P_{by},P_{bz}) \end{gathered}$$

(7)

Equation (4) is a weighted function concerning each explored node. If one explored node ${Node}_i^{explored}$ is closer to the node ${Node}_x^{explored}$, its picture number $N_i^{explored}$ has a larger weight (effect) on the estimation of ${\tilde{N}}_x$. Thus, a minus sign of $K_{est}$ is utilized to indicate the negative correlation between the Euclidean distance and weight.

Equation (4) shows that at least some initial nodes are required to be explored before their values can be used to estimate the picture numbers in other nodes. For the estimation function to work properly by covering all nodes in the camera operational space, four initial nodes are selected in the following steps:

• Find the first two nodes on the boundary of operational space so that the Euclidean distance between these two nodes is the largest;
• Select the other two nodes whose connection line is perpendicular to the previous connection line, where the Euclidean distance between these two nodes is also the largest among all possible node pairs;
• Move the camera to these four nodes in space with proper orientation. Take a single image at each location and estimate the number of pictures required at those locations. Save all four nodes in S_explored and their values in M_explored.

Figure 6b shows the selection of four initial nodes in a camera operational space and an example of the use of the estimation function to estimate the picture number in one node.

4.2. Explorable Node Selection

A stage in the algorithm, Explorable Node Selection, generates a set of nodes $S_{explorable},$ which contains all the camera’s next explorable locations from the current location $P^C$ with $\mathrm{a}\mathrm{v}\mathrm{a}\mathrm{i}\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{l}\mathrm{e}$ current energy for moving $E_{bound}.$ The explorable set is found using the following steps (Figure 7):

• With current energy bound E_bound, find all nodes in the space that can be reached from the current node P^c.

In other words, all feasible nodes are in the set:

$$S_{feasible}=\left\{{Node}_{index}\right|E(P^c,P_{index})\le E_{bound}\}$$

(8)

where $P$ is the position of a node, and $E(P_a,P_b)$ is the estimated energy cost from $P_a \mathrm{t}\mathrm{o} P_b$.

When the current energy bound $E_{bound}$ is greater than a predefined energy threshold $E_T$ ($E_{bound}>E_T$), then the algorithm is safe to explore all nodes in the feasible range $S_{feasible}$ in the next iteration loop. In other words:

$$S_{explorable}=S_{feasible}, if{ E}_{bound}>E_T$$

(9)

Note: $E(P_a,P_b)$ is generated from a dynamic model of a robot-arm control system. The details of this function are developed in the following section of this paper;

2.

When E_bound ≤ E_T, step 2 and step 3 are applied to find the explorable set. In the feasible set, look at all explored nodes and find the one that has the smallest number of pictures. In other words:

$$S_{feas\&exp}=\left\{{Node}_{index}\right|{Node}_{index}\in S_{feasible}\cap S_{explored}\}$$

(10)

$$\mathrm{e}\mathrm{x}\mathrm{p}\_min=\left\{index\right|N_{\mathrm{e}\mathrm{x}\mathrm{p}\_min}=\min \left(N_{index}|{Node}_{index}{\in S}_{feas\&exp}\right)\}$$

(11)

where $S_{explored}$ is the set of all explored nodes.

$S_{feas\&exp}$ is the intersection set between $S_{feasible}$ and $S_{explored}$;

3.

The estimation function may not give accurate results for some unexplored nodes. Therefore, it is possible that the algorithm may make the camera end up in a node that has a large number of pictures in some iteration loops. Therefore, our algorithm needs to make sure, at worst, it has enough energy to go back to the best (minimum number of pictures) node that was explored when the available energy is low (E_bound ≤ E_T). This further reduces the feasible set S_feasible. The explorable set S_explorable can be written as:

$$S_{explorable}=\left\{{Node}_{index}|E_1+E_2\le E_{bound}\right\},\hspace{1em}if{ E}_{bound}\le E_T$$

(12)

$$E_1=E(P^c,P_{index}), E_2=E(P_{index},P_{\mathrm{e}\mathrm{x}\mathrm{p}\_min})$$

(13)

Figure 7 illustrates the selection of explorable node sets.

4.3. Stochastic Modified Evaluation

For all nodes in $S_{explorable},$ we set up a hash map $M_{explorable}$. For each node, its location and number of required pictures for that node are given as: ${Node}_{index}^{explorable}$: ($P_{index}^{explorable},{\tilde{N}}_{index}^{explorable}$). The number of pictures required for nodes in $S_{explorable}$ are calculated from the estimation function in Equation (4).

The algorithm tends to select nodes located around the explored node that has the minimum number of pictures because, from Equation (4), these nodes tend to have the smallest estimated number of pictures. However, this process does not guarantee reaching the global minimum, therefore, minimizing the number of pictures. To sufficiently explore unknown nodes as well as exploit information from already explored nodes, a stochastic process is introduced to modify the picture number estimation function given in Equation (4).

Assume when the locations of a camera in space deviate a smaller amount from each other, the difference of real picture number values at those locations is also smaller. Thus, among all unexplored nodes, the estimated picture number of an unexplored node, which is closer to the explored node, is more deterministic. To emphasize this feature, we make the estimated picture number follow a normal distribution with a mean ${\mu }_{\tilde{N}}$ being equal to the value from Equation (4) and standard deviation ${\sigma }_{\tilde{N}}$. As an unexplored node is further away from explored nodes, the larger value of ${\sigma }_{\tilde{N}}$ should be assumed, which means more indeterministic estimation (illustrated in Figure 8). The following Equations (14)–(16) summarize the above discussion:

For any nodes ${Node}_{index=x}^{explorable}$ in $S_{explorable},$ its picture number estimation should follow the following normal distributions:

$$P(Z_x)=\mathcal{N}({{\mu }_{\tilde{N}}}_x,{{\sigma }_{\tilde{N}}}_x)$$

(14)

$${{\mu }_{\tilde{N}}}_x={\tilde{N}}_x^{explorable}$$

(15)

$${{\sigma }_{\tilde{N}}}_x=K_{sd}\ast {\overline{ED}}_{min}$$

(16)

where $P(Z_x)$ is the probability of a random variable $Z$ at ${Node}_x^{explorable}$. ${\tilde{N}}_x^{explorable}$ is the estimated value from Equation (4) of ${Node}_x^{explorable}.$ $K_{sd}$ is a constant parameter, and ${\overline{ED}}_{min}$ is the smallest Euclidean distance among distances between that node and explored nodes.

As discussed above, the algorithm ensures that the camera, at any iteration, always has enough energy to move back to the previous best node $N_{\mathrm{e}\mathrm{x}\mathrm{p}\_min}$. Therefore, a new distribution can be generated, which sets values bigger than the minimum ($N_{\mathrm{e}\mathrm{x}\mathrm{p}\_min}$) in the original distribution to be the minimum value. Then, Equations (14) to (16) are modified with a new random variable $Z_x^{new}$ and its expectations as follows:

$$Z_x^{new}=\left\{\begin{array}{c}{Z}_x, if Z_x<N_{\mathrm{e}\mathrm{x}\mathrm{p}\_min} \\ N_{\mathrm{e}\mathrm{x}\mathrm{p}\_min}, if Z_x \ge N_{\mathrm{e}\mathrm{x}\mathrm{p}\_min}\end{array}\right.$$

(17)

$$P\left(Z_x^{new}\right)=P(Z_x)=\mathcal{N}({{\mu }_{\tilde{N}}}_x,{{\sigma }_{\tilde{N}}}_x)$$

(18)

$${\widehat{N}}_x^{explorable}=\sum [Z_x^{new}\ast P\left(Z_x^{new}\right)]$$

(19)

where ${\widehat{N}}_x^{explorable}$ is the stochastic modified estimated value for the node ${Node}_x^{explorable}$.

Then, a new hash map ${\widehat{M}}_{explorable}$ is set up with pairs as ${Node}_{index}^{explorable}$: ($P_{index}^{explorable},{\widehat{N}}_x^{explorable}$). The next target node to be explored is the one that has the smallest modified estimated value in the map:

$$inde{x}_{Next}=\left\{index\right|{\widehat{N}}_{inde{x}_{Next}}^{explorable}=\min ({\widehat{N}}_{index}^{explorable})\}$$

(20)

$$P^{Next}=P_{inde{x}_{Next}}^{explorable}$$

(21)

4.4. Estimated Energy Cost Function

As discussed above, an estimated energy cost function $E(P_a,P_b)$ calculates the estimated energy cost from $P_a \mathrm{t}\mathrm{o} P_b$. This function is used to find the feasible node set $S_{feasible}$ and explorable node set $S_{explorable}$. In this section, the equations for the cost function are derived from the robot positioning–controlled system’s response. Also, tradeoffs between the energy cost and the settling time of moving are discussed.

The energy of moving the camera between two locations from $P_{initial}$ to $P_{final}$, results from the energy cost of DC motors in 6-DoFs robot arm joints rotating from joint angles $q_{initial}$ to $q_{final}$. Therefore, $E(P_{initial},P_{final})$ can be described as the sum of the time integration of rotational power in each joint:

$$E(P_{initial},P_{final})=\sum _{i=1}^6{\int }_{t_0^i}^{t_f^i}{V}^i·I^i·dt$$

(22)

where $V^i$ is the voltage, and $I^i$ is the current in the DC motor’s circuit of the $i^{th}$ joint. $t_0^i$ and $t_f^i$ are the initial time and final time of the $i^{th}$ joint moving from its initial angle $q_{initial}^i$ to its final angle $q_{final}^i$.

Without detailed derivation, the dynamic model of 6-dof revolutionary robotic manipulators and DC motors can be expressed as [31]:

$${\displaystyle \frac{1}{r_k}}{J_m}_k{\ddot{q}}_k+\sum _{j=1}^6{d}_{j,k}{\ddot{q}}_j+\sum _{i,j}^6{c}_{i,j,k}\left(q\right){\dot{q}}_i{\dot{q}}_j+{{\displaystyle \frac{1}{r_k}}\mathrm{B}{\dot{q}}_k+\mathsf{\Phi }}_k\left(q\right)={\displaystyle \frac{K_m}{r_{k}R}}{V}^i$$

(23)

$$J_m=J_a+J_g, {B=B}_m+{\displaystyle \frac{K_b{K}_m}{R}}$$

(24)

$$c_{i,j,k}={\displaystyle \frac{1}{2}}({\displaystyle \frac{\delta d_{k,j}}{\delta q_i}}+{\displaystyle \frac{\delta d_{k,i}}{\delta q_j}}-{\displaystyle \frac{\delta d_{i,j}}{\delta q_k}}), {\mathsf{\Phi }}_k={\displaystyle \frac{\delta V}{\delta q_k}}$$

(25)

$$i,j,k\in (1, 2, 3, 4, 5, 6)$$

(26)

where Equations (23) and (24) express the $k^{th}$ joint dynamics equation. $q_i/q_j/q_k$ is the joint revolute variable. $J_a \mathrm{a}\mathrm{n}\mathrm{d} J_g$ represent the moment of inertia of the motor and the gear of the model. $r_k$ is the gear ratio at the $k^{th}$ joint, and $B_m$ is the damping effect of the gear. $d_{i,j}$ represents the entry of the inertial matrix of the robot manipulator at the $i^{th}$ row and $j^{th}$ column. $c_{i,j,k}$ are Christoffel symbols and for a fixed $k$, with $c_{i,j,k}=c_{j,i,k}$. ${\mathsf{\Phi }}_k$ is the derivative of potential energy V with respect to $k^{th}$ joint variance. $K_m$ is the torque constant in $N-m/amp,$ $K_b$ is the back emf constant, and R is Armature Resistance.

Take $u^i$ as the actuator input to the dynamic system (23) measured at the $i^{th}$ joint from a designed controller. Therefore, $u^i$ equals the right side of Equation (23). That is:

$$V^i={\displaystyle \frac{rR}{K_m}}{u}^i$$

(27)

In addition, without derivation, the expression of current in the Laplace domain [31] is:

$$\left(Ls+R\right)I^i\left(s\right)=V^i\left(s\right)-{\displaystyle \frac{K_b}{r}}s{q}^i(s)$$

(28)

$$\mathrm{Then}: I^i\left(s\right)={\displaystyle \frac{1}{\left(Ls+R\right)}}{V}^i\left(s\right)-{\displaystyle \frac{K_b}{r}}{\displaystyle \frac{s}{\left(Ls+R\right)}}{q}^i\left(s\right)$$

(29)

$$\mathrm{Take} \mathrm{the} \mathrm{inverse} \mathrm{Laplace}: I^i\left(t\right)={\displaystyle \frac{1}{L}}{e}^{-{\displaystyle \frac{R}{L}}t}\ast V^i\left(t\right)-{\displaystyle \frac{K_b}{r}}({\displaystyle \frac{1}{L}}-{\displaystyle \frac{R}{L^2}}{e}^{-{\displaystyle \frac{R}{L}}t})\ast q^i\left(t\right)$$

(30)

where L is armature inductance, $K_b$ is the back emf constant, and $\ast$ is convolutional multiplication. Therefore, the instant current in the time domain $I^i\left(t\right)$ is a function of the instant voltage $V^i\left(t\right)$ and the instant angle $\mathrm{q}\left(t\right)$.

Various controller designs, such as the PID controller [32] and Youla controller [33], of 6-DoF revolute robotic manipulators have been well developed in many papers. In this paper, we use a previous Youla controller design [33] with feedback linearization (Figure 9) as the position controller of the robot manipulator that holds the camera.

From Figure 5, the actuator input $u^i$ can be derived from the virtual input $v^i$ so that:

$$u^i=M(q^i\left(t\right))v^i+H(q^i\left(t\right),\dot{q^i}(t))$$

(31)

where M and H are nonlinear functions of $q^i\left(t\right)$ and $\dot{q^i}(t)$, the first derivative of $q^i\left(t\right).$

Without showing the controller development in this paper, the 1-DoF controller transfer function $G_c\left(s\right)$ and 1-DoF nominal plant $G_p\left(s\right)$ with $v^i$ as the virtual actuator input in the feedback linearization are expressed as follows:

$$G_c(s)={\displaystyle \frac{3{{\tau }_{in}}^{2}s+1}{{{\tau }_{in}}^{3}s+3{{\tau }_{in}}^2}}$$

(32)

$$G_p(s)={\displaystyle \frac{1}{s^2}}$$

(33)

where ${\tau }_{in}$ is a constant parameter in the controller design.

Then, the following transfer functions can be calculated as:

$${\displaystyle \frac{v^i(s)}{{q^i}_{final}(s)}}={\displaystyle \frac{G_c}{1+G_c{G}_p}}$$

(34)

$${\displaystyle \frac{q^i(s)}{{q^i}_{final}(s)}}={\displaystyle \frac{G_c{G}_p}{1+G_c{G}_p}}$$

(35)

$${\displaystyle \frac{\dot{q^i}(s)}{{q^i}_{final}(s)}}={\displaystyle \frac{{sG}_c{G}_p}{1+G_c{G}_p}}$$

(36)

By taking inverse Laplace transform and ${q^i}_{final}$ as step input, the following Equations (37)–(39) in the time domain are:

$$v^i\left(t\right)={(q^i}_{final}-{q^i}_{intial})({\displaystyle \frac{1}{{{\tau }_{in}}^4}}{t}^2{e}^{-{\displaystyle \frac{t}{{\tau }_{in}}}}+{\displaystyle \frac{-5}{{{\tau }_{in}}^3}}t{e}^{-{\displaystyle \frac{t}{{\tau }_{in}}}}+{\displaystyle \frac{3}{{{\tau }_{in}}^2}}{e}^{-{\displaystyle \frac{t}{{\tau }_{in}}}})$$

(37)

$$q^i\left(t\right)={(q^i}_{final}-{q^i}_{intial})({\displaystyle \frac{1}{{{\tau }_{in}}^2}}{t}^2{e}^{-{\displaystyle \frac{t}{{\tau }_{in}}}}-{\displaystyle \frac{1}{{\tau }_{in}}}t{e}^{-{\displaystyle \frac{t}{{\tau }_{in}}}}-e^{-{\displaystyle \frac{t}{{\tau }_{in}}}}+1)+{q^i}_{intial}$$

(38)

$$\dot{q^i}(t)={(q^i}_{final}-{q^i}_{intial})({\displaystyle \frac{3}{{{\tau }_{in}}^2}}t{e}^{-{\displaystyle \frac{t}{{\tau }_{in}}}}-{\displaystyle \frac{1}{{{\tau }_{in}}^3}}{t}^2{e}^{-{\displaystyle \frac{t}{{\tau }_{in}}}})$$

(39)

With expressions of $v^i\left(t\right), q^i\left(t\right),$ and $\dot{q^i}(t)$, Equations (27) and (31) show that:

$$V^i(t)=F({q^i}_{intial}, {q^i}_{final},t)$$

(40)

and combine Equations (30) and (40):

$$I^i(t)=G({q^i}_{intial}, {q^i}_{final},t)$$

(41)

where F and G are nonlinear functions of ${q^i}_{initial}, {q^i}_{final},\mathrm{a}\mathrm{n}\mathrm{d} t$.

It has been shown so far that the estimated energy cost $E(P_{initial},P_{final})$ from location $P_{initial}$ to $P_{final}$ is a function of $q_{initial}$ and $q_{final}$.

$q_{initial}$ and $q_{final}$ are derived from the inverse kinematics process [31] (shown in Appendix B, Equations (A6)–(A13)):

$$q_{initial}=inverkinematics(P_{initial})$$

(42)

$$q_{final}=inverkinematics(P_{final})$$

(43)

$$\mathrm{and} q_{initial}=\left[{q^i}_{initial}|i\in \left(\mathrm{1,2},\mathrm{3,4},\mathrm{5,6}\right)\right], q_{final}=\left[{q^i}_{final}|i\in \left(\mathrm{1,2},\mathrm{3,4},\mathrm{5,6}\right)\right]$$

(44)

The development of Equations (37) to (39) assumes that the target angle of each joint ${q^i}_{final}$ is a step input. A more realistic assumption is to set ${q^i}_{final}$ as a delayed input.

$${q^i}_{final}\left(t\right)=(1-e^{-{\displaystyle \frac{t}{{\tau }_{delay}}}}){q^i}_{final}$$

(45)

where ${\tau }_{delay}$ is a time constant that measures the time delay of the target in the real positioning control. ${\tau }_{delay}\ge 0$, and when ${\tau }_{delay}=0$, it indicates no delay exists in the input.

The Laplace form of (45) is:

$${q^i}_{final}\left(s\right)=({\displaystyle \frac{1}{s}}-{\displaystyle \frac{1}{s+\frac{1}{{\tau }_{delay}}}}){q^i}_{final}$$

(46)

With the new expression of ${q^i}_{final}\left(s\right)$, Equations (37) to (39) can be developed as:

$$v^i\left(t\right)={(q^i}_{final}-{q^i}_{intial})({\displaystyle \frac{A_v}{{{\tau }_{in}}^3}}{t}^2{e}^{-{\displaystyle \frac{t}{{\tau }_{in}}}}+B_v{\displaystyle \frac{B_v}{{{\tau }_{in}}^2}}t{e}^{-{\displaystyle \frac{t}{{\tau }_{in}}}}+{\displaystyle \frac{C_v}{{\tau }_{in}}}{e}^{-{\displaystyle \frac{t}{{\tau }_{in}}}}+{\displaystyle \frac{D_v}{{\tau }_{delay}}}{e}^{-{\displaystyle \frac{t}{{\tau }_{delay}}}})$$

(47)

$$\begin{gathered} A_v={\displaystyle \frac{1}{{\tau }_{in}}}-{\displaystyle \frac{-{\tau }_{delay}}{{\tau }_{in}({\tau }_{in}-{\tau }_{delay})}}\hspace{1em}\hspace{1em}{B}_v={\displaystyle \frac{-5}{{\tau }_{in}}}-{\displaystyle \frac{7{\tau }_{in}{\tau }_{delay}-5{{\tau }_{delay}}^2}{{\tau }_{in}{({\tau }_{in}-{\tau }_{delay})}^2}} \\ {C}_v={\displaystyle \frac{3}{{\tau }_{in}}}-{\displaystyle \frac{{\tau }_{delay}(-8{{\tau }_{in}}^2+9{\tau }_{in}{\tau }_{delay}-3{{\tau }_{delay}}^2)}{{{\tau }_{in}({\tau }_{in}-{\tau }_{delay})}^3}}\hspace{1em}\hspace{1em}{D}_v=-{\displaystyle \frac{3{\tau }_{in}{\tau }_{delay}-{{\tau }_{delay}}^2}{{({\tau }_{in}-{\tau }_{delay})}^3}} \end{gathered}$$

$$q^i\left(t\right)={(q^i}_{final}-{q^i}_{intial})({\displaystyle \frac{A_Q}{{{\tau }_{in}}^3}}{t}^2{e}^{-{\displaystyle \frac{t}{{\tau }_{in}}}}+{\displaystyle \frac{B_Q}{{{\tau }_{in}}^2}}t{e}^{-{\displaystyle \frac{t}{{\tau }_{in}}}}+{\displaystyle \frac{C_Q}{{\tau }_{in}}}{e}^{-{\displaystyle \frac{t}{{\tau }_{in}}}}+{\displaystyle \frac{D_Q}{{\tau }_{delay}}}{e}^{-{\displaystyle \frac{t}{{\tau }_{delay}}}}+1)+{q^i}_{intial}$$

(48)

$$\begin{gathered} A_Q={\tau }_{in}+{\displaystyle \frac{{\tau }_{in}{\tau }_{delay}}{({\tau }_{in}-{\tau }_{delay})}}\hspace{1em}\hspace{1em}{B}_Q=-{\tau }_{in}-{\displaystyle \frac{3{{\tau }_{in}}^2{\tau }_{delay}-{\tau }_{in}{{\tau }_{delay}}^2}{{({\tau }_{in}-{\tau }_{delay})}^2}} \\ {C}_Q=-{\tau }_{in}-{\displaystyle \frac{-3{{\tau }_{in}}^2{{\tau }_{delay}}^2+{{{\tau }_{in}\tau }_{delay}}^3}{{({\tau }_{in}-{\tau }_{delay})}^3}}\hspace{1em}\hspace{1em}{D}_Q=-{\displaystyle \frac{3{\tau }_{in}{{\tau }_{delay}}^3-{{\tau }_{delay}}^4}{{({\tau }_{in}-{\tau }_{delay})}^3}} \end{gathered}$$

$$\dot{q^i}(t)={(q^i}_{final}-{q^i}_{intial})({\displaystyle \frac{A_{\dot{Q}}}{{{\tau }_{in}}^4}}{t}^2{e}^{-{\displaystyle \frac{t}{{\tau }_{in}}}}+{\displaystyle \frac{B_{\dot{Q}}}{{{\tau }_{in}}^3}}t{e}^{-{\displaystyle \frac{t}{{\tau }_{in}}}}+{\displaystyle \frac{C_{\dot{Q}}}{{{\tau }_{in}}^2}}{e}^{-{\displaystyle \frac{t}{{\tau }_{in}}}}+{\displaystyle \frac{D_{\dot{Q}}}{{{\tau }_{delay}}^2}}{e}^{-{\displaystyle \frac{t}{{\tau }_{delay}}}})$$

(49)

$$\begin{gathered} A_{\dot{Q}}= -A_Q\hspace{1em}\hspace{1em}{B}_{\dot{Q}}=2A_Q-B_Q \\ {C}_{\dot{Q}}=B_Q-C_Q\hspace{1em}\hspace{1em}{D}_{\dot{Q}}=-D_Q \end{gathered}$$

A simulation scenario is set up to calculate how estimated energy cost changes with varying ${\tau }_{delay}.$ Set $P_{initial}=[-0.30, 0.05, 1.20]$, and $P_{final}=[-0.45, 0.45, 1.20]$ and use Equations (42)–(44) and (47)–(49). Figure 10 shows the response of one angle $q^1\left(t\right)$ with varying ${\tau }_{delay}$, and

Table 1. Estimated energy cost and settling time with varying delay constant.

${\mathit{\tau }}_{\mathit{d}\mathit{e}\mathit{l}\mathit{a}\mathit{y}}$ [s]	$\mathit{E}({\mathit{P}}_{\mathit{i}\mathit{n}\mathit{i}\mathit{t}\mathit{i}\mathit{a}\mathit{l}},{\mathit{P}}_{\mathit{f}\mathit{i}\mathit{n}\mathit{a}\mathit{l}})$ [ws]	${\mathit{t}}_{\mathit{s}}$ [s]
0	10.895	0.07
0.2	0.102	0.73
0.4	0.030	1.42
0.6	0.016	2.11
0.8	0.011	2.80
1	0.009	3.49

Note: $E(P_{initial},P_{final})$ is measured as [watt-seconds].

presents the estimated energy cost $E(P_{initial},P_{final})$ and settling time $t_s$ with varying ${\tau }_{delay}$.

Table 1 shows a tradeoff between the estimated energy cost and settling time; reducing the energy cost of moving the camera inevitably increases the response time. This finding matches the results in Figure 6.

Systematic delays are inevitable in controlled system design. Delays are incurred in many sources, such as the time required for sensors to detect and process changes, for actuator responses to control signals, and for controllers to process and compute signals. In a real manufacturing environment, larger delays cause slower motion of the camera when searching the area, but a slower response results in less energy consumption from actuators in the manipulator.

If the delay of input is given or can be measured, the estimated energy cost can be calculated through the process in this section. However, if the value is unknown, the delay constant must be chosen and decided based on the following criteria:

• Select small τ_delay for a conservative algorithm that searches a small area but ensures it ends up with the minimum that has been explored;
• Select large τ_delay for an aggressive algorithm that searches a large area but risks not ending up with the minimum that has been explored.

In this section, the estimated energy cost function is well developed. However, the estimation function is developed from an ideal camera movement. The accuracy of the estimation can be negatively influenced by some unmodeled uncertainties, such as the backlash from gears in robot manipulators, unmodeled compliance components from joint vibration, etc. To tackle the problem, we can derive an online updated model of the estimation function by comparing estimated voltage and current and real-time measurements. The parameter values of motors and gears used in the simulations are summarized in Appendix A,

Table A4. Specification table of motors and gears.

Parameters	Values
DC Motor
Armature Resistance: $R$	0.03 $\mathsf{\Omega }$
Armature Inductance: $L$	0.1 mH
Back emf Constant: $K_b$	7 mv/rpm
Torque Constant: $K_m$	0.0674 N/A
Armature Moment of Inertia: $J_a$	0.09847 kg${\mathrm{m}}^2$
Gear
Gear Ratio: $r$	200:1
Moment of Inertia: $J_g$	0.05 kg${\mathrm{m}}^2$
Damping Ratio: $B_m$	0.06

.

5. Simulation Setup

The developed algorithm was simulated on an application in automated manufacturing (Figure 11). This is a multi-robot system composed of a visual system and a tool manipulation system. In the visual system, a camera is mounted on an elbow robot arm, while a tool is held by the end-effector of the robot manipulator arm. The goal of the visual system is to provide a precise estimation of the tool pose so that the tool manipulator can control the pose with guidance from the visual system. The first step of the manufacturing process is to move the camera in the space and search for the best location so that the number of images required for averaging is minimized to reduce the image noises within an acceptable limit. Then, the camera is fixed and provides continuous vision data to the tool manipulator. Therefore, the control and operation of the two manipulators are asynchronous. The model of both robot manipulators is ABB IRB 4600 (ABB, Zurich, Switzerland) [34], and the model of the stereo camera we chose in this project is Zed 2 (Stereolabs, San Francisco, CA, USA) [35].

We first define the camera’s operational space in this application and then provide the simulation results of a made-up scenario in the next.

In the above discussion, a camera operational space is gridded with nodes, and each node is a potential candidate for the camera’s optimal position. The camera operational space is defined by the three geometric constraints below:

• The camera can only be allowed to locations where fiducial markers that are attached to the tool are recognizable on the image frame. Therefore, the fiducial markers are in the angle of view of the camera, and the distance between the markers and the camera center is within a threshold;
• The camera can only be allowed to locations within the reach of the robot arm;
• The camera can only be allowed in locations where the visual system and the tool manipulation system do not physically interfere with each other.

From the specifications of the stereo camera and the robot arm (Appendix A,

Table A1. Specification table of ABB IRB 4600 45/2.05 model (dimensions).

Parameters	Values
Length of Link 1: $L_1$	495 mm
Length of Link 2: $L_2$	900 mm
Length of Link 3: $L_3$	175 mm
Length of Link 3: $L_4$	960 mm
Length of Link 1 Offset: $a_1$	175 mm
Length of Spherical Wrist: $L_t$	135 mm
Tool Length (Screwdriver): ${\overline{PJ}}_{tool}$	127 mm

and

Table A3. Specification table of stereo camera Zed 2.

Parameters	Values
Focus Length: f	2.8 mm
Baseline: B	120 mm
Weight: W	170 g
Depth Range:	0.5–25 m
Diagonal Sensor Size:	6 mm
Sensor Format:	16:9
Sensor Size: W × H	5.23 mm × 2.94 mm
Angle of View in Width: $\alpha$	$86.09°$
Angle of View in Height: $\beta$	55.35$°$

), the geometry and dimensions of the camera operational space are analyzed in the following part of the subsections.

5.1. Reachable and Dexterous Workspace of Two-Hybrid Systems

In the spherical wrist model of the robot arm, three rotational axes represent the pitch, roll, and yaw of the end-effector independently. As shown in Figure 12, these three axes intersect at point $H$. Point $P$ is the location of the end-effector, and it is the place where the camera or tool is mounted. For the elbow manipulator mounted with the stereo camera, the camera is placed so that its optical center line coincides with axis C. The location of camera optical center C is determined, and a good estimation of it can be found in this paper [36]. For now, assume the optical center is in the middle of $\overline{PJ}$ (set $\overline{PC}=17 \mathrm{m}\mathrm{m}$). For the elbow manipulator mounted with the tool, the screwdriver is attached at point $P$. Point J indicates the far end of the object (tool or camera) attached to the end-effector. Point H is kept stationary no matter which axis rotates. Therefore, the task of positioning and rotation is decoupled in the spherical wrist robot model.

The idea of reachable and dexterous workspaces of an ideal elbow manipulator has been introduced in Ref. [37]. An ideal elbow manipulator is a manipulator whose angles of rotation are free to move in the whole operational range [0, 2$\pi$]. However, a realistic elbow manipulator is limited to moving its joints within certain ranges of angle.

The workspace of an ideal elbow manipulator (Figure 13) is a sphere centered at joint 1 of the manipulator, denoted as point $o$. The reachable workspace concerning the center o of a manipulator is the aggregate of all possible locations of point $J$ attached to the end-effector and denoted as $W_o(J)$. The dexterous workspace to the center o of a manipulator is the aggregate of all possible locations that point $J$ can reach all possible orientations of the end-effector and is denoted as $W_o^d(J)$. The reachable workspace of $H$ is denoted as $W_o(H)$.

It can be shown in the figure:

$$W_o^d\left(J\right) \subseteq W_o(H) \subseteq W_o(J).$$

(50)

For an ideal elbow manipulator, the radiuses of workspaces are expressed below:

$$\mathrm{Radius} \mathrm{of} W_o\left(H\right)=L_2+\sqrt{{L_3}^2+{L_4}^2}$$

(51)

$$\mathrm{Radius} \mathrm{of} W_o(J)=L_2+\sqrt{{L_3}^2+{L_4}^2}+L_t+\overline{PJ}$$

(52)

$${\mathrm{Radius} \mathrm{of}W}_o^d\left(J\right)=L_2+\sqrt{{L_3}^2+{L_4}^2}-L_t-\overline{PJ}$$

(53)

where $L_2,$ $L_3, \mathrm{a}\mathrm{n}\mathrm{d} L_4$ are link lengths of the manipulator. $L_t$ is the length of the end-effector, and $\overline{PJ}$ is the length of the object that mounts on the end-effector.

As the camera and the tool must be able to rotate in all 3 DoFs when they are at the target position, the dexterous workspace is used as the working space of the visual system and the tool manipulation system. The dexterous workspace of the optical center is used for the visual system, and the dexterous workspace of the far endpoint of the tool is used for the tool manipulator system.

$$\mathrm{Radius} \mathrm{of} W_o^d\left(Visual\right)=r_V=L_2+\sqrt{{L_3}^2+{L_4}^2}-L_t-{\overline{PC}}_{camera} =1.716 \mathrm{m}$$

(54)

(From Specification, Appendix A, Table A1.)

$$\mathrm{Radius} \mathrm{of} W_o^d\left(Tool\right)=r_T=L_2+\sqrt{{L_3}^2+{L_4}^2}-L_t-{\overline{PJ}}_{tool} =1.606 \mathrm{m}$$

(55)

(From Specification, Appendix A, Table A1.)

Figure 14 shows the workspace setup for the hybrid system. The dexterous space of the robot arm with the camera is a sphere with a radius of $r_V$, and its sphere center is $O_V$, and the dexterous space of the robot arm with tools is a sphere with a radius $r_T$, and the center of this sphere is $O_T.$ Two systems intersect with the ground at an angle of ${\theta }_1$ and ${\theta }_2$.

$${\theta }_1=\arcsin \left({\displaystyle \frac{L_1}{r_V}}\right)=16.77°$$

(56)

$${\theta }_2=\arcsin \left({\displaystyle \frac{L_1}{r_T}}\right)=17.95°$$

(57)

To avoid interference between the visual and the tool manipulation systems (as the third geometric constraint of the operational space), the reachable workspaces (the maximum reach) of the two systems should have no overlap. The reachable workspaces of the two systems are calculated from Equation (51):

$$\mathrm{Radius} \mathrm{of} W_o\left(Visual\right)=r_{Vr}=L_2+\sqrt{{L_3}^2+{L_4}^2}+L_t+{\overline{PJ}}_{camera}=2.044 \mathrm{m}$$

(58)

(From Specification, Appendix A, Table A1.)

$$\mathrm{Radius} \mathrm{of} W_o\left(Tool\right)=r_{Tr}=L_2+\sqrt{{L_3}^2+{L_4}^2}+L_t+{\overline{PJ}}_{tool} =2.138 \mathrm{m}$$

(59)

(From Specification, Appendix A, Table A1.)

Let $L_{VT}$ be the distance between $O_V$ and $O_T$. The following relationship must be satisfied:

$$L_{VT}\ge r_{Vr}+r_{Tr}=4.182 \mathrm{m}$$

(60)

The visual system is able to detect the tool when it gets close to the target pose (just above the bolt). Also, reference points are put near the tool’s target pose. Assume a marker is put where the bolt is, and this marker represents an approximated location of all interest points and reference points. Therefore, the marker should be within the dexterous space of the tool manipulation system. Set up a coordinate system with ${\overline{O}}_V$(the projection of $O_V$ on the ground with $a_1$ offset) as the origin. The projection of $O_T$ on the ground with $a_1$ offset is ${\overline{O}}_T.$ Assume the marker is placed on the line $\overline{{\overline{O}}_V{\overline{O}}_T}$. The distance m from marker M to ${\overline{O}}_V$ should be:

$$L_M\ge {a_1+L}_{VT}-r_T\cos \left({\theta }_2\right)\ge 2.829 \mathrm{m}$$

(61)

5.2. Detectable Space for the Stereo Camera

From Appendix A, Table A3, the angle of view in width and height for both lenses in the stereo camera is $\alpha$ = $86.05°$, and $\beta =55.35°$. The detectable space for each lens can be modeled as the inner area of a cone with angles $\alpha$ and $\beta .$ And the overlapped space is the detectable space for the stereo camera. The model is shown below (Figure 15). Point C is the center of the baseline whose length is denoted as $b$ or the center of the camera system. The overlap area is also a cone with angles of vertex $\alpha$ and $\beta$, with offset $d$ from the baseline. Point Q is the vertex of the cone.

$$d={\displaystyle \frac{b/2}{\mathrm{t}\mathrm{a}\mathrm{n}(\raisebox{1ex}{$\alpha $}\!\left/ \!\raisebox{-1ex}{$2$}\right.)}}=0.064 \mathrm{m}$$

(62)

There is an upper bound of the distance between the object to the camera center; if the object is too far away from the camera center, the dimensions of the projected images are too small to be measured. Suppose to have a clear image of the fiducial markers (circle shapes), it is required that the diameter of the projected image should take at least 5-pixel numbers in the image frame:

$$N_H \ge 5 \mathrm{resolution}$$

(63)

where $N_H$ is the pixel number in the image frame.

Select the Zed camera’s mode so that its resolution is 1920 × 1082, and the image sensor size is 5.23 mm × 2.94 mm. Then, the number resolution in unit length is:

$$n=367 \mathrm{resolution}/\mathrm{mm}$$

(64)

Then, the range of the markers’ diameter on the image frame is:

$$d_H={\displaystyle \frac{N_H}{n}}\ge 0.0136 \mathrm{mm}$$

(65)

Also, assume in the inertial frame, the diameter of the attached fiducial marks is:

$$D_H=12 \mathrm{mm}$$

(66)

Then, from the pinhole model of the camera, the range of distance between the markers and camera center Z is:

$$Z={\displaystyle \frac{f_u{D}_H}{d_H}}\le 2.47 \mathrm{m}=Z_{max}$$

(67)

where $Z_{max}$ is the maximum depth of the camera to detect the markers. This parameter defines the height of the cone in Figure 15. The detectable space (abiding by the second geometric constraint) forms an elliptic cone with different angles of the vertex.

5.3. Camera Operational Space Development

As in Figure 16, set the camera lens so that it is always parallel to the face of the marker and the baseline is parallel to the ${\overline{X}}_V$ axis. In this case, the lens is kept facing downward when detecting the marker on the ground. As discussed in the above section, the detectable space of the camera is modeled as a cone. This cone intersects with the horizontal plane and forms an oval. To have the camera detect the marker, the marker must be contained inside the oval.

An inertia coordinate system ${\overline{O}}_V{\overline{X}}_V{\overline{Y}}_V{\overline{Z}}_V$ is set up with its origin seated at the bottom center of the visual system. Make point $C$ the location of the camera center, and its coordinates in the inertial frame are ($X_C^V$, $Y_C^V$, $Z_C^V$). $Q$ is the vertex of the detectable elliptic cone, and $Q’$ is the projection of $Q$ on the horizontal plane. $M$ is the marker.

The elliptic cone of the camera’s detectable space always intersects with the horizontal plane. Consider an extreme case when point $C$ is located at the apex of the visual system workspace, as shown in Figure 16. Then, the $Z_C^V$ coordinate of $C$ at the inertial frame is:

$$Z_C^V\left\{max\right\}=L_1+r_V=2.211 \mathrm{m} \le Z_{max}$$

(68)

where $Z_{max}$, defined above, is the maximum depth of the camera to detect the markers. Therefore, the cone intersects with the horizontal plane when the camera center $C$ is placed at any location of the workspace’s contour.

From the expression of Cartesian coordinates in the inertia frame:

$$a_{Q\prime }=(Z_C^V-d)tan({\displaystyle \frac{\alpha }{2}}), b_{Q\prime }=(Z_C^V-d)tan({\displaystyle \frac{\beta }{2}})$$

(69)

where $d$ is the same offset as in Equation (62). $a_{Q\prime }$ and $b_{Q\prime }$ are the major radius and the minor radius of the oval. Also, the coordinates of points $Q’$ and $M$ are $(X_C^V,Y_C^V, 0)$ and $(L_M,0, 0)$. $Q’$ is the center of the oval. To ensure the marker $M$ is inside the projected oval, it is required that:

$${\displaystyle \frac{{(X_C^V-L_M)}^2}{{a_{Q\prime }}^2}}+{\displaystyle \frac{{(Y_C^V)}^2}{{b_{Q\prime }}^2}}\le 1$$

(70)

Plug Equation (69) into Equation (70):

$${\displaystyle \frac{{(X_C^V-L_M)}^2}{{tan(\frac{\alpha }{2})}^2}}+{\displaystyle \frac{{(Y_C^V)}^2}{{tan(\frac{\beta }{2})}^2}}\le {(Z_C^V-d)}^2$$

(71)

Inequality (71) is exactly a mathematic expression of all points $C$($X_C^V$, $Y_C^V$, $Z_C^V$) within a cone whose opening is in the positive Z direction with its vertex at ($L_M$,$0,d$) and the opening parameters as $\tan ({\displaystyle \frac{\alpha }{2}})=0.934 \mathrm{and} \tan ({\displaystyle \frac{\beta }{2}})=0.524.$ Therefore, this inequality defines the second geometric constraint of the camera center. It forms an elliptical cone with its vertex located in Figure 17 at point E, which is the offset point of the marker $M$ from the ground. As $C$($X_C^V$, $Y_C^V$, $Z_C^V$) should also be within the sphere of the workspace (abiding by the first geometric constraint), the camera operational space is presented as the overlap between the cone and the sphere (the shaded area in Figure 17).

From Figure 17, the marker should not be too far from the visual system, otherwise there is no overlap region between the cone and the sphere. The largest value of $L_M$, which is the distance from the marker to the coordinate center, occurs when the cone is tangent to the sphere, as shown in Figure 18. To find the limit of $L_M$, draw a horizontal line across point $O_V$ so that it intersects the surface of the cone at point A and the center line of the cone at point B, as shown in Figure 18. Let $\overline{O_{V}A}$ and $\overline{AB}$ be the line segment lengths.

Therefore, from the geometry, the upper limit of $L_M$ is:

$$L_M<\overline{O_{V}A}+\overline{AB}+a_1={\displaystyle \frac{r_V}{\cos \left(\frac{\alpha }{2}\right)}}+\left(L_1-d\right)\tan \left({\displaystyle \frac{\alpha }{2}}\right)+a_1$$

(72)

Combining with the lower bound of $L_M$ in Equation (61), the marker M should be placed in the system so that:

$$2.829 \mathrm{m}\le L_{VT}-r_T\cos \left({\theta }_2\right)\le L_M<{\displaystyle \frac{r_V}{\cos \left(\frac{\alpha }{2}\right)}}+(L_1-d)\tan \left({\displaystyle \frac{\alpha }{2}}\right)+a_1=2.925 \mathrm{m}$$

(73)

5.4. Mathematic Expression for Node Coordinates within the Camera Operational Space

From Equation (73), the closer the marker’s distance $L_M$ to its lower bound, the larger the operational space. In general, a larger operational space is preferred because the camera has a larger space for searching the optimal location. Set $L_M$ to equal its lower bound:

$$L_M=2.83 \mathrm{m}$$

(74)

Get the cross-section area at the ${\overline{X}}_V-{\overline{Z}}_V$ plane (Figure 19).

The function of the semi-circle and the line EFG is:

$$({{\overline{Z}}_V-L_1)}^2+{\left({\overline{X}}_V-a_1\right)}^2 ={r_V}^2$$

(75)

$${\overline{X}}_V=-tan({\displaystyle \frac{\alpha }{2}})({\overline{Z}}_V-d)+L_M$$

(76)

Combine Equations (75) and (76) and solve:

$$\begin{gathered} (1+{tan({\displaystyle \frac{\alpha }{2}})}^2){{\overline{Z}}_V}^2-2({tan({\displaystyle \frac{\alpha }{2}})}^{2}d+tan\left({\displaystyle \frac{\alpha }{2}}\right)(L_M-a_1)+L_1){\overline{Z}}_V+{\tan \left({\displaystyle \frac{\alpha }{2}}\right)}^2{d}^2+ \\ 2\tan \left({\displaystyle \frac{\alpha }{2}}\right)(L_M-a_1)d+{(L_M-a_1)}^2-{r_V}^2+{L_1}^2=0 \end{gathered}$$

(77)

Solve Equation (77) by plugging numbers:

$$1.872 {{\overline{Z}}_V}^2-6.061{\overline{Z}}_V+4.670=0$$

(78)

$$\mathrm{Solve}: {{\overline{Z}}_V}_{max},{{\overline{Z}}_V}_{min}=1.972 \mathrm{m},1.265 \mathrm{m}$$

(79)

${{\overline{Z}}_V}_{max}$ and ${{\overline{Z}}_V}_{min}$ are the largest and smallest ${\overline{Z}}_V$ coordinates of the camera center $C$ that is within the camera operation space.

Take a value of $Z_C^V$ (${\overline{Z}}_V$ coordinates of the camera center $C$) in the range (${{\overline{Z}}_V}_{min}$,${{\overline{Z}}_V}_{max}$) = ($1.265 \mathrm{m}$,$1.972 \mathrm{m}$). Set a plane ${\overline{Z}}_V=Z_C^V$, and that plane intersects the camera operational space and forms a shade area, which is the overlap between the circle and the oval, as shown in Figure 14. The inequality governing points $C$ within the sphere at a specific $Z_C^V$ is:

$$({{\overline{X}}_V-a_1)}^2+{{\overline{Y}}_V}^2\le {r_V}^2-({Z_C^V-L_1)}^2$$

(80)

The inequality governing points $C$ within the cone at $Z_C^V$ is:

$${\displaystyle \frac{{({\overline{X}}_V-L_M)}^2}{{\tan (\frac{\alpha }{2})}^2}}+{\displaystyle \frac{{({\overline{Y}}_V)}^2}{{\tan (\frac{\beta }{2})}^2}}\le {(Z_C^V-\mathrm{d})}^2$$

(81)

By plotting those inequalities in Figure 20, the shaded area is where $X_C^V$ and $Y_C^V$ (${\overline{X}}_V$ and ${\overline{Y}}_V$ coordinates of the camera center $C$) should be located within. The intersection of the two curves occurs when equality holds for (80) and (81):

$$({{\overline{X}}_V-a_1)}^2+{{\overline{Y}}_V}^2={r_V}^2-({Z_C^V-L_1)}^2$$

(82)

$${\displaystyle \frac{{({\overline{X}}_V-L_M)}^2}{{\tan (\frac{\alpha }{2})}^2}}+{\displaystyle \frac{{({\overline{Y}}_V)}^2}{{\tan (\frac{\beta }{2})}^2}}={(Z_C^V-d)}^2$$

(83)

Solve by plugging numbers:

$$({{\overline{Y}}_V}_{max}, {{\overline{Y}}_V}_{min})=\pm 0.5\ast \sqrt{3.44\ast {Z_C^V}^2-2.0241\ast Z_C^V-29.802+2.451\ast \sqrt{-29.475\ast {Z_C^V}^2+23.707\ast Z_C^V+137.28}}$$

(84)

${{\overline{Y}}_V}_{max}$ and ${{\overline{Y}}_V}_{min}$ are the largest and the smallest ${\overline{Y}}_V$ coordinates of the camera center $C$ in the camera operation space with specific $Z_C^V$.

Take $Y_C^V$ in the range (${{\overline{Y}}_V}_{min}$, ${{\overline{Y}}_V}_{max}$), then for specific $X_C^V$:

$${{\overline{X}}_V}_{max}=\sqrt{{r_V}^2-({Z_C^V-L_1)}^2-{Y_C^V}^2}+a_1$$

(85)

$${{\overline{X}}_V}_{min}=L_M-\sqrt{{{(Z}_C^V-d)}^2{\tan ({\displaystyle \frac{\alpha }{2}}) }^2-{Y_C^V}^2{\displaystyle \frac{{\tan (\frac{\alpha }{2})}^2}{{\tan (\frac{\beta }{2})}^2}}}$$

(86)

${{\overline{X}}_V}_{max}$ and ${{\overline{X}}_V}_{min}$ are the largest and the smallest ${\overline{X}}_V$ coordinates of the camera center $C$ in the camera operation space with specific $Z_C^V$ and $Y_C^V$.

5.5. Numerical Solution of the Ideal Camera Operation Space

With known parameters from the camera and robot arm specifications, all possible locations of the camera center $C$($X_C^V$, $Y_C^V$, $Z_C^V$) inside the camera operational space can be derived from the following steps:

• Calculate ${{\overline{Z}}_V}_{max}$ and ${{\overline{Z}}_V}_{min}$ by Equation (77) and mesh grid $Z_C^V$ in (${{\overline{Z}}_V}_{min}$,${{\overline{Z}}_V}_{max}$);
• Take the mesh value $Z_C^V$ and calculate ${{\overline{Y}}_V}_{max}$ and ${{\overline{Y}}_V}_{min}$ by Equations (82) and (83) and mesh grid $Y_C^V$ in (${{\overline{Y}}_V}_{min}$,${{\overline{Y}}_V}_{max}$);
• Take the mesh $Y_C^V$ and calculate ${{\overline{X}}_V}_{max}$ and ${{\overline{X}}_V}_{min}$ by Equations (85) and (86) and mesh grid $X_C^V$ in (${{\overline{X}}_V}_{min}$,${{\overline{X}}_V}_{max}$).

Each above computed $C$($X_C^V$, $Y_C^V$, $Z_C^V$) is inside the camera operation space, and these three steps completely account for all the geometric constraints defined above. The number of mesh points (nodes available for searching) depends on the mesh grid size.

5.6. The Camera Operational Space from the Realistic Robot Manipulator

The above procedures of finding the camera operational space assume the use of ideal robotic manipulators, whose joints are free to move from $[0, 2\pi ]$. However, for realistic robotic manipulators, their revolutionary movement is limited within certain ranges of angles, and the ranges of motion in every joint of the specific robot manipulator ABB IRB 4600 are listed in Appendix A,

Table A2. Specification table of ABB IRB 4600 45/2.05 model (axis working range).

Axis Movement	Working Range
Axis 1 Rotation	+180$°$ to −180$°$
Axis 2 Arm	+150$°$ to −90$°$
Axis 3 Arm	+75$°$ to −180$°$
Axis 4 Wrist	+400$°$ to −400$°$
Axis 5 Bend	+120$°$ to −125$°$
Axis 6 Turn	+400$°$ to −400$°$

.

The geometric shape of the operational space for a realistic robotic manipulator is usually irregular (unlike an ideal manipulator whose operational space is the combination of a sphere and a cone), and the mathematical methods of directly finding the nodes within the space are computationally expensive. A faster way of generating realistic camera operational space is to decrease the ideal operational space by taking out nodes that are out of the operational ranges. These outlier nodes can be found in the inverse kinematic model of the robotic manipulator. Figure 21 shows the decreased nodes from the ideal camera operational space to the realistic space.

6. Simulation Results

A simulation in MATLAB was used to validate the proposed algorithm. In the simulation, the environment was set up with two minimums in the space (only one is the global minimum). $K_{est}$ and $K_{sd}$ are tunable parameters and set as $K_{est}=5, \mathrm{a}\mathrm{n}\mathrm{d} K_{sd}=50 \left({\mathrm{m}}^{-1}\right)$. The initial available energy bound is set to be $E_{bound}=$ 12 (watt-second (ws) or Joule). Figure 22 shows the sequences of nodes being explored in iteration steps by setting different energy thresholds $E_T$.

By trials of different simulation runs, it can be concluded that:

• When the E_T is large (in this scenario, 3 ws < E_T ≤ 12 ws), the algorithm is conservative, so it has only searched a small area that excludes the global minimum;
• When the E_T is in the mid-range (in this scenario, 0.5 ws < E_T ≤ 3 ws), the algorithm drives the camera to the global minimum.

When the E_T is small (in this scenario, ${0 \mathrm{ws}\le E}_T<0.5 \mathrm{ws}),$ the algorithm is aggressive. Although it has searched a large number of areas that include the global minimum, it does not have enough energy to move the camera back to the best location explored in the previous stages.

The two tunable parameters $K_{est}$ and $K_{sd}$ are selected randomly in the simulation. These parameters are used in the models to estimate the noise spatial distribution in the environment. As stated in Section 4, the positive parameter $K_{est}$ indicates that the weight of an explored node in the estimation function is based on its Euclidean distance from the node to be estimated. The larger $K_{est}$ in Equation (4) implies that Euclidean distance has a smaller influence on the weight. In other words, with increasing $K_{est}$, the estimation result of an unexplored node is more different than that of an explored node nearby. As a result, the algorithm is more likely to search nodes around the local minimum. Similarly, $K_{sd}$ is proportional to the magnitude of the standard deviation in the stochastic modified process. A smaller $K_{sd}$ causes a smaller standard deviation and thus a more deterministic evaluation of an unexplored node, which discourages the algorithm to explore areas that are away from the local minimum.

Therefore, by increasing $K_{est}$ or decreasing $K_{sd}$, we should expect that the algorithm has a higher tendency to exploit over explore. In this paper, we developed a parameter to present the degree of exploration in a simulation by measuring the average distance between the new explored node to the nearest previously explored node.

$${Avg\_Dis}_{new}= {\displaystyle \frac{\sum _{i=1}^M{\mathrm{m}\mathrm{i}\mathrm{n}}_{j\in S_{index-exp}}(\overline{ED}\left(P_j^{exp},P_i^{new}\right))}{M}}$$

(87)

where $M$ is the total number of iterations in a simulation, and $P_i^{new}$ is the position of the new explored node at the $i^{th}$ iteration. $S_{index-exp}$ is the index of explored node set as defined in Equation (6) and $\overline{ED}$ is the Euclidean distance between two nodes as defined in Equation (7).

In a simulation, a large value of the parameter ${Avg\_Dis}_{new}$ means new explored nodes in each search iteration are generally far away from previously explored nodes, and the algorithm succeeds in exploring a large area. A sensitivity test was performed by varying $K_{est}$ and $K_{sd}$ to show how these two parameters affect the searching process in the same simulation environment.

Table 2. Sensitivity analysis with varying K_est. (K_sd = 50 (m⁻¹)).

${\mathit{K}}_{\mathit{e}\mathit{s}\mathit{t}}$	Iteration Numbers	${\mathit{A}\mathit{v}\mathit{g}\_\mathit{D}\mathit{i}\mathit{s}}_{\mathit{n}\mathit{e}\mathit{w}}$ (m)	Stop at Global Minimum
90	93	0.059	NO
70	83	0.066	NO
50	67	0.073	YES
30	46	0.085	YES
10	38	0.097	YES
5	47	0.102	YES
1	20	0.131	NO

presents the sensitivity analysis of $K_{est}$ by keeping a constant $K_{sd}$ = $50 \left({\mathrm{m}}^{-1}\right),$ while

Table 3. Sensitivity Analysis with Varying K_sd. (K_est = 50).

${\mathit{K}}_{\mathit{s}\mathit{d}}$ (m−1)	Iteration Numbers	${\mathit{A}\mathit{v}\mathit{g}\_\mathit{D}\mathit{i}\mathit{s}}_{\mathit{n}\mathit{e}\mathit{w}}$ (m)	Stop at Global Minimum
90	36	0.086	NO
70	62	0.079	YES
50	67	0.073	YES
30	84	0.066	YES
10	127	0.060	YES
5	176	0.046	YES
1	264	0.037	NO

presents the sensitivity analysis of $K_{sd}$ by keeping a constant $K_{est}$ = $50$(No unit). All analysis tests are based on the same scenario in Figure 22 with $E_{bound}=$ 20 (ws) and $E_T$ = 2 (ws).

Table 2 and Table 3 show that by decreasing $K_{est}$ or increasing $K_{sd}$, the algorithm explores more areas, as indicated by the parameter ${Avg\_Dis}_{new}$. The iteration numbers reduce as more areas have been explored because fewer nodes can be reached when the total energy for moving is bounded. Those conclusions can be derived from the analysis. On one hand, the algorithm is too conservative and searches few areas when $K_{est}$ is too large or $K_{sd}$ is too small. On the other hand, the algorithm is too aggressive and skips exploiting when $K_{est}$ is too small or $K_{sd}$ is too large. Both cases will make the node not end up with the global minimum. Therefore, a moderate combination of $K_{est}$ and $K_{sd}$ is preferred. Future research can focus on the development of adaptive algorithms that tune $K_{est}$ and $K_{sd}$ over iterations based on errors between real measurements and estimations.

In addition, the numerical values of $E_T$ were also picked randomly for testing. In real applications, a fixed $E_T$ should be selected before the operation of this algorithm. The selection of $E_T$ is related to the specific application scenario, size of the camera’s operational space, and total energy available $E_{bound}$ at the beginning. However, the general rule of thumb is the following: choose small values of $E_T$ when both $E_{bound}$ and operational space is large to encourage exploring over exploiting; otherwise, choose a large $E_T$.

7. Conclusions

In this article, we developed an algorithm so that a camera can explore the workspace in a manufacturing environment and search for a location so that the image noise is minimized within the space that it can reach. This article also provides a detailed development of the camera’s operational space for a specific application. The results in a virtual environment show the algorithm succeeds in bringing the camera to the optimal (or suboptimal if the optimal one is unreachable) position in this specific scenario. By reducing the processing time of images, this algorithm can be used for various visual servoing applications in high-speed manufacturing.

However, challenges arise in some possible scenarios of real applications. For instance, if environmental factors vary too rapidly across the space, a fine gridding of the space is required to manifest those variations in the estimation function, which increases computational complexity, resulting in poor real-time performances. The same issues exist in cases where the operational space of the camera is too large.

Other limitations include that this algorithm is incapable of differentiating contrast in images at different locations. Low contrast other than image noises occurs as another issue in the visual servoing environment; for instance, low contrast in soft tissues has been shown to have negative effects on the performance of visual servoing controllers in medical applications [38]. In addition, this algorithm does not account for cases when the object is partially obstructed, which is very common in manufacturing environment. Limitations in the camera hardware that cannot account for additive noise are also not addressed in this paper. Those effects cannot be eliminated by any image denoising processes. For example, sensors with a lower dynamic range may produce images where the details are lost in extreme lighting conditions (bright sunlight or deep shadows). Another example is that some lenses may cause vignetting (darkening around the edges) or distortion, which can degrade the perceived image quality.

The simulation results show how the energy threshold $E_T$ affects the behavior of the algorithm. A moderate threshold is suggested so that the algorithm can search for enough amount of area but enables the camera to be driven back to the minimum with the limited energy available. The analysis of $E_T$’s value in different applications is a focus of our future work. This may also include adaptive algorithms that update $K_{est}$ and $K_{sd}$ online and an adaptive function for energy estimation. Also, orientations of the camera are fixed in this paper. In the future, we would like to develop advanced algorithms that provide not only the optimal of positions but also the orientations inside the space.

In summary, this paper initially explores an adaptive algorithm to search for the optimal location in space with respective to image noises for observation in ETH or ETH/EIH cooperative configurations. Although several improvements of this algorithm can be addressed in the future, it has already shown a great potential to be applied as an upgraded feature in some real manufacturing tasks.