*4.4. Standard Automation Rules Results*

For comparison purposes, the threshold is the first classification algorithm, and it consists of statistics metrics that define thresholds based on momentum *Mx* and *My* as decision constraints for the robot movements during the assembly cycle. Values of the thresholds obtained are presented in Table 2.


**Table 2.** Resulting thresholds of the data set. Values in *Nm*.

A testing algorithm that runs the data set into the threshold comparisons was used to define whether the force/torque information corresponds to a certain defined class. The Equations (10) and (11) defined the threshold limits.

#### *4.5. Deep Neural Network Training and Comparisons*

The design of the neural networks was done through iterative testing where the initial configuration was based on empirical estimations. During the iterative design phase, different layer sizes and hyper-parameters were tested. The best result was obtained with the architecture of input layer 12 neurons, 5 hidden layers with unit size of 24, 48, 48, 48, 24 neurons, and, finally an output layer of 8 neurons. All input and hidden layers use an activation function *tanh*. For the output layer, *So f tmax* activation function is selected. The Loss Function is configured as Categorical Cross-entropy, and, as a learning optimizer, the Adam algorithm is selected.

The DNN model is programmed using Python libraries and Tensor-flow Keras [37]. Then, the data set is divided into a 70/30 relationship for training and testing in 180 epochs. The data set contains the force/torque values of the two sensors. It must be considered that every time the DNN model is trained, even with the same data, different results of performance are expected.

The performance of the algorithm for training/testing is shown in Figures 8 and 9. These graphs present the accuracy of the DNN, where the blue line represents the training data and the orange line the testing data. As seen in the one sensor case, the test data fits only up to 0.8 of the value during the 180 epochs, while the training data continues to fit the model. In the case of one sensor, the model implemented in the robot controller will present at least 20% of errors in classification.

On the other hand, the graph that corresponds to the two sensors shows the training and testing data with a high accuracy value. These results show a clear advantage to using two sensors rather than one.

**Figure 8.** Single Sensor data training results 80% Accuracy.

**Figure 9.** Double Sensor data training results 97.4% Accuracy.

Some other machine learning algorithms based on the sci kit-learn [38] were designed to train/test different classification techniques and compare how well they fit the data. A list of tested machine learning algorithms with this data set is presented as follows:


In order to demonstrate the difference between using only one F/T sensor versus two F/T sensors, the classification model uses either 6 features of the peg sensor or the 12 features of both peg and hole sensors.

To show the results of the training/testing of the machine learning models, a comparison of their performance is presented in Table 3. It can be seen that the threshold algorithms based on statistical metrics performs worst. The best model corresponds to the Deep Neural Network followed by the Random Forest. The metrics reported in the table consider the accuracy defined as the number of correct predictions divided by the total number of predictions. The second metric is the F1-Score that represents the harmonic mean between precision and recall metrics.

**Table 3.** Comparison of accuracies and F1-Score on different machine learning algorithms.


#### *4.6. Validating the DNN Models*

The model was tested using extra data sets gathered after the main training data set acquisition. Using the same automated program, these new data sets with fewer contact samples were used as validation for the trained DNN model. The data sets start from 35 samples to got to 240 samples as shown in Table 4.

The algorithm to validate the trained model considers the same values of mean and standard deviation of the train data set. The Equation (12) is used to standardize the validation data sets. Then, the validation data set is used on the python Tensorflow Keras program to evaluate the correct classifications versus the incorrect classifications according to the Exact Match Ratio defined in (17).

Table 4 summarizes the accuracy results for the one sensor and two sensors. As shown in the results, there is a clear advantage to using two sensors rather than one sensor.


**Table 4.** Validation results of DNN Accuracy of one and two sensor.

#### *4.7. Testing on the Dual Arm Robot*

In order to test this methodology in a real scenario of operation with the dual arm robot, the automated recovery program in Figure 5 is executed. Two computers were used to run the experimental setup, one computer ran the automatic sequence and sensor acquisition. The whole controller is programmed in C++ running on ROS using Moveit packages and libraries to perform Cartesian commands on the robot.

The second computer that runs the service node receives the F/T data as an array of twelve elements that is computed and, at the end, gives back the class prediction. This prediction *y*ˆ is used in the assembly sequence to define the next orientation angle or movement in the corresponding direction to find the next contact states of the sequence. Table 5 shows the corresponding orientation angles according to the predicted class.

**Table 5.** Classes and angle of movement.


The last experiment considers the validation of the full assembly cycle. Logic rules used as transition constraints between the steps. The full cycle F/T data is presented in Figure 10 where the different steps and their corresponding F/T pattern can be seen. From free motion, the sequence evolves to the frontal contact state. The second step changes the orientation. The next step searches for lateral contact of the peg into the hole. At the end of the assembly sequence, the insertion phase shows a little deviation that exceeds a force limit value, which triggers the last DNN model. Reaction motions are selected according to the F/T data pattern and the DNN output. As the cycle continues, the patterns are detected and the position adjustment drives the F/T towards minimum values to lead the final assembly.

For these experiments a very slow motion is considered for safe contacts and to avoid any possible damage on the mechanical systems of the robot. Therefore, the time constraints for these experiments are not taken into account.

Finally, in order to compare the differences between the success ratio of assemblies using one sensor and using two sensors with two DNN's as classifiers, the complete assembly experiment is reported in Table 6.

**Table 6.** Final results and comparison of one vs two sensors.


**Figure 10.** Forces and torques during one assembly cycle.

These results show that the whole experiment executed 40 assembly insertion attempts, where the number of sequence steps needed to finish the insertion is counted, after which the average number of steps for the total of assembly attempts is reported as a comparison metric. The Success Ratio, is the average of the completed insertions in the experiment, is also presented.

The graphical representation of the results are presented in Figure 11a, where one can see a better performance in the Success Ratio and in Figure 11b the Average of steps for the two sensor configurations.

**Figure 11.** Experiment results. (**a**) Success Ratio and (**b**) Average of Steps.

#### **5. Conclusions**

We have presented in this paper a methodology to evaluate the effects of using double force/torque sensors on the peg-in-hole assembly and deep neural networks. An assembly strategy was proposed, which was inspired on how a human would solve the positioning error of a peg-in-hole assembly. A discrete event controller designed to run the assembly sequence was designed as it would be programmed in an industrial automated robot cell. The purpose of this strategy is to achieve the assembly task despite the aleatory positioning error of the peg and hole.

A force/torque signal acquisition program was implemented to generate the data set for training machine learning algorithms. The automated data acquisition proved very efficient if different conditions are to be considered. This allows for testing different parameters before the final experiments. It also ensures that the contact states are gathered with the same assembly conditions as it would be in a normal automated cycle.

Off-line learning of the contact states is achieved by using deep neural networks. It was found that the publicly available machine learning libraries are a great evaluation solution to integrate in a ROS-based system. Despite the good results using DNN, we encourage evaluation other machine learning algorithms in the experimental setup. We noticed how the F/T signals are less separable in a certain geometric region presenting more false positives. Further investigation of this F/T signal behavior is needed.

Additionally, it is shown that the success ratio of the assembly process improved by the use of double force/torque sensors and deep neural networks. Although the comparison between different research works is not simple due to different experimental conditions, such as robot manufacturer, developing software and perception systems among others, the findings of this research presents a contribution on the evaluation of dual force torque sensors for assembly tasks.

The results obtained by using one and two force/torque sensors to classify contact states showed an improvement of 30% of Success Rate. The assembly strategy was based on a discrete event controller represented by a Petri-Net sequence diagram. Because this sequence diagram represents an automation program, we consider it feasible to test the methodology on industrial scenarios. However, this work does not take into account the overall assembly time due to safety considerations such as working with slow speed motions to avoid any damage to the robot. For an industrial application, this needs to be investigated.

There are still many conditions that need to be taken into account to implement this strategy in an industrial application. For example, changing shapes of the mating parts, the effects of adjustments such as speed, force limits and so on. These changes would affect the performance of the classifier due to the changes in the F/T behavior.

Future work considers different challenging scenarios such as testing different machine learning algorithms, the evaluation of different assembly parameters such as higher speed movements, higher force/torque limits, shape of the mating parts, the increase of uncertain conditions such as the number of aleatory error positions and so on. Evaluation of dual arm coordination strategies will also be investigated. Moreover, integration of grippers to manipulate the objects will increase positioning uncertainties that also need to be studied.

**Author Contributions:** D.O.-A. contributed to the conceptualization, design, implementation, and python programming as well as the DNN design. I.L.-J. conceptualized the use of the DNN for peg-inhole assembly and wrote-up the first paper draft. B.N.S. and J.F.J.-V. contributed to the experimental design, scikit-learn validation and the revision of the final draft. All authors contributed to writing-up the final paper version and agreed to the published version of the manuscript. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was fully funded through CIDESI and CONACYT.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data presented in this study are openly available in https://data. mendeley.com/datasets/m7g93whs3t/1, doi:10.17632/m7g93whs3t.1. (accessed on 15 June 2021).

**Acknowledgments:** Thanks are sincerely due to Israel Uribe and Sinai Aranda for his valuable technical support. Hugo Gamez for his kind support on reviewing the writing.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Abbreviations**

The following abbreviations are used in this manuscript:


F/T Force and Torque

ROS Robot Operating System
