Learning-Based Planner for Unknown Object Dexterous Manipulation Using ANFIS

Abstract

Dexterous manipulation of unknown objects performed by robots equipped with mechanical hands represents a critical challenge. The difficulties arise from the absence of a precise model of the manipulated objects, unpredictable environments, and limited sensing capabilities of the mechanical hands compared to human hands. This paper introduces a data-driven approach that provides a learning-based planner for dexterous manipulation employing an Adaptive Neuro-Fuzzy Inference System (ANFIS) fed by data obtained from an analytical manipulation planner. ANFIS captures the complex relationships between inputs and optimal manipulation parameters. Moreover, during a training phase, it is able to fine-tune itself on the basis of its experiences. The proposed planner enables a robot to interact with objects of various shapes, sizes, and material properties while providing an adaptive solution for increasing robotic dexterity. The planner is validated in a real-world environment, applying an Allegro anthropomorphic robotic hand. A link to a video of the experiment is provided in the paper.

1. Introduction

Manipulation involves moving or rearranging objects in the environment. This term encompasses various actions, such as grasping and releasing objects, and interaction with the environment and/or objects [1]. Dexterous manipulation especially refers to the ability to achieve different configurations of the manipulated object within the hand workspace [2,3]. Several prior studies on this topic have been conducted [4,5,6,7]. Dexterous manipulation, is carried out basically in two ways, or combinations of them [8]:

• Keeping each finger in contact with the object during the manipulation, i.e., the grasp varies in a continuous way (e.g., [9,10,11]).
• “Finger gaiting“ or “in-hand regrasping“, which sequentially removes finger contact(s) and places the finger(s) at another point on the object surface, i.e., the grasp varies in a discontinuous way (e.g., [9,12,13]).

Numerous recent works focus on the dexterous manipulation of unknown objects, where “unknown object” refers to an object whose shape and other properties are unknown when the manipulation begins [8]. The application of force optimization for the manipulation of unknown objects is presented in [14], and online improvement of the hand movements while manipulating an object is described in [15,16,17]. The use of virtual contact points to solve the dexterous manipulation of unknown objects is presented in [8], and the teleoperated dexterous manipulation by an operator using an input interface, such as a keyboard, is carried out in [18], where the task is carried out by the hand following the directions of the operator.

Since the real world is so diverse, it is unrealistic for a robot to rely solely on a pre-existing accurate model of its surroundings.The robot must be able to adapt itself to new environments. In this context, machine learning may play a key role in the problem of robot manipulation. The approaches to allowing a robot to learn to manipulate objects range from learning specific manipulation skills from human demonstration to learning abstract descriptions of a manipulation task for discovering an object’s functionality through interaction, and many more in between. These can typically be categorized into five types [19]:

• Learning representations of the object and the environment: The robot learns the state characteristics and degrees of freedom of each object in its environment, e.g., learning object pose [20], shape [21], and material properties [22].
• Learning a transition model of the environment: The robot learns how its actions affect the task state [23,24].
• Learning skill policies: The robot learns to obtain a behavior or skill controller to perform a desired manipulation task [25,26,27].
• Learning to characterize skill: The robot learns a model of the results of the manipulation. A post-condition model describes the areas of the state space that the robot will go after using a skill [28,29,30].
• Learning compositional and hierarchical structure: The robot learns hierarchical knowledge that will improve its ability to solve new tasks [31,32,33].

The application of learning techniques to dexterous manipulation is the subject of many works. A framework of dexterous imitation learning for dexterous manipulation is presented in [34]; it requires a single RGB camera to observe a human operator and teleoperate the robotic hand. Reinforcement learning (RL) is used to learn dexterous manipulation policies that can perform vision-based object reorientation on a physical Shadow dexterous hand [25,35]. The use of data obtained from human demonstrations to learn the dexterous manipulation of the desired object is proposed in [36,37].

The adaptive neuro-fuzzy inference system (ANFIS) is another learning technique that was created by combining artificial neural networks (ANN) and fuzzy logic, two powerful soft computing frameworks [38]. ANFIS is a system capable of modeling complex, non-linear relationships between inputs and outputs and is commonly used in machine learning and control systems to approximate and represent functions that may be difficult to describe using mathematical equations. ANFIS requires less data for training compared to deep neural networks, making it suitable for scenarios with limited data sets. Moreover, ANFIS offers a transparent and interpretable model which is valuable in applications where understanding the reasoning behind decisions is crucial. The input and output training data are used to adjust the ANFIS parameters for robot navigation [39]. Taking advantage of the ANFIS controller for mobile robot navigation is presented in [40,41]. The application of multiple ANFIS models for object shape recognition is depicted in [42].

Data generation is another crucial component in the application of learning methods, including ANFIS. There are various methods for providing the required data for dexterous manipulation based on employing various sensing and data collection techniques. Optical tracking systems [43], magnetic sensors [44], force and tactile sensors [8], and glove-based sensors [45] are the most common approaches. Human-to-robot hand motion mapping [46,47] is frequently required in several applications, and the approaches to address this problem fall under the following six categories [45]: direct joint, direct Cartesian, task-oriented, dimensionality reduction-based, pose recognition-based, and hybrid mappings.

In this work, we use a learning-based planner for dexterous manipulation, feeding data obtained from an analytical manipulation planner that uses a heuristic to move the fingers to rotate the object in the desired direction, proposed in [8]. The proposed planner offers an adaptive method to improve robotic dexterity and is tested in the real world using an Allegro anthropomorphic robotic hand [48] with tactile sensors [49] at its fingertips. The structure of the rest of the paper is the following: Section 2 describes the problem statement. Section 3 presents the proposed manipulation strategy. Section 4 presents the experimental validation. Finally, Section 5 draws the main conclusions of the work and proposes future research paths.

2. Problem Statement

This work attempts to find a highly adaptable solution for the dexterous manipulation of unknown objects (i.e., in the absence of a precise model or information about the object geometry and properties, such as varying shapes, sizes, and material properties) with a robotic hand in real-time environmental conditions with limited sensing capabilities. Taking advantage of a data-driven approach, we employ a learning-based planner with two modules: path planner (PP) and inverse kinematics (IK). In addition, no external measurements are available as feedback to obtain information about the position of the object within the hand. The manipulation goal is to rotate the grasped unknown object around the X-, Y-, or Z-axis of a reference system attached to the robotic hand. It is assumed that the unknown object has already been grasped before starting the dexterous manipulation and that three fingers of a robotic hand are employed to grasp and manipulate the object, performing a tripod grasp [50], which operates in the same way as humans do, with the thumb working opposite to the index and middle fingers.

3. Proposed Manipulation Strategy

The manipulation strategy is based on utilizing a learning-based approach instead of a pre-defined manipulation model. ANFIS, as a machine-learning method, can perform well for limited training data, and as an inference learning system, it has the ability to provide an interpretable explicit representation of what ANFIS internally performs. The required data to train ANFIS are obtained from simulation, fixing different grasping configurations, and using a heuristic to move the fingers to rotate the object in the desired direction, as was performed in [8]. The trained ANFIS structure is expected to enable the robotic hand to carry out the same manipulation tasks by computing the finger movements to properly rotate the object during manipulation.

3.1. Overview of the Proposed Manipulation Strategy

Let us first introduce some basic nomenclature. Consider an anthropomorphic robotic hand, and let:

• $q_{ij}$ be the position of joint j of finger $f_i$, with $i=1,\dots ,n_f$ and $j=1,\dots ,n_j$.
• ${\mathbf{q}}_i=[q_{i1}\dots q_{i{n}_j}]$ be the configuration of finger i.
• $Q={[{\mathbf{q}}_1\dots {\mathbf{q}}_i\dots {\mathbf{q}}_{n_f}]}^T$ be the configuration of the hand.
• ${\mathbf{p}}_{f_i}=\left[x_{f_i}{y}_{f_i}{z}_{f_i}\right]$ be the fingertip Cartesian coordinates of $f_i$ expressed in the hand reference system.
• $P={[p_{f_i}\dots p_{f_i}\dots p_{f_i}]}^T$ be the set of all the fingertips of the hand represented in a matrix.

The main idea of the proposed approach is to use an ANFIS planner to plan the hand movements to produce the dexterous manipulation of unknown objects. The ANFIS planner is a data-driven planner based on a trained and validated ANFIS structure that aims to perform planning tasks. Unlike analytical planners, ANFIS planners have high adaptability and, therefore, can better generalize across a wider range of scenarios. They can learn and adapt their decision-making process based on the observed input–output relationships in the environment. The ANFIS planner predicts values of the next step configuration of the hand ($Q^{k+1}$) based on current fingertip positions ($P^k$). Figure 1 presents a diagram of the system using an analytical planner that works as follows: The input to the system is the desired total rotation angle of the manipulated object around a given axis, ${\varphi }_{desired}$, which is split into small rotation steps, $\Delta \varphi$, used as input for each manipulation iteration until the target, ${\varphi }_{desired}$, is achieved. The current positions of the fingertips, P, are obtained from the current configuration of the hand, Q, and used jointly with the desired rotation, $\Delta \varphi$, by a path planner to determine the proper movements of the fingers to (try) to produce the desired rotation of the object. The planner generates the desired final position of each fingertip, ${\mathbf{p}}_{f_i}$, and using inverse kinematics, the corresponding positions of the hand joints Q are obtained and fed to the control system of the hand. Now, Figure 2 illustrates the alternative proposed diagram using the ANFIS planner. The ANFIS planner replaces the two main components of the analytical planner: on the one side, the path planner is replaced by an ANFIS path planner (ANFIS-PP) that works based on learning from experience and data, and, on the other side, the IK module is also replaced by an ANFIS module (ANFIS-IK) that is previously trained. Even when the IK of a finger is analytically known, since the general goal of the work is to perform object manipulation using only learning approaches, the analytical IK module was replaced by a learning module trying to minimize the required knowledge about the system.

The steps for the development of this ANFIS machine learning planner, detailed in the next subsections, are the following:

• Step 1: Input and output data generation and preparation.
• Step 2: Training the ANFIS structure.
• Step 3: Validate the ANFIS planner in simulation.
• Step 4: Validate the ANFIS planner in a real-world experiment.

3.2. Input and Output Data Generation and Preparation

Data generation for the training phase can be performed in two ways: (a) using a dataglove to capture the movements of a human hand while performing a manipulation task; (b) running a manipulation task with the robotic hand (either real or in simulation) using any pre-defined manipulation strategy, and capturing the positions of the hand joints during the manipulation process. In any case, in this work, we consider we already have the input data independently of the way they were obtained (which is not relevant for the contribution of this work).

The considered input for the ANFIS-PP training are a data set $D\left[P^k\right]$ of fingertips in Cartesian space at time k, $P^k$, and the corresponding data set $D\left[P^{k+1}\right]$ of outputs $P^{k+1}$ after one manipulation iteration, while $\Delta \varphi$ is considered a constant (note that it could also be considered as an input variable in the problem). Regarding the ANFIS-IK, the input are a data set $D\left[P^k\right]$ of fingertips in Cartesian space at time k, $P^k$, and the corresponding data set $D\left[Q^k\right]$ of outputs $Q^k$, obtained by using the known inverse kinematics of the hand. These data sets are reprocessed to remove outlier samples using a box whisker (BW) plot [51].

3.3. Learn from Data Using the ANFIS Algorithm

ANFIS stands for “adaptive neurofuzzy inference system”, and it is used to learn a specific model from given input and output data. In this case, it is configured as a neural network that learns the manipulation behavior of the hand from the training data set. Figure 3 shows the ANFIS architecture, which is a five-layer network. Each layer $l\in \{1,\dots ,5\}$ involves node(s), input(s) and output(s), and there are also directional links that connect the nodes of each layer. A node is a fundamental element that represents a particular function in each layer. The nodes are as follows: (a) fixed (represented by circles), which have predetermined structures and parameters that do not change during training, and, (b) adaptive (represented by squares), which have parameters that are adjusted during the learning process based on the input data using the back propagation technique and the recursive least squares (RLS) method. Note that the ANFIS structure is multi-input single-output (MISO).

This type of inference system is used to learn:

• The manipulation movements of each finger $f_i$ (ANFIS-PP). In this case, an ANFIS system is used to learn each of the coordinates of the next position of a fingertip (i.e., $x_{f_i}^{k+1}$, $y_{f_i}^{k+1}$ and $z_{f_i}^{k+1}$) from the current ones (i.e., $x_{f_i}^k$, $y_{f_i}^k$ and $z_{f_i}^k$). Thus, the input to the system in Figure 3 is a fingertip position ${\mathbf{p}}_{f_i}^k=\left[x_{f_i}^k{y}_{f_i}^k{z}_{f_i}^k\right]$ and the output is a coordinate $c_{f_i}\in \{x_{f_i}^{k+1},y_{f_i}^{k+1},z_{f_i}^{k+1}\}$. This implies that the number of ANFIS systems involved in an ANFIS-PP is equal to three times the number of used fingers since a system as presented in Figure 3 is necessary for each of the three fingertip coordinates of each of the three fingers. Note that this means that the variation in each component of the fingertip depends on the current values of all three components.
• The inverse kinematics of the fingers (ANFIS-IK). In this case, an ANFIS system is used to learn each of the joint values $q_{R_{ij}}$ of a finger from the Cartesian components of the fingertip position (i.e., $x_{f_i}$, $y_{f_i}$, and $z_{f_i}$). Thus, the input to the system in Figure 3 is a fingertip position ${\mathbf{p}}_{f_i}=\left[x_{f_i}{y}_{f_i}{z}_{f_i}\right]$ and the output is $q_{R_{ij}}$. This implies that the number of ANFIS systems involved in an ANFIS-IK is equal to the number of joints of each finger.

The proposed structure of the ANFIS system is as follows (see Figure 3):

• Fuzzification layer: In this layer, there are a number of membership functions ($N_{MF}$) for each input, i.e., $A_m$, $B_m$, and $C_m$ with $m\in \{1,\dots ,N_{MF}\}$. $N_{MF}$ is a hyper-parameter selected by humans and based on compromising the accuracy and computational burden. In our particular applications, we choose $N_{MF}=2$. The selected type of these membership functions is a generalized Gaussian bell membership function (see Figure 4) defined as follows:

  $$\begin{array}{c}\hfill {\mu }_{A_m}\left(x_{f_i}\right)={\left(1+{\left(\frac{x_{f_i}-c_m}{a_m}\right)}^{2b_m}\right)}^{-1}\\ \hfill {\mu }_{B_m}\left(y_{f_i}\right)={\left(1+{\left(\frac{y_{f_i}-c_m}{a_m}\right)}^{2b_m}\right)}^{-1}\\ \hfill {\mu }_{C_m}\left(z_{f_i}\right)={\left(1+{\left(\frac{z_{f_i}-c_m}{a_m}\right)}^{2b_m}\right)}^{-1},\end{array}$$

  (1)

  where ${\mu }_{A_m}$, ${\mu }_{B_m}$ and ${\mu }_{C_m}$ are the fuzzy membership degrees for each input variable, and $a_m$, $b_m$, $c_m$ are defined as premise parameters that determine the shape of the membership function and are optimized using back-propagation to minimize the error between the model output and the actual target values.
• Product Layer: In this layer, there are eight fixed nodes (resulting from three inputs and two membership functions for each one i.e., $2^3=8$). Each node represents a fuzzy rule, labeled $\pi$, that outputs the firing strength of each rule $w_n$, $n=1,\dots ,8$, which multiplies the incoming signals and sends the product out of the layer,

  $$w_n={\mu }_{A_m}\left(x_{f_i}\right)·{\mu }_{B_m}\left(y_{f_i}\right)·{\mu }_{C_m}\left(z_{f_i}\right),m=1,2,n=1,\dots ,8$$

  (2)
• Normalization layer: In this layer, there are again eight fixed nodes, labeled N, that output normalized firing strengths ${\overline{\omega }}_n$, computed as the ratio of the $n_{th}$ rules firing strength $w_n$ to the sum of all rule’s firing strengths:

  $${\overline{\omega }}_n=\frac{{\omega }_n}{{\sum }_{n=1}^8{\omega }_n},n=1,\dots ,8$$

  (3)
• Defuzzification layer: In this layer, there are eight adaptive nodes with a function $F_j$ that outputs the weighted value of a given rule as:

  $$F_n={\overline{\omega }}_n·(p_n{x}_{f_i}+q_n{y}_{f_i}+t_n{z}_{f_i}+r_n),n=1,\dots ,8,$$

  (4)

  where $p_n$, $q_n$, $t_n$, and $r_n$ are defined as consequent parameters that will be trained by a supervised learning algorithm.
• Summation layer: In this layer, a single fixed node, labeled $\mathsf{\Sigma }$, outputs one of the coordinates of the fingertip ${\mathbf{p}}_{f_i}=\left[x_{f_i}{y}_{f_i}{z}_{f_i}\right]$ as the summation of all incoming signals, i.e., the output of the ANFIS system is either $x_{f_i}$, $y_{f_i}$, or $z_{f_i}$ depending on the case,

  $$c_{f_i}=\sum _{n=1}^8{\overline{\omega }}_n(p_n{x}_{f_i}+q_n{y}_{f_i}+t_n{z}_{f_i}+r_n) \mathrm{with} c_{f_i}\in \{x_{f_i},y_{f_i},z_{f_i}\}$$

  (5)

3.4. Validation of the Learned ANFIS Planner in Simulation

Once it is trained, the ANFIS structure must be validated. This is performed in simulation by comparing the output of the ANFIS structure, ${\mathbf{q}}_i^E$, with the corresponding actual data, ${\mathbf{q}}_i$, as:

$${\mathbf{q}}_i^E=evalfis\left(Outfis\left(ij\right),{\mathbf{p}}_{f_i}\right),$$

(6)

where evalfis is the function to evaluate the fuzzy inference system for input ${\mathbf{p}}_{f_i}$, using the trained ANFIS structure to be evaluated, em Outfis, known as the Sugeno fuzzy inference system (Sugfis), and $n_{dp}$ as the number of data points. The proposed approach is verified through the mean squared error (M.S.E.) method:

$$\mathrm{M}.\mathrm{S}.\mathrm{E}.(\%)=\left(\frac{1}{n_{dp}}\sum _{k=1}^{n_{dp}}{({\mathbf{q}}_i^E\left(k\right)-{\mathbf{q}}_i\left(k\right))}^2\right)\times 100$$

(7)

4. Experimental Validation

4.1. Hardware Set-Up

We use an Allegro Hand [48] (shown in Figure 5) to do dexterous manipulation. This is an anthropomorphic hand with four fingers, index, middle, ring, and thumb, with four degrees of freedom (DOF) per finger. The first three fingers share the same kinematic structure while the thumb has a different one. The fingers are actuated by DC motors and have potentiometers in the joints to measure their positions with a resolution of 0.002 degrees. The fingertips are equipped with WTS-FT tactile sensors from Weiss Robotics [49]. The Allegro Hand is connected to a PC via a CAN bus, and a robot operating system (ROS) [52] is used as a communication layer to integrate some necessary software modules: a module to control the Allegro Hand with a PID controller and gravity compensation, a module with a graphical user interface to command the movements of each joint of the hand, and a module to perform the object manipulation following the proposed approach. Figure 6 shows the hardware and software set-up.

4.2. Implementation

The system was implemented in MATLAB R2021a running on Linux on a PC equipped with an Intel Core i5 CPU, 16 GB of RAM, a 1 TB hard drive, and an NVIDIA GPU. Rviz is used as a graphical user interface that allows users to interact visually with the robotic systems. Figure 7 shows the applied ANFIS structure of the index finger for manipulation, where the inputs are $x_{f_1}$, $y_{f_1}$, and $z_{f_1}$ with two membership functions for each input, a Sugeno-type fuzzy inference system (FIS) with eight rules corresponding to different combinations of membership functions (see Figure 7) and input variables, and the output is a finger joint. The same structure is used for all the joints of all fingers (four structures for each finger and in total 12 for three fingers).

The implemented ANFIS structure also requires:

(a)

A partition of the input space into a grid or fuzzy regions to facilitate the creation of fuzzy rules, overlapping the membership functions, and providing a structured way to represent and model the relationships between input and output variables in a fuzzy system.

(b)

The activation of each fuzzy rule, which is determined as a fuzzy AND operation on the membership values of the input variables associated with that rule.

(c)

The combination of rules which is performed by using a fuzzy OR operation. The OR operation typically involves summing up the contributions of each rule, weighted by its activation level.

(d)

The membership degree to be constrained between 0 and 1.

(e)

The integration of multiple optimization or learning techniques to enhance the training and performance of the ANFIS model, which is known as a hybrid method.

Table 1. Specification of the ANFIS architecture.

Name	Type
Generate FIS type	Sugeno
The initial FIS model	Grid partition
Decision method for fuzzy logic operation AND (minimum)	Product
Decision method for fuzzy logic operation OR (maximum)	Probabilistic
Output defuzzification method	Weighted average
Number of membership functions for $x_{f_i}$	2
Number of membership functions for $y_{f_i}$	2
Number of membership functions for $z_{f_i}$	2
Input membership function type	Gaussian bell
Output membership function type	Constant type
Number of rules	8
Train FIS optimization method	Hybrid
Number of epochs	30

summarizes the specification of the ANFIS architecture.

4.3. Description of the Experiment

As mentioned above, in this work, dexterous manipulation is performed using only three fingers: the index ($f_1$), the thumb ($f_2$), and the middle finger ($f_3$), with the thumb working opposite to the other two fingers in the same way that humans do. The desired manipulation is the rotation of the grasped object, forward and backward, around any of the axes, X, Y or Z, with the origin located at the thumb fingertip position, the X-direction perpendicular to the hand palm plate, and the Z-direction parallel to the middle finger when it is fully extended (see Figure 8). The object is unknown, which means that its properties (like shape, weight, and center of mass) are not known during manipulation. The grasping of the object is outside the scope of this work, so it is assumed that the object is already properly grasped before starting the manipulation.

According to the problem statement mentioned in Section 2, focusing on the performance of the ANFIS planner, and avoiding imposing complexity on the system in this specific application, this work is not intended to have any feedback from tactile sensors or any other external sensors.

The set of samples $D\left[P^k{Q}^k\right]$ for training the ANFIS structure was obtained from simulations, fixing different grasping configurations and using a heuristic to move the fingers to rotate the object in the desired direction, in a similar way as was performed in [8]. We generated 15 sequences of hand configurations of about 150 samples each, while manipulating four different objects (i.e., a total of 9000 samples and 60 sequences of hand configurations), and they were filtered using a box whisker plot (Figure 9 illustrates the result for the particular case of the index finger).

After filtering, the outlier data in each sequence are replaced using linear interpolation of neighboring non-outlier values.

Table 2. Some elements of the sample data set for the index finger, $f_1$.

	${\mathit{x}}_{{\mathit{f}}_1}$	${\mathit{y}}_{{\mathit{f}}_1}$	${\mathit{z}}_{{\mathit{f}}_1}$	${\mathit{q}}_{11}$	${\mathit{q}}_{12}$	${\mathit{q}}_{13}$	${\mathit{q}}_{14}$
1	0.09363	−0.03090	0.03066	0.16250	0.53036	1.91182	−1.12669
2	0.09363	−0.03090	0.03066	0.15810	0.52958	1.91366	−1.12774
3	0.09363	−0.03132	0.03061	0.15369	0.52887	1.91548	−1.12886
⋯	⋯	⋯	⋯	⋯	⋯	⋯	⋯
9000	⋯	⋯	⋯	⋯	⋯	⋯	⋯

shows same samples for the index finger $f_1$. Data from the manipulation of three of the objects (45 sequences of hand configurations containing 6750 samples) are allocated for training and data from the remaining object (15 sequences of the hand configurations containing 2250 samples) are reserved for validation. This allocation ensures that 75% of the samples are utilized for training and 25% are reserved for validation of the system’s performance.

The data set for the training of ANFIS-PP, denoted as $D\left[P^k{P}^{k+1}\right]$, includes $P^k$, used as the input, and $P^{k+1}$ as the output. Subsequently, $P^{k+1}$ is used as the input for ANFIS-IK and $Q^{k+1}$ as the output. Furthermore, $P^{k+1}$ is fed back into ANFIS-PP to compute the positions for the next iteration. It is noteworthy that ANFIS-IK is trained using the data set $DS\left[P^k{Q}^k\right]$. A schematic of the implemented ANFIS planner is shown in Figure 10. We utilized the ANFIS structure with a hybrid optimization approach and conducted 30 training epochs for both ANFIS-PP and ANFIS-IK.

4.4. Experimental Results

The proposed approach was first validated in simulation. Throughout this simulation, the performance of the trained ANFIS planner is examined in both modules. The ANFIS planner is challenged with a previously unknown scenario, and its output (EVALFIS) is supposed to follow the expected output. The error is calculated during the operation, bearing in mind that the expected output is the output data of the analytical planner that have been processed applying statistical tools. Utilizing Rviz as a graphical user interface, the object was rotated around each of the three axes, X, Y and Z, forward and backward for five times. As an example of system performance, Figure 11 shows the evolution of the index finger next step fingertips $x^{k+1}$, $y^{k+1}$, and $z^{k+1}$ in rotation around the Z-axis and Figure 12 depicts the evolution of the index finger joints, $q_{11}$, $q_{12}$, $q_{13}$, and $q_{14}$ in rotation around the X-, Y-, and Z-axes, when the system is fed with a set of samples from the test set; the mean square error (MSE) value between the outputs calculated by trained ANFIS and the expected output is shown

Table 3. Mean square error value between the outputs calculated by trained ANFIS and the expected output for $f_1$. MSE units for ANFIS-PP and ANFIS-IK are, respectively, in mm² and rad².

	ANFIS-PP			ANFIS-IK
Output	$x_{f_1}$	$y_{f_1}$	$z_{f1}$	$q_{11}$	$q_{12}$	$q_{13}$	$q_{14}$
MSE	$2.8\times {10}^{-8}$	$1.0\times {10}^{-7}$	$5.1\times {10}^{-8}$	$1.8\times {10}^{-4}$	$1.8\times {10}^{-4}$	$1.8\times {10}^{-4}$	$1.8\times {10}^{-4}$

. It can be seen that the obtained finger joint values are quite close to the expected values according to the test set. The computational time required by the ANFIS planner in each iteration during the experiment is shown in Figure 13. Under identical conditions, the average computational time for the ANFIS planner is 4.8 ± 0.08 ms and for the analytical planner is 1.0 ± 0.08 ms. Even though the ANFIS planner is slower, it is still fast enough for real-time execution.

After the simulations, the system was tested in a real-world experiment using the real Allegro Hand. The experimental scenario is the same as that used in simulation, and the ANFIS and analytical planner outputs are compared under identical operational conditions. This real-world experiment was executed by initiating the MATLAB code to establish a connection with ROS as a publisher. This enables real-time data exchange with the other connected components. This includes:

• Selecting the desired rotation axis, X, Y, or Z, by operator.
• Selecting ${\varphi }_{desired}$ (the desired rotation angle illustrated in Figure 2) by operator. The maximum rotational angles empirically achieved in our experiment around the X-, Y- or Z-axes were 35, 18, and 60 degrees, respectively.
• Defining $\Delta \varphi$ (which is the rotation step) equal to 0.5 degree (see Figure 2).
• Importing initial fingertips point position from RVIZ.
• Defining the desired number of rotation cycles.
• Executing the MATLAB code.

Figure 14, Figure 15 and Figure 16 show images of the experiment in the real setup for rotation of the unkhown object around the X-, Y-, and Z-axes, respectively. A video of the experiment is available at youtube.com/IOC Robotics Lab (accessed on 21 May 2024). It is worth noting that, due to the kinematic structure of the hand, depending on the hand configuration, certain motions have a more limited range than others. This is what happens to the rotation of an unknown object around the Z-axis in comparison to the other axes, creating the appearance of slower movements.

As an example of system performance in the real-world experiment, Figure 17 shows a comparison of the ANFIS and analytical planner outputs for the index finger joints, $q_{11}$, $q_{12}$, $q_{13}$, and $q_{14}$ during rotation around the X-axis, under identical conditions. It can be noticed that the planner outputs are relatively similar to each other; however, the ANFIS planner attempts to do the task more smoothly.

5. Conclusions

In this paper, we present a learning-based approach, based on ANFIS, to address in-hand manipulation in real-world scenarios, producing satisfactory operational outcomes. This method provides the robots with the ability to learn from data, significantly reducing their reliance on precise and complex mathematical models for planning and execution in dexterous manipulation. The transition to data-driven learning enables the robots to respond to a wider range of situations and uncertainties present in real-world environments more effectively, providing a more adaptable and reliable method for performing intricate manipulation tasks. For future work, incorporating tactile sensor data to reach force control, exploring alternative learning methods, extension of the approach to rotate unknown objects around any desired axes, and applying four fingers in manipulation will be considered. Furthermore, we plan to merge the two “learning modules” described in this work (ANFIS-PP and ANFIS-IK) into a single one, learning directly the proper joint movements for manipulation without the need for learning, on the one side, the fingertip movements, and, then, on the other side, computing the IK (either analytically or using a learning approach).