Trigger-Based Dexterous Operation with Multimodal Sensors for Soft Robotic Hand

Abstract

This paper focuses on how to improve the operation ability of a soft robotic hand (SRH). A trigger-based dexterous operation (TDO) strategy with multimodal sensors is proposed to perform autonomous choice operations. The multimodal sensors include optical-based fiber curvature sensor (OFCS), gas pressure sensor (GPS), capacitive pressure contact sensor (CPCS), and resistance pressure contact sensor (RPCS). The OFCS embedded in the soft finger and the GPS series connected in the gas channel are used to detect the curvature of the finger. The CPCS attached on the fingertip and the RPCS attached on the palm are employed to detect the touch force. The framework of TDO is divided into sensor detection and action operation. Hardware layer, information acquisition layer, and decision layer form the sensor detection module; action selection layer, actuator drive layer, and hardware layer constitute the action operation module. An autonomous choice decision unit is used to connect the sensor detecting module and action operation module. The experiment results reveal that the TDO algorithm is effective and feasible, and the actions of grasping plastic framework, pinching roller ball pen and screwdriver, and handshake are executed exactly.

1. Introduction

Dexterous operation is one of the important abilities for a robotic hand to interact with the external environment [1,2,3,4,5,6]. A soft robotic hand (SRH) is able to perform more compliant operations than a traditional rigid robotic hand [7,8]. Therefore, SRH has attracted increasing interest from researchers [9,10,11]. Deimel and Brock designed a compliant and underactuated robotic hand to perform dexterous grasping [12,13]. Tian et al. [14] embedded inner skeletons into the soft fingers to increase the hardness to grasp eggs, pens, cups, etc. Zhou et al. [15] designed a 13 degrees of freedom (DOF) SRH for dexterous grasping and in-hand manipulation. Faudzi et al. [16] designed a human-like robotic hand using thin soft muscles. Feng et al. [17] used electromyography (EMG) to control the SRH. Devi et al. [18] designed a novel underactuated multifingered SRH for prosthetics to grasp multiple objects. However, these SRHs lacked sensors and sensing ability, which limited the dexterity of operation. To address this issue, Zhao et al. [19] developed an optoelectronically innervated soft prosthetic hand via stretchable optical waveguides that were fabricated with two soft silicone composites with different refractive indexes. The soft prosthetic had active haptic sensing to detect shape, texture, and softness and was able to select the ripest tomato among a group. Tavakoli et al. [20] performed an autonomous selection of closing posture by using soft matter capacitive sensors. Moreover, highly sensitive soft sensors are used in anthropomorphic robotic hands [21], and highly stretchable optical sensors are used to measure pressure, strain, and curvature [22]. In spite of this, there are still few SRHs that use multimodal sensors for human–robot interaction.

On the other hand, operation strategy is another problem to consider. Gu et al. [23] realized grasp configuration optimization based on haptic exploration information. Homberg et al. [24] realized robust proprioceptive grasping with an SRH. Gupta et al. [25] performed learning dexterous manipulation for an SRH from human demonstration.

In summary, single modal sensors have been introduced into the SRHs to constrain perceptual skills for dexterous operation. On the other hand, the above-mentioned SRHs are mostly used to grasp objects, not to interact with humans. Therefore, we aim to develop dexterous operation for an SRH with multimodal sensors in this paper. Firstly, multimodal sensors are designed and integrated into an SRH. An optical-based fiber curvature sensor (OFCS) is designed and embedded in the soft finger together with a gas pressure sensor (GPS) to realize curvature detection for the soft finger. A capacitive pressure contacts sensor (CPCS) is attached on the fingertip to detect the touch force of the fingertip, while a resistance pressure contact sensor (RPCS) is attached to the palm to detect the touch force of the palm. These four sensors form a multimodal sensor system for the SRH. Secondly, a trigger-based dexterous operation (TDO) strategy is proposed to improve the dexterity of operation. Operation experiments were conducted to validate the effectiveness and feasibility of dexterous operation.

The main contribution of the work is to add different sensors on the SRH to realize the multimodal dexterous operation. The paper is organized as follows: Section 2 introduces multimodal sensors integrated in the SRH; these sensors are used for curvature detection and touch force detection. Section 3 describes the framework of TDO and the design of the program flow of TDO in the embedded system. Section 4 shows the experiment results, which demonstrate that the TDO is effective and feasible. Finally, Section 5 presents the conclusions of this paper.

2. Multimodal Sensors

To realize dexterous operation with multimodal sensors for an SRH, the first step is to arrange the multimodal sensors reasonably. Therefore, four modal sensors are deployed on the SRH based on their detecting abilities to realize curvature detection and touch force detection.

2.1. The Layout of Multimodal Sensors

The layout and intended use of multimodal sensors are shown in Figure 1. There are four modal sensors, namely OFCS, GPS, CPCS, and RPCS. The OPCS embedded in the soft finger and the GPS integrated into the proportional valve are used to detect the curvature of each finger. The CPCS attached to the fingertip is employed to detect the touch force of each fingertip, and the RPCS attached to the palm is used to detect the touch force of the palm. The fingertip touch force detection determines the action trigger, while the finger curvature detection is used as gesture detection to verify whether the action of the SRH is correct or not.

2.2. Curvature Detection Sensors

Figure 2 shows the procedure of integrating the OFCS in a soft robotic hand. The optical fiber is the core component of the OFCS, which is roughened with a laser cutting machine. The silicone (Ecoflex 0030, Smooth-on, Inc., Macungie, PA, USA) is poured into a model manufactured with 3D printing technology. Then, a soft ringer with the OFCS is produced, and five fingers are integrated into a palm to form an SRH. The principle of the OFCS is based on light energy loss; the light is emitted from the gap on the roughened optical fiber when the soft finger is bent. The greater the bending degree, the more light is emitted. Therefore, the curvature of the finger is obtained by detecting the light energy loss. A detailed description can be found in [26], written by the authors of this paper.

The GPS is embedded in a proportional valve, which is used to measure the absolute pressure of gas and connected in the gas channel of each finger in series mode. The proportional valve (ITV2030-312CS, SMC, Inc., Tokyo Metropolis, Japan) is able to pass pressure of 0.5 Mpa. The control signal is an analog signal with DC 0–10 V, and the analog output signal DC 1–5 V reflects the gas pressure in the gas channel in a linear relation. The OFCS and GPS are used to detect the curvature of the finger. The gesture is recognized by merging the sensor information of five fingers. The gesture detection is compared with the action of SRH to validate the action trigger is in the correct state.

2.3. Touch Force Detection Sensors

The CPCS is a pressure sensor that uses capacitive sensitive elements to convert the measured pressure into a certain relationship with the electrical output and is used for fingertip touch force detection. The silicon membrane covers the outer contour of the CPCS. One of the aims is to protect the sensor by isolating the sensor from external contact objects. The other aim is to make the sensor easy to attach to the surface of the fingertip. The processing flow of the tectorial membrane for the CPCS is shown in Figure 3a. RPCS refers to a sensor based on the piezoresistive effect of materials and integrated circuit technology; the resistivity changes after being subjected to force, and the electrical signal output proportional to the force change can be obtained through the measurement circuit; it is used as palm touch force detection sensor. Similar to CPCS, the RPCS is also covered with a silicon membrane. The processing flow is shown in Figure 3b.

Figure 3c shows that the CPCS and RPCS are attached to the SRH. There are five CPCSs that are attached on each fingertip, while there is a CPRS with 3 × 3 array resistance strain sensors that is attached to the palm of SRH. Figure 3d shows that the CPCSs and RPCS are connected with the acquisition circuits. Fourteen analog channels are required to perform synchronous signal acquisition for the CPCSs and RPCS.

3. TDO Strategy

To improve the operation ability of SRH, the multimodal sensors must be used together to enhance perception and interaction. For this, a TDO strategy with multimodal sensors is proposed to perform autonomous choice operation.

3.1. The Framework of TDO

The framework of TDO is shown in Figure 4.

The dexterous operation includes sensor detection and action operation, and the action operation is determined by the sensor detection. The sensor detection module includes the hardware layer, information acquisition layer, and decision layer. The CPCS and RPCS constitute the hardware layer to provide sensor information. In the information acquisition layer, the sensor information acquisition is implemented, and whether the sensor trigger event happens or not is detected. In the decision layer, it is necessary to judge which sensor is triggered when the sensor trigger event happens, and an autonomous choice decision is made. Then, the sensor detection determines the action operation. The action operation module includes the action selection layer, actuator drive layer, and hardware. In the action selection layer, the operation type is selected in the operation library. Then, the control value for every finger is assigned. In the actuator drive layer, the analog voltage is controlled to drive the proportional valve, and then the corresponding gas pressure is output to drive the soft finger. In the hardware layer, the finger of SRH is bent according to the selected action.

3.2. The Implementation of TDO

In the control system of the SRH, the embedded system STM32F407 (STMicroelectronics, Inc., Geneva, Switzerland) is used as the CPU. The program flow of TDO is given in Figure 5.

In the interrupt routine, the sensor information is firstly acquired synchronously by the ADC converter. Then, the sensor trigger state is detected, and the sensor information is continuously acquired if no trigger occurs. Once the trigger occurs, we judge which sensor induces the trigger. If palm touch force triggers, the action of grasp is launched; if fingertip touch force triggers, the action of pinch is activated; if multiple fingertip touch forces trigger, the action of handshake is started. Straight after, finger curvature is detected to verify whether the action is correct or not; if it is wrong, continue to launch the corresponding action; if it is correct, end the interrupt routine. The trigger sensor judgment and action constitute autonomous choice strategy, which is the core unit of the TDO.

4. Experiment and Results

In order to verify the effectiveness of the developed TDO strategy with multimodal sensors, experiments were conducted, with three kinds of action being operated, namely the grasp action, the pinch action, and the handshake action.

4.1. The Calibration of Touch Force Detection Sensors

The CPCS and RPCS are the touch force detection sensors that are used to detect the trigger event. In the application system, the sensor characteristic needs to be calibrated.

Eleven weights were employed to calibrate the CPCS; the sensor characteristic is given in Figure 6a. The relationship between force and output voltage is given in Equation (1).

V = 0.1000 × F + 0.5000

(1)

The weights are also used to calibrate the RPCS; the sensor characteristic is given in Figure 6b. The relationship between force and output voltage is given in Equation (2).

V = 0.0220 × F² − 0.5584 × F + 4.7468

(2)

4.2. TDO Validation

The validation experiment of TDO is shown in Figure 7, with three kinds of action being operated. The grasp action is used to grasp the plastic frame when it touches the palm and the palm touch force is triggered. The pinch action is used to grasp the roller ball pen and screwdriver when the touch force of the index fingertip is triggered. The handshake action is executed when the touch force of multiple fingertips is triggered. The experimental results show that the actions are operated exactly.

The results of trigger state detection are shown in Figure 8.

As shown in Figure 8a, the palm touch force trigger is detected when the plastic frame touches the lowest row of the RPCS, and when the voltage values of the RPCS cross the threshold values (TV) at the trigger point (TP), the trigger occurs. The RPCS includes nine of the same sensors, and the palm touch force trigger detection was judged only by the signals from the three bottom sensors, as displayed in Figure 8a. As shown in Figure 8b, the index fingertip touch force trigger is detected when the roller ball pen touches the index CPCS; when the voltage value of the index CPCS crosses the threshold value 0.57 V, the trigger occurs. As shown in Figure 8c, as the screwdriver touches the index CPCS, the voltage value of the index CPCS crosses the threshold value, and the trigger occurs. As shown in Figure 8d, the multiple fingertip touch force trigger is detected. The corresponding threshold values are index fingertip with 0.57 V, middle fingertip with 0.53 V, ring fingertip with 0.58 V, and little fingertip with 0.55 V. When a human hand touches the index, middle, ring, and little fingertips, the corresponding voltage values cross the threshold values, and the SRH executes a handshake with the human hand. The subfigures in Figure 8b–d show the voltage transformation of different fingers.

5. Conclusions

In this paper, we develop dexterous operation for an SRH with multimodal sensors to improve the operation ability of the SRH. OFCS and GPS are used to detect the curvature of the soft finger. CPCS and RPCS are used to detect the touch force of the fingertip and palm. Then, a TDO strategy is proposed to perform the autonomous choice action operation. The framework of TDO includes a sensor detection module and an action operation module. The sensor detection module is divided into hardware layer, information acquisition layer, and decision layer. The action operation module is divided into three layers, namely action selection layer, actuator drive layer, and hardware layer. An operation experiment was conducted to evaluate the proposed multimodal sensor system and TDO method. The experimental results reveal that the TDO strategy is effective and feasible; the trigger event is detectable when the sensors in different parts of the SRH are touched. Then, the corresponding action is autonomously selected to make the SRH realize the gesture. However, much work is necessary to improve the dexterous operation of SRHs.

Future work will focus on designing a multijoint SRH to increase the number of action operations, introducing a machine learning algorithm to train the operations, making a complete closed loop for a sensor–actuator–sensor system to make the control system exactly, performing imitation learning from a human to guide the operation of the SRH, and conducting a quantitative comparative analysis with similar products.