Curvature Sensing and Control of Soft Continuum Robots Using e-Textile Sensors

Abstract

Soft continuum robots, with their flexible and deformable structures, excel in tasks requiring delicate manipulation and navigation through complex environments. Accurate shape sensing is vital to enhance their performance, safety, and adaptability. Unlike rigid sensors, soft sensors conform to the robot’s flexible surfaces, ensuring consistent measurement of shape and motion. This paper introduces a new approach using soft e-textile resistive sensors, which integrate seamlessly with the robot’s structure. These sensors adjust their resistance in response to movements, capturing multidimensional force data. A deep Convolutional Neural Network (CNN) decodes the sensor signals, enabling precise shape estimation and control. Our findings indicate that soft e-textile sensors may surpass traditional rigid sensors in shape sensing and control, significantly improving the functionality of soft continuum robots in challenging applications.

1. Introduction

Soft continuum robots are a type of robot defined by their flexible and deformable structures, enabling continuous curvature and smooth movements. These robots are particularly useful in applications requiring precise and delicate manipulation, as well as navigation through complex and constrained environments [1,2]. Unlike conventional robots, which are limited by their rigid structures, soft robots are adept at maneuvering through intricate spaces, performing delicate operations, and safely interacting with humans. Continuum robots, with their elastic frameworks, demonstrate this flexibility, being capable of bending in nearly limitless ways due to their elastic backbones [1]. These robots are constructed from various soft materials, such as rubber [3], polyamide [4], silicone [5], and nitinol alloy (NiTi) [6], to achieve their remarkable adaptability.

The inherent flexibility of soft robots is crucial in applications like minimally invasive surgery [7,8], search and rescue missions [9,10], and human–robot interaction [11], where the ability to adapt to and navigate unpredictable environments is vital. However, this same compliant nature also presents unique challenges in sensing and control [12,13,14]. As the demand for soft robotics grows, it is essential to develop sensing systems that not only align with their flexible characteristics but also provide the precise feedback necessary for complex tasks [15,16].

Various methods have been explored for shape sensing in continuum robotic systems, including fiber optic sensor-based approaches like Fiber Bragg Grating (FBG) sensors [17], electrical impedance techniques [18], electromagnetic (EM) tracking [19], and imaging systems [20]. However, each of these techniques has drawbacks that limit their effectiveness, particularly in Minimally Invasive Surgery (MIS), where the specific demands of the application pose challenges for their integration with continuum robots.

While FBG sensors offer notable shape-sensing precision, their widespread adoption is limited by the high costs of both the fiber material and the interrogation equipment, which hinders their practical use [21]. Similarly, electromagnetic positioning systems are frequently used but face significant challenges in surgical settings due to interference from metal objects and various equipment, which can compromise their accuracy [15]. Additionally, imaging methods, like computer tomography and magnetic resonance imaging, although effective, expose patients to ionizing radiation or X-rays, raising concerns about patient safety [22].

Accurately determining the shape of continuum robots remains a challenging task despite various existing methods. These methods must be affordable, safe for humans, and highly accurate. Since most continuum robots lack feedback control systems, integrating such systems is complex. Therefore, there is a significant need for a new method that is both flexible and cost-effective. To address the limitations of current rigid sensors, we propose a novel solution: the use of soft e-textile sensors to enhance the performance of soft continuum robots. Soft e-textile sensors are a new type of electronic fabric capable of detecting various physical properties around them and within the human body [23]. Recently, these sensors have gained popularity in wearable devices and health monitoring applications [24]. This approach is not only adaptable and cost-effective but also overcomes the limitations of existing rigid sensors. By leveraging the unique features of soft e-textile sensors, we aim to significantly improve the capabilities of soft continuum robots, addressing a crucial gap in the field of robotics.

This research investigates the potential of employing e-textile sensors to extend the range of force measurements that the sensor can accurately detect. This enhancement is facilitated by the sensor’s surface conductivity characteristics. By integrating the e-textile sensor directly into the robot’s framework, we leverage its inherent flexibility to monitor changes in resistance caused by the robot’s bending actions. To improve shape detection accuracy, we have developed a data-driven Convolutional Neural Network (CNN) approach. This CNN method analyzes the complex patterns of resistance variations recorded by the sensors, allowing it to precisely determine the robot’s shape and curvature in real-time. This advancement enhances the accuracy and adaptability of feedback control in continuum robots, enabling continuous state monitoring and necessary adjustments in dynamic environments.

The structure of this paper is organized as follows: Section 2 details the design of the e-textile sensor with its layered construction. Section 3 outlines the deep CNN learning model utilized for shape sensing and control of the robot. Section 5 showcases the experimental results, highlighting the effectiveness of the proposed soft sensor in curvature sensing and shape control of continuum robots. Finally, Section 6 concludes the paper and proposes directions for future research.

2. e-Textile Sensor Design

The soft force sensor we propose utilizes EeonTex NW170-SPLPA [25], a variable resistance stretchable sensor fabric created by Eeonyx. Comprising 72% nylon and 23% spandex fibers, this non-woven microfiber is coated with a conductive polymer that induces a piezoresistive effect. This property allows the fabric’s electrical resistance to vary with deformation, enabling precise measurement of pressure, bending, stretching, or torsion. The fabric’s flexibility and pliability, combined with its responsiveness to pressure, make it ideal for designing shape sensors. According to the manufacturer’s specifications, the sensor is highly durable, with a tensile strength of over 450 N and a tear resistance of 12 N, capable of withstanding 1000 washing cycles. Given this data, the e-textile sensor is deemed suitable for use in soft robot curvature sensing applications.

To construct the e-textile force sensor, the fabric was cut into a 2 cm by 2 cm square. This sensor layer was then placed between two vertically aligned copper electrodes, acting as a piezoresistive pressure sensor, as depicted in Figure 1. The electrodes were connected to a breadboard and a resistance measuring circuit, incorporating a Wheatstone bridge for signal conditioning and an Arduino UNO with a 10-bit Analog-to-Digital Converter (ADC). In data acquisition systems, ADCs play a crucial role in capturing and digitizing analog signals from sensors and transducers. Since the e-textile sensor readings in terms of resistance change is an analog signal that is continuous in time and amplitude, the ADC converts the sensor readings into a digital signal, which is discrete in both time and amplitude to be processed by the digital controller. Upon applying pressure, the material’s resistance changes accordingly.

To increase the ability of our soft sensor to measure a wider range of forces, we use a multi-layered design. Each layer is a 2 cm by 2 cm square of conductive textile material. The first and last layers are connected to the copper electrodes, while the intermediate layers are placed in between, as depicted in Figure 2. Adding more layers extends the sensor’s dynamic range [26,27], although it may affect the response time for detecting applied forces. Achieving the right balance of layers based on response time and dynamic range needs is crucial for optimal performance in specific applications. This balance is a common challenge in designing soft force sensors [28]. To enable more sensing points on the e-textile shape sensor, a 4 × 4 grid of conductors is arranged into a matrix above and below the soft sensing material, creating 16 sensing points, as shown in Figure 3.

3. Deep-CNN Shape Sensing

In the field of soft robotics, accurately predicting the shape of the robot without affecting its inherent properties or performance is crucial. In this study, we utilized a CNN deep learning model to estimate the curvature parameters of a soft robot based on data from the 16 sensing points of the e-textile soft sensor, which covers a single section of the continuum robot, as shown in Figure 4. Each sensing point functions as a sensor pad, created by sandwiching the soft textile material between two crossed copper connections. The selection of a CNN is due to its exceptional ability to process spatially correlated data, capturing the deformation patterns of the soft robot. CNNs have proven to be highly effective in various applications where spatial relationships are important, including image and video recognition, medical image analysis, and natural language processing.

3.1. Data Collection and Processing

Nonlinear regression models can achieve more accurate predictions by increasing the training dataset size, although this requires significant computational resources. Our research utilizes a dataset generated from experimental studies of a cable-driven continuum robot equipped with an e-textile sensor. For each data sample, we sequentially applied tension to the three servo motors actuating the robot and recorded data from the 16 sensing points on the e-textile sensor, which serve as the model features. The curvature $\kappa$ and the angle of the curvature plane $\varphi$ are the targets of our CNN model. We estimated these parameters using an Inertial Measurement Unit (IMU) sensor attached to the robot’s tip, based on the constant curvature assumption [1].

It can take a while to investigate every potential set of tensions for the robot’s actuation. In order to practically address this, we created an extensive dataset of 1330 data points. These data points were gathered inside the anticipated range of the 18 cm-long continuum robot being tensioned by the servo motors. The sampling procedure was carefully planned to provide complete coverage of the robot’s working range while respecting allowable servo restrictions. Thirty percent of the dataset was set aside for testing, fifteen percent for validation, and seventy percent for training in order to guarantee consistent model performance and generalization. The purpose of implementing this partitioning technique was to enable thorough model training and assessment.

To utilize CNNs for shape sensing of soft robots, each data sample, represented as the sensor readings $p_i\in {\mathbb{R}}^{16}$, must first be converted into a 2D feature image, denoted as $\mathbf{I}$$\in {\mathbb{R}}^{1330\times 4\times 4}$. This transformation is not merely a reshaping of data but a critical step in enabling the CNN to interpret the spatial correlations between different sensing points on the robot’s surface.

It is worth mentioning that if the sensor input falls outside the range of data that the CNN model was trained on, particularly in the presence of noise, it could affect performance. However, during dataset generation, the actuation was performed rapidly, causing disturbances in the sensor readings. These disturbances helped to broaden the range of fluctuations included in the training data, improving the model’s ability to handle such variations.

To improve the stability of the model during training, the training and testing datasets are standardized using the Z-score normalization method. This process ensures that our dataset has a mean ($\mu$) of 0 and a standard deviation ($\sigma$) of 1. The normalization of each data point $\mathbf{p}$ is conducted as follows:

$${\mathbf{p}}^{\prime }=(\mathbf{p}-\mu )/\sigma$$

It is essential to highlight that the values of $\mu$ (mean) and $\sigma$ (standard deviation) used for normalization are computed exclusively from the training dataset.

3.2. CNN Model Architectures

The CNN architecture used in our study is specifically designed for regression tasks, as our goal is to quantify the curvature of the soft robot rather than classify it into discrete categories. This regression output provides continuous values representing the curvature $\kappa$ and the angle of the curvature plane $\varphi$, which are estimated from an IMU attached to the robot’s tip during data collection. The network accomplishes this through two convolutional layers, each responsible for extracting increasingly abstract features from the input image. The first convolution layer uses kernel sizes of 16 and 8, with the hyperbolic tangent (tanh) activation function. These layers are followed by two fully connected layers with 16 and 8 units, respectively, that integrate these features to predict the robot’s curvature, as illustrated in Figure 5.

3.3. Training of CNN Model

During the training phase of our convolutional neural network (CNN) models, the Mean-Squared Error (MSE) loss function is utilized. This function is integral to the deep regressor model that estimates the shape parameters of the continuum robot. The MSE loss function quantifies the discrepancy between the CNN model’s predicted output, denoted as $\widehat{\mathit{y}}\left(\zeta \right)$, and the actual reference output $\mathit{y}$, which corresponds to the robot’s curvature. This loss function is particularly suitable for our regression task because it accurately measures the errors in continuous output values. The MSE is mathematically expressed as:

$$\mathrm{MSE}={\displaystyle \frac{1}{N}}\sum _{i=0}^{N-1}{[\widehat{\mathit{y}}{\left(\zeta \right)}^i-{\mathit{y}}^i]}^2$$

(1)

In the presented equation, the symbol $\zeta$ denotes the network parameters, encompassing the learnable weights and biases. To enhance the model’s performance, we employ the Adam optimizer [29], starting with an initial learning rate of ${10}^{-3}$. Furthermore, fine-tuning the hyper-parameters is essential for particular numerical examples. For instance, we set the training batch size to 32 and run the model for 500 epochs.

4. Jacobian-Based Shape Control

To perform shape control of a single section of a continuum robot, a Jacobian-based shape control scheme is employed. This method identifies the tendon lengths $\mathit{l}$ necessary for achieving the desired robot shape, parameterized by ${\kappa }_d$ and ${\varphi }_d$, using the Jacobian matrix that relates the velocities of the robot’s actuators to the velocities of its end-effector. The control system relies on feedback from a soft e-textile sensor that envelops the robot, providing real-time data from 16 sensing points, as depicted in Figure 6. These sensing points capture the deformations and strains experienced by the robot, which are crucial for determining its shape and movement.

To translate these sensor readings into actionable information, we utilized a CNN deep learning model. This model maps the 16-point readings from the e-textile sensor to the estimated curvature ($\widehat{\kappa }$) and angle of the curvature plane ($\widehat{\varphi }$). These estimations are essential for understanding the robot’s configuration and guiding its movements accurately. By continuously updating the Jacobian matrix with real-time feedback from the e-textile sensor, the control system can dynamically adjust the robot’s actuators, ensuring precise and smooth positioning. This approach allows the robot to adapt to various tasks and environments, maintaining high precision even in complex and unstructured settings.

According to the assumption of the constant curvature model [30], the parameters ($\kappa$) and ($\varphi$) of a continuum robot are related to the cable lengths $\mathit{l}={[l_1,l_2,l_3]}^T$ as follows:

$$\kappa =2{\displaystyle \frac{\sqrt{l_1^2+l_2^2+l_3^2-l_1{l}_2-l_2{l}_3-l_1{l}_3}}{d(l_1+l_2+l_3)}}$$

(2)

$$\varphi ={tan}^{-1}\left({\displaystyle \frac{\sqrt{3}{l}_3+l_2-2l_1}{l_2-l_3}}\right)$$

(3)

Thus, the time derivatives of the curvature and the angle of curvature plan are computed as

$$\left[\begin{array}{c}\dot{\kappa }\\ \dot{\varphi }\end{array}\right]=\mathit{J}\left(\mathit{l}\right)\dot{\mathit{l}}$$

(4)

where $\mathit{J}\left(\mathit{l}\right)\in {\mathbb{R}}^{2\times 3}$ is the Jacobian matrix computed numerically by differentiating Equations (2) and (3) with respect to $l_1,l_2,l_3$.

To eliminate any shift while pursuing the desired shape parameterized by ${\kappa }_d$ and ${\varphi }_d$, the Jacobian-based control selects the tendon lengths as follows:

$$\dot{\mathit{l}}={\mathit{J}}^{†}\left(\mathit{l}\right)\left(\left[\begin{array}{c}\dot{{\kappa }_d}\\ \dot{{\varphi }_d}\end{array}\right]+\mathit{K}\mathit{e}\right)$$

(5)

where ${\mathit{J}}^{†}\left(\mathit{l}\right)$ is the pseudoinverse of the Jacobian matrix, $\mathit{e}$ represents the error between the desired and actual shape parameters, and $\mathit{K}$ is a positive gain used to tune the controller’s performance. It is important to note that the actual shape parameters are estimated using the developed CNN shape sensing model in conjunction with the previously mentioned e-textile sensor. Since the robot’s tendons are driven by servo motors, the angle ${\theta }_m=f\left(\mathit{l}\right)$ of each servo is determined based on the mapping between the tendon length and the driving angle derived from the servo mechanism.

5. Results and Discussion

In this section, we examine the effectiveness of the proposed CNN-based method for shape sensing in continuum robots using a soft e-textile sensor. We start by assessing the ability of the e-textile sensor array to dynamically detect the robot’s shape alterations via a 4 × 4 grid of sensing points, as illustrated in Figure 7. The electrical resistance at each sensing point varies inversely with the pressure exerted by the robot’s deformation, providing a detailed insight into its structural modifications.

After evaluating the sensor, we focus on the performance of the CNN model. By carefully tracking the training and validation losses, we pinpoint key areas for improvement, enhancing the model’s robustness and effectiveness. The assessment further involves testing the model on a separate dataset to confirm its reliability and accuracy in predicting the robot’s curvature and the angle of the curvature plane.

Additionally, to determine the most effective model architecture, we utilized a 5-fold cross-validation technique across five different models. This thorough analysis provided a comprehensive understanding of each model’s strengths and weaknesses, helping us identify the most appropriate architecture for our application.

Finally, we assessed the performance of the feedback shape control for the continuum robot, which integrates Jacobian-based control with the CNN shape sensing model and the e-textile soft sensor.

5.1. e-Textile Sensor Responsiveness

This experiment examines the effectiveness of an e-textile sensor array in dynamically sensing the shape of a continuum robot using a 4 × 4 matrix of sensing points labeled from $A_0$ to $A_{15}$. Each point’s resistance inversely correlates with the pressure applied due to the robot’s deformation in terms of curvature $\kappa$. As shown in Figure 8, our results highlight the sensor’s sensitivity and accuracy in mapping curvature changes, offering valuable insights into the integration of soft robotics with the soft e-textile sensor.

The data indicate that resistance changes at the sensing points are not uniform but vary in a way that directly mirrors the robot’s deformation pattern. This variation implies that the e-textile sensor can not only detect the occurrence of bending but also distinguish its magnitude and directionality.

5.2. CNN Model Performance

The desktop TensorFlow 2.x API was used to perform the training and prediction tasks. The length of the training depends in large part on the choice of computer hardware. A CPU operating at 2.8 GHz was used in this study in order to speed up the model training procedure. The CNN model’s mean squared error (MSE) losses are shown in Figure 9 across 500 epochs, showing how they gradually approach minimum loss values for both the training and validation datasets.

To evaluate the models’ effectiveness, we used a testing dataset of 400 samples, processed in batches of 64. Figure 10 compares the CNN-based model’s predictions with the actual target values, highlighting the absolute error. Due to space constraints, only the first 100 samples are displayed. The minimal error margin of the CNN model is evidenced by the close alignment between the estimated outputs and the target signals.

Our approach of utilizing a deep learning model with a CNN architecture aims to provide a calibration-free method for sensing the robot’s shape parameters. The CNN is designed to correlate the deformation captured in the raw sensor readings, including any noise and disturbances, with the curvature measured by the IMU.

5.3. K-Fold Cross-Validation

In this subsection, we applied the K-Fold Cross-Validation technique, as detailed in [31], to a comprehensive evaluation of the suggested CNN model. This technique is essential for thoroughly evaluating the model’s shape-sensing abilities in a range of architectural layouts, guaranteeing dependability and efficacy. The dataset was split up into $k=5$ different subsets so that training and evaluating the model could be performed repeatedly. This thorough evaluation process offers insightful information on how well the model performs in various scenarios. Notably, this framework highlights the model’s applicability to real-world scenarios in addition to identifying the best architectural design.

In this work, we used feed-forward and convolutional neural network (CNN) architectures to create a set of five Deep Neural Network (DNN) models. These two- to three-layered models, which span a range of trainable parameters from 259 to 2515, are described in

Table 1. Selected architectures for the 5-fold cross-validation experiment.

Model	Layers	Size	Average MSE
1	$F_{16},F_3$	306	0.21
2	$C_{8,2},F_3$	186	0.28
3	$C_{8,2},C_{4,2},F_3$	206	0.26
4	$C_{16,2},C_{8,2},F_3$	666	0.16
5	$C_{32,2},C_{16,2},F_8,F_3$	2762	0.04

. Fully connected layers have an ‘F’ prefix and a subscript that indicates the number of neurones, whereas convolutional layers are indicated by a ‘C’ prefix followed by a subscript that indicates the size and number of kernels. Each and every convolutional layer uses a stride of 1 without any padding. All the layers employ the hyperbolic tangent (tanh) activation function, with the exception of the output layer, which uses a linear activation function. In models without a first convolutional layer, a flatten layer is included to transform the input posture image into a column vector so that it can be used with the network.

The Adam optimizer was used to train these models over 100 epochs, with the mean squared error (MSE) serving as the main loss metric. A 5-fold cross-validation approach was used to guarantee thorough examination and validation. The learning algorithm’s convergence trends across all network configurations and validation folds are shown in Figure 11a. Furthermore, a thorough examination of the average MSE losses and their standard deviations for every model architecture—with a particular emphasis on the CNN models—is shown in Figure 11b.

Notably, bigger networks tend to converge more successfully towards lower MSE values; networks with a greater parameter count are indicated in dark red. We selected a neural network that performed well in both the training and testing phases in order to keep things simple. The overall robustness of the system might be improved by investigating compact and quickly learning designs.

5.4. Performance of the Jacobian-Based Shape Control

An experiment was conducted to implement feedback curvature control using the proposed Jacobian-based controller. The trained shape sensing CNN model was integrated into the control scheme commanding the robot to reach the desired curvature and angle of curvature plane. The Arduino Uno was used to control the servo motors according to the obtained driving angles.

In Figure 12, the Jacobian-based shape control scheme is assessed for accurately reaching the desired ${\kappa }_d$ and ${\varphi }_d$. By selecting the gain $\mathit{K}=0.25$, the system initially exhibited instability with significant oscillations and prolonged response times, as shown in Figure 12a,b. As a result, the system needed tuning to improve performance. By adjusting the gain parameter, $\mathit{K}$, the system’s behavior was significantly enhanced, with setting $\mathit{K}=0.5$ resulting in a notable improvement in system response, as evidenced in Figure 12c,d. The system demonstrated stability and a more desirable response, indicating successful optimization of the system through gain tuning. In conclusion, meticulous design and tuning of the system parameters played a crucial role in achieving stable and efficient system performance. The iterative process of modeling, testing, and tuning the system parameters proved instrumental, highlighting the importance of the proposed control scheme.

5.5. End-Effector Reference Tracking

The primary objective of this experiment was to achieve accurate tracking of desired end-effector trajectories or positions in both the x- and y-directions to demonstrate the system’s overall performance. The reference trajectory was generated by moving the robot with predefined motor values while computing the robot’s tip position using the IMU sensor, as illustrated in [32].

The implemented system exhibited remarkable real-time tracking capability of the end-effector trajectory in both directions, as illustrated in Figure 13. This real-time tracking ability underscores the system’s responsiveness and effectiveness in following specified trajectories. Quantitative evaluation revealed Mean Absolute Errors (MAEs) of 0.25 and 0.13 for tracking the end-effector in the x- and y-directions, respectively. These low MAE values reflect the system’s accuracy and precision in maintaining the desired end-effector positions. The capability to accurately follow specified trajectories and maintain precise end-effector positions highlights the system’s robust design and suitability for tasks requiring intricate shape control. In conclusion, the system’s proficiency in real-time trajectory tracking, combined with its low MAE values and precise shape control capability, underscores its effectiveness and reliability in control applications demanding high precision and responsiveness.

6. Conclusions

In summary, this study introduces a novel soft e-textile resistive sensor paired with a deep convolutional neural network (CNN) specifically designed for real-time shape sensing in soft continuum robots. The CNN model is carefully crafted to handle the spatial continuities and inherent nonlinear characteristics of sensor data, significantly improving the ability to accurately detect complex shapes in soft continuum robotics. It demonstrates exceptional accuracy in determining the robot’s curvature and planar angle of curvature. A Jacobian-based shape control scheme is employed to ensure the attainment of target shape parameters and the following of specified trajectories. The system’s proficiency in real-time trajectory tracking, coupled with its low MAE values and precise shape control capability, highlights its effectiveness and reliability in control applications requiring high precision and responsiveness. Furthermore, the proposed Jacobian-based control for shape control of the continuum robot, utilizing the e-textile sensor and the CNN shape sensing model, showcases the system’s robust and adaptive nature, ensuring accurate and responsive control across various conditions.

We believe that our approach is scalable for sensing the robot’s curvature, and extending it to multiple sections is straightforward. This is particularly feasible because the e-textile sensor provides an estimation of the robot’s shape in terms of both curvature and the angle of the curvature plane. By stacking multiple sections of soft robots, each equipped with its own e-textile sensor, the curvature parameters of all sections can be estimated. Consequently, the overall shape of the robot can be derived using the forward kinematics equation. However, additional research is needed to extend this method to complex shape sensing, particularly involving variable curvatures of continuum robots. Future work could also focus on optimizing the e-textile sensor configurations to enhance sensitivity, durability, and response time. This includes investigating various materials, fabrication techniques, and sensor geometries.