Wearable Wireless Biosensors for Spatiotemporal Grip Force Profiling in Real Time ^†

Abstract

The temporal evolution of individual grip force profiles of a novice using a robotic system for minimally invasive endoscopic surgery is analyzed on the basis of thousands of individual sensor data recorded in real time through a wearable wireless sensor glove system. The spatio-temporal grip force profiles from specific sensor locations in the dominant hand performing a four-step pick-and-drop simulator task reveal skill-relevant differences in force deployment by the small finger (fine grip force control) and the middle finger (gross grip force contribution) by comparison with the profiles of a highly proficient expert. Cross-disciplinary insights from systems neuroscience, cognitive behavioral science, and robotics, with implications for biologically inspired AI for human–robot interactions, highlight the functional significance of spatio-temporal grip force profiling.

1. Introduction

Wearable wireless sensor systems are currently being developed for the monitoring of health data, exercise activities, and other performance data. Unlike conventional approaches, such devices enable the convenient, continuous, and unobtrusive monitoring of a user’s behavioral signals in real time. In previous work [1,2], we have used a wireless sensor glove system for human–robot interactions within a system prototype designed for minimally invasive endoscopic surgery. Functionally motivated analyses of thousands of human grip force data, collected from various sensor locations in the dominant and non-dominant hands of experts, trainees, and novice surgeons in image-guided task simulations, have shown specific differences between the grip force profiles of individual users as a function of task skill level and expertise in using the robotic system. Experts and non-experts employ different grip-force strategies reflected by differences in total amount of grip force deployed by the same fingers of the same hand. In this work, we show further functional analyses of thousands of individual grip force data leading the way towards task-specific grip force monitoring in real time, as clarified in the discussion further below. Task specific sensor data from a complete novice and a highly proficient expert user are exploited here in terms of spatio-temporal variations in average grip force, computed on the basis of data recorded from three task-relevant sensor locations in the dominant hand: (1) the middle phalanx of the middle finger, which mostly contributes to gross force deployment [3,4,5], necessary for lifting weighty objects but useless for skillful execution of the robotic precision task here; (2) the middle phalanx of the small finger, which allows for fine grip force control [3,4,5,6], critically important in subtle precision grip tasks like the one studied here; and (3) the middle phalanx of the ring finger, which is among the least important in grip force control across a variety of tasks and weights lifted, as discussed previously [3,4,5,6]. The spatio-temporal grip force profiles of the novice and the expert are compared across ten successive task sessions for repeated execution of the pick-and-drop robotic simulator task with four critical steps:

• Activate and move tool towards object location; requires movement ahead in depth along a virtual z-axis in 2D image plane;
• Open and close tool-tip grippers to grasp and lift object;
• Move tool with object to target location; requires lateral movement along x-axis of 2D image plane;
• Open tool-tip grippers to drop object in box.

To accomplish these steps in minimal task time and with maximal precision (i.e., no tool trajectory corrections, no incidents) requires skillful manipulation of the task-relevant (left or right) handle of the robotic system (see Materials and Methods). The novice’s dominant hand is the right hand, and the expert’s dominant hand is the left hand.

2. Materials and Methods

The robotic system is designed for bi-manual endoscopic surgical interventions; the simulator task from this study here solicits only one, in the present case the dominant hand.

2.1. Slave Robotic System

The slave robotic system is built on the Anubis® platform of Karl Storz and consists of three flexible, cable-driven sub-systems for robot-assisted endoscopic surgery, described in detail in our previous work [1] available at https://www.mdpi.com/1424-8220/19/20/4575/htm.

2.2. Master/Slave Control

The slave robot is controlled at the joint level by a position loop running at 1000 Hz on a central controller. The master side consists of two specially designed interfaces, which are passive mobile mechanical systems. Each of the two handles has 3 DoF, which translate for controlling instrument insertion, rotate around a horizontal axis for controlling instrument rotation, and rotate around a final axis (moving with the previous DoF) for controlling instrument bending. Each handle is also equipped with a trigger and with a small four-way joystick for controlling additional DoF. In the experiments here, the trigger is operated with the index finger of a given hand for controlling grasper opening and closing.

A high-level controller running in real-time Linux OS communicates with the master interfaces and provides reference positions to the slave central controller. The user sits in front of the master console and looks at the endoscopic camera view displayed on the screen in front of him/her at a distance of about 80 cm while holding the two master handles, which are about 50 cm away from each other. Seat and screen heights are adjustable to optimal individual comfort. The two master interfaces are identical, and the two slave instruments they control are also identical. Therefore, for a given task the same movements need to be produced, whatever hand the user is instructed to use (left or right). The master interfaces are statically balanced, and all joints exhibit low friction; therefore, only minimal forces are required to produce significant movement in any direction. A snapshot view of a user wearing the sensor gloves while manipulating the system is shown in Figure 1a; Figure 1b shows directions and types of tool-tip and control movements.

2.3. Sensor Glove Design

The robotic system, having its own grip style design, a specific wearable sensor glove system with inbuilt Force Sensitive Resistors (FSR), was developed, one for each hand.

2.3.1. Hardware

The gloves designed for the study each contain twelve FSR, in contact with twelve specific locations on the inner surface of the hand [1,2], including those three for which data are shown here. Two layers of cloth were used, and the FSR were inserted between the layers. The FSR were sewn to the cloth around the conducting surfaces (active areas) and did not interact, neither directly with the skin of the subject, nor with the master handles, which provided a comfortable feel when manipulating the system. Regarding FSR with a needle and thread, the electrical connections of the sensors are individually routed to the dorsal side of the hand and brought to a soft ribbon cable, connected to a small and very light electrical casing, strapped onto the upper part of the forearm and equipped with an Arduino microcontroller. Each FSR was soldered to 10 KΩ pull-down resistors to create a voltage divider, and the voltage read by the analog input of the Arduino is given by (1)

𝑉𝑜𝑢𝑡 = 𝑅𝑃𝐷𝑉3.3/(𝑅𝑃𝐷 + 𝑅𝐹𝑆𝑅)

(1)

where 𝑅_𝑃𝐷 is the resistance of the pull-down resistor, 𝑅_𝐹𝑆𝑅 is the FSR resistance, and 𝑉_3.3 is the 3.3 V supply voltage. FSR resistances can vary from 250 Ω when subject to 20 Newton (N) to more than 10 MΩ when no force is applied at all. The generated voltage varies monotonically between 0 and 3.22 Volt, as a function of the force applied, which is assumed uniform on the sensor surface. In the experiments here, forces applied did not exceed 10N, and voltages varied within the range of [0; 1500] mV. The relationship between force and voltage is almost linear within this range. It was ensured that all sensors have similar calibration curves. Thus, all following comparisons are directly between voltage levels at the millivolt (mV) scale. Regulated 3.3 V is provided to the sensors from the Arduino, and power is provided by a 4.2 V Li-Po battery enabling wireless use of the glove system. The battery voltage level is controlled during the whole duration of the experiments by the Arduino, and displayed continuously on the user interface. The glove system is connected to a computer for data storage via Bluetooth-enabled wireless communication running 115,200 bits-per-second (bps).

2.3.2. Software

The software of the glove system is divided into two parts: one running on the gloves, and one running on the computer algorithm for data collection. Each of the two gloves sends data to the computer separately, and the software reads the input values and stores them on the computer according to header values indicating their origin. The software running on the Arduino is designed to acquire analog voltages provided by the FSR every 20 milliseconds (msec) at a 50 Hz rate. In every loop, input voltages are merged with their time stamps and sensor identification. This data package is sent to the computer via Bluetooth.

3. Results

The raw data from three (see

Table 1. Task-relevant sensor locations (dominant hand) from which the spatio-temporal grip force profiles shown here were drawn.

Sensor	Finger	Anatomical Reference
S5	Middle	Middle Phalanx
S6	Ring	Middle Phalanx
S7	Pinky	Middle Phalanx

) of the twelve sensor locations in the dominant hands of the novice and the expert were analyzed.

Individual temporal grip force profiles were plotted in terms of average peak amplitudes (AmV) for fixed successive temporal windows of 2000 ms in a given individual session. With one signal per 20 ms and 100 signals per time window of 2000 ms, we have AmV = mVtotal/100, which is the total sum of mV recorded in the time window given divided by the total number of signals in the time window. Figure 2 shows these profiles for the first and the last individual sessions for sensors 5, 6, and 7.

Statistical comparison (2-Way ANOVA) between the original raw data of expert and novice from their first and last task sessions reveal significant interactions between the “expertise” (2 levels) and “session” (2 levels) factors for all three sensors considered here (S5, S6, and S7). For sensor S5 on the middle finger (gross grip force deployment), the mean (m) grip forces and their standard errors (sem) from the first session yield m = 241 mV/sem = 4.3 for the expert and m = 790 mV/sem = 2.7 for the novice, showing that the latter deploys about three times as much unnecessary gross grip force by comparison with the expert. This expertise-specific difference in proportional gross grip force deployed by the middle finger is even larger in the last session, with m = 78 mV/sem = 4.9 for the expert, and m = 640 mV/sem = 3.6 for the novice. The interaction between the “expertise” and ‘’session’’ factors for sensor S5 is significant with F(1,2880) = 28.65; p < 0.001. For sensor S6 on the ring finger (no meaningful role in grip force control), the differences between the grip force profiles of novice and expert are minimal, as would be expected, in the first session with m = 576 mV/sem = 3.8 for the expert and m = 504 mV/sem = 2.4 for the novice, and in the last session with m = 474 mV/sem = 4.5 for the expert and m = 445 mV/sem = 3.3 for the novice. The interaction between the “expertise” and “session” factors for sensor S6 is, however, significant with F(1,2880) = 35.86; p < 0.001, which is explained by the fact that grip forces, i.e., amplitudes in mV, diminish in both users from the first to the last session, but not by the same amounts. For sensor S7 on the small finger (critically important for fine grip force control), the expertise-specific difference between the two user profiles is characterized by the novice deploying largely insufficient grip forces, from the first session with with m = 98 mV/sem = 1.2 to the last with m = 78 mV/sem = 1.6, while the expert produces sufficient grip force for fine movement control from the first session with m = 594 mV/sem = 1.8 to the last with m = 609 mV/sem = 2.2. The interaction between the “expertise” and “session” factors for sensor S7 is highly significant with F(1,2880) = 188.53; p < 0.001. Task times are considerably shorter in the expert (Figure 2), with 10.4 s in the first session and 7.6 s in the last indicating a minor practice effect. The novice takes more than twice as long (26 s) in the first session compared with the expert, with a 30% time gain in the last session (18.8 s), indicating a temporal training effect. Concerning incidents (trajectory adjustments, grip failures, and drop misses), the task videos reveal a total number of 20 in the novice and only three small trajectory adjustments in the expert’s performance.

4. Discussion

The data reveal expertise-specific differences in the spatio-temporal grip force profiles of an expert and a novice repeatedly performing a 2D image-guided robot-assisted precision task. These differences can be functionally assessed under the light of previous work [3,4,5,6] on the role of finger grip forces and prehensile synergies, centrally controlled in the human brain for optimizing human motor performance and control. One major difference as a function of task expertise or skill concerns proportional gross grip force deployed by the middle finger, with the novice deploying way too much unnecessary, task-irrelevant gross grip force, while the expert has learnt to skillfully minimize those. Another functionally important, expertise-specific difference concerns precision grip force control by the small finger, critically important in surgical and other precision tasks. Here, the difference between the two users is characterized by the novice deploying insufficient grip forces, with no major evolution between the first and the last task sessions. Concerning grip forces deployed by the ring finger, which plays no major or meaningful role in grip force control, differences between grip force profiles of novice and expert are, indeed and as would be expected, minimal and do not change much across sessions although total grip force magnitudes (mV amplitude) diminish across sessions in both users. At the beginning, the novice takes more than twice as long performing the precision task by comparison with the expert, but at the end scores a 30% time gain, indicating a considerable temporal training effect, especially in the first critical task step, as indicated by the pink boxes in Figure 2. This is readily explained by the fact that the first task step is the most difficult for a novice. It consists of moving the tool along a virtual trajectory towards the object location, and requires movement away from the body in the surgeon’s peri-personal space. This implies the perceptual recovery of physically missing depth information along a virtual z-axis in the 2D image plane. Such difficulty results in longer task times and imprecise tool-movements [7,8,9,10], explaining why the largest training gain in total task time of the novice is observed for task step 1. Finally, the analyses shown here can be run in real-time to monitor manual/bimanual precision tasks, control performance quality, or prevent risks in robot assisted surgery for systems where excessive grip forces can cause tissue damage [11], which does not apply to this system here. The human hand has evolved as a function of active constraints in harmony with other sensory systems [12,13,14]. Grip force profiles are a direct reflection of the complex cognitive and behavioral synergies this evolution has produced.