*4.2. Human-in-the-Loop Haptics with ROBOTRAN*

The multibody approach described in Section 3 can be used to insert a highly dynamic model inside a real-time haptic device, which interacts with a human. Figure 7 shows the relationships between the ROBOTRAN model, the device, and the human body.

**Figure 7.** HIL haptic principle with ROBOTRAN. The light blue box, in this case, corresponds to the *Direct Dynamics* of Figure 4.

In haptics, taking the motion as input and the force as output, and vice versa, are the two possibilities, referred as impedance and admittance approaches in [41]. For the flow diagram shown in Figure 7, the device and the human are considered as an admittance operators, while the model can be assimilated to an impedance one. Other scenarios from Figure 4 are obviously possible.

The haptic device contains sensors that are used to measure the motion and to feed the multibody model. Those sensors are critical as they are in-the-loop located. Any error, noise or non-physic behavior will be reflected in the model and ultimately influence the feedback felt by the human who possesses his own somatosensory system, as discussed in Section 4.1.

Let us note of the importance of integration loop on the left in Figure 7, which must be fast, stable and precise enough. Indeed, the loop involving the human and the haptic device is instantaneous, while the integration loop is unavoidably slower, due to a fixed time-step size and thus a constant computation time.

#### *4.3. ROS-ROBOTRAN Coupling*

ROS (for "Robot Operating System") is a open-source software framework widely used in robotics. It contains numerous useful tools and libraries that allow for interactions between individual components, both in terms of software and hardware. ROS allows exploiting these tools in real-time to automate and control an electromechanical device.

Many ROS tools have been created by the ROS community to manipulate sensors or actuators and to publish or receive their data on a network. In addition, ROS offers many other benefits such as the ability to distribute the tasks requiring high processor usage across multiple hardware components. Furthermore, it makes possible to easily modify the system hardware without the need to make major changes in the software.

The coupling architecture between ROS and ROBOTRAN is presented in Figure 8.

**Figure 8.** Software architecture with ROS and ROBOTRAN.

Regarding the software, two packages constitute the basis of the coupling. In ROS, a *package* designates a folder containing one or several executable codes, a configuration file for compiling these, as well as other files useful to ROS. A ROS *node* is an executable that is able to process some computation but also to communicate with other nodes thanks to the ROS *messages* and *topics*.

As far as we are concerned, a ROS-ROBOTRAN package—called **ROSbotran**—has been created, that is able to publish/subscribe to the data of a ROS network. It contains the multibody model running in real-time, which allows to simulate the mechanical system and to forecast, analyze, and control its behavior. Inside this package, for our applications of Section 5, this node uses the multi-threads approach to parallelize the calculations.

Another package—called **ROScom**—has simply a listener–receiver role, containing two nodes: *Listener* and *Talker*. Two *topics* are used to exchange kinematic and dynamic information between the ROSbotran and ROScom packages, see Figure 8. The communication is ensured through the SSH protocol and physically via an ethernet cable.

Regarding the hardware implementation, the ROSbotran package is run on a laptop with a high-frequency *Intel Core i7-9700 CPU 3.6 GHz*. This allows to compute in real-time the multibody models, even if they are complex. An embedded processor—the *Odroid UX4*—hosts the ROScom package, which requires less computing power. This processor ensures the interface with the sensors and actuator via appropriate electronics.

This ROS-ROBOTRAN integration is already operational for several applications (see Section 5) but could be easily generalized to all types of models, sensors and actuators. Let us note that the real-time process considered in this work achieves so-called *soft* requirements. Indeed, the Linux OS used does not meet a *hard* real-time implementation. Future works should implement this approach to specific hardware and adapt the software accordingly.

#### **5. Applications**

Following the general concepts of Section 4, practical implementations have been designed in our laboratory for rather contrasting applications: a haptic piano key (Section 5.1) and a driving simulator (Section 5.2). Other realizations have also been conceived within the framework of ongoing researches dealing with specific road vehicles.

Several sensors have been utilized in these prototypes, mostly to be ran *in-the-loop*. However, some external sensors presented in this section have also been specifically designed to validate both the multibody model and the corresponding haptic device.

In this section, we take benefits of several sensors to both validate the multibody itself and the ability of the ROS-ROBOTRAN coupling to give a representative feedback to the human.

#### *5.1. Haptic Piano Key*

The *touch* of a grand piano mainly results from a dynamic transmission mechanism called *action*—which propels the hammer up to the string, given the key motion. The multibody model of Figure 9 has been developed at UCLouvain [42].

**Figure 9.** Main components of the grand piano action for the MBS model: mobile bodies (resp. other elements) are indicated by letters (resp. numbers). Circular arrows represent the DOFs (adapted from [43]).

This model finely represents the action kinematic and dynamic behavior. A first kinematic validation with an external sensor—a high-speed camera—shows that the hammer motion is well caught by the model, a little less well from the back-check capture occurring at the end of the motion, when the key is fully depressed [42].

#### 5.1.1. Previous Realizations

Several researchers have already worked to reproduce the haptic feeling of a piano keyboard thanks to active devices. In this context, the highly dynamic behavior of a piano action could clearly benefit from a multibody approach. Besides, it is difficult to reproduce its effect with only passive elements, as it is done in current digital piano for which manufacturers try to imitate the touch of a real acoustic piano.

In 1993, Gillespie presented the modeling of the piano action dynamics applied within an electromechanical apparatus [28]. The model is derived analytically and contains strong simplifications, which lack some very important dynamic features as the so-called action *escapement, Timmermans2020*. That being said, almost 30 years ago, using the technologies of that time, it was a first trial for a haptic piano key involving a dynamic model.

In 2002, Oboe developed the MIKEY keyboard [44], which allows one to simulate the type of touch of three instruments: the grand piano, the harpsichord, and the Hammond organ. He improved the grand piano action model from Gillespie's by managing the escapement phase. The considered dynamics still remains simplified compared to the complexity of the piano action. As Gillespie, the experimental prototype uses position sensors and voice coil motors.

Lozada proposed a different approach in 2007 [45], using a magneto-rheological system. He has derived an analytical model of both the physical behavior of the haptic device and the piano action. The key motion is sensed by a contactless photoelectric sensor, and the actuator needs also a force sensing. The final design differs quite significantly from a classic piano key.

With the same purpose to create a haptic piano key, Horvàth proposed in 2014 [46] an experimental fitting of a simplified model with only four constant parameters. He used force and optical sensor to relate the torque acting on the key under different impulses. Unfortunately, no further works has been published by the author to show the practical implementation inside a haptic device.

A broader platform called GENESIS-RT has been introduced by Leonard in 2015 [47] whose goal is to reproduce, among others, the piano action feedback. The piano model is based on a piano-inspired mechanics. The sensors and real-time update rates are quite high, 44 kHz and 4.41 kHz, respectively. No experimental results are shown with the device.

More recently in 2019, Adamou has suggested a two steps approach in [48]. First, an original measurement device has been used to characterize the force feedback according to the key position and velocity. Then, a replication device has been build with an electromagnetic actuator combined with a spring. A laser sensor is placed under the key to measure its position. Again, no validation with the proposed design is proposed.

Those previous attempts have used relatively few sensors, such as optical position sensors or force sensors. The goal was of course to limit the complexity and the cost of such devices. From our perspective, in a purely research context, using multiple sensors of high quality could help in quantifying the haptic feedback quality and prove the approach feasibility. Afterwards, one can imagine to simplify the design complexity for a possible industrial application. For instance, sensors could be integrated in the actuator itself, sensing the position through sensorless methods [49–51].

Using the multibody approach described in Section 3, a haptic piano key has been recently developed in our laboratory, with an experimental validation of its force feedback [30]. This implementation is described in the next section. Some improvements brought to the device, as well as further experimental validation are the subject of the subsequent Sections 5.1.3–5.1.5.

#### 5.1.2. Mechatronic Implementation

For the pianists, the force feedback principle aims at reproducing the same *touch* sensation, i.e., Fhaptic in Figure 10. An active device reproduces the force Fact on the key pilot, based on the reference force Fmod computed by the multibody model.

Kinematic sensors measure in real-time the key angular position and velocity, which are inputs of the multibody model, as shown in Figure 7.

Our prototype (Figure 11) consists of a linear actuator rigidly attached to a piano key. An embedded controller computes the model in real-time (i.e., in less than 1 ms), while the electronic boards drive the actuator accordingly. More details about the mechatronic design are given in [30].

**Figure 10.** Piano action multibody model force feedback principle, adapted from the work in [30].

**Figure 11.** Mechatronic design of one key haptic feedback device, enhanced from the work in [52].

#### 5.1.3. Model Validation

The model output force Fmod should render the force at the key pilot that would cause a force Fhaptic equivalent to that of an acoustic piano keyboard. A custom force sensor has been developed that is based on strain gauges. The sensor sensitivity is 0.765 mV/V, for a measuring range from 0 up to 50 N, with a sampling frequency of 50 kHz, and with a resolution below the milliNewton. The designed load cell of Figure 12 will be inserted instead of the key pilot number 3 in Figure 9 to capture the strain so that the sensor can measure the force applied by the whippen to the key pilot. The latter is exactly the force Fact of Figure 10 that the actuator has to apply in the haptic device. This allows us to validate the output force of the multibody model, see Figure 7.

The experimental sensor is inserted in a real piano action demonstrator, a Renner® action (Figure 13). This homemade binocular element contains four strain gauges glued to the sides, connected through a Wheatstone bridge. This setup allows to precisely measure the normal force between the whippen and the key pilot, without influencing the action dynamic behavior.

**Figure 12.** Load cell design.

**Figure 13.** Experimental homemade force sensor set-up.

The key is actuated by an external actuator shown at the right of Figure 13. A highspeed camera at 1000 fps uses the markers to retrieve the action kinematics. The key motion is then supplied as a model input. This allows to validate the model dynamics, considering the measured motion as the reference, keeping in mind the limitations of the sensing method.

In Figure 14, a first experiment shows the hammer behavior, compared to the modeling and the measured angle with the high speed camera. Hammer angles are very close in case of this relatively slow double keystroke.

**Figure 14.** Piano action model dynamic validation: hammer joint position (slow double keystroke).

Results of the measured force versus the offline model-simulated force are shown in Figure 15. A double blow pattern is visible, the first key strike starting around 0.3 s and the second around 1.5 s. In both cases, the force starts by increasing, with oscillations, until a quasi steady-state value close to 3 N. Then, when the key is released—around 0.9 s and 2.1 s—the action returns to its resting position, with forces peaks due to the hammer rebound, whose mass and inertia dynamic contributions are relatively high compared to other elements.

**Figure 15.** Piano action model dynamic validation: force (slow double keystroke).

In Figure 15, the simulation presents more oscillations all along the curve and shows higher force peaks. Apart from that, the measured force is very similar, which is encouraging for the model quality and its ability to finely reproduce the dynamic behavior of a piano action.

A faster double blow is experimented in Figures 16 and 17.

The hammer behavior is not identical between the simulation and the experiments after the hammer–string contact in Figure 16. In fact, the repetition lever acts differently: in experiments, the lever moves the hammer upwards around [0.25;0.35] s and [0.7;0.8] s, while the simulated behavior is different. Indeed, the repetition lever is difficult to tune both in practice and in simulation.

Despite that, the forces of Figure 17 are very similar, apart from the first part of the strokes, around 0.1 s and 0.5 s. Being a fast double blow, those differences are acceptable, and these results show rather good agreements with experiments, given all the remaining model parameter uncertainties.

**Figure 16.** Piano action model dynamic validation: hammer joint position (fast double keystroke).

**Figure 17.** Piano action model dynamic validation: force (fast double keystroke).

5.1.4. Real-Time Sensors Validation

The kinematic input supplied to the multibody model, see Figure 7, is of the utmost importance for the quality of the haptic feedback. Noise, oscillations, or error in the input would unavoidably result in a non-realistic force computed by the multibody model.

In the piano key haptic device of Figure 11, two position sensors and a velocity sensor capture the key kinematics to continuously drive the multibody simulation.

The key angle is retrieved through two Hall effect *Allegro A1301* sensors placed in opposition under the key, one at the front and one at the rear, as presented in Figure 11. These two sensors measure respectively the voltages Ufront and Urear.

Hall sensors use magnetic field measurements which are not linear versus displacement. However, a chosen combination Ucombined between Ufront and Urear allow obtaining experimentally a linear relation between the measurements and the key front height:

$$\mathbf{U\_{combined}} = k \cdot (\mathbf{U\_{front}^2} - \mathbf{U\_{near}^2}) + \mathbf{O} \tag{17}$$

where *k* is a proportionality factor determined by calibration and *O* is a fixed offset, also determined by experiments. The Ucombined can then be converted to a position measurement and finally to the key angle via simple trigonometry.

The velocity sensor of Figure 11 is an homemade voice coil, with a copper coil fixed on the frame and a moving permanent magnet glued on the key. The motion of the magnet creates a magnetic field variation inside the coil, inducing a varying *back-electromotive* force in the coil. This way, a voltage Ucoil is created between the coil wire edges, whose value is directly proportional to the speed of the coil:

$$\mathbf{U}\_{\rm coil} = \boldsymbol{\kappa} \ \dot{\boldsymbol{z}}\tag{18}$$

where *κ* is a proportionality factor found by experiment and linked to the coil characteristics and *z*˙ is the moving magnet vertical velocity. This velocity is considered as purely vertical because the key angle remains small.

To validate those position and velocity sensors, an external *Polytec OFV-534* vibrometer is used, as presented in Figure 18. This vibrometer independently measures the position and the velocity with a resolution of 175 μm and 4.4 mm/s in our setup.

**Figure 18.** Piano haptic key: kinematics sensing validation.

First, the position estimation is experimented by manually applying up and down movements to the key, as shown in Figure 19.

**Figure 19.** Haptic key validation of position sensors: slow motion.

The Hall effect real-time measured position is close to the reference obtained with the vibrometer. The right graph in Figure 19 shows the relation between measured and reference positions.

The experiment of Figure 19 includes a motion at slow frequency, less than 1 Hz. Performing the exact the process at a higher input frequency, around 5 Hz, provides the results shown in Figure 20. The position sensing is still consistent. However, discrepancies appear that cause the Hall effect measurement to have a maximum error of 10% between the estimation and the reality, which could be enhanced, but is sufficient for the considered application.

Furthermore, the combination of (17) allows to maintain the sensor sensitivity, i.e., the slope of the right curves in Figures 19 and 20. Indeed, the relation between the measured position and the real one of Figure 19 has almost a constant slope, except for a small quadratic deviation at the lower and upper position between [0;1] and [9;10] mm, due to the magnetic field quadratic relation with the position. This does not impact the haptic prototype behavior but it can be improved in the future by enhancing the position sensing via a better calibration.

**Figure 20.** Haptic key validation of position sensors: fast motion.

Regarding the velocity, several up and down motions are again manually applied to the key, and the homemade coil sensor output is compared to the vibrometer measurements in Figure 21.

The homemade sensor captures the velocity amplitude quite well, despite some erroneous values at some points. Indeed, some peaks are present with the homemade but not with the vibrometer, for instance, around 5.5 and 6.7 s in Figure 21, left.

Figure 22 shows the same experiment for *halfway* keystroke, for which the key is not fully depressed but the action is still able to repeat the note, as foreseen by the socalled *double escapement* grand piano action: this feature is fundamental for pianists. The validation of Figure 22 presents our sensor capability to capture this specific movement.

**Figure 21.** Haptic key validation of the velocity sensor: high amplitude, fast motion.

**Figure 22.** Haptic key validation of the velocity sensor: halfway keystroke.
