Multi-Modal Haptic Rendering Based on Genetic Algorithm

Abstract

Multi-modal haptic rendering is an important research direction to improve realism in haptic rendering. It can produce various mechanical stimuli that render multiple perceptions, such as hardness and roughness. This paper proposes a multi-modal haptic rendering method based on a genetic algorithm (GA), which generates force and vibration stimuli of haptic actuators according to the user’s target hardness and roughness. The work utilizes a back propagation (BP) neural network to implement the perception model f that establishes the mapping $(I=f(G\left)\right)$ from objective stimuli features G to perception intensities I. We use the perception model to design the fitness function of GA and set physically achievable constraints in fitness calculation. The perception model is transformed into the force/vibration control model by GA. Finally, we conducted realism evaluation experiments between real and virtual samples under single or multi-mode haptic rendering, where subjects scored 0-100. The average score was 70.86 for multi-modal haptic rendering compared with 57.81 for hardness rendering and 50.23 for roughness rendering, which proved that the multi-modal haptic rendering is more realistic than the single mode. Based on the work, our method can be applied to render objects in more perceptual dimensions, not only limited to hardness and roughness. It has significant implications for multi-modal haptic rendering.

1. Introduction

Haptic rendering aims to generate realistic virtual objects that can be felt as well as seen [1]. The tactile sensation of the object is the result of the coupling of multiple perceptual dimensions, such as roughness, compliance, coldness, and slipperiness [2]. Multi-modal haptic rendering can simulate a more natural sense of touch by targeting multiple mechanoreceptors in the skin and muscles to simultaneously transmit tactile and proprioceptive information [3]. Some work in haptic rendering tends to render one perceptual dimension, such as hardness. The rendering features are affected mainly by one perceptual dimension without considering the influence of other dimensions. According to researchers, it is crucial for haptic rendering to display multiple perception dimensions of objects [1].

The typical rendering algorithm for matching object hardness uses Hooke’s law to perform haptic rendering on a force feedback device, taking the stiffnes of the rendering object as a constant [4]. It is challenging to render objects with high hardness due to the limited rendering range of the force feedback device. Okamura enhanced hardness perception through vibrotactile stimulation by adding a decaying sinusoidal vibration waveform to a force feedback device [5]. A method of stiffness transfer has been suggested by researchers [6]. The initial stiffness was low at the time of contact, but quickly increased to a larger value, increasing the perceived hardness.

The data-driven algorithm is the representative rendering algorithm to display the roughness of texture, which is to record the data during the interaction with the real objects for haptic rendering. Kuchenbecker recorded data when the tool was in contact with the texture, generating vibration signals in response to velocity and force signals, and the vibration was encoded as autoregressive (AR) coefficients [7]. Abdulali proposed an AR acceleration storage and interpolation model based on a radial basis function network (RBFN), which was used to render anisotropic texture [8]. Osgouei proposed an inverse dynamics model to generate actuation signals using nonlinear autoregressive neural networks with external input to mimic real textures on an electrovibration display. Each network was trained under different experimental conditions using the lateral frictional force generated by applying pseudo-random binary signals [9,10]. Although the data-driven method can render fine textures, it might be over-sophisticated for rendering coarse textures [11]. Additionally, the information about hardness cannot be effectively expressed [12].

With the in-depth research on the characteristics of human tactile perception, some researchers adopt deep learning and bio-inspired models to establish a perception model between haptic data and human tactile perception. Gao proposed convolutional neural network (CNN) and long short-term memory (LSTM) to establish the relationship between tactile information obtained from BioTac and 24 haptic adjectives [13]. Ouyang proposed to use the impedance network structure to simulate the cutaneous afferent population responses (CAPR) in the peripheral nervous system to obtain tactile sensation. Ouyang utilized the CAPR model to present vibration and force feedback to help surgeons localize underlying structures in phantom tissue [14].

The research and modeling of tactile perception provide good theoretical and technical support for multi-modal haptic rendering. We can convert the perception model into the control model to obtain the control parameters of target perception intensities. This paper proposes a multi-modal haptic rendering method based on genetic algorithm (GA), which generates both force and vibration stimuli of haptic actuators according to the user’s target hardness and roughness. Back propagation (BP) neural network can be used to learn a large number of input–output mappings without prior disclosure of the mathematical equations describing these mappings [15]. We employ BP neural network to establish the perception model f that describes the mapping $(\mathit{I}=f(\mathit{G}\left)\right)$ from objective stimuli features $\mathit{G}$ to perception intensities $\mathit{I}$. We utilize the perception model f to design fitness function of GA and set physically achievable constraints in fitness calculation. GA optimizes the feature vector $\mathit{G}$, which is input into the perception model in fitness calculation to obtain an output close to the perception intensities of the target rendering sample. The perception model is transformed into the force/vibration control model by GA. A set of haptic actuators that combine force and vibration is proposed to display the hardness and roughness of the object. The multi-modal haptic rendering framework is shown in Figure 1. In the experimental verification and analysis, subjects compared the realism under the multi-modal/single rendering mode by the realism rating experiment. We analyzed the correlation between the features obtained with GA and $\mathit{I}$. The results proved that the multi-modal haptic rendering is more realistic than the single mode.

2. Data Collection and Perception Modeling

Data collection is required to build the objective-to-subjective perception model. Firstly, data collection and feature extraction were carried out under specified collection actions. Then, psychophysical experiments for hardness and roughness were performed separately to obtain subjective data. Finally, we exploited the nonlinear fitting ability of BP neural network to establish the perception model f that describes the mapping $(\mathit{I}=f(\mathit{G}\left)\right)$ from objective stimuli features $\mathit{G}$ to perception intensities $\mathit{I}$.

2.1. Objective Data Collection and Feature Extraction

Twenty-three samples (S1–S23) were made by combining different substrate materials and textures, for example soft sponge and leather 1 made up sample 1 (S1) in Figure 2. The textures included leather and fabric, and the substrate materials included sponge, EVA (ethylene vinyl acetate) plastic, rubber, and foam to ensure that the samples had a certain degree of difference in hardness and roughness. In this paper, for the properties of hardness and roughness, two interactive actions of pressing and lateral motion are selected for data acquisition [16].

As shown in Figure 3a, a three-axis accelerometer (Analog Devices ADXL345) is mounted in the front of the wearable fingertip device. A force sensor (Honeywell FSS1500) is mounted at the bottom of the wearable fingertip device. The weight of the wearable fingertip device is 37.2 g. The accelerometer was configured with a dynamic range of ±16 g and 10-bit resolution and connected to the STM32 microcontroller at the wrist over the I2C bus. The force sensor data were obtained with the Analog Digital Converter (ADC, 12 bits) of the microcontroller. We used serial communication to enable the computer to communicate with the STM32 microcontroller to transmit control commands and receive sensory signals. The SensAble Phantom Omni device can sense position, velocity in the x, y, z direction [17]. Displacement data were collected through Omni. Through a guide ball of the Labview interface, the speed of the lateral motion was limited to about 80 mm/s. We collected data for each sample 10 times during the pressing and lateral motion. The collected data were stored in the PC at the acquisition frequency of 500 Hz through Labview platform. Then the force and displacement data were lowpass filtered through a Butterworth filter with a 10 Hz cutoff frequency which was above the normal hand motion bandwidth of 2 Hz to avoid significant delay or attenuation of deliberate human motions [18]. The acceleration data were highpass filtered through a Butterworth filter with a 20 Hz cutoff frequency in order to remove a gravity component and noise [8].

After data collection, we extracted five features related to hardness and roughness in the interation data as vector $\mathit{G}$. In terms of hardness feature extraction, as in [19,20], the global stiffness $\left(GS\right)$, and work $\left(W\right)$ during the pressing process were extracted, as shown in Figure 4a. In terms of roughness feature extraction, Yoshioka and Culbertson proposed that vibration power has a high correlation with roughness [21,22]. We focused on the acceleration signal of the lateral motion direction of the three-axis acceleration, so the lateral vibration power $\left(P\right)$ during the lateral sliding process was extracted. As in [23], we extracted the peak frequency $\left(F_{peak}\right)$ and its amplitude $\left(A_{Fpeak}\right)$ after the FFT of the acceleration signal, as shown in Figure 4b. The means and standard deviations of twenty-three samples’ features are shown in Figure 5.

$$\mathit{G}=[GS,W,F_{peak},A_{Fpeak},P]$$

(1)

2.2. Subjective Data Collection

Psychophysical experiments for hardness and roughness were performed separately to obtain subjective data. We recruited 10 participants from Southeast University aged on average 25. All of them were right-handed, and reported having no cutaneous or kinesthetic problems. Two adjective rating experiments were performed on 23 samples using the fingertip device in Figure 3.

Process

Participants wore an eye mask and headphones to shield visual and auditory information from distractions. The samples were presented to the participants in an unordered order. In the first experiment, participants were required to rate the hardness of the samples on a scale of 0–100, where 100 meant “hardest” and 0 meant “softest”. In the second experiment, participants were required to rate the roughness of the samples on a scale of 0–100, where 100 meant “roughest” and 0 meant “smoothest”.

Results

The results of subjective data collection were normalized, as shown in Figure 6. Among the 23 samples, S12 has the hardest hardness score, S1 has the softest score, S7 has the roughest roughness score, and S12 has the smoothest score.

2.3. Perception Modeling

BP neural network was applied using the Levenberg–Marquardt (LM) algorithm to establish the nonlinear perception model. As shown in Figure 1a, we selected features $\mathit{G}$ as the input layer parameters, and the hardness and roughness intensities as the output layer parameters. The training was progressed with the MATLAB neural network toolbox. A total of 70% of the total samples were used for training, 15% for validation, and 15% for testing. The training parameter settings are shown in

Table 1. Training parameter settings.

Parameter	Value
Learning rate	0.01
Neural numbers in hidden layer	20
Training function	Trainlm
Hidden layer function	Tansig
Output layer function	Tansig

.

To evaluate the BP model, we use mean square error (MSE) and correlation coefficient (R) as evaluation indicators. Higher prediction accuracy can be represented by smaller MSE value and larger R value. Below are the formulas for MSE and R.

$$MSE=\frac{1}{n}\sum _{i=1}^n{(\widehat{y_i}-y_i)}^2$$

(2)

$$R=\frac{n{\sum }_{i=1}^n{y}_i{\widehat{y}}_i-{\sum }_{i=1}^n{y}_i{\sum }_{i=1}^n{\widehat{y}}_i}{\sqrt{n{\sum }_{i=1}^n{y_i}^2-{\left({\sum }_{i=1}^n{y}_i\right)}^2}\sqrt{n{\sum }_{i=1}^n{{\widehat{y}}_i}^2-{\left({\sum }_{i=1}^n{\widehat{y}}_i\right)}^2}}$$

(3)

where n is the number of tested instances, ${\widehat{y}}_i$ is the predicted value estimated by the training network for instance i, and $y_i$ is the target value of instance i. The MSE and R of the BP model is shown in

Table 2. Evaluation indicators of BP model.

Evaluation Indicators	Output1 Hardness	Output2 Roughness
MSE	0.0042	0.0073
R	0.9715	0.9343

. As shown in Figure 7, the Hardness MSE of S1 is the smallest among the 23 samples, and the Hardness MSE of S17 is the largest. Among the 23 samples, S12 has the smallest Roughness MSE, and S19 has the largest Roughness MSE.

3. Methods

The problem of GA solving can be regarded as the inverse model of the perception model. GA is utilized to obtain the sample’s control characteristics after determining the target sample’s hardness and roughness. We design the fitness function of GA based on the perceptual model and set two physically achievable constraints to limit the fitness calculation. The features of the best individual obtained from GA are transformed into the control features of the target sample.

3.1. Genetic Algorithm

GA was first proposed by John Holland in 1975. The basic idea is that the optimal solution is obtained by simulating the genetic selection process based on evolution theory [24]. The perception model based on BP network is used as the fitness function of GA. We run the algorithm using MATLAB with the settings reported in

Table 3. Settings for the genetic algorithm.

Parameter	Value
Population Size	100
Termination Generations	50
Selection Function	Tournament, 2
Crossover Function	Single-point
Crossover Friction	0.6
Mutation Function	Uniform
Mutation Friction	0.2

. As shown in Figure 1b, GA mainly consists of the following steps:

•

Chromosome encoding

•

Fitness Calculation

•

Chromosome Manipulation

•

Termination

First, the individual chromosome is encoded as vector $\mathit{G}$ (real encoding) as shown in Formula (1) and generates an initial population. The chromosomal genes of the individual are randomly generated within a specific interval that depends on the range of the extracted features in Section 2. Then, GA can evaluate the strength of an individual $\mathit{G}$ by the fitness function. The lower the fitness value of the individual, the better the individual. Excellent individuals are likely to be passed on to the next generation.

In order to obtain the features that meet the target sample’s hardness and roughness, we set two physically achievable constraints in the fitness calculation. The constraints punish the fitness value of individuals who do not satisfy the constraints. In order to make the ideal mechanical work done by the GS-controlled actuator not exceed the work done by the pressing process, Formula (4) is used to constrain $W_v$. In order to ensure that the ideal energy made by the actuator controlled by $F_{peak}$ and $A_{Fpeak}$ does not exceed the energy produced by the vibration of the sliding process, Formula (5) is applied to constrain $P_{Fpeak}$. The fitness function calculation under constraints is shown in Formula (6).

$$W\ge W_v=\frac{M{F}^2}{2GS}$$

(4)

$$P\ge P_{Fpeak}={A_{Fpeak}}^2·df$$

(5)

$$F\left(\mathit{G}\right)=\left\{\begin{array}{c}|H-I_1|+|R-I_2|,W\ge W_v,P\ge P_{Fpeak}\hfill \\ |H-I_1|+|R-I_2|+\sqrt{{W_v}^2-W^2},W<W_v,P\ge P_{Fpeak}\hfill \\ |H-I_1|+|R-I_2|+\sqrt{{P_{Fpeak}}^2-P^2},W\ge W_v,P<P_{Fpeak}\hfill \\ |H-I_1|+|R-I_2|+\sqrt{{W_v}^2-W^2}+\sqrt{{P_{Fpeak}}^2-P^2},W<W_v,P<P_{Fpeak}\hfill \end{array}\right.$$

(6)

where H and R are the hardness and roughness intensities of the target sample. $I_1$ and $I_2$ are the two outputs of the BP model after inputting individual chromosome $\mathit{G}$. $F\left(\mathit{G}\right)$ is the fitness value of chromosome $\mathit{G}$. $W_v$ is the ideal work done by Omni, and $MF$ is the maximum force 3.3 N at normal position of Omni [25]. $P_{Fpeak}$ is the vibration power at frequency $F_{peak}$, and $df$ is frequency resolution.

In order to generate a new population, it is necessary to perform selection, crossover, and mutation operations on population individuals. The range of chromosome operation to generate new individual $\mathit{G}$ is limited by the range of the extracted features in Section 2. When the GA reaches the termination condition, the individual with the best fitness in the current population is selected as the optimal solution. By setting the hardness and roughness intensities of the corresponding samples in GA, the optimal solutions of 23 samples used in this paper are obtained. The results are shown in Figure 8.

3.2. Hardness Rendering Method

GS is used to control the stiffness of Omni for the hardness rendering of the target sample. Since there are samples whose stiffness exceeds the maximum stiffness of Omni (2.3 N/mm), the paper proposes to use the voice coil motor (EAI C-2 tactor) driven by the linear amplifier for the hardness rendering of high-hardness samples. We performed a hardness discrimination experiment to determine the equivalent stiffness of the voice coil motor at different frequency. We collected samples from 10 participants with an average age of 25.

Apparatus

The voice coil motor and one fabricated object that combined with four same springs and a board were placed on a separate digital scale with a distance of about 300 mm, as shown in Figure 9b. There was only one fabricated object. The springs of the fabricated object can be changed. The digital scale’s count was reset to zero before the participants applied force. The operator did not start the voice coil motor until it was observed that the force exerted by the participant remained at around 1N. The driving waveform of the voice coil motor was generated by Labview in real time and output to the linear amplifier through the sound card of the computer at 5 kHz. The amplitude of the waveform remained constant, and the frequency of the waveform can be changed by the operator every 10 Hz between 30 and 120 Hz, with a total of 10 frequency levels. Each frequency level was paired with seven different springs for a total of 70 pairs of parameters. The spring stiffness of the seven springs is shown in

Table 4. The spring stiffness of seven springs.

Specimen No.	1	2	3	4	5	6	7
Spring Stiffness (N/mm)	0.08	0.3	0.5	0.8	1.5	2.1	4.5

.

Process

The participants were isolated from visual and auditory cues. Firstly, a frequency level of the voice coil motor was randomly selected, and then one of the seven stiffness springs was randomly selected as the springs of the board. The voice coil motor was under the left fingertip, and the board was under the right fingertip. Participants judged whether the spring was stiffer than the voice coil motor stimulus. After completing all stimuli at one frequency level, participants were allowed to rest for 5 minutes. The process was then repeated until all parameter pairs were complete. A total of 700 responses were collected.

Results

The results were analyzed by the Point of Subjective Equality (PSE) [26,27]. PSE is the “subjective equality point”, which means that the value of the subjectively perceived comparison stimulus is the same as the value of the reference stimulus, and the PSE is given by the 50% probability point. Answers from all participants were added together and probabilities were calculated based on the number of times the spring was stiffer than the stimulation of the voice coil motor, fitted with a cumulative Gaussian distribution. The corresponding PSE was calculated for each stimulation level of the voice coil motor. As shown in Figure 10, the PSE values were plotted accordingly to show trends in participants’ responses to increased vibration frequency and fitted using a two-order regression model Formula (7). The formula of the hardness rendering method is given by Formula (8).

$$G{S}_{coil}=0.00012{f_{coil}}^2+0.01f_{coil}-0.3045$$

(7)

$$F_n=\left\{\begin{array}{c}GS·Y=(G{S}_{max}+G{S}_{coil}\left(f_{coil}\right))·Y,GS>G{S}_{max}\hfill \\ GS·Y,GS\le G{S}_{max}\hfill \end{array}\right.$$

(8)

where $F_n$ is the normal force, $G{S}_{max}$ is the maximum stiffness of Omni 2.3 N/mm, Y is the displacement of the Omni y-axis, $G{S}_{coil}$ is the equivalent stiffnes of the voice coil motor, and $f_{coil}$ is the frequency of the voltage applied to the voice coil motor.

3.3. Roughness Rendering Method

The piezoelectric ceramic (Samsung PHAT423535XX) driven by piezo haptic driver (TI DRV2667) is applied to render roughness. The piezoelectric ceramic was connected to the control module over the I2C bus. We used serial communication to enable the computer to communicate with the control module to transmit control commands. $F_{peak}$ and $A_{Fpeak}$ are used to control the voltage waveform. The piezoelectric ceramic vibrated when the velocity of the x-axis of the Omni $|v_x|$ exceeded 20 mm/s. In the paper, the surface of sample is approximately defined as a sinusoidal surface with a given wavelength of $\lambda$, as shown in Figure 11. When $\lambda$ is constant, the vibration frequency f generated by the surface is proportional to the speed, $F_{peak}$ is regarded as the frequency at speed $v_{const}$. When the speed $v_x$ changes, the frequency changes proportionally [28]. The formula of the roughness rendering method is given by Formula (9).

$$\left\{\begin{array}{c}{V}_{out}=Tab(F_{peak},A_{Fpeak})\hfill \\ f_p=F_{peak}·\left|\frac{v_x}{v_{const}}\right|\hfill \\ V_t=V_{out}·sin(2\pi ·f_p·t),|v_x|\ge 20 \mathrm{m}\mathrm{m}/\mathrm{s}\hfill \end{array}\right.$$

(9)

where $V_t$ is the voltage signal applied to the piezoelectric ceramic, $V_{out}$ is the amplitude of the voltage applied, $Tab$ can obtain the corresponding voltage amplitude $V_{out}$ of the piezoelectric ceramic at different sample peak frequency and its amplitude in Figure 12, $v_{const}$ is the lateral velocity 80 mm/s, $v_x$ is the velocity of the x-axis of the Omni, and $f_p$ is the frequency of the piezoelectric ceramic voltage waveform.

4. Evaluation of Rendering

The rendering features of the target sample can be obtained through Section 3. Omni, voice coil motor, and piezoelectric ceramic are combined to render the hardness and roughness. A total of 8 representative samples (S1, S2, S7, S12, S13, S14, S16, S20) are selected from 23 samples. The realism rating experiment was performed on the 8 samples under three rendering methods. Data analysis is performed on the results of the realism rating experiment.

4.1. Realism Rating Experiment

We recruited 10 participants with an average age of 25 to participate in the realism rating experiment. All of them were right-handed, and reported having no cutaneous or kinesthetic problems. The three rendering methods in the realism rating experiment are shown as follows:

(1) No Roughness: Remove rendering of roughness component. The Omni is used to render the hardness component. The voice coil motor is utilized to render hardness when target rendering sample’s GS exceeds Omni’s maximum rendering stiffness.

(2) No Hardness: Remove rendering of hardness component. The piezoelectric ceramic is used for roughness rendering, and Omni only provides 1N of normal constant force.

(3) All: The hardness and roughness components are displayed together.

Apparatus

As shown in Figure 13c, the participant sat at a table in front of a computer and wore headphones that played white noise to mask outside sounds. The real sample was obscured to prevent the participants from being visually affected by the real sample. The Omni with an attached fingertip device (integrating a voice coil motor and piezoelectric ceramic) was used to render the virtual sample and another Omni handle was utilized to feel the real sample. The weight of the fingertip device is 45.6 g. The voice coil motor driven by the linear amplifier (Figure 9) vibrated when the Omni detected that the participant touched the virtual sample in the virtual interface. The piezoelectric ceramic vibrated when the velocity of the x-axis of the Omni $|v_x|$ exceeded 20 mm/s.

Process

Firstly, participants used the right Omni with the fingertip device to learn how to interact with a virtual sample. Then, the operator provides participants with random real samples to feel. After the participants used left Omni handle to experience the real samples, the operator randomly showed one of the three virtual rendering methods, and each rendering time was within 30 s. Participants were required to use the real sample as a reference to rate the realism of the virtual sample under the three rendering methods on a scale of 0–100, where 100 meant “completely real” and 0 meant “completely unreal”.

4.2. Result

Because the collected data passed the Shapiro–Wilk normality test but not Levene’s homogeneity test, Welch ANOVA was used to compare the means among the three rendering methods [29]. The means differed significantly among the three rendering methods $(F_{2,151.564}=105.751,p\le 0.001)$. The Games Howell multiple comparison test was conducted on the Welch ANOVA results. The results revealed statistically significant difference between No Roughness and No Hardness $(p\le 0.001)$, No Roughness and All $(p\le 0.001)$, No Hardness and All $(p\le 0.001)$ as shown in Figure 14. The mean ratings were ranked as All (70.86) > No Roughness (57.81) > No Hardness (50.23). The mean rating score for the virtual samples with All haptic rendering was higher than the virtual samples with the removal of the roughness or hardness component. This indicated that the multi-modal haptic rendering is more in line with the feeling of the real sample than the single mode.

In order to further analyze the effect of removing the roughness or hardness component on the realism rating under each sample, we used the Shapiro–Wilk normality test to verify a normal distribution of the data for each sample. Levene’s homogeneity test was utilized to check the variance homogeneity of the data for each sample. Because the data for each sample passed the Shapiro–Wilk normality test and Levene’s homogeneity test, we performed ANOVA to compare the means among the three rendering methods for each sample. A Tukey–Kramer multiple comparison test was conducted on the ANOVA results. Figure 15 shows the relative importance of matching hardness and roughness for different samples.

Realism ratings had statistically significant difference under conditions where a certain rendering component was removed. While some other samples had no statistically significant difference under removing one rendering component. Removing the rendering of the roughness component created statistically significant differences in realism rating between No Roughness and All for five samples: S1 $(p\le 0.001)$, S7 $(p\le 0.001)$, S13 $(p\le 0.001)$, S14 $(p\le 0.01)$, S16 $(p\le 0.001)$. Removing the rendering of the hardness component created statistically significant differences in realism rating between No Hardness and All for six samples: S2 $(p\le 0.001)$, S12 $(p\le 0.001)$, S13 $(p\le 0.001)$, S14 $(p\le 0.001)$, S16 $(p\le 0.001)$, S20 $(p\le 0.001)$.

To determine how the realism rating of a virtual sample changes when the roughness or hardness rendering component was removed, the realism ratings of all participants were averaged. Then, we took the same approach as [1] and treated the change in the realism rating as a function of the features (obtained by GA). The difference percentage was calculated as follows:

$$\%dif{f}_x=\frac{R_{all}-R_x}{R_{all}}$$

(10)

where $R_{all}$ is the average realism rating when all rendering components were displayed, and $R_x$ is the average realism rating when roughness or hardness component was removed. A multiple regression analysis was conducted to analyze the factors influencing the realism rating of virtual samples due to the removal of the roughness or the hardness component (

Table 5. Factors related to the realism rating of the removal of roughness (multivariate analysis).

Variable	Unstandardized Coefficient		Standardized Coefficient	T Value	p Value	${\mathit{R}}^2$ $(\mathit{adj}.{\mathit{R}}^2)$
	B	SE	$\mathit{\beta }$
Const	35.009	8.671		4.038	0.013 *
GS	−0.075	1.259	−0.012	−0.060	0.955	0.842
$F_{peak}$	−0.232	0.064	−0.722	−3.628	0.022 *	(0.724)
$A_{Fpeak}$	15.966	5.512	0.583	2.896	0.044 *

* indicates a significant difference.

and

Table 6. Factors related to the realism rating of the removal of hardness (multivariate analysis).

Variable	Unstandardized Coefficient		Standardized Coefficient	T Value	p Value	${\mathit{R}}^2$ $(\mathit{adj}.{\mathit{R}}^2)$
	B	SE	$\mathit{\beta }$
Const	3.804	6.873		0.553	0.609
GS	8.168	0.998	0.910	8.184	0.001 *	0.952
$F_{peak}$	0.060	0.051	0.129	1.179	0.304	(0.916)
$A_{Fpeak}$	−9.138	4.369	−0.232	−2.091	0.105

* indicates a significant difference.

).

5. Discussion

By analyzing the relationship between the features obtained by GA and the perceptual intensities (hardness, roughness) of real samples, we verified whether the features capture the perceptual dimensions of real samples. The relationship between the features obtained by GA and the real hardness and roughness of the object was obtained by Spearman correlation analysis in Figure 16. The results showed that the features capture the hardness and roughness of objects. The study provided valuable guidance on the importance of hardness and roughness rendering for the realism of virtual samples by evaluating sample realism changes under removing the hardness or roughness rendering component.

5.1. Correlation Analysis

The results of correlation showed that there was a significant positive correlation between GS and Hardness $(r_P=0.903,p\le 0.01)$, which indicated that larger GS would bring greater hardness perception. There was a significant positive correlation between P and Hardness $(r_P=0.548,p\le 0.01)$, between P and Roughness $(r_P=0.435,p\le 0.05)$, which was analyzed due to the greater vibration intensity transmitted by tool interaction on harder or rougher objects. There was a negative correlation between $F_{peak}$ and Roughness $(r_P=-0.394,p\le 0.05)$. Under constant interaction speed, we inferred that the larger the $F_{peak}$, the smoother an object was. There was a positive correlation between $A_{Fpeak}$ and Roughness $(r_P=0.384,p\le 0.05)$. In the case of constant interaction speed, we inferred that the larger the $A_{Fpeak}$, the rougher an object was.

5.2. Factors Related to the Realism Rating of the Removal of Roughness or Hardness Component

When the roughness component was removed, $(\beta =-0.722,p\le 0.05)$ and $(\beta =0.583,p\le 0.05)$ had a significant effect on the percent difference in realism rating. The explanatory power was 84.2% as shown by $R^2$ in Table 5. The results showed that $F_{peak}$ was highly negatively correlated with the percent difference in realism rating, and $A_{Fpeak}$ was highly correlated with the percent difference in realism rating. The results suggested that roughness rendering was more important for samples with low $F_{peak}$ and high $A_{Fpeak}$ when interacting with the samples. The Roughness rating and $F_{peak}$ were moderately negatively correlated. The Roughness rating and $A_{Fpeak}$ were moderately correlated. The results indicated that roughness rendering was more important for rougher samples. For some smooth samples, roughness rendering may not even be required.

When the hardness component was removed, GS $(\beta =0.910,p\le 0.001)$ significantly affected the percent difference in realism rating. The explanatory power was 95.2% as shown by $R^2$ in Table 6. The results showed that GS was highly negatively correlated with the percent difference in realism rating. The results suggested hardness rendering was more essential for samples with higher GS. The Hardness rating and GS were highly correlated. The results indicated that hardness rendering was more important for harder samples, and it may provide a small constant force for the rendering of some soft samples.

6. Conclusions

In this paper, we propose a multi-modal haptic rendering algorithm based on GA and BP neural network to display the multiple perceptual intensities of the object, which generate force and vibration stimuli according to the target sample’s hardness and roughness. The Omni and voice coil motor are applied to render the hardness of the object, and the piezoelectric ceramic is utilized to render the roughness. Finally, we performed a realism rating experiment in which subjects compared real samples with virtual samples under multi-modal rendering or removing the hardness or roughness component. The results proved that multi-modal rendering improves realism. Based on the work, our method can be applied to render objects in more perceptual dimensions, not only limited to hardness and roughness of the paper. It has important implications for multi-modal haptic rendering.

There are also some limitations in our work. We only performed realistic evaluation experiments for the existing 8 real samples. Since our samples are selected from 23 samples and cannot cover the haptic properties of all sample categories, the type of rendering samples has certain limitations. Through the method in the paper, control features can also be obtained for virtual samples that do not exist corresponding to real samples among our 23 samples, but the realism of such virtual samples remains to be evaluated. To better render virtual objects, other features and actuators may be tried, which is the direction of our future work.