Influence of Piano Key Vibration Level on Players’ Perception and Performance in Piano Playing

Abstract

In this study, the influence of piano key vibration levels on players’ personal judgment of the instrument quality and on the dynamics and timing of the players’ performance of a music piece excerpt is examined. In an experiment four vibration levels were presented to eleven pianists playing on a digital grand piano with grand piano-like key action. By evaluating the players’ judgment of the instrument quality, strong integration effects of auditory and tactile information were observed. Differences in the sound of the instrument were perceived by the players, when the vibration level in the keys was changed and the results indicate a sound-dependent optimum of the vibration levels. By analyzing the influence of the vibration levels on the timing and dynamics accuracy of the pianists’ musical performances, we could not observe systematic differences that depend on the vibration level.

1. Introduction

Playing the piano is a complex multi-modal task, where the pianist controls the instrument through his or her intention and perceived instrument feedback. There are four main musician-musical instrument interaction modalities: visual feedback, auditory feedback, force feedback, and vibrotactile feedback. The interaction with a musical instrument can be modeled as a feedback controller [1], where the musician’s brain controls his or her body, arms, and fingers to modify the instrument’s behavior based on changes in sensory inputs. This closed-loop model implies that if the instrument’s feedback is altered, the pianist will adapt his or her playing to compensate for and retain the desired instrument behavior. Vibrotactile feedback can support the precise control of finger force, as shown by Ahmaniemi [2] with a basic force repetition experiment on a rigid sensor box. Furthermore, Goebl and Palmer [3] demonstrated that tactile sensations from the finger-key surface interaction support some pianists to improve timing accuracy and precision of finger movements.

In piano playing, vibrotactile feedback is perceived through the fingers in contact with the keys and the feet in contact with the pedals. The keybed and soundboard vibrations excite the piano keys and pedals [4]. Askenfelt and Jansson [5] measured the vibrations of a depressed piano key and a depressed piano pedal. Piano key vibrations comprise broadband and tonal parts [4,6]. The tonal parts come from the string vibrations, and the broadband parts come from mechanical impacts (e.g., hammer-string impact and key–keybed impact) of the piano action when the piano key is played [4,6].

The levels of the vibrations’ tonal part can rise to the micrometer range [7]; the vibrations are close to the limits of human vibration perception and are often sensed subconsciously [8]. However, piano key vibrations can be detected up to the middle octave of the keyboard [9]. Further, the vibration levels vary considerably among different pianos, but it remains unclear if pianists can perceive these differences [7].

Keane and Dodd [8] found that the ratio of broadband and tonal parts of piano key vibrations influenced the instrument’s perceived sound. An upright piano was mechanically modified to reduce the broadband parts’ amplitude, with the expectation to improve the instrument’s quality. The pianists preferred the modification with regard to tone and loudness in an evaluation study. Interestingly, the participants did not report differences with regard to touch or vibrations.

Fontana et al. [10] showed by ratings of evaluation criteria that realistic piano key vibrations rendered on a digital keyboard are preferred to a no-vibration condition. In the same study, key vibrations did not show a significant effect on pianists’ timing and dynamics accuracy during a scale playing task.

In addition to the state-of-the-art, the influence of four piano key vibration levels on pianists’ personal judgment of an instrument’s sound, control, and feel is investigated in this study. We designed the experiment, such that the control of vibration levels was independent of the sound of the instrument and aimed to explore connections between vibrotactile feedback and the perceived quality of the instrument. To test if pianists adapt their playing to vibrotactile feedback and to analyze if the vibrations support the control of finger forces, the effect of the vibration levels on timing and dynamics accuracy in pianists’ performances is also studied in this paper.

2. Methods

2.1. Equipment

The pianists played on an AvantGrand N3X, a digital hybrid grand piano from Yamaha. This instrument was chosen because it simulates piano key vibrations, features state-of-the-art grand piano sound-rendering algorithms, and has a piano action resembling that of acoustic grand pianos. Musical instrument digital interface (MIDI) messages and the headphone audio output of the instrument were recorded. The pianists played with closed-back headphones to block the small amount of sound that vibrating keys radiate.

2.2. Experiment Design

The target was to control the key vibration levels independent of the sound and to cover the level range of piano key vibrations of acoustic concert grand pianos. Independent control could not be achieved with the built-in vibrotactile feedback rendering system of the AvantGrand N3X; therefore, it was extended as illustrated in Figure 1.

The mono audio output signal of the AvantGrand N3X was processed with a digital signal processor (DSP). Through a combined approach of vibrometer measurements and subjective evaluation by playing on the instrument, the DSP’s filter stage was tuned to create vibration level $V_3$ (see Figure 1), which approached the maximum vibration levels previously measured on acoustic grand pianos [7]. After implementation of $V_3$, vibration levels $V_2$ and $V_1$ were created by attenuating the signal in steps of 6 dB. The no-vibration condition $V_0$ completed the levels of the experiment. As shown in Figure 1, the vibration levels cover the range of acoustic grand pianos for notes A2, A3, and A4. For notes A0 and A1, the levels are more than 10 dB lower. The chosen music piece avoided the lowest notes. The deviations of the vibration level curves in Figure 1 are due to non-idealities of the excitation system.

The experiment was created to study the influence of four vibration levels on players’ personal judgment of the instrument quality and on the dynamics and timing of the players’ performance of a music piece excerpt. The experiment was designed so that the participants were unaware of the independent variable, and the session was split into three parts to steer the pianists’ attention to different instrument properties. Free verbalizations were used to assess the players’ judgment, allowing for unrestrained and possibly unexpected answers. Since a small influence of the piano key vibrations on musical performance and the players’ judgments was assumed—as natural levels are close to the threshold of vibration sensation [7,8,9]—numerous repetitions were included in the protocol, and participants with high levels of playing experience were selected.

2.3. Participants

Eleven pianists participated in the study: seven piano students, 22–26 years of age, with an average playing experience of 17 years; and four professional pianists, 31–40 years of age, with an average playing experience of 26 years. None of them reported having auditory or tactile impairments.

2.4. Procedure

The session for each participant lasted around 1.5 h. The participants were asked to prepare an interpretation of a music piece excerpt. The excerpt was 15 bars long and the participants were instructed to adhere to the tempo, dynamics, accents, and pedaling information. The excerpt was taken from Klage by Gretchaninov [11] and was edited to cover the dynamic range from pianissimo to fortissimo (see Figure A1 in the Appendix A). The participants were not informed about the purpose of the experiment beforehand. The participants were only told that their judgments of various settings of the instrument will be evaluated.

The experiment comprised a warm-up (with a duration of around 5–10 min), questionnaires, and three parts (A, B, and C) with a duration of roughly 20 min each.

During the warm-up, the pianists were free to play whatever they wanted and were instructed to evaluate the instrument in a way comparable to choosing an instrument for a concert or for purchase. After this familiarization, the pianists were asked to express their first impression by answering a set of questions about the sound and touch of the AvantGrand N3X and by comparing the instrument to their main instrument.

During the main parts of the experiment (A, B, and C) the pianists were asked to repeat the excerpted music piece accurately in 12 direct comparisons of different instrument settings. After each trial the pianists were asked to indicate a personal preference (“better”, “worse”, or “similar”) of the current setting relative to the previous setting and to describe their impression in a few words. The participants were told that the differences between the comparisons can be small and that some might be perceptually irrelevant. In part A the pianists were told that a slight adjustment of the instrument (not further specified in order to not suggest an answer or category) was made between each repetition, in part B it was claimed that a small adjustment to the sound was made between each trial, and in part C the pianists’ attention was directed to the keyboard by asking about the instrument’s control and feel.

In fact, the only independent variable throughout the experiment was the key vibration level ($V_0$, $V_1$, $V_2$, $V_3$). The sequence of parts (A, B, and C) was the same for all participants. Each part had a different randomized sequence of vibration levels. All pair of levels were compared twice in each part—once for each order. In total, nine trials per vibration level and per participant were recorded.

At the very end of each experiment session, personal information was collected and the participant was asked about his or her experience and preference of piano key vibrations in piano playing, before we disclosed and explained the purpose of the experiment.

3. Results

3.1. Influence of Vibration Levels on Perceived Instrument Sound, Control, and Feel

In the analysis of the preference ratings (“better”, “worse”, or “similar”), a high variance and for some participants also controversial ratings were observed. We decided to present the ratings across all parts and participants here, because it was not possible to draw conclusions from the ratings per participant or per part. The result is presented in Figure 2.

The high variance of the ratings in Figure 2 reflects the closeness of the key vibration levels to the limits of human perception. Additional factors that can have disturbed the ratings are the mood and fatigue of the player. Also a self-evaluation of the playing, the difficulty of the task, or the imposed expectation of a difference between the settings might have disturbed the ratings. However, a visual comparison of the mean values in Figure 2 indicates a tendency in the preference of the players toward vibration level $V_2$. The preference of vibration level $V_2$ is confirmed by the evaluation of the player’s verbal self-reports presented hereafter.

The free verbalizations were analyzed with an approach presented by Pate et al. [12], where concepts by Dubois [13] were applied to musical instrument evaluation.

Based on the context and for each participant, the meaning, category, and preference of each statement was identified via linguistic tools such as reformulations, oppositions, and comparatives. Thereafter, the statements were classified into positive and negative statements for three categories: sound, control, and feel. Statements covering multiple categories were split before classification. Statements not indicating a preference or a perceived difference were counted as “no difference”. The results are summarized in

Table 1. Evaluation per vibration level derived from the free verbalizations of the pianists. The number of positive and negative statements per category was counted for each vibration level ($V_0,V_1,V_2,V_3$). The frequency counts were evaluated with ${\chi }^2$-tests. The ${\chi }^2$-statistics and p-values are given for each category; p < 0.05 is highlighted in bold.

Category	Examples	${\mathit{V}}_0$	${\mathit{V}}_1$	${\mathit{V}}_2$	${\mathit{V}}_3$	${\mathit{\chi }}^2$	p
sound pos. $s_p$	“round”, “balanced”	21	26	39	24
sound neg. $s_n$	“harsh”, “artificial”	28	19	12	19
relative number of positive statements $s_p/(s_p+s_n)$		0.4	0.6	0.8	0.6	11.86	0.0079
control pos. $c_p$	“reactive”, “controllable”	25	24	25	24
control neg. $c_n$	“limited”, “hard to create dynamics”	15	11	13	16
relative number of positive statements $c_p/(c_p+c_n)$		0.6	0.7	0.7	0.6	0.69	0.88
feel pos. $f_p$	“comfortable”, “grand piano feeling”	9	8	7	11
feel neg. $f_n$	“exhausting”, “tedious”	5	4	1	4
relative number of positive statements $f_p/(f_p+f_n)$		0.6	0.7	0.9	0.7	1.52	0.68
no difference	“similar”, “somewhat different”	12	24	11	13

. Significance was evaluated with Pearson’s ${\chi }^2$-tests at a confidence level of 95%. The test was performed on all statements (positive and negative) per category.

Although we did not alter the sound throughout the experiment, Table 1 shows that the vibration levels ($V_0,V_1,V_2,V_3$) have an influence on the pianists’ sound perception. Pairwise testing with Bonferroni correction showed that the significance of the ${\chi }^2$-test in the sound category arises from the difference between $V_0$ and $V_2$.

The phenomenon of vibrotactile feedback causing a difference in sound perception is known as integration of auditory and tactile information [14,15,16] or weak synesthesia [17]. The preference of $V_2$ over $V_3$, confirming the evaluation of the preference ratings in Figure 2, was surprising because vibration level $V_3$ is closer to the levels of acoustic instruments (see Figure 1). An explanation lies in five statements about vibration level $V_3$, which were classified as negative. These statements criticized the balance of the perceived sound as having “too much bass" or being “unbalanced".

The results for control and feel are not significant according to the ${\chi }^2$-tests. For vibration level $V_1$ there were twice as many “no difference" statements than for all other levels, which indicates that $V_1$ is most difficult to differentiate.

Only two participants consciously noticed a change in vibration levels during the experiment, when vibration level $V_3$ was compared to $V_0$ and vice versa. Both recognized the vibrations during the last part, when the keyboard was the focal point.

To find possible explanations for the above presented differences, we analyzed the verbal self-reports of the participants in more specific categories. We observed that the key vibrations influence the timbre and the perceived loudness of the bass keys. Also, the timbre of treble notes was judged more pleasant when playing with $V_2$ or $V_3$. Some participants also noted a sensation of space when playing with higher vibration levels ($V_2$, $V_3$) and described it as room or reverb effect of the sound. In contrast when pianists played with vibration levels $V_0$ or $V_1$ the sound was sometimes described as dry. Comparisons to acoustic instruments and e-pianos also align with this observation.

A critical aspect for discussion is that the sequence of parts (A, B, and C) was the same for all participants, which might have influenced the results. We designed the experiment protocol to steer the players’ attention to different multi-modal aspects to discover unexpected connections between vibrotactile feedback and the players’ judgment of the instrument quality.

In part A the participants could freely describe their impressions and we did not suggest any quality criteria for the comparisons. Unbiased comparisons are only possible within the first part of the experiment. Ten out of eleven participants naturally made statements about sound and control in the first part, which justifies the suggestion of these criteria in the following parts.

We decided to put part C at the end of the experiment session, because we did not want to risk that the participants are already consciously aware of the vibrations, when comparing the levels with regard to sound. In part C the participants focused on the keyboard. Therefore we expected that it is most likely that the participants recognize the vibrations in this part (which happened in two cases).

Finally, also the difficulty of the evaluation task might have altered the judgments of the pianists. In each trial, the participant played the music piece excerpt for a duration of around 30 s, communicated his or her impression with regard the previous setting and sometimes also answered clarifying questions from the experimenter. Thereafter he or she performed the music piece for the next comparison.

3.2. Influence of Piano Key Vibration Levels on Musical Performance

To analyze the MIDI-based performance data, a custom data structure was used. The structure groups notes played at the same time (±40 ms) into clusters, removes accidentally played wrong notes, and assigns if the note was played by the left or right hand.

Key velocity $\upsilon$, a measure of a keystroke’s excitation strength, was directly extracted from the MIDI messages. For the calculation of the inter-onset interval $\tau$—the time interval between two subsequent note onsets—only notes played by the left hand were considered. The tempo of each trial was normalized.

To compare trials by the distribution of key velocity $\upsilon$ (analogously for inter-onset interval $\tau$) histogram intersection was used. Histogram intersection was introduced by Swain and Ballard [18] to identify objects by color similarity in computer vision. Histogram intersection is defined as [18]

$$H_1\left(\upsilon \right)\cap H_2\left(\upsilon \right)=\sum _{i=1}^{n}min(h_{1i}\left(\upsilon \right),h_{2i}\left(\upsilon \right)),$$

(1)

where $H_1$ and $H_2$ represent two trials by normalized discrete distributions of key velocity $\upsilon$ with n bins $h_{1i}$, $h_{2i}$. Equation (1) measures the overlap of two histograms in the range $[0,1]$. The number ‘1’ corresponds with perfect overlap; ‘0’ means no overlap. In contrast to an evaluation based on mean values only, histogram intersection also identifies differences in the distributions’ shape or offset.

To judge significance, two tests were demanded to reject the null hypothesis: non-parametric Friedman analysis of variance with a 95% confidence level in combination with pairwise Wilcoxon signed-rank tests with Bonferroni correction. We used non-parametric tests, because the evaluated quantities do not necessarily follow a normal distribution and because of the sample size.

Two approaches were used to compare the distributions of key velocity $\upsilon$ and inter-onset interval $\tau$ by histogram intersection. The distribution of both parameters was calculated for each trial and was analyzed for each participant separately.

The first approach considered if the pianists adapted their playing to the vibration levels (e.g., if a pianist perceived an overemphasis of bass notes and therefore played the bass notes with less finger force than before). This force adaption manifests in the shape of the distribution of key velocity $\upsilon$. An increase in the “amount of adaption” was expected with increasing vibration levels. The “amount of adaption” for key velocity $\upsilon$ was measured as follows.

Let $H_{P_k,V_{\ell },i}\left(\upsilon \right)$ denote the normalized histograms describing the distribution of key velocity $\upsilon$ for trial $i\in \{1,\cdots ,9\}$, vibration level $V_{\ell }$ with $\ell \in \{0,1,2,3\}$, and pianist $P_k$ with $k\in \{1,\cdots ,11\}$. Then the “amount of adaption” $A_{P_k,V_n}$ of pianist $P_k$ to vibration levels $V_n$ with $n\in \{1,2,3\}$ was estimated as the histogram difference relative to $V_0$ condition $H_{P_k,V_0,i}\left(\upsilon \right)\cap H_{P_k,V_n,j}\left(\upsilon \right)$ for all combinations of trials $i,j\in \{1,\cdots ,9\}$ and key velocity $\upsilon$. For the inter-onset interval $\tau$ the same procedure was conducted.

Figure 3 shows the “amount of adaption” of the pianist’s playing to the feedback levels for both performance parameters. The differences in Figure 3 are not significant. There is no general tendency that the participants adapt their playing to the key vibration level. Nonetheless, by analyzing the influence of the vibration levels per participant individually, three participants showed significant differences for key velocity $\upsilon$ and three for inter-onset interval $\tau$.

Possible explanations for the majority of pianists not adapting their playing to the feedback levels include that the combined task of playing, judging the impression, and adapting their playing was too difficult, that the levels were too small to cause a reaction, or that the method was not accurate enough to unveil such differences.

The second approach investigated how accurately the pianists could repeat the music piece excerpt when playing with different vibration levels. If key vibrations support the precise control of finger forces, a lower variance in the distribution of key velocity $\upsilon$ could be expected. In consequence the shape of the distribution of key velocity $\upsilon$ would be altered and hence a difference in repeatability could be detected. Likewise, if the pianist’s tempo was more stable, a different shape of the distribution of the inter-onset interval $\tau$ would occur. Indirect causes are also possible (e.g., the pianist feels more comfortable to play and therefore plays with higher repeatability). The time-point of the trials during the experiment was not taken into account. We decided to analyze and present the data for each pianist individually hereafter, because we observed a strong dependency of the repeatability on the player.

The repeatability $R_{P_k,V_0}\left(\upsilon \right)$ for participant $P_k$ playing with vibration level $V_0$ was computed by comparing the distributions of key velocity $\upsilon$ by $H_{P_k,V_0,i}\left(\upsilon \right)\cap H_{P_k,V_0,j}\left(\upsilon \right)$ for all combinations of trials $i,j\in \{1,\cdots ,9\}$, where $i\ne j$. For vibration levels $V_1,V_2$, and $V_3$, and for inter-onset interval $\tau$ similar procedures were conducted.

The resulting repeatability per vibration level and per participant is presented in Figure 4 for key velocity $\upsilon$ and inter-onset interval $\tau$. The differences in repeatability were significant for a majority of the participants. However, no consistent tendency or pattern in repeatability occurred among the pianists in Figure 4.

Therefore, the vibration levels of our experiments do not have a conclusive influence on the pianists’ repeatability, and the measured MIDI data do not support the hypothesis that key vibrations assist the precise control of finger force. Consequently the observations do not confirm the results of Ahmaniemi [2] or Galica et al. [19], for the piano playing case. Galica et al. [19] showed that even unconscious vibratory stimulation applied to the soles of the feet can cause lower variance in kinematic interactions.

In summation, the repeatability estimates $R_{P_k,V_{\ell }}\left(\upsilon \right)$, $R_{P_k,V_{\ell }}\left(\tau \right)$ for vibration levels ($V_0$, $V_1$, $V_2$, $V_3$) depended on the player. The pianists in this study were more accurate in repeating key velocity $\upsilon$ (median of $R_{P_k,V_{\ell }}\left(\upsilon \right)\in [0.85,0.88],\forall k,\ell$) than in repeating inter-onset interval $\tau$ (median of $R_{P_k,V_{\ell }}\left(\tau \right)\in [0.66,0.77],\forall k,\ell$). For the inter-onset interval $\tau$ the intra-individual variance was also considerably larger (see Figure 4), although we normalized the tempo of each trial before the analysis.

As a concluding aspect of interest, no categorical differences (in repeatability or playing adaption to feedback levels) were found between the group of students ($P_1$ to $P_7$ in Figure 4) and the group of professional pianists ($P_8$ to $P_{11}$ in Figure 4).

4. Conclusions

By systematically investigating the players’ personal judgment of the instrument quality of the vibration level in the keys, we observed strong integration effects of auditory and tactile information. The results give an illustration of the strong multi-modal effects in piano playing. The subjects perceived differences in the sound of the instrument when the vibration level in the keys was changed. The preference of vibration level $V_2$ over $V_3$ indicates an optimum or a “sweet spot” of piano key vibration levels, which depends on the instrument’s sound and sound balance.

In line with the results of Keane and Dodd [8], the vibration levels in this experiment significantly affected the instrument’s judged sound quality but not its control and feel. However, in contrast to the design of the present experiment, Keane and Dodd [8] reduced the level of the broadband part of piano key vibrations of an acoustic instrument.

An interesting direction for future research is to determine the vibration level differences that pianists can differentiate. This would help to understand if an instrument can be identified based on its vibrotactile feedback only, while the instrument’s auditory and force feedback are kept constant. Some participants in this experiment perceived a spatial sensation and described it as room or reverb effect on the sound when playing with higher vibration levels ($V_2,V_3$). For several applications it could be interesting to understand the conditions that can cause such an illusion.

We did not find systematic differences by analyzing the influence of the vibration levels on the timing and dynamics accuracy of the pianists’ musical performances. We can not exclude that such an influence exists but with the proposed measures, “amount of adaption” and repeatability, we could not measure such a relation. Furthermore, the basic results of Ahmaniemi [2], that vibrotactile feedback assists the precise control of finger forces could not be confirmed in our case. For future studies, we suggest to include a larger number of participants. This could help to identify groups reacting similarly to key vibrations. Future studies might also include multiple experiments over a certain range of time to exclude influences of physical and mental state on the day of testing. Finally, an analysis on a note-by-note basis could clarify if, for example, the vibrotactile feedback of a long-lasting bass note helps the precise control of the dynamics in subsequent keystrokes. We could not generalize such a relation with the data of the presented experiment.

If our results can be confirmed on acoustic instruments, our findings of the perception part of the experiment suggest that piano manufacturers should design the vibrations in the piano keys in balance with the sound of the lower notes of the instrument. Furthermore, it would be interesting to investigate the just-noticeable difference of piano key vibration levels, which might possibly be around 6 dB. Further research in this area could help to answer the question, if the tonal parts of the Steinway and Sons and the Yamaha concert grand pianos (the tonal parts for notes A2, A3, and A4 differ by more than 6 dB [7]) can be differentiated based on their vibrotactile feedback only by the player. Of course in such an experiment the vibrotactile feedback should be rendered on the same instrument, otherwise cues from the auditory or kinematic sensations might dominate the perceived impression.