Perceptual Relevance of Haptic Feedback during Virtual Plucking, Bowing and Rubbing of Physically-Based Musical Resonators
Round 1
Reviewer 1 Report
This submission presents the design and development and evaluation of three digital instruments with respectively plucked, bowed, or rubbed interactions augmented by haptic force-feedback. The three digital instruments are based on two software implementations (Unity and JUCE) with off-the-shelf force-feedback devices and their evaluations mix informal, qualitative and quantitative methods with human participants.
This submission is adequately within the scope of Special Issue "Feeling the Future—Haptic Audio" and serves as great use case to showcase the challenges met in the development of musical force-feedback applications.
I would recommend for this submission to be accepted with minor revisions.
Strengths
- The related work review is quite comprehensive and representative of the history of musical force-feedback digital instruments.
- The sections for the three digital instruments with respectively plucked, bowed, or rubbed interactions augmented by haptic force-feedback describe well their implementation and evaluation concisely.
- "The auditory feedback included real sounds coming from the robotic arm while actuating the string discontinuities colliding with the plucking plectrum. As a direct by-product of the haptic feedback, these sounds greatly improve the realism of the scenario." This is a great design choice, while force-feedback artifact sounds are usually considered as noise and isolated from the main musical output.
Weaknesses
- The discussion section might have been more complete with minor changes: a comparison of the three digital instruments, that are all rely on physical modelling audio-haptic synthesis, but with different implementations (Unity and JUCE) and representations (3D in virtual reality vs 2D in "desktop" settings). What would be the recommendations or implications for future related work?
Questions
- What determined the choice of 24 frames per second for the strings / masses displacement update rate in JUCE apps (bowed and rubbed)?
- In experiment 3, what would justify that the Mouse equals or outperform the Sensel across the 7 attributes? There are answers in Onofrei et al. (2022). under 4.3.1 Quantitative Data. Could it also be due to the passive haptic feedback of the mouse (button clicks, wheel detents) that better informs users that a pad without feedback if their controls are actually applied in the interactions? Could it also be due to the non-smooth surface of the Sensel Morph that passively conflicts with the rubbing metaphor?
Minor issues
- "physical interactions along with a musically unconventional one, rubbing" Maybe the glass harp (14th century), glass harmonica (1763), Crystal Baschet (1952) and others together make rubbing conventional as a musical interaction?
- "Following this design approach, an application called Keytar was developed within a popular programming environment for computer games and virtual scenarios and then validated with guitar players." Needs a reference or rephrasing into active voice if developed by authors (post-scriptum: the latter).
- "In fact, both are mediated by an object that can be grasped using two fingers (respectively the plectrum and the bow)." This sentence sounds restrictive, as plucking is also possible with two bare fingers and bowing with rosin-coated fingers (e.g. Ellen Fullman's Long String instrument). Maybe rephrase into "can be mediated"?
- "Finally, see the unlabeled plot (bottom right) in Fig. 6, the errors made on plucking a wrong instead of a visually marked string during a part of the test involving individual string plucking were logged." This sentence is a bit difficult to parse, maybe using redundancy on "string" would help, as in "plucking a wrong [string] instead of a visually marked string". Also may the unlabeled plot be labeled with something like "(f) Errors in target string selection"?
- In Figure 11, x-axis labels "Question [n]" may be replaced by attributes to avoid eye scans to/from Table 1.
Author Response
For completeness, each Reviewer receives a Response letter to both Reviewers.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Review for "Perceptual relevance of haptic feedback during virtual plucking, bowing and rubbing of physically-based musical resonators."
This paper outlines designing and creating haptic devices for plucking, bowing, and rubbing interactions with musical resonators. There are frequent minor grammatical errors, and it requires meticulous proofreading before being considered for publication. For example, the abstract alone has several grammatical errors that make it difficult to read, for example, punctuation (the use of commas), grammar (singular and plural verb forms), clarity of writing (conciseness), and unnecessarily complex sentences. Overall, I believe the central thesis or contribution needs to be made more substantial in the paper's opening section (perhaps even bullet points would help for clarity). Furthermore, the discussion requires expanding to include the broader implications of plucking, bowing, and rubbing of virtual musical instruments in the Arts field beyond musical haptics. Otherwise, this is an exciting contribution to the SI and should be included with some further editing and rewriting.
I provide below feedback on the individual sections of the manuscript.
Introduction.
Line 26: a page number is needed for the quote in the reference.
Line 121: Is there a link or reference to the Keytar project that can be shared with the reader?
Line 131: A reference should be included here for these projects, or is this reference to the work included in the current manuscript? This point needs to be clarified as in line 141; it is referenced directly. I believe (Passalenti and Fontana 2018), (Fontana et al. 2019), and (Fontana et al. 2020) can be included.
The introduction is well-paced and somewhat adequately referenced. The content sets up the scope of the manuscript logically but varies in the quality of the writing.
Section 2.
Line 144: The title of this section is either not finished or goes on too long. Either way, it is not correct.
Line 156: According to who is this requirement necessary? A little more validation is needed here.
Section 3.
Line 188: Is plucking a tactile-only form of interaction? There should be a reference to force in this scenario, as "pulling" implicitly requires pressure to be applied. Perhaps "fully haptic" would be a better description of plucking.
Line 196 to 203: The references here potentially reveal the authors' identities, given the information provided earlier on this topic.
Line 223 to 226: The sentence "For its flexibility…" is challenging to follow "already suggested/advertised/concretely" should be replaced with something more coherent.
Line 286: Remove" about"
Line 288: For "…this feature of the device…" - is this claim recorded anywhere in the device tech sheet, or is it observational?
Section three was very detailed and gave an excellent overview of the plucking mechanisms implemented. The upward trend demonstrated in Figure 6 has been observed in other experiments with Musical Haptics in other works referenced earlier (Papetti and Saitis 2018): particularly the effects of no feedback, tactile only, force only, force and vibration together. I mainly found the combination of "tech talk" and user outcomes beneficial for the journal audience; however, more could be done to show how these findings have broader implications in general Arts practice.
Section 4.
Line 382 to 394: While I appreciate the in-depth description of the synthesis model, I don’t think the Arts Journal audience will necessarily understand the importance of this model to the SI cfp. This section needs to be included but with a more cursory or succinct description of the synthesis model.
Line 490: Anonymously provide a reference for this presentation/demo. Can this observational data be formalized or quantified in some way? Can you estimate conference attendees that interacted with the demo, etc.?
Section 4 provides a firm overview of the importance of haptics in bowing interactions. While informative, it was too heavy on the equations, and the broader contributions to the field could have been hinted at before expanding in the discussion section. It also does not provide a formal evaluation, as is presented in the other two sections.
Section 5.
Line 524: Provide a reference for Bilbao.
Figure 9: The equation number is missing from the caption (??).
Line 580: Missing Figure reference (??).
Line 598: Is the evaluation published? Anonymously provide a reference here. If the study is not published, more information must be added, which may not be possible here.
Line 612: Missing Table reference (??).
Section 5 was also fascinating and provided a good description of implementing rubbing into a musical haptics system.
Discussion.
The discussion reports on the contribution of the previous sections. Still, it fails to explain why this is important in musical haptic interactions and requires more contextualization in the broader area of Arts practice. Moreover, the discussion should also tie these findings back into the presented research in the introduction and background sections of the manuscript.
Conclusion.
The conclusion could do with some reference to the opening hypothesis and central thesis of the manuscript.
Author Response
For completeness, each Reviewer receives a Response letter to both Reviewers.
Author Response File: Author Response.pdf
