Submit to Arts Review for Arts Propose a Special Issue

Journal Menu

Journal Browser

► Journal Browser

Feeling the Future—Haptic Audio

Print Special Issue Flyer
Special Issue Editors
Special Issue Information
Keywords
Published Papers

A special issue of Arts (ISSN 2076-0752).

Deadline for manuscript submissions: closed (31 January 2023) | Viewed by 23295

Share This Special Issue

Special Issue Editors

Prof. Dr. Justin Paterson

E-Mail Website
Guest Editor

London College of Music, University of West London, London W5 5RF, UK
Interests: music production; haptic audio; 3-D audio; interactive music

Prof. Dr. Marcelo M. Wanderley

E-Mail Website
Guest Editor

Music Research, McGill University, Montreal H3A 1E3, Canada
Interests: new interfaces for musical expression; musical human–computer interaction; electronic instrumentation

Special Issue Information

Dear Colleagues,

Haptics increasingly pervades our interaction with technology; the vibrating phone is a simple everyday experience for billions of people. However, increasingly sophisticated haptic applications are developing in numerous industries, from autonomous cars to surgical simulation, wearables to wellbeing, and in creative sectors such as gaming and fashion.

In music and audio, several research communities (e.g., the NIME, ICMC, and SMC conferences) act as nexuses for audio, HCI, and computer science, at times including haptics. Similarly, the IEEE gathers works in haptics which at times have links to audio and music via its Transactions on Haptics journal, the World Haptics conference, and the Haptics Symposium. The recently revived HAID workshop bridges both areas, and is a unique forum for such interdisciplinary research.

In recent decades, such research has built increasing momentum and has given rise to many novel tactile interfaces and approaches to musicking. Research on force-feedback in musical applications has traditionally suffered from issues such as hardware cost and the lack of community-wide accessibility, to software and hardware platforms for prototyping applications. Typically, associated publications often require the quantitative analysis of primary data, formal user testing in controlled conditions, or present mathematical context for novel interfaces.

Although many of these approaches have proved valuable, the field has yet to demonstrate an amalgamated and compelling case for the actual benefit of haptics—especially force-feedback—to audio and musical applications. Presenting this case represents the scope of this Special Issue.

We believe it is time to discuss future opportunities more openly, and to propose directions in which this field can blossom, and eventually precipitate more ubiquitous tools for audio and music interaction.

We invite submissions that can support this case, and whilst these might contain analyses of user testing that yield insightful results, we equally welcome case studies, conceptual visions, and discussions that lend weight to the case. Contributors should be mindful of the readership of this journal, although technical content is welcome under the established Arts theme ‘music and the machine’.

We particularly welcome contributions that address musical applications of force-feedback, telehaptics, and haptic shared control.

Prof. Dr. Justin Paterson
Prof. Dr. Marcelo M. Wanderley
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a double-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Arts is an international peer-reviewed open access semimonthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 1400 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

haptics
audio
music
touch
tactility
performance
production

Published Papers (12 papers)

Download All Papers

9 pages, 238 KiB

Open AccessEditorial

Feeling the Future—Haptic Audio: Editorial

by Justin Paterson and Marcelo M. Wanderley

Arts 2023, 12(4), 141; https://doi.org/10.3390/arts12040141 - 7 Jul 2023

Viewed by 1375

Abstract

This Special Issue in Arts aligns with the journal’s established theme of ‘Music and the Machine’ [...] Full article

(This article belongs to the Special Issue Feeling the Future—Haptic Audio)

9 pages, 449 KiB

Open AccessEditorial

Force-Feedback and Music: Five Decades of Research and Development at ACROE: An Interview with Claude Cadoz (ACROE, Grenoble, France)

by Marcelo M. Wanderley and Christian Frisson

Arts 2023, 12(4), 159; https://doi.org/10.3390/arts12040159 - 17 Jul 2023

Viewed by 1440

Abstract

Recorded on 23 March 2023 [...] Full article

(This article belongs to the Special Issue Feeling the Future—Haptic Audio)

22 pages, 3715 KiB

Open AccessArticle

Feel the Music!—Audience Experiences of Audio–Tactile Feedback in a Novel Virtual Reality Volumetric Music Video

by Gareth W. Young, Néill O’Dwyer, Mauricio Flores Vargas, Rachel Mc Donnell and Aljosa Smolic

Arts 2023, 12(4), 156; https://doi.org/10.3390/arts12040156 - 13 Jul 2023

Viewed by 2515

Abstract

The creation of imaginary worlds has been the focus of philosophical discourse and artistic practice for millennia. Humans have long evolved to use media and imagination to express their inner worlds outwardly via artistic practice. As a fundamental factor of fantasy world-building, the imagination can produce novel objects, virtual sensations, and unique stories related to previously unlived experiences. The expression of the imagination often takes a narrative form that applies some medium to facilitate communication, for example, books, statues, music, or paintings. These virtual realities are expressed and communicated via multiple multimedia immersive technologies, stimulating modern audiences via their combined Aristotelian senses. Incorporating interactive graphic, auditory, and haptic narrative elements in extended reality (XR) permits artists to express their imaginative intentions with visceral accuracy. However, these technologies are constantly in flux, and the precise role of multimodality has yet to be fully explored. Thus, this contribution to Feeling the Future—Haptic Audio explores the potential of novel multimodal technology to communicate artistic expression via an immersive virtual reality (VR) volumetric music video. We compare user experiences of our affordable volumetric video (VV) production to more expensive commercial VR music videos. Our research also inspects audio–tactile interactions in the auditory experience of immersive music videos, where both auditory and haptic channels receive vibrations during the imaginative virtual performance. This multimodal interaction is then analyzed from the audience’s perspective to capture the user’s experiences and examine the impact of this form of haptic feedback in practice via applied human–computer interaction (HCI) evaluation practices. Our results demonstrate the application of haptics in contemporary music consumption practices, discussing how they affect audience experiences regarding functionality, usability, and the perceived quality of a musical performance. Full article

(This article belongs to the Special Issue Feeling the Future—Haptic Audio)

► Show Figures

Figure 1

22 pages, 289 KiB

Open AccessArticle

Exploring the Opportunities of Haptic Technology in the Practice of Visually Impaired and Blind Sound Creatives

by Jacob Harrison, Alex Lucas, James Cunningham, Andrew P. McPherson and Franziska Schroeder

Arts 2023, 12(4), 154; https://doi.org/10.3390/arts12040154 - 13 Jul 2023

Cited by 1 | Viewed by 1891

Abstract

Visually impaired and blind (VIB) people as a community face several access barriers when using technology. For users of specialist technology, such as digital audio workstations (DAWs), these access barriers become increasingly complex—often stemming from a vision-centric approach to user interface design. Haptic technologies may present opportunities to leverage the sense of touch to address these access barriers. In this article, we describe a participant study involving interviews with twenty VIB sound creatives who work with DAWs. Through a combination of semi-structured interviews and a thematic analysis of the interview data, we identify key issues relating to haptic audio and accessibility from the perspective of VIB sound creatives. We introduce the technical and practical barriers that VIB sound creatives encounter, which haptic technology may be capable of addressing. We also discuss the social and cultural aspects contributing to VIB people’s uptake of new technology and access to the music technology industry. Full article

(This article belongs to the Special Issue Feeling the Future—Haptic Audio)

23 pages, 23465 KiB

Open AccessArticle

Design and Evaluation of a Multisensory Concert for Cochlear Implant Users

by Razvan Paisa, Doga Cavdir, Francesco Ganis, Peter Williams, Lone M. Percy-Smith and Stefania Serafin

Arts 2023, 12(4), 149; https://doi.org/10.3390/arts12040149 - 10 Jul 2023

Cited by 1 | Viewed by 1738

Abstract

This article describes the design, implementation, and evaluation of vibrotactile concert furniture, aiming to improve the live music experience of people with hearing loss using hearing technology such as cochlear implants (CI). The system was the result of a series of participatory design sessions involving CI users with different hearing assistive setups (bi-implant, bimodal, and monoimplant), and it was evaluated in a concert scenario (drums, bass, and female vocals) at the Royal Danish Academy of Music. The project aimed to improve the music appreciation for CI users by providing a multisensory concert designed with CI challenges in mind, but not excluding normal-hearing individuals or individuals with other forms of hearing aids from participating in the event. The evaluation was based on (video-recorded) observations and postexperience semistructured interviews; the data were analyzed using event analysis and meaning condensation. The results indicate that tactile augmentation provides a pleasant experience for CI users. However, concertgoers with residual hearing reported being overwhelmed if the tactile stimulation amplitude exceeds a certain threshold. Furthermore, devices that highlight instrument segregation are preferred over ones that present a tactile mixdown of multiple auditory streams. Full article

(This article belongs to the Special Issue Feeling the Future—Haptic Audio)

► Show Figures

Figure 1

21 pages, 2328 KiB

Open AccessArticle

Tactual Articulatory Feedback on Gestural Input

by Bert Bongers and G. C. van der Veer

Arts 2023, 12(4), 146; https://doi.org/10.3390/arts12040146 - 10 Jul 2023

Viewed by 1385

Abstract

The role of the sense of touch in Human–Computer Interaction as a channel for feedback in manipulative processes is investigated through the research presented here. The paper discusses how information and feedback as generated by the computer can be presented haptically, and focusses on the feedback that supports the articulation of human gesture. A range of experiments are described that investigate the use of (redundant) tactual articulatory feedback. The results presented show that a significant improvement of effectiveness only occurs when the task is sufficiently difficult, while in some other cases the added feedback can actually lower the effectiveness. However, this work is not just about effectiveness and efficiency, it also explores how multimodal feedback can enhance the interaction and make it more pleasurable—indeed, the qualitative data from this research show a perceived positive effect for added tactual feedback in the overall experience. The discussion includes suggestions for further research, particularly investigating the effect in free moving gestures, multiple points of contact, and the use of more sophisticated actuators. Full article

(This article belongs to the Special Issue Feeling the Future—Haptic Audio)

► Show Figures

Figure 1

27 pages, 8922 KiB

Open AccessArticle

Brass Haptics: Comparing Virtual and Physical Trumpets in Extended Realities

by Devon John Blewett and David Gerhard

Arts 2023, 12(4), 145; https://doi.org/10.3390/arts12040145 - 10 Jul 2023

Viewed by 1813

Abstract

Despite the benefits of learning an instrument, many students drop out early because it can be frustrating for the student, expensive for the caregiver, and loud for the household. Virtual Reality (VR) and Extended Reality (XR) offer the potential to address these challenges by simulating multiple instruments in an engaging and motivating environment through headphones. To assess the potential for commercial VR to augment musical experiences, we used standard VR implementation processes to design four virtual trumpet interfaces: camera-tracking with tracked register selection (two ways), camera-tracking with voice activation, and a controller plus a force-feedback haptic glove. To evaluate these implementations, we created a virtual music classroom that produces audio, notes, and finger pattern guides loaded from a selected Musical Instrument Digital Interface (MIDI) file. We analytically compared these implementations against physical trumpets (both acoustic and MIDI), considering features of ease of use, familiarity, playability, noise, and versatility. The physical trumpets produced the most reliable and familiar experience, and some XR benefits were considered. The camera-based methods were easy to use but lacked tactile feedback. The haptic glove provided improved tracking accuracy and haptic feedback over camera-based methods. Each method was also considered as a proof-of-concept for other instruments, real or imaginary. Full article

(This article belongs to the Special Issue Feeling the Future—Haptic Audio)

► Show Figures

Figure 1

23 pages, 6366 KiB

Open AccessArticle

Perceptual Relevance of Haptic Feedback during Virtual Plucking, Bowing and Rubbing of Physically-Based Musical Resonators

by Marius George Onofrei, Federico Fontana and Stefania Serafin

Arts 2023, 12(4), 144; https://doi.org/10.3390/arts12040144 - 7 Jul 2023

Viewed by 1220

Abstract

The physics-based design and realization of a digital musical interface asks for the modeling and implementation of the contact-point interaction with the performer. Musical instruments always include a resonator that converts the input energy into sound, meanwhile feeding part of it back to the performer through the same point. Specifically during plucking or bowing interactions, musicians receive a handful of information from the force feedback and vibrations coming from the contact points. This paper focuses on the design and realization of digital music interfaces realizing two physical interactions along with a musically unconventional one, rubbing, rarely encountered in assimilable forms across the centuries on a few instruments. Therefore, it aims to highlight the significance of haptic rendering in improving quality during a musical experience as opposed to interfaces provided with a passive contact point. Current challenges are posed by the specific requirements of the haptic device, as well as the computational effort needed for realizing such interactions without occurrence during the performance of typical digital artifacts such as latency and model instability. Both are however seemingly transitory due to the constant evolution of computer systems for virtual reality and the progressive popularization of haptic interfaces in the sonic interaction design community. In summary, our results speak in favor of adopting nowadays haptic technologies as an essential component for digital musical interfaces affording point-wise contact interactions in the personal performance space. Full article

(This article belongs to the Special Issue Feeling the Future—Haptic Audio)

► Show Figures

Figure 1

25 pages, 6905 KiB

Open AccessArticle

Design and Assessment of Digital Musical Devices Yielding Vibrotactile Feedback

by Stefano Papetti, Hanna Järveläinen and Federico Fontana

Arts 2023, 12(4), 143; https://doi.org/10.3390/arts12040143 - 7 Jul 2023

Viewed by 1379

Abstract

Touch has a pivotal importance in determining the expressivity of musical performance for a number of musical instruments. However, most digital musical devices provide no interactive force and/or vibratory feedback to the performer, thus depriving the somatosensory channel of a number of cues. Is the lack of haptic feedback only an aesthetic issue, or does it remove cues essential for digital instrument playing? If so, at which level is the interaction objectively impoverished? What are the effects on musical performance? In this survey article we illustrate our recent research about the use of vibrotactile feedback in three digital instrument interfaces, using tools that we developed over several years and made available to the community in open-source form. These interfaces span a wide range of familiarity and gestural opportunities, enabling us to explore the impact of haptic feedback on different types of digital instruments. We conducted experiments with professional musicians to assess the impact of vibratory cues on both the perceived quality of the instrument and the playing experience, as well as on musical performance. Particular attention was paid to scientific rigor and repeatability of the results, so as to serve as a reference for researchers and practitioners of the musical haptics community. Our results suggest a significant role of vibrotactile feedback in shaping the perception of digital musical instruments, although the effects on musical performance varied depending on the interfaces tested. Full article

(This article belongs to the Special Issue Feeling the Future—Haptic Audio)

► Show Figures

Figure 1

14 pages, 1518 KiB

Open AccessArticle

Music, Art Installations and Haptic Technology

by Alexandros Kontogeorgakopoulos

Arts 2023, 12(4), 142; https://doi.org/10.3390/arts12040142 - 7 Jul 2023

Viewed by 1634

Abstract

This paper presents some directions on the design, development and creative use of haptic systems for musical composition, performance and digital art creation. This research has been conducted both from an artistic and a technical point of view and its ambition, over the last decade, apart from the artistic outcome, was to introduce the field of haptics to artistic communities based on an open, do it yourself—DIY ethos. The five directions presented here are not in any sense exhaustive and are based principally on a series of collaborative works and more personal open-ended explorations with the medium of haptics and, more specifically, force-feedback interaction. They will be highlighted along with information about the interaction models and their application to artistic works created by the author and other colleagues. Those directions are (i) Haptic Algorithms and Systems; (ii) Performers Intercoupling; (iii) Haptic Interfaces as Part of the Artistic Practice; (iv) Electromechanical Sound Generation; and (v) Media Art and Art Installations. The interdisciplinary field of musical haptics still has a relatively minor position in the sound and music computing research agendas and, more importantly, its artistic dimension is very rarely discussed. The findings of this research aim to indicate and clarify potential research pathways and offer some results on the use of haptics and force-feedback systems in an artistic context. Full article

(This article belongs to the Special Issue Feeling the Future—Haptic Audio)

► Show Figures

Figure 1

13 pages, 1602 KiB

Open AccessReview

Challenges and Opportunities of Force Feedback in Music

by Christian Frisson and Marcelo M. Wanderley

Arts 2023, 12(4), 147; https://doi.org/10.3390/arts12040147 - 10 Jul 2023

Cited by 1 | Viewed by 1463

Abstract

A growing body of work on musical haptics focuses on vibrotactile feedback, while musical applications of force feedback, though more than four decades old, are sparser. This paper reviews related work combining music and haptics, focusing on force feedback. We then discuss the limitations of these works and elicit the main challenges in current applications of force feedback and music (FF&M), which are as follows: modularity; replicability; affordability; and usability. We call for the following opportunities in future research works on FF&M: embedding audio and haptic software into hardware modules, networking multiple modules with distributed control, and authoring with audio-inspired and audio-coupled tools. We illustrate our review with recent efforts to develop an affordable, open-source and self-contained 1-Degree-of-Freedom (DoF) rotary force-feedback device for musical applications, i.e., the TorqueTuner, and to embed audio and haptic processing and authoring in module firmware, with ForceHost, and examine their advantages and drawbacks in light of the opportunities presented in the text. Full article

(This article belongs to the Special Issue Feeling the Future—Haptic Audio)

► Show Figures

Figure 1

24 pages, 2723 KiB

Open AccessStudy Protocol

Feeling Connected: The Role of Haptic Feedback in VR Concerts and the Impact of Haptic Music Players on the Music Listening Experience

by Tara Venkatesan and Qian Janice Wang

Arts 2023, 12(4), 148; https://doi.org/10.3390/arts12040148 - 10 Jul 2023

Cited by 1 | Viewed by 2859

Abstract

Today, some of the most widely attended concerts are in virtual reality (VR). For example, the videogame Fortnite recently attracted 12.3 million viewers sitting in homes all over the world to a VR Travis Scott rap concert. As such VR concerts become increasingly ubiquitous, we are presented with an opportunity to design more immersive virtual experiences by augmenting VR with other multisensory technologies. Given that sound is a multi-modal phenomenon that can be experienced sonically and vibrationally, we investigated the importance of haptic feedback to musical experiences using a combination of qualitative and empirical methodologies. Study 1 was a qualitative study demonstrating that, unlike their live counterparts, current VR concerts make it harder for audiences to form a connection with artists and their music. Furthermore, VR concerts lack multisensory feedback and are perceived as less authentic than live concert experiences. Participants also identified a variety of different kinds of touch that they receive at live concerts and suggested that ideal VR concerts would replicate physical touch and thermal feedback from the audience, emotional touch, and vibrations from the music. Specifically, users advocated for the use of haptic devices to increase the immersiveness of VR concert experiences. Study 2 isolated the role of touch in the music listening experience and empirically investigated the impact of haptic music players (HMPs) on the audio-only listening experience. An empirical, between-subjects study was run with participants either receiving vibrotactile feedback via an HMP (haptics condition) or no vibrotactile feedback (control) while listening to music. Results indicated that listening to music while receiving vibrotactile feedback increased participants’ sense of empathy, parasocial bond, and loyalty towards the artist, while also decreasing participants’ feelings of loneliness. The connection between haptics condition and these dependent variables was mediated by the feeling of social presence. Study 2 thus provides initial evidence that HMPs may be used to meet people’s need for connection, multisensory immersion, and complex forms of touch in VR concerts as identified in Study 1. Full article

(This article belongs to the Special Issue Feeling the Future—Haptic Audio)

► Show Figures

Figure 1

Journal Menu

Journal Browser

Feeling the Future—Haptic Audio

Share This Special Issue

Special Issue Editors

Special Issue Information

Keywords

Published Papers (12 papers)

Further Information

Guidelines

MDPI Initiatives

Follow MDPI