Submit to Electronics Review for Electronics Propose a Special Issue

Journal Browser

Multi-Sensory Interaction for Blind and Visually Impaired People

Print Special Issue Flyer
Special Issue Editors
Special Issue Information
Keywords
Published Papers

A special issue of Electronics (ISSN 2079-9292). This special issue belongs to the section "Computer Science & Engineering".

Deadline for manuscript submissions: closed (31 December 2021) | Viewed by 62227

Printed Edition Available!
A printed edition of this Special Issue is available here.

Special issue Multi-Sensory Interaction for Blind and Visually Impaired People book cover image

Share This Special Issue

Special Issue Editor

Prof. Dr. Jun Dong Cho

E-Mail Website
Guest Editor

Department of Human ICT Convergence, Sungkyunkwan University, Suwon 16419, Korea
Interests: universal access in Human Computer Interaction; assistive technology for people with visual impairment; tactile artworks and interactions; multisensory Color Coding; haptic interface for accessibility; flexible haptic display
Special Issues, Collections and Topics in MDPI journals

Special Issue Information

Dear Colleagues,

Multi-sensory interaction aids learning, inclusion, and collaboration, because it accommodates the diverse cognitive and perceptual needs. Multisensory integration is an essential part of information processing by which various forms of sensory information, such as sight, hearing, touch, and proprioception (also called kinesthesia, the sense of self-movement and body position), are combined into a single experience. Information is typically integrated across sensory modalities when the sensory inputs share certain common features. Cross-modality refers to the interaction between two different sensory channels. Cross-modal correspondence is defined as the surprising associations that people experience between seemingly unrelated features, attributes, or dimensions of experience in different sensory modalities. One application of munti-sensory studies bridging or closing the information gap between people with visual impairments and sighted people. For visually-impaired people, the conventional human computer interaction devices are inconvenient as the devices rely heavily on visual information. Though many studies introduce the use of other modalities of sensation like haptic, sound, and scent for user interface to act as a supplement for the absence of vision, they are still not close to what the vision is to the people. The topics of interest include, but are not limited to the following:

Universal access in Human Computer Interaction
Haptic interface for accessibility
Tactile artworks and interactions
Flexible haptic display
Ambient assistive intelligence
Human-centered user accessibility for people with visual impairment
Assistive technology
Multisensory Color Coding

Prof. Dr. Jun Dong Cho
Guest Editor

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Electronics is an international peer-reviewed open access semimonthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2400 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

Universal access in Human Computer Interaction
Haptic interface for accessibility
Tactile artworks and interactions
Flexible haptic display
Ambient assistive intelligence
Human-centered user accessibility for people with visual impairment
Assistive technology
Multisensory Color Coding
Human-Centered AI

Published Papers (13 papers)

Download All Papers

Editorial

Jump to: Research, Review

9 pages, 206 KiB

Open AccessEditorial

Multi-Sensory Interaction for Blind and Visually Impaired People

by Jun-Dong Cho

Electronics 2021, 10(24), 3170; https://doi.org/10.3390/electronics10243170 - 20 Dec 2021

Viewed by 4285

Abstract

Multi-sensory interaction aids learning, inclusion, and collaboration because it accommodates the diverse cognitive and perceptual needs [...] Full article

(This article belongs to the Special Issue Multi-Sensory Interaction for Blind and Visually Impaired People)

Research

Jump to: Editorial, Review

27 pages, 1365 KiB

Open AccessArticle

Multi-Sensory Color Code Based on Sound and Scent for Visual Art Appreciation

by Luis Cavazos Quero, Chung-Heon Lee and Jun-Dong Cho

Electronics 2021, 10(14), 1696; https://doi.org/10.3390/electronics10141696 - 15 Jul 2021

Cited by 6 | Viewed by 4285

Abstract

The development of assistive technologies is improving the independent access of blind and visually impaired people to visual artworks through non-visual channels. Current single modality tactile and auditory approaches to communicate color contents must compromise between conveying a broad color palette, ease of learning, and suffer from limited expressiveness. In this work, we propose a multi-sensory color code system that uses sound and scent to represent colors. Melodies express each color’s hue and scents the saturated, light, and dark color dimensions for each hue. In collaboration with eighteen participants, we evaluated the color identification rate achieved when using the multi-sensory approach. Seven (39%) of the participants improved their identification rate, five (28%) remained the same, and six (33%) performed worse when compared to an audio-only color code alternative. The participants then evaluated and compared a color content exploration prototype that uses the proposed color code with a tactile graphic equivalent using the System Usability Scale. For a visual artwork color exploration task, the multi-sensory color code integrated prototype received a score of 78.61, while the tactile graphics equivalent received 61.53. User feedback indicates that the multi-sensory color code system improved the convenience and confidence of the participants. Full article

(This article belongs to the Special Issue Multi-Sensory Interaction for Blind and Visually Impaired People)

► Show Figures

Figure 1

20 pages, 2801 KiB

Open AccessArticle

Multi-Sensory Color Expression with Sound and Temperature in Visual Arts Appreciation for People with Visual Impairment

by Jorge Iranzo Bartolome, Gilsang Cho and Jun-Dong Cho

Electronics 2021, 10(11), 1336; https://doi.org/10.3390/electronics10111336 - 02 Jun 2021

Cited by 3 | Viewed by 2685

Abstract

For years the HCI community’s research has been focused on the hearing and sight senses. However, in recent times, there has been an increased interest in using other types of senses, such as smell or touch. Moreover, this has been accompanied with growing research related to sensory substitution techniques and multi-sensory systems. Similarly, contemporary art has also been influenced by this trend and the number of artists interested in creating novel multi-sensory works of art has increased substantially. As a result, the opportunities for visually impaired people to experience artworks in different ways are also expanding. In spite of all this, the research focusing on multimodal systems for experiencing visual arts is not large and user tests comparing different modalities and senses, particularly in the field of art, are insufficient. This paper attempts to design a multi-sensory mapping to convey color to visually impaired people employing musical sounds and temperature cues. Through user tests and surveys with a total of 18 participants, we show that this multi-sensory system is properly designed to allow the user to distinguish and experience a total of 24 colors. The tests consist of several semantic correlational adjective-based surveys for comparing the different modalities to find out the best way to express colors through musical sounds and temperature cues based on previously well-established sound-color and temperature-color coding algorithms. In addition, the resulting final algorithm is also tested with 12 more users. Full article

(This article belongs to the Special Issue Multi-Sensory Interaction for Blind and Visually Impaired People)

► Show Figures

Figure 1

21 pages, 8313 KiB

Open AccessArticle

Construction of a Soundscape-Based Media Art Exhibition to Improve User Appreciation Experience by Using Deep Neural Networks

by Youngjun Kim, Hayoung Jeong, Jun-Dong Cho and Jitae Shin

Electronics 2021, 10(10), 1170; https://doi.org/10.3390/electronics10101170 - 14 May 2021

Cited by 11 | Viewed by 3019

Abstract

The objective of this study was to improve user experience when appreciating visual artworks with soundscape music chosen by a deep neural network based on weakly supervised learning. We also propose a multi-faceted approach to measuring ambiguous concepts, such as the subjective fitness, implicit senses, immersion, and availability. We showed improvements in appreciation experience, such as the metaphorical and psychological transferability, time distortion, and cognitive absorption, with in-depth experiments involving 70 participants. Our test results were similar to those of “Bunker de Lumières: van Gogh”, which is an immersive media artwork directed by Gianfranco lannuzzi; the fitness scores of our system and “Bunker de Lumières: van Gogh” were 3.68/5 and 3.81/5, respectively. Moreover, the concordance of implicit senses between artworks and classical music was measured to be 0.88%, and the time distortion and cognitive absorption improved during the immersion. Finally, the proposed method obtained a subjective satisfaction score of 3.53/5 in the evaluation of its usability. Our proposed method can also help spread soundscape-based media art by supporting traditional soundscape design. Furthermore, we hope that our proposed method will help people with visual impairments to appreciate artworks through its application to a multi-modal media art guide platform. Full article

(This article belongs to the Special Issue Multi-Sensory Interaction for Blind and Visually Impaired People)

► Show Figures

Figure 1

24 pages, 11556 KiB

Open AccessArticle

ColorPoetry: Multi-Sensory Experience of Color with Poetry in Visual Arts Appreciation of Persons with Visual Impairment

by Jun-Dong Cho and Yong Lee

Electronics 2021, 10(9), 1064; https://doi.org/10.3390/electronics10091064 - 30 Apr 2021

Cited by 4 | Viewed by 3707

Abstract

Visually impaired visitors experience many limitations when visiting museum exhibits, such as a lack of cognitive and sensory access to exhibits or replicas. Contemporary art is evolving in the direction of appreciation beyond simply looking at works, and the development of various sensory technologies has had a great influence on culture and art. Thus, opportunities for people with visual impairments to appreciate visual artworks through various senses such as hearing, touch, and smell are expanding. However, it is uncommon to provide a multi-sensory interactive interface for color recognition, such as integrating patterns, sounds, temperature, and scents. This paper attempts to convey a color cognition to the visually impaired, taking advantage of multisensory coding color. In our previous works, musical melodies with different combinations of pitch, timbre, velocity, and tempo were used to distinguish vivid (i.e., saturated), light, and dark colors. However, it was rather difficult to distinguish among warm/cool/light/dark colors with using sound cues only. Therefore, in this paper, we aim to build a multisensory color-coding system with combining sound and poem such that poem leads to represent more color dimensions, such as including warm and cool colors for red, orange, yellow, green, blue, and purple. To do this, we first performed an implicit association test to identify the most suitable poem among the candidate poems to represent colors in artwork by finding the common semantic directivity between the given candidate poem with voice modulation and the artwork in terms of light/dark/warm/color dimensions. Finally, we conducted a system usability test on the proposed color-coding system, confirming that poem will be an effective supplement for distinguishing between vivid, light, and dark colors with different color appearance dimensions, such as warm and cold colors. The user experience score of 15 college students was 75.1%, that was comparable with the color-music coding system that received a user experience rating of 74.1%. with proven usability. Full article

(This article belongs to the Special Issue Multi-Sensory Interaction for Blind and Visually Impaired People)

► Show Figures

Figure 1

22 pages, 2836 KiB

Open AccessArticle

3D Sound Coding Color for the Visually Impaired

by Yong Lee, Chung-Heon Lee and Jun Dong Cho

Electronics 2021, 10(9), 1037; https://doi.org/10.3390/electronics10091037 - 27 Apr 2021

Cited by 4 | Viewed by 2820

Abstract

Contemporary art is evolving beyond simply looking at works, and the development of various sensory technologies has had a great influence on culture and art. Accordingly, opportunities for the visually impaired to appreciate visual artworks through various senses such as auditory and tactile senses are expanding. However, insufficient sound expression and lack of portability make it less understandable and accessible. This paper attempts to convey a color and depth coding scheme to the visually impaired, based on alternative sensory modalities, such as hearing (by encoding the color and depth information with 3D sounds of audio description) and touch (to be used for interface-triggering information such as color and depth). The proposed color-coding scheme represents light, saturated, and dark colors for red, orange, yellow, yellow-green, green, blue-green, blue, and purple. The paper’s proposed system can be used for both mobile platforms and 2.5D (relief) models. Full article

(This article belongs to the Special Issue Multi-Sensory Interaction for Blind and Visually Impaired People)

► Show Figures

Figure 1

23 pages, 1332 KiB

Open AccessArticle

Auditory Uta-Karuta: Development and Evaluation of an Accessible Card Game System Using Audible Cards for the Visually Impaired

by Haruna Miyakawa, Noko Kuratomo, Hisham E. Bilal Salih and Keiichi Zempo

Electronics 2021, 10(6), 750; https://doi.org/10.3390/electronics10060750 - 22 Mar 2021

Cited by 4 | Viewed by 3576

Abstract

Playing board games is important for people with a visually impairment, as it promotes interactive socialization and communication skills. However, some board games are not accessible to them at present. In this study, we proposed an auditory card game system that presents a card’s contents with auditory stimuli to all players, towards playing equally with others, regardless of whether they have a visual impairment or not as one of the solutions to make board games accessible. This proposal contributes significantly to expand the range of inclusive board games for the visually impaired. The purpose of this paper is to determine whether the game allows for fair competition for people with visual impairments and to clarify the effects of the valuable parameters of the system on the players. The effectiveness of the proposed system was verified by having experimental participants play “Auditory Uta-Karuta”. The results suggested that the proposed system has the potential for an accessible board game design regardless of visual impairment. In the following experiment, we investigated the impact of each valuable parameter of the system on the player’s perception of the board games to clarify the appropriate audio cue design method. The results of this experiment will greatly assist in designing an appropriate board game using the proposed system. Full article

(This article belongs to the Special Issue Multi-Sensory Interaction for Blind and Visually Impaired People)

► Show Figures

Figure 1

17 pages, 2833 KiB

Open AccessArticle

ColorWatch: Color Perceptual Spatial Tactile Interface for People with Visual Impairments

by Muhammad Shahid Jabbar, Chung-Heon Lee and Jun Dong Cho

Electronics 2021, 10(5), 596; https://doi.org/10.3390/electronics10050596 - 04 Mar 2021

Cited by 5 | Viewed by 2790

Abstract

Tactile perception enables people with visual impairments (PVI) to engage with artworks and real-life objects at a deeper abstraction level. The development of tactile and multi-sensory assistive technologies has expanded their opportunities to appreciate visual arts. We have developed a tactile interface based on the proposed concept design under considerations of PVI tactile actuation, color perception, and learnability. The proposed interface automatically translates reference colors into spatial tactile patterns. A range of achromatic colors and six prominent basic colors with three levels of chroma and values are considered for the cross-modular association. In addition, an analog tactile color watch design has been proposed. This scheme enables PVI to explore artwork or real-life object color by identifying the reference colors through a color sensor and translating them to the tactile interface. The color identification tests using this scheme on the developed prototype exhibit good recognition accuracy. The workload assessment and usability evaluation for PVI demonstrate promising results. This suggest that the proposed scheme is appropriate for tactile color exploration. Full article

(This article belongs to the Special Issue Multi-Sensory Interaction for Blind and Visually Impaired People)

► Show Figures

Figure 1

19 pages, 5255 KiB

Open AccessArticle

Accessible Visual Artworks for Blind and Visually Impaired People: Comparing a Multimodal Approach with Tactile Graphics

by Luis Cavazos Quero, Jorge Iranzo Bartolomé and Jundong Cho

Electronics 2021, 10(3), 297; https://doi.org/10.3390/electronics10030297 - 26 Jan 2021

Cited by 27 | Viewed by 8015

Abstract

Despite the use of tactile graphics and audio guides, blind and visually impaired people still face challenges to experience and understand visual artworks independently at art exhibitions. Art museums and other art places are increasingly exploring the use of interactive guides to make their collections more accessible. In this work, we describe our approach to an interactive multimodal guide prototype that uses audio and tactile modalities to improve the autonomous access to information and experience of visual artworks. The prototype is composed of a touch-sensitive 2.5D artwork relief model that can be freely explored by touch. Users can access localized verbal descriptions and audio by performing touch gestures on the surface while listening to themed background music along. We present the design requirements derived from a formative study realized with the help of eight blind and visually impaired participants, art museum and gallery staff, and artists. We extended the formative study by organizing two accessible art exhibitions. There, eighteen participants evaluated and compared multimodal and tactile graphic accessible exhibits. Results from a usability survey indicate that our multimodal approach is simple, easy to use, and improves confidence and independence when exploring visual artworks. Full article

(This article belongs to the Special Issue Multi-Sensory Interaction for Blind and Visually Impaired People)

► Show Figures

Figure 1

18 pages, 8406 KiB

Open AccessFeature PaperArticle

Sound Coding Color to Improve Artwork Appreciation by People with Visual Impairments

by Jun Dong Cho, Jaeho Jeong, Ji Hye Kim and Hoonsuk Lee

Electronics 2020, 9(11), 1981; https://doi.org/10.3390/electronics9111981 - 23 Nov 2020

Cited by 14 | Viewed by 3690

Abstract

The recent development of color coding in tactile pictograms helps people with visual impairments (PVI) appreciate the visual arts. The auditory sense, in conjunction with (or possibly as an alternative to) the tactile sense, would allow PVI to perceive colors in a way that would be difficult to achieve with just a tactile stimulus. Sound coding colors (SCCs) can replicate three characteristics of colors, i.e., hue, chroma, and value, by matching them with three characteristics of sound, i.e., timbre, intensity, and pitch. This paper examines relationships between sound (melody) and color mediated by tactile pattern color coding and provides sound coding for hue, chroma, and value to help PVI deepen their relationship with visual art. Our two proposed SCC sets use melody to improve upon most SCC sets currently in use by adding more colors (18 colors in 6 hues). User experience and identification tests were conducted with 12 visually impaired and 8 sighted adults, and the results suggest that the SCC sets were helpful for the participants. Full article

(This article belongs to the Special Issue Multi-Sensory Interaction for Blind and Visually Impaired People)

► Show Figures

Figure 1

26 pages, 6481 KiB

Open AccessFeature PaperArticle

Thermal Interaction for Improving Tactile Artwork Depth and Color-Depth Appreciation for Visually Impaired People

by Jorge Iranzo Bartolomé, Jun Dong Cho, Luis Cavazos Quero, Sunggi Jo and Gilsang Cho

Electronics 2020, 9(11), 1939; https://doi.org/10.3390/electronics9111939 - 17 Nov 2020

Cited by 11 | Viewed by 2886

Abstract

Visually impaired people can take advantage of multimodal systems in which visual information is communicated through different modes of interaction and types of feedback. Among the possible interaction modes, thermal interaction in the context of assistive devices for visually impaired people lacks research in spite of its potential. In this paper, we propose a temperature-depth mapping algorithm and a thermal display system to convey depth and depth-color of artworks’ features in the context of tactile exploration by visually impaired people. Tests with a total of 18 sighted users and six visually impaired users were performed both during the mapping algorithm design and after developing a tactile temperature prototype artwork model to assess the potentials of thermal interaction for recognizing depth and color-depth in tactile art appreciation. These tests showed both an existing correlation between depth and temperature and that the mapping based on that correlation is appropriate for conveying depth during artwork tactile exploration. Full article

(This article belongs to the Special Issue Multi-Sensory Interaction for Blind and Visually Impaired People)

► Show Figures

Figure 1

Review

Jump to: Editorial, Research

17 pages, 14704 KiB

Open AccessReview

Image Accessibility for Screen Reader Users: A Systematic Review and a Road Map

by Uran Oh, Hwayeon Joh and YunJung Lee

Electronics 2021, 10(8), 953; https://doi.org/10.3390/electronics10080953 - 16 Apr 2021

Cited by 13 | Viewed by 3738

Abstract

A number of studies have been conducted to improve the accessibility of images using touchscreen devices for screen reader users. In this study, we conducted a systematic review of 33 papers to get a holistic understanding of existing approaches and to suggest a research road map given identified gaps. As a result, we identified types of images, visual information, input device and feedback modalities that were studied for improving image accessibility using touchscreen devices. Findings also revealed that there is little study how the generation of image-related information can be automated. Moreover, we confirmed that the involvement of screen reader users is mostly limited to evaluations, while input from target users during the design process is particularly important for the development of assistive technologies. Then we introduce two of our recent studies on the accessibility of artwork and comics, AccessArt and AccessComics, respectively. Based on the identified key challenges, we suggest a research agenda for improving image accessibility for screen reader users. Full article

(This article belongs to the Special Issue Multi-Sensory Interaction for Blind and Visually Impaired People)

► Show Figures