**1. Introduction**

Multi-sensory systems, or systems that combine several modes of interaction through different senses, have been gaining popularity in recent times in the field of HCI (Human– Computer Interaction). Each one of the five senses gives humans unique ways of experiencing the world around and, as a result, researchers have seen the potential of applying those natural ways of interacting with the world into different ways of interacting with computers. In addition, multi-sensory systems can also be useful to convey complex or abstract information since each sense can congruently aid the others. Moreover, recent research has shown that humans are used to interacting with the world in a multi-sensory way and that multi-sensory experiences tend to be more engaging and easier to remember when compared to unisensory ones [1].

One of the possible applications for multi-sensory systems is the creation of novel art which creates a novel experience for the spectator by means of its multi-sensory interfaces. Because of that, artists have been showing a growing interest in multi-sensory art exhibitions. This increase of multi-sensory experiences, which makes interaction drift from the common visual and auditory types to other types of interaction, can be really convenient for minority groups that lack one of those senses, such as the visually impaired people. People with visual impairments are interested not only in performing with ease the necessary activities of everyday life but also in visiting museums and enjoying visual arts,

**Citation:** Bartolome, J.I.; Cho, G.; Cho, J.-D. Multi-Sensory Color Expression with Sound and Temperature in Visual Arts Appreciation for People with Visual Impairment. *Electronics* **2021**, *10*, 1336.https://doi.org/10.3390/ electronics10111336

Academic Editor: George A. Tsihrintzis

Received: 3 March 2021 Accepted: 28 May 2021 Published: 2 June 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

which is why many museums have been making some of their exhibitions and artworks accessible to visually impaired people by hosting specialized tours with access to tactile representations of the artworks [2–4]. Nevertheless, most of those tours and exhibitions are based solely on tactile and audio systems which do not make use of any other sensory modality for interacting with the artworks. Since visually impaired people lack only the sense of vision, we believe they could benefit greatly from multi-sensory experiences as they provide them not only with the possibility of experiencing the artwork without relying on single sense that they cannot have access to, but also with the increased engagement, gamification and artwork comprehension which multi-sensory experiences can facilitate.

In this work, we design a novel sound-temperature multi-sensory system that helps visually impaired people experience a total of 24 colors. Additionally, during the process, a throughout investigation about the relationship of the different sounds and temperatures with the warm/cold and bright/dark dimensions of colors was performed. This investigation included extensive surveys and tests with a total of 18 participants, and an additional 12 participants for the final multimodal test. The work is the continuation of a series of works whose aim is to develop a multi-modal system and algorithms that aid visually impaired people in experiencing pieces of art. In previous works, both a color-temperature mapping algorithm and a sound-color coding system were designed, implemented into physical prototypes, and tested [5–7]. This work introduces a method for finding the best way to turn those unisensory systems into an improved sound-temperature multi-sensory system for conveying colors. Both works will be explained in more detail in the following sections.

### **2. Background and Related Works**

### *2.1. Multi-Sensory Systems*

In recent times, multi-sensory systems have gained popularity among researchers thanks to strong evidence that multi-sensory experiences have some benefits over unisensory ones, especially when it pertains to experiences related to education and learning. Shams et al. [1] state that multi-sensory experiences aid in memorization by helping the brain retain the information faster and for longer times. Moreover, the authors argue that the human brain evolved to develop, adapt, learn and function optimally in multi-sensory environments. However, they also maintain that congruence between the multi-sensory cues seems to be a condition for the benefits to be optimum.

Sensory-substitution is the technique of representing the characteristics of one sensory modality into another sensory modality. The technique is strongly related to assistive multisensory systems for the visually impaired people since the main goal of those systems is to communicate through a different sense the information that cannot be acquired by the visually impaired user because of the lack of sight. The authors of [8] provide a tutorial guide on how to research sensory substitution techniques and multi-sensory systems in order to successfully design inclusive cross-modal displays. They state the three main guiding principles necessary for a multi-sensory process to occur are spatial coincidence, temporal coincidence, and inverse effectiveness. Our work takes care of following all those principles. Spatial and temporal coincidence is assured as long as the sound and the temperature are communicated to the user at the same time or in immediate succession. The inverse effectiveness guideline states that for multi-sensory information to be effective, none of the sensory modalities needs to overpower the other. Since both the musical sounds and the temperature cue are semantically congruen<sup>t</sup> to what they represent (as will be proved during the tests in later sections) and offer the user different information about the color, there is no weak/strong bias towards any of them and both sensory modalities work united as one.
